akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Combine two resnet18 models

username: vojkan7
cooked: <p>Hi,<br>
I already tried and also looked for other posts on the forum but I don’t get it.<br>
I have two models and I want to combine them.<br>
can anyone give a quick example?</p>
<pre><code class="lang-auto">class ResNet18Server(nn.Module):

    def __init__(self, config):
        super(ResNet18Server, self).__init__()
        self.logits = config["logits"]
        self.cut_layer = config["cut_layer"]

        self.model = models.resnet18(pretrained=False)
        num_ftrs = self.model.fc.in_features

        self.model.fc = nn.Sequential(nn.Flatten(),
                                      nn.Linear(num_ftrs, self.logits))

        self.model = nn.ModuleList(self.model.children())
        self.model = nn.Sequential(*self.model)

    def forward(self, x):
        for i, l in enumerate(self.model):
            if i &lt;=self.cut_layer:
                continue
            x = l(x)
            
        return x

</code></pre>

username: ptrblck
cooked: <p>Could you explain what “combine” would mean in this context?</p>

username: vojkan7
cooked: <p>Hi,<br>
I want to merge them. They are trained on different datasets and I want one model that have parameters/weights from both. I mean that new model cover both models</p>

username: InnovArul
cooked: <aside class="quote no-group" data-username="vojkan7" data-post="3" data-topic="151831">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vojkan7/48/44870_2.png" class="avatar"> vojkan7:</div>
<blockquote>
<p>I want one model that have parameters/weights from both</p>
</blockquote>
</aside>
<p>AFAIK, combining models does not work that way.</p>
<p>One of the ways to “combine” is to perform ensembling of these models as explained below:</p>
<p>Have a two-branch architecture with these 2 resnet18 models and get features from each branch (any layer of choice). Then combine these features (for example, concatenate these features) and to pass it through an MLP to predict the target variables. Finetune this setup (either by freezing the backbone models or having less Learning rate for them) in the target task.</p>
How to store gradient of entire model using model.named_parameters()? As extarcting model.named_parameters() at the end of training returns 0

username: Pavithree_shetty
cooked: <p>I am trying to store the gradients of the entire model. The code is given below:</p>
<pre><code>  for step, batch in enumerate(train_dataloader): 
    outputs = model(**batch)
    loss = outputs.loss
    loss = loss / args.gradient_accumulation_steps
    accelerator.backward(loss)
    progress_bar.update(1)
    progress_bar.set_postfix(loss=round(loss.item(), 3))
    del outputs
    gc.collect()
    torch.cuda.empty_cache()
    
    if (step+1) % args.gradient_accumulation_steps == 0 or (step+1) == len(train_dataloader):
        optimizer.step()
        scheduler.step()
        optimizer.zero_grad()
 reference_gradient = [ p.grad.view(-1) if p.grad is not None else torch.zeros(p.numel()) for n, p in 
 model.named_parameters()]
 reference_gradient = torch.cat(reference_gradient)
</code></pre>
<p>My intension is to store the model parameters of entire model to used it for further calculation in another model. But I have 2 questions here,</p>
<ol>
<li>Here the reference_gradient variable always returns 0,  I understand that this happens because, optimizer.zero_grad() is called after every gradient.accumulation steps, and all the gradients are set to 0.</li>
<li>How can I store the model parameters of the entire model?</li>
</ol>

username: InnovArul
cooked: <aside class="quote no-group" data-username="Pavithree_shetty" data-post="1" data-topic="151859">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pavithree_shetty/48/49675_2.png" class="avatar"> Pavithree_shetty:</div>
<blockquote>
<p>How can I store the model parameters of the entire model?</p>
</blockquote>
</aside>
<p>You can have a look at pytorch docs:</p>
<p><a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" class="onebox" target="_blank" rel="noopener">https://pytorch.org/tutorials/beginner/saving_loading_models.html</a></p>
<p>In a nutshell, you can use:</p>
<p><strong>Save:</strong></p>
<pre><code class="lang-auto">torch.save(model.state_dict(), PATH)
</code></pre>
<p><strong>Load:</strong></p>
<pre><code class="lang-auto">model = TheModelClass(*args, **kwargs) 
model.load_state_dict(torch.load(PATH)) 
</code></pre>
Train/test/dev splits with torchdata

username: aliutkus
cooked: <p>hi there, I’m playing with torchdata which I find quite nice, but I’m wondering: is there some convenient receipe you may know for doing train/dev/test random splits in there ?</p>
<p>My datapipe starts with some <code>dp.iter.FileLister</code> class, and so maybe there’s a way to easily split that one in three in a reproductible manner for the splits ? A random split would do fine</p>
<p>thanks for your help</p>
Value Error :Cannot take sample than population when 'replace=false'

username: asha97
cooked: <p>Following is the error trace:<br>
File “main.py”, line 267, in main<br>
_ = train(iteration,optimizer)<br>
File “main.py”, line 134, in train<br>
for j, (img,fix,img_seq,target) in enumerate(trainloader):<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 345, in <strong>next</strong><br>
data = self._next_data()<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 385, in _next_data<br>
data = self._dataset_fetcher.fetch(index)  # may raise StopIteration<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in fetch<br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/DATA/rani.1/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in <br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “./util/data_processing.py”, line 275, in <strong>getitem</strong><br>
for i,idx in enumerate(np.random.choice(np.arange(len(pt_img)),self.pt_len,replace=False)):<br>
File “mtrand.pyx”, line 1168, in mtrand.RandomState.choice<br>
ValueError: Cannot take a larger sample than population when ‘replace=False’</p>

username: ptrblck
cooked: <p>Based on the stack trace it seems you are using a custom <code>Dataset</code>, which seems to be defined in <code>./util/data_processing.py</code>. Inside the <code>__getitem__</code> method you are apparently using <code>np.random.choice</code>, which get’s an invalid argument combination.<br>
It seems that <code>len(pt_img)</code> might be smaller than <code>self.pt_len</code> which could yield a similar issue as this code snippet:</p>
<pre><code class="lang-python">np.random.choice(np.arange(10), 10, replace=False) # works
np.random.choice(np.arange(9), 10, replace=False) # fails
</code></pre>

username: Alva
cooked: <p>Thanks for your patient explanation. But if we encounter this problem, how to solve it?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>As seen in the previous example you won’t be able to sample e.g. 10 values out of 9 without replacement.<br>
Either use <code>replace=True</code> or make sure you are not trying to draw more samples than are available.</p>
Converting images to Pytorch tensors loses label data

username: Onur_Ozbek
cooked: <p>I have a dataset of images. In the below code, I am trying to convert them to Pytorch tensors by first converting them to PIL images:</p>
<pre><code class="lang-auto"># choose the training and test datasets
train_data = os.listdir('data/training/')
testing_data = os.listdir('data/testing/')
train_tensors = []
test_tensors = []

# Print out some stats about the training and test data
print('Train data, number of images: ', len(train_data))
print('Test data, number of images: ', len(testing_data))

# The transformation call to resize images and transform them into Tensors
transform = transforms.Compose([
    transforms.RandomResizedCrop((120,120)),
    transforms.PILToTensor()
])

# Converting every train/test image to a PIL image and then to a Pytorch tensor
for train_image in train_data:
    img = Image.open('data/training/' + train_image)
    train_tensors.append(transform(img))

for test_image in testing_data:
    img = Image.open('data/testing/' + test_image)
    test_tensors.append(transform(img))
</code></pre>
<p>However, in this process, the labels are completely lost. This is the output of <code>train_tensors</code></p>
<pre><code class="lang-auto"> [255, 255, 255,  ..., 255, 255, 255],
         [255, 255, 255,  ..., 255, 255, 255],
         [255, 255, 255,  ..., 255, 255, 255]],

        [[255, 255, 255,  ..., 255, 255, 255],
         [255, 255, 255,  ..., 255, 255, 255],
         [255, 255, 255,  ..., 255, 255, 255],
         ...,
         [255, 255, 255,  ..., 255, 255, 255],
         [255, 255, 255,  ..., 255, 255, 255],
         [255, 255, 255,  ..., 255, 255, 255]],

        [[255, 255, 255,  ..., 255, 255, 255],
         [255, 255, 255,  ..., 255, 255, 255],
         [255, 255, 255,  ..., 255, 254, 254],
         ...,
         [254, 254, 255,  ..., 254, 255, 255],
         [255, 255, 255,  ..., 255, 255, 255],
         [255, 255, 255,  ..., 255, 255, 255]],

        [[255, 255, 255,  ..., 255, 255, 255],
         [255, 255, 255,  ..., 255, 255, 255],
         [255, 255, 255,  ..., 255, 255, 255],
         ...,
         [255, 255, 255,  ..., 255, 255, 255],
         [255, 255, 255,  ..., 255, 255, 255],
         [255, 255, 255,  ..., 255, 255, 255]]], dtype=torch.uint8)
</code></pre>
<p>When I use this tensor in a dataloader and try to extract the labels, I get a <code>too many values to unpack</code> error.</p>
<pre><code class="lang-auto">train_loader = DataLoader(train_tensors, batch_size=batch_size, shuffle=True)

dataiter = iter(train_loader)
images, labels = dataiter.__next__() #
</code></pre>
<p>How can I maintain my label data?</p>

username: ptrblck
cooked: <p>I don’t fully understand your question as it doesn’t seem you are loading labels at all.<br>
In your currently posted code snippet you are loading images and transforming them to tensors, which seems to work fine.<br>
The <code>images, labels = dataiter.__next__()</code> operation fails as the <code>DataLoader</code> will only return the passed <code>train_tensors</code>, which is a list of input tensors without any labels.</p>
[ATen] Change tensor ordering to column-major to use standard CUDA functions

username: spaceinvader
cooked: <p>Hi!<br>
I’m using ATen and cpp-extensions with PyTorch to work with sparse matrices on GPU.<br>
But I have faced the problem: CUDA functions assume matrices to have fortran-style ordering, when PyTorch (ATen) tensors are stored in C-style.<br>
Could you tell me how can I change PyTorch tensor ordering?</p>

username: richard
cooked: <p>How exactly does changing a tensor to column-major help sparse matrices on GPU?</p>
<p>I think you could transpose a matrix: <code>matrix.t().contiguous()</code> and that would change the matrix to column-major, but I’m not sure that’s what you want.</p>

username: eduardo4jesus
cooked: <p>Sorry for resurrecting this old question. But I have a similar one.</p>
<p>Suppose I have a tensor <code>x</code> in the shape <code>B, C, H, W</code>. What is better?</p>
<p><code>x.permute({2, 3, 0, 1}).clone()</code></p>
<p>or</p>
<p><code>x.t().contiguous()</code></p>
<p>?</p>

username: ptrblck
cooked: <p><code>x.t()</code> won’t work on a tensor with 4 dimensions, as <code>&lt;=2</code> dims are expected so you should remove the <code>{}</code> and use the first approach:</p>
<pre><code class="lang-python">x.permute(2, 3, 0, 1).contiguous()
</code></pre>

username: eduardo4jesus
cooked: <p>Thank you for pointing that out.</p>
<p>I tried to use the code without the <code>{}</code> and it throw some compilation errors:</p>
<p><code>error: too many arguments in function call</code></p>
<p>That is because I am writing in C++, for which the ATen documentation expects an <code>IntArrayRef</code></p>
<blockquote>
<p><code>at::Tensor at::permute(const at::Tensor &amp;self, at::IntArrayRef dims)</code></p>
</blockquote>
<hr>
<p>Now I have this additional question:</p>
<ul>
<li>Is there any advantage between using <code>clone()</code> or <code>contiguous()</code>? Would they be equivalent performance-wise in this case?</li>
</ul>
<p>Many thanks!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="eduardo4jesus" data-post="5" data-topic="17666">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eduardo4jesus/48/20203_2.png" class="avatar"> eduardo4jesus:</div>
<blockquote>
<p>That is because I am writing in C++</p>
</blockquote>
</aside>
<p>Ah OK, I didn’t realize this.</p>
<aside class="quote no-group" data-username="eduardo4jesus" data-post="5" data-topic="17666">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eduardo4jesus/48/20203_2.png" class="avatar"> eduardo4jesus:</div>
<blockquote>
<p>Is there any advantage between using <code>clone()</code> or <code>contiguous()</code>? Would they be equivalent performance-wise in this case?</p>
</blockquote>
</aside>
<p><code>.clone()</code> will just create a copy of the tensor and will not make it contiguous, while <code>.contiguous()</code> will make create a contiguous (in memory) copy of the tensor.</p>
Autocast not working in huggingface nlp models

username: Maxwell_Albert
cooked: <p>Hi I am using roberta-base models from huggingface finetuning RTE dataset with fp16.<br>
I am using torch.cuda.amp.autocast() but the output of the model is still <code>torch.float32</code> and there is no memory saving at all.<br>
Here is the code</p>
<pre><code class="lang-auto">model = AutoModelForSequenceClassification.from_pretrained("roberta-base")
model = model.to(rank)
model = torch.nn.parallel.DistributedDataParallel(model)
for epoch in range(epochs):
    model.train()
    for i, batch in enumerate(train_dataloader):
        with autocast():
            outputs = model(batch["input_ids"], batch["attention_mask"])
           loss = outputs.loss
           ...
</code></pre>
<p>If I change my model to model.half(), the output is fp16 but will get nan gradient. I tried to use scaler, but scaler does not support fp16 gradient.<br>
How could I train hugginface nlp models in fp16?</p>

username: ptrblck
cooked: <p>I cannot reproduce the issue and get a memory saving using <code>amp</code> and this code:</p>
<pre><code class="lang-python">import torch
from transformers import AutoModelForSequenceClassification
from transformers import RobertaTokenizer


tokenizer = RobertaTokenizer.from_pretrained('roberta-base')

model = AutoModelForSequenceClassification.from_pretrained("roberta-base")
model = model.to('cuda:0')

text = ["Replace me by any text you'd like."]*1024
encoded_input = tokenizer(text, return_tensors='pt')
encoded_input  = {k: encoded_input[k].to('cuda') for k in encoded_input}

with torch.cuda.amp.autocast(enabled=True):
    outputs = model(**encoded_input)
    loss = outputs.loss

print(torch.cuda.memory_allocated()/1024**2)
# AMP: 5147.49072265625
# FP32: 7547.36669921875
</code></pre>

username: Maxwell_Albert
cooked: <p>Thank you very much!<br>
I saw the memory saving, too.<br>
It was my mistake.</p>
RuntimeError: expected dtype Double but got dtype Long

username: jabberwoo
cooked: <p>torch version: 1.4.0+cu100</p>
<p>python version: 3.6</p>
<p>Whlile running the code, encountered the error:</p>
<p><code>RuntimeError: expected dtype Double but got dtype Long</code> when loss.backward() was executed.</p>
<pre><code class="lang-auto">def train(model, train_data, train_target, len_, criterion):
   
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    model.to(device)
    train_data = torch.tensor(train_data, device=device)
    train_target = torch.tensor(train_target, device=device)
    model.double()
    train_data.double()
    train_target.double()
    model.train()
    learning_rate = 0.01

    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

    epochs = 2
    for epoch in range(epochs):
        print("epoch :", epoch)
        # forward pass and loss
        y_predicted = model(train_data)
        y_predicted.double()
        loss = criterion(y_predicted, train_target)

        # backward pass
        loss.double()
        loss.backward()

        # update
        optimizer.step()

        # init optimizer
        optimizer.zero_grad()
    return 
</code></pre>
<p>I have tried my best, but still got the same error. How to solve this error?</p>

username: ptrblck
cooked: <p>Could you update PyTorch to the latest stable version?<br>
In case you are still hitting this error, could you add the missing pieces to the code to make it executable so that we could try to reproduce the issue, please?</p>
Handling input not multiple of batch size while training

username: sandeep1
cooked: <p>How does loss is computed incase there is less number of instance say(5) in a batch of size say(10)?<br>
What does trainloader manages it? How only the loss of only last 5 instance is calculated while loss computed? And what is the standard way of handling this?</p>
<p>Thanks in advance.</p>
<pre><code class="lang-auto"> for idx,data in tqdm(enumerate(trainloader),desc="Train epoch {}/{}".format(epoch + 1, EPOCH)):

    ids = data['ids_sen'].to(device,dtype = torch.long)
    mask = data['mask_sen'].to(device,dtype = torch.long)
    token_type_ids = data['token_type_ids_sen'].to(device,dtype = torch.long)
    targets = data['targets'].to(device,dtype = torch.float)

    t1 = (ids,mask,token_type_ids)
    optimizer.zero_grad()
    out, attn_t = text_model(t1,'last')

    if (epoch+1 == EPOCH):
      train_out.append((torch.transpose(out,0,1)).detach().cpu())
      train_true.append((torch.transpose(targets,0,1)).detach().cpu())

    loss = criterion1(out, targets)
    loss.backward()
    optimizer.step()
    if idx % 100 == 0:
      scheduler.step()

    
  loss_log1.append(np.average(l1))
</code></pre>

username: AbdulsalamBande
cooked: <p>If I understood right, you mean how is loss computed when the batch size is 5 when  the batch size is always 10 for the previous data. For  batch size of 5, while computing the loss. The average of  the 5 is computed instead of 10. So it is <code>total_loss/5</code></p>
Incorrect labels

username: goksu
cooked: <p>Hi everyone!</p>
<p>I have a problem with dataloader in pytorch. I use dataloader to get data and my input dataset is CIFAR10. I get images and labels with next().iter(), but when I check the labels are incorrect. For example; the image is a cat, but its label says it is a truck. How can I fix this? If you can help me, I would be really happy <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Cheers!</p>

username: Sairam954
cooked: <p>Can you share the code you use to for dataloader and iteration of labels and images?<br>
I hope this tutorial on the CIFAR10 helps you</p>
<p><a href="https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/cifar10_tutorial.ipynb#scrollTo=aCWWR6R9hEZu" rel="noopener nofollow ugc">CIFAR10 Tutorial</a></p>

username: goksu
cooked: <p>Thank you very much, the link was really helpful.</p>
<p>Cheers!</p>
'Conv2dU' object has no attribute '_backward_hooks'

username: Sairam954
cooked: <p>Hi,</p>
<p>I tired to define a custom conv2D class, but I am getting the this object has no attribute ‘_backward_hooks’. I would greatly appreciated if someone can explain the cause for this error.</p>
<p>Thanks</p>

username: AlphaBetaGamma96
cooked: <p>Does your custom Conv2D class inherit from <code>nn.Module</code> or <code>torch.autograd.Function</code>?</p>

username: Sairam954
cooked: <p>I inherit <code>nn.Module</code> and use <span class="mention">@autocast</span> from <code>torch.cuda.amp import autocast</code> for forward method</p>

username: AlphaBetaGamma96
cooked: <p>Would you mind sharing the source code for it? It’s a bit hard to visualize the problem without the source.</p>

username: Sairam954
cooked: <pre><code class="lang-auto">class Conv2dUnary(nn.Module):
    def __init__(
        self,
        in_channels,
        out_channels,
        kernel_size,
        stride=1,
        padding=0,
        dilation=1,
        bitwidth=2,
        batch_size=1,
        kernel=None,
        bias=None,
    ):
        super(Conv2dUnary).__init__()

        self.in_channels = in_channels
        self.out_channels = out_channels
        self.kernel_h, self.kernel_w = kernel_size
        self.kernel_d = in_channels
        self.stride = stride
        self.padding = padding
        self.dilation = dilation
        self.bitwidth = bitwidth
        self.batch_size = batch_size
        self.kernel = kernel
        self.bias = bias

    @autocast
    def forward(self, input):
 
        return output
</code></pre>
<p>I trying to perform convolution by representing data in unary domain. I initially created a method to perform conv2d in unary that works fine, now trying to convert into a nn.Module and use as a layer in NN. I am not sure above the root cause of the issue.</p>

username: AlphaBetaGamma96
cooked: <aside class="quote no-group" data-username="Sairam954" data-post="5" data-topic="151821">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sairam954/48/48574_2.png" class="avatar"> Sairam954:</div>
<blockquote>
<pre><code class="lang-auto">        super(Conv2dUnary).__init__()
</code></pre>
</blockquote>
</aside>
<p>Shouldn’t this be,</p>
<pre><code class="lang-auto">Super(Conv2dUnary, self).__init__()
</code></pre>
<p>Perhaps you’ve not fully initialized the parent class?</p>

username: Sairam954
cooked: <p>Yes, I missed it. Thanks a lot <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a>. I will remove the code, as it contains some research information.</p>
CI vs. local development

username: ishaan-mehta
cooked: <p>Did I imagine this, or is there a contribution guide somewhere that advises contributors to not rely on the GitHub CI and instead test locally before pushing? I understand why that is the case—the only reason I bring that up is because I want to see that entire document again and this is all I remember from it. I’ve seen the contribution guides linked from the README.md.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The main contributing doc is: <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md" class="inline-onebox">pytorch/CONTRIBUTING.md at master · pytorch/pytorch · GitHub</a><br>
I am not sure we recommend not to rely on CI though.<br>
The main thing might be that CI testing is slower than doing the local checks. But CI will cover a lot of configurations that you (most likely) cannot test locally so it is very important today.</p>
How to pass two arguments in transforms.Lambda(<func_name>))

username: chashi
cooked: <p>I have a function like this:<br>
def max_abs_normalize(img, max_abs_val):<br>
return (img)/max_abs_val</p>
<p>I am creating a transform object like this:<br>
self.transform = transforms.Compose([<br>
transforms.Resize(image_size),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.Lambda(max_abs_normalize)<br>
])</p>
<p>def <strong>getitem</strong>(self, index):<br>
path = self.paths[index]<br>
<span class="hashtag">#img</span> = Image.open(path)<br>
img = np.load(path)<br>
img = img.f.arr_0<br>
img = img.reshape(1,img.shape[0],img.shape[0])<br>
img = torch.from_numpy(img)<br>
return self.transform(img)</p>
<p>How can I pass the two parameters in the transform.Lambda? for my max_abs_normalize function?</p>

username: Matias_Vasquez
cooked: <p>This is just a workaround, but you could do something like this.</p>
<pre><code class="lang-python"># Define your own Lambda implementation
import torch
import torchvision
from torchvision import transforms


class MyLambda(torchvision.transforms.Lambda):
    def __init__(self, lambd, max_abs_val):
        super().__init__(lambd)
        self.max_abs_val = max_abs_val

    def __call__(self, img):
        return self.lambd(img, self.max_abs_val)

def max_abs_normalize(img, max_abs_val):
    return (img)/max_abs_val
</code></pre>
<p>You can then initialize your function with some value</p>
<pre><code class="lang-python">transform = transforms.Compose([
    transforms.Resize(image_size),
    transforms.RandomHorizontalFlip(),
    MyLambda(max_abs_normalize, SOME_INITIAL_VALUE)
])
</code></pre>
<p>But if you want to change this, then you can access it like this</p>
<pre><code class="lang-python">transform.transforms[-1].max_abs_val = NEW_VALUE
</code></pre>

username: chashi
cooked: <p>Thank you. It worked!</p>
U-Net implementation training is returning a Loss of -0.0 since the first iteration

username: PabloDavidAR
cooked: <p>As I mention in the title, I’m implementing a U-Net that I’m using for segmentation with retina images. The problem appears when I’m training the model, since the loss function is 0.0 all the time. The inputs are RGB images and the output of the neural network is a 1 channel image, while the label is a 1 channel BW image.</p>
<p>I’m currently trying to use 2 channel output instead 1 after reading various posts, but with that case it is not even letting me calculate the loss itself.</p>
Undefined symbol SmallVectorBase

username: BBniccccccc
cooked: <p>Hello and thanks for reading this.<br>
I’ve been using Torchscrip for a C++ projet for 1 month now. I don’t know what i’ve done but suddenty i can’t execute my code.  I have created another small project simpler for this topic.</p>
<p>Here is my cmake</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0)

project(aanet)

# add_definitions(-D_GLIBCXX_USE_CXX11_ABI=0)

set(CMAKE_CXX_FLAGS "-D_GLIBCXX_USE_CXX11_ABI=0")
set(Torch_DIR ${CMAKE_SOURCE_DIR}/libtorch/share/cmake/Torch)
find_package(Torch REQUIRED)

add_executable(${PROJECT_NAME} main.cpp)

target_link_libraries(${PROJECT_NAME} "${TORCH_LIBRARIES}")

set_property(TARGET ${PROJECT_NAME} PROPERTY CXX_STANDARD 14)
</code></pre>
<p>my code</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main() {
  torch::Tensor tensor = torch::rand({2, 3});
  std::cout &lt;&lt; tensor &lt;&lt; std::endl;
}
</code></pre>
<p>cmake  …</p>
<pre><code class="lang-auto">-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found Torch: libtorch/lib/libtorch.so  
-- Configuring done
-- Generating done
-- Build files have been written to: build
</code></pre>
<p>make</p>
<pre><code class="lang-auto">Consolidate compiler generated dependencies of target myproject
[ 50%] Linking CXX executable myproject
[100%] Built target myproject
</code></pre>
<p>./myproject</p>
<pre><code class="lang-auto">./myproject: symbol lookup error: ./myproject: undefined symbol: _ZN3c1015SmallVectorBaseIjE8grow_podEPvmm
</code></pre>
<p>I tested all the version of libtorch and all have the same problem.</p>
<p>I would be very grateful if someone can help me. I am blocked for 2 days on this and i m totally lost.</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> !</p>

username: BBniccccccc
cooked: <p>Probleme résolu. L’executable C++ utilisait la lib python qui était dans une version différente de celle utilisait par cmake</p>
Can Torchscript take more time than eager model

username: Miguel_Campos
cooked: <p>Hello guys.</p>
<p>I’m measuring the speed of a model using this code</p>
<blockquote>
<p>def benchmark_speed(model, input_shape, precision: str = ‘fp32’, nwarmup=10, nruns=500):<br>
input_data = torch.randn(input_shape)<br>
input_data = input_data.to(‘cuda’)<br>
if precision == ‘fp16’:<br>
input_data = input_data.half()</p>
<pre><code>print("Warm up ...")
with torch.no_grad():
    for _ in range(nwarmup):
        _ = model(input_data)
torch.cuda.synchronize()
print("Start timing ...")
timings = []
with torch.no_grad():
    for i in range(1, nruns + 1):
        start_time = time.time()
        _ = model(input_data)
        torch.cuda.synchronize()
        end_time = time.time()
        timings.append(end_time - start_time)
        if i % 100 == 0:
            print('Iteration %d/%d, ave batch time %.2f ms' % (i, nruns, numpy.mean(timings) * 1000))

print("Input shape:", input_data.size())
print('Average batch time: %.2f ms' % (numpy.mean(timings) * 1000))
</code></pre>
</blockquote>
<p>The method has a first part with warming and a second part where I measure the time speed of the model resnet101 in this way</p>
<blockquote>
<p>model = torch.hub.load(‘pytorch/vision:v0.10.0’, ‘resnet101’, False)<br>
model.cuda().eval()<br>
benchmark_speed(model, (64, 3, 224, 224))</p>
<p>trt_model = torch.jit.trace(model, torch.randn(1, 3, 224, 224).cuda())<br>
benchmark_speed(model, input_shape = (64, 3, 224, 224))</p>
</blockquote>
<p>I’m getting 75.42 ms and 75.69ms for each, eager model and torchscript model.<br>
I dont see the point why I would want to use TorchScript (apart of using it with torchTensorRT)</p>
<p>PDT: I also tried</p>
<blockquote>
<p>trt_model = torch.jit.trace(model, torch.randn(<strong>64</strong>, 3, 224, 224).cuda())<br>
benchmark_speed(model, input_shape = (64, 3, 224, 224))</p>
</blockquote>
<p>but the behavior is similar, except for batch 1, in that case TorchScript seems be a bit (a bit) lightly faster.</p>
<p>Am I doing something wrong? I don’t see the advantages of using TorchScript.</p>
<p>Thank you</p>
Check if a value is present in each group of another va

username: Doulson_Mathis
cooked: <p>Hello !<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38676e0b5c19dddc6b0cf309f6f994d591173752.png" alt="forum python" data-base62-sha1="82YkOOxvVSoBQ4Ceq6oTbGdy7aa" width="196" height="206"></p>
<p>I have a dataframe with two columns : an ‘ID’ column and a column ‘V1’ with different values between 1 and 3.<br>
I’d like to mark each ID with 1 if value 3 is present for this group and 0 otherwise.</p>
<p>The attachment is more clear. I’d like to obtain the column ‘Result’, based on each group of ‘Id’ and on the occurence of “'3” in the column “Var1”</p>
<p>Thank you by advance for your help !</p>

username: Andrei_Cristea
cooked: <p>This is a pandas question rather than a PyTorch question, but you can try this:</p>
<pre><code class="lang-auto">df["Result"] = (df["Var1"] == 3).groupby(df["ID"]).transform("sum").astype(bool).astype(int)
</code></pre>

username: Doulson_Mathis
cooked: <aside class="quote no-group" data-username="Andrei_Cristea" data-post="2" data-topic="151787">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrei_cristea/48/47843_2.png" class="avatar"> Andrei_Cristea:</div>
<blockquote>
<pre><code class="lang-auto">df["Result"] = (df["Var1"] == 3).groupby(df["ID"]).transform("sum").astype(bool).astype(int)
</code></pre>
</blockquote>
</aside>
<p>Thank you very much, it works !<br>
But I don’t get what the “.transform(“sum”)” does ? What does is sum exactly ?</p>

username: Andrei_Cristea
cooked: <p>Basically what this ugly chain does is:</p>
<ul>
<li>Create a column that is populated with True if Var1 == 3, and False otherwise:<br>
<code>(df["Var1"] == 3)</code>
</li>
<li>Group by ID and, inside each group, sum these boolean values. So you’re going to get some integer that basically counts how many times 3 appears inside each ID group:<br>
<code>.groupby(df["ID"]).transform("sum")</code>
</li>
<li>Then cast this sum to bool and then back to int, as a quick and dirty way to collapse all numbers &gt;=1 to 1 and keep zeros intact:<br>
<code>.astype(bool).astype(int)</code>
</li>
</ul>
Cuda Memory why?

username: Andre_Amaral_IST
cooked: <p>I have no PID working, so why does my cuda still have memory allocated?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c4d48e63068d98647aa0d8f913805894b6fc512c.png" alt="image" data-base62-sha1="s5eYZAhgVnkzcbJKFqY3qBzq5Xe" width="568" height="310"></p>

username: sebk26
cooked: <p>Are you sure this is a PyTorch related question? If so, why?</p>
<p>Look with <code>htop</code> or <code>top</code> for ghost processes that were not properly removed and kill them.</p>
<p>Or as mentioned here [1]: <em>Open task manager(ctrl+alt+del) and then start releasing the memory by ending unnecessary tasks. This worked in my case. If you use colab, go to run time and select factory reset runtime</em></p>
<p>[1] <em>Originally posted by <span class="mention">@vignesh-creator</span> in <a href="https://github.com/pytorch/pytorch/issues/16417#issuecomment-841964979" class="inline-onebox" rel="noopener nofollow ugc">RuntimeError: CUDA out of memory. Tried to allocate 12.50 MiB (GPU 0; 10.92 GiB total capacity; 8.57 MiB already allocated; 9.28 GiB free; 4.68 MiB cached) · Issue #16417 · pytorch/pytorch · GitHub</a></em></p>
Best Practice: How to handle input data with multiple resolutions?

username: sbradshaw
cooked: <p>Hi, I am looking for some information regarding the best practice in PyTorch for data inputs into a model.</p>
<p>Problem description:<br>
I work with satellite channels of different resolutions. Because of this one input may be half the size of another input (i.e. a 0.5 km resolution channel vs. a 1 km resolution channel). When I access the data in a generator, I load a tensor of inputs from a netcdf as well as a tensor targets (i.e. x and y). In order to account for the difference of resolutions, I have previously used numpy to created a concatenated array of inputs and an initial upsampling step within my U-Net model (mode nearest). However, I want to get away from using numpy, instead using Torch tensors to help reduce i/o. Correct me if I am wrong, but I <em>believe</em> the Torch tensors need the data to be the same size for a Torch tensor with<code> torch.cat()</code>. I found the function <code>torch.nn.functional.interpolate()</code> which could be implemented before running the model (i.e. in the data generator), but I am not sure if it is <em>proper</em> to do this.</p>
<p>Thus, in the spirit of best practice, what is the best way to deal with data of multiple sizes?</p>
<p>Disclaimer:<br>
I am just switching to PyTorch from Keras for the customizability with regards to GPU use, so forgive me if this is rather obvious.</p>

username: JuanFMontesinos
cooked: <p>You can think of torch tensors as np arrays.<br>
I’m afraid tensors need to be same size to be concatenated of stacked.<br>
Using numpy is a good practice as tensors can reuse the same memory address.<br>
Usually data is padded.  Pytorch doesn’t have a very optimized IO or there is no special tool.</p>
<p>If you want optimized IO you can try nvidia’s DALI.</p>
Pre-trained Model on Spectrograms

username: _stracii
cooked: <p>Hi, are there any available pre-trained models on spectrograms?</p>
<p>All I identified here (<a href="https://pytorch.org/vision/stable/models.html" class="inline-onebox" rel="noopener nofollow ugc">Models and pre-trained weights — Torchvision 0.12 documentation</a>) are on ImageNet.</p>
<p>Thank you</p>

username: JuanFMontesinos
cooked: <p>You can find many models in audio libraries such as speechbrain or asteroid. Torchvision is for vision.</p>
Pytorch Cuda Free GPU Memory

username: karunT
cooked: <p>del model<br>
torch.cuda.empty_cache()<br>
print("After Clearing Model: ", torch.cuda.memory_allocated(device))</p>
<p>I have loaded a model to GPU (model.to(device)) and then am trying to delete it to free up some memory. However, the memory allocated does not go down. Am i missing something?</p>

username: ptrblck
cooked: <p>You might have some references stored to this model, e.g. its output.<br>
Make sure to delete all tensors etc. which might depend on the <code>model</code> instance.</p>

username: Andre_Amaral_IST
cooked: <p>Hey,<br>
You also need to do this in order to kill the processes.<br>
For that do the following:</p>
<ol>
<li><code>nvidia-smi</code></li>
<li>In the lower board you will see the processes that are running in your gpu’s</li>
<li>Check their PID</li>
<li>Delete those processes making <code>kill PID_Number</code>
</li>
</ol>
<p>Hope it helps</p>
How to make the Linear layer a pass through layer without any calculation?

username: Matthew_Wang
cooked: <p>Hi, I want to overwrite the classifier layer of Distilbert as a pass through layer that does not do any calculation as I need to extract the pre-classification embeddings…</p>
<p>I did this but it doesn’t work:</p>
<pre><code>model.classifier = torch.nn.Linear(768, 768)
</code></pre>
<p>Can anyone help please? Thanks</p>

username: ptrblck
cooked: <p>You could replace it with <code>nn.Identity</code> which will return the input without any modifications.</p>

username: Matthew_Wang
cooked: <p>It worked! Thank you !!</p>
Pytorch coordinates of matches

username: mert_yilma
cooked: <p>hi, im trying to detect objects on my screen. Everything works ok but i dont want these rectangles on my image. I just want to have x1, y1, x2, y2 coordinates. How can i get these data?</p>
<pre><code class="lang-auto">import torch
import cv2 as cv
import numpy as np
from ss import ss

model = torch.hub.load('ultralytics/yolov5', 'custom', path = 'best.pt', force_reload=True)
model.conf = 0.2
screen = ss()
screen = model(screen)
screen.render()
</code></pre>
Normalization image pixel between 0 and 255 in MNIST as it has 3 channels R, G, and B

username: S_M
cooked: <p>Hi, I want to convert the grayscale of MNIST images to RGB as its values are between 0 and 255. How is it possible for PyTorch?</p>

username: ptrblck
cooked: <p>To create the RGB channels, you could either repeat the gray channel manually or use <code>transforms.Grayscale(num_output_channels=3)</code>. I’m not sure if you want to yield tensors in the range <code>[0, 255]</code>, but if so you could use <code>transforms.Normalize(mean=0., std=(1/255., 1/255., 1/255.))</code> after applying <code>transforms.ToTensor()</code>.</p>

username: S_M
cooked: <p>Hi ptrblck,  thank you so much.</p>
Python Import not working at the top of file but works inside function

username: Mukii
cooked: <p>I was using django and ran into the following problem just wondering why Python orge acts this way. I was importing a function and I had the import at the top of the file with my other imports and my tests keep failing but my tests passes after I move the import statement to inside the function I was working on. Has anyone encountered such problems ?</p>

username: ptrblck
cooked: <p>Could you give a small example of this issue, please?</p>
Tesla K20c ; torch.cuda.is_available() returns false

username: rog12c4z
cooked: <p>I have a tesla K20c.<br>
I installed the CUDA driver from here:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://developer.nvidia.com/cuda-downloads?target_os=Windows&amp;target_arch=x86_64&amp;target_version=10&amp;target_type=exe_local">
  <header class="source">
      <img src="https://developer.nvidia.com/sites/all/themes/cuda/favicon.ico" class="site-icon" width="48" height="48">

      <a href="https://developer.nvidia.com/cuda-downloads?target_os=Windows&amp;target_arch=x86_64&amp;target_version=10&amp;target_type=exe_local" target="_blank" rel="noopener nofollow ugc" title="08:45AM - 15 April 2021">NVIDIA Developer – 15 Apr 21</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://developer.nvidia.com/cuda-downloads?target_os=Windows&amp;target_arch=x86_64&amp;target_version=10&amp;target_type=exe_local" target="_blank" rel="noopener nofollow ugc">CUDA Toolkit 11.7 Downloads</a></h3>

  <p>Resources CUDA Documentation/Release NotesMacOS Tools Training Sample Code Forums Archive of Previous CUDA Releases FAQ Open Source PackagesSubmit a BugTarball and Zip Archive Deliverables</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>&lt;nvcc -V&gt;  it is available and installed.</p>
<p>Then, I installed pytorch as:<br>
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch</p>
<p>However, torch.cuda.is_available()<br>
returns: False</p>
<pre><code class="lang-auto">(base) C:\&gt;nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_19:00:59_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0

(base) C:\&gt;conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.


(base) C:\&gt;ipython
Python 3.8.8 (default, Apr 13 2021, 15:08:03) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.22.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import torch

In [2]: torch.rand(3,5)
Out[2]:
tensor([[0.0115, 0.6534, 0.6884, 0.2533, 0.4802],
        [0.6628, 0.7389, 0.1999, 0.4942, 0.4311],
        [0.8925, 0.1638, 0.6131, 0.1957, 0.9486]])

In [3]: torch.cuda.is_available()
Out[3]: False

</code></pre>

username: ptrblck
cooked: <p>The K20 should have a compute capability of 3.5 which is not shipped in the binaries anymore (min. is 3.7), so you could try to build PyTorch from source as described <a href="https://github.com/pytorch/pytorch#from-source">here</a>.</p>
Torch not compiled with CUDA enabled... again - SOLVED

username: mark87
cooked: <p>Sorry to post this again, but after reading ALL of the available topics on this forum, I still can’t get my cuda to be recognized by pytorch. Does with every single little update the whole compatibility collapse? I don’t know anymore which torch version goes with which cuda with which cudnn with wich python version etc. This is madness.</p>

username: InnovArul
cooked: <p>Pl. try to post additional info (OS, GPU type, Python version, how are you installing, etc), so that someone who knows / who has gone thru similar things could possibly help you.<br>
Without additional info, this post will not get any reply probably.</p>

username: mark87
cooked: <p>I solved it. So, for those who are using jupyter notebook: create a venv in anaconda (NOT miniconda) and then activate it from the anaconda navigator.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="mark87" data-post="1" data-topic="151526">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mark87/48/49544_2.png" class="avatar"> mark87:</div>
<blockquote>
<p>Does with every single little update the whole compatibility collapse?</p>
</blockquote>
</aside>
<p>No, the compute capabilities weren’t changes in the last year with the exception that new Ampere devices need the CUDA11 runtime.</p>
<aside class="quote no-group" data-username="mark87" data-post="1" data-topic="151526">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mark87/48/49544_2.png" class="avatar"> mark87:</div>
<blockquote>
<p>I don’t know anymore which torch version goes with which cuda with which cudnn with wich python version etc. This is madness.</p>
</blockquote>
</aside>
<p>You could install the binaries using the provided commands from the <a href="https://pytorch.org/get-started/locally/">website</a> which mention the corresponding CUDA runtime version shipped in the binaries. Your local CUDA toolkit won’t be used unless you build PyTorch from source or custom CUDA extensions.</p>
<aside class="quote no-group" data-username="mark87" data-post="3" data-topic="151526" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mark87/48/49544_2.png" class="avatar"> mark87:</div>
<blockquote>
<p>I solved it. So, for those who are using jupyter notebook: create a venv in anaconda (NOT miniconda) and then activate it from the anaconda navigator.</p>
</blockquote>
</aside>
<p>Good to hear you solved it, as the majority of these issues are caused by a broken environment.</p>
How to avoid nan loss when using fp16 training?

username: Maxwell_Albert
cooked: <p>Hi, I am using roberta-base to train RTE dataset. When I use torch.half() to change my models parameters, I find that after first backward, the loss of model will be nan.<br>
Is there any way to solve it?</p>

username: ptrblck
cooked: <p>FP16 has a limited range of ~<code>+/-65k</code>, so you should either use the automatic mixed-precision util. via <code>torch.cuda.amp</code> (which will use FP16 where it’s considered to be save and FP32 where needed) or you would have to transform the data and parameters to FP32 for numerically sensitive operations manually in case you want to stick to a manual approach.</p>