akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Input dims and input type for speed benchmarking on Android

prerna: <p>Hi,</p>
<p>I have a custom PyTorch model, the inputs are of the model as like this:<br>
model(torch.randint(0, 10,(1,1)), torch.rand(1, 51, 2048), torch.randint(0, 10,(1, 1))).<br>
I am trying to speed benchmark this model with Android, but I am unable to understand how I should use the “input_dims” and “input_type” parameters as this model involves a mix of float and integer types.</p>
<p>Thanks!</p>

prerna: <p>An update:<br>
I checked the speed_benchmark_torch.cc file. It specifies “If multiple input needed, use semicolon to separate the dimension of different tensors.”</p>
<p>Therefore, I tried: --input_dims=“1,1;1,51,2048;1,1” --input_type=“int64;float;int64”.</p>
<p>This results in error:<br>
terminating with uncaught exception of type c10::Error: [enforce fail at speed_benchmark_torch.cc:111] input_dims_list.size() == input_type_list.size(). 1 vs 0. Input dims and type should have the same number of items.</p>
<p>(no backtrace available)</p>
<p>Aborted</p>

axitkhurana: <p>Hi <a class="mention" href="/u/prerna">@prerna</a>, what’s the full command you are running?</p>

TimPushkin: <p>I had the same problem. The solution is to put an escape character before <code>;</code>, so the parameters should look like <code>--input_dims=“1,2\;1,2,3\;5,6,7” --input_type=“float\;float\;uint8_t”</code> for example. Also, the script will require you to specify the memory format: <code>--input_memory_format="contiguous_format\;contiguous_format\;contiguous_format"</code>, for example (<code>contiguous_format</code> is the default value).</p>
Question on good practice in defining trainable parameters

nullgeppetto: <p>I have a simple model such as the one shown below, where I define a set of parameters of shape <code>(K, 2 * N * d)</code> and I initialize them so as dim=1 holds pairs of “antipodal” vectors; that is, <code>self.params[:, j+1, :] = -self.params[:, j, :]</code>. This of course results in a number of <code>K * 2 * N * d</code> trainable parameters, while at the same time, the pairs in <code>dim=1</code> are not “tied” together during training (they are only initialized to be opposite).</p>
<p>I would like to modify my model so as I define only <code>K * N * d</code> parameters, but in a way that I can form <code>self.params</code> as a <code>(K, 2 * N * d)</code> matrix holding the property described above. What would be the optimal way to do so? I could create a list of parameters maybe and then concatenate and repeat in order to get the final matrix, but would that be efficient?</p>
<p>What would be your take on this? Thanks!</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


class Model(nn.Module):
    def __init__(self, K, N, d):
        """...
        
        The model defines a set of parameters of shape (K, 2 * N * d) and initialises them so as dim=1 
        holds pairs of "antipodal" vectors; that is, self.params[:, j+1, :] = -self.params[:, j, :].
        
        Args:
            K (int) : 
            N (int) : 
            d (int) : 
            
        """
        super(Model, self).__init__()
        self.K = K
        self.N = N
        self.d = d
        
        self.params = nn.Parameter(data=torch.ones(K, 2 * N * d))
        params_init = torch.zeros(K, 2 * N, d)
        for k in range(self.K):
            vectors_k = []
            for i in range(self.N):
                v = torch.randn(1, d)
                vectors_k.extend([v, -v])
            vectors_k = torch.cat(vectors_k)
            params_init[k, :] = vectors_k
        
        self.params.data = params_init.reshape(self.K, 2 * N * d).clone()
        
        
model = Model(K=32, N=16, d=2)
trainable_parameters = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(trainable_parameters)
# 2048
</code></pre>
Loss function on the mean of model outputs

moth: <p>Hello everybody. I have a problem where a patient can have multiple images from a study and to determine if the patient has a disease or not we average all the predictions from his individual images. So to summarize:</p>
<ol>
<li>Iterate over all the patient’s images.</li>
<li>Predict whether the disease is present on each image.</li>
<li>Take the mean of all predictions as the final prediction.</li>
<li>Compute loss function by comparing the mean of predictions with ground truth.<br>
The code looks like this:</li>
</ol>
<pre><code class="lang-auto"># Get predictions in the form of logits
out = model(images)
# Sigmoid predictions to get probabilities from logits
pred = torch.sigmoid(out)
# Average all predictions for a single patient
pred = torch.mean(pred)
loss = criterion(pred, label)
loss.backward()
</code></pre>
<p>I do not know if back propagation is being computed on the mean of the output (called “pred” above) since I have not utilized the <code>requires_grad</code> parameter on <code>torch.mean()</code> nor on <code>torch.sigmoid()</code>.</p>
<p>Thanks in advance.</p>

InnovArul: <p>Ideally, you do not need to use <code>requires_grad</code> for intermediate result variables.<br>
The model parameters act as leaf variables and typically have <code>requires_grad</code> set to <code>True.</code></p>
<p>Pytorch automatically takes care of backpropagating the gradients through intermediate results to those leaf variables (i.e., parameters), given that you do not break the computation graph.</p>
AMP Scaler.step() never work

PhxxTrr: <p>Hi, I use AMP to train my model, but the parameters never update ,so the training is invalid.<br>
I find that the program always skip “scaler.step()” , but the loss is neither NAN nor INF, I don’t know the reasons.<br>
Code show as below:</p>
<p>backward:<br>
for epoch:<br>
for data in enumerate(dataloader)<br>
with autocast():<br>
loss= model(data)</p>
<pre><code>  optimizer.zero_grad()
  scaler.scale(loss).backward()
  scaler.step(optimizer)
  scaler.update()
  scheduler.step()

  optimizer = opt.AdamW([
    {'params':, ..........,'initial_lr': lr},
   ], lr=lr, weight_decay=wd)

 scheduler = opt.lr_scheduler.CosineAnnealingLR(optimizer, T_max, eta_min=min_lr, last_epoch=epoch)
</code></pre>
<p>Enviroment:<br>
OS:Ubuntu 20.04<br>
Python: 3.8<br>
Torch:1.7.1<br>
CUDA:11.4<br>
Graphics Card：3090</p>

ptrblck: <p>I don’t know why you are re-initializing the optimizer inside the training loop, but this should not be relevant to your question.<br>
Check the scale factor via <code>print(scaler.get_scale())</code> before and after <code>scaler.update()</code> was used to check if internally invalid gradients were found and thus the <code>optimizer.step()</code> call was skipped.</p>

PhxxTrr: <p>I just show which optimizer i use, but not initiallize in loop.Thank you.</p>
Change sampler every epoch

satrialoka: <p>I would like to filter out my dataset every epoch by changing the sampler for the data loader, but when I change the sampler it gives me :<br>
<code>ValueError: sampler attribute should not be set after DataLoader is initialized</code></p>
<p>here’s my sample code :</p>
<pre><code class="lang-auto">def train_test():  
        epoch = epoch_n
        for e in range(epoch):
                train_loss, train_acuracy, losses = train(train_loader,criterion,optimizer,e+1,True)
                test_loss, test_acuracy = test(test_loader,criterion,optimizer,e+1,True)
                print('Epoch %d, Training Loss/Acc: %f//%f, Testing Loss/Acc: %f//%f' % (e+1,train_loss,train_acuracy,test_loss,test_acuracy))
                indexsorted = rankloss(losses)
                indexcuted  = cutdataset(indexsorted)
                train_loader.sampler = sampler.SubsetRandomSampler(indexcuted)
</code></pre>
<p>is there a way to do this ?</p>

Oli: <p>You could create a new dataloader with the new sampler every epoch. If you don’t want to do this, do you mind sharing what the differences in the samplers are? Perhaps you can create one sampler that can take the behavior of all the samplers you need</p>

satrialoka: <p>solved by creating custom sampler by writing a method to update the sampler’s properties. Thank you</p>

mzimmer: <p>Mind to share your code?</p>
Using views on gpu to reduce memory transfers

Jakub_Mitura: <p>Hello I am training the model that is using a lot of overlapping data, and i would like to reduce memory transfers coused by reloading for each case the same data</p>
<p>More precisely<br>
I have 3d array let’s say 10000x10000x10000<br>
Model accepts 32 x32x32 arrays and is intentionally learning only local features</p>
<p>Each training data that i will put in model is 32x32x32 neighberhood of each voxel of main array, so two neighbouring voxel shere most of the data</p>
<p>Additionally i need to invoke the same 32 by 32 by 32 block 8 times for each voxel modyfing just single voxel</p>
<p>Clearly i could copy each 32 by 32 by 32 neighberhood in a loop than in inner loop stack 8 modifications of the central voxels  but it will lead by very rough approximation loading the data about the same voxels 10000 times in each epoch, what is extremely wastfull particularly on gpu</p>
<p>Is there a workaround for my case ?</p>
PyTorch tensors from C++ to Python

dno89: <p>Hi everybody,<br>
I need to embed a Python interpreter in my C++ application and I want to pass <em>Tensor</em> s back and forth from C++ and Python.<br>
I know that Boost.Python should fit this purpose, but I’m puzzled on how should I handle the Tensors exchange. My guess is that it should be relatively easy, considering that the C++ and Python frontend probably use the same underlying types. Can somebody point me in the right direction?<br>
Thanks</p>

yf225: <p><code>THPVariable_Wrap(Variable var)</code> in <a href="https://github.com/pytorch/pytorch/blob/63214b572b6a18f7328687ca69420f0be0eaaaa9/torch/csrc/autograd/python_variable.cpp#L64-L76" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/63214b572b6a18f7328687ca69420f0be0eaaaa9/torch/csrc/autograd/python_variable.cpp#L64-L76</a> is how we wrap a <code>Variable</code> (can be cast from <code>torch::Tensor</code>) into a Python object. Using Boost.Python there might be a way to pass the <code>Variable</code> C++ object to Python and construct the Python object.</p>

dno89: <p>Thank you! I will look at it!</p>

lostmsu: <p><a class="mention" href="/u/dno89">@dno89</a> have you made it work?</p>
Torch.nn.Conv3d inference time duration difference based on number of out channels

Coder_Man: <p>Hello!<br>
I have a model, through which i run inference. It uses Conv3d. When i try to run inference for out_channels=14, the time taken by it is 60ms but when i try to run it with out_channels=24, the time taken by it 100ms. Is there any way to optimize it.</p>

ptrblck: <p>In case you are using a GPU you could try to enable cuDNN benchmarking via <code>torch.backends.cudnn.benchmark = True</code> which would profile different kernels for your workload and would pick the fastest one. Using <code>to(memory_format=torch.channels_last_3d)</code> would also help in case you are using mixed-precision training.</p>

Coder_Man: <p>Thank you so much. It worked, the performance increased. but just wanted to confirm Will it have any effect on accuracy?</p>

ptrblck: <p>No, the accuracy should never be impacted by the algorithm choice. You would not expect to see bitwise identical results due to the limited floating point precision, but your accuracy should also not be sensitive to these expected small errors.</p>

Coder_Man: <p>Thank you so much, it helped me a lot!</p>
Error when I used Torchscript: Python builtin <built-in function next> is currently not supported in Torchscript

lpieri: <p>Hello I have a problem when I would like compile my custom model with TorchScript.</p>
<p>My error is:</p>
<pre><code class="lang-auto">RuntimeError:
Python builtin &lt;built-in function next&gt; is currently not supported in Torchscript:
  File "file.py", line 152
        seq_iter = enumerate(scores)
        _, inivalues = next(seq_iter)
                       ~~~~ &lt;--- HERE
</code></pre>

ptrblck: <p>TorchScript doesn’t seem to be able to script the <code>next</code> call, so you might want to script the model only assuming that the <code>next</code> call is used in the data loading part of your script.</p>
Obtain probabilities from cross entropy loss

SU801T: <p>I’m doing binary classification, however used categorical cross entropy loss rather than binary to train my model  (I believe this is ok as results appear to be normal).</p>
<p>Using a saved model, during inference however, I would like to obtain class probabilities for the outputs from the model.</p>
<p>I believe logits are output first. I then pass them through a softmax layer before selecting the top probability/predicted class.</p>
<p>Does this look correct?</p>
<pre><code class="lang-auto">outputs =  model(inputs) # obtain logits
soft_outputs = torch.nn.functional.softmax(outputs, dim=1) #pass through softmax
top_p, top_class = soft_outputs.topk(1, dim = 1) # select top probability as prediction
</code></pre>

ptrblck: <p>Yes, your usage of <code>softmax</code> looks correct, but note that the <code>topk</code> results will be the same using the logits or probabilities since <code>softmax</code> does not change the order of the logits.</p>

SU801T: <p>Ah ok, yes, I basically wanted to know the most likely predicted class and it’s associated probability score.</p>
<p>It appears that the above predicted class is the same as when I predict the class label like this:</p>
<pre><code class="lang-auto">_, pred = torch.max(outputs, 1)
</code></pre>
<p>From my understanding, the size of the logits are related to the probability score…</p>

ptrblck: <p>Yes, <code>torch.argmax</code> on the logit outputs will return the same predicted class indices as when it’s applied on the probabilities (i.e. <code>softmax</code> outputs). However, if you want to “see” the probabilities, then you can of course apply the additional <code>softmax</code>. Just make sure to not pass these probabilities to e.g. <code>nn.CrossEntropyLoss</code> during training as this criterion expects logits.</p>
<p>Yes, the size of the logits and probabilities corresponds to the number of predicted classes as <code>[batch_size, nb_classes, *]</code>.</p>
RuntimeError: cannot resize variables that require grad

chaunceyshen: <p>I am trying to implement the NMS with pytorch</p>
<p>My code is below</p>
<pre><code class="lang-auto">def nms(boxes, scores, overlap=0.5, top_k=200):
    keep = scores.new(scores.size(0)).zero_().long()
    if boxes.numel() == 0:
        return keep
    x1 = boxes[:, 0]
    y1 = boxes[:, 1]
    x2 = boxes[:, 2]
    y2 = boxes[:, 3]
    area = torch.mul(x2 - x1, y2 - y1)
    v, idx = scores.sort(0)  # sort in ascending order
    # I = I[v &gt;= 0.01]
    idx = idx[-top_k:]  # indices of the top-k largest vals
    xx1 = boxes.new()
    yy1 = boxes.new()
    xx2 = boxes.new()
    yy2 = boxes.new()
    w = boxes.new()
    h = boxes.new()

    # keep = torch.Tensor()
    count = 0
    while idx.numel() &gt; 0:
        i = idx[-1]  # index of current largest val
        # keep.append(i)
        keep[count] = i
        count += 1
        if idx.size(0) == 1:
            break
        idx = idx[:-1]  # remove kept element from view
        # load bboxes of next highest vals
        #torch.index_select(x1, 0, idx, out=xx1)
        #torch.index_select(y1, 0, idx, out=yy1)
        #torch.index_select(x2, 0, idx, out=xx2)
        #torch.index_select(y2, 0, idx, out=yy2)
        xx1 = torch.index_select(x1, 0, idx)
        yy1 = torch.index_select(y1, 0, idx)
        xx2 = torch.index_select(x2, 0, idx)
        yy2 = torch.index_select(y2, 0, idx)
        # store element-wise max with next highest score
        xx1 = torch.clamp(xx1, min=x1[i])
        yy1 = torch.clamp(yy1, min=y1[i])
        xx2 = torch.clamp(xx2, max=x2[i])
        yy2 = torch.clamp(yy2, max=y2[i])
        w.resize_as_(xx2)
        h.resize_as_(yy2)
        w = xx2 - xx1
        h = yy2 - yy1
        # check sizes of xx1 and xx2.. after each iteration
        w = torch.clamp(w, min=0.0)
        h = torch.clamp(h, min=0.0)
        inter = w*h
        # IoU = i / (area(a) + area(b) - i)
        rem_areas = torch.index_select(area, 0, idx)  # load remaining areas)
        union = (rem_areas - inter) + area[i]
        IoU = inter/union  # store result in iou
        # keep only elements with an IoU &lt;= overlap
        idx = idx[IoU.le(overlap)]
    return keep, count
</code></pre>
<p>and I got this error</p>
<pre><code class="lang-auto">w.resize_as_(xx2)
RuntimeError: cannot resize variables that require grad
</code></pre>
<p>How can I solve this problem?</p>

ptrblck: <p>Are you sure you want to use the <code>resize_</code> method and not <code>reshape</code> or <code>view</code>?<br>
If so, could you explain why it’s needed in your code as it’s usually used internally and I don’t think it’s a safe method for custom modules?</p>
Different forward( ) in train and test modes

Erica_Zheng: <p>Dear All,</p>
<p>I want to implement a two-branch architecture with <code>forward( )</code> operate differently in train and test mode:</p>
<ul>
<li>
<p>For training, <code>forward(x1, x2)</code> which allows model takes <code>x1</code>,<code> x2</code> for <code>branch_A</code> and <code>branch_B</code> at the same time.</p>
</li>
<li>
<p>For testing, I want to evaluate each branch separately, such as <code>forward(x1)</code> and <code>forward(x2)</code>.</p>
</li>
</ul>
<p>How to implement this?</p>
<p>Thank you.</p>

ptrblck: <p>You can use the internal <code>self.training</code> flag in a condition and then use the desired code path.<br>
Other layers, such as <code>nn.Dropout</code> and <code>nn.BatchNormXd</code>, are doing the same.<br>
The <code>self.training</code> attribute will be set via <code>model.train()</code> and <code>model.eval()</code>.</p>
RuntimeError when `num_workers` is larger

frederni: <p>Hello there,<br>
I’m trying to run some training on a network on a 72-core GPU so I’d prefer to use 24 as <code>num_workers</code> for the dataloader. When I’ve tested for smaller choices, e.g. 3 workers, I manage to run the code without any issue. When I however try to use 24 workers, I get a RuntimeError.</p>
<p>Traceback is as follows,</p>
<pre><code class="lang-auto">Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/local_disk0/.../torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/local_disk0/.../torch/utils/data/_utils/fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "/local_disk0/.../torch/utils/data/_utils/collate.py", line 175, in default_collate
    return [default_collate(samples) for samples in transposed]  # Backwards compatibility.
  File "/local_disk0/.../torch/utils/data/_utils/collate.py", line 175, in &lt;listcomp&gt;
    return [default_collate(samples) for samples in transposed]  # Backwards compatibility.
  File "/local_disk0/.../torch/utils/data/_utils/collate.py", line 140, in default_collate
    out = elem.new(storage).resize_(len(batch), *list(elem.size()))
RuntimeError: Trying to resize storage that is not resizable
</code></pre>
<p>Does anyone know why this happens?</p>

ptrblck: <aside class="quote no-group" data-username="frederni" data-post="1" data-topic="157255">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/frederni/48/51697_2.png" class="avatar"> frederni:</div>
<blockquote>
<p>on a 72-core GPU</p>
</blockquote>
</aside>
<p>I assume you meant CPU?</p>
<p>Are you able to reproduce the issue using a random tensor as the <code>Dataset</code> and by just increasing the number of workers?</p>
How to load tensors from a few pt files lazily into neural network DataLoader

registerrug: <p>I currently have 11 pt files of size “torch.Size([1000000, 3, 50, 40])”. Each tensor for the cnn is 3x50x40. Each pt file has 1MM of these tensors. I cannot combined them due to memory limitations and I do not want to save them as 11MM individual pt files. Can anyone help me understand how to get these into a DataLoader?</p>
<p>With a smaller dataset I have used:</p>
<pre><code class="lang-auto">data_tensor = torch.load('tensor_1.pt')  
dataset = torch.utils.data.TensorDataset(data_tensor, target_tensor)
train_set, val_set, test_set = random_split(dataset, [int(size*.8), int(size*.1), size-int(size*.8)-int(size*.1)])
train_loader = DataLoader(train_set, batch_size=128, num_workers=4, shuffle=True)
</code></pre>
<p>but with the size of these files this will not work. Thank you!</p>

ptrblck: <p>Each file should take <code>1000000 * 3 * 50 * 40 * 4 / 1024**3 ~= 22.3GB</code> of memory. I don’t know if you could load parts of this tensor into your memory in case loading the entire file already runs into the OOM.<br>
In <code>numpy</code> you could use the <code>mmap_mode</code> to keep the file on the disk and load slices only.</p>

registerrug: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>.  This file size runs smoothly.  I just cant figure out how to load 11 of them only when needed.  Any guidance is helpful.</p>
<p>BTW I have learned more from reading your responses on others posts over the past year than any online forum.  Thank you for so consistently responding to questions!</p>

ptrblck: <p>A simple approach would be to recreate the <code>dataset</code> as a new <code>TensorDataset</code> for each file, wrap it into a <code>DataLoader</code>, and train with it.<br>
You could also write a custom <code>Dataset</code> and open a single file at a time in the <code>__getitem__</code> and hold to it until all samples were used.</p>

registerrug: <p>By “wrap it into a DataLoader” do you mean by using ConcatDataset or is there another way to do this?  I have never pulled two datasets into one DataLoader.</p>

ptrblck: <p>No, I wouldn’t use a <code>ConcatDataset</code> since you would need to preload all datasets, which wouldn’t fit into your RAM.<br>
I was thinking about this simple approach:</p>
<pre><code class="lang-python">for i in range(10):
    data_tensor = torch.load('tensor_{}.pt'.format(i))  
    dataset = torch.utils.data.TensorDataset(data_tensor, target_tensor)
    train_set, val_set, test_set = random_split(dataset, [int(size*.8), int(size*.1), size-int(size*.8)-int(size*.1)])
    train_loader = DataLoader(train_set, batch_size=128, num_workers=4, shuffle=True)
    
    # train
    for data, target in train_loader:
        ...
</code></pre>
<p>Probably not the most elegant approach as you could hide this logic in a custom <code>Dataset</code>, but it might just do its job.</p>
DataLoader where each batch belongs to a group

moth: <p>Hello, I have a dataset containing images of different patients. This dataset has an <code>image_id</code> and a <code>patient_id</code>. One patient can have many images. I wish to create sort of a <strong><code>DataLoader</code> where each batch only has images from a single patient.</strong> By using a <code>DataLoader</code> in the classic way each batch may contain images from different patients and my Loss function is computed per patient.</p>
<p>One way I thought to solve this is to group by the <code>patient_id</code>, iterate over groups and at each iteration create a new <code>DataLoader</code> but i don’t know if it is the correct way.</p>
<p>Could you tell me a way to load images in the way described?</p>
Unsqueeze a certain number of times depending on a condition

aRI0U: <p>Hi,<br>
I want to create a function that takes a batch of inputs <code>X</code> and a set of coeffs <code>C</code> and then multiply each element of the batch <code>X[i]</code> by the corresponding coeff <code>C[i]</code>. One has <code>X.shape = (B, N1, ..., Nn)</code> and <code>C.shape = (B)</code>.<br>
To do so, I want to unsqueeze <code>C</code> so that it is broadcastable with <code>X</code> and then return <code>C * X</code>. However I want my function to accept any shape while <code>X.size(0) == C.size(0)</code> so I cannot predict how many times I must unsqueeze the last dimension of <code>C</code>. What would be the best way to do so?</p>

thecho7: <p>You don’t need to call <code>unsqueeze()</code> so many times.<br>
Assuming X and C are like below</p>
<pre><code class="lang-auto">X = torch.rand(2, 3, 4, 5) # [B, N1, N2, N3]
C = torch.rand(2) # [B]
B = C.size(0)

C = C.unsqueeze(1) # [B, 1]
C_expand = C.expand_as(X.view(B, -1)) # [B, N1 x N2 x N3]
C_new = C.view_as(X) # [B, N1, N2, N3]
</code></pre>
<p>Now then you can do what you want</p>

aRI0U: <p>Oh I see the trick now, thanks a lot! I was so obsessed by this <code>unsqueeze</code> that I forgot I could just expand my tensor without additional memory usage…</p>
Conditional Normalization of Pytorch Tensor

Jack_Rolph: <p>Suppose I have a tensor, <code>a</code>, I wish to normalize to the mean and standard deviation along its last axis:</p>
<pre><code class="lang-auto">a.shape

&gt; torch.Size([B,C,X,Y])

</code></pre>
<p>I can achieve this using:</p>
<pre><code class="lang-auto">s_mean = torch.mean(s, axis=-1).unsqueeze(-1)
s_std = torch.std(s, axis=-1).unsqueeze(-1)
s_norm = (s-s_mean)/(s_std)
</code></pre>
<p>However, there are several entries along the axis <code>Y</code> that have mean 0 and variance 0, ergo they are all the same value.</p>
<p>I would like to only normalize the values with a non-zero mean and standard deviation and set the remaining values to zero.</p>
<p>What functions to I require to achieve this?</p>

aRI0U: <p>I hope that you managed to find a solution since it’s been 2 years you asked but I put an answer here in case someone has the same issue.</p>
<p>You can use <a href="https://pytorch.org/docs/stable/generated/torch.where.html" rel="noopener nofollow ugc">torch.where</a> to provide a value to some indices of a tensor given a condition (i.e. a bool tensor). Also you can use the kwarg <code>keepdim=True</code> in your operations to avoid having to unsqueeze just after.<br>
Concretely, something like this should solve your issue:</p>
<pre><code class="lang-python">s_mean = torch.mean(s, dim=-1, keepdim=True)
s_std = torch.std(s, dim=-1, keepdim=True)
s_norm = torch.where(s_std != 0, (s - s_mean) / s_std, 0)
</code></pre>
Why would someone zero out the parameters of a module in the constructor?

Vedant_Roy: <p>I’m reading through <a href="https://arxiv.org/pdf/2102.09672.pdf" rel="noopener nofollow ugc">Improving Denoising Diffusion Models</a>. In the implementation, the authors sometimes zero out the parameters of a module in the constructor.</p>
<p>See this link: <a href="https://github.com/openai/improved-diffusion/blob/783b6740edb79fdb7d063250db2c51cc9545dcd1/improved_diffusion/unet.py#L158" class="inline-onebox" rel="noopener nofollow ugc">improved-diffusion/unet.py at 783b6740edb79fdb7d063250db2c51cc9545dcd1 · openai/improved-diffusion · GitHub</a></p>
<pre><code class="lang-auto">        self.out_layers = nn.Sequential(
            normalization(self.out_channels),
            SiLU(),
            nn.Dropout(p=dropout),
            zero_module(
                conv_nd(dims, self.out_channels, self.out_channels, 3, padding=1)
            ),
        )
</code></pre>
<p>where</p>
<pre><code class="lang-auto">def zero_module(module):
    """
    Zero out the parameters of a module and return it.
    """
    for p in module.parameters():
        p.detach().zero_()
    return module
</code></pre>
<p>Why would someone want to do this?</p>

shivammehta007: <p>I think this is the hack from <a href="https://paperswithcode.com/method/fixup-initialization" rel="noopener nofollow ugc">FixUp Initialization</a>. Where you initialize the last layers to zero, so that they do not contribute initially to the loss term, basically a way to make gradients flow to initial layers rather than the last layer learning everything.</p>

Vedant_Roy: <p>Thanks! This makes sense. However, if the last layers are initialized as zero, then they will never update right. How do we get the last layers to learn?</p>

Vedant_Roy: <p>There’s also an issue here, where if the weights of the last layer are 0, the <code>ResBlock</code> that the module is a part of won’t do anything since it will effectively just be as skip connection. So there must be something else going on.</p>

Vedant_Roy: <p>Marking this as the solution b/c the paper authors did say this was a part of fixup initialization.</p>

shivammehta007: <p>No, they will learn, just not during the first gradient update, after the gradients will realise that they need to update these parameters to get something meaningful and learn. Also generally, you do not zero the bias and introduce a “prior” bias value which further helps in this learning.</p>
Appllying strict=False for using a modified pretrained model

Tm4: <p>I want to remove and modify the code of a preteined model and for loading this pretrained model I want to use <code>strict=False</code> in the <code>state_dict</code>. Based on the code I’m working with and also based on the pretrained model that I need to use (<code>[TimeSformer](https://github.com/facebookresearch/TimeSformer)</code>), the <code>TimeSformer</code> model load its weights from inside itself.but in my previous experiments (using the current codes) I loaded the pretrained weights in another section of the code. In my current code, I cannot find any place to apply  <code>strict=False</code> . How can I apply  <code>strict=False</code> to avoid any mismatch because of differences between modified code of the pretrained model and its weights? Any help would be appreciated.</p>
<p>This is my code that I used for my previous experiments:</p>
<pre><code class="lang-auto">class my_model(nn.Module):
    def __init__(self, pretrained=False):
        super(my_model, self).__init__()
        
        
        self.featureExtractor =feature_extractor()
        
        
        if pretrained:
            print('Loading weights...')
            weight_dict=torch.load(os.path.join('models','vid_class.pt'))

            model_dict=self.featureExtractor.state_dict()
            
            list_weight_dict=list(weight_dict.items())
            list_model_dict=list(model_dict.items())
            
            for i in range(len(list_model_dict)):
                assert list_model_dict[i][1].shape==list_weight_dict[i][1].shape
                model_dict[list_model_dict[i][0]].copy_(weight_dict[list_weight_dict[i][0]])
            
            for i in range(len(list_model_dict)):
                assert torch.all(torch.eq(model_dict[list_model_dict[i][0]],weight_dict[list_weight_dict[i][0]].to('cpu')))
            print('Loading done!')
</code></pre>
<p>But when using <code>TimeSformer</code>, it loads the weights by itself and cannot find the place for applying <code>strict=False</code> in the <code>state_dict</code>:</p>
<pre><code class="lang-auto">class my_model(nn.Module):
    def __init__(self, pretrained=False):
        super(my_model, self).__init__()
        self.featureExtractor =TimeSformer(img_size=224, num_classes=400, num_frames=8, attention_type='divided_space_time',  
                                           pretrained_model='/models/TimeSformer.pyth')
..
..
..
</code></pre>
Train and test loss curves show convergence but huge gap between them

HegZed: <p>Hello,<br>
I’m using CNN to do image regression tasks. My loss curves show convergence after ~18 epochs, but there is a huge gap between training and testing results. I have tried to tune weight decay and increase dropout, but the gap persists.<br>
Is this due to overtraining, an unrepresentative dataset, or something else?<br>
I really appreciate any suggestion you can provide.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/6555cc3e89117afdf3c2c2ad34a69993426eda00.png" data-download-href="https://discuss.pytorch.org/uploads/default/6555cc3e89117afdf3c2c2ad34a69993426eda00" title="loss_curve"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/6555cc3e89117afdf3c2c2ad34a69993426eda00_2_537x500.png" alt="loss_curve" data-base62-sha1="ess2OXTFdTZFwIsm1ske84DnQgo" width="537" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/6555cc3e89117afdf3c2c2ad34a69993426eda00_2_537x500.png, https://discuss.pytorch.org/uploads/default/original/3X/6/5/6555cc3e89117afdf3c2c2ad34a69993426eda00.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/5/6555cc3e89117afdf3c2c2ad34a69993426eda00.png 2x" data-dominant-color="FAFAF9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss_curve</span><span class="informations">773×719 55.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
First application of PyTorch an ML

sam_6: <p>I have a (relatively small) dataset, 640,000 datapoints. I am able to create a model from this which has 8 inputs and generates 4 outputs.</p>
<p>The model predicts reasonably well, but I would like to further improve the model but I am unsure how to go about setting up the layers, number of neurons and number of layers to change or improve my model.</p>
<p>Research of done on how to set up the layers, etc… is very vague and hasn’t helped shed any light on this for me.</p>
<p>Any guidance would be greatly appreciated.</p>

mtalhaubaid: <p>The architecture of the model depends upon the dataset. If your dataset is complex you need to increase the number of layers in model. This is not the hard and fast rule but it give you the idea. There are also a lot of other parameters as well which need to me adjust or added in model architecture like dropout, batch normalization, regularization, epochs, batch size, learning rate and optimizer as well. You should know what are these and how it will effect the model performances. By doing so you can add the easily improve/made you model from scratch.</p>
Is is thread safe to do `Tensor.to(device)` from multiple threads to the same GPU device?

Fei_Liu: <p>I’m reading many small files (tensors) to be loaded to the GPU memory. In this case, can I open a thread pool and use <code>.to</code> to move all of them to GPU? I have 8 GPUs in total, each will take ~2000 such tensors, and this is done in python with a <code>'multiprocessing.pool.ThreadPool(32)</code>.</p>

Fei_Liu: <p>Follow-up question: in general, is it safe to access the same GPU device through different threads but with thread local objects (still in python, e.g. using a thread pool) ? E.g. if I do <code>x = torch.rand(2, 2)</code> in two different threads and manipulate each thread’s <code>x</code> separately, will I possibly run into any errors / undefined behaviors?</p>
Pytorch memory management

oracal: <p>I begin to read pytorch source code in github to get some details about memory management when making inference. However, I don’t know the entry of related code and vert confused. Are there some tutorials or some suggestions? Thanks so much!</p>

ptrblck: <p>Could you explain a bit more what exactly you are interested in?<br>
Are you trying to understand how the GPU memory management works with the caching allocator or anything else?</p>

oracal: <p>Right. Actually , when making inference, pytorch always allocated enough memory on device only once to meet the requirement of inference , or just allocated suitable memory while inference</p>
ROC AUC plots for multi-labels

Berny: <p>Hello, I want to do roc auc plot for every class in a problem where the labels are: 0, 1, 2, 3, 4 ,5 ,6.<br>
I found this code:</p>
<pre><code>                    label_binarize(YTest, classes=[0, 1, 2, 3, 4, 5, 6])
                    n_classes = YTest.shape[1]
                    classifier = OneVsRestClassifier(
                        svm.SVC(kernel="linear", probability=True, random_state=random_state)
                    )
                    y_score = classifier.fit(XTest, YTest).decision_function(XTest)
                    fpr = dict()
                    tpr = dict()
                    roc_auc = dict()
                    for i in range(n_classes):
                        fpr[i], tpr[i], _ = roc_curve(YTest[:, i], y_score[:, i])
                        roc_auc[i] = auc(fpr[i], tpr[i])
                    fpr["micro"], tpr["micro"], _ = roc_curve(YTest.ravel(), y_score.ravel())
                    roc_auc["micro"] = auc(fpr["micro"], tpr["micro"])
</code></pre>
<p>I have some problem with svm and decision_function. Is there a better way to solve this problem?</p>

ptrblck: <p>This issue doesn’t seem to be PyTorch-related and I think <a href="https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html#sphx-glr-auto-examples-model-selection-plot-roc-py">this <code>scikit-learn</code> tutorial</a> shows the usage for a multi class use case.</p>
Custom Sampler with Larger length than Dataset

Erica_Zheng: <p>Dear All,</p>
<p>I am defining my own Sampler. Since there are <strong>duplicated</strong> sampling, so the length will be larger than<br>
<code>self.data_source</code> (Dataset). The tricky part is that I have NOT implemented <code>def __len__(self)</code>, but no error found.</p>
<p>Any suggestions? Thank you.</p>

ptrblck: <aside class="quote no-group" data-username="Erica_Zheng" data-post="1" data-topic="157170">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/erica_zheng/48/14320_2.png" class="avatar"> Erica_Zheng:</div>
<blockquote>
<p>The tricky part is that I have NOT implemented <code>def __len__(self)</code>, but no error found.</p>
</blockquote>
</aside>
<p>The <code>__len__</code> of the <code>Dataset</code> is used by the samplers to create the indices as seen <a href="https://github.com/pytorch/pytorch/blob/18e2aeae01dd8fb7b12d118828542f1321a242b6/torch/utils/data/sampler.py">here</a>.<br>
If you create a custom sampler you might skip the <code>len(data_source)</code> usage, but would need to make sure the created indices are not out-of-bounds as seen in this example:</p>
<pre data-code-wrap="python"><code class="lang-python"># ============== Standard approach ==============
class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.randn(10, 1)
        
    def __getitem__(self, index):
        return self.data[index]
    
    def __len__(self):
        return len(self.data)
    
dataset = MyDataset()
loader = DataLoader(dataset, batch_size=5)
for x in loader:
    print(x.shape)
    

# ============== Missing __len__ ==============
class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.randn(10, 1)
        
    def __getitem__(self, index):
        return self.data[index]
    
dataset = MyDataset()
loader = DataLoader(dataset, batch_size=5)

# Fails in the sampler as __len__ is expected
for x in loader:
    print(x.shape)
# TypeError: object of type 'MyDataset' has no len()

# ============== Custom Sampler ==============
class MySampler(object):
    def __init__(self, length):
        self.length = length

    def __iter__(self):
        return iter(range(self.length))

    def __len__(self):
        return self.length
    
loader = DataLoader(dataset, batch_size=5, sampler=MySampler(10))

# works
for x in loader:
    print(x.shape)
    
# ============== Wrong length passed to the sampler ==============
loader = DataLoader(dataset, batch_size=5, sampler=MySampler(20))

# fails with IndexError
for x in loader:
    print(x.shape)
# IndexError: index 10 is out of bounds for dimension 0 with size 10
</code></pre>

Erica_Zheng: <p>Very detailed explanation! Thank you!</p>
Convert_batchnorm_modules does not exist

Zebang_Shen: <p>I am trying to use Opacus to implement DP-SGD, but I cannot find the function “convert_batchnorm_modules” anywhere in the package.<br>
Cannot find answer from google results neither.<br>
Can someone please help me with this? Thanks in advance!</p>

ptrblck: <p>I think <code>convert_batchnorm_modules</code> is deprecated and replaced with <code>ModuleValidator.fix()</code> as described <a href="https://github.com/pytorch/opacus/blob/8881261c60b42139fa2ab06350915157bbd965b1/Migration_Guide.md#your-model-has-batchnorm">here</a>.</p>

Zebang_Shen: <p>Thank you ptrblck! This is exactly what I need.</p>
Saving DataLoader (RandomSampler) State

jame: <p>Suppose a dataset is given and a RandomSampler is created by</p>
<p>torch::data::samplers::RandomSampler sampler(3);</p>
<p>Afterwards a DataLoader is initialised via</p>
<p>auto loader = torch::data::make_data_loader(<br>
dataset,<br>
std::move(sampler),<br>
torch::data::DataLoaderOptions());</p>
<p>After a couple of epochs, there is no way of saving the state of the DataLoader’s internal sampler anymore.</p>
<p>I looked at “StateLessDataLoader” and “DataLoaderBase”, but since the sample_ is a private member, there is no access to its load or save functionality.</p>
<p>Might there be a way to “fast foward” the DataLoader? But calling “next()” will load samples, which is slow.</p>
<p>Thanks in advance!</p>
How to do padding at decoder side?

Gaurav_Sharma: <p>Hi there,</p>
<p>I am working on an image to seq model, where my encoder input is an image while the decoder output will going to be a sequnece.  I am familiar with the <code>pack_padded_seq</code> for the seq2seq model. But in my case, there is no seq at the encoder side.  And I am facing some problems implementing it for sequences on the decoder side. The problem is:<br>
for pack_padded_seq, we need to provide a sorted tensor of seq_lengths, but while working with the decoder, we only pass one token at a time i.e. the shape of the decoder input tensor will going to be [batch]. That means, all the seq_lens will going to be the same i.e. 1.  How will I be able to do pack_padding with this condition?</p>
<p>for e.g.:</p>
<pre><code class="lang-auto">trg = &lt;tensor of shape [len, batch]&gt;

dec_src = trg[0,:]   # [1, batch]
 
for t in range(1, trg_len):
    output, hidden, cell = self.lstm_decoder(dec_src, encoder_out, hidden, cell)     # O: [B, out]   H: [1, B, Hid]
    outputs[t]=output
    top1 = output.argmax(1)     # [batch_size]

</code></pre>
<p>Any suggestions will be appreciated! Thank you!</p>
Does torch provide softmax and matmul with different shape for each samples?

Jinyoung_Jung: <p>HI, I am working on a dynamic model,<br>
for each sample, the input has different length, and i do cross attention and get attention map with different shape along with samples.</p>
<p>for example, in a batch we have attention map shaped (4X3) for the first sample but (4X2) for the second one.<br>
and matmul following is like (4X3)X(3Xd) and (4X2)X(2Xd), different along with samples.</p>
<p>is there convenient way to do softmax and mat mul on different length in parallel?, without applying padding for having the same shape or implementing ‘for loop’ along each sample in a batch?</p>
<p>Thanks.</p>

KFrank: <p>Hi Jinyoung!</p>
<aside class="quote no-group" data-username="Jinyoung_Jung" data-post="1" data-topic="157054" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jinyoung_jung/48/45588_2.png" class="avatar"> Jinyoung_Jung:</div>
<blockquote>
<p>is there convenient way to do softmax and mat mul on different length in parallel?</p>
</blockquote>
</aside>
<p>No, pytorch does not support “ragged tensors,” that is, tensors whose<br>
slices are of differing shapes.</p>
<aside class="quote no-group">
<blockquote>
<p>without applying padding for having the same shape or implementing ‘for loop’ along each sample in a batch?</p>
</blockquote>
</aside>
<p>Either padding (the slices of) your tensors to have the same shapes,<br>
or looping over tensors of differing shapes would be the most practical<br>
approaches.</p>
<p>Best.</p>
<p>K. Frank</p>
Hard time data parallelizing a submodule

Yash_Jakhotiya: <p>I am creating a wrapper Module over a <code>base</code> nn.Module</p>
<pre><code class="lang-auto">class SiameseModel(nn.Module):
    def __init__(self, base):
        super().__init__()
        self.base = base
    
    def forward(self, img1, img2):
        output1 = self.base.forward_features(img1)
        output2 = self.base.forward_features(img2)
</code></pre>
<p>I parallelize this model during training and save it along with the base.</p>
<pre><code class="lang-auto">model = SiameseModel(base).to(device)
if torch.cuda.device_count() &gt; 1:
    model = torch.nn.DataParallel(model)

torch.save(model.state_dict(), os.path.join(save_path, "epoch_0.pt"))
torch.save(model.module.base.state_dict(), os.path.join(save_path, "base_epoch_0.pt"))
</code></pre>
<p>Now during inference, I intend to use only the base and its specific <code>forward_features</code> method, so I can’t do something like <code>base(input)</code>, but have to use <code>base.forward_features(input)</code>.</p>
<p>In an effort to parallelize the base during inference, I do something like -</p>
<pre><code class="lang-auto">base = get_base(args.base, pretrained=False) #Instantiating the base class 
base.load_state_dict(torch.load(save_path)) #loading saved base from the last block
if torch.cuda.device_count() &gt; 1:
    base = torch.nn.DataParallel(base)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
base.to(device)
base.eval()
</code></pre>
<p>But because I have to use the <code>forward_features</code> method, I eventually end up doing</p>
<pre><code class="lang-auto">embeddings = base.module.forward_features(input)
</code></pre>
<p><strong>and only 1 GPU gets used</strong>. My hunch is that using <code>.module.forward_features</code> makes it use only 1 GPU.</p>
<p>I have tried some other things, like saving the parallelized base, and loading it, but eventually have to use <code>.module.forward_features</code>. I also tried some random things like making base.forward = base.forward_features. But none seem to make use of all the GPUs.</p>
<p>What would you suggest in this case?</p>

Yash_Jakhotiya: <p>If I try reassigning the forward method before making the base parallel -</p>
<pre><code class="lang-auto">base.forward = base.forward_features
if torch.cuda.device_count() &gt; 1:
    logger.info(f"Using {torch.cuda.device_count()} GPUs!")
    base = torch.nn.DataParallel(base)
embeddings = base(input)
</code></pre>
<p>I am hit with a <code>RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument weight in method wrapper__cudnn_convolution)</code></p>
<p>Same thing if done after parallelizing,</p>
<pre><code class="lang-auto">if torch.cuda.device_count() &gt; 1:
    logger.info(f"Using {torch.cuda.device_count()} GPUs!")
    base = torch.nn.DataParallel(base)
base.module.forward = base.module.forward_features
embeddings = base(input)
</code></pre>
Batched meshgrid?

Sam_Lerman: <p>Normally, meshgrid is computed with a list of 1d tensors. I’d like to do this for a list of <em>batched</em> 1d tensors (keeping the batches independent). Is this possible?</p>
<p>Normal:</p>
<p><code>torch.meshgrid(1d Tensor, 1d Tensor, ...)</code></p>
<p>Want:</p>
<p><code>torch.meshgrid(BxD Tensor, BxD Tensor, ...)</code></p>
<p>Related: <a href="https://discuss.pytorch.org/t/compute-cartesian-product-for-batched-tensor/128121" class="inline-onebox">Compute cartesian product for batched tensor</a></p>

Sam_Lerman: <p>Equivalent to:</p>
<pre><code class="lang-auto">meshes = []
for i in range(batch_size):
     meshes.append(torch.meshgrid(tensor1[i], tensor2[i], ...))
</code></pre>
<p>And then stacking the zipped meshes along the batched dimension. Obviously this is really inefficient though.</p>
How to generate int8 model

Rajesh_Shanmugam: <p>Trying to generate int8 model instead of quantizing fp32 model. Is there any way to generate int8 model directly ?</p>

ParGG: <p>Not really, all the quantised 8bits models are actually fake quantised. Then the backend internally uses the integer representation of the model.</p>
<p>If you would generate a quantised model directly, you would have to deal with overflow. Since the result of a matrix vector product between two int8 tensors will be a int8 tensor. Whilst to have accurate results (up to a certain dimensionality of the matrix), you would need to store the result in a int32 tensor before quantising (the previous int8 result are the 8bits starting from the LSB of the int32 result).</p>