akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.utils.data.dataloader + threading get stuck sometimes

anxu: <p>Hi,</p>
<p>When I try to create two threads and one dataloader per thread, the following warning will come out from time to time:</p>
<blockquote>
<p>OMP: Warning <span class="hashtag-raw">#190:</span> Forking a process while a parallel region is active is potentially unsafe.</p>
</blockquote>
<p>Moreover, the program can <strong>sometimes</strong> get stuck during training with two threads loading data at the same time. torch.multiprocessing can work perfectly with one dataloader per process, but I have to use threading for some reason.</p>
<p>I am using PyTorch 1.4.0 and Python 3.6. How can I make dataloader + threading work without getting stuck sometimes?</p>
<p>Thanks</p>
Modulo Linear Activation

buchholzmd: <p>I am looking to compare methods for bounding the range of neural network output for regression.</p>
<p>A naive thought is to use the modulo function (extended to the reals). It appears that torch.fmod is not-differentiable and I assume torch.remainder is not as well.</p>
<p>My question is whether my assumption is correct and whether it would be possible to implement the gradient for these functions?</p>
<p>Referencing this <a href="https://math.stackexchange.com/questions/2651437/partial-of-modulo-operator-with-non-integers" rel="noopener nofollow ugc">post</a>,  it appears that the gradient is defined on non-integer quotients. Would it be possible to implement some sort of approximation of the gradient close to integer valued quotients? Would this gradient still be useful?</p>
Sum over dim=1 every 10 matrices

NadiaMe: <p>Let’s say I have a tensor of the size<br>
torch.Size([512, 5000, 14, 14])</p>
<p>and I want to sum each 10 entries in second dim, e.g<br>
sum [512, 0:10, :, :] over dim 1<br>
sum [512, 10:20, :, :] over dim 1<br>
…</p>
<p>So the resulting tensor is of size<br>
[512, 500, 14, 14]</p>
<p>Is there any way to do it efficiently without loops?</p>

albanD: <p>Hi,</p>
<p>You can first view the Tensor before doing the reduction:<br>
<code>t.view(512, 500, 10, 14, 14).sum(2)</code></p>
Resetting the learning rate, optimiser and model states while training for every fold in cross validation

bing: <p>Hi Everyone,<br>
I am trying to use cross-validation for an image classification project.<br>
The learning rate, optimiser and model states have to be reset in every fold, but all the above parameters are continuing from the previous folds state.<br>
I am not able to figure out how to resolve this error -<br>
Below is my structure for training loop -</p>
<pre><code class="lang-auto">model_transfer = EfficientNet.from_pretrained('efficientnet-b7')
n_inputs = model_transfer._fc.in_features
model_transfer._fc = nn.Linear(n_inputs, 4)
optimizer = torch.optim.SGD(model_transfer.parameters(), lr=3e-1)
criterion_transfer = nn.CrossEntropyLoss()
scheduler = ReduceLROnPlateau(
            optimizer,
            patience=4,
            threshold=1,
            factor=0.1,
            mode='max',
            verbose=True
        )

for fold in range(5):
    print("Fold: ", fold)
    partition, labels = kfold(df_train, df_test, splits, fold, stratify_columns='labels') # noqa
    train_loader = "DataLoader containing training set"
    val_loader = "DataLoader containing validation set"
    test_loader = "DataLoader containing test set"
    data_transfer = {'train': train_loader,
                     'valid': val_loader,
                     'test': test_loader
                     }
    train_model(model=model_transfer, loader = data_transfer, optimizer = optimizer, criterion = criterion_transfer,scheduler=scheduler, n_epochs = 50,checkpoint_path='./models/checkpoint_600.pt', fold_no=fold) # noqa
</code></pre>

pchandrasekaran: <p>Try using the first block of code within the for loop. It should re-initialize everything per fold.</p>
Index problem with the Dataloader with multiple indexes

Leroy_Freyler: <p>Hello,</p>
<p>i have a problem with iterating my data. I am also roughly aware of why this is happening, but I have no solution for it.<br>
My task is to pass a dataset of hyperspectral images through a Convolution Auto Encoder. Due to the fact that I am not interested in the local information I want to create a 1D Convolutional Auto encoder. My problem now is to load the latencies in the necessary form.</p>
<p>For this purpose a data set exists in the form.<br>
[18000x91] = [Pixel, Channel]</p>
<p>Now I transform this dataset into the original form of the dataset.<br>
45x91x400] = [Images, channel, pixels in the images].</p>
<p>The code for loading the data is shown below.</p>
<pre><code class="lang-python">
class SpectralDataset(Dataset):
    def __init__(self):
        xy = np.loadtxt(Dataset_Path, delimiter=",", dtype=np.float32)
        self.len = xy.shape[0]
        # pylint: disable=E1101 # Fehler in VSC muss hier hinzugefügt werden
        x = torch.from_numpy(xy[:, 2:])
        x = torch.reshape(x, (45, 400, 91))
        self.x = x.permute(0, 2, 1)
        # pylint: enable=E1101 # Fehler in VSC muss hier hinzugefügt werden

    def __getitem__(self, index):
        return self.x[index]

    def __len__(self):
        return self.len
dataset = SpectralDataset()
</code></pre>
<p>Since I want to do batch training with shuffled data sets, I also use the Dataloader function of Pytorch. But now it gives me index values, which are far beyond the range of [0 45] .<br>
The Dataloader uses index values from the original dataset, i.e. [0 18000] .</p>
<p>How can you avoid this problem?</p>

Leroy_Freyler: <p>Hello,<br>
just found my mistake.</p>
<p>The Dataloader works with the <strong>len</strong> function.I had connected it with the previous dataform. So my problem is solved.</p>
<p>The code should look like the following.</p>
<pre><code class="lang-python">class SpectralDataset(Dataset):
    def __init__(self):
        xy = np.loadtxt(Dataset_Path, delimiter=",", dtype=np.float32)
        # pylint: disable=E1101 # Fehler in VSC muss hier hinzugefügt werden
        x = torch.from_numpy(xy[:, 2:])
        x = torch.reshape(x, (45, 400, 91))
        self.x = x.permute(0, 2, 1)
        self.len = x.shape[0]
        # pylint: enable=E1101 # Fehler in VSC muss hier hinzugefügt werden

    def __getitem__(self, index):
        return self.x[index]

    def __len__(self):
        return self.len
dataset = SpectralDataset()
</code></pre>
Log_softmax and softmax

ccalafiore: <p>I am training a model with nn.LogSoftmax(). Well, I am actually using torch.nn.CrossEntropyLoss() as loss funtion which combines nn.LogSoftmax()  and nn.NLLLoss(). However, During the test phase, I want the model to output the probabilities, not the logs of probabilities. So, I would like to use nn.Softmax() for the test phase, not nn.LogSoftmax().</p>
<p>My Question: after training with nn.LogSoftmax(), can I test the model with nn.Softmax() instead of nn.LogSoftmax()? Or do I need to train the model with nn.Softmax() to be able to use nn.Softmax() during the test phase?</p>
<p>Thanks, Carmelo</p>

KFrank: <p>Hello Carmelo!</p>
<aside class="quote no-group" data-username="ccalafiore" data-post="1" data-topic="101092" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ba9def/48.png" class="avatar"> ccalafiore:</div>
<blockquote>
<p>Or do I need to train the model with nn.Softmax() to be able to use nn.Softmax() during the test phase?</p>
</blockquote>
</aside>
<p>You can do anything you want in your test phase, regardless of how<br>
you trained your model.  (Just make sure that your test-phase results<br>
mean what you want them to mean.)</p>
<p>You say you use <code>CrossEntropyLoss</code>.  The output of your model<br>
should therefore be raw-score <em>logits,</em> and these will most likely<br>
be the output of a final <code>Linear</code> layer without any subsequent<br>
activation function.  That’s fine.</p>
<p>If you now want to do something in your test phase with the actual<br>
probabilities (instead of the logits), pass the output of your model<br>
through <code>softmax()</code> to convert them.</p>
<p>(You should think through why you want probabilities in your test<br>
phase and whether or not you need them.  If <em>all</em> you want to do is<br>
convert those probabilities to integer class-label predictions, you<br>
can do that by applying <code>agrmax()</code> directly to the logits without<br>
converting them to probabilities first.)</p>
<p>Best.</p>
<p>K. Frank</p>
How to show the image of predict.py in colab

rui_zhou: <p>!python predict.py<br>
Input image filename:</p>
<p>I want to im.show the  output picture but it repeats</p>
<p>from pspnet import PSPNet</p>
<p>from PIL import Image</p>
<p>pspnet = PSPNet()</p>
<p>while True:</p>
<pre><code>img = input('Input image filename:')

try:

    image = Image.open(img)

except:

    print('Open Error! Try again!')

    continue

else:

    r_image = pspnet.detect_image(image)

    r_image.show()</code></pre>

planet_pluto: <p>Does it display the output image more than once? Can you also attach a screenshot so that it’s easier to understand?</p>
Differentiate through complex differential equation

fewagner: <p>Hi!<br>
I’m currently working an a problem that is related to a differential equation, in which a complex-valued term appears, so the solution to the differential equation will also be complex-valued. The equation looks like this:</p>
<p>y’’(t) + (A(t) + j*B(t))**2 * y(t) = 0</p>
<p>… with j being the complex unit. I want my neural network to predict the coefficient B so that my solution y satisfies some specific conditions. For that I would like to solve the differential equation with the predicted B, check the Loss of y and then backpropagate through the DE-Solver to my model.  I think this should generally work, but I don’t know how to deal with the complex values, as PyTorch doesn’t have complex numbers implemented yet. Maybe somebody has already worked on similar problems and can help with an advice?</p>

anjali411: <p>Hi <a class="mention" href="/u/fewagner">@fewagner</a> PyTorch recently added complex numbers support so you can make tensors of complex dtype:<br>
<code>torch.randn(4, 4, dtype=torch.float)</code> and perform operations on them. you can also compute compute complex derivatives, i.e. run backward on loss functions with complex valued parameters. for more reading on complex autograd, check out: <a href="https://pytorch.org/docs/stable/notes/autograd.html?highlight=complex%20autograd" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/autograd.html?highlight=complex%20autograd</a></p>
Costumized ensemble from pre-trained models

S.silvie: <p>Hi all,</p>
<p>I have trained three separate  pre -trained  models (squeeznet, resnet, alexnet) and I want to create an ensemble.  This how my code look like but when i do model.eval(), I got the error : RuntimeError: mat1 dim 1 must match mat2 dim 0<br>
Can you please help me out? Where I am doing wrong?</p>
<p>Thank you so much!</p>
<p>class MyEnsemble(nn.Module):<br>
def <strong>init</strong>(self, modelA, modelB, nb_classes=11):<br>
super(MyEnsemble, self).<strong>init</strong>()<br>
self.modelA = modelA<br>
self.modelB = modelB<br>
self.modelC= modelC<br>
# Remove last linear layer</p>
<h1>self.modelA.fc = nn.Identity()</h1>
<h1>self.modelB.classifer = nn.Identity()</h1>
<pre><code>    # Remove last linear layer
    self.modelA.classifier = nn.Identity()
    self.modelB.fc = nn.Identity()
    self.modelC.classifier = nn.Identity()
    
    # Create new classifier
    self.classifier = nn.Linear(512 + 2048 + 4096, nb_classes)
    
def forward(self, x):
    x1 = self.modelA(x.clone())  # clone to make sure x is not changed by inplace methods
    x1 = x1.view(x1.size(0), -1)
    
    x2 = self.modelB(x)
    x2 = x2.view(x2.size(0), -1)
    
    x3 = self.modelB(x)
    x3 = x2.view(x2.size(0), -1)
    
    x = torch.cat((x1,x2,x3), dim=1)
    x = self.classifier(nn.functional.relu(x))
    return x</code></pre>

ptrblck: <p>Based on the error message I guess that the number of input features in <code>self.classifier</code> doesn’t match the feature dimension for <code>x</code>.<br>
Print the shape of <code>x</code> after using <code>torch.cat</code> and before feeding it to <code>self.classifier</code> and adapt the <code>in_features</code> if necessary.</p>

S.silvie: <p>Thank you so much! I check that out and the in_features I was passing to the self_classifier was wrong! Thank you so much for your reply, now it’s working!</p>
How to solve it?

TIAN: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9bbcf7bf0103a5e8f2f6383fcf880487cae939c4.png" alt="B)MVSZL%CUD8SB~8A3N)_{7" data-base62-sha1="mdIRUg6j85hHW1yY3S68UFfBjs8" width="460" height="32"></p>

ptrblck: <p>This seems to be a <code>PIL</code> issue, which was tracked <a href="https://github.com/pytorch/vision/issues/1712">here</a> and should be already solved. Could you update <code>torchvision</code>, if you are using an older version?</p>
Calling methods after moving model to gpu

vymao: <p>When I move a model to a gpu using <code>to(device)</code>, am I still able to call methods within the model itself (like within <code>forward</code>) that aren’t gpu compatible? For example, I’d like to process a graph Data object in the <code>forward</code> method but doing so would rely on packages not meant for the gpu.</p>

ptrblck: <p>Yes, this would be possible, but depending on the code logic your code might be synchronizing at this point. e.g. if the result of this CPU operation is needed in the <code>forward</code> for some operations.</p>
[resolved] How does one create copy of tensors in PyTorch?

Brando_Miranda: <p>I was trying to create new copies (so that python was pointing at different objects and not just updating the same copy across multiple namespaces). I tried it by getting the actual numpy data from a tensor and using that to create a brand new tensor but that didn’t work. Why did it not work? How does one do this?</p>
<pre><code>&gt;&gt;&gt; y

 3  3
 3  3
 3  3
[torch.FloatTensor of size 3x2]

&gt;&gt;&gt; yv.data = torch.FloatTensor( y.numpy() )
&gt;&gt;&gt; yv
Variable containing:
 3  3
 3  3
 3  3
[torch.FloatTensor of size 3x2]

&gt;&gt;&gt; y

 3  3
 3  3
 3  3
[torch.FloatTensor of size 3x2]

&gt;&gt;&gt; yv.data.fill_(5)

 5  5
 5  5
 5  5
[torch.FloatTensor of size 3x2]

&gt;&gt;&gt; yv
Variable containing:
 5  5
 5  5
 5  5
[torch.FloatTensor of size 3x2]

&gt;&gt;&gt; y

 5  5
 5  5
 5  5
[torch.FloatTensor of size 3x2]</code></pre>

Brando_Miranda: <p>clone maybe?</p>
<aside class="quote" data-post="10" data-topic="1603">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-copy-a-variable-in-a-network-graph/1603/10">How to copy a Variable in a network graph</a> 
  </div>
  <blockquote>
    Yes this is how you want to do it.
  </blockquote>
</aside>

<p>need to check it out…</p>
<hr>
<p>this seems to be Variables but I wanted just Tensor only…</p>

Brando_Miranda: <p>ok that works:</p>
<pre><code>&gt;&gt;&gt; yv.data = y.clone()
&gt;&gt;&gt; y

 5  5
 5  5
 5  5
[torch.FloatTensor of size 3x2]

&gt;&gt;&gt; yv
Variable containing:
 5  5
 5  5
 5  5
[torch.FloatTensor of size 3x2]

&gt;&gt;&gt; yv.data.fill_(10)

 10  10
 10  10
 10  10
[torch.FloatTensor of size 3x2]

&gt;&gt;&gt; y

 5  5
 5  5
 5  5
[torch.FloatTensor of size 3x2]

&gt;&gt;&gt; yv
Variable containing:
 10  10
 10  10
 10  10
[torch.FloatTensor of size 3x2]</code></pre>

drevicko: <p>Note that indexing with a (non-zero dimensional) tensor also results in a copy.</p>
<p>(for zero-dim tensors see <a href="https://github.com/pytorch/pytorch/issues/6217" rel="nofollow noopener">this merged commit</a>)</p>

kyteinsky: <p>You should also detach the tensor from the computational graph if it requires grad else gradients will be calculated for both that may lead to OOM issues.</p>
<pre><code class="lang-python">x = torch.tensor([1.,2.,3.], requires_grad=True)
y = x.detach().clone()
y.requires_grad = True
</code></pre>
Element wise sum of batched tensors

ed-fish: <p>I have a list of tensors <code>t_list</code> and I need to get the element wise sum. I am concerned that because the tensors are in a batch that my method is incorrect.</p>
<p><strong>Code example</strong></p>
<pre><code class="lang-auto">t_list = [t1, t2, t3, t4] #where ti is a tensor 32 x 1 x 128
t_list = torch.stack(t_list) # giving 4 x 32 x 1 x 128
sum_list = sum(t_list) # result is 1 x 32 x 1 x 128
</code></pre>
<p>Is this the correct way to sum each list within the batch or would I need to specify dimensions for sum?</p>

ptrblck: <p>Your approach might work, but if you want to set a <code>dim</code> argument you could use <code>torch.sum(t_list, dim=0)</code>.</p>
How to detect roof top objects in images using pytorch

Ajeet_Ojha: <p>Hello,<br>
I am little new to  pytorch. I have see many example of  object detection using tensforflow , Keras but could not find  one which can give me some  pointers for object detection in roof top images. any  pointers , advise.</p>

nivesh_gadipudi: <p>I assume you are looking for object detecting and localising (bounding box). If so, you can have a look at this example from pytorch here, <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a> . However, this is a fine running example and consists of block diagram of how object detection and localisation works.</p>
<p>If not try to explain the scenario a bit more contextually so that community may suggest one</p>

Ajeet_Ojha: <p><a class="mention" href="/u/nivesh_gadipudi">@nivesh_gadipudi</a> , I am looking to train the model using my custom data set which has trees, houses and some other objects. I want to identify mainly trees from those images. the example you gave does not use the custom data set.  I had gone through it before. Thank you for your response.</p>

nivesh_gadipudi: <p>For a custom dataset, you can inherit <strong>torch.utils.data.Dataset</strong> class with<br>
<strong>len</strong>(returns length of your dataset), <strong>getitem</strong>(takes in the index) and you would need to use some files libraries and image ones to open and load files from your custom dataset.</p>
<p>You can check these links out:</p>
<ol>
<li><a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a></li>
<li><a href="https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel" rel="noopener nofollow ugc">https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel</a></li>
</ol>
<p>Hopefully, these links will be helpful.</p>
Error in transferring data to cuda

ritesh313: <p>Hello,</p>
<p>I am using this in my training function:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    train_mean_loss = 0
    train_mean_acc = 0
    rand_var = 0
    
    
    for i, (train_input, train_label) in enumerate(train_dataloader):
      if(train_input.device != device_available):
        print("Train Data wasn't on cuda but now is.")
        train_input = train_input.to(device_available)
        train_label = train_label.to(device_available)
        .
        .
        .
</code></pre>
<p>When I run the training loop second time (in case of any error), I get the illegal memory access error.</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: an illegal memory access was encountered
</code></pre>
<p>I am confused about these things:</p>
<ol>
<li>Does cuda throw an error if you try to push a tensor to cuda if itt’s already on cuda?</li>
<li>I faced the same issue with my model (I had to factory reset runtime to get it running), I first created an object for my model class and then pushed it to cuda but then I changed something in my model and tried to push it again on cuda, I got the same illegal memory access error.</li>
</ol>
<p>I am using google colab and a big dataset so, it’s very difficult to debug when I have to factory reset my runtime everytime if I get error during training.</p>

ptrblck: <p>CUDA operations are asynchronous so the illegal memory access might be created in the model and the next CUDA operation would run into this error and raise it.<br>
Could you rerun your script with <code>CUDA_LAUNCH_BLOCKING=1 python scripy.py args</code> and post the stack trace here, please?</p>

ritesh313: <p>I am facing this error on colab, so I just used this in the cell:</p>
<pre><code class="lang-auto">CUDA_LAUNCH_BLOCKING = 1
model = MLP_network()
model = model.to(device_available)
</code></pre>
<p>Will this work? What does this command do?</p>

ptrblck: <p>No, this won’t work and you would have to use:</p>
<pre><code class="lang-python">os.environ['CUDA_LAUNCH_BLOCKING'] = 1
</code></pre>
<p>at the beginning of your script. Make sure to restart the runtime and set this env var before PyTorch or any other library was imported otherwise this variable might not have any effect.</p>

ritesh313: <p>Alright. Thank you so much!</p>
<p>So, I am not facing the error (the one in my question) now because I fixed the problems with my training functions and this error was showing up when I tried to run the training function second time (after getting error because of some other reason). I am gonna close the topic. Thanks again <a class="mention" href="/u/ptrblck">@ptrblck</a>  for your time, I appreciate it.</p>
How to softmax Weights of torch.nn.Parameter

ameliatqy: <p>Hey guys,</p>
<p>I was wondering, how do I softmax the weights of a torch Parameter? I want to the weight my variables A and B using softmaxed weights as shown in the code below.</p>
<pre><code class="lang-auto">class RandomClass(torch.nn.module):
     def __init__():
          ...
          self._weights = torch.nn.Parameter(0.5*torch.ones(2), requires_grad=True)
          ...

     def forward():
         # --- incorrect code to softmax parameter - tried this, doesn't work ---
         self._weights.data = torch.nn.functional.softmax(self._weights.data)
         # --- incorrect code to softmax parameter - tried this, doesn't work ---

         argument_scores = (self._weights[0]*A) + (self._weights[1]*B) 
</code></pre>

albanD: <p>Hi,</p>
<p>You should recompute the value in the forward but without trying to override the original Parameter:</p>
<pre><code class="lang-python">class RandomClass(torch.nn.module):
     def __init__():
          ...
          self._weights = torch.nn.Parameter(0.5*torch.ones(2), requires_grad=True)
          ...

     def forward():
         softmaxed_weights = torch.nn.functional.softmax(self._weights)

         argument_scores = (softmaxed_weights[0]*A) + (softmaxed_weights[1]*B) 
</code></pre>
<p>Also note that you should never use <code>.data</code> in general as it most likely means that your code will lead to more bugs.</p>

ameliatqy: <p>It worked, thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"> Good point on the <code>.data</code>, will keep that in mind.</p>
Launching torch.ddp with one command

Ilia_Karmanov: <p>I typically see a ddp script being launched by submitting multiple commands (one per process), e.g.:</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --nproc_per_node=1 --nnodes=3 --node_rank=0 --master_addr=127.0.0.1 --master_port=12345
python -m torch.distributed.launch --nproc_per_node=1 --nnodes=3 --node_rank=1 --master_addr=10.47.164.34 --master_port=12345
python -m torch.distributed.launch --nproc_per_node=1 --nnodes=3 --node_rank=2 --master_addr=10.47.164.34 --master_port=12345
</code></pre>
<pre><code class="lang-auto">torch.distributed.init_process_group(backend="nccl", init_method="env://")
</code></pre>
<p>However, I am using a cluster-management system and the admin would prefer I submit only command and hence it would have to be the same command.</p>
<p>Are there any examples of maybe using mpiexec (just to submit the command) or anything else - so that master, slaves, etc are created automatically?</p>

pritamdamania87: <aside class="quote no-group" data-username="Ilia_Karmanov" data-post="1" data-topic="101009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ilia_karmanov/48/2192_2.png" class="avatar"> Ilia_Karmanov:</div>
<blockquote>
<p>However, I am using a cluster-management system and the admin would prefer I submit only command and hence it would have to be the same command.</p>
</blockquote>
</aside>
<p>I’m assuming you mean you’d like to use the same command on all the nodes to spawn the processes. You can use this command on all nodes, but we need to do something to handle the rank:</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --nproc_per_node=1 --nnodes=3 --node_rank=&lt;rank&gt; --master_addr=10.47.164.34 --master_port=12345
</code></pre>
<p>You probably need some way of passing the appropriate unique rank in. Does your cluster-management system allow for maybe using environment variables which are different per nodes? If so, you can pass in the node_rank via an environment variable.</p>
<p>Another option might be using TorchElastic (which is a fault tolerant wrapper around DDP): <a href="https://pytorch.org/elastic/0.2.1/quickstart.html" class="inline-onebox" rel="noopener nofollow ugc">Quickstart — PyTorch/Elastic master documentation</a>. TorchElastic figures out the rank automatically, so you can use the same command on all nodes.</p>
Image Display in Pytorch

Angelina_Robert: <p>Hi,<br>
Any suggestion to display an image having a dimension of (128, 3, 32, 32) in Pytorch,<br>
It generates the following error. I think batch size (128) should be removed but how?</p>
<p>Invalid shape (128, 3, 32, 32) for image data</p>

Reuben_Wenisch: <p>have you tried <code>image[n]</code> n = 1 or … to 128</p>

Angelina_Robert: <p>I m using<br>
b = images[0]<br>
axes.append( fig.add_subplot(rows, cols, idx+1) )<br>
plt.imshow(b)</p>
<p>Now problem is the shape of image[0]= 128,3,32,32<br>
So how to discard 128 (bsz).</p>

Reuben_Wenisch: <p>what is the shape of image? usually its batch =128, rgb =3, width=32, height =32. Not sure what your 5th dimension is (shape of image) could you post the error log and shape of image (image.shape in case of numpy).<br>
also try image[0][0]</p>

Angelina_Robert: <p>Shape of image is 128,3,32,32</p>

Angelina_Robert: <p>The following code snippet could help to understand:</p>
<p>def train(train_loader, model, criterion, optimizer, epoch ):</p>
<pre><code>model.train()


losses = AverageMeter()

rows = 2
cols = 2
fig=plt.figure()
axes = []
for idx, (images, labels) in enumerate(train_loader):
    data_time.update(time.time() - end)

                               
  
    print('.........Size of image before unsqueeze : ',images[0].shape)
    b = images[0].reshape(3,32,32)
    npimg = b.numpy()
  
    axes.append( fig.add_subplot(rows, cols, idx+1) )
   
    plt.imshow(npimg)
plt.show()
exit()
</code></pre>
<p>The error is:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1cf91860c82cc6dd0e568a673f26d173816ec47e.png" alt="image" data-base62-sha1="48j3iDiyhZzD2TFxjhdkI01dVVA" width="628" height="43"></p>

BaruchG: <p>The <code>image[0]</code> should be a single item, not a batch of items.  If the shape of that is <code>[128,3,32,32]</code> then that would be the shape of your data.  If you print out <code>image.shape</code> you will get the shape of the data in the batch.  If you would like to change the shape of your data you should probably look at the construction of the <code>__getitem__</code> in your dataset and see what the shape of the data that is being put in there.</p>

ivan-jgr: <p>If <code>image[0]</code> shape is <code>(128, 3, 32, 32)</code> then you have another dimension and each batch has the shape <code>(batch_dim, 128, 3, 32, 32)</code> i.e. every batch has n  <code>128x3x32x32</code>-dimensional Tensors.</p>
<p>Are you sure this is what you want?</p>
<p>If so, you can plot <code>image[0, i, ...]</code> for <code>i = 0, 1, ..., 127</code></p>
<p>And remember that if you want to plot a 3-channel image with matplotlib you need it in shape <code>(x, y, channels)</code> e.g.</p>
<pre><code class="lang-auto"># First image of the first batch
img_plot = image[0, 0, ...].permute(1, 2, 0)
plt.imshow(img_plot)
</code></pre>
Time limit to accept/reject slack channel requests

jgbradley1: <p>Is there a time limit when we should consider a request to join the slack channel as denied? I imagine the channel receives a lot of requests and it’s hard to manage…so is there a better more automated way to join?</p>
<p>While I don’t use this forum for general help (there are plenty of other resources available), I’ve submitted two PR’s to the torchvision repo and would like to discuss future enhancements.</p>
Gradient through black box iterative algorithm

Oliver_Woodford: <p>Some recent works such as <a href="https://papers.nips.cc/paper/8358-deep-equilibrium-models.pdf" rel="noopener nofollow ugc">Deep Equilibrium Models</a> use a black-box iterative algorithm that converges to some equilibrium state, and compute gradients at the equilibrium point, without back-propping through all the (arbitrary number of) iterations. What support is there for this in PyTorch?</p>

albanD: <p>Hi,</p>
<p>It depends how the back box is implemented.<br>
If it is just pytorch code, then autograd can differentiate through the iterations (though it might be slow).<br>
If you have some properties of the equilibrium that allow you to write <code>dL/dinp = f(dL/dout)</code>, then you can use a custom autograd Function to tell the autograd how to differentiate this.</p>
Intermediate results in Checkpoint

gnadaf: <p>Can we save intermediate results in checkpoint?<br>
Intermediate results are important to save in checkpoint?</p>

ptrblck: <p>Which intermediate results do you mean?<br>
Also, do you mean <code>torch.utils.checkpoint</code> or creating a checkpoint by serializing the model?</p>

gnadaf: <aside class="quote no-group" data-username="gnadaf" data-post="1" data-topic="100691">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/cdc98d/48.png" class="avatar"> gnadaf:</div>
<blockquote>
<p>esults are important to save in checkpoint</p>
</blockquote>
</aside>
<p>torch.save()</p>
<p>Does it save only learnable parameters or intermediate results also?</p>

ptrblck: <p>Could you explain, what you mean by “intermediate results”?</p>

gnadaf: <p>For example, in the transformer model <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a></p>
<p>outputs are calculated in forward pass, and layer weights and bias are intermediate results.</p>
<p>correct me if I am wrong</p>

ptrblck: <aside class="quote no-group" data-username="gnadaf" data-post="5" data-topic="100691">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/cdc98d/48.png" class="avatar"> gnadaf:</div>
<blockquote>
<p>outputs are calculated in forward pass, and layer weights and bias are intermediate results.</p>
</blockquote>
</aside>
<p>The <code>weight</code> and <code>bias</code> would be parameters and I wouldn’t call them intermediate results.<br>
I think the intermediate activations could be called intermediate results, but I’m still unsure if that’s really what you are looking for as I haven’t heard this naming so far.<br>
By intermediate activations I mean e.g.:</p>
<pre data-code-wrap="python"><code class="lang-python">def forward(self, x):
    intermediate_act1 = self.layer1(x)
    intermediate_act2 = self.layer2(intermediate_act1)
    ...
</code></pre>

gnadaf: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, now I got clarification</p>
How can I get model's gradients in two parts?

ege-erdogan: <p>I have a neural network divided into two parts. One part computes the first n layers and the other part computes the rest of the layers.</p>
<p>I can obtain the gradients from a single model this way:</p>
<pre><code class="lang-auto">grads = torch.autograd.grad(loss, model.parameters())
</code></pre>
<p>The thing is I need the gradients of the two parts separately. The main issue I am facing seems to be continuing the first part’s gradients backwards from the second part.</p>
<p>I would appreciate any help as to how I can achieve this. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <p>Hi,</p>
<p>I am not sure to understand the issue. Can you extract the corresponding gradients from <code>grads</code>?<br>
Or provide only the subset of the parameters you are interested in to <code>autograd.grad()</code>?</p>

ege-erdogan: <p>Hi, thanks for the quick reply.</p>
<p>Say I have a single model and want to obtain gradient values from it. I can do it as I showed in main post general.</p>
<p>Now I have the model as two nn.Module instances, one computing the first part and one computing the second part.</p>
<p>I want to obtain the gradient values from both parts and combine them. Bit of a weird problem but should be possible in some way.</p>

albanD: <p>If your full model has <code>model.part1</code> and <code>model.part2</code>, then you can do:</p>
<pre><code class="lang-auto">grad_part1 = torch.autograd.grad(loss, model.part1.parameters(), retain_graph=True)
grad_part2 = torch.autograd.grad(loss, model.part2.parameters())
</code></pre>
<p>Does that match what you want?</p>
Performance of IF statements in Pytorch

iordanis: <p>I use this toy example to measure performance of if statements in the forward loop.<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20201027-085602-RC00_339240693" class="site-icon" width="16" height="16">
      <a href="https://colab.research.google.com/drive/1M6NfyPd_acnvdApgzsWfjewFxgMcY_DE?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://colab.research.google.com/drive/1M6NfyPd_acnvdApgzsWfjewFxgMcY_DE?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>At least for XLA devices (such as in COLAB) when the conditional statement is fixed the performance doesn’t seem to be affected (e.g. model properties). However when the conditional statement is highly dependant on input the performance is affected.</p>
<p>Are conditions handled in GPU or the code is optimized for GPU? Is the optimization and conversion of the code taking all of that into account?</p>
<p>Is there a general rule for code level optimization for IF statements or it doesn’t really matter?</p>

albanD: <p>Hi,</p>
<p>The main problem with conditionals is that they are handled on the python side and so the values needs to be on the CPU.<br>
So if you use an accelerator like GPU or TPU, the CPU has to wait for the value to be computed and then it can go through the conditional and continue the execution. This is sub-optimal but cannot really be worked around.</p>
<p>cc <a class="mention" href="/u/ailzhang">@ailzhang</a> there might be other details about TPU that I don’t know about?</p>
Parameter passing to registered linear module without bias

offset-null1: <p>Hey, I’m a beginner. I want to compute linear unit x dot w.T without bias. I followed this way of instantiation from tutorial:</p>
<p>struct Net : torch::nn::Module {<br>
Net(int64_t N, int64_t M)<br>
: linear(register_module(“linear”, torch::nn::Linear(N, M)))<br>
{ }</p>
<p>But according to this in Functions.h:</p>
<p>static inline Tensor linear(const Tensor &amp; input, const Tensor &amp; weight, const Tensor &amp; bias) {<br>
static auto table = globalATenDispatch().getOpTable(“aten::linear(Tensor input, Tensor weight, Tensor? bias=None) -&gt; Tensor”);<br>
return table-&gt;getOp&lt;Tensor (const Tensor &amp;, const Tensor &amp;, const Tensor &amp;)&gt;(at::detail::infer_backend(input), at::detail::infer_is_variable(input))(input, weight, bias);<br>
}</p>
<p>It doesn’t let me bypass the parameter bias. I wonder if I’m implementing it wrong? Please, can anyone tell me the approach for the same?</p>

albanD: <p>Hi,</p>
<p>The functions implemented in <code>Functions.h</code> is the “functional” version that implements the forward, not the nn layer that you create with <code>torch::nn::Linear()</code>.<br>
You can see the class doc <a href="https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_linear.html?highlight=nn%20linear#_CPPv4N5torch2nn6LinearE">here</a> and the options doc (that show to to disable bias) <a href="https://pytorch.org/cppdocs/api/structtorch_1_1nn_1_1_linear_options.html#structtorch_1_1nn_1_1_linear_options">here</a></p>

offset-null1: <p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Is nn.relu() more computationally heavy than F.relu()?

Huy_Ngo: <p>As far as I understand, nn.relu() is a layer that has weights and bias whereas F.relu is just a activation function. Doesn’t that make nn.relu() a bit more computationally heavy than F.relu because optimizer has to update the redundant weights and bias for that layer too?</p>

albanD: <p>Hi,</p>
<p><code>nn.ReLU()</code> is a layer, but it has not weights or bias.<br>
The two are exactly the same.</p>
<p>The version as a nn.Module is convenient to be able to add it directly into a <code>nn.Sequential()</code> construct for example.<br>
The functional version is useful when you write a custom forward and you just want to apply a relu.</p>

Huy_Ngo: <p>Thank you for your answer.</p>

gtimothee: <p>Be warned, Google shows " <strong>relu</strong>() is a layer that has weights and bias whereas <strong>F</strong>. <strong>relu</strong> is just a activation function." instead of <a class="mention" href="/u/huy_ngo">@Huy_Ngo</a> s  answer to my research “nn.ReLU or F.relu ?” so if someone does not click on the page he/her will see a bad answer</p>
How to define and load reduced precision LibTorch model?

abhinavkulkarni: <p>Hi,</p>
<p>Let us say I have a simple model as follows:</p>
<pre><code class="lang-cpp">auto sequential = nn::Sequential(
  nn::Linear(4, 4),
  nn::ReLU(),
  nn::GroupNorm(1, 4),
  nn::Linear(4, 4)
);

sequential.to(torch::kFloat16);
torch::load(sequential, "./sequential-fp16.pt");
</code></pre>
<p>Instead of defining model (with dtype <code>torch::kFloat32</code> by default) and converting it to <code>torch::kFloat16</code>, how do I go about defining it as a <code>torch::kFloat16</code> model in the first place?</p>
<p>I have a  large model and would like to avoid large (fp32) memory allocation before whittling it down to fp16 and loading.</p>
<p>Thanks!</p>
Download link doesn't work to download binaries from https://pytorch.org/

Paulius_Tumas: <p>Hi, tried to download stable (1.7.0) PyTorch release for Linux,  LibTorch(C++/Java) package with Cuda 11.0 and download link doesn’t work (<a href="https://download.pytorch.org/libtorch/cu110/libtorch-shared-with-deps-1.7.0.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cu110/libtorch-shared-with-deps-1.7.0.zip</a>), however if I choose Cuda 10.2 it does. Could anyone renew the link?</p>

ptrblck: <p><a class="mention" href="/u/seemethere">@seemethere</a> do you know what might have gone wrong?</p>
Can we build PyTorch extensions for c++17?

youben: <p>Hi folks,</p>
<p>I’m trying to build an extension that uses another library that was developed using c++17.</p>
<p>For now I’m just including the header of that library and the build process is complaining a lot, I tried to compile it myself by changing ‘-std=c++11’ to ‘-std=c++17’ and the compilation went fine, however, I’m wondering if it’s possible to build extensions with c++17 and if it’s the case then how to use the torch.utils.cpp_extension module to configure the build?</p>

oli4100: <p>Did you find a solution for this? I am experiencing this issue too - I need some c++17 features but can’t figure out how to supply the -std=c++17 flag to the compiler. As far as I understand the BuildExtension only allows to supply additional compiler flags, but it does not allow me to change default compiler flags.</p>

youben: <p>What I did was to pass extra compile args to use C++17, you can check my setup.py file <a href="https://github.com/youben11/th-seal/blob/master/setup.py" rel="noopener nofollow ugc">here</a> for an example. However, I stopped at the PoC level, and didn’t try to build anything realistic out of it.</p>

oli4100: <p>Thanks, very helpful to see your setup.py - it seems I should have implemented it slightly different. Using the flag as such does get me a bit further into compiling with c++17 but there are still some errors.</p>
Post_training static quantization

HUSTHY: <p>when i do static quantization in BERT like this code:</p>
<p>quantized_model = model_class.from_pretrained(args.model_name_or_path, from_tf=bool(".ckpt" in args.model_name_or_path),<br>
config=config, cache_dir=args.cache_dir if args.cache_dir else None, )<br>
quantized_model.eval()<br>
quantized_model.qconfig = torch.quantization.default_qconfig<br>
print(‘quantized_model.qconfig’,quantized_model.qconfig)</p>
<pre><code># #for X86 architectures Quantizes weights on a per-channel basis
# quantized_model.qconfig = torch.quantization.get_default_qconfig('fbgemm')
# print('quantized_model.qconfig', quantized_model.qconfig)

torch.quantization.prepare(quantized_model,inplace=True)

train_dataset = load_and_cache_examples(args, args.task_name, tokenizer, data_type='train')
args.train_batch_size = args.per_gpu_train_batch_size * max(1, args.n_gpu)
# train_sampler = RandomSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset)
train_sampler = SequentialSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset)
train_dataloader = DataLoader(train_dataset, sampler=train_sampler, batch_size=args.train_batch_size,
                              collate_fn=collate_fn)
with torch.no_grad():
    for step, batch in enumerate(tqdm(train_dataloader,desc='post-training static quantization')):
        inputs = {"input_ids": batch[0], "attention_mask": batch[1], "labels": batch[3]}
        outputs = quantized_model(**inputs)
        loss = outputs[0]  # model outputs are always tuple in pytorch-transformers (see doc)
        break
print('convert quantized model:')
# torch.quantization.convert(quantized_model, mapping={nn.Linear: nnq.Linear}, inplace=True)
torch.quantization.convert(quantized_model, inplace=True)
print('static quantized model')
print(quantized_model)

      
print_model_size(quantized_model)
t1 = time.time()
      
result = evaluate(args, quantized_model, tokenizer, prefix=prefix)
t2 = time.time()
</code></pre>
<p>an error occur:<br>
Could not run ‘quantized::layer_norm’ with arguments from the ‘CPU’ backend. ‘quantized::layer_norm’ is only available for these backends: [QuantizedCPU]<br>
how can i solve this problem?</p>
How to make weights into leaf node after using an assignment operation on them

Alexander_Soare: <p>I’m adapting a 10-class classification model to an 11-class classification model where I add one extra class. As well as keeping the prior layer weights, I want to make use of as much of the final classifier weights as possible, so I do something like this:</p>
<pre><code class="lang-auto">def make_backbone(self, load=''):
    self.backbone = SFNetV1()
    if len(load):
        self.backbone.load(load)

    # replace fully connected layer 
    if len(load):
        prior_weight = self.backbone.fc1.weight
        prior_bias = self.backbone.fc1.bias
    prior_in_features = self.backbone.fc1.in_features
    prior_out_features = self.backbone.fc1.out_features
    
    # add "empty" token
    self.backbone.fc1 = nn.Linear(prior_in_features, prior_out_features+1)

    print(self.backbone.fc1.weight.is_leaf)
    print(prior_weight.is_leaf)

    # reuse whatever weights and biases we still can
    if len(load):
        self.backbone.fc1.weight[:prior_out_features, :] = prior_weight
        self.backbone.fc1.bias[:prior_out_features] = prior_bias

    print(self.backbone.fc1.weight.is_leaf)
</code></pre>
<p>This outputs</p>
<pre><code class="lang-auto">True
True
False
</code></pre>
<p>So my problem is that last <code>False</code>. So how do I “reset” <code>self.backbone.fc1.weight</code> to be a leaf node (also <code>bias</code>)?</p>
<p>Bonus side question: Is there a better way to do what I’m trying to do?</p>

ptrblck: <p>You can wrap the assignment operation into a <code>with torch.no_grad()</code> block to make sure Autograd doesn’t record this copy as a differentiable operation:</p>
<pre><code class="lang-python">fc1 = nn.Linear(10, 10)

prior_weight = fc1.weight
prior_bias = fc1.bias
prior_in_features = fc1.in_features
prior_out_features = fc1.out_features

# add "empty" token
fc2 = nn.Linear(prior_in_features, prior_out_features+1)

print(fc1.weight.is_leaf)
print(prior_weight.is_leaf)

# reuse whatever weights and biases we still can
with torch.no_grad():
    fc2.weight[:prior_out_features, :] = prior_weight
    fc2.bias[:prior_out_features] = prior_bias

print(fc2.weight.is_leaf)

out = fc2(torch.randn(1, 10))
print(out.shape)
out.mean().backward()
print(fc2.weight.grad)
</code></pre>
Selective element-wise efficient multiplication?

Adex: <p>I have two matrices of sizes (32, 512, 7,7) and (32, 512)  respectively, where 32 is the batch size and 512 is the channel size. Let us call them A and B.<br>
Now what I need to do is this:</p>
<pre><code class="lang-auto">        for i in range(batch_size):
            for j in range(channel_size):
                A[i,j,:,:]=A[i,j,:,:]*B[i,j]
</code></pre>
<p>In other words, I want to have each elements of the 7x7 matrix (of i th batch and j th channel) multiplied by the corresponding value from B matrix’s corresponding scalar for i th batch and j th channel number.</p>
<p>Using for loops becomes higly inefficient when I have to increasing batch size.<br>
Is there any other way of doing the same?</p>
<p>Any help will be appreciated. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

tom: <p>Broadcasting: A *= B[:, :, None, None]<br>
Note that this (like your version) changes A in place, but the usual * will work similarly.</p>
<p>Best regards</p>
<p>Thomas</p>

Adex: <p>Thanks <a class="mention" href="/u/tom">@tom</a> for you reply. Is there be any other solution to this problem? I have shown a simple example here. My actual implementation is very much similar, except it requires out of place implementation (C=A * B[:, :, None, None]). As I have seen that the out place implementation of the solution given by <a class="mention" href="/u/tom">@tom</a>, in my case is highly memory inefficient for a batch size of 64. The error messege is:</p>
<p><code>RuntimeError: CUDA out of memory. Tried to allocate 256.00 GiB (GPU 0; 23.65 GiB total capacity; 2.12 GiB already allocated; 20.16 GiB free; 2.20 GiB reserved in total by PyTorch)</code></p>

tom: <p>Well, a single for loop might be a good compromise.</p>