akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to visualize CNN filter?

mbp49: <p>I wanna visualize filters from model.conv1 on the final epoch. Thanks. My CNN network looks like this.</p>
<p>class CNN(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(CNN, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(1, 40, 5, 1)<br>
self.conv2 = nn.Conv2d(40, 80, 5, 1)<br>
self.fc1 = nn.Linear(4<em>4</em>80, 500)<br>
self.fc2 = nn.Linear(500, 10)</p>
<pre><code>def forward(self, x):
    x = F.relu(self.conv1(x))
    x = F.max_pool2d(x, 2, 2)
    x = F.relu(self.conv2(x))
    x = F.max_pool2d(x, 2, 2)
    x = x.view(-1,4*4*80)
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return F.log_softmax(x, dim=1)</code></pre>

ptrblck: <p>You can directly access the filters via:</p>
<pre><code class="lang-python">filters = model.conv1.weight
</code></pre>
<p>and then visualize it with e.g. <code>matplotlib.pyplot.imshow</code>.<br>
Note that filters might be multi-dimensional, so you might need to plot each channel in a separate subplot.<br>
In your model, <code>self.conv1</code> contains one input channel, so you should be able to visualize each filer next to each other.</p>

mbp49: <p>Thank you! It works. I tend to do this：</p>
<p>filters = model.conv1.weight<br>
for i in range(40):<br>
torchvision.utils.save_image(filters[i],"\image"+str(i)+".png")</p>
<p>And I got 40 .pngs. I find the pics are too smaill. Is there any way to make it bigger? Thanks.</p>

ptrblck: <p>You could interpolate the filters first e.g. via <code>F.interpolate(..., mode='nearest')</code>.<br>
If you specify <code>mode='nearest'</code>, it’ll make sure to repeat the values instead of e.g. interpolate them linearly.</p>
<p>Alternatively you could also use <code>matplotlib</code> or <code>PIL</code> to store the filters, which would provide an argument for the desired shape.</p>

TGI_Fernando: <p>You can see a complete implementation here</p>
<p><a href="https://github.com/tgi25/pytorch/blob/master/Lesson10%20-%20LeNet.ipynb" rel="nofollow noopener">LeNet</a></p>
<p>Cheers.</p>
How can i perform cross validation with this training process

Muhammad_Izaz: <p>lbls=[]<br>
pred=[]<br>
criterion = nn.CrossEntropyLoss()<br>
optimizer = torch.optim.Adam(model.parameters())<br>
num_epochs=10<br>
<span class="hashtag">#total_step</span> = len(train_loader)<br>
total_step = len(train_data)<br>
loss_list = []<br>
acc_list = []<br>
for epoch in range(num_epochs):<br>
for i ,(images, labels) in enumerate(train_data):<br>
# Run the forward pass<br>
images = images.unsqueeze(1).float()<br>
outputs = model(images)<br>
loss = criterion(outputs, labels)<br>
loss_list.append(loss.item())</p>
<pre><code>    # Backprop and perform Adam optimisation
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
#optimizer update the weight parameters to minimize the loss function

    # Track the accuracy
    total = labels.size(0)
    _, predicted = torch.max(outputs.data, 1)
    correct = (predicted == labels).sum().item()
    acc_list.append(correct / total)
    lbls+=labels
    pred+=predicted
    if (i + 1) % 36 == 0:
        print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}, Accuracy: {:.2f}%'
              .format(epoch + 1, num_epochs, i + 1, total_step, loss.item(),
                      (correct / total) * 100))</code></pre>

ptrblck: <p>You could split the dataset before training e.g. via <code>random_split</code> or better using <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html"><code>sklearn.model_selection.KFold</code></a> and train each split separately.<br>
Alternatively, <a href="https://github.com/skorch-dev/skorch">skorch</a> provides a higher-level API for PyTorch, which is sklearn-compatible.</p>
Modifying and integrating LSTM layer

prateeth: <p>Hello, I want to modify the LSTM with a custom gate. I understand how to modify the LSTM cell. What I don’t understand is how would the back-propagation go through this modified cell, without me integrating it back with the pytorch LSTM layer? If it doesn’t, how do I integrate my changes into the original code?</p>

sharvil: <p>If you’re written a <code>forward</code> method using the Python API, you don’t really have to do much. Autograd will take care of backprop for you.</p>
Softmax assigns equal values to all classes

hamzarawal: <p>Hi, I am new to PyTroch. I am facing an issue where when I apply softmax to predicted probabilities, all the classes are assigned the same probability. For example for a 9 class problem, the output for each class is 0.111111. This results in a constant Cross entropy loss, no matter what the input is.<br>
Although when I take argmax of these same probabilities, the predicted classes are correct and the network has more than 90% accuracy. The individual predicted probability values are very small.</p>

KFrank: <p>Hello Hamza!</p>
<aside class="quote no-group" data-username="hamzarawal" data-post="1" data-topic="77684" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hamzarawal/48/23091_2.png" class="avatar"> hamzarawal:</div>
<blockquote>
<p>when I apply softmax to predicted probabilities, all the classes are assigned the same probability. For example for a 9 class problem, the output for each class is 0.111111.</p>
</blockquote>
</aside>
<p>This sounds odd and is well worth investigating.</p>
<p>Could you run your system and print out the the values <em>before</em> they go<br>
into <code>softmax()</code>?  Then make a short, self-contained, runnable script<br>
that uses those values to illustrate your issue.</p>
<p>I have in mind something like this:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

my_values = torch.FloatTensor (&lt;my_nine_values_that_I_printed_out&gt;)
my_min, my_argmin = my_values.min (dim = 0)
print  (my_min, my_argmin)
my_max, my_argmax = my_values.max (dim = 0)
print  (my_max, my_argmax)
print  (my_max - my_min)

my_values = torch.nn.functional.softmax (my_values, dim = 0)
my_min, my_argmin = my_values.min (dim = 0)
print  (my_min, my_argmin)
my_max, my_argmax = my_values.max (dim = 0)
print  (my_max, my_argmax)
print  (my_max - my_min)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

hamzarawal: <p>Thanks K. Frank, actually the problem was with the default floating point precision. When I print the values using .item() they are indeed different but very minutely. The predicted probability values are also very small.</p>
Linear on Sequential returns extra dimensions

Markus_Hiltunen: <p>As part of an assignment I constructed a simple MLP with 2 hidden, 1 output layer into a Sequential()</p>
<p>In one of the tests, it gets 1D vector [4] as input, but instead of returning [3] length output, it returns [4, 3]</p>
<p>What <em><strong>on earth</strong></em> causes this? The same construct returns 1D tensors for 1D tensors in earlier tests.</p>
<p>This is on pytorch.</p>
<blockquote>
<p>self.a_net = nn.Sequential(<br>
nn.Linear(n_node_features*2,96),<br>
nn.ReLU(),<br>
nn.Linear(96,96),<br>
nn.ReLU(),<br>
nn.Linear(96,n_edge_features)<br>
)</p>
</blockquote>

KFrank: <p>Hi Markus!</p>
<aside class="quote no-group" data-username="Markus_Hiltunen" data-post="1" data-topic="77780" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/markus_hiltunen/48/16474_2.png" class="avatar"> Markus_Hiltunen:</div>
<blockquote>
<p>it gets 1D vector [4] as input, but instead of returning [3] length output, it returns [4, 3]</p>
</blockquote>
</aside>
<p>Pytorch models always work on <em>batches</em> of samples.  If you<br>
wish to pass a single length-4 sample to a network, you have<br>
to package it as a batch of length-4 samples with <code>nBatch = 1</code>.</p>
<p>That is, for this example, your input should have a shape of<br>
<code>[1, 4]</code> (not just <code>[4]</code>).  A pytorch model will interpret an input<br>
of shape <code>[4]</code> as a batch of 4 single scalar samples, and then<br>
will return a batch of 4 outputs having shape <code>[4, nOutput]</code>.</p>
<p>I would, however, have expected you to get some sort of<br>
dimension-mismatch error (assuming that your network<br>
was expecting length-4 samples), rather than having the<br>
network accept (a batch of 4) single scalar samples.</p>
<p>Best.</p>
<p>K. Frank</p>
Normalizing weights of convolution filters

boucherm: <p>Hi,</p>
<p>I would like to normalize ( L1 norm ) the weights of my convolutional filters. Is it possible to do this for torch.nn.Conv2d ? If so, would you have some hints on the “path” I should take ?</p>
<p>Thank you</p>

boucherm: <p>It might be as simple as that:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class NormedConv( nn.Conv2d ):

    def normalize( self ):
        with torch.no_grad():
            for f in range( 0, self.weight.shape[0] ):
                s                    = torch.sum( self.weight[f,:,:,:] )
                self.weight[f,:,:,:] = torch.div( self.weight[f,:,:,:], s )
</code></pre>

M-Abdallah: <p>I have tried the similar approach :</p>
<pre><code class="lang-auto">for g in range(0, self.weight.shape[0]):
            for k in range(0, self.weight.shape[1]):
                mid_x, mid_y = int(
                    self.weight.shape[3]/2), int(self.weight.shape[2]/2)
                s = torch.sum(self.weight[g, k, :, :])
                self.weight[g, k, mid_y, mid_x] = self.weight[g, k, mid_y, mid_x] - s
</code></pre>
<p>when I started the training loop I receive this error:</p>
<p>File “train/train.py”, line 127, in <br>
opt = torch.optim.Adadelta(model.parameters(),weight_decay=1e-6)<br>
File “/home/mohamed/anaconda3/envs/cvPyt/lib/python3.6/site-packages/torch/optim/adadelta.py”, line 36, in <strong>init</strong><br>
super(Adadelta, self).<strong>init</strong>(params, defaults)<br>
File “/home/mohamed/anaconda3/envs/cvPyt/lib/python3.6/site-packages/torch/optim/optimizer.py”, line 51, in <strong>init</strong><br>
self.add_param_group(param_group)<br>
File “/home/mohamed/anaconda3/envs/cvPyt/lib/python3.6/site-packages/torch/optim/optimizer.py”, line 202, in add_param_group<br>
raise ValueError(“can’t optimize a non-leaf Tensor”)<br>
ValueError: can’t optimize a non-leaf Tensor</p>
<p>any help?</p>
CrossEntropy but for values lower than 1

Jaliborc: <p>I want to train a network that predicts the typical single-class classification, normally optimized using CrossEntropyLoss, but for which the ouput of the predicted class should try to be specific values &lt;= 1.</p>
<p>An example: for a given training sample X of class 1, the intended predicted ouput might not be [1 0 0], but specifically [0.8 0 0]. X2 of class 3 might have an intended predicted ouput of [0.6 0 0].</p>
<p>I’m thinking of making a custom modified version of crossentropy loss to fit these needs. Before I try to, does anything jump to mind that could be appropriate?</p>

ptrblck: <p>This sounds like label smoothing and you could maybe use the implementation from <a href="https://discuss.pytorch.org/t/computing-softmax-cross-entropy-for-smoothed-labels/19496">this topic</a>.<br>
Let me know, if that would work for you.</p>
Data loader will affect the random mechanism even with fixed seed

zhujiangang: <p>They said the dataloaders are using random numbers and thus change the state of rng generators.<br>
So if the validation data is not loaded by DataLoader but something iterable implemented by myself, the training loss (with only One training DataLoader) will be different from the model that the validation data is loaded by the DataLoader (Two DataLoader, one for training, another for validation)? If so, what is the insight behind such design?</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/36998" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/36998" target="_blank" rel="nofollow noopener">Data loader will affect the random mechanism even with fixed seed</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-04-21" data-time="13:54:59" data-timezone="UTC">01:54PM - 21 Apr 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-04-21" data-time="20:13:32" data-timezone="UTC">08:13PM - 21 Apr 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/zhujiangang" target="_blank" rel="nofollow noopener">
          <img alt="zhujiangang" src="https://avatars2.githubusercontent.com/u/3435057?v=4" class="onebox-avatar-inline" width="20" height="20">
          zhujiangang
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
Data loader will affect the random mechanism even with fixed seed, which will make the code unable to reproduce the...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Increase indice of tensor to specific size

JvanderLinden: <p>Hi,<br>
I have a tensor of [20, 3, 32, 32]<br>
I want to increase the 3 to 64 ([20,64,32,32], where 20 is the batch size). I tried the repeat function. But that only gives me  63 or 66, because with repeat you can only tile (multiply) indices.<br>
How can I solve this?<br>
Thanks!</p>

ptrblck: <p>How would you like to increase the channel dimension?<br>
As you’ve explained, <code>extend</code> or <code>repeat</code> won’t be possible.<br>
If you want to increase is in a “trainable” way, you could use a conv layer with a kernel size of <code>1x1</code> and <code>in_channels=3</code>, <code>out_channels=64</code>.</p>
How can I install torchvision nighly?

WaterKnight: <p>I need to install vision nightly. How can i install it?</p>
<p>!pip install git+https://github.com/pytorch/vision.git@nightly is not working!</p>

ptrblck: <p>Answered <a href="https://discuss.pytorch.org/t/pretrained-deeplabv3-resnet50-coco-is-not-supported-as-of-now/77240/9">here</a>.</p>
Train Pytorch Model on multi-TPU V3 device based TPU Pod

akashs: <p>I want to train a large model on a TPU V3 Pod with 5 TPU devices. I am very novice on TPU. I already code a model which I train on multi-gpu (4 V100) using DataParallel. I found DataParallel is very easy to incorporate. I have a couple of concerns to train this model on google cloud TPU:</p>
<ul>
<li>Can I train the same learning with <strong>DataParallel</strong>  on could TPU V3 device with 5 TPU ? or do need to do any modifications except changing the library to xla?</li>
<li>Should I use <strong>DataParallel</strong>  or  <strong>DistributedDataParallel</strong> to train the model on TPU Pod?</li>
<li>Does anyone have any experience with pythorch-lightning with multi-tpu device TPU Pod?</li>
</ul>
<p>Sorry for the novice level questions. Any types of resources, suggestions will be a great help.</p>

mrshenli: <p>cc <a class="mention" href="/u/ailzhang">@ailzhang</a> for TPU questions <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

pritamdamania87: <blockquote>
<ul>
<li>Should I use  <strong>DataParallel</strong>  or  <strong>DistributedDataParallel</strong>  to train the model on TPU Pod?</li>
</ul>
</blockquote>
<p>General guidance fo DataParallel vs DistributedDataParallel: <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#comparison-between-dataparallel-and-distributeddataparallel" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>
Backprop via layers that did not compute forward pass

pozimek: <p>By default Pytorch uses autograd to update layer weights based on their contribution to the loss, meaning that layers weights which did not contribute to computing the loss do not have any gradients flowing through them.</p>
<p>For a rather unusual research project I will need to backprop gradients through a network that didn’t compute the output. Essentially I want to define a custom, architecture-specific backward pass. My question is - can I do this in Pytorch? If not then what library would you recommend?</p>
<p>I know it sounds very unintuitive, so here’s an expanded explanation in case it’s unclear:</p>
<p>I have a process P that generates a tensor <em>Z</em>.<br>
I have a network N that maps <em>S</em> to <em>X</em>. The dimensions of <em>X</em> and <em>Z</em> are identical.<br>
I have a ground-truth label <em>Y</em> (same dimensions as <em>X</em> and <em>Z</em>) and I pass it to the loss function together with <em>Z</em> to compute the loss.<br>
I wish to backpropagate the loss through network N and update its weights as if it computed <em>Z</em>.</p>

albanD: <p>Hi,</p>
<p>I’m a bit confused in the sense that the gradient flowing back depend on the value that was forwarded. So if the last value is X or Z here, the gradient will be different. Is that fine?</p>
<p>If it is, then you can simply compute the gradient s for X first and then backprop them through Z with <code>Z.backward(grad_X)</code>.</p>
Why dropout changes weights?

csailnadi: <p>I am trying to use nn.Dropout to drop some of the weights in trained network.<br>
I expected this to result in zero-ing out some of the entries, but I also observe changes in numerical values:</p>
<pre><code>params = [p for n, p in model.named_parameters() if 'features' not in n and len(p.shape) &gt; 1]
for par in params:
    par = par.detach()
    par = torch.nn.Dropout2d(p=0.3)(par)
</code></pre>
<p>Before dropout:</p>
<p>tensor([[ 0.0022,  0.0041,  0.0004,  …, -0.0026, -0.0028, -0.0021],<br>
[ 0.0007,  0.0015,  0.0021,  …,  0.0040,  0.0016,  0.0015],<br>
[ 0.0007,  0.0005,  0.0021,  …, -0.0049, -0.0026,  0.0008],<br>
…,<br>
[-0.0003, -0.0022, -0.0003,  …,  0.0001,  0.0012, -0.0007],<br>
[ 0.0033,  0.0045,  0.0018,  …, -0.0020, -0.0036,  0.0007],<br>
[ 0.0031,  0.0045,  0.0015,  …, -0.0022, -0.0012, -0.0007]],<br>
requires_grad=True)</p>
<p>After dropout:</p>
<p>tensor([[ 0.0000,  0.0059,  0.0006,  …, -0.0037, -0.0000, -0.0030],<br>
[ 0.0011,  0.0021,  0.0030,  …,  0.0057,  0.0000,  0.0021],<br>
[ 0.0000,  0.0000,  0.0030,  …, -0.0070, -0.0038,  0.0000],<br>
…,<br>
[-0.0000, -0.0032, -0.0004,  …,  0.0000,  0.0000, -0.0010],<br>
[ 0.0047,  0.0065,  0.0026,  …, -0.0029, -0.0000,  0.0010],<br>
[ 0.0000,  0.0065,  0.0021,  …, -0.0031, -0.0018, -0.0009]])</p>
<p>Why does it happen?</p>

acarlier: <p>Hello csailnadi,</p>
<p>Take a look at the documentation for Dropout: <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Dropout" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.Dropout</a></p>
<p>To keep the same output scale (in expectation), outputs are rescaled by 1/(1-p) during training. You can see this in your example: for instance 0.0041 is mapped to 0.0041 / 0.7 = 0,00585 ~ 0.0059.</p>
<p>Hope this helped <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
nn.Linear for one-hot features

CCL: <p>After performing Global Average Pooling, I have <code>(N, C=3, 1, 1)</code> (reshaped to <code>(N, C=3, 1)</code>) dimensional features, which I would like to pass to a linear layer. My desired output is of size <code>(N, C, 1)</code> or <code>(N, C)</code>. However, I’m unsure of what dimensions to use for <code>out_features</code>.</p>
<p>Dimensions-wise, <code>nn.Linear(1, 1)</code> returns the correct dims, but I’m not sure it makes logical sense. If I’m not mistaken, features of different classes will be sharing the same weight, and this will be treated as a binary problem instead of a 3-class one. How do I perform <code>nn.Linear</code> preserving both the correct output dimensions and the multi-class nature of the problem?</p>
<p>Thanks!</p>

googlebot: <p>With nn.Linear you’re doing matrix multiplication: (N,n_in) @ (n_in,n_out) +bias = (N,n_out). Your n_out should be C(=3), unless you want to rescale+shift channels by same amounts. (n_in=1,n_out=3) makes almost no sense, this leaves two possibilities: 1)linear transformation 3x3, as done by nn.Linear(3,3) or nn.Conv1d(3,3,1) 2)independent channel transformations - nn.Linear is not suitable for this, but simple w*x+b expression does this; another fancy way to do this is is nn.Conv1d(3,3,1,groups=3) layer.</p>
<p>About shapes, for nn.Linear you should reshape input to (N,C) (or permute NCL -&gt; NLC for L&gt;1). And transformatons with nn.ConvXd work without reshape.</p>
Dist.all_gather leads to deadlock

fanl: <p>Hi everyone,<br>
If all_gather are used more than once,  the next cuda() function will deadlock. And GPU-Utils are 100%.<br>
Details:<br>
I wrapped <code>all_gather</code> in <code>collect</code>:</p>
<pre><code class="lang-auto">def collect(x):
    x = x.contiguous()
    out_list = [torch.zeros_like(x, device=x.device, dtype=x.dtype)
        for _ in range(dist.get_world_size())]
dist.all_gather(out_list, x)
return torch.cat(out_list, dim=0)
</code></pre>
<p>Next:</p>
<pre><code class="lang-auto">a_all = collect(a)
b_all = collect(b)
c = torch.rand(10,10).cuda() # deadlock! No error report.
</code></pre>
<p>The deadlock doesn’t happen if  I don’t collect <code>b_all</code> or don’t use <code>cuda()</code>.<br>
My code runs on 4 GPUs using DDP.<br>
Pytorch version: 1.4.<br>
This problem has been confusing me for a couple of days. I’ll appreciate any help!</p>

pritamdamania87: <p>Could you provide a small repro for the problem that you are seeing? I ran the following script locally on my GPU machine and didn’t notice any deadlocks:</p>
<pre><code class="lang-auto">import os
import torch
import torch.distributed as dist
from torch.multiprocessing import Process

def collect(x):
    x = x.contiguous()
    out_list = [torch.zeros_like(x, device=x.device, dtype=x.dtype)
        for _ in range(dist.get_world_size())]
    dist.all_gather(out_list, x)
    return torch.cat(out_list, dim=0)

def run(rank, size):
    """ Distributed function to be implemented later. """
    print ('START: {}'.format(rank))
    a = torch.rand(10, 10).cuda(rank)
    b = torch.rand(10, 10).cuda(rank)
    a_all = collect(a)
    b_all = collect(b)
    c = torch.rand(10,10).cuda(rank)
    print ('DONE : {}'.format(rank))

def init_process(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size)

if __name__ == "__main__":
    size = 4
    processes = []
    for rank in range(size):
        p = Process(target=init_process, args=(rank, size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()
</code></pre>
<p>Running the script:</p>
<pre><code class="lang-auto">$ python /tmp/test_allgather.py
START: 0
START: 1
START: 2
START: 3
DONE : 2
DONE : 3
DONE : 1
DONE : 0
</code></pre>
retain_graph=True for multiple LSTM input network

Lizard_King: <p>Hello. I have built a network with multiple LSTM as an input layer. Each LSTM is built for certain signal, which makes total of 10 LSTMs for each signal id. While training, certain signal and its id arrives and I want to only update weights w.r.t. signals id. Unfortunately, Im facing following error:<br>
<code>RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed</code>.<br>
Error is gone if I set <code>retain_graph=True</code> as backward parameter, but this way training speed decreases <strong>dramatically</strong> after each forward call in training loop.  How do change my training loop so I dont have to use <code>retain_graph</code> parameter?  Heres my train loop code:</p>
<pre><code>    for epoch in range(self.epochs):
        self.main_model.hidden_states = self.main_model.init_hidden()
        self.main_model.cell_states = self.main_model.init_cells()

        batch = self.generator.get_sample_batch()

        for i, batch_el in enumerate(batch):

            cum_loss = 0

            for j, input_signal in enumerate(batch_el):

                self.freeze_layers()

                signal_id = int(input_signal[0] - 1)

                self.unfreeze_layers(signal_id)

                payload_size = self.signals_per_id[signal_id]

                input_s = torch.as_tensor(np.array(input_signal[1:1+payload_size], dtype=np.float32).
                                          reshape(1, 1, payload_size)).float()

                reconstruction = self.main_model(input_s, signal_id)

                loss_value = self.loss(reconstruction, input_s) * self._dataset_inv_freqs[signal_id]

                loss_value.backward()#retain_graph=True

                cum_loss += loss_value.item()
                if (j + 1) % self.update_interval == 0:
                    self.optimizer.step()
                    self.main_model.zero_grad()</code></pre>
Using Bias=False during batch norm

Aptha_Gowda: <pre><code class="lang-auto">#model1
def getmodel():
    model = EfficientNet.from_pretrained('efficientnet-b5')
    model._fc = nn.Sequential(
         nn.Linear(in_features=2048, out_features=1024, bias=True), #first layer 
         nn.ReLU(),
         nn.BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
         nn.Dropout(p=0.5),
         nn.Linear(in_features=1024, out_features=4, bias=True)) #last layer
    model = model.to(device)
    return model
</code></pre>
<p>I have lot of question from the above code block which I want to clarify.</p>
<ol>
<li>I have read that bias should be True (bias=True) at the last linear layer. And my model also performed well when turned on.</li>
<li>Most people suggested that bias should be turned off (bias=False) before using batch norm ( Even bias in the Conv layers of EfficientNet are turned off before batch norm). But my model performed badly when I turned off the bias at the first layer. What should I follow?</li>
<li>This model (model2) just uses the single linear layer.</li>
</ol>
<pre><code class="lang-auto">#model2
def getmodel():
   model = EfficientNet.from_pretrained('efficientnet-b5')
   model._fc = nn.Sequential(
        nn.Linear(in_features=2048, out_features=4, bias=True)) #last layer
   model = model.to(device)
   return model
</code></pre>
<p>Whcih approch should I follow model1 or model2 ?</p>

albanD: <p>Hi,</p>
<ol start="2">
<li>The thing is that in your case, you have a ReLU between the Linear and Batchnorm. So that statement may not be true for your model.<br>
I think that statement comes from the fact that the batchnorm will center the values. So a bias is useless in the previous layer as it will just be cancelled by the batchnorm.</li>
<li>model2 has much less parameters. So it will perform differently. It will depend on your dataset though which one is better.</li>
</ol>

Aptha_Gowda: <p>So just to clarify, If there is an activation layer(relu) in between then there is no need to turn off the bias in layer one. Right.</p>
<p>I have this doubt that linear layer at the end (from 2048 &gt; 4 output class) learns less compared with (2048 &gt; 1024 &gt; 4 output class). If yes in what situation this is helpful? Why most people choose model2 like architecture during fine-tuning?</p>

albanD: <p>This model2 is not as powerful, so it might be worst on the training set.<br>
But because it has less parameters (and is linear) the model it is gonna learn is much simpler and so is naturally regularized (due to the structure of the function). This could be an advantage depending on your task and could lead to better validation accuracy in some cases.</p>
How to share weights in a model and then unshare them later?

kinwai_cheuk: <p>I have a model like this</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.layer_a = nn.Linear(1,1, bias=False)
        self.layer_b = nn.Linear(1,1, bias=False)
        
    def forward(self, x):
        x_a = self.layer_a(x)
        x_b = self.layer_b(x)
        
        return (x_a, x_b)
</code></pre>
<p>There are two phases of training<br>
Phase 1:<code> self.layer_a</code> and <code>self.layer_b</code> share the same weight.<br>
Phase 2: <code> self.layer_a</code> is frozen, only <code>self.layer_b</code> keeps updating.</p>
<p>To achieve Phase 1, I do something like</p>
<pre><code class="lang-auto">model = Net()
model.layer_b.weight = model.layer_a.weight
</code></pre>
<p>Then both of them share the same weight.</p>
<p>But the problem is how to freeze <code>model.layer_a.weight</code> without affect <code>model.layer_b.weight</code> in phase 2?</p>
<p>Or it there a better way to a</p>

googlebot: <p>I would remove nn.Linear altogether, and have something like</p>
<pre><code class="lang-auto">w2 = self.weight_b if stage2 else self.weight_a
x_b = x.matmul(w2.t())
</code></pre>
<p>another alternative that needs no flag passing:</p>
<pre><code class="lang-auto">w2 = self.weight_a + self.weight_b
</code></pre>
<p>init weight_b to zero and don’t train it in phase 1, detach or otherwise freeze weight_a in phase 2</p>

kinwai_cheuk: <p>Thanks for the suggestion. I was using <code>nn.Linear</code> to illustrate my problem. In reality, my model consists of two deep convolutions networks.</p>
<p>If it is simply a <code>nn.Linear</code>, your suggestions does work. But how about if it is a deep convolutions networks?</p>

googlebot: <p>Try doing sharing with</p>
<pre><code class="lang-auto">with torch.no_grad():
	model.layer_b.weight.data.copy_(model.layer_a.weight.data)
</code></pre>

kinwai_cheuk: <p>I see. Then when I want to train them separately again, I just need to activate the gradient for <code>layer_b</code>?</p>

googlebot: <aside class="quote no-group" data-username="kinwai_cheuk" data-post="5" data-topic="77374" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/13edae/48.png" class="avatar"> kinwai_cheuk:</div>
<blockquote>
<p>I see. Then when I want to train them separately again, I just need to activate the gradient for <code>layer_b</code> ?</p>
</blockquote>
</aside>
<p>Yes, probably <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-auto"></code></pre>
All functions have the same name in autograd profiler

Haxxardoux: <p>I have been looking for tools to profile the end-to-end training of a model (reduced epochs, of course), and the profiler torch.utils.bottleneck is returning some strange results.</p>
<p>There is also lots of information it is leaving out, like what runs in parallel and such (comparing to <a href="https://www.tensorflow.org/tensorboard/tensorboard_profiling_keras" rel="noopener nofollow ugc">tensorflow profiling tool</a>), so i would love to hear about any other options!</p>
<p>The bottleneck profiler has returned this output;</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d936f903616bf74786be9266a07236845b3a7b8c.png" data-download-href="https://discuss.pytorch.org/uploads/default/d936f903616bf74786be9266a07236845b3a7b8c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d936f903616bf74786be9266a07236845b3a7b8c.png" alt="image" data-base62-sha1="uZzo9vCRHLHQmToBPA1Eudtstyc" width="690" height="242" data-dominant-color="151515"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1584×556 15.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>All the functions have the same name, and it is overall a bit difficult to get a clear picture of what the model is doing. I suspect things like the input_shapes being <span class="chcklst-box fa fa-square-o fa-fw"></span> and each function getting called exactly once are signs that something is not quite right, but i have not found much documentation to confirm this suspicion.</p>
<p>Any advice or suggestions would be sincerely appreciated!</p>

albanD: <p>Hi,</p>
<p>You are running on GPU right?<br>
You can check in the doc <a href="https://pytorch.org/docs/stable/bottleneck.html">here</a> that the bottleneck tool won’t perform synchronization with the GPU.<br>
Here you see that the only expensive functions are the transfer between CPU/GPU. This is expected as these are the only place where we synchronize with the GPU and actually wait for the GPU to do the work.</p>
<p>I think you want to use the autograd profiler with cuda setting turned on to get a better idea of which function is actually taking a long time on the GPU.</p>
Squeezenet Fine Tuning

Giuseppe: <p>size mismatch, m1: [13000 x 13], m2: [1000 x 4] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:266</p>
<p>This is the error when i try to do Fine Tuning on SqueezeNet.</p>
<p>This is the code, i want to make regression with 4 output:</p>
<pre><code class="lang-auto">def squeezeNet_model():
    model_conv = models.squeezenet1_1(pretrained=True)
    mod = list(model_conv.classifier.children())
    mod.pop()
    mod.append(nn.Linear(1000, 4))
    new_classifier = nn.Sequential(*mod)
    model_conv.classifier = new_classifier


    return model_conv
</code></pre>
<p>Where i wrong?</p>

Giuseppe: <p>I solved with something more simple:</p>
<pre><code class="lang-auto">def squeezeNet_model():
    model_Squeeze = models.squeezenet1_1(pretrained=True)
    num_out = 4
    # change the last conv2d layer
    model_Squeeze.classifier._modules["1"] = nn.Conv2d(512, num_out, kernel_size=(1, 1))


    return model_Squeeze
</code></pre>
Risk Model to populate a map

QuirkyDataScientist1: <p>Good Morning,</p>
<pre><code> I want to create a risk model using pytorch in which I have data from the Global Terrorism Database (GTD) and I want to create a deep learning model that will output based on the inputs the highest risk areas (lat, lon coordinates to populate a map) but I want to be able to filter it based on the attributes. 

 How would I create such a model? Code to get me started?
</code></pre>
<p>~QuirkyDataScientist1</p>
Normalize 3D Tensor

gsouza7: <p>Hi!</p>
<p>Is there any way to min-max (actually, value / max-value) normalize a 3D Tensor by two dimensions?<br>
Let’s say we have 10x20x30 and I want to normalize it regarding the last two dimensions.<br>
I expect to have a matrix 20x30 such that each position on that matrix is the max value in the first dimension of the original tensor.<br>
This way, by doing 10x20x30 (normalized) * 20x30 = 10x20x30 (original).</p>
<p>Sorry for the silly question, but I’m struggling with it for a while.<br>
Thanks!</p>

acarlier: <p>Hello gsouza7,</p>
<p>You need to take the maximum along the first dimension, like so:</p>
<pre><code class="lang-auto">x = torch.randn(10, 20, 30)
x_max = x.max(dim=0).values
x_normalized = x / x_max
</code></pre>
How to change features size without padding in CNN?

srl123: <p>Hi,</p>
<p>I’m using a 2d CNN. My input size is a square image of 20x20 pixels and my desired output has size 400x3.</p>
<p>The most common thing I see is to introduce padding over the convolutional layers in order to reach the desired output size and let the nn train.</p>
<p>The output I get in this way is characterized by lots of zeros (sparse matrix) and obviously the performance is very low. So what else can I do? I tried to introduce more layers, each one characterized by smaller padding values but it doesn’t improve…</p>
<p>any hint will be really appreciated, thanks!</p>

Haxxardoux: <p>Is your input image actually 20x20x3? Why would you need an output to be the same size as your input?</p>

srl123: <p>My input image is just 20x20 and they have not the same size, since I my net input is 20x20 and the output is 400x3.</p>
<p>thank you</p>