akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Assertion Error Distributed Data Parallel

pklvls: <p>I’m trying to use the distributed data parallel thing with pytorch and I’m running into this error which I haven’t found a solution for elsewhere:</p>
<p><code>-- Process 2 terminated with the following error: Traceback (most recent call last):   File "/afs/csail.mit.edu/u/v/asdf/miniconda3/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap     fn(i, *args)   File "../multiprocess_model.py", line 5, in train_model     model.train(args)   File "../train.py", line 227, in train     for i, data_dict in enumerate(train_dataloader, 1):   File "/afs/csail.mit.edu/u/v/asdf/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 363, in __next__     data = self._next_data()   File "/afs/csail.mit.edu/u/v/asdf/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 402, in _next_data     index = self._next_index()  # may raise StopIteration   File "/afs/csail.mit.edu/u/v/asdf/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 357, in _next_index     return next(self._sampler_iter)  # may raise StopIteration   File "/afs/csail.mit.edu/u/v/asdf/miniconda3/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 208, in __iter__     for idx in self.sampler:   File "/afs/csail.mit.edu/u/v/asdf/miniconda3/lib/python3.8/site-packages/torch/utils/data/distributed.py", line 84, in __iter__     assert len(indices) == self.num_samples AssertionError </code></p>
<p>How would I approach fixing this problem?</p>
<p>I can comment out the assertion errors in distributed.py, but I’m hoping there is a cleaner way to do this.</p>

mrshenli: <p>This error occurs in data loader. Could you please share a repro of the data loader?</p>
<p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> for data loader questions</p>

agolynski: <p>This also looks similar to <a href="https://github.com/pytorch/pytorch/issues/49172" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/49172</a> (and proposal <a href="https://github.com/pytorch/pytorch/issues/49180" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/49180</a>). Do you use use DistributedSampler?</p>

pklvls: <p>Yeah! It ended up being a problem with my data loader.</p>
<p>I did something and now it works, although I’m not sure what I did.</p>
<p>Could a possible problem be that the data loader was returning None for the elements occasionally?</p>

pklvls: <p>On another note, I’m now getting a <code>Exception: process 0 terminated with signal SIGSEGV</code> when I run my model.</p>
LSTM Cell State/Hidden State Storage and Input

Jackson_Rusch: <p>Hello all,</p>
<p>I have a fairly simple question. When calling the model with the input and hidden parameters, does the hidden state include the hidden state and cell state or just the hidden state. I am looking to run the network in a simulation and the hidden/cell states need to be stored until the training data is collected from the simulation. As of right now, I am storing the hidden state and I just wanted to check whether or not the cell state is stored along with that.</p>
<p>Thanks!</p>

Jackson_Rusch: <p>If something in this question is unclear, please let me know</p>

Jackson_Rusch: <p>Reading some of the documentation, I’m 90% sure that the cell state is stored along with the hidden state when passing it in to the network and at the output.</p>

Jackson_Rusch: <p>Okay I’m 99.99% sure that cell state is stored with hidden, so I think I’m good</p>
Deepcopy vs. load_state_dict

rasoolfa: <p>Hi,</p>
<p>Is there any advantage to using “load_state_dict” not “deepcopy” or vice versa when one wants to ‘deep’ copy, i.e. updating model_B doesn’t change model_A after copy,  one model to another?</p>
<pre><code class="lang-auto">class DummyNet(nn.Module):
     def __init__(self,..):
         super(dummyNet, self).__init__()
         ...
     def forward(...)
          ...

</code></pre>
<p>Method 1:</p>
<pre><code class="lang-auto">model_A = DummyNet()
model_B = copy.deepcopy(model_A)
</code></pre>
<p>Method 2:</p>
<pre><code class="lang-auto">model_A = DummyNet()
model_B = DummyNet()
model_B.load_state_dict(model_A.state_dict())
</code></pre>
<p>They practically lead to the same result but want to make sure.</p>
<p>Thanks,</p>

ptrblck: <p>As you said, both codes should work, so that a parameter update in one model does not change the other and vice versa:</p>
<pre><code class="lang-python">modelA = models.resnet18()
modelB = models.resnet18()
modelA.load_state_dict(modelB.state_dict())

# Compare random parameter
print((modelA.fc.weight == modelB.fc.weight).all())
&gt; tensor(True)

# Manipulate param
with torch.no_grad():
    modelA.fc.weight.zero_()

print((modelA.fc.weight == modelB.fc.weight).all())
&gt; tensor(False)
</code></pre>

saurav.prakash16: <p>Does <code>copy.deepcopy(model_A)</code> copies the state variables as well, like in <code>model_B.load_state_dict(model_A.state_dict())</code>? If so, in what situations is one preferred over another for deep copy?</p>
Sequential model from config file with deconvolution

thomas: <p>Hello!<br>
I have a project where I have many different models, therefore I would like to store the structure in a config file. I wrote this code to generate a model, it works quite well:</p>
<pre><code class="lang-auto">layers = [
    {"type": "Conv2d", "in_channels": 1, "out_channels": 6, "kernel_size": 3},
    {"type": "ReLU"},
    {"type": "Conv2d", "in_channels": 6, "out_channels": 16, "kernel_size": 3},
    {"type": "ReLU"},
    {"type": "Flatten"},
    {"type": "Linear", "in_features": 16 * 6 * 6, "out_features": 120},
    {"type": "ReLU"},
    {"type": "Linear", "in_features": 120, "out_features": 84},
    {"type": "ReLU"},
    {"type": "Linear", "in_features": 84, "out_features": 10},
    ]

def build_model(config):
    layer_list = []
    for layer_config in config:
        layer_type = layer_config.pop('type')
        layer_list.append(getattr(torch.nn, layer_type)(**layer_config))
    return nn.Sequential(*layer_list)

model = build_model(layers)
</code></pre>
<p>However I have codes that needs “deconvolution network” and for those, I need a view/reshape layer. I am not sure how to handle that without too much trouble. Does someone have an idea ?</p>
<p>Examples of layers I would like to use:</p>
<pre><code class="lang-auto">layers = [
    {"type": "Linear", "in_features": 10, "out_features": 1024},
    {"type": "ReLU"},
    #  ? x = x.view(-1, 16, 64)
    {"type": "Conv1d", "in_channels": 16, "out_channels": 32, "kernel_size": 3},
    {"type": "ReLU"},
    {"type": "Conv1d", "in_channels": 32, "out_channels": 64, "kernel_size": 3},
    {"type": "ReLU"}
    # ...
    ]
</code></pre>
<p>Thanks!</p>

thomas: <p>I ended up using this code:</p>
<pre><code class="lang-auto">import torch
layers = [
    {"type": "Linear", "in_features": 10, "out_features": 1024},
    {"type": "ReLU"},
    {"type": "View", "shape": (-1, 16, 64)},
    {"type": "Conv1d", "in_channels": 16, "out_channels": 32, "kernel_size": 3},
    {"type": "ReLU"},
    {"type": "Conv1d", "in_channels": 32, "out_channels": 64, "kernel_size": 3},
    {"type": "ReLU"}
    # ...
    ]

class View(nn.Module):
    def __init__(self, shape):
        super(View, self).__init__()
        self.shape = shape
        
    def forward(self, x):
        return x.view(*self.shape)

def build_model(config):
    layer_list = []
    for layer_config in config:
        layer_type = layer_config.pop('type')
        if layer_type == "View":
            layer = View(**layer_config)
        else:
            layer = getattr(torch.nn, layer_type)(**layer_config)
        layer_list.append(layer)
    return nn.Sequential(*layer_list)

model = build_model(layers)
</code></pre>
<p>Unfortunately it is really not as clean as before <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
I don't know bi-lstm output?

xiashu: <p>Today I learned something about lstm. here my question<br>
As show in pytorch’s document.<br>
here comes out a tuple.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/1/11be2c6dd7216b64a30f23b5e09ab13b18e8e9ae.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/11be2c6dd7216b64a30f23b5e09ab13b18e8e9ae" title="Screenshot_20201215_180148"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/11be2c6dd7216b64a30f23b5e09ab13b18e8e9ae_2_585x500.jpeg" alt="Screenshot_20201215_180148" data-base62-sha1="2wXyQ8NG2ifE04HxYIE0aC6RVq6" width="585" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/11be2c6dd7216b64a30f23b5e09ab13b18e8e9ae_2_585x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/11be2c6dd7216b64a30f23b5e09ab13b18e8e9ae_2_877x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/1/11be2c6dd7216b64a30f23b5e09ab13b18e8e9ae.jpeg 2x" data-dominant-color="F2F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20201215_180148</span><span class="informations">1080×923 159 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>for example:<br>
here a sentence “I like eating apple” when put into a bilstm net<br>
out: lstm_out,(h_n,cell)</p>
<p>I want to know what h_n is mean？it is connect vector of “ apple” and “I” or just "apple " in back lstm and forward lstm。</p>

vdw: <p><code>h_n</code> will contain the last hidden state of the forward direction (“i like eating apple”) and the last hidden state if the backward direction (“apple eating like i”), just combined in one tensor. The hidden state for each direction might have multiple layers, hence the <code>num_layers*num_directions</code> in the shape of <code>h_n</code>.</p>
<p>Say you want to add the last hidden states of the forward and backward pass. This would look like this</p>
<pre><code>h_n = h_n.view(num_layers, num_directions, batch, hidden_size) # to separate num_layers*num_directions
h_n = h_n[-1] to consider only the last layer
h_n = h_n[0] + h_n[1] # add hidden state of forward backward direction
</code></pre>
<p>Just as an example. Up to you how you want to further process the output</p>

xiashu: <p>Thank you for your reply. i am very appreciate that you have help me resolve my question</p>
<p>And I I 'm confused that  what kind of situation  to get h_n or lstm_out? and h_0 should be random tensor or be a zero tensor ,Is there have any different?</p>
Save_image in a new folder

AleTL: <p>Hello!</p>
<p>How can I save images (from a VAE, but it doesn’t matter) in a new folder? What I want is to automatically create a new folder if it doesn’t exist, and save there the pictures. And if it exists, just to use it.</p>
<p>I’m using this but it doesn’t work:</p>
<pre><code class="lang-auto">torchvision.utils.save_image(code, f"/ImageOutput/code{epoch}.png", nrow=num_rows)   
</code></pre>
<p>The point is that I would like to create or use the folder named ImageOutput.</p>
<p>Thanks!</p>

RaLo4: <p>This might not be the most elegant way but it definitely works</p>
<pre><code class="lang-auto">if not(os.path.isdir('/ImageOutput')):
        os.mkdir('/ImageOutput')
torchvision.utils.save_image(code, f"/ImageOutput/code{epoch}.png", nrow=num_rows)
</code></pre>
>=1.6.0 UserWarning: The default behavior for interpolate/upsample with float scale_factor changed

space1panda: <p>Hello,</p>
<p>While using torchvision.detection models with torch above or equal to 1.6, I am getting the following warning on each mask interpolation in GenericRCNNTransform:<br>
UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.<br>
warnings.warn("The default behavior for interpolate/upsample with float scale_factor changed "</p>
<p>So, I am being recommended to specify recompute_scale_factor=True but this is an internal torchvision object which get’s the warning and I am not supposed to edit the code</p>
How to rearrange tensor in a certain dimension？

Niu: <p>For example：<br>
Feature is [B,C,H,W], I have weights of [B,C], I want to rearrange feature in Channel dimension with descending weights for per image. My code is as follow：<br>
I want to know that there is a more elegant way to operate all channels at the same time？<br>
Earnestly hope ！</p>
<pre><code class="lang-auto">import torch
feature = torch.rand(4,6,3,3)
weight = torch.rand(4,6)
_, indices = torch.sort(weight, dim=1, descending = True)
z1 = feature[0,indices[0],:,:]
z2 = feature[1,indices[1],:,:]
z3 = feature[2,indices[2],:,:]
z4 = feature[3,indices[3],:,:]
z = torch.stack((z1,z2,z3,z4),dim=0)
</code></pre>

ptrblck: <p>You could directly index the tensor as seen here:</p>
<pre><code class="lang-python">feature = torch.rand(4,6,3,3)
weight = torch.rand(4,6)
_, indices = torch.sort(weight, dim=1, descending = True)
z1 = feature[0,indices[0],:,:]
z2 = feature[1,indices[1],:,:]
z3 = feature[2,indices[2],:,:]
z4 = feature[3,indices[3],:,:]
z = torch.stack((z1,z2,z3,z4),dim=0)

res = feature[torch.arange(feature.size(0)).unsqueeze(1), indices]

print((z - res).abs().max())
&gt; tensor(0.)
</code></pre>

Niu: <p>Thank you very much for your quick reply, which is exactly what I want.</p>
ValueError: optimizer got an empty parameter list but len(list(model.parameters())) is not 0

R1ckShi: <p>I initialize the nn model with<br>
<code>model = ASRModel(idim_list[0] if args.num_encs == 1 else idim_list, odim, args)</code><br>
and model.parameters() returns the parameter list correctly. But when I declare an optimizer<br>
<code>optimizer = torch.optim.Adam(model.parameters(), weight_decay=args.weight_decay)</code>,<br>
an error occurs.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1fc4fe937ef91e5a60a63e2c05e03c4e1dc47bad.png" data-download-href="https://discuss.pytorch.org/uploads/default/1fc4fe937ef91e5a60a63e2c05e03c4e1dc47bad" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1fc4fe937ef91e5a60a63e2c05e03c4e1dc47bad.png" alt="image" data-base62-sha1="4x2RkGLo9t96pd4qPOEKKc3fPpr" width="690" height="194" data-dominant-color="3F3F3F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">928×262 8.69 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

R1ckShi: <p>I found that the <strong>init</strong> function for base class for Optimizer will run twice and the len(param_groups) will be 0 in the second time. Why?</p>

ptrblck: <p>Are you recreating the optimizer somewhere in your code or changing the parameters of the model?<br>
If you get stuck, could you post an executable code snippet to reproduce this issue, please?</p>
Pass grad_output of the last layer to previous layers

yYang: <p>I am trying to implement <a href="https://arxiv.org/abs/1609.01596" rel="noopener nofollow ugc">Direct Feedback Alignment</a> with a custom autograd function, a diagram of the algorithm can be found <a href="https://github.com/ChFrenkel/DirectRandomTargetProjection/blob/master/training_algorithms_topologies.png" rel="noopener nofollow ugc">here</a> in subplot c. Basically what I want to do (for example in a Linear layer) is changing the gradient computation in backward function from</p>
<pre><code class="lang-auto">grad_input = grad_output.mm(weight)
grad_weight = grad_output.t().mm(input)
</code></pre>
<p>to</p>
<pre><code class="lang-auto">grad_input = grad_output  # Directly pass down grad_output without modifiying
grad_weight = grad_output.mm(weight_fb).t().mm(input) 
</code></pre>
<p>But this won’t work as the grad_input does not have the corresponding size to the input.</p>
<p>I was wondering if there is a way to implement this with a custom autograd function. Any hint would be appreciated!</p>
Saving ensembled models correctly

tomcc: <p>Hello I am currently constructing ensembled methods and is wondering how i should save them.</p>
<p>For this example, taken from <a href="https://discuss.pytorch.org/t/custom-ensemble-approach/52024">here</a></p>
<pre><code class="lang-auto">class MyEnsemble(nn.Module):
    def __init__(self, modelA, modelB, nb_classes=10):
        super(MyEnsemble, self).__init__()
        self.modelA = modelA
        self.modelB = modelB
        # Remove last linear layer
        self.modelA.fc = nn.Identity()
        self.modelB.fc = nn.Identity()
        
        # Create new classifier
        self.classifier = nn.Linear(2048+512, nb_classes)
        
    def forward(self, x):
        x1 = self.modelA(x.clone())  # clone to make sure x is not changed by inplace methods
        x1 = x1.view(x1.size(0), -1)
        x2 = self.modelB(x)
        x2 = x2.view(x2.size(0), -1)
        x = torch.cat((x1, x2), dim=1)
        
        x = self.classifier(F.relu(x))
        return x

# Train your separate models
# ...
# We use pretrained torchvision models here
modelA = models.resnet50(pretrained=True)
modelB = models.resnet18(pretrained=True)

# Freeze these models
for param in modelA.parameters():
    param.requires_grad_(False)

for param in modelB.parameters():
    param.requires_grad_(False)

# Create ensemble model
ensembled_model = MyEnsemble(modelA, modelB)
</code></pre>
<p>If i were to save these 2 models, would it make more sense to save them seperately:</p>
<pre><code class="lang-auto">torch.save(modelA.state_dict(), "...")
torch.save(modelB.state_dict(), "...")
</code></pre>
<p>Or save the ensembled model:</p>
<pre><code class="lang-auto">torch.save(ensembled_model.state_dict(), "...")
</code></pre>
<p>Are there any difference in the two methods above?</p>

ptrblck: <p>It depends how you would like to reload and use these models afterwards.<br>
E.g. if you only care about the two submodules and would like to use them afterwards, the first approach would work. Note that you would lose the <code>self.classifier</code> as it’s a module in <code>MyEnsemble</code>.<br>
In the latter case, all parameters will be stored and you would have to recreate the ensemble model to load the <code>state_dict</code> again (you could also manipulate the <code>state_dict</code> manually, but I would consider this a workaround and not a proper way to restore the parameters).</p>

tomcc: <p>Great answer. Thank you</p>
Plot `torch.Tensor` using OpenCV

nullgeppetto: <p>I have a pytorch tensor, let’s say <code>images</code>, of type <code>&lt;class 'torch.Tensor'&gt;</code> and of size <code>torch.Size([32, 3, 300, 300])</code>, so that <code>images[i, :, :, :]</code> represents the i-th out of 32 rgb 300x300 images.</p>
<p>I would like to plot some of these images, let’s say the first one, i.e., <code>images[0, :, :, :]</code> using opencv (<code>cv2.imshow()</code>). Could you help? Thank you!</p>

ptrblck: <p>You can call <code>images[0].numpy()</code> to get the numpy array, which can be visualized using OpenCV.</p>

nullgeppetto: <p>Thanks for your response. I tried that but I got an opencv error (<code>Source image must have 1, 3 or 4 channels in function cvConvertImage</code>)… It seemed to me that it would be trivial, but it isn’t…</p>

ptrblck: <p>Sorry, my bad. You have to permute the axis:</p>
<pre><code class="lang-python">images[0].numpy().transpose(1, 2, 0)
# or
images[0].permute(1, 2, 0).numpy()
</code></pre>

nullgeppetto: <p>I see. Thank you very much, it worked!</p>

venmadh: <p>Why can’t we use torch tensors directly with Open cv without converting them to numpy? I run cv2.kmeans on large images and they’re very slow. Wondering if being able to run them on Tensors would be faster.</p>

yptheangel: <p>after converting your torch tensor back to opencv ndarray, if you do an imshow the image will appear slightly darker due to standard normalization.</p>
<pre><code class="lang-auto">def inverse_normalize(tensor, mean, std):
    for t, m, s in zip(tensor, mean, std):
        t.mul_(s).add_(m)
    return tensor

input = inverse_normalize(tensor=input, mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))
</code></pre>
<p>you should inverse normalize your torch tensor before converting to numpy array if the image colours matter to you</p>

1chimaruGin: <p>I think opencv-python package support CPU-only. So, we need to change cuda tensor to cpu.</p>
Two questions about CosineEmbeddingLoss

111437: <p>From the source code:</p>
<pre><code class="lang-auto">Tensor cosine_embedding_loss(const Tensor&amp; input1, const Tensor&amp; input2, const Tensor&amp; target, double margin, int64_t reduction) {
  auto prod_sum = (input1 * input2).sum(1);
  auto mag_square1 = (input1 * input1).sum(1) + EPSILON;
  auto mag_square2 = (input2 * input2).sum(1) + EPSILON;
  auto denom = (mag_square1 * mag_square2).sqrt_();
  auto cos = prod_sum / denom;

  auto zeros = at::zeros_like(cos, LEGACY_CONTIGUOUS_MEMORY_FORMAT);
  auto pos = 1 - cos;
  auto neg = (cos - margin).clamp_min_(0);
  auto output_pos = at::where(target == 1, pos, zeros);
  auto output_neg = at::where(target == -1, neg, zeros);
  auto output = output_pos + output_neg;
  return apply_loss_reduction(output, reduction);
}
</code></pre>
<p>I have two questions:</p>
<ol>
<li>Is it deprecate for the three steps of calculation for <code>prod_sum</code>, <code>mag_square1</code>, <code>mag_square2</code>? They all calculate <code>(input1 * input2).sum(1)</code>, or it’s because of some mechanisms like object referencing?</li>
<li>It doesn’t check the range of <code>target</code> at all. I inputed a target of 0.5, but it didn’t raise an error.</li>
</ol>

glaringlee: <p><a class="mention" href="/u/111437">@111437</a><br>
Please file an issue at<br>
‘<a href="https://github.com/pytorch/pytorch/issues" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues</a>’ , we will have people looking into it. Thanks a lot.</p>
Segmentation fault when using local pc to run experiment

Yoark_YANG: <p>Hello, Community!</p>
<p>My code works just fine when running in server (with v100). And after I move it to my local computer (2080). Except for hardware env change, I don’t think there is any software env change with respect to python env. And I have this<br>
<code> Thread 1 "python" received signal SIGSEGV, Segmentation fault. visit_reachable (op=0x2007ffebb394fb0, reachable=0x555555874360 &lt;generations+64&gt;)     at /tmp/build/80754af9/python_1588903631989/work/Modules/gcmodule.c:409 409	/tmp/build/80754af9/python_1588903631989/work/Modules/gcmodule.c: No such file or directory. (gdb) l 404	in /tmp/build/80754af9/python_1588903631989/work/Modules/gcmodule.c</code></p>
<p>Any idea why this could happen?</p>
<p><span class="hashtag">#Update</span><br>
I feel there something to do with memory…as I reduced my max_seq_length, and the error’s gone. But I am still confused</p>
Smooth L1 Loss return inf value

Can_Nguyen: <p>Hi,</p>
<p>My localization loss using Smooth L1 Loss and somehow return inf value.<br>
Can you pls explain why it happens and how to fix it?</p>
<p>Thanks in advance!</p>

Can_Nguyen: <p>I’ve figured out it myself.<br>
Having boxes with zero width, height causes this issue. Just remove it!</p>
How to return the index of max and compute acc in C++

Chen0729: <p>Hi, all</p>
<p>I would like to get maximum index from output 64 by 10 and compare with ground truth y with size 64 by 1 to obtain accuracy in libtorch c++.<br>
It seems max only return max value not index.<br>
How shall I do that please ?<br>
Thanks.</p>

ptrblck: <p><code>torch::argmax</code> should work and would give you the indices for the corresponding max. values.</p>

Chen0729: <p>Thank you for your comments.</p>
<p>Much appreciated.</p>
Multiplication two tensor in pytorch

Toby: <p>Hi,</p>
<p>Sorry for this kind of question, it is maybe because I’m too weak at linear algebra.</p>
<p>I am trying to implement <a href="https://arxiv.org/pdf/1712.07628.pdf&amp;" rel="noopener nofollow ugc">new optimizer strategy</a>. The author mentioned this formula.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/4/345951a3108755f1a2f57844bbd0ba4c29966897.png" alt="fomular" data-base62-sha1="7t6bZ5GblKXSGqY8YgfNOLmBCnl" width="295" height="91"></p>
<p>Where gk is the gradient tensor and pk is the same shape tensor as gk. For example, if the gradient tensor has the shape (c,m,n) then its transpose tensor will have the shape is (n,m,c).</p>
<p>How can I do the multiplication between two tensors to get the scalar result? In that paper:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c59082f055e3bf870cd5e6aba87184f2374c6bfe.png" alt="formula2" data-base62-sha1="sbJFUUzxfFom4iNYkuQscKStnEW" width="486" height="85"></p>
<p>The author also told that pk different from 0 and the multiplication is smaller than 0.</p>

InnovArul: <p>I am not sure about this paper.<br>
Typically, you can consider the parameters as an 1D array (vectorize the conv and fully connected layer parameters) and correspondingly the gradients will also be an 1D array.<br>
Further, you can do <code>p^T.g</code> as normal inner product.<br>
Would this work?</p>

Toby: <p>You mean I should flatten the gradient tensor?</p>
<p>Or for gk shape (c,m,n) and pk transpose shape is (c,n,m) then multiply to get the result tensor shape is (c, n, n) (equal zero if all elements are zero)</p>

InnovArul: <aside class="quote no-group" data-username="Toby" data-post="3" data-topic="105975">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/toby/48/29648_2.png" class="avatar"> Toby:</div>
<blockquote>
<p>You mean I should flatten the gradient tensor?</p>
</blockquote>
</aside>
<p>yes. To me, it seems that way.<br>
But I am not sure about the context of this operation. You might know it better.</p>

Toby: <p>At first, I was about to flatten it, but I am not sure is it correct with what the author means or not, maybe I have to dig deeper into mathematics.</p>
<p>Anyway, thank you for your help and very fast answer.</p>

Toby: <p>Okay, flatten is the solution where they implement it in <a href="https://github.com/jettify/pytorch-optimizer/blob/master/torch_optimizer/swats.py" rel="noopener nofollow ugc">this repo</a></p>
Loss function doesn't compute gradient

sigma_x: <p>The network outputs a number of vectors for which I compute distance and derive a SoftMin probability distribution (a type of clustering algorithm). This output is of the form <code>(total_clustering_prob, target_classes)</code>:</p>
<pre><code class="lang-auto">nllloss = NLLLoss(size_average=False, reduce=True)

[0.32,0.33, 0.35] [1]
....
[0.31,0.39, 0.3] [2]
</code></pre>
<p>I plug it into NLLLoss:<br>
<code>        total_clustering_loss = nllloss(total_clustering_prob, target_classes)</code><br>
and the resulting loss doesn’t have a <code>grad_fn</code>, so all the previous layers don’t compute gradients either. Why does this happen? Do I need to all computations within <code>nn.Module</code> subclass?</p>

InnovArul: <p>No. All computations need not be within <code>nn.Module</code> subclass.<br>
You have to traceback and see if the computation graph remains connected (if you use <code>detach() / .item()</code> etc, the graph disconnects) and at least one operand (parameters/inputs) in the graph has <code>requires_grad=True</code>.</p>

sigma_x: <p>I didn’t realize .<code>item()</code> detaches the graph. I owe you a beer!</p>
GRU choosing correct output

Glowtorch: <p>Hallo,</p>
<p>I am trying to implement a time-series forecasting algorithm using pytorch internal gru module.<br>
However, I am a bit confused when it comes to choosing the correct output for my specific use-case.</p>
<p>After searching for answers I came across this picture, which clarified some things for me.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e7496a33d835f085d800ee17c0ade05895a89551.png" data-download-href="https://discuss.pytorch.org/uploads/default/e7496a33d835f085d800ee17c0ade05895a89551" title="lstm-output-vs-hidden"><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e7496a33d835f085d800ee17c0ade05895a89551.png" alt="lstm-output-vs-hidden" data-base62-sha1="x03AlD3QYEjqGqTmkDe4za9w3df" width="583" height="500" data-dominant-color="EDEFF2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lstm-output-vs-hidden</span><span class="informations">640×548 20.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Problem definition:<br>
I have a time-series of lagged, hourly price-data, let´s say 24*7 = 168 datapoints as input.<br>
I then want to predict the next 24 timesteps.</p>
<p>However, I am unsure which output i should use in this case.<br>
Should I use the last hidden state of the last layer of the GRU network ?<br>
Edit: Or should I use the hidden state (=output) of every GRU cell in the network ?</p>
<p>Any help would be greatly appreciated.</p>
<p>Best Regards,<br>
Glowtorch</p>

vdw: <p>I don’t think that there is a clear-cut reason why to choose one solution over the other.</p>
<p>The basic sequence-to-sequence setup is to encode the input sequence as the last hidden state, and give it to the decoder as its first hidden state.</p>
<p>However, nobody is stopping you to use the outputs of all time steps – and probably do some max or average pooling, or push it through a linear layer (as all your input sequences seem to have the same length) – and give this to the decoder as its first hidden state.</p>
<p>I think I once tried “last hidden state” vs. “max/avg pooling” for an LSTM/GRU-based classifier with similar results. But that’s just purely anecdotal, and might differ much more in your case.</p>
<p>I would simply try both approaches and see what happens, the differences to the code are only 2-3 lines of codes anyway.</p>
How to re-iterate an iterable dataset using multiple workers

JamesDickens: <p>If I have an iterable dataset (loading tf records of images and bounding boxes), and I wish to repeat the dataset epoch after epoch, my code before realizing a problem was:</p>
<blockquote>
<p>for epoch_num in range(epochs):<br>
data_iterator = iter(data_func(*data_args))<br>
for batch in data_iterator:<br>
/// training stuff</p>
</blockquote>
<p>The issue with this approach is that my data_func gets a dataloader object where the number of workers is between 1 and 8. It seems each time I execute the line</p>
<blockquote>
<p>data_iterator = iter(data_func(*data_args))</p>
</blockquote>
<p>I am starting new processes (within a process) which is not desirable. How can I use the processes already available?</p>

ptrblck: <p>You could use <code>persistent_workers=True</code> in the <code>DataLoader</code> creation and iterate it directly via:</p>
<pre><code class="lang-python">for batch in loader:
    ...
</code></pre>
<p>which will make sure to keep the workers alive.</p>

JamesDickens: <p>Assuming I set the flag persistent_workers=True in my call to the dataloader object to get my dataloader, then how do I restart the iterator for each epoch?</p>

ptrblck: <p>The loop over the <code>DataLoader</code> will automatically reuse the iterators.<br>
I haven’t looked into the implementation of <code>persistent_workers</code>, but would assume that manually recreating the iterator wouldn’t work (but you could try it nevertheless and see, if the startup time is slower).</p>
Can I parallelize model in multiple gpu?

Hannibal046: <p>Hello,Guys.</p>
<p>I am currently stuck in a problem where I have multiple gpus with 11G memory, but the model and optimizer I use is pretty huge and complicated. So even  set the batch size  == 1, it will still yield Cuda Out of Memory.</p>
<p>What can I do to figure this out?</p>
<p>Can I put different parts of the model in different gpu? Because I notice that most of the memory consumption consists of model parameters,  grads and optimizer inner state.</p>
<p>After load the model to gpu , it consumes 3G memory. But I used 2 optimizer for different part of the model. After the first loss.backward() and optimizer.step() with batch_size == 1 ,the single gpu is almost fully occupied.</p>
<p>Or how can I restore the gpu memory to 3G after first optimization? I tried to delete the loss and zero_grad the model, but it still does not work : (</p>
<p>Looking forward to your reply!</p>

ptrblck: <aside class="quote no-group" data-username="Hannibal046" data-post="1" data-topic="106037">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hannibal046/48/32289_2.png" class="avatar"> Hannibal046:</div>
<blockquote>
<p>Can I put different parts of the model in different gpu?</p>
</blockquote>
</aside>
<p>Yes, you could use model sharding and push different parts of the model (parameters) to specific devices via <code>to('cuda:id')</code> and use the same operation in the <code>forward</code> to push the activation to the right device.</p>
<p>PyTorch Lightning provides a beta of <a href="https://medium.com/pytorch/pytorch-lightning-1-1-model-parallelism-training-and-more-logging-options-7d1e47db7b0b">sharded training</a>, which might be interesting for you.<br>
CC <a class="mention" href="/u/williamfalcon">@williamFalcon</a> for more information <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Hannibal046: <p>Thx! I will check it later.</p>
<p>And I noticed you once said in another topic that torch.utils.checkpoint may help when Cuda out of memory with batch size equals to 1. Does that fit to my situation? Appreciate so much !</p>

ptrblck: <p>Yes, <code>torch.utils.checkpoint</code> would trade compute for memory and could thus lower the memory usage.</p>
What is JIT "NNC" short for?

eval: <p>Out of curiosity, anyone knows what the abbreviation “NNC” in JIT graph fuser means? Thanks.</p>

tom: <p>Neural Net Compiler:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/6e4de445010ef5e9e0438e4a4c16f6ef3129af14/test/cpp/tensorexpr/tutorial.cpp#L13" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/6e4de445010ef5e9e0438e4a4c16f6ef3129af14/test/cpp/tensorexpr/tutorial.cpp#L13" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/6e4de445010ef5e9e0438e4a4c16f6ef3129af14/test/cpp/tensorexpr/tutorial.cpp#L13</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="3" style="counter-reset: li-counter 2 ;">
<li>// This tutorial covers basics of NNC's tensor expressions, shows basic APIs to</li><li>// work with them, and outlines how they are used in the overall TorchScript</li><li>// compilation pipeline. This doc is permanently a "work in progress" since NNC</li><li>// is under active development and things change fast.</li><li>//</li><li>// This Tutorial's code is compiled in the standard pytorch build, and the</li><li>// executable can be found in `build/bin/tutorial_tensorexpr`.</li><li>//</li><li>// *** What is NNC ***</li><li>//</li><li class="selected">// NNC stands for Neural Net Compiler. It is a component of TorchScript JIT</li><li>// and it performs on-the-fly code generation for kernels, which are often a</li><li>// combination of multiple aten (torch) operators.</li><li>//</li><li>// When the JIT interpreter executes a torchscript model, it automatically</li><li>// extracts subgraphs from the torchscript IR graph for which specialized code</li><li>// can be JIT generated. This usually improves performance as the 'combined'</li><li>// kernel created from the subgraph could avoid unnecessary memory traffic that</li><li>// is unavoidable when the subgraph is interpreted as-is, operator by operator.</li><li>// This optimization is often referred to as 'fusion'. Relatedly, the process of</li><li>// finding and extracting subgraphs suitable for NNC code generation is done by</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I’ve written a JIT tutorial on fusers (from a user perspective mostly looking into how things reach the fuser, hopefully online soon) and I standardized on calling them the TensorExpr fuser and the CUDA fuser.</p>
<p>Best regards</p>
<p>Thomas</p>

eval: <p>This is very helpful. Thanks</p>
Parallelizing models of different classes over multiple GPUs

nullgeppetto: <p>I’m interested in learning what are the good practices in parallelizing pytorch models over multiple GPUs; more specifically I have two questions.</p>
<p>My <strong>first question</strong> concerns <code>nn.DataParallel</code> and a class that inherits from it, <code>DataParallelPassthrough</code>. More specifically, in <a href="https://github.com/anvoynov/GANLatentDiscovery" rel="noopener nofollow ugc">this</a> work, a pre-trained GAN generator, <code>G</code>, which has been set to evaluation mode, is used for generating images. This model, as an instance of a specific class is <a href="https://github.com/anvoynov/GANLatentDiscovery/blob/master/trainer.py#L177" rel="noopener nofollow ugc">used</a> as:</p>
<pre><code class="lang-auto">G = DataParallelPassthrough(G)
</code></pre>
<p>where <code>DataParallelPassthrough</code> is defined as follows:</p>
<pre><code class="lang-auto">from torch import nn

class DataParallelPassthrough(nn.DataParallel):
    def __getattr__(self, name):
        try:
            return super(DataParallelPassthrough, self).__getattr__(name)
        except AttributeError:
            return getattr(self.module, name)
</code></pre>
<p>How is <code>DataParallelPassthrough</code> different than standard <code>nn.DataParallel</code>? Why should one prefer the former over the latter? I have seen it being used in some repos, but I couldn’t find any good explanation why. Could you help me understand?</p>
<p>Now, except for the aforementioned generator model <code>G</code>, there is another model (an instance of a different class), which takes <code>G</code>'s generated images as input. You may think of it as a ResNet-like model. However, this model is not set as an instance of <code>DataParallelPassthrough</code>. This causes some issues. For instance, while the whole model would fit in a 32GB Tesla V100, it doesn’t in a pair of 16GB V100’s.</p>
<p>My <strong>second question</strong> is how should I parallelize both models. Should I instantiate both models using <code>nn.DataParallel</code> or <code>DataParallelPassthrough</code>? I would like to avoid the case where I load each model to a specific GPU device (e.g., using <code>.to()</code>).</p>
<p>Thank you!</p>
Cannot find reference 'FloatTensor' in '__init__.py'

falmasri: <p>did FloatTensor changed in the new version  ?</p>

jpeg729: <p>Isn’t it called FloatTensor not TensorFloat?</p>

albanD: <p>Hi,</p>
<p>You can see that <code>FloatTensor</code> is expressed as being part of torch <a href="https://github.com/pytorch/pytorch/blob/70ba50c3d49c0454210ead1ad931aa39520b10f8/torch/__init__.py#L16-L25">here</a> and it actually comes directly from the C backend that is loaded <a href="https://github.com/pytorch/pytorch/blob/70ba50c3d49c0454210ead1ad931aa39520b10f8/torch/__init__.py#L77">here</a>.</p>

Noam_Salomonski: <p>so… Is there a solution? Pycharm thinks it doesn’t exist, probably due to <a href="https://stackoverflow.com/questions/23248017/cannot-find-reference-xxx-in-init-py-python-pycharm" rel="noopener nofollow ugc">this</a></p>
How to create a learnable input?

Boltzmachine: <p>I want to create a learnable input matrix. So I create it as nn.Parameter. But I want to use a relu to make it non-negative and a softmax to make it normalized.</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(10, 10))
    
    def forward(self, x):
        self.weight = F.softmax(self.weight.relu(), dim=1)
</code></pre>
<p>The error is</p>
<blockquote>
<p>cannot assign ‘torch.FloatTensor’ as parameter ‘weight’ (torch.nn.Parameter or None expected)</p>
</blockquote>
<p>So what is the right way to do it? <img src="https://discuss.pytorch.org/images/emoji/apple/pleading_face.png?v=12" title=":pleading_face:" class="emoji" alt=":pleading_face:" loading="lazy" width="20" height="20"></p>

InnovArul: <aside class="quote no-group" data-username="Boltzmachine" data-post="1" data-topic="106109">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/boltzmachine/48/32323_2.png" class="avatar"> Boltzmachine:</div>
<blockquote>
<p><code>self.weight = F.softmax(self.weight.relu(), dim=1)</code></p>
</blockquote>
</aside>
<p>I am not sure of the context on what you are trying to achieve.<br>
I guess, you do not need to assign <code>self.weight</code> again. It is already learnable and it will accumulate gradients when you backpropagate.</p>

Boltzmachine: <p>I want to create an adjacency matrix in GNN and want it to be learnable.<br>
If I do not assign it in <code>forward</code> again, where should I apply <code>relu</code> and <code>softmax</code></p>

InnovArul: <p>I see. One way is that, you can consider <code>self.weight</code> as logits and apply <code>softmax</code> in <code>forward</code> to get the adjacency matrix. (not sure why do we need <code>relu</code>), then continue with GNN operation.<br>
But you do not need to assign it back to <code>self.weight</code>, as the logits will be learned when you <code>loss.backward()</code> and <code>optimizer.step()</code></p>

Boltzmachine: <p>Oh, yeah. Now I find this question is stupid. I should assign it to another variable. Thanks for your reply <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
Discrepancy between theory and practice

kolorado: <p>Hi,</p>
<p>in theory, these should give the same result, but these two layers produced different (but similar) results:</p>
<pre><code class="lang-auto">import torch
from torch import nn

b=20
outc = 10
inc = 10
res = 32
input_tensor = torch.randn((b, inc, res, res))
cnn_layer = nn.Conv2d(in_channels=inc, out_channels=outc, kernel_size=res,)
out_cnn_layer = cnn_layer(input_tensor)
fc_layer = nn.Linear(in_features=inc * res * res, out_features=outc)
fc_layer.weight.data = cnn_layer.weight.data.reshape(outc, -1)
fc_layer.bias.data = cnn_layer.bias.data
out_fc_layer = fc_layer(input_tensor.view(b, -1))
assert(torch.allclose(out_cnn_layer.view(b,-1),out_fc_layer))
</code></pre>

InnovArul: <p>For me, this test passes!<br>
pytorch 1.6.0</p>

kolorado: <p>Did you run it multiple times? because mine is also: ‘1.6.0’</p>

InnovArul: <p>I ran 15 times (or more!) and all of them passed.<br>
Oh wait! I missed an assert. Sorry for the misunderstanding.<br>
I will check again.</p>

InnovArul: <p>I observe this behavior too.<br>
I can see that the CNN and FC outputs are equal up to an absolute tolerance (<code>atol</code>) of <code>10^-5</code>.</p>
<p><a class="mention" href="/u/alband">@albanD</a>: do you have some comments on this?<br>
In <code>double</code> data type, I would expect the absolute tolerance level should be even lesser than <code>10^-5</code>. Am I missing something here?</p>

albanD: <p>Hi,</p>
<p>Running on current colab, this is what I see:<br>
The same thing as you: a difference of ~1e-6 difference for float. But adding <code>torch.set_default_dtype(torch.double)</code> at the beginning, it goes down to ~1e-15. <a class="mention" href="/u/innovarul">@InnovArul</a> did you set it properly to double?<br>
So it looks like it is expected loss of precision because of floating point numbers.</p>
<p>Also note that the deeper your network is gonna be, the larger this difference is gonna be as most operation amplify a small difference that happened at the beginning.</p>

InnovArul: <p>I see. Thanks <a class="mention" href="/u/alband">@albanD</a>.  I did not set it to double. I assumed CPU has default type as Double.</p>

albanD: <p>No contrary to numpy, we default to float as it makes a big difference in terms of runtime (especially on GPU) and is enough precision for most (if not all) deep learning applications.</p>
Custom autograd function list/dict input

nchodosh: <p>The documentation for extending pytorch has this to say about inputting list or dicts to autograd functions:</p>
<p>“All kinds of Python objects are accepted here. Tensor arguments that track history (i.e., with <code>requires_grad=True</code>) will be converted to ones that don’t track history before the call, and their use will be registered in the graph. Note that this logic won’t traverse lists/dicts/any other data structures and will only consider Tensor s that are direct arguments to the call.”</p>
<p>I am interested in creating an autograd function which accepts a list of Tensors that track history which are used in the computation. The quoted documentation suggests that I would need to manually make sure that those Tensors are added to the computation graph. Is there an example of how to do this anywhere? I can’t seem to find one.</p>

albanD: <p>Hi,</p>
<p>Unfortunately, this is not possible at the moment.<br>
You can simply doing it by passing varargs: <code>YourFunc.apply(*your_list)</code>. And when creating the function <code>def forward(ctx, *your_list)</code>.</p>

Jide_Olabisi: <p>Hi, Do you know how I can register a tensor in the graph passed as an index argument to nn.ModuleList so that it can be trained?</p>

albanD: <p>Hi,</p>
<p>I am not sure to follow, ModuleList is a torch.nn construct and unrelated to this discussion about custom Functions.</p>
<p>Maybe open a new topic with a code sample that shows what you’re trying to achieve?</p>
What does torch.arccos() return?

rahulvigneswaran: <p>Does <a href="https://pytorch.org/docs/stable/generated/torch.acos.html#torch.acos" rel="noopener nofollow ugc">torch.arccos()</a> return value in radians or degrees ?</p>

rahulvigneswaran: <p>The torch.arcos() will return values in <strong>radians</strong>.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.acos(torch.tensor(0.0))
tensor(1.5708)
</code></pre>
Is there operator that can compute `a * x + b`

coincheung: <p>Hi,</p>
<p>I have 3 tensors, and I would like to compute this:</p>
<pre><code class="lang-auto">a = torch.randn(1, 32, 1, 1)
x = torch.randn(16, 32, 64, 64)
b = torch.randn(1, 32, 1, 1)
out = a *x + b
</code></pre>
<p>However, I found that this <code>a *x + b</code> is not built-in function which makes inference not as fast as expected. So is there any built in function that can compute this in the c++ side like this ?</p>
<pre><code class="lang-auto">out = some_func(a, x, b)
</code></pre>

googlebot: <p>there is b.addcmul(a,x), <a href="https://discuss.pytorch.org/t/how-to-implement-efficient-element-wise-ax-b-function/105274/5">jit code</a> is another possibility</p>