akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Training time vs number of samples

lycaenidae: <p>Hi,<br>
Does training time increase linearly with the number of samples available in the training?<br>
In the first training, I have 400 samples.<br>
In the second training, I have 800 samples.</p>
<p>I am outputting the epoch loss. However, when I double the dataset, training time increases 4 times more. I am confused. Should not I expect that time will be doubled as well? All other parameters i.e., batch size is the same.</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e51f48ace64edc755d7d12aeb58cf0d02f96b1d0.png" alt="Screenshot from 2022-06-29 15-54-51" data-base62-sha1="wGUmHuOu59BjpG7Geu4DgsC7A4w" width="651" height="482"></p>

eqy: <p>That is surprising, is the time per step changing and observable as well?</p>
Difference between tensor.new_empty(temp.size()).normal_() and torch.randn(temp.size())?

4158ndfkvBHJ1: <p>The question is basically in the title. I am reading the SWAG implementation (<a href="https://github.com/wjmaddox/swa_gaussian/blob/master/swag/posteriors/swag.py" class="inline-onebox" rel="noopener nofollow ugc">swa_gaussian/swag.py at master · wjmaddox/swa_gaussian · GitHub</a>), and the <code>sample_fullrank</code> function uses gaussian sampling at line 127. The sample vector is defined as:</p>
<pre><code class="lang-python">  cov_mat_sqrt.new_empty((cov_mat_sqrt.size(0),), requires_grad=False).normal_()
</code></pre>
<p>It seems like a really convoluted way of making a tensor of size <code>cov_mat_sqrt.size(0)</code> filled with gaussian values.</p>
<p>First I don’t really understand the point of <code>tensor.new_empty(shape)</code>, what’s the difference with <code>torch.empty(shape)</code> ? Why call this function on a specific tensor if the new empty tensor is not going to inherit it’s shape?</p>
<p>And if that’s just initializing an empty tensor, is there any functionality difference between this line and a simpler <code>torch.randn(cov_mat_sqrt.size(0))</code> ?</p>
<p>I understand that there may be different ways of achieving the same result in PyTorch, I just want to make sure I am not missing any subtle functionality of the implementation used in this codebase.</p>
<p>Thanks in advance to anybody who can clear this up <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <aside class="quote no-group" data-username="4158ndfkvBHJ1" data-post="1" data-topic="155373">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/4/d07c76/48.png" class="avatar"> 4158ndfkvBHJ1:</div>
<blockquote>
<p>First I don’t really understand the point of <code>tensor.new_empty(shape)</code>, what’s the difference with <code>torch.empty(shape)</code> ?</p>
</blockquote>
</aside>
<p>The <code>new_*</code> operations create a tensor with the same <code>dtype</code> and <code>device</code> as the source tensor it was called on, so it might be a convenient approach if you want to avoid using <code>device=src.device, dtype=src.dtype</code> in <code>torch.empty</code>.</p>
<aside class="quote no-group" data-username="4158ndfkvBHJ1" data-post="1" data-topic="155373">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/4/d07c76/48.png" class="avatar"> 4158ndfkvBHJ1:</div>
<blockquote>
<p>And if that’s just initializing an empty tensor, is there any functionality difference between this line and a simpler <code>torch.randn(cov_mat_sqrt.size(0))</code> ?</p>
</blockquote>
</aside>
<p>Yes, potentially the <code>dtype</code> and <code>device</code> as <code>torch.randn</code> would use its default (<code>float32</code> and <code>cpu</code>).</p>
Trainer.test() is not working

ranaban: <p>I want to test a trained model using the test dataloader<br>
<code>test_dl = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)</code></p>
<p>So, I tried with</p>
<pre><code class="lang-auto">results = trainer.test(dataloaders=test_dl)
</code></pre>
<p>gives the below error.</p>
<pre><code class="lang-auto">TypeError: test() got an unexpected keyword argument 'dataloaders'
</code></pre>
<p>I tried with the below instruction also</p>
<pre><code class="lang-auto">results = trainer.test(test_dl)
</code></pre>
<p>It gives.</p>
<pre><code class="lang-auto">AttributeError: 'DataLoader' object has no attribute 'log'
</code></pre>

ptrblck: <p>I guess you are using a higher-level API such as Lightning?<br>
If so, check if this argument was added in a newer version (or was removed) etc.</p>
Train multiple independent models on a single GPU

msuliman: <p>Hi,</p>
<p>I have what seems to be a very basic issue but couldn’t find a solution for it.</p>
<p>I have a GPU of 24GB, and I am trying to train two different independent models; each takes 2GB only. When I train the two models simultaneously on that GPU, the training speed on the two models reduces to around (1/5) the speed of training one of them.</p>
<p>What could be the issue?</p>

ptrblck: <p>You might be creating new bottlenecks in your code e.g. if both applications are trying to load data while your storage isn’t fast enough to feed them.<br>
You could profile the script the see where the bottleneck is created.</p>
<p>Also, to parallelize CUDA workloads on the GPU you would need to make sure enough compute resources are free. E.g. if one script uses all SMs in e.g. a matmul operation, the other kernel won’t be able to run, so you might not expect to see perfect parallelization (even without other bottlenecks) but it of course depends on the actual use case.</p>

blackbirdbarber: <aside class="quote no-group" data-username="msuliman" data-post="1" data-topic="155306">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/f9ae1b/48.png" class="avatar"> msuliman:</div>
<blockquote>
<p>What could be the issue?</p>
</blockquote>
</aside>
<p>Maybe you are not using Nvidia gpu, in which case this may be true.</p>

blackbirdbarber: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="155306">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>storage isn’t fast enough to feed them</p>
</blockquote>
</aside>
<p>What storage you recommend?</p>

ptrblck: <p>I would use an SSD at least and avoid trying to load data from a spinning disk.</p>
Index out of range error

TadpoleJJ: <p>Hi I’m getting this index out of range error when plotting the results of a 2 layer neural network.</p>
<p>The input is a CSV file with (x,y) coordinates and a y-output for whether that point is deemed correct or not according to some rule.</p>
<p>Could I have some help in understanding the error?</p>
<pre><code class="lang-auto">plt.pcolormesh(xrange,yrange,(hid.cpu()[:,node]).view(yrange.size()[0],xrange.size()[0]), cmap='Wistia', shading='auto')
IndexError: index 1 is out of bounds for dimension 1 with size 1
</code></pre>
<p>PyTorch code</p>
<pre><code class="lang-auto">class Full3Net(torch.nn.Module):
    def __init__(self, hid):
        super(Full3Net, self).__init__()
        self.in_hid = torch.nn.Linear(2,hid)
        self.hid_out = torch.nn.Linear(hid,1)

    def forward(self, input):
        output1 = self.in_hid(input)
        self.hid1 = torch.tanh(output1)
        output2 = self.hid_out(self.hid1)
        self.hid2 = torch.tanh(output2)
        output = torch.sigmoid(self.hid2)
        return output
</code></pre>

Andrei_Cristea: <p>Can you supply a bit more of the code? It looks like the error is due to the fact that your network output is 1-dimensional (number of out-features of <code>self.hid_out</code> is 1) so if you try to index it at position <code>node=1</code> you’re going to get an out-of-bounds error.</p>
<p>What shape would you like your output to have instead? Perhaps you want to use a Conv2d layer rather than a Linear one, if the dimensionality of your inputs is 2d (x, y) spatial coordinates.</p>
Got nan in forward with `torch.amp`

songyuc: <p>Hi, guys,<br>
I met nan error when using <code>torch.amp</code> like:</p>
<pre data-code-wrap="python"><code class="lang-python">...
# forward pass
with torch.cuda.amp.autocast(self.use_amp):
    x = self.backbone(img)
    x = self.neck(x)

    if self.qa.check_nan:
        assert not x.isnan().any()
        assert not self.dict_feats["layer2"].isnan().any()
        for p in self.decoder2.parameters():
            assert not p.isnan().any()

    if isinstance(self.decoder2, FuseDecoder):
        encode_data = self.decoder2(x, self.dict_feats["layer2"])
    else:
        raise NotImplementedError

if self.qa.check_nan:
    assert not encode_data.isnan().any()
...
</code></pre>
<p>, and the error is like,</p>
<blockquote>
<p>in train_epochs<br>
output = self.model(batch_img)<br>
File “/home/user/software/python/anaconda/anaconda3/envs/conda-general/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1110, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/…net.py”, line 331, in forward<br>
assert not encode_data.isnan().any()<br>
AssertionError</p>
</blockquote>

<p>What can I do to solve this issue?</p>
<p>By the way, I do not know whether these debug codes are enough to show the problem, so, any debugging suggestion will also be appreciated.</p>
<p>Thanks!</p>

ptrblck: <p>Could you describe your use case a bit more and e.g. which layer creates the invalid outputs?<br>
It seems a <code>FuseDecoder</code> is used, but I don’t know what architecture this refers to.</p>

songyuc: <p>I tested it again, and got the nan from the backbone, as</p>
<pre data-code-wrap="python"><code class="lang-python">def forward(self, img):
with torch.cuda.amp.autocast(self.use_amp):
    x = self.backbone(img)
    x = self.neck(x)

    if self.qa.check_nan:
        assert not x.isnan().any()  # Trigger the error 
        assert not self.dict_feats["layer2"].isnan().any()
        for p in self.decoder2.parameters():
            assert not p.isnan().any()
...
</code></pre>
<p>and the error is like,</p>
<blockquote>
<p>File “/home/user/software/python/anaconda/anaconda3/envs/conda-general/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1110, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/…net.py”, line 337, in forward<br>
assert not x.isnan().any()<br>
AssertionError</p>
</blockquote>

<p>where</p>
<pre><code class="lang-auto">...
self.neck = nn.Identity()
...
self.backbone = shufflenetv2.shufflenet_v2_x0_5()
# which has been modified without using FC layer
...
</code></pre>
<p>Modifiled shufflenetv2 is like:</p>
<pre data-code-wrap="python"><code class="lang-python"># Nearly same as the implementation as torchvision, 
# but without FC layer
def _forward_impl(self, x: Tensor) -&gt; Tensor:
	# See note [TorchScript super()]
	x = self.conv1(x)
	x = self.maxpool(x)
	x = self.stage2(x)
	x = self.stage3(x)
	x = self.stage4(x)
	x = self.conv5(x)
	return x
</code></pre>
How to use multilabel binary time series to predict underlying weight

voyo: <p>I have a dataset, where I simulate 8 different households, and if they walked<br>
outside that day. The next day, other people will leave their houses. They are<br>
impacted by the other people, and the different impacts are stored in a matrix<br>
(W=8x8). The people outside one day (in a 8 element 0/1 vector) is then<br>
multiplied with the matrix, to get the next day. This is then simulated for<br>
many time steps, and I also have many simulations. One simulation is in the<br>
shape (1000 x 8), where there are 1000 time steps.</p>
<p>My goal is to reconstruct the matrix from the time series data. I have thought<br>
of using a transformer based approach, and feed in only chunks from the<br>
simulation data, eg. 32x8 from the simulation.</p>
<p>I have found the <code>torch.nn.Transformer()</code>, but I don’t understand what src and tgt<br>
are supposed to be.</p>
<p>Also: Are transformer-based methods the correct architecture to attack this problem?</p>
Can Apple M1 support quantization？

Rahul_Xie: <p>Can Apple M1 support quantization？</p>

HDCharles: <p>you could run the quantization APIs but the actual quantized model you get at the end doesn’t seem like it could run since none of the backends seem to work with M1, fbgemm needs x86 with AVX, and qnnpack needs ARM.</p>

fxmarty: <p>Do you know of any backends, except fbgemm and qnnpack, supporting quantization on M1/M2 chips?</p>

Rahul_Xie: <p>I regret that I don’t know about it.</p>
DDPG agent with convolutional layers for feature extraction

andreasceid: <p>I’m trying to come up with a definition of the critic for a DDPG agent in PyTorch using a CNN as a feature extractor. It is pretty straight forward for the actor model. However, for the critic model I am not sure. Below, I’ve given my code for the actor model.</p>
<pre><code class="lang-python">class actor(nn.Module):
    def __init__(self, obs_dim, act_dim, act_limit):
        super(actor, self).__init__()

        channels, height, width = obs_dim
        self.act_limit = act_limit

         self.actor = nn.Sequential(OrderedDict([
            ('conv1', nn.Conv2d(in_channels=channels, out_channels=32, kernel_size=8, stride=4)),
            ('relu1', nn.ReLU()),
            ('conv2', nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2)),
            ('relu2', nn.ReLU()),
            ('conv3', nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1)),
            ('relu3', nn.ReLU()),
            ('flatten', nn.Flatten()),
            ('linear1', nn.Linear(3136, 512)),
            ('relu4', nn.ReLU()),
            ('linear2', nn.Linear(512, act_dim))
        ]))
    def forward(self, obs):
        # Return output from network scaled to action space limits.
        return self.act_limit * self.actor(obs)
</code></pre>
<p>My confusion lies at the part with the concatenation of the action and the observation vectors and how to actually integrate that in my CNN model. A typical critic model is a MLP that looks like this:</p>
<pre><code class="lang-python">class critic(nn.Module):
    def __init__(self, obs_dim, act_dim, act_limit):
        super(critic, self).__init__()
        self.act_limit = act_limit

        self.fc1 = nn.Linear(obs_dim + act_dim, 256)
        self.fc2 = nn.Linear(256, 256)
        self.fc3 = nn.Linear(256, 256)
        self.out = nn.Linear(256, 1)

    def forward(self, x, actions):
        x = torch.cat([x, actions / self.act_limit], dim=1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        q_value = self.out(x)

        return q_value
</code></pre>
<p>Any ideas on how to define a critic with the same (convolutional) feature extractor?</p>

vmoens: <p>Hi <a class="mention" href="/u/andreasceid">@andreasceid</a><br>
Have a look at our <a href="https://github.com/facebookresearch/rl/blob/dbc7ba0293d2d3321f4285ae2f21ee2c7d5c6dcd/torchrl/modules/models/models.py#L769-L845" rel="noopener nofollow ugc">torchrl implementation</a>: usually you would use a CNN for embedding the observation and concatenate the action with the embedded observation.<br>
Hope that helps!</p>
How to balance dataset?

farahnaz: <p>Hi<br>
I have a dataset for classification problem which it is imbalanced .it contains 68 samples for class1, 98 samples for class2 and 21 samples for class 3.how should i balance my dataset???</p>
<pre><code class="lang-auto">train_set = CostumDataset(phase='train')
validation_set = CostumDataset(phase='validation')
test_set = CostumDataset(phase='test') 
</code></pre>
<pre><code class="lang-auto">train_loader=DataLoader(train_set,batch_size=1,num_workers=1,shuffle=True)

val_loader=DataLoader(validation_set,batch_size=1,num_workers=1,shuffle=True)

test_loader=DataLoader(test_set,batch_size=1,num_workers=1,shuffle=True) 
</code></pre>

omid_khalafbeigi: <p>Hi.<br>
There are so many techniques for balancing dataset, but usage of them is depend on type of your data. In fact, these methods known as “data augmentation”.<br>
Image: <a href="https://iq.opengenus.org/data-augmentation/" class="inline-onebox" rel="noopener nofollow ugc">Data augmentation Techniques</a><br>
Text: <a href="https://arxiv.org/abs/2107.03158" class="inline-onebox" rel="noopener nofollow ugc">[2107.03158] A Survey on Data Augmentation for Text Classification</a><br>
Audio: <a href="https://medium.com/@makcedward/data-augmentation-for-audio-76912b01fdf6" class="inline-onebox" rel="noopener nofollow ugc">Data Augmentation for Audio. Data Augmentation | by Edward Ma | Medium</a></p>

farahnaz: <p>thanks .<br>
actually, my data type is correlation matrix from fmri images. should i use augmentation for image?</p>

omid_khalafbeigi: <p>What do you mean by “correlation matrix”? are you processing “correlation matrix” of FMRI using algorithms?!<br>
If your meaning is FMRI images (without correlation matrix), you can use methods such as rotation, flip, shift, and etc.<br>
Don’t forget, you can use pre-trained models when haven’t enough dataset, such as: ResNet50.</p>

farahnaz: <p>actually i use resnet18 thank you <img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=12" title=":innocent:" class="emoji" alt=":innocent:" loading="lazy" width="20" height="20"></p>
How do you debug libtorch using gdb

Kiki_Rizki_Arpiandi: <p>I wonder how do you guys debug libtorch using gdb ?</p>

albanD: <p>What do you mean?<br>
You can debug your cpp program that uses libtorch as any other cpp program. Is there something specific you want to do?</p>

Kiki_Rizki_Arpiandi: <p>Yes there is specific thing I want to do, I want to print read torch::Tensor elements using gdb. I use CLion IDE, The tensor elements and shape does not show up on CLion debugger.</p>

albanD: <p>The <code>.sizes()</code> function should return the sizes to you.<br>
You can access the values by passing the Tensor to <code>&lt;&lt;</code>, not sure how to do that in the CLion ide though <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

Kiki_Rizki_Arpiandi: <p>thank you for your response, Hmmm… it is what I usually use, or sometime I use <code>at::print()</code> to print the entire tensor … using std out will mess my code,… is there any standard or best practice to debug libtorch program</p>

sardanian: <p>Did you ever find a solution to this issue? std out is not a great solution and I am looking for something else.</p>
How can I view the contents of a C++/Libtorch tensor while debugging in an IDE like CLION?

sardanian: <p>I am trying to view the contents of tensors while debugging in CLION like I do for PyTorch tensors. However I am unable to view the contents of the tensor.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e2837f66e032c0acdc972fe608c62fed5120fad.png" data-download-href="https://discuss.pytorch.org/uploads/default/3e2837f66e032c0acdc972fe608c62fed5120fad" title="tensor_debugging"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3e2837f66e032c0acdc972fe608c62fed5120fad_2_690x473.png" alt="tensor_debugging" data-base62-sha1="8RRLOdJKGCnoztzN9XlHbKOcooZ" width="690" height="473" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3e2837f66e032c0acdc972fe608c62fed5120fad_2_690x473.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3e2837f66e032c0acdc972fe608c62fed5120fad_2_1035x709.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e2837f66e032c0acdc972fe608c62fed5120fad.png 2x" data-dominant-color="3E4347"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">tensor_debugging</span><span class="informations">1118×767 29.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have gone through these discussion boards but have not found any sort of solution.</p>
<p>I’d like to be able to view the contents like I view a vector’s contents (or something similiar and easy). My only solution at the moment is to cout at runtime, but that is unruly for long tensors and seems much more backwards than when doing it in python.</p>

sardanian: <p>This is my simple tensor in my example:</p>
<p>torch::Tensor tensor = torch::rand({2, 3});</p>
Create custom autograd functions in Pytorch

shyZam: <p>I am trying to migrate the below chainer code to pytorch.<br>
I was looking at the below chainer documentation<br>
<a href="https://docs.chainer.org/en/v7.8.1/reference/generated/chainer.FunctionNode.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.chainer.org/en/v7.8.1/reference/generated/chainer.FunctionNode.html</a></p>
<p>Class CustomErrorFunction(chainer.function_node.FunctionNode):</p>
<pre><code>         def forward(self, inputs):
                self.retain_inputs((0,1,2))
                loss=input[1]
                return loss

         def backward(self, indexes,gy):
                x,y,z = self.get_retained_inputs()
                gy0 = chainer.functions.broadcast_to(gy[0], y.shape)
                return gy0*z, None, None
</code></pre>
<p>I want to know how to create this customer function in pytorch using torch.autograd.Function</p>
<p>Are their any function in pytorch similar to these chainer functions i.e. retain_inputs() and get_retained_inputs()</p>
<p>Any help would be greatly apprciated</p>

ptrblck: <p>I’m not sure how <code>retain_inputs</code> works exactly, but would guess it’s the equivalent to <code>ctx.save_for_backward</code> as described in <a href="https://pytorch.org/tutorials/beginner/examples_autograd/polynomial_custom_function.html">this tutorial</a>.</p>
Functorch does not support torch 1.12.0+cu116

ABD-01: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30244843da220ad95c6014e1704ae1057f25b841.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/30244843da220ad95c6014e1704ae1057f25b841" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30244843da220ad95c6014e1704ae1057f25b841_2_690x482.jpeg" alt="image" data-base62-sha1="6RSEucNUz9ijdXmc9zfZcPP704N" width="690" height="482" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30244843da220ad95c6014e1704ae1057f25b841_2_690x482.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/3/0/30244843da220ad95c6014e1704ae1057f25b841.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/0/30244843da220ad95c6014e1704ae1057f25b841.jpeg 2x" data-dominant-color="44433E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">930×650 189 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I have CUDA version 11.6 hence am using torch-1.12.0+cu116<br>
But functorch requires torch&lt;1.12, &gt;=1.11</p>

ptrblck: <p>I think the <code>functorch</code> release is still blocked by <a href="https://github.com/pytorch/pytorch/issues/80489">#80489</a> so you might need to build it from source for now.</p>
Enumerate with zip for dataloader

Omar_Samir: <p>I am training a network with two separate datasets. I use this line of code to iterate through both of them at the same time:</p>
<p>for epoch in range(M):<br>
for i, (data1, data2) in enumerate(tqdm(zip(dataloader1, dataloader2), total = len(dataloader1))): <em>Rest of code</em></p>
<p>dataloader 2 has twice as much images in total. And hence, this enumerate function will iterate through the shorter dataloader (dataloader1) and will go to the second epoch once the images in dataloader1 are done.</p>
<p>My question is, when I am in the second epoch, will dataloader2 start giving the remainder of the images it has? Or will it start from the beginning, giving the same set of images as in the first epoch.</p>
<p>In other words, in this way, am I wasting half of the images in dataloader2 and not using them at all?</p>

ptrblck: <p>If you don’t shuffle the datasets you would waste 50% of the second dataset. Otherwise the 50% of the samples will be randomly drawn.</p>
Which images are used for the representative models trained in pytorch?

YuhskeHujisaki: <p>Where in pytorch are the images that VGG, MobileNet, etc. are learning?</p>

ptrblck: <p>The pretrained classification models should have been trained on <a href="http://www.image-net.org/challenges/LSVRC/2012/">ILSVRC2012</a> if I’m not mistaken.</p>
Will we get an update on MPS for M2 Chip?

Zohair_Hadi: <p>Wanted to know that will MPS work right off the shelf for the new M2 chip that Apple has just come out with? Or will we need to wait for an update on MPS to have support of it?</p>

philipturner: <p>MPS should work right off the shelf. Apple has already implemented optimized kernels for the A15 (iPhone 13), which shares the same GPU architecture as the M2. This includes <code>simd_shuffle_and_fill</code> if it’s used at all in convolutions (probably not). Otherwise, it’s the exact same kernels as M1, using hardware-accelerated matrix multiplication via <code>simdgroup_matrix</code>. All of what I’m describing should be opaque to PyTorch, as the CPU-visible API of Metal Performance Shaders never changes across architectures or operating systems.</p>
Can MultiheadAttention be optimized for sparse attention masks?

Vadim: <p>I need to apply MultiheadAttenction to batches of sequences whose lengths are very non-uniform, so I have to use a lot of padding + attention masks.   As I understand, masking tokens out does not decrease the amount of computation being done, not the amount of data saved for the backward pass.  <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>Other than trying to construct batches by grouping examples of similar length together, are there any other tricks that people have come up with to deal with this problem?</p>
Model attribute not updated correctly

Erica_Zheng: <p>I want the foward( ) have two modes depend on the <code>self.last_only</code></p>
<p>Now the issue is <code>self.last_only</code> is NOT updated according to my will.</p>
<pre><code class="lang-auto">class Model(nn.Module):

    def __init__(self)
        self.last_only = None  

    def forward(self, x1, x2):
        print('self.last_only = ', self.last_only)
</code></pre>
<pre><code class="lang-auto">def last_only(model, log=print):
    model.last_only = True
    print('model.last_only = ', model.last_only)
</code></pre>
<p><code>model(x1, x2)</code><br>
It prints out <code>None</code>.</p>

Andrei_Cristea: <p>Hi Erica -</p>
<p>You may be looking for something like this:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.last_only = None

    def forward(self, x1, x2):        
        print('self.last_only = ', self.last_only)
        return None

model = Model()
print("before making the change")
model("x", "y")

model.last_only = True
print("after making the change")
model("x", "y")

Output:
before making the change
self.last_only =  None
after making the change
self.last_only =  True
</code></pre>

Erica_Zheng: <p>Thank you, Andrei. It really helps!</p>
Trying to understand an Issue with batch Normalization

yoshcn: <p>Hi everybody,</p>
<p>I am encountering an issue while using batch normalization in my model. I am using a CNN.</p>
<p>In order to debug my network, I took a single batch of 10 samples. here each sample is a picture of (325, 40) so each tensor has dimension (1, 1, 325, 40).  and my full batch is a tensor of size (10, 1, 325,40).</p>
<p>In my training loop I am using training mode: model.train() and for validation model.eval().</p>
<p>I also know that while training, the mean average used with batch normalization  is updated and since I am using batch of 10 I am using a momentum of <strong>0.1 for my batch norm.</strong></p>
<p>At first I can see that both my training loss and my validation loss are decreasing and are of the same scale ( training_loss = O(validation_loss)).</p>
<p>But after a while my validation loss starts diverging.<br>
Once again, here my validation set and my training set are the same dataset.</p>
<p>I think this is due to the compute of my moving average or the learning of parameters gamma and beta. (parameters learnt for best normalization)</p>
<p>Would you have an idea of what might be the problem here  ?</p>
<p>PS:</p>
<p>1/ on my validation set, when I am using model.train() instead of model.eval() I obtain a training_loss and validation_loss really close to each other.</p>
<p>2/ I also tried to use bigger batches (batches of 50) but the same problem occurs.</p>
<p>thanks !</p>
How to export onnx from pytorch model?

Milad_V: <p>Hi all,</p>
<p>I would like to convert my pytorch model to onnx file but I got this error :</p>
<blockquote>
<p>TypeError: conv2d() received an invalid combination of arguments - got (list, Parameter, Parameter, tuple, tuple, tuple, int), but expected one of:<br>
(Tensor input, Tensor weight, Tensor bias, tuple of ints stride, tuple of ints padding, tuple of ints dilation, int groups)<br>
didn’t match because some of the arguments have invalid types: (list, Parameter, Parameter, tuple, tuple, tuple, int)<br>
*(Tensor input, Tensor weight, Tensor bias, tuple of ints stride, str padding, tuple of ints dilation, int groups)<br>
didn’t match because some of the arguments have invalid types: (list, Parameter, Parameter, tuple, tuple, tuple, int)</p>
</blockquote>
<p>here’s my code :</p>
<blockquote>
<p>import os<br>
import DuelingDeepQNetwork as ddqn<br>
import DuelingDDQNAgent as duelingddqnAgent<br>
from cv2 import randn<br>
import gym<br>
import utils<br>
import numpy as np<br>
import matplotlib.pyplot as plt<br>
import torch<br>
import DuelingDDQNAgent as duelingddqnAgent<br>
import DuelingDeepQNetwork as ddqn<br>
model2=ddqn.DuelingDeepQNetwork(0.0001,6,[4,200,200],‘test’,‘checkpoint/’)<br>
print(‘model2’,model2)<br>
state_dict = torch.load(‘q_eval_ddqn_5000.pth’)<br>
<span class="hashtag-raw">#model2</span>.load_state_dict(state_dict)<br>
model2.load_state_dict(state_dict[‘model_state_dict’])<br>
model2.eval()<br>
print(‘pth_loaded’)<br>
dummy_input =[4,200,200]<br>
print(‘Exporting the ONNX file…’)<br>
torch.onnx.export(model2, dummy_input,‘model.onnx’)</p>
</blockquote>
<p>How can I solve that issue ? Thanks</p>
<p>Blockquote</p>

ptrblck: <p>I guess the error is caused in:</p>
<pre><code class="lang-python">dummy_input =[4,200,200]
torch.onnx.export(model2, dummy_input,'model.onnx')
</code></pre>
<p>as <code>torch.onnx.export</code> expects the model input (e.g. a tensor) in your <code>dummy_input</code> not a list of integers.</p>

Milad_V: <p>yes exactly. the problem solved with :</p>
<blockquote>
<p>dummy_input =torch.randn([1,4,200,200], device=“cpu”)</p>
</blockquote>
<p>thanks!</p>
How to gracefully terminate a worker process in torchrun?

zizhao.mo: <p>Hi,</p>
<p>I know that I can dynamically add one worker into the worker group by executing a torchrun command. I want to know if there is a similar command to (on my initiative) terminate an existing worker without influencing other ongoing worker processes?</p>

rvarm1: <p>TorchX / TorchElastic team may have more context on this issue - cc <a class="mention" href="/u/d4l3k">@d4l3k</a></p>

d4l3k: <p>Torchrun manages worker processes on a host. If you want to terminate a single torchrun worker or the all of the workers on a host you can use <code>kill ...</code> with the PID. There’s no torch specific command for killing them.</p>
<p>Notably though, since they’re all linked if one worker dies all of the other workers will have to rerendezvous with each other (typically loading from checkpoint) so it’s possible “without influencing other ongoing worker processes”. This is an unfortunate side effect of NCCL not having any fault tolerance.</p>
<p>What’s the use case here for killing a single worker? If I know more might be able to give a better suggestion</p>

zizhao.mo: <p>Hi,</p>
<p>Thanks for your response. I am working on scheduling DDP jobs and in some of my cases, the DDP job may shrink or expand its number of workers. I know that I can expand the number of workers by executing commands on a host, but I don’t find a Pytorch way to ‘shrink’ the worker number. So according to your suggestion, I just need to kill some processes on the host and then other processes will automatically re-organize properly (e.g., # of workers: 8-&gt;4), right? Or is there any better solution to achieve my goal?</p>
Inheriting the PyTorch DataLoader class for processing "per-worker" outputs

sohomb: <p>Can we inherit the DataLoader class? If so, are there any specific restrictions to it?<br>
I know we can do so for the Dataset class, but I need to know specifically about the DataLoader.</p>
<p>Specifically I need to process the “per-worker” (DataLoader multiprocessing) return values from the dataset <code>__get_item__</code> calls and return the processed output to my main program. <br>
The only way I see it is to inherit the DataLoader module, modify the <code>worker</code> classes to include my desired changes. <br>
Are there any other “cleaner” ways to do this rather than the above? <br>
Also not sure if <code>collate_fn</code> could work here?</p>
<p>Would really appreciate the help! <br>
Thanks…</p>

ptrblck: <aside class="quote no-group" data-username="sohomb" data-post="1" data-topic="154824">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sohomb/48/45962_2.png" class="avatar"> sohomb:</div>
<blockquote>
<p>Specifically I need to process the “per-worker” (DataLoader multiprocessing) return values from the dataset <code>__get_item__</code> calls and return the processed output to my main program.</p>
</blockquote>
</aside>
<p>I think a custom <code>collate_fn</code> should work as you would get all samples from the current batch loaded in this worker process and could then process it further if needed.</p>

sohomb: <p>@<a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a> Thanks for the response…<br>
So, I guess whatever custom code I include in <code>collate_fn</code> would work on the samples at a “batch” level.<br>
Is there any alternative – if I wanted the processing to be done at a single “sample” level?<br>
Or should I employ unpacking the batch and work on each of the samples?</p>

ptrblck: <p>If you want to process each sample, you could perform these operation in the <code>Dataset.__getitem__</code> method. I’m unsure about the worker-specific processing idea, but in case you need to get the worker id you could use:</p>
<pre><code class="lang-python">worker_info = torch.utils.data.get_worker_info()
if worker_info is not None:
    worker_id = worker_info.id
</code></pre>
<p>inside the <code>__getitem__</code>.</p>

sohomb: <p>Thanks for your help…! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
I was able to work it out with a combination of namedtuple returns from the <code>__get_item__</code> calls with the specific <code>worker_id</code> calls (as mentioned above) and a custom <code>collate_fn</code> implementation.</p>
2d Convolution in expansive path in Unet from 256 to 128 feature maps

ZahAbdel: <p>Hi,<br>
how does 2d Convolution in expansive path in Unet from 256 to 128 feature maps work?<br>
I mean normally that happened if we want to extract more feature maps from an input.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3abda23e0dfe066aa41ca5ce2d1eccecc5071ca.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d3abda23e0dfe066aa41ca5ce2d1eccecc5071ca" title="Unbenannt"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3abda23e0dfe066aa41ca5ce2d1eccecc5071ca_2_576x500.jpeg" alt="Unbenannt" data-base62-sha1="ucwW8z5rvjrEuriSwXcnkdQT5X4" width="576" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3abda23e0dfe066aa41ca5ce2d1eccecc5071ca_2_576x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3abda23e0dfe066aa41ca5ce2d1eccecc5071ca.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3abda23e0dfe066aa41ca5ce2d1eccecc5071ca.jpeg 2x" data-dominant-color="EFF1F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Unbenannt</span><span class="informations">699×606 32.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The number of output channels corresponds to the number of kernels in the default setup (no grouped convs etc.).<br>
In this particular example you would use <code>128</code> kernels each with an <code>in_channels</code> attribute of <code>256</code>, which means that each kernel processes <em>all</em> input channels and creates one activation map (channel) as its output.</p>

ZahAbdel: <p>Thanks for replying. Is that meaning, when each kernel processes all input (256), the average of all 256 images is built as output map? Or there is an other way to win one activation map from 256?</p>

ptrblck: <p>Each kernel has a dimension <code>[out_channels, in_channels, height, width]</code> and thus does not take the average of the input activation but processes each input channel with a different set of weights (note the <code>in_channels</code> dimension).<br>
<a href="https://cs231n.github.io/convolutional-networks/#conv">CS231n - Conv</a> explains it in more details with a few vistualizations.</p>

ZahAbdel: <aside class="quote no-group" data-username="ZahAbdel" data-post="1" data-topic="155173">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/a3d4f5/48.png" class="avatar"> ZahAbdel:</div>
<blockquote>
<p>Hi,<br>
how does 2d Convolution in expansive path in Unet from 256 to 128 feature maps work?<br>
I mean normally that happened if we want to extract more feature maps from an input.</p>
</blockquote>
</aside>
<p>thanks a lot for helping. Now it’s clearly how does it work.</p>
Gradients not the same when using different number of GPUs despite using grad accum and same batch ordering

divinho: <p>I have a situation where using 4 GPUs leads to much worse results than when using 2 different GPUs (DDP). Gradient accumulation is used to equalize the effective batch size, and a seed is set so for each optimizer step the two different builds are seeing the same batches. But the gradients are not equal.</p>
<p>I have turned off mixed precision, and it is still the case (differences in the gradient just before updating the weights, after the gradients have been accumulated). Why could this be?</p>
<p>I’m comparing gradient norms. The differences are of the sort:</p>
<pre><code class="lang-auto">1469.666259765625 &lt;&gt; 1518.424072265625
294.58251953125 &lt;&gt; 356.9222106933594
138.08688354492188 &lt;&gt; 168.6200408935547
933.7567138671875 &lt;&gt; 957.8016357421875
</code></pre>
<p>These are gradient norms from the last layers in the model, what’s interesting is that one of these has consistently larger gradients.</p>
How do I change the input and size if I want to classify MNIST with VGG16?

YuhskeHujisaki: <p>I want to classify MNIST with VGG16, but the input is set to 3, which does not fit the size of the MNIST input.Should VGG16 be adjusted to MNIST to match this input size with the size of the image? If so, how should I do this?</p>

ptrblck: <p>You could try a few different approaches:</p>
<ul>
<li>repeat the input channels of the MNIST data</li>
<li>replace the first conv layer of VGG16 with a new one expecting a single input channel</li>
<li>reduce the <code>in_channels</code> of the first conv layer kernel e.g. via a <code>mean</code>, <code>sum</code>, <code>median</code> etc. operation</li>
</ul>

YuhskeHujisaki: <p>If I replace the first conv layer of VGG16 with a new layer that assumes one input channel, should I divide the numpy array by 225 and then Resize to 224 for the MNIST image?</p>

Anshumaan_Dash: <p>Yes, if the input is <code>[0,255]</code> you need to bring it between <code>[0,1]</code> and since VGG16 expects an input of size 224x224, you need to resize it.</p>
Pytroch's Cifar10 images as input to ResNet18

papoo13: <p>I would like to train a ResNet18(Pretrained=False) with Cifar10. Now, each input tensor that belongs to Cifar10 has orders (Height, Width, Channels). Is it necessary to permute the order and then feed images to ResNet18? does a ResNet18 model which is not pretrained limited to inputs with tensor order such as (Batch, Channels, Height, Width)?</p>

Hamzeh: <p>It does not matter which net you want to train, you have to be aware of your convolutions. I mean if your net uses <code>nn.conv2d</code>, then the input should be like <code>(N, C, W, H)</code>, which <code>N</code> is your batch size.<br>
BTW you need to prepare a <code>dataloder</code> before loading the data to your net.</p>

papoo13: <p>thank you for your answer. I noticed that when using ToTensor() in the dataloader transformations, we get the order [N,C,H,W]. In this case we don’t need to use permute().</p>

Anshumaan_Dash: <p>Nope. As long as the input is in the order <code>[N,C,H,W]</code> you can go ahead and it should work fine.</p>
Why does pytorch rotate a numpy image when converted to tensor

binbbaz: <p>I have a numpy image i tried converting to a tensor</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c5a7d0544a6bdcc5e00ef8b97656a839b7b17f4b.jpeg" alt="cats" data-base62-sha1="scxBgsKByORr1S6OcTZJ5Goa6Dh" width="224" height="224"></p>
<p><code>tensor_img = torch.from_numpy(numpy_img).permute(2, 1, 0).unsqueeze(0)</code></p>
<p>I got a rotated and flipped image. What might the problem</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/eaac20f3356ce4f4a16a22f3d5ad45b2450b6870.png" alt="Screen Shot 2022-06-28 at 11.51.19 AM" data-base62-sha1="xu0vKH8YoJmntoPJi4er4YkyvYY" width="222" height="225"></p>

binbbaz: <p>So I figured it out in case someone falls into same problem. I had earlier splitted the original image into blocks where each block is stored in a list. This list is then converted into numpy and reshaped accordingly, resulting into the first image above.</p>
<p><code>import torchvision.transforms as T</code><br>
<code>transform = T.ToTensor()</code><br>
<code>tensor_img = transform(numpy_img).unsqueeze(0)</code></p>
<p>solved the problem</p>

ptrblck: <p>While <code>ToTensor</code> should also work, I think the original issue is caused by a wrong permutation as most likely your numpy image has the shape <code>[H, W, C]</code> which you would like to transform to a tensor in <code>[C, H, W]</code>. To do so you would need to call <code>.permute(2, 0, 1)</code>.</p>
FFT2 Input range

callbarian: <p>According to the document <a href="https://pytorch.org/docs/1.8.0/fft.html" rel="noopener nofollow ugc">1.8.0-fft2</a>,<br>
two ffts and fft2 should be allclose</p>
<p>It is true when input range is 0~1.<br>
When input range is above 1 (ex.0~25), first row and first column are allclose but the rest of them are not.<br>
Is there a precision issue on this? output differs in e-07 range.</p>
<p>Tested as below</p>
<p>in_tensor = torch.arange(25).view(5,5)<br>
print(torch.allclose(torch.fft.fft(torch.fft.fft(in_tensor, dim=0), dim=1), torch.fft.fft2(in_tensor)))<br>
-False<br>
in_tensor = torch.rand(25).view(5,5)<br>
print(torch.allclose(torch.fft.fft(torch.fft.fft(in_tensor, dim=0), dim=1), torch.fft.fft2(in_tensor)))<br>
-True</p>
Merge/combine clients with server

laki: <p>Hi,<br>
i am currently trying to make split learning setup. I have 5 clients and one server.<br>
Each client has own data set (cifar10 divided on five parts))<br>
In the training: each client train network up to cut layer and outputs at the cut layer are sent to the server which completes the rest of the training .I use resnet18</p>
<p>At the end I have clients with accuracy ca 80% :client 1 recognize only plane and car, client 2 recognize only bird and cat etc</p>
<p>Now I need to merge each client with the server, so that each client can recognize whole cifar10(all classes).</p>
<p>Anyone idea how to do that?<br>
I tried to do models aggregation, but nothing is happening. Thanks in advance!</p>