akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to use LSTMCell with LayerNorm?

username: Vannila
cooked: <p>I want to use LayerNorm with LSTM, but I’m not sure what is the best way to use them together.<br>
My code is as follows:</p>
<pre><code class="lang-auto">rnn = nn.LSTMCell(in_channels, hidden_dim)

hidden, cell = rnn(x, (hidden, cell))
</code></pre>
<p>So, if I want to add LayerNorm to this model, I will do it like this?</p>
<pre><code class="lang-auto">rnn = nn.LSTMCell(in_channels, hidden_dim)
norm = nn.LayerNorm(hidden_dim)

hidden, cell = rnn(x, (hidden, cell))
hidden = norm(hidden)
cell = norm(cell)
</code></pre>
<p>Is it the right way to use it? I’m confused. Sorry for my noob question.</p>

username: richard
cooked: <p><a href="https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/" rel="nofollow noopener">https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/</a> is a good tutorial. In particular, you can copy &amp; paste from this file: <a href="https://github.com/pytorch/pytorch/blob/cbcb2b5ad767622cf5ec04263018609bde3c974a/benchmarks/fastrnns/custom_lstms.py#L60-L83" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/cbcb2b5ad767622cf5ec04263018609bde3c974a/benchmarks/fastrnns/custom_lstms.py#L60-L83</a> that has a layernorm lstm implementation.</p>

username: angerhang
cooked: <p>Unfortunately, the referred script doesn’t support padded sequence. Any ideas when will PyTorch support this formally?</p>
Subtract constant value from tensor

username: cbd
cooked: <p>“tensor_a” is of size (1,3,4,4) and “tensor_b” is of size(3).</p>
<p>I want to subtract constant value from each channel. For below code, i want to subtract constant value 1 from 1st channel, 2 from 2nd channel and 3 from 3rd channel.</p>
<pre><code class="lang-auto">import torch

a=torch.Tensor([[[[ 1.,  2.,  3.,  4.],
          [ 5.,  6.,  7.,  8.],
          [ 9., 10., 11., 12.],
          [13., 14., 15., 16.]],

         [[17., 18., 19., 20.],
          [21., 22., 23., 24.],
          [25., 26., 27., 28.],
          [29., 30., 31., 32.]],

         [[33., 34., 35., 36.],
          [37., 38., 39., 40.],
          [41., 42., 43., 44.],
          [45., 46., 47., 48.]]]])
b=torch.tensor([1., 2., 3.])
print("a shape is ",a.shape)
print("b shape is ",b.shape)</code></pre>

username: lupus83
cooked: <p>This should do the trick:</p>
<pre><code class="lang-auto">b = b.view(1, *b.size(), 1, 1).expand_as(a)
</code></pre>
Autograd profiler does not finish

username: Durotan97
cooked: <p>Hi, I need some help as I can’t figure out the issue. If I run my code with cProfile, it works fine. However, if I use the autograd profiler, it never finishes running. Any idea what the issue might be?</p>
<p>As a side note, I have similar issues when I include torch.nn.DataParallel. I can include some code if needed but it is quite long.</p>
RuntimeError: cuda runtime error (100) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:47

username: Ahtsham_Manzoor
cooked: <p>hey,<br>
I am getting this error despite trying out all the workarounds available on the forum. The status of my machine and version compatibility seems quite accurate but I still, don’t know why this error was prompted.<br>
Already tried to reinstall PyTorch but nothing works.</p>
<p>Here are the machine validations.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd8581c8314cd17166b7bb4b747d06d13632fec9.png" data-download-href="https://discuss.pytorch.org/uploads/default/cd8581c8314cd17166b7bb4b747d06d13632fec9" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd8581c8314cd17166b7bb4b747d06d13632fec9.png" alt="image" data-base62-sha1="tk7UXOATiMhX2lhqhcPis3ycHIl" width="690" height="395" data-dominant-color="0A0A0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">779×446 9.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> could you please assist in this regard to fix this?</p>

username: ptrblck
cooked: <p>Your driver seems to be too old, as CUDA 10.2 would need NVIDIA driver &gt;= 440.33.</p>

username: Ahtsham_Manzoor
cooked: <p>I solved the issue by upgrading the Pytorch from 1.5.0 to 1.6.0 with Cuda 10.1. Also, in the command, I explicitly mentioned the GPU_id=0 as I have just one GPU in my device.</p>
How do I properly use the .interpolate function?

username: shamoons
cooked: <p>I have a tensor, <code>pred</code> which has a <code>.size</code> of <code>torch.Size([8, 28, 161])</code>. I want it to match the shape of <code>outputs</code>, which has a <code>.size</code> of <code>torch.Size([8, 27, 161])</code>, so I’m doing:</p>
<pre><code class="lang-auto">pred = torch.nn.functional.interpolate(outputs, size=outputs.size())
</code></pre>
<p>But this gives me an error:</p>
<pre><code class="lang-auto">  File "train_reconstruction.py", line 204, in main
    pred = torch.nn.functional.interpolate(outputs, size=outputs.size())
  File "/home/shamoon/.local/share/virtualenvs/pytorch-lstm-audio-o2htQyti/lib/python3.6/site-packages/torch/nn/functional.py", line 2510, in interpolate
    return torch._C._nn.upsample_nearest1d(input, _output_size(1))
RuntimeError: It is expected output_size equals to 1, but got size 3
</code></pre>
<p>What am I doing incorrectly?</p>

username: ptrblck
cooked: <p><code>F.interpolate</code> applies the interpolation in the temporal/spatial/volumetric dimensions.<br>
In your case it would accept a single value:</p>
<pre><code class="lang-python">x = torch.randn(8, 28, 161)
out = F.interpolate(x, size=140)
print(out.shape)
&gt; torch.Size([8, 28, 140])
</code></pre>
<p>Since you want to interpolate in the channel dimension, you could permute the tensor and apply the interpolation on it:</p>
<pre><code class="lang-python">x = torch.randn(8, 28, 161)
x = x.permute(0, 2, 1)
x = F.interpolate(x, size=27)
x = x.permute(0, 2, 1)
print(x.shape)
&gt; torch.Size([8, 27, 161])
</code></pre>

username: shamoons
cooked: <p>Awesome! Thank you. Works perfectly.</p>

username: Peter_Pack
cooked: <p>I have also encountered this problem, I want to resize a label tensor with size of <code>[2, 3, 3]([B, H, W])</code> into a size of <code>[2, 5, 5]</code>, but the error <code>RuntimeError: It is expected output_size equals to 1, but got size 2</code> is raised, what is the proper ways to resize a label tensor? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: ptrblck
cooked: <p>Since you would like to interpolate the two last dimensions, you would have to treat the input as a spatial input which is expected to have 4 dimensions.<br>
You could temporarily <code>unsqueeze</code> the batch dimension, apply the interpolation, and <code>squeeze</code> it afterwards.<br>
Since you’ve mentioned “label tensor” I assume you are working with <code>LongTensor</code>s and are thus interested in the <code>mode="nearest"</code> interpolation. If so, this should work:</p>
<pre><code class="lang-python">x = torch.randint(0, 10, (2, 3, 3))
y = F.interpolate(x.unsqueeze(0).float(), size=(5, 5), mode='nearest').squeeze(0).long()
print(y.shape)
&gt; torch.Size([2, 5, 5])
</code></pre>

username: Peter_Pack
cooked: <p>Awesome! Thank you. Works perfectly.</p>
How can I implement this calculation without large memory usage?

username: Wang_Zhuo
cooked: <p>I have two tensors, X and W, and their shapes are (n, m) and (m, l), respectively. I want to get a new tensor R that<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/172cfaa1e6a4b3c14e740bdb24976fbd21fe00be.png" alt="22f0d7eaf4999377bbb98a34db9940a76954b25a_2_690x190" data-base62-sha1="3j1lDnu77kT37PqOA98mP6wECXs" width="690" height="190"></p>
<p>. My current solution is using <code>torch.prod(1 - X.unsqueeze(-1) * W, dim=1)</code> . However, it will generate a tensor in the shape of (n, m, l) to store intermediate results. This (n, m, l) tensor could be too large to store in my GPU. Is there any way to avoid this large intermediate result tensor and can still keep the speed? For example, the calculation to get R is very similar to matrix multiplication, but the matrix multiplication costs much less memory.</p>
Torchvision object detection models do not support batch input optimization?

username: francescoalongi
cooked: <p>Hello,<br>
I wanted to try and play with the <a href="https://pytorch.org/vision/stable/models.html#object-detection-instance-segmentation-and-person-keypoint-detection" rel="noopener nofollow ugc">object detection torchvision pretrained models</a> for a personal application in which I need to detect objects in pictures. In particular, for efficiency reasons, I need to perform batch inference. However, I noticed that the torchvision.models do not support batch inference, indeed, they simply take a list of tensors as input and they iterate over the list.</p>
<p>What am I missing?</p>
<p>In my case, I’m using retinanet_resnet50_fpn, but I believe that the aforementioned problem applies also for the other detection models.</p>
<p>Thank you very much.</p>
Missing keys when loading a model checkpoint (transformer)

username: pemfir
cooked: <p>Downloaded bert transformer model locally, and missing keys exception is seen prior to any training</p>
<p>Torch 1.8.0 , Cuda 10.1 transformers 4.6.1<br>
bert model was locally saved using git command</p>
<p><code>git clone https://huggingface.co/bert-base-uncased</code></p>
<p>in the code below i am loading the model and state_dict from the same exact folder where i have downloaded the model fresh from the huggingface repo.</p>
<pre><code class="lang-auto">from transformers import AutoModel
import torch
state_dict = torch.load(".../bert-base-uncased/pytorch_model.bin")
model = AutoModel.from_pretrained("..../bert-base-uncased/")
model.load_state_dict(state_dict, strict=False)
</code></pre>
<p>Exception</p>
<p><code>IncompatibleKeys(missing_keys=['embeddings.position_ids', 'embeddings.word_embeddings.weight'</code> …</p>
<p>Any help is appreciated.</p>

username: ptrblck
cooked: <p>It seems you might be loading a <code>state_dict</code>, which isn’t compatible to the current model definition.<br>
I would recommend to post this question in the <a href="https://discuss.huggingface.co/">HuggingFace discussion board</a> as you would find more experts knowing this model there. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>
Implementing Linear as Conv2d

username: kuzand
cooked: <p>Let’s say that we have a 4D tensor of shape <code>(B, C, H, W)</code>. We can implement a Linear layer in two ways as follows:</p>
<pre><code class="lang-auto">lin = nn.Linear(C, K)
con = nn.Conv2d(C, K, kernel_size=1)
</code></pre>
<p>In practice they are equivalent, only having different input and output shapes: for <code>lin</code> it is <code>(B, H, W, C) -&gt; (B, H, W, K)</code>, whereas for <code>con</code> it is <code>(B, C, H, W) -&gt; (B, K, H, W)</code>.</p>
<p>After running some tests (on CPU) I found out that the <code>con</code> is faster than <code>lin</code>, so I was wondering what is the reason for this difference in speed and why would someone prefer using nn.Linear at all?</p>
<p>P.s. similarly for a 3D tensor we can use Conv1d to implement a Linear layer.</p>
How to calculate the [batch_size, alphas] [batch, seq, vector]]

username: PatricYan
cooked: <p>Hi:<br>
how to calculate the A = [batch_size, alphas], B=[batch, seq, vector],<br>
to get [batch_size, seq, alpha*vector]</p>

username: Eta_C
cooked: <p><code>A.unsqueeze(1).unsqueeze(3) + B.unsqueeze(2)</code></p>
Average gradient calculated graph

username: mrswjung
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/5/0503ca98a2d3cacbc5fc205cb8b296bfebd25548.png" alt="image" data-base62-sha1="ImvDRJJz5DHI6ft5y4qXiRaTZC" width="405" height="346"></p>
<p>this is how my model’s gradient average per layer looks like during training.</p>
<p>according to this graph, can vanishing gradient be a problem?</p>
Batch loading large dataset to fit the memory

username: xdwang0726
cooked: <p>I have a very large NLP dataset in csv (if loading all into memory it causes 187G). I am wondering how do I read and batch such large dataset without reading it all into the memory, like batch read? (I do not prefer split the large file into small pieces, btw) Thanks!</p>

username: ptrblck
cooked: <p>You could use <code>pandas.read_csv</code> with its <code>chunksize</code> argument to read chunks from the file:</p>
<pre><code class="lang-python">chunksize = 10 ** 6
for chunk in pd.read_csv(filename, chunksize=chunksize):
    process(chunk)
</code></pre>
Forward (eval mode) one sample vs many samples: gpu memory

username: sbelharbi
cooked: <p>hi,<br>
any explanation to this:</p>
<p><strong>case 1: forward minbatch: gpu mem required: 3gb</strong></p>
<pre><code class="lang-auto">model = create_model()
model.eval()
x = torch.rand(32, 3, 224, 224)

with torch.no_grad():
  model(x)  # executing just this instruction takes 3gb
</code></pre>
<p><strong>case 2: forward one sample. gpu mem required: 11gb</strong></p>
<pre><code class="lang-auto">model = create_model()
model.eval()
x = torch.rand(32, 3, 224, 224)

with torch.no_grad():
  model(x[0].unsqueeze(0))  # executing only this instruction: gpu memory 11gb.
</code></pre>
<p>what possibly can cause this?</p>
<p>torch 1.10.0<br>
forward is done within ‘torch.cuda.amp.autocast(enabled=True)’</p>
<p>thanks</p>

username: ptrblck
cooked: <p>Depending on your setup and in particular if you are using <code>cudnn.benchmark</code> other kernels with a different memory requirement might be picked. Could you post a minimal executable code snippet as well as the used device, which would show the difference in memory usage, please?</p>
[Urgent]:ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead

username: praggya
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,could you please check why I am facing this issue<br>
‘’’<br>
Traceback (most recent call last):<br>
File “build.py”, line 3, in <br>
from torch.utils.ffi import create_extension<br>
File “C:\Users\pk15s\anaconda3\lib\site-packages\torch\utils\ffi_<em>init</em>_.py”, line 1, in <br>
raise ImportError(“torch.utils.ffi is deprecated. Please use cpp extensions instead.”)<br>
ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead.<br>
‘’’</p>

username: ptrblck
cooked: <p>Please don’t tag specific users, as it could discourage others to post an answer.<br>
The error is raised as you are using a deprecated utility and should use custom <code>autograd.Function</code>s as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>.</p>
Building Reversible 1d Convolutional Networks in Pytorch

username: sharmaboi
cooked: <p>Hello All,</p>
<p>I am trying to build a reversible convolutional network for 1d audio signals. I have 0 clue where to start with this, as I assume just defining an additional boolean in the forward() method won’t work too well because specifically I am trying to train from 2 pairs of losses: 1 derived from going forward in the network with an input signal (loss is backpropogated), and 1 going backward in the network (loss in a sense is forward propogated).</p>
<p>How would y’all recommend implementing a network like this? Should I custom design a reversible 1d convolution class? Or can I get away with adding a parameter to the forward function?</p>
<p>Any help would be much appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Remove a whole block of layers

username: dieptran149
cooked: <p>Hi everyone,<br>
I’m running the Densenet121 Model. After loading weights. I have 2 unwanted blocks <code>model.features.denseblock4</code> and <code>model.features.transition3</code>. I’m trying to remove these 2 blocks of layers. Could any please tell me the way to do it? I’ll very appreciate it.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/5/45c8052bc4eb5c9a3fd20284035c68b259b2fca6.png" data-download-href="https://discuss.pytorch.org/uploads/default/45c8052bc4eb5c9a3fd20284035c68b259b2fca6" title="prune"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/5/45c8052bc4eb5c9a3fd20284035c68b259b2fca6.png" alt="prune" data-base62-sha1="9Xjub6OIj9AlRmnZpXKi3IfQu2y" width="690" height="419" data-dominant-color="E8E7E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">prune</span><span class="informations">830×505 26.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>An easy way would be to replace them with <code>nn.Idenity</code> layers (assuming their input shape matches their output shape). Alternatively, you could also override the <code>forward</code> in a custom module and skip these layers directly.</p>

username: dieptran149
cooked: <p>Excuse my late reply. Thank you so much!</p>
Finding list of 2D Patterns in a 2D tensor

username: yegane
cooked: <p>I want to see if a dictionary of tensor of boolean pattern (e.g. {6x6 : freq}) exist in a larger boolean tensor (e.g. 3x256x256).</p>
<p>Then I want to update my dictionary, both pattern and frequency.</p>
<p>I was hoping that there was a pytorchi way of doing it, instead of having loops over it, or have an optimized loop for doing it.</p>
<p>As far as I know, torch.where works when we have a scalar value. I’m not sure how should I do, if I have a tensor of 6x6 instead of a value</p>

username: Eta_C
cooked: <p>I’m not sure I understand your question correctly…<br>
Is it template matching?</p>
<pre><code class="lang-python">h, w, kernel_size = 256, 256, 6
ks = kernel_size * kernel_size
tensor = torch.randn(3, h, w) &gt; 0.5 
pattern = {
    tensor[0, 0:0+kernel_size, 4:4+kernel_size]: "A",
    tensor[1, 10:10+kernel_size, 3:3+kernel_size]: "B"
}

# Step 1, unfold the tensor
nh, nw = h - kernel_size + 1, w - kernel_size + 1
u_tensor = torch.nn.functional.unfold(tensor[None, ...].float(), kernel_size).clone()
u_tensor = u_tensor[0].bool().reshape(3, ks, -1)
u_tensor = u_tensor.permute(1, 0, 2).reshape(ks, -1)

# Step 2, search
for key, value in pattern.items():
    mask = u_tensor == key.view(-1, 1)
    indices = torch.where(mask.sum(dim=0) == ks)[0]
    
    # Step 3, parse indices
    indices_x = indices % nw 
    indices_y = (indices // nw) % nh 
    indices_c = indices // (nw * nh)
   
    # all works done.

    # Now Check the Result 
    for index_c, index_y, index_x in zip(indices_c, indices_y, indices_x):
        result = tensor[index_c, index_y:index_y+kernel_size, index_x:index+kernel_size]
        print((result == key).all())
</code></pre>

username: yegane
cooked: <p>Thanks. No, I think I found the solution. It is convolution.</p>
<pre><code class="lang-auto">x = torch.tensor([[1., 0., 0.],
                      [0., 1., 0.],
                      [1., 0., 0.]])
</code></pre>
<pre><code class="lang-auto">pattern = torch.tensor([[1., 0.],
                            [0., 1.]])
</code></pre>
<pre><code class="lang-auto">img, mask = x[None, None], pattern[None, None]
M = F.conv2d(img, 1 - mask)
tensor([[[[0., 1.],
          [2., 0.]]]])
</code></pre>
<pre><code class="lang-auto"> M == mask.sum(dim&gt; =(2,3)))
  tensor([[[[ True, False],
            [False, False]]]])
</code></pre>

username: Eta_C
cooked: <p>Actually, check <a href="https://discuss.pytorch.org/t/convolution-that-only-take-channel-wise-summation/21240/4">this discuss</a>, it shows that <code>unfold</code> is one way to implement <code>conv2d</code>. So, our methods are similar <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Anyway, congratulations on finding a simpler way. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>
Index into a pytorch tensor and return array of indices of rows

username: lima
cooked: <p>Sorry for the basic question, but I’m hoping the nice people here can help me:<br>
Given the below tensor that has vectors of all zeros and vectors with ones and zeros:</p>
<pre><code class="lang-auto">tensor([[0., 0., 0., 0.],
    [0., 1., 1., 0.],
    [0., 0., 0., 0.],
    [0., 0., 1., 0.],
    [0., 0., 0., 0.],
    [0., 0., 1., 0.],
    [1., 0., 0., 1.],
    [0., 0., 0., 0.],...])
</code></pre>
<p>How can I have an array of indices of the vectors/rows with ones and zeros so the output is like this:</p>
<pre><code class="lang-auto">indices = tensor([ 1, 3, 5, 6,...])
</code></pre>

username: eqy
cooked: <p>It looks like there is an implicit reduction here; are you treating rows with multiple ones the same as those that have a single one?<br>
In that case, you could try:</p>
<pre><code class="lang-auto">import torch
a = torch.tensor([[0., 0., 0., 0.],
    [0., 1., 1., 0.],
    [0., 0., 0., 0.],
    [0., 0., 1., 0.],
    [0., 0., 0., 0.],
    [0., 0., 1., 0.],
    [1., 0., 0., 1.],
    [0., 0., 0., 0.]])
print(a)
s = torch.sum(a, dim=-1)
print(s)
inds = torch.nonzero(s).reshape(-1)
print(inds)
</code></pre>
<pre><code class="lang-auto">tensor([[0., 0., 0., 0.],
        [0., 1., 1., 0.],
        [0., 0., 0., 0.],
        [0., 0., 1., 0.],
        [0., 0., 0., 0.],
        [0., 0., 1., 0.],
        [1., 0., 0., 1.],
        [0., 0., 0., 0.]])
tensor([0., 2., 0., 1., 0., 1., 2., 0.])
tensor([1, 3, 5, 6])
</code></pre>
What is kernel in conv2d

username: dre
cooked: <p>hi i want to use conv2d on my random tensor =when use conv = nn.Conv2d(3, 1, kernel_size=3, padding=1) what is the entity of kernel i mean what is inside my 3*3 kernel?</p>

username: eqy
cooked: <p>The weights or the kernel in this case is implicitly created when the Conv layer is declared and randomly initialized. Here, it would be a CPU tensor accessible as <code>conv.weight</code>.</p>

username: Eta_C
cooked: <p>For <code>nn.Conv2d</code>, check <code>conv.weight</code> and <code>conv.bias</code>.</p>
<p>If you want to custom the kernel. See <a href="https://pytorch.org/docs/stable/nn.init.html?highlight=init" rel="noopener nofollow ugc">nn.init</a> or <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.conv2d.html?highlight=conv2d#torch.nn.functional.conv2d" rel="noopener nofollow ugc">nn.functional.conv2d</a>.</p>
Normalizing masks in image segmentation

username: Erdi
cooked: <p>Hi guys</p>
<p>I have seen so many image segmentation applications in which people apply normalization to images and “masks” at the same time. It seems to me normalizing the mask simply means messing up all the class labels encoded there, thus training the model on labels which has no meaning. Am I missing something? Thank you for your help!</p>

username: ptrblck
cooked: <p>Your concern is correct assuming the mask contain class indices. In these cases transformations such as resizing can still be applied to a mask tensor but the <code>nearest</code> mode should be selected so that class indices are kept and are not interpolated. Often if a wrong transformation is used float values will be created in the mask, which would then fail during the loss calculation as e.g. <code>nn.CrossEntropyLoss</code> expects <code>LongTensor</code>s for the target. In any case, I would expect official examples as well as popular repositories to take care of these mistakes.</p>

username: Erdi
cooked: <p>Thank you so much for your insight <a class="mention" href="/u/ptrblck">@ptrblck</a> . Resizing is alright but I saw lots of notebooks, posts etc applying the very same transformation to masks as well such as  adding noise or blurring the class indices. Your clarification was important!</p>

username: ptrblck
cooked: <p>These transformations applied on the mask (again, assuming the mask contains class indices) sounds quite dangerous, but I would also expect to see <code>dtype</code> mismatches.<br>
In case you are seeing these transformation applied on a mask with and additional transformation into <code>LongTensor</code>s via <code>mask = mask.long()</code>, I’m sure the authors of the repository would be happy to hear about your valid concern. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
During the Build LibTorch-Lite for arm64 Devices pytorch

username: patel_shahrukh
cooked: <p>When I am executing this command → BUILD_PYTORCH_MOBILE=1 IOS_ARCH=arm64 ./scripts/build_ios.sh<br>
I am getting this error</p>
<h2>
<a name="file-usersappledocumentsgithubpytorchtoolscodegengenpy-line-81-1" class="anchor" href="#file-usersappledocumentsgithubpytorchtoolscodegengenpy-line-81-1"></a>File “/Users/apple/Documents/GitHub/pytorch/tools/codegen/gen.py”, line 81</h2>
<p>def parse_native_yaml(path: str) → ParsedYaml:<br>
^<br>
SyntaxError: invalid syntax</p>
<p>CMake Error at cmake/Codegen.cmake:232 (message):<br>
Failed to get generated_cpp list<br>
Call Stack (most recent call first):<br>
caffe2/CMakeLists.txt:2 (include)</p>
<p>– Configuring incomplete, errors occurred!</p>

username: hanton
cooked: <p>Hi <a class="mention" href="/u/patel_shahrukh">@patel_shahrukh</a>, did you update submodules before running the build command? Like</p>
<pre><code class="lang-auto">git submodule sync &amp;&amp; git submodule update --init --recursive --jobs 0
</code></pre>
<p>Tested at v1.10.0( 36449ea931) and it works for me.</p>
NLLLoss vs CrossEntropyLoss

username: dachosen1
cooked: <p>I’m comparing the results of NLLLoss and CrossEntropyLoss and I don’t understand why the loss for NLLLoss is negative compared to CrossEntropyLoss with the same inputs.</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch

label = torch.tensor([3, 0, 1, 1, 4])
output = torch.tensor([[0.5073, 0.4838, 0.5053, 0.4839, 0.5183],
        [0.5072, 0.4849, 0.4933, 0.4809, 0.5148],
        [0.5020, 0.4836, 0.5021, 0.4829, 0.5162],
        [0.5023, 0.4801, 0.4994, 0.4805, 0.5174],
        [0.5024, 0.4899, 0.4932, 0.4835, 0.5148]])

criterion = nn.NLLLoss()
loss = criterion(output, label)
loss
tensor(-0.4939)

criterion = nn.CrossEntropyLoss()
loss = criterion(output, label)
loss
tensor(1.6128)
</code></pre>

username: mariosasko
cooked: <p><code>CrossEntropyLoss</code> applies <code>LogSoftmax</code> to the output before passing it to <code>NLLLoss</code>.<br>
This snippet shows how to get equal results:</p>
<pre><code class="lang-python">nll_loss = nn.NLLLoss()
log_softmax = nn.LogSoftmax(dim=1)
print(nll_loss(log_softmax(output), label))

cross_entropy_loss = nn.CrossEntropyLoss()
print(cross_entropy_loss(output, label))
</code></pre>

username: dana_bezalel
cooked: <p>Hello,<br>
Is there a difference in terms of running time or accuracy of using CrossEntropyLoss vs. LogSoftmax + NLLLoss (on CPU or GPU)?<br>
Which option is considered more conventional / recommended?<br>
Thanks</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> uses <code>F.log_softmax</code> and <code>F.nll_loss</code> internally, so there wouldn’t be a difference in using the latter ops explicitly.</p>
How to upgrade the NCCL version of pytorch 1.7.1 from 2.7.8 to 2.11.4?

username: nxcvbc
cooked: <p>I’ve tried upgraded NCCL in the system, but not sure how to let the Pytorch use the latest version of the NCCL.</p>

username: ptrblck
cooked: <p>You could use the system NCCL installation via:</p>
<pre><code class="lang-auto">NCCL_INCLUDE_DIR="/usr/include/" NCCL_LIB_DIR="/usr/lib/" USE_SYSTEM_NCCL=1 python setup.py install
</code></pre>
<p>where the env vars point to the folders containing the corresponding NCCL files.</p>
Multivariate Student's t-distribution in pytorch?

username: robmok
cooked: <p>Hi there,</p>
<p>I have been using MultivariateNormal from pytorch distributions. I was hoping to find an implementation of more heavy-tailed distributions like the student’s t, but it looks like it doesn’t exist? It looks like there is one in TensorFlow (<a href="https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/MultivariateStudentTLinearOperator" class="inline-onebox" rel="noopener nofollow ugc">tfp.distributions.MultivariateStudentTLinearOperator</a>).</p>
<p>Does anyone know if pytorch actually has something like this? Or if it’s possible to request it?</p>
<p>Thank you!<br>
Rob</p>
Installing torch on the jetson NX

username: ELSH
cooked: <p>I am trying to download pytorch on a jetson NX to use a tracker i found, problem is that my conda is not able to find the correct cudatoolkit, i’ve trying downloading with pip but when running<br>
torch.cuda.is_available()</p>
<p>i get false<br>
i’ve tried downloading a separate cudatoolkit from another conda channel, and succeeded but when trying to install pytorch without cudatoolkit in the conda command, i get false again when trying to use cuda.<br>
i tried using a different cudatoolkit version (no conda channel seem to have 10.2) and still false.<br>
any help would be appreciated.</p>
<p>after downloading i get this:</p>
<blockquote>
<blockquote>
<blockquote>
<p>print(torch.<strong>version</strong>)<br>
1.8.1<br>
print(torch.version.cuda)<br>
None<br>
torch.backends.cudnn.enabled<br>
True</p>
</blockquote>
</blockquote>
</blockquote>
<p>i saw a different topic suggesting removing cpuonly, my conda doesnt have it.<br>
I created a completely new environment and did a clean downloaded and still nothing.</p>
<p>as it is a jetson i dont have nvcc or nvidia-smi i can only give you my cuda version which is 10.2.300 which came with the original build of the jetpack.</p>

username: ELSH
cooked: <p>managed to solve it by following the official Nvidia guide for pytorch on jetson:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-10-now-available/72048">
  <header class="source">
      <img src="https://aws1.discourse-cdn.com/nvidia/optimized/2X/0/0372ccc95874f71d7fbff64bbbff6f8c69dd850b_2_32x32.png" class="site-icon" width="32" height="32">

      <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-10-now-available/72048" target="_blank" rel="noopener nofollow ugc" title="12:51AM - 27 March 2019">NVIDIA Developer Forums – 27 Mar 19</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:237/42;"><img src="https://aws1.discourse-cdn.com/nvidia/original/2X/8/8f17cc8f1a724d6ecea8a197a267ec8a05ef1490.png" class="thumbnail" width="237" height="42"></div>

<h3><a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-10-now-available/72048" target="_blank" rel="noopener nofollow ugc">PyTorch for Jetson - version 1.10 now available</a></h3>

  <p>Below are pre-built PyTorch pip wheel installers for Python on Jetson Nano, Jetson TX1/TX2, and Jetson Xavier NX/AGX with JetPack 4.2 and newer.   Download one of the PyTorch binaries from below for your version of JetPack, and see the installation...</p>

  <p>
    <span class="label1">Reading time: 232 mins 🕑</span>
      <span class="label2">Likes: 123 ❤</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Multiple loss, multiple backwards, multiple optimizers

username: yegane
cooked: <p>I have three models:<br>
M1 gives the output(OUT1) to (IN2) M2<br>
M1.requires_grad=False<br>
M2.requires_grad=True<br>
M2 uses the output of M1, and I used OUT1.detach() to stop backpropagtion<br>
Then I backward M2<br>
I use the output of M2 for M3, and again with .detach()<br>
Then I calculated Loss1, and backpropagte M1, based on output of M2 and M3.</p>
<p>Everything works and it is fine.</p>
<p>Now, I want to do a backward through all of the M1&gt;M2&gt;M3, and I don’t want to use detach(), but it returns different errors, like <code>is at version 3; expected version 2 instead. </code><br>
I expected something like this:</p>
<p>M1.requires_grad(True)<br>
M2.requires_grad(False)<br>
M3.requires_grad(False)<br>
M1&gt;OUT1</p>
<p>M1.requires_grad(False)<br>
M2.requires_grad(True)<br>
M3.requires_grad(False)<br>
OUT1&gt;M2&gt;OUT2<br>
calculate loss2<br>
backpropagate(loss2)<br>
optim2.step()</p>
<p>M1.requires_grad(False)<br>
M2.requires_grad(False)<br>
M3.requires_grad(True)<br>
OUT2&gt;M3&gt;OUT3<br>
calculate Loss3<br>
backpropagate(Loss3)<br>
optim3.step()</p>
<p>M1.requires_grad(True)<br>
M2.requires_grad(False)<br>
M3.requires_grad(False)<br>
loss1=f(OUT2,OUT3,OUT1, loss2,loss3)<br>
backpropagete(loss1)<br>
optim1.step()</p>
<p>So in here, I want to have multiple stage losses. I want to keep model1, and its out1, out of backwards of M2 and M3, but I want out1, inside of their calculation, and at the end, I want to have a loss that goes through M3 and M2 (but without affecting their parameters), but I want loss1 would be able to see how it can affect loss2, loss3, and only affect M1. e.g. I want M2 works on its own loss2 to lower it, and I want loss1 see how it can change loss2 and loss 3.</p>
PyTorch vs Tensorflow gives different result in Conv2d

username: Kyriakos_Fabionus
cooked: <p>Hi,</p>
<p>I found this issue, when trying to load trained PyTorch model’s weights into Tensorflow Keras model.<br>
Following is the code to investigate this issue:</p>
<pre><code class="lang-auto">import numpy as np
import torch
import tensorflow as tf

tf.enable_eager_execution()

th_x = torch.randn(1, 3, 8, 8)
tf_x = np.float32(th_x.numpy())

s = (1, 1) # strides
th_conv = torch.nn.Conv2d(3, 1, 3, s, 1, bias=False)
tf_conv = tf.keras.layers.Convolution2D(1, 3, s, 'same', data_format='channels_first', use_bias=False)
tf_conv(tf_x)
tf_conv.set_weights([th_conv.weight.data.permute(2, 3, 1, 0).detach().numpy()])

th_y = th_conv(th_x).detach().numpy()
tf_y = tf_conv(tf_x).numpy()

print(th_y.shape, th_y.ravel()[:10])
print(tf_y.shape, tf_y.ravel()[:10])
print('Error Rate:', np.mean(np.abs(th_y - tf_y) &gt; 1e-6))
</code></pre>
<p>The code above produces same results for PyTorch’s Conv2d and Tensorflow’s Convolution2D operations. However, when I set the strides to (2, 2), it gives totally different results.</p>
<p>I’m looking forward to hear any solution to this issue, thanks in advance.</p>
<p>I’m using <code>tensorflow==1.14.0</code> and <code>torch==1.1.0</code>.</p>

username: ArseniiP
cooked: <p>Most likely it is related to different ‘same’ padding behavior in TF and Pytorch. It is symmetric in Pytorch and asymmetric in TF. I also faced this issue and solved it by adding manual padding in symmetric manner before convolution using tf.pad. In conv layer padding should be ‘valid’ after that.</p>
Pytorch c++ Frontend pretrained model

username: DSL
cooked: <p>Hello together,</p>
<p>I worked out this blog <a href="https://krshrimali.github.io/posts/2019/08/applying-transfer-learning-on-dogs-vs-cats-dataset-resnet18-using-pytorch-c-api/" class="inline-onebox" rel="noopener nofollow ugc">Applying Transfer Learning on Dogs vs Cats Dataset (ResNet18) using PyTorch C++ API :: Kushashwa Ravi Shrimali (Kush) — Hello Friend NG HZ Theme</a> and i am little bit confused. In this blog the transfer learning is just a converted network with jit plus an additional layer. Is it possible to change the weights in the converted model? Or just i can change the weights of additional layers?</p>
<p>kind regards</p>
Tensorboard graph for sequential model with multiple inputs

username: aimep
cooked: <p>The <a href="https://pytorch.org/docs/stable/tensorboard.html#torch.utils.tensorboard.writer.SummaryWriter.add_graph" rel="noopener nofollow ugc"><code>torch.utils.tensorboard.writer.SummaryWriter.add_graph</code></a> method takes a <code>input_to_model</code> which is "a variable or a tuple of variables to be fed, and <em>this tuple is expanded into multiples arguments.</em></p>
<p>But for <code>torch.nn.Sequential</code> models, the only way to have models that takes multiple arguments is to make models that takes only one argument (a tuple) and deconstruct it explicitly in the <code>forward</code> code (<a href="https://github.com/pytorch/pytorch/pull/9979#" rel="noopener nofollow ugc">see here</a>).</p>
<p>Why aren’t both behavior consistent?</p>
<p>As for now, if I want to make both work together, I need to give <code>input_to_model=((x, y),)</code> to <code>add_graph</code>, which is not nice.</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
from torch.utils.tensorboard.writer import SummaryWriter

class Model_X_x(nn.Module):
    def forward(self, X):
        x, y = X
        return x

class Model_x_xx(nn.Module):
    def forward(self, x):
        return x, x

x, y = torch.randn(3), torch.randn(5)
model = nn.Sequential(Model_X_x(), Model_x_xx())
pred = model((x, y))
SummaryWriter().add_graph(model, ((x, y),))
</code></pre>
About BCEWithLogitsLoss's pos_weights

username: Wangt-CN
cooked: <p>Hello everyone, I’m a new in pytorch and it’s quite great.But I have some problem when using the BCELoss and I have tried to find answer in the forum but it doen’t work. So I have to creat a topic.</p>
<p>Now I am solving a problem about the multilabel classification and use the BCEWithLogitsLoss ( just define learning rate).  I have 500 label, each label can be 0 or 1, and 0 is much more than 1.(Only in first 2 label, 0 and 1 are approximatly equall).</p>
<p>When I try to train it. First, the loss is about 0.69, and it decreace quickly to 0.1 in 1 epoch. And then loss basically do not fall.  And i try to print the output (before the sigmoid) and the label. Find that all the value are negative.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/c/c08208bc1d291902607f4d41f2cccc38dc0ea24a.jpg" alt="image" data-base62-sha1="rt0gOhgZ4dbgDfm622pm9HbAJ0C" width="474" height="277"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/5/54c9b15ba5e9a96c0bfbbfe68261fa5d40416b25.jpg" alt="image" data-base62-sha1="c64epbC7jactdXN1cXl1uVFgufr" width="494" height="165"></p>
<p>I feel very strange. And I just guess if it’s because that in label 0 is much more than 1? And the output almost are negative. If it is, whether i should use the pos_weight?</p>

username: ptrblck
cooked: <p>If your classes are imbalanced, setting the <code>pos_weigth</code> to <code>number_neg_samples / number_pos_samples</code> should give a more diverse prediction.</p>

username: Wangt-CN
cooked: <p>Thankyou very much for reply.<br>
I have a try but a little worried, because the pos_weight on some class can be 100, more than the three in the docs.<br>
Does it matter?</p>
<p>Best regards</p>

username: ptrblck
cooked: <p>You would have to check your per-class accuracies or the confusion matrix.<br>
The weighting will trade one accuracy for another, so it’s a way to balance your predictions.<br>
Currently your model would have a perfect true negative rate, while the true positive rate is most likely low.</p>

username: codeXiaoLi
cooked: <p>hello, i am doing a task of multilabel classification, may i have your wechat to chat with you about the problem of unbalanced data?  Thank you, i am master candidate.</p>

username: Sampath_Kumaran
cooked: <p>Hi… We need to use pos_weight from only train_dataset or for entire dataset for a multilabel approach?</p>
How to implement n-Step learning?

username: TheUnnamed22
cooked: <p>Hello,</p>
<p>i am a bit confused about the implementation of n-step learning. Lets assume we have n=4 and a constant reward of +1. lets also assume gamma=1 for simplification. So if we have the following:</p>
<p>s_1=[1,1,1,1]<br>
s_2=[1,1,1,T] Close to target</p>
<p>s_1 = 1+1+1+1 = 4<br>
s_2= 1+1+1 = 3</p>
<p>So State s_1 would be much more desirable although s_2 is close to the aim of the task. Or lets put the question different. How would you calculate state s_2 or a state s_3 = [1,T,1,1] ?</p>
Broadcasting in mask select

username: haoran-hash
cooked: <p>The code snippet is below.</p>
<pre><code class="lang-auto">import torch

data = torch.randn((10, 2))
data_norm = torch.linalg.vector_norm(data, dim=1)

mask1 = (data_norm &lt;= 1).unsqueeze(dim=1)   # mask1.shape=(10,1)
small_data = data[mask1]
</code></pre>
<p>After running, the following error is reported.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "practice_14.py", line 8, in &lt;module&gt;
    small_data = data[mask1]
IndexError: The shape of the mask [10, 1] at index 1 does not match the shape of the indexed tensor [10, 2] at index 1
</code></pre>
<p>But i remember mask select have broadcasting mechanism.<br>
Can someone explain that？</p>
<p>(By the way, if i modify <code>mask1 = (data_norm &lt;= 1).unsqueeze(dim=1)</code> to <code>mask1 = (data_norm &lt;= 1)</code>, i.e. <code>mask.shape=(10,)</code>, above code can run correctly.)</p>
Export on mobile errors

username: Matt_Davis
cooked: <p>Hey,<br>
I have been having a hard time used the export to get it on mobile. First it said missing LA pack. We did a pytorch build for CPU and then it says model version supported is 2 - 5 but ours is 7. Which I assume is now the libtorch needs to be rebuilt to match the export version. Does that sound correct?</p>
Fast implementation of minibatch discrimination layer (for GAN training)?

username: f3ba
cooked: <p>The minibatch discrimination layer (suggested by <a href="http://papers.nips.cc/paper/6124-improved-techniques-for-training-gans" rel="nofollow noopener">http://papers.nips.cc/paper/6124-improved-techniques-for-training-gans</a>), lets a discriminator look at a complete batch of examples and inspect their proximity, thus helping to avoid mode collapse in the training of GANs.</p>
<p>Unfortunately the minibatch discrimination layer does not seem to be available in the main pytorch distribution. But I bet there are fast implementations available out there that one can look at?</p>

username: Oguz_Vuruskaner
cooked: <p>You can check out my implementation at <a href="https://github.com/eyecuvision/good_guys/blob/main/good_guys/layers/complex/MinibatchDiscrimination.py" rel="noopener nofollow ugc">good_guys/MinibatchDiscrimination.py at main · eyecuvision/good_guys (github.com)</a></p>
Five Class Classification not working well

username: Naveed_Ilyas
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bca695c7c1df32e2d778deb5305e347c011f1b93.png" alt="loss" data-base62-sha1="qUSFCAVoJQimn4VgLnS2tiRjX7Z" width="640" height="480"></p>
<p>I am trying to train five class classification model. However the validation loss is not converging at all. What could be appropriate reason. Please guide me.</p>
<p>Thank you</p>

username: Artlind
cooked: <p>Hello, it looks like your model is overfitting.</p>

username: Naveed_Ilyas
cooked: <p>What should i do to overcome overfitting.<br>
I already added regularization + dropout.<br>
Do you have any other idea.<br>
Please share</p>
Alternatives to for-looping over a ModuleList

username: PyScorch
cooked: <p>I have a neural network where the neurons of a single layer (layer 2 in the example) are arranged into some n number of “blocks”. I want to forward propagate through them individually to get to n outputs. Presently, I am doing that using ModuleList and a for loop to go over all the individual blocks. But the code is terribly slow because of this. What is a more efficient way to do this?</p>
<p>Any help would be appreciated, thanks a lot!</p>
<pre><code class="lang-auto">class small_net(nn.Module):

    def __init__(self, num_classes=10):
        super(small_net, self).__init__()

        self.bCov1 = BinarizeConv2d(3, 50, kernel_size=3, stride=1, padding=1,bias=True)
        self.bn1 = nn.BatchNorm2d(50, affine=False)
        self.htan1 = nn.Hardtanh(inplace=True)


        self.layer2_neuron_blocks = fixed_block_size(50, 10)
        self.layer2_convs = nn.ModuleList(
                    [BinarizeConv2d(i, 100, kernel_size=3, padding=1, bias=True) for i in self.layer2_neuron_blocks])
        self.mp2 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.layer2_bns = nn.ModuleList([nn.BatchNorm2d(100, affine=False) for _ in self.layer2_neuron_blocks])
        
        
        self.fc3 = BinarizeLinear(100 * 16 * 16, 20, bias=True)
        self.bn3 = nn.BatchNorm1d(20, affine=False)
        self.htan3 = nn.Hardtanh(inplace=True)


        self.fc4 = BinarizeLinear(20, num_classes, bias=True)
        self.bn4 = nn.BatchNorm1d(num_classes, affine=False)
        
    def forward(self, x):
        x = self.bCov1(x)
        x = self.bn1(x)     
        x = SignActivation.apply(x)
        
        y2 = [0]+list(np.array(self.layer2_neuron_blocks).cumsum())

        out_layer2 = torch.zeros([x.shape[0], len(self.layer2_neuron_blocks), 100, 16, 16])

        for i in range(len(y2)-1):
            x2 = x[:, y2[i]:y2[i+1], :, :]
            x2 = self.layer2_convs[i](x2)
            x2 = self.mp2(x2)
            x2 = self.layer2_bns[i](x2)
            x2 = SignActivation.apply(x2)
            out_layer2[:, i, :, :, :]  = x2.clone()
        
        x2 = torch.mean(out_layer2, dim=1).to(device)

        x = x2.view(-1, 100 * 16 * 16)

        x = self.fc3(x)
        x = self.bn3(x)
        x = self.htan3(x)

        x = self.fc4(x)
        x = self.bn4(x)

        return x
</code></pre>
How to get dot product with different shape

username: PatricYan
cooked: <p>Hi:<br>
a=[batch_size, seq_len, embedding]<br>
b=[batch_size, 1, embedding]</p>
<p>how to get the dot product of [1, embedding] and  [1,embedding],  [2,embedding],…<br>
[seq,embedding]</p>

username: Eta_C
cooked: <p>Try</p>
<pre><code class="lang-auto">batch_size, seq_len, embedding = 3, 4, 5
a = torch.randn([batch_size, seq_len, embedding])
b = torch.randn([batch_size, 1, embedding])

print((a * b).shape)  # torch.Size([3, 4, 5])
print(torch.bmm(a, b.permute(0, 2, 1)).shape)  # torch.Size([3, 4, 1])

</code></pre>
Numpy or pytroch squeeze

username: Aamir_M1
cooked: <p>In numpy or torch API , what is  squeeze used for.</p>

username: ptrblck
cooked: <p><code>squeeze</code> removes a dimension with a size of 1.<br>
E.g. if you have a tensor with a shape of <code>[2, 1, 3, 1, 4, 1]</code>, you could <code>squeeze</code> dim1, dim3, and dim5 by passing the <code>dim</code> argument to <code>squeeze</code>. Calling <code>squeeze()</code> on a tensor without specifying a dimension will remove all dimensions with a size of 1. The <a href="https://pytorch.org/docs/stable/generated/torch.squeeze.html#torch.squeeze">docs</a> also explain this behavior.</p>

username: Aamir_M1
cooked: <p>Yes I get that part , however my question was, under what conditions would one use  this.</p>

username: ptrblck
cooked: <p>The use case explains itself, so whenever you would need to remove a size 1 dimension, you could use <code>squeeze</code>. E.g. if you are dealing with a single sample somewhere in your code and would like to remove the batch size in order to plot the image tensor via <code>plt.imshow</code>.</p>