akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Total number of parameters with shared weights

sashapiv: <p>Hi,</p>
<p>I try to use shared weights and display correct amount of parameters.<br>
A simple two linera layers network with the same weights.</p>
<pre><code class="lang-auto">def __init__(self):
        super(testModule, self).__init__()
        self.fc1 = nn.Linear(10, 10, bias=True)
        self.fc2 = nn.Linear(10, 10, bias=False)

        # Remove the weights as we override them in the forward
        # so that they don't show up when calling .parameters()
        del self.fc1.weight
        del self.fc2.weight

        self.fc2_base_weights = nn.Parameter(torch.randn(10, 10))
        # self.shared_weights = nn.Parameter(torch.randn(10, 5))

        self.fc1.weight = self.fc2_base_weights
        self.fc2.weight = self.fc2_base_weights
</code></pre>
<p>Yet I receive 210 parameter’s instead of 110…</p>
<pre><code class="lang-auto">----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Linear-1               [-1, 10, 10]             110
            Linear-2               [-1, 10, 10]             100
================================================================
Total params: 210
Trainable params: 210
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.00
Forward/backward pass size (MB): 0.00
Params size (MB): 0.00
Estimated Total Size (MB): 0.00
----------------------------------------------------------------
</code></pre>
<p>Thank you in advance!</p>

ptrblck: <p>I get the expected <code>110</code> total parameters using the latest version of <code>torchinfo</code>:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(10, 10, bias=True)
        self.fc2 = nn.Linear(10, 10, bias=False)

        # Remove the weights as we override them in the forward
        # so that they don't show up when calling .parameters()
        del self.fc1.weight
        del self.fc2.weight

        self.fc2_base_weights = nn.Parameter(torch.randn(10, 10))
        # self.shared_weights = nn.Parameter(torch.randn(10, 5))

        self.fc1.weight = self.fc2_base_weights
        self.fc2.weight = self.fc2_base_weights
        
model = MyModel()
summary(model)
# =================================================================
# Layer (type:depth-idx)                   Param #
# =================================================================
# MyModel                                  --
# ├─Linear: 1-1                            110
# ├─Linear: 1-2                            100
# =================================================================
# Total params: 110
# Trainable params: 110
# Non-trainable params: 0
# =================================================================
</code></pre>
Loss.backward throws "mps.concat" error on M1, but works well on CPU and GPU

wwydmanski: <p>Hi!<br>
I’ve been trying to run a Seq2Seq model on my M1 mac, but for some reason I’m getting an error while running loss.backward().<br>
The error:</p>
<pre><code class="lang-auto">(mpsFileLoc): /AppleInternal/Library/BuildRoots/0aa643d0-625a-11ed-b319-a23c4f261b56/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm:636:0: error: invalid input tensor shapes, all input shapes must match except at axis
(mpsFileLoc): /AppleInternal/Library/BuildRoots/0aa643d0-625a-11ed-b319-a23c4f261b56/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm:636:0: note: see current operation: %28 = "mps.concat"(%26, %24, %27) : (tensor&lt;3x1x8xf32&gt;, tensor&lt;1x4x8xf32&gt;, tensor&lt;si32&gt;) -&gt; tensor&lt;4x1x8xf32&gt;
</code></pre>
<p>I can see the mismatch between 3x1x8xf32 and 1x4x8xf32, but it isn’t present anywhere in the code, and it works well outside MPS.</p>
<p>Code:</p>
<pre><code class="lang-auto">class OneLayerSeq2Seq(torch.nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(OneLayerSeq2Seq, self).__init__()
        self.hidden_size = hidden_size
        self.encoder = torch.nn.LSTM(input_size, hidden_size, batch_first=True)
        self.decoder = torch.nn.LSTM(hidden_size, hidden_size, batch_first=True)
        self.relu = torch.nn.ReLU()
        self.linear = torch.nn.Linear(hidden_size, output_size)

    def forward(self, input, future=0):
        outputs = []
        encoder_output, (hidden, cell) = self.encoder(input)
        encoder_output = self.relu(encoder_output)

        decoder_input = encoder_output[:, -1, :].unsqueeze(1)
        for i in range(future):
            decoder_output, (hidden, cell) = self.decoder(decoder_input, (hidden, cell))
            decoder_input = self.relu(decoder_output)
            outputs += [self.linear(decoder_input)]
        outputs = torch.cat(outputs, dim=1)
        return outputs
</code></pre>
<p>Do you have any idea what might be causing it?</p>
GradCAM zero attributions: GAP vs no-GAP Neural Network

vggls: <p>I am styding the well-known Kaggle’s medical dataset</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.kaggle.com/datasets/tawsifurrahman/covid19-radiography-database">
  <header class="source">
      <img src="https://www.kaggle.com/static/images/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://www.kaggle.com/datasets/tawsifurrahman/covid19-radiography-database" target="_blank" rel="noopener nofollow ugc">kaggle.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://storage.googleapis.com/kaggle-datasets-images/576013/1042828/70f123e67b39cb794a711ed8fb85c491/dataset-card.jpg?t=2020-03-29-00-11-42" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://www.kaggle.com/datasets/tawsifurrahman/covid19-radiography-database" target="_blank" rel="noopener nofollow ugc">COVID-19 Radiography Database</a></h3>

  <p>COVID-19 Chest X-ray images and Lung masks Database</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
and I have trained <strong>two variations of an EfficientNet-B2</strong> network;</p>
<ul>
<li>one structured as “<strong>conv layers - GAP layer - class scores</strong>” (<strong>Network 1</strong>)<br>
(i.e. as downloaded from PyTorch torchvision)</li>
<li>one structured as “<strong>conv layers - Flatten - class scores</strong>” (<strong>Network 2</strong>)<br>
(i.e. I have removed the GAP layer and have flattened the output of the last conv layer)</li>
</ul>
<p>The networks are comparable in terms of generalization ability as they yield<br>
similar classification report on the test set. The test set consists of 10% data per class resulting in 2119 images it total.</p>
<ul>
<li>Network 1 test accuracy: 1997/2119</li>
<li>Network 2 test accuracy: 1971/2119.</li>
</ul>
<p>Then, in both networks I applied GradCAM (ReLU version) on the correctly classified test images,<br>
and then I observed the following weird result regarding the number of GradCAM maps that are produced<br>
by the two structures:</p>
<ul>
<li>Network 1: 8 out of the 1997 attribution maps have 0 values (sum of all elements is 0)</li>
<li>Network 2: 1037 out of the 1971 attribution maps have 0 values</li>
</ul>
<p>In other words, replacing GAP with Flattening seems to have a great influence on the quality of the produced maps as Network 2 produces almost 1000 more 0-valued maps.</p>
<p>Is there an intuition/explanation on this ? What could be the cause ?</p>
Training LSTM with multiple GPU and get this Warning

Peterwisu: <p>Hi,</p>
<p>I am trying to train an LSTM network using DataParalell with multiple GPUs and this warning shows up what it means. Will this effect the result of the training?</p>
<pre><code class="lang-auto">UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters(). (Triggered internally at  /opt/conda/conda-bld/pytorch_1659484809535/work/aten/src/ATen/native/cudnn/RNN.cpp:968.)
  self.dropout, self.training, self.bidirectional, self.batch_first)
</code></pre>
<p>And Can I train LSTM in parallel since I remember seeing a post about the disadvantages of RNN and its variants that “In practice, however, LSTMs are much slower to train than self-attention networks as they can- not be parallelized at sequence level”</p>
<p>Thank you</p>

ptrblck: <aside class="quote no-group" data-username="Peterwisu" data-post="1" data-topic="168949">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peterwisu/48/52366_2.png" class="avatar"> Peterwisu:</div>
<blockquote>
<p>Will this effect the result of the training?</p>
</blockquote>
</aside>
<p>No, it will only potentially increase the memory usage as explained in the error message:</p>
<aside class="quote no-group" data-username="Peterwisu" data-post="1" data-topic="168949">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peterwisu/48/52366_2.png" class="avatar"> Peterwisu:</div>
<blockquote>
<p><code>possibly greatly increasing memory usage.</code></p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="Peterwisu" data-post="1" data-topic="168949">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peterwisu/48/52366_2.png" class="avatar"> Peterwisu:</div>
<blockquote>
<p>And Can I train LSTM in parallel</p>
</blockquote>
</aside>
<p>Yes, you can use data parallel approaches, but I would generally recommend to use <code>DistributedDataParallel</code> for a better performance than <code>DataParallel</code>.</p>
Why VAE model in pytorch dosn't use torch.nn.KLDivLoss?

laro: <ul>
<li>I saw several examples of VAE implementations.</li>
<li>In VAE we use loss with 2 parts: MSE and KLDivLoss</li>
<li>In all the examples I saw, they wrote the VAE loss (MSE+KL LOSS) in their own implementation and didn’t used: <code>torch.nn.KLDivLoss</code>
</li>
</ul>
<p>One example can be found here:<br>
<a href="https://github.com/AntixK/PyTorch-VAE/blob/master/models/vanilla_vae.py" rel="noopener nofollow ugc">https://github.com/AntixK/PyTorch-VAE/blob/master/models/vanilla_vae.py</a></p>
<p>Why did they implement the part of the KLDivLoss and didn’t use <code> torch.nn.KLDivLoss</code> ?</p>
Get well adjusted confidence scores from similarity of CLIP encodings

Johnny_Cash: <p>I am using CLIP to check similarity between text and an image. Now for example I have list of words (objects) I want to check against. For example (“elephant”, “tiger”, “giraffe”).</p>
<p>By taking the dot product of the encodings I get the similarity value. To evaluate the “confidence” I take the softmax over the outputs and it works very well predicting which class is in the image. But it could happen that the classes are not mutually exclusive. In that case softmax doesn’t make sense. I tried to use sigmoid as it is used with multi-label classification, but it seems to give me values all around 0.55 (so classes that were correct around around 0.56 and classes that are wrong 0.54), so in the example (0.565, 0.55, 0.62) if elephant and giraffe are in the picture. Thus it is hard to set a threshold there.</p>
<p>I would like to have something like (0.95, 0.05, 0.98) if elefant and giraffe are in the picture, thus the similarity is high for both words.</p>
<p>Am I thinking too complicated and there is a standard way to do this? Is it even possible to get this well adjusted confidence score?</p>
Element-wise matrix addition of two differently shaped matrices (tensors) based on index

cmetzner: <p>Hello all,</p>
<p>I am curious if you guys have a better (i.e., faster, computationally more efficient) solution than me.</p>
<p>Situation:<br>
I have two tensors with size A=[bs, n, k] and B=[bs, m, k], where bs: batch size, n: number of low-level labels, m: number of high-level categories, and k: hidden dimension. I want to add the elements of B to specific elements/vectors of A based on a known index. I know exactly which element of B has to be added to one or more of the elements of A. Please see a working example of how I approach this problem (disregard the batch size as it is not important for the problem) - please note that I perform these operations on multiple GPUs:</p>
<p>import torch</p>
<p>A = torch.tensor([[0, 0, 0,], [1, 1, 1], [2, 2, 2], [3, 3, 3], [4, 4, 4]])<br>
B = torch.tensor([[1, 1, 1], [2, 2, 2]])</p>
<p>indices = [0, 1, 1, 0, 0]</p>
<p>for i, index in enumerate(indices):<br>
A[i] += B[index]</p>
<p>print(A)</p>
<p>Result:<br>
tensor([[1, 1, 1],<br>
[3, 3, 3],<br>
[4, 4, 4],<br>
[4, 4, 4],<br>
[5, 5, 5]])</p>
<p>Is there a more efficient way to this?</p>
<p>Thank you,<br>
Christoph</p>

ptrblck: <p>Direct indexing should work:</p>
<pre><code class="lang-python">A_ = torch.tensor([[0, 0, 0,], [1, 1, 1], [2, 2, 2], [3, 3, 3], [4, 4, 4]])
A_[torch.arange(len(indices))] += B[indices]

print((A_ == A).all())
# tensor(True)
</code></pre>

cmetzner: <p>That will work, thank you!</p>
Why the input and output gradient size of LPPool are the same

skyleaworlder: <pre><code class="lang-python">def save_gradient(module, grad_input, grad_output):
    print(f"{module.__class__.__name__} input grad:\n{grad_input}")
    print(f"{module.__class__.__name__} output grad:\n{grad_output}")

m = nn.LPPool2d(3, 2, stride=2)
m.register_backward_hook(save_gradient)
input = torch.tensor([float(i) for i in range(1, 50)], requires_grad=True).reshape(1,1,7,7)
input = torch.transpose(input, 2, 3)

output = m(input)
output.backward(gradient=output)
</code></pre>
<p>I got:</p>
<pre><code class="lang-auto">LPPool2d input grad:
(tensor([[[[0.0309, 0.0107, 0.0063],
          [0.0248, 0.0097, 0.0059],
          [0.0206, 0.0089, 0.0056]]]]),)
LPPool2d output grad:
(tensor([[[[10.7722, 31.1708, 52.9788],
          [13.4294, 34.2547, 56.1203],
          [16.2184, 37.3533, 59.2653]]]], grad_fn=&lt;PowBackward0&gt;),)
</code></pre>
<p>If I don’t misunderstand the parameters of backward hook, here LPPool2d input grad size should be (1, 1, 7, 7), while output grad size should be (1, 1, 3, 3).</p>
<p>My torch version is <code>1.13.0+cu116</code>. Actually I don’t know LPPool mechanism very clearly. I just found that the input and output gradient size of AvgPool or MaxPool are different, and I want to know the reason why. I have tried to search this question but I failed to find my answer I expect.</p>
<p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Your code raises a warning:</p>
<pre><code class="lang-python"># UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
</code></pre>
<p>and I would recommend not to ignore it.<br>
Using <code>register_full_backward_hook</code> shows:</p>
<pre><code class="lang-python">LPPool2d input grad:
(tensor([[[[ 0.0928,  5.9412,  7.2183, 15.5274, 15.8743, 24.4626,  0.0000],
          [ 0.3713,  7.5194,  8.2128, 16.9710, 16.9879, 25.8405,  0.0000],
          [ 0.6702,  7.4463,  8.4368, 16.8152, 17.1239, 25.7304,  0.0000],
          [ 1.1914,  9.0101,  9.4586, 18.2457, 18.2465, 27.1025,  0.0000],
          [ 1.5415,  8.8788,  9.6645, 18.0975, 18.3750, 26.9972,  0.0000],
          [ 2.2197, 10.4203, 10.7086, 19.5163, 19.5055, 28.3640,  0.0000],
          [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000]]]]),)
LPPool2d output grad:
(tensor([[[[10.7722, 31.1708, 52.9788],
          [13.4294, 34.2547, 56.1203],
          [16.2184, 37.3533, 59.2653]]]],
       grad_fn=&lt;BackwardHookFunctionBackward&gt;),)
</code></pre>

skyleaworlder: <p>Thank you!</p>
<p>Now I know that it’s because I haven’t understood corresponding API enough.</p>
nn.Upsample not working right

Belial: <p>Hello,<br>
I am trying to use nn.Upsample to make image tensor twice bigger.<br>
But it seems only upsample the width.<br>
Below is the code I use:</p>
<pre><code class="lang-auto">u = nn.Upsample(scale_factor=2)
t = torch.randn([1, 28, 28])
print(u(t).shape) # output shape is [1, 28, 56], I want it to be [1, 56, 56]
</code></pre>
<p>Thanks!</p>

ConvolutionalAtom: <p><code>nn.Upsample</code> will consider the channel dimension, which in pytorch is the dimension in second index.<br>
The channel dimension will not be doubled, but the dimension after this will be doubled, which are height and weight dim. This is based on the following notation</p>
<pre><code class="lang-auto">(Batch_dim, Channel_dim, Height_dim, Width_dim)
</code></pre>
<p>In your problem, adding the channel dimension would be enough</p>
<pre><code class="lang-auto">u = nn.Upsample(scale_factor=2)
t = torch.randn([1, 1, 28, 28])
print(u(t).shape) # output shape is [1, 1, 56, 56]
</code></pre>
<p>If you dont want channel dimension, you can get away with using view to add channel dim</p>

Belial: <p>Got it. Thank you so much!</p>
Question of concatdataset

ws_f: <p>when I don’t use concatdataset, it’s ok.<br>
But when I use concatdataset, I get the error RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument tensors in method wrapper___cat).<br>
I don’t know why, as the bug happens when the code executes at the following<br>
for i, (images, labels) in enumerate(data_loader):</p>

ptrblck: <p>I guess one of your internal datasets might use samples stored on the CPU while another one tries to use samples stored on the GPU. Check how each dataset loads the sample in its <code>__getitem__</code> and make sure to use the same device.</p>

ws_f: <p>I just load CIFAR10 dataset partly using different transforms. How to solve it?</p>

ptrblck: <p>Try to narrow down the issue using my previous suggestion. If you get stuck you can also post a minimal, executable code snippet reproducing the issue.</p>
(facebook/tts_transformer-zh-cv7_css10) Expected tensors should all on the same device

Fahad_Ali1: <p>The error appears when we work on the below code:</p>
<p><code>wav, rate = TTSHubInterface.get_prediction(task, models[0], generator, sample)</code></p>
<p>in the <strong>sample argument</strong> we have three values:</p>
<ol>
<li>src_tokenns</li>
<li>src_lengths</li>
<li>speaker</li>
</ol>
<p>these are tensors that should be on the same device like in below:</p>
<pre><code class="lang-auto">models = [models[0].to(device)]
TTSHubInterface.update_cfg_with_data_cfg(cfg, task.data_cfg)
generator = task.build_generator(models, cfg)

sample = TTSHubInterface.get_model_input(task, text)
device = 'cuda' if torch.cuda.is_available() else 'cpu'
sample['net_input']['src_tokens'] = sample['net_input']['src_tokens'].to(device)

sample['net_input']['src_lengths'] = sample['net_input']['src_lengths'].to(device)

sample['speaker'] = sample['speaker'].to(device)
</code></pre>
Check if tensor elements in a value list

oyyqofuow: <p>Hi, is there any way to see if elements in a tensor are in a value list?<br>
e.g., a tensor a = ([1, 2, 3, 4, 5]), and I want to check if each element in a is in value list [1, 2, 3]<br>
here we can see that element 4 and 5 are not in value list [1, 2, 3], and element 1 and 2 and 3 are in that value list.</p>
<p>It is actually can be written as:  (a == 1) or (a == 2) or (a == 3), while sometimes I don’t want to iterate values in the value list [1, 2, 3] thus I want to do the task without iterating values in the list [1, 2, 3], any function to do that?</p>

Cami_Williams: <p>I found this response from <a class="mention" href="/u/ptrblck">@ptrblck</a> in 2018 that could be a solution: <a href="https://discuss.pytorch.org/t/how-to-judge-a-tensor-is-in-a-list/15998/3" class="inline-onebox">How to judge a Tensor is in a list?</a></p>

iacob: <p>You can achieve this with a <a href="https://stackoverflow.com/q/66779647/9067615" rel="noopener nofollow ugc">for loop</a>:</p>
<pre><code class="lang-python">&gt;&gt;&gt; sum(a==i for i in vals).bool()
tensor([[ True,  True, False],
        [ True,  True, False]])
</code></pre>

Saruar_Alam: <p>You can do achieve this way as well</p>
<pre><code class="lang-auto">setensor = torch.randint(0, 10, size = (5, 20))
selist = [1, 2, 4, 6]
print(setensor)
semasks = torch.any(torch.stack([torch.eq(setensor, aelem).logical_or_(torch.eq(setensor, aelem)) for aelem in selist], dim=0), dim = 0)
print(semasks)
</code></pre>

jiangsijia9495: <p>You help me a lot!<br>
Clever solution!</p>

fleance: <p>You can also achieve this without an explicit for loop:</p>
<pre><code class="lang-auto">t = torch.randint(0, 10, size=( 10, 10))
print(t)
search = torch.tensor([0, 5, 9])
mask = ((search.view(-1, 1) - t.view(-1)).transpose(-1, -2) == 0).sum(dim=-1).view(t.shape) != 0
print(mask.int())  # 1s and 0s for readability =)
print(t[mask].unique())  # matches the search tensor!
</code></pre>
<p>We just subtract every value in t by every value in search, if the result is 0 the value is the same!</p>
Slow arm64 Conv3d on Apple Silicon CPU

ath: <p>I’m noticing that for large input images, Conv3d is substantially slower (4x) and more memory intensive when using python/torch built for arm64. I’m seeing this on a 32GB M1 Pro running macOS 13.0.1.</p>
<p>I’ve installed a prebuilt x86_64 python distribution and a native arm64 python distribution. Both have torch 1.13.1 installed via conda. Here’s the info I get from <code>/usr/bin/time</code> running the below example with each distribution:</p>
<p><strong>python_x86_64:</strong> 31.96 seconds, 6.6 GB peak memory<br>
<strong>python_arm64:</strong> 131.09 seconds, 26.2 GB peak memory</p>
<pre><code class="lang-auto">import torch


class ConvNet(torch.nn.Module):

    def __init__(self, ndims):
        super().__init__()
        channels = [1, 32, 32]
        self.convs = torch.nn.ModuleList()
        for i in range(len(channels) - 1):
        	self.convs.append(torch.nn.Conv3d(channels[i], channels[i + 1], 3, 1))
        	self.convs.append(torch.nn.LeakyReLU(0.2))

    def forward(self, x):
        for conv in self.convs:
            x = conv(x)
        return x

device = 'cpu'
torch.set_num_threads(1)

image = torch.rand(1, 1, 192, 256, 256)
model = ConvNet(ndims=3).to(device)
output = model(image)
</code></pre>
<p>This only seems to be the case for large inputs. In fact, when using an input of size (128, 128, 128), the arm64 version is faster (but still larger memory footprint). I don’t notice this for 2D convolutions. Any idea what is causing this?</p>
How to compute second order gradient

dumbo: <p>Hi Folks,</p>
<p>Is that feasible to use  ∇x∇θL(z, θ) as a regularizer in a classical loss function (e.g. cross entropy)? If so, how to compute it efficiently in Pytorch? Any suggestions would be highly appreciated!</p>

AlphaBetaGamma96: <p>Have a look a the functorch extension, it allows for efficient higher-order gradients</p>
<p>The link to functorch: <a href="https://pytorch.org/functorch/stable/install.html" class="inline-onebox" rel="noopener nofollow ugc">Install functorch — functorch 1.13 documentation</a></p>

dumbo: <p>Thank you <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> , I find a way to compute 2nd order gradient wrt inputs <a href="https://pytorch.org/functorch/stable/generated/functorch.grad.html" class="inline-onebox" rel="noopener nofollow ugc">functorch.grad — functorch 1.13 documentation</a>. Not sure whether it can be used for my aforementioned case: compute 2nd order gradient wrt model parameters and followed by inputs.</p>

AlphaBetaGamma96: <p>You can just wrap different commands together like so,</p>
<pre><code class="lang-auto">model = Model()

fnet, params = make_functional(model)

per_sample_gradients = vmap(jacrev(jacrev(fnet, argnums=0), argnums=1), in_dims=(None, 0))(params, x)
</code></pre>
<p>Functorch is compositional, so you can just compose different <code>jacrev</code> calls to get what you want.</p>

dumbo: <p>This is exactly what I am looking for. Thanks a lot!</p>
Can I use index_copy_ asynchronously in a stream?

rinkujadhav2013: <p>I need to use <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.index_copy_.html#torch.Tensor.index_copy_" rel="noopener nofollow ugc">Tensor.index_copy_</a> in a stream. Will the function be asynchronous or synchronous when I do that?<br>
Also, is it blocking when used outside a stream?<br>
If it’s asynchronous then can I assume that it’s same for all the other similar functions - that they’ll be synchronous outside the stream and asynchronous when a stream is set except for the functions that provide a <code>non_blocking</code> argument?</p>
Extending tensor class to integrate a simple attribute/function

anima: <p>Hi,<br>
I am trying to extend the tensor class, as a simple toy experiment to understand basics of the pytorch, to have a new function “say()”, which will only print the contents of the tensor or a related info. Of course, just calling <strong>repr</strong> would do a similar behaviour, however I am trying the extend the class.<br>
If the tensor class were a common python class, which has an <strong>init</strong> function, then the following might have worked:<br>
def new_tensor(torch.Tensor):<br>
<strong>init</strong>(self, an_array):<br>
super(self).<strong>init</strong>(an_array)<br>
def say(self):<br>
print(super(self))<br>
a desired usage would be:<br>
a = new_tensor([0, 1, 3])<br>
a.say() // prints tensor([0.0000, 1.0000, 3.0000])<br>
How can we extend the tensor class?<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/tensor.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/tensor.py" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/torch/tensor.py</a></h4>
<pre><code class="lang-py">import sys
import torch
import torch._C as _C
from collections import OrderedDict
import torch.utils.hooks as hooks
import warnings
import weakref
from torch._six import imap
from torch._C import _add_docstr


# NB: If you subclass Tensor, and want to share the subclassed class
# across processes, you must also update torch/multiprocessing/reductions.py
# to define a ForkingPickler serialization mode for the class.
#
# NB: If you add a new method to Tensor, you must update
# torch/__init__.py.in to add a type annotation for your method;
# otherwise, it will not show up in autocomplete.
class Tensor(torch._C._TensorBase):
    def __deepcopy__(self, memo):
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/torch/tensor.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
How can I make a single LSTM cell model?

moth: <p>I wish to make a <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTMCell.html" rel="noopener nofollow ugc">single cell LSTM</a> model. So far, I’ve managed to make this model but I am not sure if it is implemented correctly:</p>
<pre><code class="lang-auto">class SingleLSTMCell(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.model = nn.LSTMCell(config.INPUT_SIZE,
                                 config.HIDDEN_SIZE) # (input_size, hidden_size)
        self.h_0 = torch.randn(config.BATCH_SIZE,
                               config.HIDDEN_SIZE,
                               requires_grad=True) # Initial hidden state: (batch, hidden_size)
        self.c_0 = torch.randn(config.BATCH_SIZE,
                               config.HIDDEN_SIZE,
                               requires_grad=True) # Initial cell state: (batch, hidden_size)

    def forward(self, x):        
        x = self.model(x, (self.h_0, self.c_0))
        return x
</code></pre>
<p>Where <code>config</code> is a class which holds parameters to create the model.</p>
<ul>
<li>I want my model to be input a tensor of size (1, time_steps) which represents a one dimensional time series. Example: how price of a product fluctuates in 100 days.</li>
<li>The model should output N time steps in the future, tensor of size (1, N). Example: output how the price of a product will be in the next 7 days.</li>
</ul>
Update binary tensor during training

fabiola: <p>I want to minimize 2 variables, and one is binary. It is not a probability vector for binary classification, but an input to my network who must only contain values 0 and 1. How can I achieve this? I am doing this, but I get the error <code>RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.</code>. The binary variable is <code>mask</code>.</p>
<pre><code class="lang-auto">mask = torch.ones(32, 32)
mask.requires_grad = True
for epoch in range(epochs):
    for d, t in dataloader:
        optimizer.zero_grad()
        output = net(mask*d)
        loss = criterion(output, t)
        loss.backward()
        optimizer.step()
        mask.data[mask &gt;= mask.float().mean()] = torch.Tensor([1])
        mask.data[mask &lt; mask.float().mean()] = torch.Tensor([0])
</code></pre>
<p>I would like that before each forward pass, mask is binary (only 0 and 1 values). What is the best way to do this?</p>
<p>Thanks!</p>

KFrank: <p>Hi Fabiola!</p>
<aside class="quote no-group" data-username="fabiola" data-post="1" data-topic="168868" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fabiola/48/30499_2.png" class="avatar"> fabiola:</div>
<blockquote>
<pre><code class="lang-auto">        mask.data[mask &gt;= mask.float().mean()] = torch.Tensor([1])
        mask.data[mask &lt; mask.float().mean()] = torch.Tensor([0])
</code></pre>
</blockquote>
</aside>
<p>Two comments:</p>
<p>Don’t use <code>.data</code>.  It is deprecated and can break things.</p>
<p>Wrap your modification of <code>mask</code> in a <code>no_grad()</code> block to eliminate the<br>
<code>leaf Variable</code> error:</p>
<pre data-code-wrap="python"><code class="lang-python">    with torch.no_grad():
        mask.data[mask &gt;= mask.float().mean()] = torch.Tensor([1])
        mask.data[mask &lt; mask.float().mean()] = torch.Tensor([0])
    
</code></pre>
<p>Also, it’s possible that you might get better training results if you permit mask<br>
to run from 0.0 to 1.0 and then add a penalty or Lagrange-multiplier constraint<br>
to your <code>loss</code> so that the optimization itself pushes <code>mask</code> to be either 0.0 or<br>
1.0.   For example, <code>mask**2 + (mask - 1.0)**2</code> takes on its minimum of<br>
0.0 exactly when <code>mask</code> is equal to either 0.0 or 1.0.</p>
<p>Best.</p>
<p>K. Frank</p>
Learn Pytorch: Training your first deep learning models step by step

black0017: <p>Hello all, here is a comprehensive pytorch guide. Hope you find it usefull!</p>
<p>Learn Pytorch: Training your first deep learning models step by step</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://theaisummer.com/learn-pytorch/">
  <header class="source">
      <img src="https://theaisummer.com/static/55137e36f7262e3cc1523f2f65e84429/android-icon-192x192.png" class="site-icon" width="192" height="192">

      <a href="https://theaisummer.com/learn-pytorch/" target="_blank" rel="noopener nofollow ugc" title="12:00AM - 22 December 2022">AI Summer – 22 Dec 22</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/284;"><img src="https://theaisummer.com/static/71cdab8c4f3e60a548525635a4b69b04/ee604/learn-pytorch.png" class="thumbnail" width="690" height="284"></div>

<h3><a href="https://theaisummer.com/learn-pytorch/" target="_blank" rel="noopener nofollow ugc">Learn Pytorch: Training your first deep learning models step by step | AI Summer</a></h3>

  <p>This blogpost is about starting learning pytorch with a hands on tutorial on image classification.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
How to use pytroch to realize custom learning a, b parameters?

Gorgen: <p>My task is multimodal emotion recogntion. How to fuse different modality is challenging and difficult.<br>
the dimension of audio or text modality is sequence, batch, feature dimention<br>
Question. bimodality = a<em>audio_modality + b</em>text_modality.</p>
<p>how to use pytroch to realize custom learning a, b parameters? thanks</p>

J_Johnson: <p>For text, you will need a tokenizer and transformer. You can use this tutorial to get started:</p>
<p><a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a></p>
<p>Change the final Softmax layer to ReLU and make the last Linear layer output larger, maybe 64 neurons wide.</p>
<p>For the audio, you can use this tutorial:</p>
<p><a href="https://pytorch.org/tutorials/intermediate/speech_command_classification_with_torchaudio_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/speech_command_classification_with_torchaudio_tutorial.html</a></p>
<p>And change the final layer to ReLU, and change the final Linear layer to 64 neurons wide.</p>
<p>Combine both of the above into one model with one forward pass.</p>
<p>Then concat both of those outputs, and send through a final linear layer, 128 neurons wide for input and output should be however many classifications you need, followed by a Softmax.</p>

Gorgen: <p><a class="mention" href="/u/j_johnson">@J_Johnson</a> Thank you for your detailed advice<br>
best wishes</p>
Calculating the required parameters of transposed convolution for upsampling

Tm4: <p>When I want to use <a href="https://pytorch.org/docs/1.13/generated/torch.nn.ConvTranspose2d.html#torch.nn.ConvTranspose2d" rel="noopener nofollow ugc">the equations</a> for obtaining the values of the parameters of the transposed convolution, I face some problems. For example, I have a tensor with the size of [16,256,16,160,160] and I want to upsample that to the size of [16,256,16,224,224]. Based on the equation of the transposed convolution, when, for solving the equations for the height, I select stride+2 and I want to find the k (kernel size), I have the following equation that the kernel size will have a large and also negative value.</p>
<p>224 = (160 - 2)x (2) + 1x(k - 1) + 1</p>
<p>What is wrong with my calculations and how I can find the parameters.</p>
Will running torch.cuda.synchronize slow down inference performance?

Vedant_Roy: <p>I’m doing inference in a loop. Think:</p>
<pre><code class="lang-auto">for batch in dl:
    outputs = model(batch)
    np.savez(outputs.cpu().numpy())
</code></pre>
<p>I’m trying to benchmark how long it takes for my code to do inference on a single batch.</p>
<p>If I add a <code>torch.cuda.synchronize()</code> before the <code>np.savez</code> line, will this slow down my inference code at all?</p>

ptrblck: <p>No, you should not see any additional slowdown by adding <code>torch.cuda.synchronize()</code> since pushing the <code>CUDATensor</code> to the CPU via <code>outputs.cpu()</code> will already synchronize your code.</p>
Matmul slow for small matrices

edgarriba: <p>I’m doing a small benchmark for small matrices (3x3, 4x4) comparing against numpy and turns out that we are doing quite slow. Hardware is a razer lambda.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/077dcdd47023952299d5cc2959ad93b093c0ba39.png" alt="image" data-base62-sha1="14gSjZAcw7E9IvYrCxOa23KM1bb" width="645" height="337"></p>
<p>to reproduce: <a href="https://github.com/kornia/kornia-benchmark/blob/master/dynamo/test_matmul.py" class="inline-onebox" rel="noopener nofollow ugc">kornia-benchmark/dynamo/test_matmul.py at master · kornia/kornia-benchmark · GitHub</a></p>
<pre data-code-wrap="bash"><code class="lang-bash">pip3 install numpy --pre torch[dynamo] --force-reinstall --extra-index-url https://download.pytorch.org/whl/nightly/cu117
</code></pre>
<p>wondering whether we can improve performance for small matrices to target vision/robotics applications.</p>

marksaroufim: <p>Would you suggest you try out <code>torch.compile(mod, mode="reduce-overhead")</code> for anything on the smaller end. Although I’m not sure if a matmul is the most meaningful benchmark since inductors benefits mostly come from fusions</p>

edgarriba: <p>I’ll try — just trying to find the best way to use torch for small matrices operations.</p>
Apply multiple regularization values to the same layer

Antonio_Ossa: <p>Hi everyone,</p>
<p>Say I’m training a model using triplet loss <a href="https://gombru.github.io/2019/04/03/ranking_loss/" rel="noopener nofollow ugc">1</a>, trying to learn distances between pairs of objects. While training, I have an anchor <code>a</code>, a positive item <code>pos</code>, and a negative item <code>neg</code>. Also, for simplicity, let’s say that the model is this one:</p>
<pre><code class="lang-python">class Model(nn.Module):

    def __init__(self):
        super().__init__()
        self.common_layer = CNN()
        
    def forward(self, anchor, pos, neg):
        anchor = self.common_layer(anchor)
        pos = self.common_layer(pos)
        neg = self.common_layer(neg)
        return anchor * (pos - neg)
</code></pre>
<p>Notice that in the forward step I’m using the same layer with 3 different objects.</p>
<p>While training, I would like to apply L2 regularization (using the <code>weight_decay</code> argument of the optimizer) but using a different lambda value for the anchor (lambda_a), the positive item (lambda_pos), and the negative item (lambda_neg). Is that possible? Is this somehow a “bad practice”?</p>
<p>If my understanding is correct, <a href="https://github.com/pytorch/pytorch/issues/40967#issuecomment-653778463" rel="noopener nofollow ugc">this comment</a> in a GitHub issue seems to imply that applying multiple optimizers on the same layer is undesired (that’s why the suggestion was raising an exception).</p>
<p>PS: This is a simplification of something I saw in a paper implementation. The paper was not implemented in PyTorch, and other implementations seem to ignore this detail. This is a version of the snippet, where the forward step was implemented manually (C++):</p>
<pre><code class="lang-cpp">double anchor = common_repr[anchor_id][f];
double pos_item = common_repr[pos_item_id][f];
double neg_item = common_repr[neg_item_id][f];

common_repr[anchor_id][f]   += learn_rate * ( deri * (pos_item - neg_item) - lambda1 * anchor);
common_repr[pos_item_id][f] += learn_rate * ( deri * anchor - lambda2 * pos_item);
common_repr[neg_item_id][f] += learn_rate * (-deri * anchor - lambda2 / 10.0 * neg_item);
</code></pre>
Two decoders sharing the same encoder

MahaA: <p>Hi. I am trying to implement a network with a shared single encoder and two decoders. Both decoders are trained simultaneously on the same data. I have figured out that if I total the loss of both decoders and perform backpropogation, the gradients will be accumulated on the encoder as well. However, I am confused how validation will be performed for such network. Any help will be greatly appreciated. Thank you!</p>

vdw: <p>I’m not sure, why this a problem. I’ve never tried it, but that’s how I would do it. Below is a code snippet of a RNN-based encoder-decoder model</p>
<pre><code class="lang-auto">class RnnAttentionSeq2Seq(nn.Module):

    def __init__(self, params, criterion):
        super().__init__()
        self.params = params
        self.criterion = criterion

        self.encoder = Encoder(params)
        self.decoder = Decoder(params, self.criterion)
        
        
    def forward(self, X, Y):
        
        # Push through encoder
        encoder_outputs, encoder_hidden = self.encoder(X)

        # Push through decoder
        loss = self.decoder(Y, encoder_hidden, encoder_outputs)

        return loss
</code></pre>
<p>If I had 2 decoders, I would simply change it to:</p>
<pre><code class="lang-auto">class RnnAttentionSeq2Seq(nn.Module):

    def __init__(self, params, criterion):
        super().__init__()
        self.params = params
        self.criterion = criterion

        self.encoder = Encoder(params)
        self.decoder1 = Decoder(params, self.criterion)
        self.decoder2 = Decoder(params, self.criterion)        
        
    def forward(self, X, Y1, Y2):
        
        # Push through encoder
        encoder_outputs, encoder_hidden = self.encoder(X)

        # Push through decoders
        loss1 = self.decoder1(Y1, encoder_hidden, encoder_outputs)
        loss2 = self.decoder2(Y2, encoder_hidden, encoder_outputs)        

        return loss1 + loss2
``</code></pre>

dreidizzle: <p>The above is the right way to go I think. You might want something like t*loss1 + (1-t)*loss2 so you balance the two losses (t is a fraction between 0 and 1).</p>
<p>For validation, you have (after you train), the ability to utilize the model given some input. For example maybe loss 1 has to do with translation and loss 2 has to do with classification. Then, given an input, you’d get a BLEU for translation and some accuracy for the classification task (task 2). I’d imagine the tasks should be kind of related actually (the encoder parameters are shared so it should “gain” by being good on each task), but I have to think more. Let me know if this might be helpful to you …</p>
Memory grows with loop with recursively assigned attribute

David_Van_Dijcke: <p>I have the following function inside a class:</p>
<pre><code class="lang-auto">    def parabola(self):
        # Initialize u1, u2, and u3 with zeros
        u1 = torch.cat((torch.diff(self.ubar.detach(), dim=0).detach().clone(), torch.zeros_like(self.ubar[:1,:,:,:], dtype = torch.float32)), dim = 0)
        u2 = torch.cat((torch.diff(self.ubar.detach(), dim=1).detach().clone(), torch.zeros_like(self.ubar[:,:1,:,:], dtype = torch.float32)), dim = 1)
        u3 = torch.cat((torch.diff(self.ubar.detach(), dim=3).detach().clone(), torch.zeros_like(self.ubar[:,:,:,:1], dtype = torch.float32)), dim = 3)

        
        # Stack the values of mu1 and mu2 for each value of z in the loop
        mu1sum = torch.stack([self.mu1[:,:,:,self.k_indices[z]].sum(dim=-1) for z in range(self.l)], dim=-1)
        mu2sum = torch.stack([self.mu2[:,:,:,self.k_indices[z]].sum(dim=-1) for z in range(self.l)], dim=-1)


        # Calculate u1, u2, and u3 using the formulas in the original function
        u1 = self.p1 + self.sigmap * (u1 + mu1sum)
        u2 = self.p2 + self.sigmap * (u2 + mu2sum)
        u3 = self.p3 + self.sigmap * u3
        
        #B = self.bound(u1,u2,self.lmbda,z+1,self.l,img)
        # Calculate B using self.bound and broadcast it along the last dimension
        k_indices = torch.arange(1, self.l+1, dtype=torch.int64).repeat(self.h, self.w, self.nc, 1)
        B = self.bound(u1, u2, self.lmbda, k_indices, self.l, self.f.unsqueeze(-1).repeat(1,1,1,self.l))

        # Use mask to select elements where u3 &lt; B
        mask = u3 &lt; B
        img = self.f.unsqueeze(self.f.dim()-1).expand(mask.shape)[mask]
        
        
        self.p1[~mask] = u1[~mask]
        self.p2[~mask] = u2[~mask]
        self.p3[~mask] = u3[~mask]
</code></pre>
<p>which is called in a loop to recursively update self.p1, self.p2, self.p3:</p>
<pre><code class="lang-auto">        p1 = torch.zeros((self.h,self.w,self.nc,self.l,), dtype = torch.float32)
        p2 = torch.zeros((self.h,self.w,self.nc,self.l,), dtype = torch.float32)
        p3 = torch.zeros((self.h,self.w,self.nc,self.l,), dtype = torch.float32)

        for iter in range(self.repeats):
            p1, p2, p3 = self.parabola(p1, p2, p3, self.ubar) # project onto parabola (set K)s
</code></pre>
<p>All tensors are created on the GPU with a Mac M1 chip (“mps”). This loop is giving me substantial memory leak, with the GPU memory basically increasing linearly and by several order of magnitude over the course of ~500 iterations.</p>
<p>Any suggestions on what may be causing this? I have tried to detach and clone pretty much everything in this code, to no avail. Autograd is globally disabled.</p>
Quicksom based Clustering

Rohit_Gupta1: <p>I am trying to cluster similar data points using quicksom package which is GPU based SOM using Pytorch. All the functions are running smoothly except when I am using som.predict() function that calculates quantization errors and topographic errors. But the function stops in the middle and throws the following errors:</p>
<p><strong>When I am using batch size =25</strong><br>
‘IndexError: too many indices for tensor of dimension 1’</p>
<p><strong>And when I am using default batch size = 100</strong><br>
‘ValueError: too many values to unpack (expected 2)’</p>
<p>I have attached the snippet of my data, and the errors I am getting.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7de2f51370cf2978399aa0fc11fc214934dd0229.png" data-download-href="https://discuss.pytorch.org/uploads/default/7de2f51370cf2978399aa0fc11fc214934dd0229" title="Screenshot_20221223_020143"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7de2f51370cf2978399aa0fc11fc214934dd0229_2_383x500.png" alt="Screenshot_20221223_020143" data-base62-sha1="hXDW8hwoLMNmtyo2y7cJ2MjDrnP" width="383" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7de2f51370cf2978399aa0fc11fc214934dd0229_2_383x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7de2f51370cf2978399aa0fc11fc214934dd0229_2_574x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7de2f51370cf2978399aa0fc11fc214934dd0229_2_766x1000.png 2x" data-dominant-color="F6EDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20221223_020143</span><span class="informations">784×1023 206 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>