akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to add L2 regularizer?

huynth: <p>This is the code in Tensorflow</p>
<pre><code class="lang-auto">class DownShiftedConv2d(tf.keras.layers.Layer):
    def __init__(self, num_filters, filter_size=(2, 3), dilation_rate=(1, 1), kernel_regularizer='L2', padding='VALID',
                 **kwargs):
        super().__init__(**kwargs)
        if isinstance(filter_size, int):
            self.filter_size = (filter_size, filter_size)
        else:
            self.filter_size = filter_size
        self.wnconv = tf.keras.layers.Conv2D(num_filters, filter_size, strides=dilation_rate,
                                             kernel_regularizer=kernel_regularizer, padding=padding)

    def call(self, inputs, *args, **kwargs):
        output = tf.pad(inputs, [[0, 0], [self.filter_size[0] - 1, 0],
                                 [int((self.filter_size[1] - 1) / 2), int((self.filter_size[1] - 1) / 2)],
                                 [0, 0]])
        output = self.wnconv(output)
        return output

</code></pre>
<p>I want to translate this code from Tensorflow to Pytorch but don’t know the correct way to add L2 regularizer.<br>
This is my code :</p>
<pre><code class="lang-auto">class DownShiftedConv2D(nn.Module):
    def __init__(self, in_channels, num_filters, filter_size=(2, 3), dilation_rate=(1, 1), padding='valid',
                 **kwargs):
        super(DownShiftedConv2D, self).__init__(**kwargs)
        self.in_channels = in_channels
        self.num_filters = num_filters
        self.filter_size = filter_size
        self.dilation_rate = dilation_rate
        
        if isinstance(filter_size, int):
            self.filter_size = (filter_size, filter_size)
        else:
            self.filter_size = filter_size
        self.wnconv = nn.Conv2d(in_channels=in_channels, out_channels=num_filters, kernel_size=filter_size,
                                stride=dilation_rate, padding=padding)

    def forward(self, inputs, *args, **kwargs):
        output = F.pad(inputs, (0,0, 
                                self.filter_size[0] - 1, 0,
                                int((self.filter_size[1] - 1) / 2), int((self.filter_size[1] - 1) / 2),
                                0, 0))
</code></pre>
<p>Can anyone help me ? Thank you</p>

suraj.pt: <p>You could add the weight’s L2 norm to the loss before optimizing</p>
<pre><code class="lang-auto">l2_lambda = 0.01
l2_reg = torch.tensor(0.)
for name, param in model.named_parameters():
    if 'conv' in name:
        l2_reg += torch.linalg.norm(param)
loss += l2_lambda * l2_reg
</code></pre>
<p>AFAIK the <code>weight_decay</code> parameter in your <code>optimizer</code> will apply to all parameters in the network, and not just the conv layers.</p>

huynth: <p>Thanks for your answer <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"></p>
Is NNAPI conversion an all-or-nothing in terms of operators?

carrythetorch: <p>Hi everybody, first post.<br>
I have a question regarding NNAPI conversion: do all operators need to be supported by NNAPI for the conversion to be successful? I am coming from the Tensorflow Lite world, where NNAPI unsupported operators get pushed back to CPU (with obvious performance implications, but at least the model can be run).<br>
Since <code>torch.backends._nnapi.prepare.convert_model_to_nnapi</code> doesn’t take any additional arguments, it seems it’s all-or-nothing. E.g. I was trying to convert MobileNet V3 to NNAPI, but getting an error that <code>quantized::hardswish</code> is not supported.</p>

axitkhurana: <p>Yes <a class="mention" href="/u/carrythetorch">@carrythetorch</a>, it’s all the operators need to be supported at this point. They aren’t that hard to implement, we’re definitely open to PRs: <a href="https://github.com/pytorch/pytorch/blob/master/torch/backends/_nnapi/serializer.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/serializer.py at master · pytorch/pytorch · GitHub</a></p>
Building a Look-up Table for Latency

saadabbasi: <p>I’m interested in building a look-up table (LUT) to estimate the latency for a given neural network architecture. The idea is that network architectures consist of discrete operations combined in a variety of ways. If we assume that the total latency of an architecture is the sum of the latency of its individual operations, we can have a simple way of estimating its latency.</p>
<p>My question is, what is the best way to measure the latency on a GPU? I use the following code to measure the latency of a complete end-to-end architecture. Is this synchronous code valid for single operations on GPUs as well?</p>
<pre><code class="lang-auto">def latency_on_gpu(net, input_v, N = 100):
    input_v = input_v.to('cuda:0')
    net.to('cuda:0')
    timings = np.zeros((N,1))
    starter, ender = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True) 
    for _ in range(10):
        _ = net(input_v)
    for n in range(N):
        starter.record()
        y = net(input_v)
        ender.record()
        torch.cuda.synchronize()
        curr_time = starter.elapsed_time(ender)/1000
        timings[n] = curr_time
    
    return timings
</code></pre>
[SOLVED] No torch version in MacOS Catalina

eastwing: <p>I’m trying to restore all python packages on MacOS 10.15.7 Catalina and now I faced the following error:</p>
<p>ERROR: Could not find a version that satisfies the requirement torch==1.10.0 (from versions: none)<br>
ERROR: No matching distribution found for torch==1.10.0</p>
<p>I’ve tried to point Pip to the page with -f parameter. Is there any way to install it to Catalina?</p>
<p>MacOS 10.15.7 Catalina<br>
Python 3.10.1</p>

eastwing: <p>Ok, I’ve figured it out. I had to downgrade python to 3.9.9 so it has nothing to do with MacOS.</p>
Why does BCEWithLogitsLoss give negative values for the input?

shamoons: <p>My loss function is:</p>
<pre><code class="lang-auto">def loss_fn(inp, target):
    zeros_sum = (target == 0).sum(dim = 0).float()
    one_sum = (target == 1).sum(dim = 0).float()

    pos_weight = zeros_sum / (one_sum + 1e-2)
    loss_fn = torch.nn.BCEWithLogitsLoss(reduction='mean', pos_weight=pos_weight)

    loss = loss_fn(inp, target)

    return loss
</code></pre>
<p>When passing to my model, I’m passing in the result of a convolution and no Sigmoid activation:</p>
<pre><code class="lang-auto">            pred = model(inputs)
            
            loss = loss_fn(pred, outputs)
</code></pre>
<p>So then why does my <code>pred</code> have negative values? I thought <code>Sigmoid</code> squashes between 0 and 1?</p>

ptrblck: <p>Could you check the range of your <code>target</code> tensor and make sure it’s in <code>[0, 1]</code>?</p>
<p>Also, could you explain the two code snippets, as the second one uses <code>outputs</code> as the target tensor?</p>

shamoons: <p>I see the issue. I was doing:</p>
<p><code>print(pred)</code></p>
<p>I should be doing <code>print(torch.nn.Sigmoid()(pred))</code></p>

Madhu_Varun: <p>I have a doubt. According to Pytorch documentation for BCEWithLogitsLoss, sigmoid calculation will be done. My question is why are you taking sigmoid again in torch.nn.Sigmoid()(pred)?</p>
<p>Please help me.</p>

JapinderSandhu: <p>I have the same question,</p>
<p>why use torch.nn.Sigmoid()(pred) after using BCEwithLogitLoss? I thought BCEwithLogitLoss already used Sigmoid built in ?</p>

ptrblck: <p><code>nn.BCEwithLogitLoss</code> applies the sigmoid internally and calculates the loss given the logits as inputs. If you want to use the probabilities given by the model, you would have to apply the sigmoid manually.</p>
How to Reverse a Torch Tensor

peak: <p>How to Reverse a Torch Tensor</p>

fmassa: <p>There is not yet slicing with negative indices in pytorch, but this issue is being tracked in <a href="https://github.com/pytorch/pytorch/issues/229">https://github.com/pytorch/pytorch/issues/229</a> .<br>
In the mean time, you can use <code>index_select</code> for that</p>
<pre><code class="lang-python">tensor = torch.rand(10) # your tensor
# create inverted indices
idx = [i for i in range(tensor.size(0)-1, -1, -1)]
idx = torch.LongTensor(idx)
inverted_tensor = tensor.index_select(0, idx)
</code></pre>

qianguih: <p>Hi, it looks like this can’t work on torch.autograd.variable.Variable. Is there any way to reverse a specific dimension in a Variable? Thanks!</p>

apaszke: <p>It should work once you wrap <code>idx</code> in a Variable</p>

DiffEverything: <p>If your use case is to reverse sequences to use in Bidirectional RNNs, I just create a clone and flip using numpy.</p>
<pre><code class="lang-python">rNpArr = np.flip(fTensor.numpy(),0).copy()   #Reverse of copy of numpy array of given tensor
rTensor = torch.from_numpy(rNpArr)          
</code></pre>

Sunil_Sharma: <p><strong>Can you try something like:</strong></p>
<blockquote>
<p>import torch as pt</p>
<p>aa = pt.tensor([[1,2,3],[4,5,6],[7,8,9]])<br>
idx = [i for i in range(aa.size(0)-1, -1, -1)]<br>
idx = pt.LongTensor(idx)<br>
inverted_tensor = pt.index_select(aa,0,idx)<br>
print(“Before ::”,aa)<br>
print(“After ::”,inverted_tensor)</p>
</blockquote>

aRI0U: <p>Use <a href="https://pytorch.org/docs/stable/torch.html#torch.flip" rel="nofollow noopener">torch.flip</a></p>

chorus12: <p>torch.flip does not do the task but <a href="https://pytorch.org/docs/stable/generated/torch.fliplr.html" rel="noopener nofollow ugc">fliplr</a> does</p>
Access the optimizer state_dict

afshin67: <p>In Python one can get the the underlying data of an optimizer by calling the <code>optimizer.state_dict()</code>. For example for the Adam optimizer, it returns the number of steps, exponential moving average of gradient values, and exponential moving average of squared gradient values. I was trying to access the same values in the C++ version of torch, but it seems that it is not straight-forward to get them. I looked into the Adam optimizer implementation at:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/src/optim/adam.cpp">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/src/optim/adam.cpp" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/src/optim/adam.cpp" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/torch/csrc/api/src/optim/adam.cpp</a></h4>


    <pre><code class="lang-cpp">#include &lt;torch/optim/adam.h&gt;

#include &lt;torch/csrc/autograd/variable.h&gt;
#include &lt;torch/nn/module.h&gt;
#include &lt;torch/serialize/archive.h&gt;
#include &lt;torch/utils.h&gt;

#include &lt;ATen/ATen.h&gt;
#include &lt;c10/util/irange.h&gt;

#include &lt;cmath&gt;
#include &lt;functional&gt;

namespace torch {
namespace optim {

AdamOptions::AdamOptions(double lr) : lr_(lr) {}

bool operator==(const AdamOptions&amp; lhs, const AdamOptions&amp; rhs) {
  return (lhs.lr() == rhs.lr()) &amp;&amp;
</code></pre>


  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/src/optim/adam.cpp" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
But, still it is not clear to me how to get those.</p>
<p>I looked into <code>optimizer-&gt;state()</code> which can be obtained by:<br>
<code>ska::flat_hash_map&lt;std::string, std::unique_ptr&lt;torch::optim::OptimizerParamState&gt;&gt;&amp; state_ = optimizer-&gt;state();</code><br>
and it does not look like what I was expecting.</p>
<p>Does anyone know how can I access the number of steps, exponential moving average of gradient values, and exponential moving average of squared gradient values in C++ API?</p>
Model sharding, data parallelism and NVLink

royve: <p>Hi Guys,<br>
I read the following thread:<br>
<a href="https://discuss.pytorch.org/t/split-single-model-in-multiple-gpus/13239">https://discuss.pytorch.org/t/split-single-model-in-multiple-gpus/13239</a></p>
<p>and also watched the following video:<br>
<a href="https://youtu.be/_d3xs1L4jeA" rel="noopener nofollow ugc">https://youtu.be/_d3xs1L4jeA</a></p>
<p>You guys provided great information, but I still have some questions. In my group, we are interested in buying a server with 8 Nvidia A40 GPUs, such that those 8 GPUs are split into 4 groups of 2, where each pair of GPUs are physically connected using a NVLink bridge.<br>
I wonder how using 4 pairs of NVLink GPUs will affect the utilization of data parallelism and model sharding. Will the code snippets provided in the thread above work as is? For example, will I be able to achieve model sharding between two pairs of GPUs?<br>
In general, I wonder how the usage of NVLink will accelerate training and inference with respect to the common usage of multi-gpu scenarios such as data parallelism and model sharding (when comparing it to the case where there is no usage of NVLink at all).</p>
<p>Thanks!</p>
Unable to calculate differential using torch.autograd.grad

Vedaant_Joshi: <p>Hello all, I’m trying to calculate the differential of the output with respect to one of the column in the dataset. I am not able to produce the differential. It is a regression problem and the network has two outputs.  There are more than 2000 columns and only 1 column is required in the calculation of differential.</p>
<p><strong>Error produced is as follows:</strong></p>
<blockquote>
<p>RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</p>
</blockquote>
<p><strong>Model :</strong></p>
<blockquote>
<p>class Model2(nn.Module):</p>
<pre><code>def __init__(self, input_dim):

        super(Model2, self).__init__()
        self.layer1 = nn.Linear(input_dim, 512)
        self.layer2 = nn.Linear(512, 256)
        self.layer3 = nn.Linear(256, 128)
        self.layer4 = nn.Linear(128, 64)
        self.layer5 = nn.Linear(64, 2)
        
    def forward(self, x):
        x = F.relu(self.layer1(x))
        x = F.relu(self.layer2(x))
        x = F.relu(self.layer3(x))
        x = F.relu(self.layer4(x))
        x = self.layer5(x)
        return x
</code></pre>
</blockquote>
<p><strong>Training</strong>:</p>
<blockquote>
<p>EPOCHS = 1000<br>
fourth_loss_train = np.zeros((EPOCHS,))<br>
fourth_loss_val = np.zeros((EPOCHS,))<br>
fourth_train_mape = np.zeros((EPOCHS,))<br>
fourth_val_mape = np.zeros((EPOCHS,))</p>
<p>x1 = X_train[:,-2].requires_grad_(True)<br>
x2 = 1 - x1 #.requires_grad_(True)</p>
<p>for epoch in tqdm.trange(EPOCHS, position=0, leave=True):</p>
<pre><code>model_PINN_5.train()

y_pred = model_PINN_5(X_train)
loss1 = loss_fn(y_pred, y_train)

d_lngamma1_x1 = torch.autograd.grad(y_pred[:,0], x1, torch.ones_like(y_pred[:,0]), create_graph=True)[0]
d_lngamma2_x2 = torch.autograd.grad(y_pred[:,1], x2, torch.ones_like(y_pred[:,1]), create_graph=True)[0]

physics = x1@d_lngamma1_x1 # - x2@d_lngamma2_x2
loss2 = (1e-4)*torch.mean(physics**2)

loss = loss1 + loss2

fourth_loss_train[epoch] = loss.item()

# Zero gradients
loss.backward()
optimizer.step()
optimizer.zero_grad()


# Calculate train accuracy
fourth_train_mape[epoch] = mean_absolute_percentage_error(y_train, y_pred)

with torch.no_grad():
    model_PINN_5.eval()
    
    y_pred = model_PINN_5(X_val)
    val_loss = loss_fn(y_pred, y_val)
    fourth_loss_val[epoch] = val_loss
    
    fourth_val_mape[epoch] = mean_absolute_percentage_error(y_val, y_pred)
</code></pre>
</blockquote>
Concerning unknown errors with Torch.abs() and F.elu()

tsuijenk: <p>Hello everyone,</p>
<p>I’ve been decoding for many hours but I still haven’t been able to figure out why.</p>
<ol>
<li>
<p>Torch.abs() returns unknown error when running in terminal, with PyTorch Lightning, and Wandb (if that matterrs).</p>
<ul>
<li>no NaN values</li>
<li>range is 0,1.</li>
<li>size is (20,1600,16)</li>
<li>tensor sum to 12700, meaning that we probably have a Sparse Tensor<br>
* This “sparse tensor” thing is the only cause for the error that came to mind, as I find several relevant posts online.</li>
</ul>
</li>
<li>
<p>F.elu(x) produces unknown error.</p>
<ul>
<li>x is also range (0,1), with no NaN values.</li>
<li>this is taking place after the two down-sampling stages, in the first up-sampling stage.</li>
<li>within the first up-sampling stage, this is after 2 convolutional layers + 2 F.elu(x) + 3rd convolutional layer.</li>
</ul>
</li>
</ol>
<p>I am very puzzled…Could both of these be due to memory error?</p>

tsuijenk: <p>Update: Sad…but I am pretty sure Out of Memory error is involved in this.</p>
<p>I have tried clearing cache, but no luck. (<a href="https://github.com/pytorch/fairseq/blob/50a671f78d0c8de0392f924180db72ac9b41b801/fairseq/trainer.py#L283" class="inline-onebox" rel="noopener nofollow ugc">fairseq/trainer.py at 50a671f78d0c8de0392f924180db72ac9b41b801 · pytorch/fairseq · GitHub</a>)</p>
<p>Currently, I have been using batch size 64, but I guess I might have to try batch size 32.</p>
<p>Still open to more solutions and tips.</p>
<p>P.s. My code is all vectorized.</p>
Custom loss function no_grad and OOM error

wbsst: <p>I want to use a custom loss function, where I do a compute condition before using the <code>MSELoss</code> what ever I do getting error <code>no_grad</code> if I used the <code>requires_grad=True</code> I get OOM error, note my batch size was working fine before the custom loss_fn, when I reduce the batch size to 2 it works but the loss doesn’t improve at all, for about 10k steps nothing changes same value.</p>
<pre><code class="lang-auto">pred = model(input)
loss_fn = nn.MSELoss()
diff = torch.gt(torch.sub(pred, target), 0).type(pred.type())
diff.requires_grad=True
tar = torch.ones(diff.size()).type(diff.type()).cuda()
loss = loss_fn(diff, tar)
</code></pre>

cbd: <p>I think “diff” is True or False and “tar” is float. MSE might give error. Isn’t it? Can you post “tar” and “loss” value if its not giving error?</p>

wbsst: <p>I have changed the diff type by <code>.type(pred.type())</code> to float, I don’t think that’s the problem, it’s like the training graph is detached when using the <code>torch.gt</code> and <code>torch.sub</code></p>

cbd: <p>torch.gt is not differentiable.</p>

wbsst: <p>I also try using math operations <code>((pred-target)&gt;0).float()</code> but still the same error</p>

cbd: <p>Greater than itself is non differentiable. You should not use it if you want gradient flow.</p>

wbsst: <p>so there’s no solution for me to use this loss function, without this step my loss function it’s not complete</p>
Spatially-varying filters for Conv2d

Alan_Wang: <p>Is there an accepted way to implement spatially-varying filters for Conv2d? Or is that something that needs a custom implementation? If so, what would be the best way to go about doing that?</p>
<p>Just to be clear, by spatially-varying filters, I mean a convolutional layer where the filter changes as it slides across the feature map.</p>
<p>Thanks in advance for any suggestions.</p>

saisushanth: <p>Hi Alan !! Did you find the solution for the above question? If so, please let me know that</p>
How can I get the sum of gradients immediately after loss.backward()?

JimW: <p>I am trying to do some importance sampling experiments:<br>
During an evaluation epoch, I calculate the loss for each training sample, and obtain the sum of gradients for this training sample.  Finally, I will sort the training samples based on gradients they introduced. For example, if sample A shows a very high gradient sum, it must be an important sample to training. Otherwise, it is not a very important sample.</p>
<p>Note that, the gradients calculated here will not be used to update parameters. In other words, they are only used for selecting importance samples.</p>
<p>I know gradients will be ready somewhere after loss.backward(). But what is the easiest way to grab the summed gradients over the entire model?  In my current implementation,  I am only allowed to modify one small module with only loss availble, so I don’t have “inputs” or “model”. Is it possible to get the gradients from only “loss”?</p>

huahuanZ: <p>It’s easy to get the sum of loss with a model</p>
<pre data-code-wrap="python"><code class="lang-python"># make gradients ready
loss.backward()
grad_sum = sum(param.grad.sum() for param in model.parameters() if param.grad is not None)
</code></pre>
<p>What do you mean</p>
<blockquote>
<p>so I don’t have “inputs” or “model”</p>
</blockquote>
<p>Where is your learnable parameters?</p>

JimW: <p>Thanks.  I am working on some existing codebase, but I am only allowed to modify the evaluate_metric() module.  It has GT and Prediction as inputs,  but I don’t have the access to the network or the input data of the network. Therefore,  I cannot do “model.parameters()” at this point, because I don’t have “model” passed to this evaluate_metric() module.  For “inputs”,  I meant to say the data (i.e. RGB image tensor), which is not accessible inside evaluate_metric(), either.</p>

huahuanZ: <p>Sorry for the late reply. I don’t know any approach that can sum up the gradients through only a function interface. Maybe you should ask the owner of the NN for model access.</p>
Conv2dTranspose vs Conv2d

pamdwj: <p>Are they the same thing if i want to apply a convolution layer with kernel_size 1 and stride 1?</p>

ptrblck: <p>No, as the input and output channels will be transposed in the transposed conv layer compared to the plain conv one.<br>
If you permute it back, the operations would be equal for this setup:</p>
<pre><code class="lang-python">conv = nn.Conv2d(3, 6, 1)
conv_trans = nn.ConvTranspose2d(3, 6, 1)

with torch.no_grad():
    conv_trans.weight.copy_(conv.weight.permute(1, 0, 2, 3).contiguous())
    conv_trans.bias.copy_(conv.bias)
    
x = torch.randn(2, 3, 24, 24)
out1 = conv(x)
out2 = conv_trans(x)
print((out1 - out2).abs().max())
# &gt; tensor(2.3842e-07, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
Transfomer model IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Oscar_Garcia: <p>Hi.</p>
<p>I’m trying to use the Transfomers class as in <a href="https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html" class="inline-onebox" rel="noopener nofollow ugc">Transformer — PyTorch 1.10.1 documentation</a></p>
<p>I have some pair of of embeddings of 512 and 256 dims. My problem is that when doing this to a given pair:</p>
<pre><code class="lang-auto">transformer_model = nn.Transformer(nhead=16, num_encoder_layers=12)
a = torch.Size([512])
b  = torch.Size([256])
out = transformer_model(a, b)
</code></pre>
<p>I get the following:</p>
<p><code> IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)</code></p>
<p>I have seen some related issues, but dont know how to fix it in this case. How could I solve this dimension problem?</p>
<p>When using just the encoder, I face another issue:</p>
<pre><code class="lang-auto">encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8)
transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=6)
a = torch.Size([512])
transformer_encoder(a)
</code></pre>
<p>I get:</p>
<pre><code class="lang-auto">ValueError: not enough values to unpack (expected 3, got 1)
</code></pre>
<p>How should I reshape the tensor in this second case. Is something like this <code>v = torch.reshape(a, (1, 1, 512))</code> the right approach? Also, could I directly project the output to 256 dims?</p>
<p>Thanks.</p>

ptrblck: <p>As given in the linked docs, the shapes for <code>src</code> and <code>tgt</code> are expected as:</p>
<blockquote>
<ul>
<li>src: <code>[S,N,E]</code>, <code>[N, S, E]</code> if <code>batch_first</code>.</li>
<li>tgt: <code>[T,N,E]</code>, <code>[N, T, E]</code> if <code>batch_first</code>.</li>
</ul>
</blockquote>
<p>while <code>a</code> and <code>b</code> seem to have a single dimension with a shape of <code>256</code> which raises the indexing error.</p>
Timing pytorch inference for concurrent python processes

copah: <p>Say I have python process A and python process B scheduled concurrently like this:</p>
<pre><code class="lang-auto">#!/bin/bash

python my_model_A.py &amp;
python my_model_B.py &amp;
</code></pre>
<p>I want to understand the inference time for each of the models while they run standalone and concurrently.</p>
<p>For a standalone pytorch model I follow the guides <a href="https://discuss.pytorch.org/t/how-to-measure-time-in-pytorch/26964/2">here</a>, that essentially tells me to synchronize the CUDA device.</p>
<p>Now the same code still works for two concurrently pytorch models. The issue here is that the inference isn’t really reported correctly, because the call <code>torch.cuda.synchronize()</code> blocks the other process, so there is a lot of idling reported in the timing.</p>
<p>An example: Say Model A takes 150ms and so does Model B. For standalone application, both inference time is correctly reported using above link. If I schedule them concurrently, the above linked code will report 300ms for both models, since it is essentially reporting both Model A and Model B inference due to the blocking of <code>torch.cuda.synchronize()</code>.</p>
<p>Is there a way to schedule these two processes concurrently and still measure the inference separately?</p>

ptrblck: <p>I think it depends a bit what you actually want to measure. If you want to measure the entire runtime of the script you could run it via <code>time python script.py</code> and make sure the execution is indeed finished e.g. by manually synchronizing or by printing a value of the result.<br>
Manual syncs are needed, if you want to measure the actual GPU runtime since CUDA code is executed asynchronously.</p>
How to use f1 metric with PyTorch?

shallowLearning: <p>I am currently working on a multi-label binary classification problem.</p>
<p>The output of my model is a tensor like this:</p>
<p><code>tensor([[3.0914e-08, 3.2459e-17]])</code></p>
<p>and the ground truth label looks like this:</p>
<p><code>tensor([[1, 1]])</code></p>
<p>I iterate over a custom validation DataLoader (after training for one epoch) and for every input and label I execute:</p>
<pre><code class="lang-auto">prediction = self._model(x)
loss = self._crit(output, y.float())
return loss, prediction
</code></pre>
<p>Where <code>self._crit</code> is BCELoss. Now I am trying to calculate the f1-score using <code>sklearn.metrics</code> via:</p>
<p><code>f1_score(label[0], (prediction[0] &gt; 0.5).float() * 1)</code></p>
<p>I don’t know whether this is the correct approach but the f1_score always ends up being 0.</p>
<p>Instead of calculating this for every validation sample, should I maybe do this only for the whole validation set?</p>
<p>If so - how?</p>

ptrblck: <p>You could store the predictions in a new tensor (or append them to a <code>list</code> and create the tensor after the validation loop is done) and calculate the F1 score afterwards on the entire validation set.</p>
RuntimeError: Expected hidden[0] size (2, 1, 15), got [2, 16, 15]

ashcher51: <p>I’m building a seq2seq model for time series data, and I am getting this error:<br>
<code>RuntimeError: Expected hidden[0] size (2, 1, 15), got [2, 16, 15] </code></p>
<p>I can tell that the error has something to do with the LSTM in my decoder, but I am unable to resolve this.<br>
Any help with this would be greatly appreciated.</p>
<p>Here are the encoder and decoder classes:</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    def __init__(self, input_dim, hid_dim, n_layers, dropout):
        super().__init__()
        
        self.hid_dim = hid_dim
        self.n_layers = n_layers
        self.rnn = nn.LSTM(input_dim, hid_dim, n_layers, dropout=dropout, batch_first=True)
    
    def forward(self, x):
        outputs, (hidden, cell) = self.rnn(x)
        return hidden, cell

class Decoder(nn.Module):
    def __init__(self, output_dim, input_dim, hid_dim, n_layers, dropout):
        super().__init__()
        
        self.output_dim = output_dim
        self.hid_dim = hid_dim
        self.n_layers = n_layers
        self.rnn = nn.LSTM(output_dim, hid_dim, n_layers, dropout=dropout, batch_first=True)
        self.fc_out = nn.Linear(hid_dim, output_dim)
        
    def forward(self, input, hidden, cell):
        input = input.unsqueeze(0).view(1, -1, 3)
        output, (hidden, cell) = self.rnn(input, (hidden, cell))
        prediction = self.fc_out(output.squeeze(0))
        return prediction, hidden, cell
</code></pre>
<p>and here is the seq2seq class if necessary:</p>
<pre><code class="lang-auto">class Seq2Seq(nn.Module):
    def __init__(self, encoder, decoder, device):
        super().__init__()
        
        self.encoder = encoder
        self.decoder = decoder
        self.device = device
    
    def forward(self, X, y, teacher_forcing_ratio = 0.5):
        batch_size = 3
        y_len = y.shape[0]
        y_size = self.decoder.output_dim
        
        outputs = torch.zeros(y_len, batch_size, y_size).to(self.device)
        hidden, cell = self.encoder(X)
        input = y[0, :]
        
        for t in range(1, y_len):
            output, hidden, cell = self.decoder(input, hidden, cell)
            outputs[t] = output
            teacher_force = random.random() &lt; teacher_forcing_ratio
            input = y[t] if teacher_force else output
        return outputs
</code></pre>

ashcher51: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> would you be able to help me out with this?</p>

ptrblck: <p>I don’t know which input shaoe you are passing to the model, but would guess that <code>hidden</code> or <code>cell</code> is creating the shape mismatch.<br>
Here is a small examples showing the <code>hidden</code> and <code>cell</code> outputs in the expected shape:</p>
<pre><code class="lang-python">model = nn.LSTM(input_size=3, hidden_size=15, num_layers=2, batch_first=True)
x = torch.randn(1, 100, 3)
out, (hidden, cell) = model(x)
</code></pre>
<p>so check the shape of <code>hidden</code> and <code>cell</code> and make sure their shape matches the expected ones.</p>
How can I make npz dataloader?

wonchulSon: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa45eeded124048dda30f3390800b159daecc85b.png" alt="image" data-base62-sha1="zI1cjgEEShsImbssxyTtcPWen2P" width="289" height="244"></p>
<p>I have these ImageNet 32x32 npz dataset.<br>
How can I concatenate these batches and make one dataloader?<br>
And npz file contains the label information too?</p>
<p>Please help.<br>
Thank you!</p>

ptrblck: <p>The <code>.npz</code> file format is usually used by <a href="https://numpy.org/doc/stable/reference/generated/numpy.savez.html"><code>numpy.savez</code></a> so we cannot know, what’s inside the data.</p>
<p>You can use <code>np.load</code> to load each file and inspect it.<br>
Once you got the numpy arrays, you could transform them to tensors via <code>torch.from_numpy</code> and create your <code>Dataset</code>.</p>

DuaneNielsen: <p>Here’s an NPZ loader I wrote for my own dataset…</p>
<p>Like Peter said, it’s going to be different for each archive structure… but perhaps this will be useful to get started…</p>
<pre><code class="lang-auto">import torch
import numpy as np
from pathlib import Path


class NPZLoader(dataloader.Dataset):
    def __init__(self, path, transform=None):
        self.path = path
        self.files = list(Path(path).glob('*/*.npz'))
        self.transform = transform

    def __len__(self):
        return len(self.files)

    def __getitem__(self, item):
        numpy_array = np.load(str(self.files[item]))['arr_0']
        torch_array = torch.from_numpy(numpy_array)
        if self.transform is not None:
            torch_array = self.transform(torch_array)
        return torch_array, 0
</code></pre>
Weight tensor stored on CPU despite model being stored on GPU

namiyousef: <p>Hi,</p>
<p>I run into the following error:</p>
<pre><code class="lang-python">RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same
</code></pre>
<p>when I try to run my model on Colab (with GPU runtime enabled).<br>
During my training loop I send my batches to the GPU using:<br>
<code>X = X.to(device)</code><br>
<code>y = y.to(device)</code></p>
<p>and the model is also on the GPU, using:<br>
<code>model.to(device)</code></p>
<p>Sources online indicate that this error arises because the network isn’t on the GPU (see <a href="https://stackoverflow.com/questions/59013109/runtimeerror-input-type-torch-floattensor-and-weight-type-torch-cuda-floatte" rel="noopener nofollow ugc">S/O</a>, and <a href="https://discuss.pytorch.org/t/runtimeerror-input-type-torch-cuda-floattensor-and-weight-type-torch-floattensor-should-be-the-same/21782">PT forums</a>. However, I’m almost certain that mine is, having verified with <code>next(model.parameters()).is_cuda</code>.</p>
<p>Any advice would be appreciated.</p>

namiyousef: <p>The problem was that one of the submodules was saved in local function so it was not bound to the main nn.Module class.</p>
Multiple labels compressed during inference

smrolfe: <p>Hello,</p>
<p>I followed the <a class="mention" href="/u/monai">@MONAI</a> <a href="https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/unetr_btcv_segmentation_3d.ipynb" rel="noopener nofollow ugc">BTCV UNETR tutorial</a> to segment my own dataset as described in this <a href="https://discuss.pytorch.org/t/monai-tutorial-debugging/140469/4">thread</a>. The training results look reasonable, so now I’d like try the model with new data.</p>
<p>I adapted the <a href="https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/torch/unet_inference_dict.py" rel="noopener nofollow ugc">UNET inference exampl</a>e for my UNETR model, but have made some error, as the output is [0-1], rather than the expected [0-50]. Any pointers on why this is occurring would be appreciated. I have posted my notebook <a href="https://github.com/smrolfe/MultiOrganSegmentation/blob/main/unetr_inference.ipynb" rel="noopener nofollow ugc">here</a>.</p>

ptrblck: <p>Could you verify that:</p>
<ul>
<li>your model has indeed 51 output channels and returns the logits in the shape <code>[batch_size, nb_classe=51, height, width]</code>
</li>
<li>the target tensors contain all class indices in <code>[0, 50]</code>
</li>
</ul>
<p>If so, could you try to overfit a single sample with a target containing a variety of class indices and check if the model is able to learn it?</p>
How to use LSTMCell with variable length sequence?

beann: <p>I try to use LSTMCell to produce results for variable-length sequences, and get multiple predictions by adding a linear layer after it,<br>
I take inspiration from this codebase <a href="https://discuss.pytorch.org/t/how-to-obtain-memory-states-from-pack-padded-sequence/58283/2" class="inline-onebox">How to obtain memory states from pack padded sequence - #2 by Fawaz_Sammani</a>, and what I do is as follows,</p>
<pre><code class="lang-python">import torch
from torch import nn

sequences = torch.LongTensor([[1, 2, 0, 0, 0, 0], 
                              [3, 4, 5, 0, 0, 0],  
                              [5, 6, 0, 0, 0, 0],  
                              [8, 9, 10, 11, 12, 0]])  
seq_lengths = torch.LongTensor([2, 3, 2, 5])

class PackedMemory(nn.Module):
    def __init__(self, vocab_size, embed_dim, decoder_dim):
        super(PackedMemory, self).__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTMCell(embed_dim, decoder_dim)
        self.decoder_dim = decoder_dim
        self.linears = [nn.Linear(decoder_dim, 10), nn.Linear(decoder_dim, 10)]
        
    def forward(self, seq, seq_len):
        batch_size = seq.size(0)
        sorted_lengths, sort_indices = torch.sort(seq_len, descending=True) 
        sorted_sequences = seq[sort_indices]
        hidden_states = torch.zeros(batch_size, max(sorted_lengths), self.decoder_dim)
        memory_states = torch.zeros(batch_size, max(sorted_lengths), self.decoder_dim)
        final_hidden = torch.zeros(batch_size, self.decoder_dim)  # 这个用于存储 hidden
        embeddings = self.embedding(sorted_sequences)
        h,c = [torch.zeros(batch_size, self.decoder_dim), torch.zeros(batch_size, self.decoder_dim)]
        res = []
        for t in range(11):
            if t &gt;= max(sorted_lengths):  # TODO for some short sequence
                pass
            else:
                batch_size_t = sum([l &gt; t for l in sorted_lengths])
                print(batch_size_t)
                h, c = self.lstm(embeddings[:batch_size_t, t, :], (h[:batch_size_t], c[:batch_size_t])) 
                hidden_states[:batch_size_t, t, :] = h
                memory_states[:batch_size_t, t, :] = c
                final_hidden[:batch_size_t] = h.clone()
            if t%5 == 0:
                res.append(self.linears[t//10](final_hidden)) 
        return res

packed_memory = PackedMemory(13,512,512)
res = packed_memory(sequences, seq_lengths)
print(res)
</code></pre>
<p>This code is runnable, but it seems the learned results are not as I imagined.<br>
I guess the error may occur in the line marked as TODO. Does anyone have some suggestions?</p>
Pytorch gradient calculation

Shreya_Saha: <p>Hi, I am new to pytorch and this is what I want to do:</p>
<p>I have two models that are related to one another: modelA and modelB. I want to get three separate gradients. I am able to get the first two without any issues, but I am not sure how I can get the third one.</p>
<ol>
<li>
<p>differentiating the loss of modelA wrt modelA.parameters<br>
[v.grad.data for v in modelA.parameters()]</p>
</li>
<li>
<p>differentiating the loss of modelB wrt modelB.parameters<br>
[v.grad.data for v in modelB.parameters()]</p>
</li>
<li>
<p>differentiating the loss of modelB wrt modelA.parameters</p>
</li>
</ol>
<p>This is what I tried: torch.autograd.grad(lossB, modelA.parameters())<br>
However, I get the following error: grad can be implicitly created only for scalar outputs</p>
<p>Any help would be great!</p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="Shreya_Saha" data-post="1" data-topic="141138">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shreya_saha/48/45583_2.png" class="avatar"> Shreya_Saha:</div>
<blockquote>
<p>grad can be implicitly created only for scalar outputs</p>
</blockquote>
</aside>
<p>you need to make sure that <code>lossB</code> is a scalar (i.e. a single number) so make sure you sum over your batch dimension as well. If you want per-sample gradients, this can be done via the use of hooks (but gets quite messy to implement)</p>
<p>Also, if <code>lossB</code> doesn’t depend on the parameters of <code>modelA</code> the gradients will be zero by definition. So, check they depend on each other!</p>
Mixing bool indexing with normal indexes. Bug?

Anders_G: <p>I want to do something like this:</p>
<pre><code class="lang-auto">  import torch
  
  x = torch.randn((5, 4, 3))
  mask = torch.randn((5, 4)) &lt; 0
  
  # This is ok
  print(x[mask, :].shape)
  
  # This is not, why?
  print(x[mask, 0].shape)
  
  # I like to set a value like this.
  x[mask, 0] = 2
</code></pre>
<p>But I get an error for the line x[mask, 0]: “IndexError: The shape of the mask [5, 4] at index 1 does not match the shape of the indexed tensor [5, 3] at index 1”</p>
<p>What is going on here. Is it a bug or feature and is there some way to accomplish x[mask, 0] = 2 without an intermediate tensor?</p>
Are there any constraints to the image sizes for torchvision models?

maflobra: <p>The <code>torchvision.models</code> documentation mentions that all models require an image input size of <em>“at least 224x224”</em>, with some exceptons, like the <em>Inception</em> model, which requires exactly (?) 299x299. Now I am wondering, for all other models, are <strong>really any</strong> other image dimensions fine as well?</p>
<p>I understand that technically these “modern” models can all handle other dimensions as well. But are there any constraints to the image dimensions, or better: are there any dimensions that work better than others? Or can I use any “random” sizes, like for example 474x474 or 834x834, or whatever? Or should I follow some “best practices” when choosing the image dimensions?</p>

J_Johnson: <p>If I am not mistaken, they use an average pooling layer after convolutions. So the image data can go through the model without popping up errors. That is not to say that the comprehension of the model will be the same for all image sizes. It depends on the structure of the model.</p>
<p>For instance, a Conv2d layer with a given kernel may be able to capture a particular feature of the image important to classification at one size, but not when that same feature is much larger. (Read up on kernels to understand why.)</p>
<p>Due to the way neural networks learn, it’s not like there is a one size fits all answer to this. It may also be dataset dependent. But generally speaking, if you have enough compute resources, you should trying resizing beforehand and see what size works best for your application. Or if you don’t have those resources, just preprocess your images to the smallest size allowed, as that will also equate to less calculations/compute time during training or inference.</p>
What is the purpose of a mask in time series forecasting?

NullPointer: <p>Why do we train with the mask and then make predictions with it during inference (the ladder idea is clearer). Why can’t we train the model and create a mask such that it extends an initial period?</p>
<p>Ex:</p>
<pre><code class="lang-auto"># mask would be the same dimensions except a different length, and filled with 0
torch.cat(previousTimeTensor, mask)
</code></pre>
Modifing segmentation masks for only a few specific classes

drydenwiebe: <p>Hello,</p>
<p>Right now I have pseudo-labels for an image dataset from DeepLabV3+ that are of shape [H W, 1] where each pixel is an integer from 0 to 18 representing the class of the pixel (using cityscapes pretraied models).</p>
<p>Now, what I want to do is focus on only a few classes (say for example, 10, 12, 13) and map the rest of the classes to some background class. What is the best way to do this in PyTorch?</p>
<p>I guess numpy would also work.</p>
<p>Thanks!</p>

KFrank: <p>Hi Dryden!</p>
<aside class="quote no-group" data-username="drydenwiebe" data-post="1" data-topic="141093" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/50afbb/48.png" class="avatar"> drydenwiebe:</div>
<blockquote>
<p>Now, what I want to do is focus on only a few classes (say for example, 10, 12, 13) and map the rest of the classes to some background class.</p>
</blockquote>
</aside>
<p>You therefore want four classes, “background”, “class 10”, “class 12”, and<br>
“class 13.”  You should then label them with with the four integer class<br>
labels <code>0</code>, <code>1</code>, <code>2</code>, <code>3</code>.</p>
<p>Set up a translation table that maps the integers <code>{0, 1, 2, ..., 17, 18}</code><br>
to your desired smaller set of class labels, <code>{0, 1, 2, 3}</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">translate_table = torch.tensor ([0] * 19)
translate_table[10] = 1
translate_table[12] = 2
translate_table[13] = 3
</code></pre>
<p>Let’s say that <code>labels</code> is your pseudo-labels tensor (in your example<br>
of shape <code>[H, W, 1]</code>).  Then:</p>
<pre data-code-wrap="python"><code class="lang-python">mapped_labels = translate_table[labels]
</code></pre>
<p>will be the class labels for your reduced four-class problem (and will<br>
still have shape <code>[H, W, 1]</code>).</p>
<p>Best.</p>
<p>K. Frank</p>
Google Colab crashing again and again

shivani_sharma: <p>I am trying to extract features.</p>
<p>extracted_features = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
main_extracted_features = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for batch in range(100):<br>
print(“Batch Number”,batch)<br>
data_for = new_data[batch*100:(batch+1)*100]<br>
for i in range(len(data_for)):<br>
img = torch.tensor(data_for[i]/255).float()<br>
ft_extract = feature_extractor(img.to(device))<br>
extracted_features.append(ft_extract)<br>
main_extracted_features.append(extracted_features)<br>
print(np.array(extracted_features).shape)<br>
print(np.array(main_extracted_features).shape)</p>
<p>and then crashes</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b2b34c1aa9f05da7987adad08a33e948310aeb9.png" data-download-href="https://discuss.pytorch.org/uploads/default/6b2b34c1aa9f05da7987adad08a33e948310aeb9" title="Screenshot 2021-01-09 at 1.16.28 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/b/6b2b34c1aa9f05da7987adad08a33e948310aeb9_2_690x363.png" alt="Screenshot 2021-01-09 at 1.16.28 PM" data-base62-sha1="fi3EN6ZDEby9aUxyOKlShZVhhIB" width="690" height="363" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/b/6b2b34c1aa9f05da7987adad08a33e948310aeb9_2_690x363.png, https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b2b34c1aa9f05da7987adad08a33e948310aeb9.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b2b34c1aa9f05da7987adad08a33e948310aeb9.png 2x" data-dominant-color="E4E4E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-01-09 at 1.16.28 PM</span><span class="informations">996×524 73.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

user_123454321: <p>It would seem like colab is running out of memory. I think it is because of not using .detach() function but not sure, have to look at more of the code.</p>

lvkd84: <p>Did it say because Colab is out of RAM? I have had Colab crashing from last night for “unknown reason”. What is your runtime log output? For me, it’s always crashes after “zmq.error.ZMQError: Address already in use”.</p>
How to access grad of view of tensor

AreTor: <p>Hi everyone,</p>
<ul>
<li>I have a tensor <code>t</code>  with <code>requires_grad = True</code>
</li>
<li>I apply a boolean advanced indexing to <code>t</code>, say <code>t1 = t[mask]</code>.</li>
<li>The mask is regular so I can reshape <code>t1</code> to have a shape <code>s</code>, that is <code>t2 = t1.reshape(s)</code> (or <code>t1.view(s)</code>)</li>
</ul>
<p>Now, I want to access the gradient of <code>t2</code>, that is, to retrieve a tensor <code>g</code> with shape <code>s</code> containing the gradient of <code>t</code> at the corresponding positions of <code>mask</code>.</p>
<p>The <code>view</code>/<code>reshape</code> operators make <code>t2</code> a non-leaf tensor, thus calling <code>t2.grad</code> returns <code>None</code>.</p>
<p>Is there a way to get these gradients?</p>

tom: <p>You can do <code>t2.retain_grad()</code> before the backward.</p>
OOM when allocating tensor with shape

samm: <p>i got this problem with using embedding for 150.000 sentences using BERT model<br>
<code>tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1619381,768] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:RandomUniform]</code></p>
<p>as the layer of embedding is<br>
<code>    embed = Embedding(1619381, 768, mask_zero=True)(sentence)</code><br>
i got this number <strong>1619381</strong> from     <code>embeddings = torch.cat(embeddings,dim=0)</code><br>
which contains all embedding for 150.000 sentences .</p>

tom: <p>Note that this is a tensorflow error…</p>