akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
L_inf norm equivalent of L_2 norm. in higher dimensions

username: jarvico
cooked: <p>Hi all,</p>
<p>I am sorry, my question is not related with Pytorch at all, but I would be glad if somebody could help on my problem.</p>
<p>I am working on a research subject where I need to implement different adversarial attack types on MNIST data.</p>
<p>I use max perturbation limit “epsilon” under L_infinity norm. This L_inf norm is limiting the max amount of allowed perturbation to each pixel of 28x28 image using a chosen epsilon parameter.</p>
<p>For example: I use epsilon value of 0.15 with L_inf norm metric, use PGD attack type and get an attack success rate around %60 for all MNIST test data.</p>
<p>But I am not sure which epsilon I need to chose when I need to implement the same attack under L2_norm metric. ( to simulate the same volume of perturbation)</p>
<p>I know that in higher dimensions the volume of L2 norm ball is smaller than the L_inf norm cube. And for the same amount of perturbation ( to make the volumes of L_2 ball equal to L_inf cube), I need to use a bigger epsilon value for L_2 norm.</p>
<p>For example, if I chose an epsilon of 0.1 for L_inf norm, then may be I need to chose an epsilon value of around 1.3 for L_2 norm.</p>
<p>But I don’t know the exact conversion formula for the correct epsilon from L_inf norm to L2_norm.</p>
<p>I would be very happy if somebody could help me on this.</p>
<p>Thanks in advance…</p>

username: jarvico
cooked: <aside class="quote no-group" data-username="jarvico" data-post="1" data-topic="107131">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jarvico/48/28509_2.png" class="avatar"> jarvico:</div>
<blockquote>
<p>But I am not sure which epsilon I need to chose when I need to implement the same attack under L2_norm metric. ( to simulate the same volume of perturbation)</p>
</blockquote>
</aside>
<p>I still couldn’t figure out how to calculate necessary l2 norm perturbation epsilon_1 value which will result in the same volume of perturbation for l’inf norm under epsilon_2 value.</p>
<p>I need to calculate epsilon_1 using epsilon_2 and the image dimensions. Can somebody help me to find out the conversion formula?</p>
<p>Thanks in advance…<br>
Jarvico</p>
Translating keras.backend.gradients to pytorch

username: kdebugging
cooked: <p>I am trying to translate this keras code to pytorch:</p>
<pre><code class="lang-auto">from keras import backend as K
y_hat = model.get_layer('fc_out').output[0, 0]
conv_output = model.get_layer(activation_layer).output

grads = K.gradients(y_hat , conv_output)[0]
grad_function = K.function([model.get_layer('input_1').input], [conv_output, grads])
output, grads_val = grad_function([img])
</code></pre>
<p>so far I manage to get the <code>output</code> variable correctly but not the <code>grads_val</code>:</p>
<pre><code class="lang-auto">activation = {}

def get_activation(name):
    def hook(model, input, prediction):
        activation[name] = prediction
    return hook

model.fc_out.register_forward_hook(get_activation('fc_out'))
model.last_swish.register_forward_hook(get_activation(activation_layer))
model.eval()
prediction = model(img)

y_hat = activation_layer['fc_out']
output = activation_layer[activation_layer']
grads_val = torch.autograd.grad(y_hat , output, grad_outputs=torch.ones_like(y_hat))
</code></pre>
<p>Any clue what I did wrong here? Thank you</p>
DataLoader num_workers in a multi-node distributed setup

username: rina
cooked: <p>I am following the setup used here: <a href="https://github.com/pytorch/examples/tree/ee964a2eeb41e1712fe719b83645c79bcbd0ba1a/imagenet" class="inline-onebox" rel="noopener nofollow ugc">examples/imagenet at ee964a2eeb41e1712fe719b83645c79bcbd0ba1a · pytorch/examples · GitHub</a> and I am trying to run it on 2 nodes, with 4 GPUs each.</p>
<p>When using <code>DistributedDataParallel</code>, the example calculates num_workers as follows:</p>
<pre><code class="lang-auto">workers = int((args.workers + ngpus_per_node - 1) / ngpus_per_node)
</code></pre>
<p>Otherwise, it uses the number entered by the user, with <code>4</code> as the default.</p>
<p>I am not sure why they followed this calculation.<br>
What are the guidelines for setting the number of workers, especially in the distributed setup?</p>
<p>Thank you</p>
How to save a model in an s3 bucket?

username: spadel
cooked: <p>The title says it all - I want to save a pytorch model in an s3 bucket. What I tried was the following:</p>
<pre><code>import boto3

s3 = boto3.client('s3')
saved_model = model.to_json()
output_model_file = output_folder + "pytorch_model.json"
s3.put_object(Bucket="power-plant-embeddings", Key=output_model_file, Body=saved_model)
</code></pre>
<p>Unfortunately this doesn’t work, as <code>.to_json()</code> only works for tensorflow models. Does anyone know how to do it in pytorch?</p>

username: spadel
cooked: <p>Does anyone have an answer for this? This seems to be quite an important issue and I believe many people will benefit from a solution.</p>
Applying functions to certain columns

username: asr
cooked: <p>Hi,<br>
I have problem that requires applying function objects (like<code> torch.sin</code>) to some columns of a matrix <code>X</code>. Right now I solve this by having a list of tuples like this: <code>[(function, col)]</code> where function is the function object and col is the column this function acts on.<br>
In each step I stack the resulting columns and return the matrix. Now I’m not sure this is the most performant way because it’s doing a list comprehension over all columns for every evaluation. Doing something like</p>
<pre><code class="lang-auto">X[:,i] = torch.sin(X[:,i]) 
</code></pre>
<p>is also not possible because it breaks autograd. I was thinking the new <a href="https://pytorch.org/docs/master/generated/torch.vmap.html#torch.vmap" rel="noopener nofollow ugc">torch.vmap</a> could somehow help here?<br>
Another problem with my approach is that due to the list mentioned above I can’t jit the module because the functor datatype is not supported.</p>
<p>Do you have any tips on how to improve the implementation?</p>
RuntimeError: The size of tensor a (128) must match the size of tensor b (332) at non-singleton dimension 2

username: Alymostafa
cooked: <p>I tried to plug the linformer implementation from this repo  <a href="https://github.com/kuixu/Linear-Multihead-Attention" class="inline-onebox" rel="noopener nofollow ugc">GitHub - kuixu/Linear-Multihead-Attention: Reproducing the Linear Multihead Attention introduced in Linformer paper (Linformer: Self-Attention with Linear Complexity)</a> into PyTorch, replacing the <code>MultiheadAttention</code> in <code>transformer.py</code> by <code>linear_multihead_attention</code> class and forward function. Using <code>seq_len</code>= 332 and <code>project_k</code>=128.I printed the shape of <code>attn_output_weights</code>: torch.Size([64, 332, 128]), and the <code>attn_mask</code>: torch.Size([1, 332, 332]).</p>
How to implement sum fusion for two stream CNN

username: rubijade
cooked: <p>I have implemented 2 streams CNN for action recognition. The first stream is for the spatial model for features and the second one is for temporal features (optical flow)<br>
I did train each model separately and for the next step, I want to apply sum fusion and then make the prediction for the class using softmax.<br>
my questions are:<br>
1- for the fusion, I just get the weights for the classifier of each stream and then I did a simple tensor addiction. Is that the correct way to do it? I<br>
2- for the class prediction in the last step using the softmax, should I make the training with both spatial and optical flow data? I am so confused.<br>
I did read papers but they did not go into detail and the codes that I find in GitHub did not make sense to me<br>
Your help is welcome. Thank you in advance</p>
Create a neural network that tells you that he saw this state before

username: epic_spaces
cooked: <p>I’m really stuck creating a neural network that tells you that he saw this state before, or he never saw it,  if not tells not. Like numbers from 0 to 10, I’ve thought of one output neuron that has 0 if number not shown or 1 so, but I’m stuck, maybe 2 outputs (0,1) without softmax or argmax ?</p>

username: epic_spaces
cooked: <p>ahem ! well don’t let this topic die</p>

username: Napam
cooked: <p>I am not sure how I would do it your way with a neuron that directly classifies seen or unseen.</p>
<p>However, an unsupervised way would be to overfit an autoencoder to some dataset. The idea is that given a sample from the training set, it will achieve a very low loss. Given an unseen sample it would get a very high loss.</p>
How to push data and model onto the same GPU if using multiple GPUs via nn.DataParallel(model)

username: minawenli
cooked: <p>Hello,</p>
<p>I have a CNN code which works fine on CPU but doesn’t on multiple GPUs.</p>
<p>The error message is:<br>
RuntimeError: Expected tensor for argument <span class="hashtag">#1</span> ‘input’ to have the same device as tensor for argument <span class="hashtag">#2</span> ‘weight’; but device 1 does not equal 0 (while checking arguments for cudnn_convolution)</p>
<p>Except using the following two lines in my code, I didn’t do anything specifically for the multiple GPU.<br>
model = nn.DataParallel(model)<br>
data, label = data.to(device), label.to(device)</p>
<p>I have searched around. It seems it is because the input data are not on the same GPU as the model. So in my case, how to push the data and model on the same GPU? Many thanks in advance.</p>

username: Sayed_Nadim
cooked: <p>You can try the following if you want to have both data and label on the same device.</p>
<pre><code class="lang-auto">device = torch.device("cuda:0")
data, label = data.to(device),  label.to(device)
</code></pre>

username: minawenli
cooked: <aside class="quote no-group" data-username="Sayed_Nadim" data-post="2" data-topic="111765">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sayed_nadim/48/34051_2.png" class="avatar"> Sayed_Nadim:</div>
<blockquote>
<p><code>to(device)</code></p>
</blockquote>
</aside>
<p>Hi Sayed,</p>
<p>Thank you for the reply. I am using multiple GPUs, and because it is on server, every time, I got different server. So I don’t necessarily have GPU0 to use. Also, the message shows that the input and weight are not on the same device. How does this thing happen? And what does device 1 for and why it should be 0?</p>
Training input from frozen model?

username: chenjesu
cooked: <p>I have a model that I trained on MNIST… what I’m trying to do is to use an optimizer to adjust the <em>input</em> so that it looks more like a picture of a 6.<br>
I’m not sure if this is possible, since I looked at GANs and they all seem to have two networks to do what I am trying with one network.<br>
Starting from random input…</p>
<pre><code class="lang-auto">for parameter in model.parameters():
    parameter.requires_grad = False
    print((param.shape, param.requires_grad))

input_data = torch.rand(1,1,28,28)
optimizer = optim.Adadelta(model.parameters(), lr=args.lr)
input_data.requires_grad = True
model.train()
print(input_data.requires_grad)
data = input_data.to(device)
target = torch.tensor([6], dtype=torch.long).to(device)

# the main training loop
for i in range(10):
    optimizer.zero_grad()
    output = model(data)
    loss = F.nll_loss(output, target)
    loss.backward()
    optimizer.step()
    pred = output.argmax(dim=1, keepdim=True)  # get the index of the max log-probability
    print(pred, output)
ninput = data.clone().detach().cpu().numpy()[0][0]
plt.imshow(ninput)
</code></pre>

username: Nikronic
cooked: <p>Hi,</p>
<p>There are few things:</p>
<ol>
<li>GANs are a different architecture/loss to deal with generating images, you can do it with a simple MLP too (input, output and loss functions are the only things that matter - at least to me!).</li>
<li>You have mentioned that you want use a trained network (freezed weights) and only update input to satisfy a loss (similar to number 6). But you have passed <code>model.parameters()</code> instead of <code>input</code> to the optimizer, so your optimizer does not know anything about input. Just pass inputs to optimizer!</li>
<li>Put your model in <code>model.eval()</code> mode as it could have batch norm or dropout layers.</li>
</ol>
<p>Other parts seem fine to me.</p>
<p>Bests</p>
Checkerboard artifacts when using GANs (for image-to-image translation)

username: Li-Yun
cooked: <p>Can we categorize some causes, for having artifacts in generative models? Instead of replacing transposed convolutions with upsampling and convolutional layers, are any empirical “ways” to search in the right direction? More specifically I noticed some border checkerboard artifacts that do not disappear with the aforementioned use of upsampling.</p>
Training disables if turn of if add method for Agent

username: lemikhovalex
cooked: <p>Hello! I’ve been following <a href="https://github.com/bentrevett/pytorch-rl" rel="noopener nofollow ugc">bentrevett’s repo on RL</a> and got a crazy idea.<br>
Having class Agent(nn.Module) implement a method which will do sampling and making log_prob for reinforcement learning. But it ruined everything. It ruins only if I use it for training(where I need gradients), in other places it does’t change anything.</p>
<pre><code>def sample_action(self, state, stoch=True):
    self.__action_pred, value_pred = self.forward(state)
    self.__action_prob = F.softmax(self.__action_pred, dim = -1)
    dist = distributions.Categorical(self.__action_prob)
    if stoch:
        action = dist.sample()
    else:
        action = torch.argmax(self.__action_prob, dim = -1)
    return action, value_pred, dist
</code></pre>
<p>here is the function that ruins learning.<br>
More, if I even return<br>
self.__action_pred, value_pred = self.forward(state)<br>
learning stops(</p>
<p>You can find problem example <a href="https://github.com/lemikhovalex/pytorch-rl/blob/master/5_tr%20-%20Proximal%20Policy%20Optimization%20(PPO)%20%5BCartPole%5D-Copy1.ipynb" rel="noopener nofollow ugc">here</a><br>
Dont you know what is this?</p>
Optimizing sorting dataset into tensors by target class

username: Dylan_Yung
cooked: <pre><code class="lang-auto">import torch
import random
from torchvision import datasets, transforms

class DataManager():
    def __init__(self, data: datasets):
        self.data = data
        self.data_len = len(data)
        # (Class, 4D Tensor)
        self.tensor_dict = dict()
        self.avg_tensor_dict = dict()

        for i in range(self.data_len):
            tensor, target = data[i]

            if target in self.tensor_dict:
                old_tensor = self.tensor_dict[target]
                self.tensor_dict[target] = torch.cat([old_tensor, tensor.unsqueeze(0)],0)
            else:
                self.tensor_dict[target] = tensor.unsqueeze(0)
    
    def get_class_tensor(self, key):
        return self.tensor_dict[key]
    
    def get_average_class_tensor(self, key):
        tensor = self.tensor_dict[key]
        tensor = torch.mean(tensor, 0)
        self.avg_tensor_dict[key] = tensor

        return tensor
    
    def sample_class_tensor(self, key, num: int):
        class_tensor = self.tensor_dict[key]
        num_tensors = class_tensor.shape[0]
        samples = random.sample(range(0, num_tensors), num)
        ret_tensor = None

        for i in range(len(samples)):
            tensor = class_tensor[samples[i]]
            if i == 0:
                ret_tensor = tensor.unsqueeze(0)
            else:
                ret_tensor = torch.cat([ret_tensor, tensor.unsqueeze(0)], 0)
            
        return ret_tensor
</code></pre>
<p>I have this datamanager that takes in a dataset and sorts cifar10 into a dictionary where each key is associated with a 4D tensor of all the tensors with a label of the same class. The for loop makes it pretty slow as cifar is quite a large dataset. Is there a way to optimize this?</p>
Network in for loop memory usage during training

username: Jonathan_Brokman
cooked: <p>Hi,<br>
given a nn block “B” an input image “x” is forwarded like so: x = B(x) for T iterations.</p>
<p>This is in contrast to the more common setting x = B_T(…(B_2(B_1(x))…), where all B_i’s share architecture, but not weights.</p>
<p>I am using the former, with an additional constrain: I need access to all mid-outputs B(x), B(B(x), etc…</p>
<p>I implemented this via a for loop, and noticed it is slower and consumes more memory during training than the “common case”. How can I alleviate this problem?</p>
<p>.</p>
<p>Thanks, Jonathan</p>

username: ptrblck
cooked: <p>Could you post (pseudo) code, which would show the difference in the performance?<br>
If I understand your use case correctly, you are comparing these approaches:</p>
<pre><code class="lang-python"># approach 1
model = nn.Linear(1, 1)
x = torch.randn(1, 1)

for _ in range(100):
    x = model(x)

# approach 2
models = nn.ModuleList([nn.Linear(1, 1) for _ in range(100)])
x = torch.randn(1, 1)
for m in models:
    x = m(x)
</code></pre>

username: Jonathan_Brokman
cooked: <p>Yes, this is the case, only with AdaIn-like res-blocks instead of f.c.’s. Interestingly approach 1 consumes more memory during training.</p>
<p>How can I implement approach 1 more efficiently? Note again, that I need access to all mid-outputs</p>
<p>Thank you very much, Jonathan</p>
Beginner question: How to handle tensor input shapes for a simple FFNN?

username: jooragbo
cooked: <p>Hello,</p>
<p>I’m new to NNs and pytorch!</p>
<p>I have some data being provided to my model via a DataLoader with the batch size set to 8. The shape of my input is (8, 15, 50).</p>
<p>I want to make a simple FFNN to classify my data, there are 50 different classes for my data.<br>
So, I’m trying to do Linear, ReLU, Linear, ReLU, Linear (two hidden layers and then a final Linear that outputs a 50-dimensional vector).</p>
<p>Given that the initial input is (batch_size, 15, 50), what do I need to do to the shape so that the initial Linear will work on it? I thought it should be 15*50 on the first layer? But I think I need to reshape the (8, 15, 50) input?</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>I don’t know what the two dimensions in your input mean, but assuming they represent independent features, you could use:</p>
<pre><code class="lang-python">x = x.view(x.size(0), -1)
</code></pre>
<p>to flatten the input into the shape <code>[batch_size=8, features=15*50]</code>.<br>
The <code>in_features</code> of the first linear layer would also be set to <code>15*50</code> in this case.</p>
<p>Note that <code>nn.Linear</code> layers accept multiple dimensions, where the last dim would correspond to the <code>in_features</code> while the other additional dimensions would be used to reapply the linear layer on them.<br>
This could be used in case your <code>dim1</code> dimension represent a temporal dimension and you would like to apply the model on all time steps in it.</p>
Torch-nightly1.8 with cudnn 8.1 doesn't recognize rtx 30xx and 20xx series

username: dmdmello
cooked: <p>Hello,</p>
<p>I’m eager to get performance improvements for my rtx 3070. So I tried today the torch-nightly1.8-cudnn8.1 version via miniconda, it installed successfully, but I got the following warning when setting the cuda device:</p>
<p>import torch<br>
torch.cuda.set_device(1)<br>
/home/corsair/miniconda3/envs/torch1.8_cudnn8.1/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py:104: UserWarning:<br>
GeForce RTX 3070 with CUDA capability sm_86 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_37 sm_50 compute_37.<br>
If you want to use the GeForce RTX 3070 GPU with PyTorch, please check the instructions at <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
<p>warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))<br>
/home/corsair/miniconda3/envs/torch1.8_cudnn8.1/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py:104: UserWarning:<br>
GeForce RTX 2070 SUPER with CUDA capability sm_75 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_37 sm_50 compute_37.<br>
If you want to use the GeForce RTX 2070 SUPER GPU with PyTorch, please check the instructions at <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
<p>warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))</p>
<p>I think something is off, I was working with nightly 1.8-cudnn8.0.5 just fine with both my gpus. 1.9-cudnn8.1 also works fine.</p>

username: ptrblck
cooked: <p>The current conda nightly binaries are broken, as they only contain two old compute capabilities.<br>
CC <a class="mention" href="/u/seemethere">@seemethere</a></p>
Even though I use torch.no_grad(): one of the variables needed for gradient computation has been modified

username: HmmRfa
cooked: <p>I return all activations like this:</p>
<pre><code class="lang-auto">def forward(self, input):
        x = input
        allActivations = {}
        for i in range(self.n_layers):
            x = getattr(self, 'layer_' + '{0:02d}'.format(i))(x)
            allActivations['layer_' + '{0:02d}'.format(i)] = x
        return x, allActivations
</code></pre>
<p>No I want to set them to a different value like this:</p>
<pre><code class="lang-auto">        with torch.no_grad():
            for key, value in allActivations.items():
                value.div_(someValue)
</code></pre>
<p>Do I need to call torch.no_grad or not? Thanks for your help!</p>

username: ptrblck
cooked: <p>The <code>torch.no_grad()</code> guard will make sure that Autograd doesn’t track any new operations applied inside the block. However, inplace operations on tensors, which are needed for gradient computation, can still be disallowed.<br>
In your particular case a specific <code>value</code> tensor is needed during the backward pass to calculate the gradient, while you are modifying it inplace, which would yield the wrong result.</p>
How to downsample an image with interleaved output

username: vic_rain
cooked: <p>I have an single-channel image with size 32x32. I want to down-sample it into 16x16 as 4 images or 1 image has four channels such that no pixel in the original 32x32 image is lost.</p>
<p>The four images or 4-channel image should have each pixel at the same location is interleaved with each other in the original image.</p>
<p>For example, below 4x4 image are downscaled into  4 2x2 images. Each number represents the new image index.</p>
<p>1 2 1 2 1 2 1 2 1 2<br>
3 4 3 4 3 4 3 4 3 4<br>
1 2 1 2 1 2 1 2 1 2<br>
3 4 3 4 3 4 3 4 3 4<br>
1 2 1 2 1 2 1 2 1 2<br>
3 4 3 4 3 4 3 4 3 4<br>
1 2 1 2 1 2 1 2 1 2<br>
3 4 3 4 3 4 3 4 3 4</p>

username: Jonathan_Brokman
cooked: <p>Hi! I’m doing the same… Did you find an efficient solution? Im currently indexing and concatenating which feels expensive, given the fact this is a form of reshape…</p>
What does it contain when using torch.save to save the dataset

username: 111429
cooked: <p>Hi, my question is regarding this <a href="https://github.com/pytorch/vision/blob/master/references/video_classification/train.py#L128" rel="noopener nofollow ugc">code</a><br>
Later the script torch.save the dataset as dataset cache to facilitate the data loading next time.</p>
<p>The second time I load the dataset from cached is indeed much faster.<br>
But when I try to remove the data in the <code>traindir</code>, then load the cache, there is an error.<br>
So the code still needs to visit the dataset path to read the video on the fly, correct?<br>
Then how can it claim to “pre-compute” the data?<br>
And the saved dataset looks much smaller than actual dataset.<br>
so what’s under the hood? I only know the transform can be saved in it.<br>
Thanks.</p>
The dimension and size of autograd.grad

username: israrbacha
cooked: <p>Hi everyone, I am facing a problem regarding the shape and dimension of autograd.grad tensor. I wonder it maybe a python related issue but i failed to find any satisfactory answer. So kindly help me.</p>
<p>'import torch</p>
<p>from torch.autograd import grad</p>
<p>import torch.nn as nn</p>
<h1>Create some dummy data.</h1>
<p>x = torch.ones(2, 2, requires_grad=True)</p>
<p>gt = torch.ones_like(x) * 16 - 0.5  # “ground-truths”</p>
<h1>We will use MSELoss as an example.</h1>
<p>loss_fn = nn.MSELoss()</p>
<h1>Do some computations.</h1>
<p>v = x + 2</p>
<p>y = v ** 2</p>
<h1>Compute loss.</h1>
<p>loss = loss_fn(y, gt)</p>
<p>print(f’Loss: {loss}’)</p>
<h1>Now compute gradients:</h1>
<p>d_loss_dx = grad(outputs=loss, inputs=x)</p>
<p>print(len(d_loss_dx))</p>
<p>print(f’dloss/dx:\n {d_loss_dx}’)’</p>
<p>when i run the above code the output is:<br>
Loss: 42.25<br>
1<br>
dloss/dx:<br>
(tensor([[-19.5000, -19.5000],<br>
[-19.5000, -19.5000]]),)<br>
but in the same code when i change the line:<br>
d_loss_dx = grad(outputs=loss, inputs=x)[0]<br>
the output is:<br>
Loss: 42.25<br>
2<br>
dloss/dx:<br>
tensor([[-19.5000, -19.5000],<br>
[-19.5000, -19.5000]])</p>
<p>I didn’t understand the difference please help Thanks!!!</p>

username: israrbacha
cooked: <p>d_loss_dx = grad(outputs=loss, inputs=x)[0]<br>
The difference is here, when i remove “[0]” it change the tensor shape but the values are same. I put the results in the above question.</p>

username: AlphaBetaGamma96
cooked: <p><code>torch.autograd.grad</code> will returns your gradients in a <code>tuple</code>. Which for a single gradient Tensor will be like what you have above</p>
<pre><code class="lang-auto">(tensor([[-19.5000, -19.5000],
[-19.5000, -19.5000]]),)
</code></pre>
<p>(A Tensor wrapped inside brackets with a comma at the end.)</p>
<p>However, when you add <code>[0]</code> at the end of <code>d_loss_dx = grad(outputs=loss, inputs=x)[0]</code>, you’re telling PyTorch to return the 0th element of that tuple which in your case is the tensor within it, i.e.,</p>
<pre><code class="lang-auto">tensor([[-19.5000, -19.5000],
[-19.5000, -19.5000]])
</code></pre>
<p>I hope that makes sense?</p>
<p>You can get similar behaviour without adding <code>[0]</code> at the end by placing a comma after your <code>d_loss_dx</code> variable, i.e.,</p>
<pre><code class="lang-auto">d_loss_dx,  = grad(outputs=loss, inputs=x)
</code></pre>
<p>will return</p>
<pre><code class="lang-auto">tensor([[-19.5000, -19.5000],
[-19.5000, -19.5000]])
</code></pre>
Create AnyModule from generic pointer?

username: meganset
cooked: <p>Given:<br>
<em>std::shared_ptr ptr</em><br>
that was created from a concrete type, e.g. Linear<br>
is there a way to create an AnyModule other than going through possible types,</p>
<p>e.g.<br>
<em>if(const auto&amp; a=std::dynamic_pointer_cast<a>torch::nn::Linear</a>(m))</em></p>
<ul>
<li>AnyModule …*<br>
<em>else if(…)</em>
</li>
</ul>
<p>Thanks</p>
Number of parameters of MultiheadAttention

username: Elidor
cooked: <p>Hi everyone,</p>
<p>doing some tests, I noticed that increasing or decreasing the number of heads of the multi head attention the total number of learnable parameters of my model does not change.<br>
Is this behavior correct? And if so, why?</p>
<p>Shouldn’t the number of heads affect the number of parameters the model can learn?</p>
<p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>EDIT: obv I use the pytorch implementation for the <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html" rel="noopener nofollow ugc">MultiheadAttention</a></p>

username: ptrblck
cooked: <p>If I understand the <a href="https://github.com/pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/torch/nn/modules/activation.py#L871-L872">code</a> correctly, it seems the <code>embed_dim</code> will be split by the number of heads, which would not increase the parameter size.</p>

username: Elidor
cooked: <p>Yes exactly, even if it leaves me a little perplexed …<br>
For example in transformers, changing the number of heads it seems to me that the total number of trainable parameters change</p>
Groupby aggregated topk in pytorch

username: MushroomHunting
cooked: <p>Hi there,</p>
<p>bit of an odd question but I’m wondering if this is possible to do with pytorch out of the box.<br>
Basically I’m trying to replicate an operation I can do quite easily using pandas groupby operations.</p>
<p>Let’s say I have a tensor that’s full of people’s ages in a particular class which is simply identified by an integer. So in the following example the class identifiers are in column 0: 99 and 55. Note they are of different sizes. The ages are in column 1.</p>
<pre><code class="lang-auto">ages_by_class = [[ 99,  24 ],
                 [ 99,  13 ],
                 [ 55,  33 ],  #&lt;--- ages not necessarily sorted in any order apriori
                 [ 55,  43 ],
                 [ 55,  36 ],
</code></pre>
<p>I’m trying to get the indexes or boolean mask or values corresponding to the topk ages <em>within each group</em>. So for the above, the boolean mask would look something like</p>
<pre><code class="lang-auto">mask_top2 := [[ 1 ],
              [ 1 ],
              [ 0 ],
              [ 1 ],
              [ 1 ],
</code></pre>
<p>In pandas the solution is simply to groupby() the sorted class id column, and then get the .head(k) but it’s not clear how I could translate this logic into pytorch. Any help would be greatly appreciated!</p>
<p>cheers</p>
Channel dimension mismatch while implementing Squeezenet

username: manix
cooked: <p>Error :</p>
<blockquote>
<p>RuntimeError: Given groups=1, weight of size [16, 128, 1, 1], expected input[32, 64, 126, 126] to have 128 channels, but got 64 channels instead</p>
</blockquote>
<p>The above images are the errors that i got while trying to run a SqueezeNet model , i have tried various means to solve but no idea where it has gone wrong , i have taken the implementation from</p>
<p><a href="https://github.com/pytorch/vision/blob/master/torchvision/models/squeezenet.py" rel="noopener nofollow ugc">Github link</a></p>
<p>Could anyone help me by telling where i went wrong .<br>
The images dimensions are in batch of 16  → [16,3,512,512]</p>
<p>The error seems to be coming with the second hidden layer somewhere …</p>

username: ptrblck
cooked: <p>The error is raised by a conv layer, which is expecting 128 input channels and returns 16 output channels.<br>
Did you modify the model in any way and if so, could you post the applied changes?</p>
<p>The <code>torchvision</code> implementation work fine for me:</p>
<pre><code class="lang-python">model = models.squeezenet1_0()
x = torch.randn(1, 3, 224, 224)
out = model(x)

model = models.squeezenet1_1()
x = torch.randn(1, 3, 224, 224)
out = model(x)
</code></pre>
PyTorch accuracy lower compared to tensorflow

username: Zafar
cooked: <p>I was going through the tensorflow keras tutorials, and tried repro their numbers in pytorch. It looks like the keras <code>.fit</code> gets better accuracy out of the box compared to the pytorch flow. I tried making sure that the optimizer parameters are the same, and even tried initializing the pytorch weights to the ones in keras – still same.</p>
<p>This is the tutorial I was going through: <a href="https://www.tensorflow.org/tutorials/keras/classification" class="inline-onebox" rel="noopener nofollow ugc">Basic classification: Classify images of clothing &nbsp;|&nbsp; TensorFlow Core</a></p>
<p>Here is the notebook gist that shows the training and inference results for the same number of epochs: <a href="https://gist.github.com/z-a-f/8f285e45052a75c4920d2ee47b698fd4" class="inline-onebox" rel="noopener nofollow ugc">TF_vs_PyTorch.ipynb · GitHub</a></p>
<p>Is there anything I can do with the pytorch flow to roughly match the keras numbers?</p>
Multiple weights of each layer with a learnable parameter

username: Naruto-Sasuke
cooked: <pre><code class="lang-python">net2 = copy.deepcopy(net1)

old_dict = collections.OrderedDict()
for k in net1.module.state_dict().items():
    key = k[0]
    value = k[1]
    old_dict[key] = value

# network contains `bn` layers
for k in net1.module.state_dict().items():
     key = k[0]
     value = k[1]
     value_old = old_dict[key]
     if key.find('num_batches_tracked') != -1:
            continue
     # update param
     old_dict[key] = alpha * value_old + (1 - alpha) * value
net2.module.load_state_dict(old_dict, strict=True)
</code></pre>
<p>In the original version, <code>alpha</code> is fixed and shared across all the layers, however, I want to set <code>alpha</code> as a learnable parameter for each layer for now.<br>
How to do it?</p>
Variable gradient not updating

username: ameertg
cooked: <p>Hi,</p>
<p>I’ve trained a CNN for regression on 3D images with no gradient issues. My goal is now to visualise the gradients of some input image to identify the most important pixels. The output of my network is a tensor of size 1 and so I use this output directly to do the backward pass.<br>
The issue is that the gradients that I get out are all zero no matter what I do. Can anyone help me understand why this is?</p>
<pre><code class="lang-auto">    image_tensor = torch.Tensor(image_tensor)  # convert numpy or list to tensor
    if cuda:
        image_tensor = image_tensor.cuda()
    X = Variable(image_tensor[None], requires_grad=True)  # add dimension to simulate batch
    output = model(X)

    # Backward pass.
    model.zero_grad()
    output.backward()
    relevance_map = X.grad.cpu().numpy()[0]

    return relevance_map
</code></pre>
<p>FYI, the model is a sequence of 3D conv blocks, relus and then a sequence of 3 linear layers + relus reducing the dimension down to 1.</p>
<p>Thanks a lot! <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

username: ptrblck
cooked: <p>I would recommend to remove the usage of <code>Variable</code>s, as they are deprecated since PyTorch <code>0.4</code>, but your code should generally work.<br>
Here is a small example showing that gradients are accumulated in the input using a simple model:</p>
<pre><code class="lang-python">x = torch.randn(1, 10, requires_grad=True)
model = nn.Linear(10, 10)
output = model(x)

# Backward pass.
model.zero_grad()
output.mean().backward()
relevance_map = x.grad.cpu().numpy()[0]
print(relevance_map)
</code></pre>
About torch.manual_seed() and cudnn.deterministic = True

username: 111429
cooked: <p>Hi,<br>
What’s the difference between just setting torch.manual_seed(seed),<br>
and just setting cudnn.deterministic = True,<br>
and setting both torch.manual_seed(seed) and cudnn.deterministic = True?<br>
If I want a reproducible result, what is the minimum setting?<br>
Also, <a href="https://github.com/yechengxi/deconvolution/blob/master/main_imagenet.py#L154" rel="noopener nofollow ugc">here</a> the author<br>
says “‘You have chosen to seed training. This will turn on the CUDNN deterministic setting, ’<br>
which can slow down your training considerably! You may see unexpected behavior when restarting            from checkpoints.’”<br>
How does that affect speed and create unexpected behavior?<br>
Thanks!</p>

username: ptrblck
cooked: <p>The <a href="https://pytorch.org/docs/stable/notes/randomness.html">reproducibility docs</a> explain the difference.<br>
Using <code>torch.manual_seed</code> will seed the psuedorandom number generator, while setting <code>cudnn.deterministic</code> will pick deterministic algorithms in cudnn and skip potentially faster but non-deterministic ones.</p>
LSTM for predicting sine wave

username: kevs
cooked: <p>Hi, i am trying to create a LSTM model for predicting a sine wave function. I want to eventually be able to implement this using only basic operations within the autograd system.</p>
<p>But for testing out things i am using the torch.nn.LSTM for now.</p>
<pre><code class="lang-auto">import torch
import matplotlib.pyplot as plt

# Parameters
INPUT_LEN = 400
SEQ_LEN = 20
INPUT_SIZE = 1
NUM_LAYERS = 1
HIDDEN_SIZE = 1
BATCH_SIZE = 40

# training data
X = (torch.arange(2 * INPUT_LEN, dtype=torch.get_default_dtype()) / INPUT_LEN).reshape(SEQ_LEN, BATCH_SIZE, INPUT_SIZE)
y = torch.sin(2 * np.pi * X).reshape(SEQ_LEN, BATCH_SIZE, HIDDEN_SIZE)

rnn = torch.nn.LSTM(INPUT_SIZE, HIDDEN_SIZE)

optim = torch.optim.Adam(rnn.parameters())

for epoch in range(20):
    # the first cell will neither have a previous output or 
    # a previous hiddens state, set them to zero
    h0 = torch.zeros((NUM_LAYERS, BATCH_SIZE, HIDDEN_SIZE))
    c0 = torch.zeros((NUM_LAYERS, BATCH_SIZE, HIDDEN_SIZE))

    out, (h_out, c_out) = rnn(X, (h0, c0))

    optim.zero_grad()

    # Mean squared error loss
    loss = torch.tensor(y).sub(out).pow(torch.tensor(2)).sum().div(torch.tensor(SEQ_LEN*BATCH_SIZE))
    loss.backward()
    
    optim.step()
    
N_STEPS = 20 # number of time steps to predict

X = (torch.arange(2 * INPUT_LEN, dtype=torch.get_default_dtype()) / INPUT_LEN)
y = torch.cos(2 * np.pi * X)
y_init = y[:N_STEPS]

# start with initial values, rest will be predicted
y_pred = y_init
for i in range(len(X) - N_STEPS):
    inp = y_pred[i:i + N_STEPS].reshape(N_STEPS, INPUT_SIZE, HIDDEN_SIZE)
    out, (h_out, c_out) = rnn.forward(inp)
    # h_out is the output for t = seq_len
    y_pred = torch.cat((y_pred, h_out.flatten()))

# detach results
X = X.detach().numpy()
y = y.detach().numpy()
y_pred = y_pred.detach().numpy()
y_init = y_init.detach().numpy()

# plot result
plt.plot(X[N_STEPS:], y_pred[N_STEPS:], label="prediction")
plt.plot(X, y, label="ground truth")
plt.plot(X[:N_STEPS], y_init, label="initial sequence", color="red")
plt.legend(loc="upper left")
plt.show()
</code></pre>
<p>This code does not seem to train the model at all. I suspect there might be something wrong with the defined loss function (MSE). The loss stays around 0.6.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38b5df302c14a205b63ac4960d9cfc3f4a56a1b0.png" alt="plot" data-base62-sha1="85GoBQdtEtWnLD87hPZu79GyUzC" width="567" height="420"></p>
<p>I want the prediction (blue line) to correctly predict the ground truth (orange line).</p>
Insert an element to tensor

username: Adriano_Santos
cooked: <p>Hi,</p>
<p>How could I do this?</p>
<p>values = [[1, 2, 3], [9, 8, 7]]</p>
<p>ex = torch.zeros([2, 5], dtype=torch.int32)</p>
<p>Before<br>
tensor([[0, 0, 0, 0, 0],<br>
[0, 0, 0, 0, 0]], dtype=torch.int32)</p>
<p>After<br>
tensor([[0, 0, 0, 0, 0, 1, 2, 3],<br>
[0, 0, 0, 0, 0, 9, 8, 7]], dtype=torch.int32)</p>

username: Dwight_Foster
cooked: <p>You can do something like this</p>
<pre><code class="lang-auto">x = torch.cat([ ex, torch.tensor(values)], dim=1)
</code></pre>
Sometimes loss term suddenly jumps and then back to normal? Is there any reasons why this happens?

username: klory
cooked: <p>Sometimes when I train certain models I see loss terms like this?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/664652b793d0346354e3b7be6436d61f3ab8267d.png" data-download-href="https://discuss.pytorch.org/uploads/default/664652b793d0346354e3b7be6436d61f3ab8267d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/664652b793d0346354e3b7be6436d61f3ab8267d_2_690x384.png" alt="image" data-base62-sha1="eALmQKDx5X40iNtCZS9JuSK2yN7" width="690" height="384" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/664652b793d0346354e3b7be6436d61f3ab8267d_2_690x384.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/664652b793d0346354e3b7be6436d61f3ab8267d_2_1035x576.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/6/664652b793d0346354e3b7be6436d61f3ab8267d.png 2x" data-dominant-color="FCFDFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1072×598 50 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This happens not quite offen, but it does not restricted to a certain network and dataset, is there any insight why and when this would happen and why the loss will back to normal after that batch?</p>
Intercept before optimizer.step()

username: HmmRfa
cooked: <p>Hey, I want to intercept before the before the optimizer.step() call where the weights are updated.<br>
My issues are at when trying to access the values. Escpecially the following:</p>
<ul>
<li>Access input/output weights of all layers.</li>
<li>Access loss of all layers</li>
<li>Access activations of all layers.</li>
<li>Set those values to new values.</li>
</ul>
<p>If found this: [<a href="https://discuss.pytorch.org/t/how-to-split-backward-process-wrt-each-layer-of-neural-network/7190" class="inline-onebox">How to split backward process wrt each layer of neural network?</a>] , but it says that the solution is just a hack. What would be a propper approach to handle this?</p>
Improve speed and code for ensemble update

username: Se_di
cooked: <p>Hello,<br>
Im training an ensemble of networks and wanted to know if there are some possible improvements in the code that i wrote in terms of speed following this scheme:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92dae43e5e363766266e71fd6a159992790a0c3e.png" alt="image" data-base62-sha1="kX8GoeYjZlO6Uez6DD0BogsjIVo" width="369" height="114"></p>
<p>so I have an ensemble of N(=10) networks with N target networks.<br>
calculating the loss is quite easy but then i have to do the backward pass… currently my schedule looks like this:</p>
<pre><code class="lang-auto">        # Compute critic losses and update critics 
        for critic, optim, target in zip(self.critics, self.optims, self.target_critics):
            Q = critic(states, actions).cpu()
            Q_loss = F.mse_loss(Q, Q_targets)
        
            # Update critic
            optim.zero_grad()
            Q_loss.backward()
            optim.step()
            # soft update of the targets
            self.soft_update(critic, target)
</code></pre>
<p>Especially concerning for me is the need of also 10 optimizers, is there an easier way to do it?<br>
Thanks a lot!</p>
Basic torch.cat question -- trying to append repeated tensor to larger tensor

username: wallyxie
cooked: <p>Hi,<br>
Newbie question.</p>
<p>Suppose I have</p>
<pre><code class="lang-auto">test_short = torch.rand([1, 1, 3])
test_long = torch.rand([5, 100, 3])
</code></pre>
<p>If I want to append <code>test_short</code> to the beginning of <code>test_long</code> such that I repeat the elements of <code>test_short</code> 5 times and end up with a third tensor of <code>torch.Size([5, 101, 3])</code>, how would I go about that in the most concise manner? I figure I could do</p>
<pre><code class="lang-auto">test_short_repeat = test_short.repeat([5, 1, 1])
test_combined = torch.cat([test_short_repeat, test_long], 1)
</code></pre>
<p>but was not sure if there was a convention that was more accepted for carrying something out like this.</p>
<p>Thank you very much for your help.</p>

username: ptrblck
cooked: <p>Your code looks correct to me and I’m not aware of a better way to write it.</p>
Java tensor concatenation

username: lev
cooked: <p>I have the following Java commands:</p>
<p>import org.pytorch.Tensor;</p>
<p>Tensor var1 = …(some definition);<br>
Tensor var2 = …(some definition);<br>
Tensor var3;</p>
<p>I want var3 to be the concatenation tensor of var1 and var2</p>
<p>What is the command for that and where can I find a list of possible commands?<br>
Thank you.</p>
Java tensor indexing

username: lev
cooked: <p>I am defining a java tensor:<br>
FloatBuffer buffer = Tensor.allocateFloatBuffer(10<em>3</em>80*80);<br>
Tensor input = Tensor.fromBlob(buffer, new long[]{10, 3, 80, 80});</p>
<p>How can I get an element of the tensor?<br>
input[0,0,0,0], input[0], input[0][0][0][0] all do not work. Any other suggestions? There is nothing in the documentation.</p>
Is the collate function executed by every DataLoader worker?

username: ataxias
cooked: <p>Is the collate function executed by every DataLoader worker, or only by the main process?<br>
So basically, does each worker only call <code>__getitem__</code> and then put fetched samples into a queue for the main process to use the collate function and make a batch, or each worker process calls the collate function, creates batches and puts entire batches into a queue?</p>
<p><a class="mention" href="/u/vincentqb">@vincentqb</a></p>

username: wayi
cooked: <p>cc <a class="mention" href="/u/vincentqb">@vincentqb</a> for dataloader questions</p>
Linking model and class labels?

username: FaultyBagnose
cooked: <p>I’m about to implement a simple model-label-wrapper that outputs a classification probability vector in the form of a dictionary with class labels as keys (rather than a ‘dictionary’ indexed by integers). Semantically, the label set is a part of the model, IMO, so they should be linked together at the lowest possible code level.</p>
<p>Is there a standard way of bundling a classification model with its label set?</p>

username: comely420x
cooked: <p>I use .json file to do it. I don’t know any other way of solving it sorry.</p>
Masked median filter/pool

username: Raghavendra_Singh
cooked: <p>I am trying to write a masked median filter/pool based on this code<br>
<a href="https://gist.github.com/rwightman/f2d3849281624be7c0f11c85c87c1598" rel="noopener nofollow ugc">PyTorch MedianPool (MedianFilter) (github.com)</a></p>
<p>mask is as simple as all nonzero() values in the input data.</p>
<p>Question is how to apply mask after <code>unfold</code> that is along the last dimension of<br>
<code>x.contiguous().view(x.size()[:4] + (-1,))</code><br>
so that median is taken over only non-masked values?</p>
<p>I can <code>gather</code>, <code>median</code> and <code>scatter</code> along the last dimension but that would imply that I loop over other dimensions?</p>
<p>Thanks!</p>