akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Adding time-step feature to input of sequence generator LSTM

username: boris.mtdv
cooked: <p>I’m training an LSTM using word embeddings as input and output, and wanted to try adding a feature to my input to represent the position in the sequence that the word has, with the intention of teaching the network to predict words that appear at the end of the sentence. Does anyone know if this is likely to improve the performance of the model? If so, how would one go about it? I was thinking about just adding a value to the input tensor to represent the psotion and scaling it between 0 and 1. Would this be a good approach?</p>
Silencing step calculations

username: Felipe_bivort_haiek
cooked: <p>Hello, is there anyway to slence pytorchs printing each backprop step to terminal? I am afraid it is cnsuming more resources than I would like by printing all backprops for resnet 50.</p>

username: ptrblck
cooked: <p>PyTorch does not automatically print during the backward call.<br>
Most likely you are using some <code>print</code> statements in your training loop, which print some information.</p>
<p>Are you using plain PyTorch or some higher level API?<br>
In the latter case, the wrapper might print something by default, which could probably be disabled.</p>
Appropriate model and image dataset for H/W profiling

username: jiseongg
cooked: <p>Hi, I’m trying to write a report about GPU/CPU performance analysis.</p>
<p>I’ve used CIFAR-10 dataset, and I’ve got some data from profiling tool (VTune Amplifier, Nvidia Nsight Compute), while executing ResNet18.</p>
<p>I’m looking for bigger images. (CIFAR-10 is only 32x32) I’ve heard about ImageNet, but I think it is too heavy for me. I’m a beginner in deep learning. Are there any popular image datasets used in analyzing computing performance? Also I wanna get recommendation about the models. Maybe I’ll use one or more CNN models for it.</p>
<p>The last question is, for performance analysis, should I use training code or just do inference?</p>

username: tom
cooked: <p><a href="https://github.com/fastai/imagenette/" rel="nofollow noopener">Imagenette</a> tries to fill the gap of “small dataset with imagenet-like images”.</p>
<p>You would have to decide what performance you want to analyse. If you plan to deploy a model into production, inference might be most important, otherwise training matters most.</p>
<p>Best regards</p>
<p>Thomas</p>

username: jiseongg
cooked: <p>Thank you very much. The size and the data amount are desirable for me.</p>
<p>I have one question more. I’ve checked Imagenette-320 and I found that the images have various size. But all have the form of (320, xxx) or (xxx, 320). As far as I know, if the network has fully connected layer, this different size may be issue. But I’m a newbie in the world of neural network model, so may I ask you for some reference of the way to transform this kind of various image files to tensor? Or the reference of the code using some famous cnn models dealing with this kind of image data?</p>
<p>Thank you</p>

username: tom
cooked: <p>Yeah, I can recommend the fast.ai courses for that.</p>
<p>Best regards</p>
<p>Thomas</p>
How to copy pretrained model to parallel model?

username: John1231983
cooked: <p>Hello all, I have trained my model with my dataset. At that time, my model is</p>
<pre><code class="lang-auto">class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1   = nn.Linear(16*5*5, 120)
        self.fc2   = nn.Linear(120, 84)
        self.fc3   = nn.Linear(84, 10)

    def forward(self, x):
        out = F.relu(self.conv1(x))
        out = F.max_pool2d(out, 2)
        out = F.relu(self.conv2(out))
        out = F.max_pool2d(out, 2)
        out = out.view(out.size(0), -1)
        out = F.relu(self.fc1(out))
        out = F.relu(self.fc2(out))
        out = self.fc3(out)
        return out
</code></pre>
<pre><code class="lang-auto">net = LeNet()
net.train()
</code></pre>
<p>The network has trained and save the model to <code>lenet.pth</code>. Now, I changed the network to support multiple GPU and I want to load the pre-trained model <code>lenet.pth</code>. Then I change it to</p>
<pre><code class="lang-auto">net = LeNet()
net = torch.nn.DataParallel(net, [0,1])
ckpt = torch.load('./lenet.pth', map_location=lambda storage, loc: storage)
net.load_state_dict(ckpt['checkpoints'])
</code></pre>
<p>However, it does not allow to load it. How to fix it? Thanks</p>

username: ptrblck
cooked: <p>Could you try to restore the <code>state_dict</code> before wrapping the model in <code>nn.DataParallel</code>?<br>
Also, what kind of error are you getting?</p>

username: John1231983
cooked: <aside class="quote no-group quote-modified" data-username="John1231983" data-post="1" data-topic="59528">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john1231983/48/4040_2.png" class="avatar"> John1231983:</div>
<blockquote>
<p>net.load_state_dict(ckpt[‘checkpoints’])</p>
</blockquote>
</aside>
<p>No. I load the model after nn.DataParallel, I guess we should use net.model.load_state_dict instead of net.load_state_dict. Am I right?</p>

username: ptrblck
cooked: <p>This would be one approach or load the model before wrapping it in <code>nn.DataParallel</code>.</p>
Apply user defined function to each row (or any dimension)

username: TinfoilHat0
cooked: <p>I’ve bunch of column vectors where I construct a matrix out of them by concatenating them side by side.<br>
That is, if my column vector is of dimension (nx1) and if I have m of them, I end up with a matrix of size (nxm).</p>
<p>On this final matrix, I’d like to apply the same operation to each row. The way I currently do it is simply via iterating over each row manually.</p>
<p>Does PyTorch have a nicer way of doing this?</p>

username: raghavendragaleppa
cooked: <p>It depends on the kind of function you want to apply. Give me an example of what function you want to apply.</p>

username: JuanFMontesinos
cooked: <p>Depending of the operation you may be able to use einsum or some other operator.</p>

username: TinfoilHat0
cooked: <p>I want to apply linear regression to each row.</p>

username: raghavendragaleppa
cooked: <p>Well if the output off your row 1 is needed for the output of your row 2 then you have to loop over them. Other wise go for direct Matrix multiplication</p>

username: JuanFMontesinos
cooked: <p>Then I’d say you can use <code>torch.</code> <code>solve</code><br>
<a href="https://pytorch.org/docs/stable/torch.html#torch.solve" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#torch.solve</a><br>
It works with batch, thus, in the worst case, you can reshape your matrix to create a batch and to pose the problem that way.<br>
Obviously it depends on the method you want to use, the best you can do is to code it your self if you have a custom one based on pytorch functions.</p>
Update layer parameters manually during forward pass

username: buddhiw
cooked: <p>Hi all, I would like to know whether it is possible to update the parameters of a layer manually during forward pass of a network. I need to calculate and update the parameters of a layer based on a previous layer output.<br>
Thank you.</p>

username: tom
cooked: <p>You can do that (before you use the layer) if you want and then use something like</p>
<pre><code class="lang-python">with torch.no_grad():
   # you might also do the computation of new_value  here
   self.nex_layer.my_param.copy_(new_value)
</code></pre>
<p>or so.</p>
<p>Best regards</p>
<p>Thomas</p>

username: buddhiw
cooked: <p><a class="mention" href="/u/tom">@tom</a> thanks for the reply. Are you suggesting to do this within the training loop? When I use <code>torch.no_grad()</code> wrapper, how is it going to affect the backpropagation of the whole network?</p>

username: tom
cooked: <p>No, sorry, I edited the answer to clarify. So the gradient would be w.r.t. the new value of <code>my_param</code> and you could compute the <code>new_value</code> in the <code>no_grad</code> block.<br>
The remaining computation - the bit that needs to be backproped - would not be in <code>no_grad</code>, of course.</p>
<p>Best regards</p>
<p>Thomas</p>
Using DDP for training a pytorch model on distributed GPU system

username: Cooper_Lee
cooked: <p>We use DDP  to train our model based on pytorch on a distributed GPU system. I discover that some demos for distributed GPU system based on pytorch sync initial weights for each node and compute the average grad of each node to update the patamters of model. I wonder whether we need to use these extra operation, when we use DDP to train a model based on pytorch on distributed GPU system.</p>
How to select the larger GPU to train the model？

username: beitian
cooked: <p>I have two GPUs，one is of 11G, the other is of 23G. In previous test, my code always runs in the GPU of 11G, it is out of memory. What can I do to select the second GPU?</p>

username: ptrblck
cooked: <p>You could specify the device when pushing the model parameters and data to the device via <code>.to('cuda:1')</code> or use <code>CUDA_VISIBLE_DEVICES=1 python script.py args</code> to execute your script. In the latter approach GPU1 will be mapped to <code>'cuda:0'</code> in your script.</p>
How extract all hidden state outputs instead of last hidden state output in LSTM?

username: Arun_Sagar
cooked: <p>I am new to pytorch . I am trying to grasp how the input and output shapes work in encoder - decoder model. while implementing attention model , i am stuck here where i need to get all hidden state outputs …</p>
<p><strong>My code for decoder</strong><br>
class DecoderLSTM(nn.Module):</p>
<pre><code>def __init__(self,embed_size, vocab_size, hidden_size):
    super(DecoderLSTM, self).__init__()
    self.embed_size = embed_size
    self.vocab_size = vocab_size
    self.word_embeddings = nn.Embedding(vocab_size, embed_size)
    self.decoderlstm = nn.LSTM(input_size = embed_size,
                               hidden_size = hidden_size,
                               num_layers = 1,
                               batch_first = True,
                               bias =  True)
    self.fc_layer = nn.Linear(hidden_size, vocab_size)
    
    self.attention = BahdanauAttention(hidden_size)
    
def init_hidden(self, enc_hidden):
    enc_h_t, enc_c_t = enc_hidden
    h_t = torch.tensor(enc_h_t).requires_grad_()
    c_t = torch.tensor(enc_c_t).requires_grad_()
    return (h_t,c_t)

def forward(self, enc_out, enc_hidden, captions):
    
    (h_t,c_t) = self.init_hidden(enc_hidden)
    
    embeds = self.word_embeddings(captions)
    inputs = torch.tensor(embeds)
    print('Embed Tensor Shape:',inputs.shape)
    dec_out, (dec_h_t, dec_c_t) = self.decoderlstm(inputs, (h_t.detach(), c_t.detach()))
    print('dec_h_t :',dec_h_t.shape)
    print('dec_c_t :',dec_c_t.shape)
    attention_out = self.attention(enc_out, dec_out)
    outputs = self.fc_layer(attention_out)
    return outputs
</code></pre>
<p>instead of getting all hidden state ouputs, i am getting only last state output (num_layer, batch_size, hidden_state )</p>
<p>How to get all hidden state outputs ?</p>

username: tom
cooked: <aside class="quote no-group" data-username="Arun_Sagar" data-post="1" data-topic="59390">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arun_sagar/48/17064_2.png" class="avatar"> Arun_Sagar:</div>
<blockquote>
<p>How to get all hidden state outputs ?</p>
</blockquote>
</aside>
<p>You don’t get the cell state <code>(h_t, c_t)</code> from the <code>LSTM</code> for intermediates.<br>
Thus you would want to loop over t yourself, using either <code>LSTM</code> with sequence length of 1 or <code>LSTMCell</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: sharjeel
cooked: <p>your can try to run all of your hidden layers in your forward function. here’s the sudo code:<br>
x = F.relu(self.conv1(x))<br>
x = F.relu(self.conv2(x))<br>
x = F.relu(self.conv2(x))</p>
<p>and then apply a for loop on the output of every layer and iterate it up to the number of layers you have and keep displaying all of he layers.</p>
Save model state in memory (cpu and gpu)

username: jojono
cooked: <p>I would like to save and load the weights of my model several time for each epochs during training. At the moment I save and load the model from a file but it is slow.<br>
I know that it is possible to save it inside a IOBuffer object but it is not possible on GPU. Is there a way to save and load a model from a variable on both CPU and GPU quickly ?</p>
<p>Thank you</p>

username: donCalculator
cooked: <p>Just assign model.state_dict() to a variable, and then call model.load_state_dict() on the variable to load the weights back. See the example below.</p>
<pre><code class="lang-auto">from torchvision.models import resnet
states =[]
model = resnet.resnet18()

states.append(model.state_dict())

model.load_state_dict(states[0])
</code></pre>
<p>Edit: If you want to store the weight on the GPU then:</p>
<pre><code class="lang-auto">model = resnet.resnet18().cuda()
model2 = model
...
model = model2
del model2
</code></pre>

username: jojono
cooked: <p>Thank you.<br>
But it does not work, the weights of the model do not change after the loading for both solutions :</p>
<pre><code class="lang-auto">model = torch.nn.Linear(4,4)
save = model
#training here
print(net.weight.data)
model = save
print(net.weight.data)
#Same result for both print
</code></pre>
<pre><code class="lang-auto">model = torch.nn.Linear(4,4)
save = model.state_dict()
#training here
print(net.weight.data)
model.load_state_dict(save)
print(net.weight.data)
#Same result for both print
</code></pre>
Concatenation while using Data Loader

username: anandrox7
cooked: <p>Hello everyone,</p>
<p>I had a small question with respect to data augmentation and dataloader. In order to use use data augmentation in addition to the unaltered set of original images for training, I am using ConcatDataset in Data Loader, which consists of two data_transform operations on same dataset. (data_transforms_A contains many augmentation techniques while data_transforms contains only scaling to leave original image intact.) Is this the right approach? Also in each batch, lets say of size 64,what is the distribution of amount of images fetched from each dataset?<br>
Please find below the code i am currently using,</p>
<p>train_loader = torch.utils.data.DataLoader(<br>
torch.utils.data.ConcatDataset([datasets.ImageFolder(args.data + ‘/train_images’,<br>
transform=data_transforms_A),<br>
datasets.ImageFolder(args.data + ‘/train_images’,<br>
transform=data_transforms)]), batch_size=args.batch_size, shuffle=True, num_workers=1)</p>
<p>Thank you.</p>

username: SimonW
cooked: <p>yes, this seems correct to me. the entire training set should contain half of unaugmented images and half of augmented images.</p>

username: anandrox7
cooked: <p>Hi Simon, thanks for the reply. Just to confirm, can we say the same about the way images are sampled in a particular batch (around half from each dataset)<br>
Thanks.</p>

username: SimonW
cooked: <p>Yes. What happens is there will be a random permutation using all data of these two datasets, and each batch will sequentially fetch segments of the permuted data.</p>

username: rzhang63
cooked: <p>So does it mean in the usual case where we only define one data loader for the training images (with some augmentations such as random crop or horizontal flip), then in the training process only transformed images are fed into the optimization loop and the original images are never used?</p>

username: SimonW
cooked: <p>yes, this is typical in deep learning. however, most common transforms have probability of being an identical transform so the original data has a probability of being used.</p>
Torch.nn.functional's softmax does not sum to 1

username: RylanSchaeffer
cooked: <p>I started receiving negative KL divergences between a target Dirichlet distribution and my model’s output Dirichlet distribution. Someone online suggested that this might be indicative that the parameters of the Dirichlet distribution don’t sum to 1. I thought this was ridiculous since the output of the model is passed through</p>
<p><code>output = F.softmax(self.weights(x), dim=1)</code></p>
<p>But after looking into it more closely, I found that  <code>torch.all(torch.sum(output, dim=1) == 1.)</code> returns False! Looking at the problematic row, I see that it is <code>tensor([0.0085, 0.9052, 0.0863], grad_fn=&lt;SelectBackward&gt;)</code>. But <code>torch.sum(output[5]) == 1.</code> produces <code>tensor(False)</code>.</p>
<p>What am I misusing about softmax such that output probabilities do not sum to 1?</p>

username: RylanSchaeffer
cooked: <p>I’m using version ‘1.2.0+cpu’</p>

username: RylanSchaeffer
cooked: <p>Full model code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F



def assert_no_nan_no_inf(x):
    assert not torch.isnan(x).any()
    assert not torch.isinf(x).any()


class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.weights = nn.Linear(
            in_features=2,
            out_features=3)

    def forward(self, x):
        output = F.softmax(self.weights(x), dim=1)
        assert torch.all(torch.sum(output, dim=1) == 1.)
        assert_no_nan_no_inf(x)
        return output

</code></pre>

username: KFrank
cooked: <p>Hi Rylan!</p>
<aside class="quote no-group quote-modified" data-username="RylanSchaeffer" data-post="1" data-topic="59516" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rylanschaeffer/48/17130_2.png" class="avatar"> RylanSchaeffer:</div>
<blockquote>
<p><code>output = F.softmax(self.weights(x), dim=1)</code></p>
<p>But <code>torch.sum(output[5]) == 1.</code> produces <code>tensor(False)</code>.</p>
<p>What am I misusing about softmax such that output probabilities do not sum to 1?</p>
</blockquote>
</aside>
<p>You are testing for <em>exact equality</em> with a floating-point result.</p>
<p>Try looking at  <code>torch.sum(output[5]) - 1.0</code> and see if you<br>
get numbers that are small, that is, on the order of floating-point<br>
round-off error (about 10^-7, relative).</p>
<p>Best.</p>
<p>K. Frank</p>
Memory Error while trying to save resnet features to a dictionary

username: bat34
cooked: <p>I’m fairly new to PyTorch and I’m trying to extract the features of my training data using a ResNet101 and save them to a dictionary. I will then intend to use this dictionary as a lookup table for another model.</p>
<p>However, I keep running into memory problems after only processing 8 images on a 12GB GPU machine. The for loop is:</p>
<pre><code class="lang-auto">splits = ['train2014', 'val2014', 'test2015']
for split in splits:
    RES_FILE = os.path.join(D_ROOT, 'BaselineTraining', split, 'baseline_{}_cnn_features.pth'.format(split))
    IMAGE_DIR = os.path.join(D_ROOT, 'Images', 'mscoco', split)
    images = os.listdir(IMAGE_DIR)
    images_path = [os.path.join(IMAGE_DIR, file) for file in images]
    res = torch.load(RES_FILE) if os.path.exists(RES_FILE) else {}
    for i in progressbar.progressbar(range(len(images))):
        #Checkpoint every 2,500 images
        if i % 2500 == 0 and i &gt; 0:
            torch.save(res, RES_FILE)
        image_name, image = images[i], cv2.imread(images_path[i])
        if image_name in res:
            continue
        image = process_image(image)
        image = np.expand_dims(image, axis=0)
        image = torch.from_numpy(image)
        _, H, W, C = list(image.size())
        image = image.view(1, C, H, W)
        image = image.float()
        image = image.cuda()
        out = resnet101.forward(image)
       #res[image_name] = out.cpu()
    print('Successfully processed {}.'.format(split))
print('Successfully processed all images.')
</code></pre>
<p>Also, I have noticed that the problem only persists when I uncomment the commented line. Without it, extraction goes smoothly. What is happening here?</p>

username: albanD
cooked: <p>Hi,</p>
<p>What happens is that you seem to run this code with the autograd enabled.<br>
Since the parameters in the resnet require gradients, your <code>out</code> does as well (and so contains all the history to compute these gradients if needed). When you save <code>out.cpu()</code> in your dictionary, you still hold onto this history. So the memory usage increases.<br>
The best thing to do here I think is to decorate your function with <code>@torch.no_grad()</code> or run the model within the <code>with torch.no_grad():</code> context manager. to disable autograd.</p>
<p>PS: you should not call the <code>.forward()</code> method of modules directly, but call them with the input: <code>out = resnet101(image)</code> in your case. Otherwise features like hooks won’t work.</p>

username: bat34
cooked: <p>Thanks! It worked like a charm. Memory usage is now more or less constant <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Getting assertion error while converting nd array to tensor

username: Prak_kas
cooked: <p>I have made train and validation splits of data using sklearn splits. The results of sklearn splits are of nd array type , i am converting them to tensor before building data loader , but  I am getting an assertion error</p>
<pre><code>from torch.utils.data import TensorDataset
from torch.utils.data import DataLoader


x_tr = torch.tensor(x_tr, dtype=torch.long)
y_tr = torch.tensor(y_tr, dtype=torch.float32)
Train = TensorDataset(x_tr, y_tr)
Trainloader = DataLoader(Train, batch_size=128)

x_valid2 = torch.tensor(x_valid2, dtype=torch.long)
y_valid2 = torch.tensor(y_valid2, dtype=torch.float32)
valid2 = TensorDataset(x_valid2, y_valid2)
validloader2 = DataLoader(valid2, batch_size=128)
</code></pre>
<p>Error is as follows:<br>
AssertionError                            Traceback (most recent call last)<br>
 in ()<br>
32 x_tr = torch.tensor(x_tr, dtype=torch.long)<br>
33 y_tr = torch.tensor(y_tr, dtype=torch.float32)<br>
—&gt; 34 Train = TensorDataset(x_tr, y_tr)<br>
35 Trainloader = DataLoader(Train, batch_size=128)<br>
36</p>
<pre><code>/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py in __init__(self, *tensors)
    156 
    157     def __init__(self, *tensors):
--&gt; 158         assert all(tensors[0].size(0) == tensor.size(0) for tensor in tensors)
    159         self.tensors = tensors
    160 

AssertionError: 
</code></pre>
<p>I also tried to convert the the nd arrays to tensors using torch.numpy(), to mitigate the issue, but still this error at  Tensordataset before data loader persists.<br>
Any help is appreciated.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You need to use <code>torch.from_numpy()</code> to have proper conversion behavior.</p>
<p>The error you see is because the TensorDataset expects all inputs to contain the same number of elements. In particular, here, I expect <code>x_valid2.size(0) != y_valid2.size(0)</code>.</p>
Channels in MFCC, and how to use them

username: Rohan_Kumar
cooked: <p>what are channels in mfcc, because when i use transform.mfcc i get the output as [2, n_mfcc, time], my question is what are the channels, and are the number of channels consistent in similar type of audio (basically in a dataset), how do use channel in models concatenate mfcc end-to-end?</p>

username: vincentqb
cooked: <p>The conventions we use for dimensions are given in the <a href="https://github.com/pytorch/audio/blob/master/README.md" rel="nofollow noopener">README</a>. In particular, a waveform is (channel, time), and <code>MFCC</code> : (channel, time) -&gt; (channel, mfcc, time), and so MFCC is applied per channel. Your original waveform must therefore have had 2 channels.</p>
<p>In the datasets we provide, the number of channels are the same across all waveforms.</p>
<p>Since the output of MFCC is just a tensor, you can use torch.cat to concatenate two MFCCs along a given axis.</p>
<p>Is that what you were asking?</p>

username: Rohan_Kumar
cooked: <p>It’s pretty much what I asked, thank you. I still want to know what channels are actually. And if you use log2 with mfcc, which I think is used, how do you handle the ban values, currently I have replaced them with 0s.</p>
Loss goes to Zero in first epoch!

username: Rafael_R
cooked: <p>Hi,</p>
<p>I am getting a weird case in which my loss just vanishes within an epoch:</p>
<pre><code class="lang-auto">
Train Epoch: 1 [0/33005 (0%)] Loss: 84108.492188

Train Epoch: 1 [32000/33005 (97%)] Loss: 0.000000

Epoch: 2 LR: [0.001]

Starting Training for Epoch 2

Train Epoch: 2 [0/33005 (0%)] Loss: 0.000000

Train Epoch: 2 [32000/33005 (97%)] Loss: 0.000000

Epoch: 3 LR: [0.001]

Starting Training for Epoch 3

Train Epoch: 3 [0/33005 (0%)] Loss: 0.000000
</code></pre>
<p>I am at loss for what is the issue, I tried different random initializations as well, same thing happens.</p>
<p>Any common tips on debugging the model?</p>

username: distant1219
cooked: <p>Can you print your output to check if there are some unusual things or paste it to discuss？</p>

username: raghavendragaleppa
cooked: <p>Can you post model architecture.  And also the training loop</p>
Core Dumped: Modify Parameters Manually

username: MarkD
cooked: <p>For every layer of ResNet, I would like to assign the average value of convs to each conv, every time after they’re updated by <code>backward</code>.</p>
<pre><code class="lang-python">class ResNet(nn.Module):
    def __init__(self, block, num_blocks, num_classes=10):...
    def _make_layer(self, block, planes, num_blocks, stride):...
    def forward(self, x):...

    def setAvg(self):
        for layer in [self.layer1,self.layer2,self.layer3,self.layer4]:
            weight_conv_average = torch.mean(
                torch.stack([torch.Tensor(p) for n, p in layer.named_parameters() if 'conv' in n and (p.size()[0]==p.size()[1])]), 0)
            for n, p in layer.named_parameters():
                if 'conv' in n and (p.size()[0]==p.size()[1]):
                    #print(n,'      ', p.size())
                    p.data.copy_(weight_conv_average)
</code></pre>
<p>And in the main part of training, I use</p>
<pre><code class="lang-python">rawnet = ResNet18()
net = rawnet.to(device)
if device == 'cuda':
    net = torch.nn.DataParallel(net)
    cudnn.benchmark = True
...
def train(epoch):
    ...
    loss.backward()
    with torch.no_grad():
        rawnet.setAvg()
    optimizer.step()
</code></pre>
<p>However this will cause <code>Segmentation fault (core dumped)</code> with Linux cuda env.</p>
<p>I’ve tried to invoke <code>setAvg</code> in <code>forward</code> but also failed.</p>
<p>And the reason why I distinguish <code>net</code> and <code>rawnet</code> is that <code>net</code> is <code>DataParallel</code> object, which has no attribute <code>'setAvg'</code>.</p>
<p>Big Thanks!</p>

username: MarkD
cooked: <p>I’ve figured it out:</p>
<p>1.use <code>net.module.setAvg</code> to deal with attributes for DataParalleled net</p>
<p>2.no need to call <code>torch.Tensor(p)</code> for Parameter <code>p</code>, just <code>p</code> is ok</p>
OnnxToCaffe2 - Incompatible ConvTranspose operator

username: paolor
cooked: <p>Hello, noob here - apologies in advance if this is a stupid question.</p>
<p>I’m attempting to load an ONNX model in Caffe2 using: caffe2::onnx::Caffe2Backend::Prepare(const std::string &amp; onnx_model_str, const std::string &amp; device, const std::vector&lt;caffe2::onnx::Caffe2Ops,std::allocator<a>caffe2::onnx::Caffe2Ops</a>&gt; &amp; extras)</p>
<p>I’m getting the following error:<br>
“Don’t know how to map unexpected argument ‘group’ (from operator ConvTranspose)”.</p>
<p>If I load the same model as Caffe2 prototext (by converting offline to init/predict via c2.onnx_graph_to_caffe2_net), I don’t have any issues.</p>
<p>Investigating the error, it seems that Caffe2’s ConvTranspose expects the following args: legacy_pad, kernels, strides, pads, adjs, order, shared_buffer, no_bias, whereas ONNX’s ConvTranspose assumes the following: auto_pad, dilations, group, kernel_shape, output_padding, output_shape, pads, strides.</p>
<p>Looks like the ‘group’ argument was added on April 4 2019 <strong>to the Caffe2 operator</strong> :<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="15" height="15">
      <a href="https://github.com/pytorch/pytorch/commits/master/caffe2/operators/conv_transpose_op_impl.h" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/pytorch/pytorch/commits/master/caffe2/operators/conv_transpose_op_impl.h" target="_blank" rel="nofollow noopener">pytorch/pytorch</a></h3>

<p>Tensors and Dynamic neural networks in Python with strong GPU acceleration - pytorch/pytorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>But the <strong>Caffe2 ONNX backend schema</strong> wasn’t changed and it doesn’t seem to have kept up with ONNX itself.</p>
<p>Questions:</p>
<ol>
<li>Is my interpretation correct?</li>
<li>If yes, is OnnxNodeToCaffe2Ops (caffe2\onnx\backend.cc) maintained/supported? Can I rely on loading ONNX models directly in Caffe2?</li>
</ol>
<p>Thanks so much!</p>
How to create a network with (batch x X x 6) output size

username: donCalculator
cooked: <p>I’m want to create a network which outputs a tensor shaped like this:<br>
batch x X x 6: where X is a number which is between 1 and 50</p>
<p>Sort of like a network which predicts the bounding boxes for a give shape in an image. But Im not sure which layer should be the last, because for example a nn.Linear() have a fixed output size.</p>
<p>Thank you!<br>
DC</p>

username: distant1219
cooked: <p>Assume that a conv layer’s output is b<em>c</em>h<em>w, you can set the last channel is 6, and using global pooling by grouped to change output from h</em>w to h<em>1 or w</em>1, then you do some dimensionality reduction. you will get the b<em>X</em>6©.</p>

username: donCalculator
cooked: <p>Thank you for your reply, but I need to accommodate for the pooling layer needs to have an output size of b x c x (1,1) but it could be b x c x (1,5) or b x c x (1,10)</p>
<p>Or you suggest I set the output shape dynamically in each forward pass?</p>

username: distant1219
cooked: <p>No，i just suggest that you can get the shape what you want in the last layer.</p>

username: donCalculator
cooked: <p>But this is the issue, sometimes I need 4 * 6 detection for an image or sometime 8 * 6. Is this possible?</p>
ValueError: too many values to unpack (expected 2) from torch.utils.data.DataLoader output

username: shivsd
cooked: <p>I’m using VOCSegmentation() from here  <a href="https://pytorch.org/docs/stable/_modules/torchvision/datasets/voc.html" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torchvision/datasets/voc.html</a> to get the object and pass this instance into torch.utils.data.DataLoader. After loading i get following error while enumerating the " <strong>self.train_loader</strong> ". I’m loading  VOC2012 from pascal_voc dataset. Please can you correct me if doing anything wrong , Here is the code snippet. I appreciate any help on this.</p>
<pre><code class="lang-auto">  def __init__()
       train_dataset = get_segmentation_dataset(args.dataset, split=args.train_split, mode='train', **data_kwargs) # this get_segementation_dataset() will call VOCSegmentation()
       self.train_loader = data.DataLoader(dataset=train_dataset,
                                           batch_size=args.batch_size,
                                           shuffle=True,
                                           drop_last=True)

  def train(self):
       cur_iters = 0
       start_time = time.time()
       for epoch in range(self.args.start_epoch, self.args.epochs):
           self.model.train()

           **for i, (images, targets) in enumerate(self.train_loader):**

               cur_lr = self.lr_scheduler(cur_iters)
               for param_group in self.optimizer.param_groups:
                   param_group['lr'] = cur_lr

               images = images.to(self.args.device)
               targets = targets.to(self.args.device)

               outputs = self.model(images)
</code></pre>

username: albanD
cooked: <p>When you write <code>for i, (images, targets) in enumerate(self.train_loader)</code>, you expect the train_loader to return 2 elements at every iteration. It looks like it returns more in your case.<br>
You can for <code>for i, loader_data in enumerate(self.train_loader)</code> to get the loader datas and inspect them to see why there isn’t the number you expect.</p>
LSTM Input Shape Query

username: Abheesht_Sharma1
cooked: <p>I am new to LSTM and PyTorch’s implementation of LSTM using <code>torch.nn.LSTM()</code> has confused me further.<br>
I am implementing an LSTM model for predicting the speeds of different frames (<a href="https://arxiv.org/pdf/1801.06734.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1801.06734.pdf</a>).</p>
<p>The paper does not mention much about the LSTM part of the model. All it says is that I have to consider the “speeds of 10 previous timestamps”. Also, in the image of the architecture, they have mentioned “LSTM 128”.</p>
<p>What should my <strong>input_size</strong>, <strong>hidden_size</strong> and <strong>num_layers</strong> be? What is “128” here?</p>
<p>Since I have essentially only one feature, i.e., speed, I am guessing the <strong>input_size</strong> to be 1. Is this correct?</p>
<p>I can have the <strong>seq_len</strong> to be 10 (for 10 previous timestamps).</p>
<p>Is this correct or am I missing something here?</p>
<p>If that is correct, what will the <strong>hidden_size</strong>, <strong>batch_size</strong> and <strong>num_layers</strong> be?</p>
Demo for GPU distributed system with Pytorch

username: Cooper_Lee
cooked: <p>We are going to train our model based on pytorch on a GPU distributed system. There are several nodes with 4 v100 GPUs for each node on the GPU distributed system.  The GPU distributed system takes slurm as resource scheduling system. I wonder whether there is a example program about pytorch on slurm GPU distributed system. I will appreciate it if any give me some advice.</p>
Architecture visualization

username: Ivanita
cooked: <p>May I ask if there is any way to visualize the architecture of the model? I know that net.state_dict()  is able to call out all the weighting of each neuron. But is there a way to visualize? Thanks.</p>

username: tom
cooked: <p>Printing a model will give you a “visualization” of how the modules and submodules are composed. Given the flexibility of the computation (which might reorder modules etc.),<br>
this isn’t authoritative in any way, but for a well designed model (and yours would be, would it?), it goes a long way.<br>
There also is considerably lower level way of visualizing the computational graph recorded by autograd, see e.g. (but a bit old) <a href="https://github.com/szagoruyko/pytorchviz/blob/master/torchviz/dot.py" rel="nofollow noopener">https://github.com/szagoruyko/pytorchviz/blob/master/torchviz/dot.py</a> (it’s a whole package with instructions how to use etc, but the linked file has the meat, really).</p>
<p>Best regards</p>
<p>Thomas</p>
Get Batch statistics over multiple GPUs

username: nbansal90
cooked: <p>Hey Folks,</p>
<p>I was looking to implement class imbalanced loss for a classification model, where the cross entropy loss would be weighted with class frequency.(Inverse frequency). Now, Since I want to implement this at batch level, I need to know the frequency of each class in all the batches. But at a given time, i know this information one a given gpu only. Is there a way to get this information, during runtime.</p>

username: JuanFMontesinos
cooked: <p>You can use hooks. They are called at the time of computing backward or forward. If you use multi-gpu they will be called once for each gpu.</p>
RNNs: Understanding differences of feeding sequence as a whole VS step-by-step

username: da-bu
cooked: <p>Hello,</p>
<p>I’m a bit lost with understanding what PyTorch does internally in these two variations of passing a sequence through an RNN model:</p>
<p>Version 1:<br>
<code>out, hidden = model(sequence)</code></p>
<p>Version 2:</p>
<pre><code class="lang-auto">for element in sequence:
    out, hidden = model(element, hidden)
</code></pre>
<p>In my understanding, version 1 and 2 should be the same - I’m just manually unrolling it over time in version 2. Version 2 also allows us to implement stuff like modifying the hidden state before passing it to the next timestep (which is what I would like to do for my use case), and also to train it without teacher forcing (by passing in the last “out” instead of “element”). However, I noticed that version 1 trains much faster (about 8x on my machine).</p>
<p>Question 1: Is version 1 faster simply because e.g. it has less forward-calls, or does PyTorch internally somehow parallelise something across timesteps (which I don’t see how it could do that given that the RNN’s computations depend on the output of the previous timestep)?</p>
<p>Question 2: More generally, is version 2 indeed the way to go if I need to train without teacher forcing or if I need to modify “hidden” between timesteps?</p>
<p>Thank you for your help!</p>

username: JuanFMontesinos
cooked: <p>If you look at source code version 1 internally runs version 2.<br>
There is a Base_RNN class or something like that. I cannot find it right now but at least it used to be like that. Anyway according to the theory it has to work like that. It cannot be paralelized .</p>

username: da-bu
cooked: <p>Thank you for your reply! What do you think causes the speed difference if V1 runs V2 internally?</p>

username: tom
cooked: <p>So the stock LSTM will use CuDNN which <a href="https://devblogs.nvidia.com/optimizing-recurrent-neural-networks-cudnn-5/" rel="nofollow noopener">has been heavily optimized</a> but is inflexible.<br>
But the JIT can make custom RNNs almost as fast: Some of this is discussed in <a href="https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/" rel="nofollow noopener">the LSTM with JIT</a> blogpost and we <a href="https://lernapparat.de/more-jit-optimizations/" rel="nofollow noopener">also have optimizations for the backward</a>.<br>
You can look at <a href="https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py" rel="nofollow noopener">the benchmark code</a> to get ideas how to use them.</p>
<p>Best regards</p>
<p>Thomas</p>

username: JuanFMontesinos
cooked: <p>As I said it used to be like that, at least last time I checked source code but it seems it has been improved <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Thanks <a class="mention" href="/u/tom">@tom</a> Awesome info as always</p>
[Bug] Unable to install pytorch 1.3.0 via pip on Windows 10 due to malformed URL on mirror

username: holgerschmeisky
cooked: <p>First of all thank you very much for the great work <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>installing pytorch 1.3.0 via pip in Windows 10 fails with</p>
<blockquote>
<p>ERROR: Could not find a version that satisfies the requirement torch&gt;=1.1.0 (from flair==0.4.4-&gt;-r requirements.txt (line 30)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)<br>
ERROR: No matching distribution found for torch&gt;=1.1.0 (from flair==0.4.4-&gt;-r requirements.txt (line 30))</p>
</blockquote>
<p>I fixed it by downloading and installing the wheel manually from <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a></p>
<p>The reason pip can not find it is in the URL:</p>
<p><a href="https://download.pytorch.org/whl/cpu/torch-1.3.0%2Bcpu-cp36-cp36m-win_amd64.whl" class="onebox" target="_blank" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cpu/torch-1.3.0%2Bcpu-cp36-cp36m-win_amd64.whl</a></p>
<p>The %2B somehow snuck in there and breaks pip’s lookup mechanism. It only occurs for some versions, not all of them.</p>
Unknown type name '__torch__.Module'

username: pascal.soveaux
cooked: <p>Hello,</p>
<p>The following code works with 1.2.0 but not with 1.3.0</p>
<pre><code class="lang-auto">    torch::serialize::OutputArchive archive;
    {
        torch::serialize::OutputArchive slot;
        module-&gt;save(slot);
        archive.write(KEY_MODULE, slot);
    }
    archive.save_to(filename);
</code></pre>
<pre><code class="lang-auto">    torch::serialize::InputArchive archive;
    archive.load_from(filename, tensors::device()); // tensors::device() returns the correct device
</code></pre>
<pre><code class="lang-auto">Unknown type name '__torch__.Module':
at code/__torch__.py:47:60
import __torch__.___torch_mangle_5
import __torch__.___torch_mangle_6
import __torch__.___torch_mangle_7
import __torch__.___torch_mangle_8
import __torch__.___torch_mangle_9
import __torch__.___torch_mangle_10
class Module(Module):
  __parameters__ = ["a6d20b65-2d16-4fbd-bce3-6886baed0c39", "2779f61f-c777-452e-a375-64c2c247be72", ]
  __annotations__ = []
  __annotations__["bc95c774-407e-4b95-91b7-9692585d89d0"] = __torch__.Module
                                                            ~~~~~~~~~~~~~~~~ &lt;--- HERE
  __annotations__["a6d20b65-2d16-4fbd-bce3-6886baed0c39"] = Tensor
  __annotations__["2779f61f-c777-452e-a375-64c2c247be72"] = Tensor
Compiled from code 
</code></pre>
<p>Is there something obvious that I am missing ?</p>
<p>Thank you,</p>
<p>Pascal</p>

username: albanD
cooked: <p>What is <code>__torch__.py</code> supposed to be? This is some of your code right? It looks like the <code>Module</code> class there got removed.</p>

username: pascal.soveaux
cooked: <p>Hello Alban,</p>
<p>Thank you for your help.<br>
My code is plain C++, I am not using python at all so I have no idea of what <strong>torch</strong>.py is supposed to be.<br>
I guess this is related to jit.</p>
<p>Pascal</p>

username: albanD
cooked: <p>Where the sample of python code that you posted above come from?</p>

username: pascal.soveaux
cooked: <p>This is the message for the exception thrown by load_from</p>
<pre><code class="lang-auto">archive.load_from(filename); // &lt;= throws
</code></pre>

username: Michael_Suo
cooked: <p>Can you try creating a compilation unit that all the <code>OutputArchive</code>s share? Something like:</p>
<pre><code class="lang-auto">    auto cu = std::make_shared&lt;torch::jit::script::CompilationUnit&gt;();
    torch::serialize::OutputArchive archive(cu);
    {
        torch::serialize::OutputArchive slot(cu);
        module-&gt;save(slot);
        archive.write(KEY_MODULE, slot);
    }
    archive.save_to(filename);
</code></pre>

username: pascal.soveaux
cooked: <p>It works fine now.<br>
Thank you Michael</p>
Predict multiple input to multiple output of LSTM very hard

username: pahome
cooked: <p>I try to predict access times of all blocks of one disk in one lstm model.</p>
<p>Assume that some blocks have similar access pattern A, and some blocks have similar access pattern B…etc.</p>
<p>Features I have is io_size, io_latency, io_counts, block_ID, and it works when I predict multiple to one.<br>
Ex: input is last 4 timestamps, and predict access times of next 1 timestamp.</p>
<p>But when I change to input 10 timestamps and output 4 timestamps, it all not obvious and patterns don’t show well. When I expect zero access times in specific timestamps, it have some edge shape of access times.</p>
<p>Should I change something or validate others?</p>
How to Extract 3d patches using unfold?

username: beitian
cooked: <p>First, how can deal with the margin problem when I using unfold function for many times to extract 3d patches, because I found that some of pixels in margin will be ignored when the patch size can not be exactly divided.<br>
Second, in the test set, how can I recover the original Image size after predict label through networks?</p>

username: ptrblck
cooked: <p>I’m not sure, if there is an easy way to create patches and reconstruct the input image without padding, if the shapes do not match.<br>
Have a look at <a href="https://discuss.pytorch.org/t/creating-nonoverlapping-patches-from-3d-data-and-reshape-them-back-to-the-image/51210/6">this post</a> to see, how to use padding to reconstruct the input again.</p>

username: beitian
cooked: <p>Thank you, you really help me .</p>
Custom activation functions with Learnable Parameter

username: Syzygianinfern0
cooked: <p>How can I implement something like PRelu by inheriting autograd.Function?</p>

username: tom
cooked: <p>You likely want <code>nn.Module</code> for holding the parameters, in fact see the PyTorch implementation of <code>nn.PReLU</code> and also the <a href="https://pytorch.org/tutorials/beginner/nn_tutorial.html" rel="nofollow noopener">What is torch.nn tutorial</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
Whats the proper way to push all data to GPU and then take small batches during training?

username: TinfoilHat0
cooked: <p>I’m working with the MNIST dataset where the memory of my GPU can accommodate the entire dataset.<br>
Naturally, I’d like to push everything to GPU before starting to train my model to make things faster. However, even though I want everything in GPU, I’d like to take small batches of data to ensure good generalization. What’s the nicest way of doing so? As of now, what I do is this:</p>
<pre><code class="lang-auto">apply_transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])
 train_dataset = datasets.MNIST(data_dir, train=True, download=True, transform=apply_transform)
self.train_loader = DataLoader(data.DatasetSplit(train_dataset, data_idxs), 
                                             batch_size=60000, shuffle=True)
    
# load the entire data to GPU and get it in a different  with desired batch size
 _, (images, labels) = next(enumerate(self.train_loader))
images, labels = images.to(args.device), labels.to(args.device)
self.train_loader = DataLoader(torch.utils.data.TensorDataset(images, labels), batch_size=args.local_bs, shuffle = True)
</code></pre>
<p>So briefly, first I wrap the dataset with a loader where the batch size is the size of the entire dataset. I then fetch tensors from this loader and then push them to the GPU. Finally, I create a tensor dataset with tensors I have and wrap the tensor dataset with another loader where the batch size is my desired size.<br>
Is there a nicer way of doing this? Preferably, without meddling with data loaders at all, as profiling my code shows data loaders are the bottleneck (could be because I use them in a wrong way).</p>

username: tom
cooked: <p>For MNIST, the dataset already stores everything in Tensors, so you can grab the <code>ds.data</code> and <code>ds.targets</code> Tensors from the torchvision MNIST dataset directly and stick them into your TensorDataset.</p>
<p>Best regards</p>
<p>Thomas</p>