akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to convert the code in 0.4.1 to 1.1.0

username: lunasdejavu
cooked: <p>I want to do mobilenetv1-yolov2 by this github <a href="https://github.com/marvis/pytorch-yolo2" rel="nofollow noopener">project</a><br>
but it is in 0.4.1 but my environment is in CUDA 10<br>
so that I want to do modify the code little by little<br>
but are there more documents to help this work except for the official documents about 1.1?<br>
How people usually do it?</p>
Passing generated output as input to t+1 without a for loop

username: ram
cooked: <p>Is it possible to make lets say LSTMs in pytorch to pass the generated output as input to the next time step without using a for loop?<br>
For example, if the decoder_input_emb in the following example is a 3d tensor (batch, seq_leng, dim) , the LSTM will always take the the input from the  decoder_input_emb not from the generated output.</p>
<blockquote>
<p>outputs, hidden = nn.LSTM(decoder_input_emb, hidden)</p>
</blockquote>
<p>Is there a way to force the LSTM to take its out prediction as next input?</p>
<p>I know that it can be done in a loop like this (simplified):</p>
<blockquote>
<pre><code>decoder_input = decoder_input_emb[:, 0]
  	for di in range(max_length):
  		step_output, hidden = nn.LSTM(decoder_input, hidden)
  		decoder_input = step_output
</code></pre>
</blockquote>
<p>But this is significantly slower.</p>

username: SimonW
cooked: <p>nn.LSTM does that for you. you don;t need to run in a loop</p>

username: jrichterpowell
cooked: <p>Sorry Simon, would you mind explaining this? I thought the nn.LSTM and nn.RNN used the input tensor at the next sequence index as input, and only passed the hidden layer forward.</p>
Adding the metadata information into CNN

username: prashant_kumar
cooked: <p>I have a CNN model in pytorch. My dataset comes with a lot of metadata information. Besides using the image I want to use this metadata information to improve my accuracy. Something like a small SVM classifier that uses last convolution layer features + metadata information to finish up the model and train on that. Any codebase links/paper would be helpful which can guide me in this regard.</p>

username: bhushans23
cooked: <p>Please refer to following tutorial <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#finetuning-the-convnet" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#finetuning-the-convnet</a></p>
Error: Cudnn RNN backward can only be called in training mode

username: dldefy
cooked: <p>I’m running into the following error:</p>
<p><code>Cudnn RNN backward can only be called in training mode</code></p>
<p>The first epoch trains fine, and the error only occurs on the 2nd epoch. I’ve made sure to call <code>model.train()</code> prior to generating the hidden layer inputs, calculating loss, and before <code>loss.backward()</code>, and <code>model.training</code> displays as <code>True</code> when I check it. If it’s useful I have to use <code>loss.backward(retain_graph=True)</code>.</p>
<p>What other reasons might be causing this error? Are there other items I need to make sure I run <code>.train()</code> on?</p>

username: jerinphilip
cooked: <p>Chances are high that there maybe a <code>model.eval</code> while computing validation loss running in between epochs (perhaps based on updates rather than epochs) which could be doing this?</p>

username: dldefy
cooked: <p>In my epoch loop I have train and validate as separate function calls. In the train function I call <code>model.train()</code> at the start, and at the start of the validate function I call <code>model.eval()</code>. I thought this would automatically handle making the model trainable again…however when I comment out the validate function (so the model only trains) it actually seems to work.</p>
<p>Do I need to run <code>model.train()</code> somewhere again?</p>

username: yuanzhoulvpi
cooked: <p>you should read this link,I suppose it can solve your problems <a href="https://discuss.pytorch.org/t/cudnn-rnn-backward-can-only-be-called-in-training-mode/37622/2">https://discuss.pytorch.org/t/cudnn-rnn-backward-can-only-be-called-in-training-mode/37622/2</a></p>

username: dldefy
cooked: <p>Thanks yuanzhoulvpi and jerinphilip for your input.</p>
<p>I was using a custom loss function and as it turns out this was the issue - data I was tracking was not being properly detached and this somehow affected the model being able to be fully put back into training mode.</p>
<p>All fixed now!</p>
Running batch through custom function

username: ahmed
cooked: <p>I am trying to run a batch of tensors from my dataloader through a custom function. I have normalised them and simply want to unnormalise it. This is the function:</p>
<pre><code class="lang-auto">def unnormalise(height, width, x):
    x = np.asarray(x.cpu().detach().numpy())
    if sum(x) == 0:
        x = x
    else:
        x1 = ((x[0] + 1) * width) / 2
        x2 = ((x[1] + 1) * height) / 2
        x3 = ((x[2] + 1) * width) / 2
        x4 = ((x[3] + 1) * height) / 2
        x = [x1, x2, x3, x4]
</code></pre>
<p>It works fine as part of my dataloader (I believe it is because only one value is given rather than a batch) but it doesn’t work when there are multiple values. The batch size i am currently using is 4.</p>
<p>This is the unnormalised values:</p>
<pre><code class="lang-auto">tensor([[200.90,  85.31,  14.35,  43.75],
        [  0.00,   0.00,   0.00,   0.00],
        [ 53.20,  97.12,   9.45,  22.75],
        [114.10,  88.81,  13.30,  14.88]]) 
</code></pre>
<p>I tried this:</p>
<pre><code class="lang-auto">i = 0
for i in range(4):
    x_ = unnormalise(224, 224, x[i])
    print(x)
    i += 1
</code></pre>
<p>but that only gives me the data for the final tensor. The <code>print(x)</code> shows that it is correctly unnormalising each tensor in the batch but only outputs the final tensor as <code>x_</code>.</p>
<pre><code class="lang-auto">print(x):

tensor([200.90,  85.31,  14.35,  43.75])
tensor([0., 0., 0., 0.])
tensor([53.20, 97.12,  9.45, 22.75])
tensor([114.10,  88.81,  13.30,  14.88])
</code></pre>
<pre><code class="lang-auto">print(x_): tensor([114.10,  88.81,  13.30,  14.88])
</code></pre>
<p>Any advise to help me obtain all 4 values would be greatly appreciated.</p>

username: ahmed
cooked: <p>I have managed to work it out:</p>
<pre><code class="lang-auto">list = []
i = 0
for i in range(4):
    list.append(unnormalise(224, 224, x[i]))
    i += 1
</code></pre>
Grad values are None when tensors undergo shallow copy

username: Siddhesh_Mhatre
cooked: <p>Hi. I have a model who’s forward method performs a shallow copy of the tensors into a dictionary before returning like so -</p>
<pre><code class="lang-auto">def forward(self, input):
    block0 = self.block0(input)
    block1 = self.block1(block0)
    self.end_points = {}
    self.end_points['block0'] = (block0, 0)
    self.end_points['block1'] = (block1, 1)
    return block0 
</code></pre>
<p>Where <strong>self.block0</strong> and <strong>self.block1</strong> are <strong>nn.Conv2d</strong> layers followed by <strong>batch norm</strong> and <strong>leaky relu</strong></p>
<p>If now I do -</p>
<pre><code class="lang-auto">output = model(input)
loss = output.mean()
loss.backward()
print (model.block0.conv.bias.grad) #block0 is an nn.Module with contains a class attribute conv which is nn.Conv2d
</code></pre>
<p>The grad value is None. There is a similar outcome if I return just return <strong>self.end_points</strong> dict.</p>
<p>One the other hand with the following forward function -</p>
<pre><code class="lang-auto">def forward(self, input):
    block0 = self.block0(input)
    block1 = self.block1(block0)
    self.end_points = {}
    self.end_points['block0'] = (block0, 0)
    return block0
    self.end_points['block1'] = (block1, 1) 
</code></pre>
<p>The grad attribute of <strong>model.block0</strong> gets accumulated with the correct gradient.</p>
<p>I have this problem when I wrap the module in nn.DataParallel only. I’m using the following workaround since I have some custom functions.</p>
<pre><code class="lang-auto">class MyDataParallel(torch.nn.DataParallel):
    """
    Allow nn.DataParallel to call model's attributes.
    """
    def __getattr__(self, name):
        try:
            return super().__getattr__(name)
        except AttributeError:
            return getattr(self.module, name)
</code></pre>
<p>I am not able to understand why this is the case. Please help! Thank you in advance.</p>

username: ptrblck
cooked: <p>Could you post the definition of <code>self.block0</code> and <code>self.block1</code>?<br>
Based on your description, I would assume you’ve defined them as <code>nn.Sequential</code> modules, but then you would get an error calling <code>model.block0.grad</code>.</p>

username: Siddhesh_Mhatre
cooked: <p>My apologies. I did not mention earlier that I encounter this problem only when I wrap my module inside nn.DataParallel. I have updated the description above.</p>
<p>As to your question both of those blocks are nn.Modules which have an instance of nn.Conv2d, nn.LeakyRelu and nn.BatchNorm2d which are called on the input in the forward method in that order.</p>

username: Siddhesh_Mhatre
cooked: <p>The problem is fixed by not setting end_points as a class variable and returning the entire dict. I suspect the problem is along the lines of <a href="https://github.com/pytorch/pytorch/issues/16532" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/16532</a>. Not sure though where the tensor is going out of scope and triggering a recursive deletion of the rest of the graph though.</p>
Extremely slow inference speed with dynamic inference graph

username: TaojiannanYang
cooked: <p>I am trying to train a network, but training different portion of the whole network in each iteration. Let’s say <code>1x</code> represents the whole network, <code>0.5x</code> means half of the network(just use the first <code>0.5x</code> weights of the <code>1x</code> network). I randomly sample a <code>width</code> in each iteration, and do forward and backward only with that part of the whole network. But I found the forward and backward speed is very slow. However, if I sample <code>width</code> from a fixed set, e.g. <code>[0.25, 0.5, 0.75, 1.0]</code>, the speed will be much faster. Is this caused by the dynamic computational graph in pytorch? But I think even I fix the <code>width_set</code>, it still need to create a different graph in every iteration. I don’t know why there is a different between these two situations.</p>
How to generate LLVM IR from the pytorch code

username: ritesh_gupta
cooked: <p>I’ve seen in forums that people used -dump-ir command on the command line to get it. Have no idea where to specify it. Can anyone explain? or give a sample code to generate the IR?</p>

username: jfix
cooked: <p><code>-dump-ir</code> is a command line option specified in <code>lib/Optimizer/IROptimizer.cpp</code>. It should be available on the command line for any binary that links against the <code>Optimizer</code> lib. For example, you can specify it when running the <code>image-classifier</code>. There are examples of running the <code>image-classifier</code> at <code>tests/images/run.sh</code>, so you could append it to any of those. It also should work when running unit tests that link against <code>Optimizer</code>, e.g. <code>./tests/OperatorTest -dump-ir</code>.</p>

username: ritesh_gupta
cooked: <p>Thanks <a class="mention" href="/u/jfix">@jfix</a>. I’ll try to execute and get back to you. If you have some good tutorial on this please share I need it.</p>

username: jfix
cooked: <p>I would suggest reading our <a href="https://github.com/pytorch/glow/blob/master/docs/Testing.md" rel="nofollow noopener">doc on Testing</a>.</p>
Ludwig but on PyTorch

username: torrmal
cooked: <p>Hi Everyone,</p>
<p>I am working on an implementation of Ludwig but based on Pytorch (any feedback is welcome):</p>
<p>The goal is to facilitate Pytorch for those that know little about ML on the one hand, and on the other to make experiments easy for those with more advanced knowledge of Pytorch and ML.</p>
<p>Although it is inspired on Ludwig, it’s not only that the soul is on Pytorch, also going for a minimalist approach and more flexible in the sense that anything can be replaced and customized.</p>
<p>You can see it here:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/mindsdb/lightwood/" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/31035808?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="63" height="63">

<h3><a href="https://github.com/mindsdb/lightwood/" target="_blank" rel="nofollow noopener">mindsdb/lightwood</a></h3>

<p>A minimalist framework to streamline the work with PyTorch - mindsdb/lightwood</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>It seems to be functional ATM, and out of the box can model input and output with any of the following data types: (NUMERIC, CATEGORICAL, DATETIME, IMAGE, TEXT, TIME_SERIES)</p>
<p>Currently working on:</p>
<ul>
<li>Putting documentation together</li>
<li>Examples</li>
<li>A simple abstraction of Botorch/Ax such that optimal hyperparameters can be found in a similar frictionless fashion.</li>
</ul>
<p>Looking forward to hearing your thoughts,</p>
<p>Best regards,</p>
<p>Jorge</p>
Data_source parameter in Sampler.__init__

username: marchinidavide
cooked: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Quick question about the torch.utils.data.Sampler class and the DataLoader.</p>
<p>What is the behaviour of<br>
<code>self.data_source = data_source</code> in the definition of all Samplers?<br>
I have my custom data.Dataset class that needs a very expensive initialisation: does calling<br>
<code>self.data_source = data_source</code> in the <code>__init__</code> of the sampler init a copy of the dataset?<br>
My custom data.Dataset also has a variable <code>self.dataset_index</code> (instance of <code>pandas.DataFrame</code>) that is pretty much all I would need in to access from the sampler. Does the <code>data_source</code> parameter strictly need to be an instance of <code>data.Datset</code> or anything with a <code>__len__</code> method?</p>
<p>As always thanks in advance!</p>

username: marchinidavide
cooked: <p>Also found that <code>len(pd.DataFrame)</code> is a single number while <code>pd.DataFrame.__len__()</code> returns the two dimensions and the whole table …</p>

username: Oli
cooked: <p>Hi,</p>
<p>the data_source is referring to a Dataset object. This dataset won’t become reinitialised in the sampler. The sampler simply calls the dataset.len() function so that it knows how many indexes it is supposed to produce.</p>
<aside class="quote no-group" data-username="marchinidavide" data-post="1" data-topic="47755">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marchinidavide/48/15221_2.png" class="avatar"> marchinidavide:</div>
<blockquote>
<p>Does the <code>data_source</code> parameter strictly need to be an instance of <code>data.Datset</code> or anything with a <code>__len__</code> method?</p>
</blockquote>
</aside>
<p>I’m not sure, but I’m guessing that it would work with anything with a <code>__len__</code>. Report back if you find out <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: marchinidavide
cooked: <p>I find difficult to relate <strong>len</strong> method of the Dataset class and <strong>len</strong> method of the Sampler class?<br>
What happens when they differ?</p>

username: Oli
cooked: <p>From phone. So if your sampler is longer than dataset it would produce an index that would be out of range. Like ~&gt; dataset[high_ind] ~&gt; out of range error</p>
<p>If lower, some data points would never be used. You could handle these cases in your dataclass I guess, but not sure if there is a situation where this makes sense</p>
Using models in a row

username: barthelemymp
cooked: <p>Hello I d like to stack models .<br>
For example:</p>
<pre><code class="lang-auto">class m1(nn.Module):
    def __init__(self, n_wire):
        super(m1, self).__init__()
        ...
    def forward(self,):
        ...

class m2(nn.Module):
    def __init__(self, n_wire):
        super(m2, self).__init__()
        ...
    def forward(self,):
        ...

class m3(nn.Module):
    def __init__(self, n_wire):
        super(m3, self).__init__()
        ...
    def forward(self,):
        ...
</code></pre>
<p>and then optimise a cost between label and m3(m2(m1(x)))<br>
PS: I want to keep them separatedbecause some of them are usual nn and other are using an interface specific for qml.</p>
<p>What are the condition in order not to break the backprop ?</p>
<p>Best regards,</p>
<p>B</p>

username: Balamurali_M
cooked: <p>What do you mean by break the backprop ? This equation itself will work good provided you handle the dimensions well. loss.backward () will be applied on a graph. This three models are part of the graph</p>

username: barthelemymp
cooked: <p>thank you!<br>
It just that I wanted to combine different combination of models. To do so I want to make a big model which will call m1(m2()) or m2(m1()) or m1(m1()) etc according to some trainable parameters. And i was wondering if the backward was not stuck to one model. But according to your answer the compose of model will make only graph so one backward. That s exactly what I need <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
thank you again!</p>
State_dict() changes with the model

username: lei_bai
cooked: <p>Hi all, I want to save the best model and then load it during the test. So I use the following method:</p>
<p>def train():<br>
<span class="hashtag">#training</span> steps …<br>
if acc &gt; best_acc:<br>
best_acc = acc<br>
best_state = model.state_dict()<br>
return best_state</p>
<p>then in the main function I use:<br>
model.load_state_dict(best_state)<br>
to resume the model.<br>
However, I found that best_state is always the same as the last state during training, not the best state. Is anyone know the reason and how to avoid it?  (Version: 1.1.0, Linux, GPU)</p>
<p>By the way, I know I can use torch.save(the_model.state_dict(), PATH) and then load the model by<br>
the_model.load_state_dict(torch.load(PATH)).<br>
However, I don’t want to save the parameters to file as train and test functions are in one file.</p>

username: JuanFMontesinos
cooked: <p>I don’t remember right now but it may map to same memory address. Try to make a deep copy of state edict when you save it</p>

username: iArunava
cooked: <p>Hey there! Welcome to the community!</p>
<pre><code class="lang-auto">if acc &gt; best_acc:
  best_acc = acc # you missed this
  best_state = model.state_dict()
  . . . 
</code></pre>
<p>Happy Coding <img src="https://discuss.pytorch.org/images/emoji/apple/man_technologist.png?v=9" title=":man_technologist:" class="emoji" alt=":man_technologist:"></p>

username: lei_bai
cooked: <p>Thanks for your answer, I have update the sentence in the question. I have best_acc = acc in my code, it doesn’t work.</p>

username: iArunava
cooked: <p>Please paste the code properly in ```</p>
<p>P.S. I see that the<br>
<code>return best_state</code><br>
is also there, are you sure you are returning after several epochs? And not just after 1 epoch? Or whenever the <code>acc &gt; best_acc</code> ? ( It can’t be figured out as you have no indentations in your question.)</p>

username: god_sp33d
cooked: <p>Make a deep copy of state_dict ?</p>
<p>best_model = copy.deepcopy(model.state_dict())</p>

username: iArunava
cooked: <p>Yeah! Since you are <em>not</em> saving it to a file, might be the case that <a class="mention" href="/u/god_sp33d">@god_sp33d</a> and <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> is right, that you need to make a deepcopy. DeepCopy should work, if you have no leaks elsewhere in the code.</p>

username: lei_bai
cooked: <p>Thanks, deepcopy works</p>

username: lei_bai
cooked: <p>I should use deepcopy, otherwise the saved state keeps changing with training</p>
About ImbalancedDataSampler

username: msrdinesh
cooked: <p>I want to  know whether ImbalancedDataSampler in this <a href="https://github.com/ufoym/imbalanced-dataset-sampler" rel="nofollow noopener">link</a> is present in pytorch are not. When I run (?torch.utils.data.Sampler), I am seeing it as one of the subclasses of torch.utils.data.Sampler. But, there is nothing mentioned about it in pytorch documentation and also in source code. I am confused. Please help. Thanks in<br>
Advance!!</p>
<pre><code class="lang-auto">Init signature: torch.utils.data.Sampler(data_source)
Docstring:     
Base class for all Samplers.

Every Sampler subclass has to provide an __iter__ method, providing a way
to iterate over indices of dataset elements, and a __len__ method that
returns the length of the returned iterators.
File:           /home/AIX_Common/__files__/opt/anaconda2/envs/ai-gpu/lib/python3.6/site-packages/torch/utils/data/sampler.py
Type:           type
Subclasses:     SequentialSampler, RandomSampler, SubsetRandomSampler, WeightedRandomSampler, BatchSampler, DistributedSampler, SortSampler, SortishSampler, ImbalancedDatasetSampler
</code></pre>
How to construct model based on my formula in Pytorch

username: Sirui_Li
cooked: <p>I am totally new to Pytorch and machine learning. I am trying to construct my model from scratch. The model is not CNN or RNN, just based on my formula. The input is two matrixes. What I want to do in my hidden layer is multiplying these two matrixes, and then output the result in the output layer.</p>
<pre><code class="lang-python">class myModel(nn.Module):
    def __init__(self, matrix_a, matrix_b):
        super(myModel, self).__init__()
        self.matrix_a = matrix_a
        self.matrix_b = matrix_b
        self.build_model()


    def build_model(self):
        self.layers = nn.ModuleList()
        #i2h
        i2h = self.build_input_layer()
        self.layers.append(i2h)
        # h2h
        h2h = self.build_hidden_layer()
        self.layers.append(h2h)
        # h2o
        h2o = self.build_output_layer()
        self.layers.append(h2o)


    def build_input_layer(self):
        input_matrix_a = self.matrix_a
        input_matrix_b = self.matrix_b
        return


    def build_hidden_layer(self):
        # I want to multiply matrix a and b
        return


    def forward(self,matrix_a, matrix_b):
        for layer in self.layers:
            output = layer(matrix_a, matrix_b) # not sure if it is correct
        return output
</code></pre>

username: ptrblck
cooked: <p>I’m not sure, how <code>build_model</code> is supposed to work, as <code>build_input_layer</code> does not return anything.<br>
However, if you would just like to pass two matrices, create a single one, and use it as a linear layer, this code should work:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self, matrix_a, matrix_b):
        super(MyModel, self).__init__()
        self.matrix = nn.Parameter(torch.matmul(matrix_a, matrix_b))
        
    def forward(self, x):
        x = x.matmul(self.matrix)
        return x
        
    
matrix_a = torch.randn(2, 5)
matrix_b = torch.randn(5, 10)
model = MyModel(matrix_a, matrix_b)
x = torch.randn(1, 2)
output = model(x)
loss = output.mean()
loss.backward()
</code></pre>

username: Sirui_Li
cooked: <p>Thank you very much!! Yes, you are right. My input layer is wrong. Could you please explain the <code>loss = output.mean()</code> for me? As the output does not have a label, I am not clear why there is a loss function.</p>

username: ptrblck
cooked: <p>I used it just to calculate some valid gradients without a target, so you can just ignore it and use your target and a proper criterion to calculate the loss. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Why do we need to send the model to a device if we are using DataParallel?

username: Brando_Miranda
cooked: <p>I have access to multiple GPUs. I wanted to use them. However, I was reading the <a href="http://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="nofollow noopener">tutorial</a> and saw that it was not explained why one needed the device id. It was really weird to me because I would have thought that the whole point was that pytorch handles that for you and just uses the GPUs for you without you having to worry about it…thus my question is, why does it need to give the device to the model if one is using DataParallel and how to use that so that it does it properly and uses the GPUS available.</p>
<p>What confuses me is the following line in the tutorial:</p>
<pre><code class="lang-auto">model = Model(input_size, output_size)
if torch.cuda.device_count() &gt; 1:
  print("Let's use", torch.cuda.device_count(), "GPUs!")
  # dim = 0 [30, xxx] -&gt; [10, ...], [10, ...], [10, ...] on 3 GPUs
  model = nn.DataParallel(model)

model.to(device) ## &lt;- DO WE REALLT NEED THIS?
</code></pre>

username: kangkang_li
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/11d0d6ab35e6b1e4e05ea1e7efdf34838b842b97.png" data-download-href="https://discuss.pytorch.org/uploads/default/11d0d6ab35e6b1e4e05ea1e7efdf34838b842b97" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/11d0d6ab35e6b1e4e05ea1e7efdf34838b842b97_2_690x481.png" alt="image" data-base62-sha1="2xByc4q1fxRbBiEtq4PkCepp2Bh" width="690" height="481" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/11d0d6ab35e6b1e4e05ea1e7efdf34838b842b97_2_690x481.png, https://discuss.pytorch.org/uploads/default/original/2X/1/11d0d6ab35e6b1e4e05ea1e7efdf34838b842b97.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/1/11d0d6ab35e6b1e4e05ea1e7efdf34838b842b97.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">766×535 42.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It seems that you don’t have to.</p>
Attention mechanisms in Image

username: Mdhvince
cooked: <p>Hi,</p>
<p>I would like to know if a CNN can output le (x,y) point in an image that represents where the attention is on this image.</p>
<p>I’ve heard about attention mechanisms, but most of the example is for NLP.</p>
<p>So, maybe there is a way to output the « point of attention » through the last conv layer of a CNN as an (x,y) point.</p>
<p>I think this can be done by looking at what part of the image the CNN pay attention to classify an image. But my task is more complex. Here I want to do this without the classification task. I just want to know what is the point on this image we are looking at (something like visual perception), I mean, where the user is looking at!</p>
<p>It will be great if you can give me some resources. Example with PyTorch will be excellent.</p>
<p>Thank you.</p>
<p>Medhy</p>
Rendering using Batch of normals and batch of coefficient matrices

username: watchdog
cooked: <p>Hello, i am trying to build code for do a basic rendering using:<br>
A batch of normal maps, <strong>Normals</strong> with size:<br>
<strong>[batch_size, Width, Height, channels=3]</strong><br>
and a coefficient Matrix, <strong>M</strong>, with size: <strong>[batch_size, 4, 4, channels=3]</strong>.</p>
<p>So, to put it simple i want to do a matrix multiplication, between every normal map with its corresponding Matrix <strong>M</strong>. I used **bmm()<strong>and  more specifically after permuting the axes i got :<br>
For the normal map. <strong>Normals.shape = [batch_size, Width*Height, channels=3], and for M = [batch_size, 16, channels=3]</strong> .<br>
But god a batch of images with shape ** [batch_size, Width<em>Height, 16] ** and after taking first 3 channels, **[batch_size, Width</em>Height, :3]</strong> , i don’t get the expected result. Any suggestion or help would be appreciated.</p>
CUDA Memory Overflow during LSTM + Attention

username: Soumyadip_Ghosh
cooked: <p>Hi,</p>
<p>My LSTM Module runs fine without attention, but it runs into CUDA out of memory error with attention. The memory used increases with every forward pass.<br>
Code snippet below. I pass the encoder hidden states to the decoder in a utility function.</p>
<p>LSTM Forward Pass</p>
<pre><code class="lang-auto">for di in range(target_length):
                #Get context vectors
                context_vectors = self.attention(lstm_hidden[0]) #Pass only the hidden state
                #combine attention with input
                lstm_input = self.attention_combine(torch.cat((context_vectors, lstm_input), 2))
                lstm_output, lstm_hidden = self.lstm(lstm_input, lstm_hidden) #Feed to lstm
                lstm_input = embeddings[:,di,:].unsqueeze(1) #Next input if teach forcing = false
                decoded_sentences.append(lstm_output)
</code></pre>
<p>Attention Code</p>
<pre><code class="lang-auto">def attention(self, states):
        '''
        Calculate attention weights
        1. Take one step of the hidden states and calculate for all in the batch 
        2. Linearly transform to one weight
        3. Append attention weights
        '''
        states_temp = states.transpose(1,0) #decoder hidden states and transform shape to (31 * 1 * 1024) !!!
        encoder_state_num = self.encoder_hidden.shape[0] #Number of hidden states in encoder
        
        encoder_hidden_temp = self.encoder_hidden.unsqueeze(0) #encoder hidden shape (1*31*1024)
        states_temp = states_temp.repeat(1, encoder_state_num, 1)
        encoder_hidden_temp = encoder_hidden_temp.repeat(encoder_state_num, 1, 1)
        encoder_decoder_hidden_matrix = torch.cat((states_temp, encoder_hidden_temp), dim=2)

        attn_weights = torch.tanh(self.attention_weights(encoder_decoder_hidden_matrix)).squeeze() #Attention weights shaep (31 *31) ith row indexes decoder hidden states jth column indexes encoder hidden states

        attn_weights = self.mask_softmax(attn_weights, self.mask_vector)

        "Apply Attention Weights"
        #Efficient Implementation
        context_vectors = torch.matmul(attn_weights.permute(1,0).unsqueeze(2), self.encoder_hidden.unsqueeze(1)) #Apply attention weights to encoder_hidden vectors
        context_vectors = torch.sum(context_vectors, 0)
        context_vectors = context_vectors.unsqueeze(1) #context vector shape (31*1*1024) for all sentences - one context vector
        
        return context_vectors
</code></pre>
<p>Best,<br>
Soumyadip</p>
Regularization on a subset of parameters

username: hosseinshn
cooked: <p>Hello,</p>
<p>I was wondering, what is the best way to apply L2 or L1 regularization to only some of the parameters of a network instead of all of them? For example, for a network with 3 fully connected layers.</p>
<p>Thanks,</p>
<p>Hossein</p>

username: alex.veuthey
cooked: <p>Have a look at this topic / answer: <a href="https://discuss.pytorch.org/t/dont-regularize-me-bro/4946/4" class="inline-onebox">Don't Regularize Me, Bro</a></p>
VAE using CNN on CIFAR10 dataset

username: Ravi_Gupta
cooked: <p>Can anyone pls tell me why my loss is coming negative. And my output is very bad…any suggestion.</p>
<p>link of code:- <a href="https://drive.google.com/open?id=1SqN6ruqdmWg_Y7rd_S_n-bKTtCACDKkr" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p><span alt="" role="presentation" class="broken-image" title="This image is broken"><svg class="fa d-icon d-icon-unlink svg-icon" aria-hidden="true"><use href="#unlink"></use></svg></span></p>
<p><span alt="" role="presentation" class="broken-image" title="This image is broken"><svg class="fa d-icon d-icon-unlink svg-icon" aria-hidden="true"><use href="#unlink"></use></svg></span></p>
Sparse activations matrix multiplication, and how do I find matching indices to zero out weights?

username: tsauri
cooked: <p>I am doing weight pruning, by sparsifying activations with mask – If the activations have lots of zeros, the weights can be safely zeroed out too.<br>
And I need way to calculate weight parameters sparsity percentage.</p>
<p>Say I have y(x) = ReLU(Wx+b)<br>
x is activations vector from previous layer, and have lots of zero values (due to mask)</p>
<p>From matmul Wx, how do I find elements (and its indices) in W that match with zero-valued activations in x,<br>
So I can sparse W, and count zero values to calculate sparsity percentage.<br>
Does pytorch have such function to do so?</p>
<p>Also how do I do the same for convolution layers (They have filters, filter size, stride, pad, etc)</p>
<p>Thank you</p>
nn.Cov2D how to decide the output image with different size

username: ElleryL
cooked: <p>Consider I have decoder that maps the latent space to some image</p>
<p><code>latent_variable.shape = (batch_size=64, latent_dim=10)</code></p>
<p>I want to decoding it into an image with size <code>(batch_size=64, channels=3, height=32, width=64)</code> using 1 linear layers and 7 layers of <code>nn.ConvTranspose2d</code></p>
<p>My current code is</p>
<p><code>layer1 = nn.Linear(latent_dim, 256) # gives (batch_size, 256)</code></p>
<p>My goals is to add 7 more <code>nn.ConvTranspose2d</code> layers after this;</p>
<p>But I’m very confused how to add these layers up to make my <code>(batch_size, channels=3, height=32, width=64)</code> matched up (especially the height and width are different)</p>
<p>Can someone help me clarify how does <code>nn.ConvTranspose2d</code> work ?</p>
Variable sized input for LSTMCell

username: Abhishek_Das
cooked: <p>Suppose, I have sentences of variable length. Each of them is a 1x50 dim tensor. Now if I have 6 words in a sentence, would my sentence dimension be 6X1X50?<br>
What if I want it to process it in batch size of 2 as an input to LSTMCell ?</p>

username: anantguptadbl
cooked: <p>Hello Abhishek,</p>
<p>The dimensions of any document are NumSentences X NumWords_in_a_sentence X EmbeddingLengthForWord</p>
<p>I would recommend you going through some of the examples out there w.r.t to processing sentences using LSTM. The following can be helpful<br>
<a href="https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html#example-an-lstm-for-part-of-speech-tagging" rel="nofollow noopener">LSTM Tagger</a></p>

username: Abhishek_Das
cooked: <p>If I say I have just 10 sentences what would be the <strong>input</strong> dimension to LSTM/LSTMCell in above case?<br>
My embedding for each word is 1x50.<br>
Sentence have variable length i.e the number of words vary from sentence to sentence.<br>
Batch =2<br>
Please tell me what would be my input tensor.</p>
Pytorch C++ frontend : How to save tensor as numpy array

username: jkmm
cooked: <p>While this is easily possible in Pytorch, how do I a save a tensor as a numpy array in C++?<br>
torch::save() method saves as a zip archive data, and I don’t know how to convert this to numpy array.</p>
<p>torch::save(combined_tensor,“combined_tensor.pt”);<br>
*$file combined_tensor.pt *<br>
<em>combined_tensor.pt: Zip archive data</em><br>
Any help/pointers ??</p>
<p>Regards<br>
JK</p>
Including the PIL filename in transforms

username: ajwitty
cooked: <p>I am trying to include the filename path in a custom image transform (so that I can compare an image with its binary map during loading) but the PIL image object does not include the filename of the image. I am wondering if there is any way to pass the corresponding file path to the transform with the PIL image.</p>

username: Balamurali_M
cooked: <p>You can use dataloader. Set the dataloader in such a way that it can return image, binary mask and image filename. I am assuming there are two folders, one for image and other for masks and both have same filenames.</p>
<pre><code class="lang-auto">class DataLoaderEx(Dataset):
    
    def __init__(self,img_path,mask_path):
               self.img_path = img_path
               self.mask_path = mask_path
               self.files = glob.glob(img_path + '*.bmp')
    def __len__(self):
              return len(self.files)
    def __getitem__(self, index):
             img_path = self.files[index]
            mask_path = os.path.join(self.mask_path,os.path.basename(img_path))
             # Here you can read the image using PIL with img and mask path
            return image,mask.filename
</code></pre>
<p>Hope this one helps.</p>
How can I create a many to many RNN with fix number of unrolling steps in decoder?

username: leolcling
cooked: <p>I suppose I should define a <code>RNNCell</code> (or <code>LSTMCell</code>/<code>GRUCell</code>) in my decoder and use a for loop in the forward method in decoder. Something like this:</p><aside class="quote" data-post="4" data-topic="5657">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/moskomule/48/4429_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/different-between-lstm-and-lstmcell-function/5657/4">Different Between LSTM and LSTMCell Function</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    LSTMCell is more flexible and you need less code with LSTM . 
So with LSTMCell, 
def forward(self, x):
        h = self.get_hidden() 
        for input in x:  
            h = self.rnn(input, h) # self.rnn = self.LSTMCell(input_size, hidden_size)

while with LSTM it is 
def forward(self, x):
        h_0 = self.get_hidden()
        output, h = self.rnn(x, h_0) # self.rnn = self.LSTM(input_size, hidden_size)

output is the blue rectangles in your fig.
  </blockquote>
</aside>

<p>But when I look at the tutorial of <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="noopener nofollow ugc">seq-seq translation</a>. I got confused because the tutorial use nn.GRU as oppose to GRUCell, and a for loop was used in the training loop… Is is related to attention?</p>
<p>If I want to implement a many-to-many RNN with attention and control the output to be 5 steps, should I use <code>RNN</code> or <code>RNNCell</code>? Thanks.</p>
Questions about create a new loss function

username: njuww
cooked: <p>Suppose the original loss of the model  is loss_ori<br>
I want to add something to the loss function  about  the weight of every Conv2d layer just like:</p>
<p>before training:</p>
<pre><code>if isinstance(m, nn.Conv2d):
    I, O, W, H = m.weight.shape
    U = torch.randn(I, O, W, H)
    m.register_buffer("U", U)
    re_loss = torch.FloatTensor(0)
    m.register_buffer("re_loss",re_loss)
</code></pre>
<p>when training:</p>
<pre><code class="lang-auto"> def admm_loss(m):
      loss = nn.MSELoss()
      if isinstance(m,(nn.Conv2d,nn.Linear)):
          m.re_loss = loss(m.weight,m.U)
 net.apply(admm_loss)
 for name, para in net.state_dict().items():
       if "re_loss" in name:
           re_loss += para
 loss = loss_ori + 1.5e-4 * re_loss
 loss.backward()


</code></pre>
<p>but the program doesn’t work as excepted<br>
i wonder if my change to the loss function is right?（regardless of U,just focus on if the grad of weigts involve both loss_ori and re_loss)<br>
it seems both loss_ori and re_loss do not converge</p>

username: Balamurali_M
cooked: <p>What are you trying to do ? Can you explain it with better ?</p>

username: njuww
cooked: <p>trying to update weigths  by   model_loss  and  a  sparse matrix U, U  will be updated by myself ,i just want  to know  if  the way  i compute the grads is right</p>
Commputing the difference among one channel of a tensor and all other channels within that tensor

username: isalirezag
cooked: <p>How can I compute the difference among one channel of a tensor and all other channels within that tensor.<br>
In other words, what is the best way to do this::<br>
B,C,W,H = 2,3,20,20<br>
A = torch.rand(B,C,W,H)<br>
I want to have tensor D which is defined as following:</p>
<pre><code class="lang-auto">D[0,0,:,:] = torch.max(A[0:1,0:1,:,:]-A[0:1,:,:,:],dim=1)[0]
D[0,1,:,:] = torch.max(A[0:1,1:2,:,:]-A[0:1,:,:,:],dim=1)[0]
D[0,2,:,:] = torch.max(A[0:1,2:3,:,:]-A[0:1,:,:,:],dim=1)[0]
D[1,0,:,:] = ...
D[1,1,:,:] = ...
D[1,2,:,:] =...
</code></pre>
<p>Now imagine in real case C and B are so large.<br>
What is the fastest way to compute D?</p>

username: isalirezag
cooked: <p>The answer is something like this:</p>
<pre><code class="lang-auto">A_1 = torch.unsqueeze(A,1)
A_2 = torch.unsqueeze(A,2)
D = torch.max(A_1-A_2,dim = 2)[0]
</code></pre>
Overcommit_memory setting crashes pytorch with 'Cannot allocate memory' when forking

username: techknowfile
cooked: <p><strong>PyTorch:</strong> 1.1.0 (both built from github and d/l from pip)<br>
<strong>Python:</strong> 3.6.8 and 3.7.1<br>
<strong>System:</strong> 92GB RAM, 1 to 4 Tesla V100s.</p>
<p><strong>Issue:</strong> Using the default forking method for torch.multiprocessing,  num_workers &gt; 1 for DataLoader, and /proc/sys/vm/ <strong>overcommit_memory=2</strong> results in “Cannot allocate memory” error on fork.</p>
<p>The strange thing is that my dataset could fit into memory 100 times without using it up, and indeed less than 2GB of memory are taken up when running my code. This issue came up when I tried running my code on a node of a distributed cluster with the hardware mentioned above, after running perfectly fine on my 32GB/Titan X system. I had the system administrators confirm the issue was with the overcommit_memory settings by flipping them off for me yesterday.</p>
<p>There are several ways for us to get around it (changing the overcommit ratio, increasing the swap, or changing the overcommit_memory value), but I’m still confused as to why it seems Python  / Pytorch is requesting so much memory that it isn’t actually using in the first place, and if there is something I can change in my code that would resolve the issue without server-side changes.</p>
<p>Enabling torch.multiprocessing.set_start_method(“spawn”) or “forkserver” prevents the memory calculation error / os.fork() crash, but also slows my code down for a crawl (even and especially when trying to run on multigpu with DataParallel), so I’m assuming whichever solution method is preferred will be one that doesn’t involve these settings.</p>
Where is CUDAToolKit path when installed via conda?

username: Alex_Fann
cooked: <p>I installed my PyTorch 1.0 using the command <code>conda install pytorch torchvision cudatoolkit=9.0 -c pytorch</code> while my system has an existing <code>cudatoolkit</code> already, which causes a CUDA version mismatch in my current application. I am using Ubuntu 18.</p>
<p>I am wondering where can I find the <code>cudatoolkit</code> installed via the above conda command? Specifically, I am looking for: <code>cuda/bin</code> ,  <code>cuda/include</code>  and  <code>cuda/lib64</code> in  <code>$PATH</code> ,  <code>$CPATH</code>  and  <code>$LD_LIBRARY_PATH</code>  respectively.</p>

username: ptrblck
cooked: <p>The shipped CUDA libraries in the PyTorch binaries should not create any conflicts with your system-wide CUDA install.<br>
What kind of version mismatch do you get?</p>

username: Alex_Fann
cooked: <p>Thanks for the quick reply!</p>
<p>I was trying to install the library <code>pytorch geometric</code> on a server (without root access).</p>
<p>I have the situation detailed in <a href="https://github.com/rusty1s/pytorch_scatter/issues/19#issuecomment-501453905" rel="nofollow noopener">this</a> comment.</p>
<p>In short, the problem was that my system has already installed a <code>cudatoolkit</code>, but when installing pytorch with conda, a second <code>cudatoolkit</code> was installed (different toolkit versions). Then, when I install pytorch geometric, I have to specify the PATH of my toolkit, but I can only find the system-installed toolkit path instead of the conda-installed toolkit path.</p>
<p>Therefore, I am looking for the conda-installed toolkit path to make sure <code>pytorch geometric</code> uses the correct <code>cudatoolkit</code>.</p>

username: ptrblck
cooked: <p>The PyTorch binaries do not ship with <code>nvcc</code>, so you won’t be able to compile other libraries with it.<br>
The solution would be to install the same CUDA version locally on your machine as is used by PyTorch or build PyTorch and the other lib from source using your system-wise CUDA install.</p>

username: Alex_Fann
cooked: <p>Thanks for the reply!</p>
<p>However, in my case, I don’t have root access. I am wondering if it is possible to still use conda to install pytorch but specify the <code>cudatoolkit</code> as <code>9.2.148</code>?</p>
<p>I tried <code>conda install pytorch torchvision cudatoolkit=9.2 -c pytorch</code>, but get</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.version.cuda

'9.0.176'
</code></pre>

username: ptrblck
cooked: <p>Is building from source not an option?<br>
If so, could you run a docker container with e.g. CUDA10.0 and install pytorch using the same version?</p>
L2 norms for each layer

username: Maria_Fernanda_De_La
cooked: <p>Hi, I am wanting to obtain the L2 norms at each layer for all epochs. Would either of these be correct or should I access the data of the parameters to obtain the weights?</p>
<p>torch.sum(torch.norm(net.conv_layer1[0].weight, p=2, dim=1))<br>
.<br>
.<br>
.<br>
torch.sum(torch.norm(net.fc[0].weight, p=2, dim=1))</p>
<p>OR<br>
torch.sum(net.conv_layer1[0].weight.pow(2)).pow(1/2)<br>
.<br>
.<br>
.<br>
torch.sum(net.fc[0].weight.pow(2)).pow(1/2)</p>
<p>Thank you in advance!</p>
Slow training speed with dynamic network

username: TaojiannanYang
cooked: <p>I am trying to reproduce the paper <a href="https://arxiv.org/abs/1903.05134" rel="nofollow noopener">https://arxiv.org/abs/1903.05134</a>, where the idea is to train 4 random width sub-networks in each iteration so that the full-network can execute at any widths. However, I found the forward and backward speed of the random width sub-network is very slow. The code for training is shown below</p>
<pre><code class="lang-auto">min_width, max_width = 0.25, 1.0  # min width 0.25x, max width 1.0x
width_mult_list = [min_width, max_width]
sampled_width = list(np.random.uniform(FLAGS.width_mult_range[0], FLAGS.width_mult_range[1], 2))
width_mult_list.extend(sampled_width)
for width_mult in sorted(width_mult_list, reverse=True):
     model.apply( lambda m: setattr(m, 'width_mult', width_mult))
     output = model(input)
     loss = criterion(output, target)
     loss.backward()
optimizer.step()
</code></pre>
<p>The issue is that the speed of min_width and max_width is normal, but the two random widths are very slow(3-4 times slower).<br>
However, if I fix the <code>samped_width</code> to some random number, let’s say <code>sampled_width=[0.25, 0.6543, 0.76354, 1.0]</code>, the speed will be normal.<br>
This is strange to me, since I think the speed should be similar. Is this caused by the dynamic graph in pytorch?<br>
My pytorch version is 1.0.1</p>