akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to roll back the impact of a loss function with less cost?

JTiC: <p>I am new in pytorch. I want to know how to roll the impact of a loss back with very less cost.<br>
For example, a model H with parameters p at a moment. Then a loss L is calculated based on the parameters and the model. It works like:</p>
<pre><code class="lang-auto">L =  criterion(prediction, target)
L.backward()
optimizer.step()
</code></pre>
<p>Then the parameters p become p’, and I want to get the parameters p back, or parameters approximating to p. How can I do?<br>
How to analyze a operation like:</p>
<pre><code class="lang-auto">L =  criterion(prediction, target)
L1 = -L
L.backward()
optimizer.step()
optimizer.zero_grad()
L1.backward()
optimizer.step()
</code></pre>
<p>Is it ok? Thank you!</p>
RuntimeError: Function FillBackward1 returned an invalid gradient at index 1 - expected type torch.cuda.FloatTensor but got torch.FloatTensor

kikiloverafa: <p>Hello,</p>
<p>I am new to Pytorch, and I am doing some tests in autograd. I keep getting the following error using the following codes. Could someone explain why the function fill_() not working with GPU?</p>
<p>Code:</p>
<blockquote>
<p>import torch<br>
xx = torch.Tensor([1]).cuda()<br>
xx.requires_grad = True<br>
b = torch.Tensor(2).fill_(xx[0]).cuda()<br>
c = sum(b)<br>
c.backward()<br>
xx.grad</p>
</blockquote>
<p>Error:</p>
<blockquote>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
4 b = torch.Tensor(2).fill_(xx[0]).cuda()<br>
5 c = sum(b)<br>
----&gt; 6 c.backward()<br>
7 xx.grad</p>
<p>1 frames<br>
/usr/local/lib/python3.6/dist-packages/torch/autograd/<strong>init</strong>.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)<br>
91     Variable._execution_engine.run_backward(<br>
92         tensors, grad_tensors, retain_graph, create_graph,<br>
—&gt; 93         allow_unreachable=True)  # allow_unreachable flag<br>
94<br>
95</p>
<p>RuntimeError: Function FillBackward1 returned an invalid gradient at index 1 - expected type torch.cuda.FloatTensor but got torch.FloatTensor</p>
</blockquote>

kikiloverafa: <aside class="quote no-group" data-username="kikiloverafa" data-post="1" data-topic="52715">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kikiloverafa/48/13012_2.png" class="avatar"> kikiloverafa:</div>
<blockquote>
<p>b = torch.Tensor(2).fill_(xx[0]).cuda()</p>
</blockquote>
</aside>
<p>Actually I figured it out. need to change</p>
<blockquote>
<p>b = torch.Tensor(2).fill_(xx[0]).cuda()<br>
into</p>
</blockquote>
<blockquote>
<p>b = torch.Tensor(2).cuda().fill_(xx[0])</p>
</blockquote>
How do I customize my module's behavior for train and eval?

ibraheemmoosa: <p>Dropout and Batch-Norm already have this kind of behavior. In other words what a Dropout or Batch-Norm module outputs when it is in eval mode is different from when it is in train mode. How can I achieve something similar with my custom module?</p>

ptrblck: <p>You could use the internal <code>self.training</code> attribute.<br>
Here is a dummy example:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc = nn.Linear(10, 10)
        
    def forward(self, x):
        x = self.fc(x)
        if self.training:
            x = x * 1000
        return x

model = MyModel()
x = torch.randn(1, 10)

output = model(x)
print(output)
&gt; tensor([[ -151.6117,    20.6451,  -589.1161,  -120.6478,   395.1652,  -950.3046,
         -1062.1073,   973.9295,    61.4954,  -412.5521]],
       grad_fn=&lt;MulBackward0&gt;)

model.eval()
output = model(x)
print(output)
&gt; tensor([[-0.1516,  0.0206, -0.5891, -0.1206,  0.3952, -0.9503, -1.0621,  0.9739,
          0.0615, -0.4126]], grad_fn=&lt;AddmmBackward&gt;)
</code></pre>

ibraheemmoosa: <p>Thank you. That seems very nice and simple.</p>
Recursive @torch.jit.script export to ONNX

dashesy: <p>I have a <a href="https://github.com/microsoft/pytorch_od/blob/master/mictorch/softmaxtree.py" rel="nofollow noopener">SoftmaxTree implementation in C++ extension</a>, that I cannot convert to ONNX. I can change the implementation to use recursive jit script, but before that I wanted to see if then it can be exported to ONNX.</p>
<p>I have tried simple <code>@torch.jit.script</code> with an if-statement can be exported and results in an <a href="https://github.com/onnx/onnx/blob/master/docs/Operators.md#If" rel="nofollow noopener">if node</a> in hte graph. But anything more complicated I am not sure.</p>

driazati: <p>It depends on your code, the TorchScript compiler always makes the same type of graph, and there are various limitations on what can be exported to ONNX. Looking at the link you posted, TorchScript currently doesn’t support custom autograd <code>Functions</code>, but this is something we are working on supporting.</p>
Char-level seq2seq translation (eng-fra)

Robin888: <p>Hi,</p>
<p>I am new to pytorch and want to implement a pytorch version of below seq2seq model (char-level translation, a keras example):</p>
<p><a href="https://keras.io/examples/lstm_seq2seq/" class="onebox" target="_blank" rel="nofollow noopener">https://keras.io/examples/lstm_seq2seq/</a></p>
<p>I defined the encoder and decoder but does not know how to continue, anyone can help or is there an online implementation? Many thanks.</p>
<pre><code class="lang-auto">
class encoder(nn.Module):
  def __init__(self):
    super(encoder,self).__init__()
    self.LSTM=nn.LSTM(input_size=num_encoder_tokens,hidden_size=256,batch_first=True)
    
  def forward(self,x):
    out,(h,c)=self.LSTM(x)
    return h,c
  
class decoder(nn.Module):
  def __init__(self):
    super(decoder,self).__init__()
    self.LSTM=nn.LSTM(input_size=num_decoder_tokens,hidden_size=256,batch_first=True)
    self.FC=nn.Linear(256,num_decoder_tokens)
   
  def forward(self,x,hidden_states):
    out,(h,c)=self.LSTM(x,hidden_states)
    out=self.FC(out[:, -1, :])
    return out,(h,c)
</code></pre>
nn.LSTMCell example in the documentation

Ammola: <p>Hi everyone,<br>
Excuse me If you find my question very intuitive because I’m still new to Pytorch.<br>
I have some confusion about torch.nn.LSTMCell Class.<br>
According to the documentation, the inputs to the LSTM cell as follows: input and (h_0, c_0), where input is of shape (batch, input_size) which is a tensor containing input features.<br>
The problem is that I can’t understand why in the provided example the given input has different dimensions, instead of being:<br>
input(batch,input_size)<br>
the given input is:<br>
input(seq_len,input_size)<br>
As long as I understand, the number 6 in the example represents the batch size.<br>
Here is the example used in the documentation:</p>
<pre><code class="lang-auto">rnn = nn.LSTMCell(10, 20)
&gt;&gt;&gt; input = torch.randn(6, 3, 10)
&gt;&gt;&gt; hx = torch.randn(3, 20)
&gt;&gt;&gt; cx = torch.randn(3, 20)
&gt;&gt;&gt; output = []
&gt;&gt;&gt; for i in range(6):
        hx, cx = rnn(input[i], (hx, cx))
        output.append(hx)
</code></pre>
Controlling memory usage when training many models

ga1ois: <p>I have an application where I am training and discarding a large number of models (neural architecture search). After running for a while, I begin to get GPU OOMs. I have not completely ruled out the idea that the models are getting excessively large, but based on what I’ve been able to debug I do not think this is the issue (especially since the models are not particularly large…). My guess is that each model allocates a chunk of the GPU’s RAM and then does not release the memory when it is destroyed (or “relases” it by PyTorch doesn’t actually free up the memory). My question is: is there a simple way to make sure the allocate memory is released?</p>
PyTorch tensor to List of Lists

ssheshap: <p>I have a PyTorch tensor of shape ((1,512,16,3)).<br>
The (16,3) part of the tensor represents 16 - (x,y,z) points.<br>
These 16 points could be unique points or repeated.<br>
I want to convert it to a list of 512 lists only with unique points.<br>
Dimensions of the resulting list of lists (1,512,p,3) ==&gt; 1 list which has 512 lists each of varying “p” size which in turn contain a list of 3 points.</p>
<p>So far I tried using two for loops to fetch the (16,3) points and use torch.unique to pick the unique points. But 2 for loops make my code slower.</p>
<p>Please let me know if there is a function to do it in PyTorch.</p>
<p>Thanks in advance.</p>

ptrblck: <p>If I understand the use case correctly, <code>p</code> won’t be a fixed number, but is varying for each of the <code>512</code> entries, e.g.:</p>
<pre><code class="lang-python">x = torch.randn(1, 512, 16, 3)
# your operation
result = ...
print(result[0, 0].shape) # [4, 3]
print(result[0, 1].shape) # [7, 3]
...
</code></pre>
<p>If that’s the case, you won’t be able to create the result as a tensor without padding.</p>
<p>Let me know, if I misunderstood the use case.</p>

ssheshap: <p>You understood it correct.<br>
Thanks a lot. I am using padding now.</p>
Out of GPU Memory during test, Model ouput (eval) & require_grad = True

Pirate-Roberts: <p>Hi everyone!</p>
<p>I have several questions for you:</p>
<p>I’m new with pytorch and I’m trying to perform a test on my NN model with JupyterLab and there is something strange happening. It seems to require the same GPU memory capacity as training (for a same input size and a batch size of 1 for the training). So I think it could be due to the gradient maps that are saved during the back propagation (I can see that the GPU memory explodes just after I run the output = model(input) line). Do you know a way to see if the gradient maps are actually computed and saved?</p>
<p>If I do<br>
<code>my_model.eval()</code></p>
<p>to avoid the back propagation, do I also need to do something like</p>
<pre><code class="lang-auto">with torch.no_grad:
       model_input = some_tensor
</code></pre>
<p>or</p>
<pre><code class="lang-auto">with torch.no_grad:
       model_output = my_model(model_input)
</code></pre>
<p>?</p>
<p>And If I do something like this:</p>
<pre><code class="lang-auto">my_model.eval()
with torch.no_grad:
      model_output = my_model(model_input)
</code></pre>
<p>Is <code>model_ouput.requires_grad</code> supposed to be <code>False</code> ?</p>
<p>I’m working with pytorch 0.4.0 on Windows 10 and I have a 6Gb GPU.</p>
<p>Thank you!</p>

albanD: <p>Hi,</p>
<p>You have some more informations <a href="https://discuss.pytorch.org/t/model-eval-vs-with-torch-no-grad/19615" class="inline-onebox">'model.eval()' vs 'with torch.no_grad()'</a>) .<br>
Let me know if you have more questions.</p>

Pirate-Roberts: <p>Hi,</p>
<p>Thank you, it helped and I finally figured out my problem!</p>
Seq2seq LSTM with inputs conditioned on last output

Xema: <p>I am trying to implement models from the paper <a href="https://github.com/mansimov/unsupervised-videos" rel="noopener nofollow ugc">Unsupervised Learning of Video Representations using LSTMs</a>.</p>
<p>In particular, the authors describe a encoder-predictor LSTM model (see image below) where the sequence is first encoded and the next steps of the sequence are then predicted. When doing the predictions, at each time step they feed the input either the ground truth of the last step (training time) or the predicted output of the last step (testing time).</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/393c2b6b3d780f1d7b4c92cd33af6f89838b96ad.png" alt="image" data-base62-sha1="8ak7YkYHlocautwJWqAt4a63VOB" width="349" height="197"></p>
<p>While it seems straightforward to implement the training stage, I am stuck with the testing stage. How to dynamicaly feed the output of a step to the input of the next step? I could do a for loop and computing the steps by hand one after the other, but it does not seem very sweet.</p>
<p>What do you think?</p>

Xema: <p>What I wanted to do is, in fact, called “teacher forcing”. An official <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="nofollow noopener">PyTorch tutorial</a> covers this topic.</p>
Pytorch 1.1.0 cpu usage is much higher than 1.0.1

yuval: <p>I am using a code that consumes much more CPU resources on 1.0.1 than 1.1.0, while keeping the GPU usage at ~40% and same runtime on both.<br>
Any idea why, where and what to look in order to make it consume less resources on 1.1.0?</p>
<p><strong>In detail:</strong></p>
<p>Originally the code was developed for 0.3.1 (<a href="https://github.com/akuzeee/AFLAC/" rel="nofollow noopener">link</a>), and I upgraded it to 0.4.1 according to the migration guide.</p>
<p>Specifically:<br>
(1) I changed all “volatile” usages to “with torch.nograd()”<br>
(2) replaced “.data[0]” to “.item()”<br>
(3) replaced all “.cuda()” to “.to(device)”.</p>
<p>The exact same code uses much more CPU resources on 0.4.1 or 1.0.1 vs 1.1.0<br>
pytorch was installed on anacoda with the following command.<br>
for v1.1.0: <code>conda install pytorch==1.1.0 torchvision cudatoolkit=10.0 -c pytorch</code><br>
for v1.0.1: <code>conda install pytorch==1.0.1 torchvision cudatoolkit=10.0 -c pytorch</code><br>
for v0.4.1: <code>conda install pytorch==0.4.1 torchvision cudatoolkit=10.0 -c pytorch</code></p>
<p>I am not the one who developed the code, and I am new to pytorch (but experienced with TF). So I am unsure on how to debug this problem, what to look, or how to create a minimal example to reproduce it. I’ll be thankful for any guidance here.</p>
<p><strong>EDIT</strong></p>
<ol>
<li>
<p>Following the advice on <a href="https://github.com/pytorch/pytorch/issues/20311" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/20311</a>, setting <code>OMP_NUM_THREADS=1</code> for 1.1.0 reduces CPU usage, but makes train time slower by 7.5% compared to 1.0.1 (and 16% compared to 0.4.1)</p>
</li>
<li>
<p>For the same code, the total runtime of pytorch 1.0.1 is slower than 0.4.1 by ~8% [notes: (a) in this bullet, I refer to 1.0.1, not 1.1.0, (b) I repeated this measure 3 times and it was consistent, with low variance]</p>
</li>
</ol>
Backprop in a network with two heads

D0miH: <p>I am currently trying to understand the <a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py" rel="nofollow noopener">actor critic example</a> for the cart-pole environment.</p>
<p>I understand the general principle and how the algorithm works. However in this code we have a neural net with two heads. One output for our actions and another one for our predicted future reward based on the current state.</p>
<p>My question is about these few lines of code:</p>
<pre><code class="lang-python"># reset gradients
    optimizer.zero_grad()

    # sum up all the values of policy_losses and value_losses
    loss = torch.stack(policy_losses).sum() + torch.stack(value_losses).sum()

    # perform backprop
    loss.backward()
    optimizer.step()

    # reset rewards and action buffer
    del model.rewards[:]
    del model.saved_actions[:]
</code></pre>
<p>Here we are adding the policy loss and the value loss. But as i understood it, we should call backward once for the loss of the policy and once for the value loss? Why is it sufficient here to just add them up?</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Boazrciasn: <p>If you add both losses, gradient magnitude and direction for weights do not change when compared to doing backwards twice for both loss values. Mathematically resulting gradients will be the same in both cases thus taking derivatives once is more efficient.<br>
From another point of view is that your network can not optimize two loss functions separately, in your case you only have one loss function with two sub loss functions which is policy_losses+value_losses.</p>
<p>Hope this helps.<br>
Best.</p>

D0miH: <p>Yes, your explanation makes absolutely sense.</p>
<p>Thank you very much</p>
Initialize weights of convolution layer

zahra: <p>Hi,</p>
<p>I want to assign filter = [[[[1, 2, 3], [4, 5, 6], [7, 8, 9]]]]<br>
as weight of convolution layer. Is this command ok?</p>
<pre><code class="lang-auto">self.conv1 = conv2d(...)
self.conv1.weight = Parameter(filter)
</code></pre>
<p>Thanks</p>

ptrblck: <p>I would recommend to wrap the assignment in a <code>with torch.no_grad()</code> block just to make sure Autograd won’t complain later.</p>

zahra: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="52672">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>with torch.no_grad()</p>
</blockquote>
</aside>
<p>Thanks a lot.<br>
Is it ok?</p>
<pre><code class="lang-auto">class Net(Module):
    def __init__(self):
        super(Net, self).__init__()	
        self.convZ = nn.Conv2d(...)
        with torch.no_grad():
             self.convZ.weight = Parameter(filter)        
        self.pool = nn.AvgPool2d(...)
        self.fc1 = nn.Linear(....)

    def forward(self, x):
        out = torch.atan(self.convZ(x))
        out = self.pool(out)
        out = torch.atan(self.fc1(out))
        return out

</code></pre>
<p>Best Regards</p>

ptrblck: <p>Looks alright, although I wouldn’t assign <code>output_after_convZ</code> back to <code>self.convZ</code>, but rather a new attribute.</p>

zahra: <p>Yes, you are right. It is  just a mistake. I edited it.<br>
Many thanks, you are the best <img src="https://discuss.pytorch.org/images/emoji/apple/shamrock.png?v=9" title=":shamrock:" class="emoji" alt=":shamrock:"></p>
Cross Entropy and Binary Cross Entropy do not give the same result

Haydnspass: <p>Hey,<br>
Until now I used Binary Cross entropy loss but since I need to use some other loss function I need to change my output so that it conforms to the Cross Entropy format. The result should be exactly the same, right?</p>
<p>When I tried a fake / handcrafted example I do not get the same results for both of the loss functions, probably I am just overseeing something …<br>
Suppose in binary format my ground truth is [1., 0.] and my prediction [0.5, 0], then in normal CE format my ground truth should be the same (but with integers since those are now class indices, so [1 0]). My prediction however should be for the first element: [0.5, 0.5] and for the second element [1.0, 0.0] right?</p>
<p>Below you can find it in a jupyter notebook.</p>
<p>Cheers,<br>
Lucas</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f1d037b95e41eace989fea2ed1956f53cdccdb29.png" data-download-href="https://discuss.pytorch.org/uploads/default/f1d037b95e41eace989fea2ed1956f53cdccdb29" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f1d037b95e41eace989fea2ed1956f53cdccdb29_2_472x500.png" alt="image" data-base62-sha1="yvbaPtZtyRQabwiQcd5gPTtgcAh" width="472" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f1d037b95e41eace989fea2ed1956f53cdccdb29_2_472x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f1d037b95e41eace989fea2ed1956f53cdccdb29_2_708x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f1d037b95e41eace989fea2ed1956f53cdccdb29_2_944x1000.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1063×1126 123 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

InnovArul: <p>Remember that cross entropy loss = ‘log softmax’ + ‘negative log likelihood’.</p>
<p>Hence,  you may need to set<br>
<code>prediction[1,1]=float("-inf")</code></p>
BERT positional embeddings

Philipp_Friebertshau: <p>Hi,</p>
<p>I just embedded the BERT positional embeddings into the 2D space (with umap) for different BERT models that are trained on different languages (I use “pytorch_transformers”).</p>
<p>It’s obvious that the embedded positional embeddings for the german model ist way more unstructred than for the other language models. Why is that? Is the german model not that well trained?</p>
<p>I also checked the word embeddings, these are looking good.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e11d11d6607932305ff96f9435ed07ebed285d82.png" data-download-href="https://discuss.pytorch.org/uploads/default/e11d11d6607932305ff96f9435ed07ebed285d82" title="posi_emb_lang"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e11d11d6607932305ff96f9435ed07ebed285d82_2_690x493.png" alt="posi_emb_lang" data-base62-sha1="w7rIagni2hCW5acW8URSG2IW2hI" width="690" height="493" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e11d11d6607932305ff96f9435ed07ebed285d82_2_690x493.png, https://discuss.pytorch.org/uploads/default/optimized/2X/e/e11d11d6607932305ff96f9435ed07ebed285d82_2_1035x739.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/e/e11d11d6607932305ff96f9435ed07ebed285d82_2_1380x986.png 2x" data-dominant-color="EDEEEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">posi_emb_lang</span><span class="informations">2940×2104 1.05 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Why max_memory_allocated() after reset doesn't return 0?

sublee: <p>I’m trying to measure the memory usage of each layer by <code>torch.cuda.max_memory_allocated()</code> and <code>torch.cuda.reset_max_memory_allocated()</code>:</p>
<pre><code class="lang-python">for layer in sequential_module:
    torch.cuda.reset_max_memory_allocated(0)
    x = layer(x)
    size = torch.cuda.max_memory_allocated(0)
    sizes.append(size)
</code></pre>
<p>I understood <code>torch.cuda.reset_max_memory_allocated()</code> can be used to reset the starting point to track the maximum memory allocations. So I’ve expected that <code>torch.cuda.max_memory_allocated()</code> after reset always returns 0. But it doesn’t.</p>
<pre><code class="lang-python">In [1]: import torch

In [2]: torch.cuda.max_memory_allocated(0)
Out[2]: 0

In [3]: x = torch.rand(100, 100, 100, device=0)

In [4]: torch.cuda.max_memory_allocated(0)
Out[4]: 4865536

In [5]: torch.cuda.reset_max_memory_allocated(0)

In [6]: torch.cuda.max_memory_allocated(0)
Out[6]: 4865536
</code></pre>
<p>How do I understand this behavior?</p>

ptrblck: <p>Since you are not freeing any memory, <code>torch.cuda.max_memory_allocated</code> will still return the currently used memory, as it’s still the peak.<br>
Have a look at this code snippet:</p>
<pre><code class="lang-python"># Check for empty
torch.cuda.max_memory_allocated(0)

# Create one tensor
x = torch.rand(100, 100, 100, device='cuda:0')
# should yield the same value
torch.cuda.memory_allocated(0)
torch.cuda.max_memory_allocated(0)

# Create other tensor
y = torch.rand(100, 100, 100, device='cuda:0')
# Should be same, but higher
torch.cuda.memory_allocated(0)
torch.cuda.max_memory_allocated(0)

# Delete one tensor
del y
# max_memory_allocated should keep it's old value
torch.cuda.memory_allocated(0)
torch.cuda.max_memory_allocated(0)

# Reset to track new peak memory usage
torch.cuda.reset_max_memory_allocated(0)
torch.cuda.memory_allocated(0)
torch.cuda.max_memory_allocated(0)
</code></pre>

sublee: <p>That’s clear explanation and code snippet. Thank you so much!</p>
How to choose distributions for multi-dimensional output?

Xiaodong_MEI: <p>As for my reinforcement learning network, the dimension of “action” is 2.<br>
I have to chose a distribution for it, then sample from the distribution to choose action</p>
<p>Can I just  choose to generate two seperate distributions for it, or have to choose other distributions, such as GMM or multivariat normal?</p>
Neural Network doesn't learn anything in regression problem

domi: <p>I am playing with a simple problem of regression with neural network. Here is the function to generate the data:</p>
<pre><code class="lang-auto">def generate_data():
    while(1):
        x,y,z = [random.uniform(0,100) for _ in range(3)]
        gt = (x**3 + math.log(y)) * math.cos(z)
        yield x,y,z,gt
</code></pre>
<p>I generated 36000 data points for this problem. To solve this problem, I proposed this network:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, ):
        super(Net, self).__init__()
        self.l1 = nn.Sequential(nn.Linear(3,500), nn.ReLU())
        self.l2 = nn.Sequential(nn.Linear(500,300), nn.ReLU())
        self.l3 = nn.Sequential(nn.Linear(300,100), nn.ReLU())
        self.l4 = nn.Linear(100,1)

    def forward(self, x):
        x = self.l1(x)
        return self.l4(self.l3(self.l2(x)))
</code></pre>
<p>The settings for training as below:</p>
<pre><code class="lang-auto">net = Net()
loss_fn = nn.MSELoss()
optimizer = optim.SGD(net.parameters(), lr = 0.0001)
### some code for dataset and dataloader here

for _ in range(3000):
    loss_avg = 0
    for x, gt in dataloader:
        pred = net(x)
        loss = loss_fn(pred, gt)
        loss_avg += loss/1000.0
        c+=1
        loss.backward()
        optimizer.step()
    print(f"Loss : {loss_avg}")
</code></pre>
<p>However, my model doesn’t learn anything. The loss doesn’t reduce but bounce up and down. What I already tried:</p>
<ol>
<li>Change the learning rate =&gt; doesn’t work</li>
<li>Add the norm layers after RELU =&gt; loss quickly grows up to inf.</li>
<li>Divided the input by its max value, doesn’t work either.</li>
<li>Normalize both input and output =&gt; loss up and down, didn’t reduce.</li>
</ol>
<p>Could you help me point down what I did wrong? Or any suggestions to debug or hints?</p>

Link88: <p>I think the problem is that you are accumulating the gradients. You need to set them to zero after each gradient update. Try just adding before loss.backward, optimizer.zero_grad() .</p>

domi: <p>Thanks <a class="mention" href="/u/link88">@Link88</a>. Indeed you pointed out what I missed on the pipeline. I put back it into my code. The problem still remains as before. I am still looking for a robust solution (and trying different ways) so if you could find anything else, please let me know.</p>
How to change the VGG19 network in an existing program into a self-defined training network when performing convolutional neural network visualization training

zhao_jing: <p>How to change the VGG19 network in an existing program into a self-defined training network when performing convolutional neural network visualization training<br>
code show as below</p>
<h1>定义一个可视化的函数</h1>
<p>def deepcamvis(imagepath):<br>
import matplotlib.pyplot as plt<br>
import numpy as np<br>
import cv2<br>
from keras import backend as K<br>
from keras.applications.vgg19 import VGG19, preprocess_input, decode_predictions<br>
from keras.preprocessing import image</p>
<pre><code># 导入模型
model = VGG19(weights="imagenet")
# 读取一张待判断的图像
im = image.load_img(imagepath, target_size=(224, 224))
# 转化为数组
imx = image.img_to_array(im)
# 添加一个纬度，得到1*224*224*3的矩阵
imx = np.expand_dims(imx, axis=0)
# 对图像进行预处理，即标准化
imx = preprocess_input(imx)
# 使用VGG19网络进行预测图像的类别
impre = model.predict(imx)
# 对预测的结果进行解码
imprename = decode_predictions(impre, top=5)
# 最有可能的
print(imprename)
# 计算图像所在位置的编码
impreindex = np.argmax(impre)
# 可视化类激活图像
# 计算预测向量种种的老虎元素
impre_output = model.output[:, impreindex]
## 输出的卷基层特征 block5_conv4 (Conv2D)
conv_femap = model.get_layer("block5_conv4")
## 输出特征图的梯度
grads = K.gradients(impre_output, conv_femap.output)[0]
## 计算每个元素在特定通道的梯度平均大小,shape=(512,)的向量
conv_grads = K.mean(grads, axis=(0, 1, 2))
## 访问刚定义变量的数值
iterate = K.function([model.input], [conv_grads, conv_femap.output[0]])
conv_grads_val, conv_femap_val = iterate([imx])
## 将特征图的每个通道乘以类别重要程度
for ii in range(conv_femap_val.shape[2]):
    conv_femap_val[:, :, ii] = conv_femap_val[:, :, ii] * conv_grads_val[ii]
## 计算热力图
heatmap = np.mean(conv_femap_val, axis=-1)
## 对热力图进行标准化，数值到0～1之间
heatmap = np.maximum(heatmap, 0)
heatmap = heatmap / np.max(heatmap)
## 将生成的热力图叠加到原始图像上
imag = cv2.imread(imagepath)
heatmap = np.uint8(cv2.resize(heatmap, (imag.shape[1], imag.shape[0])) * 255)
## 生成伪彩色热力图
heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)
## 新的图像
imagnew = np.uint8(heatmap * 0.45 + imag)
## OpenCV image to Matplotlib
imagnewrgb = imagnew[..., ::-1]
plt.figure()
plt.imshow(imagnewrgb)
plt.show()
return imagnewrgb</code></pre>
Video frame recommendation

111127: <p>Hi!<br>
Now I’m working on video understanding<br>
In my work, My goal is video frame recommendation in certain videos.<br>
For example, inputs are 64 frames in each videos which are 8.<br>
So my inputs are [64 frames x 8 videos].<br>
Ground Truth labels are like this form. e.g. [0, 0, 0, 0, … 1, 1, 1, 1].<br>
0 means first video, 1 means second videos.<br>
Neural Net recommends video frames. Generated outputs like this form [batch_size, 64, 8], 64 means number of frames, 8 means number of input videos.<br>
After that, I want to measure a loss with labels.<br>
But I can’t use Cross-Entropy loss in my works.<br>
How can I use Cross-Entropy loss in my task?</p>
Convert output to binary values

shadowhy: <blockquote>
<p>a = torch.randn(2,3,2,2, requires_grad=True)    <span class="hashtag-raw">#i</span>.e.(batch,chnl,h,w)<br>
amax = torch.argmax(a, dim=1)</p>
</blockquote>
<p>print(a) : <img src="https://discuss.pytorch.org/uploads/default/original/2X/c/cd27b5b712e8a3c9ac12541e39c8a45c1b741ca8.png" alt="image" data-base62-sha1="tgSXtJLLSTEmEDDg1OV0rKXrmis" width="418" height="289"><br>
print(amax): <img src="https://discuss.pytorch.org/uploads/default/original/2X/3/3cba3a8838d8f8111ac3bcceaa784308ac7a8203.png" alt="image" data-base62-sha1="8FdDKXUjPSvkRi1916a2MqwyIQb" width="148" height="83"></p>
<p>My aim is to convert the results to binary values: such that value at max index (along dim=1) to be 1 and others to be 0.<br>
For example, based on the above, i want to get this: <img src="https://discuss.pytorch.org/uploads/default/original/2X/9/9ea589f2a77bd44287efb70aae0d04f809535ce3.png" alt="image" data-base62-sha1="mDs6yjm0ymJ0hcAM4kz5mPCyDyr" width="305" height="286"></p>
<p>How do i achieve that?<br>
Somewhat felt i am missing on a simple way to do it, but i just couldn’t figure out.</p>

AnBucquet: <p>Hi <a class="mention" href="/u/shadowhy">@shadowhy</a> ,</p>
<p>You can try this:</p>
<pre><code class="lang-auto"> output = (a == torch.max(a, 1 , keepdim=True)[0])
</code></pre>
<p>Let me know if it’s not what you’re looking for.</p>
F1-score Error for MultiLabel Classification

chrome: <p>I am trying to calculate the F1-score of a multilabel classification problem using <code>sklearn.metrics.f1_score</code> but I am getting the error <code>UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples. 'precision', 'predicted', average, warn_for)</code>. My <code>y_pred</code> was  probability values and I converted them using <code>y_pred &gt; 0.5</code>.<br>
<code>y_pred[0]</code>   is <code>tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=torch.uint8)</code> and</p>
<p><code>y_ground_truth[0]</code>   is <code>tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])</code></p>
<p>The error is occuring at:</p>
<pre><code class="lang-auto">f1_score(y_ground_truth.to('cpu'), y_pred.to('cpu'), average='samples')
</code></pre>
<p>I am using <code>nn.BCELoss()</code><br>
I went through <a href="https://discuss.pytorch.org/t/calculating-precision-recall-and-f1-score-in-case-of-multi-label-classification/28265/3" class="inline-onebox">Calculating Precision, Recall and F1 score in case of multi label classification</a> but it didn’t help.</p>

Aiman_Mutasem-bellh: <p>Try this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">target_   = y_ground_truth.data.cpu().numpy()
output_  = y_pred.data.cpu().numpy()
f1_score(output_  , target_   , average='samples')
</code></pre>
C++ using JIT but we have four tensor return problem?

lingtengQIu: <p>because my model coming from refinedet net<br>
so the return have four tensor.<br>
when I return the result to tensor ,I meeted some problem like follow:<br>
auto input = torch::randn({1,3,512,512});<br>
auto flag = torch::tensor(true);<br>
auto output = model-&gt;forward({input,flag}).toTensor();<br>
std::cout&lt;&lt;output&lt;&lt;std::endl;</p>
<p>terminate called after throwing an instance of ‘c10::Error’<br>
what():  isTensor() ASSERT FAILED at /home/qlt/LibTorch/include/ATen/core/ivalue.h:182, please report a bug to PyTorch. (toTensor at /home/qlt/LibTorch/include/ATen/core/ivalue.h:182)<br>
frame <span class="hashtag">#0:</span> std::function&lt;std::string ()&gt;::operator()() const + 0x11 (0x7f06fd782fe1 in /home/qlt/LibTorch/lib/libc10.so)<br>
frame <span class="hashtag">#1:</span> c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x2a (0x7f06fd782dfa in /home/qlt/LibTorch/lib/libc10.so)<br>
frame <span class="hashtag">#2:</span> c10::IValue::toTensor() &amp;&amp; + 0xa8 (0x443c2a in ./build/launch)<br>
frame <span class="hashtag">#3:</span> main + 0x452 (0x440788 in ./build/launch)<br>
frame <span class="hashtag">#4:</span> __libc_start_main + 0xf0 (0x7f06fb5e9830 in /lib/x86_64-linux-gnu/libc.so.6)<br>
frame <span class="hashtag">#5:</span> _start + 0x29 (0x43ee09 in ./build/launch)<br>
how can i deal with this more tensor return question?</p>

justusschock: <p><a class="mention" href="/u/smth">@smth</a> told me yesterday, that there have been some fixes which recently went into master. Using the master from github fixed this issue for me (the latest nightly would probably also work).</p>

lingtengQIu: <p>OK,I Find a method to deal with this problem.<br>
fisrt convert Ivalue to tuple<br>
and get element can deal with this problem.<br>
auto tpl = output.toTuple();<br>
auto arm_loc = tpl-&gt;elements()[0].toTensor();<br>
auto arm_conf = tpl-&gt;elements()[1].toTensor();<br>
auto odm_loc = tpl-&gt;elements()[2].toTensor();<br>
auto odm_conf = tpl-&gt;elements()[3].toTensor();</p>

smth: <p><a class="mention" href="/u/lingtengqiu">@lingtengQIu</a> if you update to latest Preview build, we have fixed this bug.</p>
Using different modules for a condition

Eric_Taw1: <p>I have a bunch of values I’d like to predict, and each value belongs to a category that has its own distribution of values. Instead of creating a new model per category, I’d like to incorporate weight sharing, then have the model apply a different fully connected layer based on the category. What’s the most efficient way to implement this?</p>

ptrblck: <p>Could you explain your use case a bit?<br>
How would you select the category? Do you have this information for each sample, i.e. training and test samples?</p>

Eric_Taw1: <p>Sure. For context, this has to do with the CHAMPS competition on Kaggle.</p>
<p>I have a tensor of shape NxF, where N is the number of nodes in a graph and F is the length of features. Per graph, I have a set of attributes that I am regressing against, the scalar coupling constant. Each attribute involves a pair of nodes, and I am given the type of coupling for each attribute I am trying to predict. Ideally, I’d like to have a model that does convolutions on the entire graph, then use different fully connected networks per coupling type. The only weight sharing across coupling types is the convolution/pooling part.</p>
<p>Currently, I am doing this by concatenating every combination of 2 nodes possible, forming an NxNxF tensor, then passing that through a fully connected network that outputs 8 values, one for every coupling type (now NxNx8). Then, since I have the indices of each pair of nodes to train on, I can index that tensor with <code>torch.take</code> and get the right values. However, I’m wondering if there’s a more efficient way to do this. Doing a pairwise concatenation of every node is very memory intensive.</p>
<p>Do you have any suggestions? Thanks!</p>
Batches of different distributions

Andreas_Georgiou: <p>Let’s say that I have a fixed dataset of length N. I want to apply a distribution over this N samples and then sample from this distribution to create mini-batches. I want to continuously change the distribution over my data. One option is to change it after each epoch. This can be easily done by just generating new probabilities for each sample every epoch and then my <strong>getitem</strong> will just use this distribution to sample an item. However, what I want is to change the distribution after each mini-batch. Is there a way I can do that so that I don’t lose the nice parallelization and multi-threading DataLoader offers? I am not sure where to put the code that generates the probabilities over each sample. Any ideas?</p>
Backprop through weights of a second network

MrSirLRD: <p>Hi, my question might be bit odd but i’m hoping it is possible with pytorch<br>
Say i have two networks, the first of which outputs a bunch of values, one for each parameter of the second network (say a classification network) i then add these values to the parameters of the second network and perform a forward pass of the second network and compute a loss, is it possible to backprop the loss through to the first network to perform an optimization step?<br>
Thanks</p>

Ran: <p>Yes, the first network can be optimized by the loss.</p>

MrSirLRD: <p>Ok, how would you do it?<br>
I’m currently looping through the parameters of the second network and adding on the output of the first, but when I backprop the loss and step the optimizer (which has the parameters of the first network) the parameters of the first network don’t change</p>

Ran: <p><img src="https://discuss.pytorch.org/uploads/default/original/2X/9/93310f823879567a1e82cda89e6d172f0f6f695d.jpeg" alt="2019-08-05_192621" data-base62-sha1="l07iB4bfuu5WGXrRrOzXCUhtkol" width="456" height="142"></p>
<p>Hope it helps you out.</p>

MrSirLRD: <p>Hi Ran,<br>
Thanks for your response,<br>
unfortunately thats not quite what i’m after, i know you can backprop through the input of the second network, but what i’d like to do is to add the output of the first to the weights and bias of the second and then forward pass a separate input to the second network. see example below</p>
<pre><code class="lang-auto">def add_weights(weights, network):
    c = 0
    for p in network.parameters():
        num_weights = p.data.view(-1).shape[0]
        params = weights[c:c+num_weights].view(p.data.shape)
        c +=num_weights
        p.data += params
    return network

def SR_loss(network, images, labels, params):
    network = add_weights(params, network)
    pred_lables = network(images.to(device))
    loss = criterion(pred_lables, lables.to(device))
    return loss, pred_lables

params_fc, _ = SR_C_net(input)
loss, pred_lables = SR_loss(C_net, image, lables, params_fc)
</code></pre>
<p>i can see that simply adding the output to p.data won’t allow for backprop to the first network, but i’m not sure how it can be done.</p>
How to not use shared memory in PyTorch multiprocessing

HMEIatJHU: <p>When I use multiprocessing on a remote server, I got some error messages like:</p>
<pre><code class="lang-python">File "/job/.local/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 315, in reduce storage
fd, size = storage._share_fd_()
RuntimeError: unable to write to file &lt;/torch_1_3660435083&gt;
</code></pre>
<p>What works: 1) run it locally with same # of processes/workers; 2) run it on the server with fewer processes/workers.<br>
This seems that shared memory is not enough.</p>
<p>Since I use ‘spawn’ method to initialize a process, I actually did not use any <code>model.share_memory()</code>.</p>
<p>What might be the problem and solution?<br>
Anything else that I should do to make sure I do not use shared memory?</p>
<p>(I can not set the shared memory size on this remote server…)</p>
<p>Thanks very much in advance!</p>
Deeplabv3 resnet wrong input size error

Nishanth_Sasankan: <p>I am using the deeplabv3+ resnet 101  to perform semantic segmentation. I was using the FCN_resnet101 model and everything was working, but when switching the model to deeplabv3_resnet101 gives me this following error</p>
<pre><code class="lang-auto">File "train.py", line 45, in &lt;module&gt;
    outt = model(i)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torchvision/models/segmentation/_utils.py", line 22, in forward
    x = self.classifier(x)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torchvision/models/segmentation/deeplabv3.py", line 91, in forward
    res.append(conv(x))
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torchvision/models/segmentation/deeplabv3.py", line 60, in forward
    x = super(ASPPPooling, self).forward(x)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 83, in forward
    exponential_average_factor, self.eps)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 1693, in batch_norm
    raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))
ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 256, 1, 1])
</code></pre>
<p>my input size is actually [1 3 417 417].</p>
<p>Any and all help is appreciated</p>

ptrblck: <p>Try to pass more than a single sample to your model while training.<br>
The batch norm layer complains, since the activation only contains a single pixel and a single sample.</p>
Numerical floating difference between Conv with kernel size 1 and Linear

dem123456789: <pre><code class="lang-auto">batch_size = 3
num_layers = 2
in_channels = 10
out_channels = 20
kernel_size = 1
stride = 1
padding = 0
dilation = 1
sharing_rates = [0, 0]
bias = True
input = torch.randn(batch_size, in_channels, 1, 1).to(device)
o = nn.Conv2d(in_channels,out_channels,kernel_size,stride,padding,dilation,1,bias).to(device)
output = o(input)
g = nn.Linear(in_channels,out_channels,bias).to(device)
g._parameters['weight'].data.copy_(o._parameters['weight'].squeeze().data)
g._parameters['bias'].data.copy_(o._parameters['bias'].squeeze().data)
goutput = g(input.squeeze())
print((output.squeeze()-goutput).abs().sum())
print(torch.eq(output.squeeze(),goutput).all())
</code></pre>
<p>I get following printout</p>
<pre><code class="lang-auto">tensor(1.9670e-06, grad_fn=&lt;SumBackward0&gt;)
tensor(0, dtype=torch.uint8)
</code></pre>
<p>I think Conv with kernel size 1 is the same as Linear operation. Am I wrong?</p>

ptrblck: <p>A difference of <code>1e-6</code> points to the limited floating point precision.<br>
You could run your code again with <code>DoubleTensors</code>, which should yield a smaller difference.</p>
Linearly transform image data

DDong: <p>What kind of transformation would linearly map an RGB image with values in [0,256] linearly to values in [-1,1]?</p>

ptrblck: <p>I’m not sure, if that’s what you are looking for, but this should work:</p>
<pre><code class="lang-python">x = torch.arange(0, 256).float()
x /= x.max() / 2. 
x -=1
x.min() # tensor(-1.)
x.max() # tensor(1.)
</code></pre>
.npy as input of ResNet

Giuseppe: <p>Hy guys, i have a problem, i would to pass many  distributions that are storage like .npy as input of a ResNet. It’s a new thing for me the type of input. How can do this? Is it possible or i can transformate my input first? The code that i wrote is ready for images…</p>

JuanFMontesinos: <p>You can pass whatever you want, for reading numpy files you will have to replace the reading function to adapt it to npy files.<br>
What’s the exact question about?</p>

Giuseppe: <p>The problem in another, i have the .npy as images of 9 channels.<br>
I can’t change them<br>
How can i change the resnet50 model pretrained?</p>

JuanFMontesinos: <p>When you load the model model.load(state_dict) use the arg strict=False</p>
<p><code>model.load(sd,strict=False)</code><br>
Anyway you wiol have to fije tune</p>
Customized criterion calculation is slow in cuda

garyzcc: <p>I customized a criterion function as my loss, while it turned out slower on GPU than running on CPU, in both calculating loss value and do loss.backward(). Here is my code defining the criterion.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.distributions.negative_binomial import NegativeBinomial


class NegtiveBinomialLoss(nn.Module):
    def __init__(self):
        """
        this Module is designed as a customized loss criterion for negative binomial distribution,
        implementing -sum(log(likelihood))

        """
        super(NegtiveBinomialLoss, self).__init__()

    def forward(
        self,
        net_output: torch.autograd.Variable,
        y: torch.Tensor,
    ) -&gt; torch.autograd.Variable:
        """
        Args:
            net_output: predict values, aka network output
            y: targets

        Returns: loss function variable

        """
        # alpha is number of success, lambda is number of failure
        param_alpha = net_output.t()[0]
        param_lambda = net_output.t()[1]
        log_likelihoods = torch.Tensor(len(y)).cuda()
        for i in range(len(y)):
            # In torch.distributions.negative_binomial.NegativeBinomial(total_count, probs),
            # total_count = success number, probs = failure prob
            nb = NegativeBinomial(total_count=param_alpha[i], probs=param_lambda[i] / (param_alpha[i] + param_lambda[i]))
            log_likelihoods[i] = nb.log_prob(y[i])
        return -log_likelihoods.sum()
</code></pre>
<p>I tried to put the criterion on cuda as well, but nothing changed.</p>
<pre><code class="lang-auto">criterion = NegtiveBinomialLoss().cuda()
loss = criterion(output, targets)
loss.backward()
</code></pre>
<p>Is there any method to speed it up?</p>

garyzcc: <p>Seems figure out by avoiding to use the for loop in criterion</p>