akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Long to wide tensor according to indices

ngravindra: <p>I want to create a wide array from long data.</p>
<p>I have some edge indices, <code>edge_index=tensor([[0,0,0,1,1,1...], [0,2,3,1,2,3,..]])</code> for node 0 connected to node 0,2,3</p>
<p>associated with three edge features, <code>edge_attr=tensor([[0.1,0.2,0.3], [0.4,0.5,0.6],...])</code></p>
<p>such that edge_index.shape = 2 x n_edges, and edge_attr.shape= n_edges x 3 (using pytorch geometric’s data object).</p>
<p>I am trying to re-shape the edge attributes to have shape [n_nodes, max_n_edges, 3] so each node, e.g., node 0, has its 3 connections 1st edge feature in one row, and its next edge feature concatenated in the third dimension.</p>
<p>I can think of a pandas pivot way of doing this but the duplicates in the node indices (node 0 has 3 connections) poses somewhat of a problem and is slow… Is there a torch way of doing this?</p>
Torch geometric on CUDA 10.2

lepoeme20: <p>Hi,</p>
<p>Recently, I studied graph and learned that torch has torch geometric.</p>
<p>I’m using CUDA 10.2 on ubuntu 18.04 and I tried  installed using conda, but it failed.</p>
<p>How can I use torch geometric with CUDA10.2?</p>

ptrblck: <p>Based on <a href="https://github.com/rusty1s/pytorch_geometric#pytorch-150">this information</a>, their binaries should be available with CUDA10.2 for Windows and Linux.<br>
What kind of error did you get?</p>
What happens when Conv2d filter number is not a multiple of the input channel?

jeffreyng99: <p>A bit of context; reading literature, an input image 256x256x3 is passed through a convolution that produces 64 filters, 256x256x64. What happens when the number of filter is not a multiple of the input channels?</p>
<pre><code class="lang-python">m = nn.Conv2d(3, 2, 3, 1)
input = torch.randn(1, 3, 10, 10)
output = m(input)
</code></pre>
<p>The code above has 2 filters and 3 layers: my current understanding is the last channel is not touched at all and if there was 5 filters, the channels before would each get convoluted twice.<br>
Is this a problem? How does PyTorch handles this?</p>

futscdav: <p>You misunderstand the application of Conv2d in virtually all ML frameworks today. Each filter takes all channels as input as opposed to just one, therefore, it has <code>CxKxK</code> parameters. If you want to use a subset, that’s what the <code>groups</code> parameter is for. Then it needs to be a multiple.</p>
Mask R-CNN Evaluations metrics

mbordi: <p>Hi!</p>
<p>I am following the tutorial TorchVision 0.3 Object Detection finetuning tutorial and try to train the network with customized data.</p>
<p>After the first epoch I obtain evaluation results and they seem to be not okay. Please refer to the image.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/1/6137b79afe0cdc0b94863816003983861db4d145.png" alt="Bildschirmfoto 2020-05-14 um 00.29.27" data-base62-sha1="dS1GFrZ5QdAxCqE5yW8rQW7D7vL" width="685" height="449"></p>
<p>What does the -1 in evaluation mean?<br>
And what do I do wrong?</p>
<p>Thank you!</p>
Understanding how pytorch connects densenet layers

Guilherme_Martins: <p>Assuming I have a network where I have 2 conv. layers (cv1, cv2) and 1 batch norm layer (bn). That are connected in the following way:<br>
cv1 --&gt; cv2 --&gt; cv3 and cv1 —&gt; cv3<br>
And that cv1 has 64 output layers, cv2 has 32 output layers and bn has 64 +32 = 96 input layers.<br>
Can I say that weight with index 63 is applied to the layer number 64 of the cv1 and that weight with index 64 is being applied to layer number 1 of the cv2 layer?</p>
<p>Thank you!</p>

ptrblck: <p>You are not using <code>bn</code> in your figure, so I assume <code>cv3</code> should be the batchnorm layer?<br>
The <code>weight</code> and <code>bias</code> parameters of the batchnorm layer will be applied per input channel, so your assumption should be correct as shown in this code snippet:</p>
<pre><code class="lang-python"># setup
bn = nn.BatchNorm2d(32+64).eval()

# change the weight and bias to see an effect
with torch.no_grad():
    nn.init.normal_(bn.weight)
    nn.init.normal_(bn.bias)

# use case 1
x1 = torch.randn(1, 32, 4, 4)
x2 = torch.randn(1, 64, 4, 4)
x = torch.cat((x1, x2), dim=1)
out = bn(x)

# manual approach
out1 = x1 * bn.weight[:32][None, :, None, None] + bn.bias[:32][None, :, None, None]
out2 = x2 * bn.weight[32:][None, :, None, None] + bn.bias[32:][None, :, None, None]
out_manual = torch.cat((out1, out2), dim=1)

# print absolute error
print((out - out_manual).abs().max())
&gt; tensor(3.3379e-05, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

Guilherme_Martins: <p>Yes thats right cv3 should be bn.<br>
Alright, thank you!</p>
How to check corresponding labels for predictions/softmax values?

Talita: <p>Suppose that I have a tensor produced by <code>F.softmax(nn.Linear())</code> of shape [64, 2], where 64 is the batch size and 2 is the number of classes;</p>
<pre><code class="lang-auto">tensor([[0.5382, 0.4618],
        [0.5180, 0.4820],
        [0.5294, 0.4706],
        [0.5272, 0.4728],
        [0.5307, 0.4693],
        [0.5291, 0.4709],
        [0.5215, 0.4785],
        [0.5300, 0.4700],
        [0.5330, 0.4670],
</code></pre>
<p>And that my label/class can either be 0 or 1.<br>
How can I know which probability corresponds to which label? I suppose that at training time, I need to pick the probability value of the true tag and feed that to the loss function.</p>

LeviViana: <aside class="quote no-group" data-username="Talita" data-post="1" data-topic="81074">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/977dab/48.png" class="avatar"> Talita:</div>
<blockquote>
<p><code>tensor([[0.5382, 0.4618],</code></p>
</blockquote>
</aside>
<p>For the first sample, you have 0.5382 probability for class 0 and 0.4618 probability for class 1.</p>
Entangled SGD optimizer

polo5: <p>In Pytorch the optim.SGD update rule has the form:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/7/273eb5ac305fa8c54824b02202ee561ae57116e4.png" data-download-href="https://discuss.pytorch.org/uploads/default/273eb5ac305fa8c54824b02202ee561ae57116e4" title="Screenshot from 2020-05-13 16-41-46"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/273eb5ac305fa8c54824b02202ee561ae57116e4_2_690x29.png" alt="Screenshot from 2020-05-13 16-41-46" data-base62-sha1="5BaYwp0p9G3Q8ZoNaIHg7fuhySg" width="690" height="29" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/273eb5ac305fa8c54824b02202ee561ae57116e4_2_690x29.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/273eb5ac305fa8c54824b02202ee561ae57116e4_2_1035x43.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/7/273eb5ac305fa8c54824b02202ee561ae57116e4.png 2x" data-dominant-color="E6E5E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-05-13 16-41-46</span><span class="informations">1108×47 11.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there any reason why it is common practice to couple the learning rate with momentum (beta) and weight decay (mu)? As it is, I can’t increase momentum (smoothing of gradients) without increasing the magnitude of the velocity (v), i.e. effective step size.</p>
<p>Is there a good reason why Pytorch hasn’t opted for a more ‘disentangled’ update rule like this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fcd2cb66699ab0b3bbf5ff54feece5673be4241.png" data-download-href="https://discuss.pytorch.org/uploads/default/8fcd2cb66699ab0b3bbf5ff54feece5673be4241" title="Screenshot from 2020-05-13 16-56-03"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fcd2cb66699ab0b3bbf5ff54feece5673be4241_2_690x24.png" alt="Screenshot from 2020-05-13 16-56-03" data-base62-sha1="kw7RvbE90D6fm9scT6b2ya6pay5" width="690" height="24" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fcd2cb66699ab0b3bbf5ff54feece5673be4241_2_690x24.png, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fcd2cb66699ab0b3bbf5ff54feece5673be4241.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fcd2cb66699ab0b3bbf5ff54feece5673be4241.png 2x" data-dominant-color="E4E4E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-05-13 16-56-03</span><span class="informations">977×34 9.05 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I seem to be getting a similar perf on CIFAR-10 with it, although it requires different values of mu and beta.</p>
Pytorch 1.5 - is Torchscript still required? Do pad tokens require computation?

Damiox: <p>I see Pytorch 1.5 has brought a lot of improvements for C++ APIs. I’ve got a model that I’m using in Python. Would I still need to use Torchscript in the middle if I wanted to load this model from C++/Java? I guess so because it’s in Python and I need it to be compiled in JIT so I can load it from other languages. Please confirm this.</p>
<p>Also, my current concern about Torchscript is committing to a given sentence length. My model is GPT-2 and I just want to avoid padding as much as possible because <em>I think</em> pad tokens are still being calculated in the neural network. I’m not sure about this point though. Should I have multiple Torchscript models with different sentence lenghts? I believe having a single Torchscript model with sentences of 1024 tokens may affect performance for sentences with &lt; 100 tokens as the majority of those tokens will be pad tokens e.g… If there’s no computation for them, then I’ll be safe. How is this concept usually being handled?</p>

albanD: <p>Hi,</p>
<p>For the first point, I think the cpp API has a very different goal. If you were using it, you would need to reimplement your model in CPP and then load_state_dict like you would do in python.<br>
TorchScript does not require that as it allows you to serialize the whole model with the weights.</p>
<p>For the tokens in your model I’m not sure <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"> But if you use torchscript, it should just behave like the python version of your model. So you just need to make sure not to do extra padding on the python side.</p>

Damiox: <p>What if I would be using scripting (torch.jit.script) instead of tracing (torch.jit.trace) with Torchscript? Should I still have to commit to a fixed sentence length if I were using torch.jit.script ?</p>

albanD: <p>if you use script, you won’t. The branching will work fine.<br>
If you use tracing, you will have to keep fixed sequence indeed.</p>
Which best model I need to save

lycaenidae: <p>Hi</p>
<p>Do I need to save best model on training set, or best model on validation set?</p>

futscdav: <p>Provided you have a representative validation set, it’s normal practice to consider the model which performs best on validation set to be better. Note that if you are comparing many models on a small validation set, you run the risk of basically comparing noisy random variables.</p>

lycaenidae: <p>Is it a really bad to save the model which is best on the train set?</p>

futscdav: <p>It’s not bad per se. But usually, you care about performance on unseen data (which are what validation set is supposed to represent), not so much about performance on specific training dataset.</p>
Support for implementing Meta-Learning Algorithm

Asichurter: <p>When I implemented a meta-learning algorithm called “Reptile”, I got the following error:</p>
<p>File “/home/asichurter/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/asichurter/GitHub/APISeqFewShot/models/Reptile.py”, line 163, in forward<br>
grads = t.autograd.grad(loss, self.Learner.parameters())<br>
File “/home/asichurter/anaconda3/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 157, in grad<br>
inputs, allow_unused)<br>
RuntimeError: cudnn RNN backward can only be called in training mode</p>
<p>I indeed know what I am doing because I need to backpropagate and compute the gradient of RNN models like LSTM. It is because this kind of meta-learning algorithm indeed needs these gradients to adapt even in the inference stage! However it seems to be unfortunate that this is not supported by PyTorch framework up to now (I use PyTorch 1.4, Cuda 10.2 with corresponding CuDnn on Ubuntu 18.04). One more thing, even though running “model.train()” can solve this problem, I can not actually do this because I have some normalization layer like Layer Normalization in my architecture and setting it to training mode can somehow cause some problems(it will keep tracking the statistics during evaluation).</p>
<p>To conclude, it seems that PyTorch can’t completely support some keys features to implement meta-leraning algorithm (one more problem I encountered was being unable to compute high-order derivatives on RNN models). Wish some improvements on this!</p>

albanD: <p>Hi,</p>
<p>This is a limitation of cudnn that does not implement it.<br>
You can disable cudnn during the forward computation of that RNN if you want to be able to do that by falling back to our own implementation that does support it</p>
<pre><code class="lang-auto">with torch.backends.cudnn.flag(enabled=False):
    # Your rnn forward code
</code></pre>
Fft: ATen not compiled with MKL support

mahmoodn: <p>May I know what should I do prior to PyTorch installation to get rid of this error:</p>
<pre><code class="lang-auto">  File "main.py", line 15, in &lt;module&gt;
    specgram = torchaudio.transforms.Spectrogram()(waveform)
  File "/home/mnaderan/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "build/bdist.linux-x86_64/egg/torchaudio/transforms.py", line 70, in forward
  File "build/bdist.linux-x86_64/egg/torchaudio/functional.py", line 259, in spectrogram
  File "build/bdist.linux-x86_64/egg/torchaudio/functional.py", line 52, in _stft
  File "&lt;__torch_function__ internals&gt;", line 6, in stft
  File "/home/mnaderan/.local/lib/python2.7/site-packages/torch/_overrides.py", line 140, in _implement_torch_function
    return implementation(*args, **kwargs)
  File "/home/mnaderan/.local/lib/python2.7/site-packages/torch/functional.py", line 427, in stft
    return torch._C._VariableFunctions.stft(input, n_fft, hop_length, win_length, window, normalized, onesided)
RuntimeError: fft: ATen not compiled with MKL support
</code></pre>

miken: <aside class="quote no-group" data-username="mahmoodn" data-post="1" data-topic="64596">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahmoodn/48/13302_2.png" class="avatar"> mahmoodn:</div>
<blockquote>
<p>:</p>
</blockquote>
</aside>
<p>bump! any solution to this problem?</p>

miken: <p>for anyone coming across this, the issue seems to be a non intel cpu or you haven’t compiled with MKL support</p>
Save memory during backward through shared weight

maaft: <p>Hi,</p>
<p>I have two convolutions which have the same shared weight. One is using dilation=1, the other is using dilation=2.</p>
<p>Let <code>x</code> be an input tensor, and <code>w</code> the shared weight.</p>
<p>Is there anything I can do to save memory when calculating:</p>
<pre><code class="lang-python">y = F.conv2d(x, weight=w, dilation=1) + F.conv2d(x, weight=w, dilation=2)
</code></pre>
<p>and doing the backward pass?</p>
<p>Right now the memory consumption is doubled which feels unnecessary.</p>

albanD: <p>Hi,</p>
<p>The memory consumption is doubled most likely because both outputs are saved.<br>
Indeed, to compute the backward of the convolution, we need to know what was the output. So if you do two of them, we need to know both outputs.</p>
Torch<1.0 to torch torch>1.0 conversion

bartmch: <p>First post <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
Hi, I’d like to convert <a href="https://github.com/arnoweng/CheXNet/blob/master/model.pth.tar" rel="nofollow noopener">these</a> torch&lt;1.0 pth.tar weight to torch&gt;1.0 pth.tar weight. Can anyone point me in the right direction to do this easily? Thanks!</p>

bartmch: <p>Update:<br>
So I managed to change the keys of the state_dict by doing:</p>
<pre><code class="lang-auto">checkpoint = torch.load(checkpoint_pth)
pattern = re.compile(r'^(.*denselayer\d+\.(?:norm|relu|conv))\.((?:[12])\.(?:weight|bias|running_mean|running_var))$')
state_dict = checkpoint['state_dict']
for key in list(state_dict.keys()):
    res = pattern.match(key)
    if res:
        new_key = res.group(1) + res.group(2)
        state_dict[new_key] = state_dict[key]
        del state_dict[key]
return checkpoint['state_dict']
</code></pre>
<p>But when I compare pre-trained densenet121 model (torch&lt;1.0, see link previous message) with a torch.models.densenet121 (torch&gt;1.0) it looks like the torch&lt;1.0 model contains extra layers such as<br>
‘module.densenet121.features.denseblock2.denselayer12.norm2.num_batches_tracked’ and ‘module.densenet121.features.transition2.norm.num_batches_tracked’.</p>
<p>The mismatched layers are all of “.num_batches_tracked” any idea on how to solve this?</p>

albanD: <p>Hi,</p>
<p>This was added fairly recently to batchnorm. As you can see in the code <a href="https://github.com/pytorch/pytorch/blob/4fef3763dd7266195e8ef20c0a5d4dd1219afeb0/torch/nn/modules/batchnorm.py#L60-L73">here</a>, the batchnorm layer should be able to load this properly even though the key is missing.</p>
Mypy error with reproducible code

RMeli: <p>When I use <code>mypy</code> with the following <a href="https://pytorch.org/docs/stable/notes/randomness.html#reproducibility" rel="nofollow noopener">reproducible</a> code</p>
<pre><code class="lang-auto">import torch
from torch.backends import cudnn
torch.manual_seed(0)

cudnn.deterministic = True
cudnn.benchmark = False
</code></pre>
<p>I get an error</p>
<pre><code class="lang-auto">test.py:5: error: Module has no attribute "deterministic"
test.py:6: error: Module has no attribute "benchmark"
Found 2 errors in 1 file (checked 1 source file)
</code></pre>
<p>I think the problem arises from the fact that  <code>deterministic</code> and <code>benchmark</code> are not defined in <code>torch.backends.cudnn.__init__</code>. Is there a workaround for this error which is making my CI fail?</p>

albanD: <p>Hi,</p>
<p>We are currently working on having a mypy compatible code base. Could you open an issue on github for this please?</p>
Building a NN that classifies images based on human preference

Zador: <p>Hi,<br>
I have a data set containing triplets of three images of food A, B and C where it is given that a human thinks A and B are closer in taste than A and C.<br>
I want to build a neural net that learns from these preferences and will be able to predict, if given a triplet of images of food, if A &amp; B or A &amp; C are closer in taste.</p>
<p>Does any one have any ideas on how I might do this?<br>
I am new to machine learning so any suggestions on network architecture or loss function would be most helpful</p>

simaiden: <p>You could look siamese networks and <a href="https://arxiv.org/abs/1503.03832" rel="nofollow noopener">triple loss</a> to start</p>
Cuda context size per process

seliad: <p>Hi,<br>
I noticed that there is a large (~500MB) cuda context created per process on GPU.<br>
can see it simply by doing:</p>
<pre><code class="lang-auto">import torch
torch.randn(1,device=0)
</code></pre>
<p>it takes 500MB (used to take 750MB in previous versions).<br>
when multiprocessing on same GPU this is a lot of unneeded memory.<br>
How can we work around this?</p>

mrshenli: <p>Do multiple processes have to work on the same set of GPUs? Can each process work on an exclusive set of GPUs and use <code>CUDA_VISIBLE_DEVICES</code> to control which devices they see?</p>

seliad: <p>I explicitly want multi processes using the same GPU.</p>

mrshenli: <p>I am not aware if there is a way to avoid the per-process CUDA context or reduce its size. <a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/alband">@albanD</a> might know more.</p>

ptrblck: <p>I don’t think it’s possible to reuse a single CUDA context between processes, but haven’t looked deeply into it.<br>
We expect the best performance using a single process per GPU.<br>
What is your use case <a class="mention" href="/u/seliad">@seliad</a> that you want to use multiple processes on the same device?<br>
Are you seeing any performance gains (regardless of the wasted memory)?</p>

seliad: <p>For processes: A,B<br>
Each has to do<br>
(1) distributed communication (e.g A–&gt;C , B-&gt;D)<br>
(2) share parameters (A&lt;-&gt;B)</p>
<p>For the distributed communication I need different ranks (Im currently using cuda-aware MPI).</p>
<p>Even if there is an option to use distributed communication with threads (i think that there is in mpi, not sure if Pytorch supports it), in python it is a pretty bad Idea.</p>

seliad: <p>I also noticed that when using 2 processes communicating through a <code>Queue</code>,<br>
The sender process (e.g sending from device0 to device1, with <code>copy_</code>) thholds this cuda contex on both devices. I created the buffer at the sender and sent it  through a queue (reusing it) as recommended  in the docs.<br>
(push communication model)</p>
<p>Another option is to have a thread in the receiver waiting on that queue and pulling from it, I guess that won’t cause this extra memory?</p>
<p>So, I wonder, maybe it could be (theoretically, and very partially) solved by creating all tensors in a single process (single owner), and sending them to all processes sharing the device.</p>
Transformer for classification

slavavs: <p>I want to use a transformer for classification. I want to submit a proposal at the entrance, and at the exit I want to classify it into two classes.<br>
Do I have to use a decoder or can I get a softmax at the output of the encoder?</p>

slavavs: <p>if I input 10 words, I get 10 vectors at the output of the encoder. Then you need to convert this vector to get 2 classes. How can I convert these vectors? Somewhere I read that you can average. Someone writes that you can use CNN, RNN. I don’t know how to do it right. How does BERT make a classification?</p>
Bug? nn.Parameter doesn't work with sparse input

cod3licious: <p>I have a very small network like this:</p>
<pre><code class="lang-python">import torch.nn as nn

class ConEcNet(nn.Module):

    def __init__(self, input_dim, embedding_dim):
        super(ConEcNet, self).__init__()
        self.W_inputs = nn.Parameter(0.1*torch.randn(input_dim))
        self.W_0 = nn.Linear(input_dim, embedding_dim)

    def forward(self, inputs):
        return self.W_0(inputs * self.W_inputs)
</code></pre>
<p>i.e. the input vector is first multiplied elementwise by the vector in <code>W_inputs</code> (i.e. each input feature has an individual importance weight) and then passed to the linear layer <code>W_0</code>. This works fine if my inputs are a regular tensor, however, if the inputs are a sparse tensor, the first multiplication (<code>inputs * self.W_inputs</code>) results in a <code>RuntimeError: mul operands have incompatible sizes</code>.</p>
<p>Since I’m working with BOW features, I would really like to use sparse inputs. Does <code>nn.Parameter</code> in general not work with sparse tensors? Is there a way around this besides casting the inputs to a regular tensor?</p>
<p>I’m using torch version <code>1.5.0</code>.</p>
<p>Example:</p>
<pre><code class="lang-python">import numpy as np
from scipy.sparse import coo_matrix
import torch

def coo2tensor(coo):
    """
    convert scipy sparse coo matrix to sparse torch tensor
    """
    values = coo.data
    indices = np.vstack((coo.row, coo.col))
    i = torch.LongTensor(indices)
    v = torch.FloatTensor(values)
    shape = coo.shape
    return torch.sparse.FloatTensor(i, v, torch.Size(shape))

cnet = ConEcNet(10, 5)
a = torch.randn(3, 10)
cnet(a)  # works
b = coo2tensor(coo_matrix(np.random.randn(3, 10)))
cnet(c)  # runtime error
cnet.W_0(c)  # works
</code></pre>
How to compute per-channel sum fast?

Liangzhi_Tang: <p>suppose the shape of x is [2, num_channels, 4, 5], and I compute per-channel summation as follows:</p>
<blockquote>
<p>s = torch.zeros(num_channels)<br>
for i in range(num_channels):<br>
s[i] = x[:, i].sum()</p>
</blockquote>
<p>It turns out this implement is too slow. Is there a faster implement?</p>

Nikronic: <p>Hi,</p>
<p>Using built-in functions, it would be like this:</p>
<pre><code class="lang-auto">num_channels = 10
x = torch.randn(2, num_channels, 4, 5)
result = torch.sum(x, dim=[0, 2, 3])
</code></pre>
<p><code>torch.sum</code> accepts a parameter <code>dim</code> which can be a list. If you pass a list, it will reduce the tensor over all the indices in <code>dim</code>. For instance,</p>
<pre><code class="lang-auto">x = torch.randn(2, 4, 5)
x[0,:,:] = 1
x[1,:,:] = 2
result = torch.sum(x, dim=[1,2])  # tensor([20., 40.])
</code></pre>
<p>And explanation would be we have 4*5 cells for each channel (2 in this case) so as I set dim=0 all equal to 1 and dim=1 all equal to 2, so sum over all dims except channel will reduce it in that way.</p>
<p>bests</p>

Liangzhi_Tang: <p>Thanks a lot and It saved half of the time! I also need to optimize per-channel torch.max() and torch.max doesn’t have such features you mentioned, any suggestion for per-channel torch.max?</p>

Nikronic: <p>glad it worked,<br>
Actually, there is no built-in function for <code>max</code> or <code>argmax</code> like <code>sum</code>. But you can do the trick something similar to this post:</p><aside class="quote" data-post="5" data-topic="65122">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/ecccb3/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-efficiently-normalize-a-batch-of-tensor-to-0-1/65122/5">How to efficiently normalize a batch of tensor to [0, 1]</a> 
  </div>
  <blockquote>
    <a class="mention" href="/u/ptrblck">@ptrblck</a> Here it is 
batch_size = 3
height = 2
width = 2

A = torch.randint(2, 11, (batch_size, height, width)).float()
AA = A.clone()
print(A)

# I can get what I want from below for-loop solution
for i in range(batch_size):
    A[i] -= torch.min(A[i])
    A[i] /= torch.max(A[i])

# Your solution
AA -= AA.min(1, keepdim=True)[0]
AA /= AA.max(1, keepdim=True)[0]

print(A)  # A and AA are different
print(AA)
  </blockquote>
</aside>
View_classify in helper is not defined

AmirHusain: <p>hi  how are you ? thanks for helping<br>
i wrote my network but im having difficulty in plotting the results by helper module as View_classify is not defined  . can you please help me to solve the problem ?<br>
i downloaded helper and copied to  work place and it didnt work</p>
<p>imported everything needed<br>
transform = transforms.Compose([transforms.ToTensor(),<br>
transforms.Normalize((0.5,), (0.5,))])<br>
trainset = datasets.MNIST(‘MNIST_data/’, download=True, train=True, transform=transform)<br>
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)</p>
<p>def Sigmoid(x):<br>
return 1 / (1 + torch.exp(-x))  # rewrited</p>
<p>def Softmax(x):<br>
return torch.exp(x) / torch.sum(torch.exp(x), dim=1).view(-1, 1)</p>
<p>model = nn.Sequential(nn.Linear(784, 256),<br>
nn.ReLU(),<br>
nn.Linear(256, 64),<br>
nn.ReLU(),<br>
nn.Linear(64, 10),<br>
nn.LogSoftmax(dim=1))<br>
criterian = nn.NLLLoss()<br>
optimizer = optim.SGD(model.parameters(), lr=0.1)</p>
<p>epochs = 1<br>
for e in range(epochs):<br>
running_loss = 0<br>
for images , labels in trainloader:<br>
images = images.view(images.shape[0], -1)<br>
optimizer.zero_grad()</p>
<pre><code>    output = model.forward(images)
    loss = criterian(output, labels)

    loss.backward()

    optimizer.step()

    running_loss += loss.item()


else:
    print(f"Training loss : {running_loss/ len(trainloader)}")
</code></pre>
<p>images , labels = next(iter(trainloader))<br>
img = images[0].view(1 , 784)<br>
with torch.no_grad():<br>
output = model.forward(img)<br>
ps = F.softmax(output , dim = 1)<br>
helper.view_classify(img.view(1 , 28 , 28) , ps)</p>

ptrblck: <p>Could you check your current working directory via <code>!ls</code>, if you are using IPython?<br>
I guess you might have copied the file to a different location?<br>
Also, did you include the necessary methods or classes to use <code>view_classify</code>?</p>

AmirHusain: <p>hello , thank you so much for helping</p>
<p>i read the helper module i downloaded  it didn’t contain view_classify in it</p>
<p>sorry but i didn’t understand your last sentence</p>
<p>im not good at drawing in python i just downloaded helper and wrote the code i saw in instructon  . im coding with pycharm ,</p>
<p>can we use pyplot instead ? what should i write ?</p>
<p>thanks again<br>
regards</p>

ptrblck: <p><a href="https://raw.githubusercontent.com/udacity/deep-learning-v2-pytorch/3bd7dea850e936d8cb44adda8200e4e2b5d627e3/intro-to-pytorch/helper.py">This file</a> should contain the <code>view_classify</code> method. You could either download it again and import the method or copy-paste it to your script.</p>

AmirHusain: <p>Hi<br>
thank you so much<br>
it worked brilliantly</p>
How to use libtorch trace model

fengweigang: <p>I installed libtorch1.5 and try to use it train model, I want use C++ API to trace the trained model and save it. But I just find the API torch::jit::tracer::trace, how to use it? I can’t get any explain about these API, who can give me some advice?</p>

hktxt: <pre><code class="lang-auto"># An instance of your model.
model = torch.load('my_model.pth')
model.eval()
# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)
 
# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model.cuda(), example.cuda())

# save it
traced_script_module.save("model_cpp.pt")
</code></pre>

fengweigang: <p>First of all, thank you for your answer, I actually ues libtorch(c++ API), so I want to train and save finally model in the same platform, but I can’t find any instructions about how to trace or scrpt model in C++ platform. You may not get my problems.</p>
conv2D RuntimeError: Expected 4-dimensional input for 4-dimensional weight [128, 10, 2, 2], but got 2-dimensional input of size [4, 10] instead

John_J_Watson: <p>I am trying to pass an output of a pretrained classifier model to a bunch of conv2D and linear layers (to get some embeddings eventually), but since the classifier outputs a dim of [4,10], I am having trouble passing this to the conv layers.</p>
<p>So, I have:</p>
<pre><code class="lang-auto">   def forward(self, data):
         x = data
         x = self.model_ft(x) # model_ft is the pretrained model; 
         print(x.shape)
        # output of the above is : torch.Size([4, 10])
         x = self.conv1(x) # conv2D defined as: self.conv1 = nn.Conv2d(10, 128, 2)
         x = F.relu(x)
         x = self.pool1(x) # defined as: self.pool1 = nn.MaxPool2d(2)
         x = self.conv2(x) # defined as: self.conv2 = nn.Conv2d(128, 64, 5)
</code></pre>
<p>However, when I run this, I get:</p>
<blockquote>
<p>RuntimeError: Expected 4-dimensional input for 4-dimensional weight [128, 10, 2, 2], but got 2-dimensional input of size [4, 10] instead</p>
</blockquote>
<p>Not sure what it is that the conv2D layer is expecting and I could not really figure this out reading the docs <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20">  - any pointers would be great!</p>

albanD: <p>Hi,</p>
<p>The conv2d expects an input of size: <code>(batch, channel, height, width)</code>.</p>

John_J_Watson: <p>Thanks albanD - I should be using conv1D not 2D!</p>
Why doesn't this custom softmax negative loglikelihood work?

jwillette: <p>I have realized that the negative log likelihood function in pytorch has a reduction argument which can be used instead of my method below but  I still cannot justify why this doesn’t work  when it looks like it should.</p>
<p>In this case <code>y</code> was a vector with all of the integer class labels in it. I verified that the answers were very close to what I would get from the <code>functional.nll_loss</code> in torch, but the model failed to learn in this case so there must have been some problem which didn’t cause an error, but IDK what it is. Can anyone see why this didn’t work?</p>
<pre><code class="lang-auto">loss = -torch.log(F.softmax(logits, dim=1)[:, y])
loss.mean().backward()
optimizer.step()
</code></pre>

ptrblck: <p>You would get approx. the same results assuming your indexing is right (I had to use <code>[torch.arange(y.size(0), y]</code> if y has the shape <code>[batch_size]</code>).<br>
However, you might run into numerical stability issues, as you are applying <code>torch.log</code> and <code>F.softmax</code> separately.</p>
<p>You could try to use <code>loss = -F.log_softmax(logits, dim=1)[torch.arange(y.size(0)), y]</code> instead and compare your results with the unreduced reference loss.</p>

jwillette: <p>I see thanks, I guess I didn’t notice a mistake in my indexing which caused it to get random gradients and not learn anything. Thanks!</p>
How to quickly make a mask tensor using index tensor?

jwchoi384: <p>Hi, I used pytorch 1.1 and I have a question about making mask tensor using index tensor.</p>
<p>I have a simple example for understanding.<br>
For example, I have index tensor (size is [2, 4]) which is look like below:</p>
<p><code>tensor([[0, 0, 2, 4], [0, 0, 0, 3]])</code></p>
<p>and finally, I want to get mask tensor (size is [2, 4, 6]) like this:</p>
<p><code>tensor([[[1., 0., 0., 0., 0., 0.],</code><br>
<code> [1., 0., 0., 0., 0., 0.],</code><br>
<code> [0., 0., 1., 0., 0., 0.],</code><br>
<code>[0., 0., 0., 0., 1., 0.]],</code><br>
<code>[[1., 0., 0., 0., 0., 0.],</code><br>
<code>[1., 0., 0., 0., 0., 0.],</code><br>
<code>[1., 0., 0., 0., 0., 0.],</code><br>
<code>[0., 0., 0., 1., 0., 0.]]])</code></p>
<p>I implemented it using a for loop statement twice like below, but if the tensor size is large (for example, mask size is [32, 8732, 21]), it is too slow to use for training.</p>
<blockquote>
<p>Blockquote<br>
for i in range(mask.size(0)):<br>
for j in range(mask.size(1)):<br>
mask[i,j,b[i,j]] = 1</p>
</blockquote>
<p>Is there any way to implement the action quickly without using the for loop?</p>
<p>Thank you in advance.</p>

ptrblck: <p>You could use <code>scatter_</code> as shown here:</p>
<pre><code class="lang-python"># setup
index = torch.tensor([[0, 0, 2, 4], [0, 0, 0, 3]])
mask = torch.zeros(2, 4, 6)

# loop
for i in range(mask.size(0)):
    for j in range(mask.size(1)):
        mask[i,j,index[i,j]] = 1

# scatter
mask2 = torch.zeros(2, 4, 6)
mask2.scatter_(2, index.unsqueeze(2), 1)

print((mask == mask2).all())
&gt; tensor(True)
</code></pre>

jwchoi384: <p>It works!!<br>
Thank you for your answer!</p>
Torchvision Faster Rcnn tutorial

Arun_Mohan: <p>Could some one provide me sorce for any Faster rcnn tutorial using torchvision. I had gone through one in pytorch docs(<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a>) but it left me with many doubts. So if any other good sources are there please help me out.</p>

ptrblck: <p>What kind of doubts do you have? Maybe we could clear them here? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Arun_Mohan: <p>How to use a different backbone for training.is there any doc explaining that.If you know please let me know</p>
Loading images with csv file

Muhammad_Izaz: <p>i have CSV file which contain information about the images. i want to load those images and also assign the true or false based on the CSV file.</p>

ptrblck: <p>The loading of the <code>csv</code> file can be done by using e.g. <code>pandas.read_csv</code>.<br>
You could read the <code>csv</code> in your custom <code>Dataset.__init__</code> method and once this is done, you could load load each row using <code>self.df.iloc[index]</code> in <code>Dataset.__getitem__</code>, and load the images as well as create the target.</p>
Save variable as image & difference between variable and tensor

cbd: <p>I am running code on torch==0.3.1 &amp; torchvision==0.2.1.</p>
<p>(1) I have output of CNN stored in “var_1” having torch.Size([3, 512, 512]) &lt;class ‘torch.cuda.FloatTensor’&gt;. I used “save_image(zz,‘t1.png’)” command to save as image but shows the error “TypeError: tensor or list of tensors expected, got &lt;class ‘torch.autograd.variable.Variable’&gt;”</p>
<p>I try to use “var_1= torch.from_numpy(numpy.array(var_1))” command before save_image but system hang. How can i save it as image?</p>
<p>(2) I found difference between variable and tensor is gradient propagation. But i think that difference is not present in latest version. If it is then after which version the vector and variable are the same?</p>

ptrblck: <ol start="2">
<li>
<p><code>Variables</code> were deprecated in PyTorch <code>0.4</code>, and you can use tensors in newer versions.</p>
</li>
<li>
<p>Could you update to the latest stable PyTorch (<code>1.5</code>) and torchvision (<code>0.6</code>) release and retry your code?</p>
</li>
</ol>

cbd: <p>Code is not supported in that version and required many changes. It will be fine if there is method to visualize the variable in form of image.</p>

ptrblck: <p>I’m not sure, what causes this error in these old versions, but you should be able to transform the tensor to a numpy array, permute its dimensions to <code>[height, width, channels]</code> and use <code>matplotlib.pyplot.imshow</code> to visualize the image.</p>

cbd: <p>In pytorch we can’t see the variable as image? If not i will not try much.</p>

ptrblck: <p>PyTorch or <code>torchvision</code> don’t reimplement visualization libraries and use e.g. <code>PIL</code> or other standard libraries to visualize tensors.</p>

cbd: <p>Okay ptrblck,<br>
Thanks.</p>
Normalization vs clipping

cbd: <p>NOTE: I am doing post processing for restoration after CNN output. Lets say CNN output is A. Then final output is Final_OUTPUT  = A + B.</p>
<p>During restoration of image, Final_OUTPUT values range goes beyond -1 to 1. When converted to image it looks like noise in image where the value exceed to less than -1 and greater than 1.  Is it better to<br>
(1) normalize the data ( x/torch.max(mod(x))  OR<br>
(2) clip the less than -1 value to -1 and greater than 1 value to 1.</p>
<p>Which will be better when loss is backpropogate? Can CNN learn after many epoch to range the data between -1 to 1 in case of (1). I tried (2), but color changes as value clip to -1 and 1.</p>
Create Dataloader with IDs

verance: <p>Hello, I am trying to create a dataloader with the following code:</p>
<pre><code class="lang-auto">data_loader = DataLoader(dataset, batch_size=1, num_workers=num_workers,
                                          sampler=SubsetRandomSampler(pool_idx))
</code></pre>
<p>where pool_idx = <code>pool_idx = random.sample(range(1, len(test_dataset)), pool_size)</code></p>
<p>Example of pool_idx:</p>
<p><code>[9, 6, 2, 1, 3, 5, 7, 4, 8, 0]</code></p>
<p>Is it possible to have the image + each corresponding id so when iterating through the dataset I can have access to each id?</p>
<p>Kind regards</p>

ptrblck: <p>How is the ID defined?<br>
If it’s an ID of the image, you could return it in the <code>Dataset.__getitem__</code> method with the data and target tensors.</p>
How to load a model when the layer names are not the same?

Umair_Javaid: <p>How to load a model when the layer names are not the same?<br>
<code>model7.load_state_dict(torch.load(path)["state_dict"], strict=False)</code><br>
<strong><code>output:</code></strong><br>
<code>_IncompatibleKeys(missing_keys=['features.26.weight', 'features.26.bias', 'features.28.weight', 'features.28.bias', 'features.30.weight', 'features.30.bias'], unexpected_keys=['features.25.weight', 'features.25.bias', 'features.27.weight', 'features.27.bias', 'features.29.weight', 'features.29.bias'])</code><br>
If I have to change the name of the layers, how do I do that?</p>

ptrblck: <p>The <code>state_dict</code> is an instance of an <code>OrderedDict</code>, so you could iterate all keys and change their name as done <a href="https://discuss.pytorch.org/t/how-do-i-update-old-key-name-in-state-dict-to-new-version/76026/3">here</a>.</p>
Custom dataset os.path.join () returns type error

olisp: <p>I am writing custom dataset but it returns type error when I am merging root directory <code>path</code> with pandas <code>iloc</code> of image name in csv file:</p>
<pre><code> img_path = os.path.join(self.root_dir, self.annotations.iloc[index,0])
</code></pre>
<p>error:  <code>TypeError: join() argument must be str or bytes, not 'int64'</code></p>
<p>I have tried converting the annotation.iloc to string type but it’s still giving me the same error.</p>
<p>custom dataset class:</p>
<pre><code>class patientdataset(Dataset):

    def __init__(self, csv_file, root_dir, transform=None):  
            self.annotations = pd.read_csv(csv_file)
            self.root_dir = root_dir
            self.transform = transform



    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):
        img_path = os.path.join(self.root_dir, self.annotations.iloc[index,0])  
        image= np.array(np.load(img_path)) 
        y_label = torch.tensor(self.annotations.iloc[index, 1]).long()


        if self.transform:
            imagearrays = self.transform(image)
            image = imagearrays[None, :, :, :]
            imaget = np.transpose(image, (0, 2, 1, 3))
            image = imaget


        return (image, y_label)
</code></pre>
<p>CSV file with image names and labels<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8acf68c16a451d7b9a4dc5eeea0f35420d1a24dd.png" data-download-href="https://discuss.pytorch.org/uploads/default/8acf68c16a451d7b9a4dc5eeea0f35420d1a24dd" title="CSV file with image names and labels"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8acf68c16a451d7b9a4dc5eeea0f35420d1a24dd.png" alt="CSV file with image names and labels" data-base62-sha1="jNYgjiPqaTPqaf2eMwImTe5x6Lr" width="128" height="500" data-dominant-color="E7E8E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">CSV file with image names and labels</span><span class="informations">179×698 5.27 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <aside class="quote no-group" data-username="olisp" data-post="1" data-topic="80870">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/e79b87/48.png" class="avatar"> olisp:</div>
<blockquote>
<p>I have tried converting the annotation.iloc to string type but it’s still giving me the same error.</p>
</blockquote>
</aside>
<p>What kind of error message do you get, as this should be working?</p>
You need to specify the dataset ('--dataroot')

ky_Pa: <p>How can I add a path in this form? Is it in the parameters here or do I need to write some code below? How does it work?</p>
<pre><code class="lang-auto">parser.add_argument('--dataroot', required=True, help='path to images')</code></pre>

ptrblck: <p><code>parser</code> is used to grab the arguments you pass to run the script from e.g. a terminal:</p>
<pre><code class="lang-auto">python script.py --dataroot /my/path/to/images
</code></pre>
<p>This will add the provided path to <code>args.dataroot</code> after you’ve called <code>args = parser.parse_args()</code>.</p>

ky_Pa: <p>Can you help me instantiate and write the code? I don’t know how to write the format of the path. If my path is ‘F:\MADAN’. Do I need to use ‘default’?</p>

ptrblck: <p>I’m not sure what <code>default</code> refers to, but you should be able to pass the path as given in my example.<br>
If that doesn’t work for you, you might need to add two backslashes, as a single backslash might be interpreted as an escape sequence.</p>

ky_Pa: <p>I still do n’t know where the path should be added?Is it written like this?</p>
<pre><code class="lang-auto">parser.add_argument('--dataroot', /path/to/images, help='path to images')</code></pre>

ptrblck: <p>The <code>parser.add_argument</code> call should be inside your script (e.g. <code>script.py</code>).<br>
Now you would execute this script from a terminal via:</p>
<pre><code class="lang-auto">python script.py --dataroot /your_path/
</code></pre>

ky_Pa: <p>Thanks for your method. Can be successfully run on the terminal. Is there any way to run on python IDE?</p>

ptrblck: <p>Depending on your IDE you might have some run options, where you can specify the arguments which should be passed to the script before execution.</p>

ky_Pa: <p>I probably understand. Thanks for your help!</p>
Is there any function to shift image by x axis or y asix?

Wei_Wong: <p>I want to do image augmentation, there are function available to do the job in PIL library, I wonder if there is any function like this in pytorch?</p>

ptrblck: <p>You could check out <a href="https://github.com/kornia/kornia">kornia</a>, which provides a lot of differentiable transformations.</p>
<p>CC <a class="mention" href="/u/edgarriba">@edgarriba</a> who might throw in more information. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Error loading state_dict ; GPT2 Finetuning

Kiran_Ikram: <p>Hi,<br>
I was using this snippet of code to load my finetuned GPT2 and it was working absolutely fine:<br>
tokenizer = GPT2Tokenizer.from_pretrained(‘gpt2-medium’)<br>
model = GPT2LMHeadModel.from_pretrained(‘gpt2-medium’)<br>
model_path = ‘/content/drive/My Drive/PREDICTION/POETRY/EDset3_trained_models/gpt2_medium_SET3_ED.pt’<br>
model.load_state_dict(torch.load(model_path))<br>
model = model.to(device)</p>
<p>As of this morning, having made no changes, I get the following error:<br>
RuntimeError: Error(s) in loading state_dict for GPT2LMHeadModel:<br>
Missing key(s) in state_dict: “transformer.h.0.attn.masked_bias”, “transformer.h.1.attn.masked_bias”</p>

ptrblck: <p>Hi Kiran,</p>
<p>is this issue solved <a href="https://github.com/huggingface/transformers/issues/4309">here</a> already or are you still running into these issues?</p>