akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Model Outputting the same value for every training value

dcorv: <p>I am a newbie to Pytorch and I’m trying to build a cat dog classifier by training a model with the VGG16 architecture. However, when I run my model, it doesn’t really learn anything and always outputs either all ones or all zeros in any given batch in the test set which are a mix of cat and dog.</p>
<p>I have been stuck on this for a while, if anyone could give me any help. It would be much appreciated.</p>
<p>Code:</p>
<p><a href="https://colab.research.google.com/drive/1WbixZzZYiRPASMVlqbV3br8_axVipHpG" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1WbixZzZYiRPASMVlqbV3br8_axVipHpG</a></p>
<p>Please find it here. Thanks a lot in advance.</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects raw logits as the model output, since internally <code>F.log_softmax</code> and <code>nn.NLLLoss</code> will be applied.<br>
Remove the <code>F.softmax(x,dim=1)</code> in the <code>forward</code> method of your model and rerun the code.<br>
Let me know, if this helps.</p>

dcorv: <p>Hi,</p>
<p>Thanks for getting back. Still having issues, the loss on the training set is descreasing so seems as if the NN is learning but still on the test set, predicts all the same class for every member of the batch. Something weird still going on.</p>

ptrblck: <p>In that case your model might just be overfitting.<br>
Does the validation loss go down at the beginning of your training and stays after a certain amount of epochs constant or goes up again?<br>
You might need to play around with some hyperparameters or add regularization (e.g. via dropout).</p>
RuntimeError: CUDA error: no kernel image is available for execution on the device in the second GPU

Abdelrahman_Elsayed: <p>Traceback (most recent call last):<br>
File “train.py”, line 264, in <br>
).to(device)<br>
File “C:\Users\a.abdallah\Anaconda3\envs\ocr2\lib\site-packages\torch\nn\modules\module.py”, line 425, in to<br>
return self._apply(convert)<br>
File “C:\Users\a.abdallah\Anaconda3\envs\ocr2\lib\site-packages\torch\nn\modules\module.py”, line 201, in _apply<br>
module._apply(fn)<br>
File “C:\Users\a.abdallah\Anaconda3\envs\ocr2\lib\site-packages\torch\nn\modules\rnn.py”, line 140, in _apply<br>
self.flatten_parameters()<br>
File “C:\Users\a.abdallah\Anaconda3\envs\ocr2\lib\site-packages\torch\nn\modules\rnn.py”, line 129, in flatten_parameters<br>
self.batch_first, bool(self.bidirectional))<br>
RuntimeError: CUDA error: no kernel image is available for execution on the device</p>

ptrblck: <p>This error is raised if some CUDA code wasn’t compiled for the compute capability of your device.<br>
How did you install PyTorch and which GPU are you using?</p>

Abdelrahman_Elsayed: <p>the error becuase i installed pytorch=1.5 and the code was wriiten in 1.4 i solved it thank</p>

wnnaone: <p>I’m  running into similar issues while trying to use PyTorch 1.5. How did you solve it?</p>
Explicitly Define Gradient for Certain Computation

sait: <p>Say there is function F . which cannot be computed exactly but it’s gradient can be perfectly defined can I define function  s.t I tell the autograd , how gradient should be computed instead of autograd figuring about by computation graph. So my requirement is as following,</p>
<pre><code class="lang-auto">function F(x) :
      #cannot be computed exactly 
      approx = some_decent_approx_of_F(x)
      return approx
</code></pre>
<pre><code class="lang-auto">function Grad_of_F(x):
      #can be computed exactly
     grad  = exact_grad_of_F(x)
     return grad
</code></pre>
<p>so instead of gradient computed based on computational graph of F. I want to explicitly say use Grad_of_F for computation of gradient in backward pass ? It would really helpful if someone  could point to how to do it.  Thanks In advance ?</p>

albanD: <p>Hi,</p>
<p>You will need to use a custom autograd.Function for that. <a href="https://pytorch.org/docs/stable/notes/extending.html">This</a> section of the doc shows you how to do so and how to use it.</p>
Multihead anomaly detector using PyTorch

Lizard_King: <p>Hello.<br>
I want to implement the model that consists of 5 different LSTMs as inputs where 5 corresponds to number of incoming signals of different shapes and these are followed by a joint vector, 2 FC-layers and reconstruction in order to perform some anomaly detection. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b437e4dc95e0247c503fd80131f5196bfb8bddc3.png" data-download-href="https://discuss.pytorch.org/uploads/default/b437e4dc95e0247c503fd80131f5196bfb8bddc3" title="arch"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b437e4dc95e0247c503fd80131f5196bfb8bddc3_2_690x191.png" alt="arch" data-base62-sha1="pIhHdhyQX9URA9eYWDD4tery4r9" width="690" height="191" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b437e4dc95e0247c503fd80131f5196bfb8bddc3_2_690x191.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b437e4dc95e0247c503fd80131f5196bfb8bddc3_2_1035x286.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/4/b437e4dc95e0247c503fd80131f5196bfb8bddc3.png 2x" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">arch</span><span class="informations">1353×376 96.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I assume this could be achieved by creating 5 different models sharing their weights. At each timestamp just one signal arrives and only the corresponding weights are to be updated.<br>
How should I approach this strategy using torch properly? Particularly I don`t understand how to associate joint vector “parts” with corresponding lstm cells?<br>
Thank you for your attention.</p>
Onnx to mlmodel conversion fails to generate .mlmodel file

malemo: <p>I’m trying to convert Pytorch model to MLModel with Onnx.</p>
<p>My code:</p>
<pre><code class="lang-auto">import torch
from onnx_coreml import convert
import coremltools

net = BiSeNet(19)
net.cuda()
net.load_state_dict(torch.load('model.pth'))

dummy = torch.rand(1,3,512,512).cuda()

torch.onnx.export(net, dummy, "Model.onnx", input_names=["image"], output_names=["output"], opset_version=11)

finalModel = convert(model='Model.onnx', minimum_ios_deployment_target='12')
finalModel.save('ModelML.mlmodel')
</code></pre>
<p>After the code runs Model.onnx is generated, however, .mlmodel file is not generated. There’re no errors in the console.</p>
<p>What could be the issue?</p>

ptrblck: <p>Did you verify that the generated ONNX model is running fine?<br>
If that’s the case, it seems that <code>onnx_coreml</code> fails to store it properly, so I would recommend to create an issue <a href="https://github.com/onnx/onnx-coreml">here</a>.</p>

malemo: <p>How can I verify ONNX model? This is my <a href="https://drive.google.com/file/d/1thEITb-MlbbCtbzoU-yEJkugxijXn70R/view" rel="nofollow noopener">model</a>.</p>

ptrblck: <p>You could run it with e.g. <code>onnxruntime</code> as described <a href="https://pytorch.org/docs/stable/onnx.html">here</a>.</p>
Each GPU gets some processes, occupy 12MB memory

saywwhaaaat: <p>I use pytorch set cuda device to select the gpu I want to work on.</p>
<p>But the run will allocate processes on each GPU. For the GPU I selected, the data will be loaded there, uses as much memory as the training required. For those GPUs I didn’t select, each of them will occupy 12 MB, and have processes running, all processes have the same PID.</p>
<p><code>CUDA_VISIBLE_DEVICES=3</code> can’t solve my problem, because when I do this I will get <code>invalid device ordinal</code> error, meaning my training script is somehow doing some mysterious jobs as long as I start training,  that it will “duplicate” the processes on each GPUs, and therefore all GPUs have to be visible.</p>
<p>Anyone can help this? It is quite embarrassing to let everyone on the server can see my jobs is everywhere. Thank you for your help.</p>

ptrblck: <p>If <code>CUDA_VISIBLE_DEVICES=3</code> yields an error, you would have to check your code, since you are indeed using different devices.<br>
Check the code for all <code>.cuda()</code>, <code>to()</code> and <code>device</code> usages and use the default device instead of different ones.</p>

saywwhaaaat: <p>All of them are using the device I specified, I do this by using <code>grep -r</code> <code>.cuda()</code> ,  <code>to()</code>  and  <code>device</code></p>
<p>And it is not only about the default device, we have 9 gpus here, all of them get duplicate processes.</p>
<p>Each of them will occupy 12 MB, all processes have the same PID.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bcb7ffb02472f506bf55765d9c46803c0040df9.png" data-download-href="https://discuss.pytorch.org/uploads/default/1bcb7ffb02472f506bf55765d9c46803c0040df9" title="WX20200416-142757@2x"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1bcb7ffb02472f506bf55765d9c46803c0040df9_2_690x207.png" alt="WX20200416-142757@2x" data-base62-sha1="3XST4xyzbh9WAWwwyN9b791PKUp" width="690" height="207" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1bcb7ffb02472f506bf55765d9c46803c0040df9_2_690x207.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1bcb7ffb02472f506bf55765d9c46803c0040df9_2_1035x310.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bcb7ffb02472f506bf55765d9c46803c0040df9.png 2x" data-dominant-color="E7B756"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WX20200416-142757@2x</span><span class="informations">1246×374 103 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You could try to update to the latest nightly and recheck it.<br>
Otherwise, I would recommend to use <code>CUDA_VISIBLE_DEVICES</code> to mask all unwanted devices, which will make sure to not use any memory on the masked devices.</p>
Using autograd in customized autograd Function

ttoosi: <p>Hi,</p>
<p>I am trying to define customized modules (Linear, Convolutional, etc) where I would like to have all sorts of regularizations on parameters. I provided a snippet below to clarify my question.<br>
I don’t want to calculate gradients of those regularization loss functions manually, so I am using <strong>autograd</strong>. The problem is that although the parameters of the local net I define within the backward function have <strong>requires_grad=True</strong> the <strong>output</strong> does not have a <strong>grad_fn</strong>. So, It seems to me that <strong>autograd</strong> doesn’t work inside the customized <strong>autograd</strong> function. How am I supposed to use <strong>autograd</strong> in this case?</p>
<p>Thank you,<br>
Tahereh</p>
<pre><code class="lang-auto">class LinearCustomFunc(autograd.Function):
    @staticmethod
    def forward(context, input, weight, weight_cu, bias=None):
        context.save_for_backward(input, weight, weight_cu, bias)
        output = input.mm(weight.t())
        if bias is not None:
            output += bias.unsqueeze(0).expand_as(output)
        
        return output

    @staticmethod
    def backward(context, grad_output):
        input, weight, weight_fa, bias = context.saved_tensors
        grad_input = grad_weight = grad_weight_cu = grad_bias = None

        if context.needs_input_grad[0]:
            pass 
        if context.needs_input_grad[1]:
            pass
        if context.needs_input_grad[2]:         
            local_net = nn.Sequential(nn.Linear(2,3))
            list(net.parameters())[0].register_hook(save_grad('0.wieght'))
            output = local_net(input)
            loss = nn.MSELoss()
            loss.backward()            
            grad_weight_cu = grads['0.wieght']
                     
        if bias is not None and context.needs_input_grad[3]:
            grad_bias = grad_output.sum(0).squeeze(0)

        return grad_input, grad_weight, grad_weight_cu, grad_bias
</code></pre>

albanD: <p>Hi,</p>
<p>If you’re not running with <code>creat_graph=True</code>, then the backward runs with autograd disabled.<br>
If you need to enable it locally you can use <code>with torch.enable_grad():</code>. Also I would advise you to use autograd.grad here to get the gradient more easily:</p>
<pre><code class="lang-auto">            local_net = nn.Sequential(nn.Linear(2,3)) # This will contain random weights. Is it expected?
            with torch.enable_grad():
                output = local_net(input)
                loss = nn.MSELoss()
            grad_weight_cu = autograd.grad(loss, local_net[0].weight)[0]

</code></pre>
Load a model with modified dimensions (but same structure) onto old model

sinang: <p>Hello everyone,</p>
<p>I have a model and I prune it so that its inference time gets shorter. Pruned model doesn’t change the general structure (same sequence of conv, batchnorm, relu) but dimensions for convolutional layers, batch normalization change due to pruning. When my model is pruned, I save it in the following way:</p>
<pre><code class="lang-auto">torch.save(model.state_dict(), "model_pruned")
</code></pre>
<p>Once my model is pruned and saved, I try to load it into my old model in this way:</p>
<pre><code class="lang-auto">network = OldNet().cuda()
dest = 'path/to/pruned_network'
network.load_state_dict(torch.load(dest))
network = network.eval()
</code></pre>
<p>This gives then a Runtime Error:</p>
<pre><code class="lang-auto">RuntimeError: Error(s) in loading state_dict for OldNet:
	size mismatch for features.0.weight: copying a param with shape torch.Size([28, 1, 5, 5]) from checkpoint, the shape in current model is torch.Size([32, 1, 5, 5]).
	size mismatch for features.0.bias: copying a param with shape torch.Size([28]) from checkpoint, the shape in current model is torch.Size([32]).
</code></pre>
<p>I understand why I encounter this error but I don’t know how to load my new model for inference.</p>
<p>Any help would be really appreciated! Thank you!</p>

simaiden: <p>Prune again the model instance, and then load the “pruned” weights</p>
How to fix pylint complaints like "torch.tensor is not callable", and "Pylint: Parameters differ from overridden 'forward' method"?

potatolige: <p>Hi,<br>
I am new to PyTorch and I am using Pycharm with Pylint plugin to write my code.</p>
<p>I have some annoying fake errors and warnings coming from Pylint, saying that I have some problem like “torch.tensor is not callable”, and “Pylint: Parameters differ from overridden ‘forward’ method”. However my code runs indeed properly.</p>
<p>So far, I have tried to add numpy and torch to the .pylintrc file at “ignored-classes”, “ignored-modules” and “generated-members”, but cannot ignore all the fake msgs. So any other ideas?</p>
<p>Thx in advance.<br>
Geli</p>
nn.CrossEntropyLoss equation

spaul13: <p>can anyone please tell me the underlying loss equation invoked by torch.nn.CrossEntropyLoss() in pytorch?</p>

albanD: <p>The formula is given in the doc for CrossEntropyLoss: <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss">https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss</a></p>
Question on Autograd -- Compute Jacobian, non scalar output

pomonam: <p>I have a neural network $f: \mathbb{R}^n \times \mathbb{R}^c \to \mathbb{R}^m$ and in each layer, I have $W_1 b + W_2$, where $b \in \mathbb{R}^c$ is some parameter. I would like to compute a derivative (jacobian) with respect to $b$ (e.g. $d/db f(x)$).</p>
<p>When I try to use autograd.grad (e.g. autograd.grad(output which is $\mathbb{R}^n$, $b$ which is $\mathbb{R}^c$) it says, “grad can be implicitly created only for scalar output”. Are there any methods to achieve this? Thank you.</p>

albanD: <p>Hi,</p>
<p>Yes this is a limitation of AD that it only does “vector times Jacobian” product. If you want to get the full jacobian you will need to use special code.<br>
Note that if you use nightly builds of pytorch, we added this <a href="https://pytorch.org/docs/master/autograd.html#torch.autograd.functional.jacobian">here</a>.</p>
When to use clone()

chinmay5: <p>I am really confused when and why to use clone() method. The official documentation for creating a custom autograd function makes use of clone in the backward pass <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" rel="nofollow noopener">link</a>. Why is that needed? What happens if we do not use a clone because the computation graph is not affected by the clone method. Any help is highly appreciated</p>

albanD: <p>Hi,</p>
<p>You should use <code>clone()</code> to get a new Tensor with the same value but that is backed by new memory.<br>
The tutorial uses it because it later modifies the Tensor inplace and it is forbidden to modify the gradient given to you inplace. So it first clone it to get new memory. Then the inplace change won’t break that rule.</p>
Is Sorting Differentiable?

Hakjin_Lee: <p>It might be inappropriate for this forum.</p>
<p>Now, I am implementing rotated iou loss which is used for some works on oriented object detection.<br>
But to compute intersection of two rotated boxes, it must use convex hull algorithm including sorting process.<br>
But, to my knowledge, sorting is not differentiable.</p>
<p>Is that possible to implement rotated IoU Loss?</p>
<p>You can find a example of cuda implementation on <a href="https://github.com/facebookresearch/detectron2/blob/master/detectron2/layers/csrc/box_iou_rotated/box_iou_rotated_utils.h" rel="nofollow noopener">https://github.com/facebookresearch/detectron2/blob/master/detectron2/layers/csrc/box_iou_rotated/box_iou_rotated_utils.h</a> .</p>

albanD: <p>It depends what you mean by sorting.<br>
If you just want gradients to flow from the sorted output back to the unsorted output, it works fine. As this can be seen as just shuffling the Tensor values.<br>
And the sort operator in pytorch does exactly that.</p>
Repeated matrix multiplication

superjunnis: <p>I am currently working on implementing Attention for a Variation auto encoder, as implemented in <a href="https://joao-pereira.pt/docs/accepted_version_ICMLA18.pdf" rel="nofollow noopener">https://joao-pereira.pt/docs/accepted_version_ICMLA18.pdf</a></p>
<p>I am having a hard time identifying an efficient solution for the following.</p>
<p>I have a tensor A with dimension (11000, 11000)<br>
I also have a tensor H with dimension (11000, 20)<br>
I want to populate a tensor C with the following:<br>
The dimension of C should be (11000, 20)<br>
Where the rows in C are:</p>
<p>C[j,:] = A[0,j] * H[0,:] + A[1,j] * H[1,:] + … + A[11000,j] * H[11000,:]</p>
<p>for all j in range(1,11000)</p>
<p>I tried double for-loops but it was tediously slow.</p>
<p>Hope you can provide some help, thanks in advance!</p>
<p>Best Regards, Jonas</p>

albanD: <p>Hi,</p>
<p>You want to mulitply all the values in A by all values in H in a batched manner. And then reduce over the dimension of A:<br>
<code>C = (A.unsqueeze(1) * H.unsqueeze(2)).sum(dim=1)</code>.</p>

superjunnis: <p>Thanks for your reply!</p>
<p>I tried it but it doesn’t give the desired output.</p>
<pre><code class="lang-auto">A = torch.rand(11000,11000)
H = torch.rand(11000,20)

C = (A.unsqueeze(1) * H.unsqueeze(2)).sum(dim=1)

C.shape
</code></pre>
<p>torch.Size([11000, 11000])</p>
<p>But the desired shape of C was (20, 11000).</p>
<p>Should dim=0 with the example i provided?</p>

albanD: <p>Ho sorry, the sum should be over <code>dim=2</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

superjunnis: <p>Thank you so much! Really helped out.</p>

superjunnis: <p>Hi again.</p>
<p>I tried increasing the value 20 (this is my lstm_size in my VAE model) to a higher number and it causes very high usage of memory in the calculation.</p>
<p>Below is the code to reproduce the high memory consumption</p>
<pre><code class="lang-auto">A = torch.rand(11000,11000)

H = torch.rand(11000,200)

C = (A.unsqueeze(1) * H.unsqueeze(2)).sum(dim=2)
</code></pre>
<p>The above code is an example of how the code is written in my model, which causes the session to crash.</p>
<p>When the model is ran i get the following error message, with the error referring to the calculation of C.</p>
<p>CUDA out of memory. Tried to allocate 180.30 GiB (GPU 0; 15.90 GiB total capacity; 6.81 GiB already allocated; 8.10 GiB free; 7.10 GiB reserved in total by PyTorch)</p>
<p>Is there a way to optimize the calculation of C?</p>
<p>Thanks in advance.</p>
<p>Regards Jonas</p>

albanD: <p>Yes  the intermediary Tensor will be quite big in this case!</p>
<p>Looking back at it, you can get the same result by doing: <code>A.sum(-1, keepdim=True) * H</code> which will use much less memory.</p>
Dilated Conv2d training is very slow

alexan: <p>Hi! Whenever I add convolutional layers with dilation &gt; 1, my training slows down up to 5 times. The same happens with GPU utilization: it usually stays under 20% vs 95+% without dilated convs. Using dense kernels of the same size (e. g. 5x5 instead of 3x3 with dilation = 2) eliminates the effect. <code>torch.utils.bottleneck</code> shows that dilated convs are done with <code>slow_conv_dilated2d</code>.</p>
<p>Is this slowdown an expected behavior, or have I configured something wrong?</p>
<p>System configuration:<br>
OS: Ubuntu 16.04.5<br>
Kernel: 4.15.0-91-generic<br>
NVidia driver: 418.87.01<br>
CUDA: 10.0.130<br>
Pytorch: 1.4.0, installed with pip from repo in instruction</p>

ptrblck: <p>Could you rerun your code with <code>torch.backends.cudnn.benchmark = True</code> and check, if you are seeing a speedup?<br>
This setting will profile different cudnn kernls for your current workload and will select the fastest one.<br>
Also, could you post the definition of your dilated convolutions, please?</p>

alexan: <p>With <code>benchmark = True</code> time stays the same<br>
I’m not sure what you mean by definition. I use <code>nn.Conv2d(..., dilation=2)</code> inside a <code>nn.Sequential</code></p>

ptrblck: <p>What are the other arguments you are passing the the conv to create it (<code>kernel_size</code> etc.)?</p>

alexan: <p>Here is a more complete fragment</p>
<pre><code class="lang-auto">def _conv_pad_size(k, d=1):
    return (k - 1) // 2 + k // 2 * (d - 1)

class ConvBlock(nn.Sequential):
    def __init__(self, in_, out, kernel_size=3, dilation=1, pool=1):
        padding = _conv_pad_size(kernel_size, dilation)
        super().__init__(
            nn.Conv2d(in_, out, kernel_size, padding=padding, dilation=dilation),
            nn.BatchNorm2d(out),
            nn.MaxPool2d(pool),
            nn.ReLU(inplace=True)
        )
</code></pre>
<p>I use a series of blocks with increasing number of channels. All blocks have <code>dilation=2, kernel_size=3</code></p>

alexan: <p>Could you give some pointers where to investigate further?<br>
UPD: oh. The previous message was not a reply, my bad</p>
Variational autoencoder: from LSTM to Linear

lenka: <p>Hi,</p>
<p>I am making a simple Variational autoencoder with LSTM’s where I want to take a time series as the input and generate the same time series as the output.</p>
<p>I am confused with the decoder part - I feed it with the sampled latent vectors and as the LSTM output I get hidden_size number of features per each time point. My question is how to connect this with a Linear layer whose output would be size 1, but per each time point, with shared weights?<br>
Something like TimeDistributed(Dense(1)) in keras.</p>
<p>Here is example code:</p>
<pre><code class="lang-auto">class Decoder(nn.Module):
    """Converts latent vectors into the output time series."""
    def __init__(self, ..):
          super(Decoder, self).__init__()
         ...
         self.lstm = nn.LSTM(latent_length, self.hidden_size)
           
    def forward(self, latent):
        ....
        decoder_output, _  = self.lstm(decoder_inputs)
</code></pre>
<p>decoder_output is of shape (sequence_length, n_batches, hidden_size) and I would like to connect it to a linear layer so that the output is (n_batches, n_input_features=1). Does someone know how to this correctly?</p>
<p>Thanks.</p>
Cannot Install - Wheel not supported

lucas_stokesberry: <p>I have checked all the other posts on this topic saying the same error. I have tried their solutions and none work.</p>
<p>I am using Python 3.8.2 64 bits on Windows 10.</p>
<p>I am attempting to use the wheel “torch-1.4.0+cpu-cp38-cp38-win_amd64.whl”.</p>
<p>I also tried using Python 3.7 and installing the wheel for that one, which gave the same error “…wheel not supported…”.</p>
<p>I also tried using the code given on the chooser portion of this site which gave me a different error “No matching distribution found for torch==1.4.0”.</p>
<p>Please help.</p>
<p>EDITED:</p>
<ol>
<li>
<p>I attempted:<br>
pip install torch<br>
*which gave the error: ModuleNotFoundError: No module named ‘tools.nnwrap’</p>
</li>
<li>
<p>And then I tried:<br>
pip install torch-1.4.0+cpu-cp38-cp38-win_amd64.whl<br>
*which gave the error: torch-1.4.0+cpu-cp38-cp38-win_amd64.whl is not a supported wheel on this platform.</p>
</li>
</ol>

peterjc123: <p>Please share the exact commands you used and their result. Thank you.</p>

lucas_stokesberry: <p>Edited my response, thanks.</p>

peterjc123: <p>Please try <code>python -m pip install "torch-1.4.0+cpu-cp38-cp38-win_amd64.whl" </code>.</p>

lucas_stokesberry: <aside class="quote no-group quote-modified" data-username="peterjc123" data-post="4" data-topic="76897">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peterjc123/48/2790_2.png" class="avatar"> peterjc123:</div>
<blockquote>
<p>-m pip install “torch-1.4.0+cpu-cp38-cp38-win_amd64.whl”</p>
</blockquote>
</aside>
<p>It gave "-m is not recognized as an internal or external command, operable program or batch file.</p>

peterjc123: <p>Which terminal are you using? CMD, powershell, or sth else?</p>

peterjc123: <p>It is <code>python -m pip install "torch-1.4.0+cpu-cp38-cp38-win_amd64.whl"</code> , not <code>-m pip install “torch-1.4.0+cpu-cp38-cp38-win_amd64.whl”</code>.</p>

lucas_stokesberry: <aside class="quote no-group quote-modified" data-username="peterjc123" data-post="7" data-topic="76897">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peterjc123/48/2790_2.png" class="avatar"> peterjc123:</div>
<blockquote>
<p>python -m pip install “torch-1.4.0+cpu-cp38-cp38-win_amd64.whl”</p>
</blockquote>
</aside>
<p>Wow, that worked thanks! Could you explain why that worked when I couldn’t install it using pip nor inside of PyCharm?</p>

peterjc123: <p><code>pip install torch</code> won’t work because we don’t host Windows wheels on PyPI.</p>
<p><code>pip install torch</code> doesn’t work because your <code>pip</code> points to a different python environment. You can verify the result using <code>where pip</code> in CMD.</p>
Pytorch equivalent to tensorpack's DataFlow/LMDB Serializer

glefundes: <p>Hi!</p>
<p>When using the regular Dataset/Dataloader interfaces, the IO speed is a clear bottleneck for me (no access to SSD) and training leaves the GPU very hungry. I have somewhat overcome this by using <a href="https://tensorpack.readthedocs.io/tutorial/efficient-dataflow.html#sequential-read" rel="nofollow noopener">tensorpack’s LMDB serializer</a> to store the data and load the data as bytes sequentially from a single LMDB db file.</p>
<p>The thing is, tensorpack cannot serialize PyTorch tensors or PIL images, which leaves me having to do the preprocessing (convert from numpy array to tensor, and apply torchvision transforms) during training time. While this method provides a significant performance increase speed-wise, the preprocessing is still huge bottleneck. Besides, the network for some reason does not converge as well as when using PyTorch’s regular dataset/dataloader (even though it takes 6x more time for a single epoch).</p>
<p>My question is, is there an equivalent package to tensorpack that can serialize native PyTorch fomats and read them from an LMDB database or similar? If not, is there a more efficient way to do what I’m doing here? I realize the simple answer would be to get a SSD drive for the machine, but that’s out of my control.</p>
Random cropping

sampa: <p>I have image with different image size, I want to add random cropping in my data_transform part in such a way that it will random crop 60% of the original images and then resize.<br>
I used transforms.RandomResizedCrop(256). But is it mean that it will always crop the size 256?</p>

onomatopoeia: <p>according to the <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.RandomResizedCrop" rel="nofollow noopener">documentation</a> 256 means, that the output image will have side lengths of 256 pixels. The original random cropping is determined by the other parameters <code>scale</code> and <code>ratio</code>.</p>

sampa: <p>is it possible that e output image will have side lengths of 60% of the original image? suppose the image length is 1024 * 1024 then the cropped image will be 615<em>615 or if the original image size is 512</em> 512 it will be 307*307.</p>

onomatopoeia: <p>My understanding is, if we look at the function</p>
<pre><code class="lang-auto">torchvision.transforms.RandomResizedCrop(size, scale=(0.08, 1.0), ratio=(0.75, 1.3333333333333333), interpolation=2)
</code></pre>
<p>That the image will be randomly cropped between 0.08 to 1 of the original, then given a random aspect ratio of 0.75 to 1 1/3. Then this final image is RESIZED to 256.</p>
<p>So taking your example, yes, if the original image is 1024 * 1024, it can be cropped down to 60%, then a random aspect ratio is applied, and then it is finally resized to 256 (if you set <code>size=256</code> in the function).</p>
<p>You can of course double check all of this by putting through some images and checking the outputs.</p>
<p>EDIT: the interplay between aspect ratio and scale are somewhat non-obvious tbh. On second reading of the source code it seems that the AREA of the image is scaled down, and then this is fit the an aspect ratio, and then finally resized to 256 *256.</p>
<p>So in your example scaling the image by 60% would mean (keeping aspect ratio at 1), each side would be: sqrt(0.6) * 1024.</p>
How does loss.backward() work for batches?

spadel: <p>So I am training a DDQN to play connect four at the moment. At each state, the network predicts the action the best action and moves accordingly. The code looks basically like follows:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
        for i in range(batch_size):
                while game is not finished:
                        action = select_action(state)
                        new_state = play_move(state, action)
                        pred, target = get_pred_target(state, new_state, action)
                        preds = torch.cat([preds, pred])
                        targets = torch.cat([targets, target)]
        loss = loss_fn(preds, targets)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>While training, the network is getting a little bit better, but nowhere as good as I would expect. Thinking about it, I am wondering now, whether I have actually correctly implemented the loss.backward() call. The point is, I am saving all the predictions and targets for each move in the tensors preds and targets. However, I am not tracking the states, that have led to these predictions and targets. But isn’t that necessary for the backward propagation, or is this information somehow saved?</p>
<p>Thank you very much!</p>
Training pytorch implementation of 'Tacotron 2' with custom data

Md.Yasin_Arafat_Yen: <p>Can we train the pytorch version of Tacotron 2 with our own Data?</p>

ptrblck: <p>Yes, you should be able to swap the data from your current script to your custom data.<br>
Do you see any issues with this approach?</p>

Md.Yasin_Arafat_Yen: <p>Thanks a lot for responding!</p>
<p><code>tacotron2 = torch.hub.load('nvidia/DeepLearningExamples:torchhub', 'nvidia_tacotron2')</code></p>
<p>This line loads the pre-trained tacotron2 over LJ Speech dataset. How do I load raw untrained model in order to train with my own data? And what is the line for that? Can you help?!</p>

Md.Yasin_Arafat_Yen: <p>I guess torch.hub contains all pretrained models, right? If so, torch.load(PATH) should load the desired raw model. If so, what is the path I have to give for loadin tacotron2?</p>

ptrblck: <p>If you want to use the raw model and train it, I would recommend to check out <a href="https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2">this repository</a>, which provides the model definition as well as training code.</p>

Md.Yasin_Arafat_Yen: <p>Can we train this model with JSUT dataset?</p>

kdot: <p>Hi, I am a bit curious for the default data LJSpeech we need the step bash scripts/prepare_mels.sh. In my opinion we would need the mels as well with a different dataset, but in the documentation under the  point Multi-dataset it does not implicitly name this step. Also, I was able to start training Tacotron as well as WaveGlow with my own data. So to phrase a question do we need to run prepare_mels.sh on our own data to run both models correctly?<br>
Thanks for your time.</p>

ptrblck: <p>I would assume you need to recreate the mel spectra, but feel free to create an issue with this question in the repository.</p>
<p>EDIT:<br>
I just talked to Grzegorz (author of the repo), who explained, that <code>prepare_mels.sh</code> allows you to load mels directly from you dist instead of processing wav files on the fly and is therefore recommended.</p>
<blockquote>
<p>With mels on disk use --load-mel-from-disk --training-files=filelists/ljs_mel_text_train_filelist.txt --validation-files=filelists/ljs_mel_text_val_filelist.txt<br>
The filelists specify the paths inside the dataset, just check filelists/ljs_mel_text_val_filelist.txt as an example</p>
</blockquote>
GeneralizedRCNNTransform scale_factor error

jtburden: <p>Hi All , I am new here.<br>
I currently am getting a TypeError: object of type ‘float’ has no len()</p>
<p>I assumed that this error was on my end, so I looked through torchVision to see what was causing it and thus, could correct my error.</p>
<p>However it seemed that the error was being caused by variables created within torchVision</p>
<p>GeneralizedRCNNTransform is setting  scale_factor = size / min_size</p>
<p>this then falls over in ConvTranspose2d.interpolate</p>
<p>def _check_size_scale_factor(dim, size, scale_factor):<br>
# type: (int, Optional[List[int]], Optional[float]) -&gt; None<br>
if size is None and scale_factor is None:<br>
raise ValueError(“either size or scale_factor should be defined”)<br>
if size is not None and scale_factor is not None:<br>
raise ValueError(“only one of size or scale_factor should be defined”)<br>
if not (scale_factor is not None and len(scale_factor) != dim):<br>
raise ValueError(<br>
"scale_factor shape must match input shape. "<br>
“Input is {}D, scale_factor size is {}”.format(dim, len(scale_factor))<br>
)</p>
<p>I expect this complete to be an error on my side however I can not find it , any help / guidance would be appreciated</p>

ptrblck: <p>In the current master, the lines of code seem to have been changes ~last month as seen <a href="https://github.com/pytorch/vision/blob/1b9f25111415dee8772e7b2e9c1962b7fa7349b1/torchvision/ops/misc.py#L80-L86">here</a> so it could have been a bug, which was fixed recently.<br>
Could you install the latest nightly binary (or build from source) and rerun your code?</p>

jtburden: <p>thank you this solved the issue</p>
Permute fails At Inference Time on PyTorch-XLA

ecdrid: <p>I have a model where the forward pass has something like the below.</p>
<pre><code class="lang-auto">        ............
        .............
        all_feats, _ = self.rnn(sequence_output)
        conved = all_feats.unsqueeze(1)
        conved = F.relu(self.conv(conved))
        conved = conved.squeeze()
        conved = conved.permute(0, 2, 1) #  -- it blows here;
        ...........
        ...........
</code></pre>
<p>The traceback XLA throws is weird;</p>
<pre><code class="lang-auto">Exception in device=TPU:7: torch_xla/csrc/helpers.cpp:86 : Check failed: min_shape_dim &lt;= dim &amp;&amp; dim &lt;= max_shape_dim 

*** Begin stack trace ***
	tensorflow::CurrentStackTrace[abi:cxx11]()
	torch_xla::XlaHelpers::GetCanonicalDimensionIndex(long long, long long)
	torch_xla::XlaHelpers::GetCanonicalDimensionIndices(absl::Span&lt;long long const&gt;, long long)
	torch_xla::XLATensor::permute(torch_xla::XLATensor const&amp;, absl::Span&lt;long long const&gt;)
	torch_xla::AtenXlaType::permute(at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;)
	c10::impl::wrap_kernel_functor_unboxed_&lt;c10::impl::detail::WrapFunctionIntoRuntimeFunctor_&lt;at::Tensor (*)(at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;), at::Tensor, c10::guts::typelist::typelist&lt;at::Tensor const&amp;, c10::ArrayRef&lt;long&gt; &gt; &gt;, at::Tensor (at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;)&gt;::call(c10::OperatorKernel*, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;)

  File "&lt;ipython-input-26-582cd081b9b1&gt;", line 67, in forward
    conved = conved.permute(0, 2, 1)

Value out of range (expected to be in range of [-2, 1], but got 2)Exception in device=TPU:4: torch_xla/csrc/helpers.cpp:86 : Check failed: min_shape_dim &lt;= dim &amp;&amp; dim &lt;= max_shape_dim 

*** End stack trace ***
</code></pre>
<p>NB There’s no error when the model is training; Error happens only at the inference time; I cannot understand why will that happen at the very first place;</p>
<p>I can share the model details further via a private gist or something if it’s needed;</p>
<p>Thanks;</p>

ptrblck: <p>I assume the <code>conved.squeeze()</code> might remove unwanted dimensions during inference, since <code>coved</code> has only two dimensions after the squeeze.<br>
I’m not sure, which dimensions you would like to squeeze in this line of code, but note that all dimensions with size==1 will be removed.<br>
If you are using a batch size of 1 during inference, this dimension will be removed, too.<br>
I would recommend to pass the <code>dim</code> argument to <code>tensor.squeeze</code> to make sure only the desired dims are removed.</p>

ecdrid: <blockquote>
<p>I’m not sure, which dimensions you would like to squeeze in this line of code, but note that all dimensions with size==1 will be removed.</p>
</blockquote>
<p>I will be precise from next time!</p>
<p>Yep that was the bug precisely; I was using a <code>bs</code> of 1 at inference;</p>
<p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> &lt;3;</p>
GroupNorm don't support half precision input

happyboyneu: <p>I set net.half(),and groupnorm is float,it got this error:<br>
File “/root/miniconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/root/miniconda3/lib/python3.6/site-packages/torch/nn/modules/normalization.py”, line 232, in forward<br>
input, self.num_groups, self.weight, self.bias, self.eps)<br>
File “/root/miniconda3/lib/python3.6/site-packages/torch/nn/functional.py”, line 1736, in group_norm<br>
torch.backends.cudnn.enabled)<br>
RuntimeError: Expected object of scalar type Float but got scalar type Half for argument <span class="hashtag">#3</span> ‘tensor1’<br>
but when I use batchnorm,it can run normally.How to fix this question?</p>

ptrblck: <p>Could you post a minimal code snippet to reproduce this issue, please?</p>
Implementing Weight Dropout in Linear and Conv Layers

boris: <p>How do you implement weight dropout (i.e., “DropConnect”) for linear and convolutional layers? Some simple example code would be extremely helpful.</p>

boris: <p>Still waiting for help on this. Anybody?!</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/dropconnect-implementation/70921">This topic</a> links to an implementation and discusses a custom approach.</p>
Custom regularizer

yegane: <p>I want to make sure that a model that I am training does not diverge from my main model.<br>
I was thinking that if I could add [parameters of main model] - [parameters of the model], as a regularizer, then I might be able to do this.<br>
However, I am not sure if there is any better way to do that.<br>
Is there any implementation similar to that in Pytorch?</p>

ptrblck: <p>Would it be possible to calculate a p-norm between the parameters as described <a href="https://discuss.pytorch.org/t/how-does-one-implement-weight-regularization-l1-or-l2-manually-without-optimum/7951/2">here</a> and add it to your loss?</p>
AssertionError: target labels must of int64 type

burakalp: <p>Hi there,</p>
<p>I am just a newbie in Pytorch. I cloned prajjwal1’s repository from here <strong><a href="https://github.com/prajjwal1/autonomous-object-detection" rel="nofollow noopener">https://github.com/prajjwal1/autonomous-object-detection</a></strong> and was trying to run it for bdd100k dataset. However, I keep getting this below error. It might be a simple error but I could not fix it. It is appreciated if you could help me.</p>
<p>Training started<br>
0%|                                                                                                                              | 0/25 [00:00&lt;?, ?it/s]<br>
Traceback (most recent call last):<br>
File “train_baseline.py”, line 81, in <br>
train_one_epoch(model, optimizer, dl, device, epoch, print_freq=200)<br>
File “C:\Users\Burak\autonomous-object-detection\engine.py”, line 42, in train_one_epoch<br>
loss_dict = model(images, targets)<br>
File “C:\Users\Burak\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\Burak\AppData\Local\Continuum\anaconda3\lib\site-packages\torchvision\models\detection\generalized_rcnn.py”, line 71, in forward<br>
detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)<br>
File “C:\Users\Burak\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\Burak\AppData\Local\Continuum\anaconda3\lib\site-packages\torchvision\models\detection\roi_heads.py”, line 745, in forward<br>
assert t[“labels”].dtype == torch.int64, ‘target labels must of int64 type’<br>
AssertionError: target labels must of int64 type</p>

mohaimen: <p>Check your targets data type using <code>targets.dtype</code>.<br>
If not int64, then convert it to int64 using <code>targets = targets.long()</code></p>
<p>Or even, when you create the tensor, you do it like this:<br>
<code>targets = torch.tensor([[1,0,0,0,0],[0,1,0,0,0]], dtype=torch.long)</code></p>
Reshaping Tensor: Basic

Rana_Fida: <p>Hi,<br>
If we have a linear tensor of 24 elements from 0 to 23.<br>
X = [0 1 2 3 …23]<br>
How can we reshape it to something like that<br>
Y = [[0,1 2 3 12 13 14 15],<br>
[4 5 6 7 16 17 18 19],<br>
[8 9 10 11 20 21 22 23]]</p>
<p>Thanks</p>

abodh_ltd: <pre><code class="lang-python">x = torch.range(1,24)
print (x)
x = x.view(3,-1)
print (x)


Output:
tensor([[ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.],
        [ 9., 10., 11., 12., 13., 14., 15., 16.],
        [17., 18., 19., 20., 21., 22., 23., 24.]])

</code></pre>

Rana_Fida: <p>Well The issue is we don’t want it [1 2 3 4 5 6 7 8]<br>
we want it to be [0 1 2 3 <strong>12 13 14 15</strong>]<br>
and so on… there is gap. See expected output given in the question</p>

vdw: <p>Well you can first split your original tensor in half, reshape both halves and concatenate them again:</p>
<pre><code># initilizing list 
X = list(range(0, 24))

# converting list to array 
X = np.array(X) 

# converting array to tensor
X = torch.from_numpy(X)

# split tensor in half
X_split = torch.split(X, int(X.shape[0]/2), dim=0)

# reshape the two tensors
X_left = X_split[0].view(3,-1)
X_right = X_split[1].view(3,-1)

# concatenate the reshaped tensors again
X = torch.cat((X_left, X_right), dim=1)
</code></pre>
<p>That wil give you:</p>
<pre><code>tensor([[ 0,  1,  2,  3, 12, 13, 14, 15],
        [ 4,  5,  6,  7, 16, 17, 18, 19],
        [ 8,  9, 10, 11, 20, 21, 22, 23]])</code></pre>

abodh_ltd: <p>Ohh I didn’t notice. Sorry about that. Chris (<a class="mention" href="/u/vdw">@vdw</a>)  has just posted a solution. I was about to write the same. See if that works for you.</p>
What is the meaning of Kenel size 1x7 in CNN (Inception Network)

sgaur: <p>Hi Guys,</p>
<p>I am building Inception network for my project and come through the kernel size 1x7 or 7x1 and some of its variant. what does it mean.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b11e69b794ac3e9146f50c0b6362482336d4d35b.png" data-download-href="https://discuss.pytorch.org/uploads/default/b11e69b794ac3e9146f50c0b6362482336d4d35b" title="Screenshot from 2020-04-15 16-40-34"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b11e69b794ac3e9146f50c0b6362482336d4d35b_2_564x499.png" alt="Screenshot from 2020-04-15 16-40-34" data-base62-sha1="pgRFE89lweubvUVSQAUhVtsP5B9" width="564" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b11e69b794ac3e9146f50c0b6362482336d4d35b_2_564x499.png, https://discuss.pytorch.org/uploads/default/original/3X/b/1/b11e69b794ac3e9146f50c0b6362482336d4d35b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/1/b11e69b794ac3e9146f50c0b6362482336d4d35b.png 2x" data-dominant-color="F8F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-04-15 16-40-34</span><span class="informations">657×582 73.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

mohaimen: <p>7 x 1 means its width is 1 and depth is 7 (a column vector)<br>
On the other hand, 1 x 7 mean its width is 7 and depth is 1 (a row vector)</p>
<p>Both of them are one dimensional but using conv2d.</p>
Question about concatenate the FC layer features of each model and use then train the concatenated features with FC layer

raghunath_puttagunta: <p><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/db5451df223c0f01b8d43a8702521817fb08b794.png" alt="Untitled%20Diagram" data-base62-sha1="vihdFbagI40LOTubZ6FrGr3U75O" width="471" height="321"><br>
I have 2 different models as shown above and I want to take features from FC layer of each model and concatenate them and pass it through another FC layer and train them. The question I have is if I should have 3 loss functions (one loss function for each network ) or one loss function at the end of the whole network. The concern I have about using one loss function is while backward propagation does the network choose to change the weights of only one network because it might lead to more decrease in loss</p>

luciaL: <p>I think you should try. Try a lot, select one.</p>
<p>I have the same models as shown above, but I don’t know how to extract the FC layers, could you please help me with that?</p>