akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torchaudio in win10

username: YH-WEI
cooked: <p>Dear experts, when will the torchaudio be compatible with win10? I really need it. Thank you very much!</p>

username: vincentqb
cooked: <p>You can track the progress <a href="https://github.com/pytorch/audio/issues/425" rel="nofollow noopener">here</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to optimize variable input, variable output network?

username: Ognjen_Kocic1
cooked: <p>Current optimization techniques, that I’m aware of, are not applicable to the wide range of networks that are built for image-to-sequence or sequence-to-sequence problems where both image dimensions and output sequences can vary. Some applications that can utilize such models are:</p>
<ol>
<li>image captioning</li>
<li>speech-to-text and vice versa</li>
<li>OCR</li>
<li>image-to-speech (e.g. describing photos to blind or almost blind people)</li>
</ol>
<p>Such models can for example be built from fully convolutional backend (spatial or temporal - it doesn’t really matter), after which some attention and RNN decoder are applied. Additional complexity with variable sized input comes from the fact that real user data cannot be efficiently batched all the time due to large difference in input shapes (huge padding introduces too much wasted computation).</p>
<p>As specified here: <a href="https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936" class="inline-onebox">What does torch.backends.cudnn.benchmark do?</a>, cudnn benchmark which does wonders for some class of models doesn’t work. Switching to float16 inference might give minimal boost on Volta+ architectures, but TensorCores require some strict shape restrictions to be properly utilized and that might be problematic with variable batch dimension. Exporting and converting to TensorRT network is again problematic for variable in/out shape networks.</p>
<p>Am I missing something or there are no good general strategies for optimizing these and alike models?</p>
Pytorch ImageNet dataset

username: Zend_AI
cooked: <p>Hi all;<br>
I am Unable to download the orignal ImageNet dataset from their official website, However, I found out that pytorch has ImageNet as one of it’s torch vision datasets.<br>
Q1. Is that the original ImageNet dataset?<br>
Q2. How do I get the classes for the dataset like it’s being done in Cifar-10<br>
classes = [‘airplane’, ‘automobile’, ‘bird’, ‘cat’, ‘deer’,<br>
‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’]</p>
Different Optimisers In Different Layers

username: James_e
cooked: <p>Hi,</p>
<p>I am currently looking into hybrid neural networks. I would like to apply a different optimizer to quantum circuit compared to the classical layers.</p>
<p>The quantum circuit has classical input and parameters so it can be optimised by an classical optimiser.</p>
<p>My network looks like</p>
<pre><code class="lang-auto">class NeuralNet(nn.Module)

  def __init__(self):
    super(NeuralNet, self).__init__()
    self.pre_net = nn.Linear(28*28, 4)
    self.q_params = nn.Parameter(0.01 * torch.randn(q_depth * 4)) #Quantum circuit parameters 
    self.post_net = nn.Linear(4, 10)
</code></pre>
<p>1.) How can I apply the same optimiser to all layers, but have a different step size for the quantum circuit parameters</p>
<p>2.) How can I apply a completely different optimiser to the quantum circuit parameters?</p>
<p>Thanks for your help</p>
<p>James</p>

username: ptrblck
cooked: <p>You could pass only a subset to the corresponding optimizers.<br>
<a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options">This per-param example</a> shows how to use different arguments for specific parameter sets, but you could use the syntax to create two optimizers with different parameters.</p>

username: James_e
cooked: <p>Thanks for the reply! Could you possibly provide a quick example for the two optimiser case?</p>
<p>Really appreciate the help <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Torchscript with spconv

username: Jakov_Gaspar
cooked: <p>Hello,</p>
<p>I’m trying to convert my model via jit.trace(). I’m currently using the spconv library for my Network. Training the network works fine and with no errors, but using jit.trace() throws a size mismatch error.</p>
<p>Here are the layers of my model:</p>
<pre><code class="lang-auto">self.net = spconv.SparseSequential(
            spconv.SparseConv2d(1, 64, 1),
            nn.PReLU(),
            spconv.SparseConv2d(64, 256, 2, padding=(1, 0)),
            nn.PReLU(),
            spconv.SparseConv2d(256, 512, 2, padding=(1, 0)),
            nn.PReLU(),
            spconv.SparseConv2d(512, 256, 2, padding=(0, 1)),
            nn.PReLU(),
            spconv.SparseConv2d(256, 64, 2, padding=(0, 1)),
            nn.PReLU(),
            spconv.SparseConv2d(64, 1, 1),
            # nn.Tanh(),
        )
</code></pre>
<p>Here is the error message:</p>
<pre><code class="lang-auto">RuntimeError: size mismatch, m1: [375 x 375], m2: [1 x 64] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:290
</code></pre>
<p>The input consists of 375x375 matrices. The network operates on CUDA.</p>
<p>Maybe someone has an idea why this error comes up.</p>
<p>Thank you</p>

username: driazati
cooked: <p>That error looks like it’s coming from the actual operation, not anything to do with tracing. Are you tracing it with the same inputs you’re using to run it successfully in normal PyTorch?</p>
<p>If your model has any control flow that depends on the inputs tracing may not work correctly. Can you also post the full code you’re using to run/trace your model?</p>

username: Jakov_Gaspar
cooked: <p>Hi,</p>
<p>I already solved the problem, you were right. I accidentally passed a float as batch size, which caused this error.</p>
How is sequencing established in pytorch RNNs?

username: tom_q
cooked: <p>From what I understand, unlike feedforward nerual networks, the sequence of data is really important in RNNs because prediction and data from observation 1 of X is used in the prediction of subsequent rows.</p>
<p>But in tutorials I’ve seen, the forward function in RNNs still seem to operate on the whole training matrix X like it’s about to do everythign in parallel.</p>
<p>So I tried to create an empty list to hold all the predictions for the calculation of the loss as well as a loop to iterate through the rows in order like this:</p>
<pre><code class="lang-auto">    def forward(self, X0):
        predictions = []
        for i in range(X0.shape[0]):
</code></pre>
<p>But doing things this way, the predictions are disconnected from the parameters and the process of backpropogation will not work. So my questions are, how should I design my model to process each row of the training matrix in order from top to bottom if not with a loop and how should the predictions in each RNN cell be recorded?</p>
Loading & Freezing a Pretrained Model to Combine with a New Network

username: CCL
cooked: <p>I have a pretrained model and would like to build a classifier on top of it. I’m trying to load and freeze the weights of the pretrained model, and pass its outputs to the new classifier, which I’d like to optimise. Here is what I have so far, I’m a little stuck on a <code>TypeError: forward() missing 1 required positional argument: 'x'</code> error from the <code>nn.Sequential</code> line:</p>
<pre><code class="lang-auto">import model #model.py contains the architecture of the pretrained model

class Classifier(nn.Module):
    def __init__(self):
        ...
    def forward(self, x):
        ...

net = model.Model()
net.load_state_dict(checkpoint["net"])

for c in net.children():
    for param in child.parameters():
        params.requires_grad = False

model = nn.Sequential(nn.ModuleList(net()), Classifier())
</code></pre>

username: ptrblck
cooked: <p>I assume, the error is thrown in <code>nn.ModuleList(net())</code>, as you are trying to call <code>net()</code> without any inputs.<br>
Note that <code>net(input)</code> will return the output of the forward pass, so wrapping it in an <code>nn.ModuleList</code> won’t work.<br>
Try to pass both models directly to <code>nn.Sequential</code>.</p>

username: CCL
cooked: <p>I’m unfamiliar with how weights are loaded into a model in Pytorch behind the scenes. Could I still load &amp; freeze the state_dict of the model without instantiating it (ie. directly passing the class)?</p>

username: ptrblck
cooked: <p>You would need to create an instance of your class, which will create the parameters, which you could freeze thereafter.<br>
The <code>state_dict</code> contains tensors mapped to the module names from its model, which do not require gradients by default and only store the values.</p>

username: CCL
cooked: <p>I think I’m misunderstanding something here… If I do the following, will I still be able to use the pretrained parameters of <code>model.Model()</code>?</p>
<pre><code class="lang-auto">net = model.Model()
net.load_state_dict(checkpoint["net"])
for c in net.child ... #freeze

model = nn.Sequential(model.Model(), Classifier())
</code></pre>

username: CCL
cooked: <p>Ah I should be doing <code>nn.Sequential(net, Classifier())</code> instead… Thanks for all your help!</p>
Code autocomplete problem

username: somethingForNothing
cooked: <p>I just started learning pytorch, I don’t quite understand why I can’t use auto-completion for some functions in Pycharm and VS Code.Such as torch.empty(), torch.zeros(), etc.<br>
Can you help me with this problem?<br>
Thanks~   o(<em>￣︶￣</em>)o</p>

username: ptrblck
cooked: <p>This issue is tracked <a href="https://github.com/pytorch/pytorch/issues/7318">here</a> and fixes are actively developed. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

username: tom
cooked: <p>Here is a pyi that PR12500 generates:<br>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/t-vi/0d0ae013072f96f50fa11fbc2287e33b" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/t-vi/0d0ae013072f96f50fa11fbc2287e33b" target="_blank" rel="nofollow noopener">https://gist.github.com/t-vi/0d0ae013072f96f50fa11fbc2287e33b</a></h4>
<h5>__init__.pyi</h5>
<pre><code class="Python">
from typing import List, Tuple, Optional, Union, Any, ContextManager, Callable, overload

import builtins
import math
import pickle


class dtype: ...
_dtype = dtype</code></pre>
This file has been truncated. <a href="https://gist.github.com/t-vi/0d0ae013072f96f50fa11fbc2287e33b" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>As far as I know it works with mypy and PyCharm and has reasonable coverage.</p>
<p>Best regards</p>
<p>Thomas</p>

username: binbin
cooked: <p>This pyi file solves most built-in functions autocomplete problem. Many thanks to tom. However, when I try to type torch.nn, Pycharm shows warning ‘cannot find reference nn in __init__.pyi’. It seems this pyi file shadows all package of torch like torch.nn and torch.autograd. Is there any way to solve this? I will be very appreciated with that!</p>

username: yannick
cooked: <p>the pytorch 1.0.1 still not works. the torch.randn, torch.FloatTenser() can not be autocompleted by vscode or wingide.</p>

username: tom
cooked: <p>torch.FloatTensor is deprecated, so not recommending that is a feature.<br>
There are bugs to be expected, but I see that you(?) already found <a href="https://github.com/pytorch/pytorch/issues/16996" rel="nofollow noopener">issue 16996</a>, which is a good place to report those.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Umar_Karim
cooked: <p>I faced a similar problem with <em>pytorch 1.4</em>, <em>vs code 1.42</em> with <em>torch.nn</em> as it was not showing autocomplete. I changed the language server from <em>microsoft</em> to <em>jedi</em> and now autocomplete works.</p>
SPP Layer and adaptiveAvgPool layer to have a fixed shape,which will be preferred?

username: liwm29
cooked: <ul>
<li>To replace the use of <code>resize</code> or <code>crop</code>  , I can use spp layer, i.e. spational pyramid pooling layer<br>
or adaptiveAvgPool layer to have a fixed shape which can be used as the input of FC layer after the Conv layer.</li>
<li>But which will be preferred?</li>
</ul>
<hr>
<ul>
<li>I simply know that spp layer will extract input features using some different  kernel shape into some fixed output shape and then flatten them</li>
<li>adaptiveAvgPool can auto compute the kernel size and stride to get a fixed output shape</li>
</ul>
How the gradient in Linear layer be computed using transpose?

username: liwm29
cooked: <h3><a name="im-wondering-why-i-need-to-transpose-the-grad_output-when-calculate-grad_weight-1" class="anchor" href="#im-wondering-why-i-need-to-transpose-the-grad_output-when-calculate-grad_weight-1"></a>I’m wondering why i need to transpose the grad_output when calculate grad_weight?</h3>
<ul>
<li>the linear layer is to apply a linear transformation to the incoming data: y = xA^T + b</li>
<li>What i’m wondering is that how the gradient of weight matrix be computed by code like the following:<br>
If i define a backward func,i should write like this:</li>
</ul>
<pre data-code-wrap="py"><code class="lang-py">def backward(ctx , grad_output)
  imput,weight,bias = ctx.saved_tensors

  grad_input = grad_output.mm(weight)
  grad_weight = grad_output.t().mm(input)
  grad_bias = grad_output.sum(0).squeeze(0)
  return grad_input,grad_weight,grad_bias
</code></pre>
<blockquote>
<p>weight ,bias and input are cached by the forward action and use ctx.saved_tensors to reuse</p>
</blockquote>
<ul>
<li>I know it do make it right because only we transpose it can we have the correct shape to have a matrix multiply(torch.mm)</li>
<li>But is there any mathematical reason for the code of  transpose?</li>
</ul>
<hr>
<p>THANKS!</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you write down the values for each entry in the matrix, you will see that the indices a flipped. Which means that in code, you should add a Transpose.</p>

username: liwm29
cooked: <p>Thanks a lot , I will have a try</p>
Use DenseNet121 with MKL-DNN

username: nexgus
cooked: <p>Hi there,</p>
<p>Consider the simple Python script as following:</p>
<pre><code class="lang-python">import torch
import torchvision

from torch.utils import mkldnn as mkldnn_utils

net = torchvision.models.densenet121()
net.eval()
net = mkldnn_utils.to_mkldnn(net)

data = torch.randn(1, 3, 224, 224)
data = data.to_mkldnn()

_ = net(data)
</code></pre>
<p>We retrieve a DensNet 121 by <code>torchvision</code> and convert it to MKL-DNN data format. Besides, we generate a data and reorder it to MKL-DNN data layout.</p>
<p>When the last line of the script is executed, an exception is raised:</p>
<pre><code class="lang-auto">RuntimeError: Could not run 'aten::_cat' with arguments from the 'MkldnnCPUTensorId' backend. 'aten::_cat' is only available for these backends: [CPUTensorId, VariableTensorId].
</code></pre>
<p>Obviously, <code>torch.cat()</code> is not supported while backend is MKL-DNN.</p>
<p>Well, my problem is, how do I use DenseNet 121 (from <code>torchvision</code>) and also get the benefit of MKL-DNN?</p>
<p>B.Rds,<br>
Augustus</p>
Is gradient calculated based on the last forward operation?

username: neouyghur
cooked: <p>Hi I have a bad feeling about on my code, although it does not return any error.</p>
<pre><code class="lang-auto">    for idx, inputs in enumerate(data_loader):
        data_time.update(time.time() - end)
        bsz = inputs.size(0)

        config.STEP = config.STEP + 1
        inputs = inputs.to(config.DEVICE)
        # forward
        # TODO: update???
        x1, x2 = torch.split(inputs, [3, 3], dim=1)
        x1.contiguous()
        x2.contiguous()
        x1 = x1.cuda(non_blocking=True)
        x2 = x2.cuda(non_blocking=True)

        with torch.set_grad_enabled(config.TRAIN_BACKBONE):
            fea1 = net(x1)
            fea2 = net(x2)
</code></pre>
<p>I am calculating the loss based on the <code>fea1</code> and <code>fea2</code>. Is that correct?  Does <code>fea1</code> contribute to the gradient?</p>

username: albanD
cooked: <p>Anything that is computed in a differentiable way and that contribute to the loss will also contribute to the computes gradients. So yes, both will participate in the gradients in the paramters of <code>net</code>.</p>

username: neouyghur
cooked: <p><a class="mention" href="/u/alband">@albanD</a> then can we split a large batch to <code>N</code> small batches, then get an accumulated results after <code>N</code> forwards to increase our batch size? I didn’t see some one used such a way to increase the batch size, instead they do accumulated based on the back propagated gradient.</p>

username: albanD
cooked: <p>You can do any combination, depending on what your constraints are. You can see this post for a more detailed description: <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/19" class="inline-onebox">Why do we need to set the gradients manually to zero in pytorch?</a></p>

username: neouyghur
cooked: <p>Thanks for answering. Find out many interesting related discussion. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
RuntimeError: cuda runtime error (214) : uncorrectable ECC error encountered at /pytorch/aten/src/THC/generic/THCTensorMathPointwise.cu:207

username: LucasX
cooked: <p>I have 4 V100 gpus on my machine. When I run with <code>CUDA_VISIBLE_DEVICES=1(3) python3 train.py</code> it works. But <code>CUDA_VISIBLE_DEVICES=0(2) python3 train.py</code> it raises error:</p>
<blockquote>
<p>RuntimeError: cuda runtime error (214) : uncorrectable ECC error encountered at /pytorch/aten/src/THC/generic/THCTensorMathPointwise.cu:207</p>
</blockquote>
<p>It seems only work on the 2nd and 4th devices but fails on the 1st and 3rd devices.</p>
<p>Environment:</p>
<ul>
<li>CUDA 10.1</li>
<li>PyTorch 1.3.1</li>
<li>NVIDIA DRIVER 418.116.00</li>
</ul>
<p>How to fix this problem?</p>

username: ptrblck
cooked: <p>This error refers to a hardware failure.<br>
How often do you see these errors and are these devices warmer than the other ones?</p>

username: LucasX
cooked: <p>Almost every time. The 1st and 3rd devices are not warmer than others.</p>

username: ptrblck
cooked: <p>Could you try to use the 1st and 3rd device in another workstation and run some tests to see, if you still see these memory errors?</p>
RuntimeError: std::bad_alloc: temporary_buffer::allocate: get_temporary_buffer failed in RPN

username: Deltap
cooked: <p>Hi,<br>
Here is the error message:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/shiqian/zhangzh/FAFRCNN/train.py", line 28, in &lt;module&gt;
    train()
  File "/home/shiqian/zhangzh/FAFRCNN/train.py", line 24, in train
    transfer_train_fast('/home/shiqian/zhangzh/FAFRCNN/checkpoints/fs_40tr_27.5con_d1_0.75l-0.20a-1.00b-1.00/03090209_0.310')
  File "/home/shiqian/zhangzh/FAFRCNN/transfer_train_fast.py", line 82, in transfer_train_fast
    adv_meters, meters, ins_adv_optimizer, img_adv_optimizer, train_num=cfg.dis_train_num)
  File "/home/shiqian/zhangzh/FAFRCNN/dis_train.py", line 61, in dis_train
    faster_rcnn_mp.rpn_features(t_images, t_features, t_targets)
  File "/home/shiqian/zhangzh/FAFRCNN/model/faster_rcnn/faster_rcnn_model_parallel.py", line 269, in rpn_features
    proposals, proposal_losses, iou, anchor_len = self.rpn(images, features, targets)
  File "/home/shiqian/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/shiqian/zhangzh/FAFRCNN/model/faster_rcnn/faster_rcnn_model_parallel.py", line 344, in forward
    objectness.to(cfg.gpu5), pred_bbox_deltas.to(cfg.gpu5), labels, regression_targets)
  File "/home/shiqian/.local/lib/python3.5/site-packages/torchvision/models/detection/rpn.py", line 371, in compute_loss
    reduction="sum",
  File "/home/shiqian/.local/lib/python3.5/site-packages/torch/nn/functional.py", line 2179, in l1_loss
    ret = torch._C._nn.l1_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
RuntimeError: std::bad_alloc: temporary_buffer::allocate: get_temporary_buffer failed
</code></pre>
<p>I encountered this error many times. It always happens at RPN’s loss computation. Sometimes I can train my model for many epochs without this error, but sometimes it occurred at the second epoch. I am training my model on a server, which should have enough memory. This is the output of <code>free -h</code> in bash:</p>
<pre><code class="lang-auto">              total        used        free      shared  buff/cache   available
Mem:           376G        103G         79G         10G        193G        259G
Swap:          7.6G        5.4G        2.2G

</code></pre>
<p>I modified the <code>FasterRCNN</code> in pytorch to enable model parallel manually, other components are basically the same.</p>
How to check if the preceding layer is ReLU?

username: Liangzhi_Tang
cooked: <p>For example, the following is part of ResNet18, is there a simple code snippet to check if the inputs of “layer1.0.conv2” are all greater than 0 (ReLU is used before)?</p>
<blockquote>
<p>ResNet(<br>
(conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)<br>
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)<br>
(layer1): Sequential(<br>
(0): BasicBlock(<br>
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
(1): BasicBlock(<br>
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
)<br>
(layer2): Sequential(<br>
(0): BasicBlock(<br>
(conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU(inplace=True)<br>
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(downsample): Sequential(<br>
(0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)<br>
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)</p>
</blockquote>

username: albanD
cooked: <p>Hi,</p>
<p>I don’t think there is such a thing that you can use a very reliable way.<br>
Is <code>input.ge(0).all()</code> good enough for you?</p>

username: Liangzhi_Tang
cooked: <p>Thanks for the replay. Which way is faster, input.min() == 0  or  input.ge(0).all()?</p>

username: albanD
cooked: <p>Not, sure <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> You can try both. They should be fairly similar.</p>
Pytorchaudio Spectrogram Output Size:- Unexpected number of SFTs

username: Paul_Creaser
cooked: <p>Hi,</p>
<p>I am applying the pyaudio spectrogram to a 512 sample audio sample.</p>
<p>An example below:-</p>
<p>waveform = 512 samples<br>
specgram = torchaudio.transforms.Spectrogram(hop_length=64)(waveform)</p>
<p>I thought this would generate 512/64=8 Hops, so 8 SFTs for the spectrogram, however it generates 9.</p>
<p>waveform = 512 samples<br>
specgram = torchaudio.transforms.Spectrogram(hop_length=128)(waveform)</p>
<p>I thought this would generate 512/128=4 SFTs for the spectrogram, however it generates 5.</p>
<p>I guess it may start at the  -hop_length/2 and finishes at 512 + hop_length/2</p>
<p>Pad is set to zero by default. If it is set to 32, it increases the number of SFTs by 1 as expected.</p>
Recommended ways to calculate statistics of model?

username: kirk86
cooked: <p>I was wondering if anyone has any suggestions on recommend ways to calculate some statistics of a model which do not require gradient to be calculated for those?</p>
<p>I’ve been looking at some code base which does this for vgg16 but the way they are doing it seems a bit odd to me.</p>
<p>First they instantiate the model, then thy instantiate a replica of the same model which they pass into a <code>class ComputeStats(nn.Module)</code>.</p>
<p>Second, the above class in its init(), actually strips away all the params of the replica model and puts in their place placeholds for the stats to be computed using zero tensors applying the following function recursively to every module of the replica model.</p>
<pre><code class="lang-auto">def module_parameters(module, params):                 |                                                                                                               
      for name in list(module._parameters.keys()):
          if module._parameters[name] is None:
             continue

          data = module._parameters[name].data
          module._parameters.pop(name)                                  

         module.register_buffer("%s_dummy1" % name, data.new data.size()).zero_())
         module.register_buffer("%s_dummy2" % name, data.new(data.size()).zero_())
                                                                                                                                                                                                                                                      
         params.append((module, name))
</code></pre>
<p>Finally, in the <code>class ComputeStats(nn.Module)</code> in the forward function they just return the forward of the replica model.<br>
<code>def forward(self, *args): return self.replica(*args)</code></p>
<p>MWE:</p>
<pre><code class="lang-auto">1. model = vgg16()
2. replica = vgg16()
3. class ComputeStats(replica)
4. in the __init__() of ComputeStats execute replica.apply(lambda module: module_parameters(module=module, params=list()))
5. in the forward of ComputeStats::forward(self, *args): return replica(*args)
</code></pre>
<p>Is this practice common or is a better simpler way to achieve this?</p>
Difference between model.children() and model.features

username: nihal-rao
cooked: <p>I am using a pretrained mobilenet as follows :</p>
<pre><code class="lang-auto">model = torchvision.models.mobilenet_v2(pretrained=True)
</code></pre>
<p><code>model.children()</code> gives all the layers, including the last classification head.<br>
However , <code>model.features</code> gives all the layers <strong>excluding</strong> the classification head.<br>
Why is this so? Are there any cases where both give the same result?<br>
I would also be thankful if anyone pointed me to the PyTorch documentation for <code>.features</code> (I couldn’t seem to find it).</p>

username: ptrblck
cooked: <p><code>model.features</code> is defined inside the model’s <code>__init__</code> method and thus not a general attribute of <code>nn.Module</code>s.<br>
Depending how you are designing your model you might want to create certain “blocks” e.g. features, classifier etc. In autoencoder-like models you would most likely find a <code>model.encoder</code> and <code>model.decoder</code> attribute.</p>
Index concept in torch.utils.data.DataLoader

username: Niki
cooked: <p>Hi,</p>
<p>I got confused about the image index concept by using <code>torch.utils.data.DataLoader</code>. This index is the index of image for the entire training/testing dataset or just index of image for the mini_batch? If it’s for the mini_batch then it means in the next mini-batch, the image with index 1 is the same as the previous image with index 1?</p>

username: ptrblck
cooked: <p>The <code>DataLoader</code> uses a sampler (e.g. <code>RandomSampler</code> if you specify <code>shuffle=True</code>) to create indices in the range <code>[0, len(dataset)]</code>. These indices will be used to index the passed <code>Dataset</code> instance, which will call into its <code>__getitem__(self, index)</code> method.<br>
So yes, if the <code>DataLoader</code> uses the same index in the next epoch, <code>Dataset.__getitem__</code> will also see the same index.</p>

username: Niki
cooked: <p>Thank you for your reply, <a class="mention" href="/u/ptrblck">@ptrblck</a>.  So this means using <code>torch.utils.data.DataLoader</code> for <code>trainloader </code>, then <code>batch_idx</code>s in</p>
<p><code>for batch_idx, (images, targets) in enumerate(trainloader):</code></p>
<p>are indices for entire dataset?</p>

username: ptrblck
cooked: <p>No. The custom or internal <code>sampler</code> will create the indices and pass them to the <code>Dataset</code>.<br>
<code>batch_idx</code> is created by your <code>enumerate</code> call and will assign an increasing index to each batch, which usually contains more than a single sample.</p>

username: Niki
cooked: <p>Then how I can get the index of each image?</p>

username: ptrblck
cooked: <p>You could write a custom <code>Dataset</code> and return the index with the data and target.</p>
<pre><code class="lang-python">def __getitem__(self, index):
    # your data loading logic
    x = self.data[index]
    y = self.target[index]
   
    # transformations
    ...

    return x, y, index
</code></pre>
<p>This would yield an additional batch of the used indices in your training loop.</p>
How to safely load training data from memory

username: R_Tux
cooked: <p>Hi</p>
<p>My training data has the shape <code>torch.Size([330000,7225])</code> and is currently saved and retrived by using <code>torch.save()</code> and <code>torch.load()</code> before being split into batches using <code>TensorDataset</code> and <code>DataLoader</code> . However when loading i get the error:</p>
<pre><code class="lang-python">RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:47] 
((ptrdiff_t)nbytes) &gt;= 0. alloc_cpu() seems to have been called with negative number: 18446744066066682432
</code></pre>
<p>I suspect this is because the file is too large to be loaded all at once (it is 8.88 GB). Is there an alternative way in which I don’t have to load all of the training data first or would you recommend I save my data using an alternative approach?</p>

username: ptrblck
cooked: <p>I tried to reproduce this error my storing and loading a tensor with the posted shape, but cannot reproduce this issue.<br>
Could you post some information about your system, i.e. OS, PyTorch version, etc.?</p>

username: R_Tux
cooked: <p>Here is some information about my system:</p>
<p>PyTorch version: <code>1.4.0</code></p>
<p>Hardware and system information:</p>
<pre><code class="lang-python">======================================== System Information ========================================
System: Windows
Node Name: LAPTOP-LFIC0N5G
Release: 10
Version: 10.0.18362
Machine: AMD64
Processor: Intel64 Family 6 Model 158 Stepping 13, GenuineIntel
======================================== Boot Time ========================================
Boot Time: 2020/3/2 12:30:58
======================================== CPU Info ========================================
Physical cores: 6
Total cores: 12
Max Frequency: 2592.00Mhz
Min Frequency: 0.00Mhz
Current Frequency: 2592.00Mhz
CPU Usage Per Core:
Core 0: 12.5%
Core 1: 3.4%
Core 2: 12.5%
Core 3: 5.9%
Core 4: 14.3%
Core 5: 2.1%
Core 6: 5.6%
Core 7: 1.4%
Core 8: 3.2%
Core 9: 2.2%
Core 10: 3.7%
Core 11: 2.4%
Total CPU Usage: 5.8%
======================================== Memory Information ========================================
Total: 15.84GB
Available: 4.18GB
Used: 11.66GB
Percentage: 73.6%
==================== SWAP ====================
Total: 52.62GB
Free: 37.13GB
Used: 15.49GB
Percentage: 29.4%
======================================== Disk Information ========================================
Partitions and Usage:
=== Device: C:\ ===
  Mountpoint: C:\
  File system type: NTFS
  Total Size: 476.15GB
  Used: 250.62GB
  Free: 225.53GB
  Percentage: 52.6%
Total read: 451.03GB
Total write: 450.79GB
</code></pre>

username: ptrblck
cooked: <p>Could you try to free more system memory, as you only have ~4GB + swap?</p>
Concatenate parameter and tensor in pytorch

username: mikeaalv
cooked: <p>It is an RNN type of structure. Because of the question setting, it is reasonable to make the initial value of the hidden state to be a concatenation of tensor and parameters. For the tensor part, the initial value is known, and for the parameter part, it is trainable. I try to torch.cat() parameter and tensor, and it seems not possible.</p>
<pre><code class="lang-auto">initialvec=torch.zeros(1,1,3)
hidden_train=torch.zeros(1,1,3)
hidden_train=nn.Parameter(hidden_train,requires_grad=True)
torch.cat(hidden_train,initialvec) 
</code></pre>
<p>Error message</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-34-1f7bbc984497&gt; in &lt;module&gt;
      2 hidden_train=torch.zeros(1,1,3)
      3 hidden_train=nn.Parameter(hidden_train,requires_grad=True)
----&gt; 4 torch.cat(hidden_train,initialvec)

TypeError: cat(): argument 'tensors' (position 1) must be tuple of Tensors, not Parameter
</code></pre>
<p>What’s a good way to concatenate a parameter and tensor?</p>
<p>Thank you.<br>
Best.<br>
Yue</p>

username: albanD
cooked: <p>Hi,</p>
<p><code>torch.cat</code> takes a tuple as input, not multiple parameters:<br>
<code>torch.cat([hidden_train,initialvec] dim=X)</code> in your case <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: mikeaalv
cooked: <p>Thank you. Sorry I didn’t realize the grammar error ;-&gt;</p>
Error index 101 is out of bounds for axis 0 with size 101

username: cdalvarezb
cooked: <p>Hello guys, i´m trying to solve the Poisson´s equation by using the relaxation method to know the potential and electric field matrixs but i have an error that say  “index 101 is out of bounds for axis 0 with size 101” and i dont understand what is the problem. ¿Can anyone helpme please?</p>
<p>import numpy as np<br>
import math</p>
<h1>SYMBOLS USED IN THIS CODE</h1>
<h1>E = Total electric field matrix using Poisson’s equation</h1>
<p>#% V = Potential matrix</p>
<p>#% Nx = Number of grid points in X- direction<br>
#% Ny =Number of grid points in Y-Direction</p>
<p><span class="hashtag">#Enter</span> the dimensions</p>
<p>Nx = 101 # Number of X-grids<br>
Ny = 101 # Number of Y-grids<br>
mpx = math.ceil(Nx/2) # Mid-point of x<br>
mpy = math.ceil(Ny/2) <span class="hashtag">#Mid</span> point of y</p>
<p>Ni = 750 <span class="hashtag">#Number</span> of iterations for the Poisson solver<br>
V = np.zeros((Nx,Ny))</p>
<p>T = 0 # Top-wall potential<br>
B = 0 # Bottom-wall potential<br>
L = 0 # Left-wall potential<br>
R = 0 # Right-wall potential</p>
<p><span class="hashtag">#-------------------------------------------------------------------------</span></p>
<h1>Initializing edges potentials</h1>
<p><span class="hashtag">#-------------------------------------------------------------------------</span></p>
<p>V[1,:] = L<br>
R= V[Nx,:]<br>
V[:,1] = B<br>
V[:,Ny] = T</p>
<p><span class="hashtag">#-------------------------------------------------------------------------</span></p>
<h1>Initializing Corner potentials</h1>
<p><span class="hashtag">#-------------------------------------------------------------------------</span></p>
<p>V[1,1] = 0.5*(V[1,2]+V[2,1])<br>
V[Nx,1] = 0.5*(V[Nx-1,1]+V[Nx,2])<br>
V[1,Ny] = 0.5*(V[1,Ny-1]+V[2,Ny])<br>
V[Nx,Ny] = 0.5*(V[Nx,Ny-1]+V[Nx-1,Ny])</p>
<p>length_plate = 51 # Length of plate in terms of number of grids<br>
lp = math.floor(length_plate/2)</p>
<p>position_plate = 15 # Position of plate on x axis<br>
pp1 = mpx+position_plate<br>
pp2 = mpx-position_plate</p>
<p>for z in range(1,Ni): <span class="hashtag">#Number</span> of iterations<br>
for i in range(2,Nx-1):<br>
for j in range(2,Ny-1):</p>
<p><span class="hashtag">#The</span> next two lines are meant to force the matrix to hold the</p>
<h1>potential values for all iterations</h1>
<p>V[pp1,mpy-lp:mpy+lp] = 100<br>
V[pp2,mpy-lp:mpy+lp] = -100</p>
<p>V[i,j]=0.25*(V[i+1,j]+V[i-1,j]+V[i,j+1]+V[i,j-1])</p>
<h1>Take transpose for proper x-y orientation</h1>
<p>V = np.transpose(V)</p>
<p>[Ex,Ey]=np.gradient(V)<br>
Ex = -Ex<br>
Ey = -Ey</p>
<h1>Electric field Magnitude</h1>
<p>E= math.sqrt( Ex<em>Ex + Ey</em>Ey )</p>

username: albanD
cooked: <p>Hi,</p>
<p>You should have a stack trace of where the error happened right? The error means that the index you use is out of bound of the Tensor you try to read from (python is 0 indexed, so for a Tensor of size 101, the valid indices are 0 to 100 <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Is there any API can work for this?

username: entslscheia
cooked: <p>I have a tensor <code>A</code> of shape <code>m*n</code> and a long tensor <code>B</code> of shape <code>m*1</code>, each element in <code>B</code> is in <code>[0, n-1]</code> indicating the last unmasked element in each row of <code>A</code>, and then I can return a mask of <code>m*n</code> for <code>A</code> based on <code>B</code>.</p>
<p>For example, the shape of <code>A</code> is <code>4*3</code>, and <code>B</code> is <code>[[2], [1], [0], [1]]</code>, then I want to return a mask as follows,</p>
<pre><code class="lang-auto">[[1, 1, 1],
[1, 1, 0],
[1, 0, 0],
[1, 1, 0]]
</code></pre>
<p>Is there any similar API provided by PyTorch?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I don’t think there is I’m afraid.<br>
The best way I can think of to do this is:</p>
<pre><code class="lang-python">out = torch.zeros_like(A)
n = A.size(1)
out.scatter_(1, n - 1 - B, 1) # Work in reverse order
out = out.cumsum(dim=1)
out.flip(1)
</code></pre>
Error of Cross-entropy function

username: Cross-entropy
cooked: <p>criterion = nn.CrossEntropyLoss()</p>
<p>print(y)<br>
print(score)</p>
<p>Output:<br>
tensor([0])<br>
tensor([-0.0582, -0.0136, -0.0489,  0.0052, -0.0133, -0.0383,  0.0546, -0.0046,<br>
-0.0712, -0.0451,  0.0127, -0.0537,  0.0170, -0.0693, -0.0291,  0.0012,<br>
-0.0209, -0.0789, -0.0262, -0.0246, -0.0059,  0.0020, -0.0286,  0.0061,<br>
0.0107, -0.0366, -0.0331,  0.0319,  0.0519,  0.0404, -0.0104, -0.0366,<br>
0.0209,  0.0137, -0.0235, -0.0217, -0.0268, -0.0281, -0.0365, -0.0068,<br>
-0.0292, -0.0203,  0.0088,  0.0109, -0.1082, -0.0077, -0.0582,  0.0276,<br>
0.0150, -0.0218], grad_fn=)</p>
<p>But if I run<br>
loss = criterion(score, y) :</p>
<p>IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)</p>
<p>But if I use output “score2” from other model’s output:</p>
<p>tensor([[ 0.1487, -0.0896, -0.4345,  1.4411, -0.0286, -0.8751,  0.8499,  0.3635,<br>
0.1626, -0.6802, -0.5571,  0.1824,  0.3580,  0.1138, -1.4639,  0.4943,<br>
-2.2529, -0.5330, -0.3036, -0.8565, -0.5219, -0.0841, -1.0277,  0.0677,<br>
-0.8914, -0.6413, -0.6800, -0.9659, -0.6177, -0.5056, -0.1912, -0.9347,<br>
-0.6115, -1.3679, -0.7902, -1.4691, -1.1015, -1.0402, -1.6861, -1.2793,<br>
-1.1746, -1.6846, -1.2551, -1.9724, -1.1744, -1.6048, -1.4779, -1.9207,<br>
-1.9850, -2.2973]], grad_fn=)</p>
<p>and put score2 and y in loss function, it works.</p>
<p>Can anybody help? If you need rest of the code, please tell.</p>

username: Oli
cooked: <aside class="quote no-group" data-username="Cross-entropy" data-post="1" data-topic="72502">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cross-entropy/48/21481_2.png" class="avatar"> Cross-entropy:</div>
<blockquote>
<p>[</p>
</blockquote>
</aside>
<p>vs</p>
<aside class="quote no-group" data-username="Cross-entropy" data-post="1" data-topic="72502">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cross-entropy/48/21481_2.png" class="avatar"> Cross-entropy:</div>
<blockquote>
<p>[[</p>
</blockquote>
</aside>
<p>Looks like you need to <code>score = score.unsqueeze(0)</code> if that second example was working for you</p>

username: Cross-entropy
cooked: <p>Thank you very much!</p>
Data Parallel Operation Giving error

username: Rafi_zunaed
cooked: <p>my below code snippet:</p>
<p>model = … <span class="hashtag">#somemodel</span><br>
model = model.cuda()<br>
model, optimizer = amp.initialize(model, optimizer, opt_level=“O2”)<br>
model = nn.DataParallel(model)</p>
<h1>in dataloader batch</h1>
<p>images = images.cuda()<br>
output = model(images)</p>
<p>last line is giving me this error:<br>
Expected tensor for argument <span class="hashtag">#1</span> ‘input’ to have the same device as                                                                                                                                                              tensor for argument <span class="hashtag">#2</span> ‘weight’; but device 1 does not equal 0 (while checking                                                                                                                                                              arguments for cudnn_convolution)</p>
<p>i am running this code on gcp 2xT4 server. What portion of the above mentioned code is wrong or needs to be corrected to use multi gpu training?<br>
[pytorch version 1.1.0]</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think this is a known limitation of amp. You can see this issue on their repo for more info: <a href="https://github.com/NVIDIA/apex/issues/503">https://github.com/NVIDIA/apex/issues/503</a></p>
Gradient w.r.t inputs

username: feribg
cooked: <p>I currently have a model that outputs a single regression target with mse loss. I can get the derivatives with respect to the inputs like so</p>
<pre><code class="lang-auto">x = x.requires_grad_(True).cuda()
output = model.eval()(None, x)
output[0].backward()
x.grad[0]
</code></pre>
<p>However this works one element at a time, if x is a batch, then</p>
<pre><code class="lang-auto">x = x.requires_grad_(True).cuda()
output = model.eval()(None, x)
output.backward()
x.grad[0]
</code></pre>
<p>fails with a runtime error that implicit derivatives are only supported for scalar outputs. I have 2 questions:</p>
<ol>
<li>How to get a batch of target + it’s derivatives  wrt to inputs?</li>
<li>How to get higher order derivates wrt to inputs?</li>
</ol>
<p>Important to note that if I aggregate the loss then that works, but I need the specific input &lt;&gt; loss pair derivatives not the averaged one.</p>

username: feribg
cooked: <p>Here seems to  be an example of that in tensorflow, what  would the Pytorch equiv be ?</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/google/tf-quant-finance/blob/ed17cb35e263dcb4d27339b5c88cae3f2129ca7d/tf_quant_finance/math/gradient.py#L24" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/google/tf-quant-finance/blob/ed17cb35e263dcb4d27339b5c88cae3f2129ca7d/tf_quant_finance/math/gradient.py#L24" target="_blank" rel="nofollow noopener">google/tf-quant-finance/blob/ed17cb35e263dcb4d27339b5c88cae3f2129ca7d/tf_quant_finance/math/gradient.py#L24</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="14" style="counter-reset: li-counter 13 ;">
<li># limitations under the License.</li>
<li>"""Helper functions for computing gradients."""</li>
<li>
</li>
<li>
</li>
<li>import functools</li>
<li>
</li>
<li>import tensorflow.compat.v2 as tf</li>
<li>from tensorflow_probability.python.math import value_and_gradient</li>
<li>
</li>
<li>
</li>
<li class="selected">def fwd_gradient(func, x, input_gradients=None, use_gradient_tape=False):</li>
<li>  """Computes forward mode gradient.</li>
<li>
</li>
<li>  Implementation based on suggestions in</li>
<li>  [this thread](https://github.com/tensorflow/tensorflow/issues/19361).</li>
<li>
</li>
<li>  TensorFlow computes gradients using the reverse mode automatic</li>
<li>  differentiation which is suitable for typical machine learning situations</li>
<li>  where one has a scalar loss function that one wants to differentiate with</li>
<li>  respect to the parameters. In some cases, one needs to be able to compute</li>
<li>  directional derivatives of non-scalar functions. Suppose F is a function from</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">def fwd_gradient(func, x, input_gradients=None, use_gradient_tape=False):
</code></pre>
Can I duplicate parameter tensor whose values are the same across duplicates while training?

username: yjkimnada
cooked: <p>So let’s say I have an input vector of size 100. Specifically, the input vector is size [50 + 50]. Then I want to create a parameter tensor of size [50 x 2] that is applied to the first 50 inputs and second 50 inputs. such that we get a vector of size [2 + 2] afterward. I want the first 50 inputs and last 50 inputs of the input vector to be multiplied by the same [50 x 2] parameter tensor and that this remains true while parameter updating. Can you “duplicate” a parameter tensor like so or is there a way to program this? I’m not sure if the “.clone()” function does what I want…</p>
<p>Thanks!</p>