akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Profiling memory consumption of forward and backward pass

username: adi8862
cooked: <p>I’m trying to profile the memory used by PyTorch for the forward and backward pass of one minibatch for various CNN layers. When doing so I found that if a certain layer configuration doesn’t fit on the GPU and it throws an out of memory error, the GPU memory does not fully get freed and so if I continue executing with new layer configurations eventually the GPU completely runs out of memory.</p>
<p>I was wondering if there was a way to deallocate all the tensors put on by PyTorch for that layer so the only memory remaining is the default CUDA context?</p>

username: ptrblck
cooked: <p>Do you have a reproducible code snippet for this behavior?<br>
If PyTorch encounters an OOM, it should delete the current allocation, clear the cache and retry the allocation.</p>
Chunked loss produces error with DistributedDataParallel

username: davidstap
cooked: <p>I am having a problem with chunked loss calculation when using DistributedDataParallel (the code works fine on a single GPU). I use a single node with 4 GPU’s, and am training a transformer model for NLP. Instead of feeding the whole batch to the final linear layer that maps to the vocabulary dimension (called generator in the code below), I split the batch up in chunks. This is common practice, see e.g. <a href="http://nlp.seas.harvard.edu/2018/04/03/attention.html" rel="nofollow noopener">http://nlp.seas.harvard.edu/2018/04/03/attention.html</a> (class MultiGPULossCompute).</p>
<p>The code I use for loss calculation (where <code>x</code> are the model activations):</p>
<pre><code class="lang-auto">x_copy = x.clone().detach()
x_copy.requires_grad = True

chunk_loss_all = 0.0
for chunk_start in range(0, batch_size, chunk_size):
	# Calculate loss per chunk
	chunk_end = min(chunk_start + chunk_size, batch_size)
	chunk_predictions = generator(x_copy[chunk_start:chunk_end])
        chunk_loss = criterion(chunk_predictions.contiguous().view(-1, chunk_predictions.size(-1)),
        		                              y[chunk_start:chunk_end].contiguous().view(-1))
	chunk_loss_all += chunk_loss

# backward for chunk losses
chunk_loss_all.backward()

# backward through rest of the model
x_gradients = x_copy.grad.view_as(x)
x.backward(gradient=x_gradients)
optimizer.step()
optimizer.zero_grad()
</code></pre>
<p>The error that is produced:</p>
<pre><code class="lang-auto">  File "loss/compute.py", line 75, in chunked_loss
    x.backward(gradient=x_gradients)
  File "/home/dstap1/anaconda3/envs/logos/lib/python3.8/site-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/dstap1/anaconda3/envs/logos/lib/python3.8/site-packages/torch/autograd/__init__.py", line 97, in backward
    Variable._execution_engine.run_backward(
RuntimeError: has_marked_unused_parameters_ INTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch_1579022027550/work/torch/csrc/distributed/c10d/reducer.cpp:290, please report a bug to PyTorch.
</code></pre>
<p>The problem is probably in the multiple <code>.backward()</code> calls. I don’t know how to rewrite my code to solve this problem. Any ideas?</p>
Grad updated after converting tensor to numpy, modifying, converting back?

username: ngravindra
cooked: <p>If I do some operations on a tensor in numpy/pandas, <code>new_stuff = X.detach().cpu().numpy()</code>, then re-assign to <code>Xprime = torch.tensor(new_stuff, requires_grad=True)</code>, will the params update when I feed it into the model, <code>output = model(Xprime)</code>?</p>

username: ptrblck
cooked: <p>All operations inside <code>model</code> will be tracked by Autograd and the parameters will get a valid gradient during the backward pass.<br>
However, the operations preceding the <code>Xprime</code> creation will not be tracked, since you detached the tensor and also used numpy (through pandas) to calculate <code>new_stuff</code>.</p>
Data loading with massive batches?

username: Sam_Lerman
cooked: <p>Hi, I am using a batch size of 640 on the following model:<br>
</p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/rosinality/relation-networks-pytorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/4343568?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/rosinality/relation-networks-pytorch" target="_blank" rel="nofollow noopener">rosinality/relation-networks-pytorch</a></h3>

<p>Relation Networks for CLEVR implemented in PyTorch - rosinality/relation-networks-pytorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>It’s a simple CNN with a relation network. Most of the computation time is spent merely on loading the data. In fact, I checked the GPU usage and most of the time they’re not being used. I have 4 GPUs and they seem to process the data quickly, once it’s loaded. I also have 40 CPUs serving as workers.</p>
<p>However, I am in a hurry and the training time is too long with my current setup (3 weeks!). The bottleneck is the data loading I believe. See the <code>__getitem__</code> and <code>collate</code> functions in the dataset.py of the link above.</p>
<p>Is there any way to speed up data loading with such large image batches? Perhaps a way to leverage my 40 CPUs? Any help would be greatly appreciated; this is particularly urgent.</p>

username: ptrblck
cooked: <p>Generally, multiple workers should speedup your data loading, although the sweet spot depends on your setup.<br>
You could have a look at <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a> for more information about potential bottlenecks.<br>
Also, you might want to use <a href="https://github.com/NVIDIA/DALI">DALI</a> in case it fits your use case.</p>

username: Sam_Lerman
cooked: <p>Not to be a nuisance, but is there anything obvious that stands out to you in this data loader? <a href="https://github.com/rosinality/relation-networks-pytorch/blob/master/dataset.py" rel="nofollow noopener">https://github.com/rosinality/relation-networks-pytorch/blob/master/dataset.py</a></p>
<p>The <code>collate_data</code> function uses a sort, but getting rid of it didn’t fix the intermittent pauses throughout training. If you could take a glimpse and if something stands out to you that I didn’t catch, it would be really appreciated.</p>

username: ptrblck
cooked: <p>The <code>Dataset</code> looks good and apparently the custom <code>collate_fn</code> doesn’t change the slow down.<br>
You could install <code>PIL-SIMD</code> to speed up the image transformations.<br>
<code>PIL-SIMD</code> is a drop-in replacement for <code>PIL</code> and you don’t have to change any calls, so this might be worth a try.</p>
How to get label from rcnn detection label?

username: cjwcommuny
cooked: <p>I’m using <code>torchvision.models.detection.maskrcnn_resnet50_fpn</code> to detect object, and get detection output (<code>type=List[Dict]</code>)，but <code>output[image_id]["labels"]</code> returns <code>int</code>, so how can I get the label in <code>str</code> ?</p>
<p>I found that <code>#class</code> of default pretrained MaskRCNN in COCO_train_2017 is 91, but actually the <code>#class</code> of the dataset is 80, and plus the background is 81, it’s weird.</p>
Running out of memory when GPU training

username: Eric_Montufar
cooked: <p>Hi, I want to train a big dataset with 1M images. I have a RTX2060 with 6Gbs of VRAM.</p>
<p>I am using a pretrained Alexnet with some extra layers and once I upload my model to my GPU It uses approximately 1Gb from it leaving 4.4 Gbs free.</p>
<p>Then I try to train my images but my model crashes at the first batch when updating the weights of the network due to lack of memory in my GPU. I tried even with batch_size = 1.</p>
<p>I am using Adam optimizer set to update only the parameters that requires_grad = True which are just the last two layers. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c7bb30a6b914c09fa515a247c993debabbfa71aa.png" data-download-href="https://discuss.pytorch.org/uploads/default/c7bb30a6b914c09fa515a247c993debabbfa71aa" title="Screenshot from 2020-05-18 17-19-13"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c7bb30a6b914c09fa515a247c993debabbfa71aa_2_690x354.png" alt="Screenshot from 2020-05-18 17-19-13" data-base62-sha1="suU4iVfDxq7ZXcm56TFcT1X5026" width="690" height="354" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c7bb30a6b914c09fa515a247c993debabbfa71aa_2_690x354.png, https://discuss.pytorch.org/uploads/default/original/3X/c/7/c7bb30a6b914c09fa515a247c993debabbfa71aa.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/7/c7bb30a6b914c09fa515a247c993debabbfa71aa.png 2x" data-dominant-color="232525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-05-18 17-19-13</span><span class="informations">872×448 50 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Am I doing something wrong or can I do something to fix this issue?</p>
<p>Am I doing something wrong or simply my hardware isn’t enough for such a problem?</p>
<p>How can I solve this issue?</p>

username: ptrblck
cooked: <p>Your GPU might not have enough memory for the current training.<br>
You could try to use another model, which is not so memory hungry or alternatively you could also try to use <a href="https://pytorch.org/docs/stable/checkpoint.html"><code>torch.utils.checkpoint</code></a> to trade compute for memory.</p>
Optimizing pytorch model for single board computers (Jetson nano)

username: Sundar_Krishna
cooked: <p>Hi ,<br>
I dont know whether my question is out of scope to this forum so pls bear with me. I am trying to load a resnet34 model which was trained and saved on my laptop (with pytorch) . I copied the the .pkl (pickle) file and .pth (pytorch model) files from my laptop to jetson nano. Now I ran my code which would implement these models . But the code ran very slow . I think the model loading would have taken so much time. It took more than a minute to detect and give me the output. My question is can i use tensorrt to optimize my model? if so how as I already have my .pth file ready , so i think i dont have to retrain my model in jetson nano as that would be tedious , please help!!!</p>
<p>NOTE: what exactly I did was in my laptop collected various environmental sound dataset and converted them into mel-spectogram using librosa and trained those spectograms in my laptop using pytorch after which i got my .pkl and .pth files</p>

username: ptrblck
cooked: <p>You could try to convert your PyTorch model to ONNX and then TensorRT as described in <a href="https://devblogs.nvidia.com/how-to-deploy-real-time-text-to-speech-applications-on-gpus-using-tensorrt/">this blog post</a> for WaveGlow.<br>
Also, you could have a look at <a href="https://github.com/NVIDIA-AI-IOT/torch2trt">this repository</a>, which seems to be a bit older by now, so I’m not sure which operations will work and which won’t.</p>

username: Sundar_Krishna
cooked: <p>Thank you so much sir <a class="mention" href="/u/ptrblck">@ptrblck</a> I already have my “.pth” file trained in my laptop I just used the same file in jetson nano which was found to be very slow comparatively. Is there any way to optimize only the “.pth” file?</p>

username: ptrblck
cooked: <p>Assuming you are storing the <code>state_dict</code> in the file, I think you would have to recreate the model and follow the export pipeline.</p>

username: Sundar_Krishna
cooked: <p>Thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Softmax calculation is slightly off

username: pytorcher
cooked: <p>I am using cross entropy loss on two computers and one of them is returning different numbers.  I believe I have tracked it down to one of the computers returning slightly wrong values for softmax.  Any reason why this would be?  The computer which is wrong, with numbers displayed here, is running pytorch 1.3.1.<br>
(Pdb) (X[0].exp() / X[0].exp().sum()).tolist()<br>
[0.09599629789590836, 0.06680507212877274, 0.09601865708827972, 0.06676748394966125, 0.12830372154712677, 0.06672990322113037, 0.13807427883148193, 0.06669234484434128, 0.16462643444538116, 0.1099857762455940<br>
2]<br>
(Pdb) F.softmax(X[0]).tolist()<br>
[0.09599630534648895, 0.06680507957935333, 0.09601867198944092, 0.06676748394966125, 0.12830372154712677, 0.06672991067171097, 0.13807427883148193, 0.06669234484434128, 0.16462644934654236, 0.1099857836961746<br>
2]<br>
(Pdb) X[0].softmax(0).tolist()<br>
[0.09599630534648895, 0.06680507957935333, 0.09601867198944092, 0.06676748394966125, 0.12830372154712677, 0.06672991067171097, 0.13807427883148193, 0.06669234484434128, 0.16462644934654236, 0.1099857836961746<br>
2]</p>

username: KFrank
cooked: <p>Hi py!</p>
<aside class="quote no-group quote-modified" data-username="pytorcher" data-post="1" data-topic="82083" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/7ea924/48.png" class="avatar"> pytorcher:</div>
<blockquote>
<p>I believe I have tracked it down to one of the computers returning slightly wrong values for softmax.  Any reason why this would be?<br>
(Pdb) (X[0].exp() / X[0].exp().sum()).tolist()<br>
[0.09599629789590836, …<br>
…<br>
(Pdb) F.softmax(X[0]).tolist()<br>
[0.09599630534648895, …</p>
</blockquote>
</aside>
<p>This is floating-point round-off error.  First cast <code>X</code> to a<br>
<code>dtype = torch.double</code> tensor, and then redo your test.<br>
Your “right” numbers and “wrong” numbers will now agree<br>
to about twice as many decimal places.</p>
<p>Best.</p>
<p>K. Frank</p>
Is it possible to change Optimizer from Adam to SGD for continue training

username: Wei_Wong
cooked: <p>Save state_dict of model and Adam, and pause training. Load state_dict of model and change optimizer to SGD.</p>

username: Nathan_Wood
cooked: <p>SGD does not keep track of extra variables relating to weights (unless you’re using momentum). This means you can simply create a new SGD optimizer.</p>
<pre><code class="lang-python">torch.save({'model': model.state_dict(), 'optim': optim.state_dict()}, '...')
</code></pre>
<p>To switch to SGD, use:</p>
<pre><code class="lang-python">state_dict = torch.load('...')
model.load_state_dict(state_dict['model'])
optim = torch.optim.SGD(model.paramters(), new_learning_rate)
</code></pre>

username: Wei_Wong
cooked: <p>Thank you!! <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_three_hearts.png?v=9" title=":smiling_face_with_three_hearts:" class="emoji" alt=":smiling_face_with_three_hearts:"></p>

username: Tuna
cooked: <p>Your idea is using Adam to fast init training and turn to SGD at the end. I admit that is good idea and there is a paper called “<a href="https://arxiv.org/pdf/1902.09843.pdf" rel="noopener nofollow ugc"><em>Adaptive Gradient Methods with Dynamic Bound of Learning Rate. In Proc. of ICLR 2019.</em></a>” has same idea.</p>
<blockquote>
<p>As described in the paper, AdaBound is an optimizer that behaves like Adam at the beginning of training, and gradually transforms to SGD at the end</p>
</blockquote>
<p>Please refer to this code: <a href="https://github.com/Luolc/AdaBound" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Luolc/AdaBound: An optimizer that trains as fast as Adam and as good as SGD.</a></p>
Expected device cuda:0 but got device cpu though the two tensors are on cuda

username: jpainam
cooked: <p>Hi, I thought i’m doing think right, but got this error</p>
<pre><code class="lang-python">features = torch.from_numpy(features).cuda()
print(features.device) #  shows cuda:0
dists = self.calculate_distance(features).cuda()
print(dists.device) # shows cuda:0
dists.add_(torch.tril(100 * torch.ones(len(features), len(features))))
</code></pre>
<p>But the last line fails with error <code>RuntimeError: expected device cuda:0 and dtype Float but got device cpu and dtype Float</code></p>
<p>Anything I’m doing wrong?</p>
<p>Thank you</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/jpainam">@jpainam</a>,</p>
<p>You have to move the <code>torch.ones</code> Tensor to the GPU:</p>
<pre><code class="lang-auto">dists.add_(torch.tril(100 * torch.ones(len(features), len(features)).cuda()))
</code></pre>

username: jpainam
cooked: <p>Thank you, It  works,<br>
but, is there a way to just move everything to cuda if my class  doesn’t extend <code>nn.Module</code>? Doing it for each tensor can be a little bit too much.</p>

username: spanev
cooked: <p>You should be able by doing:</p>
<pre><code class="lang-auto">torch.set_default_tensor_type('torch.cuda.FloatTensor')
</code></pre>

username: vijetadeshpande
cooked: <p>I am doing</p>
<ol>
<li>torch.set_default_tensor_type(‘torch.cuda.FloatTensor’)</li>
<li>source, target = source.cuda(), target.cuda()</li>
</ol>
<p>But still getting the following error on google colab</p>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
7     start_time = time.time()<br>
8     # train<br>
----&gt; 9     train_loss = train(model, data_train, optimizer, criterion, CLIP, DEVICE)<br>
10     # stop clock<br>
11     end_time = time.time()</p>
<p>3 frames<br>
/content/train.py in train(model, data, optimizer, criterion, clip, cohort_size)<br>
46<br>
47         # calculate loss<br>
—&gt; 48         loss = criterion(output, trg)<br>
49<br>
50         # backward propogation</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in <strong>call</strong>(self, *input, **kwargs)<br>
548             result = self._slow_forward(*input, **kwargs)<br>
549         else:<br>
–&gt; 550             result = self.forward(*input, **kwargs)<br>
551         for hook in self._forward_hooks.values():<br>
552             hook_result = hook(self, input, result)</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)<br>
430<br>
431     def forward(self, input, target):<br>
–&gt; 432         return F.mse_loss(input, target, reduction=self.reduction)<br>
433<br>
434</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in mse_loss(input, target, size_average, reduce, reduction)<br>
2541     else:<br>
2542         expanded_input, expanded_target = torch.broadcast_tensors(input, target)<br>
-&gt; 2543         ret = torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))<br>
2544     return ret<br>
2545</p>
<p>RuntimeError: expected device cuda:0 but got device cpu</p>
Ask for advice in pytorch extension

username: JTiC
cooked: <p>Hello, I am new at pytorch extension.  I want some advice in pytorch extension development. Assume I propose a new convolution kernel (for example, assume I am the first one who propose the deformable conv <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"> <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:">), and I have to implement it in cuda and C++.</p>
<p>As the deformable conv is complex,</p>
<p>(1) how can I know if my implement is right?<br>
(2) how can I know if the gradient (in backward) is right?</p>
<p>Any helpful experience is ok.</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/jtic">@JTiC</a><br>
I suggest you to take a look at this instruction first.<br>
<a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_extension.html</a><br>
And then take a look at how to install the libtorch here:<br>
<a href="https://pytorch.org/cppdocs/installing.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/cppdocs/installing.html</a></p>
<p>We can not automatically test whether your code’s logic is correct or not since that’s your own implementation. But you can write test case to run your code to check whether the output result is what you want.</p>

username: JTiC
cooked: <p>Thank you, it is helpful! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Torch not compiled with CUDA enabled---CUDA

username: jasg
cooked: <p>Friends i have an imac with a built-in GPU card Radeon Pro Vega 56, but I get this error:<br>
AssertionError: Torch not compiled with CUDA enabled.<br>
I have installed pytorch = 1.5.0, cudatoolkit = 9.0 and torchvision = 0.4, please support me</p>

username: miker256
cooked: <p>Hi <a class="mention" href="/u/jasg">@jasg</a>, did you <a href="https://pytorch.org/get-started/locally/#mac-from-source" rel="nofollow noopener">Build from Source</a>? You probably need to build from the source since PyTorch is installed without CUDA support by default on MacOS. This is due to CUDA being a Nvidia proprietary technology.</p>
Remove all occurences of an item from LongTensor

username: alabijesujoba
cooked: <p>Is there an efficient way of removing occurences of items in a LongTensor<br>
For example</p>
<pre><code class="lang-auto">#input
x = torch.LongTensor([0,2,3,1,4,2,7,1,8])

#output
x = torch.LongTensor([0,3,4,7,8])
</code></pre>
<p>In the example above I want to remove 2 and 1 from the tensor. Is there any method in pytorch for this? or any efficient way?</p>

username: vainaijr
cooked: <p>maybe,</p>
<pre><code class="lang-auto">x[x.ge(3) + x.le(0)]
</code></pre>
<p>or</p>
<pre><code class="lang-auto">x[~((x!=1) ^ (x!=2))]
</code></pre>

username: alabijesujoba
cooked: <p>Hi and thanks <a class="mention" href="/u/vainaijr">@vainaijr</a>. The two methods works well. Do you have an idea of how I can get the values between <code>2</code> and <code>1</code> as a longtensor. For example in this case, the output would be  <code> x = [torch.LongTensor([3]), torch.LongTensor([7])]</code>? <code>2</code> and <code>1</code> can be seen as a kind of boundary marker.</p>
Can i vary the size of nn.TransformerEncoder outputs if not, then how do i calculate loss?

username: Rohan_Kumar
cooked: <p>I have followed this <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">guide</a> on transformers, and found out that the input/ouput size of transformerEncoder layer is -</p>
<pre><code class="lang-auto">Input -&gt; [source sequence length, batch size, embedding dimension ]
Output -&gt; [source sequence length, batch size, embedding dimension]
</code></pre>
<p>My question is that i have target sequence of length different than source sequence,</p>
<blockquote>
<p>target → [target sequence length ,batch size, embedding dimension]</p>
</blockquote>
<p>so can i somehow change the code, so that my output matches the dimension of transformer, or how do i calculate the loss, given the dimensions dont match?</p>
Access layers in pretrained Resnet50

username: phantom90
cooked: <p>Hi, I am using the pretrained Resnet50 wrapped by pretrainedmodels (<a href="https://pypi.org/project/pretrainedmodels/" rel="nofollow noopener">https://pypi.org/project/pretrainedmodels/</a>)。</p>
<p>Now I want to access the intermidate result of each layer, I try the following code:</p>
<pre><code class="lang-auto">for module_pos, module in self.model._modules.items():
    if module is not None:
        x = module(x)  # Forward
</code></pre>
<p>However, there are two issues.</p>
<p>First, some of the layers in <em>self.model._modules.items()</em> is None (that is why I use  <em>if module is not None:</em></p>
<p>Second, when I run the code, there is an error at the very last layer.</p>
<p><em>RuntimeError: size mismatch, m1: [2048 x 1], m2: [2048 x 1000] at /opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/TH/generic/THTensorMath.cpp:197</em></p>
<p>How comes the pretrained model encountered with this problem? I know reshape m1 to [2048] may solve the problem. But why does this happen?</p>
<p>Thank you very much!</p>

username: albanD
cooked: <p>Hi,</p>
<p>Such chaining of each layer one by one is only expected to work if the original module is a <code>nn.Sequential()</code> as this is how the <code>forward</code> method is implemented <a href="https://github.com/pytorch/pytorch/blob/858b2010aeaa63d2680ac8fd09f59a69ac899026/torch/nn/modules/container.py#L90-L93" rel="nofollow noopener">here</a>.<br>
For general modules, you should check how the <code>forward</code> method is implemented to know how they should be used.<br>
In the case of resnet, the repo you linked seem to use the torchvision version, whose forward function can be found <a href="https://github.com/pytorch/vision/blob/95131de394543a7c34bd51932bdfce21dae516c1/torchvision/models/resnet.py#L197-L212" rel="nofollow noopener">here</a>. It looks like by change, because the modules were defined in the right order, it works up until the <code>.reshape()</code>. But this is only by chance and is not guaranteed.<br>
I think the best way to access internal values here would be to change the <code>forward</code> method to either return or print the intermediate values you’re interested in.</p>

username: phantom90
cooked: <p>Thanks very much! Really appreciate.</p>

username: Sam_Lerman
cooked: <p>I copied the forward code from the link you provided exactly and still got a size mismatch error at that point…</p>

username: Sam_Lerman
cooked: <p>Oh it seems that <code>torch.flatten(x, 1)</code> was just not expecting a batch size of 1 input, so changing it to <code>torch.flatten(x)</code> fixed it. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
LowRankMultivariateNormal example raises ValueError

username: smonsays
cooked: <p>I am trying to make use of LowRankMultivariateNormal but I have trouble to correctly specify <code>cov_factor</code> and <code>cov_diag</code>. Unfortunately, the example in the documentation also raises a <code>ValueError</code> for me (pytorch version 1.3.1):</p>
<pre><code class="lang-auto">m = LowRankMultivariateNormal(torch.zeros(2), torch.tensor([1, 0]), torch.tensor([1, 1]))
# ValueError: cov_factor must be at least two-dimensional, with optional leading batch dimensions
</code></pre>
<p>I haven’t been able to figure out the right amount of (un-) squeezing to make it work. Can someone help?</p>

username: ptrblck
cooked: <p>Thanks for reporting this issue!<br>
The example doesn’t seem to work indeed.<br>
Could you use <a href="https://github.com/pytorch/pytorch/blob/93acd1998f342fffcbf7f861a3bb62265519102b/test/test_distributions.py#L285-L295">these test use cases</a> as an example code, while this issue is being fixed?</p>
<p>CC <a class="mention" href="/u/fehiepsi">@fehiepsi</a></p>

username: fehiepsi
cooked: <p>Oops… the cov_factor in the example is missing a trailing dimension (which has size “rank”). Thanks for pointing it out! I’ll fix it soon. In the meantime, you can simply unsqueeze cov_factor to say that the distribution has rank 1. In general, cov_factor will have shape event_shape + rank</p>

username: smonsays
cooked: <p>Ah great, thank you! This works for me (I had to change the integers for the tensor creation of <code>cov_factor</code> to floats as otherwise one gets a <code>RuntimeError: "cholesky_cpu" not implemented for 'Long'</code>)</p>
<pre><code class="lang-auto">LowRankMultivariateNormal(torch.zeros(2), torch.tensor([1.0, 0.0]).unsqueeze(-1), torch.tensor([1, 1]))
</code></pre>

username: He_Zhao
cooked: <p>Dear smonays</p>
<p>May I ask: if we have a low-rank cov-matrix at hand, what is the correct way to find out cov_factor and cov_diag, in order to use this LowRankMultivariateNormal() function ??</p>
<p>The example here is a very simplified case I guess.</p>
<p>Thanks</p>
LBFGS dataloader question

username: ericrhenry
cooked: <p>This might work in a general forum, but I am focusing on the C++ frontend, so:<br>
From all I can tell, there is no mini-batch version of the LBFGS optimizer at the moment. (Please correct if not true.) My question is pretty simple: When creating a dataloader to use with this optimizer, do I simply create the same kind of Batch-type dataloader I would use with a stochastic optimizer, and force the batch size to equal the total training-set size? (I’m guessing I would have to Stack-map that as well.) Or is there an alternative approach to creating a dataloader for use with this optimizer?<br>
Sorry if this is somehow trivial…</p>
<p>Thanks,<br>
Eric</p>
Accelerate Spectrograms with GPU and PyTorch?

username: Mason7Acree
cooked: <p>I am working with data that I spend a lot of time converting 1D signals to 2D spectrograms that are then fed to a CNN in Pytorch. I have a 2nd GPU and wanted to know if it was reasonable to accelerate my fft/spectrogram workflow through PyTorch on my GPU instead of what I am currently doing in my dataloader?</p>
<p>What I am doing now:<br>
-In my custom dataset (using torch.utils.data dataset class) I initialize and build my dataset of time-series audio data before training begins (fast). Then inside my <strong>getitem</strong> method I have some things that get done to the time-series data before I pass the data sample to a custom fft function that builds the spectrograms in the way I need and then returns that spectrogram as the data sample.<br>
-My spectrogram function is custom built for various things I need but the main workhorse is built around the np.fft.fft() function, which is where I need to accelerate.<br>
-I am using the “num_workers” kwarg in the PyTorch dataloader to better use my cores.</p>
<p>What I would like to do:<br>
Either in my <strong>getitem</strong> or elsewhere, I’d like to send it to my 2nd GPU (GTX 980ti) in hopes to faster generate the spectrograms then send them to my main GPU to pass the data through the model.</p>
<p>Is this reasonable? Can I expect to see accelerations in contrast to a Ryzen 3950x and utilizing the “num_workers” kwarg in the dataloader? How would I do this? I’m open to ideas for other libraries and tools that exist out of PyTorch, but not sure where to start and seeking insight from this community.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think the best way to speed this up would be to move it as preprocessing.<br>
Have a seperate script that converts your audio data to the spectrogram and save them to disk.<br>
Then your dataloader in the training script will just load the spectrograms directly.</p>

username: Mason7Acree
cooked: <p>Inconveniently this will not be accessible because I have certain transformations that are applied to my time-series data before I generate a spectrogram. Also this would eliminate any data augmentation that I have available in the time-series data as well.</p>

username: JuanFMontesinos
cooked: <p>Hi,<br>
You can indeed operate on the main thread and to process that in a second gpu.<br>
Even if I didn’t a proper profiling from my experience it may be worse to pre process them as the dimensionality is usually way higher. For example in my case an wavelength of 16k elements becomes into a 512x256x2.<br>
Anyway I only recommend this is the main workload is the stft.if you have additional heavy preprocessing multiprocessing may be better</p>
Autograd.function with parameter

username: satubus
cooked: <p>Hello there,</p>
<p>I am trying to implement a custom activation function (similar to relu) with defined forward/backward static methods.<br>
Now when I call Function.apply  the following code throws errow self.param1 not found.</p>
<p>pseudo-code:</p>
<p>class F1(Function):</p>
<pre><code> def __init__(self, PARAM):
       self.param

 @staticmethod
def forward(ctx, input):
    output = output * (output &gt; self.PARAM1)
    return output

@staticmethod
def backward(ctx, grad_output):
    grad_input[input&lt;= self.PARAM1] = 0
    return grad_input
</code></pre>
<p>I would like to pass a treshold parameter upon which input values are filtered…<br>
any ideas? thanks in advance</p>

username: satubus
cooked: <p>Found the <a href="https://discuss.pytorch.org/t/how-to-save-a-list-of-integers-for-backward-when-using-cpp-custom-layer/25483">solution</a> in forum…</p>
Is Python a must for building Libtorch from source?

username: WMF1997
cooked: <p>Hello,<br>
I manage to build pytorch(libtorch) from source on my Raspian (armv7l).<br>
After finding CMakeLists.txt, I just use cmake to build it. (I knew the official instructions of building it, I just want to have another try)</p>
<p>However, I find that if directly use cmake to get the configure. I have the following error (I only build libtorch):</p>
<pre><code class="lang-auto">Failed to get generated_cpp list. 
</code></pre>
<p>I checked the cmake file, and I found the reason. in cmake/codegen.cmake:166-182 . The code generation process indeed needs python.</p>
<p>And I found that using  <code>python3 tools/build_libtorch.py</code> to get a libtorch build. Then rebuild libtorch or pytorch in another directory, cmake+make actually works. (because of the <strong>cached</strong> generated_cpp.txt ?)</p>
<p>So I want to ask that… Are there some other methods (except for the method mentioned above) to get generated_cpp.txt? Or, are there any other methods to build libtorch? (For example, a device with no python installed at all, but cmake and make are installed)</p>
<p>yours sincerely,<br>
<a class="mention" href="/u/wmf1997">@wmf1997</a></p>

username: albanD
cooked: <p>Hi,</p>
<p>I never asked that question but given that all of the codegen we do is implemented in python, I would say that python is required to build libtorch yes (even though it is not required to use it!).</p>
Flatten upper triangular matrix

username: blade
cooked: <p>Given an <code>NxN</code> matrix <code>Mat</code>, I would like to have a vector of upper diagonal elements <code>Vec</code> of the matrix. Are there anyways, without using a nested loop, to do this? With a nested loop</p>
<pre><code class="lang-auto">Vec = None
for i in range(N):
    for j in range(N):
        if i&gt;j:
            if Vec is None:
                Vec = adjacency[:, i, j].unsqueeze(1)
            else:
                Vec = torch.cat((Vec, adjacency[:, i, j].unsqueeze(1)), dim=1)
</code></pre>
<p>where Vec is an <code>N(N-1)/2</code>-dimensional vector.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You should be able to use <code>triu_indices()</code> <a href="https://pytorch.org/docs/stable/torch.html#torch.triu_indices">here</a> to get the indices and then just index adjacency with it using a single function call.</p>
Difference between error rate and accuracy?

username: Janashree_Ananthan
cooked: <p>Hai, I am working on image classification task. I have referred many papers, in some papers they have proposed to achieve accuracy of 83.4% but they have no mention of error rate they achieved, while in some they propose they have achieved error rate of 6.54% but they have no mention of accuracy obtained.<br>
My question is whether 1-error rate = accuracy?<br>
Please do help<br>
Thank you</p>

username: Isaac_Kargar
cooked: <p>I think it should be the same.</p>

username: Janashree_Ananthan
cooked: <p>Thanks for the reply, I have another doubt ,if both are same, why they are computed differently???</p>

username: Isaac_Kargar
cooked: <p>I think the better way is to see what is exactly the definition for both. But in general, I think they are the same. You can check <a href="https://stats.stackexchange.com/questions/133458/is-accuracy-1-test-error-rate" rel="nofollow noopener">this</a> and <a href="https://stackoverflow.com/questions/52865390/definition-of-error-rate-in-classification-and-why-some-researchers-use-error-ra" rel="nofollow noopener">this</a>.</p>

username: Janashree_Ananthan
cooked: <p>Thanks, the links explain what i needed.</p>
Pytorch -Visualising test error

username: alameer
cooked: <p>I am running code that trains a model which print a test count error and count error. I would like to visualise this data. What is the best way to do this please? I am quite new to Pytorch and help would be greatly appreciated.</p>

username: dachosen1
cooked: <p>Have you looked into <a href="https://pytorch.org/docs/stable/tensorboard.html" rel="nofollow noopener">Tensorboard</a>?</p>

username: alameer
cooked: <p>Thanks! I have had a look but not sure which bit of the code i need to integrate to my code?</p>

username: dachosen1
cooked: <p>Cnange the randoms to values to your loss values.</p>
<pre><code class="lang-auto">from torch.utils.tensorboard import SummaryWriter
import numpy as np

writer = SummaryWriter()

for n_iter in range(100):
    writer.add_scalar('Loss/train', np.random.random(), n_iter)
    writer.add_scalar('Loss/test', np.random.random(), n_iter)
    writer.add_scalar('Accuracy/train', np.random.random(), n_iter)
    writer.add_scalar('Accuracy/test', np.random.random(), n_iter)
</code></pre>

username: alameer
cooked: <p>Thanks for your help! Do i include this in the same file as my training model? And once the training is done i issue the command line tensorboard --logdir=runs to get the dashboard? Thanks</p>

username: dachosen1
cooked: <p>Yep, that’s correct. Check out the documentation for a full list of what you can do.</p>
What is the role of bottleneck layer in UNet architecture?

username: aditya_raj
cooked: <p>Viewed 3 times</p>
<p>0</p>
<p>The vary famous UNet model has three parts: 1 Encoder, 2 Decoder, and 3 Bottleneck.<br>
.<br>
As far as I understand …<br>
Encoder does the contextual feature extraction.<br>
.<br>
Decoder does the localization of features</p>
<p>but I do not understand the role of bottleneck… I want to know that is there any specific role of bottleneck layer in the UNet model ?? Anyone please make me understand. Please correct me for encoder and decoder as well if I am wrong…</p>
<p>… … Thank you everyone</p>

username: ptrblck
cooked: <p>Bottlenecks in Neural Networks are a way to force the model to learn a compression of the input data.<br>
The idea is that this compressed view should only contain the “useful” information to be able to reconstruct the input (or segmentation map).</p>

username: aditya_raj
cooked: <p>Thanks Sir <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"><img src="https://discuss.pytorch.org/images/emoji/apple/v.png?v=9" title=":v:" class="emoji" alt=":v:"> Your answer will help me to understand this more clearly</p>