akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Unable to load torch model trained on CUDA with a CPU only machine

username: 111532
cooked: <p>I’ve a similar issue to this: <a href="https://discuss.pytorch.org/t/import-torch-file-with-trained-model-in-cuda-in-a-cpu-machine/97591" class="inline-onebox">Import torch file with trained model in CUDA, in a CPU machine</a></p>
<p>I’d trained a model on Colab with CUDA service, and saved as model.pickle file.</p>
<p>When I want to load the model on my local computer via:</p>
<pre><code class="lang-auto">with open("D:/models/resnet34_new.pickle", 'rb') as file:
        model = torch.load(file, map_location=torch.device('cpu'))
</code></pre>
<p>It shows the error:</p>
<pre><code class="lang-auto">RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
</code></pre>
<p>I’d searched multiple solutions online, but none works for me.</p>
<p>What could I do the load the model?<br>
Thank you for any help!</p>

username: pchandrasekaran
cooked: <p>Have you tried saving the model as a <em>.pt</em> file? I’ve used that method and it works.</p>
<pre><code class="lang-auto"># Saving
path = "./your_name.pt"
torch.save(model.state_dict(), path)

# Loading
model.load_state_dict(torch.load(path, map_location=torch.device("cpu")))

</code></pre>

username: 111532
cooked: <p>I would try it, thanks!</p>
Converting to onnx with dynamic batch has trouble

username: fwy134679
cooked: <p>i  converted this model to onnx with dynamic batch as below ,and it succeeded.<br>
The model come from <a href="https://github.com/facebookresearch/detr.git" rel="noopener nofollow ugc">https://github.com/facebookresearch/detr.git</a><br>
and my converted model is in <a href="https://drive.google.com/file/d/12v98uC0jxfktdRc3aJnpuwhT0k2Yj7P7/view?usp=sharing" rel="noopener nofollow ugc">https://drive.google.com/file/d/12v98uC0jxfktdRc3aJnpuwhT0k2Yj7P7/view?usp=sharing</a></p>
<p>my script as below:</p>
<pre><code class="lang-auto">output_onnx = 'det.onnx'
print("==&gt; Exporting model to ONNX format at '{}'".format(output_onnx))
input_names = ["input"]
output_names = ["scores", "boxes"]
dynamic_dic = {'input': {0: 'batch_size'}, 'scores': {0: 'batch_size'}, 'boxes': {0: 'batch_size'}}

inputs = torch.randn(1, 3, 800, 800).to('cpu')
torch_out = torch.onnx.export(net, inputs, output_onnx, export_params=True, verbose=False,
                               input_names=input_names, output_names=output_names, opset_version=11, dynamic_axes=dynamic_dic)

onnx_model = onnx.load('det.onnx')
print(onnx_model.graph.input[0])


However, when i run it in triton server using onnxruntime_onnx, i got an error that told me outputs of the model have fixed dimension but dynamic .    And i don`t know what is wrong with it?</code></pre>
Code reproducibility strange behaviour

username: Stainley
cooked: <p>Hey, I have a strange code reproducibility issue:</p>
<pre><code class="lang-python">
torch.manual_seed(0)
torch.cuda.manual_seed(0)
np.random.seed(0)
random.seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
</code></pre>
<p>All these options are set at the beginning of the training function, num of workers for dataloader is set to zero. I use only one GPU for training. Dataloader loads the data once and then only utilizes it.</p>
<p>At the first launch of training function I have RESULT 1, but at the second and all next launches I got RESULT 2.</p>
<p>If I increase the number of training epochs, at the first launch I have RESULT 2 + some epochs passed, at all next launches I got RESULT 3.</p>
<p>Any ideas what can cause this issue?</p>

username: ptrblck
cooked: <p>You might get non-deterministic results, if you don’t set all required flags mentioned in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">reproducibility docs</a>.<br>
Could you take a look at these docs and add e.g. <code>torch.use_deterministic_algorithms(True)</code> to the script?</p>
Class init func with self.hparams = hparams raising an "AttributeError: can't set attribute"

username: Binge_Kenny
cooked: <p>In my LightningModule subclass, I used ‘self.hparams = hparams’ which worked fine a week ago but now it unexpectedly returns an Exception saying “AttributeError: can’t set attribute”.</p>
<p>I believe something has been changed recently in the father class <strong>LightningModule</strong> as this issue should be solved by add a “<span class="mention">@ATTRI.setter</span>” but here I cannot access the father class.</p>
<pre><code class="lang-auto">class T5FineTuner(pl.LightningModule):
  def __init__(self, hparams):
    super(T5FineTuner, self).__init__()
    self.hparams = hparams
</code></pre>
<p>Any way to solve this given a recent update on source code, please?</p>

username: ptrblck
cooked: <p>Is this issue already solved <a href="https://discuss.pytorch.org/t/pytorch-lightning-module-cant-set-attribute-error/121125/5">here</a>?</p>

username: Ajinkya_Ambatwar
cooked: <p>Yes <a class="mention" href="/u/ptrblck">@ptrblck</a>, it is solved.</p>
How to test quantization performance

username: 0Chen
cooked: <p>I have quantization a model from 32-bit float to int8. I want to test the quantization performance, such as latency. Is there any way to inference the model with 8bit fix point?</p>

username: Vasiliy_Kuznetsov
cooked: <p>Hi <a class="mention" href="/u/0chen">@0Chen</a> , have you tried the autograd profiler (example: <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Profiler — PyTorch Tutorials 1.8.1+cu102 documentation</a>)?</p>

username: 0Chen
cooked: <p>Thanks！<br>
That can test time and memory usage. However, I want to test in low-bit compute, such as 8bit or 6bit. Could pytorch just allocate a tensor with 8bit.</p>

username: Rushirajsinh_Parmar
cooked: <p><a class="mention" href="/u/0chen">@0Chen</a> I am facing the same issue too! Does anyone know how to perform inference after quantizing the model?? When I try to get the output, I get this error</p>
<pre><code class="lang-auto">Error(s) in loading state_dict for Module:
	 ('Copying from quantized Tensor to non-quantized Tensor is not allowed, please use dequantize to get a float Tensor from a quantized Tensor',)
</code></pre>

username: HDCharles
cooked: <p>If you have a quantized model, then it is doing low-bit computation depending on what format it was quantized to. Otherwise, I’m not sure I understand what you are trying to do if your quantized model isn’t already doing what you want.</p>

username: HDCharles
cooked: <p>I don’t think this is related to OP’s issue. But it looks like you are trying to load a saved quantized model into a non-quantized model. the state_dict to my knowledge is just a record of the parameters for different modules. Since quantized modules have different parameters than non-quantized ones, you won’t be able to load one into another. I believe that when you create the model before loading the state dict, you have to fuse/prepare/convert it, at which point you should be able to load the state dict per <a href="https://discuss.pytorch.org/t/how-do-i-save-and-load-quantization-model/64188" class="inline-onebox">How do I save and load quantization model</a></p>
<p>If this is not helpful, can you make a separate post within the <a class="hashtag" href="/c/quantization/17">#<span>quantization</span></a> category so as not to hijack this thread? I am the oncall for quantization so I am responding to any new posts.</p>

username: Rushirajsinh_Parmar
cooked: <p><a class="mention" href="/u/hdcharles">@HDCharles</a> Thanks! I have created the post here - <a href="https://discuss.pytorch.org/t/how-to-load-a-quantised-model-in-pytorch-or-pytorch-lightning/121702">Load Quantised model</a></p>
Question about .to(device)

username: Tejan_Mehndiratta
cooked: <p>Hi,</p>
<p>If I move a tensor to Cuda using <code>.to(device)</code>, does it mean that then all the arithmetic operations done with that tensor will happen on Cuda?</p>
<p>Thanks.</p>

username: jmandivarapu1
cooked: <p>My understanding is that any operation you do on the device will happen on GPU only, you can also use all the torch methods on that tensor and all the operations will also happen on GPU</p>

username: Tejan_Mehndiratta
cooked: <p>Yes, It’s True. If you transfer a tensor on Cuda and then do any operation on that tensor, the operation happens on Cuda.</p>
<p>Thanks.</p>
What's the difference between get_lr and get_last_lr?

username: UCDuan
cooked: <p>Hi all,<br>
I’m new to <code>lr_scheduler</code> and I get different results from <code>get_lr</code> and <code>get_last_lr</code>. What’s the true learning rate? And why do they generate different results?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>I think you should rely on calling <code>get_last_lr</code>, since using <code>get_lr</code> outside of the <a href="https://github.com/pytorch/pytorch/blob/0362b753dba22c1ccab2f9f7ee9f6de2ffe45368/torch/optim/lr_scheduler.py#L1027-L1044">internal manipulation of the learning rate</a> would yield a <a href="https://github.com/pytorch/pytorch/blob/0362b753dba22c1ccab2f9f7ee9f6de2ffe45368/torch/optim/lr_scheduler.py#L245-L248">warning</a>.</p>
Installing Pytorch cuda

username: Md_Shahriar
cooked: <p><code>windows 10</code></p>
<p>I have install Pytorch cuda by this …<br>
<code>conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia</code></p>
<p>Now …</p>
<pre><code class="lang-auto">torch.cuda.is_available()
&gt;&gt;&gt;False
</code></pre>
<p>But</p>
<pre><code class="lang-auto">torch.backends.cudnn.enabled
&gt;&gt;&gt;True
</code></pre>

username: ptrblck
cooked: <p>Did the install logs show that the <code>cudatoolkit</code> was indeed installed or did you see a fallback to a CPU version?<br>
If you’ve verified that the CUDA version was indeed picked, the error might be raised by a setup issue, i.e. your GPU cannot be detected.<br>
Is <code>nvidia-smi</code> working properly and do you have a sufficiently new NVIDIA driver to run the CUDA11.1 runtime?</p>

username: Md_Shahriar
cooked: <p>Please see the picture …</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/7/77008fe91cee8cd066ce2f591bde86b639a5365d.png" data-download-href="https://discuss.pytorch.org/uploads/default/77008fe91cee8cd066ce2f591bde86b639a5365d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/7/77008fe91cee8cd066ce2f591bde86b639a5365d.png" alt="image" data-base62-sha1="gYK1rTw66Kh1fV8uuYGBG44Tf0p" width="690" height="84" data-dominant-color="161616"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">938×115 4.09 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Md_Shahriar
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7f6eab52dcfd9b376c4773ea4009821d0e0e4a5.png" data-download-href="https://discuss.pytorch.org/uploads/default/f7f6eab52dcfd9b376c4773ea4009821d0e0e4a5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7f6eab52dcfd9b376c4773ea4009821d0e0e4a5.png" alt="image" data-base62-sha1="znAX1JivCbYcPBKgyzbJaD7T4QB" width="690" height="354" data-dominant-color="161616"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">712×366 8.54 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And also if i run a model in GPU , it show <code>AssertionError: Torch not compiled with CUDA enabled</code></p>

username: ptrblck
cooked: <p>This could mean that a CPU-only package is already installed. Create a new virtual environment and reinstall PyTorch there (and check the install logs for <code>cudatoolkit</code> or the corresponding CUDA tag in the wheel name) or alternatively uninstall all PyTorch packages in your current environment and reinstall it again.</p>
Is there a discord server for Pytorch Community?

username: mfcs
cooked: <p>I would like to know if Is there a discord server for Pytorch Community.</p>
<p>Some technologies have created their own servers on discord.</p>
<p>What do you all think?</p>
<p>Sorry for the off topic.</p>
<p>Best regards,</p>
<p>Matheus Santos.</p>

username: JuanFMontesinos
cooked: <p>There is a slack server you can apply to.</p>
Understanding Time Performance of F.Conv2d

username: eduardo4jesus
cooked: <p>In my project, I am trying to compare two techniques:</p>
<ul>
<li>
<code>spatial convolution</code> and <code>filtering via FFT</code>.</li>
</ul>
<p>At this point I am only focused on the number of operations. Optimizing parallelism is not part of my scope.</p>
<p>So, in theory, the techniques I am comparing against the standard ones should be fasts due to have less operation. But that hasn’t be the case when I time both executions. I am assuming that, though running in CPU, there is still a lot of parallelism going on.</p>
<p>So, how do I turn that off?</p>
<p>From the <a href="https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html" rel="noopener nofollow ugc">torch notes</a> I am assuming it would be something like this:</p>
<pre><code class="lang-auto">torch.set_num_interop_threads(1)
torch.set_num_threads(1)
</code></pre>
<p>But when timing both techniques, that still doesn’t give the results I am looking for, meaning, <code>spatial convolution</code> is still taking less time than <code>filtering via FFT</code>.</p>
<hr>
<p>PS: These are the accumulated timing for <code>128</code> runs for</p>
<ul>
<li>Standard Convolution</li>
<li>Custom implementation of Convolution, based on GEMM method</li>
<li>
<code>FFT filtering python</code> implementation</li>
<li>
<code>FFT filtering C++ extension</code> implementation</li>
</ul>
<pre><code class="lang-auto"># time_default, time_custom, time_python, time_cpp
0.8500628471374512, 5.313140630722046, 2.029076337814331, 1.942748785018921

# time_default, time_custom, time_python, time_cpp
# Setting threads num. to 1
0.894277811050415, 5.291566371917725, 1.9720547199249268, 1.8894264698028564
</code></pre>
<p>How come the standard conv2d is way faster when it suppose to be slower?</p>

username: eqy
cooked: <p>What does “standard conv2d” mean in this case? Are you referring to just calling <code>F.conv2d</code> as-is? Even with a single thread, this will typically dispatch to an MKLDNN implementation on many systems, which means it will use a highly optimized vectorized and memory-hierarchy-aware implementation. You are correct in that there will still be implicit parallelism here due to the use of vectorized instructions. On the other hand, even without parallelism, performance is going to be highly dependent on how the computation aligns with a CPU’s caches (which again is going to be highly optimized here).</p>
<p>So in practice, even if there is some theoretical complexity improvement, this improvement can often be difficult to realize given that different algorithms will often have different memory access patterns and different levels of vectorization friendliness. It becomes even more difficult to compare when the baseline is a highly tuned library implementation (as F.conv2d will use in this case).</p>
Autograd.grad memory leak when using sum, but no memory leak when using norm

username: NadavSellam
cooked: <p>i am using pytorch, and i need to calculate the gradients of the network at a series of points to add to the loss.<br>
i used the code in the function get_sdf_gradient_dont_work:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4eca130447f389f9320df22cb537a5afc6811527.png" data-download-href="https://discuss.pytorch.org/uploads/default/4eca130447f389f9320df22cb537a5afc6811527" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4eca130447f389f9320df22cb537a5afc6811527.png" alt="image" data-base62-sha1="bf0bszotaF7Fooi2zKa9NW6yhW7" width="690" height="314" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">942×429 43.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and i got a memory leak in the GPU.<br>
but when i use the code in the function get_sdf_gradient_work, it seems to work (not the functionality i needed, but no memory leak)</p>
<p>the network is a simple MLP using only linear layers and LeakyReLU.<br>
and the multiresolution function is:<br>
<span class="mention">@staticmethod</span><br>
def multi_resolution(points, resolutions):<br>
resolutions_sin = torch.cat([torch.sin((2 ** i) * pi * points) for i in range(1, resolutions)], dim=-1)<br>
resolutions_cos = torch.cat([torch.cos((2 ** i) * pi * points) for i in range(1, resolutions)], dim=-1)<br>
return torch.cat([points, resolutions_sin, resolutions_cos], dim=-1)<br>
does someone knows how to solve this ?</p>
Revert .pth model to FP32

username: Hiperdyne19012
cooked: <p>The .pth was train with mixed precision. How to I make the model into FP32 again?</p>

username: HDCharles
cooked: <p>generally my understanding is that training with mixed precision occurs as follows (pseudocode):</p>
<pre><code class="lang-auto">class Model:
  &lt;model code&gt;

def train(net):
  with torch.cuda.autocast()
    &lt;train loop&gt;

main:
  net = Model()
  train(net)
  save(net)
  
</code></pre>
<p>in which case, your model should be saved in fp32, since the autocast part just alters the calculations in there, not the model itself. If you did something different, i’m not sure if its possible to revert. I would double check the documentation thought to make sure you are doing things as expected:</p>
<p><a href="https://pytorch.org/docs/stable/amp.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/amp.html</a><br>
<a href="https://pytorch.org/docs/stable/notes/amp_examples.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/amp_examples.html</a></p>
Pytorch >= 1.7.0 support for A100 GPUs?

username: amirhf
cooked: <p>I read in the blog when pytorch 1.7 was released that now pytorch supports A100 GPUs. However the pre-built binaries apparently do not support sm_80 arch and pytorch has to be built from source to support that. Is this accurate? Is there a comprehensive guide on how to build from source and enable sm_80 arch support? Or even better, is there a proper pytorch docker image that supports sm_80?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="amirhf" data-post="1" data-topic="108211">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amirhf/48/5634_2.png" class="avatar"> amirhf:</div>
<blockquote>
<p>However the pre-built binaries apparently do not support sm_80 arch and pytorch has to be built from source to support that. Is this accurate?</p>
</blockquote>
</aside>
<p>No, that’s not true and the binaries are shipping with <code>sm_80</code> since the <code>1.7.0</code> release.</p>

username: aroibu1
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>  - I just updated my pytorch to version 1.8.1 and I am getting an error which basically says that on my A100 GPU sm_80 is not supported. I opened a separate thread here about this: <a href="https://discuss.pytorch.org/t/nvidia-a100-gpu-runtimeerror-cudnn-error-cudnn-status-mapping-error/121648" class="inline-onebox">NVIDIA A100 GPU - RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR</a></p>

username: aroibu1
cooked: <p>Right - I think I have fixed this now. I’ll post the solution in the other thread.</p>
How to mask linear layer input to prevent invalid feature input from updating parameters

username: Fly2Fly
cooked: <p>I have video features (Batch, Time, Spatial, Feature_dim), like (32,15,4,2048).<br>
and I need first a dimension transform using a <code>nn.Liear(2048,512)</code> .<br>
But some features are masked according the associating mask (B,T,S) (True means yes,False means no)<br>
How do I avoid updating this layer with unwanted features ?</p>

username: pascal_notsawo
cooked: <p>How is the mask applied? How do you update the parameters of this layer? Is there a loss function?</p>
<p>For example :</p>
<ul>
<li>
<p>in attention models the mask is multiplied by the attention scores to avoid taking them into account in the calculation of the output: everything is managed at the model level</p>
</li>
<li>
<p>in classification it is also common to provide a weight parameter to the loss function (binary_cross_entropy_with_logits, cross_entropy…) to avoid the update of some neuron in output : everything is managed at the loss level.<br>
Or if we have to deal with a regression, we can play on the loss like this:</p>
</li>
</ul>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

B, n = 4, 6

torch.manual_seed(0) 
y = torch.empty((B, n)).uniform_(-10, 10)
torch.manual_seed(1) 
y_pred = torch.empty((B, n)).uniform_(-10, 10)

if True :
    # if the mask is batch wise
    mask = torch.empty(B).random_(2).bool()
    loss = F.mse_loss(y_pred, y, reduction='none').sum(dim=1) * mask
else :
    # if the mask is element wise
    mask = torch.empty((B, n)).random_(2).bool() 
    loss = ( F.mse_loss(y_pred, y, reduction='none') * mask ).sum(dim=1) 

loss = loss.mean() #  loss.sum() 
loss
</code></pre>
<p>If possible provide more details about your task.</p>
How to update the resnet backbone of deeplabv3 with pretrained resnet weights?

username: seyeeet
cooked: <p>I was wondering if it is possible to update the resnet part of deeplabv3 with the pretrain weights of resnet50?</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.models as models

model = models.segmentation.deeplabv3_resnet50(num_classes=21, pretrained=False)
res50 = torchvision.models.resnet50(pretrained=False)
</code></pre>
<p>doing something like below is not working correctly:</p>
<pre><code class="lang-auto">model.backbone.load_state_dict(res50.state_dict(), strict=False)
_IncompatibleKeys(missing_keys=[], unexpected_keys=['fc.weight', 'fc.bias'])
</code></pre>

username: ptrblck
cooked: <p>I think it’s working as expected, since apparently the classifier of the resnet seems to be replaced with the <code>DeepLabHead</code>, so you won’t be able to load the <code>fc</code> parameters into the backbone.</p>

username: seyeeet
cooked: <p>oh I thought there was something wrong <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
Thanks for clarification, I can confirm that the weights of the backbone are getting updated  by checking<br>
<code>print(res50.conv1.weight[0,0,...]==model.backbone.conv1.weight[0,0,...])</code></p>
Feeding None as hidden state to LSTM/GRU

username: leckofunny
cooked: <p>Hello everybody,</p>
<p>I just came across a behavior that I would expect to throw an exception.<br>
If you feed None as hidden state to nn.GRU or nn.LSTM, it won’t throw an exception.<br>
Instead it will use a hidden state made of zeros.</p>
<p>I’m wondering whether this behavior is intentionally or not.</p>
<p>torch version 1.8.1+cu111</p>

username: pascal_notsawo
cooked: <p>This is intentional.<br>
If you look at the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html#LSTM" rel="noopener nofollow ugc">source code</a> you will see :</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py#L659" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py#L659" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/torch/nn/modules/rnn.py#L659</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="649" style="counter-reset: li-counter 648 ;">
<li>if isinstance(orig_input, PackedSequence):</li>
<li>    input, batch_sizes, sorted_indices, unsorted_indices = input</li>
<li>    max_batch_size = batch_sizes[0]</li>
<li>    max_batch_size = int(max_batch_size)</li>
<li>else:</li>
<li>    batch_sizes = None</li>
<li>    max_batch_size = input.size(0) if self.batch_first else input.size(1)</li>
<li>    sorted_indices = None</li>
<li>    unsorted_indices = None</li>
<li>
</li>
<li class="selected">if hx is None:</li>
<li>    num_directions = 2 if self.bidirectional else 1</li>
<li>    real_hidden_size = self.proj_size if self.proj_size &gt; 0 else self.hidden_size</li>
<li>    h_zeros = torch.zeros(self.num_layers * num_directions,</li>
<li>                          max_batch_size, real_hidden_size,</li>
<li>                          dtype=input.dtype, device=input.device)</li>
<li>    c_zeros = torch.zeros(self.num_layers * num_directions,</li>
<li>                          max_batch_size, self.hidden_size,</li>
<li>                          dtype=input.dtype, device=input.device)</li>
<li>    hx = (h_zeros, c_zeros)</li>
<li>else:</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Thoughts about RNN training with different batch_size

username: KendrickQ
cooked: <p>After reading RNN source code in Pytorch and some blogs regarding with RNN, I want to verify my thoughts about RNN training. That is, for example, <strong>if we set batch_size from 1 to 4, the training process is totally different.</strong></p>
<p>Here is the explanation about <em>nn.RNNN()</em> in Pytorch documentation.</p>
<blockquote>
<p><code>h_n</code> of shape (num_layers * num_directions, batch, hidden_size)</p>
</blockquote>
<p>if <code>batch_size = 1</code>, then we only get one init hidden state (typically is a tensor with all zeros). While if <code>batch_size = 4</code>, we get four init hidden states, which means the input hidden state of RNN training in the fourth word now is different from the input hidden state of previous training process.</p>
<p>Here is my reason. Considering the former one, the input hidden state of fourth word in this process (<code>batch_size=1</code>) <strong>contains the information of first three words</strong>, while the other one is totally zero. Therefore, two training processes are totally different.</p>
<p>Can anybody tells me whether my thoughts are right or not?</p>
Which way should I use to loop model using K-fold Cross Validation

username: sguerin
cooked: <p>Hi! I’m performing K-fold cross validation on inception v3 model and I’m a little confused about how to loop train, val and test modes in the model. Bellow I’ve written some pseudocode to show the two options that I’ve tried. In <em>Option A</em> train and validation are performed for every epoch in every fold. In <em>Option B</em> validation is perform for every fold.</p>
<p><strong>Option A</strong></p>
<pre><code class="lang-python">for fold in folds:
    for epoch in epochs:
        train
        val
# Load best model
test
</code></pre>
<p><strong>Option B</strong></p>
<pre><code class="lang-python">for fold in folds: 
    for epoch in epochs:
        train
    val
# Load best model
test
</code></pre>
<p>The model gets better results on validation set with option B. ¿Is there a correct/better way?</p>
<p>Thanks for the help!</p>
Layer not freezing

username: jpj
cooked: <pre><code class="lang-auto">        def forward(self, input):
           H1 = self.hidden_layer_1(input)
           H1 = self.relu(H1)
           final_inputs = self.output(H1)
  
        optimizer = torch.optim.SGD(self.parameters(), lr =lr,momentum=momentum)

        weights = np.load('./w0.npy')
        bias = np.load('./b0.npy')

        self.hidden_layer_1.weight.data = torch.from_numpy(weights).float()
        self.hidden_layer_1.bias.data = torch.from_numpy(bias).float()

        # freeze
        for param in self.hidden_layer_1.parameters():

          param.requires_grad = False
</code></pre>
<p>I am trying to freeze the layer but the loss is still changing after epochs.</p>

username: pascal_notsawo
cooked: <p>Try to do it (freeze the parameters) before passing them to the optimizer.</p>

username: jpj
cooked: <p>I tried that as well but training and validation loss still changing</p>

username: pascal_notsawo
cooked: <p>Is <code>hidden_layer_1</code> the only layer in your model?<br>
Show a more complete code if possible</p>

username: jpj
cooked: <p>Yes. I have included the forward function in the edited post.</p>

username: pascal_notsawo
cooked: <p>But I saw that you call <code>self.output</code> in this forward function.  <code>self.output</code> does not have a trainable parameters?<br>
Because if your model is still learning, it means that something is changing somewhere. If you only have one layer and you want to freeze it, what is the point of training the model here then?</p>

username: jpj
cooked: <p>I was checking if the freezing part is working for the first layer then I would add a second hidden layer which I would not freeze.</p>
Shape of input data to a LSTM with very long sequences

username: Planty
cooked: <p>Hello,</p>
<p>I’m new to pytorch and I have trouble to understand how my LSTM is working with different input shapes of my data.</p>
<p>My Data consist of signals where each sequence has a lengths of ~300 000. The sequence length differs between 5000 and 500 000 but manly the length is around 300 000. I try to solve a many-to-one task. So one sequence belongs to one class. In total I have 65 000 different sequences (each around 300 000 time steps long).</p>
<p>Example:<br>
Sequence 1 (length: 300 000) belongs to class 0<br>
Sequence 2 (length: 310 000) belongs to class 1<br>
Sequence 3 (length: 290 000)  belongs to class 2<br>
Sequence 4 (length: 280 000)  belongs to class 0<br>
…</p>
<p>I was running my LSTM with two different input shapes, while batch_first = true:</p>
<p><strong>(1,sequence,1)</strong> : Here I had to limit the sequence length to 500 to be able to train the network, I’m planning to use batches here, where one batch contains the howl sequence (600,500,1). I assume that the network is getting one value at a time but the training curve do not decrease.</p>
<p><strong>(1,1,sequence)</strong> :<br>
Here I padded or cut the sequence to have equal length (250 000) for all sequences<br>
I assume that the network is getting the howl sequence at the same time, the training curve decreases.</p>
<p>Both implementations were working and giving some results (not good ones but results).</p>
<p>Could someone explain me the difference of how the network is working with the different input shapes?<br>
Are both making sense?</p>
<p>Thanks for any help</p>
3D RoI Alignment in TorchVision

username: medvisioner
cooked: <p>Hi, I know there is a 2D RoI Alignment in TorchVision (torch.ops.torchvision.roi_align).</p>
<p>However, I am using 3D data (feature map with shape Batch, Channel, Depth, Height, Width).</p>
<p>Is there a 3D RoI Alignment available ?  Or how can I modify the 2D RoI Aligment to make it work for 3D data ?</p>
<p>Many Thanks.</p>
Find version of cuda library?

username: johann-petrak
cooked: <p>When pytorch gets installed, it will always also install its own cuda libraries for a specific cuda version, and for one pytorch version, it is possible to install that pytorch version with various versions of the cuda library included.</p>
<p>I have pytorch version 1.7.1, how can I figure out from within python what the version of the cuda library that was installed with it is?</p>

username: eqy
cooked: <p>Does <code>torch.version.cuda</code> work?</p>

username: johann-petrak
cooked: <p>Yes that worked, thank you!<br>
I tried all variations of <code>torch.cuda.__version__</code> instead <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
Question about using transforms with RCNN

username: Lelouch1
cooked: <p>Hello,</p>
<p>I am currently using a pretrained RCNN ( torchvision.models.detection.fasterrcnn_resnet50_fpn ) and I noticed that i could not use the transforms functions defined in <a href="https://pytorch.org/vision/stable/transforms.html" class="inline-onebox" rel="noopener nofollow ugc">torchvision.transforms — Torchvision master documentation</a> but only those defined in <a href="https://github.com/pytorch/vision/blob/master/references/detection/transforms.py" class="inline-onebox" rel="noopener nofollow ugc">vision/transforms.py at master · pytorch/vision · GitHub</a> otherwise I get an error. Do you know why the official functions does not work and do you know any way to make them functionnal because currently I can only use the 3 or 4 functions from the github.</p>
<p>Thanks</p>
Pretrained MobilenetV2 depth 0.35 support

username: IgorKasianenko
cooked: <p>Hello!<br>
I’ve been working with <a href="https://pytorch.org/vision/0.8/_modules/torchvision/models/mobilenet.html#mobilenet_v2" rel="noopener nofollow ugc">MobilenetV2</a> model and noticed that there is a pretrained version for it in</p>
<pre><code class="lang-auto">model_urls = {
    'mobilenet_v2': 'https://download.pytorch.org/models/mobilenet_v2-b0353104.pth',
}
</code></pre>
<p>and it works great!</p>
<p>Now I want to train 0.35 depth, I was able to create model passing <code>width_mult=0.35, </code> as <code>**kwargs</code> . It works.<br>
But when I try to create it with <code>pretrained=True</code> it crashes on size mismatch from checkpoint and model.</p>
<p>I found some <a href="https://github.com/d-li14/mobilenetv2.pytorch/blob/master/pretrained/mobilenetv2_0.35-b2e15951.pth" rel="noopener nofollow ugc">pre-trained models</a> with needed depth multiplier, but I’d like so guidance how to load them to instance of torchvision model.</p>
<p>If you guide me - would be happy to implement loading of this model it torch code, but first of all I need to find matching 0.35 weights.</p>
<p>Thank you for any recommendations.</p>

username: anargu
cooked: <p>Hi, searching for the same task. Did you find a solution to load the weights on 0.35 depth?</p>

username: IgorKasianenko
cooked: <p>Not really, I use pytorch at work for research and tf for production, so I took tf model 0.35 depth. If I will decide to train it from scratch, I will update</p>
How to implement timestopping and earlystopping callback on pytorch

username: Mars_Su
cooked: <p>Hi, I want to ask some problem about Callback.<br>
It i need to implement timestopping and earlystopping on native pytorch without pytorch-lightning or ignite.<br>
How do i implement? and why native pytorch does not support Callback object like tensorflow?<br>
I will be appreciate for any answer.</p>
Treating tensor as intervals

username: Eilson
cooked: <p>Hi,</p>
<p>I have an integer tensor T of shape Nx2 (e.g. [[2,5], [3,7], [1,4]] for N=3) and I want to convert it to a tensor of size NxC such that for every row n, only indices in the interval T[n] are assigned 1 and otherwise 0. I also know that every entry in T is between 0 and C.</p>
<p>Using the example above, when C = 8, the ideal output is:<br>
[[0, 0, 1, 1, 1, 1, 0, 0],<br>
[0, 0, 0, 1, 1, 1, 1, 1],<br>
[0, 1, 1, 1, 1, 0, 0, 0]].</p>
<p>Is there a way to do this without explicit loops?</p>

username: googlebot
cooked: <p>I = torch.arange(C,dtype=T.dtype)<br>
bools = (( I &gt;= T [:,0,None]) &amp; ( I &lt;= T[:,1,None])) # broadcasting (1,C) with (N,1)<br>
ints = bools.long()</p>
Loss between unordered matrices

username: sguysc
cooked: <p>What is a good loss function between a pair of two matrices that row <em>i</em> in the target matrix does not necessarily correspond to row <em>i</em> in the trained matrix? More specifically, I’m looking to minimize the sum of all errors between target and trained tensors, where row <em>i</em> in the target corresponds to row <em>j</em> in the trained tensor where the sum of all “distances” between <em>(i,j)</em> is minimal, and if <em>j</em> is already matched with <em>i</em>, then it cannot match another <em>k</em> in the target matrix as well (“without replacement”).</p>
<p>Is there a built-in function for this in torch?<br>
I’ve implemented maximum mean discrepancy (MMD) distance metric, but it’s not giving me satisfactory results.</p>
Run multiple independent models on same input data

username: oezguensi
cooked: <p>Hi am using <a href="https://github.com/jfzhang95/pytorch-deeplab-xception" rel="nofollow noopener">this</a> implementation of DeepLab for PyTorch and would like to perform hyperparameter optimization by training multiple models at the same time.<br>
This implementation uses <code>nn.DataParallel</code> to train a model on multiple GPUs.</p>
<p>I start one training process after executing <code>export CUDA_VISIBLE_DEVICES=0,1</code>. When I want to start the other training process, I get two different errors depending on the GPU ids in <code>CUDA_VISIBLE_DEVICES</code>.</p>
<p>If I type <code>export CUDA_VISIBLE_DEVICES=0,1,2,3</code> and train the second model on GPU 2 and 3, I receive:<br>
<code>RuntimeError: module must have its parameters and buffers on device cuda:2 (device_ids[0]) but found one of them on device: cuda:0</code></p>
<p>On the other hand if I execute <code>export CUDA_VISIBLE_DEVICES=2,3</code> I receive <code>AssertionError: Invalid device id</code>.</p>
<p>How can I train two models at the same time while another process already loaded the input data in a GPU.<br>
Do I have to specify on which device the model should run using <code>.to_device(cuda:x)</code>?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="oezguensi" data-post="1" data-topic="80554">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oezguensi/48/18134_2.png" class="avatar"> oezguensi:</div>
<blockquote>
<p>How can I train two models at the same time while another process already loaded the input data in a GPU.<br>
Do I have to specify on which device the model should run using <code>.to_device(cuda:x)</code> ?</p>
</blockquote>
</aside>
<p>Yes, I would recommend to use a single script with the <code>DataLoader</code>, create multiple model, push each one to the desired device via <code>to('cuda:id')</code> and just pass the data to each model.<br>
Since the training is done on different devices, it should be executed in parallel.</p>
<p>Your approach of running multiple scripts with <code>CUDA_VISIBLE_DEVICES</code> would make it unnecessary complicated to share the data between these processes.</p>

username: oezguensi
cooked: <p>Thank you. As soon as I didn’t use <code>CUDA_VISIBLE_DEVICES</code> and specified ‘.to(device)’ I can train multiple models, each on multiple GPUs, at the same time.</p>
<p>It looks like <code>_check_balance(device_ids)</code> somehow gets the device id 0 even though <code>nn.DataParallel(model, device_ids=[2,3])</code> has GPU IDs 2 and 3. That was the problem why using <code>CUDA_VISIBLE_DEVICES=2,3</code> was resulted in an <code>AssertionError: Invalid device id</code> .</p>

username: Carty-Bao
cooked: <p>Hello, Have you solved this problem? I am new at this field. I would like to inference two models on two gpus with same input. Which method do you recommend? cause I saw Distributed Data Parallel is only available on inferencing only one model on multiple gpus.</p>
Bazel build is not incremental

username: 111531
cooked: <p>hi,  last week I ran command “bazel build //:torch”, it works fine. but today, I ran the command again without any code change.  I found “bazel build //:torch” will rebuild the whole project.  why bazel build is not incremental ?</p>

username: 111531
cooked: <p>sorry，my fault.   bazel build //:torch is incremental if you have not restart laptop or shutdown the server.  that is why i found my ‘bazel build //:torch’ is not incremental</p>
Is there fp8 inference in Pytorch

username: 111179
cooked: <p>Similar to fp16 inference in pytorch framework like [<a href="https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html" rel="noopener nofollow ugc">Training With Mixed Precision :: NVIDIA Deep Learning Performance Documentation</a>](<a href="https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html" class="inline-onebox" rel="noopener nofollow ugc">Training With Mixed Precision :: NVIDIA Deep Learning Performance Documentation</a>), is there any framework about fp8 inference in pytorch?</p>
<p>Thank you for your time.</p>

username: HDCharles
cooked: <p>To my knowledge, PyTorch’s mixed precision support (<a href="https://pytorch.org/docs/stable/amp.html" class="inline-onebox" rel="noopener nofollow ugc">Automatic Mixed Precision package - torch.cuda.amp — PyTorch 1.8.1 documentation</a>) does not handle fp8 either.</p>
<p>For 8 bit precision you’d need to look towards quantization to integers or fake quants but that doesn’t really fall under the umbrella of mixed precision, though I’m not sure if that’s core to your request or just ancillary.</p>
<p>For info about quantization, you can see this: <a href="https://pytorch.org/docs/stable/quantization.html" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 1.8.1 documentation</a></p>

username: 111179
cooked: <p>I see. Thank you so much</p>
FasterRCNN - half the batch disappears when using DataParallel

username: Lakshay_Sharma
cooked: <p>I trained a FasterRCNN model with a single GPU. At inference time, I’d like to use multi-GPU prediction. However, I find that for a batch of say 50 images, the model’s output only has 25 predictions.</p>
<p>Here’s a part of my code</p>
<pre><code class="lang-auto">    model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False)

    N_CLASSES = 2 # 1 + background

    INP_FEATURES = model.roi_heads.box_predictor.cls_score.in_features

    model.roi_heads.box_predictor = FastRCNNPredictor(INP_FEATURES, N_CLASSES)

    device = torch.device('cuda') if torch.cuda.is_available() else torch.device

    if torch.cuda.device_count() &gt; 1:
        model = nn.DataParallel(model)
    print("Using", torch.cuda.device_count(), "GPUs")

    model.to(device)
    checkpoint = torch.load(args.pretrained_path)

    if torch.cuda.device_count() &gt; 1:
        model.module.load_state_dict(checkpoint['model_state_dict'])
    else:
        model.load_state_dict(checkpoint['model_state_dict'])

</code></pre>
<p>Any clues?</p>
<p>Thanks!</p>
KLDiv loss reduction

username: Aryan_Asadian
cooked: <p>Hi,</p>
<p>In <code>torch.nn.KLDivLoss</code>, when I set “reduction = mean”, I receive this warning:</p>
<p>UserWarning: reduction: ‘mean’ divides the total loss by both the batch size and the support size.‘batchmean’ divides only by the batch size, and aligns with the KL div math definition.‘mean’ will be changed to behave the same as ‘batchmean’ in the next major release.<br>
warnings.warn(“reduction: ‘mean’ divides the total loss by both the batch size and the support size.”</p>
<p>What does <strong>Support size</strong> mean in this context?</p>

username: InnovArul
cooked: <p>“Support size” refers to the dimension of the probability vector.</p>
<p>For example, if <code>x</code>, <code>y</code> are of dimension <code>B x D</code> (batch size = <code>B</code>, support size = <code>D</code>),</p>
<pre><code class="lang-auto">F.kl_div(x, y, reduction='mean') * D is equal to (F.kl_div(x, y, reduction='batchmean') 
</code></pre>

username: Aryan_Asadian
cooked: <p>Thank you for your reply. Does this “mean” reduction in KLDivLoss is equal to the functionality of “mean” in CrossentropyLoss as well?<br>
In the documentation for <code>torch.nn.CrossEntropyLoss</code> it has mentioned :</p>
<blockquote>
<pre><code class="lang-auto">'mean': the weighted mean of the output is taken
</code></pre>
</blockquote>

username: InnovArul
cooked: <p>In my understanding, <code>batchmean</code> of <code>KLDiv</code> is similar to the <code>mean</code> of <code>CrossEntropyLoss</code>.</p>

username: karl7
cooked: <p>hi, <a class="mention" href="/u/innovarul">@InnovArul</a> , i also met this warning in computing kl loss, do i need to set the reduction to ‘none’ to avoid this warning? or just ignore it? which is the right way?</p>

username: InnovArul
cooked: <p>In my experience, for most cases, <code>reduction='batchmean'</code> would suffice.</p>
Embedding pytorch for machine learning iusing DSP

username: Saulo_Barreto
cooked: <p>Dear PyTorch development community,<br>
We are developing an application of neural network to run on an embedded device with microprocessor + DSP. Our application will perform the learning process so we need to perform the forward and backward propagation on the embedded device. We are assessing the possibility of using Pytorch and would like to know from you if your experts have any suggestions regarding this task and/or if there is any kind of incompatibility that we can face due to the use o DSPs with the library.</p>
DataLoader produces NaNs with DDP

username: jopo666
cooked: <p>I have a weird and persistent problem where everything works fine when training with one GPU but when I move onto multi-GPU training some <strong>individual pixels</strong> in my input images become NaNs and this ofc crashes the training. It happens with random images and there is nothing wrong with the images as I check for NaNs in my <code>Dataset.__call__</code> function. Then during <code>training_step</code> the NaNs magically appear, with random inputs at random pixels. So there may be problems inside the <code>collate_fn</code>?</p>
<p>Has anyone encountered anything similar?</p>

username: wayi
cooked: <p><a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> for dataloader</p>

username: jopo666
cooked: <p>Seems to be the same problem as <a href="https://discuss.pytorch.org/t/dataset-input-nan-but-fine-on-index/120733">120733</a></p>
How to get the instruction trace from running

username: Edward_Zhou
cooked: <p>Hi,</p>
<p>I am currently trying to get the instruction trace of running ML model inference or training. Then I decided to port these instruction traces to gem5 (architecture simulator). Can I ask how I can do this? This is not clear now because this is not like C, C++, where I can compile the program to get the assembly. I don’t even know whether it is possible to get the instruction trace or assembly from interpreter language like Python.</p>
<p>Many thanks!!<br>
Edward</p>
What are the differences between the namespaces of C++ Extensions?

username: eduardo4jesus
cooked: <p>I notice that we have some components in more than one namespace.</p>
<p>Such as <code>torch::Tensor</code> and <code>at::Tensor</code>, or <code>torch::fft::fft</code> and <code>at::_fft_c2c</code>. In general, it seems we have four namaspaces in <code>Aten</code>.</p>
<ul>
<li>
<a href="https://pytorch.org/cppdocs/api/namespace_at.html#namespace-at" rel="noopener nofollow ugc">at</a>;</li>
<li>
<a href="https://pytorch.org/cppdocs/api/namespace_c10.html#namespace-c10" rel="noopener nofollow ugc">c10</a>;</li>
<li>
<a href="https://pytorch.org/cppdocs/api/namespace_caffe2.html#namespace-caffe2" rel="noopener nofollow ugc">caffe2</a> which is empty; and</li>
<li><a href="https://pytorch.org/cppdocs/api/namespace_torch.html#namespace-torch" rel="noopener nofollow ugc">torch</a></li>
</ul>
<p><strong>Questions</strong></p>
<ol>
<li>Why do we have four of them?</li>
<li>Does each of them have a different purpose that I am not realizing yet?</li>
</ol>

username: eduardo4jesus
cooked: <p>Apparently, <code>at</code> namespace is old, and <code>torch</code> namespace is a new implementation, as <a href="https://github.com/pytorch/pytorch/issues/14257#issuecomment-440487374" rel="noopener nofollow ugc">per this link</a>.</p>