akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch equivalent of TF reduce_max

sukanya_kudi: <p>Hi,<br>
Is there a reduce_max equivalent of TF(can take multiple dims as input)?<br>
the torch.max operation allows only one dim as input.<br>
I require this for implementation of RMAC. If there are any other alternatives please suggest.</p>

krishnavishalv: <p>Is this what you are looking for ?</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/filipradenovic/cnnimageretrieval-pytorch/blob/master/cirtorch/layers/functional.py#L22" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/filipradenovic/cnnimageretrieval-pytorch/blob/master/cirtorch/layers/functional.py#L22" target="_blank" rel="nofollow noopener">filipradenovic/cnnimageretrieval-pytorch/blob/master/cirtorch/layers/functional.py#L22</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="12" style="counter-reset: li-counter 11 ;">
<li># return F.adaptive_max_pool2d(x, (1,1)) # alternative</li>
<li>
</li>
<li>def spoc(x):</li>
<li>return F.avg_pool2d(x, (x.size(-2), x.size(-1)))</li>
<li># return F.adaptive_avg_pool2d(x, (1,1)) # alternative</li>
<li>
</li>
<li>def gem(x, p=3, eps=1e-6):</li>
<li>return F.avg_pool2d(x.clamp(min=eps).pow(p), (x.size(-2), x.size(-1))).pow(1./p)</li>
<li># return F.lp_pool2d(F.threshold(x, eps, eps), p, (x.size(-2), x.size(-1))) # alternative</li>
<li>
</li>
<li class="selected">def rmac(x, L=3, eps=1e-6):</li>
<li>ovr = 0.4 # desired overlap of neighboring regions</li>
<li>steps = torch.Tensor([2, 3, 4, 5, 6, 7]) # possible regions for the long dimension</li>
<li>
</li>
<li>W = x.size(3)</li>
<li>H = x.size(2)</li>
<li>
</li>
<li>w = min(W, H)</li>
<li>w2 = math.floor(w/2.0 - 1)</li>
<li>
</li>
<li>b = (max(H, W)-w)/(steps-1)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
What is the tgt in the torch.nn.Transformer?

shamoons: <p>Specifically, I see <a href="https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html#torch.nn.Transformer" rel="noopener nofollow ugc">here in the docs</a>:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; transformer_model = nn.Transformer(nhead=16, num_encoder_layers=12)
&gt;&gt;&gt; src = torch.rand((10, 32, 512))
&gt;&gt;&gt; tgt = torch.rand((20, 32, 512))
&gt;&gt;&gt; out = transformer_model(src, tgt)
</code></pre>
<p>I’m unsure what the <code>tgt</code> is. The docs say <code>tgt – the sequence to the decoder (required).</code>. But I’m not passing a sequence to the decoder. I want the decoder to give me an output, don’t I?</p>
Why do PyTorch Transformer tutorials typically not have a Decoder?

shamoons: <p>I’m trying to find a Transformer tutorial with PyTorch that doesn’t use embeddings. I have time-series data, but every tutorial I find (<a href="https://github.com/kuberlab-catalog/pytorch-tutorials-code/blob/9e5d67daaa819a24f50bea5f44e1afb490c71092/beginner_source/transformer_tutorial.py" rel="noopener nofollow ugc">https://github.com/kuberlab-catalog/pytorch-tutorials-code/blob/9e5d67daaa819a24f50bea5f44e1afb490c71092/beginner_source/transformer_tutorial.py</a>) has the Encoder, but ends up with a <code>Linear</code> layer. No <code>decoding</code>. I’m wondering why that is?</p>
Size of libtorch/CUDA DLLs

StefanD: <p>Hi,</p>
<p>we are currently upgrading from libtorch 1.4 to 1.7 in order to support Ampere GPUs.<br>
We noticed that torch.dll is larger, new CUDA dlls (e.g. cusolver) have been introduced and existing ones (e.g. cuDNN) are much larger.<br>
This is problematic because we need to ship some of our products on DVD and we are running out of space.</p>
<p>We have analyzed which DLLs are actually called at runtime and it seems like only cudart and cuDNN (except adv) are used. We are only using CNNs (training and inference).</p>
<p>It would be great if libtorch could be configured at compile-time so that only required functionality (e.g. CNNs) is enabled. As a workaround we could link the unused DLLs with delay loading and not deploy them, but we aren’t sure if this is safe.</p>

ptrblck: <p>Could you please create a feature request on GitHub and tag me there, so that we can track it?</p>

StefanD: <p>Done.<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/47751" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/47751" target="_blank" rel="noopener nofollow ugc">Make libtorch modular</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-11-11" data-time="13:52:28" data-timezone="UTC">01:52PM - 11 Nov 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/stefandiep" target="_blank" rel="noopener nofollow ugc">
          <img alt="stefandiep" src="https://avatars2.githubusercontent.com/u/74306785?v=4" class="onebox-avatar-inline" width="20" height="20">
          stefandiep
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🚀 Feature
libtorch binaries have grown substantially over the last releases, to a point where this is a problem because we need...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Understanding Model Structure/ModuleList

L-Reichardt: <p>Hello,<br>
I have the following code, which I do not understand completely:</p>
<pre><code class="lang-auto">class ResMod(nn.Module):
    def __init__(self, cin, cout, reps=5):
        super(ResMod, self).__init__()
        self.expand = BottleneckSparse(cin, cout, use_norm=False)
        self.pool = MASPool(cout, 2, 2)
        self.layers = nn.ModuleList([BottleneckSparse(cout, cout, use_norm=False) for _ in range(reps)]) 

    def forward(self, input):
        x, m = input
        x, m = self.expand((x, m))
        x, m = self.pool((x, m))

        for L in self.layers:
            x, m = L((x, m))

        return x, m

</code></pre>
<p>Do I understand this correctly, that the custom BottleneckSparse Layer is used 5 times in the ModuleList?</p>
<p>Is the forward path then:<br>
Bottleneck &gt; Pool &gt; 5x Bottleneck?<br>
with the bottlenecks being added in the “for L”-part?</p>

klory: <p>yes, but they are five different BottleneckSparse</p>

L-Reichardt: <p>Thank you. I was confused because the layer order seems very impractical.</p>
About converting PIL Image to PyTorch Tensor

zhuhonglinX: <p>I use PIL open an image:</p>
<pre><code class="lang-python">pic = Image.open(...).convert('RGB')
</code></pre>
<p>Then I want to convert it to tensor, I have read <code>torchvision.transforms.functional</code>, the function <code>to_tensor</code> use the following way：</p>
<pre><code class="lang-python">img = torch.ByteTensor(torch.ByteStorage.from_buffer(pic.tobytes()))
img = img.view(pic.size[1], pic.size[0], len(pic.getbands()))
</code></pre>
<p>Why not use the following method directly:</p>
<pre><code class="lang-python">img = torch.from_numpy(np.array(pic))
</code></pre>
<p>I want to know what is the difference, is it because of the difference in efficiency？</p>

klory: <p>Did not know about the first line, but the second line change [W,H,C] =&gt; [C,W,H]</p>
Copy weights from Conv2D with bias=True to Conv2D with bias=False

JPM: <p>Hi,</p>
<p>I have a model that is a sequence of Conv2D with bias=True, I have another model which also is a sequence of Conv2D but with bias=False.</p>
<p>How can I copy the weights from the first model (incorporating the bias) to the second model that just has weights?</p>
<p>Any pointers are greatly appreciated.</p>
<p>Thanks!</p>

ptrblck: <p>Yes, you can skip the additional parameters by using <code>strict=False</code> in <code>load_state_dict</code> as shown here:</p>
<pre><code class="lang-python">conv_bias = nn.Conv2d(3, 3, 3, 1, 1, bias=True)
conv_nobias = nn.Conv2d(3, 3, 3, 1, 1, bias=False)

conv_nobias.load_state_dict(conv_bias.state_dict(), strict=False)
&gt; _IncompatibleKeys(missing_keys=[], unexpected_keys=['bias'])
</code></pre>
Random questions

OuisYasser: <p>Hello everyone, I have a random question if anyone can respond it and thank you.</p>
<ol>
<li>let’s say I’m doing social distanciation project, but how for example we can do it ? Like I don’t think that training a model on a dataset will be able to detect the distance between two persons??<br>
What really I’m asking, how to train a model to detect the distance? If there is any other solution please provide it.</li>
</ol>

Kushaj: <p>You can detect person/face/something to get (x,y) value for each person of interest. After that you need to transform your image from image to planar scale. Homography is the term you are looking for to do that. See <a href="https://docs.opencv.org/master/d9/dab/tutorial_homography.html" rel="noopener nofollow ugc">opencv</a> docs for info on this. Now you can compute the distances between all pairs of points.</p>
Get RuntimeError: CUDA error: no kernel image is available for execution on the device on weight initialisation with cuda 10.2, torchvision 0.6.0 and torch 1.5.0

VFernandez: <p>I am deploying a neural network in my Ubuntu machine and when the weights are initialised, I get an error: “RuntimeError: CUDA error: no kernel image is available for execution on the device”.</p>
<p>In nvidia-smi, the CUDA version is 10.2.<br>
I have 2 GPUs. One of them (K40c) is very old and requires a low version of torchvision (0.4.0) but I am not using it: I specify cuda:1 and make sure that 1 points to the newest GPU device (Titan V).</p>
<p>Going to <a href="https://pytorch.org/get-started/previous-versions/" rel="noopener nofollow ugc">https://pytorch.org/get-started/previous-versions/</a>, I made sure I had installed torch and torchvision so that they were compatible with version 10.2:</p>
<ul>
<li>torch version: 1.5.0</li>
<li>torchvision: 0.6.0</li>
</ul>
<p>Still, I have this issue, in particular in line:</p>
<p>init.xavier_normal_(m.weight.data, gain=gain)<br>
File “…/python3.6/site-packages/torch/nn/init.py”, line 282, in xavier_normal_<br>
return <em>no_grad_normal</em>(tensor, 0., std)<br>
File “…/python3.6/site-packages/torch/nn/init.py”, line 19, in <em>no_grad_normal</em><br>
return tensor.normal_(mean, std)<br>
RuntimeError: CUDA error: no kernel image is available for execution on the device.</p>
<p>Is it possible that the old GPU, despite not being used, is still causing this problem?</p>
<p>Thank you.</p>

ptrblck: <aside class="quote no-group" data-username="VFernandez" data-post="1" data-topic="101499">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/f6c823/48.png" class="avatar"> VFernandez:</div>
<blockquote>
<p>Is it possible that the old GPU, despite not being used, is still causing this problem?</p>
</blockquote>
</aside>
<p>Could you rerun the code with</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=id python script.py args
</code></pre>
<p>where <code>id</code> should be 0 or 1 depending which GPU is mapped to the device id and make sure that only the TitanV is found?</p>
<p>The error could also point to an NVIDIA driver, which is too old.<br>
For CUDA10.2, you would need <code>&gt;=440.33</code> as given in <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html">this table</a>.</p>

VFernandez: <p>Hello! Thank you for your answer.<br>
Using CUDA_VISIBLE_DEVICES doesn’t work, same error is popping out.<br>
The driver version is 440.100, so it should be okay according to the NVIDIA website.</p>

ptrblck: <p>I don’t know why it shouldn’t work, as I’m using a TitanV myself with the binaries, different CUDA versions, and builds from source.<br>
If <code>CUDA_VISIBLE_DEVICES</code> is not working, you could try to disable or remove the old GPU for the sake of debugging and rerun your script.</p>
[data paralled] RuntimeError: Gather got an input of invalid size: got [2, 4, 6, 300, 128], but expected [2, 5, 6, 300, 128]

Hu_Penglong: <p>I’m training a language model using gpt2, when i used multi-gpus<br>
RuntimeError: Gather got an input of invalid size: got [2, 4, 6, 300, 128], but expected [2, 5, 6, 300, 128]<br>
gpus=2, the batch_size=9. It seems the two gpu get different size of bath_size：one is 4 , the other is 5.<br>
how can i fix this.</p>
'ValueError: optimizer got an empty parameter list' when it's clearly not empty

Asa-Nisi-Masa: <p>Here is the code I’m trying to run on Google Colab:</p>
<pre><code class="lang-auto">import torch


class Model(torch.nn.Module):

    def __init__(self):
        super(Model, self).__init__()

        self.U = torch.nn.Embedding(12, 8, sparse=True)
        self.V = torch.nn.Embedding(18, 8, sparse=True)

    def forward(self):
        # to be implemented
        pass

model = Model()
optimizer = torch.optim.SparseAdam(model.parameters(), lr=2e-3)
</code></pre>
<p>and I get a ValueError. However, printing <code>list(model.parameters())</code> shows that the parameters clearly exist. What could be the issue here?</p>

Nikronic: <p>Hi,</p>
<p>I am not sure but it may be related to an issue in this thread:</p>
<aside class="quote" data-post="6" data-topic="1501">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/error-optimizer-got-an-empty-parameter-list/1501/6">ERROR:optimizer got an empty parameter list</a> 
  </div>
  <blockquote>
    Do: 
G_params = list(G.parameters())
D_params = list(D.parameters())

.parameters() is a generator, and probably for debugging purposes you are pre-populating it somewhere.
  </blockquote>
</aside>

<p>I have no clue but apparently it works!</p>
<p>Bests</p>

Asa-Nisi-Masa: <p>Thank you sir. It works (I also have no idea why).</p>

ptrblck: <p>This seems to be a newly introduces bug, so could you please create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> so that we can track and fix it?<br>
I can reproduce it in <code>1.7.0</code>, <code>1.8.0.dev20201110</code>, but not <code>1.7.0.dev20200830</code>.</p>
Using Torchtext to Load Sentences and Graphs in Batches

John_Watkins: <p>Hi,</p>
<p>I am wondering if there is anyway to use existing torchtext modules (Fields, Datasets, Iterators) to efficiently load data batches containing both sentences and graphs. Specifically, I am trying to achieve the following:</p>
<pre><code class="lang-auto"> for batch in bucket_iterator:
        src_sents = batch.src_sents # This is the src sentence, a LongTensor
        src_nodes = batch.src_nodes # This is the src graph node labels (will convert to embedding), a LongTensor
        src_edges = batch.src_edges # This is the src graph edges, a SparseTensor (from torch_sparse)
        tgt_sents = batch.tgt_sents # This is the tgt sentence
        # do model training with above information
</code></pre>
<p>The main issue here is that the torchtext modules have many features that are necessary for preprocessing the text portion of my data. However, I need to also load the graph associated with each text. I have not yet found an efficient way to load the graph. Is there perhaps a way to build custom fields for graph nodes and edges?</p>
<p>Any help would be appreciated.</p>
<p>For clarification, please note that the graph is intended to work with modules in the pytorch_geometric library.</p>
Adam optimizer doesn't converge while SGD works fine

Dawid_S: <p>I am training a seq2seq model using SGD and I get decent results. My batch size is 2, and I don’t average the loss over the number of steps.</p>
<p>I am using PyTorch this way:</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3)
</code></pre>
<p>SGD works fine, I can observe losses decreasing slowly, and the final accuracy is pretty good.</p>
<p>Obviously, I wanted to try Adam optimizer to check if I can get the results any faster.</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3)
</code></pre>
<p>But Adam doesn’t work at all. None of the training/dev/test losses decreases (only noise) and they have far too big values.</p>
<p>Do you have any idea what may cause such behaviour?</p>

ptrblck: <p>Have you played around with the learning rate of Adam?<br>
I assume besides changing the optimizer the code stays the same, so that would be my first hyperparameter to tweak.</p>

Dawid_S: <p>I think 0.001 is pretty standard value, so it’s weird for me that it doesn’t work at all. Yes, the same with 0.0001.</p>
<p>Also I think the network is very sensitive to the SGD learning rate. It barely works with any changes</p>

Thibault_Escobar: <p>Hello, I have the same probleme… Did you find an explanation ?</p>
<p>Thanks.</p>
<p>Thibault</p>

Dawid_S: <p>Well, eventually I was able to train an almost sensible neural net using Adam with 0.0001 or 0.00001 lr, I don’t remember. It was still clearly worse than SGD so I abandoned it, but I was comfortable with the fact that it’s probably possible, so maybe I don’t have any NN bugs</p>

Dawid_S: <p>But I encourage you to try your own ideas and share insights! I think seq2seq networks have a lot of specific issues I couldn’t find on the internet.</p>

Albert_Christianto: <p><a class="mention" href="/u/dawid_s">@Dawid_S</a> try to change the learning rate a bit.<br>
I also have the same problem.<br>
In a while, I trust SGD more than adam,<br>
but it doesn’t make sense when the others report that adam have better performance than SGD. And it turns out that we must set correct learning rate to start.<br>
source: from my bloody experience using adam.<br>
CMIIW. I also want to know why this bug happen all of the time</p>
<p>best regards,<br>
Albert Christianto</p>

TANKUN_LI: <p>I also have the same problem. I implement a ResNet+LSTM model. It’s funny that it seems adam is more unstable, or converge really slow while SGD only takes 6 epoches to converge.<br>
But most of the report claimed that Adam should converge more quickly.</p>
Bugs on ".cuda()"

lzypku: <p>When I use " .cuda()", something gets wrong.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; A = torch.tensor([1,2])
&gt;&gt;&gt; B = A.cuda()
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1579040055865/work/aten/src/THC/THCGeneral.cpp line=50 error=71 : operation not supported
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/lzy/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/cuda/__init__.py", line 197, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (71) : operation not supported at /opt/conda/conda-bld/pytorch_1579040055865/work/aten/src/THC/THCGeneral.cpp:50
</code></pre>

ptrblck: <p>This error is sometimes raised by a wrong usage of multiprocessing and CUDA, but this doesn’t seem to be the case here.<br>
What does <code>torch.cuda.is_available()</code> return, which PyTorch and CUDA runtime versions are you using, and which GPU?</p>

lzypku: <pre><code class="lang-auto">&gt;&gt;&gt; torch.cuda.is_available() 
False
&gt;&gt;&gt; print(torch.__version__)
1.4.0

cat /usr/local/cuda/version.txt 
CUDA Version 10.0.130

nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130
</code></pre>
<p>My gpu is 1080Ti.</p>
<p>It seems like there is no use of cuda.</p>

ptrblck: <p>Your local CUDA toolkit will not be used, if you installed the conda binaries or pip wheels.<br>
Did you properly select the desired CUDA runtime <a href="https://pytorch.org/get-started/locally/">here</a>, which should ship in the binaries? If so, what NVIDIA driver are you using?</p>
Transform CNN input dimension

snowe: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have a question regarding possible transformations of a CNN input. I know that the input should be in the form <code>batch_size x number_of_channels x height x width</code>. Let’s assume I have a tensor of shape [2, 3, 5, 5]. So two images with three channels each and the image is of size 5 by 5.</p>
<p>Is there a neat way to transform this input into 3 x [2, 1, 5, 5]? Basically, I would like to split each channel of every image in the batch and then have 3 such inputs. All I can think of are a bunch of nested for loops but maybe there is a better / more common way to do this?</p>
<p>Any help is very much appreciated!</p>
<p>All the best<br>
snowe</p>

klory: <pre><code class="lang-auto">x = torch.randn(2,3,5,5)
x_ = x.transpose(0,1).unsqueeze(2)
x_ = list(x_)
</code></pre>

snowe: <p><a class="mention" href="/u/klory">@klory</a> Thank you so much, that is exactly what I was looking for! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Transformer Encoderlayer outputs nan

Sine: <p>Hi everyone,<br>
I am trying to use Transformer Encoder Layer with src_key_padding_mask to be the encoder in the multi-turned dialogue generation task, but i get NaN.</p>
<p>I found the attention output is NaN when the sentence is all  PAD. Because all tokens in the sentence is converted to -inf, the <code>Softmax</code> returns NaN as results.</p>
<p>However, to fix the size of all contexts in the batch, I need sentences with all PAD tags.</p>
<p>There is an simple example as follow:</p>
<pre><code class="lang-auto">context: [[[1,2,3,0,0],[1,1,0,0,0],[0,0,0,0,0]],
          [[1,2,3,0,0],[1,1,0,0,0],[2,5,6,0,0]]]
mask: [[[False,False,False,True,True],[False,False,True,True,True],[True,True,True,True,True]],
       [[False,False,False,True,True],[False,False,True,True,True],[False,False,False,True,True]]]
</code></pre>
<p>0 is the PAD id.</p>
<p>I input the sentences to the encoder layer one by one, so [0,0,0,0,0] get NaN.</p>
<p>How can I solve this?<br>
Is it essential to use the mask in the Transformer Encoder Layer?</p>

Abhilash_Srivastava: <ol>
<li>Are the values in context, independent of each other? ie does context[1] has any relation to context[0]? Padding is typically used when we want to uncover one token at a time (and hide the other tokens for the future).</li>
<li>If you’re simply inputting this sequence one at a time, you may not use a mask at all. Simply padding it with <code>0</code> values at the end, would let your model learn that these tokens (PAD tokens) are not useful. The softmax can still be applied and it should work just fine.</li>
</ol>

Sine: <p>Thank you for your answer. I will try it.</p>
<p>Appreciate it.</p>
Fastest way to implement

kolorado: <p>Hi,</p>
<p>I want to fill a tensor like this:</p>
<pre><code class="lang-auto">gainW = torch.tensor(1.1)
gainH = torch.tensor(2.5)
res = 5
a = torch.zeros(res,res)
a[0,0] =0.1
for i in range(res):
    for j in range(res):
        if i==0 and j==0:
            continue
        if j==0:
            a[i,j] = torch.cos(a[i-1,0]*gainH)
        else:
            a[i,j] = torch.sin(a[i,j-1]+gainW)

print(a)
</code></pre>
<pre><code class="lang-auto">tensor([[ 0.1000,  0.9320,  0.8955,  0.9112,  0.9046],
        [ 0.9689,  0.8785,  0.9180,  0.9016,  0.9086],
        [-0.7523,  0.3408,  0.9916,  0.8674,  0.9224],
        [-0.3049,  0.7139,  0.9706,  0.8777,  0.9184],
        [ 0.7233,  0.9683,  0.8788,  0.9179,  0.9017]])
</code></pre>
<p>But this is extremely inefficient, How can I improve it? Any ideas would be appreciated!</p>
<p>BTW, I want to calculate later the gradient with respect to gainW,gainH!</p>

seungjun: <pre><code class="lang-python">import torch

gainW = torch.tensor(1.1)
gainH = torch.tensor(2.5)
res = 5
a = torch.zeros(res,res)
a[0,0] = 0.1
for i in range(1, res):
    a[i, 0] = torch.cos(a[i-1, 0] * gainH)

for j in range(1, res):
    a[:, j] = torch.sin(a[:, j-1] + gainW)
</code></pre>
<p>This might be faster.<br>
As your case requires the previous computation result to get the next, it’s difficult to avoid for loops entirely.</p>

kolorado: <p>Thnx, Do you think this is the best I can do? Do you think putting it in JIT (trace) would help? Does trace keep the gradient flow?</p>
Conditional Generative Adversarial Network output on non-trained condition

Prateek_Kukreja: <p>I am new and currently learning about GANs, I have a query as follows:</p>
<p>Suppose I have trained my conditional GAN on MNIST dataset where I have conditioned on gaussian noise with different means. Ex:</p>
<p>Gaussian noise with mean 1 will give output as image of 1.</p>
<p>Gaussian noise with mean 2 will give output as image of 2. and so on.</p>
<p>After training , suppose I give gaussian noise with mean 1.5, what output image should I expect?</p>
<p>This logic I am extending for an Image to Image translation task. Suppose I have conditioned my GAN(pix2pix) model to add <em>effect1</em> on input image for gaussian noise with mean 1, and add <em>effect2</em> on input image for gaussian noise with mean 2. What output image can I expect if I give input image with gaussian noise with mean 1.5? will it be combination of effect1 and effect2 or can be anyone of them or some random output?</p>

ptrblck: <p>My guess is that the output might not be a clear interpolation between the two original classes, since your model got a new and unknown input.<br>
Similar behavior can be seen, if you train a GAN with a latent vector of dim <code>Z</code> by sampling from a Gaussian distribution. If you try to directly interpolate between two valid points, your input would leave the “dense” part of the distribution (i.e. the high-dimensional bubble) and your model could output artifacts.</p>
<p>PS: let us know what the outcome is, in case you are running these experiments! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Model working on Windows but not Linux

EmreTokyuez: <p>I am trying to deploy my flask application using Docker, but before running it in a container, I would like to verify that it works in a linux environment.</p>
<p>After installing all the packages in the right versions (pytorch 1.6, torchtext 0.7.0) as I did in windows, I get the following error for my exported model:</p>
<pre><code class="lang-auto">untimeError: Error(s) in loading state_dict for RNN:
        size mismatch for embedding.weight: copying a param with shape torch.Size([101616, 100]) from checkpoint, the shape in current model is torch.Size([100944, 100]).
</code></pre>
<p>How can i fix this?</p>

ptrblck: <p>One <code>nn.Embedding</code> layer changed its size and thus the <code>state_dict</code> cannot be loaded.<br>
Check, how the <code>num_embeddings</code> are calculated on your systems and make sure to use the same value (e.g. <code>101616</code> from the checkpoint).<br>
Often these errors between Windows and Linux happen, because your code might read/sort/find different files or files in the wrong order etc., as these system calls behave differently on these OSes.</p>
ValueError: can't optimize a non-leaf Tensor” for multidimensional tensor

dhyey: <p>i have output x from network 1 and another tensor y (with require grad = True) ! now  i want to concatenate tensor x and y to feed in network2 but when optimizing it raises an error and y.grad tensor is full of zeroes!!<br>
how to solve this problem??</p>

ptrblck: <p>Based on the error it seems you are trying to pass a non-leaf tensor (e.g. an output activation) to the optimizer, which is not supported. Could this be the case?</p>
Will a model trained using the CUDA 10.1 version of pytorch work properly on a pytorch version using CUDA 11?

BaruchG: <p>Basically what the title asks.  Is this guaranteed to work and is this documented somewhere?  Also, will the pytorch cuda 10.1 version work on a system with cuda 11 installed?</p>

ptrblck: <p>PyTorch conda binaries and pip wheels ship with their own CUDA (, cudnn, NCCL, etc.) runtime and your local CUDA toolkit will not be used unless you build PyTorch from source or any custom CUDA extension.</p>
<p>Yes, CUDA11 should work, if it supports your GPU architecture. If not, please create a topic here or an issue on GitHub and we’ll take a look.</p>
How to convert a fully supervised landmark estimation model to a semi-supervised one?

Mona_Jalal: <p>Imagine that I have a model that can do 2D landmark predictions for a moving deformable object (e.g. facial landmarks). How can I change this model such that it would work in a semi-supervised fashion in which I only have annotation for 500-1000 frames but I want the algorithm to make annotations on the rest of frames (in which rest of the frames don’t have groundtruth)?</p>
<ol>
<li>How the current model in the link should be changed?</li>
<li>What evaluation score should be used since we don’t have the grountruth for rest of the frames and we can’t use something like MSELoss?</li>
<li>Assume that the 500-1000 frames to be annotated is selected via a clustering algorithm such as K-means.</li>
</ol>
<p>Here’s an example a fully supervised 2D facial landmark estimation problem:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://i0.wp.com/thecleverprogrammer.com/wp-content/uploads/2020/10/cropped-logo_transparent-1.png?fit=32%2C32&amp;ssl=1" class="site-icon" width="32" height="32">
      <a href="https://thecleverprogrammer.com/2020/07/22/face-landmarks-detection/" target="_blank" rel="noopener nofollow ugc" title="06:27AM - 22 July 2020">Data Science | Machine Learning | Python | C++ | Coding | Programming |... – 22 Jul 20</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/388;"><img src="https://i0.wp.com/thecleverprogrammer.com/wp-content/uploads/2020/07/un-113.png?fit=1280%2C720&amp;ssl=1" class="thumbnail" width="690" height="388"></div>

<h3><a href="https://thecleverprogrammer.com/2020/07/22/face-landmarks-detection/" target="_blank" rel="noopener nofollow ugc">Face Landmarks Detection</a></h3>

<p>In this article I will guide you how you can detect face Landmarks with Machine Learning. Have you ever thought how Snapchat manage to apply amazing filters</p>

  <p><span class="label1">Est. reading time: 7 minutes</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Recurrent dropout in LSTM

kontrabas: <p>Hello!</p>
<p>I’ve found something called “recurrent dropout” in the model which I’d like to implement. But I cannot find it in pytorch documentation (and I’ve found it in tensorflow documentation). Is there something like this in pytorch?</p>
<p>Best regards</p>

Abhilash_Srivastava: <p>I don’t think Pytorch supports recurrent dropout of the shelf. See if <a href="https://discuss.pytorch.org/t/implementing-recurrent-dropout/5343">this</a> is helpful.</p>
ValueError: Using a target size (torch.Size([5, 1])) that is different to the input size (torch.Size([5])) is deprecated. Please ensure they have the same size

best.travel_ever: <p>I am confused now having this error:</p>
<p>ValueError: Using a target size (torch.Size([5, 1])) that is different to the input size (torch.Size([5])) is deprecated. Please ensure they have the same size.</p>
<p>As some days ago this code run with no error at all. How is it possible?</p>
<p>The error happens in the following lines when I try to train my GAN</p>
<pre><code class="lang-auto">       # Train the Generator
        model.zero_grad()
        errG = criterion(fake, hr_img)

        out_Dis_G_1 = Dis(fake)
        print(out_Dis_G_1.shape)
        labels1 = torch.full((bsize, 1), 1, dtype=torch.float, device='cuda')
        **e_g = loss_clf(out_Dis_G_1, labels1)**
</code></pre>
<p>Although, when I have the similar lines above but about D model, there is no problem</p>
<pre><code class="lang-auto">labels0 = torch.full((bsize,1), 0, dtype=torch.float, device='cuda')

errDis_fake = loss_clf(out_Dis_G, labels0)
</code></pre>

BaruchG: <p>What are the shapes of <code>out_Dis_G_1</code> and <code>labels1</code>?  Also, what line is the error pointing to?</p>

best.travel_ever: <p>out_Dis_G_1 = torch.Size([5])<br>
labels1 = torch.Size([5, 1])</p>
<p>eroor is here:  e_g = loss_clf(out_Dis_G_1, labels1), loss_clf is nn.BCELoss()</p>

BaruchG: <p>And what happens if you make them the same shape using <code>squeeze</code> or <code>unsqueeze</code> ?</p>

best.travel_ever: <p>Yeah, I think this trick works !<br>
Thank you a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to simultaneously train two different models on two different GPUs, using the same data?

GustavoVargasHakim: <p>Hi there!</p>
<p>This is a question that might sound simple, but I can’t find any information that actually works when I implement it.</p>
<p>So, I have to models (net1 and net2), and I have two GPUs (“cuda:0” and “cuda:1”). I have the training and test dataloaders (train_dl, test_dl). I just want to train both models simultaneously, like:</p>
<p>net1.to(decive1)<br>
net2.to(device2)</p>
<p>accuracy1 = training(net1)<br>
accuracy2 = training(net2)</p>
<p>How do I do this? How do I make sure these last two functions don’t execute sequentially? I want them to execute at the same time. I have read about something related to “spawn” but I have really no idea how to implement that.</p>
<p>Thank you!</p>

agolynski: <p>Hi,</p>
<p>If net1 and net2 do not depend on each other you can just spawn 2 processes and train each model in its own process. You can do</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp

def your_func(int rank):
   if rank == 0: 
      train net1
   else:
      train net2
...
    mp.spawn(
        your_func,
        args=(),
        nprocs=2,
        join=True)
</code></pre>
<p>Would that work for you?</p>

GustavoVargasHakim: <p>Thank you very much! This is exactly what I needed. I have only one more question:</p>
<p>If I wanted to return something from my function (such as the accuracy value), what would be the best way to do so? I was thinking about having a global list a updating that list inside the function, but this might be a little bit odd probably.</p>
<p>Thanks again!!</p>

rvarm1: <p>A global list may be tricky since each process would have its own copy of the global list. Overall you would need to communicate the accuracy across processes to aggregate them, you can look into python multiprocessing such as <code>mp.Manager()</code>: <a href="https://docs.python.org/3/library/multiprocessing.html" rel="noopener nofollow ugc">https://docs.python.org/3/library/multiprocessing.html</a>.</p>
Replace elements of tensor with variable number of zeros to along a certain dimension

PyScorch: <p>Suppose there is an array A = tensor([[0.4869, 0.5144, 0.9086, 0.6139],<br>
[0.5103, 0.8270, 0.4832, 0.8980],<br>
[0.5234, 0.1135, 0.1037, 0.7451]])</p>
<p>And I want to replace the elements in each row with zeros, depending on another tensor t = tensor([0, 1, 3])</p>
<p>The output should be like out = tensor([[0.4869, 0.5144, 0.9086, 0.6139],<br>
[0, 0.8270, 0.4832, 0.8980],<br>
[0, 0, 0, 0.7451]])</p>
<p>I have already tried an implementation that uses the torch.gather function but that operation seems to consume a lot of memory and it runs into memory overflow when dealing with huge tensors.</p>

Nikronic: <p>Hi,</p>
<p>I think this thread can help you with this:</p><aside class="quote quote-modified" data-post="2" data-topic="102097">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/set-value-of-torch-tensor-up-to-some-index/102097/2">Set value of torch tensor up to some index</a> <a class="badge-category__wrapper " href="/c/nlp/8"><span data-category-id="8" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to Natural Language Processing"><span class="badge-category__name">nlp</span></span></a>
  </div>
  <blockquote>
    Hi, 
You can do this by creating binary mask. 
First we create a matrix with same shape as X filled with zeros, then put 1s where index matches with ind tensor and finally by using cumsum, set 1 after previously located points. Finally, we can mask X: 
X = torch.tensor(
    [[1,0,4,5,6],
     [3,6,7,10, 13],
     [1,4,2,8,21]])


ind = torch.tensor([2,3, 1])


mask = torch.zeros_like(X)
mask[(torch.arange(X.shape[0]), ind)] = 1
mask = 1 - mask.cumsum(dim=-1)
mask
# mask
# tensor([[1, 1, 0, 0, 0]…
  </blockquote>
</aside>

<p>But I have not tested it on large tensors.</p>
<p>Bests</p>
Store low bitwidth models

AfonsoSalgadoSousa: <p>Hello. I am struggling to assess the storage memory footprint of quantized models. I would like to compare the storage savings of using quantization. We theoretically can manage to know this, but I would also like to save my model for deployment.<br>
So, basically, my question is, how can I store a Pytorch quantized model (not quantized using the built-in quantization methods) encoded as to minimize its size in accordance with the bitwidth used?</p>
Threshold_mode in ReduceLROnPlateau

aroibu1: <p>Hi all. For my project, I want to use the <code>torch.optim.lr_scheduler.ReduceLROnPlateau</code> learning rate scheduler to adjust my learning rate based on my validation loss. Looking at the documentation, however, it is not very clear to me what the <strong>threshold_mode</strong> parameter does, and which version of it, <em>rel</em> or <em>abs</em> is best to use?</p>
<p>The <em>rel</em> parameter is the default one, however, doing some quick calculations with pen and paper it appears that this version might lead to some very small margins when comparing validation losses between two iterations, much smaller than the threshold defined by me for measuring the new optimum.</p>
<p>Can anybody please advise on this? Cheers!</p>
Slicing 2D tensor using list of start/end indices

marianlonga: <p>Given a 2D tensor x and 2D tensors of start and end indices, how can I get a tensor with the slices of x? I am looking for a vectorized function that can do this:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.arange(0, 20).view(4, 5)
tensor([[ 0,  1,  2,  3,  4],
        [ 5,  6,  7,  8,  9],
        [10, 11, 12, 13, 14],
        [15, 16, 17, 18, 19]])
&gt;&gt;&gt; starts = torch.tensor([[0,0], [0,3], [2,1]])
tensor([[0,0], [0,3], [2,1]])
&gt;&gt;&gt; ends = torch.tensor([[0,2], [0,5], [2,5]])
tensor([[0,2], [0,5], [2,5]])
&gt;&gt;&gt; slices = some_function(x, starts, ends)
tensor([[0, 1], [3, 4],
           [],
           [11, 12, 13, 14],
           []])
</code></pre>
<p>starts and ends have the same length and each slice given by the pair start[i], end[i] only runs over the same zeroth dimension. Any help would be appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

klory: <p>First, your output is not a tensor because each row don’t have the same lenght, if they have the same length, you can refer <a href="https://stackoverflow.com/questions/46091111/python-slice-array-at-different-position-on-every-row" rel="noopener nofollow ugc">here</a>, otherwise I’m afraid it has to be a loop to solve it</p>
Setting Parameters of DistributedDataParallel

Shen: <p>Hi, I’m trying to use <code>DistributedDataParallel</code> to enable training on multiple GPUs on a single node. According to what I read from online documents, the procedure is the following (please correct me if it’s not true):</p>
<pre><code class="lang-auto">dist.init_process_group(backend=backend, init_method=init_method, rank=rank, world_size=world_size)
model = DistributedDataParallel(model, device_ids=[local_rank])
</code></pre>
<p>And then spawn the process in the main function:</p>
<pre><code class="lang-auto">multiprocessing.spawn(fn, args=args, nprocs=world_size, join=True)
</code></pre>
<p>I have two questions regarding this:</p>
<ul>
<li>
<p>What should I set these parameters to? e.g., <code>backend</code>, <code>init_method</code>, <code>rank</code>, <code>world_size</code>… I didn’t find an example showing details of setting these parameters.</p>
</li>
<li>
<p>I saw some examples online used <code>DistributedSampler</code>. Is this necessary for using <code>DistributedDataParallel</code>?</p>
</li>
</ul>
<p>Thanks!!</p>

agolynski: <p>Hi,</p>
<p>You can take a look at <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html</a></p>
<p>Your params: backend is a type of communication between nodes (depends on your hardware and setup), e.g. backend = ‘nccl’; local rank is given to your function (fn) by multiprocessing.spawn(); rank is computed based on node rank and local rank; world_size is how many ranks you want to run in total (typically = number of nodes * number of gpus in a node).</p>
<p>DistributedSampler is useful for loading data across the ranks you’ve created by multiprocessing.spawn(), so they don’t share the same pieces of data in training.</p>

Shen: <p>Hi, thanks for your reply! I’m still a bit confused about the concept <code>rank</code>. Does it correspond to the number of gpus available?</p>
<p>For example, if I’m using 1 node with 4 gpus to train, how should I set those parameters?</p>

agolynski: <p>you can think of rank as an ID of a process controlling one (preferred) or multiple GPUs.<br>
local rank is a local ID of such process.</p>
<p>if you have 1 node with 4GPUs we prefer that you use DDP with 4 ranks (processes)</p>
Change of 3D feature-map/ image orientation after call F.grid_sample with identity transformation

Sureerat_Reaungamorn: <p>Hi</p>
<p>I would like to use <code>torch.nn.functional.grid_sample</code> to resample a 3D image volume, so I tested it with identity displacements (see the code below). This should not spatially transform the image; however, it did re-orient  the image. The attached figure shows the image after resampling using <code>F.grid_sample</code> on the first row. The second row shows the original image.</p>
<p>Would there be something that I missed or I did wrong? Any suggestions on how to fix this.</p>
<pre><code class="lang-auto">def coordinates_map(size, start=-1.,end=1.):
    '''This is a function to create a 3D grid '''
    batch_size, channels = size[:2]
    d,h,w = size[2:]

    w_p = np.linspace(start, end, num=w)
    h_p = np.linspace(start, end, num=h)
    d_p = np.linspace(start, end, num=d)
    coords = np.stack(np.meshgrid(h_p,d_p,w_p, indexing='xy'), axis=-1)[np.newaxis,...]

    return torch.from_numpy(coords).float().expand((batch_size,)+coords.shape[1:])

image = torch.rand((1, 1, 48, 65, 64)) # I used a real image not random tensor
grid = coordinates_map(size=image.shape)
deformed_image = F.grid_sample(x,grid) 
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/8/789679859fa463bba8177cc79c97853b39dc807c.png" data-download-href="https://discuss.pytorch.org/uploads/default/789679859fa463bba8177cc79c97853b39dc807c" title="pytorch_bug"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/8/789679859fa463bba8177cc79c97853b39dc807c.png" alt="pytorch_bug" data-base62-sha1="hcLGAPGxURDmu61RpzyVClnGoLW" width="690" height="452" data-dominant-color="222224"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch_bug</span><span class="informations">902×592 61.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

gkuling: <p>Hey Sureerat, Not sure if you’ve already solved this, but I was just working on the exact same problem today. There is just one more step at the end that you could implement. The grid sample changes the orientation of the scan that you input so you use <code>.permute()</code> on your output tensor to reorient it.</p>
<p>I found to get the proper resampling I had to rearrange just the last three dimensions</p>
<pre><code class="lang-auto">&gt;d1 = torch.linspace(-1, 1, self.shape[2])
&gt; d2 = torch.linspace(-1, 1, self.shape[1])
&gt; d3 = torch.linspace(-1, 1, self.shape[0])
&gt; meshx, meshy, meshz = torch.meshgrid((d1, d2, d3))
&gt; grid = torch.stack((meshx, meshy, meshz), 3)
&gt; grid = grid.unsqueeze(0)  # add batch dim
&gt; out = torch.nn.functional.grid_sample(x, grid)
&gt; out = out.permute(0,1,4,3,2)
</code></pre>
What does "#type: " mean?

five_lu: <pre><code class="lang-auto">    def max(self, list_in):
        # type: (List[List[int]]) -&gt; List[int]
        maxes = the_list[0]
        for sublist in list_in[1:]:
            for index, item in enumerate(sublist):
                maxes[index] = max(maxes[index], item)
        return maxes
</code></pre>
<p>In this func ,  if i dont follow the rule (input dtype is List[List[int]] &amp; output type is List[int]),but nothing happened … so  what’s the meaning of “# type”?</p>

klory: <p>it’s just a comment to make you understand what are the types of the input and output</p>
Models loaded on on Windows are different to that loaded on Linux

jolinxql: <p>I found that when I load and evaluate models trained and saved on windows use different operating systems, results are <strong>inconsistent</strong>.</p>
<p><strong>Phenomenon:</strong><br>
I trained a model on Windows 10 and tested it on the test set before saving and got <code>accuracy = 63.91%</code>. Then I saved it to “model_epoN”.</p>
<ol>
<li>When I load and evaluate it on the same Windows 10, the results are <strong>consistent</strong>. i.e., <code>accuracy = 63.91%</code>, too.</li>
<li>But when I load and evaluate it on Ubuntu, the results are <strong>inconsistent</strong>. <code>accuracy =0.1%</code>, like randomly guessing.</li>
</ol>
<p>I’ve noticed the difference between CR LF and LF line break types and made small changes to the datasets, so I’m sure the problem lies somewhere else.</p>
<p><strong>Version information:</strong><br>
The pytorch version on Windows 10 python3.6 is <code>0.3.0b0+591e73e</code> (peterjc123).<br>
The pytorch version on Ubuntu 16.4 python3.5 is <code>0.3.0.post4</code> (official).</p>
<p><strong>Save function:</strong></p>
<pre><code class="lang-auto">def save_checkpoint(parser, epoch):
    torch.save({'state_dict': parser.state_dict()},
    'model_epo' + str(epoch + 1))
</code></pre>
<p><strong>Load function:</strong></p>
<pre><code class="lang-auto">def load_checkpoint(filename, parser):
    checkpoint = torch.load(filename)
    parser.load_state_dict(checkpoint['state_dict'])
    return parser
</code></pre>
<p>No error or warning was raised. I guess the formats for saving state_dict are different according to the OS?<br>
Currently, I’m not able to write a minimal working example because the model is too complicated and it builds dynamic nets in each iteration. I haven’t trained the model on Linux and test it on Win. But I will try and I still need some time.<br>
I hope the problem can be solved in pytorch v0.4.0.</p>

EmreTokyuez: <p>Did you find a workaround? I have the same problem as well and need to load my model on linux to dockerize.</p>