akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How can I push a module into a sequential?

Mabinogiysk: <p>Is there any example about pushing a module into a sequential? I have been stuck here two days.<br>
“”"<br>
torch::nn::Sequential layer1{ nullptr };<br>
auto conv1 = torch::nn::Conv2d(torch::nn::Conv2dOptions(64, 64, 1).stride(1).bias(false));<br>
layer1-&gt;push_back(conv1);<br>
“”"<br>
Then I got error: Accessing empty ModuleHolder<br>
I have tried push_back(&amp;conv1),push_back(std::move(conv1)), none of them worked and I cannot find any example on internet, is anyone can help?</p>

Mabinogiysk: <p>I have solved it, layer1 was initiated with null in my code.</p>

Lin_Jia: <p>For future reference,<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html</a></p>
<h1>Example of using Sequential</h1>
<p>model = nn.Sequential(<br>
nn.Conv2d(1,20,5),<br>
nn.ReLU(),<br>
nn.Conv2d(20,64,5),<br>
nn.ReLU()<br>
)</p>
<h1>Example of using Sequential with OrderedDict</h1>
<p>model = nn.Sequential(OrderedDict([<br>
(‘conv1’, nn.Conv2d(1,20,5)),<br>
(‘relu1’, nn.ReLU()),<br>
(‘conv2’, nn.Conv2d(20,64,5)),<br>
(‘relu2’, nn.ReLU())<br>
]))</p>
How to implement distributed model parallel using torch.distributed

xdwang0726: <p>I am trying to implement model parallel using torch.distributed(DistributedDataParallel), and I am wondering is there a tutorial for that (single node multiple GPUs)? I know nn.DataParallel is easier to use, however, I use another package that only support torch.distributed. Thanks!</p>

mrshenli: <p>Yep, here is the tutorial: <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#combine-ddp-with-model-parallelism" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#combine-ddp-with-model-parallelism</a></p>
<p>It should work if you place different layers of the model on different GPUs and pass it to DDP. DDP should be able to detect it is a multi-GPU model. One caveat is that, please make sure no GPUs are shared across processes.</p>

xdwang0726: <p>I followed the tutorial online, however I got the error message <code>RuntimeError: Model replicas must have an equal number of parameters.</code> in <code>model = torch.nn.parallel.DistributedDataParallel(model)</code><br>
Any idea what might cause this issue? Thanks!</p>

pritamdamania87: <p><a class="mention" href="/u/xdwang0726">@xdwang0726</a> Do you see this error when using the code in the tutorial? Or is it some custom code based on the tutorial? If it is the latter, could you provide a minimal repro of the issue?</p>
SyncBatchNorm across multiple batches of the SAME gpu - is it possible?

Alon: <h3>Fake Distributed on 1 GPU</h3>
<p>I have big samples, so I can’t use a big batch size. I virtually increase the batch size simply by calling the <code>optimizer.step()</code> every N batches. However, that of course doesn’t help the statistics of BatchNorm that are calculated per batch, and suffer from that. There is only so much I can do with the batchnorm momentum… I would like to simulate a distributed system on 1 GPU and sync the BN layers across multiple fake-parallel batches.<br>
Is that possible?</p>

pritamdamania87: <p>Is using multiple GPUs to increase the effective batch size not an option for you?</p>
Libtorch, how to add a new optimizer

freezek: <p>For test, I copy the file “adam.h” and “adam.cpp”, and change all Related keyword “Adam” to “MyAdam”, and include “adam.h” in “optim.h”. After compiling, when I use “MyAdam” in new code, the compiler aborted undefined symbols:</p>
<pre><code class="lang-auto">Undefined symbols for architecture x86_64:
  "torch::optim::MyAdamOptions::MyAdamOptions(double)", referenced from:
      _main in Net_test.cc.o
  "torch::optim::MyAdam::step(std::__1::function&lt;at::Tensor ()&gt;)", referenced from:
      _main in Net_test.cc.o
  "vtable for torch::optim::MyAdamOptions", referenced from:
      torch::optim::MyAdamOptions::MyAdamOptions(torch::optim::MyAdamOptions const&amp;) in Net_test.cc.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for torch::optim::MyAdam", referenced from:
      torch::optim::MyAdam::MyAdam(std::__1::vector&lt;torch::optim::OptimizerParamGroup, std::__1::allocator&lt;torch::optim::OptimizerParamGroup&gt; &gt;, torch::optim::MyAdamOptions) in Net_test.cc.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
ld: symbol(s) not found for architecture x86_64
</code></pre>
<p>Is there something I miss ?</p>

freezek: <p>For example:</p>
<pre><code class="lang-auto">    auto opt = torch::optim::MyAdam(param);
    auto options = static_cast&lt;torch::optim::MyAdamOptions&amp;&gt;(opt.defaults());
</code></pre>

Lin_Jia: <p><a class="mention" href="/u/freezek">@freezek</a>,  the implementation for certain libtorch classes are not strictly contained in single cpp file. You might need to copy all implementation files to complete MyAdam. If you want to do this kind of copying, you have to search the files realted to Adam in the source code level (directly downloaded from pytorch github codebase), instead of searching Adam implementation in distributed libtorch version, cause distributed version already compiles some Adam class implementation, and you cannot access those.</p>

freezek: <p>hi, <a class="mention" href="/u/lin_jia">@Lin_Jia</a>, thanks for reply. I find add <code>MyAdam.cpp</code> path into <code>caffe2/CMakeLists.txt</code> works .</p>
One of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [768, 64]] is at version 3; expected version 2 instead

Amy_Lapi: <p>Hi! I’m pretty new to PyTorch and I’ve come across this error in my network:</p>
<p><code>one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [768, 64]] is at version 3; expected version 2 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</code></p>
<p>I have already read several forums on this topic. They all generally seem to say that there is an in place operation someplace in the code that is messing up the backward operation, but I can’t seem to find any in place operations.</p>
<p>Here is my learn function where I call backward:</p>
<pre><code class="lang-auto">#updating the q-network. IE, learning from past experiences
def learn(batch, optim, optimGru,optimLilGru, net, target_net, gamma, global_step, target_update):
    #separate batch
    s,a,s_,r,d,a_ = batch #in addtion to the batch I will need to return the next action for the next state
    #zero the grad
    optim.zero_grad()
    optimLilGru.zero_grad()
    optimGru.zero_grad()
    
    ##calculate loss
    x = torch.tensor(s_, dtype = torch.float).cuda()

    Q_hat_mean = target_net.forward(x).mean(axis=1)[0] #We don't want the max, we want something different. I picked me#an() since it isn't the max and would likely give amiddle range result
    true = r + gamma * (1-d) * Q_hat_mean
    
    est2 = net(s)[range(len(a_)),a_] 

    loss = torch.mean((true-est2)**2) #found the MSE
    loss.backward()
    optim.step()
    optimGru.step()
    optimLilGru.step()
    torch.autograd.set_detect_anomaly(True)

    if global_step % target_update == 0:
        target_net.load_state_dict(net.state_dict())

    return loss.item()
</code></pre>
<p>I am using two GRU and a DQN in this code. If it is helpful, I can post more code snippets, but I’m not sure which parts will be useful in solving the problem.</p>
<p>I’m looking for a way to resolve the error, but I am also very interested in understanding more about what this error means and how I can avoid it in future projects I work on. Any advice on either of those topics will be greatly appreciated.</p>
<p>Thanks!</p>
Confusion about GEMM in Pytorch

Shen: <p>Hi all,<br>
I recently encountered the word GEMM. I’m a bit confused about the usage of GEMM in Pytorch: how does it differ from the normal matrix-matrix multiplication? For example, I’ve read something about turning the convolution to a matrix multiplication, i.e., unfold + GEMM + reshape procedure. I’m wondering how is the GEMM implemented in Pytorch.</p>
<p>Suppose I have a Conv layer, if I first unfold the input to a 3D matrix (with the 1st dimension to be batch dimension), and reshape the weight to a 2D matrix, then perform a multiplication</p>
<pre><code class="lang-auto">conv_mat = torch.matmul(input_unfold, weight_reshape)
</code></pre>
<p>Finally reshape <code>conv_mat</code> back to the shape of the convolution output. Will this be regarded as using GEMM? If not, how to get GEMM involved?<br>
Thanks!</p>

ptrblck: <p>GEMM (general matrix multiplication) is used as a synonym for “matrix multiplication” and comes from the “BLAS world” if I’m not mistaken.<br>
Have you seen GEMM in the docs somewhere? If so, I think we should replace it.</p>

Shen: <p>Thanks for your reply. No, I didn’t see GEMM in Pytorch docs. But I saw there are some matrix multiplication functions in Pytorch for BLAS operations, such as <code>mm</code>, <code>bmm</code> and etc. It seems they are dealing with matrix multiplications with different dimensions (2D/3D). I have two questions:</p>
<ol>
<li>From your description, <code>matmul</code> is also a kind of GEMM, right? Since it is performing matrix multiplication.</li>
<li>How does <code>matmul</code> differ from functions for BLAS operations (e.g., <code>mm</code>, <code>bmm</code>) in terms of the performance? Does BLAS function have a faster implementation for matrix multiplication?</li>
</ol>

ptrblck: <ol>
<li>
<p>Yes, these are just different names from different domains. Maybe similar to image processing filters and convolution kernels.</p>
</li>
<li>
<p><code>matmul</code> calls into different implementations as seen <a href="https://github.com/pytorch/pytorch/blob/a153eafab752cdd9762dc4cb5b4fb4f78f255e8a/aten/src/ATen/native/LinearAlgebra.cpp#L567-L695">here</a>.</p>
</li>
</ol>

Shen: <p>Thanks for your help!</p>
PyAudio tutorials

danielius: <p>Hello,</p>
<p>What is up with the audio tutorials?</p>
<p>When trying to search for pytorch audio quickstart-ish code only found <code>https://pytorch.org/tutorials/beginner/audio_classifier_tutorial.html</code> which seems to be old and missing code.</p>
<p>The one tutorial that is in pytorch/tutorials is <code>https://pytorch.org/tutorials/beginner/audio_preprocessing_tutorial.html</code> which only shows file loading and few transforms.</p>
<p>Is there a way one could contribute on writing those?</p>

ptrblck: <p>Contributions are always welcome!<br>
Could you open an issue <a href="https://github.com/pytorch/tutorials/issues">here</a> and describe what kind of tutorial you would like to write to get some feedback? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

danielius: <p>Will do. Initially started to write and issue and its template got me here.</p>

Steve_B: <p>Has there been any progress on this front? This would be most helpful if so.</p>
Reproducibility in multiagent environments

Kimonili: <p>I need to create a reproducable code for PPO algorithm but in a multiagent environment. When I applied the same algorithm in a single agent setting the reproducibility was succesfull.</p>
<p>I think the problems stands in the initialization of the actor and critic networks.</p>
<p>I  create the networks using the code snippet bellow, similar code snippet for the actor networks as well.</p>
<pre><code class="lang-auto">            for _ in range(len(self.env.agents)):
                torch.manual_seed(self.seed)
                critic = Critic(sum(obs_size), self.hidden_size).to(self.device)
                self.critics.append(critic)
</code></pre>
<p>Also, the environment is initialized using a seed as well, as shown bellow:</p>
<pre><code class="lang-auto">def init_env(gym_id: str, seed: int):
    env = make_env(gym_id, discrete_action=True)
    env.seed(seed)
    np.random.seed(seed)
    return env, seed
</code></pre>
<p>Additionally, the buffer from which the agents update their policy is also set to a specific seed, as shown bellow:</p>
<pre><code class="lang-auto">ids = np.arange(self.trajectory_size)
        for agent, _ in enumerate(self.env.agents):
            for epoch in range(self.epochs):
                np.random.seed(self.seed)
                np.random.shuffle(ids)
</code></pre>
<p>Is there anything else that is sensitive to randomness (and has to be set to a seed) in the PPO algorithm that I cannot see?</p>
How do I change it with c++ code?

sangwo: <p>image = torch.from_numpy(image.transpose(2, 0, 1)).unsqueeze(0).float()</p>

ptrblck: <p><code>image.transpose</code> with more than 2 dims is a numpy method and thus not usable in libtorch.<br>
You would have to either transpose two dimensions (using the same syntax as in Python) or use <code>permute</code> instead (<code>tensor.oermute({2, 0, 1})</code>).<br>
<code>torch.from_numpy</code> is not available in libtorch, if I’m not mistaken.</p>

Lin_Jia: <p>unsqueeze(0) is available in C++, and you can just directly write it as tensor.unsqueeze(0)</p>
Batch norm mix precision jit bug

Fangzou_Liao: <p>batch norm can receive half tensor and also return half tensor in normal cases,</p>
<p>but after tracing, and under torch.no_grad(), it return float instead of half (see img)</p>
<pre><code class="lang-auto">bn_layer = torch.nn.BatchNorm2d(10).cuda().float()
x = torch.rand(1,10,10,10).cuda().half()
o1 = bn_layer(x)
print(o1.dtype)

with torch.no_grad():
    trace = torch.jit.trace(bn_layer, torch.rand(1,10,10,10).cuda().half())
    
with torch.no_grad():
    o2 = trace(x)
print(o2.dtype)

o3 = trace(x)
print(o3.dtype)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/d/dd806f07ae1b4be0405d23cc164828bfc1854882.png" data-download-href="https://discuss.pytorch.org/uploads/default/dd806f07ae1b4be0405d23cc164828bfc1854882" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd806f07ae1b4be0405d23cc164828bfc1854882_2_690x317.png" alt="image" data-base62-sha1="vBuGIek5hiMAtnKbjjxaEnmx5Rw" width="690" height="317" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd806f07ae1b4be0405d23cc164828bfc1854882_2_690x317.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd806f07ae1b4be0405d23cc164828bfc1854882_2_1035x475.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/dd806f07ae1b4be0405d23cc164828bfc1854882_2_1380x634.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1624×748 73.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Multiple torch.save torch.load in one file fails with 1.6.0

antigol: <p>Dear all,</p>
<p>The following code was working fine with pytorch 1.5.0 and is now failing:</p>
<pre><code class="lang-auto">with open('test.pkl', 'wb') as f:
    torch.save(4, f)
    torch.save(5, f)

with open('test.pkl', 'rb') as f:
    assert torch.load(f) == 4
    assert torch.load(f) == 5
</code></pre>
<p>I get the following error</p>
<pre><code class="lang-auto">~/anaconda3/envs/env38/lib/python3.8/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    583                     return torch.jit.load(opened_file)
    584                 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
--&gt; 585         return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
    586 
    587 

~/anaconda3/envs/env38/lib/python3.8/site-packages/torch/serialization.py in _legacy_load(f, map_location, pickle_module, **pickle_load_args)
    753             "functionality.".format(type(f)))
    754 
--&gt; 755     magic_number = pickle_module.load(f, **pickle_load_args)
    756     if magic_number != MAGIC_NUMBER:
    757         raise RuntimeError("Invalid magic number; corrupt file?")

UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.
</code></pre>

albanD: <p>Hi,</p>
<p>Thanks for reporting this. It does look like a regression. Could you open an issue on github please?</p>

antigol: <p>OK! Done here <a href="https://github.com/pytorch/pytorch/issues/45136" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/45136</a></p>
Differentiable Sign or Step Like Function

HaydenW: <p>Hi, I’m very new to PyTorch and I have been trying to extend an autograd function that tunes multiple thresholds to return a binary output and optimize using BCELoss, but I’ve been struggling with the fact that any sign or step function I apply always returns a gradient of 0. In some instances I’ve been able to get it to work with ReLu and Trigonometric functions; however, it then returns the exact same (very small) gradient for every threshold. The framework of the sign function follows this idea: Output = Sign(Sum(Sign(Sign(X - Threshold)*-1 + 1))). If I wanted to apply an Or function with multiple thresholds, I would then apply this function Sign(Output1 + Output2). I would like to define the optimal threshold cut-off point for separate arrays and then combine them through And/Or logic. Is there any operation in PyTorch that works in autograd to do so?</p>

albanD: <p>Hi,</p>
<p>The problem with the sign function is that it return 1 or -1 and so is piecewise constant. So the gradient of the function almost everywhere is 0. So it is expected that you get a gradient of 0 if you use that function.</p>
<p>In general, if your loss is piecewise constant, you won’t be able to optimize it with gradient-based optimizers as all gradients are always 0 I’m afraid.</p>

HaydenW: <p>That makes sense why I’ve always returned a gradient of 0, I’ve done some reading on <a href="https://arxiv.org/pdf/1602.02830.pdf" rel="nofollow noopener">Binarized Neural Networks</a>. They overcome this issue through a quantized sign function and a hard tanh gradient. I’ve looked through various GitHub implementations of the suggested method; however, I can’t seem to figure out how to apply this to my threshold problem without returning a 0 gradient again. Is this a possible solution to my 0 gradient problem in autograd? Or should i try to approximate a probability using the hyperbolic tangent atan, which mostly returns a gradient larger than 0?</p>

albanD: <p>Hi,</p>
<p>You can either design a smoothed version of your loss function that will have non zero gradients.</p>
<p>Or you can use this piecewise constant function but design a backward to compute something that is not the true gradient but will point in the right direction using a <a href="https://pytorch.org/docs/stable/notes/extending.html">custom autograd.Function</a>.</p>
Speeding up epochs for test runs

mcfn: <p>say i added a code that runs every epoch and i want to test/debug it. epoch takes long to run so i wanna shorten it. i am willing to give up “actual” learning since this is only a test run.<br>
i thought of taking a subset of my data but it’s breaking a couple of places in my code so i’d rather not.<br>
i thought of skipping the backprop phase (loss.backwards()) but somehow it take up extra space exceeding my gpu’s limit.<br>
what other options do i have?</p>
Sampling from a tensor in Torch

ariG23498: <p>I wanted to know whether we could sample elements of a tensor given a probability distribution of the tensor.<br>
In numpy we would do something like this:</p>
<pre><code class="lang-auto">a = np.array([1,2,3,4])
b = np.random.choice(a, p=np.array([0.1, 0.1, 0.1, 0.7]))
</code></pre>
<p>In torch I would like to have the array <code>a</code> and <code>p</code> to be of <code>torch.tensor</code>. The numpy function works well with CPU torch tensors translated to numpy arrays, but unfortunately, using GPUs the trick fails.</p>

Sayak_Paul: <p>Might be helpful? <a href="https://discuss.pytorch.org/t/torch-equivalent-of-numpy-random-choice/16146" class="inline-onebox">Torch equivalent of numpy.random.choice?</a></p>

ariG23498: <p>Have seen, did not do the trick.</p>

phan_phan: <p>Hi !<br>
If you want to do :</p>
<pre><code class="lang-python">a_numpy = np.array([1, 2, 3, 4])
p_numpy = np.array([0.1, 0.1, 0.1, 0.7])
b_numpy = np.random.choice(a_numpy, p=p_numpy)
</code></pre>
<p>You can use <a href="https://pytorch.org/docs/stable/generated/torch.multinomial.html#torch.multinomial" rel="nofollow noopener">torch.multinomial</a> :</p>
<pre><code class="lang-auto">a = torch.tensor([1, 2, 3, 4])
p = torch.tensor([0.1, 0.1, 0.1, 0.7])
index = p.multinomial(num_samples=1, replacement=True)
b = a[index]
</code></pre>
<p>Here I used <code>replacement=True</code> because it is the default behavior of <code>numpy.choice</code>, even though is makes no difference when <code>num_samples=1</code></p>
.cpu().detach().numpy() vs .data.cpu().numpy()

alwynmathew: <p>What is the difference between <code>.cpu().detach().numpy()</code> and <code>.data.cpu().numpy()</code> ?</p>
<p>When should I used one of these over the other?</p>

albanD: <p><code>.data</code> is an old api and should not be used anymore.<br>
So the first one is the right way to go.</p>

pytorching: <p>How about <code>.cpu().detach()</code> vs <code>.detach().cpu()</code> ?</p>

albanD: <p>The end result is the same.<br>
The second one is going to be imperceptibly faster because you don’t track the gradients for the <code>cpu()</code> op. But nothing else.</p>

pytorching: <p>Thank you very much. That answers my question exactly.<br>
BTW, I don’t know if it’s allowed or forbid. May I ask you to look at another question I post?</p><aside class="quote quote-modified" data-post="1" data-topic="97082">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/85f322/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/nn-dataparallel-typeerror-expected-sequence-object-with-len-0-or-a-single-integer/97082">nn.DataParallel: TypeError: expected sequence object with len &gt;= 0 or a single integer</a> 
  </div>
  <blockquote>
    In my forward function: 
def __call__(self, train=True):
    if train:
        predicted = self.forward(...)
        loss = ....
        return loss # return a single value that's fine
        # loss.size() = the number of my GPUs.
    else:
        predicted = self.forward(...)
        return predicted # expected sequence object with len &gt;= 0 or a single integer
        # In validation step I want to return the whole predict labels for other purpose
        # predicted with shape [16, 1] on eac…
  </blockquote>
</aside>
Input for conv3d

arjun_pukale: <p>I have video data where each video is in the form of <code>(30,224,224)</code> where 30 is the number of frames and each frames are grayscaled and have size of <code>(224,224)</code> .<br>
I want to use conv3d but the conv3d input is of the form <code>(N,Cin​,D,H,W)</code>.<br>
here H,W = <code>224,224</code><br>
N is the batch size.<br>
what does Cin and D represent? and what must be their values in my case?</p>

mariosasko: <p>Cin is the number of channels in the image/frame (I assume 1 if they are grayscaled), D is the frame dimension (this dimension considers relationship among the frames in the input).</p>
<p>So your input should have the shape (N, 1, 30, 224, 224).</p>
Loss.backward() failure after pruning

TTyRAeLL: <p>I prune a model by setting a certain channel to zero, and then the  <code>loss.backward()</code>  function isn’t working. It may due to the pruned model can’t be derivated. How can I use BP regardless of the pruned channel?</p>

tom: <p>If you do this for an intermediate, you should multiply with a mask rather than assigning to the value. Quite likely the operation that generated the non-pruned output wants it to compute the backwards. (Search for “inplace” in the forums.)</p>
<p>Best regards</p>
<p>Thomas</p>

TTyRAeLL: <p>Thanks for help, I referred to the inplace operation and solved my problem.</p>
Values randn in pytorch

SAI_VARSHITTHA: <pre><code class="lang-auto">x = torch.randn(1)
print(x)
print(x.item())
</code></pre>
<p>The output of the above code is</p>
<pre><code class="lang-auto">tensor([0.3369])
0.3368943929672241  # this is python number
</code></pre>
<p>When doing operations on x , what values will be used for the values iniside x? the rounted value( 0.3369) will be used or the number (0.3368943929672241 ) is used?</p>

ptrblck: <p>The complete <code>float32</code> number will be used. The <code>print(x)</code> uses the default print options, which truncate the output.<br>
You can increase the print precision via: <code>torch.set_printoptions(precision=10)</code> (or to any other value).</p>
RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.cuda.FloatTensor) should be the same

Feywell: <p>I use torch.cuda.amp.autocast before model forward, But I get a error like topics.<br>
The code is :</p>
<pre><code class="lang-auto">class SuperConv2d(nn.Conv2d):
    def __init__(self, in_channels, out_channels, kernel_size, stride=1,
                 padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros'):
        super(SuperConv2d, self).__init__(in_channels, out_channels, kernel_size,
                                          stride, padding, dilation, groups, bias, padding_mode)

    def forward(self, x, config):
        in_nc = x.size(1)
        out_nc = config['channel']
        weight = self.weight[:out_nc, :in_nc]  # [oc, ic, H, W]
        if self.bias is not None:
            bias = self.bias[:out_nc]
        else:
            bias = None
        return F.conv2d(x, weight, bias, self.stride, self.padding, self.dilation, self.groups)
</code></pre>
<p>What can I change to use amp for this module?<br>
Is it F.conv2d support amp?</p>
<p>My environment is:</p>
<blockquote>
<p>gpu:  rtx2080ti<br>
torch:  py3.7_cuda10.1.243_cudnn7.6.3_0<br>
cuda: 10.1</p>
</blockquote>

ptrblck: <p>The code seems to work for me using your custom module:</p>
<pre><code class="lang-python">conv = SuperConv2d(10, 10, 3, 1, 1).cuda()
x = torch.randn(10, 10, 24, 24).cuda()

config = {'channel': 2}

with torch.cuda.amp.autocast():
    out = conv(x, config)

print(out.dtype)
&gt; torch.float16
</code></pre>
<p>Could you check, what might be the difference in my code snippet?</p>

Feywell: <p>Maybe, The difference is my input is float16, but it is float32 in your case.<br>
Because, I use the module inside the network. For me, the input of this module is float16.</p>

ptrblck: <p>Doesn’t seem to be the issue, as the input is not also <code>float16</code>:</p>
<pre><code class="lang-python">plain_conv = nn.Conv2d(10, 10, 3, 1, 1).cuda()
conv = SuperConv2d(10, 10, 3, 1, 1).cuda()
x = torch.randn(10, 10, 24, 24).cuda()

config = {'channel': 2}

with torch.cuda.amp.autocast():
    out = plain_conv(x)
    print(out.dtype)
    out = conv(out, config)

print(out.dtype)
</code></pre>
Data Loading through Sliding window in Pytorch

Sarath_Raveendran: <p>I have dataset(PAMAP2) as CSV file for implementing Human Activity Recognition. I was wondering how to load the data to a data loader using a sliding window of 2 seconds with a 50% overlap in PyTorch. I am trying to feed this particular data set to a CNN model for activity prediction.</p>

ptrblck: <p>If you are able to load the complete data into your RAM, <a href="https://discuss.pytorch.org/t/dataloader-for-a-lstm-model-with-a-sliding-window/22235/2">this code snippet</a> would give you a basic idea of a sliding window approach using a custom <code>Dataset</code>.</p>
The outputs of fasterRCNN is none

lan1: <p>My model is below:<br>
bb_model = torchvision.models.resnet101()<br>
bb_model_path = ‘/model/pretrain/resnet101-5d3b4d8f.pth’<br>
bb_model.load_state_dict(torch.load(bb_model_path))<br>
backbone = nn.Sequential(bb_model.conv1,<br>
bb_model.bn1,<br>
bb_model.relu,<br>
bb_model.maxpool,<br>
bb_model.layer1,<br>
bb_model.layer2,<br>
bb_model.layer3,<br>
bb_model.layer4)<br>
backbone.out_channels = 2048<br>
anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256, 512),),aspect_ratios=((0.5, 1.0, 2.0),))<br>
roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[‘0’],output_size=7,sampling_ratio=2)<br>
model = FasterRCNN(backbone,num_classes=201, rpn_anchor_generator=anchor_generator, box_roi_pool=roi_pooler)<br>
model_path = args.model_dir<br>
model.load_state_dict(torch.load(model_path))</p>
<p>When I give the net a input, the output of th network is none:<br>
[{‘boxes’: tensor([], size=(0, 4), grad_fn=), ‘labels’: tensor([], dtype=torch.int64), ‘scores’: tensor([], grad_fn=)}]</p>
<p>What cause the problem?</p>

ptrblck: <p>The output will be empty, if your model isn’t able to detect any objects as seen here using an untrained model and random inputs:</p>
<pre><code class="lang-python">model = models.detection.fasterrcnn_resnet50_fpn()
model.eval()
x = [torch.rand(3, 300, 400), torch.rand(3, 500, 400)]
out = model(x)
print(out)
&gt; [{'boxes': tensor([], size=(0, 4), grad_fn=&lt;StackBackward&gt;), 'labels': tensor([], dtype=torch.int64), 'scores': tensor([], grad_fn=&lt;IndexBackward&gt;)}, {'boxes': tensor([], size=(0, 4), grad_fn=&lt;StackBackward&gt;), 'labels': tensor([], dtype=torch.int64), 'scores': tensor([], grad_fn=&lt;IndexBackward&gt;)}]
</code></pre>
Logo usage in publication

peterneher: <p>Dear Pytorch Team,</p>
<p>we want to use the Pytorch logo in a scientific publication in JCO CCI to indicate the usage of Pytorch for our methods. Is this permitted? <a class="mention" href="/u/smth">@smth</a></p>
<p>Cheers,<br>
Peter</p>

peterneher: <p>Can you help, <a class="mention-group" href="/groups/admins">@admins</a> ?</p>

ptrblck: <p>CC <a class="mention" href="/u/jspisak">@jspisak</a> for visibility</p>

jspisak: <p>Yes but please follow the branding guidelines here: <a href="https://pytorch.org/assets/brand-guidelines/PyTorch-Brand-Guidelines.pdf">https://pytorch.org/assets/brand-guidelines/PyTorch-Brand-Guidelines.pdf</a></p>
<p>Thanks for checking!</p>
Can not load and test CNN model

iamneerav: <p>I am trying to load a model saved in a location as .pt file. I am trying to read the model as but it is giving me an AttributeError and/or a TypeError stating “missing 1 required positional argument: ‘self’” at .cuda() and .eval() lines.</p>
<pre><code>import sys
sys.path.insert(0, '/home/nkaushal/network/models')
sys.path.insert(0, '/home/nkaushal/network/utils')
import yadoc
from   utils     import *
from   libraries import *
torch.cuda.empty_cache()
from model_3 import UNet

param_file  = sys.argv[1]
params      = yadoc.parameters('/home/nkaushal/network/params/test_params_{}.yml'.format(param_file))

test_set     = high_dimensional_dataset(params.data, params.target, params.test.lIndex, params.test.hIndex, params.test.aug)
test_loader  = DataLoader(test_set, batch_size=params.test.batch_size, shuffle=True, num_workers=params.test.num_workers)


def test_prediction(path, model, test_loader):
    net = torch.load(model)
    net.cuda()
    net.eval()

    for test_batch, test_data in enumerate(test_loader, 0):
        print(test_batch)
        inputs, labels = test_data
        NetInput = torch.autograd.Variable(inputs,requires_grad=False).cuda()
        Y_pred = net(NetInput)
    np.save(path+'test_'+str(test_batch)+'.npy',np.concatenate((np.squeeze(Y_pred.data.cpu().numpy()$

return None


if params.test.is_test:
    test_prediction(params.out_path, params.model_path, test_loader)</code></pre>

ptrblck: <p>The recommended way to serialize the model is to store and load the <code>state_dict</code> and to recreate the model object as explained <a href="https://pytorch.org/docs/stable/notes/serialization.html">here</a>.<br>
Based on the error message I guess that loading the complete model fails.</p>
Transformer dropout at inference time

ales004: <p>Hi,<br>
looking at the <a href="https://github.com/pytorch/pytorch/blob/eace0533985641d9c2f36e43e3de694aca886bd9/torch/nn/modules/transformer.py#L282" rel="nofollow noopener"><code>TransformerEncoderLayer</code></a> and <a href="https://github.com/pytorch/pytorch/blob/eace0533985641d9c2f36e43e3de694aca886bd9/torch/nn/modules/transformer.py#L348" rel="nofollow noopener"><code>TransformerDecoderLayer</code></a> code, it seems that at inference time dropout is applied without change.</p>
<p>I thought that the dropout is not used at inference time. Is it correct or am I missing something?</p>

mariosasko: <p>The forward pass looks the same, but during inference you should change the module’s mode to <code>eval</code> (call <code>model.eval()</code>). This will update the module’s internal flag <code>self.training</code> and do the same for each submodule (this means you can call this once on the top-level module). This flag is used if the module behavior differs during training (call <code>module.train()</code> to enter the train mode) and evaluation (call <code>module.eval()</code> to enter the eval mode). So <code>nn.Dropout</code> (or e.g. <code>nn.BatchNorm</code>) relies on that functionality to change its behavior during training/evaluation.</p>
<p>This means that calling <code>module.eval()</code> is not neccesary in cases where there is no layer that has different behavior between training and evaluation (e.g. all layers are <code>nn.Linear</code> or <code>nn.Conv</code>) but still it’s a good practice.</p>

ales004: <p>Yes this was clear, I did not understood that this is enforced inside the Dropout module.<br>
Thanks.</p>
Model.eval() but layer have different weight for each trial

woong97: <p>I load pytorch model, and use model.load_state_dict(). And use model.eval()<br>
And I print weight of linear layer in order to check the weight of model<br>
However, the output of weight is continuously if I execute python file again.<br>
Thus the output of ‘forward’ is different for same input</p>
<p>How is this possible?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/b/ebace954d57b43817d1014d675d89aa86af9c19a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ebace954d57b43817d1014d675d89aa86af9c19a" title="스크린샷 2020-09-21 오전 5.12.03"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/ebace954d57b43817d1014d675d89aa86af9c19a_2_690x229.jpeg" alt="스크린샷 2020-09-21 오전 5.12.03" data-base62-sha1="xCSFkwCOarTmIfbWjb8D6OJMn4S" width="690" height="229" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/ebace954d57b43817d1014d675d89aa86af9c19a_2_690x229.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/ebace954d57b43817d1014d675d89aa86af9c19a_2_1035x343.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/ebace954d57b43817d1014d675d89aa86af9c19a_2_1380x458.jpeg 2x" data-dominant-color="4F5F7D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">스크린샷 2020-09-21 오전 5.12.03</span><span class="informations">1700×566 218 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In addition, I have two pytorch model (pt file). One is made from my local computer. And the other is made from google colab. Interestingly, pt file from colab has constant weight if I execute file again. But pt file from my local computer has different weight if I execute file again.<br>
Thus the outputs are always same if I use model from colab. But the outpust are always different if I use model from local computer</p>

ptrblck: <aside class="quote no-group" data-username="woong97" data-post="1" data-topic="96959">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/woong97/48/53120_2.png" class="avatar"> woong97:</div>
<blockquote>
<p>But pt file from my local computer has different weight if I execute file again.</p>
</blockquote>
</aside>
<p>Could you check if your script is saving the (updated) <code>state_dict</code> in each run?</p>
Why nn.BatchNorm2d returns different values comparing with the one calculated by math equation defined in paper

ftian1: <p>May I know why?</p>
<p>Below is the one calculated by following the formula define in paper.</p>
<p>… math::</p>
<pre><code>    y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta
</code></pre>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        #self.conv2_bn = nn.BatchNorm2d(16, track_running_stats=False)
        #nn.init.constant_(self.conv2_bn.weight, 1.135)
        #nn.init.constant_(self.conv2_bn.bias, 0.2756)

    def forward(self, x):
        m = torch.mean(x)
        v = torch.var(x, unbiased=False)
        z = (x - m) / math.sqrt(v + 1e-5)
        x = z * 1.135 + 0.2756
        return x

torch.manual_seed(1)
n = Net()
n.eval()
x = torch.rand((1, 16, 2, 2))
y = n.forward(x)
print(y)

</code></pre>
<p>The one is calculated by nn.batchnorm2d():</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv2_bn = nn.BatchNorm2d(16, track_running_stats=False)
        nn.init.constant_(self.conv2_bn.weight, 1.135)
        nn.init.constant_(self.conv2_bn.bias, 0.2756)

    def forward(self, x):
        x = self.conv2_bn(x)
        return x

torch.manual_seed(1)
n = Net()
n.eval()
x = torch.rand((1, 16, 2, 2))
y = n.forward(x)
print(y) 
</code></pre>
<p>These two values are different… May I know why?</p>

chetan_patil: <p>The output for <code>m</code> and <code>v</code> end up to be a single-valued Tensor.<br>
Instead they must be <code>16-valued Tensors</code>.</p>
<p>You must replace the code to</p>
<pre><code class="lang-auto">m = torch.mean(x, dim= (0, 2,3))
v = torch.var(x, dim=(0, 2, 3), unbiased=False)
</code></pre>

ftian1: <p><a class="mention" href="/u/chetan_patil">@chetan_patil</a> Many thanks for your prompt. I ignored this and after fixed that, the results are same.</p>
Is it possible for pytorch>1.4 built with CUDA9.0 to have anomaly behavior?

zehongs: <p>Hi!<br>
I built pytorch 1.5 with CUDA9.0 myself. Recently I encountered NaN/inf values when doing network forward. But the same code works fine on other servers(pytorch1.5+CUDA10.2).<br>
Is it possible for pytorch&gt;1.4 built with CUDA9.0 to have anomaly behavior?</p>
<ul>
<li>The random NaN was introduced by 1 of 64 weights in first conv layer suddenly became NaN.</li>
</ul>

zehongs: <p>It seems one of my gpus has been broken…</p>

ptrblck: <p>How did you figure it out?</p>

zehongs: <p>I ran the same code/data on different gpus, by setting the CUDA_VISIBLE_DEVICES. And only the gpu:0 gave me NaN after random epochs.<br>
I never met this after a server reboot. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
PyTorch Geometric - Data Split and Negative Sampling for edges

arko: <p>Hello,</p>
<p>I am currently using PyTorch Geometric for some experiments and wanted to get some clarity on how to use my own train / test splits as well as assign positive and negative labels.</p>
<p>I am creating my own InMemoryDataset from a DataFrame contains columns like IsTrain which is set to 1 / 0 depending on my Train / Test split. It also has a column IsPositive that tracks whether the edge is being supposed to be used as a positive or negative input.</p>
<p>Currently I am just using the utils methods like train_test_split_edges, negative_sampling which ignores my flags as mentioned below.</p>
<p>Any help is much appreciated. Many thanks in advance.</p>
<p>Thanks &amp; regards<br>
Arko</p>
Learning rate causes validation loss to go high

ManuelMendoza: <p>Hi there, I’m facing some strange behavior in my training code: by increasing the learning from 0.01 to 0.02, the validation loss gets way too high (from 100 to even 1000), but after a couple of iterations (between 20 and 30), it descends to 2.0 and starts decreasing as expected.</p>
<p>Currently, I’m working with GoogLeNet (no pretraining), but I’d been dealing with this problem for a while with different architectures like ResNet or LeNet. It’s not a problem that persists over time, but I’m sure it indicates that something is not right.</p>
<p>Until now, I’d been choosing the highest learning rate possible before things messed up, but for this case with GoogleNet, I need the learning rate to be higher.</p>
<p>My training code is as follows:</p>
<pre><code class="lang-auto">def train_step(model, criterion, data_set, device, optimizer):
    model.train()
    for images, labels in data_set:
        # Stating variables
        images = images.to(device)
        labels = labels.to(device)

        # Clear gradients
        optimizer.zero_grad()
        
        # Forward step
        output = model(images)
        
        # Loss step
        loss = criterion(output, labels)
        
        # Backward step
        loss.backward()
        optimizer.step()
        
        # Accuracy
        top_p, prediction = output.topk(1, dim=1)
        equals = (prediction == labels.view(*prediction.shape))

        # Saving data
        total_loss = loss.item()
        total_acc = torch.mean(equals.type(torch.FloatTensor))

        break

    return model, total_loss, total_acc

def val_step(model, criterion, data_set, device):
    with torch.no_grad():
        model.eval()
        for images, labels in data_set:
            # Stating variables
            images = images.to(device)
            labels = labels.to(device)
            
            # Forward steo
            output = model(images)
            
            # Loss step
            loss = criterion(output, labels)
            
            # Accuracy
            top_p, prediction = output.topk(1, dim=1)
            equals = (prediction == labels.view(*prediction.shape))

            # Saving data
            total_loss = loss.item()
            total_acc = torch.mean(equals.type(torch.FloatTensor))

            break

    return total_loss, total_acc
</code></pre>
<p>(I break the loop to get just one batch sample, I know there are better ways but for this case I’ve just been lazy)</p>
<p>Hyperparameters:</p>
<p>num_classes = 7<br>
batch_size = 128</p>
<pre><code class="lang-auto">criterion = CrossEntropyLoss()
optimizer = SGD(models[0].parameters(), lr=0.01, momentum=0.7, weight_decay=0.001)
</code></pre>
<p>I also have a lr_scheduler but that is just besides the point here. I would appreciate it if someone could tell a way to solve this problem or if, in any case, things just work like this, and I have to keep going with low learning rates.</p>
How to make a loss avoid some of the regions for back propagation

seyeeet: <p>I have inputs with different sizes. In order to be able to use them in a batch, I padded them to have the same size. now at the end of each iteration, I have <code>loss(output, label)</code>, where both output and label are in the shape of <code>B,C,W,H</code> but there are just certain part o each image that I care about and don’t want to penalize the model for making mistake on the padded parts.<br>
Is there a way to modify the loss (in my case MSE) to not care about the regions that are padded?</p>
<p>does it make sense if I just multiply the output with a mask at the end that has zero for the regions that I don’t care before computing the loss?</p>

mariosasko: <p>Yes, using a mask to ignore the padded parts seems right. At the moment, you can’t specify for MSELoss that some parts of the input should be ignored so zeroing those parts out is the only solution.</p>
Generalized unsqeeze to many dimensions

minimalist: <p>Hi,</p>
<p>Is there any function (or any simple way) to expand an $m$ dimensional tensor to $m + n$ dimensions?<br>
Basically I want to achieve the following code,</p>
<pre><code class="lang-auto">foo = torch.ones((2,2))
bar = foo.unsqueeze(-1).unsqueeze(-1).unsqueeze(-1).unsqueeze(-1).unsqueeze(-1)
</code></pre>
<p>by something like</p>
<pre><code class="lang-auto">foo = torch.ones((2,2))
bar = foo.unsqueeze(-1,n=5)
</code></pre>
<p>by an imaginary overload of <code>unsqueeze</code> that inserts <code>n</code> new dimensions at the end of tensor <code>foo</code>.<br>
So far the only solution I found was</p>
<pre><code class="lang-auto">bar = foo[(None,)*5].permute(-1,-2,0,1,2,3,4,5)
</code></pre>
<p>That is looks weird and I do not want to do the <code>permute</code> at the end also something cleaner would be nice.</p>
<p>Thanks</p>

mariosasko: <p>You can do something like this (it looks kind of messy though):</p>
<pre><code class="lang-python">foo[(...,) + (1,) * n]
</code></pre>
<p>or you can use view/reshape (a bit cleaner):</p>
<pre><code class="lang-python">foo.view(foo.shape + (1,) * n)
</code></pre>
Dense layer for many-to-many LSTM

suo: <p>My current LSTM has a many-to-one structure (please see the pic below). On the top of LSTM layer, I added one dropout layer and one linear layer to get the final output, so in PyTorch it looks like</p>
<blockquote>
<p>self.dropout = nn.Dropout(0.2)<br>
self.ln = nn.Linear(hidden_size, 1)</p>
</blockquote>
<blockquote>
<p>h, c = self.lstm(x)<br>
last_h = self.dropout(h[:, -1, :])<br>
out = self.ln(last_h)</p>
</blockquote>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/6415da0424dd66f2f5b134709b92baa59e604c55.jpg" data-download-href="https://discuss.pytorch.org/uploads/default/6415da0424dd66f2f5b134709b92baa59e604c55" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/6415da0424dd66f2f5b134709b92baa59e604c55_2_690x215.jpg" alt="image" data-base62-sha1="ehoyXM0mJXx0LvGo0XGnNJKovU9" width="690" height="215" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/6415da0424dd66f2f5b134709b92baa59e604c55_2_690x215.jpg, https://discuss.pytorch.org/uploads/default/optimized/1X/6415da0424dd66f2f5b134709b92baa59e604c55_2_1035x322.jpg 1.5x, https://discuss.pytorch.org/uploads/default/original/1X/6415da0424dd66f2f5b134709b92baa59e604c55.jpg 2x" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1329×416 85.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Now, I want to modify my LSTM to simulate many-to-many; let’s say many-to-10.</p>
<p>Q1. Is it correct if I add one dropout and one dense layer for the 10-cell outputs? Something like</p>
<blockquote>
<p>self.dropout = nn.Dropout(0.2)<br>
self.ln = nn.Linear(hidden_sizex10, 10)</p>
</blockquote>
<blockquote>
<p>h, c = self.lstm(x)<br>
last_h = self.dropout(h[:, -10, :])<br>
out = self.ln(last_h)</p>
</blockquote>
<p>Q2. Or, should I add one dropout and dense layer for each cell output (i.e., 10 dropout and 10 dense layers)? In this case, how I can implement them in the code?</p>

maamli: <p>Did you find a solution to this?</p>