akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Build Glow on Windows

milivoje96: <p>Hi,</p>
<p>I would like to build a Glow on my Windows x64 machine. For now, I’m working inside a virtual machine.</p>
<p>I went to <a href="https://github.com/pytorch/glow" rel="nofollow noopener">https://github.com/pytorch/glow</a> and I followed <strong>docs/GlowWindowsBuildx86.md</strong>.<br>
I used the <strong>prebuild</strong> version of LLVM and I did other steps successfully, but I have a problem with building the Glow.<br>
There are missing <strong>pthread.h</strong> and failed tests <strong>HAVE_GNU_POSIX_REGEX</strong> and <strong>HAVE_POSIX_REGEX</strong> during the execution of <em>CMake</em> command and a lot of warnings and linker errors(multiple definitions of symbols, etc.) during the <em>build</em> step.<br>
Also, only four executable files are generated, but I suppose there should be a lot.</p>
<p>Can you please tell me are there some additional steps that should be done in order to compile Glow properly.</p>
<p>I installed the newest Windows SDK, CMake, Python, and Visual Studio Community 2019 with C/C++ Development Kit.</p>
<p>Thanks in advance.</p>

jfix: <p>Hi, I’d suggesting posting an issue on GitHub and tagging mciprian13 and ayermolo. They’re some of the primary users that run Windows.</p>
Can Glow compile training of dynamically scheduled networks?

dbanas: <p>If I understand things correctly, Glow can be used to compile both the inference and the training of <em>statically</em> scheduled networks.<br>
Can it also be used to compile the training of <em>dynamically</em> scheduled networks?</p>

jfix: <p>Can you define what exactly you mean by static and dynamic scheduling here? Mostly likely the answer is no – currently Glow does not support dynamic shapes or control flow, so in order to support e.g. different batch sizes you need to compile different models (though you could dynamically dispatch to different compiled models for different batch sizes at runtime).</p>
Can you compare MLIR vs. GLOW?

Peter_Peter: <p>Hello guys,</p>
<p>before going to deep study of GLOW, can you please somehow elaborate on MLIR and compare it to GLOW?<br>
What are the benefits or possible drawbacks of GLOW vs. MLIR?<br>
Are these comparable technologies or these serve other purposes indeed?</p>

jfix: <aside class="quote no-group" data-username="Peter_Peter" data-post="1" data-topic="85549">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peter_peter/48/1360_2.png" class="avatar"> Peter_Peter:</div>
<blockquote>
<p>Are these comparable technologies or these serve other purposes indeed?</p>
</blockquote>
</aside>
<p>There’s some overlap, but in general they serve different purposes. MLIR is compiler infrastructure – it’s intended to be a useful multi-level IR that can be used across different SW stacks. Meanwhile, Glow is itself a compiler framework that has its own two levels of Glow-specific IR, and also includes optimizations written specifically for Glow IR, has features such as AOT compilation for LLVM-based backends, a high-performance runtime for server-oriented deployment, etc. In theory we could for example replace Glow’s IR with MLIR (but this is not something we’re planning).</p>

Peter_Peter: <p>Hello Jordan,</p>
<p>What you mean by  <strong>LLVM-based backends</strong>?</p>

jfix: <p>Glow uses LLVM in two ways: 1) Across the whole compiler as a support library e.g. for many of its efficient data structures, CL options, serialization, etc. 2) Separately, backends may use LLVM for low-level codegen for specific architectures once they go below Glow Instruction IR. E.g. the Glow CPU backend uses this, and other private backends may as well (maintained privately by OSS users so you don’t see them on Github).</p>
Is there any list of mapping functions for converting tensorflow codes to pytorch codes?

JTiC: <p>I try to convert tensorflow codes to pytorch codes, but I am not familar with tensorflow. Is there any list of mapping functions for converting tensorflow codes to pytorch codes? Something like:   tf.reshape&lt;-&gt; torch.reshape</p>
<p>Thank you very much!</p>

Nikronic: <p>Hi,</p>
<p>I do not think so, because there are many functions in PyTorch or TF which are not available in the other one. Also, many of similar functions work differently and use different semantic. Converting TF code to PyTorch needs deep understanding of both libraries which mostly can be achieved by reading documentations.<br>
Although they are some methods for instance converting a model to an intermidate state then converting to another model but codes will be generated automatically which may not be your case.</p>
<p>Bests</p>
Understanding PyTorch operators

flynntax: <p>Mainly looking for information about aten and prim operators. Is there documentation to look at all of them? What is the total number of them? It seems there’s a C API to use them directly but is there a Python API as well? Basically I want to be able to build a torchscript graph node by node with specific operators.</p>

Michael_Suo: <p>The aten and prim namespaces are considered internal details so we don’t really have user-facing documentation explaining them. The closest (for aten) is maybe <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/README.md" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/README.md</a>.</p>
<p>There is are Python bindings to the PyTorch IR in (<a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/python/python_ir.cpp" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/python/python_ir.cpp</a>), but once again they are considered internal.</p>
Verifying Model Architecture

mayurat22: <p>I am implementing CNN based Sentence Classification. I would request people to let me know, if i have implemented the batch normalization and dropout in the right order ? Any Suggestion is welcomed !</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7f57f6f43e01413510a4772cc603d130e4e1061e.png" data-download-href="https://discuss.pytorch.org/uploads/default/7f57f6f43e01413510a4772cc603d130e4e1061e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f57f6f43e01413510a4772cc603d130e4e1061e_2_624x500.png" alt="image" data-base62-sha1="iax6kKfU59fVckNZApkAAjA459s" width="624" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f57f6f43e01413510a4772cc603d130e4e1061e_2_624x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f57f6f43e01413510a4772cc603d130e4e1061e_2_936x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/f/7f57f6f43e01413510a4772cc603d130e4e1061e.png 2x" data-dominant-color="EDEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">946×758 79.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

harsha_g: <p><a class="mention" href="/u/mayurat22">@mayurat22</a> Looks Good To Me.</p>

mayurat22: <p>Thank you for the reassurance.</p>
How to add "to" attribute to Dataset

bfeeny: <p>I have a custom Dataset:</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, data, window):
        self.data = data
        self.window = window
        self.shape = self.__getshape__()
        self.size = self.__getsize__()

    def __getitem__(self, index):
        x = self.data[index:index+self.window]
        y = self.data[index+self.window]
        return x, y

    def __len__(self):
        return len(self.data) -  self.window 
    
    def __getshape__(self):
        return (self.__len__(), *self.__getitem__(0)[0].shape)
    
    def __getsize__(self):
        return (self.__len__())
</code></pre>
<p>I want to be able to have it copied to an alternate device (such as Cuda)</p>
<pre><code class="lang-auto">my_array = np.arange(100)
my_dataset=MyDataset(my_array,10)
my_loader = torch.utils.data.DataLoader(my_dataset, batch_size=20, shuffle = False, drop_last=True)
for data, target in my_loader.to(device):
</code></pre>
<p><code>AttributeError: 'DataLoader' object has no attribute 'to'</code></p>
<p>Even though the error says “DataLoader”, I assume it means the Dataset, since I am using a standard DataLoader I always use (torch.utils.data.DataLoader), but I am using a custom Dataset.</p>

Scott_Hoang: <p>The typical way you would go about it is:</p>
<pre><code class="lang-auto">for data in dataloader:
    data = data.to(device) # send to cuda 
</code></pre>
<p>if you want to access function within your dataset from your data loader</p>
<pre><code class="lang-auto">yourdataloader.dataset.&lt;your function / dataset attributes here&gt;
</code></pre>
<p>you can send your data to ‘cuda’ but not your dataset class to cuda.</p>

bfeeny: <p>I am not sure I understand what you are saying.  Do I have to add some sort of “to” attribute to my Dataset?  Perhaps I just take the output of my Dataset and make it a Tensor?</p>

Scott_Hoang: <p>your data coming from your dataloader is a torch.Tensor type.<br>
torch.Tensor type has built-in .to functionality.</p>

bfeeny: <p>Thanks, I was really tired the other night and having a moment.  That is how I always send to a device and yes it works with my dataset!</p>
Can quantized model be exported and used?

Shani_Gamrian: <p>Is it possible to transform the quantization ability to Caffe? Let’s say I created a quantized model using PyTorch and now I want to export the model to Caffe, can I do that by using the scale/zero_point parameters or it’s mandatory to use PyTorch for their quantization?</p>

jerryzh168: <p>you can take a look at ONNX, but we don’t have very good quantization support in ONNX right now, I’m not sure about the ONNX - caffe path either.</p>

Shani_Gamrian: <p>Is the quantization done once and then can be used (with the scale and zero_point) or it should have special support that make it int8 during inference?</p>

jerryzh168: <p>quantization is done before inference, it transforms a floating point model to a quantized model.</p>
AttributeError: 'torch.distributed.rpc.Future' object has no attribute 'then'

aguirguis: <p>Hello,</p>
<p>I am trying to use <code>then</code> method that is explained <a href="https://pytorch.org/docs/master/futures.html" rel="nofollow noopener">here</a> along with <code>rpc_async</code>. Yet, I get this error:<br>
AttributeError: ‘torch.distributed.rpc.Future’ object has no attribute ‘then’</p>
<p>I tried just importing <code>torch.futures.Future</code> and I got also an error:<br>
ModuleNotFoundError: No module named ‘torch.futures’</p>
<p>Yet, importing <code>torch.distributed.rpc.Future</code> works. But, the imported class does not have a constructor, where running <code>inspect.getmembers(Future, predicate=inspect.isfunction)</code> gives an empty array.</p>
<p>I have the latest torch version (1.5.1).</p>
<p>Would you please help me with this issue?</p>
<p>Thanks,<br>
Arsany</p>

mrshenli: <p>Hey <a class="mention" href="/u/aguirguis">@aguirguis</a>, the <code>torch.futures</code> package is introduced as an experimental feature in v1.6. If you use the nightly binary or compile from master, the <code>then</code> API should be there.</p>
<p>Sorry, we should have mentioned that it is only available for v1.6+</p>
Debugging JIT tracing/scripting

austinmw: <p>Hi, I’m currently trying to JIT compile <a href="https://github.com/xingyizhou/CenterTrack" rel="noopener nofollow ugc">CenterTrack</a> with either tracing or scripting.</p>
<p>CenterTrack uses <a href="https://github.com/CharlesShang/DCNv2" rel="noopener nofollow ugc">DCNv2</a>. I’ve found some resources that mentioned I needed to fix a couple things first:</p>
<ul>
<li>change code that uses Python integers instead of tensors. Got that from: <a href="https://github.com/xi11xi19/CenterNet2TorchScript" class="inline-onebox" rel="noopener nofollow ugc">GitHub - xi11xi19/CenterNet2TorchScript: centernet pytorch model to torch script model</a></li>
<li>possibly update to torch 1.5 by making code changes described here: <a href="https://github.com/CharlesShang/DCNv2/pull/58" class="inline-onebox" rel="noopener nofollow ugc">add pytorch 1.5 supported by lbin · Pull Request #58 · CharlesShang/DCNv2 · GitHub</a> (applied these to the above DCNv2 version instead of the original version)</li>
</ul>
<p>Now I’ve removed some of the initial errors I got when trying to trace, but am stuck the current ones.<br>
When I try to trace:</p>
<pre><code class="lang-auto"># Try trace
input_var = torch.zeros([1, 3, 736, 1280], dtype=torch.float32).cuda()
try:	
    traced_script_module = torch.jit.trace(detector.model, input_var)
except Exception as e:
    print(f'Exception (trace): {e}')
</code></pre>
<p>The only thing that is returned is:</p>
<blockquote>
<p>Exception (trace): Only tensors, lists and tuples of tensors can be output from traced functions</p>
</blockquote>
<p>And when I instead try torch script I have two issues, a small one being requirements of default values for tensors, but a larger one being heavy use of *args as inputs throughout the codebase.</p>
<p>I suspect that the torch trace issue comes from some forward() methods returning dictionaries. Is that not supported in v1.5? Is there a recommended alternative? And also is there a way to find out where in my python code the error “Only tensors, lists and tuples of tensors can be output from traced functions” refers to?</p>
<p>Any help is greatly appreciated!</p>
Is PyTorch tutorial wrong?

I_H_Yoo: <p>The <a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html?highlight=chatbot" rel="nofollow noopener">PyTorch chatbot tutorial</a> implements Luong attention and getting the encoder’s hidden state via:</p>
<pre><code class="lang-python"># Set initial decoder hidden state to the encoder's final hidden state
decoder_hidden = encoder_hidden[:decoder.n_layers]
</code></pre>
<p>I believe it must be <code>decoder_hidden = encoder_hidden[-decoder.n_layers:]</code> if it want to extract the last hidden state of last stacked LSTM. Am I missing something?</p>

harsha_g: <p>In this case, it would not make much difference because- the number of layers for both the encoder and decoder is the same. However, if the number of layers is unequal then the choice of which layers (the first <code>decoder.n_layers</code> or the last <code>decoder.n_layers</code>) to use is purely dictated by personal design choice.</p>

I_H_Yoo: <p>Thanks again. By the way, if I desing bi-directional encoder and stacked RNN, the hidden state of the encoder ($h_n$) must be <code>[# layers * # dir, Batch, Hidden]</code> and it doesn’t match to $H$'s dimension <code>[Seq_len, Batch, # dir * hid]</code> for feed to initial decoder’s hidden state. How can I solve it? Just employ a FC layer?</p>

harsha_g: <p>The answer to this question depends on how many layers and directions your decoder has. If you did not come across this earlier, I highly recommend <a class="mention" href="/u/bentrevett">@bentrevett</a>’s (a huge shoutout to him assuming he is the same as the owner of this repo) <a href="https://github.com/bentrevett/pytorch-seq2seq" rel="nofollow noopener">pytorch-seq2seq tutorials</a> where he walks you through all the shape transformations inside the encoder and decoder.</p>

I_H_Yoo: <p>I actually saw that and couldn’t find anything that helping me solve my problem. The link only implements 1 layer LSTM.</p>
How Preprocess video and work with that?

Andrea_Pelliccia: <p>Hi! I’m new of this forum.<br>
I’m doing a video classification, I don’t know how to load video and how to preprocess video from my dataset.<br>
My dataset has 99 videos divide in 6 classes, my big problem is how to divide video in training, validation and test.How to pre-process data video and how to upload my video in PyTorch?<br>
Thanks a lot!!</p>

ptrblck: <p>NVIDIA has a video loader for PyTorch: <a href="https://github.com/NVIDIA/nvvl">nvvl</a>.</p>
<p>The splitting depends on your use case. Would it be possible to split the videos based on the classes or do you want to split the videos frame-wise?</p>

Andrea_Pelliccia: <p>Thanks for answering me immediately.<br>
I would like to divide them into frames so that I can process them as images using a rnn. The problem is that I do not know how to load them directly in python or process them since the videos are integral and already divided into classes. But to train them I have to do a shuffle,I think, at the same time I should keep the video separate for supervised learning, for the test. How can I start?<br>
Thanks again</p>

ptrblck: <p>So as I understand your use case you would like to classify each frame separately.<br>
The video sequence is therefore not important, i.e. each frame has the target of the video target?</p>
<p>If so, you could just load the video, e.g. with OpenCV, and save each frame of the video into separate folders.<br>
Using these images, you could then train a supervised model on the data.</p>
<p>I would split the videos in a stratified fashion, i.e. approx. using the same class distribution for training, validation and test data.</p>

Andrea_Pelliccia: <p>I apologize if I explained myself badly.<br>
I meant that to classify the video I need every frame or at least to train a RNN, I use one frame every 5s for example, and I use the other 4 to train an RNN (like LSTM).<br>
My goal is to train an RNN that allows me to classify 6 classes in my dataset composed of 99 videos.<br>
My problem is that I don’t know how to upload videos since I have 99 videos tagged(like label), but all together and not divided into train, test and validation.<br>
I don’t know how to load videos in python and how to divide them into train, validation and test.<br>
I assumed that to classify a video I have to divide it into frames, am I wrong? How would you do it?<br>
Initially I would like to train it using the CPU.<br>
Could you recommend some approach for this project?<br>
Thanks for the answer!!</p>

Etienne_Perot: <p>hello Andrea, can you try perhaps <a href="https://github.com/etienne87/pytorch-streamloader" rel="nofollow noopener">https://github.com/etienne87/pytorch-streamloader</a> and tell me if that fits your need?</p>
CrossEntropyLoss in train vs validation

rbapat: <p>The documentation specifies that CrossEntropyLoss combines both LogSoftmax and NLLLoss, which means that I don’t have to implement nn.LogSoftmax into my model; i can just return the output of my last dense layer. However, since I’m not actually running the loss function for the validation stage, would I need to call nn.LogSoftmax on my model output? This is what the main part of my validation loop looks like:</p>
<pre><code class="lang-auto"># model returns nn.Linear(..., num_outputs)
probs = model(data)

label = torch.argmax(label, dim = 1)
preds = torch.argmax(probs, dim = 1)

running_correct += (preds == label).sum().item()
</code></pre>
<p>Basically, since im not running the loss function, there is no softmax being run on the model outputs; should i call LogSoftmax on <code>probs</code> before calculating the predictions? If this is the case then I assume it would be better to use NLLLoss and throw a softmax in my model.</p>

Chirath_Pansilu: <p>Yes it will be easier to use logsoftmax on model output and NLLLoss as loss function.<br>
Then when you do validation you should use exponential on model output to get actual probabilities</p>
List of JIT-supported operations

tengerye: <p>Hi, may I know where I can find a list of JIT-supported or not supported operations in Pytorch please?</p>

Michael_Suo: <p>supported PyTorch stuff: <a href="https://pytorch.org/docs/master/jit_builtin_functions.html#builtin-functions" rel="nofollow noopener">https://pytorch.org/docs/master/jit_builtin_functions.html#builtin-functions</a><br>
unsupported PyTorch stuff: <a href="https://pytorch.org/docs/master/jit_unsupported.html#jit-unsupported" rel="nofollow noopener">https://pytorch.org/docs/master/jit_unsupported.html#jit-unsupported</a></p>

tengerye: <p>Thank you so much. But I can’t find <code>SyncBatchNorm</code> in either page. Is there something wrong?</p>
Optimizing parameters wrapped in torch.distributions

smonsays: <p>I want to optimize the parameters of a probability distribution using <code>rsample()</code> in the computational graph. Ideally, I would just do the following:</p>
<pre><code class="lang-python">import torch
mu = torch.randn(2, requires_grad=True)
dist = torch.distributions.normal.Normal(loc=torch.sigmoid(mu), scale=torch.ones(2))

for _ in range(3):
    loss = torch.sum(dist.rsample())
    loss.backward()

    with torch.no_grad():
        mu.add_(mu.grad, alpha=-0.1)
</code></pre>
<p>This returns the</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed.
</code></pre>
<p>The transformation <code>torch.sigmoid()</code> inside the definition of the distribution is causing this problem. Is there a nicer way of accomplishing this other than reconstructing the distribution on every call of <code>rsample</code> as suggested <a href="https://discuss.pytorch.org/t/detach-rsamples-from-distributions-parameters/50251/11">here</a>?</p>

Scott_Hoang: <p>you should try loss.backward(retain_graph=True)</p>

smonsays: <p>Wouldn’t that just leave me with the complete computational graph and make each iteration slower and slower as a result?</p>

Scott_Hoang: <p>yes. but as the the error descriptors said, you are backwarding through the graph a second time. So you need to keep computational graph intact to do that.</p>

smonsays: <p>In that case I prefer the workaround I mentioned as it does not suffer from this problem. I was hoping there was a more elegant solution.</p>

Scott_Hoang: <p>you could have mu.detach() and passed into torch.sigmoid</p>

googlebot: <p>I often just use rsample/log_prob code without distribution objects. Like:</p>
<pre><code class="lang-auto">def normal_rsample(m,s,shape):
	return torch.randn(shape) * s + m
</code></pre>
Back propagation of model partition training on different mobile devices

torch_fresh: <p>I have DNN model, Layer {1, 2, 3, 4, 5}. I separate the model into two parts: fore part, layer {1, 2, 3} and hind part, layer {4, 5}. I deploy these two parts to different mobile devices. For example, fore part, layer {1, 2, 3} is deployed on device A and hind part, layer {4, 5} is deployed on device B. Training has two parts: forward pass and backward pass.</p>
<p><strong>Forward pass:</strong> one mini-batch flows from layer 1 to layer 3 on device A. The intermediate output of layer 3 is sent to device B by wireless connection. Then, the intermediate output of layer 3 flows from layer 4 to layer 5 and loss function.</p>
<p><strong>Backward pass:</strong> calculate the gradient and update the weight from layer 5 to layer 4  on mobile device B. The intermediate gradient of layer 4 is sent back to device A by wireless connection. Then, the mobile device A finishes the following back propagation.</p>
<p>I can realize the the forward pass. However, I does not understand how pytorch can realize the backward pass. For example, how the device B finishes the back propagation and get the intermediate gradient of layer 4. How the device A feeds intermediate gradient to layer 3 and finishes the back propagation.</p>
<p>Thank you for any suggestion and code example.</p>
How to mark argument as nondifferentiable in a custom autograd Function

pyscorcher: <p>I’m trying to implement a custom autograd.Function, where one of the arguments for the forward pass does not need a gradient (let’s say an index or a string or something like that). To illustrated I just modified the <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" rel="nofollow noopener">example from the tutorial</a>  as follows:</p>
<pre><code class="lang-auto">class MyReLU(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, second):
        print(second)   # &lt;- second nondifferentiable argument
        ctx.save_for_backward(input)
        return input.clamp(min=0)

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad_input[input &lt; 0] = 0
        return grad_input  # as a "hack"(?) you could just add None as a second output
</code></pre>
<p>Now I get the error<br>
<code>RuntimeError: function MyReLUBackward returned an incorrect number of gradients (expected 2, got 1)</code></p>
<p>Is there a way to define that the second argument doesn’t need a gradient?</p>
<p>I noticed that you can just use <code>return grad_input, None</code> in the <code>backward()</code> function, but that seems just like a hack. Or is this the way you’re supposed to do it?</p>

albanD: <p>Hi,</p>
<p>No need to do anything specific in the forward. And in the backward, returning <code>None</code> is the right thing to do <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

pyscorcher: <p>ah cool, then it is simpler than I imagined, thanks a lot for you answer! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Gradients are zero when calling autograd on torch.bernoulli()

smonsays: <p>I was surprised to realise that it is possible to call <code>backward()</code> in a computational graph that invokes torch.bernoulli(). However, I am not completely sure what happens in the background.</p>
<p>In this small example the gradients wrt to the Bernoulli probabilities are as far as I can tell always zero.</p>
<pre><code class="lang-python">import torch
p = torch.rand(2, 2, requires_grad=True)
loss = torch.sum(torch.bernoulli(p))
loss.backward()
torch.allclose(p.grad, torch.zeros_like(p))
</code></pre>
<p>So I assume this is not invoking some form of surrogate gradient, is it? Is it intentional that the gradients are always 0 in this case?</p>
<p>I would also be grateful if someone could point me to the respective <code>*.cpp</code> file that implements the <code>backward()</code> call.</p>
<p>Many Thanks,<br>
Simon</p>

albanD: <p>Hi,</p>
<p>Yes this is expected.<br>
It is specified here: <a href="https://github.com/pytorch/pytorch/blob/727463a727e75858809a325477ac2b62ccd08e7e/tools/autograd/derivatives.yaml#L270-L271">https://github.com/pytorch/pytorch/blob/727463a727e75858809a325477ac2b62ccd08e7e/tools/autograd/derivatives.yaml#L270-L271</a></p>
Another pip install problem

Ashley_Reid: <p>hay,<br>
struggling to install pytorch using any of the methods I have found so far. I am wondering if this is because my CUDA is not supported yet? what is my best route to getting this going?</p>
<p><code>PS C:\&gt; nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2020 NVIDIA Corporation Built on Wed_May__6_19:10:02_Pacific_Daylight_Time_2020 Cuda compilation tools, release 11.0, V11.0.167 Build cuda_11.0_bu.relgpu_drvr445TC445_37.28358933_0</code><br>
<code>PS C:\&gt; pip install torch===1.5.1 torchvision===0.6.1 -f https://download.pytorch.org/whl/torch_stable.html Looking in links: https://download.pytorch.org/whl/torch_stable.html ERROR: Could not find a version that satisfies the requirement torch===1.5.1 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2) ERROR: No matching distribution found for torch===1.5.1</code></p>
<p>that said, i feel “no matching distribution” dosn’t feel right to me?</p>

albanD: <p>Hi,</p>
<p>I would expect this to be because you are using a 32-bit version of python and we only provide binaries for 64-bit python?</p>
Faster/Mask RCNN RPN custom AnchorGenerator

sigma_x: <p>Every time I define a new Anchor Generator, I get a CUDA OOM problem. I suspect it’s nothing to do with memory, there’s a weight mismatch somewhere. Here’s the code:</p>
<pre><code class="lang-auto">mrcnn_args = {'num_classes':63}
icdar_anchor_generator = AnchorGenerator(
      sizes=tuple([(4, 8, 16, 32, 64, 128, 256, 512) for r in range(5)]), 
      aspect_ratios = tuple([(0.25, 0.5, 1, 1.5, 2) for rh in range(5)]))
mrcnn_args['rpn_anchor_generator'] = icdar_anchor_generator
maskrcnn_model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=False, **mrcnn_args)
</code></pre>
<p>I followed the suggestion here:</p>
<blockquote>
<p><a href="https://github.com/pytorch/vision/issues/978" class="inline-onebox" rel="noopener nofollow ugc">Load pre-trained fasterrcnn with AnchorGenerator() · Issue #978 · pytorch/vision · GitHub</a></p>
</blockquote>
<p>If I simply defined one tuple for sizes of aspect views, I got a size mismatch:</p>
<pre><code class="lang-auto">icdar_anchor_generator = AnchorGenerator(
      sizes=(4, 8, 16, 32, 64), 
      aspect_ratios = (0.5, 1, 1.5))

RuntimeError: shape '[840000, -1]' is invalid for input of size 4476540
</code></pre>
<p>So what else should I define?</p>

chhaya_kumar_das: <p>Did you resolve this?</p>

ham952: <p>Here is the correct way to do so</p>
<pre><code>def get_instance_segmentation_model_anchors(num_classes):
 #load an instance segmentation model pre-trained on COCO
 model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=False)

 #create an anchor_generator for the FPN which by default has 5 outputs
 anchor_generator = AnchorGenerator(
 sizes=((16,), (32,), (64,), (128,), (256,))
 aspect_ratios=tuple([(0.25, 0.5, 1.0, 2.0) for _ in range(5)]))

 model.rpn.anchor_generator = anchor_generator

 # 256 because that's the number of features that FPN returns
 model.rpn.head = RPNHead(256, anchor_generator.num_anchors_per_location()[0])

 # get the number of input features for the classifier
 in_features = model.roi_heads.box_predictor.cls_score.in_features

 # replace the pre-trained head with a new one
 model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

 # now get the number of input features for the mask classifier
 in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
 hidden_layer = 256

 # and replace the mask predictor with a new one
 model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask,
                                                   hidden_layer,num_classes)

 return model
</code></pre>
<p>You can now call the model and validate the anchors sizes and aspect ratios as</p>
<pre><code>model = get_instance_segmentation_model_anchors(num_of_classes)
print('Anchor Size :',model.rpn.anchor_generator.sizes)
print('Anchor Aspect ratio :',model.rpn.anchor_generator.aspect_ratios[0])</code></pre>
Exploding memory when using variable size batches

wouterM: <p>Hi all,</p>
<p>Currently I am working on a computer vision project where during inference the number of patches to be classified differs quite heavily (in the 20 - 400) range. When I run these variable sized batches through the network the used memory absolutely explodes. The memory usage (checked with htop) is almost twice as high as when I use a fixed size input batch of 400 (the max).</p>
<p>Is there a logical reason for why this happens? I could split the batches into equally size chunks but this would hamper performance since more forward passes are required.</p>
<p>Here is some code for reproduction, change the constant 250 to np.random.randint(1, 250) to see the difference:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch
from torchvision import models
import numpy as np

class network(nn.Module):
    def __init__(self):
        super(network, self).__init__()
        self.features = models.alexnet(pretrained=True).features[:5]
        for layer in self.features.parameters():
            layer.requires_grad = False
        self.forward_pass = nn.Sequential(
            nn.Linear(192 * 5 * 5, 10000),
            nn.ReLU(),
            nn.Linear(10000, 5000),
            nn.ReLU(),
            nn.Linear(5000, 1000)
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(-1, 192 * 5 * 5)
        return self.forward_pass(x)

net = network()
for layer in net.parameters():
    layer.requires_grad = False
net.eval()

with torch.no_grad():
    while True:
        dataset = torch.rand((250, 3, 50, 50))
        net.forward(dataset)
</code></pre>
Opinion new (sinusoid) activation function

ab-10: <p>I would like to implement a sinusoid activation function in <code>torch.nn.functional</code>, this is motivated by the strong results demonstrated using the activation function in <em>Implicit Neural Representations with Periodic Activation Functions</em> (<a href="https://arxiv.org/abs/2006.09661" rel="nofollow noopener">ArXiv</a>, <a href="https://vsitzmann.github.io/siren/" rel="nofollow noopener">paper’s webiste</a>, and <a href="https://github.com/lucidrains/siren-pytorch" rel="nofollow noopener">Github repo</a>).</p>
<p>I’m curious about whether this looks like a valuable contribution to PyTorch?</p>

albanD: <p>Hi,</p>
<p>The general policy we have here is the following:</p>
<ul>
<li>We expect the given paper to have a significant amount of citations and have proven to be valuable over time (we want the core library to remain small enough)</li>
<li>Exceptions can happen when it is hard/impossible to implement a given paper with the tools we currently provide.</li>
</ul>
<p>In this case, you can use the regular sin function from torch right? What would be different if you add it to <code>nn.functional</code>?</p>

ab-10: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,<br>
Thank you for informing me on the reasoning behind adding new features to the library. Keeping in mind the policy, I have to agree with you that there’s not much reason to add sin activation function as of yet.</p>
Toy example for pack_padded_sequence

praneethsv: <p>I was trying to run the working example on how to use packing for variable-length sequence inputs for rnn taken from this link (<a href="https://discuss.pytorch.org/t/simple-working-example-how-to-use-packing-for-variable-length-sequence-inputs-for-rnn/2120" class="inline-onebox">Simple working example how to use packing for variable-length sequence inputs for rnn</a>).</p>
<p>Here is the code:</p>
<p>import torch<br>
import torch.nn as nn<br>
from torch.autograd import Variable<br>
from torch.nn import functional as F</p>
<p>batch_size=4<br>
max_length = 3<br>
hidden_size = 2<br>
n_layers = 1</p>
<p>vec_1 = torch.FloatTensor([[1, 2, 3]])<br>
vec_2 = torch.FloatTensor([[1, 2, 0]])<br>
vec_3 = torch.FloatTensor([[1, 0, 0]])<br>
vec_3 = torch.FloatTensor([[4, 0, 0]])</p>
<p>print(vec_1)</p>
<p>batch_in[0] = vec_1<br>
batch_in[1] = vec_2<br>
batch_in[2] = vec_3<br>
batch_in[3] = vec_4</p>
<p>batch_in = Variable(batch_in)</p>
<p>print(batch_in)</p>
<p>seq_lengths = [3,2,1, 1]  # list of integers holding information about the batch size at each sequence step</p>
<p>pack = torch.nn.utils.rnn.pack_padded_sequence(batch_in, seq_lengths, batch_first=True)<br>
print(pack)</p>
<p>rnn = nn.RNN(max_length, hidden_size, n_layers, batch_first=True)<br>
h0 = Variable(torch.randn(n_layers, batch_size, hidden_size))</p>
<p>out, _ = rnn(pack, h0)</p>
<p>batch_in = torch.zeros((batch_size, max_length, 1))</p>
<p>However, When I execute the code. It throws me the following error.</p>
<p>RuntimeError: The expanded size of the tensor (1) must match the existing size (3) at non-singleton dimension 1.  Target sizes: [3, 1].  Tensor sizes: [3]</p>
<p>But, If I change the vec_1, vec_2, vec_3, vec_4 to the following way:</p>
<p>vec_1 = torch.FloatTensor([[1], [2], [3]])<br>
vec_2 = torch.FloatTensor([[1], [2], [0]])<br>
vec_3 = torch.FloatTensor([[1], [0], [0]])<br>
vec_3 = torch.FloatTensor([[4], [0], [0]])</p>
<p>I don’t get the above error but a different error. Here it is:</p>
<p>RuntimeError: input.size(-1) must be equal to input_size. Expected 3, got 1</p>
<p>Can someone explain what is wrong in here? In the discussion form, it seems for everyone else it is working.</p>
Rebooted while training

jaemin: <p>Hi.<br>
I have the system shutdown problem while training. After training a few batches, the computer shut down itself and rebooted. I think the custom layer causes this problem because training works well without that layer. The custom layer applies self-attention to compute the weighted mean and standard deviation in the time domain.</p>
<p>I trained with 4 GPUs using nn.DataParallel and the Pytorch version is 1.5.0.</p>
<pre><code class="lang-auto">class AttnPooling(nn.Module):
    def __init__(self, din, dh=500, eps=1e-12):
        super(AttnPooling, self).__init__()
        self.eps = eps
        self.w1 = Parameter(torch.Tensor(dh, din))
        self.w2 = Parameter(torch.Tensor(1, dh))
        self.reset_parameters()

    def reset_parameters(self) -&gt; None:
        init.kaiming_uniform_(self.w1, a=math.sqrt(5))
        init.kaiming_uniform_(self.w2, a=math.sqrt(5))

    def forward(self, x, dim):
        # x : (Batch, F_dim, Time)
        # h : (Batch, Time, F_dim)
        h = x.transpose(1, 2)
        attn = F.relu(F.linear(h, self.w1))
        attn = F.softmax(F.linear(attn, self.w2), dim=1)

        mean = torch.bmm(x, attn)
        variance = (x - mean).pow(2).mean(dim=-1)
        mean = mean.squeeze(dim)

        mask = (variance &lt;= self.eps).type(variance.dtype).to(variance.device)
        variance = (1.0 - mask) * variance + mask * self.eps
        stddev = variance.sqrt()

        # mean: (B, F_dim), stddev: (B, F_dim)
        pooling = torch.cat((mean, stddev), dim=-1)
        return pooling
</code></pre>

Scott_Hoang: <p>I think your computer overheated from training.</p>

jaemin: <p>It could be one of the reason, but using mean+stddev layer is works well.<br>
If the overheat is the problem, I think that mean+stddev layer also shows rebooting problem.</p>

ptrblck: <p>Besides overheating your PSU might be too weak for the setup. This would be visible if your workload creates some peaks in GPU utilization, which might need more power for this time frame.<br>
Could you check the max. power requirement for the GPUs and compare it to your PSU?</p>

jaemin: <p>There are 4 TITAN Xp GPUs. The power capacity of a GPU is 250W and PSU is 1600W model.<br>
Considering the other parts of the computer, it can be the reason.<br>
Thanks for your help.</p>
"Missing key(s) in state_dict: "features.0.weight" ..." Error exporting VGG19 to ONNX

RIP_Dude: <p>While attempting to convert PyTorch VGG19 model to onnx with a weighed Pre-trained checkpoint from <a href="https://drive.google.com/drive/folders/1PyQJmkdCsAkOYwUyaj_l-l0as-iLDgeH" rel="nofollow noopener">https://drive.google.com/drive/folders/1PyQJmkdCsAkOYwUyaj_l-l0as-iLDgeH</a></p>
<p>Specifically the vox-cpk.pth.tar. I think it’s a vgg19 because that’s but I am not a 100% sure.</p>
<p>my code:</p>
<pre><code class="lang-auto">import torch
from vgg_pytorch import VGG
from torch.autograd import Variable
import torchvision.models as models


dummy_input = torch.randn(16, 3, 224, 224)

model = models.vgg19(pretrained=False)

#I am running on a MBP. I can run it on Google Colab if needed
state_dict = torch.load('./vox-cpk.pth.tar',  map_location=torch.device('cpu')) 

model.load_state_dict(state_dict)

torch.onnx.export(model, dummy_input, "demo19.onnx", verbose=True)


</code></pre>
<p>Error message:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "export2onnx.py", line 12, in &lt;module&gt;
    model.load_state_dict(state_dict)
  File "/Users/MYNAME/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 847, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for VGG:
	Missing key(s) in state_dict: "features.0.weight", "features.0.bias", "features.2.weight", "features.2.bias", "features.5.weight", "features.5.bias", "features.7.weight", "features.7.bias", "features.10.weight", "features.10.bias", "features.12.weight", "features.12.bias", "features.14.weight", "features.14.bias", "features.16.weight", "features.16.bias", "features.19.weight", "features.19.bias", "features.21.weight", "features.21.bias", "features.23.weight", "features.23.bias", "features.25.weight", "features.25.bias", "features.28.weight", "features.28.bias", "features.30.weight", "features.30.bias", "features.32.weight", "features.32.bias", "features.34.weight", "features.34.bias", "classifier.0.weight", "classifier.0.bias", "classifier.3.weight", "classifier.3.bias", "classifier.6.weight", "classifier.6.bias". 
	Unexpected key(s) in state_dict: "discriminator", "kp_detector", "optimizer_discriminator", "optimizer_kp_detector", "epoch", "generator", "optimizer_generator". 
</code></pre>
<p>My goal:<br>
is to convert the first-order-motion project into ONNX then into CoreML to deploy this <a href="https://github.com/AliaksandrSiarohin/first-order-model" rel="nofollow noopener">https://github.com/AliaksandrSiarohin/first-order-model</a> but on iOS.</p>

ptrblck: <p>Based on the error message it seems the loaded object doesn’t hold the model’s <code>state_dict</code> only, but:</p>
<pre><code class="lang-python">"discriminator", "kp_detector", "optimizer_discriminator", "optimizer_kp_detector", "epoch", "generator", "optimizer_generator"
</code></pre>
<p>I guess <code>discriminator</code> and <code>generator</code> might be the keys for <code>state_dicts</code>, but based on their name I don’t think they are used in a “normal” VGG model.</p>
What is the batch text tuple composed of?

Jeremy_Rutman: <p>I am using the iterators generated by torchtext and am wondering about the batch.text tuple - the first element of the tuple seems to have a batch of word indices (as judging by the size, [batchsize*sentence_length] ) but the second element is a mystery to me - it is of length batchsize, but what are the values in there?  Is this covered in the docs somewhere I cound’t find?</p>
<pre><code>    for idx, batch in enumerate(val_iter):
        text = batch.text[0]    # this is the first element of the batch.text tuple - what is the second??</code></pre>

harsha_g: <p>It is difficult to answer the question without much context. For example, a quick google search of your code snippet brings up <a href="https://github.com/microsoft/samples-for-ai/blob/master/examples/NLPTutorials/language_model/language_model.py#L96-L97" rel="nofollow noopener">this</a> snippet of code</p>
<pre><code class="lang-auto">for b, batch in enumerate(train_iter):
            x, y = batch.text, batch.target
</code></pre>
<p>from where it’s easy to make out what is what. A similar context (source code or task) will help answer your question.</p>

Jeremy_Rutman: <p>Here’s some more context - the iterator is coming from the torchtext bucketiterator .  The label has its own field , batch.label so that is not it.  The values I see are integers, as though a single numericalized  sentence from the batch were being sent as the mystery  second tuple element .</p>
<pre><code class="lang-auto">fields = {'text: ('text',TEXT), 'label': ('label',LABEL)}
train_data, test_data, validate_data = data.TabularDataset.splits(
        path=path,
        train=trainfile,
        test=testfile,
        validation=validationfile,
        format='csv',  #csv_reader_params=
        fields=fields
    )
train_iterator, test_iterator,valid_iterator = data.BucketIterator.splits(
        (train_data, test_data,validate_data),
        batch_size=BATCH_SIZE,
        sort_key=lambda x: len(x.text),
        device=device)
for idx, batch in enumerate(valid_iterator):
        text = batch.text[0]    # this is the first element of the batch.text tuple - what is the second??
        target = batch.label
</code></pre>

harsha_g: <p>Length of the sentence</p>

Jeremy_Rutman: <p>Thanks - is this referred to in the docs somewhere? I couldnt  find much  info on the iterators themselves.</p>

harsha_g: <p>Same here. In the past, I too struggled with a lack of good documentation about <code>torchtext</code> but my goto resources were the following and some deep digging <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<ol>
<li>
<a href="http://anie.me/On-Torchtext/" rel="nofollow noopener">http://anie.me/On-Torchtext/</a> (the answer to your question is in here)</li>
<li><a href="https://mlexplained.com/2018/02/08/a-comprehensive-tutorial-to-torchtext/" rel="nofollow noopener">https://mlexplained.com/2018/02/08/a-comprehensive-tutorial-to-torchtext/</a></li>
</ol>
A question on how batches are passed through a PyTorch network

JonathanHampton: <p>So when a batch of data is passed through a network, is each sample in the batch passed through its own instance of the network in parallel? My assumption is that it is. How does PyTorch go about this? Is it a multithreaded process? How does it work on a GPU?</p>

googlebot: <p>The concept is called <a href="https://en.wikipedia.org/wiki/Data_parallelism" rel="nofollow noopener">https://en.wikipedia.org/wiki/Data_parallelism</a>. Batch dimension of tensors is assumed to be independent, meaning that operations that transform other dimensions can be executed in parallel . And threading to do this is “implementation detail”, i.e. it is backend specific, and also operation specific (e.g. vector/matrix operations treat some dimensions as special)</p>
TypeError: len() of a 0-d tensor

JSP: <p>Hi,<br>
Kindly help me to solve the error arising due to assert len(…) = len(…) stmt which was initially written using torch 0.2.0 torchvision 0.1.9 but is causing an issue in torch 1.5.1 torchvision 0.5.0 (for compatibility with cuda 10.1).</p>
<p>Traceback (most recent call last):<br>
File “./main.py”, line 186, in<br>
cuda=cuda<br>
File “/home/js/GR/train.py”, line 102, in train<br>
collate_fn=collate_fn,<br>
File “/home/js/GR/dgr.py”, line 130, in train_with_replay<br>
collate_fn=collate_fn,<br>
File “/home/js/GR/dgr.py”, line 205, in _train_batch_trainable_with_replay<br>
callback(trainable, progress, batch_index, result)<br>
File “/home/js/GR/train.py”, line 157, in cb<br>
result[‘g_loss’], ‘generator g loss’, iteration, env=env<br>
File “/home/js/GR/visual.py”, line 88, in visualize_scalar<br>
[name], name, iteration, env=env<br>
File “/home/js/GR/visual.py”, line 93, in visualize_scalars<br>
assert len(scalars) == len(names)<br>
File “/home/js/anaconda3/envs/env_con/lib/python3.5/site-packages/torch/tensor.py”, line 445, in  <strong>len</strong><br>
raise TypeError(“len() of a 0-d tensor”)<br>
TypeError: len() of a 0-d tensor</p>

tom: <p>A long time ago, PyTorch, here 0.2, didn’t have scalar (0-dimensional) Tensors and so you would have to use tensors of shape <code>[1]</code>. Thus tensors could always act like sequences.<br>
Nowadays, we do have scalar Tensors and these don’t act like sequences.</p>
<p>Likely you want to look more closely at what type of shapes can happen there and why.</p>
<p>Porting from PyTorch 0.2 does take some effort (there also is the Variable/Tensor merge that such code would not know about yet, see the 0.4 migration guide).</p>