akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Jensen Shannon Divergence

kko: <p>Hi,</p>
<p>I am trying to implement Jensen Shannon Divergence (JSD) in Pytorch:<br>
class JSD(torch.nn.Module)</p>
<p>def forward(self, P, Q):<br>
kld = KLDivLoss().cuda()<br>
M = 0.5 * (P + Q)<br>
return 0.5 * (kld(P, M) + kld(Q, M))<br>
When I run the above code, I am getting the below error:<br>
AssertionError: nn criterions don’t compute the gradient w.r.t. targets - please mark these variables as volatile or not requiring gradients.</p>
<p>I guess KL expect second term to be not requiring gradients. But, in JSD M term also contains gradient requiring variable. Is there an easy way of dealing with this? Or, should I write my KLD function?</p>
<p>Thanks.</p>
<p>(I created this post by mistake, it has already been answered. But, I dont know how to remove this one.)</p>

kamata1729: <p>You should add</p>
<pre><code class="lang-auto">super().__init__()
</code></pre>
<p>at the first line of <strong>init</strong></p>
The ordinal 961 could not be located in the dynamic link library

Shisho_Sama: <p>This is a follow up question about <a href="https://discuss.pytorch.org/t/is-building-from-source-for-windows-up-to-date-subprocess-calledprocesserror-command-cmake-returned-non-zero-exit-status-1-error/90769">this one</a> . I gave up on that momentiraliy and instead followed <a href="https://medium.com/@albertsundjaja/installing-pytorch-c-api-d52c722f47ec" rel="nofollow noopener">this guide</a><br>
and created a simple cmake project, built it and opened the project in visual studio for further development.<br>
However, After the successful build, executing the executable fails with this error message :</p>
<pre><code class="lang-auto">The ordinal 961 could not be located in the dynamic link library
D:\Codes\fac_ver\cpp\port\ModelEmbeder\libtorch\lib\torch_cpu.dll
</code></pre>
<p>Why am I seeing this? I’m using Visual Studio 2019 v142 and downloaded the latest libtorch (nighlies) (the <a href="https://download.pytorch.org/libtorch/nightly/cpu/libtorch-win-shared-with-deps-debug-latest.zip" rel="nofollow noopener">debug version</a> currently)</p>
<p>Any help is greatly appreciated<br>
Thanks a lot in advance</p>

Shisho_Sama: <p>OK, Thanks to God I found out the issue.<br>
This happens becasue of a conflict between two versions of <code>libiomp5md.dll</code> . I had previously installed Anaconda so I had this , when I added the libtorch , there is a nother version there. so since Anaconda was in the <code>PATH</code> variables higher than my <code>libtorch</code> lib folder! it always tried to use the anacondas and it’d fail with that error.<br>
moving the lib higher than anaconda fixed this for me !</p>
Gradient checkpointing conflict with `no_grad()`

R0dluvan: <p>I have a model that uses gradient checkpointing (1.7.0.dev20200709) in its <code>forward</code> method.<br>
My training framework uses <code>with torch.no_grad():</code> during evaluation/sample generation. This triggers the <code>None of the inputs have requires_grad=True. Gradients will be None</code> warning. This is slightly annoying, but the worse thing is that it silences any subsequent, more meaningful occurrences of that warning message.</p>
<p>What is the clean way to solve this? I can make my <code>checkpoint</code> calls contingent on <code>self.training</code> since we also use <code>eval()</code> but it seems to me a well-written module should be able to be called under <code>no_grad</code> without causing warnings.<br>
Maybe there’s a way to specifically detect that you’re under <code>no_grad</code> - but if so, shouldn’t <code>checkpoint</code> suppress the warning in that case?</p>

albanD: <p>Hi,</p>
<p>The way I would do it is to make the use of <code>checkpoint</code> conditional on the fact that at least one input does require gradients. If none of them do, there is no point to use the checkpoint.</p>

R0dluvan: <p>That works, but it diminishes the value of the warning - if I make a mistake in some refactor and the inputs end up being <code>requires_grad=False</code> even for the training case, it will no longer warn me of that.</p>
<p>I don’t trust myself, so I think I’ll go with <code>self.training</code> - it may still trigger the warning incorrectly if my module is called in some different way that I’m not aware of (with <code>no_grad</code> without <code>eval()</code>), but I’d rather have too many warnings than too few.</p>
Ignore zeros or some elements in the input

sdota16: <p>Hi,</p>
<p>I am wondering is there a way with which I can give a layer in Pytorch a mask using which it can ignore certain values of the input?</p>
<p><a href="https://discuss.pytorch.org/t/masking-optimizers-for-sparse-input-data/1188">I saw a post regarding creating a custom layer to ignore zeros, however</a>, I am wondering if this functionality is prebuilt in the layers of Pytorch (say Conv2d or linear).</p>
ISTFT not available in PyTorch C++ frontend?

scarecrow1123: <p><code>at::istft</code> doesn’t seem to available in libtorch and I’m encountering this error:</p>
<p><code>error: ‘istft’ is not a member of ‘at’</code></p>
<p>I searched the <code>include</code> dir of libtorch distribution to find any declarations of the istft function, but there doesn’t seem to be any. I’m not sure if this was intentional or if I’ve missed something.</p>
<p>P.S: I’m using the <code>&lt;torch/torch.h&gt;</code>  header<br>
Version: libtorch 1.5.1</p>
<p>EDIT: This seems to be available in the latest nightly(1.7.0.dev20200727+cpu) and works fine</p>
A newbie question regarding torch tensor add

torching: <p>Perhaps a newbie question here, I checked the documentation here <a href="https://pytorch.org/docs/stable/torch.html#torch.add" rel="nofollow noopener">torch.add</a>.</p>
<p>Question:</p>
<ol>
<li>
<p>what is the star “*” argument within the function as shown in the documentation?<br>
<code>torch.add(input, other, *, alpha=1, out=None)</code></p>
</li>
<li>
<p>from the documentation, what is the meaning of below lua statement?<br>
tt means timestep, gradInput[tt][1], input[tt][2], cumreward are all torch.Tensor()</p>
</li>
</ol>
<pre><code class="lang-auto">      self.gradInput[tt][1]:add(
         input[tt][2]:squeeze(), -1, self.cumreward:select(2, shifted_tt))
</code></pre>
<p>Thank you in advance!!</p>
From_dlpack + ctypes

vadimkantorov: <p>I’m planning to have a plain C library that would return <code>struct DLManagedTensor</code> instances. I interface with it in Python with <code>ctypes</code>.</p>
<p>How would it be possible to use <code>from_dlpack(...)</code> on such ctypes DLManagedTensor objects? How do I construct a “PyCapsule” from such a <code>ctypes</code> object? Should I use <a href="https://stackoverflow.com/questions/60330422/how-to-set-the-name-argument-while-using-ctypes-pythonapi-pycapsule-new" rel="nofollow noopener">ctypes.pythonapi.PyCapsule_New</a>? Would there be any memory leaks of the capsule object themselves?</p>
<p>Thanks!</p>

vadimkantorov: <p>Yes, it seems that I would need to create the capsule manually. I found the CuPy support PR: <a href="https://github.com/cupy/cupy/pull/1082/files" rel="nofollow noopener">https://github.com/cupy/cupy/pull/1082/files</a> that does something similar on CuPy side</p>

vadimkantorov: <p>Do I understand correctly that PyTorch + DLPack implicitly assume Little Endian?</p>

vadimkantorov: <p>I made an example of reading audio with ffmpg in C (deallocation is also done in C), calling it from Python, and reading it into a PyTorch tensor via DLPack: <a href="https://github.com/vadimkantorov/readaudio" rel="nofollow noopener">https://github.com/vadimkantorov/readaudio</a></p>
<p>A simpler example of just demonstating my DLPack Python / NumPy bindings: <a href="https://github.com/vadimkantorov/pydlpack" rel="nofollow noopener">https://github.com/vadimkantorov/pydlpack</a></p>
<p>Managing memory is not very sure, as I’m not very confident about NumPy / Python array / object lifecycle. If you take a look or find problems, feel free to drop into GitHub issues!</p>
Where to do text pre-processing, inside __getitem__ or before loading dataset?

abhigenie92: <p>General question:</p>
<ol>
<li>If you want to do some pre-processing of text, would you do in before the loading of the dataset in the Dataset init method, or inside the __getitem__method?</li>
</ol>
<p>Specific questions:</p>
<ol>
<li>Say you filter some samples, say anything less than a length of x. Then you do that in <strong>getitem</strong>() method since you need to return a sample.</li>
<li>Say you are text pre-processing like sentence tokenization, if you do in getitem method, you would do everytime(whenever it is sampled again) right or probably inefficient? Or it cached?</li>
</ol>

vdw: <p>To be honest, I do the preprocessing of the training data as a completely separate step. Seems cleaner to me.</p>
Operator to convert intlist to tensor?

vdantu: <p>Is it possible to cast intlist (int []) to a tensor and vice versa in the JIT scriptmodule’s graph? Curious to know if we can insert any operator which does this for us.</p>

tom: <p>tensor, probably:</p>
<pre><code class="lang-python">@torch.jit.script 
def fn(l : List[int]): 
     return torch.tensor(l) 
fn.graph                                                                                                                                                                                                  
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">graph(%l.1 : int[]):
  %4 : bool = prim::Constant[value=0]()
  %2 : None = prim::Constant()
  %5 : Tensor = aten::tensor(%l.1, %2, %2, %4)
  return (%5)
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

vdantu: <p>Nice example. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

vdantu: <p>I was going through the source code and found the following two operators</p>
<pre><code class="lang-auto">"aten::_list_to_tensor(int[] self) -&gt; Tensor"

"aten::_tensor_to_list(Tensor self) -&gt; int[]"
</code></pre>
<p>I don’t see these used anywhere in the code itself. Can these be used to convert an int[] to Tensor and if needed a Tensor to int[]?</p>

eellison: <p>you can use::</p>
<pre><code class="lang-auto">torch.jit.script
def foo(x: torch.Tensor):
    out: List[int] = x.tolist()
    return out
</code></pre>

vdantu: <p>Were you able to script this method? I had tried this and got errors before.</p>
How to get the indices of a [batch_size, 3] matrix in dimension 0 that match a [1,3] vector

whoab: <p>How to get the indices of a [batch_size, 3] matrix in dimension 0 that match a [1,3] vector.</p>
<p>E.g.<br>
Matrix A = [[1,2,3],<br>
[4,5,6],<br>
[1,2,3]]</p>
<p>function(A, [1,2,3]) -&gt; indices [0,2]</p>

Unity05: <p>Hi <a class="mention" href="/u/whoab">@whoab</a>,</p>
<p>most likely, there is a better solution somewhere out there, but it should do the job:</p>
<pre><code class="lang-auto">torch.unique((a == torch.Tensor([1, 2, 3])).nonzero()[:, 0])
</code></pre>
<p>I hope I could help you.</p>
<p>Regards,<br>
Unity05</p>
Memory Issues with RandomSampler

kathrynchapman: <p>I am working with long documents and BERT, so I used the sliding window approach and fit each entire document into a minibatch. Since the documents are different lengths, the minibatches are different lengths. I had to make my own Dataset class and also rewrite the forward method in the DataParallel to scatter the pre-made mini batches across multiple GPUs, yada yada.</p>
<p>Anyway, I am still using the Dataloader, and when I set sampler=None (bad, I know), the model trains fully with no issue. When I use RandomSampler, I run into CUDA out of memory issues after maybe 25 steps or so.</p>
<p>I decided to just turn the train_dataloader into a list and call random.shuffle(train_dataloader) after every epoch. The model trains fine, and I specifically do not what the items inside the minibatches to randomly shuffle across minibatches.</p>
<p>My question then is, are using RandomSampler and my list/shuffle approach the same? In either case, why is using RandomSampler running into memory issues?</p>
LSTM hidden_states issue

hang_yu: <p>For LSTM in pytorch, first I figure out that the size of LSTM hidden_states is [num_layers*num_direction, batch_size, hidden_features]. So how do I extract the hidden state of the sequence? Now my understanding of doing this is as follow, is that right? I cannot find any explanatory documents about this please help me.</p>
<pre><code class="lang-auto"># For multi-layer single directional LSTM network
rnn = nn.LSTM(input_size, hidden_features, num_layers, batch_first=True, bidirectional=False);
input = initializeInput();
# input.size() = [batch_size, seq_len, hidden_features]
hidden_cell = initializeHiddenCell(); 
# hidden_cell.size() = ([num_layers*num_direction, batch_size, hidden_features], [num_layers*num_direction, batch_size, hidden_features])

# input_size == hidden_features
output, hidden_cell = rnn(input, hidden_cell);
hidden_state for current timestep input is: hidden_cell[0][-1];


# For multi-layer bidirectional LSTM network
rnn = nn.LSTM(input_size, hidden_features, num_layers, batch_first=True, bidirectional=False);
input = initializeInput();
# input.size() = [batch_size, seq_len, hidden_features]
hidden_cell = initializeHiddenCell(); 
# hidden_cell.size() = ([num_layers*num_direction, batch_size, hidden_features], [num_layers*num_direction, batch_size, hidden_features])

# input_size == hidden_features
output, hidden_cell = rnn(input, hidden_cell);
forward and backward hidden_state for current timestep input is: hidden_cell[0][-1], hidden_cell[0][-2];
</code></pre>

vdw: <p>If you look at the docs, you can resolve the <code>num_layers*num_directions</code> with</p>
<pre><code>h = h.view(num_layers, num_directions, batch, hidden_size)
</code></pre>
<p>After that, the last layer is simply <code>h[-1]</code>, the last layer for the forward pass is <code>h[-1][0]</code>, and the last layer for the backward pass is <code>h[-1][1]</code>.</p>
<p>Does this answer your question? I’m not quite if I understood what you need to know.</p>

hang_yu: <p>Exactly! That is what I am trying to figure out. Thank you so much!</p>
This is an invalid model. Graph output (1) does not exist in the graph

anico: <p>Does anyone know why this code gives the following error when loading the ONNX model?</p>
<pre><code class="lang-auto">@torch.jit.script
def bar(x):
    zero = torch.tensor(0.0, dtype=torch.float32)
    one = torch.tensor(1.0, dtype=torch.float32)
    if x.eq(zero):
        y = zero
    else:
        y = one
    return y
        

class Foo(nn.Module):
    
    def forward(self, x):
        return bar(x)
    
foo = Foo()

dummy_x = torch.tensor(0.0, dtype=torch.float32)

torch.onnx.export(foo, dummy_x, "./foo.onnx", input_names=["x"], output_names=["y"])

foo_onnx = onnxruntime.InferenceSession("./foo.onnx")
</code></pre>
<pre><code class="lang-auto">InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Failed to load model with error: D:\2\s\onnxruntime\core\graph\graph.cc:912 onnxruntime::Graph::InitializeStateFromModelFileGraphProto This is an invalid model. Graph output (1) does not exist in the graph.
</code></pre>

neginraoof: <p>I just tested this repro with PyTorch and ONNXRuntime nightly.<br>
The error I see is:<br>
onnxruntime.capi.onnxruntime_pybind11_state.InvalidGraph: [ONNXRuntimeError] : 10 : INVALID_GRAPH : This is an invalid model. Type Error: Type ‘tensor(float)’ of input parameter (x) of operator (Equal) in node (Equal_3) is invalid.</p>
<p>However, by changing dtype to torch.int in the test, this ONNXRuntime error is no longer thrown.<br>
Looks like ONNX Equal op takes float input data type, so this might be an issue with ONNXRuntime.<br>
Will follow op on this.</p>

neginraoof: <p>Actually setting opset_version=11 would fix this issue. ONNX Equal op supports float types starting from opset 11.</p>

anico: <p>Got it. The change is as follows:</p>
<pre><code class="lang-auto">torch.onnx.export(foo, dummy_x, "D:/foo.onnx", input_names=["x"], output_names=["y"], opset_version=11)
</code></pre>
<p>Now it runs correctly:</p>
<pre><code class="lang-auto">x = np.asarray(1.0, dtype=np.float32)
foo_onnx.run(["y"], {"x": x})
</code></pre>
how to increase number of images with data augmentation

Marco_De_Simone: <p>I’m trying to apply data augmentation with pytorch. In particular, I have a dataset of 150 images and I want to apply 5 transformations (horizontal flip, 3 random rotation ad vertical flip) to every single image to have 750 images, but with my code I always have 150 images.</p>
<p>‘train’: transforms.Compose([<br>
transforms.Resize(224),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.RandomRotation(degrees = (90,90)),<br>
transforms.RandomRotation(degrees = (180,180)),<br>
transforms.RandomRotation(degrees = (270,270)),<br>
transforms.RandomVerticalFlip(p=1),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
])</p>

MariosOreo: <p>It performs the transformations on fly in each iteration. So it will not increase the actual scale of your data on your disk</p>

achaiah: <p>If you want more images you need to generate them ahead of time. There are a number of libraries that can help you do this. <a href="https://github.com/albu/albumentations" rel="nofollow noopener">Albumentations</a> and <a href="https://github.com/aleju/imgaug" rel="nofollow noopener">Imgaug</a> come to mind.</p>
<p>But the greater question is what are you trying to accomplish with transformations? If you just want more random samples then train your networks for a longer number of epochs. Every epoch your dataset will be transformed differently (since you have random transformations there) so you’ll get a fresh set of 150 images for your network to chew on.</p>
<p>Lastly, you don’t want to be chaining your transforms as you’ve done above since you are doing the rotations one after another and thus potentially undoing your intended result.</p>

beitian: <p>is there any package for 3d image?</p>

rishabhramteke: <p>I also want to know how to do that. Did you find the solution?</p>
`requires_grad` becomes False inside gradient checkpointing

R0dluvan: <p>I’m using gradient checkpointing (1.7.0.dev20200709) and I’m observing a behavior that I don’t understand.</p>
<p>Basically, I have a code snippet in my <code>forward</code> that goes<br>
<code>foo = bar * baz</code><br>
where <code>bar</code> is <code>requires_grad=False</code> and <code>baz</code> is <code>requires_grad=True</code>.<br>
I call this code with the same inputs first outside, then inside a <code>checkpoint()</code> call.<br>
In the first case, <code>foo</code> ends up being <code>requires_grad=True</code> which is what I would expect.<br>
In the second, <code>foo</code> ends up being <code>requires_grad=False</code>.</p>
<p>Is this expected? Will this not cause failure of the model to learn? Have I discovered a bug, or is this <code>checkpoint</code> magic that will all turn out fine in the end?</p>

albanD: <p>Hi,</p>
<p>When you do checkpointing, the forward is called twice.<br>
Once during the forward without tracking gradients (and so requires_grad=False) and once, during the backward, with tracking enabled to recover all the buffers.</p>

R0dluvan: <p>Checkpoint magic, got it <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Speeding up huge linear layer?

Felix_Labelle: <p>Edit: Replaced an earlier post after some investigating.</p>
<p>I’ve been trying to predict a 24x32 map with 42 classes from a 256 dimensional vector. That’s about 32,000 parameters, however the hidden layers come in BATCH_SIZE x 36 groups. This has become a bottleneck in my model. What are my options for speeding this up? Are there options other than parallelizing (I have access up to 4 GPUs).</p>
<p>Here is the bottleneck with dimensions in the comments:</p>
<pre><code class="lang-auto"># visual_feat.shape = BATCH_SIZE x 36 x 256
# img_features =  BATCH_SIZE x 36 x 2048
# map_dims = 24x32
# num classes = 42
self.map_pred(visual_feat).reshape(img_features.shape[0], self.num_classes, *img_features.shape[1:-1],*self.map_dims)
</code></pre>
ROI Extraction - subregion of feature map?

soulslicer: <p>Is there a pytorch module that lets me extract a subregion of a conv feature map?</p>
<p>I have a feature map of [B, 64, 100, 100]</p>
<p>I have a set of bounding boxes as follows: [B, N, 4] (x1, y1, x2, y2 format)</p>
<p>Is there any function that lets me extract these bboxes differentiabily?</p>
How to extend model with Pad layer?

odats: <p>I have input vectors with different size: [1,0,1], [1,0,1,1,1,1]…<br>
Is there a way to add additional NN layer that Pads vectors on the right side with 2?</p>
<pre><code class="lang-auto">[1,0,1] -&gt; [1,0,1,2,2,2,2,2] 
[1,0,1,1,1,1] -&gt;  [1,0,1,1,1,1,2,2]
</code></pre>
<pre><code class="lang-auto">MAX_STRING_SIZE = 8
model = torch.nn.Sequential(
    torch.nn.Linear(MAX_STRING_SIZE, 1),
    torch.nn.Sigmoid()
)
</code></pre>

Sagnik_Mukherjee: <p><code>torch.nn.functional.</code> <code>pad</code> ( <em>input</em> ,  <em>pad</em> ,  <em>mode=‘constant’</em> ,  <em>value=2</em> )</p>
<p>you can also try</p>
<pre><code class="lang-auto">tensor1 = torch.Tensor([1,0,2])
max_seq_len = 7
tensor2 = torch.ones(max_seq_len)*2
tensor1 = torch.cat((tensor1,tensor2))[:max_seq_len]
</code></pre>
<p>change the max_seq_len to get padding till different sizes</p>

odats: <p>How to use it with batches?<br>
<code>tensor1 = torch.Tensor([[1,0,2],[1,0,2,1,0,2]])</code></p>
How to convert output of each hidden layer to local binary pattern (LBP) and feed it to next layer's input in pytorch

VahidZarghami: <p>Hi<br>
I want to convert output of each hidden layer to local binary pattern (LBP) and feed it to next layer’s input in PyTorch,<br>
In the other word I want to classify my input images (RGB or LBP) via extract the features in each layer in ResNet18!</p>
<p>Thanks.</p>
Is this minmal LSTM classifier correctly set up?

HenryPowell: <p>Hi,</p>
<p>I’ve written a basic LSTM classifier and a few more complex models based of this basic code. This basic model should take in a batch of sequences of size [200 x 128] and assign to each sequence one of 6 classes. I’m fairly sure this is correct and the model learns on the dataset. I’ve been looking around at other LSTM implementations and have seen that in some cases there are a few things included which I haven’t and I’m worried I’m missing something that could be effecting model training.</p>
<p>The basic model is as follows:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn


class Net(nn.Module):
    def __init__(self, num_classes=6, hidden_size=256, steps=200, bidirectional=False):
        super(Net, self).__init__()

        self.bidirectional = bidirectional
        self.hidden_size = hidden_size

        self.LSTM_one = nn.Sequential(
            nn.LSTM(input_size=128, hidden_size=hidden_size, num_layers=1, batch_first=True, bidirectional=bidirectional)
        )

        self.classifier = nn.Sequential(
            nn.Dropout(0.2),
            nn.Linear(in_features=steps*hidden_size, out_features=500),
            nn.ReLU(inplace=True),
            nn.Dropout(0.2),

            nn.Linear(in_features=500, out_features=num_classes)
        )

    def forward(self, x):
        x, hidden = self.LSTM_one(x)
        x = torch.flatten(x, 1)
        x = self.classifier(x)
        return x, hidden

</code></pre>
<p>I’ve seen in other places that a separate init function is called that initalizes the hidden state of the LSTM. Something like:</p>
<pre><code class="lang-python"> def init_hidden(self):
    return (torch.zeros(1 + int(self.bidirectional), 1, self.hidden_size),
    torch.zeros(1 + int(self.bidirectional), 1, self.hidden_size))
</code></pre>
<p>and further ones that pass a <code>hidden</code> argument to the <code>forward</code> method like this:</p>
<pre><code class="lang-python">  def forward(self, inputs, hidden):
    output, hidden = self.lstm(inputs.view(1, 1, self.input_size), hidden)
    return output, hidden
</code></pre>
<p>So my question is: are these two addition steps that my LSTM model is missing necessary or have they been deprecated?</p>
<p>Many thanks.</p>
Can I use 'torch.cuda.amp.autocast' with 'torch.einsum'?

Cuistiano: <p>Can I use ‘torch.cuda.amp.autocast’ with ‘torch.einsum’ ? Will this work ?</p>

smth: <p>since einsum is a fairly dynamic op, i think <code>amp.autocast</code> might keep it in the “safe” region and do the computation in fp32.</p>
<p>It should be quick to try and find out.</p>
Gradient Checkpointing returning values

sajastu: <p>I have a checkpoint callback function (i.e, custom_dec) that returns a Tensor, and a dictionary. But it seems like this function does not return dictionaries (or other data types), but only tensors. What is the workaround to this, as the module that I want to checkpoint is returning a tensor, plus a data type as dictionary:</p>
<pre><code class="lang-auto">    def custom_dec(self, module):
        def custom_forward(*inputs):
            output = module(inputs[0], inputs[1],
                            encoder_attn_mask=inputs[2],
                            decoder_padding_mask=inputs[3],
                            layer_state=inputs[4],
                            causal_mask=inputs[5],
                            output_attentions=inputs[6],
                            )
            # output[2] is a python dictionary
            return output[0], output[2]
</code></pre>
<p>The following is the checkpoint call:</p>
<pre><code class="lang-auto">x, layer_past = \
                checkpoint.checkpoint(
                    self.custom_dec(decoder_layer),
                    x,
                    encoder_hidden_states,
                    encoder_padding_mask,
                    decoder_padding_mask,
                    layer_state,
                    decoder_causal_mask,
                    output_attentions,
                )
</code></pre>
<p>The error:</p>
<blockquote>
<p>TypeError: CheckpointFunctionBackward.forward: expected Variable (got dictionary) for return value 1</p>
</blockquote>

albanD: <p>Hi,</p>
<p>Assuming this is not a differentiable output, you can return it by side effect:</p>
<pre><code class="lang-python">    def custom_dec(self, module, out_dict):
        def custom_forward(*inputs):
            output = module(inputs[0], inputs[1],
                            encoder_attn_mask=inputs[2],
                            decoder_padding_mask=inputs[3],
                            layer_state=inputs[4],
                            causal_mask=inputs[5],
                            output_attentions=inputs[6],
                            )
            # output[2] is a python dictionary
            out_dict[0] = output[2]
            return output[0]

out_dict = []
x = checkpoint.checkpoint(
                    self.custom_dec(decoder_layer, out_dict),
                    x,
                    encoder_hidden_states,
                    encoder_padding_mask,
                    decoder_padding_mask,
                    layer_state,
                    decoder_causal_mask,
                    output_attentions,
                )
layer_past = out_dict[0]</code></pre>
Indexing embedding using longtensor loses gradient_fn

spacegoing: <p>When I use a long tensor to index embedding, it loses its grad_fn:</p>
<pre><code class="lang-auto">embed_option = nn.Embedding(4, 10)
vec = embed_option(torch.LongTensor([[0,1,2]]))
</code></pre>
<p><code>vec.requires_grad</code> is <code>Flase</code>.</p>
<p>When I tried in main scope, everything worked fine. However, if I define this in a nn.Module class, it loses its gradient function. Any idea how this could happen?</p>

spacegoing: <p>fk me. I wrapped it in <code>with torch.no_grad()</code> in caller function.</p>
Type of x.size(1)

Alpha: <p>Hi,</p>
<p><code>x</code> is a <code>torch.Tensor</code> and I define <code>c=x.size(1)</code>.</p>
<p>I found that there are two possible types of <code>c</code>:</p>
<ul>
<li><code>&lt;class 'torch.Tensor'&gt;</code></li>
<li><code>&lt;class 'int'&gt;</code></li>
</ul>
<p>How can I get my desired type <code>&lt;class 'int'&gt;</code> rather than  <code>&lt;class 'torch.Tensor'&gt;</code>?</p>

vainaijr: <p>if</p>
<pre><code class="lang-auto">x = torch.randn(5, 5)
</code></pre>
<p>then</p>
<pre><code class="lang-auto">type(x.size(1))
</code></pre>
<p>would return int, how do you get torch.Tensor?</p>

Alpha: <p>Thank you for your reply, but I do get the torch.Tensor.</p>
<p>Here is part of my code:</p>
<pre><code class="lang-auto">    def forward(self, x):
        # Explicitly forward every layer.
        # Branch2a.
        in_channel = x.size(1)
        l_size = list(x.size())
        import torch
        fake = torch.ones(3,4,5).cuda()
        type(fake.size(2))
        import pdb; pdb.set_trace()
</code></pre>

vainaijr: <p>it might be for x, for example,</p>
<pre><code class="lang-auto">class A(nn.Module):
  def __init__(self):
    super().__init__()
    self.a = 10
  def forward(self, x):
    print('x type:', type(x))
    # Explicitly forward every layer.
    # Branch2a.
    in_channel = x.size(1)
    print('in_channel type:', type(in_channel))
    l_size = list(x.size())
    print('l_size type:', type(l_size))
    import torch
    fake = torch.ones(3,4,5).cuda()
    print('fake type:', type(fake.size(2)))
    # import pdb; pdb.set_trace()
    return x
</code></pre>
<pre><code class="lang-auto">net = A()
net(torch.randn(1, 2, 3))
</code></pre>
<p>would return</p>
<pre><code class="lang-auto">x type: &lt;class 'torch.Tensor'&gt;
in_channel type: &lt;class 'int'&gt;
l_size type: &lt;class 'list'&gt;
fake type: &lt;class 'int'&gt;
tensor([[[-1.2244, -1.2652, -0.6877],
         [ 1.5777, -1.5478,  1.1681]]])
</code></pre>

Alpha: <p>Thanks very much for your kind reply.</p>
<p>After debugging, I found my problem is resulted from the <a href="https://github.com/facebookresearch/fvcore/blob/master/fvcore/nn/flop_count.py" rel="nofollow noopener">fvcore.nn.flop_count.flop_count</a>.</p>
<p>During the forward in above function, the type will be <code>&lt;class 'torch.Tensor'&gt;</code></p>
Determining last linear layer dimenions automatically

Rahul_Seetharaman: <pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self,in_channels=1,num_classes=10):
        super(CNN,self).__init__()
        self.conv1=nn.Conv2d(
            in_channels=1,
            out_channels=8,
            kernel_size=(3,3),
            stride=(1,1),
            padding=(1,1))
        self.pool=nn.MaxPool2d(kernel_size=(2,2),stride=(2,2))
        self.conv2=nn.Conv2d(
            in_channels=8,
            out_channels=16,
            kernel_size=(3,3),
            stride=(1,1),
            padding=(1,1))
        self.num_classes=num_classes
        self.fc1=nn.Linear(16*7*7,num_classes)
    def forward(self,X):
        X=F.relu(self.conv1(X))
        X=self.pool(X)
        X=F.relu(self.conv2(X))
        X=self.pool(X)
        X=X.reshape(X.shape[0],-1)
        print(X.shape)
        self.fc1=nn.Linear(X.shape[1],self.num_classes).to(device=device)
        X=self.fc1(X)
        return X
model=CNN().to(device=device)
x=torch.randn(64,1,28,28).to(device=device)
input_size=784
num_classes=10
learning_rate=0.001
batch_size=64
num_epochs=1

train_dataset=datasets.MNIST(root="datasets/",train=True,transform=transforms.ToTensor(),download=True)
train_loader=DataLoader(dataset=train_dataset,batch_size=batch_size,shuffle=True)
test_dataset=datasets.MNIST(root="datasets/",train=False,transform=transforms.ToTensor(),download=True)
test_loader=DataLoader(dataset=test_dataset,batch_size=batch_size,shuffle=True)
criterion=nn.CrossEntropyLoss()
optimizer=optim.Adam(model.parameters(),lr=learning_rate)

for epoch in range(num_epochs):
    for batch_idx,(data,targets) in enumerate(train_loader):
        data=data.to(device=device)
        targets=targets.to(device=device)
        # forward propagation
        scores=model(data)
        loss=criterion(scores,targets)
        # backward propagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>I have written the following code for MNIST classification. This gives really poor accuracy of 0.09.<br>
But when I define the last Linear layer inside the constructor instead by giving dimensions 16 X 7 X 7 , then the accuracy is 0.96. I felt it was rather strange. What could be the reason for this. ?</p>

Unity05: <p>Hi <a class="mention" href="/u/rahul_seetharaman">@Rahul_Seetharaman</a>,</p>
<p>If I’ve understood your question correctly, you first defined the linear layer inside the forward() method, and after seeing the bad accuracy, you defined it in the constructor, what gave you a much better result?<br>
If yes, everything that has trainable parameters should be defined in the constructor. By defining it in the forward method, you define your <code>self.fc1</code> new every iteration, therefore training the layer will not make any difference as you override it in the next iteration.<br>
I hope I could help you.</p>
<p>Regards,<br>
Unity05</p>
Bidirectional RNN Implementation

sgaur: <p>Hi I am trying to understand bidirectional RNN.</p>
<pre><code class="lang-auto">&gt; class RNN(nn.Module):
&gt; 

&gt; 
&gt;   def __init__(self,n_vocab,n_embed,hidden_size,output_size):
&gt; 
&gt;     super().__init__()
&gt; 
&gt;     self.hidden_size = hidden_size
&gt; 
&gt;     self.embedding = nn.Embedding(n_vocab+1,n_embed) ## n_vocab is unique words in dictionary ## n_embed is hyperparameter

&gt;     self.rnn = nn.RNN(n_embed, hidden_size, num_layers = 1, batch_first = True,bidirectional = True) # 
&gt; 
&gt;     self.fc =  nn.Linear(hidden_size,output_size)   
&gt; 
&gt;   def forward(self,x):&gt;
&gt; 
&gt;     x = x # input batch_size * seq_length
&gt; 
&gt;     batch_size = x.size(0) 
&gt; 
&gt;     #print('Batch Size is',batch_size)
&gt; 
&gt;     x = self.embedding(x) # batch-size x seq_length x embedding_dimension
&gt; 
&gt;     x,hidden =self.rnn(x)   #batch-size x seq_length x hidden_size
&gt; 
&gt; 
&gt; 
&gt;     return x,hidden
</code></pre>
<p>I am returning both hidden state and output while going through tutorials some says that I need to concatenate hidden state (<strong>torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1</strong>)) and in some tutorials take output state (<strong>x[:,-1,:]</strong>) but both of results come difference.</p>
<p>What is the correct way of doing Bidirectional RNN.</p>
Question regarding arXiv submission

vainaijr: <p>What do I do if my paper is on hold for entire eternity, plus do I need to mention my implementation in the arXiv paper?<br>
Plus who reviews the arXiv paper?<br>
Plus is the use of LaTeX a necessity, or is a word file converted to pdf fine?</p>

TinfoilHat0: <p>Uh wrong forum man? you’d better email arxiv moderators.</p>

vainaijr: <p>ok, delete your post then i delete the question</p>
Are the outputs of Dropout2d scaled?

b1d4fc9f712b67d006e1: <p>The outputs of Dropout are scaled by 1 / (1 - p) to make the layer indentity during testing. Is it same case for Dropout2d?</p>

ptrblck: <p>Yes, seems to be the case as seen here:</p>
<pre><code class="lang-python">drop = nn.Dropout2d()
x = torch.ones(2, 2, 2, 2)
out = drop(x)
print(out)
&gt; tensor([[[[0., 0.],
          [0., 0.]],

         [[0., 0.],
          [0., 0.]]],


        [[[2., 2.],
          [2., 2.]],

         [[0., 0.],
          [0., 0.]]]])
</code></pre>

b1d4fc9f712b67d006e1: <p>Thanks. So stupied of me not running some code .</p>
The last layer of model

skyraker: <p>Hi,<br>
The output of my model is single channel image, and the last layer is a conv. Do I need to add other layer sunch as BN and ReLU after the last layer?</p>

JuanFMontesinos: <p>Well it depends on several things, the loss, what do you want to predict etcetera…<br>
If you want to predict RGB values then it can be nice and you remove negative values this way. If you want to predict audio between -1 and 1  that would be impossible using a ReLu.<br>
The last activation may be something different from ReLu depending on your output. Even no activation.</p>
how to self-define the autograd to realize the CNN + SVRG

yujun: <p>Hi , all.<br>
I want modify the way of grandient update.<br>
After the CNN model is setted, when it comes to training, we need to update the gradient by the following algorithm in picture<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5dceb41be96e9e47159fac445ce3af454579fb1f.png" data-download-href="https://discuss.pytorch.org/uploads/default/5dceb41be96e9e47159fac445ce3af454579fb1f" title="algorithm of SVRG"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dceb41be96e9e47159fac445ce3af454579fb1f_2_690x273.png" alt="algorithm of SVRG" data-base62-sha1="dnRgqUUMRSN9UwZPli7Z07oh65V" width="690" height="273" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dceb41be96e9e47159fac445ce3af454579fb1f_2_690x273.png, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5dceb41be96e9e47159fac445ce3af454579fb1f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5dceb41be96e9e47159fac445ce3af454579fb1f.png 2x" data-dominant-color="F5F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">algorithm of SVRG</span><span class="informations">1020×404 81.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So, I don’t know how to calculate the average gradient of all samples under the last period’s parameter, and how to transfer the gradient (the fomula w_t  in the Line 9 of the picture ) to the model</p>
<p>the follow is my code and it remains to be modified.</p>
<pre><code class="lang-auto"># 主要的训练过程
for epoch in range(n_epoch) :
    for data in img_dataloader :
        img = data
        img = img.cpu()  # 读入到gpu
        output1, output = model(img)  # 编码结果、解码结果
        loss = criterion(output, img)  # MSE
        optimizer.zero_grad()  # 梯度置零
        loss.backward()  # 梯度回传
        #after calculate the ori grad and before update the next parameter, we need to amend the gradient
        #get current grad
        #amended grad = current_grad - mu_grad
        #but I dont know how to transfer the amended grad to model
        optimizer.step()  # 更新参数
        #there maybe need to save the last period's gradient but I dont know how to do
        #parameter = last paramater
        #mu_grad = calcultegrad(sample,last_parameter).mean()
</code></pre>
Train Graph Neural Network model by optimizing edge features in PyTorch Geometric

Udith_Haputhanthri: <p>I am new to the field of Graph Representation Learning. I am trying to train the GNN by optimizing only the edge features without optimizing the node feature vectors. Basically I want to obtain <strong>a set of edge feature vectors that minimizing the loss function</strong>. I was not able to find out a way to do this in the geometric documentation. So I want to know whether there is a way to achieve this.</p>
Speech embedding LSTM model giving different results during inference(I am using model.eval() )

ashu170292: <p>I am using an LSTM model with 3 hidden layers to produce speech embeddings. The input to the model are MFCC features taken on a window of 0.025 seconds and the hop length is 0.01 seconds. I have two inputs for inference and their size is as follows:</p>
<ol>
<li>torch.Size([54, 24, 40])</li>
<li>torch.Size([439, 24, 40])</li>
</ol>
<p>The first 18  frames in both the inputs is same. I load the model as follows:</p>
<pre><code class="lang-auto">embedder_net = SpeechEmbedder()
embedder_net.load_state_dict(torch.load(hp.model.model_path))
embedder_net.eval()
</code></pre>
<p>The state_dict looks like following:</p>
<pre><code class="lang-auto">LSTM_stack.weight_hh_l0 	 torch.Size([3072, 768])
LSTM_stack.bias_ih_l0 	 torch.Size([3072])
LSTM_stack.bias_hh_l0 	 torch.Size([3072])
LSTM_stack.weight_ih_l1 	 torch.Size([3072, 768])
LSTM_stack.weight_hh_l1 	 torch.Size([3072, 768])
LSTM_stack.bias_ih_l1 	 torch.Size([3072])
LSTM_stack.bias_hh_l1 	 torch.Size([3072])
LSTM_stack.weight_ih_l2 	 torch.Size([3072, 768])
LSTM_stack.weight_hh_l2 	 torch.Size([3072, 768])
LSTM_stack.bias_ih_l2 	 torch.Size([3072])
LSTM_stack.bias_hh_l2 	 torch.Size([3072])
projection.weight 	 torch.Size([256, 768])
projection.bias 	 torch.Size([256])

</code></pre>
<p>When I use the model for inference on the two inputs, I get results of the following size:</p>
<ol>
<li>torch.Size([54, 256])</li>
<li>torch.Size([439, 256])</li>
</ol>
<p>Since the first 18 frames are same in both the inputs, I expect the first 18 embeddings in both results to be the same. But this is not the case.</p>
<p>In fact the results in both cases are very different from the result I get when I just take frames 18 frames as the  input. Any idea why that would happen?</p>
<p>Help appreciated</p>
Error when build cpp extension: expected constructor, destructor?

NguyenVanThanhHust: <p>I’m not very good at C++. I’m trying to follow this <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">cpp_extension_tutorial</a>.</p>
<p>This is my cpp_file: <a href="https://github.com/NguyenVanThanhHust/DeepLearning/blob/master/PytorchTutorial/Cpp_Extension/lltm.cpp" rel="nofollow noopener">lltm.cpp</a><br>
and here is my <a href="https://github.com/NguyenVanThanhHust/DeepLearning/blob/master/PytorchTutorial/Cpp_Extension/setup.py" rel="nofollow noopener">setup.py</a></p>
<p>I can’t compile code. Here is error when i compile:</p>
<pre><code class="lang-auto">running install
running bdist_egg
running egg_info
writing lltm_cpp.egg-info/PKG-INFO
writing dependency_links to lltm_cpp.egg-info/dependency_links.txt
writing top-level names to lltm_cpp.egg-info/top_level.txt
reading manifest file 'lltm_cpp.egg-info/SOURCES.txt'
writing manifest file 'lltm_cpp.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'lltm_cpp' extension
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.7/dist-packages/torch/include -I/usr/local/lib/python3.7/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/lib/python3.7/dist-packages/torch/include/TH -I/usr/local/lib/python3.7/dist-packages/torch/include/THC -I/usr/include/python3.7m -c lltm.cpp -o build/temp.linux-x86_64-3.7/lltm.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=lltm_cpp -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
lltm.cpp:1:9: warning: #pragma once in main file
 #pragma once
         ^
lltm.cpp:95:17: error: expected constructor, destructor, or type conversion before ‘(’ token
 PYBNIND11_MODULE("lltm_cpp", m)
                 ^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

</code></pre>
<p>What is wrong with this code? How can i fix it?</p>
What does cudnn.fastest = True work?

GB_K: <p>Hello.</p>
<p>I saw the code</p>
<pre><code class="lang-python">cudnn.fastest = True
</code></pre>
<p>in the repo. <a href="https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch" rel="nofollow noopener">https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch</a>.</p>
<p>What is the meaning of cudnn.fastest?</p>
<p>Thank you for your answer in advance.</p>

Shisho_Sama: <p>It just signals the Pytorch to use the fastest implementation available for operations such as Convolution etc. when enabled, they usually consume more memory (that is cudnn.benchmark and cudnn.fastest)</p>

ptrblck: <p>It shouldn’t have any effect, as <code>torch.backends.cudnn.fastest</code> is not implemented.<br>
In your IDE or <a href="https://github.com/pytorch/pytorch/blob/b7bda236d18815052378c88081f64935427d7716/torch/backends/cudnn/__init__.py#L86-L93">here</a> you can see, that <code>benchmark</code>, <code>deterministic</code>, and <code>enabled</code> are known flags.</p>
<p><code>benchmark = True</code> will use <code>cudnnFind</code> to profile all available Kernels and select the fastest one.</p>
<p>The repository seems to only set this new argument without using it in the code.</p>

GB_K: <p>Thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a>.<br>
So it means that torch.backends.cudnn.fastest can be deleted in my coding set. right?</p>

ptrblck: <p>You should be able to delete it, as I cannot find any usage of it.<br>
Since the PyTorch backends doesn’t use this flag, I searched for any usage in the repository, but couldn’t find it either.</p>
<p>I’m not familiar with the repository, but I couldn’t find any “custom PyTorch” installation, which would use this flag.</p>

GB_K: <p>Thank you so much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>Anyway, let me know, if you see any difference after removing the flag.<br>
Also, <a class="mention" href="/u/shisho_sama">@Shisho_Sama</a> seems to have seen it before, so it would be interesting where it was used before.</p>

Shisho_Sama: <p>Actually I guess I mistook torch with Pytorch and though this is available here as well (as cudnn.benchmark is!) . This was the case for torch back in the day if I’m not mistaken.<br>
Since Pytorch released I have only used <code>cudnn.benchmark</code> myself and never used the <code>cudnn.fastest</code> as I remember benchmark handled everything by itself . I have never seen cudnn.fastes in any official examples and chances are those repos that do use cudnn.fastes, do it as a habit comming from torch!</p>
feature fusion

111186: <p>I want to use Feature Fusion to improve the VGG19’s performance in classification. my  plan is to extract the Conv layers 8，12, 16 and use the torch.cat()  to imply the feature fusion. but i don’t konw how to change my code to finish it. this is my code, it will be very grateful to tell me how to do it.</p>
<pre><code class="lang-auto">class VGG(nn.Module):
    def __init__(self, vgg_name):
        super(VGG, self).__init__()
        self.features = self._make_layers(cfg[vgg_name])
        self.classifier = nn.Linear(512, 7)

    def forward(self, x): 
        out = self.features(x)
        #print("out:-&gt;", out)
        out = out.view(out.size(0), -1)
        out = F.dropout(out, p=0.5, training=self.training)
        out = self.classifier(out)
        return out

    def _make_layers(self, cfg):
        layers = []
        in_channels = 3
        for x in cfg:
            if x == 'M':#64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M'
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
            else:
                layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
                           nn.BatchNorm2d(x),
                           nn.ReLU(inplace=True)]
                in_channels = x
        layers += [nn.AvgPool2d(kernel_size=1, stride=1)]
        return nn.Sequential(*layers)
</code></pre>