akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Multilabel classification with weighted labels

henryvdv: <p>Hello everyone,</p>
<p>I’m currently trying to build a classifier for a multilabel classification where multiple annotators annotated one label each.<br>
Let’s say the label vectors for annotators 1, 2, and 3 look like this:<br>
A1: [1, 0, 0, 0]<br>
A2: [1, 0, 0, 0]<br>
A3: [0, 0, 0, 1]</p>
<p>I could only use the majority label and use [1, 0, 0, 0] as my label vector - or I could make it a multi-label classification problem with [1, 0, 0, 1] as my labels.<br>
But is there a way to weight the loss based on the distribution of labels?<br>
For instance, a loss function that could use [0.66, 0, 0, 0.33] as the label vector?</p>
<p>Thanks!</p>
How to use orch.autograd.Function to change the gradient

isalirezag: <p>I have a huge network (e.g. resnet)<br>
I like to change every gradient for each function which is computed via  back-propagation.<br>
Let say I like to add a constant <code>C</code> to it.<br>
can someone help me to understand how via <code>torch.autograd.Function</code> and <code>aaply</code> i can do it?<br>
I think my function should be like this?</p>
<pre><code class="lang-auto">class Change(torch.autograd.Function):
	
    @staticmethod
    def forward(ctx, input,C):
		
		ctx.save_for_backward(input)
        return input

    @staticmethod
    def backward(ctx, grad_output):
		
        input, = ctx.saved_tensors # why we should do input, not intpu?
        grad_input = grad_output.clone()
        
		
        return grad_input+C
</code></pre>
<p>I am not sure where should i put/apply it though</p>

spanev: <p>Hi <a class="mention" href="/u/isalirezag">@isalirezag</a>,</p>
<p>Do you want to apply it during back-propagation or after?</p>
<p>In the first case, you should be able to achieve it with a <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.register_backward_hook" rel="nofollow noopener">backward hook</a>:</p>
<pre><code class="lang-auto">def hook_fn(module, grad_input, grad_output):
    if isinstance(grad_input, (list, tuple)):
        return tuple((grad + C if not grad is None else None for grad in grad_input))
    elif isinstance(grad_input, torch.Tensor):
        return grad_input + C

# Add the backward hook on all the layers
for l in rn.modules():
    l.register_backward_hook(hook_fn)
</code></pre>
<p>in the second case, you can do this with:</p>
<pre><code class="lang-auto">for p in model.parameters():
    if p.requires_grad:
        p.grad.data = p.grad.data + C
</code></pre>
Efficient implementation using smart indexing

agadetsky: <p>Hello, everyone!<br>
Is it possible to implement following function in vectorized fashion, without for cycles, using advanced indexing?</p>
<pre><code class="lang-auto">def dumb_foo(x, permutation):
    assert x.ndimension() == permutation.ndimension()
    ret = torch.zeros_like(x)
    if x.ndimension() == 1:
        ret = x[permutation]
    elif x.ndimension() == 2:
        for i in range(x.size(0)):
            ret[i] = x[permutation[i]]
    elif x.ndimension() == 3:
        for i in range(x.size(0)):
            for j in range(x.size(1)):
                ret[i, j] = x[i, j, permutation[i, j]]
    else:
        ValueError("Only 3 dimensions maximum")
    return ret
</code></pre>

agadetsky: <p>Now I end up with something like this. Can it be implemented in more efficient way?</p>
<pre><code class="lang-auto">def smart_foo(x, permutation):
    assert x.ndimension() == permutation.ndimension()
    if x.ndimension() == 1:
        ret = x[permutation]
    elif x.ndimension() == 2:
        d1, d2 = x.size()
        ret = x[
            torch.arange(d1).unsqueeze(1).repeat((1, d2)).flatten(),
            permutation.flatten()
        ].view(d1, d2)
    elif x.ndimension() == 3:
        d1, d2, d3 = x.size()
        ret = x[
            torch.arange(d1).unsqueeze(1).repeat((1, d2 * d3)).flatten(),
            torch.arange(d2).unsqueeze(1).repeat((1, d3)).flatten().unsqueeze(0).repeat((1, d1)).flatten(),
            permutation.flatten()
        ].view(d1, d2, d3)
    else:
        ValueError("Only 3 dimensions maximum")
    return ret
</code></pre>
Which dataset has been used to pre-train the vision models?

eFroD: <p>Hello,</p>
<p>in order to evaluate the performance of different models (not only pre-trained models),  I want to use datasets from ImageNet.<br>
The problem is, that I want to go sure, that the images I present, are ‘new’ to the model in order to control the bias but it could be very likely, that popular datasets were used to train the models.<br>
The papers related to the models only mention the datasets used to benchmark the models.<br>
Is there a way to get informations about how these networks have been trained?</p>
<p>Thank you very much in advance.</p>

ptrblck: <p>The classification models were pretrained on ImageNet, while the segmentation and detection models use e.g. COCO. The <a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="nofollow noopener">pretrained</a> argument gives you information about the used dataset.</p>
Running ONNX model with the Caffe2 backend

Luca_Pamparana: <p>I am trying to upgrade my existing pytorch 0.4 model to 1.0 and am attempting to use the Caffe2 backend to run the models in production on the GPU.</p>
<p>So, what I did is as follows:</p>
<pre><code class="lang-auto"># Export my model to ONNX
torch.onnx._export(model, args, "test.pnnx", export_params=True)

import caffe2.python.onnx.backend as onnx_caffe2_backend
# Load the ONNX model from file.
model = onnx.load("test.onnx")
# We will run our model on the GPU with ID 3.
rep = onnx_caffe2_backend.prepare(model, device="CUDA:3")

outputs = rep.run(np.random.randn(1, 3, 128, 64).astype(np.float32))
</code></pre>
<p>Now, I have a couple of questions about this:</p>
<p>1: What if my input data already resides on the GPU? How can I pass that data to the model rather than moving it to CPU with numpy and then passing it to the executor? I tried tjhe following:</p>
<pre><code class="lang-auto">args = torch.randn(1, 3, 128, 64, dtype=torch.float32).cuda(3)
print(args.dtype)
outputs = rep.run(args)
</code></pre>
<p>This prints <code>torch.float32</code>. However, I get the error:</p>
<pre><code class="lang-auto">if arr.dtype == np.dtype('float64'):
TypeError: data type not understood
</code></pre>
<p>I am not sure why the array is being interpreted as a <code>double</code> array.</p>
<p>2: I noticed that the call to the <code>prepare</code> is rather slow. So, it seems my old pytorch code is faster than running it on the backend. I will do more exhaustive timing comparisons but is this the right way to export the model and have it running on the GPU with pytorch/onnx/caffe?<br>
So, regarding this point. If I call <code>prepare</code> without the GPU option, the call is fast but specifying GPU with <code>onnx_caffe2_backend.prepare(model, device="CUDA:3")</code> is very slow.</p>
<p>My system is using</p>
<ul>
<li>python 3.6.8</li>
<li>pytorch 1.0.0</li>
<li>onnx 1.3.0</li>
<li>ubuntu 16.04</li>
<li>cuda 9.0</li>
</ul>

Mdhvince: <p>Hi,<br>
Have you fixed this ? I have the same problem, very slow when using caffe2 backend on GPU. And I think it’s because the input is on CPU <code>.numpy()</code>, so the question can be how to move the input on gpu in order to use it with caffe2 backend.</p>
<p>with a similar code I have this result about inference speed:<br>
pytorch CPU (3s) &gt; onnx/caffe2 backend GPU (600ms) &gt; onnx/caffe2 backend CPU (200ms)&gt; pytorch GPU (50ms).</p>
<p>I would expect onnx/caffe2 backend GPU to have a better speed but it is not the case.</p>
Unexpected nn.Linear behaviour

mike-shvets: <p>Hi, is nn.Linear expected to have that large relative difference when running on a 3d tensor, as opposed to looping through the dimension? Interestingly, the latter matches to numpy matrix multiplication, as shown below.</p>
<pre><code class="lang-python">import torch
import numpy as np

fc = torch.nn.Linear(256, 128)

inp = torch.rand(3, 10, 256)

out1 = fc(inp).detach().numpy()

out2 = []
for i in range(3):
    out2.append(fc(inp[i]))
out2 = torch.stack(out2).detach().numpy()

w = fc.weight.detach().numpy()
b = fc.bias.detach().numpy()
out3 = inp.numpy() @ w.T + b 

# passes this line
np.testing.assert_allclose(out3, out2)
# fails here
np.testing.assert_allclose(out3, out1)
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "tmp.py", line 47, in &lt;module&gt;
    np.testing.assert_allclose(out3, out1)

AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatch: 78.8%
Max absolute difference: 5.9604645e-07
Max relative difference: 0.04850746
 x: array([[[-3.184696e-01,  4.671749e-01, -3.306221e-01, ...,
         -3.613108e-01,  3.210519e-01, -4.924317e-01],
        [-5.997717e-06,  7.380165e-02,  6.725912e-02, ...,...
 y: array([[[-3.184695e-01,  4.671748e-01, -3.306221e-01, ...,
         -3.613108e-01,  3.210520e-01, -4.924318e-01],
        [-5.986542e-06,  7.380170e-02,  6.725915e-02, ...,...
</code></pre>

ptrblck: <p>The difference is most likely created by the limited precision using FP32.<br>
If you use <code>DoubleTensors</code>, your difference should be smaller:</p>
<pre><code class="lang-python">fc = torch.nn.Linear(256, 128).double()
inp = torch.rand(3, 10, 256).double()
</code></pre>
Custom Optimizer in PyTorch

artix41: <p>For a  project that I have started to build in PyTorch, I would need to implement my own descent algorithm (a custom optimizer different from RMSProp, Adam, etc.). In TensorFlow, it seems to be possible to do so (<a href="https://towardsdatascience.com/custom-optimizer-in-tensorflow-d5b41f75644a" rel="nofollow noopener">https://towardsdatascience.com/custom-optimizer-in-tensorflow-d5b41f75644a</a>) and I would like to know if it was also the case in PyTorch.</p>
<p>I have tried to do it by simply adding my descent vector to the leaf variable, but PyTorch didn’t agree:  “a leaf Variable that requires grad has been used in an in-place operation.”. When I don’t do the operation in-place, the “new” variable loses its position of leaf, so it doesn’t work neither…</p>
<p>Is there an easy way to create such a custom optimizer in PyTorch?</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Latope2-150: <p>You may want to look at that post: <a href="https://discuss.pytorch.org/t/regarding-implementation-of-optimization-algorithm/20920">Regarding implementation of optimization algorithm</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

artix41: <p>Thanks for pointing it out, I will try to do what they suggest <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

tux: <p>Hi Artix !</p>
<p>You can write your own update function for sure !<br>
To update your weights, you might use the optimiser library. But you<br>
can also do it yourself. For example, you can basically code the<br>
gradient descent, the SGD or Adam using the following code.</p>
<blockquote>
<p>net = NN()<br>
learning_rate = 0.01<br>
for param in net.parameters():<br>
weight_update = smth_with_good_dimensions<br>
param.data.sub_(weight_update * learning_rate)</p>
</blockquote>
<p>As you can see, you have access to your parameters in net.parameters(), so you can update them like you want.</p>
<p>If you want more specific examples, you can go here where I implemented both SVRG and SAGA : <a href="https://github.com/kilianFatras/variance_reduced_neural_networks" class="inline-onebox" rel="noopener nofollow ugc">GitHub - kilianFatras/variance_reduced_neural_networks: Implementation of SVRG and SAGA optimization algorithms for deep learning topics.</a> (variance reduced algorithmes) ! If you have any further question, do not hesitate <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

artix41: <p>Thank you so much, that’s exactly what I needed <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=5" title=":grinning:" class="emoji" alt=":grinning:"> !</p>

111150: <p>Is these code is useful for image classification also?</p>
Different outputs for `torch.pdist` between gpu and cpu

wandering007: <p>The code snippet is shown as follows:</p>
<pre><code class="lang-auto">a = torch.randn(100, 20)
b = torch.pdist(a)
c = torch.pdist(a.cuda()).cpu()
print(torch.sum(torch.abs(b - c)))  # tensor(0.0007)
</code></pre>
<p>The output difference is quite large between gpu and cpu computation. What’s the cause of it?</p>
No grad accumulator for a saved leaf! error for layernorm script

asdfqwer: <p>Hi,<br>
I tried the script from [<a href="https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/" rel="nofollow noopener">https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/</a>](Optimizing CUDA Recurrent Neural Networks with TorchScript), and tried to do backward, simply add “out.sum().backward()” below [<a href="https://github.com/pytorch/pytorch/blob/cbcb2b5ad767622cf5ec04263018609bde3c974a/benchmarks/fastrnns/custom_lstms.py#L454" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/cbcb2b5ad767622cf5ec04263018609bde3c974a/benchmarks/fastrnns/custom_lstms.py#L454</a>], I got an error “No grad accumulator for a saved leaf!”.<br>
Anyone knows how to solve it?<br>
My pytorch version: 1.1.0 installed from conda<br>
Thanks.</p>
Regarding resizing with black bars at top and bottom

vainaijr: <p>how do I resize an image which is rectangle in shape, let us assume 40x60 to a square image, let us assume 30x30, without stretching it, and have black bars at top and bottom instead of stretch?</p>

dsuess: <p>Have a look <a href="https://github.com/ultralytics/yolov3/blob/master/utils/datasets.py#L489" rel="nofollow noopener">here</a>. It’s even doing a bit more than that, but you should be able to find the relevant bits.</p>
Updating frozen parameters / Fine Tuning

haffybra: <p>Hi,</p>
<p>This question is related to a question that came up to me after reading the official tutorial on fine tuning.</p>
<p>Suppose we follow this tutorial desiring only the <strong>feature extraction</strong> from the convolutional layers and updating the weights of the fully-connected layer.</p>
<p>By that, if we only freeze the freeze the weights on the convolutional layers with <code>param.requires_grad = False</code> and only the fully-connected layer will have <code>param.requires_grad = True</code>, as seen <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html#set-model-parameters-requires-grad-attribute" rel="nofollow noopener">here</a>.</p>
<p>But <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html#create-the-optimizer" rel="nofollow noopener">here</a> the tutorial “forces” the weight to be updated only in the list of parameters that <code>requires_grad == True</code>. Is this really necessary, once we froze the weights in all layers but the fully-connected? Couldn’t we simply have:</p>
<p><code>optimizer_ft = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)</code></p>
<p>I appreciate all comments on that.</p>

ptrblck: <p>It’s just to prevent “accidental” updates (e.g. by weight decay, if these parameters had valid gradients before) and will just skip there parameters in the update loop.</p>
<p>Have a look at <a href="https://discuss.pytorch.org/t/finetuning-torchvision-models-tutorial/55475/5">this topic</a> for more information.</p>

haffybra: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you for the clarification. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to freeze layer update?

wonchulSon: <p>I want to freeze several CNN layer,<br>
how can I do?</p>

DoubtWang: <p>1: optimizer does not include the parameters of CNN layer.<br>
2: set the gard to 0 before updating model parameters or set requires_grad=False for the parameters of CNN  before model training.</p>

wonchulSon: <p><a class="mention" href="/u/doubtwang">@DoubtWang</a><br>
Thank you for help.<br>
But can you show me the example code…?<br>
I’m beginner^^</p>

spanev: <p>Hi <a class="mention" href="/u/wonchulson">@wonchulSon</a> ,</p>
<p>Please find two threads about it here:</p><aside class="quote" data-post="7" data-topic="7088">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vmirly1/48/3309_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088/7">How the pytorch freeze network in some layers, only the rest of the training?</a> 
  </div>
  <blockquote>
    Setting .requires_grad = False should work for convolution and FC layers. But how about networks that have instanceNormalization? Is setting .requires_grad = False enough for normalization layers too?
  </blockquote>
</aside>

<aside class="quote quote-modified" data-post="2" data-topic="26714">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kelam_goutam/48/21842_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/correct-way-to-freeze-layers/26714/2">Correct way to freeze layers</a> 
  </div>
  <blockquote>
    I would like to do it the following way - 
# we want to freeze the fc2 layer this time: only train fc1 and fc3
net.fc2.weight.requires_grad = False
net.fc2.bias.requires_grad = False

# passing only those parameters that explicitly requires grad
optimizer = optim.Adam(filter(lambda p: p.requires_grad, net.parameters()), lr=0.1)

# then do the normal execution of loss calculation and backward propagation

#  unfreezing the fc2 layer for extra tuning if needed
net.fc2.weight.requires_grad = True
n…
  </blockquote>
</aside>

<p>The second one is doing exactly what <a class="mention" href="/u/doubtwang">@DoubtWang</a> is suggesting above. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>
<p>Let us know if you need more details about it!</p>

DoubtWang: <p>Hi, <a class="mention" href="/u/wonchulson">@wonchulSon</a> ,<br>
I believe u can find some example and code in these threads given by <a class="mention" href="/u/spanev">@spanev</a>.</p>
Will model.eval() sets all the networks in the model to evaluation mode?

Weize: <p>If I have</p>
<pre><code class="lang-auto">class A(nn.Module)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">class B(nn.Module):
    def __init__(self):
        ....
        self.a_network = A()
</code></pre>
<p>Will <code>B.eval()</code> also set <code>a_network</code> to evaluation mode?</p>
<p>Thanks!</p>

DoubtWang: <blockquote>
<p>Sets the module in evaluation mode.</p>
</blockquote>
<p>I think this function is also effective for sub-modules.  In my previous research, there was a similar operation.  WHY do you doubt it ?</p>
Mutrix multiplication in tensor 4D

isalirezag: <p>if i have <code>A=torch.rand(2,3,5,6)</code> and <code>B = torch.rand(2,3,6,7)</code> how I can do multiplication in a way that my output <code>C</code> has <code>2x3x5x7</code> dimension. in other words, i   do matrix multiplication in a way that</p>
<pre><code class="lang-auto">C[0,0,:,:] = A[0,0,:,:]xB[0,0,:,:]
C[0,1,:,:] = A[0,1,:,:]xB[0,1,:,:]
C[0,2,:,:] = A[0,0,:,:]xB[0,2,:,:]
C[1,0,:,:] = A[1,0,:,:]xB[1,0,:,:]
C[1,1,:,:] = A[1,1,:,:]xB[1,1,:,:]
C[1,2,:,:] = A[1,2,:,:]xB[1,2,:,:]
....
</code></pre>

vainaijr: <p>I think einsum could be used here</p>
<pre><code class="lang-auto">x = torch.randn(2, 3, 5, 6)
y = torch.randn(2, 3, 6, 7)
z = torch.einsum('abcd, abde -&gt; abce', x, y)
z.shape
</code></pre>
Slice a batch with different coordinates with no loop

pedroveloso13: <p>Hi everyone,</p>
<p>I am trying to slice a batch tensor (batch, channels, size, size) with a batch of coordinates (batch, 2, 2). This is similar to the this <a href="https://discuss.pytorch.org/t/how-to-slice-multiple-blocks-efficiently-from-a-torch-tensor/19680/3">topic</a>, which does not seem to have a proper solution.</p>
<p>Here is a toy example:</p>
<pre><code class="lang-auto">batch = 6
channels = 5
board_size = 4
agent_size = 2
iterations = 1000

B = torch.FloatTensor(batch, channels, board_size, board_size).uniform_(-100, 100)
pos = torch.LongTensor([[0, 1], [1, 3], [2, 0], [2, 2], [2, 1], [2, 2]])
# pos = torch.randint(0, board_size - agent_size, (batch, 2))
</code></pre>
<p>In this example ,the resulting tensor would have shape (batch, channel, agent_size, agent_size) = (6, 5, 2, 2) and would be formed by blocks (5, 2, 2) that are not aligned in the original tensor.</p>
<p>The problem is that while indexing accepts a multi-element tensor, slicing does not. Therefore, the solution using numpy slicing notation is not valid and triggers “TypeError: only integer tensors of a single element can be converted to an index”.</p>
<pre><code class="lang-auto">M = B[:, :, pos[:, 0]: pos[:, 0] + agent_size, pos[:, 1]: pos[:, 1] + agent_size]
</code></pre>
<p>One alternative would be to pass all the indices of the sliced tensor axis by axis, but that would require building one tensor per axis with the total number of elements in the final slice (see this <a href="https://stackoverflow.com/questions/46091111/python-slice-array-at-different-position-on-every-row" rel="nofollow noopener">post</a>). While this might be simple for lower dimensions and small slices, in my toy example it would require building 4 tensors, each with  6 x 5 x 2 x 2 = 120 indices and each following a different logic.</p>
<p>My current solutions are:</p>
<ol>
<li>use a loop to traverse the batches</li>
<li>Use the tensors for indexing. I do it twice to get the sliced “frame”.</li>
</ol>
<pre><code class="lang-auto">#method 1: loop along batch dimension
def multiSlice1(B, pos, size):
	s = B.shape
	M = torch.zeros(s[0], s[1], size, size)
	for i in range(B.shape[0]):
		M[i] = B[i, :, pos[i, 0]: pos[i, 0] + size, pos[i, 1]: pos[i, 1] + size]
	return M

#method2 
def multiSlice2(B, pos, size):
	pos_row = pos[:,0]
	pos_row = pos_row.view(-1, 1) + torch.arange(size)
	pos_row = pos_row.view(pos_row.shape[0], 1, pos_row.shape[1], 1)
	expanse = list(B.shape)
	expanse[0] = -1
	expanse[2] = -1
	pos_row = pos_row.expand(expanse)
	M1 = torch.gather(B, 2, pos_row)
	
	pos_col = pos[:,1]
	pos_col = pos_col.view(-1, 1) + torch.arange(size)
	pos_col = pos_col.view(pos_col.shape[0], 1, 1, pos_col.shape[1])
	expanse = list(M1.shape)
	expanse[0] = -1
	expanse[3] = -1
	pos_col = pos_col.expand(expanse)
	M2 = torch.gather(M1, 3, pos_col)
	return M2
</code></pre>
<p>Is there any simpler solution that is more efficient?</p>
Sampling from an arbitrary pdf in pytorch

Subho: <p>Is there a fast and efficient way in PyTorch to sample a vector (of potentially large dimension), given only its pdf in closed-form? In my case, it is intractable to analytically compute the inverse of the cdf.</p>

KFrank: <p>Hi Subho!</p>
<aside class="quote no-group" data-username="Subho" data-post="1" data-topic="55567" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a3d4f5/48.png" class="avatar"> Subho:</div>
<blockquote>
<p>Is there a fast and efficient way in PyTorch to sample a vector (of potentially large dimension), given only its pdf in closed-form? In my case, it is intractable to analytically compute the inverse of the cdf.</p>
</blockquote>
</aside>
<p>Well, it depends, of course, on your probability density function …</p>
<p>You say nothing about it other than that you have a closed-form<br>
expression for it.</p>
<p>Given that, pre-compute the following:</p>
<p>Analytically or numerically compute its integral to get the<br>
cumulative distribution function.  You say that you cannot invert<br>
this analytically, so invert it numerically, storing the (pre-computed)<br>
inverse as a look-up / interpolation table.  (The granularity and<br>
interpolation scheme for the interpolation table will depend on<br>
the accuracy required of your inverted cumulative distribution<br>
function.)</p>
<p>Then on a tensor (vector) basis, generate a bunch of uniform<br>
deviates and pump them through (on a tensor basis) your inverse<br>
cumulative distribution function.  You now have a tensor (vector)<br>
of samples from your original probability density function.</p>
<p>Good luck!</p>
<p>K. Frank</p>

Subho: <p>Turns out that I can use simple rejection sampling in my case. Thanks anyways.</p>

KFrank: <p>Hello Subho!</p>
<aside class="quote no-group" data-username="Subho" data-post="3" data-topic="55567" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a3d4f5/48.png" class="avatar"> Subho:</div>
<blockquote>
<p>Turns out that I can use simple rejection sampling in my case. Thanks anyways.</p>
</blockquote>
</aside>
<p>Rejection sampling doesn’t parallelize naturally.  You would<br>
therefore typically run the rejection-sampling loop individually<br>
for each element of the vector you want to populate with random<br>
deviates, thereby forgoing the benefits of using pytorch’s tensor<br>
operations.</p>
<p>Best.</p>
<p>K. Frank</p>
Global max pool in pytorch

slavavs: <p>How do I write Global max pool code?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/851747497becc60f943d8699d0b86ca17cfed158.png" data-download-href="https://discuss.pytorch.org/uploads/default/851747497becc60f943d8699d0b86ca17cfed158" title="gmp"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/8/851747497becc60f943d8699d0b86ca17cfed158_2_330x500.png" alt="gmp" data-base62-sha1="iZnnoV9fQxzXZdXd083CTC5sQ0o" width="330" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/8/851747497becc60f943d8699d0b86ca17cfed158_2_330x500.png, https://discuss.pytorch.org/uploads/default/original/2X/8/851747497becc60f943d8699d0b86ca17cfed158.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/8/851747497becc60f943d8699d0b86ca17cfed158.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gmp</span><span class="informations">377×570 24.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Oli: <p>You don’t have to, its already in pytorch! <a href="https://pytorch.org/docs/stable/nn.html?highlight=max%20pool#torch.nn.AdaptiveMaxPool2d" rel="nofollow noopener">docs</a></p>
How to get started preforming inference with outside images

nlam: <p>My model does a pretty good making inferences on images provided in the dataset that I used to train and test the model, but I was wondering if it is possible to get a prediction using an outside image. If so, I was hoping someone can get me started in the right direction on how to load and properly prepare the image. I am currently working with the MNIST dataset by the way,</p>
<p>My notebook if interested:<br>
<a href="https://colab.research.google.com/drive/16RFi_fERIyHMpiNXlGJb01faUsJWrkot" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/16RFi_fERIyHMpiNXlGJb01faUsJWrkot</a></p>
Custom class layering autograd functionals

TreeLimes: <p>I’m needing higher order gradients of my loss function w.r.t all the parameters of my model. So, as suggested, I’m trying to make use of autograd functionals. So far, I’ve created a class that defines a forward function that takes in data and weights as input. Then it passes the data through layers of functionals parameterized appropriately by the weights and returns the models output. Then this output is used along with a target value to define the loss. Each parameter is set to require grads during initialization of a class object.</p>
<p>The class looks like the following.</p>
<pre><code class="lang-auto">class M():
    def __init__(self):
        #initialize parameters

    def forward(self, x, weights=None):
        #pass x through functional layers parameterized by weights
        return output
</code></pre>
<p>I try to use this class in the following way:</p>
<pre><code class="lang-auto">m = M()

loss = loss_fn(m.forward(data, weights), target)

grad = autograd.grad(loss, weights, create_graph=True)
</code></pre>
<p>I get an error however:</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>I think I must be doing something obviously wrong, so I’d appreciate any help.</p>
Backward with Autograd

maralm: <p>Hi,</p>
<p>I am looking for an example to do the complete backward with autograd.grad instead of loss.backward for a network like this:</p>
<pre><code class="lang-auto">class DeepNN(nn.Module):
    def __init__(self):
        super(DeepNN, self).__init__()
        self.fc_1 = nn.Linear(D_in, H)
        self.fc = nn.Linear (H,H)
        self.fc_end = nn.Linear(H,D_out)

        self.module_list = []

        self.module_list.append(self.fc_1)
        for i in range(num_hidden):
            self.module_list.append(nn.Linear (H,H))

        self.layers = nn.ModuleList(self.module_list)
        self.f = nn.Sequential(*self.module_list)
        self._name = "DeepNN"
        
    
    def forward (self, x):
 
        return self.f(x)
</code></pre>
<p>Can anyone help me with that?<br>
Thanks.</p>
Function forward(self, inputs) in (RNN + Linear) model

shirui-japina: <p>I want to build a model like</p>
<pre><code class="lang-auto">Model(
  (rnn): LSTM(25088, 5, num_layers=2, batch_first=True)
  (output_layer): Linear(in_features=5, out_features=2, bias=True)
)
</code></pre>
<p>and I know when train a RNN-like model, we can use functions like</p>
<pre><code class="lang-auto">torch.nn.utils.rnn.pad_sequence()
torch.nn.utils.rnn.pack_padded_sequence()
torch.nn.utils.rnn.pad_packed_sequence()
</code></pre>
<p>My question is about the forward() in class Model()</p>
<pre><code class="lang-auto">class Model(nn.Module)
  def __init__(self):
    super(Model, self).__init__()
    self.rnn = nn.LSTM(25088, 5, 2, batch_first=True)
    self.output_layer = nn.Linear(5, 2)
  def forward(self, inputs, hidden_0=None)
    output, (hidden, cell) = self.rnn(inputs, hidden_0)
    output = self.output_layer(output) # AttributeError: 'PackedSequence' object has no attribute 'dim'
</code></pre>
<p>I know I can get packed sequence by pack_padded_sequence(), and also can feed the part of model, <strong>self.rnn</strong>. But when feed the part of model, <strong>self.output_layer</strong>, it will error as above. The point is that, there is <strong>batch-size info</strong> in the output of self.rnn(inputs, hidden_0), but self.output_layer(output), which is a nn.Linear() has no info about batch. It just has parameters ( <em>in_features</em> ,  <em>out_features</em> ,  <em>bias=True</em>), which meas there is mistake that <strong>batch-size info</strong> and <strong>no-batch-size info</strong>. How can train the model properly ?<br>
Thank you for your help!</p>
Memory error when trying to train with two different dataloaders in parallel

apap: <p>I am trying to train my model using 2 dataloaders from different image datasets.</p>
<p>I use  <code>cycle() and zip()</code>  to do so because my datasets are not the same length from here: <a href="https://stackoverflow.com/questions/51444059/how-to-iterate-over-two-dataloaders-simultaneously-using-pytorch" rel="nofollow noopener">How to iterate over two dataloaders simultaneously using pytorch?</a></p>
<p>But I am getting the following error:</p>
<pre><code class="lang-auto">  File "/home/Desktop/example/train.py", line 229, in train_2
    for i, (x1, x2) in enumerate(zip(cycle(train_loader_1), train_loader_2)):
  File "/home/.conda/envs/3dcnn/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 346, in __next__
    data = self.dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/.conda/envs/3dcnn/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/.conda/envs/3dcnn/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 80, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "/home/.conda/envs/3dcnn/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 80, in &lt;listcomp&gt;
    return [default_collate(samples) for samples in transposed]
  File "/home/.conda/envs/3dcnn/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 56, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 154140672 bytes. Error code 12 (Cannot allocate memory)
</code></pre>
<p>I tried to solve that by setting <code>num_workers=0</code> , decreasing the batch size, using <code>pinned_memory=False</code> and <code>shuffle=False</code> … But none of it worked…</p>
<p>My current setup is 256GB of RAM and 4 NVIDIA TESLA V100 GPUs and when I train my model with the two datasets but not in parallel - i.e. i train one epoch first on the one dataloader and then on the other one- it works totally fine (I consume only 21GB ) .</p>
<p>Is there a more efficient way to have the same functionality as when using <code>enumerate(zip(cycle(train_loader_1), train_loader_2)):</code>  ?</p>

apap: <p>solution: <a href="https://stackoverflow.com/a/57890309/11090634" rel="nofollow noopener">https://stackoverflow.com/a/57890309/11090634</a></p>
What are dense CPU tensor?

fruits_punch_samurai: <p>Cannot iterate over train_dl where dataset is moved to GPU.<br>
train_dl=  DataLoader(dataset, batch_size, sampler=train_sampler, pin_memory=True)</p>
<p>working over MNIST data</p>
<pre><code class="lang-auto">for images,labels in train_dl:
    outputs=model(images)
    loss=F.cross_entropy(outputs,labels)
    print("loss",loss.item())
    break
    
print(outputs.shape)
print(outputs[:10].data)
</code></pre>
<p>RuntimeError: cannot pin ‘torch.cuda.ByteTensor’ only dense CPU tensors can be pinned</p>

ptrblck: <p>Since your tensors are already CUDATensors, you cannot use <code>pin_memory=True</code> (this will use pinned host memory for fast data transfer).<br>
The error message points rather to CPU tensors (with the additional requirement that they should be dense, not sparse).</p>
nn.CrossEntropyLoss to update a generator

apap: <p>Would it be possible to use <code>nn.</code> <code>CrossEntropyLoss</code>  to update the generator of a GAN? for example if I have 5 classes and I want to update the generator based only on the results  for a specific class I get from the discriminator.</p>

InnovArul: <p>I think its similar to what Auxilliary GANs do.<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://arxiv.org/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://arxiv.org/abs/1610.09585" target="_blank" rel="nofollow noopener">arXiv.org</a>
  </header>
  <article class="onebox-body">
    

<h3><a href="https://arxiv.org/abs/1610.09585" target="_blank" rel="nofollow noopener">Conditional Image Synthesis With Auxiliary Classifier GANs</a></h3>

<p>Synthesizing high resolution photorealistic images has been a long-standing
challenge in machine learning. In this paper we introduce new methods for the
improved training of generative adversarial networks (GANs) for image
synthesis. We construct a...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Np.asarray behaviour in pytorch and tensorflow

muhibkhan: <p>Hello. I want my as output same as tensorflow output, but i’m unable to figuring out how array behaves? Why np.array perform the same way as in tensorflow and what is the difference between the output of tensorflow and pytorch, desdpite that both are tensors with dim = 10? Is there any other thing which i’m doing wrong</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5c0699ad7ce28ddefe7f79a5b1b7f691f3cd6c99.png" data-download-href="https://discuss.pytorch.org/uploads/default/5c0699ad7ce28ddefe7f79a5b1b7f691f3cd6c99" title="output"><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5c0699ad7ce28ddefe7f79a5b1b7f691f3cd6c99.png" alt="output" data-base62-sha1="d864eNAVNrXlist8UQgHs7zEfnX" width="690" height="433" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">output</span><span class="informations">773×486 15.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>tensorflow output = [1. 2. 3. 4. 5. 6. 7. 8. 9. 10.]<br>
pytorch output = [1., 2., 3., 4., 5., 6., 7., 8., 9., 10.]</p>

spanev: <p>Hi <a class="mention" href="/u/muhibkhan">@muhibkhan</a>,</p>
<p>There are both the same, of shape <code>(10)</code>. The difference you see is just an aesthetic difference in the printing.<br>
If you print <code>tensor1</code> or <code>tensor2</code> shape and type in PyTorch you will have:</p>
<pre><code class="lang-auto">torch.Size([10]), torch.float32
</code></pre>
<p>which is strictly equivalent to TensorFlow’s <code>TensorShape([Dimension(10)])</code> that you see.</p>

muhibkhan: <p>So logically both outputs like:<br>
tensorflow output = [1. 2. 3. 4. 5. 6. 7. 8. 9. 10.]<br>
pytorch output = [1., 2., 3., 4., 5., 6., 7., 8., 9., 10.]</p>
<p>are same. Comma separation (<strong>,</strong>) doesn’t have any affect.</p>

spanev: <aside class="quote no-group" data-username="muhibkhan" data-post="3" data-topic="55665">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/muhibkhan/48/18379_2.png" class="avatar"> muhibkhan:</div>
<blockquote>
<p>are same. Comma separation ( <strong>,</strong> ) doesn’t have any affect.</p>
</blockquote>
</aside>
<p>Correct, that’s what I meant. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>
Access the weights before last FC layer of a trained model

Rosa: <p>So, I have a trained classification model. Which works pretty well, given any unknown image. We can use class activation map to see which features are getting activated to classify a given object.</p>
<p>In case of binary classification, some features play role to decide ‘0’, some play role to decide ‘1’. This features/weights are decided just before the fully connected classification layer, per my understanding.</p>
<blockquote>
<ol>
<li>so In case of densenet169 trained model, How can I get access to those weights before the last FC layer ?</li>
<li>Can you give any idea/code/examples to distinguish the deciding features and plot them which are contributing to make decision.</li>
</ol>
</blockquote>
<p>Thanks in advance.</p>

spanev: <p>Hi <a class="mention" href="/u/rosa">@Rosa</a>,</p>
<p>Weights/parameters and features/activations are (usually) not the same thing. The features are (usually) the result of previous <code>weights op features</code> where <code>op</code> can be any binary operator: in Python, weights are contained in the operators and features are the results of operators.<br>
In both cases you will be able to access them with <a href="https://discuss.pytorch.org/t/get-the-activations-of-the-second-to-last-layer/55629/2">hooks</a>.</p>
<p>In the case of a loaded trained <code>densenet169</code>, you should first identify the classifier Linear layer by printing the architecture:</p>
<pre><code class="lang-auto">net = models.densenet169()
print(net)
</code></pre>
<p>which seems to be</p>
<pre><code class="lang-auto">net.classifier
Linear(in_features=1664, out_features=1000, bias=True)
</code></pre>
<p>If you wanna look into the input features and/or weights, your hook should look like this:</p>
<pre><code class="lang-auto">global classifier_input_features # to be able to access it anywhere
global classifier_weights
def hook_fn(module, input, output):
    global classifier_input
    global classifier_weights

    classifier_input_features = input
    classifier_weights = module.weight.data

     # you can use matplotlib or TensorboardX to visualize them

hook = net.classifier.register_forward_hook(hook_fn)
</code></pre>
<p>after that, yo can access  <code>classifier_input_features</code> and <code>classifier_weights</code> after the forward pass.</p>

Rosa: <p>Thanks, for clear explanation. I was mixing up those  terms. Also thanks for the code snippet. However I was already able to extract both weights and features.</p>
<p>But can you give me a idea how to distinguish the factors from which a classifier taking the decision to tell a given input either o or 1 ?</p>
Read mat files and convert to tensor

carrot: <p>Hi, I want to read mat files in jupyter and convert as tensor. So I use <a href="http://scipy.io" rel="nofollow noopener">scipy.io</a> but I can’t convert to tensor.</p>
<p>import <a href="http://scipy.io" rel="nofollow noopener">scipy.io</a> as spio<br>
import torch<br>
import numpy as np<br>
data = spio.loadmat(‘train.mat’)<br>
np_data = np.array(data)<br>
tensor_data = torch.Tensor(np_data)</p>
<p>but result is</p>
<hr>
<p>TypeError                                 Traceback (most recent call last)<br>
 in <br>
----&gt; 1 tensor_data = torch.Tensor(np_data)</p>
<p>TypeError: can’t convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, int64, int32, int16, int8, uint8, and bool.</p>
<p>what should I have to do?</p>

ptrblck: <p>What is currently stored in <code>np_data</code>?<br>
It looks like it’s stored as a <code>numpy.object_</code>, which happens if not all elements have the same <code>dtype</code>.<br>
Dummy example:</p>
<pre><code class="lang-python">x = np.array([1.0, np.sum])
print(x)
&gt; array([1.0, &lt;function sum at 0x7fb77bcdbd08&gt;], dtype=object)
</code></pre>
Expand an existing Embedding and linear layer - NaN loss value

Giang_Nguyen: <p>Hi,</p>
<p>I am working on an image captioning stuff.<br>
Now as the size of vocab increases, I have to expand the Embedding layer and my last linear layer. My decoder is like:</p>
<pre><code class="lang-auto">def __init__(self, embed_size, hidden_size, vocab_size, num_layers, max_seq_length=20):
 """Set the hyper-parameters and build the layers."""
 super(DecoderRNN, self).__init__()
 self.embed = nn.Embedding(vocab_size, embed_size)
 self.lstm = nn.LSTM(embed_size, hidden_size, num_layers, batch_first=True)
 self.linear = nn.Linear(hidden_size, vocab_size)
 self.max_seg_length = max_seq_length
</code></pre>
<p>And now when the vocab increases its size by new_dim_num, I copy the old weights and concatenate with the new weights to build a new Embedding and linear layer like this:</p>
<pre><code class="lang-auto">from copy import deepcopy
# expand decoder layers
# new_dim_num is the additional number of entries to vocab
old_embed = deepcopy(decoder.embed.weight.data)
new_row_embed = torch.Tensor(new_dim_num, args.embed_size).to(device)
new_embed = torch.cat((old_embed, new_row_embed), 0)
old_linear = deepcopy(decoder.linear.weight.data)
new_row_linear = torch.Tensor(new_dim_num, args.hidden_size).to(device)
new_linear = torch.cat((old_linear, new_row_linear), 0)
old_bias = deepcopy(decoder.linear.bias.data)
new_row_bias = torch.Tensor(new_dim_num).to(device)
new_bias = torch.cat((old_bias, new_row_bias))


 decoder.embed = nn.Embedding.from_pretrained(new_embed)
 decoder.linear.weight = nn.Parameter(new_linear)
 decoder.linear.bias = nn.Parameter(new_bias)
 decoder.linear.out_features = vocab_size

 for param in decoder.parameters():
      param.requires_grad_(True)
</code></pre>
<p>But now the loss is always NaN or really large. I’ve check the output of RNN of the output value is abnormally huge.<br>
Could someone point what is wrong here?<br>
Thank you.</p>

Giang_Nguyen: <p>I think my approach was wrong.<br>
This is how I make this work:</p>
<pre><code class="lang-auto">old_embed = decoder.embed.weight.data
old_weight = decoder.linear.weight.data
old_bias = decoder.linear.bias.data


 decoder.embed = nn.Embedding(vocab_size, args.embed_size)
 decoder.linear = nn.Linear(args.hidden_size, vocab_size)

 decoder.embed.weight.data[:tmp_dim, :] = old_embed
 decoder.linear.weight.data[:tmp_dim] = old_weight
 decoder.linear.bias.data[:tmp_dim] = old_bias
</code></pre>
<p>much more easy and elegant!</p>
How install torchvison of a specific version via conda?

Zhang_Chi: <p>Hi, i am going to use torch 0.3.1 but i dont know how to install torchvison 0.2.0</p>

qiangqiang_wu: <p>Hello, do you solve the problem? I just want to use torch 0.4.1, but when I use “pip install TorchVision”, it will automatically install TorchVision-0.4.0 and update my PyTorch to the latest version (1.2.0).</p>
LSTM layer dimensionality

sigma_x: <p>Why is the default dimensionality of GRU/LSTM layers (seq.length x batch_size x hidden_dim)? This is quite confusing, as I need to do more layer permutations, to make sure that the next layer (e.g. fc) takes in the batch rather than the sequence as an input.</p>
<p>What’s the rationale for this?</p>

DoubtWang: <p>the next layer will takes in the batch although the shape of rnn layer is (seq_length, batch_size, hidden_dim) when rnn run all time step.</p>

sigma_x: <p>OK so is this correct?</p>
<pre><code class="lang-auto">&gt; 
&gt;     def forward(self, x)
&gt;         b = x.size()[0]
&gt;         out_batch = self.embedding(x.long())
&gt;         # lstm - convert input to (seq_lengthxbatch sizexnum of features) dims
&gt;         out_lstm, (h_t, o_t) = self.lstm(out_batch.view(-1, b, self.embedding_dim))
&gt;         out_lstm = out_lstm[-1,:,:]
&gt;         out_fc1 = self.fc1(out_lstm)
&gt;         return out_fc1
</code></pre>
<p>where <code>self.lstm = nn.LSTM(embedding_dim, hidden_dim_lstm, n_layers, batch_first=True)</code></p>

vdw: <p><code>view()</code> can be a dangerous function. Just because it throws nor error and the next layer accept it as input doesn’t necessarily mean that that the reshaping is “semantically” correct.</p>
<p>I’m not say that your use of <code>view()</code> in this case is wrong, but I would strongly recommend <code>transpose()</code> to swap the <code>batch_size</code> and <code>seq_len</code> dimension. In your case that would be:</p>
<pre><code>out_batch = out_batch.transpose(0,1)
</code></pre>
<p>Alternatively, you can create <code>out_lstm</code> with <code>LSTM(..., batch_first=True)</code> to accept your <code>out_batch</code> directly after the embedding layer.</p>

DoubtWang: <p>if u set batch_first=True, the shape of input must be (B, L, D).</p>
<blockquote>
<p><strong>batch_first</strong>  – If  <code>True</code> , then the input and output tensors are provided as (batch, seq, feature). Default:  <code>False</code></p>
</blockquote>