akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pass None to Gradient Checkpointing

agemagician: <p>Hello,</p>
<p>I am trying to convert T5 from Google model using HuggingFace library to use gradient checkpointing. However, some variables might be None which lead to an error from checkpointing. Is there any solutions to this issue.</p>
<p>Code:<br>
<a href="https://github.com/agemagician/transformers/blob/patch-2/src/transformers/modeling_t5.py#L758" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/agemagician/transformers/blob/patch-2/src/transformers/modeling_t5.py#L758</a></p>
<p>Error message :</p>
<blockquote>
<p>TypeError: CheckpointFunctionBackward.forward: expected Variable (got NoneType) for return value 1</p>
</blockquote>

agemagician: <p>Any help here will be highly appreciated .</p>

googlebot: <p>You can use torch.zeros(0) to “encode” Nones. Or try the below code, this was copy-pasted from checkpoint.py, I removed preserve_rng_state functionality (was not needed for my use case) and added handling of python arguments (scalars at least).</p>
<pre><code class="lang-python">from torch.utils.checkpoint import detach_variable

class CheckpointXFunction(torch.autograd.Function):

	@staticmethod
	def forward(ctx, run_function, *args):
		ctx.run_function = run_function
		
		te_args = []
		py_args = []
		for iarg,arg in enumerate(args):
			if torch.is_tensor(arg):
				te_args.append(arg)
			else:
				py_args.append((iarg,arg))
		
		ctx.save_for_backward(*te_args)
		ctx.py_args = py_args
		with torch.no_grad():
			outputs = run_function(*args)
		return outputs

	@staticmethod
	def backward(ctx, *args):
		if not torch.autograd._is_checkpoint_valid():
			raise RuntimeError("Checkpointing is not compatible with .grad(), please use .backward() if possible")
		inputs = detach_variable(ctx.saved_tensors)
		if ctx.py_args:
			inputs = list(inputs)
			for i,arg in ctx.py_args:
				inputs.insert(i, arg)
			
		with torch.enable_grad():
			outputs = ctx.run_function(*inputs)

		if isinstance(outputs, torch.Tensor):
			outputs = (outputs,)
		
		flt_outputs = []
		flt_args = []
		for ioutput, output in enumerate(outputs):
			if output is not None and output.requires_grad:
				flt_outputs.append(output)
				flt_args.append(args[ioutput])
				
		if flt_outputs:
			torch.autograd.backward(flt_outputs, flt_args)
		grads = tuple(inp.grad if isinstance(inp, torch.Tensor) else None for inp in inputs)
		return (None,) + grads

def checkpointx(function, *args):
	return CheckpointXFunction.apply(function, *args)
</code></pre>
How to use Generic transforms for the following preprocessing step

Rishav_Sapahia: <p>I want to apply the following transformation to the image dataset.</p>
<ol>
<li>N(w, h) = I(w, h) − G(w, h), (1) where N is the normalized image, I is the original image, and G is the Gaussian blurred image with kernel size 65*65 and 0 mean and standard deviation 10.</li>
</ol>
<p>The code for gaussian blur is-</p>
<pre><code class="lang-auto">def gaussian_blur(img):
    image = cv2.GaussianBlur(image,(65,65),10)
    new_image = img - image
return image
</code></pre>
<p>I am really not sure how to convert it into lambda function as to use in generic transform.Any other advice on how to apply the above preprocessing step is also welcomed.</p>

ptrblck: <p>This code should work:</p>
<pre><code class="lang-python">def gaussian_blur(img):
    image = np.array(img)
    image_blur = cv2.GaussianBlur(image,(65,65),10)
    new_image = image_blur
    return new_image

x = torch.randn(3, 224, 224)
img = TF.to_pil_image(x)
transform = transforms.Lambda(gaussian_blur)
img = transform(img)
</code></pre>

Rishav_Sapahia: <p>Thanks, it worked. I am observing a peculiar behavior, don’t know whether a gap in my knowledge or not.</p>
<pre><code>data_transforms = transforms.Compose([transforms.RandomCrop(512,512),
                             transforms.ToTensor(),
                             transforms.Lambda(gaussian_blur),
                             transforms.Normalize(mean=train_mean, std=train_std),
                             transforms.RandomRotation([+90,+180]),
                             transforms.RandomRotation([+180,+270]),
                             transforms.RandomHorizontalFlip(),
                           ])
</code></pre>
<p>When I am showing the shape of the images, it is coming out to be not 512,512. Does Random Crop  doesn’t take the required size and then crop it or I am doing something wrong.</p>
<pre><code>for images, labels in final_train_loader:  
    print('Image batch dimensions:', images.shape)

Image batch dimensions: torch.Size([3, 3, 584, 565])
Image label dimensions: torch.Size([3])</code></pre>

ptrblck: <p>Make sure this code is really called in your <code>Dataset</code>, as it should throw an error.<br>
Image transformations like <code>RandomRotation</code> and <code>RandomHorizontalFlip</code> are only defined for <code>PIL.Image</code>s.</p>
<p>Thus you would have to use <code>ToTensor()</code> and <code>Normalize()</code> as the last transformations.<br>
I’m not sure if your <code>Lambda(gaussian_blur)</code> transform works on tensors or <code>PIL.Image</code>s.</p>

Rishav_Sapahia: <p>Yes, you were right. I was calling some other dataset and there was also a typo in my implementation.<br>
<code>Lambda(gaussian_blur)</code>  works on PIL image but we have to add a conversion of numpy array to PIL image using <code>im = Image.fromarray(new_image)</code> in the gaussian_blur function.</p>

Ripley: <p>Thank you! for this code!</p>
Input(t) = output(t-1) without "for"

111353: <p>This question was updated, I write in following comment .</p>
<p>I want to make RNN to be as follow<br>
input(t) = output(t-1)</p>
<p>In pytorch’s RNN, the time series data, the input are generally obvious from the beginning.<br>
But this case isn’t because the input depend on its output.<br>
So, I think, There are only way to do this is for statement in this case,<br>
but I also know that for statement in python is too slow,<br>
and this is the reason why I’m thinking to be used PyTorch.</p>
<p>Is there any ideas?<br>
Thank you for reading.</p>

albanD: <p>Hi,</p>
<p>It is not very clear from your question what kind of objects input and output are. But adding an extra entry at the beginning will do this: <code>[None,] + output</code> for a list for example. Or <code>torch.cat([torch.tensor(whatever_is_the_right_size), output], 0)</code> for Tensors.</p>

111353: <p>Thank you and sorry for the unclear question.<br>
I will update my question.</p>

111353: <p>This is the algorism what I want to create.</p>
<p>There are one layler RNN, called Reservoir computing<br>
like this,<br>
Recent advances in physical reservoir computing: A review<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://linkinghub.elsevier.com/retrieve/pii/S0893608019300784" target="_blank" rel="noopener nofollow ugc">linkinghub.elsevier.com</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://linkinghub.elsevier.com/retrieve/pii/S0893608019300784" target="_blank" rel="noopener nofollow ugc">Redirecting</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>In my case, for example, I want this network to work like below,<br>
This is the input,<br>
<code>torch.tensor[x1(t),x2(t),......,x50(t)]</code></p>
<p>And output will be<br>
<code>torch.tensor[x1(t+1),x2(t+1),......,x50(t+1)]</code></p>
<p>Then, the next input should be<br>
<code>torch.tensor[x1(t+1),x2(t+1),......,x50(t+1)]</code><br>
(equal to the before output.)</p>
<p><br>
If it goes, there are no input from external, so the following external inputs are separately given as triggers to drive this network.<br>
<code>torch.tensor[x51(t),x52(t),x53(t)]</code></p>
<p>Thank you for your attention.</p>
Recommendations

Azmi_Anis: <p>Pytorch should streamline their guidelines on GitHub to support first-time contributors. In addition to that, the application should also articulate their application for users. I would also recommend the site to use better SEO to attract more users. Furthermore, I believe that the site needed a proper introduction about what Pytorch is basically about so i did update some of the language on the documentation. However, according to me much more work is needed.</p>

ptrblck: <p>Hi,</p>
<p>could you explain a bit what you are missing and would wish to see in the <a href="https://pytorch.org/docs/stable/community/contribution_guide.html">contribution guide</a>?</p>
<p>Could you post a link to your PR, which improves the introduction to PyTorch or are you referring to <a href="https://github.com/pytorch/pytorch/pull/46417/files">this one</a>?</p>
<aside class="quote no-group" data-username="Azmi_Anis" data-post="1" data-topic="99531">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/azmi_anis/48/29874_2.png" class="avatar"> Azmi_Anis:</div>
<blockquote>
<p>However, according to me much more work is needed.</p>
</blockquote>
</aside>
<p>Constructive feedback and contributions are always more than welcome! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Pytorch distribution - TransformedDistribution

saandeep_aathreya: <p>Can anyone explain how <code>torch.distributions.transformed_distribution.</code> <code>TransformedDistribution</code> works in pytorch?</p>
<p>example code:<br>
prior = TransformedDistribution(Uniform(torch.zeros(2), torch.ones(2)), SigmoidTransform().inv)</p>
Torch.sum over multidimensional tensor

JamesDickens: <p>I have a tensor t of shape (38, 38, 7, 7, 21) -&gt; (x, y, i, j, c)</p>
<p>Supposing I have x_min, x_max, y_min, y_max and two values i,j, and a class num c</p>
<p>I want to sum all elements in the range</p>
<p>(x_min : xmax, y_min:y_max, i, j, c)</p>
<p>I am thinking something like<br>
torch.sum(t[x_min : xmax, y_min:y_max, i:i+1, j:j+1, c:c+1]) but this would not work<br>
if i = 6 for instance, so I would need if statements. Is there a cleaner way of achieving this?</p>

albanD: <aside class="quote no-group" data-username="JamesDickens" data-post="1" data-topic="99544">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jamesdickens/48/45317_2.png" class="avatar"> JamesDickens:</div>
<blockquote>
<p>if i = 6 for instance, so I would need if statements.</p>
</blockquote>
</aside>
<p>Why wounldn’t it?<br>
I think that selecting the subset you care about and calling sum is the right thing to do here.<br>
Note that you don’t need to give ranges for dimensions where you want a single value: <code>torch.sum(t[x_min : xmax, y_min:y_max, i, j, c])</code> but that will give the same result <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Distributed Data Parallel over the Internet

arshagarwal: <p>I am trying to train a network with “Distributed Data Parallel” on multiple nodes, each having a different public IP address by sshing simultaneously into these nodes using “pdsh” coordination tool as suggested in <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html#communication-backends" rel="noopener nofollow ugc">this tutorial</a>.</p>
<p>Specifically, given a local machine with public IP address “Ip0” and 2 remote nodes with Public IP address “Ip1” and  “Ip2” respectively on which training is to be performed remotely from the local machine, how to go about making such a set-up?</p>
<p>Also, how to ensure before running the training script on each remote node that the 2 remote nodes have access to each other?</p>
<p>Thanks in advance.</p>

mrshenli: <aside class="quote no-group" data-username="arshagarwal" data-post="1" data-topic="99504">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arshagarwal/48/29861_2.png" class="avatar"> arshagarwal:</div>
<blockquote>
<p>Also, how to ensure before running the training script on each remote node that the 2 remote nodes have access to each other?</p>
</blockquote>
</aside>
<p>Can you try ssh to one of the remote machine and then ping/ssh another remote machine?</p>
<blockquote>
<p>Specifically, given a local machine with public IP address “Ip0” and 2 remote nodes with Public IP address “Ip1” and “Ip2” respectively on which training is to be performed remotely from the local machine, how to go about making such a set-up?</p>
</blockquote>
<p>One of the remote machine can serve as the master, i.e., using it’s IP address as the <code>MASTER_ADDR</code> and pick a port for <code>MASTER_PORT</code>.</p>

arshagarwal: <p>If you mean ssh-ing into node-2 from node-1, yes I can do that. However, what if I were to do this with n (say 10) nodes? Should I then ssh into remote node-1 and then from the remote node-1 terminal  use “pdsh” to ssh into all other nodes simultaneously? <a class="mention" href="/u/mrshenli">@mrshenli</a></p>

mrshenli: <aside class="quote no-group" data-username="arshagarwal" data-post="3" data-topic="99504">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arshagarwal/48/29861_2.png" class="avatar"> arshagarwal:</div>
<blockquote>
<p>Should I then ssh into remote node-1 and then from the remote node-1 terminal use “pdsh” to ssh into all other nodes simultaneously?</p>
</blockquote>
</aside>
<p>No, you don’t need to ssh from node-1 to other nodes to launch the script. The ping/ssh I mentioned is only to check what IP would work. If you confirm that the IP of one node is accessible for all other nodes, you can set that node as master. This is only for rendezvous, and all nodes will use the rendezvous process to discover each other automatically.</p>

arshagarwal: <p>Understood, thanks for such a subtle answer. <a class="mention" href="/u/mrshenli">@mrshenli</a></p>
How to import a quantized TFLite model into PyTorch

algal: <p>Hi,</p>
<p>I’m fairly new to PyTorch and I’d like to understand how to import a quantized TFLite model into PyTorch so I can work on it in PyTorch.</p>
<p>I already have a PyTorch model definition which matches the model used to create the <code>.tfilte</code> file – except for the fact that this tflite file has been quantized, presumably automatically at export time.</p>
<p>There are two aspects of this I want to understand better.</p>
<p>First, the conv2d kernels and biases in the TFLite file are float16. Of course, I load these tensors’ buffers into float16 numpy arrays when I am reading from the tflite file. But is it enough to use these float16 numpy arrays as the values when I am populating the state_dict for my PyTorch model? Or do I need to define the torch model differently, for instance when I initialize the <code>nn.Conv2d</code> modules?</p>
<p>Second, I notice that this TFLite model has many (about 75) automatically generated “dequantize” layers after the normal-seeming part of the model. Do I need to manually add layers to my PyTorch model to match all these TFLite dequantization layers?</p>
<p>I’d appreciate any advice, especially pointers to any examples of how to do this.</p>

Zafar: <p>You can get the TF model weights, and load the into the non-quantized PyTorch model. After that you can just quantize the model using the PTQ flow described in here:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" target="_blank" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials...</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
RAM gets filled even though everything is shifted to GPU

a_d: <p>Hello all,<br>
I have a model and a dataset class. On a fresh boot, the system takes up 1.8GB of ram with no process running. The dataset class upon initialization takes up an additional 2 ~2.5 GB as it stores some variables for further reference. I instantiate the model class and pass it to the training function which looks as follows -</p>
<pre><code class="lang-auto">def trainer(model, train_dataloader, val_dataloader, num_epochs):
    torch.backends.cudnn.benchmark = True 
    model.train()
    model.cuda()
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=0.00009)
    criterion = nn.CrossEntropyLoss().cuda() 
        model.train()
        epoch_loss_train = 0
        epoch_acc_train = 0

        for _, (image, label) in enumerate(train_dataloader):
            optimizer.zero_grad()
            image = image.cuda()
            label = label.cuda()
            output = model(image)

            loss = criterion(output, label)
            loss.backward()  
            optimizer.step()
            
            del image, label

</code></pre>
<p>As seen above, the model is shifted to the GPU and the dataloader returns the image and label which are shifted on to the GPU. The dataloader’s runs on a single thread and the system monitor reflects that.<br>
Once the training loop starts around 2.8 GB of GPU memory is utilized. However RAM gets filled up to 7 GB. I was wondering where is this additional 2.5 GB is  coming from ??</p>

albanD: <p>Hi,</p>
<p>Can you try checking the ram usage if you just do a simple cuda op like <code>torch.rand(10, device="cuda")</code>? The cpu size memory usage of the cuda driver is known to be very large. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

a_d: <p>Damn you are right, my ram usage shot up from 3.1GB to 5.2GB. So this is where the extra 2.1GB comes from.</p>
Memory allocation for backward pass

Stefaanhess: <p>I am trying to estimate the memory allocation of my GPU before training. Therefore I count the number of model-parameters, the dimensions of the in/out-tensors and I track all intermediate computations. When I run <code>model(inputs)</code> with <code>torch.no_grad()</code>, the estimated memory allocation in byte is (model_params + input_params + intermediate_params) * 32 / 8. This also matches the observation with <code>torch.cuda.memory_allocated()</code>.<br>
When I do the same with building the graph, I would expect the allocation to be (model_params + input_params + 2 * intermediate_params) * 32 / 8, because of the backward pass. But the observation with <code>torch.cuda.memory_allocated()</code> is about (model_params + input_params + 3 * intermediate_params) * 32 / 8. Are there any other things allocated on the GPU, or am I missing something?<br>
Thanks for helping out,<br>
Stefaan</p>

albanD: <p>Hi,</p>
<p>All the intermediary states do get allocated but they should all be freed when you run in <code>no_grad()</code> (unless you keep reference to them).<br>
And when grad mode is enabled, most of these will be kept alive by the autograd to be able to compute the backward indeed. But nothing else no.</p>
Module 'torch' has no attribute 'view_as_complex'

sohaib_attaiki: <p>Hi,</p>
<p>I’m using Pytorch 1.5.1.</p>
<p>I’m having a problem using some functions that exist is the documentation, but I got the error <code> 'torch' has no attribute</code>.<br>
For example, the code provided in this <a href="https://pytorch.org/docs/stable/generated/torch.view_as_complex.html" rel="noopener nofollow ugc">pytorch link</a> doesn’t work:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; x=torch.randn(4, 2)
&gt;&gt;&gt; torch.view_as_complex(x)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
AttributeError: module 'torch' has no attribute 'view_as_complex'
&gt;&gt;&gt; 
</code></pre>
<p>How can I solve this problem?</p>
<p>Thank you in advance!</p>

sohaib_attaiki: <p>It appears that this function is only available in torch 1.6. Is there a way to use in an anterior version of Pytorch?</p>

albanD: <p>Hi,</p>
<p>I am afraid this is not something that can easily be backported to older versions of pytorch as they don’t have complex support.<br>
You will have to use 1.6+ I’m afraid</p>

sohaib_attaiki: <p>Ok, thanks for your response!</p>
Does this torch implementation of tf.scan break the backporp?

blade: <p>I have written the following code to mimic <code>tf.scan</code></p>
<pre><code class="lang-auto">    def scan(foo, x):
        res = []
        res.append(x[0].unsqueeze(0))
        a_ = x[0].clone()

        for i in range(1, len(x)):
            res.append(foo(a_, x[i]).unsqueeze(0))
            a_ = foo(a_, x[i])

        return torch.cat(res)
</code></pre>
<p>It generates the desired output for a number of examples. My only question is if the <code>append</code> and <code>torch.cat</code> part in this work break the backpropagation computations.</p>

KFrank: <p>Hi Blade!</p>
<aside class="quote no-group" data-username="blade" data-post="1" data-topic="99500" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blade/48/23861_2.png" class="avatar"> blade:</div>
<blockquote>
<p>My only question is if the <code>append</code> and <code>torch.cat</code> part in this work break the backpropagation computations.</p>
</blockquote>
</aside>
<p>No, backpropagation will not be broken by <code>append()</code>.  Even though<br>
you are appending to a python list, you’re still (presumably) appending<br>
a valid pytorch tensor, and <code>cat()</code> is a valid pytorch tensor operation.<br>
(Of course, something in <code>foo()</code> might break backpropagation.)</p>
<p>Here is a simple script that illustrates backpropagating through<br>
<code>append()</code> and <code>cat()</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

t1 = torch.autograd.Variable (torch.FloatTensor ([1.0]), requires_grad = True)
t2 = torch.autograd.Variable (torch.FloatTensor ([2.0]), requires_grad = True)
l = []
l.append (t1)
l.append (t2)
t = torch.cat (l)
t.prod().backward()
t1.grad
t2.grad
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt;
&gt;&gt;&gt; t1 = torch.autograd.Variable (torch.FloatTensor ([1.0]), requires_grad = True)
&gt;&gt;&gt; t2 = torch.autograd.Variable (torch.FloatTensor ([2.0]), requires_grad = True)
&gt;&gt;&gt; l = []
&gt;&gt;&gt; l.append (t1)
&gt;&gt;&gt; l.append (t2)
&gt;&gt;&gt; t = torch.cat (l)
&gt;&gt;&gt; t.prod().backward()
&gt;&gt;&gt; t1.grad
Variable containing:
 2
[torch.FloatTensor of size 1]

&gt;&gt;&gt; t2.grad
Variable containing:
 1
[torch.FloatTensor of size 1]
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Graph Mode Quantization Aware Training in PyTorch

mathmanu: <p>Hi,</p>
<p>The Graph Mode Post Training Quantization in Pytorch is awesome because it doesn’t need the model to be defined in certain way only using Modules. (<a href="https://pytorch.org/tutorials/prototype/graph_mode_static_quantization_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/prototype/graph_mode_static_quantization_tutorial.html</a>)</p>
<p>Wondering if there is a plan for Graph Mode Quantization Aware Training as well in Pytorch?</p>
<p>Thanks,<br>
Manu</p>

dskhudia: <p>Hi Manu,</p>
<p>Yes. It’s very much work in progress and the code is at <a href="https://github.com/pytorch/pytorch/tree/master/torch/quantization/fx" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/tree/master/torch/quantization/fx</a> Tutorials and docs will be released once it’s ready.</p>

dskhudia: <p>You can monitor the progress here: <a href="https://github.com/pytorch/pytorch/issues/45635" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/45635</a></p>

mathmanu: <p>Thanks Daya,</p>
<p>What would be the rough timeline for graph mode QAT to be available - will it be included in PyTorch 1.7 release?</p>
<p>Also wondering what would be the difference between quantize_jit and quantize_fx</p>

dskhudia: <p>It’s not a part of 1.7. quantize_jit is the current api that you used for graph-mode post-training quantization and it will be deprecated once quantize_fx is available (i.e., new graph mode).</p>
Traverse a model to get each module names and its parents

ucohen1: <p>I would like to connect each nn.Module in a model to its named parent.</p>
<pre><code class="lang-auto">class AddBlock(nn.Module):
    def forward(self, x, y):
        return x+y

class multi_inp(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(3,32,kernel_size=3)
        self.add = AddBlock()
    def forward(self, x, y):
        return self.conv(self.add(x,y))

a = torch.rand(3,128,128)
b = torch.rand(3,128,128)
model = multi_inp()
</code></pre>
<p>using the flowing:</p>
<pre><code class="lang-auto"> for n,m in model.named_modules():
     if n:
         print(n,m)
</code></pre>
<p>prints:</p>
<blockquote>
<p>conv Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1))<br>
add AddBlock()</p>
</blockquote>
<p>I would also like to get another column stating a list of named modules for each node</p>
<pre><code class="lang-auto">module-name | parents | module
=========== | ======= | ======
conv        |  None   |  Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1))
add         | [conv]  |  AddBlock()
</code></pre>

KarthikR: <p>“…I would also like to get another column…” - just for clarity, could you type the full output (for the code above) you are expecting?</p>

ucohen1: <p>thanks, I’ve updated the question</p>

KarthikR: <p>Please see if this helps:</p>
<p><span class="hashtag">#Variables</span><br>
level = 0<br>
strParent = ‘’<br>
lstModules = list( model.named_modules())[1:]</p>
<p><span class="hashtag">#Initial</span> Print Statements<br>
print(’{0:&lt;20}’.format(‘Module Name’), ‘|’, ‘{0:&lt;10}’.format(‘Parent’) , ‘|’,  ‘{0:&lt;10}’.format(‘Module’, ’ ‘*10))<br>
print(’{0:&lt;20}’.format(’===========’), ‘|’,  ‘{0:&lt;10}’.format(’======’), ‘|’,  ‘{0:&lt;10}’.format(’======’))</p>
<p><span class="hashtag">#Loop</span> through the module<br>
for i in range(len(lstModules)):<br>
if level==0:<br>
strParent=‘None’</p>
<p>module_name = lstModules[i][0]<br>
parent_name = strParent<br>
module      = str(lstModules[i][1])<br>
print(’{0:&lt;20}’.format(module_name), ‘|’ ,  ‘{0:&lt;10}’.format(parent_name), ‘|’,  ‘{0:&lt;100}’.format(module))<br>
strParent = lstModules[i][0]<br>
level=1</p>

ucohen1: <p>thanks <a class="mention" href="/u/karthikr">@KarthikR</a>.<br>
your snippet works fine for hierarchical architectures where all children are structured properly below their parents.<br>
I am looking for a way to list all parents that would work also for complex scenarios such as resnets, where the connection scheme have multiple parents in non-sequential order.<br>
try:</p>
<blockquote>
<p>import torchvision.models as models<br>
model = models.resnet18()</p>
</blockquote>
<p>I think this would require the inspection of the actual tensor flow along the net.</p>
PyTorch implementation of Stacked Capsule Auto-Encoders

bdsaglam: <p>Hi,<br>
I have implemented Stacked Capsule Auto-Encoder (Kosiorek et al, 2019) in PyTorch. The original implementation by the authors of paper was created with TensorFlow v1 and DeepMind Sonnet. I could not find a good PyTorch implementation, so hope this provides an easier-to-understand alternative.</p>
<p>I mostly kept the architecture of model and hyper-parameters same with the original implementation. However, some parts are refactored for ease of use.</p>
<p>I am kind of new to PyTorch, so there might be some not-so-best practices <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">. Feel free to give any feedback.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/bdsaglam/torch-scae" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/9764782?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/bdsaglam/torch-scae" target="_blank" rel="nofollow noopener">bdsaglam/torch-scae</a></h3>

<p>PyTorch implementation of Stacked Capsule Auto-Encoders - bdsaglam/torch-scae</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>Thanks for sharing the code, Baris! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Chatura_Madhusanka: <p>Thanks Baris! <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"> <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"></p>

JTiC: <p>Beautiful code, like a poem. But the performances seems not comparable to the reported results in the paper.</p>

bdsaglam: <p>Yes, you’re right. There must be a bug somewhere causing worse performance, but I haven’t had the opportunity to spend some time to find it yet.</p>

123465: <p>I reimplement it according to the official code, but I only get .6 accuracy also.</p>
Is there a way to make the forward pass through a network faster?

Supreet_Pravin_Kurde: <p>Assuming I do not need to do backprop, is it possible to do a forward pass faster? I understand that it beats the purpose of creating a computational graph and having an autoGrad engine. I’m not a computer scientist, so I just want to know, whether its possible? For example while running in model.eval mode. Is it viable to have a different algorithm to do the forward pass when backprop is not required?</p>

ptrblck: <p>You can wrap the forward pass into <code>with torch.no_grad()</code> to  avoid storing the intermediate activations and create the computation graph. However, this shouldn’t change the used operations.</p>
<p>If you are using a GPU, you could use <code>torch.backends.cudnn.benchmark = True</code> to let cudnn benchmark all workloads and select the fastest algorithm. Note that the first iteration will observe an overhead due to the profiling.</p>
Specialized Normalization Running Slow in Pytorch

JamesDickens: <p>I have a tensor of shape  <code>z = (38, 38, 7, 7, 21) = (x_pos, y_pos, grid_i, grid_j, class_num)</code>, and I wish to normalize it according to the formula:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc3b5007d69567f5456f7e3818f991ab3b1f24fc.png" alt="image" data-base62-sha1="vqg7s34N2K0zKlNl6qyVGH60sTW" width="320" height="93"></p>
<p>I have produced a working example of what I mean here, and the problem is that it is extremely slow, approximately 2-3 seconds for each grid entry (of which there are 49, so 49*3 seconds = 147 seconds, which is way too long, considering I need to do this with thousands of image feature maps).<br>
Any optimizations or obvious problems very much appreciated. This is part of a Pytorch convolutional neural network architecture, so I am using torch tensors and tensor ops.</p>
<pre><code>import torch
def normalizeScoreMap(score_map):
    for grid_i in range(7):
        for grid_j in range(7):
            for x in range(38):
                for y in range(38):
                    grid_sum = torch.tensor(0.0).cuda()
                    for class_num in range(21):
                        grid_sum += torch.pow(score_map[x][y][grid_i][grid_j][class_num], 2)
                    grid_normalizer = torch.sqrt(grid_sum)
                    for class_num in range(21):
                        score_map[x][y][grid_i][grid_j][class_num] /= grid_normalizer
    return score_map

random_score_map = torch.rand(38,38,7,7,21).cuda()
score_map = normalizeScoreMap(random_score_map)
</code></pre>
<p>Edit: For reference I have an i9-9900K CPU and a nvidia 2080 GPU, so my hardware is quite good. I would be willing to try multi-threading but I am looking for more obvious problems/optimizations.</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">x = random_score_map.clone()
s = (x**2).sum(4, keepdims=True)
n = torch.sqrt(s)
x /= n

print(torch.allclose(x, score_map))
&gt; True
</code></pre>
<p>Note that you should avoid for loops where possible and try to use vectorized code instead.</p>
A few question about sharing weights of convolution kernels and dilation convlution in pytorch

zhongqiu1245: <p>1,Could you tell me how can I achieve multiple convolution kernels sharing the same weight? If I want to make 4 conv to share 1 conv weight.<br>
for example, I want to keep conv1 and conv2,3,4 using the same weights(using the weight of conv1)<br>
self.conv1 = nn.Conv2d(3, 3, 3)<br>
self.conv2 = nn.Conv2d(3, 3, 3)<br>
self.conv3 = nn.Conv2d(3, 3, 3)<br>
self.conv4 = nn.Conv2d(3, 3, 3)</p>
<p>2,May I ask whether the implementation of dilated convolution is to generate a convolution kernel with a dilation rate = r as in the original paper or to periodically sample the feature map at equal intervals, then convolve with a common convolution kernel, and then resample the image Spell back the original feature map size（As mentioned in “Smoothed Dilated Convolutions for Improved Dense Prediction”?）?</p>
<p>Thank you in advance!</p>
Simple PytorchBenchmark script gives CUDA forked subprocess error

Claartje_Barkhof: <p>I am trying to run a simple benchmark script, but it fails due to a CUDA error, which leads to another error:</p>
<pre><code class="lang-auto">Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
Traceback (most recent call last):
  File "/home/cbarkhof/code-thesis/Experimentation/Benchmarking/benchmarking-models-claartje.py", line 23, in &lt;module&gt;
    benchmark.run()
  File "/home/cbarkhof/.local/lib/python3.6/site-packages/transformers/benchmark/benchmark_utils.py", line 674, in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
ValueError: too many values to unpack (expected 2)
</code></pre>
<p>My script is simply:</p>
<pre><code class="lang-auto">from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments

benchmark_args = PyTorchBenchmarkArguments(models=["bert-base-uncased"],
                                           batch_sizes=[8],
                                           sequence_lengths=[8, 32, 128, 512],
                                           save_to_csv=True,
                                           log_filename='log',
                                           env_info_csv_file='env_info')

benchmark = PyTorchBenchmark(benchmark_args)
benchmark.run()
</code></pre>
<p>I am not aware of doing any multi-processing, so why is this happening?</p>
<p>If anyone can point me to why this might be happening. Please let me know :). Cheers!</p>
Could not load library cudnn_ops_infer64_8.dll. Error code 126

KagamiMiro: <p>After upgrade to rtx3090, I reinstalled my Windows10 and pytorch with</p>
<blockquote>
<p>conda install pytorch torchvision cudatoolkit=11 -c pytorch-nightly</p>
</blockquote>
<p>When I use nn.Conv/nn.BatchNorm layer in my module and do the forward propagation, it shows</p>
<blockquote>
<p>Could not load library cudnn_ops_infer64_8.dll. Error code 126<br>
Please make sure cudnn_ops_infer64_8.dll is in your library path!</p>
</blockquote>
<p>in Vscode</p>
<blockquote>
<p>Process finished with exit code -1073740791 (0xC0000409)</p>
</blockquote>
<p>in Pycharm</p>
<p>I’ve tried reinstalling pytorch and installed cuda toolkit 11 from <a href="http://developer.nvidia.com" rel="noopener nofollow ugc">developer.nvidia.com</a>, but the problem remains unsolved.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class test(nn.Module):
    def __init__(self):
        super(test, self).__init__()
        self.l = nn.Conv2d(1,1,3)

    def forward(self, input):
        return self.l(input)

net = test().cuda()
print(net(torch.rand(1,1,5,5).cuda()))
</code></pre>

ptrblck: <p>Double post from <a href="https://github.com/pytorch/pytorch/issues/46312">here</a> with follow-up.</p>
1x1 Convolution with padding = 0, stride = 1, dilation = 2

JamesDickens: <p>I am trying to understand what is happening in the scenario where I have a 2d<br>
convolutional layer with the following parameters:</p>
<p>2d_conv = nn.Conv2d(3, 3, kernel_size=1, stride=1,<br>
padding=0, bias=False, dilation=2)</p>
<p>Is this just the same as a regular 1x1 2-d convolution, since I’m not sure what effect<br>
the dilation would have.</p>
<p>I am asking since in the paper describing the a-trous algorithm modification of a resnet-101 backbone in R-FCN the authors write:</p>
<blockquote>
<p>All layers before and on the conv4 stage [9] (stride=16) are unchanged; the stride=2 operations in the first conv5 block is modified to have stride=1, and all convolutional filters on the conv5 stage are modified by the “hole algorithm” [15, 2] (“Algorithme à trous”) to compensate for the reduced stride</p>
</blockquote>

phan_phan: <p>When <code>kernel_size=1</code>, dilation has no effect.<br>
See <a href="https://github.com/vdumoulin/conv_arithmetic/blob/master/README.md" rel="noopener nofollow ugc">this page</a> for a visualization of dilation : for only 1 pixel as input, should not change a thing.</p>
Just a quick question on Pytorch Lighting, is the model saved in these director?

JonathanSum: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5dd6840850f34cef7e76fddf5c73960b1d0d073c.png" data-download-href="https://discuss.pytorch.org/uploads/default/5dd6840850f34cef7e76fddf5c73960b1d0d073c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dd6840850f34cef7e76fddf5c73960b1d0d073c_2_690x351.png" alt="image" data-base62-sha1="do80a6VXNrf1gJnEUy5uN3WFVxy" width="690" height="351" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dd6840850f34cef7e76fddf5c73960b1d0d073c_2_690x351.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dd6840850f34cef7e76fddf5c73960b1d0d073c_2_1035x526.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5dd6840850f34cef7e76fddf5c73960b1d0d073c.png 2x" data-dominant-color="404040"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1174×598 63.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
aren’t the model and its weights automatically saved in Pytorch lighting?</p>

ptrblck: <p>The <code>events...</code> file should be created by tensorboard, while the second file seems to contain some hyperparameters, not the <code>state_dict</code>.</p>
<p>Since I’m not deeply familiar with Lightning, CC <a class="mention" href="/u/williamfalcon">@williamFalcon</a>.</p>
Convert Torch Tensor to flattened C++ array

kumar-akshay324: <p>How do I convert a Torch tensor to a C++ array without looping over all the dimensions ( I can do this, but is terribly slow) and <code>std::memcpy</code> has undefined behavior.</p>
<p>Using memcpy like this.</p>
<pre><code class="lang-auto">std::array&lt;float, 327600&gt; output_data;
at::Tensor output = *from some instructions*
std::memcpy(output_data.data(),output.data_ptr(),327600*sizeof(float));
</code></pre>

kaihsinwu: <p>I didn’t get the same undefine behavior using g++. But I guess</p>
<p>Maybe you missed the usage of template version of data_ptr:</p>
<p>output.data_ptr&lt;float&gt;()</p>
<p>The output.data_ptr() returns a void pointer, and sometimes memcpy btwn pointers with different types has undefined behavior</p>

kumar-akshay324: <p>Thanks for the reply.</p>
<p>The problem was the non-contiguous memory distribution for the tensor. So the raw memory copy failed. Using the contiguous method on the tensor and returning the data pointer worked</p>

bluove: <p>Could you help me to understand in details ?</p>
<p>I use these code, but it do not work:</p>
<h1>result is a torch::Tensor object, it is on cpu.</h1>
<p>result = result.contiguous();<br>
std::memcpy(ret_array,_result.data_ptr(),sizeof(float)*_result.numel());</p>

a_d: <p>Like python, <code>torch::Tensor</code> has the view method to<br>
You could do the following -</p>
<pre><code class="lang-auto">#include&lt;torch/torch.h&gt;

auto tensor = torch::rand({1, 2, 3 ,5});
tensor = tensor.view(tensor.size(0), -1); //Here the shape of the data is batch, num_tensors
tensor = tensor.contiguous();
std::vector&lt;float&gt; vector(tensor.data_ptr&lt;float&gt;(), tensor.data_ptr&lt;float&gt;()+tensor.numel());
</code></pre>
Multiple add_graph calls

111242: <p>Is it possible to add multiple graphs (multiple calls to add_graph(model_A), add_graph(model_B), etc). I tried it, but only the last called add_graph() gets shown on tensorboard. Thanks!</p>
<p>redirected by <span class="mention">@bdhirsh</span> from <a href="https://github.com/pytorch/pytorch/issues/46239" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/46239</a></p>
Numerical unstability of calculating KL(1-P||1-Q)

lemon: <p>Given two raw logit vectors p and q, I would like to calculate <code>-(P-Q)*log((1-P)/(1-Q))</code>, which is equals to <code>KL(1-P||1-Q) + KL(1-Q||1-P)</code>, where P/Q is the probability distribution of p/q after softmax().</p>
<p>Here is my code:</p>
<blockquote>
<p>-(F.softmax(p, dim=1) - F.softmax(q, dim=1)) * torch.log((1.0001 - F.softmax(p, dim=1)) / (1.0001 - F.softmax(q, dim=1)))</p>
</blockquote>
<p>but it turns out to be numerical unstable. How can I fix it?</p>
Why torch.dot(a,b) == torch.sum(a * b) returns False?

RajXantar: <p>Hi, why torch.dot(a,b) == torch.sum(a * b) returns False?<br>
e.g.:<br>
a = torch.rand(1000)<br>
b = torch.rand(1000)<br>
print(torch.dot(a,b) == torch.sum(a * b)),</p>
<p>Thanks!</p>

fadetoblack: <p>Welcome to the community!</p>
<p>The values of both those expression are not equal after a certain decimal point. Please run the following to understand:</p>
<pre><code class="lang-auto">a = torch.rand(1000)
b = torch.rand(1000)

x1 = torch.dot(a,b)
x2 = torch.sum(a*b)

torch.set_printoptions(precision=10)

print(x1.item())
print(x2.item())
</code></pre>
How to update weight of model A based on gradient of model B

Toan_Bui: <p>Model A and B have same structure, same initialization. (A does not has data).<br>
After model B trains and do model_B.backward(), it has gradient.<br>
How to copy all gradient from B to A so that after optimizer_A.step() and optimizer_B.step(), both have same weight.<br>
I tried:</p>
<pre><code class="lang-auto">for A, B in zip(model_A.named_parameters(), model_B.named_parameters()):
    A[1].grad = B[1].grad
</code></pre>
<p>But it didn’t work. The result is like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/9/593bd1f9e31514f65d45c0c001227c714d0591fc.png" alt="image" data-base62-sha1="cJoERzDRaAEH4yk2NL1QqWFpqaw" width="224" height="356"><br>
On each round, the first line is accuracy of A and the second line is accuracy of B.</p>
Optimize a part of a tensor with selected index

MoyGcc: <p>Hey,</p>
<p>I want to optimize a tensor but not all elements of that tensor. For example:</p>
<p>test_tensor = torch.zeros((1, 10475,3), requires_grad = True)<br>
optimizer = torch.optim.Adam([test_tensor[:,rest_idx,:]], 0.005, betas=(0.9, 0.999))</p>
<p>where rest_idx is a sorted array with any possible elements between 0 and 10474. The code I wrote above will not work because the slices are no leafs anymore. The way that I divide the test_tensor into two parts firstly and concatenate two parts of this tensor afterward is not applicable I guess because the selected indices can be very random.</p>
<p>Is there any efficient way to solve this problem? Thanks a lot in advance!</p>
<p>Best</p>

googlebot: <p>binary_mask = torch.zeros_like(test_leaf)<br>
binary_mask[idx].fill_(1.)<br>
test_tensor = test_leaf * binary_mask<br>
<span class="hashtag">#if</span> other values are needed<br>
test_tensor += test_leaf.detach() * binary_mask.logical_not()</p>

MoyGcc: <p>Hey Alex,</p>
<p>could you explain further? I think the way you suggested is only to set the selected elements of the test_tensor to zero. I do want the rest parts to not be optimized.</p>

googlebot: <p>given test_leaf=[1,2,3] and mask [1,1,0], that code does<br>
test_tensor = [1,2,0] + [0,0,3] = [1,2,3]</p>
<p>because the second summand was detached, test_leaf’s gradient will be zero at positions where mask is zero</p>
<p>note that usual adaptive optimizers like Adam won’t calculate gradient moments correctly for such parameters. There is optim.SparseAdam, but I’m not sure how to use it.</p>
nn.TransformerEncoderLayer input/output shape

lisyuan: <p>In the <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoderLayer.html" rel="noopener nofollow ugc">official website</a>, it mentions that the <code>nn.TransformerEncoderLayer</code> is made up of self-attention layers and feedforward network. The first is self-attention layer, and it’s followed by feed-forward network. Here are some input parameters and example</p>
<ul>
<li>
<strong>d_model</strong>  – the number of expected features in the input (required).</li>
<li>
<strong>dim_feedforward</strong> -  the dimension of the feedforward network model (default=2048)</li>
</ul>
<pre><code class="lang-auto">encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8, dim_feedforward=1024)
src = torch.rand(10, 32, 512) 
out = encoder_layer(src)
print(out.shape)
</code></pre>
<p>the output shape is</p>
<pre><code class="lang-auto">torch.Size([10, 32, 512])
</code></pre>
<p>there are two questions</p>
<ol>
<li>Why is the output shape [10, 32, 512]? Isn’t it [10, 32, 1024] because the feedforward network is connected after self-attention layer ?</li>
<li>In this example, is the input shape [10, 32, 512] corresponding to [batchsize, seq_length, embedding]?<br>
In my experience, using <code>nn.Embedding</code> the shape is [batchsize, seq_length, embedding]<br>
but it looks like the shape is [seq_length, batchsize, embedding] in this <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">tutorial</a>.</li>
</ol>

phan_phan: <p>Hi,</p>
<ol>
<li>The TransformerEncoder “<em>transforms</em>” each input embeddings with the help of neighboring embeddings in the sequence, so it is normal that the output is homogeneous with the input : it should be the same shape as the input.<br>
You can look at the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/transformer.html#TransformerEncoderLayer" rel="noopener nofollow ugc">implementation of <code>nn.TransformerEncoderLayer</code></a> for more details : you can see where <code>dim_feedforward</code> is used.</li>
<li>See the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html#torch.nn.Transformer" rel="noopener nofollow ugc">full <code>nn.Transformer</code> documentation</a> for details on required shapes : It is effectively <code>[seq, batch, emb]</code>.<br>
Note that <code>nn.Embedding</code> works with any shape : <code>[size1, size2] -&gt; [size1, size2, emb]</code><br>
So you can use <code>[seq, batch, emb]</code> or <code>[batch, seq, emb]</code> for <code>nn.Embedding</code>.</li>
</ol>

lisyuan: <p>Thank you very much!! This really helps me a lot</p>