akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to create bilingual LSTM

username: nkn002
cooked: <p>I am trying to create a bilingual LSTM that is trained on a bilingual corpus in English and Spanish. For monolingual datasets in English, we only need to use a pre-trained English embedding like fasttext to get the corresponding word vectors from the ids from the tokenized text. For example, for the sentence “I like dog”, I could tokenize them into [“i”, “like”, “dog”], get the corresponding ID to retrieve the vectors.</p>
<p>However, I don’t think we can do that for bilingual datasets, so I need a bilingual embedding in English and Spanish. For example, for the sentence “I like dog, me gusta el perro.”, I could tokenize them into [“i”, “like”, “dog”, “me”, “gusta”, “el”, “perro”]. But, I don’t know what to do next.</p>
<p>However, another problem arises: I have already looked at commonly used multilingual embedding techniques like MUSE. However, I am not quite sure how to incorporate them into my model. Another approach I have in mind is to create a joint vocab in English and Spanish and train an embedding from scratch based on that.</p>
<p>So, my questions are (1) How can I use multilingual embedding in English and Spanish from MUSE to process the sentence “I like dog, me gusta el perro.” in the example, and (2) Does create a joint vocab embedding work for this problem?</p>

username: J_Johnson
cooked: <p>Work through the tutorial examples:</p>
<p><a href="https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html</a></p>
<p>They will likely answer most of your questions.</p>
<p>Additionally, have a look at the paper “Attention is All You Need(2017)”. This might persuade you to consider Transformers instead of LSTMs. Pytorch has tutorials for those, too.</p>
How to index for sparse tensor

username: 111319
cooked: <p>I want to index a sparse tensor and propagate gradients through it.<br>
Now, I implement it like this:</p>
<pre><code class="lang-auto"> import torch

i = torch.tensor([[0, 1]])
v = torch.tensor([2., 2.], requires_grad=True)
s = torch.sparse_coo_tensor(i, v, size=[4], requires_grad=True)

print(s)
r = torch.index_select(s, 0, torch.tensor([1, 2])).to_dense()
print(r)
r.sum().backward()
print(v.grad)
</code></pre>
<p>But it returns the following error when call the “backward()”:</p>
<pre><code class="lang-auto">tensor(indices=tensor([[0, 1]]),
       values=tensor([2., 2.]),
       size=(4,), nnz=2, layout=torch.sparse_coo,
       grad_fn=&lt;SparseCooTensorWithDimsAndTensorsBackward0&gt;)
tensor([2., 0.], grad_fn=&lt;ToDenseBackward0&gt;)
Traceback (most recent call last):
  File "/home/pb/slidr/test.py", line 11, in &lt;module&gt;
    r.sum().backward()
  File "/home/pb/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/pb/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Tensors of type SparseTensorImpl do not have strides
</code></pre>
<p>I need a dense results after “index_select”, so I called the “to_dense()”. Is there a way to realize indexing through a list for sparse tensors and support back-propagation?</p>
Multiprocessing Across Multiple Dataloaders

username: ParasKoundal
cooked: <p>I have a dataset, which I can weight in different ways.</p>
<p>I want to test performance of model (no training, only testing) on different weighting schemes.</p>
<p>Hence, I want to run different Dataloaders (for different weighting schemes ; done using sampler in Dataloader) in parallel (using multiprocessing).</p>
<p>Is there a working example of such use case.</p>
Is it feasible to call nn.Module().forward in an autograd.Function's forward?

username: yzs
cooked: <p>The scenario is to add a hook in the model forward and backward. Is it feasible to call <code>real_model(x)</code> inside <code>hook_func.func()</code> to perform forward of <code>real_model</code> as bellow?</p>
<p>Pseudo code:</p>
<pre><code class="lang-python">class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        # model init
        ...
    
    def forward(self, x):
        # computation graph and returns x
        return x

real_model = Model()

class hook_func(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        # some operation on x
        print(x)
        # call the real forward of a custom nn.Module()
        x = real_model(x)
        return x
    ...
</code></pre>

username: soulitzer
cooked: <p>If your use case is to hook into the module’s forward and backward, check out the note on module hooks here <a href="https://pytorch.org/docs/master/notes/modules.html#module-hooks" class="inline-onebox" rel="noopener nofollow ugc">Modules — PyTorch master documentation</a></p>

username: yzs
cooked: <p>Thanks for your information! Unfortunately, the <code>register_full_backward_hook()</code> can only be called <strong>after</strong> backward, which cannot fully satisfy my needs. Currently, I separate <code>hook_func</code> into two <code>torch.autograd.Function</code> and customize the Module.</p>
RuntimError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 1

username: srd
cooked: <p>I’ve read all the similar topics here, but can’t figures this out. How do I change my labels (y_train) to the correct dimensions?</p>
<pre><code class="lang-auto">data = ImageFolder(data_dir, transform=transforms.Compose([transforms.Resize((224,224)),transforms.ToTensor()]))

trainloader = torch.utils.data.DataLoader(data, batch_size=3600,
                                          shuffle=True, num_workers=2)


dataiter = iter(trainloader)
x_train, y_train = dataiter.next()

print(x_train.size())
print(y_train.size())
</code></pre>
<p>torch.Size([3600, 3, 224, 224])<br>
torch.Size([3600])</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        # here we set up the tensors......
        self.layer1 = torch.nn.Linear(224, 12)
        self.layer2 = torch.nn.Linear(12, 10)

    def forward(self, x):
        # here we define the (forward) computational graph,
        # in terms of the tensors, and elt-wise non-linearities
        x = F.relu(self.layer1(x))
        x = self.layer2(x)
        return x

net = Net()
y = net.forward(x_train)
lossFn = torch.nn.CrossEntropyLoss() 
loss = lossFn(y, y_train)
print(loss)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c66553d6bb1aeb2a162968f9469ced985f050dc9.png" data-download-href="https://discuss.pytorch.org/uploads/default/c66553d6bb1aeb2a162968f9469ced985f050dc9" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c66553d6bb1aeb2a162968f9469ced985f050dc9.png" alt="image" data-base62-sha1="sj5DcAiQhY1cXwvGCyPTbSc5tKp" width="690" height="258" data-dominant-color="373738"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1328×497 27.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>You didn’t explain your use case so I guess you are working on a multi-class classification. If that’s the case, keep the target tensor shape but flatten the input tensor to a 2-dimensional one and fix the <code>in_features</code> of the first linear layer in the model as it’ll raise a shape mismatch.</p>
How to sort a tensor of n dimension

username: Sumesh_Sankar
cooked: <p>I have a tensor  like this  [[1,2],[-1,2],[1.3,3],[-2,5]] i have to sort this tensor with respect to the first value(ie, 1,-1,1.3,-2) while not changing the other value in the pair. The result should looks like this [[-2.5],[-1,2],[1,2],[1.3,3]].</p>

username: tom
cooked: <p>You could use something like</p>
<pre data-code-wrap="python3"><code class="lang-nohighlight">t = torch.tensor([[1,2],[-1,2],[1.3,3],[-2,5]])
idx = t[:,0].argsort()
t_sorted = t[idx]
t_sorted
</code></pre>
<p>Best regards</p>
<p>Thomas</p>
When freeze layers of Resnet50(pretrained), are there total 10 layers to freeze?

username: Jus
cooked: <p>I found a good explanation of freezing resnet50 layers.<br>
I am wondering that as far as I know, there are 50 layers inside resnet50, evev if I consider conv layers and fc layer, at least there are 20 layers to freeze grads.<br>
Please teach me how come resnet50 total 10 layers as we freeze the model.<br>
Thank you in advance!! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Below is the post I saw!</p>
<pre><code class="lang-auto">model_ft = models.resnet50(pretrained=True)
ct = 0
for child in model_ft.children():
    ct += 1
    if ct &lt; 7:
        for param in child.parameters():
            param.requires_grad = False
This freezes layers 1-6 in the total 10 layers of Resnet50</code></pre>

username: ptrblck
cooked: <p>You can <code>print</code> the model and would see all defined layers of it.<br>
<a href="https://arxiv.org/pdf/1512.03385.pdf">This paper</a> shows the architecture in Figure 3 for ResNet34 and you can see that each “conv block” is defined as a layer.</p>

username: TTAyanlade
cooked: <p>Thanks for the response and time. I now understand the conv clock representation from the paper. I am just curious as well to know why there are 10 children (which everyone translates as layers) instead of 50 in resnet50 or say 34 for resnet34.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Each registered block is counted as a child module and contains more registered modules internally. If you would flatten these blocks you’ll get the corresponding 50 or 34 layers. The current implementation reuses common blocks for a better and clean coding style instead of manually defining each of the 50 layers separately.</p>
"buffer is not large enough" error

username: lule
cooked: <p>When running on <code>device = 'mps'</code> on my M1 macbook, I get the following error:</p>
<pre><code class="lang-auto">/AppleInternal/Library/BuildRoots/a0876c02-1788-11ed-b9c4-96898e02b808/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSCore/Types/MPSNDArray.mm:782: failed assertion `[MPSNDArray, initWithBuffer:descriptor:] Error: buffer is not large enough. Must be 8000 bytes
</code></pre>
<p>This seems to occur when running the following code (I had to do this workaround of creating a one-hot encoding with numpy and then converting it back to a <code>torch.Tensor</code> because the built-in pytorch one-hot converter seems not to be supported for <code>device = 'mps'</code> yet):</p>
<pre><code class="lang-auto">gen_labels = np.random.randint(0, n_classes, current_batchSize)
cls_one_hot = np.zeros((gen_labels.size, gen_labels.max() + 1))
cls_one_hot[np.arange(gen_labels.size), gen_labels] = 1
gen_labels = torch.Tensor(gen_labels).to(device)
cls_one_hot = torch.Tensor(cls_one_hot).to(device)
cls_one_hot = cls_one_hot.type(torch.LongTensor)
</code></pre>
<p>I am on Pytorch version 1.13.0.dev20220713.<br>
Thankful for any help!</p>
How to compile in debug mode?

username: babuya
cooked: <p>Hi, I wanted to transfer pytorch model to torchscript, so I used</p>
<pre><code class="lang-auto">test_model = torch.jit.script(model)
</code></pre>
<p>which model is a <em>.pth</em> checkpoint.<br>
However, I got following errors:</p>
<pre><code class="lang-auto">File "/xxx.py", line 54, in eval_one_epoch
    test_model = torch.jit.script(model)
  File "/xx/anaconda3/envs/xxx/lib/python3.7/site-packages/torch/jit/_script.py", line 898, in script
    obj, torch.jit._recursive.infer_methods_to_compile
  File "/xx/anaconda3/envs/xxx/lib/python3.7/site-packages/torch/jit/_recursive.py", line 352, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, stubs_fn)
  File "/xx/anaconda3/envs/xxx/lib/python3.7/site-packages/torch/jit/_recursive.py", line 406, in create_script_module_impl
    script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
  File "/xx/anaconda3/envs/xxx/lib/python3.7/site-packages/torch/jit/_script.py", line 388, in _construct
    init_fn(script_module)
  File "/xx/anaconda3/envs/xxx/lib/python3.7/site-packages/torch/jit/_recursive.py", line 373, in init_fn
    cpp_module.setattr(name, orig_value)
RuntimeError: Unable to cast Python instance to C++ type (compile in debug mode for details)
</code></pre>
<p>It seemed to tell no more useful trace back. What means to compile in debug mode and how can I do? Or any other suggestions?</p>

username: ptrblck
cooked: <p>I guess the error message suggests to rebuild PyTorch from source with <code>DEBUG=1</code> to get more debug information. Could you post the model definition which raises this error, please?</p>
Fail to build torchvision from source on linux

username: Tianyu9748
cooked: <p>NAME=“CentOS Linux”<br>
VERSION=“7 (Core)”<br>
CUDA=11.4<br>
cuDNN=8.4.1<br>
cmake=3.23.4<br>
g++=9.3.0<br>
torch=1.14.0a0+git4dc5798</p>
<p>Command I used is:<br>
<code>python setup.py install</code></p>
<p>``<br>
/bin/ld: /usr/local/contrib/lib/libswscale.a(scale.o): relocation R_X86_64_32 against `.rodata’ can not be used when making a shared object; recompile with -fPIC</p>
<p>/bin/ld: /usr/local/contrib/lib/libswscale.a(yuv2rgb.o): relocation R_X86_64_32S against symbol `ff_dither8’ can not be used when making a shared object; recompile with -fPIC</p>
<p>/bin/ld: /usr/local/contrib/lib/libswscale.a(gamma.o): relocation R_X86_64_32S against `.text’ can not be used when making a shared object; recompile with -fPIC</p>
<p>/bin/ld: /usr/local/contrib/lib/libswscale.a(hscale.o): relocation R_X86_64_32S against `.text’ can not be used when making a shared object; recompile with -fPIC</p>
<p>/bin/ld: final link failed: Nonrepresentable section on output<br>
collect2: error: ld returned 1 exit status<br>
error: command ‘/afs/usr0/home/GCC9/bin/g++’ failed with exit code 1<br>
``</p>
<p>Is there anyone had this issue before?</p>
<p>Best<br>
Max</p>
ImageFolder return type

username: Ishal_Garg
cooked: <p>What does Datasets.ImageFolder() returns</p>

username: theairbend3r
cooked: <p>It returns Dataset object with contains tuples =&gt; <code>(image, class)</code>.<br>
<a href="https://pytorch.org/docs/stable/torchvision/datasets.html#torchvision.datasets.ImageFolder" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/datasets.html#torchvision.datasets.ImageFolder</a></p>

username: Ishal_Garg
cooked: <p>Thanks. But as i was doing a competition on kaggle. Dataset was a folder of images with as imageID and a csv file of labels with the respective imageID. As i loaded the images from the folder which only had images no classes. How would classes of the images be refered. And all the classes were zero</p>

username: theairbend3r
cooked: <p><code>datasets.</code> <code>ImageFolder</code> will infer classes ‘cat’ and ‘dog’ if your directory is organized as follows -</p>
<pre><code class="lang-auto">root/dog/xxx.png
root/dog/xxy.png
root/dog/xxz.png

root/cat/123.png
root/cat/nsdf3.png
root/cat/asd932_.png
</code></pre>
<p>Here’s a related thread - <a href="https://discuss.pytorch.org/t/custom-dataset-labeling-from-csv/55565/2" class="inline-onebox">Custom Dataset labeling from CSV</a></p>

username: Ishal_Garg
cooked: <p>Thanks for the link. That is what i m having trouble in.</p>

username: TTAyanlade
cooked: <p>Thanks airbend3r. I would also like to add that it does not return that, but the information can be gotten using the “.imgs” attribute.</p>
Classification accuracy is significantly different on different platforms

username: Neo
cooked: <p>Greetings,</p>
<p>I have a two-classes classification problem where I use the following model in training and testing:</p>
<p>def get_model():<br>
# Load the model based on VGG19<br>
vgg_based = torchvision.models.vgg19(pretrained=True)<br>
# freeze the layers<br>
<span class="hashtag">#for</span> param in vgg_based.parameters():<br>
#   param.requires_grad = False<br>
# Modify the last layer<br>
number_features = vgg_based.classifier[6].in_features<br>
features = list(vgg_based.classifier.children())[:-1] # Remove last layer<br>
features.extend([nn.Linear(number_features, 1), nn.Sigmoid()])<br>
vgg_based.classifier = torch.nn.Sequential(*features)<br>
return vgg_based</p>
<p>The question is why I am getting different classification accuracies on the testing set: first is 94.5% and the second is 74%?<br>
The first platform is Windows, python 3.8, pytorch 1.12.1<br>
The second platform is Ubuntu on AWS, python 3.9, pytorch 1.12.1</p>
<p>I use shuffling and augmentation on the training set. I repeated the experiments several times and I get the same different results. The code and the data are the same.</p>
Transfer different shape weights?

username: 3DTOPO
cooked: <p>It seems all the answers I can find on the subject essentially state to not transfer weights if the models have different shapes, but I would like to try.</p>
<p>For my case, it either will usable or not but I feel like it is worth trying and in any event I might learn something.</p>
<p>Let’s say I have two RRDBNet models, one with 32 feature layers and the other with 64, all else identical:</p>
<pre><code class="lang-auto">model32 = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=32, num_block=23, num_grow_ch=32)
model64 = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=23, num_grow_ch=32)

loadnet32 = torch.load(model_path32, map_location=torch.device('cpu'))
loadnet64 = torch.load(model_path64, map_location=torch.device('cpu'))

model32.load_state_dict(loadnet32["params_ema"], strict=False)
model64.load_state_dict(loadnet64["params_ema"], strict=True)

for n, p in model64.named_parameters():
   ...interpolate weights (or skip every other value)?
</code></pre>
Different gradients

username: InternalInstance
cooked: <p>To overcome a memory problem I split my training data into batches and process it in a sequential manner instead of feeding the whole data into the network.<br>
But I get different gradients between the first and second examples given below.<br>
I think that they are mathematically equivalent and should give the same result. Сorrect me if i’m wrong.<br>
1.</p>
<pre><code class="lang-auto">for epoch in range(epochs):
    optimizer.zero_grad()
    for b_data in splitted_data:
        ...
        out = model(b_data)
        err = loss_fn(out, gt) / number_of_batches
        err.backward()
    optimizer.step()
</code></pre>
<ol start="2">
<li>
</li>
</ol>
<pre><code class="lang-auto">for epoch in range(epochs):
    optimizer.zero_grad()
    out = model(all_data)
    err = loss_fn(out, target)
    err.backward()
    optimizer.step()
</code></pre>

username: imxtx
cooked: <p>I think the accumulated total error in the first example equals the error in the second example divided by number of batches in the first example.</p>

username: soulitzer
cooked: <p><a class="mention" href="/u/imxtx">@imxtx</a> is correct, you should not need to divide by number_of_batches in the first example. The amount of accumulated gradient is proportional to the batch size, so the two would be equivalent without this extra scaling.</p>
How to get sorted indices

username: zzzf
cooked: <p>According to the documentation of <code>argsort</code></p>
<pre><code class="lang-auto">a = torch.randn(4, 4)
a
tensor([[ 0.0785,  1.5267, -0.8521,  0.4065],
        [ 0.1598,  0.0788, -0.0745, -1.2700],
        [ 1.2208,  1.0722, -0.7064,  1.2564],
        [ 0.0669, -0.2318, -0.8229, -0.9280]])

torch.argsort(a, dim=1)
tensor([[2, 0, 3, 1],
        [3, 2, 1, 0],
        [2, 1, 0, 3],
        [3, 2, 1, 0]])
</code></pre>
<p>However, I want the sorted indices. That is</p>
<pre><code class="lang-auto">tensor([[1, 3, 0, 2],
        [3, 2, 1, 0],
        [2, 1, 0, 3],
        [3, 2, 1, 0]])
</code></pre>

username: KFrank
cooked: <p>Hi Z!</p>
<aside class="quote no-group" data-username="zzzf" data-post="1" data-topic="163911" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zzzf/48/14158_2.png" class="avatar"> zzzf:</div>
<blockquote>
<p>However, I want the sorted indices. That is</p>
<pre><code class="lang-auto">tensor([[1, 3, 0, 2],
        [3, 2, 1, 0],
        [2, 1, 0, 3],
        [3, 2, 1, 0]])
</code></pre>
</blockquote>
</aside>
<p>Slap on a second <code>argsort()</code> to get (the indices of) your sorted indices:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.12.0'
&gt;&gt;&gt; a = torch.tensor([[ 0.0785,  1.5267, -0.8521,  0.4065],
...         [ 0.1598,  0.0788, -0.0745, -1.2700],
...         [ 1.2208,  1.0722, -0.7064,  1.2564],
...         [ 0.0669, -0.2318, -0.8229, -0.9280]])
&gt;&gt;&gt; a.argsort (dim = 1)
tensor([[2, 0, 3, 1],
        [3, 2, 1, 0],
        [2, 1, 0, 3],
        [3, 2, 1, 0]])
&gt;&gt;&gt; a.argsort (dim = 1).argsort (dim = 1)
tensor([[1, 3, 0, 2],
        [3, 2, 1, 0],
        [2, 1, 0, 3],
        [3, 2, 1, 0]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Error in exporting onnx

username: rabbit_hog
cooked: <p>I try to export quantized shufflenetv2 to onnx file and get error as below. It seems that this error only happens in quantized shufflenetv2 because I succeed in exporting float shufflenetv2.</p>
<p>RuntimeError: ONNX symbolic expected node type <code>prim::TupleConstruct</code>, got <code>%414 : Tensor(*, *, *, *) = onnx::Slice(%x.8, %410, %413, %402), scope: __module.stage2/__module.stage2.1 # D:\ProgramData\Anaconda3\envs\nanodet\lib\site-packages\torchvision\models\quantization\shufflenetv2.py:42:0 </code></p>

username: ddang
cooked: <p>see <a href="https://discuss.pytorch.org/t/quantized-pytorch-model-exports-to-onnx/144968/5" class="inline-onebox">Quantized Pytorch model exports to onnx - #5 by Deepak_Ghimire1</a><br>
we do not support ONNX</p>
Namedtuple in C++ interface

username: Siddhesh_Mhatre
cooked: <p>Hi,</p>
<p>I have traced and saved a model that gives as output a python namedtuple. How do I go about accessing the fields of this output in the C++ api?</p>
<p>For eg -<br>
from collections import named_tuple<br>
NT = namedtuple(‘NT’, [‘output1’, ‘output2’])<br>
def f(x):<br>
output_tuple = NT(x * 2, x / 2)<br>
return output_tuple</p>
<p>I am able to trace and save the above function using torch.jit.trace and would like to index into this output using the keyword arguments for NT in C++.</p>
<p>Thanks</p>

username: driazati
cooked: <p><code>namedtuple</code> does not work out of the box yet (though it is on our roadmap). In your example it is de-sugared to a regular tuple. You can see what’s going on under the hood with the <code>.code</code> property of the traced code. For example</p>
<pre><code class="lang-python">from collections import namedtuple
MyTuple = namedtuple('MyTuple', ['output1', 'output2'])
def f(x):
    output_tuple = MyTuple(x * 2, x / 2)
    return output_tuple
traced = torch.jit.trace(f, (torch.ones(2, 2)))
print(traced.code)
</code></pre>
<p>Outputs</p>
<pre><code class="lang-auto">def forward(self,
    x: Tensor) -&gt; Tuple[Tensor, Tensor]:
  _0 = (torch.mul(x, CONSTANTS.c0), torch.div(x, CONSTANTS.c0))
  return _0
</code></pre>
<p>In C++ you can access the tuple like this</p>
<pre><code class="lang-cpp">torch::IValue output = my_module-&gt;forward(...);
std::vector&lt;torch::IValue&gt;&amp; tuple_elements = output-&gt;toTuple().elements();
</code></pre>

username: dorianhenning
cooked: <p>Thanks <a class="mention" href="/u/driazati">@driazati</a> , this almost got me there! I got compilation errors when I used your snippet and had to change it to either:</p>
<ol>
<li>make the <code>lvalue</code> reference constant:</li>
</ol>
<pre><code class="lang-auto">const std::vector&lt;torch::IValue&gt; &amp; tuple_elements = ...
</code></pre>
<ol start="2">
<li>or remove the reference and use a normal vector with copy construction:</li>
</ol>
<pre><code class="lang-auto">std::vector&lt;torch::IValue&gt; tuple_elements = ...
</code></pre>
<p>Furthermore, the rest of the line should be changed as well:</p>
<pre><code class="lang-auto">... = output.toTuple()-&gt;elements()
</code></pre>
<p><code>output</code> is a <code>torch::IValue</code>/<code>c10::IValue</code>, not a pointer, so pointer access does not work. However, <code>.toTuple()</code> returns a <code>c10::intrusive_ptr&lt; ...&gt;</code> and you can access its member functions with <code>-&gt;</code>.</p>
Dataloader with multitask loss?

username: Prakhar_Sharma
cooked: <p>I am trying to implement minibatch using <code>torch.utils.data.DataLoader</code>. I have a multi-task loss where each loss term requires an input of different size. More specifically, I am constructing a Physics-informed neural network.</p>
<p>For example, for loss_1, I have 5000 samples, and for loss_2 I have 50 samples. I have both these samples in form of <code>npz</code> files.</p>
<p>I have implemented my training epochs without the minibatch as follows:</p>
<pre><code class="lang-python">for i in range(max_iter):
    loss_1 = loss_A (dataset_1, output_1)  # minimises the third derivative of output using torch.auograd.grad()
    loss_2 = loss_B (dataset_2, output_2)  # standard MSE loss
    total_loss = loss_1 +  loss_2
    # minimise the total loss
</code></pre>
<p>How do I implement this with a minibatch using the <code>torch.utils.data.DataLoader</code>? In minibatch, you backpropagate the loss for each batch as follows:</p>
<pre><code class="lang-python">for i in range(max_epoch):
    training_loader = torch.utils.data.DataLoader(training_set, batch_size=4, shuffle=True)
    for i, data in enumerate(training_loader):
        inputs, labels = data
        loss_1 = loss_A (inputs, output_1) 
        # minimise the total loss in each batch
</code></pre>
<p>I want to access minibatches for two different datasets simultaneously. I hope my problem is clear. I know there are <a href="https://github.com/lululxvi/deepxde/blob/d1649313f93e9c80d858d33ba616cdb1e76bec17/deepxde/data/sampler.py" rel="noopener nofollow ugc">solutions</a>, just wanted to ask if someone has some trick.</p>
Knowledge Distillation

username: ageryw
cooked: <p>Hello everyone, I’m using knowledge distillation to train a model. The teacher model has previously been trained and is designed to guide the student. However, when the student trains, the total loss is negative. Is this normal behaviour during training? Any suggestions are welcome. Thank you very much.<br>
Here is a section of the code.</p>
<pre><code class="lang-auto">temp = 5
alpha = 0.9
for phase in ['train', 'val']:
    ......
    losses = []
    for data, targets in pbar:
        data = data.to(device)
        targets = targets.to(device)
    
        with torch.no_grad():
            teacher_preds = teacher(data)
    
        student_preds = student(data)
        _, preds = torch.max(student_preds, 1)
        student_loss = student_loss_fn(student_preds, targets)

        ditillation_loss = divergence_loss_fn(
            F.softmax(student_preds / temp, dim=1),
            F.softmax(teacher_preds / temp, dim=1)
        )

        loss = alpha * student_loss + (1-alpha) * ditillation_loss
        losses.append(loss.item())
        running_corrects += torch.sum(preds == targets.data)
        curr_train_samples += len(targets)
        # backward
        optimizer.zero_grad()
        if phase == 'train':
            loss.backward()
            optimizer.step()

    epoch_loss = sum(losses) / len(losses)
    epoch_acc = running_corrects.double() / curr_train_samples
     ....
    print(f'\t{phase}: Loss: {epoch_loss:.4f} Acc: {epoch_acc:.4f}')
</code></pre>
<p>Sample output</p>
<blockquote>
<p>Epoch 1</p>
<blockquote>
<p>train: Loss: -0.2539 Acc: 0.3342<br>
val:    Loss: -0.2725 Acc: 0.4614</p>
</blockquote>
</blockquote>
<blockquote>
<p>Epoch 2</p>
<blockquote>
<p>train: Loss: -0.4222 Acc: 0.5226<br>
val:    Loss: -0.4394 Acc: 0.5932</p>
</blockquote>
</blockquote>

username: ageryw
cooked: <p>I think I found this one helpful, and solved the issue I faced.</p>
<aside class="quote" data-post="11" data-topic="16791">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mfajcik1/48/6772_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/kl-divergence-produces-negative-values/16791/11">KL Divergence produces negative values</a> 
  </div>
  <blockquote>
    Did you normalized values with log_softmax? 
torch.nn.KLDivLoss(size_average=False)(F.log_softmax(scores, -1), targets)
  </blockquote>
</aside>
Received 0 items of ancdata Pytorch 0.4.0

username: aradhyamathur
cooked: <p>Upon iterating over the dataloader I receive <code>RuntimeError: received 0 items of ancdata</code>. I tried by increasing number of open file descriptors as shown in <a href="https://github.com/pytorch/pytorch/issues/973" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/973</a> but to no avail. Can someone please guide how this error could be prevented.<br>
<code>x, (y_l, y_ab) = next(loader)</code><br>
where <code>loader</code> is defined as following<br>
<code>loader = cycle(train_dataloader)</code></p>
<p>The complete error traceback is given below.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main_mse.py", line 267, in &lt;module&gt;
    main()
  File "main_mse.py", line 263, in main
    train(g_model, d_model, learning_rate_ae, learning_rate_color, train_dataloader, test_dataloader, now)
  File "main_mse.py", line 111, in train
    x, (y_l, y_ab) = next(loader)
  File "/home/thor/anaconda3/envs/torch3env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 280, in __next__
    idx, batch = self._get_batch()
  File "/home/thor/anaconda3/envs/torch3env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 259, in _get_batch
    return self.data_queue.get()
  File "/home/thor/anaconda3/envs/torch3env/lib/python3.6/multiprocessing/queues.py", line 337, in get
    return _ForkingPickler.loads(res)
  File "/home/thor/anaconda3/envs/torch3env/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 70, in rebuild_storage_fd
    fd = df.detach()
  File "/home/thor/anaconda3/envs/torch3env/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/home/thor/anaconda3/envs/torch3env/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/home/thor/anaconda3/envs/torch3env/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
</code></pre>

username: penguinshin
cooked: <p>Also having this issue. My sharing strategy is file_descriptor</p>

username: aradhyamathur
cooked: <p>Can you share your snippet as well ?</p>

username: Diego999
cooked: <pre><code class="lang-auto">torch.multiprocessing.set_sharing_strategy('file_system')
</code></pre>
<p>solved the problem for me.</p>
<p>or</p>
<pre><code class="lang-auto">echo "ulimit -n 4096" &gt;&gt; .bashrc
echo "ulimit -n 4096" &gt;&gt; .bash_profile
source ~/.bashrc
</code></pre>

username: 47759476
cooked: <p>thank you for your method !!! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>
The impact of Beta value in adam optimizer

username: israrbacha
cooked: <p>Hello all, I went through StyleGAN2 implementation. In adam optimizer, they used Beta_1=0. What’s the reason behind the choice? in terms of sample quality or convergence speed?</p>

username: ptrblck
cooked: <p>I guess a hyperparameter turning showed this setup worked fine starting apparently in the ProgGAN implementation.</p>
<p><a href="https://arxiv.org/pdf/1912.04958.pdf">Analyzing and Improving the Image Quality of StyleGAN</a>:</p>
<blockquote>
<p>We kept most of the details unchanged […] Adam optimizer [25] with the same hyperparameters (β1 = 0, β2 = 0.99,  = 10−8, minibatch = 32) […]</p>
</blockquote>
<p><a href="https://arxiv.org/pdf/1812.04948.pdf">A Style-Based Generator Architecture for Generative Adversarial Networks</a>:</p>
<blockquote>
<p>We build upon the official TensorFlow [1] implementation of Progressive GANs by Karras et al. […] In particular, we use the same discriminator architecture, resolution-dependent minibatch sizes, Adam [33] hyperparameters, […]</p>
</blockquote>
<p><a href="https://arxiv.org/pdf/1710.10196.pdf">PROGRESSIVE GROWING OF GANS FOR IMPROVED QUALITY, STABILITY, AND VARIATION</a>:</p>
<blockquote>
<p>We train the networks using Adam (Kingma &amp; Ba, 2015) with α = 0.001, β1 = 0, β2 = 0.99,<br>
and  = 10−8.</p>
</blockquote>

username: israrbacha
cooked: <aside class="quote no-group quote-modified" data-username="ptrblck" data-post="2" data-topic="153757">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>We train the networks using Adam (Kingma &amp; Ba, 2015) with α = 0.001, β1 = 0, β2 = 0.99,<br>
and = 10−8.</p>
</blockquote>
</aside>
<p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>  for your response, I read these papers but they didn’t mention any scientific method to choose these hyperparameters, should we decide by trial and error? I read in a blog post that choosing beta somewhere near to 1 (i.e 0.98) speeds up convergence then why do we have to choose  Beta_1=0. And why not Beta_2?</p>

username: zhangmiaochang
cooked: <p>It also coufused me quite a while</p>
<p>Adam with β1 = 0, β2 = 0.99 is equvalent to RMSprop with alpha=0.99. (Adam ≈ RMSprop with momentum)<br>
what the original author (in [PROGRESSIVE GROWING OF GANS FOR IMPROVED QUALITY, STABILITY, AND VARIATION] , thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> ) really need is RMSprop, and he achieved his goal by the Adam optimizer.</p>
Can I explicitly set dtype for torch.nn.Conv2d?

username: han324solo
cooked: <p>How do I set dtype for torch.nn.Conv2d?<br>
It seems that default dtype for torcn.nn.Conv2d is float.</p>
<p>I want to explicitly set dtype for torch.nn.Conv2d ,<br>
as I set dtype for tensor as below:</p>
<blockquote>
<p>a = torch.tensor([1,2,3], dtype=torch.int8)</p>
</blockquote>

username: albanD
cooked: <p>Hi,</p>
<p>You can use <code>your_mod.type(dtype)</code> to change its type. Or <code>your_mod.double()</code> to change to double precision.<br>
Be careful that the conv module is not implemented for every types <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: han324solo
cooked: <p>Super simple<br>
Thanks!</p>

username: christophezei
cooked: <p>Hello thank you for this solution but can I specify which layers? since in my application I need some layers in int8 and the others I want to keep in float</p>

username: christophezei
cooked: <p>I found the solution if some one need it later on:</p>
<pre><code class="lang-auto">    with torch.no_grad():
        for layer_name, param in model.named_parameters()():
            if "weights" in layer_name: # you can apply any condition you want 
                param.data = param.type(torch.uint8)
</code></pre>
How to quantize torch.nn.functional.grid_sample?

username: vivian528899
cooked: <p>Hi all,<br>
I want to convert the inputs of grid_sample from float32 to quint8 but it seems that the pytorch doesn’t support grid_sample in quint8.<br>
How to quantize torch.nn.functional.grid_sample in quint8?<br>
It’s there any other operation that can replace torch.nn.functional.grid_sample and can be quantized in quint8?<br>
Below is the part of my code.<br>
Thanks.</p>
<blockquote>
<p>from re import T<br>
import torch.nn.functional as F<br>
import torch.nn as nn<br>
import torch<br>
from torch import Tensor</p>
<p>from torch.quantization import QuantStub, DeQuantStub<br>
import torch.quantization</p>
<p>use_quan = True</p>
<p>class FlowWarp(nn.Module):<br>
def <strong>init</strong>(self, interpolation=‘bilinear’, padding_mode=‘zeros’, align_corners=True):<br>
super().<strong>init</strong>()<br>
self.interpolation=interpolation<br>
self.padding_mode=padding_mode<br>
self.align_corners=align_corners</p>
<pre><code>    if(use_quan):
        self.quant = QuantStub()
        self.quant_x = QuantStub()
        self.dequant = DeQuantStub()
        self.f = torch.nn.quantized.FloatFunctional()

def forward(self, x, flow: Tensor) -&gt; Tensor:
    if flow.shape[-1] != 2:
        flow = flow.permute(0, 2, 3, 1)

    if(use_quan):
        x = self.dequant(x)
    n, c, h, w = x.size()

    grid_y, grid_x = torch.meshgrid(
        torch.linspace(-1, 1, h),
        torch.linspace(-1, 1, w),
        indexing='ij'
    )

    grid = torch.stack((grid_x, grid_y), 2).type_as(x).expand(n, h, w, 2)
    grid.requires_grad = False

    if(use_quan):
        grid = self.quant(grid)
        x = self.quant_x(x)
        grid_flow = self.f.add(grid, flow)
    else:
        grid_flow = grid + flow
    output = F.grid_sample(
        x,
        grid_flow,
        mode=self.interpolation,
        padding_mode=self.padding_mode,
        align_corners=self.align_corners
    )
    return output
</code></pre>
</blockquote>
How to collect custom batches

username: malfonsoarquimea
cooked: <p>Hi!<br>
I am currently working on a system that requires me to load batches composed of random pixels of a given set of images. More precisely, if there are N images in the dataset and batch size is B, for each epoch I need to collect N batches of size B, each one composed of B random pixels of the corresponding image. E.g. first batch is composed of B randomly selected pixels of the first image, etc.<br>
What would be the recommended way to do this?</p>

username: nivek
cooked: <p>I am not very familiar with this domain, but it seems like for each image, you can perform <code>B</code> <a href="https://pytorch.org/vision/main/generated/torchvision.transforms.RandomCrop.html" rel="noopener nofollow ugc">random crops</a> of the image to get a batch. You can potentially do that within <code>__getitem__</code> or <code>__iter__</code> of your custom Dataset (or DataPipe).</p>

username: malfonsoarquimea
cooked: <p>I ended up with this solution using a DistributedSampler</p>
<pre><code class="lang-auto">def train_dataloader(self):
    """returns a dataloader for training according to hparams

    Returns:
        DataLoader: DataLoader ready to deliver samples for training
    """
    # define a distributed sampler in case we are using multiple GPUs
    if self.hparams.num_gpus&gt;1:
        sampler = torch.utils.data.distributed.DistributedSampler(
            self.train_dataset, shuffle=False)
    # only use the sampler if using multiple GPUs
    return DataLoader(
        self.train_dataset,
        shuffle=False,
        num_workers=self.hparams.num_workers,
        batch_size=self.hparams.batch_size,
        pin_memory=False,
        sampler=sampler if self.hparams.num_gpus &gt; 1 else None)
</code></pre>
Visualize t-sne plot for last layer of features of CNN_LSTM network

username: Atefeh_Rashidi
cooked: <p>I have this CNN_LSTM classifier and I want to plot t-sne for the last layer  of this classifier (not the classification layer, last features layer. Something <a href="https://becominghuman.ai/visualizing-representations-bd9b62447e38" rel="noopener nofollow ugc">like this</a> but in PyTorch, not Keras). How can I do that?</p>
<pre><code class="lang-auto">class CNN_LSTM(nn.Module):

  def __init__(self,vocab_vectors,padding_idx,batch_size):
      super(Sentiment,self).__init__()
      print('Vocab vectors size:',vocab_vectors.shape)
      self.batch_size = batch_size
      self.hidden_dim = 128
      
      self.embedding = nn.Embedding.from_pretrained(vocab_vectors)
      self.embedding.weight.requires_grad = False
      self.embedding.padding_idx = padding_idx
      
      self.cnns =  nn.ModuleList([nn.Conv1d(in_channels=vocab_vectors.shape[1], out_channels=32, kernel_size=k) for k in [3,4,5]])
      
      self.lstm = nn.LSTM(input_size=32, 
                          hidden_size=self.hidden_dim,                             
                          bidirectional=True,  
                          batch_first=True)        
      self.linear1 = nn.Linear(2*self.hidden_dim,self.hidden_dim)        
      self.linear2 = nn.Linear(self.hidden_dim,1)
      self.dropout = nn.Dropout(0.2)

  @staticmethod
  def conv_and_max_pool(x, conv):
      """Convolution and global max pooling layer"""
      return F.max_pool1d(F.elu(conv(x)),1).permute(0, 2, 1)
      
  # https://github.com/gaussic/text-classification/blob/master/cnn_pytorch.py
  def forward(self,x):
      hidden = (torch.zeros(2, x.shape[0], self.hidden_dim).cuda(),
              torch.zeros(2, x.shape[0], self.hidden_dim).cuda())
      e = self.embedding(x)
      # Conv1d takes in (batch, channels, seq_len), but raw embedded is (batch, seq_len, channels)
      e = e.permute(0,2,1)
      cnn_outs = []
      for conv in self.cnns:
          f =self.conv_and_max_pool(e,conv)
          cnn_outs.append(f)
      out = torch.cat(cnn_outs, dim=1).cuda()
      print(out.shape)
      _, hidden = self.lstm(out,hidden)
      out = torch.cat((hidden[0][-2,:,:], hidden[0][-1,:,:]), dim=1).cuda()
      
      return self.linear2(self.dropout(F.relu(self.linear1(out))))
</code></pre>
How to generate vision transformer attention maps for 3D grayscale MRI data

username: Yuda_Bi
cooked: <p>My problem is how can I generate attention maps for 3D grayscale MRI data after training with vision transformer for a classification problem?</p>
<p>My data shape is (120,120,120) and the model is 3D ViT. For example:</p>
<pre><code class="lang-auto">img = nib.load()
img = torch.from_numpy(img)
model = torch.load...
model.eval()

output, attn = model(img)
</code></pre>
<p>After this, because I have 6 transformer layers and 12 heads, so the attn I got the shape that is</p>
<pre><code class="lang-auto">(6,12,65,65)
</code></pre>
<p>Then I don’t know how to apply this to original 3D grayscale images. I got several examples online that only deal with images from ImageNet.</p>
<p>For example:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/tczhangzhi/VisionTransformer-Pytorch">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/tczhangzhi/VisionTransformer-Pytorch" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/bd829119ce835fe211e24f027aa3d55b06745217c7e788be40e6d9b8d6d68ab2/tczhangzhi/VisionTransformer-Pytorch" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/tczhangzhi/VisionTransformer-Pytorch" target="_blank" rel="noopener nofollow ugc">GitHub - tczhangzhi/VisionTransformer-Pytorch</a></h3>

  <p>Contribute to tczhangzhi/VisionTransformer-Pytorch development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/jacobgil/vit-explain">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/jacobgil/vit-explain" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/af019407cd26ad8aca86977564c55f56d9868a41767db99a5576537c4edf5a0f/jacobgil/vit-explain" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/jacobgil/vit-explain" target="_blank" rel="noopener nofollow ugc">GitHub - jacobgil/vit-explain: Explainability for Vision Transformers</a></h3>

  <p>Explainability for Vision Transformers. Contribute to jacobgil/vit-explain development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Can anyone help me with this?</p>
How to make a 2D toy example like this?

username: BobKim
cooked: <p>Hello everyone!<br>
I want to make a 2D toy example like this</p>
<p><a href="https://clovaai.github.io/AdamP/static/img/momentum90.mp4" rel="noopener nofollow ugc">2D toy example video</a><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/6/567c8503586cae7d0b73dae602ce2e5a111c4823.png" alt="image" data-base62-sha1="cl5Qm06ey05DoPhTwHncGKmDsZ5" width="313" height="326"></p>
<p>This problem is maximize the cosine similarity between two vectors.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18db90068ad4446362a22e8ed1a9b3cae04d137a.png" alt="image" data-base62-sha1="3xTRWfjcnfhm1ayYkGvW2VbswMG" width="189" height="81"></p>
<p>Then the optimal vector is w* or cw*. (c &gt; 0)<br>
I want to make this example using Adam or SGD and plot the results like that video.</p>
<p>So, how to build the example??</p>
Functorch jacrev is taking too much memory

username: wasabi
cooked: <p>I’m trying to get the gradient of a pretrain VGG net on cifar10, the code is the following</p>
<pre><code class="lang-auto">net = torch.hub.load('pytorch/vision:v0.10.0', 'vgg11', pretrained=True).to(device)
net.eval()
fnet, params = make_functional(net)

def fnet_single(params, x):
    return fnet(params, x.unsqueeze(0)).squeeze(0)

def get_ntk_feature(fnet_single, params, x1):
    # Compute J(x1)
    jac1 = vmap(jacrev(fnet_single), (None, 0))(params, x1)
    jac1 = [j.flatten(2) for j in jac1]

    return jac1

x, y = next(iter(cifar_trainLoader))
o=get_ntk_feature(fnet_single, params, x.cuda())
</code></pre>
<p>However, CUDA tries to allocate 152.59 GB memory. Here x is a batch of size 10, so shouldn’t the memory be just 10*sizeof(net)?</p>
nn.Embedding layer

username: NoWay2Guess
cooked: <p>Hello guys, I have a stupid question. I am currently building a seq2seq model to predict the time-series data. Both input and target are sinusoidal signals. As I was going through many examples  online, pretty much all of them start with an embedding layer. I assume this is because those examples are language translations, which the input text needs to be vectorized. But for my case, where the inputs are numbers already, so I presume that I don’t need embedding layer for both encoder and decoder, is that correct?</p>

username: IneedMrmeeseeks
cooked: <p>Yes, I think so. Good Luck with that!</p>

username: vdw
cooked: <p>Yes, for your case you indeed won’t need an embedding layer.</p>