akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Self attention implementation

youwillknovv: <p>Hi community,</p>
<p>I am working on a bi-lstm+ attention NER model and I am following this paper: Long Short-Term Memory-Networks for Machine Reading (<a href="https://arxiv.org/pdf/1601.06733.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1601.06733.pdf</a>)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e2e0b880e29ccd87e88477f1e134752ab35d276.png" data-download-href="https://discuss.pytorch.org/uploads/default/3e2e0b880e29ccd87e88477f1e134752ab35d276" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e2e0b880e29ccd87e88477f1e134752ab35d276.png" alt="image" data-base62-sha1="8S4fLe3M7asDkhRJSRjShT0JGZg" width="546" height="499" data-dominant-color="272B31"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">678×620 19.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I just wonder am I implmenting correctly? And the performance is not improving the model, what else I should change? Thanks</p>
Anomaly Detection of IT assets

Parkz: <p>I am looking into anomaly detection of IT assets using machine learning for my organization. We have hourly updates on device hostnames, manufacturer, model, and 60 other fields. I want to build a model that detects anomalies if a device changes models/manufacturers from the last hour’s upload. Eventually, I’d like to be able to recommend these to security personel who can vote if the model output is security related or not. What models should I consider?</p>
Direct access to the bytes of a serialized model

jimoskar: <p>Is it possible to access the the bytes of a serialized model directly, without saving it to file first? We want to get access to the serialized bytes in program, e.g. by an address and size, so we can store it in our own data structure. Is this possible?</p>
<p>Thank you.</p>

Matej_Kompanek: <p>I don’t think you can get direct access to address but you can use stringstreams to save it to memory<br>
<a href="https://discuss.pytorch.org/t/can-libtorch-load-model-from-memory/55226/3">Example</a></p>

Bohaterowicz: <p>Thank you for your answer, I work with Jim on a project and your answer was very helpful!</p>
Fbgemm not available

junliangyu96: <p>torch.backends.quantized.supported_engines<br>
Out[3]: [‘qnnpack’, ‘none’]</p>
<p><code>fbgemm</code> is not supported.</p>
<p>I tried to install <code>fbgemm</code> on my Ubuntu 20 and reinstalled pytorch 1.8. The problem is still there.</p>
<p>and also I tried to use <code>qnnpack</code> to run Quantization Aware Training on Bert. No matter I set the Embedding layer to be ‘None’ or ‘float16_dynamic_qconfig’,  I got a error message:<br>
Didn’t find engine for operation quantized::linear_prepack NoQEngine</p>
Multiclass classification

jpj: <p>Is my code for getting the predicted class correct? I don’t understand how softmax is being used here in cross entropy to get the class with the highest probability</p>
<pre><code class="lang-auto">for i, (x,y) in enumerate(zip(feature_trainloader,label_trainloader), 0):

                optimizer.zero_grad()   

                output = self.forward(x)

                target = y

                loss = self.CrossEntropyLoss(output, target)

                epoch_loss_train.append(loss.item())

                y_true_train = torch.cat((y_true_train, target))

                _, pred_class = torch.max(output, dim=1)

                y_pred_train = torch.cat((y_pred_train, pred_class))

                loss.backward()

                optimizer.step()
</code></pre>

ptrblck: <p><code>nn.CrossEntropyLoss</code> will apply <code>F.log_Softmax</code> and <code>nn.NLLLoss</code> internally to calculate the loss, which is why you shouldn’t apply a softmax activation on the model outputs.<br>
To get the predictions you can apply <code>torch.argmax</code> directly on the logits, since the order won’t change compared to the probabilities (the highest/lowest logit will also be the highest/lowest probability).</p>

Kapil_Rana: <p>You cant use nn.CrossEntropyLoss with softmax in pytorch.<br>
use as per details given by <a class="mention" href="/u/ptrblck">@ptrblck</a> .</p>
<p>If you really need softmax probabilities at the end then you can get using torch.exp(output)</p>
<p>and if you really want to train with softmax function than use custom cross entropy loss.<br>
something like</p>
<p>def custom_categorical_cross_entropy(y_pred, y_true):<br>
y_pred = torch.clamp(y_pred, 1e-9, 1 - 1e-9)<br>
return -(y_true * torch.log(y_pred)).sum(dim=1).mean()</p>

jpj: <p>Thanks. So that means my way of getting pred_class is correct right?</p>
Skip weights in covolution

NPC: <p>Hey,</p>
<p>i am sorry if i missed a similar topic.<br>
I want to skip a weight in a convolution layer. (I know bad description <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ).</p>
<p>Lets say i have a kernel like [a, b, c].<br>
But i want  a == c (or more precisely I want the optim. to threat it as one parameter).<br>
Is there a proper way to archive this?<br>
I am currently just setting them to (a + c) / 2 every step but this is not very satisfactoring.</p>
<p>Sorry if i am just a little bit stupid<br>
And thank you very much for your time<br>
NPC</p>

KFrank: <p>Hi NPC!</p>
<aside class="quote no-group" data-username="NPC" data-post="1" data-topic="123504" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/ce7236/48.png" class="avatar"> NPC:</div>
<blockquote>
<p>Lets say i have a kernel like [a, b, c].<br>
But i want  a == c (or more precisely I want the optim. to threat it as one parameter).</p>
</blockquote>
</aside>
<p>To me the conceptually cleanest approach would be to build your<br>
constrained kernel from a separate two-element <code>Parameter</code> that<br>
you actually optimize.  Something like:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from itertools import chain

# initialize  a = 1.1, b = 2.2
k = torch.nn.Parameter (torch.tensor ([1.1, 2.2]))
opt = torch.optim.SGD (chain (my_model.parameters(), (k, )), lr = 0.1)
# ...
# then in your forward pass
    # x = apply_some_model_layers (x)
    ker = torch.cat ((k[:2], k[0:1]))
    x = torch.nn.functional.conv1d (x, ker)
    # x = apply_some_more_model_layers (x)
    return x
</code></pre>
<p>I believe that the <code>weight</code> of a <code>torch.nn.Conv1d</code> has to<br>
be a leaf tensor, so we have to use the functional form,<br>
<code>torch.nn.functional.conv1d ()</code>.  By using differentiable<br>
pytorch tensor operations to build <code>ker</code> from <code>k</code>, you will be able<br>
to backpropagate through the construction of <code>ker</code> and properly<br>
optimize the two elements of <code>k</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

NPC: <p>Hello KFrank,</p>
<p>thank you very much. I will try it this way <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Best.<br>
NPC</p>
How select uneven indices alongside a dimension in batches, and apply pooling?

FeryET: <p>Hi.</p>
<p>I have to select certain hidden states from a LSTM outputs, indices of which are different for each sample,  and apply pooling on them. A simple way is to write a for loop and select indices for each sample in the batch. But I’d like to use pytorch’s api to minimize python overhead. Is there any option to do this?</p>
How cancle below ```UserWarning```?

jaffe-fly: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5170fc09f9e1243052f3f9fa3ceff6c04ec7ec3.png" data-download-href="https://discuss.pytorch.org/uploads/default/a5170fc09f9e1243052f3f9fa3ceff6c04ec7ec3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a5170fc09f9e1243052f3f9fa3ceff6c04ec7ec3_2_690x107.png" alt="image" data-base62-sha1="nysbT0LjVczIfvmULunE279GVcn" width="690" height="107" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a5170fc09f9e1243052f3f9fa3ceff6c04ec7ec3_2_690x107.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a5170fc09f9e1243052f3f9fa3ceff6c04ec7ec3_2_1035x160.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a5170fc09f9e1243052f3f9fa3ceff6c04ec7ec3_2_1380x214.png 2x" data-dominant-color="E9E9EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2098×328 81.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>when use DistributedDataParallel</p>
<pre><code class="lang-auto">scaler = GradScaler()
for epoch in range(args.epochs):
    model.train()
    if num_distrib() &gt; 1:
        train_loader.sampler.set_epoch(epoch)
    for i, (input, target) in enumerate(train_loader):  
        with autocast():  # mixed precision
            output = model(input)
            loss = loss_fn(output, target)  # note - loss also in fp16
        model.zero_grad()
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()
        reduced_loss = reduce_tensor(loss, args.gpus)
        losses.update(reduced_loss.item(), input.size(0))
    scheduler.step()
</code></pre>
<p><code>scheduler.step()</code>  has after <code>scaler.step(optimizer)</code>  why appearance <code>UserWarning</code></p>
<p>how can i do ?</p>

ptrblck: <p>The warning is raised, because the <code>GradScaler</code> might use an initial scaling factor, which could be too large for the first batches and will thus reduce it as well as skip the <code>optimizer.step()</code>. Due to this, the <code>scheduler.step()</code> would be executed before the first <code>optimizer.step()</code>, which will raise the warning.<br>
You could ignore or disable it, or on the other hand, check if the <code>scaler</code> decreased the scale factor and also skip the <code>scheduler.step()</code>.</p>

jaffe-fly: <p>i want to known  Correct front-rear order  for my code ,thank you</p>
Gradients in the background in NN's

Luesch: <p>hello,<br>
In the past few days i tried to rebuild an autograd engine in the style of pytorch.<br>
All of the computed gradients do actually match  pytorch’s gradients so it should all be fine.<br>
Now i decided to build a little nn library ontop of my autograd engine to train some small networks.<br>
So my problem is that due to operations like the .mean() in the end the gradients get scaled down to basically nothing so it takes a huge amount of optimizer steps to learn anything. So my question is if<br>
pytorch somehow scales gradients when optimizing parameters and if so how?<br>
have a nice day <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Luesch: <p>Update:</p>
<p>I compared my linear layers gradients with similar weights and bias with torch and they again were the<br>
same so it should be something in optimizer.</p>

ptrblck: <p>No, PyTorch doesn’t scale the gradients (unless you are using mixed-precision training with the <code>GradScaler</code>, but in any case the gradients would be unscaled before the update, so you can skip this side note) and you could take a look at the <code>SGD</code> implementation <a href="https://github.com/pytorch/pytorch/blob/1faba1e4cce2c6dfb3d89a19e7d709fd9af5fe67/torch/optim/sgd.py">here</a> and <a href="https://github.com/pytorch/pytorch/blob/1faba1e4cce2c6dfb3d89a19e7d709fd9af5fe67/torch/optim/_functional.py#L146-L180">here</a> to check this implementation and compare it against yours.</p>

Luesch: <p>Thank you very much for your reply!<br>
In the end it was just a stupid broadcasting error becuase i have to sum the bias gradients to counter the<br>
batch size and i summed the weight gradients aswell.<br>
have a nice day!</p>
RuntimeError on running ALBERT for obtaining encoding vectors from text

puranjay_mishra: <p>I’m trying to get feature vectors from the encoder model using pre-trained ALBERT v2 weights. i have a nvidia 1650ti gpu (4 GB) , and plenty RAM but for some reason I’m getting Runtime error saying -<br>
RuntimeError: [enforce fail at …\c10\core\CPUAllocator.cpp:75] data. DefaultCPUAllocator: not enough memory: you tried to allocate 491520000 bytes. Buy new RAM!</p>
<p>I’m really new to pytorch and deep learning in general. Can anyone please tell me what is wrong?</p>
<p>My entire code -<br>
encoded_test_data = tokenized_test_values[‘input_ids’]<br>
encoded_test_masks = tokenized_test_values[‘attention_mask’]</p>
<p>encoded_train_data = torch.from_numpy(encoded_train_data).to(device)<br>
encoded_masks = torch.from_numpy(encoded_masks).to(device)</p>
<p>encoded_test_data = torch.from_numpy(encoded_test_data).to(device)<br>
encoded_test_masks = torch.from_numpy(encoded_test_masks).to(device)</p>
<p>config  = EncoderDecoderConfig.from_encoder_decoder_configs(BertConfig(),BertConfig())<br>
EnD_model = EncoderDecoderModel.from_pretrained(‘albert-base-v2’,config=config)<br>
feature_extractor = EnD_model.get_encoder()<br>
feature_vector = feature_extractor.forward(input_ids=encoded_train_data,attention_mask = encoded_masks)<br>
feature_test_vector = feature_extractor.forward(input_ids = encoded_test_data, attention_mask = encoded_test_masks)</p>
<p>Also 491520000 bytes is about 490 MB which should not be a problem.</p>

ptrblck: <p>How much host RAM does <code>htop</code> or any other tool report?<br>
As the error message suggest, PyTorch cannot allocate the ~500MB of RAM, so I guess you might be indeed running out of memory or the process is not allowed to allocate more RAM for some reason.</p>

puranjay_mishra: <p>Im using windows. I have around 2.8 GB available RAM. Pycharm IDE is using 600 MB. I doubt there is a cap set on the RAM usage.</p>

ptrblck: <p>2.8GB seems to be quite low, since the OS (and other processes) would also use host memory, so the OOM error seems to be valid.</p>

puranjay_mishra: <p>Oh no. Its 2.8 GB memory of actually unused RAM. I have 8 GB RAM, and background process are using about 5 GB of it. And also, I tried reducing the number of training examples and also the length of the maximum padded input . The OOM error still exists even though the required space now is 153 MB , which should easily be managable.<br>
I also have maxed out the RAM limit of the heap of pycharm software to 2048 MB. I really dont know what to do now… Anyways thank you for spending the time to help me!</p>
Can we use complex weights with convolution layers?

raunak: <p>Hi,</p>
<p>My ground truth is complex-valued. The input can be real valued or complex but the output needs to be complex. Making the output as two channel image and using [</p>
<h3>
<a name="torchview_as_complex-1" class="anchor" href="#torchview_as_complex-1"></a>torch.view_as_complex</h3>
<p>](<a href="https://pytorch.org/docs/stable/generated/torch.view_as_complex.html" class="inline-onebox" rel="noopener nofollow ugc">torch.view_as_complex — PyTorch 1.8.1 documentation</a>). But that doesn’t give good results.</p>
<p>so, I am trying to make the input as well the weights of the NN to be complex. But I get the following error message:<br>
RuntimeError: “unfolded2d_copy” not implemented for ‘ComplexFloat’</p>
<p>(minimal program to reproduce the error)<br>
import torch<br>
import torch.nn as nn</p>
<p>c1 = nn.Conv2d(1, 32, 3, 1)<br>
c1.type(torch.cfloat)<br>
c1(torch.ones([3,1,20,20], dtype=torch.cfloat))</p>

ptrblck: <p>The feature request seems to be tracked <a href="https://github.com/pytorch/pytorch/issues/52983">here</a> (as well as the related issues posted in the linked issue) and so far the workaround seems to be to split these operations into the <code>re</code> and <code>im</code> part (as you already did).</p>
Torch.argmax - I want more than just one value

Zoran_Peric: <p>I have a pytorch tensor as output language model (GTP2) and I use argmax to get index with the highest probability.</p>
<p>predicted_index = torch.argmax(predictions_2[0, -1, :]).item()</p>
<p>But I need all the indexes sorted by probability.</p>
<p>How do I do this in pytorch?</p>
<p>#####################################################<br>
print (“predicted_index:”,predicted_index)<br>
print (“predictions_2[0, -1, :]:”,predictions_2[0, -1, :])</p>
<p>I get:<br>
predicted_index: 484<br>
predictions_2[0, -1, :] field is: tensor([-122.9283, -124.4627, -128.4069,  …, -131.5974, -128.7110,<br>
-125.7269], device=‘cuda:0’)</p>

InnovArul: <p>Have a look at <a class="mention" href="/u/ptrblck">@ptrblck</a>’s answer:</p><aside class="quote" data-post="2" data-topic="19170">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/advanced-indexing-with-torch-topk/19170/2">Advanced indexing with torch.topk</a> 
  </div>
  <blockquote>
    Would this work: 
x = torch.randn(3, 10, 10)
idx = torch.topk(x, k=2, dim=0)[1]
x.scatter_(0, idx, 100)
print(x)
  </blockquote>
</aside>

<p>you can get the <code>topk</code> values and indices using <code>torch.topk()</code> API.</p>

Zoran_Peric: <p>It, works now. Thanks very much.</p>
<p>predicted_k_indexes = torch.topk(predictions_2[0, -1, :],k=3)<br>
prk_0 = predicted_k_indexes[0]<br>
prk_1 = predicted_k_indexes[1]<br>
for item11 in prk_1:<br>
print (item11.item())</p>
<p>output:<br>
484<br>
523<br>
35075</p>

vinsent_paramanantha: <p>Can you tell me how to store the top 3 max  indexes values in a List or Tensor</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/store-index-of-max-values/123514">here</a>.</p>
Store index of max values

vinsent_paramanantha: <p>How can find the top 3 max values in the tensor and how can I store the index values in one variable.</p>

eqy: <p>The <code>topk</code> function probably does something similar to what you want: <a href="https://pytorch.org/docs/stable/generated/torch.topk.html" rel="noopener nofollow ugc">torch.topk — PyTorch 1.8.1 documentation</a></p>

vinsent_paramanantha: <p>Thank you <a class="mention" href="/u/eqy">@eqy</a></p>
<p>But my question is about how to pick the top 3 max values of the indexes and store that indexes in a variable . This is a solution I found</p>
<p>frame = sorted(range(len(mul_reward)), key=lambda i: mul_reward[i], reverse=True)[:4]<br>
print(frame)<br>
con_frame = torch.Tensor(frame)</p>

eqy: <p>I’m not sure I understand; what happens when you run<br>
<code>_, frame = torch.topk(mul_reward, 3)</code> ?</p>
How to download a pic from URL

looper: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a95f7d003c8500fd577c34ce6f2c88b7b4aef9ab.png" data-download-href="https://discuss.pytorch.org/uploads/default/a95f7d003c8500fd577c34ce6f2c88b7b4aef9ab" title="1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a95f7d003c8500fd577c34ce6f2c88b7b4aef9ab.png" alt="1" data-base62-sha1="oalh3USTVyzM9hQFD4wPAjyg1g7" width="690" height="87" data-dominant-color="F6EBEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1382×175 6.15 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
why its not working, and how to solve it, please.</p>

looper: <p>I am using juypter notebook from windows.</p>

ptrblck: <p>This question doesn’t be PyTorch-specific, so I would recommend to post it in a general discussion/help board (or a Jupyter-specific one).</p>
Reapply torchvision.transforms to downloaded data

mbacher: <p>Hi, I am trying to run a simple VAE model written in pytorch. To that end, I use the classical MNIST dataset by means of torchvision.datasets. Now, the data have been already downloaded and recorded in its raw values i.e., [0, 255]). Once I load it from the directory, the transforms defined to normalize the data into [-1, 1] with zero mean and unit variance, is not applied. Ergo, the data is in its original range [0, 255]. How could I do the scaling into [0,1] and the to reapply the transforms.Normalize? Here is the code:</p>
<pre><code class="lang-auto">transform_train = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=(0.1307,), std=(0.3081,))
])

transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=(0.1307,), std=(0.3081,))
])

use_cuda = torch.cuda.is_available()

trainset = datasets.MNIST(root='../../data/MNIST', train=True, download=False, transform=transform_train)
valset = datasets.MNIST(root='../../data/MNIST', train=False, download=False, transform=transform_test)
</code></pre>

ptrblck: <p>The provided code is normalizing the data and it’s not in its original range as seen here:</p>
<pre><code class="lang-python">x, y = trainset[0]
print(x.min(), x.max())
&gt; tensor(-0.4242) tensor(2.8215)
</code></pre>
<p>Are you sure this code snippet was returning the raw data?</p>

mbacher: <p>The data I have, was downloaded without any transformation, and the values I see are in the range [0, 255], even after executing the code I put in the thread. Notice that I set the download flag to False since I have already the data locally recorded… If the data were in [0, 1] then I see that the mean and std are respectively, 0.1307 and 0.3081.</p>

ptrblck: <p>Your code unfortunately doesn’t show how you are checking the data stats. As you can see in my code snippet the <code>min</code> and <code>max</code> values show the transformed results, not the original data values.<br>
In case you are checking the internal <code>.data</code> attribute, then note that this is the original data, where no transformations were applied. Also, feel free to post an executable code snippet to reproduce this issue.</p>
Error message: Failed to parse the network descriptor

JUNSUNGKIM99: <p>When I execute this command " ./image-classifier tests/images/imagenet/cat_285.png -use-imagenet-normalization -image-mode=0to1 -m=resnet50 -model-input-name=gpu_0/data -dump-llvm-ir" , this error message "<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d98c641ffd9ef153b4f0b71dd1100045c9ecb649.png" data-download-href="https://discuss.pytorch.org/uploads/default/d98c641ffd9ef153b4f0b71dd1100045c9ecb649" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d98c641ffd9ef153b4f0b71dd1100045c9ecb649.png" alt="image" data-base62-sha1="v2woEBxJLp3Mei0968VB1VfD4EN" width="690" height="202" data-dominant-color="282827"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">839×246 8.25 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
" occur.</p>
<p>why this error occur??</p>

jfix: <p>I think this is because you don’t have the directory <code>resnet50/</code> containing the model protobuf files in the directory you’re running the command from.</p>
Multi-gpu parallalization for online learning

takieddine_soualhi: <p>Hello everyone,</p>
<p>I’m newbie to pytorch. i’m working currently on untrimmed video classification. the network i’m implementing does online learning which means a batch size of 1. In addition, the loss is calculated on all frames of a single video (the loss is equal to the mean of cross entropy across all frames).<br>
the tutorial : <a href="https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html" class="inline-onebox" rel="noopener nofollow ugc">DataParallel — PyTorch 1.8.1 documentation</a> was not<br>
of a great help since my batch size is 1 so the network runs only on GPU 1.</p>

mrshenli: <p>Hey <a class="mention" href="/u/takieddine_soualhi">@takieddine_soualhi</a> I am trying to understand the use case.</p>
<blockquote>
<p>In addition, the loss is calculated on all frames of a single video (the loss is equal to the mean of cross entropy across all frames).</p>
</blockquote>
<p>Is it possible to treat each frame as a sample, so that the number of frames in the video will be the batch size? If that’s possible, then you can let each GPU to take care of a subset of the frames?</p>
World_size and rank torch.distributed.init_process_group()

nexgus: <p>Hi there,</p>
<p>To read the official doc, it totally confuse me.</p>
<p>What’s the definition of ‘world_size’ and ‘rank’ torch.distributed.init_process_group()?</p>
<p>Regarding the argument ‘world_size’, is it cross machine total device count? Or just total machine count?</p>
<p>Regarding the argument ‘rank’, is it an index for each machine, or an index for each devices?</p>
<p>For instance, if I have 2 machines and there are 4 GPUs per machine. What’s the value of ‘world_size’ and ‘rank’ I have to set when I call torch.distributed.init_process_group()?</p>

mrshenli: <p>The concepts of world_size and rank are defined on processes (hence the name process_group). If you would like to create 8 processes, then the world_size should be 8, and the ranks for them should range from 0 to 7. It is up to the application to determine how to place processes to machines. In the above cluster (2 machines, and 4 GPUs each), the best setup would be creating 4 processes on each machine, with each exclusively working on a different GPU.</p>

HuangLED: <p>“… the best setup would be creating 4 processes on each machine, …”</p>
<p>In this case, world_size should be 4, each process on every machine should have a rank from 0 to 4?</p>
<p>Then with this setup, how to define the rank of these two machines?</p>

mrshenli: <blockquote>
<p>“… the best setup would be creating 4 processes on each machine, …”</p>
</blockquote>
<p>Hey <a class="mention" href="/u/huangled">@HuangLED</a>, in this case, the <code>world_size</code> should be 8, and the ranks should range from 0-3 on the first machine and 4-7 on the second machine. This page might help explain:</p><aside class="onebox githubfolder" data-onebox-src="https://github.com/pytorch/examples/tree/main/distributed/ddp">
  <header class="source">

      <a href="https://github.com/pytorch/examples/tree/main/distributed/ddp" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/pytorch/examples/tree/main/distributed/ddp" target="_blank" rel="noopener nofollow ugc"></a></h3>

  <p><a href="https://github.com/pytorch/examples/tree/main/distributed/ddp" target="_blank" rel="noopener nofollow ugc">//github.com/pytorch/examples/tree/main/distributed/ddp</a></p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Is `[m(x) for m in self.sub_module_list]` run parallel or sequential?

JiaRu2016: <p>My model has several sub-modules, stored in a <code>nn.ModuleList</code> member, in <code>forward()</code> if I calling each sub-module like this</p>
<pre><code class="lang-python">def forward(self, x):
    out = [m(x) for m in self.sub_module_list]   # where self.sub_module_list is nn.ModuleList object
    out = torch.cat(out)
</code></pre>
<p>is <code>forward()</code> of each submodule run parallel or sequential?</p>
<p>I suppose if module is jit traced the answer is clear: it is parallel since that’s exactly what static computation graph want to do. But if not jit traced, will this line of code run sequentially?</p>

eqy: <p>It looks like there is one tensor and all of them are on the same device so I would expect them to run sequentially. However, there can be some pipeline parallelism because there aren’t dependencies between the module lists.</p>
Pack padded sequence sorting indices and their effect on hidden states

FeryET: <p>Hi</p>
<p>I’m trying to implement a model that works on the hidden states of the LSTM. What I’m trying to do is to use pack_padded_sequence to make my LSTM model learn better. The problem I have right now is that I cannot understand how this packing (and sorting) affects the hidden states generated by the model. Are they returned to the right order? or should I rearrange them accordingly?</p>

googlebot: <p>yes, states are transparently reordered back and forth IIRC</p>
Optimized double ‘for’ pixel loops

Ella: <p>Hi All,<br>
I have a layer for the U-Net model; the kernel of the layer must loop through all the pixels in the image (image size is 1024x1024) -  it is really slow. It takes 2.15 (minutes on average) for one image per one iteration. Any suggestions on how can I optimize the double loop over the image matrix?</p>

eqy: <p>Can you post the code? there might be a way to vectorize it or compose it from faster native operations</p>

Ella: <pre><code>def forward(self, details, norms, max_pixels, min_pixels):

    return torch.FloatTensor([[torch.where(dtl_pixel &lt; 0, weight(dtl_pixel, nrm, max_pixel, min_pixel), dtl_pixel) for j, (dtl_pixel, nrm, max_pixel, min_pixel) in
                               enumerate(zip(detail, norm, max_pixel, min_pixel))] for i, (detail, norm, max_pixel, min_pixel) in
                              enumerate(zip(details, norms, max_pixels, min_pixels))]).reshape((1024, 1024))
</code></pre>
<p>I organized dataset (with torch.utils.data.Dataset) for details, norms, min pixels, max pixels as separated class, that I calculate in the beginning, and is used by the kernel.<br>
weight function is:<br>
weight = lambda detail, norm, max, min: detail / torch.exp(torch.pow((max * self.lamda) / (norm * min + self.epsilon), 0.01))</p>

eqy: <p>It is a bit difficult to reason about the computation without seeing some concrete shapes, but I think the general pattern might be amenable to vectorization as I don’t see any obvious tricky control flow.</p>
<p>Is where being called on just a single pixel? I think it would likely be faster to precompute both branches first and then select them later.<br>
I think this can be done if the weight function can be vectorized across pixels (just make sure that all of the inputs have the same shape or can be broadcast in a way that makes sense).</p>
How to use make_grid to display the tensors in color map?

seyeeet: <p>Does anyone know if I have a tensor of dimention <code>20x1x256x256</code> how I can use <code>make_grid</code> and <code>save_image</code> to save the images in a specific color map (e.g. viridis)? 20 represents 20 different predictions and <code>1x256x256</code> represents the prediction.</p>

eqy: <p>I’m not sure those functions support using a custom color map (it sounds like you want to generate a heatmap first). In this case you might consider producing a heatmap (e.g., with something like matplotlib or altair) before saving the prediction images.</p>
Is it possible to vectorize the inference time of a transformer when generating a new sequence?

Brando_Miranda: <p>My feeling is that this is not possible. During training I believe this possible because we are feeding the transformer the true tokens (basically doing teacher forcing). However, during testing we don’t have the truth and we have to truly be auto-regressive so we have to really generate one token at a time. Is this right?</p>
<p>Note this is a code example showing what I believe the common transformer test time:</p>
<p>-<a href="https://github.com/azadyasar/NeuralMachineTranslation/blob/7fd9450c88d833748218c1678124cc67e3303065/src/nmt/translation/translator.py#L37" class="inline-onebox" rel="noopener nofollow ugc">NeuralMachineTranslation/translator.py at 7fd9450c88d833748218c1678124cc67e3303065 · azadyasar/NeuralMachineTranslation · GitHub</a></p>
<pre><code class="lang-auto">    for i in range(max_len):
      trg_tensor = torch.LongTensor(trg_indexes).unsqueeze(0).to(self.config.device)
      trg_mask = self.model.make_trg_mask(trg_tensor)
      with torch.no_grad():
        output, attention = self.model.decoder(trg_tensor, enc_src, trg_mask, src_mask)
      pred_token = output.argmax(2)[:,-1].item()
      trg_indexes.append(pred_token)

      if pred_token == self.config.trg_vocab.eos_idx:
        break
</code></pre>
<p>other example and discussion: <a href="https://github.com/azadyasar/NeuralMachineTranslation/issues/1" class="inline-onebox" rel="noopener nofollow ugc">How to vectorize decoder translation in transformer? · Issue #1 · azadyasar/NeuralMachineTranslation · GitHub</a></p>
Torchaudio.load() in C++

Oktai15: <p>I want to convert ogg-file to <code>torch.Tensor</code> using <code>torchaudio.load</code> function, but in deployment I need to do it in C++ (I converted original model to TorchScript). Can I use it from C++? Or what can I do as alternative?</p>

vincentqb: <p>torchaudio does not yet have a C++ interface. However, <code>torchaudio.load</code> is a wrapper around <a href="https://github.com/pytorch/audio/blob/master/torchaudio/torch_sox.cpp" rel="nofollow noopener">https://github.com/pytorch/audio/blob/master/torchaudio/torch_sox.cpp</a> which you could use.</p>

Nasim_Alam: <p>Any update on this?<br>
I also need to convert “.wav” file to torchaudio tensor in c++ while deployment on mobile device. Is there any way to do this.</p>
<p>This link is no more: <a href="https://github.com/pytorch/audio/blob/master/torchaudio/torch_sox.cpp" rel="noopener nofollow ugc">https://github.com/pytorch/audio/blob/master/torchaudio/torch_sox.cpp</a><br>
Is there any alternate link or implementation of this?</p>
<p>Thanks.</p>

QuantScientist: <p>Same here. Is there any built-in way to read a wav file into a torch tensor in C++?</p>
Multiheadattention module - how to set kdim, vdim?

dlsf: <p>I am wondering how to set the values in the module, i.e. I tried it but got errors (cryptic) that the dimensions do not match.</p>
<p>I looked at the source code, now I am wondering if I misunderstand the implementation or if there is an error in the implementation? The dimension for the q embedding should be the same as kdim in my opinion (in red), as q and v are used for calculating the correlations. Why is it set to embed_dim?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/7/37d1396a39783d6fed0086b89b189d0b3e627a91.png" data-download-href="https://discuss.pytorch.org/uploads/default/37d1396a39783d6fed0086b89b189d0b3e627a91" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37d1396a39783d6fed0086b89b189d0b3e627a91_2_606x500.png" alt="image" data-base62-sha1="7XMwlSz3wPtXVtaixR0X5XNcOIh" width="606" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37d1396a39783d6fed0086b89b189d0b3e627a91_2_606x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37d1396a39783d6fed0086b89b189d0b3e627a91_2_909x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37d1396a39783d6fed0086b89b189d0b3e627a91_2_1212x1000.png 2x" data-dominant-color="E9E8EB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1360×1122 164 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Flexible API for layers

amaleki: <p>I am interested in implementing a more flexible API for the Linear layer, where the inputs are only the output feature size. The input feature size is inferred from the size of input tensor. I have a minimal implementation:</p>
<pre><code class="lang-auto">from torch.nn import Module, Linear
class FlexibleLinear(Module):
    def __init__(self, out_feats):
        super(FlexibleLinear, self).__init__()
        self.out_feats = out_feats
        self.initialized = False
        self.linear = None

    def build(self, x):
        if self.initialized:
            return

        in_feats = x.shape[1]
        out_feats = self.out_feats
        self.linear = Linear(in_feats, out_feats)
        self.initialized = True

    def forward(self, x):
        self.build(x)
        y = self.linear(x)
        return y
</code></pre>
<p>I am wondering if there is any (better) way to do this, and/or if this can create any problem for a larger scale network.</p>

JuanFMontesinos: <p>I’m not really convinced about this code. There are issues:</p>
<ul>
<li>Layer is not allocated properly. It will raise an Exception.</li>
<li>Layer is not tracked by the nn.Module. Thus, optimizer won’t track it and this will create a silent bug.</li>
</ul>
<p>Despite I don’t know the details of the new lazy modules, I would recommend you to check them to code this.<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html#LazyConv2d" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html#LazyConv2d</a></p>

amaleki: <p>Thank you for the issues mentioned and the link.<br>
It seems like <code>LazyLinear</code> is a new feature added since 1.8</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.LazyLinear.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.LazyLinear.html</a></p>
Torch.norm & torch.linalg.norm

Ashima_Garg: <p>While using <code>torch.norm</code> my code is giving nans. So, I tried changing the <code>dtype</code> argument  as mentioned here <a href="https://pytorch.org/docs/stable/generated/torch.norm.html" class="inline-onebox" rel="noopener nofollow ugc">torch.norm — PyTorch 1.7.0 documentation</a> to <code>dtype=torch.float64</code>.<br>
But it is giving the following error:<br>
<code>ValueError: dtype argument is not supported in frobenius norm</code>.</p>
<p>As mentioned as WARNING on this page as <a href="https://pytorch.org/docs/stable/generated/torch.norm.html" class="inline-onebox" rel="noopener nofollow ugc">torch.norm — PyTorch 1.7.0 documentation</a> torch.norm is deprecated and maybe removed in future release.</p>
<p>But the following snippet is also giving as <code>ImportError : cannot import name 'linalg'</code><br>
<code>from torch import linalg</code></p>
<p>Please look into the issue.<br>
Thanks.</p>

InnovArul: <aside class="quote no-group" data-username="Ashima_Garg" data-post="1" data-topic="107305">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/bc8723/48.png" class="avatar"> Ashima_Garg:</div>
<blockquote>
<p>But the following snippet is also giving as <code>ImportError : cannot import name 'linalg'</code><br>
<code>from torch import linalg</code></p>
</blockquote>
</aside>
<p>Which torch version are you using?<br>
<code>linalg</code> is introduced in pytorch 1.7.0 I think.</p>
<p>Also, can you give a short snippet of code to reproduce the issue?</p>

Ashima_Garg: <p>I’m using torch 1.3.1 version</p>

Lakshmi_Jayalal: <p>Even I am facing the same issue.</p>
<pre><code class="lang-auto">
from torch import linalg 
</code></pre>
<p>gives me the error</p>
<pre><code class="lang-auto">​

from torch import linalg 

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
&lt;ipython-input-11-fe7ec55944c3&gt; in &lt;module&gt;
----&gt; 1 from torch import linalg

ImportError: cannot import name 'linalg' from 'torch' (/home/nithin/anaconda3/envs/Pytorch_cuda_env/lib/python3.7/site-packages/torch/__init__.py)


</code></pre>

ptrblck: <p>Which PyTorch version are you using?<br>
As given by <a class="mention" href="/u/innovarul">@InnovArul</a> it seems that this namespace was introduced in PyTorch&gt;=<code>1.7</code>.</p>

Lakshmi_Jayalal: <p>Oh. Okay.<br>
I’m using 1.3.1.<br>
Thanks</p>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128, 512]], which is output 0 of ViewBackward, is at version 4; expected version 0 instead

adrilang: <p>Dear all,<br>
I’m getting the following error message: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128, 512]], which  is output 0 of ViewBackward, is at version 4; expected version 0 instead.<br>
I know that this question has been asked several times but I tried to remove all implace operations in my code and I still get the same error. Can someone give me a hint how to solve this problem? I’m using python=3.6, torch = 1.2.0, and tochvision=0.4.0. Here is the code I’m using:</p>
<p>‘’‘VGG11/13/16/19 in Pytorch.’’’<br>
import torch<br>
import torch.nn as nn</p>
<p>cfg = {<br>
‘VGG11’: [64, ‘M’, 128, ‘M’, 256, 256, ‘M’, 512, 512, ‘M’, 512, 512, ‘M’],<br>
‘VGG13’: [64, 64, ‘M’, 128, 128, ‘M’, 256, 256, ‘M’, 512, 512, ‘M’, 512, 512, ‘M’],<br>
‘VGG16’: [64, 64, ‘M’, 128, 128, ‘M’, 256, 256, 256, ‘M’, 512, 512, 512, ‘M’, 512, 512, 512, ‘M’],<br>
‘VGG19’: [64, 64, ‘M’, 128, 128, ‘M’, 256, 256, 256, 256, ‘M’, 512, 512, 512, 512, ‘M’, 512, 512, 512, 512, ‘M’],<br>
}</p>
<p>class VGG(nn.Module):<br>
def <strong>init</strong>(self, vgg_name):<br>
super(VGG, self).<strong>init</strong>()<br>
self.features = self._make_layers(cfg[vgg_name])<br>
self.classifier = nn.Linear(512, 10)</p>
<pre><code>def forward(self, x):
    out = self.features(x)
    emb = out.view(out.size(0), -1)
    out = self.classifier(emb)
    return out, emb

def _make_layers(self, cfg):
    layers = []
    in_channels = 3
    for x in cfg:
        if x == 'M':
            layers = layers + [nn.MaxPool2d(kernel_size=2, stride=2)]
        else:
            layers = layers + [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
                       nn.BatchNorm2d(x),
                       nn.ReLU(inplace=False)]
            in_channels = x
    layers = layers + [nn.AvgPool2d(kernel_size=1, stride=1)]
    return nn.Sequential(*layers)

def get_embedding_dim(self):
    return 512
</code></pre>
<p>def test():<br>
net = VGG(‘VGG11’)<br>
x = torch.randn(2,3,32,32)<br>
y = net(x)<br>
print(y.size())</p>

ptrblck: <p>The posted code snippet does not reproduce the issue and will raise an error in:</p>
<pre><code class="lang-python">print(y.size())
&gt; AttributeError: 'tuple' object has no attribute 'size'
</code></pre>
<p>since the <code>forward</code> method is returning a <code>tuple</code>.<br>
Could you post a code snippet, which would reproduce this issue, please?<br>
PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

adrilang: <p>Thanks for the answer. Actually I found a hidden inplace operation in some other part of the code, which I didn’t posted. I didn’t realize that this was an inplace operation:<br>
<code>feature[0] = feature[0].detach()</code></p>
Problem in disjoint training of autoencoder and MLP networks

mortezamg63: <p>Hello,<br>
I have implemented a network that has an autoencoder and after that a Multiple layer perceptron (MLP). Before autoencoder I used embedding layers. The embeddings’ outputs are inputs of  autoencoder. I use embeddings’ outputs for calculating MSE in the autoencoder. The embeddings are trained by  autoencoder at the same time. My goal is to train the autoencoder, then using encoder’s output as input to train the MLP. The autoencoder seems to be trained good because there is reduction in loss. But when train autoencoder and freeze the autoencoder layers, training only MLP on pre-trained autoencoder does not improvement in accuracy.</p>
<p>To make sure my implementation is on the right direction and trained features by autoencoder does not make problem in MLP classification part of my model, I have tried joint training by using the following loss functions and weights:<br>
Total Loss = 0.02 * classification loss + autoencoder loss<br>
Total Loss = classification loss + 0.2 * autoencoder loss<br>
Everything works fine.</p>
<p>As I said, my problem is that If I train autoencoder first and then train classification, the classification can not learn. I can not figure the issue out. Does it expect to be helpful if I use embedding output as input data for training autoencoder?<br>
Because there are some variations in the embedding, at first I expected no loss reduction in training autoencoder. But it can learn and the loss reduces, so it shows that autoencoder and embeddings are learning representation of data. Does it a true conclusion?</p>
<p>Can anyone help me on this issue?</p>
<p>I hope my question is clear and you can help me. If you need more details , please let me know.</p>
Batched tensordot?

jw3126: <p>I would like to do a “batched” tensordot. For instance say I have ten 3d vectors packed into a tensor <code>x</code> with <code>x.size() = (10,3)</code>.<br>
And I have ten 2x3 matrices packed into a tensor <code>A</code> with <code>A.size() = (10, 2, 3)</code>.<br>
I would like to multiply the ith matrix with the ith vector and get the results packed into a tensor <code>y</code> with <code>y.size() = (10, 2)</code>.<br>
How to do this?</p>

jw3126: <p>One way to do this is using <code>torch.einsum</code>:</p>
<pre><code class="lang-python">import torch
x = torch.randn(2,3)
A = torch.randn(2,2,3)
res = torch.einsum('abc,ac -&gt; ab', A, x)

assert torch.allclose(torch.matmul(A[0], x[0]), res[0])
assert torch.allclose(torch.matmul(A[1], x[1]), res[1])
</code></pre>
How to make the output of the lstm network become the input of the next step

YH-WEI: <p>I need to move the value output by lstm to the next input, and then make predictions step by step. How do I describe the code?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a16df803b11bc5c910712df7167ac40247512b5.png" alt="LSTM" data-base62-sha1="jHATz15mVMCbPubs4GiDrxB7Rrv" width="340" height="297"></p>