akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Learnable parameters

oasjd7: <p>I want to train learnable parameter <code>w</code> to re-weight loss.<br>
But <code>w</code> is never changed.<br>
What`s wrong?</p>
<pre><code class="lang-auto">w = nn.Parameter(torch.tensor(0.1), requires_grad=True).cuda()
optimizer = optim.SGD(list(network.parameters()), lr=0.1, momentum=0.9, weight_decay=5e-4)
...
loss = loss * w
optimizer.zero_grad()
loss.backward()
optimizer.step()

</code></pre>

ptrblck: <p><code>w</code> doesn’t seem to be passed to the optimizer, so that even if this parameter gets valid gradients, it won’t be updated.<br>
Also, call <code>cuda()</code> on the tensor, not the <code>nn.Parameter</code>, as this operation will create a non-leaf variable.</p>

emad: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> how to update that to pass it to the optimizer?</p>

ptrblck: <p>You could pass <code>w</code> as an element of a <code>list</code> to the <code>optimizer</code>:</p>
<pre><code class="lang-python">optimizer = torch.optim.SGD([w], lr=0.1)
</code></pre>
<p>Of course concatenating <code>w</code> with all other model parameters into a <code>list</code> would also work.</p>

DK13: <aside class="quote no-group" data-username="oasjd7" data-post="1" data-topic="70114">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oasjd7/48/5053_2.png" class="avatar"> oasjd7:</div>
<blockquote>
<p><code>w = nn.Parameter(torch.tensor(0.1), requires_grad=True).cuda()</code></p>
</blockquote>
</aside>
<p>Hi! It’s not clear to me, how <code>w</code> and the list of model parameters can be concatenated?</p>
<p>Best,<br>
Dimitri</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/dk13">@DK13</a>,</p>
<p>You can try something like</p>
<pre><code class="lang-auto">params = [p for p in model.parameters()]
params.append(w)
optim = torch.optim.Adam(params)
</code></pre>
<p>This is without moving the tensors to cuda but just move them before you pass them to the optimizer and all should work! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

DK13: <p>Thanks a lot! I tried also <code>list(model.parameters())+[w]</code>, I think it does the same thing.</p>
PyTorch in-place operator issue in NumPy conversion

One4ever: <p>I converted a PyTorch <code>Tensor</code> into a NumPy <code>ndarray</code>, and as it’s shown below ‘<code>a</code>’ and ‘<code>b</code>’ both have different ids so they point to different memory locations.</p>
<p>I know in PyTorch underscore indicates in-place operation so <code>a.add_(1)</code> does not change the memory location of <code>a</code>, but why does it change the content of the array <code>b</code> (due to the difference in id)?</p>
<pre><code>import torch
a = torch.ones(5)
b = a.numpy()
a.add_(1)

print(id(a), id(b))
print(a)
print(b)
</code></pre>
<p>Results:</p>
<pre><code>139851293933696 139851293925904
tensor([2., 2., 2., 2., 2.])
[2. 2. 2. 2. 2.]
</code></pre>
<p><a href="https://pytorch.org/tutorials/beginner/former_torchies/tensor_tutorial.html#numpy-bridge" rel="noopener nofollow ugc">PyTorch documentation</a>:</p>
<blockquote>
<p>Converting a torch Tensor to a NumPy array and vice versa is a breeze.<br>
The torch Tensor and NumPy array will share their underlying memory<br>
locations, and changing one will change the other.</p>
</blockquote>
<p>But why?<br>
(They have different IDs so must be independent) <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

googlebot: <p>id() is inappropriate because python objects are not value objects, i.e. they link to other objects, and you just have multiple links here (see .storage().data_ptr() to reason about address identities)</p>

One4ever: <p>Thanks a lot for your reply</p>
How to visualize the attention heatmap for textual entailment

anis016: <p>Hello!</p>
<p>I am working on textual entailment. Can anyone please help on how to visualize the premise and hypothesis of activated neurons? similarly like this:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/67e00f81b7116b03d72407e7e9fa1354b6cbc128.jpeg" alt="image" data-base62-sha1="eOVe4FeupdeR7Cftfs4YT4U9yli" width="602" height="390"></p>
<p>Best Regards</p>

hH1sG0n3: <p>Did you end up finding a solution for this at all <a class="mention" href="/u/anis016">@anis016</a> ?</p>
How do we detect an object that is partially visible in a picture?

Oguzhan: <p>How do we detect an object that is partially visible in a picture? I am working on Faster RCNN, YOLOv5, Swin-L Transformer, EfficentNet etc. and is there any method to detect objects that are partially visible in the picture? The dataset contains images from videos, so maybe tracking works someway but I do not know how to do that.</p>

KFrank: <p>Hi Oğuzhan!</p>
<aside class="quote no-group" data-username="Oguzhan" data-post="1" data-topic="124641" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oguzhan/48/34324_2.png" class="avatar"> Oguzhan:</div>
<blockquote>
<p>How do we detect an object that is partially visible in a picture?</p>
</blockquote>
</aside>
<p>As a general rule you want the images in your training set to be<br>
as representative as possible of the images on which you want<br>
to perform inference.</p>
<p>So you want to train on a set of images that includes a good<br>
assortment of <em>occluded</em> (“partially visible”) objects.</p>
<p>Consider training a cow detector, and, for the sake of argument,<br>
let’s assume that the cows in your images are always facing<br>
sideways to the camera.  Let’s say that all the cows in your training<br>
images are full, unoccluded cows.  It is plausible that your model<br>
might “learn” that a cow has a head, a tail, and four legs, but<br>
doesn’t pay attention to the shape of the cow’s head, or its ears,<br>
or its nose.</p>
<p>If you present your model with an inference image of a cow sticking<br>
its head out from behind a barn, it’s very reasonable to expect that<br>
your model won’t detect that cow.  But if you had trained your model<br>
on a mixture of cows that included both full cows and cows peeking<br>
their heads out from behind barns, your model would likely have<br>
learned that some cows have heads and tails and four legs, but that<br>
other cows have heads with ears and noses of certain shapes, and<br>
would likely be successful at detecting occluded cows.</p>
<p>Best.</p>
<p>K. Frank</p>

Oguzhan: <p>We have developed a simulation (that imitate a city) to provide data using game development so if we take pictures as you said above the model should work fine. I will inform about this. Thanks.</p>
How to pack_padded_sequence and pad_packed_sequence?

Sanbot-kk: <p>Hello!</p>
<p>I am new to PyTorch and I am trying to implement a Bidirectional LSTM model with input sequences of varied length. I wanted to mask the inputs to avoid influencing the gradient calculation with the padding information. Is this the right way to proceed?</p>
<pre><code class="lang-auto">class Bi_RNN(nn.Module):

    def __init__(self, input_dim1, input_dim2, hidden_dim, batch_size, output_dim, num_layers=2, rnn_type='LSTM'):
        super(Bi_RNN, self).__init__()
        self.input_dim1 = input_dim1
        self.input_dim2 = input_dim2
        self.hidden_dim = hidden_dim
        self.batch_size = batch_size
        self.num_layers = num_layers

        #Define the initial linear hidden layer
        self.init_linear1 = nn.Linear(self.input_dim1, self.input_dim1)
        self.init_linear2 = nn.Linear(self.input_dim2, self.input_dim2)

        # Define the LSTM layer
        self.lstm1 = eval('nn.' + rnn_type)(self.input_dim1, self.hidden_dim, self.num_layers, batch_first=True, bidirectional=True)
        self.lstm2 = eval('nn.' + rnn_type)(self.input_dim2, self.hidden_dim, self.num_layers, batch_first=True, bidirectional=True)
        
        self.lstm3 = eval('nn.' + rnn_type)(self.hidden_dim *4, self.hidden_dim, self.num_layers, batch_first=True, bidirectional=True)
        
        # Define the output layer
        self.linear = nn.Linear(self.hidden_dim * 2 , output_dim)
        
        self.log_softmax = nn.LogSoftmax(dim=1)

    def init_hidden(self):
        # This is what we'll initialise our hidden state as
        return (torch.zeros(self.num_layers, self.batch_size, self.hidden_dim),
                torch.zeros(self.num_layers, self.batch_size, self.hidden_dim))

    def forward(self, input1, input2, x_lens):
       
        x1_packed = pack_padded_sequence(input1, x_lens, batch_first=True, enforce_sorted=False)
        x2_packed = pack_padded_sequence(input2, x_lens, batch_first=True, enforce_sorted=False)
        

        # Forward pass through LSTM layer
        lstm_out1, self.hidden1 = self.lstm1(x1_packed)
        lstm_out1_padded, output1_lengths = pad_packed_sequence(lstm_out1, batch_first=True)
        lstm_out2, self.hidden2 = self.lstm2(x2_packed)
        lstm_out2_padded, output2_lengths = pad_packed_sequence(lstm_out2, batch_first=True)
        

        lstm_out3 = torch.cat((lstm_out1_padded, lstm_out2_padded), 2)
        lstm_out, self.hidden3 = self.lstm3(lstm_out3)
        
        y_pred = self.log_softmax(self.linear(lstm_out))
        return y_pred
</code></pre>
Gradient per example before and after clipping

AliSh: <p>Hello,</p>
<p>I wonder if there is an option in Opacus to access the per example gradient before and after clipping during the training please?</p>
<p>Thanks,<br>
Ali</p>

ffuuugor: <p>Hi<br>
Accessing per sample gradients before clipping is easy - they’re available between <code>loss.backward()</code> and <code>optimizer.step()</code> calls. Backward pass calculates per sample gradients and stores them in <code>parameter.grad_sample</code> attribute. Optimizer step then does the clipping and aggregation, and cleans up the gradients.<br>
For example:</p>
<pre><code class="lang-auto">m = Module()
optimizer = optim.SGD(m.parameters(), &lt;...&gt;)
privacy_engine = PrivacyEngine(&lt;...&gt;)
privacy_engine.attach(optimizer)

&lt;...&gt;
output = m(data)
loss = criterion(data, labels)
loss.backward()
print(m.fc.weight.grad_sample) # print per sample gradients
optimizer.step()
</code></pre>
<p>Post-clip values are more tricky - it’s not something we support out of the box.<br>
<code>optimizer.step()</code> does three things at once:</p>
<ol>
<li>clips per sample gradients</li>
<li>accumulates per sample gradients into <code>parameter.grad</code>
</li>
<li>adds noise</li>
</ol>
<p>Which means that there’s no easy way to access intermediate state after clipping, but before accumulation and noising.</p>
<p>I suppose, the easiest way to get post-clip values would be to take pre-clip values and do the clipping yourself, outside of opacus code.<br>
All you need to do is replicate a small bit from <code>opacus/opacus/per_sample_gradient_clip.py:clip_and_accumulate()</code>:</p>
<pre><code class="lang-python"># step 0 : calculate the layer norms
all_norms = calc_sample_norms(
  named_params=self._named_grad_samples(),
)

# step 1: calculate the clipping factors based on the noise
clipping_factor = self.norm_clipper.calc_clipping_factors(all_norms)
</code></pre>
<p>You can then simply multiply your pre-clipping per-sample gradients <code>p.grad_sample</code> by <code>clipping_factor</code> tensor to get the same clipping that’s happening inside opacus.</p>
<p>Hope this helps</p>

AliSh: <p>Thanks <a class="mention" href="/u/ffuuugor">@ffuuugor</a>!<br>
I wonder when each layer has both weights and biases, how opacus computes the l2 norm of gradients per example for the clipping? Does it compute separately for biases and weights please?</p>

ffuuugor: <p>Actually, by default (and as per <a href="https://arxiv.org/pdf/1607.00133.pdf" rel="noopener nofollow ugc">Abadi et al.</a>) we clip the L2 norm of the entire gradient vector for a given sample, i.e. vector consisting of gradients for all trainable parameters in the model stacked together.</p>
<p>We’ve tried experimenting with different approaches to clipping, e.g. dynamic per-layer thresholds (see implementations in <a href="https://github.com/pytorch/opacus/blob/master/opacus/utils/clipping.py" rel="noopener nofollow ugc">clipping.py</a>), but it wasn’t too fruitful</p>
Train YOLOv5 from scratch (not pre-trained)

almog: <p>Hello,<br>
I want to train the YOLOv5 model (<a href="https://pytorch.org/hub/ultralytics_yolov5/" class="inline-onebox" rel="noopener nofollow ugc">YOLOv5 | PyTorch</a>)  from scratch (not using the pretrained weights) on my own dataset and classes for a task of Face Mask Detection.</p>
<p>I have seen that in  order to train I should load:<br>
model = torch.hub.load(‘ultralytics/yolov5’, ‘yolov5s’, autoshape=False, pretrained=False)  # load scratch</p>
<p>However, How do I actually train it? Can I use it as one layer in my model?</p>
<p>Thank you,<br>
Almog</p>

ptrblck: <p>I don’t believe you could use the model as one “layer” in another model, as I would assume that the <code>YOLOv5</code> model expects image tensor inputs and returns the predictions (not any intermediate activations), so I’m unsure how it should be used as a layer in a larger model.</p>
<aside class="quote no-group" data-username="almog" data-post="1" data-topic="124561">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/9fc29f/48.png" class="avatar"> almog:</div>
<blockquote>
<p>However, How do I actually train it?</p>
</blockquote>
</aside>
<p><a href="https://github.com/ultralytics/yolov5/blob/e8810a53e83ddb5dd6bf8e871c2ede701007047c/train.py">This <code>train.py</code></a> script from the repository seems to show the training of the model.</p>

almog: <p>Thank you for you reply <a class="mention" href="/u/ptrblck">@ptrblck</a> !<br>
I have seen theirs but it requires more modifications to my code… Can I use it as my model but train it with my own train.py code?</p>
<p>I have also tried to use both their model and their loss function existed in train.py (class ComputeLoss in loss.py) but I haven’t found documentation on what the input ‘targets’ to this function should be.</p>
<p>What I mean to do is:</p>
<pre><code>for epoch in tqdm(range(train_params.num_epochs)):
    model.train()
    for i, (images, bboxs, labels) in enumerate(train_loader):
        preds = YOLO_model(images)
        loss = compute_loss(preds, targets)
        loss.backward()
</code></pre>
<ul>
<li>What should I give in ‘targets’?</li>
<li>And how do I separate ‘preds’  to labels and bboxes?</li>
</ul>
<p>Thank you!</p>

ptrblck: <p>Yes, you should be able to adapt your custom training script using the reference.<br>
I’m not familiar with the repository so don’t know which shapes and objects are expected.<br>
I would probably start by checking <a href="https://github.com/ultralytics/yolov5/wiki/Train-Custom-Data">the “train custom data” tutorial</a> and in case that doesn’t help you could try to execute their script and check the expected shapes etc.</p>
What are the best practices to do before moving the model to TensorRT?

UmairJavaid: <p>I have a PyTorch model and I want to deploy it in TensorRT. I want to know what are the best practices to carry out before moving the model to ONNX (should I train it in fp32 or fp16 using AMP), what are the best practices to do while the model is in ONNX(use anything from the ONNX optimization toolkit), and what are the best practices to carry out while the model is in TensorRT?<br>
The goal is to maximize inference time in the deployment stage. Since, there are so many different articles and methods to optimize a PyTorch model, include PyTorch JIT and torchscript, I am confused about which steps to carry out and which to not. The goal is to maximize inference time in the deployment stage.</p>
Best way to deal with 1 channel images

kl_divergence: <p>I have a lot of one channel images and wanted to make use of transfer learning. All the pretrained models that are available in torchvision expects 3 channel images, so for time being I have added padding of zeros to make 1 channel image,3 channel image but seems that it’s not helping as my loss isn’t going down. Is there any better way to deal with this ? I am using ResNet50 for this purpose.</p>

ptrblck: <p>Instead of filling the additional channels with zeros you could convert the grayscale image to an RGB one with <code>Image.open('PATH').convert('RGB')</code>.<br>
This might work in most use cases.<br>
Alternatively, you could keep your grayscale image and add a conv layer in front of your model, which outputs 3 channels. For this approach you would need to train this layer though.</p>

kl_divergence: <p>I dont’t have images, I have a dataframe which contains strokes. So I’m using this, so not sure how to use your first approach. The function which converts strokes to images is:</p>
<pre><code class="lang-auto">def get_ims(raw_strokes):
    image = Image.new("P", (28,28), color=255)
    image_draw = ImageDraw.Draw(image)

    for stroke in eval(raw_strokes):
        for i in range(len(stroke[0])-1):

            image_draw.line([stroke[0][i], 
                             stroke[1][i],
                             stroke[0][i+1], 
                             stroke[1][i+1]],
                            fill=0, width=6)
    return np.array(image)
</code></pre>

Swift2046: <p>I’m sure someone will be able to tell you if there’s a smarter way to do this, but I think what you can do is:</p>
<pre><code class="lang-auto">RGB_image = [image] + [image] + [image]
</code></pre>

JoshuaC3: <p>RGB_image = [image] * 3 <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

grisaitis: <p>You can make your own transform that accomplishes it for you:</p>
<p><code>transform_rgb = transforms.Lambda(lambda image: image.convert('RGB'))</code></p>
<p>I’m not sure how performant this is, though, given the lambda operation and how it’s on the PIL Image and not the tensor.</p>

Pablo1: <p>What is the difference between:</p>
<ul>
<li>Converting the image to an RGB <code>Image.open('PATH').convert('RGB')</code><br>
and</li>
<li>
<code>RGB_image = [image] * 3</code> ?</li>
</ul>
<p>How did the Pytorch’s pretrained models manage this?</p>
How to make torchtext input data from scratch

lesshaste: <p>I am trying to modify <a href="https://github.com/bentrevett/pytorch-seq2seq/blob/master/5%20-%20Convolutional%20Sequence%20to%20Sequence%20Learning.ipynb" rel="noopener nofollow ugc">https://github.com/bentrevett/pytorch-seq2seq/blob/master/5%20-%20Convolutional%20Sequence%20to%20Sequence%20Learning.ipynb</a> to use my own training data.</p>
<p>The code makes variables <code>train_iterator, valid_iterator, test_iterator</code> using the function BucketIterator.splits.  The type of <code>train_iterator</code> is torchtext.data.iterator.BucketIterator.</p>
<p>In my case I have two long strings made up of sentences, one in English and one in a dialect of English that is a translation of the one in plain English.  Let us call these string1 and string2.  How can I make the variables  <code>train_iterator, valid_iterator, test_iterator</code> from these two strings?</p>

mmg: <p><a href="https://github.com/mmg10/pytorch_text_new" rel="noopener nofollow ugc">This might help</a>. It reads a CSV and builds ‘from scratch’ <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Torchtext alternative to TabularDataset, BucketIterator

Stephen_Fernandes: <p>utnil now ive been using the torchtext BucketIterator and TabularDataset  for machine translations,<br>
but the problem is the BucketIterator cannot be used with TPUs and it doesnt have a sampler and DistributedDataSampler cannot be used over that, also tried using it with Lightning but stuck to ony single GPU .</p>
<p>is there any better alternative DataLoaders for seq2seq translation task that can also batch data according to similar lengths like BucketIterator and that also can be compatible for distributed training on TPUs and GPUs.</p>
<p>i have a csv file with parallel texts to be used for seq2seq translations</p>

mmg: <p><a href="https://github.com/mmg10/pytorch_text_new" rel="noopener nofollow ugc">This might help</a>.</p>
<p>However, I haven’t used this on TPUs. Please let me know if it works (or not!)</p>
Issue with pad_sequence

Tejan_Mehndiratta: <p>Hi,<br>
I m using this code to pad the sequences to the maximum length, but the issue is that the sequences are getting padded to the max len based on the max len it found in the mini-batch it is.<br>
Is there a way to pad every sequence in the dataset to the given length?</p>
<pre><code class="lang-auto">def collate_batch(batch):
    text_list, label_list = [], []

    for text, label in batch:
        processed_text = torch.Tensor(text_transform(text))
        text_list.append(processed_text)
        label_list.append(label)
    return pad_sequence(text_list, padding_value=3.0, batch_first=True), torch.Tensor(label_list)

trainloader = torch.utils.data.DataLoader(trainset, batch_size=32, shuffle=True, collate_fn=collate_batch)

</code></pre>

mmg: <p>I don’t think so. In Keras, there is a max length parameter, but not in PyTorch. TO my knowledge, what you are suggesting is unnecessary. Sequence lengths need be equal only for mini-batches, not the entire batch</p>
How can I Construct a vocabulary from the train and test datasets

Alston: <p>I want to build a vocabulary from my training and test datasets using the torchtext. Of course, I can do it as follow:</p>
<pre><code class="lang-auto">TEXT.build_vocab(train, test)
</code></pre>
<p>where TEXT is a Field object, and the train and test are Dataset objects. But I do not have enough memory to load training and test datasets at the same time. When I performed like this:</p>
<pre><code class="lang-auto">TEXT.build_vocab(train)
del train
TEXT.build_vocab(test)
del test
</code></pre>
<p>it only builds the vocab from the test data.<br>
How can I build the vocab in 2 steps so that I can release the memory after I create the corresponding vocab?</p>

mmg: <p>I am not sure why you are building vocab from the test data. Only the train data is used for building vocabulary. Any words present in test but missing in train must be rendered as unknown</p>
Bleu Score in torchtext

Tejan_Mehndiratta: <p>Hi,</p>
<p>In the example given on the pytorch website for calculating the bleu_score:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; from torchtext.data.metrics import bleu_score
&gt;&gt;&gt; candidate_corpus = [['My', 'full', 'pytorch', 'test'], ['Another', 'Sentence']]
&gt;&gt;&gt; references_corpus = [[['My', 'full', 'pytorch', 'test'], ['Completely', 'Different']], [['No', 'Match']]]
&gt;&gt;&gt; bleu_score(candidate_corpus, references_corpus)
    0.8408964276313782

</code></pre>
<p>My question is:<br>
For the sentence <code>['My', 'full', 'pytorch', 'test']</code>, the Bleu is getting calculated against <code>[['My', 'full', 'pytorch', 'test'], ['Completely', 'Different']]</code> and for the sentence <code>['Another', 'Sentence']</code> the Bleu is getting calculated against <code>[['No', 'Match']]</code>, Right ?</p>

mmg: <p>Refer to <a href="https://cloud.google.com/translate/automl/docs/evaluate#bleu" rel="noopener nofollow ugc">this post</a>. The scores are calculate for 1-,2-,3- and 4-gram sequences then (sort of) averaged out</p>
Enforce pad_sequence to a certain length

jiwidi: <p>I have a set of tensor that I’m padding with <code>pad_sequence</code> but I need to guarantee a fixed length for them. I can’t do it right now as <code>pad_sequence</code> with extend the shorter tensors up to the longest, if that longest tensor doesn’t reach the length I want them I’m screwed. I thought a solution could be adding zeros to one of the tensors to padd up to the length I want so the result of that padding will have my desired length. I don’t know how to do it</p>
<p>So lets say I have a tensor with shape <code>torch.Size([44])</code> and a desired length 50, how can I add zeros to it to reach a shape of <code>torch.Size([50])</code>? This needs to hold regardless of the initial tensor shape.</p>

Unity05: <p>Hi,</p>
<p>you could add a dummy tensor and remove it afterwards. For example: <code>torch.empty(50)</code>.</p>
<p>Regards,<br>
Unity05</p>

mmg: <p>Keras had an option like this. But not in PyTorch. I guess Unity05’s workaround is a good fix.</p>
How does torchtext encode labels?

mark_eu: <p>torchtext has a common way to map text and labels:</p>
<pre><code class="lang-auto">fields = [('text',TEXT), ('label',LABEL)]
</code></pre>
<p>In case I have string labels, is there a rule for encoding labels? For example, how does it encode this data:</p>
<pre><code class="lang-auto">text1   'AB'
text2   'AA'
</code></pre>
<p>Does it first sort labels and ‘AA’ gets label 0 and ‘AB’ gets 1 or ‘AB’ gets 0 and ‘AA’ gets 1? Also, in case of labels are already encoded (0,1,2…) does torchtext just take these labels? Thank you</p>

mmg: <p><code>Field</code> is now part of torchtext.legacy. I personally won’t recommend it. It is better to create your own dataset/dataloader.</p>
How to collect tensors in all gpus for each batch and save them

richcmwang: <p>Hi, I want to collect tensors in all GPUs for each minibatch and save them.  Can someone suggest how to do that?</p>

Ali1234: <p>If you are using DDP (<code>DistributedDataParallel()</code>) then you can simply save them like you do without DDP (using torch.save), because every process (i.e. GPU) will run it. Use gpu index to prevent multiple saving<br>
of the same name</p>

gcramer23: <blockquote>
<p>I want to collect tensors in all GPUs for each minibatch and save them.</p>
</blockquote>
<p>Do you want all tensors to be on a single process before saving?</p>
<p>You can save a tensor using <a href="https://pytorch.org/docs/stable/generated/torch.save.html" class="inline-onebox" rel="noopener nofollow ugc">torch.save — PyTorch 2.1 documentation</a>.</p>

richcmwang: <p>Yes, DDP is fine.  The tensors can stay on GPUs. Each tensor should be saved in a file, and I want to make sure I save them without repetition or missing a tensor.  Can you suggest a sample code?</p>

Ali1234: <p>To avoid repetition you can use gpu index when saving files. Something like the following:</p>
<pre><code class="lang-auto">def save_tensors(gpu, total_gpus):
    torch.cuda.set_device(gpu)
    torch.distributed.init_process_group(backend = 'nccl', init_method='env://', world_size=total_gpus, rank=gpu)
    for i in range(100):
        tensor = torch.rand(2,3)
        torch.save(tensor, f'tensor_{i}_gpu_{gpu}.pt')
def main():
    gpu_count = torch.cuda.device_count()
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '1234'
    torch.multiprocessing.spawn(save_tensors, nprocs = gpu_count, args = (gpu_count, ))
</code></pre>
RuntimeError: Found dtype Half but expected Float backward

mellamodios: <p>i’m trying to train my model in PyTorch but I have the following error:</p>
<pre><code>---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-47-4c3349f66d9b&gt; in &lt;module&gt;
     26 
     27             # Backpropagate loss
---&gt; 28             loss.backward()
     29 
     30             # Steps

~/anaconda3/lib/python3.8/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    219                 retain_graph=retain_graph,
    220                 create_graph=create_graph)
--&gt; 221         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    222 
    223     def register_hook(self, hook):

~/anaconda3/lib/python3.8/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    128         retain_graph = create_graph
    129 
--&gt; 130     Variable._execution_engine.run_backward(
    131         tensors, grad_tensors_, retain_graph, create_graph,
    132         allow_unreachable=True)  # allow_unreachable flag

RuntimeError: Found dtype Half but expected Float
</code></pre>
<p>My code:</p>
<pre><code>    criterion_KL = torch.nn.KLDivLoss()

    logits_per_image, logits_per_text = model(images.to(device), texts.to(device)) # both NxN matrix
    
    optim.zero_grad()
    
    labels = labels_matrix(tax) # returns a NxN matrix, len(tax) = N

    loss_i = criterion_KL(logits_per_image, labels)
    loss_j = criterion_KL(logits_per_text, labels)
    loss = (loss_i + loss_j)/2


    print(loss_i, loss_j, loss)
   # tensor(-0.6089, device='cuda:0', grad_fn=&lt;KlDivBackward&gt;) tensor(-0.6089, device='cuda:0', grad_fn=&lt;KlDivBackward&gt;) tensor(-0.6089, device='cuda:0', grad_fn=&lt;DivBackward0&gt;)

    print(loss_i.dtype, loss_j.dtype, loss.dtype)
   # torch.float32 torch.float32 torch.float32
</code></pre>
<p>I have no idea why this error is happening. Tried to cast the <code>labels</code> tensor to float, but didn’t have results.<br>
Any help?</p>

ptrblck: <p>Are you manually casting the model to <code>half()</code> or any parts of it?<br>
Also, could you update to the latest PyTorch release (<code>1.9.0</code>), if you are using an older one? In case you are still hitting this issue in the latest release, could you post an executable code snippet to reproduce this issue?</p>
Running a vocoder in C++

pak123: <p>Hello,<br>
I have a mel file which I generated using Python and saved using torch.save().<br>
Now i want to load this file in C++ into a torch:;Tesnsor.<br>
I looked at from_blob but it does not seem to fit this scenario.<br>
How can I load such a file (or pickled npy?) from disk into a tensor in C++?</p>
<p>Thanks</p>

ptrblck: <p>I’m not sure, if <a href="https://github.com/pytorch/pytorch/issues/36568#issuecomment-613688287">this comment</a> is still up-to-date, but if so you should use <code>torch.jit.save</code> on the Python side and load it via <code>torch::load</code> in C++.</p>
Adding Swallow Layer at the end of NN

Tzeni_Botsi: <p>Hello, i want to add a swallow classifier (SVM, RF, etc) as the last layer of my NN and back propagate. My initial thought is to use <code>predict_proba</code> of sklearn which can be viewed as the output of a softmax layer.</p>
<p>Lets say i have my output tensor from forwarding the input data through the basic network and then i feed it to sklearn. After that i change the output tensor’s values to match the sklearn predictions. Will this mess the computation graph? I mean i can’t use the new tensor to a BCE loss and back propagate, right?</p>
<p>Maybe instead of replacing the network’s output tensor with sklearn’s predictions i could piecewise multiply  the networks tensor with the appropriate vector in order to match sklearn’s predictions. Any ideas?</p>
<p>Thanks in advance!</p>
How can i get class_id in the file?

samm: <p>How can i dump in the file <code>class_id</code> or <code>category_id</code> of the coco image , if i’m using mmdetection. Should i change in class <code>aips/test.py</code> or <code>datasets/coco.py</code> or the <code>config</code> file for the model ? as what i’m got this array for every image</p>
<pre><code class="lang-auto">[3.41282135e+02, 2.03359509e+01, 4.95520966e+02, 3.14121826e+02,
        8.74738395e-01]
</code></pre>
<p>and i need the structure of the file contains</p>
<pre><code class="lang-auto">x1,y1,x2,y2,class_id,bbox_score
</code></pre>
<p>and sorry if the post in not appropriate here if someone can help or guide me i will appreciate that</p>
Difference between Vocab and vocab

mmg: <p>Does anyone know the difference between torchtext.vocab.Vocab and torchtext.vocab.vocab?</p>
<p>All I know is that I can’t pass the <code>min_freq</code> parameter to the *.Vocab.</p>

pascal_notsawo: <p>torchtext.vocab.Vocab is a class while torchtext.vocab.vocab is a method that returns an object of type torchtext.vocab.Vocab.</p><aside class="onebox githubblob">
  <header class="source">

      <a href="https://github.com/pytorch/text/blob/master/torchtext/vocab.py#L225-L230" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/text/blob/master/torchtext/vocab.py#L225-L230" target="_blank" rel="noopener nofollow ugc">pytorch/text/blob/master/torchtext/vocab.py#L225-L230</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="225" style="counter-reset: li-counter 224 ;">
          <li>tokens = []</li>
          <li>for token, freq in ordered_dict.items():</li>
          <li>    if freq &gt;= min_freq:</li>
          <li>        tokens.append(token)</li>
          <li>
          </li>
<li>return Vocab(VocabPybind(tokens, None))</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>A bit like tensor and Tensor.</p>
Tensor concatenation

alam4545: <p>How to get the follwing output?</p>
<p>Inputs:</p>
<p>bx1 = torch.tensor([[1,2],[3,4],[5,6],[7,8]])<br>
tensor([[1, 2],<br>
[3, 4],<br>
[5, 6],<br>
[7, 8]])<br>
bx2 = torch.tensor([[11,22],[33,44]])<br>
tensor([[11, 22],<br>
[33, 44]])</p>
<p>OUtput:</p>
<p>tensor([[[ 1,  2, 11, 22],<br>
[ 1,  2, 33, 44]],</p>
<pre><code>    [[ 3,  4, 11, 22],
     [ 3,  4, 33, 44]],

    [[ 5,  6, 11, 22],
     [ 5,  6, 33, 44]],

    [[ 7,  8, 11, 22],
     [ 7,  8, 33, 44]]])</code></pre>

eqy: <p>I think this can work:</p>
<pre><code class="lang-auto">import torch
bx1 = torch.tensor([[1,2],[3,4],[5,6],[7,8]])
bx2 = torch.tensor([[11,22],[33,44]])
temp = torch.repeat_interleave(bx1, 2, dim=0)
temp = temp.reshape(4, 2, -1)
output = torch.cat((temp, bx2.repeat(4, 1, 1)), dim=2)
print(output)
</code></pre>
<pre><code class="lang-auto">tensor([[[ 1,  2, 11, 22],
         [ 1,  2, 33, 44]],

        [[ 3,  4, 11, 22],
         [ 3,  4, 33, 44]],

        [[ 5,  6, 11, 22],
         [ 5,  6, 33, 44]],

        [[ 7,  8, 11, 22],
         [ 7,  8, 33, 44]]])
</code></pre>
CUDA ran out of memory even though it has free space

Lucky_Magna: <p>Don’t know why this is happening, it used to work fine before.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/466a81fee80f2b66208e680e8a7b98adf04e3d27.png" data-download-href="https://discuss.pytorch.org/uploads/default/466a81fee80f2b66208e680e8a7b98adf04e3d27" title="Screenshot (1)"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/6/466a81fee80f2b66208e680e8a7b98adf04e3d27.png" alt="Screenshot (1)" data-base62-sha1="a2VC7pzExYki5oJWu2JMQ62ri5x" width="690" height="395" data-dominant-color="161616"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (1)</span><span class="informations">1103×633 51.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b43e4b4b97d392718d0ac3ac241b65e1f9841c0d.png" data-download-href="https://discuss.pytorch.org/uploads/default/b43e4b4b97d392718d0ac3ac241b65e1f9841c0d" title="Screenshot (3)"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b43e4b4b97d392718d0ac3ac241b65e1f9841c0d.png" alt="Screenshot (3)" data-base62-sha1="pIvpn35GLDHUgB75M0SVnQtYj8x" width="690" height="393" data-dominant-color="161616"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (3)</span><span class="informations">713×407 16.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Need help ASAP.</p>

Gantavya_Bhatt1: <p>Could you please provide more details, such as source code and exactly what snippet is taking this memory?</p>
How to convert torch._C.graph text into dot language

charlesjiangxm: <p>Hi,<br>
Is there any way to convert a <code>torch._C.Graph</code> text like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9ea852805303a68ea9e629770b4ce20b992f451.png" data-download-href="https://discuss.pytorch.org/uploads/default/f9ea852805303a68ea9e629770b4ce20b992f451" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9ea852805303a68ea9e629770b4ce20b992f451.png" alt="image" data-base62-sha1="zERlwKhJELmdHZ2oQ3VAZX7Thip" width="598" height="500" data-dominant-color="EAEBEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">630×526 15.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
generated by <code>torch.jit.trace(model, input)</code> directly into .dot language used by networkX, GraphViz, etc. like this</p><aside class="onebox wikipedia" data-onebox-src="https://en.wikipedia.org/wiki/DOT_(graph_description_language)">
  <header class="source">

      <a href="https://en.wikipedia.org/wiki/DOT_(graph_description_language)" target="_blank" rel="noopener nofollow ugc">en.wikipedia.org</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://en.wikipedia.org/wiki/DOT_(graph_description_language)" target="_blank" rel="noopener nofollow ugc">DOT (graph description language)</a></h3>

<p>DOT is a graph description language, developed as a part of the Graphviz project. DOT graphs are typically stored as files with the .gv or .dot filename extension — .gv is preferred, to avoid confusion with the .dot extension used by versions of Microsoft Word before 2007. dot is also the name of the main program to process DOT files in the Graphviz package.
 Various programs can process DOT files. Some, such as dot, neato, twopi, circo, fdp, and sfdp, can read a DOT file and render it in graphic...</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Thank you</p>
Profiling a scripted/frozen models

saar-eliad: <p>Hi,<br>
We want to profile the execution of scripted/frozen models.<br>
I know that the profiler <code>record_function</code> op can be scripted, but I want to avoid having 2 versions of the model (one for profiling, and one for deployment).<br>
Is there a suggested way to do this?</p>
Skip connection implementation

promach: <p>how to implement skip connection for this <a href="https://gist.github.com/promach/b6f526c56e20f029d68e6f9041c3f5c0#file-gdas-py-L124" rel="noopener nofollow ugc">coding</a> ?</p>
<pre><code class="lang-auto">class SkipEdge(Edge):
    def __init__(self):
        super().__init__()
        self.f =
</code></pre>
NN-generated tensor to image bytes

kashik00007: <p>Hello all.</p>
<p>I am having some questions regarding my approach to generate real-time image from a neural network. It works quite well (for a portable macbook, but after 2 or 3 seconds after starting the fan gets much faster and I can get frame drops). The process is:</p>
<p>Step 1) I have a tensor with pixel values in the “result” variable. with the shape <code>torch.Size([1, 1, 310, 310])</code> being 310x310 the size and with <code>grad_fn=&lt;PermuteBackward&gt;</code><br>
Step 2) then I create <code>x = torchvision.utils.make_grid(result)</code><br>
Step 3) then I create an image with <code>img = F.to_pil_image(x)</code><br>
Step 4) then I pass everything to bytes <code>data = img.tobytes("raw", "RGBX", 0, -1)</code> to be displayed</p>
<p>And I do this in real time to give display in a rendering system (for the moment a GUI). I confess this seems like a bit of an intensive approach to generate frame by frame and get a decent framerate without the computer starting to complain.</p>
<p>Any help?<br>
Thanks</p>
Define input shape and units of conv2d layers

kashik00007: <p>Hello all.</p>
<p>I’m trying to define units, input shape, activations and kernel_initializer for conv2d layers in pytorch, and I think this is something I got from tensorflow and might not work this way here (units as in filters and input shape e.g. [h, w, 2 + zdimension]).</p>
<p>Defining only these parameters <code>torch.nn.Conv2d (*in_channels*, *out_channels*, *kernel_size*, *stride=1*, *padding=0*, *dilation=1*, *groups=1*, *bias=True*, *padding_mode='zeros'* )</code> doesn’t make much sense to me.</p>
<p>Sorry if this is a silly question, new to torch.<br>
Thanks!</p>

ptrblck: <p>I’m not sure what’s currently not working. The posted code looks alright and should initialize an <code>nn.Conv2d</code> layer. Could you explain your issue a bit more, please?</p>

kashik00007: <p>It was working, thank you for the help!</p>
AttributeError: 'SegmentConsensusBackward' object has no attribute 'consensus_type'

Mai_Magdy: <p>I have this error appears after i have added . apply to object class calling as it was written this function has been deprecated when i used torch.autograde.function<br>
the following is the code:<br>
import torch</p>
<p>import math</p>
<p>import numpy</p>
<p>class Identity(torch.nn.Module):</p>
<pre><code>def forward(self, input):

    return input
</code></pre>
<p>class SegmentConsensus(torch.autograd.Function):</p>
<pre><code>def __init__(self, consensus_type, dim=1):

    self.consensus_type = consensus_type

    self.dim = dim

    self.shape = None

def forward(self, input_tensor):

    print(input_tensor)

    self.shape=numpy.array(input_tensor)

    #self.shape = len(input_tensor)

    print(self.shape)

    if self.consensus_type.apply == 'avg':

        output = input_tensor.mean(dim=self.dim, keepdim=True)

    elif self.consensus_type == 'identity':

        output = input_tensor

    else:

        output = None

    return output

def backward(self, grad_output):

    if self.consensus_type == 'avg':

        grad_in = grad_output.expand(self.shape) / float(self.shape[self.dim])

    elif self.consensus_type == 'identity':

        grad_in = grad_output

    else:

        grad_in = None

    return grad_in
</code></pre>
<p>class ConsensusModule(torch.nn.Module):</p>
<pre><code>def __init__(self, consensus_type, dim=1):

    super(ConsensusModule, self).__init__()

    self.consensus_type = consensus_type if consensus_type != 'rnn' else 'identity'

    self.dim = dim

def forward(self, input):

    return SegmentConsensus((self.consensus_type, self.dim)).apply(input)
</code></pre>
<p>and this is the error<br>
f self.consensus_type.apply == ‘avg’:<br>
AttributeError: ‘SegmentConsensusBackward’ object has no attribute ‘consensus_type’</p>

ptrblck: <p>I guess the error might be raised, since you are using the legacy definition of custom <code>autograd.Function</code>s and would have to adapt the code to the new approach as explained <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>.</p>
Torch serve: dynamic batching?

johann-petrak: <p>I have been unable to figure out if torch serve supports dynamic batching and if yes how:</p>
<p>I have some model where throughput could be optimized if we always run batchsize &gt; 1 intances through the model at once.</p>
<p>So it would be cool if torchserve can collect requests that are received within a certain amount of time and group them into batches for processing.</p>
<p>This should be something that can be configured through the torch serve config file or through command line options when starting, not through the torchserve API itself, as it should be set up when the torch serve serve is started with the initial model.</p>
<p>But I cannot find any documentation about how to do this.</p>
<p>Is this supported and if yes, how?</p>
Have trouble getting correct dimensions/shape to feed into neural network in Pytorch

vanstorm9: <p>I am trying to feed a tensor of a certain size into my linear layer, but I am struggling with getting the correct dimensions (not too experienced with this)</p>
<p>I realize that I have to change the dimensions of my linear layer (self.linear1), but I don’t know what dimensions I should set my linear layer to. I am currently feeding in a vector of size [32, 1164].</p>
<p>This is summary of the print output as well as error I get.</p>
<pre><code class="lang-auto">torch.Size([32, 1152])
torch.Size([32, 1])
x.shape:  torch.Size([32, 1164])


 File "/home/vanstorm/Documents/Programming/reinforcement-learning/curiousModule.py", line 113, in forward
    y = F.relu(self.linear1(x))
  File "/home/vanstorm/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/vanstorm/.local/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/vanstorm/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 1690, in linear
    ret = torch.addmm(bias, input, weight.t())

RuntimeError: mat1 dim 1 must match mat2 dim 0
</code></pre>
<p>Apparently my dimensions are incorrect, and I am not too experienced with handling different dimensional sizes in Pytorch. What would be a suitable dimension to set my self.linear1 and self.linear2?</p>
<p>Can anyone also explain the logic of the correct dimensional numbers as well so that I can avoid this. How would I calculate the desired dimensions of a linear layer?</p>
<p>Code I am using:</p>
<pre><code class="lang-auto">class Fnet(nn.Module): #C     Forward network
    def __init__(self):
        super(Fnet, self).__init__()
        self.linear1 = nn.Linear(300,256)
        self.linear2 = nn.Linear(256,288)



    def forward(self,state,action):
        action = action.unsqueeze(1)
        action_ = torch.zeros(action.to(device).shape[0],12).to(device) #D


        
        indices = torch.stack( (torch.arange(action.shape[0]).to(device), action.squeeze().to(device)), dim=0)
        indices = indices.tolist()
        action_[indices] = 1.
        
        x = torch.cat( (state,action_) ,dim=1)
        print(state.shape)
        print(action.shape)
        x = torch.flatten(x,start_dim = 1)
        print('x.shape: ',x.shape)

        y = F.relu(self.linear1(x))
        y = self.linear2(y)
        return y
</code></pre>
<p>Can anyone help me?</p>

eqy: <p>It looks like your first two linear layers match each other in terms of dimension, but the first layer does not match the input size. (e.g., in this case to match it should have an input dimension of 1164 ：</p>
<pre><code class="lang-auto">self.linear1 = nn.Linear(1164,256)
</code></pre>
<p>)</p>
Ground truth shape for multiclass segmentation

sloan: <p>I am setting up a basic Unet model to segment three classes. The individual mask data is set up with shape (H, W, 1) where each pixel has value between 0 and 3. I am wondering whether I can use masks as they are or do they need to be transformed to one-hot-encoded format?<br>
I.e. can the target simply be (H, W, batch_size)?</p>

ptrblck: <p>Assuming you are using <code>nn.CrossEntropyLoss</code> for the multi-class segmentation, the model output should contain logits in the shape <code>[batch_size, nb_classes, height, width]</code> and the target should contain values in the range <code>[0, nb_classes-1]</code> and have the shape <code>[batch_size, height, width]</code>.<br>
Given that you are using class indices in <code>[0, 3]</code> you would be dealing with 4 classes (maybe you implicitly explained it as 3 classes + background).</p>