akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Reading image data through dataset class

israrbacha: <pre><code class="lang-auto">class RDataset(Dataset):
    def __init__(self, data_path, data_name, data_type, patch_size=None, length=None):
        super().__init__()
        self.data_name, self.data_type, self.patch_size = data_name, data_type, patch_size
        self.A_images = sorted(glob.glob('{}/{}/{}/rain/*.png'.format(data_path, data_name, data_type)))
        self.B_images = sorted(glob.glob('{}/{}/{}/norain/*.png'.format(data_path, data_name, data_type)))
        # make sure the length of training and testing different
        self.num = len(self.A_images)
        self.sample_num = length if data_type == 'train' else self.num

    def __len__(self):
        return self.sample_num

    def __getitem__(self, idx):
        image_name = os.path.basename(self.A_images[idx % self.num])
     A = T.to_tensor(Image.open(self.A_images[idx % self.num]))
     B = T.to_tensor(Image.open(self.B_images[idx % self.num]))
        h, w = A.shape[1:]
return A,B
</code></pre>
<p>I went through the above code and it works fine but my confusion is, after wrapping the dataset class in <strong>Datloader(RDataset, batch_size=10)</strong> how the value of idx is generated? my understanding is that idx will take values between 0 and len(self.num), and idx%self.num will always be <strong>zero</strong> means it will always refer to data sample at idx 0?</p>

ptrblck: <aside class="quote no-group" data-username="israrbacha" data-post="1" data-topic="156180">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/israrbacha/48/34290_2.png" class="avatar"> israrbacha:</div>
<blockquote>
<p>how the value of idx is generated?</p>
</blockquote>
</aside>
<p>It’s generated by the <code>sampler</code> in the range <code>[0, len(dataset)]</code>. E.g. the default <code>RandomSampler</code> creates the indices <a href="https://github.com/pytorch/pytorch/blob/3f56a1b8c0e9038e80bb12ceab1aea7e114f6e09/torch/utils/data/sampler.py#L117-L136">here</a>.</p>
<aside class="quote no-group" data-username="israrbacha" data-post="1" data-topic="156180">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/israrbacha/48/34290_2.png" class="avatar"> israrbacha:</div>
<blockquote>
<p>and idx%self.num will always be <strong>zero</strong></p>
</blockquote>
</aside>
<p>No, that should not be the case and seems to be used if <code>self.sample_num</code> (and thus the length of the dataset) doesn’t match the actual number of samples to avoid an out-of-bounds indexing operation:</p>
<pre data-code-wrap="python"><code class="lang-python">A_images = torch.randn(10, 1)
sample_num = 20

for idx in range(sample_num):
    print(idx % len(A_images))
``
so it can be used for a repeated sampling procedure.</code></pre>
Weighted Binary Cross Entropy

Can_Keles: <p>Hi, i was looking for a Weighted BCE Loss function in pytorch but couldnt find one, if such a function exists i would appriciate it if someone could provide its name.</p>

ptrblck: <p><a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCEWithLogitsLoss">nn.BCEWithLogitsLoss</a> takes a <code>weight</code> and <code>pos_weight</code> argument.<br>
From the docs:</p>
<blockquote>
<p><strong>weight</strong> (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor"> <em>Tensor</em> </a> <em>,</em> <em>optional</em> ) – a manual rescaling weight given to the loss of each batch element. If given, has to be a Tensor of size nbatch.<br>
<strong>pos_weight</strong> (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor"> <em>Tensor</em> </a> <em>,</em> <em>optional</em> ) – a weight of positive examples. Must be a vector with length equal to the number of classes.</p>
</blockquote>
<p>Is one of these weights what you are looking for?</p>

Can_Keles: <p>I think pos_weight is the one I was looking for, thank you for your time</p>

AshviniKSharma: <p>Can you explain why they say positive example not just example?</p>

ptrblck: <p>For a binary classification, you would often hear positive and negative example, which would represent the classes 1 and 0, respectively.<br>
I think it’s the standard terminology, which is also used in e.g. confusion matrices and to calculate other metrics such as “True positive rate”, “True negative rate”, “False positive rate”, etc.</p>

AshviniKSharma: <p>Thanks, but that was not what I was looking for. To be more clear, can you give me example to calculate weights for multilabel case. For example suppose I have example 10 examples and each example can belong to multiple label/class. In that situation what should be the process to calculate pos weights that can be used in loss function?</p>

ptrblck: <p>You could treat each occurrence of a class as the positive sample and could calculate the <code>pos_weight</code> for each class.<br>
I.e. if your complete dataset contains 100 samples in total, 90 class0 samples, and 80 class1 samples, your <code>pos_weight</code> could be calculated as <code>negative/positive = [10/90, 20/80]</code>.</p>

PNayak: <p>Sorry, the last bit is confusing. Can you elaborate ?<br>
100 Samples = 90 Class0 samples + 10 Class1 samples; Or 100 samples = 20 Class0 samples + 80 Class1 samples.</p>
<p>But how 100 samples = 90 Class0 + 80 Class1 ?</p>

PNayak: <p>Okay I see, if the samples have multiple targets, it makes sense. Sorry for the stupid Q !</p>

Maxisawesome: <p>if you only have one class you can pass in a tensor of length one into pos_weight, and the 1s of that class will be correspondingly upweighted with the 0s at weight of 1</p>
<p>FYI for future ppl finding this post I struggled with it for a bit lol</p>
Trying to use ConvNeXt as Faster-RCNN backbone

peggs: <p>I’m having a little trouble trying to train a Faster-RCNN model on COCO, with an ImageNet-pretrained <code>torchvision</code> ConvNeXt as the backbone, as shown below:</p>
<pre><code class="lang-python">import torch
import torchvision.models.detection as torchdet
from torchvision.models import convnext_tiny, ConvNeXt_Tiny_Weights

backbone = convnext_tiny(weights=ConvNeXt_Tiny_Weights.DEFAULT).features
# 768 determined using torchinfo.summary(backbone, (3,300,300))
backbone.out_channels = 768

# 5x3 per location
anchor_generator = torchdet.rpn.AnchorGenerator(
    sizes=((32, 64, 128, 256, 512),), aspect_ratios=((0.5, 1.0, 2.0),))

roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=['0', '1', '2', '3'],
                                                output_size=7,
                                                sampling_ratio=2)

# 91 classes in MS COCO
model = torchdet.FasterRCNN(backbone=backbone, num_classes=91,
                            rpn_anchor_generator=anchor_generator, box_roi_pool=roi_pooler)
</code></pre>
<p>I’m trying to emulate the <a href="https://github.com/pytorch/vision/blob/e75a333782bb5d5ffdf5355e766eb5937fc6697c/references/detection/README.md" rel="noopener nofollow ugc">training recipes</a> used by the Torchvision team, so my setup looks like this:</p>
<pre><code class="lang-python">params = [p for p in model.parameters() if p.requires_grad]
# 0.0025 LR used because only using 1 GPU, Facebook used 0.02 for 8 GPUs
optimizer = torch.optim.SGD(params, lr=0.0025, momentum=0.9, weight_decay=1e-4)
lr_scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[16, 22], gamma=0.1)

#Using batchsize 2
</code></pre>
<p>However, the loss will only drop to about 0.5 before jumping back up to 1.8, across iterations in just one epoch. <strong>Is there something else I should change about the training to better match what the Torchvision team did for training its other FasterRCNN models?</strong></p>
<p><strong>I’m also curious what</strong> the <code>featmap_names</code> in <code>torchvision.ops.MultiScaleRoIAlign(featmap_names=['0', '1', '2', '3'],...)</code> is supposed to do?</p>
<p>Thanks!</p>
Multitask Regression

maxrivera: <p>I have CSV data with 2 target columns, each is continuous data that can be modeled as a regression problem. How would I structure a neural network in PyTorch with 2 output targets? What would the final layer look like? What would the loss function be? Thanks</p>

KFrank: <p>Hi Max!</p>
<aside class="quote no-group" data-username="maxrivera" data-post="1" data-topic="156125" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/da6949/48.png" class="avatar"> maxrivera:</div>
<blockquote>
<p>I have CSV data with 2 target columns, each is continuous data that can be modeled as a regression problem.</p>
</blockquote>
</aside>
<p>As I understand it you have it, you have input data – perhaps images,<br>
perhaps measurements or statistics describing some object, etc. – and<br>
for each sample, you have two target values.</p>
<p>You also say that your target values are continuous and can be modeled<br>
as “regression.”</p>
<p>From this I understand that if your target is, say, 17.20, then a predicted<br>
value of 17.21 would be quite good, 17.30 would be slightly less good,<br>
while 112.00 would be quite bad.  That is, closer is better, and almost<br>
right is almost as good as exactly right.</p>
<p>For such a situation, the upstream part of your network would be whatever<br>
is appropriate for processing your input data.  For example, if your input<br>
samples are images, your network might start out with a number of<br>
convolutional layers.  On the other that, if your input data is simply vectors<br>
of “unstructured” numerical data, your network might begin immediately<br>
with fully-connected layers.</p>
<p>Towards the end of your network, you would typically have just vectors of<br>
numerical “features.”  Your final layer should be fully-connected with two<br>
outputs:</p>
<pre data-code-wrap="python"><code class="lang-python">final_layer = torch.nn.Linear (in_features = n_features, out_features = 2)
</code></pre>
<p>This will output two numerical predictions, one each for each of the two<br>
values in your target.  You would then typically use <code>MSELoss</code> as your loss<br>
criterion.</p>
<p>Best.</p>
<p>K. Frank</p>

maxrivera: <p>Thanks for the reply. That’s exactly what I ended up doing. PyTorch’s MSELoss also handles vectors, so it can average the loss of both features at the same time. Nice to know it was recommended.</p>
Custom Function with the same backward as Softmax

kibrq: <p>Suppose, I have a variable x of a shape (L, N) and a following sampling operation:</p>
<ul>
<li>softmaxed = softmax(x, dim=1)</li>
<li>sampled = torch.mutlinomial(softmaxed, k)</li>
<li>one_hot_encoded = torch.one_hot(sampled, N).to(torch.float64)</li>
</ul>
<p>I want to make this sequence of operation <em>diffirentiable</em>. To do so one may say, that the derivative is approximately the same as derivative of softmax.</p>
<p>So I want to inherit the Function class and make a function <code>sample</code> and I want to reuse softmax derivative calculation. I didn’t find a calculation of softmax derivative in pytorch. Any advice?</p>

kibrq: <p>I came up with the following solution:</p>
<h1>
<a name="inputshape-l-n-where-l-is-a-length-of-a-sequence-and-n-is-a-number-of-symbols-1" class="anchor" href="#inputshape-l-n-where-l-is-a-length-of-a-sequence-and-n-is-a-number-of-symbols-1"></a>input.shape = [L, N] where L is a length of a sequence and N is a number of symbols</h1>
<p>class SoftmaxSample(Function):</p>
<pre><code>@staticmethod
def forward(ctx, input, num_samples):
    L, N = input.shape

    with torch.enable_grad():
        softmaxed = torch.softmax(input, dim = 1)
    sampled   = torch.multinomial(softmaxed, num_samples, replacement = True).transpose(1, 0)
    output    = F.one_hot(sampled, N).to(torch.float64)

    ctx.save_for_backward(input, softmaxed)
    return output

@staticmethod
def backward(ctx, grad_output):
    input, softmaxed = ctx.saved_tensors
    grad_input = grad_num_samples = None
    
    grad_input, = torch.autograd.grad(softmaxed, input, grad_outputs=grad_output.mean(dim=0))
    return grad_input, None
</code></pre>
<p>softmax_sampler = SoftmaxSampler.apply</p>
Convert a piece of code from TensorFlow to PyTorch

DRISS_ELALAOUI: <p>please, I want to convert this piece of code from TensorFlow to PyTorch:</p>
<pre><code class="lang-auto">with tf.name_scope("embedding"):
            self.embedding_matrix = tf.get_variable(
                    "W", shape=[self.vocab_size, self.embedding_size],
                    initializer=tf.contrib.layers.xavier_initializer(seed=1234))

            self.softmax_biases = tf.Variable(tf.zeros([self.vocab_size]))
</code></pre>
<p>How can I do it ? and thank you</p>

ptrblck: <p>Something like this should work:</p>
<pre><code class="lang-python">embedding_matrix = torch.empty([self.vocab_size, self.embedding_size])
torch.nn.init.xavier_normal_(embedding_matrix)
softmax_biases = torch.zeros([self.vocab_size])
</code></pre>

DRISS_ELALAOUI: <p>Well, thank you very much for your help</p>
[DDP] should I do mp.spawn when there is only 1 GPU per node?

B.W_Zhang: <p>In a multi-node multi-GPU scenario, I use mp.spawn to spawn nproc_per_node processes on each node. But when I’m using multiple nodes with only 1 GPU on each node, should I still do the spawn? because spawning 1 process sounds redundant. Will spawning in this case be slower or faster or have little effect?<br>
Thanks!</p>

kwen2501: <p>I’d say you wouldn’t need to spawn processes in general one-GPU-per-node cases.</p>
Create_feature_extractor, nn.Parameter, DataParallel are not compatiable together

lemon1: <p>Here is the code I run:</p>
<pre><code class="lang-auto">import torch
from torchvision.models.feature_extraction import create_feature_extractor
from torchvision.models import resnet50, vit_b_16
from torch.nn import DataParallel

return_nodes = {"heads": "0"}
# return_nodes = {"fc": "0"}
device = torch.device("cuda")
backbone = vit_b_16()
backbone = create_feature_extractor(backbone, 
        return_nodes=return_nodes)
model1 = DataParallel(backbone).to(device)
x = torch.rand(2, 3, 256, 256).to(device)
out = model1(x)
</code></pre>
<p>and the error message is</p>
<pre><code class="lang-auto">  File "/home/ubuntu/models/test_param_error.py", line 60, in &lt;module&gt;
    out = model1(x)
  File "/home/ubuntu/anaconda3/envs/torchenv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/torchenv/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 167, in forward
    replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
  File "/home/ubuntu/anaconda3/envs/torchenv/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 172, in replicate
    return replicate(module, device_ids, not torch.is_grad_enabled())
  File "/home/ubuntu/anaconda3/envs/torchenv/lib/python3.9/site-packages/torch/nn/parallel/replicate.py", line 148, in replicate
    setattr(replica, key, param)
  File "/home/ubuntu/anaconda3/envs/torchenv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1206, in __setattr__
    raise TypeError("cannot assign '{}' as parameter '{}' "
TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'class_token' (torch.nn.Parameter or None expected)
</code></pre>
<p>My torch and torchvision version:</p>
<pre><code class="lang-auto">pytorch                   1.11.0          py3.9_cuda11.5_cudnn8.3.2_0
torchvision               0.12.0               py39_cu115
</code></pre>
<p>Here is what I find in a multi-gpu setting.</p>
<ol>
<li>If the model is resnet50 (and the return node is some resnet50 layer), this code has no error. For ViTs, they have defined some parameters using <code>nn.Parameter (self.class_token = nn.Parameter(torch.zeros(1, 1, hidden_dim)))</code>. That is where the error occurs.</li>
<li>
<code>create_feature_extractor</code>, <code>nn.Parameter</code>, <code>DataParallel</code>: either two of them would work together but not all three.</li>
<li>From the error message we can know that the program tries to assign some tensor to the parameter <code>class_token</code>. If we don’t use <code>create_feature_extractor</code>, this would work. Why does <code>create_feature_extractor</code> change the behavior?</li>
</ol>
<p>Thanks for help.</p>
How do I get Predictions and error rate from the fasterrcnn model?

StrafeNDestroy: <p>Hello I am new to pytorch and training AI I was wondering how I get my predictions and error rate from the Fasterrcnn model. This code is the return of sending a single batch to the model in the eval state. How come I am getting so many boxes, scores and labels. I only have 3 class [Truck,Car,Jeep]=[0,1,2]<br>
should only have one bounding boxes per image and one score?</p>
<pre><code class="lang-python">[{'boxes': tensor([[6.3082e+02, 6.6109e-01, 1.0240e+03, 9.4249e+02],
        [3.1181e+02, 2.9275e-01, 1.0240e+03, 6.0196e+02],
        [0.0000e+00, 9.7500e-02, 6.8195e+02, 6.7823e+02],
        [3.1015e+02, 3.8843e+02, 1.0240e+03, 1.0119e+03],
        [0.0000e+00, 3.8825e+02, 6.8305e+02, 1.0115e+03],
        [1.8268e+02, 2.1860e+02, 1.0240e+03, 7.1305e+02],
        [1.7617e+02, 1.0996e-01, 8.9179e+02, 9.5223e+02],
        [6.5779e+02, 0.0000e+00, 1.0240e+03, 1.0240e+03],
        [3.6129e+02, 0.0000e+00, 1.0240e+03, 6.5571e+02],
        [3.6079e+02, 3.6690e+02, 1.0240e+03, 1.0240e+03],
        [0.0000e+00, 0.0000e+00, 6.4789e+02, 7.4001e+02],
        [0.0000e+00, 3.6656e+02, 6.4843e+02, 1.0240e+03],
        [2.2603e+02, 0.0000e+00, 8.5769e+02, 1.0240e+03]], device='cuda:0',
       grad_fn=&lt;StackBackward0&gt;),
  'labels': tensor([1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2], device='cuda:0'),
  'scores': tensor([0.3015, 0.3010, 0.3006, 0.3003, 0.3002, 0.2989, 0.2988, 0.1948, 0.1933,
        0.1914, 0.1914, 0.1901, 0.1872], device='cuda:0',
       grad_fn=&lt;IndexBackward0&gt;)}
</code></pre>

Andrei_Cristea: <p>Generally you need to do more post-processing steps to convert this raw output (which is a continuous set of beliefs over a lot of candidate bounding boxes) to specific predictions. This requires things like having a confidence threshold (a minimum score below which you throw out results) and in some cases may require doing things to consolidate boxes that are “close enough”.</p>
<p>Here are some resources:</p>
<ul>
<li>region proposal <a href="https://towardsdatascience.com/faster-rcnn-1506-01497-5c8991b0b6d3" rel="noopener nofollow ugc">here</a>
</li>
<li>simple “predict” function <a href="https://debuggercafe.com/faster-rcnn-object-detection-with-pytorch/" rel="noopener nofollow ugc">here</a>
</li>
<li>NMS explanation <a href="https://medium.com/swlh/object-detection-with-deep-learning-rcnn-anchors-non-maximum-suppression-ce5a83c7c62b" rel="noopener nofollow ugc">here</a>
</li>
</ul>

StrafeNDestroy: <p>Thank you so much for your help and the resources</p>
Stateful LSTM training

bid_dash: <p>I am trying to train a stateful ConvLSTM module. Basically for each batch in mini-batch training, I want to save the hidden states for next batch to be used. If I do that without <code>hidden_state.detach()</code>, it throws an error. But, if I do with the detach function, the training goes on!</p>
<ol>
<li>My understanding is that we definitely need hidden_state in computing gradients wrt the loss. So why do we need to <code>hidden_state.detach()</code>? Won’t this detach the hidden state from gradient computation for this variable?</li>
<li>Secondly, <code>optimizer.zero_grad()</code> should be called when I really want to free up the gradients and not in every batch forward pass as I need the accumulated gradients until the TBPTT is over.</li>
<li>Should I call <code>optimizer.step()</code> every forward pass or when the TBPTT window is completed?</li>
</ol>

pragyan_dahal: <p>Hi Bidya, I am facing the exact same issue with my Network. Can you please tell me where exactly do you put the .detach()? The snippet of the code would be really helpful. Thanks</p>
Question about autograd and leaf tensors

vikigenius: <p>I have always been confused about this behaviour</p>
<p>Let us say I am implementing an autoencoder. My source and target are going to be same, let us say I want to reconstruct some vector x.</p>
<p>So something like<br>
z = Encoder(x)<br>
y = Decoder(z)</p>
<ol>
<li>
<p>loss = LOSS(y, x)</p>
</li>
<li>
<p>loss = LOSS(y, x.clone())</p>
</li>
<li>
<p>loss = LOSS(y, x.clone().detach())</p>
</li>
</ol>
<p>What is the difference in behavior in each of the cases, x is of course a leaf tensor, but I don’t understand the gradients aspect is it 1) going to form a cycle ?</p>

ptrblck: <p>It shouldn’t be a problem, as usually <code>x</code> won’t require gradients in an autoencoder setup.<br>
The first two approaches should yield the same result, while the last one should be different.<br>
Here is a small dummy code:</p>
<pre><code class="lang-python">x = torch.randn(1, requires_grad=True)
param = nn.Parameter(torch.randn(1))
output = x * param

# 1.
loss = F.mse_loss(output, x)
loss.backward()
print(param.grad)
print(x.grad)

x.grad.zero_()
param.grad.zero_()

# 2.
output = x * param
loss = F.mse_loss(output, x.clone())
loss.backward()
print(param.grad)
print(x.grad)

x.grad.zero_()
param.grad.zero_()

# 3.
output = x * param
loss = F.mse_loss(output, x.clone().detach())
loss.backward()
print(param.grad)
print(x.grad)

&gt; tensor([-3.3104])
tensor([0.6608])

&gt; tensor([-3.3104])
tensor([0.6608])

&gt; tensor([-3.3104])
tensor([-0.9747])
</code></pre>

vikigenius: <p>Thanks this answers my question. But since x is a leaf variable, it’s grad value serves no purpose right?</p>

ptrblck: <p>Usually you don’t need gradients in your input tensor.<br>
However there are some use cases, e.g. in adversarial training, where these gradients can be used for small perturbation of the input.</p>

doojin: <p>It’s an old article, but I’m asking because it’s similar to my situation!</p>
<p>Is the last case wrong?<br>
I’ve only seen the first case.</p>

ptrblck: <p>I wouldn’t claim it’s wrong as it depends on your use case.</p>
How to test model that has a sub model in it?

Ulwan_M_Rasyid: <p>So I have this repo,</p>
<details>
<summary>
link</summary>
<p><a href="https://github.com/eugenelet/Meta-rPPG" class="inline-onebox" rel="noopener nofollow ugc">GitHub - eugenelet/Meta-rPPG: Implementation of "Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-Learner"</a></p>
</details>
<p>that only gives me a sample of their dataset as pth but not a pre-trained model… I already analyzed what the input shape is and was able to recreate it with another dataset, but now I’m kinda stuck on how to test this model… any idea how to do it? any suggestions are welcomed!</p>
Focal Loss: how to compute alpha?

_stracii: <p>I am using the pytorch implementation of focal loss (<a href="https://pytorch.org/vision/main/generated/torchvision.ops.sigmoid_focal_loss.html" class="inline-onebox" rel="noopener nofollow ugc">sigmoid_focal_loss — Torchvision main documentation</a>), but I am not sure how to compute the alpha weight.</p>
<p>It says: “Weighting factor in range (0,1) to balance positive vs negative examples or -1 for ignore.” and the authors say: it can be set by inverse class frequency.</p>
<p>How do I compute the class frequency? I have [36413,  8291] samples for class 0 and 1, respectively.</p>
<p>Thank you.</p>
ONNX export error | tensors from F.gelu() and F.layer_norm stay in CUDA

mvalente: <p>Hi!</p>
<p>I already opened an issue on PyTorch’s github but thought that someone might have a solution in the meanwhile.  The issue is explained in this <a href="https://github.com/pytorch/pytorch/issues/81096" rel="noopener nofollow ugc">link</a>.</p>
<p>onnx.export | onnx | cuda | cpu | device</p>
Dataset overwrite is not possible when `persistent_workers=True`

eliaz: <p>Hi,</p>
<p>I have an issue when using <code>DataLoader</code>, and I am not sure if this is an expected behavior.</p>
<p>I use <code>torch==1.11.0</code></p>
<p>I am trying to update labels of a datasets at each epoch.<br>
I tried:</p>
<pre><code class="lang-auto">dataset.labels = new_labels
</code></pre>
<p>and:</p>
<pre><code class="lang-auto">dataloader.dataset.labels = new_labels
</code></pre>
<p>But it was not working. I found that the issue comes from the fact that I use <code>persistent_workers=True</code>. If I set <code>persistent_workers=False</code> this works but the training becomes too long.</p>
<p>Here is a quick way to reproduce the issue:</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset, DataLoader


class Foo(Dataset):

    def __init__(self, n=15):
        self.data = [1] * n

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        return self.data[idx]


def bar(persistent_workers=False):
    dts = Foo()
    loader = DataLoader(dts, batch_size=5, num_workers=5, persistent_workers=persistent_workers)

    for batch in loader:
        print(batch)

    dts.data = [0] * len(dts.data)

    print("Changing data")
    for batch in loader:
        print(batch)


bar(False)
print()
print("Now it is bugging")
print()
bar(True)
</code></pre>
<p>So both call have different output, with the first call having the expected behaviour :</p>
<pre><code class="lang-auto">tensor([1, 1, 1, 1, 1])
tensor([1, 1, 1, 1, 1])
tensor([1, 1, 1, 1, 1])
Changing data
tensor([0, 0, 0, 0, 0])
tensor([0, 0, 0, 0, 0])
tensor([0, 0, 0, 0, 0])

Now it is bugging

tensor([1, 1, 1, 1, 1])
tensor([1, 1, 1, 1, 1])
tensor([1, 1, 1, 1, 1])
Changing data
tensor([1, 1, 1, 1, 1])
tensor([1, 1, 1, 1, 1])
tensor([1, 1, 1, 1, 1])
</code></pre>
<p>Would there be a way to still overwrite the dataset while using <code>persistent_workers=True</code> ?</p>
<p>Thank you for any help,<br>
Elias</p>

ptrblck: <p>No, I don’t think that’s easily possible, as each worker would use a copy of the dataset so manipulations would not be reflected. You would either need to recreate the <code>DataLoader</code> (after manipulating the <code>Dataset</code>) or would need to use your first approach and avoid persistent workers.<br>
There might be a hacky way to use the <code>Dataset.__getitem__</code> method with the worker information, but I don’t know if this would work (and it doesn’t sound too clean to be honest).</p>

eliaz: <p>Hi, thank you for your answer!</p>
What is different?

doojin: <p>Hi,</p>
<p>What is different below two case? Is it the difference between being a leaf tensor or not? Or is there any other difference?</p>
<p>I don’t know what it is, but I think both of them can calculate gradient. (If not, let me know)</p>
<p><strong>case1)</strong></p>
<pre><code class="lang-auto">a = torch.randn(3,4,4, device='cuda', requires_grad=True)
</code></pre>
<p><strong>case2)</strong></p>
<pre><code class="lang-auto">a = torch.randn(3,4,4).cuda().requires_grad_(True)
</code></pre>
<p>Actually, I’m more curious about what kind of situation the tensor is after network learning.</p>
<p>Thank you for your time.</p>

ptrblck: <p>Both are leaf tensors as seen by <code>print(a.is_leaf)</code>.<br>
The difference is the creation. The first tensor will be created directly on the GPU with the desired flags, the second one will be sampled on the CPU, pushed to the GPU, and then its <code>requires_grad</code> attribute will be changed inplace.</p>
<aside class="quote no-group" data-username="doojin" data-post="1" data-topic="156156">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/9f8e36/48.png" class="avatar"> doojin:</div>
<blockquote>
<p>Actually, I’m more curious about what kind of situation the tensor is after network learning.</p>
</blockquote>
</aside>
<p>I don’t understand the question, as both tensors are equal besides their chain of operations during the creation.</p>

doojin: <p>Thank you very much for your reply.<br>
My last question is explained in detail as follows.<br>
When I input an image into the network. After the input passes through the network, the loss function is calculated. At this time, the question is whether the situation of the tensor used in the calculation corresponds to the above case.</p>

ptrblck: <p>No, the model output will not be a leaf tensor as it’s computed by the input as well as the parameters and will thus have a history (i.e. it’s attached to a computation graph).</p>

doojin: <p>I really want to thank you for your help!</p>

ptrblck: <p>Sure, happy to help. Let me know if you have more questions or my answers weren’t detailed enough (or didn’t fit your use case).</p>
Performing Convolution on a batch of images with respective weight from a batch of custom weight filter

Sheikh_Talha: <p>I want to perform a convolution on a batch of images with respective weight from a batch of weight filters for example:</p>
<p>images = torch.ones(8, 32, 160, 608) <span class="hashtag">#dimensions</span> are B,C,H,W#<br>
weights = torch.randn(8, 1, 32, 32) <span class="hashtag">#dimensions</span> are B,C,H,W#</p>
<p>out=[]<br>
for i in range(images.shape[0]):<br>
out.append( torch.nn.conv2d(images[i] , weights[i]) ) <span class="hashtag">#I</span> want to use a single filter for all the 32 channels of input image to get an output like (1,1,<em>,</em>)#</p>
<p>out = torch.stack(out)</p>
<p><span class="hashtag">#required</span> output dimension should be like out.shape = (8,1,<em>,</em>)</p>

ptrblck: <p>The current <code>weight</code> shape is using <code>[out_channels=8, in_channels=1, height=32, width=32]</code> which would create 8 output channels (activation maps) for an input with a single channel.<br>
Your current input has 32 channels so I’m unsure how these weights should be applied.<br>
Would you like to create a separate conv kernel for each sample in the input batch <em>and</em> each channel?</p>

Sheikh_Talha: <p>I have a batch size of 8 for input as well as weights (dim=0). So for each batch I want to use the respective weight of size (1,1,32,32) and convolve it with the respective input image of size (1,32,160,608). And for performing colvoution I want to use the single channel weight to be convolved with all the 32 channels of the input image and then sum all the resultants into a single channel output.</p>
Calling copy.deepcopy on FSDP raised error

Xiangfan_Li: <p>Hi,</p>
<p>When I tried to call copy.deepcopy on a torch.distributed.fsdp.FullyShardedDataParallel model wrapper.<br>
The following error was raised:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/ema_pytorch/ema_pytorch.py", line 59, in __init__
    self.ema_model = copy.deepcopy(model)
  File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.8/copy.py", line 296, in _reconstruct
    value = deepcopy(value, memo)
  File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.8/copy.py", line 296, in _reconstruct
    value = deepcopy(value, memo)
  File "/usr/lib/python3.8/copy.py", line 153, in deepcopy
    y = copier(memo)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/parameter.py", line 55, in __deepcopy__
    result = type(self)(self.data.clone(memory_format=torch.preserve_format), self.requires_grad)
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/fsdp/flatten_params_wrapper.py", line 112, in __new__
    raise ValueError("An non-empty list or tuple argument is needed")
ValueError: An non-empty list or tuple argument is needed
</code></pre>
<p><code>Parameter.__deepcopy__</code> will call  <code>fsdp.FlatParameter.__new__</code> when being deep copied, however <code>self.data.clone()</code>(which is a Tensor) is passed to <code>fsdp.FlatParameter.__new__</code> which expects a list or tuple as parameter.</p>
<p>Seems that <code>Parameter.__deepcopy__</code> was not implemented correctly to copy <code>fsdp.FlatParameter</code> ?<br>
Or if <code>torch.distributed.fsdp.FullyShardedDataParallel</code> should not be deep copied anyway?</p>
What is the difference between output and h_n in LSTM?

matic_lauko: <p>I’m new to deep learning and have gone into learning LSTMs. But, reading the code, I have just found out that LSTMs return output and h_n, but, from as far as I knew h_n is the output:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc9adb0b5dd85cd7aad8ee72777009126091d88a.png" alt="image" data-base62-sha1="tc1bcr6WbtlxeemZq6zbCDOsjbQ" width="226" height="300"><br>
What am I missing?<br>
Thanks<br>
Image source: <a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/" class="inline-onebox" rel="noopener nofollow ugc">Understanding LSTM Networks -- colah's blog</a></p>

IneedMrmeeseeks: <p>I also recently studying LSTM. Here is <a href="https://wandb.ai/authors/rnn-viz/reports/Under-the-Hood-of-LSTMs--VmlldzoyNjY0MTg" rel="noopener nofollow ugc">the full picture of LSTM!</a><br></p>
<p>So, The output of LSTM is just some random weight matrix multiply with hidden state h_n.</p>
<p>I hope this help</p>
Debug on process 3 terminated with signal SIGTERM

Bruce_Muller: <p>Trying to train using ddp on 4 GPUs but I’m getting a: process 3 terminated with signal SIGTERM<br>
Which happens most the way through validation for some reason. Does anyone have any idea why this might happen or how I can debug it easier?</p>
<blockquote>
<p>File “train_gpu.py”, line 210, in <br>
main_local(hparam_trial)<br>
File “train_gpu.py”, line 103, in main_local<br>
trainer.fit(model)<br>
File “/scratch/staff/brm512/anaconda3/envs/ln1/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 343, in fit<br>
mp.spawn(self.ddp_train, nprocs=self.num_gpus, args=(model,))<br>
File “/scratch/staff/brm512/anaconda3/envs/ln1/lib/python3.7/site-packages/torch/multiprocessing/spawn.py”, line 171, in spawn<br>
while not spawn_context.join():<br>
File “/scratch/staff/brm512/anaconda3/envs/ln1/lib/python3.7/site-packages/torch/multiprocessing/spawn.py”, line 107, in join<br>
(error_index, name)<br>
Exception: process 3 terminated with signal SIGTERM</p>
</blockquote>

ptrblck: <p>Is the validation loop running correctly on a single device?<br>
Usually the error messages might be better when disabling multi-GPU runs and multiprocessing.</p>

oneriver: <p>Hi, Bruce_muller:<br>
Was your problem solved?<br>
I have the same issue: 4 GPUs DDP training and terminated with signal SIGTERM.</p>
Parameters of the network

zecheng_zhang1: <p>Hi,</p>
<p>I have a question about getting all parameters of the network. My network is defined as follow:</p>
<pre><code class="lang-auto">activation = nn.ReLU()
class OneInputBasis(nn.Module):
    def __init__(self):
        super().__init__()
        
        bo_b = True
        bo_last = False

        self.l1 = nn.Linear(200, 100, bias = bo_b).to(device)
        self.l4 = nn.Linear(100, 100, bias = bo_last).to(device)
        
    def forward(self, v):
        v = activation ( self.l1(v) )
        v = ( self.l4(v) )        

        return v
</code></pre>
<p>and</p>
<pre><code class="lang-auto">class node(nn.Module):
    def __init__(self):
        super().__init__()
        
        bo_b = True
        bo_last = False
        
        self.set_lay = []
        
        for jj in range(dim_output_space_basis):
            self.set_lay.append(OneInputBasis())
        
        
    def forward(self, v):

        w = self.set_lay[0](v)

        for ii in range(dim_output_space_basis-1):
            w = torch.cat((w, self.set_lay[ii+1](v)), dim = 1 )
        
        return w
</code></pre>
<p>and</p>
<pre><code class="lang-auto">class mesh(nn.Module):
    def __init__(self):
        super().__init__()
        
        bo_b = True
        bo_last = False

        self.l3 = nn.Linear(2, 100, bias = bo_b).to(device)
        self.l4 = nn.Linear(100, 100, bias = bo_b).to(device)
        self.l7 = nn.Linear(100, 10, bias = bo_last).to(device)
        
    def forward(self, w):
        w = activation ( self.l3(w) )
        w = activation ( self.l4(w) )
        w =  ( self.l7(w) )
        
        return w
</code></pre>
<p>finally, I have</p>
<pre><code class="lang-auto">activation = nn.ReLU()
class Test(nn.Module):
    def __init__(self):
        super().__init__()
        
        bo_b = True
        bo_last = False

        self.top = node()
        self.bottom = mesh()

    def forward(self, v, w, y):
        v = self.top(v)
        w = self.bottom(w)
        e = torch.bmm(w ,torch.bmm(v, y))

        return e[:, :, 0]
</code></pre>
<p>Now I define the network:</p>
<pre><code class="lang-auto">fnn_adam = Test()
</code></pre>
<p>When I print the parameters of the network, as</p>
<pre><code class="lang-auto"> for p in fnn_adam.parameters():
     print(p)
</code></pre>
<p>I can only see the parameters associated with fnn_adam.bottom, how can I print out the parameters associated with fnn_adam.top which is defined by the Class node()? Are the parameters associated with .top trainable? Thank you so much!</p>

ptrblck: <p>You are not properly registering the submodules (and thus their parameters) in <code>node</code>.<br>
Using a plain Python <code>list</code> won’t work:</p>
<pre><code class="lang-python">        self.set_lay = []
        
        for jj in range(dim_output_space_basis):
            self.set_lay.append(OneInputBasis())
</code></pre>
<p>and you would need to use an <code>nn.ModuleList</code> instead.</p>
Non_blocking in pytorch

shrutishrestha: <p>non_blocking=True<br>
The pytorch document says that "GPU copies are much faster when they originate from pinned method, that returns a copy of the object, with data put in a pinned region.<br>
Also, once you pin a tensor or storage, you can use asynchronous GPU copies. Just pass an additional <code>non_blocking=True</code> argument to a [<code>to()</code>] used to overlap data transfers with computation.</p>
<p>So this means the time should be faster. Instead, it taking more time. After placing non_blocking=True, code takes 1 hour+ to run whereas non putting this takes 11 mins</p>
<p>Link<a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-nn-ddp-instead:~:text=Host%20to%20GPU%20copies%20are%20much%20faster%20when%20they%20originate%20from%20pinned%20(page%2Dlocked)%20memory.%20CPU%20tensors%20and%20storages%20expose%20a%20pin_memory()%20method%2C%20that%20returns%20a%20copy%20of%20the%20object%2C%20with%20data%20put%20in%20a%20pinned%20region" rel="noopener nofollow ugc">:</a></p>

ptrblck: <p>Non-blocking copies would require page-locked memory, which won’t be available to the system anymore. If you are decreasing the host RAM too much, your OS might suffer and start to use the swap (which could explain the drastic slowdown). Could you check if that’s the case in your setup?</p>
Pipeline for working with tabular (CSV) data

maxrivera: <p>I’d like to train on a tabular dataset (CSV), but I’m not sure the best way to turn the pandas dataframe into a PyTorch dataset. With image datasets, I simply use torchvision.datasets.ImageFolder to create a PyTorch dataset directly from my data directory. Then I can use torch.utils.data.random_split to split into train, validation, and test sets. I would like to follow a similar workflow for CSV files, but all the tutorials I’ve seen use Scikit-learn to split the data first and apply normalization, then create a custom PyTorch dataset class… why isn’t there a way to do this without scikit-learn or custom dataset classes, similar to the way I was working with images?</p>
TypeError when passing list of tensors to C++

prockenschaub: <p>I am trying to pass a TensorList to a C++ extension, as done for example in <a href="https://github.com/pytorch/pytorch/blob/bfb39e577c98a4e137c1e38029ac042358a3609f/aten/src/ATen/native/PackedSequence.cpp#L188" rel="noopener nofollow ugc">torch.nn.utils.rnn.pad_sequence</a>. I am new to extending pytorch and have been following <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">this tutorial</a>.</p>
<p>A very basic function to test if I can pass a list of tensors could be</p>
<pre><code class="lang-cpp">#include &lt;torch/extension.h&gt;
#include &lt;pybind11/pybind11.h&gt;

bool test_tensorlist(torch::TensorList x) {
  return true
}

PYBIND11_MODULE(tensorlist, m) {
    m.doc() = "Test how to pass tensor list to C++ extension.";
    m.def("test_tensorlist", &amp;test_tensorlist, "Pass list of tensors (as torch::TensorList).")
}
</code></pre>
<p>Unfortunately, attempting to call this in python results in</p>
<pre><code class="lang-auto">TypeError: incompatible function arguments. The following argument types are supported: 
   1. (arg0: c10::ArrayRef&lt;at::Tensor&gt;) -&gt; bool  
Invoked with: [tensor([1]), tensor([2])]
</code></pre>
<p>I get the exact same result if I use <code>at::TensorList</code> or <code>c10::ArrayRef&lt;at::Tensor&gt;</code>. Passing a single tensor via <code>torch::Tensor</code> or a list of integers via <code>c10::ArrayRef&lt;int64_t&gt; </code>works like charm, though.</p>
<p>Any ideas what I might be doing wrong?</p>
<p>Edits: formatting</p>

juanpaez22: <p>I’m having the same issue. Maybe the pybind binding for TensorList is not implemented yet. I opened an issue on GitHub here: <a href="https://github.com/pytorch/pytorch/issues/80979" rel="noopener nofollow ugc">TensorList Python Binding Encountering C++ TypeError · Issue #80979 · pytorch/pytorch (github.com)</a></p>
<p>As a workaround, I was able to use the TORCH_LIBRARY bindings successfully ( <a href="https://github.com/pytorch/pytorch/blob/38350acf8fddc350c893e23a8e77bed2e59f5b06/torch/library.h" rel="noopener nofollow ugc">pytorch/library.h at 38350acf8fddc350c893e23a8e77bed2e59f5b06 · pytorch/pytorch (github.com)</a>). Here is a tutorial that helped: <a href="https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html" rel="noopener nofollow ugc">Extending TorchScript with Custom C++ Operators — PyTorch Tutorials 1.12.0+cu102 documentation</a></p>
<p>The basic function you listed above works with this approach. The downside is that there is now the extra step of calling torch.ops.load_library at runtime</p>
Why doesn't Pytorch automatically initialize nn.Parameter?

Vedant_Roy: <p>I have learned positional embeddings in my transformer with the following:</p>
<pre><code class="lang-python">        self.positional_embedding = nn.parameter.Parameter(
            torch.zeros((sequence_len, d_model)), requires_grad=True
        )
</code></pre>
<p>For embeddings, linear layers, and convolutions it seems like Pytorch (well, I’m using Pytorch Lightning), automatically initializes the tensors with random values.</p>
<p>But, for this positional embedding Pytorch does not initialize the tensor with random values. (I.e, if I call <code>print(self.positional_embedding)</code> inside of the forward method, the output will be all 0s.)</p>
<p>Does this mean I did something wrong / this embedding will not automatically update during backprop, or is everything still fine and this is just a special case where I need to manually initialize the embedding.</p>

ptrblck: <p>The <code>nn.Parameter</code> class does not initialize the internal tensor and will use its values directly. Modules are implementing a <code>reset_parameters</code> function to initialize all parameters as seen in e.g. <a href="https://github.com/pytorch/pytorch/blob/2beb57a823f6bbe1824371291686727aab71815d/torch/nn/modules/linear.py#L90-L111"><code>linear.py</code></a>.<br>
In your use case you are explicitly initializing the <code>positional_embedding</code> parameter with <code>torch.zeros</code> so it’s expected to see zeros afterwards. I’m not familiar with your use case so don’t know if a random initialization would perform better.</p>

Vedant_Roy: <p>Got it, so the values in a <code>nn.parameter.Parameter</code> will update during back propogation. They just won’t be initialized automatically, which is fine by me, since on the next line I initialize the values using <code>kaiming_he</code>.</p>
Create torch.Tensor from Optional[torch.Tensor] in jit.script

kvr777: <p>Hello, I have an optional torch.Tensor that should be processed with standard functional functions (e.g. torch.nn.functional.interpolate). When I try to write condition like: if tensor is not None: x = F.interpolate(x), I’ve got error like “Expected a value of type ‘Tensor’ for argument ‘input’ but instead found type ‘Optional[Tensor]’”. When I try to do something like new_x: torch.Tensor = x.clone() , I’ve got the error like “‘Optional[Tensor]’ object has no attribute or method ‘clone’” . How can I cast optional tensor to tensor when I’m sure it is not Null?</p>
Pytorch developer within Pl@ntNet (open-source project)

ajoly_PlantNet: <p>The team developing the Pl@ntNet project (<a href="http://plantnet.org" rel="noopener nofollow ugc">plantnet.org</a>) is looking for a post doc/data scientist with strong skills in python and pytorch development.<br>
Pl@ntNet is a citizen observatory of plant biodiversity and a research platform at the crossroads of data science, ecology and artificial intelligence. The Pl@ntNet mobile application is used by tens of millions of users worldwide.</p>
<p>You will work mainly in the Montpellier Computer Science Laboratory (LIRMM) as a research engineer at Inria. Inria is a major player in computer science research worldwide; it is the managing organization of the Pl@ntNet consortium.<br>
You will work in a quite unique environment at the forefront of digital technologies for biodiversity management. You will interact regularly with the engineers and developers of the Pl@ntNet platform as well as with the researchers and ecologists in charge of its management.</p>
<p>More information here:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://jobs.inria.fr/public/classic/en/offres/2022-05126">
  <header class="source">
      <img src="https://jobs.inria.fr/favicon-16x16.png" class="site-icon" width="16" height="16">

      <a href="https://jobs.inria.fr/public/classic/en/offres/2022-05126" target="_blank" rel="noopener nofollow ugc">Inria</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://jobs.inria.fr/public/classic/en/offres/2022-05126" target="_blank" rel="noopener nofollow ugc">Pytorch developer / Data scientist within Pl@ntNet team</a></h3>

  <p>Offre d'emploi Inria</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>