akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
ZeroRedundancyOptimizer consolidate_state_dict warning

kleingeo: <p>I have been trying to minimize my memory footprint while training with DDP and came across <code>ZeroRedundancyOptimizer</code>. I implemented it similarly to that from the example, by simply wrapping an <code>AdamW</code> class into it. This is the (rough) example of the training loop I am using that is called from <code>torch.multiprocessing.spawn</code>.</p>
<pre><code class="lang-python">device = f'cuda:{gpu}'
torch.cuda.set_device(gpu)  # From 0 to N-1 for the number of GPUs

net = net.to(device)
net = torch.nn.parallel.DistributedDataParallel(net, device_ids=[rank],
                                                gradient_as_bucket_view=True,
                                                find_unused_parameters=True)
scaler = GradScaler()
optimizer = ZeroRedundancyOptimizer(net.parameters(), 
                                    optim=torch.optim.AdamW, lr=learning_rate,)

for sample, ground_truth in train_loader:
    optimizer.zero_grad(set_to_none=True)
    
    with autocast(): 
        pred = net(sample)
        loss = loss_fn(pred, ground_truth)
   
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

</code></pre>
<p>However, I am getting the error <code>Optimizer state has not been consolidated. Returning the local state Please call "consolidate_state_dict()" beforehand if you meant to save the global state</code>.</p>
<p>Am I doing something wrong with this implementation?</p>
<p>I am running <code>torch == 1.8.1</code> and <code>cudatoolkit==10.2</code></p>

wanchaol: <p>cc <a class="mention" href="/u/mrshenli">@mrshenli</a> <a class="mention" href="/u/rvarm1">@rvarm1</a> if you have any insights on what might be wrong there?</p>

rvarm1: <p>I’m guessing that some part of the autocast/grad scaling is attempting to call <code>optimizer.state_dict</code> but ZeRo requires state dict to be consolidated first. This seems like it might be a limitation in terms of autocast and ZeRo working together.</p>
<p>@ <a href="https://discuss.pytorch.org/u/kleingeo">kleingeo</a> Could you please file an issue with a full repro to the PyTorch GH repo? Thanks!</p>

kleingeo: <p>I found the error. It was stemming from the fact that I was saving the optimizer after each epoch without calling <code>optimizer.consolidate_state_dict()</code> first. After I included that before saving the warning went away.</p>
Issue with DataParallel and module returning dict in forward

Thomas_Ricatte: <p>Hello,</p>
<p>I am trying to use DataParallel to learn my model on multiple GPUs. The forward function of my model is returning a dict() with two keys “predicted_tags” and “likelihood” (it’s basically an NER model) and I thought this use case was supported but I am getting the following error</p>
<pre><code class="lang-auto">  File "/mnt/nfs/home/foo/myenv/lib64/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 163, in forward
    return self.gather(outputs, self.output_device)
  File "/mnt/nfs/home/foo/myenv/lib64/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 175, in gather
    return gather(outputs, output_device, dim=self.dim)
  File "/mnt/nfs/home/foo/myenv/lib64/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 73, in gather
    res = gather_map(outputs)
  File "/mnt/nfs/home/foo/myenv/lib64/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 68, in gather_map
    raise(e)
  File "/mnt/nfs/home/foo/myenv/lib64/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 64, in gather_map
    for k in out))
  File "/mnt/nfs/home/foo/myenv/lib64/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 64, in &lt;genexpr&gt;
    for k in out))
  File "/mnt/nfs/home/foo/myenv/lib64/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 68, in gather_map
    raise(e)
  File "/mnt/nfs/home/foo/myenv/lib64/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 65, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/mnt/nfs/home/foo/myenv/lib64/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 68, in gather_map
    raise(e)
  File "/mnt/nfs/home/foo/myenv/lib64/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 65, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/mnt/nfs/home/foo/myenv/lib64/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 68, in gather_map
    raise(e)
  File "/mnt/nfs/home/foo/myenv/lib64/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 65, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
TypeError: zip argument #1 must support iteration
</code></pre>
<p>Note that I am not getting this error if I don’t set the value of predicted_tags and return for instance</p>
<pre><code class="lang-auto">{ predicted_tags: None, loss: sth }
</code></pre>
<p>Do you have any idea ? I added some torch.save by looking at the stack trace and all that I can say is that in <code>data_parallel.py</code>, before running <code>return self.gather(outputs, self.output_device)</code> outputs is still the correct dict, e.g.</p>
<pre><code class="lang-auto">[
{'predicted_tags': [[0, 0, 0], [0, 0], [0, 0, 0, 0]], 'loss': 0},
{'predicted_tags': [[0, 0, 0, 0, 0]], 'loss': 0}
]
</code></pre>
<p>but later in the process just before the final exception, outputs is just <code>(0, 0)</code>. Hope that these details help.</p>

samin_hamidi: <p>I am facing the same issue. Have you been able to solve the problem?</p>
UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()

talhaanwarch: <p>When using mix precison, i am getting this warning. Without mix precision, there is no such warning</p>
<pre><code class="lang-auto">UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()
</code></pre>
<pre><code class="lang-auto">scaler = torch.cuda.amp.GradScaler()
with experiment.train():
for batch_idx, _data in enumerate(train_loader):
    image, labels= _data 
    image, labels = image.to(device), labels.to(device)
    optimizer.zero_grad()
    with torch.cuda.amp.autocast():
        output = model(image)
        loss = criterion(output, labels)
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()
    scheduler.step()
</code></pre>
<p>This is how i defined the learning rate</p>
<pre><code class="lang-auto">
scheduler = optim.lr_scheduler.OneCycleLR(optimizer, max_lr=hparams['learning_rate'], 
             steps_per_epoch=int(len(train_loader)),
              epochs=hparams['epochs'],
               anneal_strategy='linear')
</code></pre>
<p>Here is warning i am getting</p>
<pre><code class="lang-auto">/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:1565: UserWarning: To get the last learning rate computed by the scheduler, please use `get_last_lr()`.
  "please use `get_last_lr()`.", UserWarning)
/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate scheduler
</code></pre>

my3bikaht: <p>Look at this one:</p><aside class="quote quote-modified" data-post="1" data-topic="92930">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gil_fernandes/48/27322_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/optimizer-step-before-lr-scheduler-step-error-using-gradscaler/92930">`optimizer.step()` before `lr_scheduler.step()` error using GradScaler</a> 
  </div>
  <blockquote>
    Even though I think my code calls the optimizer.step via Gradscaler function before the lr_scheduler.step() function I am still getting this warning: 

/opt/anaconda3/envs/huggingface/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:123: UserWarning: Detected call of lr_scheduler.step() before optimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step() before lr_scheduler.step().  Failure to do this will result in PyTorch skipping the first v…
  </blockquote>
</aside>
Layers in BERT problem

Tim5: <p>i have a problem in BERT with getting the meaning of encoded_layers[11] while I got the first element is the hidden state of the last layer of the Bert model got by <code>encoded_layers = outputs[0]</code>, so what does those mean</p>
<pre><code class="lang-auto">layers,_ = BERTModel(tokens_tensor)
embedding = layers[11].squeeze(0)
</code></pre>
Torch.index_put on GPU with accumulate=True

oneilllml: <p>I’ve seen two posts about index_put on GPU: <a href="https://github.com/pytorch/pytorch/issues/39227" rel="noopener nofollow ugc">here</a> and <a href="https://discuss.pytorch.org/t/torch-index-put-on-gpu/140211">here</a>. They are both marked as solved, but it’s not working for me. It’s fine on the CPU, and on GPU with accumulate=False, but when accumulate=True I get</p>
<pre><code class="lang-auto">RuntimeError: linearIndex.numel()*sliceSize*nElemBefore == value.numel()INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/Indexing.cu":250, please report a bug to PyTorch. number of flattened indices did not match number of elements in the value tensor251
</code></pre>
<p>Steps to reproduce:</p>
<pre><code class="lang-auto">t1 = torch.zeros(10).long().cuda()
t2 = torch.randint(10, size=(25,)).cuda()
t1.index_put(indices=[t2], values=torch.tensor(1), accumulate=True)
</code></pre>
<p>Pytorch version: 1.10.1</p>

ptrblck: <p>Could you update to the latest nightly as I cannot reproduce the issue, please?</p>

oneilllml: <p>Hi, yes I just tried and it works fine on nightly</p>

anantguptadbl: <pre><code class="lang-auto">t1 = torch.zeros(10).cuda()
t2 = torch.randint(10, size=(25,)).cuda()
print("T2 is {0}".format(t2))
t1.index_put(indices=[t2], values=torch.ones(25).type(torch.FloatTensor).cuda(), accumulate=True)
</code></pre>

oneilllml: <p>The use case is that I want to count the number of occurrences of each number from 1 to 10 in the size 25 tensor <code>t2</code>. It’s a bit counterintuitive to have to allocate a new tensor of 25 ones to do that, but this does work, so thanks.</p>
Avoiding loop for sequential update on tensors

Apocalypto: <p>I have a tensor <code>a</code> with shape (100,140,140) which is actually a batch (size 100) of matrices of size (140x140). You can assume these matrices are actually vectors with 140x140 = 19600 components.</p>
<p>I also have a function <code>myfunc(a, b, c, d)</code> where <em>b, c, d</em> are some constant tensors. <code>myfunc()</code> outputs a batch of scalers. Now I want to find the gradient of <code>myfunc(a, b, c, d)</code> with respect to the vectors in tensor <code>a</code> using <em>Finite Difference Method</em>:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50292d6c8b01bba039594b45c97e3a0feb97f626.png" alt="image" data-base62-sha1="br8q23gR1WD9IhPjYl9ABl0SjAO" width="280" height="137"></p>
<p>What I am trying to do right now is something like this:</p>
<pre><code class="lang-auto">grad_a = torch.empty_like(a)

for i in range(a.shape[1]):
        for j in range(a.shape[2]):
              a[:,i,j] += h    #h is the step size
              out1 = myfunc(a, b, c, d)
              a[:,i,j] -= 2*h
              out2 = myfunc(a, b, c, d)
              grad_a[:,i,j] = (out1 - out2)/(2*h)
              a[:,i,j] += h    #resetting the value to original 
            
</code></pre>
<p>I am fairly new to pytorch and don’t really know how to do these avoiding a loop. For loop is really slow on this and may not even finish on my system with larger tensors. Is there any torch functionality to do these efficiently without a loop? I have found this <a href="https://wesselb.github.io/fdm/docs/_build/html/readme.html#multivariate-derivatives" rel="noopener nofollow ugc">FDM Library</a> but I don’t know how to use the gradient method from that library on functions with more than 1 parameter.</p>
<p>Any help is appreciated. Thank you!</p>
Faster RCNN with resnet101 FPN

Nathane_Berrebi: <p>Hello,<br>
I trained a pretrained faster RCNN with resnet50 FPN and I would try to do the same thing with a resnet101.</p>
<ol>
<li>
<p>Is there a way to train a pretrained faster RCNN with resnet101 FPN as backbone ?</p>
</li>
<li>
<p>And ideally to change the anchor boxes sizes (with a pretrained mode on resnet 101 or resnet 50)? Because I know that my objects are supposed to have always the same shape.</p>
</li>
</ol>
<p>Thank you very much</p>

einsteinxx: <p>Would this help, for ResNet50?</p>
<pre><code class="lang-auto">anchor_sizes = ((32,), (64,), (128,), (256,), (512,)) 
aspect_ratios = ((0.5, 0.75, 1.0),) * len(anchor_sizes) 
anchor_generator = AnchorGenerator(anchor_sizes, aspect_ratios)
# get the number of input features 
in_features = modelr.roi_heads.box_predictor.cls_score.in_features
# define a new head for the detector with required number of classes
modelr.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes) 
modelr.rpn.anchor_generator = anchor_generator #adding in customized anchors
</code></pre>
Correct labeling for two class Resnet50 FasterRCNN setup

einsteinxx: <p>I am using Resnet50 as the backbone for a faster-rcnn setup using ultrasound images. I have done initial training (after downloading the default weights) with the BUSI ultrasound set and now have another set to further refine the model.  My data is composed of two classes of ultrasound images (benign and malignant), where each is composed of various frames from US video. Only some of the frames have annotated bounding boxes.<br>
I’m using two classes, one for normal (any image that does not have an annotation) and class 2 for anything with annotations (even benign with annotations). To get the label 1 classes through, I’ve put them as</p>
<p>Loading the default model and snippet for setting the background dataloader values:</p>
<pre><code class="lang-auto">#model load

modelr=torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False, 
                                                            num_classes=2,
                                                            pretrained_backbone=True)
#############
#Assigning data
            #for a Normal (label 0)
            boxes = torch.zeros((0, 4), dtype=torch.float32)
            area = nr*nc #number of image rows x columns #using full image size for area
            boxes = torch.as_tensor(boxes, dtype=torch.double)
            area = torch.as_tensor(area, dtype=torch.double)

            label = torch.zeros((1,), dtype=torch.int64)

            iscrowd=torch.ones((1,), dtype=torch.int64)
</code></pre>
<p>Since I have two classes and each test image is from a series of images that may or may not have a small selection with ground truth, is it correct to assign normal to “background” and not normal to label 1 (does num classes of 2 signify background and another label?)? Inputting images without annotations in the fasterRCNN format required some trickery (area size, iscrowd), which I’m not sure is entirely correct.<br>
I get decent IOU with one dataset, but I’ve got a suspicion I have a core setup issue somewhere. Occasionally, I will get a test output that returns an empty value for predicted bounding box, label, and score. Is this normal for a dual label FasterRCNN setup?  Having a series of images all taken from one video with one overall label and only some of those frames having a truth annotation is a bit confusing.</p>
Wrongly placed `model.train()` but training/validation losses/metrics all improving

jasperhyp: <p>Hi,  I accidentally forgot to put <code>model.train()</code> within each epoch, and yet in each epoch my evaluation function calls <code>model.eval()</code>.  Weirdly, the train &amp; evaluation losses and metrics are all improving as training proceeds (which makes me omit this issue for a month), but when I extract the final node embeddings (I’m training a GNN) they are definitely not the well-trained one.  Of course, everything is back to normal when I put <code>model.train()</code> back to the right place.  Am a little bit confused.</p>
<p>Demo code:</p>
<p><code>main</code> function</p>
<pre><code class="lang-python"># main()

model.train()  # this is put in the wrong place!

for i in range(num_epoch):
    train(model, optim, ...)

edge_all_loader, node_embeddings = generate_train_batch(xxx)

with torch.no_grad():  # generate final embeddings for all nodes
    model.eval()
    out_all = model(edge_all_loader, node_embeddings, xxx)
</code></pre>
<p>while the <code>train</code> function looks like:</p>
<pre><code class="lang-python">edge_train_loader, node_embeddings = generate_train_batch(xxx)
edge_val_loader, node_embeddings = generate_val_batch(xxx)  # the two initial node embeddings are the same

# model.train()  # It should've been here

node_embeddings_train = run_train(model, edge_train_loader, node_embeddings)  # another function to run batch training, where the loss is backproped and the model is (thought to be) updated

with torch.no_grad():
    model.eval()
    out_val = model(edge_val_loader, node_embeddings, xxx)

xxx
</code></pre>

ptrblck: <p><code>model.train()</code> will change the behavior of some layers, such as enabling dropout and using the batch stats in batchnorm layers while also updating the running stats. Depending on the model architecture and which layers are used, <code>model.train()/.eval()</code> might or might not have any effect (you should still use it in case you’ll add e.g. dropout layers later).</p>

jasperhyp: <p>Thanks Piotr, you reminded me that <code>dropout</code> should have been causing this weird behavior.  But it is still quite intriguing to see the link prediction task going so well without <code>dropout</code> while the embeddings are updated so badly.  Such an interesting behavior…</p>
How do you update batchnorm statistics of a SWA model when using DDP?

nsriniva03: <p>Once training is done and SWA model is computed, how do you update the batchnorm statistics when using DDP ?</p>
<p>Do you still just use, torch.optim.swa_utils.update_bn(train_loader, swa_model) and the parameters are updated across different ranks?</p>

H-Huang: <p>Since each rank contains a copy of the model I believe you just have to call <code>update_bn</code> on each rank as you would for a single model and it should work.</p>
Torch AutoGrad behaviour after torch.detach()

mrityu: <p>Hey,</p>
<p>Please refer to two different code snippets.</p>
<pre><code class="lang-auto">Y_prob = model(X) #Getting the output from the model
loss = criterion(Y_true, Y_prob) #Some arbitrary loss function used
.
.
.
loss.backward()
</code></pre>
<p>Snippet 2:</p>
<pre><code class="lang-auto">Y_prob = model(X) #Getting the output from the model
loss = criterion(Y_true, Y_prob) #Some arbitrary loss function used
Y_prob =  Y_prob.detach() #detaching the output from the graph
.
.
.
loss.backward()
</code></pre>
<p>I expected that the first snippet would obviously be error-free. Still, snippet 2 should have thrown an error as I am removing the model output from the graph itself, which will throw an error when I will call <code>loss.backward()</code> which will call the <code>backward()</code> of <code>criterion</code> and which will in-turn call for the backward of <code>Y_prob</code> but since <code>Y_prob</code> is already detached, it will throw an error as the <code>grad_fn</code> of the detached object will be None.</p>
<p>But surprisingly, both the snippets work well.  What is it that I am getting wrong here?</p>

JuanFMontesinos: <p>Basic python and even coding.<br>
You are overwritting variable called Y_prob which equals the older Y_prob but detached.<br>
Your computations were calculated with previous one.</p>
<p>it’s like if you do</p>
<pre><code class="lang-auto">x=5
print(f'id x: {id(x)}')
w=[x]
print(f'id w: {id(w[0])}')
x=6
print(f'id new x: {id(x)}')
</code></pre>
<pre><code class="lang-auto">id x: 10914624
id w: 10914624
id new x: 10914656
</code></pre>
<p>When you overwrite x, it no longer points to the same object.<br>
Analogously</p>
<pre><code class="lang-auto">x=tensor(5)
print(f'{X old: id(x)}')
x=x.detach()
print(f'{X new: id(x)}')
</code></pre>

mrityu: <p>I get that the computations were performed using the older tensor, my doubt is, is that the older object is not required again by the autograd during the backward computation?</p>

JuanFMontesinos: <p>Well it is.<br>
That you overwrite the doesn’t mean that the object disappears.<br>
The object is still there while it’s required.<br>
The corresponding memory will be freed once it’s no longer pointed by an alive variable in the environment.</p>
<pre><code class="lang-auto">import ctypes
a = "hello world"
a_id = id(a)
a=141
print( ctypes.cast(a_id, ctypes.py_object).value)
print(a)
</code></pre>
<pre><code class="lang-auto">hello world
141
</code></pre>
<p>In fact this is sometimes a source of bugs in pytorch when people drag tensors without detaching them.</p>
Converting tensors into numpy array and vise versa

Gaborandi: <p>i’m working on distilbert for text classification, and want to save the tensors into a data frame so i can use it again as feature generator<br>
to illustrate my question more:<br>
below are token IDs for one sentence in a row in csv file i’m working on :<br>
Token IDs: tensor([  101, 11113,  2080, 25876,  2542, 15009, 11290, 22291,  3370,  2478,<br>
7037, 22160,  3215,  1998,  5760,  5001, 10464, 24895, 15009,  2157,<br>
2002, 17585,  6593, 16940,  1037,  2553,  3189,  1026,  1013,  1055,<br>
1028,  1026,  1013,  1055,  1028, 11581,  2063,  2138,  1997,  1996,<br>
15843,  1997, 10181, 17843,  2542, 15009, 11290, 22291,  3370, 25510,<br>
7096,  2038,  2468,  1996,  2364,  7709,  2000,  7438,  5022,  2007,<br>
2203,  1011,  2754,   102])<br>
1- i want to replace the text in the csv file into their tokens by converting these tensors into np.array and save this CSV file<br>
2- then i need to know how to convert this column back from np.array to tensors (input ids and attention masks) so i can use them for fine tuning tasks</p>

ptrblck: <p>1 To convert a tensor to a numpy array use <code>a = tensor.numpy()</code>, replace the values, and store it via e.g. <code>np.save</code>.<br>
2. To convert a numpy array to a tensor use <code>tensor = torch.from_numpy(a)</code>.</p>
Applying Smote technique (class imbalance) in Pytorch?

RAFAIL_MAHAMMADLI: <p>Hello,</p>
<p>I want to apply Smote technique on my training data . I got following error. Can anyone asses me to solve this error.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e33d07d7bd2ab36f0ba0ca19e3ae8a78285b251.png" data-download-href="https://discuss.pytorch.org/uploads/default/2e33d07d7bd2ab36f0ba0ca19e3ae8a78285b251" title="Screenshot 2022-01-11 at 00.54.08"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e33d07d7bd2ab36f0ba0ca19e3ae8a78285b251_2_690x350.png" alt="Screenshot 2022-01-11 at 00.54.08" data-base62-sha1="6AIYtRSvUvnmvxS985OqOmY2mDn" width="690" height="350" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e33d07d7bd2ab36f0ba0ca19e3ae8a78285b251_2_690x350.png, https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e33d07d7bd2ab36f0ba0ca19e3ae8a78285b251.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e33d07d7bd2ab36f0ba0ca19e3ae8a78285b251.png 2x" data-dominant-color="313232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-01-11 at 00.54.08</span><span class="informations">1005×511 66.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thank you</p>
Performance issue of RTX 3070 compared to 2070 SUPER

YunusCC: <p>Hello,</p>
<p>I have a flask server running inference for several different models. Two identical computers that only differs in terms of GPU, one RTX3070 and the other 2070 SUPER. Both computers have i9-10900X and nvme2 SSDs and same RAM. Both computers are up to date with every library and pytorch 1.10.0 with 11.3 cuda and 8.2.0 cudnn. So in summary, they are exactly the same with everything else.</p>
<p>However, RTX 3070 performs significantly slower and uses much more VRAM (~7.6GB) than 2070SUPER (~6GB VRAM). 2070SUPER can handle %50 more load and still perform better.</p>
<p>Are there any known issues regarding the performance issues on 3070?</p>
<p>Note: I use Transformers and Attention networks which I heard should run faster on 30** series GPUs.<br>
Note2: I use half precision for CNN models as well. Can the issue be related with fp16?<br>
Note3: I regularly clear gpu cache release some memory, otherwise VRAM keeps bloating under heavy load.<br>
Note4: Flask runs multithreaded if that helps.<br>
Note5: This issue is impossible to reproduce due to complexity of the code.</p>

ptrblck: <p>I’m not sure what “Note 3” means and how you are releasing the memory, but in case you are clearing the cache this would hit the performance.<br>
Note 5 makes it quite hard to give you a valid answer. Would it be possible to get a model with the input shapes showing the performance difference?</p>

YunusCC: <p>If clearing the cache was hitting the performance, wouldn’t it decrease the performance on both of the computers? They are running the same code at the same time, and there is a huge performance difference both in terms of allocated VRAM which leads to reduced inference speed due to lack of gpu VRAM available.</p>
<p>I realized when the gpu VRAM is full while using Flask server, whenever a request is made that requires more space on VRAM, it queues the operation which leads to reduced inference speed.</p>
<p>So the issue is with RTX3070 allocating more VRAM compared to 2070 SUPER.</p>
Different weights in loss.backward() and optimizer.update()

Babi: <p>I want to use different weights in the backward pass and the update pass as</p>
<blockquote>
<p>w1.grad = dloss/dw1<br>
w(new) = <em><strong>w2</strong></em> - lr * w1.grad</p>
</blockquote>
<p>instead w(new) = <em><strong>w1</strong></em> - lr * w1.grad</p>
<p>How can I do that?</p>
Pass transformation to Cityscapes dataset

talhaanwarch: <p>Hi, greetings, i want to transform both image and mask. I tried this way</p>
<pre><code class="lang-auto">from torchvision.datasets import Cityscapes
from torchvision import transforms as T
aug=T.Compose([T.Resize((256,512))])
dataset = Cityscapes('../data/', split='val', mode='fine',
                     target_type='semantic',transforms=aug)
</code></pre>
<p>but it gives me following error</p>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_348578/797377544.py in &lt;module&gt;
      7                      target_type='semantic',transforms=aug)
      8 
----&gt; 9 out= dataset[2]
     10 print(out[0].size,out[1].size)
     11 #img,seg=augment(np.array(out[0]),np.array(out[1]))

~/venv/lib/python3.8/site-packages/torchvision/datasets/cityscapes.py in __getitem__(self, index)
    190 
    191         if self.transforms is not None:
--&gt; 192             image, target = self.transforms(image, target)
    193 
    194         return image, target

TypeError: __call__() takes 2 positional arguments but 3 were given
</code></pre>

L-Reichardt: <p>A little late, but maybe someone will read it. You could try removing the keyword <code>transforms=aug</code>. Instead try <code>Cityscapes(...., transform=aug, target_transform=aug)</code>. Note that there are <code>transform</code> and <code>transforms</code> as two different keywords.</p>
<p>If that doesnt work, you could try removing <code>aug=T.Compose(...)</code>, and instead create a class.</p>
<pre><code class="lang-auto">class CityTransform:
    def __call__(self, image, target):
        image = *your transformation*  (image)
        target = *your transformation* (target)
        return image, target
</code></pre>
<p>then insert it</p>
<pre><code class="lang-auto">dataset = Cityscapes('../data/', split='val', mode='fine',
                     target_type='semantic',transforms=CityTransform)
</code></pre>
Modifing gradients between backeard and optimization step procedure

hamedB: <p>Hello</p>
<p>I have seen a couple of previous topics regarding observing or modifying the gradients of non-leaf tensors. some of them are for the year 2018 or 2019 which suggest function .grad. However this function returns None. And through the autograd doc. I have seen much explanation about the required_grad function.<br>
My question is that is it possible to see the values of the gradients of the tensor in computational graph and is it possible to modify the gradients after loss.backward() and before optimizer.step() ?</p>
<p>Thanks.</p>

anantguptadbl: <p><a class="mention" href="/u/hamedb">@hamedB</a> you need to register hooks on the intermediate variables to get the grad</p>
<aside class="quote quote-modified" data-post="1" data-topic="141869">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/e19b73/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-calculate-get-intermediate-results-of-backpropagation/141869">How to calculate / get intermediate results of backpropagation</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    I’ve found this one example which illustrates my problem: 
import torch
from torch.autograd import grad
import torch.nn as nn


# Create some dummy data.
x = torch.ones(2, 2, requires_grad=True)
gt = torch.ones_like(x) * 16 - 0.5  # "ground-truths"

# We will use MSELoss as an example.
loss_fn = nn.MSELoss()

# Do some computations.
v = x + 2
y = v ** 2

# Compute loss.
loss = loss_fn(y, gt)

print(f'Loss: {loss}')

# Now compute gradients:
d_loss_dx = grad(outputs=loss, inputs=x)

d_loss_dv = g…
  </blockquote>
</aside>

<p><a href="https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html</a></p>

hamedB: <p>Thanks <a class="mention" href="/u/anantguptadbl">@anantguptadbl</a> for your reply. I am familiar with the hook. But I am not sure how to apply the modifications on the actual tensor. Hook can records the values on an external list. However I want after computing the gradients by loss.backward(), add some values to each gradients computed and then call optimizer.step()<br>
if you have any idea, I appreciate it.</p>
<p>Thanks</p>
Nested Models or Combining RNN and linear model

mephisto: <p>Hi,</p>
<p>I am trying to build a model that can both handel string and float values.</p>
<p>My plan is to turn the strings, character wise in to a one hot encoded tensors which then could be fed to a RNN. Afterwards that result would be merged with the other float values in a regular linear network.</p>
<p>But I seem to be getting a bit confused on how to implement it correctly.<br>
Looking forward to your advice.</p>
<p>This is what I have so far:</p>
<pre><code class="lang-auto">class NeuralNet(nn.Module):
    def __init__(self, input_size_int, input_size_str, hidden_size_rnn, hidden_size_ln):
        super(NeuralNet, self).__init__()

        self.num_layers = 5
        self.hidden_size_rnn =hidden_size_rnn
        self.rnns = nn.ModuleList()
        self.fully_connected = nn.ModuleList()

        self.h = []

        for i in range(input_size_str):
            self.rnns.append(nn.RNN(input_size=len(str_m.ALL_LETTERS) ,hidden_size=hidden_size_rnn, num_layers=self.num_layers, batch_first=True))
            self.fully_connected.append(nn.Linear(hidden_size_rnn, 1))

        self.input_size = input_size_int + input_size_str


        self.l1 = nn.Linear(input_size_int + input_size_str, hidden_size_ln)
        self.l2 = nn.Linear(hidden_size_ln, 2)  
    
    def forward(self, x1,  s):

        self.hidden = [None for _ in range(len(self.rnns))]
        x2 = []

        for idx, rnn in enumerate(self.rnns):
            self.hidden[idx] = torch.zeros(self.num_layers, s[idx].size()[0], self.hidden_size_rnn).to(device)

            out_rnn, _ = rnn(s[idx],self.hidden[idx])
            # out: batch_size, seq_legnth, hidden_size

            out_rnn = out_rnn[-1::]

            out_rnn = self.fully_connected[idx](out_rnn)
            x2.append(out_rnn.reshape(-1))

        # Merge X1 and X2
        x = torch.cat([x1, torch.cat(x2)])

        out = F.relu(self.l1(x))
        out = F.relu(self.l2(out))
        
        return out
</code></pre>
Dynamic vs static set of images

safin_salih: <p>I have a problem where I want to create a network that classify a set of images, however the number of images are not fixed, How do I come up with a solution such that the network deals with this dynamic behavior of set of inputs.</p>
<p>Thank you.</p>

JuanFMontesinos: <p>You can train for a large set and pad shorter ones (that’s the typical approach when working with sequences)<br>
Another idea is to compute N features for N images and then doing an average (which is somehow a good option)</p>

safin_salih: <p>average the features? Thank you for your help</p>

JuanFMontesinos: <p>Well, average pooling is nothing but an average and it works.<br>
Averaging will just generate a global descriptor for your set.<br>
Then you need just to classify that global descriptor.</p>
<p>Ofc this approach works for stable scenarios where each element in the set does a soft contribution.</p>
<p>So imagine you have a set which is {penguin, penguin, ostrich, chicken,…, }<br>
and you have 20 elements like these. You would classify it as “birds”.</p>
<p>If all of a sudden you add a human as  21st element, its features will be dissolved and prob still classified as birds. Although you may want to classify that as “biped”.</p>
<p>In case where single elements may change radically  the meaning of the set, average won’t work very well.</p>
How to check libtorch version

tholzmann: <p>Hi,</p>
<p>In Python I can get the PyTorch version string with:<br>
<code>print(torch.__version__)</code><br>
However, I did not figure out a way to do the same with the C++ libtorch interface?</p>
<p>Does someone know how to retrieve the libtorch version string using the C++ interface?</p>
<p>Thanks!<br>
Thomas</p>

Willem: <p>Was wondering about the same. Did you find out whether this is possible yet?</p>

bigcmos: <p>Did you figure out a way to do it?  I filed a feature request for this issue:</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/44365" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/44365" target="_blank" rel="nofollow noopener">Add ability to determine libtorch version from a c++ API or preprocessor macro.</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-09-09" data-time="06:08:10" data-timezone="UTC">06:08AM - 09 Sep 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/bigcmos" target="_blank" rel="nofollow noopener">
          <img alt="bigcmos" src="https://avatars1.githubusercontent.com/u/1166657?v=4" class="onebox-avatar-inline" width="20" height="20">
          bigcmos
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🚀 Feature
I want to be able to determine the version of libtorch in my c++ binary in cases where I don't...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Shisho_Sama: <p>Starting from 1.8, one can use <code>TORCH_VERSION_{MAJOR,MINOR,PATCH}</code> for this.</p>
<pre><code class="lang-cpp">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main() {
  std::cout &lt;&lt; "PyTorch version: "
    &lt;&lt; TORCH_VERSION_MAJOR &lt;&lt; "."
    &lt;&lt; TORCH_VERSION_MINOR &lt;&lt; "."
    &lt;&lt; TORCH_VERSION_PATCH &lt;&lt; std::endl;
}
</code></pre>
<p>Prints:<br>
<code>PyTorch version: 1.8.0</code></p>

Luke_Vassallo: <p>Is it possible to also report the CUDA version the library was compiled for?</p>
Get batch's datapoints across all GPUs

fermoren: <p>Hello,</p>
<p>I´m running my model (using pytorch lightning) in a cluster with multiples GPUs (2). My problem is that I would like to access all the datapoints in the batch. Because I´m using more than 2 GPUs, my batch in divided between those two devices for parallelisation purposes, which means than when I access the data in the batch in eval/training, I´m getting just half the batch.</p>
<p>How could I obtain the complete batch and the predictions of the model that are divided among different devices/GPUs?</p>
<p>Thanks!</p>

ptrblck: <aside class="quote no-group" data-username="fermoren" data-post="1" data-topic="142694">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/dbc845/48.png" class="avatar"> fermoren:</div>
<blockquote>
<p>Because I´m using more than 2 GPUs, my batch in divided between those two devices for parallelisation purposes, which means than when I access the data in the batch in eval/training, I´m getting just half the batch.</p>
</blockquote>
</aside>
<p>Are you using <code>nn.DataParallel</code> or <code>DistributedDataParallel</code>?<br>
In the former (not recommended) case the batch would be split while passing it to the <code>DataParallel</code> model, so inside the <code>forward</code> of the models you would only see a chunk of the data. The output however is gathered again on the default device and you will see the output for the entire batch.<br>
In the latter case, each process would use a single GPU so would be responsible to load the entire batch.</p>

fermoren: <p>Actually, I´m not quite sure of what lightning is using. I tried to set the flag  accelerator=“ddp” on the Trainer but the problem persists.</p>
<p>Could I force “ddp_model = DDP(model, device_ids=[rank])” over my model or may that results in conflict in lightning?</p>

ptrblck: <p>I’m not familiar enough with Lightning so don’t know what it uses internally or how it would interact if you manually wrap the model into e.g. <code>DDP</code>.</p>
PyTorch distributed with MPI on Multi-node Multi-GPUs

krishmani.85: <p>Hi,<br>
I’m trying to run a PyTorch DDP code on 2 nodes with 8 GPUs each with mpirun. I want to use 1 mpi<br>
rank per node to launch the DDP job per node and let DDP launch 8 worker threads in each node. The command I’m using is</p>
<pre><code class="lang-auto">let N_NODES=2
let N_RANKS=2
for (( node_rank=0; node_rank&lt;$N_NODES; node_rank++ ))
do
        mpirun -np $N_RANKS -npernode 1 -hostfile $HOSTFILE --map-by node \ 
        python -u -m torch.distributed.launch  --master_addr=$master_addr --nproc_per_node=8  --nnodes=$N_NODES \
        --node_rank=$node_rank train.py &amp;

        pids[${node_rank}]=$!
done
</code></pre>
<p>When I check the activity on the GPUs with nvidia-smi, I see the code running on few GPUs on both nodes (6 on first node and 4 on second node). What is missing here to run on all 8 GPUs in each node, 16 in total? Any help is kindly appreciated.</p>
<p>Also, if I just launch 2 ranks with 1 rank per node with the below command, I see the expected execution, one instance running on a single gpu in each node.</p>
<blockquote>
<p>mpirun -np 2 -npernode 1 -hostfile $HOSTFILE --map-by node \ <br>
python -u -m torch.distributed.launch  --master_addr=$master_addr  train.py</p>
</blockquote>
<p>With the following way, I see 8 process on 1st node and 16 processes on 2nd node</p>
<blockquote>
<p>mpirun -np 2 -npernode 1 -hostfile $HOSTFILE --map-by node \ <br>
python -u -m torch.distributed.launch --nproc_per_node=8  train.py</p>
</blockquote>
<p>Or is there another way to launch the distributed implementation across multi-node multi-gpu system with mpirun.</p>
<p>Thanks,<br>
K</p>

Rakshith_V: <p>Did you find any drop in data loader performance while running through mpirun ??</p>
.transforms.ToTensor() Issue on Datasets.Cityscapes

L-Reichardt: <p>Hello Forum,</p>
<p>Im attempting to implement the Cityscapes dataset using <code>torchvision.datasets.Cityscapes</code>, but I keep getting errors when I try to load 2 labels. Loading a single label works fine. However, from my understanding, <code>datasets.Cityscapes</code> would support this. I have tried the following variants:</p>
<pre><code class="lang-auto">train_transform = transforms.ToTensor()
</code></pre>
<pre><code class="lang-auto">    train_dataset = datasets.Cityscapes('./data', split='train', mode='fine',
                     target_type='semantic', transform=train_transform, target_transform=train_transform)
    train_loader = DataLoader(dataset=train_dataset, batch_size = mini_batch_size, shuffle=shuffle, **kwargs)
</code></pre>
<p><strong> → Works as intended, I get my image and the semantic label</strong></p>
<pre><code class="lang-auto">    train_dataset = datasets.Cityscapes('./data', split='train', mode='fine',
                     target_type=['semantic','instance], transform=train_transform, target_transform=train_transform)
    train_loader = DataLoader(dataset=train_dataset, batch_size = mini_batch_size, shuffle=shuffle, **kwargs)
</code></pre>
<p><strong>TypeError: pic should be PIL Image or ndarray. Got &lt;class ‘tuple’&gt;</strong></p>
<p>How can I apply the transform to both labels?</p>
<p>Thank you in advance.<br>
L</p>
Legacy autograd function with non-static forward method - old pytorch code

qqaazz: <p>Trying to run some code from <a href="https://github.com/sbarratt/torch_cg/tree/master/torch_cg" rel="noopener nofollow ugc">here</a>. If I run the test.py after removing the require_gradients() on line 39 (which I’m not sure how to avoid the leaf variable error otherwise for) I still receive the following error message:</p>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
Input In [23], in <br>
61 print(f"Solving K={K} linear systems that are {n} x {n} with {As[0].nnz} nonzeros and {m} right hand sides.")<br>
63 cg = CG(A_bmm, M_bmm=M_bmm, rtol=1e-5, atol=1e-5, verbose=True)<br>
—&gt; 64 X = cg(B_torch)<br>
66 start = time.perf_counter()<br>
67 X_np = np.concatenate([np.hstack([splinalg.cg(A, B[:, i], M=M)[0][:, np.newaxis] for i in range(m)])[np.newaxis, :, :]<br>
68                        for A, B, M in zip(As, Bs, Ms)], 0)</p>
<p>File ~/.local/lib/python3.9/site-packages/torch/autograd/function.py:261, in Function.<strong>call</strong>(self, *args, **kwargs)<br>
260 def <strong>call</strong>(self, *args, **kwargs):<br>
 → 261     raise RuntimeError(<br>
262         "Legacy autograd function with non-static forward method is deprecated. "<br>
263         "Please use new-style autograd function with static forward method. "<br>
264         “(Example: <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 1.10.1 documentation</a>)”)</p>
<p>RuntimeError: Legacy autograd function with non-static forward method is deprecated. Please use new-style autograd function with static forward method.</p>
<hr>
<p>I’ve read some on this forum about this error, but cannot single out the problem in the definition of the forward, backward functions:</p>
<p>class CG(torch.autograd.Function):</p>
<pre><code>def __init__(self, A_bmm, M_bmm=None, rtol=1e-3, atol=0., maxiter=None, verbose=False):
    self.A_bmm = A_bmm
    self.M_bmm = M_bmm
    self.rtol = rtol
    self.atol = atol
    self.maxiter = maxiter
    self.verbose = verbose

def forward(self, B, X0=None):
    X, _ = cg_batch(self.A_bmm, B, M_bmm=self.M_bmm, X0=X0, rtol=self.rtol,
                 atol=self.atol, maxiter=self.maxiter, verbose=self.verbose)
    return X

def backward(self, dX):
    dB, _ = cg_batch(self.A_bmm, dX, M_bmm=self.M_bmm, rtol=self.rtol,
                  atol=self.atol, maxiter=self.maxiter, verbose=self.verbose)
    return dB</code></pre>

ptrblck: <p>The error message links to an example of a custom <code>autograd.Function</code>, which e.g. includes the <code>@staticmethod</code> decorators which are missing in your code.</p>

qqaazz: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks. when I add the <span class="mention">@staticmethod</span> bits in before the forward and backward definitions I still get the same error. What other decorators are important?</p>
Advantage of creating multiple layers with same arguments

tanvir_14: <p><strong>Case 1:</strong></p>
<pre><code class="lang-auto">class _ResGraphConv(nn.Module):
    def __init__(self, adj, input_dim, output_dim, hid_dim, p_dropout):
        super(_ResGraphConv, self).__init__()

        self.gconv1 = _GraphConv(adj, input_dim, output_dim, p_dropout)
        self.gconv2 = _GraphConv(adj, input_dim, output_dim, p_dropout)

    def forward(self, x):
        residual = x
        out = self.gconv1(x)
        out = self.gconv2(out)
        return residual + out
</code></pre>
<p><strong>Case 2:</strong></p>
<pre><code class="lang-auto">class _ResGraphConv(nn.Module):
    def __init__(self, adj, input_dim, output_dim, hid_dim, p_dropout):
        super(_ResGraphConv, self).__init__()

        self.gconv1 = _GraphConv(adj, input_dim, output_dim, p_dropout)

    def forward(self, x):
        residual = x
        out = self.gconv1(x)
        out = self.gconv1(out)
        return residual + out
</code></pre>
<p>As you can see in case 1, I have created 2 layers and calling them from forward. And in case 2, I’m creating only 1 layer as they take same arguments but calling that from forward twice to decrease the number of parameters. Is there any disadvantage of doing case 2?</p>

anantguptadbl: <p><a class="mention" href="/u/tanvir_14">@tanvir_14</a></p>
<p>In the second case, since your are reusing the same layer, your learnable parameters have been reduced.<br>
Imagine this way, you are using the same filters on the image twice.</p>

tanvir_14: <p><a class="mention" href="/u/anantguptadbl">@anantguptadbl</a>  thanks for the answer…I got that part…but does it have any disadvantage (case 2) in the case of accuracy or something?</p>

anantguptadbl: <p><a class="mention" href="/u/tanvir_14">@tanvir_14</a> yes it will definitely have. The output of the two layers might be capturing some essence of the image by filters, but since you are using the same conv layer, it will be restrained to use the same filter to extract features out of the image for 2 layers</p>
<p>In ideal scenarios it is not a good idea to use Case 2</p>

tanvir_14: <p><a class="mention" href="/u/anantguptadbl">@anantguptadbl</a> thanks so much for the answer…I got ur point</p>