akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
torchmetrics: Multi label Confusion matrix different device error

Sreeharan: <p>I’m trying to calculate the confusion matrix using torchmetrics for my multi-label output, but I get the following error:</p>
<pre><code>File "/home/antpc/.local/lib/python3.8/site-packages/torchmetrics/metric.py", line 394, in wrapped_func
    raise RuntimeError(
RuntimeError: Encountered different devices in metric calculation (see stacktrace for details).This could be due to the metric class not being on the same device as input.Instead of `metric=ConfusionMatrix(...)` try to do `metric=ConfusionMatrix(...).to(device)` where device corresponds to the device of the input.
</code></pre>
<p>My code:</p>
<pre><code>from torchmetrics import ConfusionMatrix
def calculate_metrics(predictions, targets):
	cm = ConfusionMatrix(num_classes=34, multilabel=True)
	matrix = cm(predictions, targets)
	return matrix
</code></pre>
<p>Then I tried to change my code as:</p>
<pre><code>from torchmetrics import ConfusionMatrix
def calculate_metrics(predictions, targets):
	cm = ConfusionMatrix(num_classes=34, multilabel=True).to(device='cpu')
	matrix = cm(predictions.detach().cpu(), targets.detach().cpu())
	return matrix
</code></pre>
<p>Still it shows the same error. Can anyone help me out with this?</p>

ptrblck: <p>Could you post a minimal, executable code snippet by adding the missing definitions, which would reproduce the issue, please?</p>

Sreeharan: <p>The code is written in pytorch lightning.</p>
<pre><code class="lang-auto">from torch import optim, nn
import pytorch_lightning as pl
from torchmetrics import ConfusionMatrix

class ModelClassifier(pl.LightningModule):
    def __init__(self):
        super(ModelClassifier, self).__init__()
        self.model = nn.Linear(3*512*512 ,out_features=34)
        self.loss_fn = nn.BCEWithLogitsLoss()
        self.cm = ConfusionMatrix(num_classes=34, multilabel=True).to(device='cpu')

    def forward(self, x):
        batch_size, _, _, _ = x.size()
        x = x.view(batch_size,-1)
        x = self.model(x)
        return x
    
    def configure_optimizers(self):
        optimizer = optim.Adam(self.parameters(), lr=0.01)
        return optimizer
    
    def loss_func(self, pred, labels):
        return self.loss_fn(pred,labels)
    
    def calculate_metrics(self, pred, labels):
        confusion_matrix = self.cm(pred.detach().cpu(), labels.detach().cpu())
        return confusion_matrix

    def training_step(self, batch, batch_idx):
        inputs, labels = batch

        outputs = self(inputs)
        loss = self.loss_fn(outputs, labels.float())
        metrics = self.calculate_metrics(outputs, labels)
        
        return loss

model = ModelClassifier()

trainer = pl.Trainer(strategy='dp', max_epochs=150, gpus=8, fast_dev_run=True)

trainer.fit(model, train_loader)
</code></pre>
<p>Here the train_loader contains images of size: (3 X 512 X 512) with a batch size of 32.</p>
Failed to create the filter from "fps=1,format=pix_fmts=rgb24" (Invalid argument)

tommas: <p>Hi there, I’m using StreamReader in torchaudio to process video. However, when using method add_basic_video_stream(), I encountered an error: <strong>Failed to create the filter from “fps=1,format=pix_fmts=rgb24” (Invalid argument.).</strong></p>
<p>Here is my source code creating the config:</p>
<pre><code class="lang-auto">s = StreamReader(src)
s.add_basic_video_stream(
        frames_per_chunk=1,
        format="rgb24",
        decoder="h264_cuvid",
        frame_rate=1,
        hw_accel="cuda:0"
    )
</code></pre>
<p>Here is my torch and torchaudio nightly version:</p>
<pre><code class="lang-auto">1.13.0.dev20220914+cu113
0.13.0.dev20220914+cu113
</code></pre>
<p>Can someone help me with this? Thank you all in advance.</p>

tommas: <p>For more information about my device information, I am running on <strong>Ubuntu 20.04.3 LTS</strong>, 2 NVIDIA GeForce RTX 3090.</p>
Updating two models at the same time

lemon_pi: <p>Hi, I’m new to pytorch and had no luck following similar threads. I’m trying to jointly train two models in the same loop, and the model updates involve a different computation that takes in some combined loss from both model_a and model_b. However, I’m not sure how to go about training them at the same time. Any advice would be greatly appreciated!</p>
<pre><code class="lang-auto">self.optimiser_a.zero_grad()
loss_a = calc_loss_a(output_a, output_b)
loss_a.backward()
self.optimiser_a.step()

self.optimiser_b.zero_grad()
loss_b = calc_loss_b(output_a, output_b)
loss_b.backward()
self.optimiser_b.step()
</code></pre>
<p>The error I get from the above is</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling .backward() or autograd.grad() the first time.
</code></pre>
<p>From the advice in some threads, I tried using retain_graph=True, but received this error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [4, 10]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>

ptrblck: <p>Using <code>retain_graph=True</code> is often used as a workaround which is unfortunately usually wrong.<br>
I don’t know how the outputs are calculated, but I guess you might be running into <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">this error</a>, which tries to use stale forward activations while calculating the gradients.</p>

lemon_pi: <p>Thank you for such a prompt and helpful reply! In that case, I’m still not very sure what the best practice in this scenario would be if I want the two losses to be back propagated independently.</p>
Printing weights after integer quantization

Vikram_M: <p>I loaded a model which is in FP32 and converted it to int8. But after printing the weights, I am getting floating point values as output, while I expected integers. Can someone let me know what I am missing?</p>
<pre><code>model = torch.ao.quantization.convert(model,inplace=True)
for i in model.parameters():
  print(i)
</code></pre>

jerryzh168: <aside class="quote no-group" data-username="Vikram_M" data-post="1" data-topic="162521">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vikram_m/48/44629_2.png" class="avatar"> Vikram_M:</div>
<blockquote>
<p>I loaded a model which is in FP32 and converted it to int8. But after printing the weights, I am getting floating point values as output, while I expected integers. Can someone let me know what I am missing?</p>
<pre><code class="lang-auto">
</code></pre>
</blockquote>
</aside>
<p>can you print the model, just to double check that the model is quantized?<br>
also I think the quantized weights are probably not parameters.</p>

Vikram_M: <p>How to do that?<br>
print(model) would just print the model’s architecture<br>
It would be helpful if someone provides the code on how to quantize the model given below which is pretrained</p>
<pre><code class="lang-auto">from torchvision.models import resnet50, ResNet50_Weights

weights = ResNet50_Weights.DEFAULT
model = resnet50(weights=weights)
</code></pre>
Passing per-parameter options within FSDP wrapped model

rowhanm: <p>Hi, I’m training a model using FSDP and wanted to use different learning rates for different parameters.</p>
<p>In order to get a list of the full names of the parameters, I used the <code>summon_full_params()</code> context manager and then I filter my params as per their names into two buckets <code>param_group_1</code> and <code>param_group_2</code>.</p>
<p>Then I pass these groups into the optimizer as:</p>
<pre><code class="lang-auto">torch.optim.SGD(
[{"params": param_group_1, "lr": 1e-3}, {"params": param_group_2, "lr": 1e-4}]
)
</code></pre>
<p>However this fails because -</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>If I pass the full model parameters into 1 group then it works but I would like to fine tune the LRs</p>
<pre><code class="lang-auto">torch.optim.SGD(
fsdp_model.parameters(), lr=1e-3
)
</code></pre>
<p>Is there a correct way of doing this? Am I finding the parameter groups correctly?</p>

agu: <p>Different learning rates for different parameters is not well-supported with FSDP at the moment, but we are working on it. This support may be available in the release <strong>after</strong> 1.13 (where 1.13 is the upcoming release) or earlier in nightlies. When the support lands and stabilizes, the documentation will include how to apply different hyperparameters to different parameters with FSDP.</p>
Reduce training runtime when set requires_grad=False for intermediate layers

leigao97: <p>If I freeze the beginning a few consecutive layers by setting requires_grad=False, the autograd engine will exclude those layers from the graph, so the computation and runtime during model training will be reduced. However, if I freeze some intermediate layers, the autograde engine will still compute the full graph. Thus, although frozen layers doesn’t get updated, the gradients with respect to the weights will still be computed and no runtime saving. Is my understanding correct? How to save computation in the second case? Is it possible that I can skip the computation of dL/dW? The layer that I am trying to freeze is conv2d, and the model I am doing experiment is resnet.</p>
<p>Reference:</p><aside class="quote quote-modified" data-post="24" data-topic="7088">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/l0sg/48/1481_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088/24">How the pytorch freeze network in some layers, only the rest of the training?</a> 
  </div>
  <blockquote>
    I’m afraid I have no definitive answer for this since I don’t know your exact model setup, but several suggestions: 

Every single tensor before the frozen part in the computational graph must also be requires_grad=False <a href="https://pytorch.org/docs/stable/notes/autograd.html" rel="noopener nofollow ugc">so that the frozen subgraph gets excluded in the autograd engine.</a> If there exists any tensor that requires grad, It’ll need all the backward pass of the graph anyway.
I would check which part of your model is a major bottleneck in speed. For example, if the unfrozen parts may co…
  </blockquote>
</aside>


leigao97: <p>For the first case, here is my code snippet to freeze the first 50 parameters in resnet34 model:</p>
<pre><code class="lang-auto">model = torchvision.models.resnet34()

for i, (name, param) in enumerate(model.named_parameters()):
    param.requires_grad = False
    if i &gt; 50:
        break

optimizer = optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=0.1)
</code></pre>
<p>For the second case, here is my code snippet to freeze all convolutional parameters in resnet34 model:</p>
<pre><code class="lang-auto">model = torchvision.models.resnet34()

for i, (name, param) in enumerate(model.named_parameters()):
    if param.shape == 4:
        param.requires_grad = False

optimizer = optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=0.1)
</code></pre>
<p>The runtime for training the first case model was reduced significantly compared with full model training, but for the second case, there is no runtime difference compared with full model training.</p>
How to calculate two tensors with different shape?

evilroach: <p>Sorry about post so simple question, I’m a freshman with PyTorch.</p>
<p>I need to normalize a feature tensor at inference stage.</p>
<p>Every average and STD values of every columns have been calculated already when training.</p>
<p>Now I get a row of new data to be inferred, and I need to normalize them at first.<br>
But the shape of input is difference to the average tensor(same for STD), I can’t finger it out how to make that.</p>
<p>For example:<br>
The average tensor is 1 dimension, and each element in it is for each column of the feature tensor in same order. The input has 2 dimensions.</p>
<pre><code>avg = torch.rand(5)
input = torch.rand(3, 5)
</code></pre>
<p>I know that can be done with a loop, but it’s not pythonic.<br>
I guess that there must be a solution in vector way.</p>
<p>Further more, when the input has 3 dimensions, i.e. [batch_size, row_count, column_count], how do it?</p>
<pre><code>avg = torch.rand(5)
input = torch.rand(1, 3, 5)
</code></pre>
<p>Thanks!</p>

ptrblck: <p>You can <code>unsqueeze</code> the smaller tensor explicitly and/or allow broadcasting to apply the operation elementwise:</p>
<pre><code class="lang-python">avg = torch.rand(5)
input = torch.rand(1, 3, 5)

input - avg
# tensor([[[ 0.0108,  0.4138,  0.0683,  0.1598, -0.1582],
#         [ 0.5192,  0.3414, -0.3722,  0.8293,  0.1087],
#         [-0.1535, -0.0606,  0.5320,  0.1019, -0.2201]]])

input - avg[None, None, :]
# tensor([[[ 0.0108,  0.4138,  0.0683,  0.1598, -0.1582],
#         [ 0.5192,  0.3414, -0.3722,  0.8293,  0.1087],
#         [-0.1535, -0.0606,  0.5320,  0.1019, -0.2201]]])
</code></pre>
How to make an attntion mask

skerlet_flandorle: <p>I want to make a mask to use for attntion etc.<br>
I used to mask with a for loop, but that would be too slow, so I’d like to change</p>
<pre><code class="lang-auto">[4,7,2,5,9]
</code></pre>
<p>The length of each data like</p>
<pre><code class="lang-auto">[1,1,1,1,1, 0,0,0,0,0]
[1,1,1,1,1, 1,1,1,0,0]
[1,1,1,0,0, 0,0,0,0,0]
[1,1,1,1,1, 1,0,0,0,0]
[1,1,1,1,1, 1,1,1,1,1]
</code></pre>
<p>I want to convert it so that it can be handled like 0 and 1 data=data*mask like<br>
Is there such a function in pytorch? Also, is there such a method other than for?</p>
Multi label text classification using Pytorch

pztcookie: <p>Hi. Would someone be so kind to provide me some project examples or tutorials on how to create the Pytorch dataset for multilabel text dataset, as well as trainer class? I am new to Pytorch, and first time doing multi-label text classification, it would be nice if there are some step-by-step tutorials or project examples. Thank you so much! I appreciate your help a lot!<br>
Here is what the data frame looks like. It needs to be preprocessed into a dataset that Pytorch accepts.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d24942bf29307abe0fb794cc83d188b644687dde.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d24942bf29307abe0fb794cc83d188b644687dde" title="Xnip2022-10-01_18-00-25"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d24942bf29307abe0fb794cc83d188b644687dde_2_690x243.jpeg" alt="Xnip2022-10-01_18-00-25" data-base62-sha1="u0hecYKFJRwO5VtZrKV8s6xLHum" width="690" height="243" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d24942bf29307abe0fb794cc83d188b644687dde_2_690x243.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d24942bf29307abe0fb794cc83d188b644687dde_2_1035x364.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d24942bf29307abe0fb794cc83d188b644687dde_2_1380x486.jpeg 2x" data-dominant-color="424242"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Xnip2022-10-01_18-00-25</span><span class="informations">1456×514 97.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Model training in Image reconstruction problem

Hafsa_Moontari_Ali: <p>I am working on an image reconstruction problem where I am generating single images. I have saved the model for each epoch and tested using five image. The images are generated for some modesl(such as models saved at epochs 171) but just for the model on 172 or 173 epoch, the reconstructed images are very obscure or completely dark. Again, the model saved at epoch ~200 can reconstruct almost perfect images. I am wondering if it means that my model is not really learning properly? I want to mention that the training loss and validation losses are decreasing.</p>
<p>Any suggestions would be much appreciated!</p>

ptrblck: <aside class="quote no-group" data-username="Hafsa_Moontari_Ali" data-post="1" data-topic="162591">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hafsa_moontari_ali/48/20614_2.png" class="avatar"> Hafsa_Moontari_Ali:</div>
<blockquote>
<p>but just for the model on 172 or 173 epoch, the reconstructed images are very obscure or completely dark.</p>
</blockquote>
</aside>
<p>Does this mean that the model stored at other epochs close to 172 is creating “good” outputs while only these two epochs are “bad”? Did you see a loss spike for these epochs?</p>
Input/Target size mismatch when training a downstream BERT for classification (huggingface pretrained)

Eylon_Ben_shitrit: <p>I am training a BERT model with a downstream task to classify movie genres. I am using HuggingFace pretrained model (aleph-bert since data is in Hebrew)</p>
<p>When training, I get the following error:</p>
<p>ValueError: Expected input batch_size (3744) to match target batch_size (16).</p>
<p>This is my notebook: <a href="https://colab.research.google.com/drive/1mqIUPnLOu_H-URn5tzE6gGySsW3oAcRY?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p>The error happens in the compute_loss functions, while performing the cross_entropy step.</p>
<p>My batch size is 16 but for some reason the bert output returns a different size.</p>
Error in creating engine from QAT model in pytorch 1.12 in TensorRT

Mahsa1994: <p>Hi,<br>
I’ve trained a quantized model (ResNet50) and exported to ONNX. I want to infer this model in TensorRT. But the ONNX model is not parsed correctly, so the TensorRT engine is not created.<br>
I test the model (from training to inference) with previous version of pytorch and it works correctly.<br>
The error is here:</p>
<blockquote>
<p>[09/20/2022-08:16:58] [TRT] [E] onnx::QuantizeLinear_1259: invalid weights type of Int8<br>
ERROR: Failed to parse the ONNX file.<br>
In node 0 (parseGraph): INVALID_NODE: Invalid Node - Identity_0<br>
onnx::QuantizeLinear_1259: invalid weights type of Int8</p>
</blockquote>
<p>The current version that gives me the error is:</p>
<blockquote>
<p>Torch==1.12.0<br>
ONNX==1.12<br>
TensorRT==8.2.3<br>
Opset==12</p>
</blockquote>

jerryzh168: <p>please take a look at FQA for GPU and ONNX: <a href="https://discuss.pytorch.org/t/quantization-frequently-asked-questions/161251#quantized-inference-in-gpu-3" class="inline-onebox">[quantization] Frequently Asked Questions</a></p>

Mahsa1994: <p>Thank you for your reply. I think it’s a version conflict, I get correct result when running with Torch==1.9.0.</p>
Best value for 'reduction' parameter?

ran_ran1: <p>hello,<br>
which is the best value to reduction instead of size_average=‘false’ to solve this error:<br>
warnings.warn(warning.format(ret)) UserWarning: size_average and reduce args will be deprecated, please use reduction=‘mean’ instead</p>
<blockquote>
<p>def loss_function(recon_x, x, pad_index):<br>
MSE = F.mse_loss(recon_x<em>pad_index, x</em>pad_index, size_average=‘false’)<br>
return MSE</p>
</blockquote>
<p>reduction=‘mean’ as mentioned in error message or reduction=‘none’</p>

ksmdanl: <p>To me it is not an error. The solution is already shown in the warning message, which is to replace <code>size_average='false'</code> with <code>reduction='none'</code>, if you wish to keep your input dimension:)</p>

ran_ran1: <p>thank you for reply but in warning message  said (reduction=‘mean’)not ‘none’ if I should track the warning,and  when I put reduction=‘none’ H get this error</p>
<blockquote>
<p>File “C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\autograd_<em>init</em>_.py”, line 67, in _make_grads<br>
raise RuntimeError(“grad can be implicitly created only for scalar outputs”)<br>
RuntimeError: grad can be implicitly created only for scalar outputs</p>
</blockquote>
Pytorch DDP bucket size does not match model size

yuri123: <p>Hi,</p>
<p>I’m using pytorchDDP with VGG16. I set the bucket_size=25mb. and the size of VGG16 is about 528mb. but all_reduce kernel is launched 6 times. (and 5 times for 50mb bucket size, 3 times for 10mb bucket size.) I think 6x25 does not match 528.</p>
<p>and Communication operations Stats of Distributed tap on Tensorboard says the Avg Size(bytes) of all_reduce is nearly 92mb. I understand that Avg Size column is the size of transferred Data. 92mb also does not match 25mb(bucket size)</p>
<p>I understand the all_reduce is launched when the bucket is full of gradient. Is the model size different with gradient size? I’m confused. Any help would be greatly appreciated.</p>
<p>Thank you.</p>

aazzolini: <aside class="quote no-group" data-username="yuri123" data-post="1" data-topic="162420">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuri123/48/53013_2.png" class="avatar"> yuri123:</div>
<blockquote>
<p>confused</p>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/yuri123">@yuri123</a>, thanks for your question! Could you share a small code for repro? Specially on how you set the bucket size at initialization and point out to what model implementation you’re using?</p>

yuri123: <p>Hi <a class="mention" href="/u/aazzolini">@aazzolini</a>, Thanks for your reply.<br>
I just set the bucket size like below</p>
<pre><code class="lang-auto">    model = VGGD.vgg16bn().cuda()
    ddp_model = DDP(model, device_ids=[rank], broadcast_buffers=True, 
    bucket_cap_mb=25, gradient_as_bucket_view=True)
</code></pre>
<p>and I’m sorry but I don’t understand what exactly model impletation means. ;(</p>
<p>as you can see the above code, I made VGGD.py and that’s like this</p>
<pre><code class="lang-auto">class vgg16bn(torch.nn.Module):
    def __init__(self):
        super(vgg16bn, self).__init__()
        self.model = nn.Sequential(OrderedDict([
                        ("block1_conv1", nn.Conv2d(3, 64, 3, padding = 1)),
                        ("block1_conv1_batchnorm", nn.BatchNorm2d(64)),
                        ("block1_conv1_ReLU", nn.ReLU(True)),
                        ("block1_conv2", nn.Conv2d(64, 64, 3, padding = 1)),
.
.
.

</code></pre>
<p>.+ To clarify the question I asked, when I set the bucket size 25mb, all_reduce kernel is launched 6 times <strong>per step.</strong><br>
(model size is about 528mb, so I think all_reduce is need to be excuted 528mb/25mb=21 or 22 times per step… please let me know what I’m wrong ;_; …)</p>
<p>Thank you!</p>
Training pauses after leaving my computer idle for a while, but resumes after I move the mouse or keyboard

Xin_Zhi_Tan: <p>I’m training a simple network on a single GPU overnight, however I notice that the training would stop after an hour or so and would continue when I move the mouse or keyboard (attached screenshot, the training pauses at epoch 76 and resumes after I moved the mouse).</p>
<p>So last night I decided to run a separate timing script (attached below) whilst training the network to check whether this is a PyTorch problem or something else, the training ran without any problem.</p>
<pre><code class="lang-auto"># timing script
from datetime import datetime
import time

while(1):
    now = datetime.now()
    current_time = now.strftime("%H:%M:%S")
    print("Current Time =", current_time)
    time.sleep(300)

</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/3/53e11489ee4008981d31b70e4c3dac7916f0b67d.png" data-download-href="https://discuss.pytorch.org/uploads/default/53e11489ee4008981d31b70e4c3dac7916f0b67d" title="Screenshot from 2022-09-30 21-09-12"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/53e11489ee4008981d31b70e4c3dac7916f0b67d_2_690x335.png" alt="Screenshot from 2022-09-30 21-09-12" data-base62-sha1="bY1Rsm2CJF85DXNMGJVMg874RhX" width="690" height="335" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/53e11489ee4008981d31b70e4c3dac7916f0b67d_2_690x335.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/53e11489ee4008981d31b70e4c3dac7916f0b67d_2_1035x502.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/53e11489ee4008981d31b70e4c3dac7916f0b67d_2_1380x670.png 2x" data-dominant-color="EFEEED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-09-30 21-09-12</span><span class="informations">1382×672 50.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Maybe your system is configured to shut down the GPU or your overall system if no user input is noticed. PyTorch itself won’t just pause your training.</p>
Accessing all matrix multiplication done by pytorch

highfrequencyfan: <p>Perhaps a vague and ambitious question, but for a research project, I am effectively looking to access all matrix multiplication done by a pytorch model under the hood and store it in memory. The use case is to run my own simulations with these matrices.</p>
<p>In the ideal case, I would be able to generate a python list of all matrix multiplication done by a particular pytorch model (in both training and inference) after running it, along with the order in which they occur. How would I begin to go about doing something like this? Any suggestions would be much appreciated!</p>

ptrblck: <p>Would “access all matrix multiplication” mean you would like to store the outputs of these matmuls or somehow the calls?</p>

highfrequencyfan: <p>Both the inputs and outputs, and the sequence in which they are multiplied (if possible). This would be sufficient for now.</p>
<p>To motivate this question, in the long run I will be attempting to perform all matrix multiplication on a processor currently inaccessible by pytorch. So what I’d like to do is take any two matrices that pytorch multiplies, multiply them myself (rather than through GPU as pytorch would usually do) and reroute the result back into the model for further training/inference. So I suppose for this purpose, access to the calls would be necessary.</p>
<p>Any direction or advice would be greatly appreciated. Thank you for your help!</p>

ptrblck: <p>Maybe overriding <code>__torch_function__</code> as described <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch">here</a> could work.</p>
Can not send dataset to GPU

salidw: <p>Hello,</p>
<p>I am a newbies for ML. I started training MNIST dataset. I can send my model to GPU by using this code.</p>
<blockquote>
<p>network.to(device) # Returned… true</p>
</blockquote>
<p>When I sent the dataset to GPU by this testing code.</p>
<p>for batch_idx, (data, target) in enumerate(train_loader):<br>
data.to(device)<br>
target.to(device)<br>
if batch_idx == 100:<br>
print(data.device)<br>
print(type(data))<br>
pass</p>
<p><span class="hashtag-raw">#Returned</span> … cpu<br>
<span class="hashtag-raw">#Returned</span> … &lt;class ‘torch.Tensor’&gt;</p>
<p><strong>Question</strong>: Why does “data.to(device)” still be on CPU?<br>
Note: I use RTX30xx</p>

ptrblck: <p>The <code>to()</code> operation is not applied inplace on tensors (unlike on modules as it’s recursively applied on all submodules, parameters, and buffers internally), so you need to reassign the tensor:</p>
<pre><code class="lang-python">x = x.to(...)
</code></pre>

salidw: <p>Thank you very much , I got it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">  solved</p>
Calculating MSE between 2 Tensors

teohongwei898: <p>Hello all!<br>
Currently doing some image deblurring and wanted to find out what is wrong.<br>
Using this code:</p>
<pre><code class="lang-auto">Y = np.square(np.subtract(sharpimg,img)).mean()
</code></pre>
<p>I have three images for reference:</p>
<ol>
<li>A original sharp image</li>
<li>A blurred version of the image</li>
<li>A deblurred image from (2) with a trained model.</li>
</ol>
<p>By plotting the images, it is evident that (3) is clearer than (2)<br>
However, calculating the Mean Square Error is a big problem. The MSE between (1) and (2) is 20+, while my MSE between (3) and (1) is 16000+<br>
Plotting my tensors out also showed a big difference in values.<br>
Does anyone know why, and how do I potentially fix this problem?<br>
Thank you!</p>

KFrank: <p>Hi Hong!</p>
<aside class="quote no-group" data-username="teohongwei898" data-post="1" data-topic="162517" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/teohongwei898/48/53587_2.png" class="avatar"> teohongwei898:</div>
<blockquote>
<p>The MSE between (1) and (2) is 20+, while my MSE between (3) and (1) is 16000+</p>
</blockquote>
</aside>
<p>The most likely explanation is that the input and output images<br>
of your deblurring network have different normalizations.  (This is<br>
certainly possible; your network outputs whatever you train it to<br>
output, so if you train it to output an image whose normalization<br>
differs from that of the input, it will.)</p>
<p>For example, your network might take in an unnormalized grayscale<br>
image whose pixel values range from 0 to 255, normalize it internally,<br>
and then output a normalized, deblurred image whose pixel values<br>
range from -1.0 to 1.0.</p>
<p>(I note that your reported mean-squared-error is close to (255 / 2)**2.)</p>
<p>Print out the <code>min()</code>, <code>max()</code>, and <code>mean()</code> of your three images (original,<br>
blurred, and deblurred) to see if inconsistent normalization explains<br>
your issue.</p>
<p>If this is the cause, you might consider retraining your network to produce<br>
a deblurred image with the desired normalization (or, alternatively, accept<br>
an input image with the desired normalization).</p>
<p>But, as a work-around, you could try just normalizing your original and<br>
deblurred images consistently before computing the mean-squared-error<br>
between them.</p>
<p>Best.</p>
<p>K Frank</p>

teohongwei898: <p>Hi Frank!<br>
Thanks for the suggestion! Normalizing seems to do the trick, MSE seems reasonable now. Unfortunately, the value of my MSE between (3) and (1) is a bit higher than between (1) and (2). Guess my model is not performing as well <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"> Thanks for the reply though!</p>
Shape of data affects speed of batch normalization

TriKri: <p>When looking for the reason for why the inference time of my network had suddenly increased by 25% compared to a previous version of itself, I realized that one of the changes I had made was to rearrange the layers and put each <code>PixelShuffle</code> layer <em>before</em> each corresponding <code>BatchNorm2d</code> layer instead of after it as it was before. Simply by making sure the batch normalization was done before the pixel shuffling I was able to reduce the inference time back to what it was before.</p>
<p>Since I doubt that the speed of the pixel shufflig depends on whether the data has been batch normalized or not (please correct me if I’m wrong), I supose that the speed of the batch normalization operation must depend on the shape of the data, which is affected by the pixel shuffling. Specifically, if the image data is reshaped to be higher and wider but have fewer feature maps, batch normalization of the data goes significantly slower.</p>
<p>Is this your experience too? Why is the speed of the batch normalization operation so dependent of the shape of the data, even though it contains equally many pixels in total? And can this knowledge be used to make the <code>BatchNorm2d</code> operation go even quicker?</p>

ptrblck: <p>Yes, this might be expected as even though “equally many pixels” are used on both approaches, the reductions and elementwise operations would not be the same.<br>
I don’t know which backend you are using or if you are on the GPU, but you might want to take a look at <a href="https://pytorch.org/tutorials/intermediate/nvfuser_intro_tutorial.html">this nvFuser tutorial</a> which explains how PyTorch uses it to automatically generate fast kernels for these kind of operations (normalizations are already working) to avoid these performance cliffs.</p>

TriKri: <p>What do you mean by which backend I’m using? Yes, I’m working with the data on the GPU. Thanks for the link; I will take a look!</p>
RuntimeError: CUDA error: operation would make the legacy stream depend on a capturing blocking stream

groucho64738: <p>I haven’t gotten a lot of info in searching for an error we’ve been experiencing on our systems (once we added some NVidia A40 GPUs to our mix). This error appears when running some simple code like this:</p>
<pre><code class="lang-auto">import torch
device=torch.device("cuda")
x = torch.randn(10, 100).to(device)
</code></pre>
<p>If we use pytorch 1.8.1-cu113 then there is no issue, any version of pytorch more recent causes this problem. If we use the older V100 GPUs (and pytorch 1.11-cu102) then it works. I’ve tried pytorch 1.9 through the latest with cu113 and none of it works…except 1.8.1</p>
<p>Is there a special flag or option to enable that will make this start working again?</p>
<p>If there’s more info you need on my environment I can let you know. This was using miniconda with python 3.9, and I upgraded to python 3.10 and tried the same stuff and it didn’t work (RHEL8).</p>
<p>Any help someone can offer would be greatly appreciated.</p>

ptrblck: <p>No, there is no special flag and we couldn’t reproduce the issue as described in your <a href="https://github.com/pytorch/pytorch/issues/65528">cross post</a>. The user provided the information about their setup, installed driver, used PyTorch binary etc. and even a matching system on our side wasn’t able to raise this error.<br>
I would thus recommend to try a clean reinstall of the drivers and to retry it again.</p>
Torch JIT Static Runtime Restrictions

ryandeng: <p>I am looking at the new static runtime feature inside torch.jit on the C++ side. I was wondering if there are any restrictions on the model being run? For example, if the model does a backwards pass during inference, does the static runtime still support this?</p>
<p>If it doesn’t, is there a reliable way to run inference on the model on many CPU threads simultaneously? Deepcopying the torch script module doesn’t seem to create a new underlying graph.</p>
Word Embedding using transformers (BERT, Roberta, or any other model) or Fast-text, Glove etc?

RAJA_PARIKSHAT: <p>I am training a model to learn title similarities between two titles. I have around 2-3 Million lines of text data, scraped from multiple sources. Now I want to use that data to train a model that can learn title similarity. For example Finance officer is close to the Finance lead compared to the sales officer.</p>
<p>Which models should I go to? Fast text, Glove, or transformer-based models.<br>
Note: at inference time I will give titles only and compute some similarity metrics like cosine distance or something else.</p>

dleemiller: <p>The sentence transformers library has many pretrained models, and is implemented in pytorch. It might suit your needs:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/UKPLab/sentence-transformers">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/UKPLab/sentence-transformers" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/3e03f33e5aae0e15fc5490929583ab1db2ff5bf76af8099e2069cd30e566bd0d/UKPLab/sentence-transformers" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/UKPLab/sentence-transformers" target="_blank" rel="noopener nofollow ugc">GitHub - UKPLab/sentence-transformers: Multilingual Sentence &amp; Image...</a></h3>

  <p>Multilingual Sentence &amp; Image Embeddings with BERT - GitHub - UKPLab/sentence-transformers: Multilingual Sentence &amp; Image Embeddings with BERT</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Strict=false in load_stat_dict

cbd: <p>What strict=false do in load_stat_dict?</p>
<p>I read it load with missing parameter. For an example if i have module of 4 convolution layer followed by BN and RelU. Then if i have pth file of 3 convolution layer followed by BN and RelU OR  5 convolution layer followed by BN and RelU then it is possible to load weights using this argument. Am i right?</p>

albanD: <p>It will if the ones that match have the same name and the same paramters.</p>
<p>In general, it is used if you extend a given Module to add extra stuff but you still want to be able to load a checkpoint from the original module that contains the paramters from all the common pieces.</p>

futscdav: <p>To expand, <code>state_dict</code> is like a normal python dictionary. Default <code>strict=True</code> means that when the model loads, it will work if and only if the dictionary has keys with the exact same name as the parameters of the model AND nothing else.</p>
<p>With <code>strict=False</code>, you are saying that you don’t care if the parameters which are not included (by name) in the dictionary don’t get loaded and to ignore any keys which the module does not use.</p>
<p><code>strict=False</code> is useful for example when you created a Module definition which only covers part of a previous module (think feature extractor made from a classification net), or when you want to expand a network and preload some of the weights from an existing module. In both cases, you need to be very careful to get the naming right or manually rename the keys in the dictionary.</p>

juvekaradheesh: <p>EDIT: Nvm, found my answer in the official tutorials<br>
<a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#id4" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/saving_loading_models.html#id4</a></p>
<blockquote>
<p>Whether you are loading from a partial <em>state_dict</em> , which is missing some keys, or loading a <em>state_dict</em> with more keys than the model that you are loading into, you can set the <code>strict</code> argument to <strong>False</strong> in the <code>load_state_dict()</code> function to ignore non-matching keys.</p>
</blockquote>
<p>Just to be sure, will it work with strict=False for this case:</p>
<p><code>model = ['x', 'y', 'z' , 'a']</code><br>
<code>state_dict = ['x', 'y', 'z', 'b']</code></p>
<p>I don’t care about ‘a’ and ‘b’, I just want to load x, y, z</p>

albanD: <p>Yes it will just ignore these if strict=False!</p>
Strange warning when working with pretrained models

meshghi: <p>Hi, I’m receiving a series of warnings like the following. I was wondering if it is safe to disregard them? Thanks.<br>
<code>/lib/python3.9/site-packages/pretrainedmodels/models/dpn.py:258: SyntaxWarning: "is" with a literal. Did you mean "=="?   elif block_type is 'down':</code></p>

ptrblck: <p>The warning is raised in the 3rd party lib <code>pretrainedmodels</code> in <a href="https://github.com/Cadene/pretrained-models.pytorch/blob/8aae3d8f1135b6b13fed79c1d431e3449fdbf6e0/pretrainedmodels/models/dpn.py#L255">this line of code</a>:</p>
<pre><code class="lang-python">        if block_type is 'proj':
            self.key_stride = 1
            self.has_proj = True
        elif block_type is 'down':
            self.key_stride = 2
            self.has_proj = True
        else:
            assert block_type is 'normal'
            self.key_stride = 1
            self.has_proj = False
</code></pre>
<p>since a <code>str</code> is compared using the “identity” comparison via <code>is</code> instead of the “equality” comparison via <code>==</code>.<br>
The former checks if the variables are pointing to the exact same object while the latter compares if the variables point at values which are equal.<br>
The repository author might need to fix this, but given the repository wasn’t updated in ~3+ years I doubt the author is still interested in it.</p>

meshghi: <p>Thank you. What surprises me is that I didn’t get this error before. I’m running the code with the same everything, and now I suddenly got this error. I really am wondering why?</p>

ptrblck: <p>I guess your Python version might have been updated and raises the warning now (rightfully) or maybe you’ve (accidentally) silenced these warnings before somehow. This warning is real but doesn’t depend on any PyTorch-specific code and is raised by plain Python:</p>
<pre><code class="lang-python">if "a" is "b":
    print("c")
# SyntaxWarning: "is" with a literal. Did you mean "=="?
#  if "a" is "b":
</code></pre>

meshghi: <p>Got it. Thank you so much for the reply.</p>
Pip install torchviz==0.0.2

yviiendaleee: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e01366a64adc3cfe8d5ee64dc9783de0b1a421cb.png" data-download-href="https://discuss.pytorch.org/uploads/default/e01366a64adc3cfe8d5ee64dc9783de0b1a421cb" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e01366a64adc3cfe8d5ee64dc9783de0b1a421cb.png" alt="image" data-base62-sha1="vYgwe1zg6yS7HFLkOVMcgTcTu8X" width="690" height="104" data-dominant-color="161213"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1040×157 6.81 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am getting  this mistake when trying to run the command pip install torchviz==0.0.2. I am using python version 3.9.12 and torch version 1.12.0.<br>
Does anyone know why I can’t install torchviz via the command window or what the problem is?</p>

ptrblck: <p>It seems to be working for me:</p>
<pre><code class="lang-python">pip install torchviz==0.0.2
Collecting torchviz==0.0.2
  Downloading torchviz-0.0.2.tar.gz (4.9 kB)
  Preparing metadata (setup.py) ... done
Collecting torch
  Downloading torch-1.12.1-cp38-cp38-manylinux1_x86_64.whl (776.3 MB)
</code></pre>
<p>but I don’t know why it’s failing on your system. Maybe try to download the wheel and install it directly from <a href="https://pypi.org/project/torchviz/#files">here</a>.</p>
Transforming a 3D Convolutional encoder layer to a 3D Deconvolutional layer

Tm4: <p>I have a 3D convolutional network that is composed of an encoder and a decoder. In the decoder part, I want to use the type of convolutional layers that are used in the decoder part but making them to do deconvolution. In other words, I want that the mentioned convolutional layers increase the spatial size as like as the converse rate that is used in the encoder while decrease the number of channels.<br>
The convolutional layer that I use inside my encoder contains sequential layers:</p>
<pre><code>Conv_layer = nn.Sequential(
            BasicConv3d(64, 64, kernel_size=1, stride=1),
            SepConv3d(64, 192, kernel_size=3, stride=1, padding=1),
            nn.MaxPool3d(kernel_size=(1,3,3), stride=(1,2,2), padding=(0,1,1)),
        )
</code></pre>
<p>The type of convolutional layers that are used:</p>
<pre><code>    class BasicConv3d(nn.Module):
        def __init__(self, in_planes, out_planes, kernel_size, stride, padding=0):
        super(BasicConv3d, self).__init__()
        self.conv = nn.Conv3d(in_planes, out_planes, kernel_size=kernel_size, stride=stride, padding=padding, bias=False)
        self.bn = nn.BatchNorm3d(out_planes, eps=1e-3, momentum=0.001, affine=True)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.conv(x)
        x = self.bn(x)
        x = self.relu(x)
        return x
    
 class SepConv3d(nn.Module):
        def __init__(self, in_planes, out_planes, kernel_size, stride, padding=0):
        super(SepConv3d, self).__init__()
        self.conv_s = nn.Conv3d(in_planes, out_planes, kernel_size=(1,kernel_size,kernel_size), stride=(1,stride,stride), padding=(0,padding,padding), bias=False)
        self.bn_s = nn.BatchNorm3d(out_planes, eps=1e-3, momentum=0.001, affine=True)
        self.relu_s = nn.ReLU()

        self.conv_t = nn.Conv3d(out_planes, out_planes, kernel_size=(kernel_size,1,1), stride=(stride,1,1), padding=(padding,0,0), bias=False)
        self.bn_t = nn.BatchNorm3d(out_planes, eps=1e-3, momentum=0.001, affine=True)
        self.relu_t = nn.ReLU()

    def forward(self, x):
        x = self.conv_s(x)
        x = self.bn_s(x)
        x = self.relu_s(x)

        x = self.conv_t(x)
        x = self.bn_t(x)
        x = self.relu_t(x)
        return x
</code></pre>
<p>My question is that how I should chnge the kernel_size, stride and padding that change the mentioned layers to deconvolution that incease the spatial size of feature maps as the same with the converse rate of convolutional layers.</p>
Class name difference between original and quantized version of Sigmoid

CRPS: <p>Hello, i`m interested in fx quantization of pytorch model. In my project i need to register model by its class name, for example:<br>
I define module class from GraphModule by calling node.target. For quantized version of ConvBnReLU output would be QuantizedConvBnReLU.</p>
<p>Despite quantized Sigmoid my method works for any quantized module because of “Quantized” prefix in module class name. But for quantized Sigmoid class name would be “Sigmoid”, not “QuantizedSigmoid”. Why and is it error in naming?</p>

CRPS: <p>I checked realization of quantized activations and Sigmoid doesn’t contain method _get_name.</p>