akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
TypeError: 'tuple' object is not callable when using flatten layer

laro: <p>I Have the following model:</p>
<p>import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F</p>
<p>class MyModel(nn.Module):</p>
<pre><code>def __init__(self, input_size, num_classes):
    super(MyModel, self).__init__()
    
    
    
    self.layer_1      = nn.Conv1d(1,   16,  3, bias=False, stride=2)
    self.activation_1 = F.relu       
    self.adap         = nn.AdaptiveAvgPool1d(1)
    self.flatten      = nn.Flatten          (),  
    
    self.layer_2      = torch.nn.Linear(2249, 500)
    self.activation_2 = F.relu
    self.layer_3      = torch.nn.Linear(500, 2)
   
    
    pass

def forward(self, x, labels=None):
    
    
    x = x.reshape(256, 1, -1)                        
    x = self.layer_1(x)     
    x = self.activation_1(x)            
    x = self.flatten(x)
    return x
</code></pre>
<p>When running torchinfo<br>
model = MyModel(input_size=4500, num_classes=2)<br>
torchinfo.summary(model, (256, 4500))</p>
<p>I’m Getting error:</p>
<p>Input In [101], in MyModel.forward(self, x, labels)<br>
30 x = self.activation_1(x)<br>
—&gt; 31 x = self.flatten(x)<br>
32 return x</p>
<p>TypeError: ‘tuple’ object is not callable</p>
<ol>
<li>What is wrong ?</li>
<li>What do I need to change ?</li>
</ol>

ptrblck: <p>Remove the comma after <code>nn.Flatten()</code> and the error should be gone:</p>
<pre><code class="lang-python">self.flatten      = nn.Flatten          (),  # &lt;- note the comma
</code></pre>
Brevitas Trained Model Does Not Get Exported To ONNX

mrFoxx: <p>I have trained my brevitas model with Brevitas. Now I am trying to export my onnx model to use it with finn. As I try to export my model with the command below:</p>
<pre><code class="lang-auto">build_dir = "./"

base_file_name = "my_model"

bo.export_finn_onnx(my_model, (1, 3, 1024), build_dir + "/{}.onnx".format(base_file_name))
</code></pre>
<p>I get the error message as:</p>
<blockquote>
<p>TypeError: max_pool1d(): argument ‘kernel_size’ (position 2) must be tuple of ints, not Tensor,</p>
</blockquote>
<p>I couldn’t figure it out how to export my .pth extended file. If you help I will be appreciated. Thank you</p>

mrFoxx: <p>Here is how my max pool 1 d line works by the way if it helps.</p>
<pre><code class="lang-auto">   pool_size = int(xb.size(-1))
        xb = F.max_pool1d(xb, pool_size).squeeze()
</code></pre>
<p>I have changed my code to that after I have read a bit of different comments. On its old version the code was a below, but I was still getting the same error with that too.<br>
<code>xb = nn.MaxPool1d(xb.size(-1))(xb)</code></p>
RuntimeError: the derivative for 'target' is not implemented for Auto Encoder

desert_ranger: <p>I am getting the following error - RuntimeError: the derivative for ‘target’ is not implemented<br>
I did have a look at similar posts however they are different from my problem. I’m trying to code a AE from scratch. Here’s my code -</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
from torch import nn
from torchviz import make_dot

trainset = torchvision.datasets.FashionMNIST(root='./data', train=True,
                                        download=True)
data = trainset.data.float()
# print(trainset.data.shape)
# plt.imshow(trainset.data[9])
# plt.show()
device = "cuda" if torch.cuda.is_available() else "cpu"
data = data.to(device)
print(f"Using device = {device}")
class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.encode = nn.Sequential(
            nn.Linear(28*28, 512),
            nn.ReLU(),
            nn.Linear(512, 30),
            nn.ReLU()
        )
        self.decode = nn.Sequential(
            nn.Linear(30, 512),
            nn.ReLU(),
            nn.Linear(512, 28*28),
            nn.ReLU()
        )

    def forward(self, x):
        x = self.flatten(x)
        encoded = self.encode(x)
        decoded = self.decode(encoded)
        return decoded

model = NeuralNetwork().to(device)
# print(model)

lossFn  = nn.BCELoss()

optimizer = torch.optim.SGD(model.parameters(), lr = 1e-3)

for epoch in range(1000):
    optimizer.zero_grad()
    outputs = model(data)
    loss = lossFn(outputs, outputs)
    loss.backward()
    optimizer.step()
    model.forward(data)
</code></pre>

ptrblck: <p>I cannot reproduce the error using your code snippet, but would guess you could <code>detach()</code> the <code>outputs</code> passed as the target to <code>nn.BCELoss</code>. However, I also don’t quite understand why you would like to calculate the loss between the output and itself. Shouldn’t it be the output and the input ot a target?</p>
<p>However, I’m running into an error complaining about the input value range as <code>nn.BCELoss</code> expects probabilities while you are returning the output of an <code>nn.ReLU</code> layer:</p>
<pre><code class="lang-python">RuntimeError: all elements of input should be between 0 and 1
</code></pre>
<p>You could try to use a <code>sigmoid</code> activation and revisit your use case.</p>
<p>Note that the <code>data</code> input tensor is not normalized, so you might also want to divide it by <code>255</code>.</p>

desert_ranger: <p>Thank you for your reply. Your suggestions worked and the code is running now!</p>
DataParallel Blocks Internal LSTM Weights

andre99amaral: <p>Hello,</p>
<p>I am getting this problem when using DataParallel I can not access my LSTM internal weights. The following picture exemplifies the error.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6ecdfd1e9efd924cd9347af64c6f8a65a22a9416.png" data-download-href="https://discuss.pytorch.org/uploads/default/6ecdfd1e9efd924cd9347af64c6f8a65a22a9416" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6ecdfd1e9efd924cd9347af64c6f8a65a22a9416_2_690x167.png" alt="image" data-base62-sha1="fOdQJJCqGyCLWCyZTQQUBV0nfim" width="690" height="167" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6ecdfd1e9efd924cd9347af64c6f8a65a22a9416_2_690x167.png, https://discuss.pytorch.org/uploads/default/original/3X/6/e/6ecdfd1e9efd924cd9347af64c6f8a65a22a9416.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/e/6ecdfd1e9efd924cd9347af64c6f8a65a22a9416.png 2x" data-dominant-color="F1F6F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">823×200 66.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks in advance for your help!<br>
Regards,<br>
André</p>

ptrblck: <p>If you are using <code>nn.DataParallel</code> the model will be wrapped into the <code>.module</code> attribute so you might need to access the parameter via <code>model.module.layer.weight</code>.</p>

andre99amaral: <p>The LSTM formulas are the following:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2dbae9d6bc3dacf5b07b405db2c47238f346c62.png" alt="image" data-base62-sha1="u5kW03Xny09itI68UxzyGqLYDqG" width="375" height="255"></p>
<p>However, if I get the LSTM internal weights and bias, for the first expression, the matrices sizes do not match.</p>
<p>Wi_i is [450 by 2]<br>
Wh_i is [450 by 450]<br>
bi_i is [450 by 1]<br>
bh_i is [450 by 1]</p>
<p>x = [5000 by 3 by 2]<br>
h = [1 by 5000 by 450]</p>
<p>How is it possible to perform the i_t equation for example?</p>
<p>Thanks for the help!</p>

ptrblck: <p>The shape of the parameters is described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html?highlight=lstm#torch.nn.LSTM">docs</a> as e.g.:</p>
<blockquote>
<p><strong>~LSTM.weight_ih_l[k]</strong> – the learnable input-hidden weights of the kth layer (W_ii|W_if|W_ig|W_io), of shape (4*hidden_size, input_size) for k = 0…</p>
</blockquote>
<p>so they are in a packed format.</p>
Is this dice loss function correct?

yx_L: <p>I try to use the dice loss to calculate the distance between the true mask and the predicted mask,<br>
code:</p>
<pre><code class="lang-auto">#inputs.shape = [batch_size, cls, w, h] input(y_pred)
#target.shape = [batch_size, 1, w, h] target(y_true)
class DiceLoss(torch.nn.Module):
    def __init__(self, weight=None, size_average=True):
        super(DiceLoss, self).__init__()

    def forward(self, inputs, targets, num_classes, smooth=1):
        
        inputs = torch.sigmoid(inputs)
        print("inputs.shape:",inputs.shape)
        targets = targets.squeeze(1) # target：[batch_size, 1, w, h] -&gt; [batch_size, w, h]
        targets = torch.nn.functional.one_hot(targets, num_classes) # [batch_size, w, h, cls]
        targets = targets.permute(0,3,1,2) # [batch_size, w, h, cls] -&gt; [batch_size, cls, w, h]
        targets = targets.contiguous()
        print("targets.shape:",targets.shape)
        #flatten label and prediction tensors
        inputs = inputs.view(-1)
        targets = targets.view(-1)
        intersection = (inputs * targets).sum()               
        dice = (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth)  
        return 1 - dice
dice_loss = DiceLoss()


class_num = 2

w = 2

h = 2

batch_size = 1

A = torch.ones([batch_size,class_num,w,h]) # A.shape = torch.Size([batch_size, cls, w, h])

B = torch.zeros([batch_size,1,w,h]).long() # B.shape = torch.Size([batch_size, 1, w, h])

dice_loss(A,B,num_classes = 2)
</code></pre>
What is feature for image recognition

tom7: <p>I’m thinking what feature is in image recognition</p>
<p>Machine is checking rgb gradient among pixels in convolution neural network of PyTorch ?</p>
<p>There is method such as SIFT（Scale-Invariant Feature Transform) and HOG（Histograms of Oriented Gradients)</p>
<p>Which one is used in PyTorch ?</p>

ptrblck: <aside class="quote no-group" data-username="tom7" data-post="1" data-topic="155043">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom7/48/50012_2.png" class="avatar"> tom7:</div>
<blockquote>
<p>Machine is checking rgb gradient among pixels in convolution neural network of PyTorch ?</p>
</blockquote>
</aside>
<p>No, while the “RGB” representation could be valid for the input to a CNN, the intermediate activations are created by the conv layers and have multiple channels which cannot be mapped to the colorspace again.</p>
<aside class="quote no-group" data-username="tom7" data-post="1" data-topic="155043">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom7/48/50012_2.png" class="avatar"> tom7:</div>
<blockquote>
<p>Which one is used in PyTorch ?</p>
</blockquote>
</aside>
<p>Neither, as the conv layers are using a trainable conv kernel and perform a convolution (or rather a cross-correlation) using the input activation.</p>
<p>I would recommend to take a look at e.g. <a href="https://cs231n.github.io/">CS231n</a> as it explains the concepts of CNNs pretty well.</p>
Would the eps of 1e-8 in AdamW be rounded to zero when open `torch.amp`?

songyuc: <p>Hi, guys,<br>
I heard from the Internet that, the eps of 1e-8 in <code>torch.optim.AdamW</code> be rounded to zero when open <code>torch.amp</code>, is this true?</p>
<p>Your answer and guide will be appreciated!</p>

ptrblck: <p>Could you post a reference you saw as I would expect to see some discussion about it there, too?</p>
LBFGS closure in TorchScript?

fdan: <p>Hello folks,</p>
<p>I have a model which I’m refactoring to support torchscript for use in a c++ library.  An issue I have run into is <em>function definitions aren’t supported</em> when using a closure for the LBFGS optimiser (<a href="https://pytorch.org/docs/stable/_modules/torch/optim/lbfgs.html#LBFGS.step" rel="noopener nofollow ugc">which according to the docs is necessary</a>).  So I’m wondering if it’s possible to use the LBFGS optimiser in torchscript?</p>
<pre><code class="lang-auto">import torch

class Test(torch.nn.Module):
    def __init__(self):
        super(Test, self).__init__()      
        
    def forward(self, 
                input_: torch.Tensor, 
                target: torch.Tensor) -&gt; torch.Tensor:
        
        optim = torch.optim.LBFGS(tensor)
        max_iter = 50
        n_iter = 0
        
        def closure():
            optim.zero_grad()
            loss = input_ - target
            loss.backward()
            n_iter += 1
            return loss  
        
        while n_iter &lt;= max_iter:
            optim.step(closure)
        
        return tensor
    
    
test = torch.jit.script(Test())
test.save('test.pt')
</code></pre>
<p>The truncated output is:</p>
<pre><code class="lang-auto">UnsupportedNodeError: function definitions aren't supported:
  File "/tmp/ipykernel_4162/3211224755.py", line 15
        n_iter = 0
        
        def closure():
        ~~~ &lt;--- HERE
            optim.zero_grad()
            loss = input_ - target
'Test.forward' is being compiled since it was called from 'Test.forward'
  File "/tmp/ipykernel_4162/3447201789.py", line 20
                target: torch.Tensor) -&gt; torch.Tensor:
        
        optim = torch.optim.LBFGS(tensor)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
        max_iter = 50
        n_iter = 0
</code></pre>
Install from source - problems

stefaniebosch: <p>Hello everyone, I am working on a laptop that runs Windows 11. I am trying to install pytorch from source, but when I run “python setup.py install” or “python setup.py develop” it will install for a while but always fail and give me these errors:</p>
<p>“C:\Users\stefa\Desktop\pytorch\c10/test/util/complex_test_common.h(431): fatal error C1001: Internal compiler error.<br>
(compiler file ‘D:\a_work\1\s\src\vctools\Compiler\CxxFE\sl\p1\c\p1_init.c’, line 3668)”</p>
<p>and</p>
<p>“[18/1846] Building CXX object c10\cuda\CMakeFiles\c10_cuda.dir\CUDACachingAllocator.cpp.obj<br>
ninja: build stopped: subcommand failed.”</p>
<p>Does anyone know what to do to fix this? I already tried a few fixes, like the fix from mrshenli in <a href="https://github.com/pytorch/pytorch/issues/32694" class="inline-onebox" rel="noopener nofollow ugc">ninja: build stopped: subcommand failed. · Issue #32694 · pytorch/pytorch · GitHub</a>, but still keep getting these errors.</p>
How to obtain the tensor from the middle part of a CNN model (EfficientNet_b0), instead of the last layer?

John_M: <p>How to obtain the tensor from the middle part of a CNN model (EfficientNet_b0), instead of the last layer?</p>
<p>I saw code from some researchers that obtain the tensor from the middle part of ResNet50 in the following method:<br>
To draw ResNet50 in a simple way (there are other ways of drawing it) based on the PyTorch model:<br>
(first 4 structures)<br>
(layer1) (Bottleneck 012)<br>
(layer2) (Bottleneck 0123)<br>
(layer3) (Bottleneck 012345)<br>
(layer4) (Bottleneck 012)<br>
(last 2 structures)<br>
We can obtain the tensor right after layer1,2,3,4 by modifying the lines below and writing our own resnet.py and then load the ResNet50’s pretrained weights:<br>
def forward(self, x): …<br>
x = self.layer1(x)<br>
x = self.layer2(x)<br>
x = self.layer3(x)<br>
x = self.layer4(x) …<br>
(from <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py" class="inline-onebox" rel="noopener nofollow ugc">vision/resnet.py at main · pytorch/vision · GitHub</a>)</p>
<p>If I load EfficientNet_b0 (shown in the below EfficientNet_b0 graph) by: efficientnet_b0 = models.efficientnet_b0(pretrained=True) (which means the EfficientNet_b0’s model is already fixed):<br>
(0) (first 3)<br>
(1) (MBConv 0)<br>
(2) (MBConv 01)<br>
(3) (MBConv 01)<br>
(4) (MBConv 012)<br>
(5) (MBConv 012)<br>
(6) (MBConv 0123)<br>
(7) (MBConv 0)<br>
(8)<br>
(last 2)<br>
Is there a simple way of obtaining the tensor right after (5)(6)(7)(8)?</p>

ptrblck: <p>You could use forward hooks as described e.g. <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a> or manipulate the <code>forward</code> method directly by creating a new module deriving from the <code>EfficientNet_b0</code> implementation.</p>

John_M: <p>Thanks. Sorry that I forgot to confirm the solution. Forward Hook is the correct answer to my problem. However, I studied the implementation of Forward Hook from here: <a href="https://web.stanford.edu/~nanbhas/blog/forward-hooks-pytorch/" class="inline-onebox" rel="noopener nofollow ugc">Intermediate Activations — the forward hook | Nandita Bhaskhar</a> ,<br>
just in case someone wants to know how to use it, because there’s no much documentation about Forward Hook.</p>
How obtain many to many lstm layers predictions as output?

Nidhi_Joshi: <p>So I have trained an LSTM model which takes time-series (C-MAPSS dataset) data that have sliding window formulation (somewhat similar to the picture below)</p>
<p>model input : <code>(batch_size x sliding_window x feature_size)</code><br>
model output: (batch_size x sliding_window_outputs)</p>
<p>, which is good for training the model. Helps in the reduction of MSE loss and r2 loss seems alright. Now I have run into another complication, i.e., because of such dataset/dataloaders formulation, the predictions are also going to be sliding window technique (because test dataset also has to be in the format since model accepts such dimensionality)  and cannot be really used on test dataset to check the model performance in the end.<br>
Can anybody suggest how to obtain predictons just for the time-series data and not according to the sliding window approach? Basically, how to formulate the test-data loaders or any post testing modification so that I can get predictions serially and not sliding window fashion?</p>
<p>Thank you</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/7670fcaf6cfaf8923df42088aaab2e06286333b5.png" data-download-href="https://discuss.pytorch.org/uploads/default/7670fcaf6cfaf8923df42088aaab2e06286333b5" title="A-two-stage-sliding-window-to-create-mini-batches-that-overlap"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/7670fcaf6cfaf8923df42088aaab2e06286333b5_2_690x333.png" alt="A-two-stage-sliding-window-to-create-mini-batches-that-overlap" data-base62-sha1="gTMpJR3P04xK8pTl1cbI1KftHet" width="690" height="333" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/7670fcaf6cfaf8923df42088aaab2e06286333b5_2_690x333.png, https://discuss.pytorch.org/uploads/default/original/3X/7/6/7670fcaf6cfaf8923df42088aaab2e06286333b5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/6/7670fcaf6cfaf8923df42088aaab2e06286333b5.png 2x" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">A-two-stage-sliding-window-to-create-mini-batches-that-overlap</span><span class="informations">850×411 49.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to use and finetune XLS-R wave2vec2 feature extractor in PyTorch?

Wafaa_Wardah: <p>Please help me to some resource that shows how to use XLS-R (wave2vec2) in PyTorch training setup. I’ve looked all over but found no guide or sample code for:</p>
<ol>
<li>PyTorch dataset and data loader to load waveforms in batch (can be integrated with <span class="hashtag">#2</span> below)</li>
<li>Model part 1 - XLS-R pretrained feature extractor</li>
<li>Model downstream task - something simple like language classification (just depends on data/labels)</li>
<li>Training, validation, metrics, …</li>
</ol>
<pre><code class="lang-auto">    bundle = torchaudio.pipelines.WAV2VEC2_XLSR53
    model = bundle.get_model().to(args['device'])

    # Resample audio to the expected sampling rate
    sample_path = args['trainsets'][0] + os.listdir(args['trainsets'][0])[0]

    sample_waveform, sample_samplerate = torchaudio.load(sample_path)
    sample_waveform = sample_waveform.to(args['device'])
    waveform = torchaudio.functional.resample(sample_waveform,
                                              sample_samplerate, bundle.sample_rate)

    # Extract acoustic features
    with torch.inference_mode():
        features, _ = model.extract_features(waveform)

    print(len(features))
    for x, element in enumerate(features):
        print(x, element)
</code></pre>
<p>The above is what I have , it applies the feature extractor to the sample waveform and produces 24 tensors (the extracted features I believe). I need to apply it to the whole dataset in batches, and add a few more layers to the model, and fine-tune the whole model. Once this works, I also need to use the new XLS-R (larger one not in PyTorch yet). Any help is appreciated.</p>
<p>Thanks.</p>

nateanl: <p>Hi <a class="mention" href="/u/wafaa_wardah">@Wafaa_Wardah</a>, The 24 tensors are the outputs of the 24 transformer layers in the XLS-R model. You can choose one of them, or weighted-sum the tensors as the final feature. The shape should be <code>(batch, frame, feature_dim)</code>.</p>
<p>For 1) PyTorch dataset and data loader to load waveforms in batch, you can check the code of <a href="https://github.com/pytorch/audio/blob/main/torchaudio/datasets/librispeech.py#L79" rel="noopener nofollow ugc">LibriSpeech dataset</a> in torchaudio for dataset, and the <a href="https://github.com/pytorch/audio/blob/main/examples/asr/emformer_rnnt/librispeech/lightning.py#L197-L204" rel="noopener nofollow ugc">LibriSpeech ASR</a> training recipe for DataLoader usage.</p>
<p>For 3), I’m not sure what downstream task you want to apply, you can check <a href="https://github.com/s3prl/s3prl" rel="noopener nofollow ugc">S3PRL repository</a>, which aims to benchmark self-supervised learning models in various downstream tasks.</p>
<p>For 4), I recommend using PyTorch-Lightning, which provides simplified APIs for training, validation, and tensorboard support.</p>
Training a model on sparse binary BOW vector of length 7K

MJJJ: <p>I would like to compare the performance of a model that captures semantic relationships, such as a transformer, and a model trained on a bag of words vector. The BOW vector is of length 7K with about 100K examples and 21 classes.</p>
<p>Whats a reasonable starting point for architecture when dealing with a large binary vector like this as input? I’m still fairly new at pytorch so any guidance is appreciated!</p>
What should be the ideal constraint for calculating weighted loss?

Jimut123: <p>We can calculate weighted loss by defining a custom loss like this:</p>
<pre><code class="lang-auto">custom_loss = l_1 * loss_1 + l_2 * loss_2 + …. + l_N * loss_N
</code></pre>
<p>Say we are using <code>loss_1</code> as MAE and <code>loss_2</code> as MSE, etc.</p>
<p>I wanted to know that, should the constraint for l_i’s always needs to be 1 for weighted average?</p>
<p><code> l_1 + l_2 + … + l_N = 1</code></p>
<p>Or it could be greater than 1?</p>
<p>What are the pros and cons for choosing a value larger than 1?</p>
<p>Also note that I will be using Adam optimizer for this purpose.</p>

ptrblck: <p>I think the weighting can sum to any value and would just scale the loss and thus the gradients.<br>
If you’ve adapted the learning rate already to the loss scale you might want to keep the sum of the weights as <code>1</code> to avoid scaling it further.</p>

Jimut123: <p>Great, thanks for the reply. I was thinking of doing Bayesian optimization for finding the optimal weights of combined loss functions. I was worried about the domains, some researchers used 10^5 as constraints somewhere, and I was thinking if there is some standard constraints. So, increasing the constants will just increase the gradients. Hence, it is better to keep the constraints convex.</p>
<p>Also, in case of very deep architectures, will it be convenient to use number greater than 1 to minimize the vanishing gradient problem? Or it will be okay to keep the sum of the constrains = 1?</p>
Some issues when doing vector dot product

Yikai_Li1: <p>Hi everyone. Recently I got some issues when doing dot product using the following code.</p>
<blockquote>
<p>hidden_states[batch][i] @ hidden_states[batch][0]</p>
</blockquote>
<p>The answer is quite weird cause the element in the vector is no more than 1e-2 and the length is 768, but I got the answer of 100+.</p>
<p>I guess it may caused by the floating point overflow. So I try to print something and that’s what I get.</p>
<blockquote>
<p>print(hidden_states[batch][i][589], hidden_states[batch][0][589])<br>
<strong>#</strong> output:tensor(0.0248, device=‘cuda:0’, dtype=torch.float16, grad_fn=) tensor(0.0248, device=‘cuda:0’, dtype=torch.float16, grad_fn=)</p>
</blockquote>
<blockquote>
<p>print(hidden_states[batch][i][:588] @ hidden_states[batch][0][:588], hidden_states[batch][i][:589] @ hidden_states[batch][0][:589])<br>
<strong>#</strong> output:tensor(3.1836, device=‘cuda:0’, dtype=torch.float16, grad_fn=) tensor(104.6250, device=‘cuda:0’, dtype=torch.float16, grad_fn=)</p>
</blockquote>
<blockquote>
<p>print(hidden_states[batch][i][:588] @ hidden_states[batch][0][:588] + hidden_states[batch][i][589] * hidden_states[batch][0][589])<br>
<strong>#</strong> output:tensor(3.1836, device=‘cuda:0’, dtype=torch.float16, grad_fn=)</p>
</blockquote>
<p>So it seems that when I sum the first 588 elements of the product result, the answer is all right. But when I try to sum the first 589 element, I got the overflow issue and incorrect answer.</p>
<p>Is there any way to solve such issue? Thank you very much for helping.</p>

ptrblck: <p>I’m unsure if overflows are seen as I would expect to see an invalid result in this case.<br>
I assume you are not using <code>torch.amp</code> but are manually casting the tensors to <code>float16</code>?<br>
If so, could you post a minimal, executable code snippet which would reproduce the issue?</p>

Yikai_Li1: <p>Sorry bro, it seems that’s my fault to misestimate the value of the element in the vector. The value of hidden_states[batch][i][588] increases dramatically to about 10.x but waht I check is the value of  hidden_states[batch][i][589] in my code above <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20">. Anyway, thank you very much for helping.</p>
VXLAN impact on nccl broad casting

wengyao04: <p>Hi, we launch PytorchJob on our kubernetes cluster (1.20) to fine tune bert model with nccl backend <a href="https://github.com/huggingface/transformers/blob/v2.8.0/examples/run_language_modeling.py" class="inline-onebox" rel="noopener nofollow ugc">transformers/run_language_modeling.py at v2.8.0 · huggingface/transformers · GitHub</a><br>
We use <code>NCCL version 2.10.3+cuda10.2</code></p>
<p>We explicitly schedule master pod and worker pod on two different kuberntes node with v100 GPU, after we enable VXLAN we find that the data transfer time is 2-3 times slower. We then enable jumbo frame and increase MTU to 9000, but it still doesn’t not have any improvement.</p>
<p>Does anyone experience same issue with VXLAN ?</p>

ptrblck: <p>I would recommend to post this issue in the NCCL repository as it doesn’t seem to be PyTorch-specific and the NCCL devs might help out.</p>
Accumulating results with different data loaders

KanZa: <p>I want to do training with a small part of the dataset let’s say 5000 images and then I would like to add 2000 images during each cycle</p>
<p>Let’s say Training Images is 50,000<br>
Cycle 1: Training in the first step only with 5000 images (now the images will be 45,000)<br>
from cycle 2 to 9:<br>
Cycle 2: Training in the next cycle with 2000 images and accumulate the results with previous learning (now the remaining images will be 43,000 and will decrease in every cycle)</p>
<p>How can I add the results of the first training into the next training cycles from 2 to 9</p>
Normal noise into part of tensor

Handa_gou: <p>Hello,<br>
I want to input some normal noise into part of tensor of some inner layer, for example, I have hooked the first 16  of a layer tensor, How could I add the noise?<br>
I notice</p>
<p>class AddGaussianNoise(object):<br>
def <strong>init</strong>(self, mean=0., std=1.):<br>
self.std = std<br>
self.mean = mean</p>
<pre><code>def __call__(self, tensor):
    return tensor + torch.randn(tensor.size()) * self.std + self.mean

def __repr__(self):
    return self.__class__.__name__ + '(mean={0}, std={1})'.format(self.mean, self.std)
</code></pre>
<p>is adding noise to orignal input, so can you help me out in the inner layer, part of tensor? Thank you.</p>

ptrblck: <p>I’m unsure what “inner layer” means in your context, but in case you want to add noise to an intermediate activation you could write <code>AddGaussianNoise</code> as an <code>nn.Module</code> and apply it in the forward pass or you could also add the noise directly to the activation in the <code>forward</code> method.</p>

Handa_gou: <p>sorry for my expression, The inner layer what I want to mean is, for example, I have 5 layers NN, and I hook out the least second layer, in this layer I have a total of 200 nodes, and I only want to add noise to the first 16 nodes. Based on this situation, could you help me out? Thank you so much!</p>

ptrblck: <aside class="quote no-group" data-username="Handa_gou" data-post="3" data-topic="154929">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/handa_gou/48/47516_2.png" class="avatar"> Handa_gou:</div>
<blockquote>
<p>in this layer I have a total of 200 nodes, and I only want to add noise to the first 16 nodes</p>
</blockquote>
</aside>
<p>I’m not sure what “nodes” refer to, but in case it’s the output activation, you could use the previously suggested method of adding the noise directly to the output in the <code>forward</code> method.</p>
How do I encode string values in pytorch?

harshadbarge: <pre><code class="lang-auto">df_train.head():
  country            league        home_team   away_team  home_odds  draw_odds  away_odds  home_score  away_score  dow  month
0  Brazil  Copa do Nordeste     Sport Recife  Imperatriz       1.36       4.31       7.66           2           2    4      2
1  Brazil  Copa do Nordeste              ABC  America RN       2.62       3.30       2.48           2           1    6      2
2  Brazil  Copa do Nordeste  Frei Paulistano     Nautico       5.19       3.58       1.62           0           2    6      2
3  Brazil  Copa do Nordeste      Botafogo PB   Confianca       2.06       3.16       3.50           1           1    6      2
4  Brazil  Copa do Nordeste        Fortaleza       Ceara       2.19       2.98       3.38           1           1    6      2

df_test.shape:
(76544, 11)
</code></pre>
<pre><code class="lang-auto">df_test.head()
     country          league      home_team      away_team  home_odds  draw_odds  away_odds  home_score  away_score  dow  month
0      World   Club Friendly       Westerlo           Gent       2.93       3.47       2.19         NaN         NaN    4      6
1   Malaysia    Super League       Johor DT       Selangor       1.27       5.59       8.26         NaN         NaN    4      6
2  Argentina  Reserve League        Lanus 2  River Plate 2       2.54       3.12       2.65         NaN         NaN    4      6
3       Asia         AFC Cup    Bali United          Kedah       1.58       4.08       4.93         NaN         NaN    4      6
4   Ethiopia  Premier League  Defence Force     Adama City       2.93       2.16       3.38         NaN         NaN    4      6

df_test.shape:
(599, 11)
</code></pre>
<p>I perform encoding in sklearn using pandas as:</p>
<pre><code class="lang-auto">def encode_features(df_train, df_test):
    features = ['country', 'league', 'home_team', 'away_team']
    df_combined = pd.concat([df_train[features], df_test[features]])

    for feature in features:
        le = preprocessing.LabelEncoder()
        le = le.fit(df_combined[feature])
        df_train[feature] = le.transform(df_train[feature])
        df_test[feature] = le.transform(df_test[feature])
    return df_train, df_test


df_train, df_test = encode_features(df_train, df_test)
</code></pre>
<p>What is the best way to encode string values using pytorch?</p>
How to improve accuracy through stacked generalization

vishnu_vardhan1: <p>Hi,</p>
<p>I have developed two models from scratch, one with covolutions and other with vision transformers, these 2 models gave me a test accuracy of 85% and 88%. I have saved these 2 models and created an ensemble by freezing the layers, chopping of the output layers and concatenated them and added 2 dense layers and an output layer.</p>
<p>However, when I started training this ensemble its giving me a train and val accuracy of 100% for 2 epochs but my test accuracy dropped to 80%</p>
<p>I know that its overfitting, but I don’t know what else to do to improve it. Can anyone give me any suggestions on how to use those 2 models to improve the accuracy even further. I tried cross validation as well, still no improvement.</p>
<p>My dataset:<br>
370 samples for train<br>
24 samples for val<br>
60 samples for test</p>
<p>Thank you</p>
Efficient way to manipulate latent space of a pre-trained network

mojzaar: <p>Hi,<br>
I have a pre-trained Unet model that I want to do some extra calculations on its bottleneck, multiply the output of that block to a new vector, and pass the resulting vector to the decoder for reconstruction. What is the most efficient way to do that without decomposing the Unet into encoders and docoders and then assembling them back into a new module?</p>
Can't link pytorch library to existing codebase

will-leeson: <p>I’m attempting to link PyTorch to an existing fairly sophisticated tool’s code base. I need to be able to call the model based on some data that the tool generate. I’m doing this inside of the code and not externally calling a python script in an attempt to be more efficiency since I don’t want to have to incur the costs of loading the model every time I want to call it. I also need to do this with a static PyTorch library. When I attempt to build, the build makes it until the final linking phase, and it give me the following error:</p>
<p>/usr/bin/ld: -f may not be used without -shared</p>
<p>The only flag I can see with -f is -fPIC. Without adding torch’s libraries and include directories, it build’s fine and this flag still occurs. This leads me to believe there is something going wrong with PyTorch. Here is the CMakeLists.txt File I’m using to build the library which includes pytorch. I link against this library.</p>
<pre><code class="lang-auto">add_library(prediction gat.cpp)

target_include_directories(prediction 
    PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}
    PRIVATE ${TORCH_INCLUDE_DIRS}
)

target_link_libraries(prediction "${TORCH_LIBRARIES}")

set_property(TARGET prediction PROPERTY CXX_STANDARD 14)
</code></pre>

will-leeson: <p>I unfortunately could not get this working with static libraries, but dynamic libraries loaded fine. I set the CMake option of shared to true.</p>
How to inspect whethet there is NaN or Inf in gradients after amp?

songyuc: <p>Hi, guys,<br>
I want to know how to inspect whethet there is NaN or Inf in gradients after amp?</p>
<pre><code class="lang-python">optimizer.zero_grad()
with autocast():
    output = model(input)
    loss = loss_fn(output, target)
scaler.scale(loss).backward()
# To Insepct Whethet There is Infs or NaNs in Gradients After AMP?
if contain_inf_nan():
    # Do not skip if there is infs or NaNs
    raise RuntimeError
scaler.step(optimizer)
scaler.update() 
</code></pre>
<p>Your answer or guide will be appreciated!</p>

ptrblck: <p>You can inspect the <code>.grad</code> attributes of each parameter (as is also done in the <code>scaler.step()</code> to avoid updating the parameters if invalid gradients were found e.g. due to a too large scaling factor) e.g. via:</p>
<pre><code class="lang-python">if not all([torch.isfinite(p.grad) for p in model.parameters()]):
    print("invalid gradients")
</code></pre>

songyuc: <p>Thank you sincerely!</p>

ptrblck: <p>Quick follow-up in case it was missed: note that the <code>scaler.step(optimizer)</code> will already check for invalid gradients and if these are found then the internal <code>optimizer.step()</code> call will be skipped and the <code>scaler.update()</code> operation will decrease the scaling factor to avoid overflows in the next training iteration.<br>
If you are skipping these steps manually, you might get stuck with the same (high) scaling factor and might continue to run into overflows and thus invalid gradients, so keep this in mind for your use case.</p>

songyuc: <p>Appreciate for your guide. I have noticed it by using <code>apex</code>.</p>
Why few epochs in starting has more validation accuracy than training accuracy

saurav_kumar: <p>I am trying to build network with graph layer.In starting few epochs has lower loss on validation data as compared to training data.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7032fb8b88a6b97e61dd6d640c79496f6efc684.png" data-download-href="https://discuss.pytorch.org/uploads/default/b7032fb8b88a6b97e61dd6d640c79496f6efc684" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7032fb8b88a6b97e61dd6d640c79496f6efc684.png" alt="Capture" data-base62-sha1="q70cDYNIoDpl0DWQSW6VKhigyPy" width="689" height="234" data-dominant-color="E7E7E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1293×440 39.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is this code error or data issue?</p>

saurav_kumar: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48e14bdef8de63c5cc2bbb1a0a2ca56557ec5f46.png" data-download-href="https://discuss.pytorch.org/uploads/default/48e14bdef8de63c5cc2bbb1a0a2ca56557ec5f46" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48e14bdef8de63c5cc2bbb1a0a2ca56557ec5f46.png" alt="Capture" data-base62-sha1="aoJ4x4O3hwGBG7n6wLZSfJBFSGq" width="690" height="447" data-dominant-color="F0F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1051×681 28.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

saurav_kumar: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1be0a11d71f4d5e3da2c880eba8544631b2f9899.png" data-download-href="https://discuss.pytorch.org/uploads/default/1be0a11d71f4d5e3da2c880eba8544631b2f9899" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1be0a11d71f4d5e3da2c880eba8544631b2f9899.png" alt="Capture" data-base62-sha1="3YC9MSsLJ4kbmVzBht6oplCSL2h" width="557" height="500" data-dominant-color="EEEEEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">822×737 30.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Register_hook not working on slice of embedding layers

pratyush.sinha: <p>I am adding extra vocabulary tokens to embedding layer in T5 model on huggingface repo. During fine tuning , I do not want any updates on pre trained embeddings but only on added extra vocabulary tokens. I am using register_hook to set gradients to 0 on pre trained embeddings. However while fine tuning all embeddings are updated.<a href="https://colab.research.google.com/drive/1urLFzeMtOJTc0FYGpb_WNyduqfdWf4RI#scrollTo=n1_mlWRp2vb_" rel="noopener nofollow ugc">Colab notebook showing that in simple cases register_hook on slice of embedding works</a></p>
<p>However does not work on T5<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3767bbc44011f843b87f33352ba9e894d7c79f1.png" data-download-href="https://discuss.pytorch.org/uploads/default/f3767bbc44011f843b87f33352ba9e894d7c79f1" title="Screenshot 2022-06-24 at 11.49.34"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3767bbc44011f843b87f33352ba9e894d7c79f1_2_362x500.png" alt="Screenshot 2022-06-24 at 11.49.34" data-base62-sha1="yJLSenh9qvTEtgADDbajE3xtwlj" width="362" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3767bbc44011f843b87f33352ba9e894d7c79f1_2_362x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3767bbc44011f843b87f33352ba9e894d7c79f1_2_543x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3767bbc44011f843b87f33352ba9e894d7c79f1_2_724x1000.png 2x" data-dominant-color="2B2B2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-06-24 at 11.49.34</span><span class="informations">812×1120 212 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Best way to list all classification models?

ashishdatta: <p>Hi all,</p>
<p>Just curious if there is a better way to list all classification models in <code>torchvision</code> besides something like <code>torchvision.models.__dict__.keys()</code> I am trying to find something similar to <code>pytorch-image-models</code> you can do <code>timm.list_models()</code> to get a list of all available models.</p>
<p>You used to be able to query all segmentation models via something like <code>__all__ = ['fcn_resnet50', 'fcn_resnet101', 'deeplabv3_resnet50', 'deeplabv3_resnet101']</code> from <a href="https://github.com/pytorch/vision/blob/v0.6.0/torchvision/models/segmentation/segmentation.py#L8" class="inline-onebox" rel="noopener nofollow ugc">vision/segmentation.py at v0.6.0 · pytorch/vision · GitHub</a>, but I see that was removed in v0.12.0</p>
<p>Thanks!</p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x3 and 2048x1000)

donparker1: <p>I am new to AI and python, I’m trying to build an architecture for inference but up till now, I couldn’t understand how to get the inputs and outputs correctly. I keep seeing the error:</p>
<p>RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x3 and 2048x1000)</p>
<p>Any suggestions on how to resolve ?</p>
<pre><code class="lang-auto">class RN50(torch.nn.Module):

def __init__(self, rn50):
    super(RN50, self).__init__()
    self.avgpool = rn50.avgpool
    self.fc = rn50.fc

def forward(self, x):
    x = self.avgpool(x)
    x = torch.flatten(x, 1)
    x = self.fc(x)

    return x

def calibration(model_path):
    end_model=torch.jit.load(model_path)

    for i, (images, target) in enumerate(cal_loader):
       output = end_model(images) 

r50end = RN50(rn50)
x = torch.rand([112, 2048, 7, 7 ], dtype=torch.float32).contiguous()
with torch.no_grad():
    model = torch.jit.trace(r50end, x).eval()

model.save("rn50.pt")
calibration("rn50.pt")
</code></pre>

ptrblck: <p>This issue seems to be a duplicate (or at least very similar) to <a href="https://discuss.pytorch.org/t/runtimeerror-mat1-and-mat2-shapes-cannot-be-multiplied-1x1-and-2048x1000/154887">this topic</a> you’ve created and I guess both issues can be solved with the same approach so let’s follow up in the other thread.</p>
Would `torch.amp` cause a slower convergence?

songyuc: <p>Hi, guys,<br>
I want to know whether <code>torch.amp</code> would cause a slower convergence. I found after using <code>torch.amp</code>, my custom model seems to converge slower than training it without <code>torch.amp</code>.</p>
<p>Your answer and guide will be appreciated!</p>

ptrblck: <p>No, this shouldn’t be the case as seen e.g. in <a href="https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets/resnet50v1.5#example-plots">these loss curves</a> for RN50.</p>

songyuc: <p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a>, I noticed, at the beginning of the the curve, the loss convergence with FP32 seems to be a little faster than Mixed Precision, while they reach similar accuracy eventually.</p>

ptrblck: <p>That sounds reasonable assuming you are not seeing a large divergence.<br>
Using <code>amp</code> will not create bitwise-identical results to the <code>float32</code> run, so the loss curves will see a bit of jitter during the training and will not map perfectly.</p>

songyuc: <p>Thank you sincerely.</p>
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph

Harry-MG: <p>I am facing the error</p>
<p><code>RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.'</code></p>
<p>when executing the code below. I have tried setting <code>allow_unused=True</code> but this just returns a None grad. I thought that <code>torch.argmin</code> may be causing an issue, but <code>autograd.grad</code> works fine if I differentiate w.r.t the second argument, just not the first. In the setup below I have <code>n=5</code> and <code>inv_cov</code> is an inverse covariance matrix of some data.</p>
<pre><code class="lang-python">def score_fn(diagonal, invcov):
    # inputs
    # invcov: dim x dim torch array
    # diagonal: 1 x dim torch array
    # output
    # score: 1 dimensional tensor

    dim = diagonal.shape[0]
    dag = torch.zeros((dim, dim))

    for t in range(dim):
        ind = torch.argmin(torch.diag(invcov * torch.diag(diagonal)))
        dag[ind, :] = - invcov[ind, :] / invcov[ind, ind]
        dag[ind, ind] = 0
        invcov = invcov - torch.outer(invcov[:, ind].T, invcov[ind, :]) / invcov[ind, ind]
        invcov[ind, ind] = 1e6

    score = torch.linalg.norm(dag)

    return score


d = torch.ones(n)
d.requires_grad = True
sc = score_fn(d, inv_cov)
print(torch.autograd.grad(sc, d))

</code></pre>
<p>I have looked over the other forum posts for this error but the suggestions do not resolve my issue. Any help would be appreciated. Thanks!</p>

ptrblck: <p>The <code>d</code> tensor which requires gradients seems to be only used in:</p>
<pre><code class="lang-python">ind = torch.argmin(torch.diag(invcov * torch.diag(diagonal)))
</code></pre>
<p>which will detach it from the computation graph since <code>torch.argmin</code> is not differentiable.</p>