akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Combine Train and Test data

shivangi: <p>What is the best way to combine train and test data in torchvision and how can we do it?</p>
<p>Can someone explain how can we do it using <code>ConcatDataset</code> ?</p>
<p>I tried the following:<br>
<code>train = datasets.MNIST(root=dirpath, train=True, download=True, transform=trans)</code><br>
<code>test = datasets.MNIST(root=dirpath, train=False, download=True, transform=trans)</code><br>
<code>data_list = list()</code><br>
<code>data_list.append(train)</code><br>
<code>data_list.append(test)</code><br>
<code>total = ConcatDataset(data_list)</code></p>
<p>But it did not work.</p>

ptrblck: <p>Did you get an error?<br>
Your code should work.<br>
Alternatively try this:</p>
<pre><code class="lang-python">train_dataset = datasets.MNIST(root=dirpath, train=True, transform=trans)
test_dataset = datasets.MNIST(root=dirpath, train=False, transform=trans)

dataset = ConcatDataset([train_dataset, test_dataset])
</code></pre>
<p>Both codes should be equivalent, so let me know, if mine works.</p>

Ana_Beatriz: <p>Can someone explain how can we do it using <code>ConcatDataset</code> ?</p>
<pre><code class="lang-auto">  DATASET_DIR = '...'
  def load_data(data_dir=DATASET_DIR):
      transform = transforms.Compose([
          transforms.ToTensor(),
          transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
      ]),
      train_data = dataset(
          root_dir=data_dir,
          fold_nums=[1,2,3,4,5],
          transforms=transform,
          albumentations_package=False
      ),
  
      test_data = dataset(
          root_dir=data_dir,
          fold_nums=[6],
          transforms=transform,
          albumentations_package=False
      ),
      train_loader = torch.utils.data.DataLoader(
          train_data, 
          batch_size=batch_size, 
          num_workers=num_workers),
</code></pre>
<pre><code>test_loader = torch.utils.data.DataLoader(
    val_data, 
    batch_size=batch_size, 
    num_workers=num_workers)
</code></pre>
<h1>
<a name="it-is-not-working-here-with-this-code-why-1" class="anchor" href="#it-is-not-working-here-with-this-code-why-1"></a>It is not working here with this code. Why?</h1>
<p>dataset = ConcatDataset([train_data, test_data])</p>

ptrblck: <p>What kind of error are you seeing?</p>

Ana_Beatriz: <pre><code class="lang-auto">&gt; ---------------------------------------------------------------------------
&gt; NameError                                 Traceback (most recent call last)
&gt; /tmp/ipykernel_15260/1455441349.py in &lt;module&gt;
&gt;       3 criterion = nn.CrossEntropyLoss()
&gt;       4 
&gt; ----&gt; 5 dataset = ConcatDataset([train_data, test_data])
&gt;       6 
&gt;       7 num_epochs=10
&gt; 
&gt; NameError: name 'train_data' is not defined
</code></pre>

ptrblck: <p>It seems you did not execute the previous code snippet which creates <code>train_data</code>.</p>

Ana_Beatriz: <p>It looks like it, but I’ve already done that. I have already restarted the kernel.</p>
How to calculate partial columns hessian

Mrinmay_sen: <p>Hi,<br>
I am using following code to generate hessian with 10 columns only, not full hessian (dimension is d x m) and a m x m matrix M which should be symmetric.</p>
<p>import torch<br>
import warnings<br>
torch.set_default_dtype(torch.float64)<br>
import torchvision<br>
import torchvision.models as models<br>
from torchvision import transforms<br>
import torch.utils.data<br>
import torch.nn as nn<br>
import numpy as np<br>
device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)</p>
<p>transform = transforms.Compose([transforms.ToTensor(), transforms.Lambda(lambda x: x.repeat(3, 1, 1) ), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])<br>
trainset = torchvision.datasets.MNIST(root=‘data’, train=True, download=True, transform=transform)<br>
train_loader=torch.utils.data.DataLoader(trainset, batch_size=512, shuffle=True, num_workers=2)<br>
size=len(trainset)<br>
testset = torchvision.datasets.MNIST(root=‘data’, train=False, download=True, transform=transform)<br>
testloader = torch.utils.data.DataLoader(testset, batch_size=128, shuffle=True, num_workers=2)<br>
cnn = models.resnet18(pretrained=False)<br>
classifier = nn.Linear(in_features=512, out_features=10, bias=True)<br>
cnn.fc=classifier<br>
cnn=cnn.to(device)</p>
<p>loss_fn = nn.CrossEntropyLoss()<br>
p= torch.cat([gi.data.view(-1) for gi in cnn.parameters()])<br>
column_indx= np.random.choice(len(p), 20, replace= False)</p>
<p>for batch, (X, y) in enumerate(train_loader):<br>
cnn.train()<br>
wt1= torch.cat([gi.data.view(-1) for gi in cnn.parameters()]).view(-1, 1)<br>
X = X.to(device)<br>
y = y.to(device)<br>
pred=cnn(X)<br>
loss=loss_fn(pred, y) <span class="hashtag">#mean</span> loss for one batch<br>
l2_norm = sum(p.pow(2.0).sum() for p in cnn.parameters())<br>
loss = loss + l2_lambda * l2_norm<br>
env_grads = torch.autograd.grad(loss, cnn.parameters(), retain_graph=True, create_graph=True)<br>
g= torch.cat([gi.reshape(-1) for gi in env_grads])<br>
mini_batch_indx+=1<br>
C=torch.zeros(10, len(p)).to(device)<br>
for i in range(10):<br>
h_col=torch.autograd.grad(g[column_indx[i]], cnn.parameters(), retain_graph=True, create_graph=False)<br>
C[i]= torch.cat([gi.reshape(-1) for gi in h_col])</p>
<pre><code>with torch.no_grad():
    M=C[: , column_indx]
</code></pre>
<h5>
<a name="c-is-the-partial-columns-hessian-size-d-x-m-where-m-is-number-of-required-columns-and-m-which-is-formed-by-the-intersection-between-those-m-columns-of-h-1" class="anchor" href="#c-is-the-partial-columns-hessian-size-d-x-m-where-m-is-number-of-required-columns-and-m-which-is-formed-by-the-intersection-between-those-m-columns-of-h-1"></a>C is the partial columns hessian (size d x m , where m is number of required columns) and M , which is formed by the intersection between those m columns of H.</h5>
<p>If i am not wrong,  hessian is always symmetric. Then M also should be symmetric. But i am not getting symmetric.<br>
Please help me, how to get symmetric M.</p>
Hyperparameter tuning with ray tune

Ana_Beatriz: <p>Hello! I’m trying to implement an EfficientNet-B7 in this code here <a href="https://pytorch.org/tutorials/beginner/hyperparameter_tuning_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Hyperparameter tuning with Ray Tune — PyTorch Tutorials 1.12.1+cu102 documentation</a><br>
and it is not working because I don’t know how to put this ‘L1’ and ‘L2’:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, l1=120, l2=84):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, l1)
        self.fc2 = nn.Linear(l1, l2)
        self.fc3 = nn.Linear(l2, 10)
</code></pre>
<p>in the EfficientNet-B7. Also, how can I modify this part later using L1 and L2?</p>
<p><code>net = Net(config["l1"], config["l2"])</code></p>
Re-running a trained network

raesol: <p>Does re-running an already trained(run ) network affect the accuracy? if so why is it affecting it? so to be specific, I was running an image classification network, and I run the trainer part of the code and was just wondering and just re-run the same part of the cell(in google colab) and the accuracy was changed from the previous run by a bigger number( that is in the first run, the accuracy was around 45% ~ 50% in half of the epoch and reached around 68% in about 50 epochs but in the second run, the accuracy was 68% throughout the whole epoch of running)? so what is really happening there, is it like the network is saved or sth else? thank you</p>

ptrblck: <p>No, it will not affect the accuracy unless you keep the model in <code>.train()</code> mode and are thus updating e.g. the running stats of batchnorm layers etc.<br>
As long as no gradients are computed and no weight updates performed, the accuracy of the model will not change.</p>
MFCC using PyTorch

talmaashani2016: <p>Hello,<br>
I would like to generate like librosa MFCC images using TorchAudio:</p>
<p>Librosa Code:</p>
<pre><code class="lang-auto">y,sr = librosa.load (file_path)
S = librosa.feature.mfcc(y=y, sr=sr)
 img = librosa.display.specshow(S, sr=sr)
plt.axis('off')
 plt.savefig(f'image.png', bbox_inches='tight', pad_inches = 0)
plt.show()
</code></pre>
<p>I have tried using torchaudio, but it is not giving the same results:</p>
<pre><code class="lang-auto">    waveform, sample_rate = torchaudio.load (file_path)
            spectrogram_tensor = torchaudio.transforms.MFCC(sample_rate = 22050, n_mfcc = 256,   melkwargs={
                "n_mels": 256,
                "n_fft": 2048,
                "win_length": None,
                "hop_length":512,
                "mel_scale": "htk",
            }, )(waveform)
            plt.imsave (f'test.png', spectrogram_tensor[0,:,:].numpy(),  vmin = -80, vmax=0, origin="lower", cmap='viridis')


</code></pre>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/mfcc-extracterted-by-librosa-pytorch/161180/6">here</a>.</p>
Optimizer issues

raesol: <p>Does the order of optimizer.zero_grad() and optimizer.step() matter? so in the training part of the code, when i put optimizer.zero_grad() before optimizer.step() i got an accuracy of 52% and 50% for training and testing respectively. however, when i reverse the order of step and zero_grad, the accuracty increased to 99% and 68% for training and testing respectively. if zero_grad is actually zeroing the parameters, then why are we getting an accuracy as big as 50%? thank you in advance</p>

srishti-git1110: <p>The order definitely matters.</p>
<p>The rule is to zero out the previous gradients before the next backward step to prevent accumulation of gradients, and then take the step to update the parameters.</p>
<p>An order that works fine -</p>
<pre><code class="lang-auto">optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>Can you please send your two code versions that produced different results?</p>

ptrblck: <aside class="quote no-group" data-username="raesol" data-post="1" data-topic="161468">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raesol/48/54218_2.png" class="avatar"> raesol:</div>
<blockquote>
<p>if zero_grad is actually zeroing the parameters</p>
</blockquote>
</aside>
<p><code>zero_grad()</code> is setting the gradients to zero, not the parameters.</p>
<aside class="quote no-group" data-username="raesol" data-post="1" data-topic="161468">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raesol/48/54218_2.png" class="avatar"> raesol:</div>
<blockquote>
<p>then why are we getting an accuracy as big as 50%?</p>
</blockquote>
</aside>
<p>If you are working on a 2-class classification and the dataset is balanced, a 50% accuracy would be considered a “random” classifier and the worst case. A lower accuracy shouldn’t be possible since your model would then start to “learn” to predict the opposite labels.</p>
'RuntimeError: Expected object of scalar type Long but got scalar' for torch.nn.CrossEntropyLoss()

Nil_MSh: <p>Hi<br>
I’m using this loss function for xlm-roberta-large-longformer and it gives me this error:</p>
<pre><code class="lang-auto">    import torch.nn.functional as f
    from scipy.special import softmax
    
    loss_func = torch.nn.CrossEntropyLoss()
    output = torch.softmax(logits.view(-1,num_labels), dim=0).float()
    target = b_labels.type_as(logits).view(-1,num_labels)
    loss = loss_func(output, target)
    train_loss_set.append(loss.item()) 
</code></pre>
<p>when I try</p>
<pre><code class="lang-auto">b_labels.type_as(logits).view(-1,num_labels).long()
</code></pre>
<p>it tells me</p>
<blockquote>
<p>RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15</p>
</blockquote>
<p>What should I do?</p>

ptrblck: <p>For a multi-class classification use case <code>nn.CrossEntropyLoss</code> expects logits in the shape <code>[batch_size, nb_classes, *]</code> and targets as a <code>LongTensor</code> in <code>[batch_size, *]</code> containing class indices in <code>[0, nb_clsses-1]</code>. Newer PyTorch versions also allow you to pass “soft” labels as the target, but let’s skip it here.<br>
The error indicates that your target shape is wrong which is also shown via:</p>
<pre><code class="lang-python">target = b_labels.type_as(logits).view(-1,num_labels)
</code></pre>
<p>since the <code>target</code> should not contain a class dimension as previously explained.</p>
Retraining head for ssdlite-mobilenetv3

vccheng2001: <p>Hi! I have a pretrained ssdlite320_mobilenet_v3_large model with a classification/regression head.<br>
The model performs object detection + classification among several tasks. However, now I want to finetune the model so that it works on a reduced number of classes.</p>
<p>How should I go about retraining/finetuning the head part only? Is retraining even required?</p>
<p>I am trying something like this, but the output dimensions during training seem to be off.</p>
<pre><code class="lang-auto">
 model_old = SSDMobilenet(num_classes=20) # 
 model_old.load_state_dict(model_path, map_location=self.device)
      
  # New model with user-defined number of classes
  model = SSDMobilenet(num_classes=4) 
  # Use old model backbone with new classifier
  model.backbone = model_old.torchvision_model.backbone
  for param in model.parameters():
      param.requires_grad = False
  for param in model.torchvision_model.head.classification_head.parameters():
      param.requires_grad = True 
</code></pre>
Regression: at::cuda::getCurrentCUDASparseHandle() link error on linux

Mathieu_Lamarre: <p>I have a C++ extension containing the following code:</p>
<pre><code class="lang-auto">auto handle = at::cuda::getCurrentCUDASparseHandle();
</code></pre>
<p>It was working fine until version Pytorch 1.11.0, now with 1.12.1 linking with the same .so files I get <code>undefined reference to at::cuda::getCurrentCUDASparseHandle()</code> but only on my linux build. Windows is fine.</p>
<p>Did this function move to a new .so file? Maybe I need to explicitly include some header, instead of the all-in-one <code>#include &lt;torch/cuda.h&gt;</code>?<br>
Thanks!</p>

Mathieu_Lamarre: <p>Solution to the above is to add <code>--no-as-needed</code> to my CMake linker params. For some yet unknown reason, the default <code>--as-needed</code> option for <code>ld</code> would discard <code>torch_cuda_cpp.so</code> in my build with <code>1.12.1</code>.</p>
SlowFast networks Input

Gianluca_Maselli: <p>Hi,<br>
I’m trying to use SlowFast50 for video classification, however, I don’t get how properly pass the input to the pre-trained network. In fact, I’ve already extracted the desired number of frames and put it into <code>segment = torch.stack(frames,0).to(device)</code>  where <code>frames</code> is a list and each element is a tensor already normalized between 0 and 1.<br>
Can I pass then <code>segment</code> directly into <code>model(input)</code>  without performing the step <code>PackPathway</code>?<br>
Sorry but I am new into the Slowfast networks.</p>
New CUDA releases change performance

nazareem: <p>Hi,</p>
<p>I have a code base, which is stable, and where the cpu and gpu seeds are fixed.</p>
<p>I used to run this code on my 2080Ti gpu with a cuda 10.2, but recently I upgraded my cuda version to 11.6, and I noticed that the performance changed, on the majority of experiments it got better, but for one experiment it got worse. Note that nothing changed in the code, seed or hardware, only the version of cuda.</p>
<p>Is this a known issue? If so, is there any way to understand what happened?</p>
<p>Thank you in advance.</p>

ptrblck: <p>No, this is not a known issue. Could you post a reproduction example which shows the different behavior in convergence, please?</p>
Torch has no attribute 'cov'?

flying_life: <p>In new version  1.18,  torch has no attribute ‘cov’, but vervison 1.11 has torch.cov.</p>

ptrblck: <p>The latest stable release is <code>1.12.1</code> which also has <code>torch.cov</code> so unsure where <code>1.18</code> is coming from.</p>

flying_life: <p>OK.  There’s no such a problem in the latest version.  Many thanks.</p>
ValueError: `logits` and `labels` must have the same shape, received ((None, 256, 256, 4) vs (None,))

Aditi_Kumari: <p>def conv_block(input, num_filters):<br>
x = Conv2D(num_filters, 3, padding=“same”)(input)<br>
x = BatchNormalization()(x)   <span class="hashtag">#Not</span> in the original network.<br>
x = Activation(“relu”)(x)</p>
<pre><code>x = Conv2D(num_filters, 3, padding="same")(x)
x = BatchNormalization()(x)  #Not in the original network
x = Activation("relu")(x)

return x
</code></pre>
<p>def encoder_block(input, num_filters):<br>
x = conv_block(input, num_filters)<br>
p = MaxPool2D((2, 2))(x)<br>
return x, p</p>
<p>def decoder_block(input, skip_features, num_filters):<br>
x = Conv2DTranspose(num_filters, (2, 2),strides=2,padding=“same”)(input)<br>
x = Concatenate()([x, skip_features])<br>
x = conv_block(x, num_filters)<br>
return x</p>
<p>def build_unet(input_shape, n_classes):<br>
inputs = Input(input_shape)</p>
<pre><code>s1, p1 = encoder_block(inputs, 64)
s2, p2 = encoder_block(p1, 128)
s3, p3 = encoder_block(p2, 256)
s4, p4 = encoder_block(p3, 512)

b1 = conv_block(p4, 1024) #Bridge

d1 = decoder_block(b1, s4, 512)
d2 = decoder_block(d1, s3, 256)
d3 = decoder_block(d2, s2, 128)
d4 = decoder_block(d3, s1, 64)

if n_classes == 1:
  activation = 'sigmoid'
else:
  activation = 'softmax'

outputs = Conv2D(n_classes, 1, padding="same", activation=activation)(d4)  


model = Model(inputs, outputs, name="U-Net")


return model
</code></pre>
<p>my_unet = build_unet(input_shape=(256,256,3), n_classes=4)</p>
<p>print(my_unet.summary())</p>
<p>my_unet.compile(optimizer=tf.keras.optimizers.Adam(), loss=‘binary_crossentropy’, metrics=[‘accuracy’])</p>
<p>history = my_unet.fit(<br>
train_ds,<br>
batch_size=BATCH_SIZE,<br>
validation_data=val_ds,<br>
verbose=1,<br>
epochs=50,</p>
<p>)</p>
<p>can somebody help me with this code</p>

ptrblck: <p>It seems you are using <code>Keras</code> so I would recommend to post the question in the <a href="https://discuss.tensorflow.org/">TensorFlow discussion board</a> as you will find the experts there.<br>
Based on the error message I would recommend to check the model output and compare it to the targets as it seems your loss function expects the same shapes for both.</p>
How to use own model for transfer learning

exponential: <p>I have a model whose last layers are like seen below</p>
<pre><code class="lang-auto">'
'
'
out = self.relu(self.bn1(out))
out = F.avg_pool2d(out, 8)
out = out.view(-1, self.nChannels)
out = self.fc(out)
return out
</code></pre>
<p>The last layer’s output is of <code>torch size [18, 2048]</code><br>
I have trained the model and have a .pth file, I would like to do finetuning/transfer learning for a downstream task by adding a Linear layer and then train.<br>
What is the right way to do this?<br>
Do I initialize the whole network first and thereafter load the model_state_dict or just load the model from path?</p>
<p>Essentially, I would like to have something like this for the transfer learning/downstream training:</p>
<pre><code class="lang-auto">import torch.nn as nn

model = Net(args1, args2, args3)
model = model.load_state_dict(torch.load(PATH))
model.eval()

class final_network(nn.Module):
    def __init__(self, use_face=False, num_glimpses=1):
        super(final_network, self).__init__()
        self.pre_network = model

        self.final_task = nn.Sequential(
            nn.Linear(2048, 468),
        )

    def forward(self, x):
        feature = self.pre_network(x)
        feature = feature.view(feature.size(0), -1)
        out = self.final_task(feature)

        return out
</code></pre>
<p>Do I use the entire model (checkpoint) as seen here <a href="https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_models_for_inference.html#save-and-load-entire-model" class="inline-onebox" rel="noopener nofollow ugc">Saving and loading models for inference in PyTorch — PyTorch Tutorials 1.12.1+cu102 documentation</a> ? I am a little confused as all the info found online are for popular network architecture.</p>
<p>Many thanks.</p>

ptrblck: <aside class="quote no-group" data-username="exponential" data-post="1" data-topic="161386">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/exponential/48/50889_2.png" class="avatar"> exponential:</div>
<blockquote>
<p>Do I initialize the whole network first and thereafter load the model_state_dict or just load the model from path?</p>
</blockquote>
</aside>
<p>I would recommend to initialize the model objects first, then to load their <code>state_dict</code>s. Storing and loading the model directly can break in many different ways and I would thus not use it.</p>
<p>Your code snippet looks alright, but I would also explicitly pass <code>model</code> as an argument to <code>final_network.__init__</code>.</p>
"predict_proba"

Zuly: <p>Hello,<br>
I am working multi-class. When I study at Keras I can use “predict_proba” function for can see probability of  every class.<br>
Model is Sequential() and I used CNN.<br>
code: y_score = model.predict_proba(testX)</p>
<p>I want to learn, is there a function at Pytorch like “predict_proba” .</p>
<p>Thank you for helping</p>

ptrblck: <p>Assuming you are working on a multi-class classification use case, you can pass the input to the model directly and check the logits, calculate the probabilities, or the predictions:</p>
<pre><code class="lang-python">model.eval()
logits = model(data)
probs = F.softmax(logits, dim=1) # assuming logits has the shape [batch_size, nb_classes]
preds = torch.argmax(logits, dim=1)
</code></pre>

Zuly: <p>Thank you so much. I will try and return back. You are amazing man.</p>

ahmediqbal: <p>And what will be the code, if we have single class problem, and using torch.<strong>sigmoid</strong>() ?</p>

ptrblck: <p>In that case you would use a threshold to get the predictions using a desired probability.<br>
E.g.:</p>
<pre><code class="lang-python">probs = torch.sigmoid(output)
preds = probs &gt; 0.5 # adapt threshold if needed
</code></pre>

ahmediqbal: <p>Thanks for reply,</p>
<p>Because i want to compute sklearn.metrics.roc_curve(), so i am using my code like this, it is right?</p>
<blockquote>
<p>output = torch.sigmoid( model(image) )</p>
<p>y_pred = output.cpu().numpy().flatten() &gt;= 0.5</p>
<p>y_true = mask.flatten()</p>
<p>roc_curve(y_true , y_pred)</p>
</blockquote>

ptrblck: <p><code>roc_curve</code> expects the scores as an input, not the predictions, since the curve itself will be created using different thresholds.<br>
From the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html">docs</a>:</p>
<blockquote>
<p>y_score <code>ndarray</code> of shape (n_samples,)<br>
Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).</p>
</blockquote>

ahmediqbal: <p>According to scikit-learn</p>
<blockquote>
<p>Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).</p>
</blockquote>
<p>From Pytorch what output i can expect as scores? i think torch.sigmoid return probabilities?</p>

ptrblck: <p>Yes, you could pass the probabilities into this method created by <code>sigmoid</code> without applying the threshold.</p>

alijawad07: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> I have a multi class classification setup and I want to deal with out of distribution detection, my model gives unrealistically high confidence for out of distribution as well. I am fairly new in this and I was hoping if you could help me with this? thanks</p>
Pytorch t-shirts?

nafizh1: <p>Sorry for the non-technical post here. I love pytorch. As a researcher who uses DL rather than do research in DL itself, pytorch was so easy to integrate into my workflow.</p>
<p>I was wondering if there are pytorch t-shirts available anywhere to buy or if there is a non-profit project where the donation buys you a t-shirt, and a portion also goes to some worthy cause.</p>

AbdulsalamBande: <p>Check out <a href="https://www.redbubble.com/shop/pytorch+t-shirts" rel="noopener nofollow ugc">https://www.redbubble.com/shop/pytorch+t-shirts</a></p>

SteveDigital: <p>Here’s also one I bought recently <a href="https://www.artificial-intelligence.store/store/p/pytorch-t-shirt-unisex" rel="noopener nofollow ugc">https://www.artificial-intelligence.store/store/p/pytorch-t-shirt-unisex</a></p>

gug: <p>Now that pytorch is officially part of Linux foundation, hopefully we could soon find official merchandise on <a href="https://linuxfoundation.store/" rel="noopener nofollow ugc">store</a>.</p>
3d medical images classifications

CHOONG_HONG_LEE: <p>I am trying to classify medical dicom images using Conv3D. I have a few questions, not sure if I am asking the question in the right way. The image size is (512 x 512 x 3 channels). Each scan has no of slices 28 - 40 slices in DICOM format, and I have around 500 datasets</p>
<ol>
<li>How should I structure the datasets? Is a absent or present classification for the medical images. I understand if simple 2D images (for e.g. dog and cat, I can put the right images in the respective folders). But in my case, I have a stack of 2D images for one scan.</li>
<li>The no of slices varies for all the scans, what is the correct way to use the dataloader?</li>
</ol>
<p>My questions may not be very clear, but I am more than happy to give more clarification if needed. Or if there is any example that will be great.</p>

my3bikaht: <p>I think you can look here for a starting point</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://stanfordmlgroup.github.io/projects/mrnet/">
  <header class="source">
      <img src="https://stanfordmlgroup.github.io/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://stanfordmlgroup.github.io/projects/mrnet/" target="_blank" rel="noopener nofollow ugc">stanfordmlgroup.github.io</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/361;"><img src="https://stanfordmlgroup.github.io/projects/mrnet/img/team_fb.jpg" class="thumbnail" width="690" height="361"></div>

<h3><a href="https://stanfordmlgroup.github.io/projects/mrnet/" target="_blank" rel="noopener nofollow ugc">Deep learning to assist clinicians in reading knee MRI</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


CHOONG_HONG_LEE: <p>Thanks, will take a look</p>

fepegar: <p>I know it’s a bit late, but you can take a look at TorchIO: <a href="https://torchio.readthedocs.io/" rel="noopener nofollow ugc">https://torchio.readthedocs.io/</a></p>
RuntimeError: grad can be implicitly created only for scalar outputs in one GPU

E4CE: <p>Hello, i have encountered this error.</p>
<p>I am trying to train my own dataset and encountered this error.</p>
<pre><code class="lang-auto">Ignoring class  0  in IoU evaluation
[IOU EVAL] IGNORE:  tensor([0])
[IOU EVAL] INCLUDE:  tensor([1, 2])
Lr: 3.106e-05 | Update: 2.258e-01 mean,4.181e-01 std | Epoch: [0][0/322] | Time 3.170 (3.170) | Data 0.154 (0.154) | Loss 1.9250 (1.9250) | acc 0.533 (0.533) | IoU 0.363 (0.363) | [1 day, 20:35:54]
Traceback (most recent call last):
  File "/content/LiDAR-MOS/mos_SalsaNext/train/tasks/semantic/train.py", line 178, in &lt;module&gt;
    trainer.train()
  File "../../tasks/semantic/modules/trainer.py", line 274, in train
    show_scans=self.ARCH["train"]["show_scans"])
  File "../../tasks/semantic/modules/trainer.py", line 391, in train_epoch
    loss_m.backward()
  File "/usr/local/lib/python3.7/dist-packages/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py", line 166, in backward
    grad_tensors_ = _make_grads(tensors, grad_tensors_, is_grads_batched=False)
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py", line 67, in _make_grads
    raise RuntimeError("grad can be implicitly created only for scalar outputs")
RuntimeError: grad can be implicitly created only for scalar outputs
</code></pre>
<pre><code class="lang-auto">  optimizer.zero_grad()
            if self.n_gpus &gt; 1:
                idx = torch.ones(self.n_gpus).cuda()
                loss_m.backward(idx)
            else:
                loss_m.backward() #here i got the error
            optimizer.step()
</code></pre>
<p>I have looked the error in google, and it usually happens when you use two or more GPUs. However, I am using only one GPU and got this error. Also, when I try to train it with my dataset 1 which has around 400 frames, it works. However, when I try to train it with dataset 2 which has around 2400 this throws an error.<br>
Could you please help me to solve this error?</p>

ptrblck: <p>The error is raised if the <code>loss</code> tensor contains more than a single element. You would have to either pass the gradient directly to the <code>backward</code> call or reduce the tensor first e.g. via <code>loss_m.mean().backward()</code>.</p>
<p>In any case, you should also try to figure out why the shape of the <code>loss_m</code> tensor changes as it seems to be unexpected.</p>
Can I set torch.randint's requires_grad=True?

Marshall_stan: <p>In the PyTorch 1.12 documentation, <strong>torch.randint</strong> have the parameter-<strong>requires_grad=False</strong> as  default, but as I know <strong>only</strong> the “floatTensor” can set the requires_grad=True.Does anybody explain this issue for me or <strong>show me how to autograd the intTensor</strong> generated by “<strong>torch.randint</strong>”?</p>

srishti-git1110: <p>Hi,<br>
Is this relevant to something that you are looking for?</p>
<pre><code class="lang-auto">import torch
x = torch.randn(3, requires_grad=True)
x.data = torch.randint(5, (3,))
print(x)
print(x.dtype)
</code></pre>
<p>out -</p>
<pre><code class="lang-auto">tensor([2, 1, 0], requires_grad=True)
torch.int64
</code></pre>
<p>Edit: As a disclaimer, I don’t think this is an intended behaviour though. Looks like a bug to me.<br>
See -</p>
<pre><code class="lang-auto">y = (x+ float(5.0)).sum()
y.backward()
</code></pre>
<p>gives an error -</p>
<pre><code class="lang-auto">RuntimeError: isDifferentiableType(grad.scalar_type()) INTERNAL ASSERT FAILED at "../torch/csrc/autograd/engine.cpp":746, please report a bug to PyTorch. 
</code></pre>

ptrblck: <p>Manipulating the <code>.data</code> attribute can yield unwanted side effects and is deprecated.<br>
The argument might just be there for consistency reason to match other calls (that would be my guess at least), but you are right that integer tensors cannot require gradients:</p>
<pre><code class="lang-python">x = torch.randint(3, 5, (3,), requires_grad=True)
# RuntimeError: Only Tensors of floating point and complex dtype can require gradients
</code></pre>
Can't use PyTorch from latest docker image

Diego: <p>When I try to run PyTorch using the latest docker image (<a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch" rel="noopener nofollow ugc">nvcr.io/nvidia/pytorch:22.08-py3</a>) it breaks on <code>import torch</code> giving the following stacktrace:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "/mnt/home/.local/lib/python3.8/site-packages/torch/__init__.py", line 811, in &lt;module&gt;
    from .functional import *  # noqa: F403
  File "/mnt/home/.local/lib/python3.8/site-packages/torch/functional.py", line 7, in &lt;module&gt;
    import torch.nn.functional as F
  File "/mnt/home/.local/lib/python3.8/site-packages/torch/nn/__init__.py", line 1, in &lt;module&gt;
    from .modules import *  # noqa: F403
  File "/mnt/home/.local/lib/python3.8/site-packages/torch/nn/modules/__init__.py", line 2, in &lt;module&gt;
    from .linear import Identity, Linear, Bilinear, LazyLinear
  File "/mnt/home/.local/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 7, in &lt;module&gt;
    from .. import functional as F
  File "/mnt/home/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 18, in &lt;module&gt;
    from .._jit_internal import boolean_dispatch, _overload, BroadcastingList1, BroadcastingList2, BroadcastingList3
  File "/mnt/home/.local/lib/python3.8/site-packages/torch/_jit_internal.py", line 25, in &lt;module&gt;
    import torch.distributed.rpc
  File "/mnt/home/.local/lib/python3.8/site-packages/torch/distributed/__init__.py", line 55, in &lt;module&gt;
    from .distributed_c10d import *  # noqa: F403
  File "/mnt/home/.local/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 188, in &lt;module&gt;
    reduce_op = _reduce_op()
  File "/mnt/home/.local/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 176, in __init__
    for k, v in ReduceOp.__members__.items():
AttributeError: type object 'torch._C._distributed_c10d.ReduceOp' has no attribute '__members__'
</code></pre>
<p>any ideas?</p>

ptrblck: <p>It’s working for me:</p>
<pre><code class="lang-python">Status: Downloaded newer image for nvcr.io/nvidia/pytorch:22.08-py3

=============
== PyTorch ==
=============

NVIDIA Release 22.08 (build 42105213)
PyTorch Version 1.13.0a0+d321be6
...
root@99ec9b88acce:/workspace# python 
Python 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10) 
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__path__
['/opt/conda/lib/python3.8/site-packages/torch']
&gt;&gt;&gt; 
</code></pre>
<p>Could you check if you are mounting any folders which might overwrite oder files (and maybe the installation) inside the container? If not, did you install any packages, which might do the same?</p>

Diego: <p>Silly me, I was exporting my $PATH variable to the container, making it believe that python was installed somewhere else, whoops!</p>
Why `atomicAdd` cause non-determinstic

Rita: <p>I found in pytorch source code that some operations are non-deterministic because they use <code>atomicAdd</code>, like this one <a href="https://github.com/pytorch/pytorch/blob/3b78c5682b483086f66d875749f94b7551072a05/aten/src/ATen/native/cuda/NLLLoss2d.cu#L232" rel="noopener nofollow ugc">pytorch/NLLLoss2d.cu at 3b78c5682b483086f66d875749f94b7551072a05 · pytorch/pytorch (github.com)</a></p>
<p>Does anyone know why <code>atomicAdd</code> can cause non-deterministic. Thanks!</p>

ptrblck: <p>The order of summands is not known and thus causes the non-deterministic behavior.<br>
While a simple addition is deterministic: <code>y = x1 + x2</code>, it’s not if more summands can change their order of execution: <code>y = x1 + x2 + x3 + x4</code>.</p>
Gradient computation when using forward hooks

Naman-ntc: <p>Suppose I have a custom <code>nn.Module</code> –</p>
<pre><code class="lang-auto">class Identity(nn.Module):
    def __init__(self):
        pass
    def forward(x):
        return x

hooked_layer = Identity()
hookfn = lambda model,input,output: output*2    
### hookfn can in principle be complex function 
### even non differentiable such as quantization 
hooked_layer.register_forward_hook(hookfn)
 
</code></pre>
<p>Now consider activation <code>A</code>, which passes through <code>hooked_layer</code> to become A_hooked. I understand that in the computation of loss, <code>A_hooked</code> is used.<br>
Now I want to understand how will back-propagation happen.<br>
1.) How will the gradient of <code>A</code> be computed? Will gradient of <code>A_hooked</code> be copied to <code>A</code> or will gradient of <code>A</code> equal to half of the gradient of <code>A_hooked</code>.<br>
2.) In the above case if gradient of <code>A_hooked</code> is not copied to gradient of <code>A</code> – then what will happen if I use some non-differentiable hook – such as quantization<br>
3.) Lastly in the layers following our original <code>hooked_layer</code>, will <code>A</code> or <code>A_hooked</code> be used in backpropagation</p>

albanD: <p>Hi,</p>
<p>I think the simplest way to understand what will happen here is to know that the autograd lives below torch.nn and is completely unaware of what torch.nn does.<br>
So in this case, whatever is the Tensor you give to the rest of the net is the one that will get gradients (it does not matter if it comes from a hook or not).<br>
And in this case, since A_hooked depends on A, then the gradients will flow back from A_hooked to A.</p>

goksinan: <p>Can somebody confirm that forward hook supports backpropagation? For example, I obtain activations (or weights) from particular layers using forward hooks. Then, I compute the <em>L1 norm</em> of these activations (or weights) and added it to my main loss term, i.e. <code>loss = loss + L1_loss</code>. Now, when I call <code>loss.backward()</code> the gradients will flow through the hooks and then the activations (or weights) will be penalized accordingly right?<br>
Another option I came across was iterating over model parameters (maybe with some if-else statements), which should work just fine, but it doesn’t provide the flexibility I am looking for.</p>
Split dataset (always same images)

Alphonsito25: <p>Hi.</p>
<p>I would like to split my dataset between training and test, but I don’t want it to be done randomly, like:</p>
<p>torch.utils.data.random_split(dataset, [train_size, val_size])</p>
<p>This is because I want to perform several trainings under the same conditions (test images always the same in each training).</p>
<p>Is there a pytorch function that does it?</p>
<p>Thank you very much!</p>

ptrblck: <p>You can wrap your dataset into a <code>Subset</code> and pass the corresponding indices to them to create the training and validation splits.</p>

Alphonsito25: <p>I am sorry but, how can I put this into code?</p>

ptrblck: <p>Here is an example:</p>
<pre><code class="lang-python">dataset = TensorDataset(torch.arange(10))
train_dataset = torch.utils.data.Subset(dataset, indices=torch.arange(5))
val_dataset = torch.utils.data.Subset(dataset, indices=torch.arange(5, 10))

for d in train_dataset:
    print(d)
    
for d in val_dataset:
    print(d)
</code></pre>

Alphonsito25: <p>Perfect! Thank you very much!</p>
RuntimeError: Expected hidden[0] size (1, 13, 128), got [13, 128]

Jul14n: <p>Good day, I am trying to build a seq2seq LSTM model but get the following error from the code snippet of my decoder:</p>
<pre><code class="lang-auto">print("entered")
print(h0.shape)
output, (h,c) = self.lstm(seq_embed, h0)
output_pred = self.fc(output)
ans = output_pred.argmax(-1)
</code></pre>
<p>Where h0 is simply obtained from my encoder through:</p>
<pre><code class="lang-auto">output, (h,c) = self.lstm(seq_embed, h0)
</code></pre>
<p>which yields the following when printed:</p>
<pre><code class="lang-auto">entered
torch.Size([1, 13, 128])
</code></pre>
<p>It complains about my h0 tensor being of the wrong size but the from the printed outputs below my dimensions seem to be correct so not sure how to proceed further?</p>
<p>Upon investigating some of the other errors that are displayed I came upon this section below. My 2nd question is does my h0 tensor require two “layers” for hidden[0] and hidden[1] which would essentially require it to be a (2,13,128)? Any assistance would be apprecriated</p>
<pre><code class="lang-auto">--&gt; 533 self.check_hidden_size(hidden[0], expected_hidden_size,
    534                        'Expected hidden[0] size {}, got {}')
    535 self.check_hidden_size(hidden[1], expected_hidden_size,
    536                        'Expected hidden[1] size {}, got {}')
</code></pre>
PINN does not follow boundary condition

yviiendaleee: <p>Hi everybody,<br>
I am using the code <a href="https://github.com/maziarraissi/PINNs" rel="noopener nofollow ugc">GitHub - maziarraissi/PINNs: Physics Informed Deep Learning: Data-driven Solutions and Discovery of Nonlinear Partial Differential Equations</a> as a base.<br>
Instead of the burgers equation I am using the fick’s law of diffusion to calculate the concentration.</p>
<p>I don’t have any measured data so my PINN only uses the initial condition, the boundary conditions and the fick’s law of diffusion as physical losses.<br>
However my result of the concentration does not follow the change over time, only the change of the radial coordinate.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/1/91e4d76d15fb7a877d09c0eef04fb7aca5a70755.png" data-download-href="https://discuss.pytorch.org/uploads/default/91e4d76d15fb7a877d09c0eef04fb7aca5a70755" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/1/91e4d76d15fb7a877d09c0eef04fb7aca5a70755.png" alt="image" data-base62-sha1="kODwtOypPze855VTDzzgxTO4lrn" width="690" height="368" data-dominant-color="303343"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">850×454 17.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
These are my initial and boundary conditions. To make it easier, I used normalized values (from 0 to 1).</p>
<p>My training data is based on the initial and boundary conditions, my testing data are random values.</p>
<p>What could be the problem why my model does not follow the maximum boundary condition? What is the reason the concentration does not reach the value 1?</p>
<p>Thank you in advance</p>
How does Quantization convert method

Rami_Ismael: <p>I have successfully called torch.ao.quantization.convert properly. However, I thought the converted model would transform my nn.layer and nn.Conv2d with the weight as int8 datatype. Instead, my layers are QuantizeLinear and QuantizeConv.</p>

ddang: <p>see <a href="https://pytorch.org/docs/stable/quantization.html" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.1 documentation</a>, in particular,</p>
<blockquote>
<h1>Convert the observed model to a quantized model. This does several things:</h1>
<h1>quantizes the weights, computes and stores the scale and bias value to be</h1>
<h1>used with each activation tensor, and replaces key operators with quantized</h1>
<h1>implementations.</h1>
<p>model_int8 = torch.quantization.convert(model_fp32_prepared)</p>
</blockquote>
C++ extension is slower than python

fangwei123456: <p>Hi everyone, I write a C++ extension to accelerate, but I find C++ is slower than python. Here are codes:<br>
test.cpp:</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;torch/extension.h&gt;
#include &lt;math.h&gt;
using namespace torch::autograd;

torch::Tensor heaviside_step(const torch::Tensor &amp; x)
{   
    return x.ge(0).to(x);  // bool -&gt; float
}
class sigmoid_atf: public Function&lt;sigmoid_atf&gt;
{
    public:
  static torch::Tensor forward(AutogradContext *ctx, const torch::Tensor &amp; x, const torch::Tensor &amp; alpha)
  {
      if (x.requires_grad())
      {
        ctx-&gt;save_for_backward({x, alpha});
      }
      return heaviside_step(x);
  }
  static tensor_list backward(AutogradContext *ctx, tensor_list grad_outputs)
  {
      auto saved = ctx-&gt;get_saved_variables();
      auto x = saved[0];
      auto alpha = saved[1];
      auto grad_x = alpha * torch::sigmoid_backward(grad_outputs[0], torch::sigmoid(x * alpha));  // main contribution to acceleration
      return {grad_x, torch::Tensor()};
  }
};

torch::Tensor sigmoid_apply(const torch::Tensor &amp; x, const torch::Tensor &amp; alpha)
{
    return sigmoid_atf::apply(x, alpha);
}

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
    m.def("sigmoid_apply", &amp;sigmoid_apply);
}
</code></pre>
<p>test.py:</p>
<pre><code class="lang-auto">from torch.utils import cpp_extension
import torch
import time
import numpy as np
def heaviside_step(x: torch.Tensor):
    return x.ge(0).to(x)

class sigmoid(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, alpha):
        if x.requires_grad:
            ctx.save_for_backward(x, alpha)
        return heaviside_step(x)

    @staticmethod
    def backward(ctx, grad_output):
        grad_x = None
        if ctx.needs_input_grad[0]:
            sgax = (ctx.saved_tensors[0] * ctx.saved_tensors[1]).sigmoid_()
            grad_x = grad_output * (1 - sgax) * sgax * ctx.saved_tensors[1]

        return grad_x, None
    
def cal_fun_t(n, f, *args, **kwargs):
    # warm up
    f(*args, **kwargs)
    torch.cuda.synchronize()
    t_start = time.time()
    for _ in range(n):
        f(*args, **kwargs)
    torch.cuda.synchronize()
    return (time.time() - t_start) / n

cext = cpp_extension.load(name='test',
                          sources=['./test.cpp'], verbose=True)

device = 'cuda:0'
x = torch.rand([1024], device=device)
alpha = torch.rand([1], device=device)
with torch.no_grad():
    t1 = cal_fun_t(100, sigmoid.apply, x, alpha)
    t2 = cal_fun_t(100, cext.sigmoid_apply, x, alpha)
    print(f'python:{t1}, c++:{t2}')
</code></pre>

woong97: <p>I have similar problem with you…<br>
Did you resolve this problem?</p>

fangwei123456: <p>Hi, I have not solved this problem.</p>