akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to calculate sensitivity, precision, recall and F1 score of my binary dataset with output labels 0 and 1

AP_M: <p>How can I calculate sensitivity, precision, recall and F1 score of my binary dataset. The output values are 0 and 1.</p>
<pre><code class="lang-auto">model = Net()
optimizer = optim.Adam(model.parameters(), lr=0.001)
loss_func = nn.NLLLoss()
criterion = nn.CrossEntropyLoss()  

epochs = 2
loss_list = []

model.train()
for epoch in range(epochs):
    total_loss = []
    for i, data in enumerate(train_ldr, 0):
  # get the inputs; data is a list of [inputs, labels]
      X_train,Y_train = data.values()
      X_train = X_train.unsqueeze(0)
      X_train = X_train.unsqueeze(1)
    
      optimizer.zero_grad()
        # Forward pass
      output = model(X_train)
      # Calculating loss
      loss = criterion(output, Y_train) 
        
       # Backward pass
      loss.backward()
        # Optimize the weights
      optimizer.step()
        
      total_loss.append(loss.item())
    loss_list.append(sum(total_loss)/len(total_loss))
    print('Training [{:.0f}%]\tLoss: {:.4f}'.format(
        100. * (epoch + 1) / epochs, loss_list[-1]))

model.eval()
criterion = nn.CrossEntropyLoss()
with T.no_grad():
     
    correct = 0
    for i, data in enumerate(test_ldr, 0):
        
        X_test,Y_test = data.values()
        X_test = X_test.unsqueeze(0)
        X_test = X_test.unsqueeze(1)
        output = model(X_test)
        
        pred = output.argmax(dim=1, keepdim=True) 
        correct += pred.eq(Y_test.view_as(pred)).sum().item()
        
        loss = criterion(output, Y_test)
        total_loss.append(loss.item())
        
    print('Performance on test data:\n\tLoss: {:.4f}\n\tAccuracy: {:.1f}%'.format(
        sum(total_loss) / len(total_loss),
        correct / len(test_ldr) * 100)
        )
</code></pre>

sh0416: <p>In this case, I recommend you use the scikit-learn package for computing some evaluation metric.</p>
<p>It would be great unless you have to accelerate the evaluation process due to the large data.</p>
Using autoencoder and classifier to reduce crossdataset difference of CT-Scans

Stebi: <p>Hello there,<br>
In my recent research i tried to minimize the corssdataset difference of CT-Scans made by different CT-Maschines in different countries with the goal of making the data perform better when used together to train an Classifier.<br>
For this i tried using an classic VGG16 Autoencoder and an imaging technoloy classifier to train this autoencoder before actually training the medical Classifier.<br>
Our result are sadly quite unstable and we can not achieve good performance with same and cross dataset images at the same time.<br>
Is the approach generaly flawed or do you have some recommondations and tipps ?</p>
DDP on 8 GPUs vs. Single GPU training speed

FruitVinegar: <p>Hi.<br>
As I mentioned at title, I trained my model in 2 different device environments to compare training speed.<br>
I used torchvision official resnet50 model and CIFAR10 for dataset, which is enough small to run in single GPU.<br>
I found that DDP on 8 GPUs are about 2x slower than single GPU.<br>
Is this expected for small models? Or am I using something wrong with DDP?</p>
<p>+)<br>
I used same parameters for 2 environments, including batch size. Should I incrase batch size for 8 GPUs DDP 8x times? Does this (hopefully always) guarantees similar accuracy with single GPU cases?</p>

mrshenli: <blockquote>
<p>I used same parameters for 2 environments, including batch size. Should I incrase batch size for 8 GPUs DDP 8x times? Does this (hopefully always) guarantees similar accuracy with single GPU cases?</p>
</blockquote>
<p>If this is per-process batch size, DDP batch size should actually be 1/8 compared to local training, so that DDP and local both collectively process the same number of samples in each iteration.</p>
<p>If this is already global batch size, can you try to increase the batch size for both DDP and local trianing and see how that changes the perf numbers? DDP would run allreduce on model gradients. So if the batch size is too small, the communication overhead can overshadow the speedup from parallelizing computations.</p>

headkit: <p>Maybe you can find a solution together:</p><aside class="quote quote-modified" data-post="3" data-topic="121624">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wayi/48/32374_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/distributeddataparallel-training-not-efficient/121624/3">DistributedDataParallel training not efficient</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    Very interesting project! 

So basically training with 4 GPUS needs 4 epochs to get the same results like a single GPU achieves in only 1 epoch. 

This is not true if you consider the sync among 4 GPUs per epoch. It should be equivalent to running 4 epochs on a single GPU. 
Can you confirm if there is any communication between different processes (by printing the gradient values of different ranks after backward)? Gradients of different ranks should be the same after backward. 
Additionally, you…
  </blockquote>
</aside>
Logic of Group Formation in Group Normalization

moutan: <p>For my purposes I want to apply Group Normalization to three specific channel groups of a tensor. This groups are contiguous so for a Tensor with dimensions <code>(N,C,H,W)</code>. Group 1 would be<code> (N, : C /3,H,W)</code>, Group 2<code> (N, C/3: 2C/3, H,W)</code> and Group 3 <code>(N; 2C/3 : C, H , W)</code>. I tried looking in the code base but did not find what the logic of group creation  is.</p>
<p>Can someone explain to me the logic for the formation of groups in GroupNorm? Does it match my contiguous scenario?</p>

ptrblck: <p>Your assumption seems to be correct based on this small experiment:</p>
<pre><code class="lang-python">N, H, W = 16, 224, 224

# increasing groups
norm = nn.GroupNorm(num_groups=3, num_channels=6)
x = torch.cat((
    torch.randn(N, 2, H, W) * 2 + 5,
    torch.randn(N, 2, H, W) * 4 + 10,
    torch.randn(N, 2, H, W) * 6 + 15,
), dim=1)

out = norm(x)
for i, o in enumerate(out.split(1, dim=1)):
    print("channel {}: mean: {:.5f}, {:.5f}".format(i, o.mean(), o.std()))
&gt; channel 0: mean: 0.00021, 0.99910
  channel 1: mean: -0.00021, 1.00089
  channel 2: mean: -0.00071, 0.99964
  channel 3: mean: 0.00071, 1.00036
  channel 4: mean: -0.00172, 1.00064
  channel 5: mean: 0.00172, 0.99936

# mix channels
norm = nn.GroupNorm(num_groups=3, num_channels=6)
x = torch.cat((
    torch.randn(N, 1, H, W) * 2 + 5,
    torch.randn(N, 1, H, W) * 1/2 - 5,
    torch.randn(N, 1, H, W) * 4 + 10,
    torch.randn(N, 1, H, W) * 1/4 - 10,
    torch.randn(N, 1, H, W) * 6 + 15,
    torch.randn(N, 1, H, W) * 1/6 - 15,
), dim=1)

out = norm(x)
for i, o in enumerate(out.split(1, dim=1)):
    print("channel {}: mean: {:.5f}, {:.5f}".format(i, o.mean(), o.std()))
&gt; channel 0: mean: 0.96000, 0.38414
  channel 1: mean: -0.96000, 0.09611
  channel 2: mean: 0.96217, 0.38453
  channel 3: mean: -0.96217, 0.02406
  channel 4: mean: 0.96211, 0.38544
  channel 5: mean: -0.96211, 0.01068
</code></pre>
<p>As you can see, the first approach gives a standardized (zero mean, unit variance) output per channel, while the second approach using the mixed stats does not.</p>

moutan: <p>Thank you for your reply. Yes, it seems almost certain the group formation is done for contiguous channels.</p>
Image Detection with Coordinate as Label

sleepyo: <p>Hi guys,</p>
<p>I’m handling a detection problem with image coordinate (x,y with range -1 to 1)  as a label.<br>
The model is something like this :</p>
<pre><code class="lang-auto">class BB_model(nn.Module):
    def __init__(self):
        super(BB_model, self).__init__()
        resnet = models.resnet34(pretrained=False)
        layers = list(resnet.children())[:8]
        self.features1 = nn.Sequential(*layers[:6])
        self.features2 = nn.Sequential(*layers[6:])
        self.bb = nn.Sequential(nn.BatchNorm1d(512), nn.Linear(512, 2))
        
    def forward(self, x):
        x = self.features1(x)
        x = self.features2(x)
        x = F.relu(x)
        x = nn.AdaptiveAvgPool2d((1,1))(x)
        x = x.view(x.shape[0], -1)
        x = self.bb(x)
        return x
</code></pre>
<p>With <code>L1loss</code> as the loss function and <code>MAE</code> as the metrics. Somehow the model is not learning anything…<br>
Do I have to use tanh / sigmoid too?</p>
Torchvision.transforms.functional Tensor Parameters

Jannis: <p>Hi! I would like to calculate the gradient of torchvision.transforms.functional transformation parameters. For example, I would like to use <code>torchvision.transforms.functional.rotate</code> and provide a tensor for the angle parameter so I can calculate the gradient of it. Is there any easy way of doing this in PyTorch right now?</p>

ptrblck: <p>I don’t think this is supported in <code>torchvision</code> transformations, as literals are expected and tensor inputs would raise an error.<br>
However, <a href="https://github.com/kornia/kornia">Kornia</a> is a differentiable computer vision library and might be helpful. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
CC <a class="mention" href="/u/edgarriba">@edgarriba</a> as the core dev</p>

Jannis: <p>Thank you, Kornia looks exactly like what I’ve been looking for!</p>
Custom loss function using Scipy operations

nisharaichur: <pre><code class="lang-auto">
from scipy.spatial.transform import Rotation as R

class MNCriterion(nn.Module):

    def __init__(self, t_loss_fn=nn.L1Loss(), q_loss_fn=nn.L1Loss(), sax=0.0, saq=0.0, srx=0, srq=0.0, learn_beta=False, learn_gamma=False):
        super(MapNetCriterion, self).__init__()
        self.t_loss_fn = t_loss_fn
        self.q_loss_fn = q_loss_fn
        self.sax = nn.Parameter(torch.Tensor([sax]), requires_grad=learn_beta)
        self.saq = nn.Parameter(torch.Tensor([saq]), requires_grad=learn_beta)
        self.srx = nn.Parameter(torch.Tensor([srx]), requires_grad=learn_gamma)
        self.srq = nn.Parameter(torch.Tensor([srq]), requires_grad=learn_gamma)
        self.register_buffer("conj", torch.tensor([1,-1,-1,-1]))

    def pose_loss(self, pose_a, pose_b):
        quat_a = pose_a[3:]
        quat_b = pose_b[3:]
        r = R.from_quat(quat_a) 
        q_mat = r.as_matrix().T
        q_conj = torch.mul(quat_a, self.conj)
        sub = pose_b[:3] - pose_a[:3]
        delta_position = torch.matmul(q_mat, sub.T)
        delta_quat = self.hamilton_product(q_conj, quat_b)
        return np.concatenate((delta_position, delta_quat)).unsqueeze(0)

    def forward(self, pred, targ):
        p_loss =  self.pose_loss(pred, targ)
        ......


</code></pre>
<p>I get an error</p>
<blockquote>
<p>r = R.from_quat(quat_a))<br>
TypeError: can’t convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.</p>
</blockquote>
<p>Since Scipy expects the numpy array, how do I deal with this?</p>

Sanjan_Das: <p>Have you tried the suggestion from the error, ie using <code>Tensor.cpu()</code>? In your case, assuming <code>quat_a</code> is a Tensor, you could just add <code>quat_a.cpu()</code> before the line which you get your error on.</p>

nisharaichur: <p>I tried that.<br>
That would be a problem if I convert them to cpu tensors and then to numpy arrays, since gradients willl not be computed respect to them and I get an error.</p>

notprime: <p>Going from tensors to numpy arrays and viceversa is not recommended, especially when you wan to compute gradients over this tensors.<br>
Have you tried to work only on tensors?<br>
Maybe you can code that <code>Rotation</code> using pytorch functions, or maybe there are some libraries written in pytorch you could use.<br>
For instance, I found this method from <strong>PyTorch3D</strong> library: <a href="https://pytorch3d.readthedocs.io/en/latest/modules/transforms.html" rel="noopener nofollow ugc">pytorch3d.transforms</a></p>

nisharaichur: <p>Implementing the Rotation using pytorch helps.</p>
<p>Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Why does conv2d() use kernel space when there is only one thread?

lynn2089: <p>Hi all,<br>
I am currently in touch with the use of pytorch. When I was running the convolutional neural network code, I tested the conv2d() function on the CPU. I used “top” to check performance.<br>
When I set the number of threads equal to 1, I found that “%sy” in the information printed by “top” reached 40-60% (sy: system cpu time (or)% CPU time spent in kernel space).<br>
Is it normal? And can anyone explain why the kernel space is used?</p>
<pre><code class="lang-auto">cpu_num = 1 
print("cpu_num: ",cpu_num)
os.environ ['OMP_NUM_THREADS'] = str(cpu_num)
os.environ ['OPENBLAS_NUM_THREADS'] = str(cpu_num)
os.environ ['MKL_NUM_THREADS'] = str(cpu_num)
os.environ ['VECLIB_MAXIMUM_THREADS'] = str(cpu_num)
os.environ ['NUMEXPR_NUM_THREADS'] = str(cpu_num)
torch.set_num_threads(cpu_num)
</code></pre>
<p>Many thanks!</p>
Getting too many values to unpack while freezing the parameters

bing: <p>I am trying to use transfer learning. I want to freeze the parameters of the model apart from the batch norm layers. I am using the below code but getting <code>too many values error</code>.</p>
<pre><code class="lang-auto">for name,param in model_transfer.parameters():
    if ("bn" not in name):
        param.requires_grad = False

</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">ValueError                                Traceback (most recent call last)
 in 
----&gt; 1 for name,param in model_transfer.parameters():
      2     if ("bn" not in name):
      3         param.requires_grad = False
      4 
      5 

ValueError: too many values to unpack (expected 2)

</code></pre>
<p>How to resolve it?</p>

albanD: <p>Hi,</p>
<p>If you want both the name and the parameter, you need to use <code>model_transfer.named_parameters()</code>.</p>

ricslator: <aside class="quote no-group" data-username="bing" data-post="1" data-topic="80717">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/839c29/48.png" class="avatar"> bing:</div>
<blockquote>
<p><code>ValueError: too many values to unpack (expected 2)</code></p>
</blockquote>
</aside>
<p>Python functions can return multiple variables . These variables can be stored in variables directly. This is a unique property of Python , other programming languages such as C++ or Java do not support this by default.</p>
<p>The valueerror: <a href="http://net-informations.com/python/err/value.htm" rel="noopener nofollow ugc">too many values to unpack</a> occurs during a multiple-assignment where you either don’t have enough objects to assign to the variables or you have more objects to assign than variables. If for example myfunction() returned an iterable with three items instead of the expected two then you would have more objects than variables necessary to assign to.</p>
Convolutional Neural Network (CNN) on numerical data

Ajinkya_Anjikar: <p>Hello, I am a newbie to deep learning.<br>
I have numerical data with 300 rows and 667 columns. The last column contains labels with the help of which, I want to classify the datasets into two classes. I am having problems with data input to the convolutional layer. I am having difficulty applying filters to numerical data.<br>
Can anyone please suggest a proper implementation of CNN to numerical data to me?</p>
<p>P.S.: I know that the dataset is small. I have data augmentation in the to-do task but before that, I want to understand the concepts related to the input layer.</p>

pchandrasekaran: <p>Wouldn’t a Mutil-Layered Perceptron be more suitable for your task at hand? I’m not sure how a CNN could be implemented properly.</p>

Ajinkya_Anjikar: <p>Thank you so much for the response.<br>
I have already implemented MLP. I want to go a step ahead and want to know what changes in accuracy and performance can CNN make.</p>

EvanZ: <p>A CNN assumes there is some spatial similarity or continuity between pixels. What is the relationship between the rows and columns in your dataset? A CNN is not just a magical recipe.</p>

Ajinkya_Anjikar: <p>Thank you for sharing the information.<br>
Each row is a spectrum of cells. So, each row with the mentioned number of columns can give spectral information of respective cells.</p>
Stupid Question!

aaaaab: <p>Hi there,<br>
Sorry for the stupid and elementary question!<br>
I am totally new to Python and PyTorch.</p>
<p>My question is that I want to multiply a <strong>[128,10,512]</strong> tensor by a tensor with the length of <strong>10</strong> such that the final output to be a <strong>[128*512]</strong> tensor. I know how to do that using for loops. However, I am wondering whether there is a direct PyTorch command in this regard.<br>
Thank you in advance for your kind help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrmcl: <p>How are you reducing along that dimension? sum?</p>

aaaaab: <p>Thank you for your reply.<br>
Yes, It is Sum</p>

ptrmcl: <p>First thing that comes to mind:</p>
<pre><code class="lang-auto"># length 10 vector
ten = torch.ones((10))
# 128x10x512 tensor
big = torch.ones((128,10,512))

result = (big*ten.view(1,-1,1)).sum(1).flatten()
</code></pre>
<p>We reshape the tensor to be 1x10x1 to align with the other one. Then we sum across that dimension and flatten.</p>
<p>Makes sense?</p>

aaaaab: <p>Thank you!<br>
That’s great.<br>
What about if “big” is a tensor shaped [batchsize, num_windows, CNN_features], “ten” is a tensor shaped [batchsize, num_windows], and using “ten” we want to map “big” to the “result” tensor with the shape of [batchsize, CNN_features]? i.e.<br>
big = [128,10,512]<br>
ten = [128,10]<br>
result = [128,512]</p>

ptrmcl: <p>sum again? I think it’s the same idea.</p>
<pre><code class="lang-auto">ten = torch.ones((128,10))
big = torch.ones((128,10,512))

result = (big*ten.view(128,10,1)).sum(1)
</code></pre>

aaaaab: <aside class="quote no-group" data-username="ptrmcl" data-post="6" data-topic="121364">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrmcl/48/32259_2.png" class="avatar"> ptrmcl:</div>
<blockquote>
<p><code>result = (big*ten.view(128,10,1)).sum(1)</code></p>
</blockquote>
</aside>
<p>Thank you again!<br>
I learned a lot!</p>

Lamb_Sauce: <p>No question is stupid. I know its overrepeated but <strong>No Question Is Stupid!</strong></p>

Lamb_Sauce: <aside class="quote no-group" data-username="aaaaab" data-post="5" data-topic="121364">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/a88e57/48.png" class="avatar"> aaaaab:</div>
<blockquote>
<p>result = [128,512]</p>
</blockquote>
</aside>
<p>btw, use the &lt;/&gt; button to add code or three backticks ``` to start and finish the thing. It just makes everything easier and clean formatting. Easier for other people to understand everything.</p>
Parameter Tensor grad func conflict

SachinKonan: <p>I was wondering what happens if you have a parameter (P), that has required_grad to be set to true, and manually set its associated tensor, P.data, to requires_grad False. So P = parameter with requires_grad:True, but P.data.requires_grad = False. What happens if I back-propagated on a Neural Network that had this phenomenon.</p>
<p>Also i noticed that when i printed out the .data value of one of the parameters in my nn.Module, its requires grad property is set to False. Does this mean that requires_grad of .data is not used if it is embedded in a Parameter whose requires_grad property is True?</p>

ptrblck: <p>The <code>.data</code> attribute is used internally and only its parent would reflect the <code>.requires_grad</code> attribute.<br>
Note that the usage of <code>.data</code> is deprecated and could yield to many side effects, as Autograd won’t be able to track operation performed on <code>.data</code>.</p>
I can't plot images with make_grid

Anderson_Ferreira: <p>I have a dataset of images in the “dataset” and was trying to use “make_grid” like in this link: <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Training a Classifier — PyTorch Tutorials 1.8.1+cu102 documentation</a>", but but there’s something wrong and I don’t know what.</p>
<pre><code class="lang-auto">dataiter = iter(dataset)
images, labels = dataiter.next()
plt.imshow(torchvision.utils.make_grid(images.permute(1,2,0)))
    # show images
</code></pre>
<p>My dataset contains 3 images of shape (3, 40, 40).<br>
The error is:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
&lt;ipython-input-92-0a9e2d56a78a&gt; in &lt;module&gt;
      1 dataiter = iter(dataset)
----&gt; 2 images, labels = dataiter.next()
      3 plt.imshow(torchvision.utils.make_grid(images.permute(1,2,0)))
      4     # show images

AttributeError: 'iterator' object has no attribute 'next'
</code></pre>

eqy: <p>Did you mean to use <code>images, labels = next(dataiter)</code> by chance?</p>