akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
I am getting a noisy classified image inspite of getting test accuracy of 99 percent

Sandeep_Kumar_GITAM: <p>Dear friends, I am getting a noisy classified image inspite of getting test accuracy of 99 percent. I am using the following function for obtaining the classification map.</p>
<pre><code class="lang-auto">def classified_pixels(FPatches,gPatches,g):
  FPatches=np.reshape(FPatches,(FPatches.shape[0],FPatches.shape[3],FPatches.shape[1],FPatches.shape[2]))
  data_test=MyDataset(FPatches, gPatches)
  test_loader = torch.utils.data.DataLoader(data_test,batch_size=10,shuffle=False, num_workers=2)
  with torch.no_grad():
    correct = 0
    total = 0
    predicted_numpy=[]
    for images, labels in test_loader:
      images = images.to(device)
      labels = labels.to(device)
      outputs = model(images)
      _, predicted = torch.max(outputs.data, 1)
      predicted_numpy.append(predicted.cpu().numpy())
      total += labels.size(0)
      correct += (predicted == labels).sum().item()
  classification_map=np.array(predicted_numpy)
  cm=[]
  for arr in classification_map:
    cm.append(arr.tolist())
  cm=list(itertools.chain.from_iterable(cm))
  classification_map=np.array(cm)

  height=g.shape[0]
  width=g.shape[1]
  outputs = np.zeros((height,width))
  k=0
  for i in range(height):
    for j in range(width):
       target = g[i][j]
       if target == 0 :
         continue
       else :
        outputs[i][j]=classification_map[k]
       k=k+1
  return classification_map,outputs
</code></pre>

gphilip: <p>If you add more context to your question then people who are not very familiar with the exact setting of your problem, may also be able to help you. This could increase the chances of you getting useful inputs from the forum.</p>
<p>Examples of more context include: What is the problem that you are trying to solve? What does the input look like? What is the output that you expect? What does your model look like? And so on.</p>

Sandeep_Kumar_GITAM: <p><a class="mention" href="/u/gphilip">@gphilip</a>, I have performed pixel classification of Hyperspectral Image using the above mentioned model. I have extracted the patches mentioned as FPatches around the neighbourhood of the pixel and consider them as samples. The label of the center pixel of the FPatch is considered as the label for the entire FPatch.  The model is trained using training samples and tested using test samples. Test accuracy is more than 99 percent. For obtaining the classification map I have taken the entire FPatches and predicted the labels using the same trained model. Finally I have converted the predicted classes into array to get the classification map. One important thing to mention is that the groundtruth contains background pixels denoted as 0 so when the classification map array is generated we will keep those classes as zero.</p>

gphilip: <p>Thank you for the details.</p>
<p>I don’t have enough knowledge in your domain to answer your question; let us hope others can help you figure this out.</p>
How can I detect alloc/dealloc in GPU?

SungBalance: <p>Hi! I’m new to pytorch, and I’m trying to figure out how pytorch works.</p>
<p>I want to know when pytorch exactly alloc or dealloc tensors(specially output gradients and weight gradients) during training.</p>
<p>For this work, I’m trying to get logs for allocation/deallocation. I used some hooks with torch.cuda.memory_allocated(), but it’s hard to get exact moment and target tensor.</p>
<p>Is there a way to get logs for VRAM allocation/deallocation? (like vlog in tensorflow) or What do you think is appropriate to achieve my goal?</p>
<p>Thanks.</p>
Init_process_group freezing

sleepybear: <p>I use the following repo. to train a TTS on my own dataset : <a href="https://github.com/NVIDIA/tacotron2" rel="noopener nofollow ugc">Tacotron 2</a></p>
<p>All stuffs are going well, but when I put <code>distributed_run = True</code> in <code>hparams.py</code> the process has been stucked at <code>init_process_group</code> function in <code>train.py</code> at line 35 (I put my <code>world_size = 3</code>). My GPU driver and cuda are okey! Is this a bug? If yes how can I fix it?</p>
<p>Thanks</p>

gcramer23: <p>Hi, can you confirm that all ranks reach line 35.</p>

sleepybear: <p>No. Thanks, good point! The bug is in Nvidia code.</p>
How to train checkpoint with a different dataset

root_access: <p>Hello,</p>
<p>I am training a unet model, which is fine. am also able to load the checkpoint and resume training .<br>
Say i train the model on Animal datasets up to 100 epochs. Then i want to train Car Datasets after , do i need to resume training with both datasets mixed ? or i can only add car datasets and the next checkpoint to be saved will have learned on both ?</p>
<p>Thank You</p>

eqy: <p>Generally speaking there is no fixed recipe you have to follow here, you can train on both datasets in any order or with different degrees of mixing. It’s hard to know ahead of time what will give the best results as it is likely dataset and use-case dependent. The only thing to pay attention to is how the last layers are swapped in an out (as I imagine the classes in Cars and Animals are different and there are different numbers of classes between the datasets).</p>
How can i select feature map channels through mask map

QianMo-wangxiaoyu: <p>for example<br>
mask_map.shape=(1,H,W)<br>
feature_map.shape=(1,C,H,W)<br>
I use a picture to explain my question<br>
in this , I make H=2,W=2,C=2<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/1/219a2dbbcb478bae3bbe43102f5dd8ad246ed49e.png" data-download-href="https://discuss.pytorch.org/uploads/default/219a2dbbcb478bae3bbe43102f5dd8ad246ed49e" title="1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/1/219a2dbbcb478bae3bbe43102f5dd8ad246ed49e.png" alt="1" data-base62-sha1="4Ng58689lpk6mm3QSIwp8M9XxVk" width="690" height="499" data-dominant-color="F0EDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">830×601 7.11 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

eqy: <p>Does broadcasting work if you reshape the mask?</p>
<pre><code class="lang-auto">reshaped_mask = mask.reshape(1, 1, H, W)
out = feature_map * reshaped_mask
</code></pre>
What's the best practice to change the values of a tensor?

a514514772: <p>I am looking into a project that has to re-initialize weights at some point. A problem suddenly occurs to me.</p>
<p>I used to change values of tensors in the following way, which I believe outdated/not recommended now:</p>
<pre><code class="lang-auto">m = nn.Linear(100, 100)
m.weight.data.fill_(1)
</code></pre>
<p>I am wondering what is the correct/encouraged way to manage it in the current version. My current idea is like this:</p>
<pre><code class="lang-auto">m.weight.detach().fill_(1)
</code></pre>
<p>Are there any more recommendations?</p>
Bug when test solov2

Sylvain_Ard: <p>Hello,<br>
I am trying to test <a href="https://github.com/WXinlong/SOLO" class="inline-onebox" rel="noopener nofollow ugc">GitHub - WXinlong/SOLO: SOLO and SOLOv2 for instance segmentation, ECCV 2020 &amp; NeurIPS 2020.</a><br>
I have do training but :<br>
I have the following bug when I launch<br>
python tools/test.py configs/solov2/solov2_r50_fpn_8gpu_3x.py demo/epoch_36.pth --json_out output --eval segm<br>
:<br>
[&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;] 30/30, 5.1 task/s, elapsed: 6s, ETA:     0sTraceback (most recent call last):<br>
File “tools/test.py”, line 282, in <br>
main()<br>
File “tools/test.py”, line 273, in main<br>
results2json(dataset, outputs, args.json_out)<br>
File “c:\solo-master\mmdet\core\evaluation\coco_utils.py”, line 224, in results2json<br>
json_results = det2json(dataset, results)<br>
File “c:\solo-master\mmdet\core\evaluation\coco_utils.py”, line 153, in det2json<br>
for i in range(bboxes.shape[0]):<span class="hashtag">#changed</span> by SA<br>
AttributeError: ‘NoneType’ object has no attribute ‘shape’</p>
<p>and when I run the demo I have always 0 instances<br>
Please help me !<br>
Best regards</p>

Sylvain_Ard: <p>I have instances but always only one. That I want is to segment the leaflets of a leaf but if the leaf is simple it works but if the leaf is compound all the leaflets are in the same instance, an idea ?</p>

ptrblck: <p>I don’t quite understand the last sentence, but the previous error is raised if no <code>bboxes</code> are detected and they are thus <code>None</code>. My guess is that the code you are using isn’t properly guarding against this case and thus fails.</p>
No instances detected with soloV2

Sylvain_Ard: <p>I tested <a href="https://github.com/aim-uofa/AdelaiDet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - aim-uofa/AdelaiDet: AdelaiDet is an open source toolbox for multiple instance-level detection and recognition tasks.</a> on my own dataset with respect to <a href="https://medium.com/@chengweizhang2012/how-to-train-detectron2-with-custom-coco-datasets-4d5170c9f389" class="inline-onebox" rel="noopener nofollow ugc">How to train Detectron2 with Custom COCO Datasets | by Chengwei Zhang | Medium</a> but it predicts always 0 instances !! please help me !<br>
Best regards</p>

ptrblck: <p>I would recommend to contact the authors of this repository/model, as your request is not easily actionable here.</p>
Torch.mode - what does indices hold

bigThrum: <p>Hi All,</p>
<p>Can’t get my head around the return result (indices) - from docs “mode … a value which appears most often in that row, and <code>indices</code> is the index location of each mode value found.”<br>
my test:<br>
<span class="hashtag">#test</span><br>
a = torch.tensor([4,1,2,3,4,5,4,4,4,5,6,7,8,4,4,9,4])<br>
print(a)<br>
print(a.size())<br>
torch.mode(a)</p>
<p>result<br>
tensor([4, 1, 2, 3, 4, 5, 4, 4, 4, 5, 6, 7, 8, 4, 4, 9, 4])<br>
torch.Size([17])<br>
torch.return_types.mode(<br>
values=tensor(4),<br>
indices=tensor(16))</p>

ptrblck: <p>The <code>indices</code> seem to point to any index containing the <code>mode</code> value. Based on some runs the last index seems to be used, but I guess there is no guarantee to it and any index of the mode value could be returned:</p>
<pre><code class="lang-python">x = torch.zeros(2, 5)
torch.mode(x)
&gt; torch.return_types.mode(
  values=tensor([0., 0.]),
  indices=tensor([4, 4]))
</code></pre>
How can I make black a bounding box area of all images?

chopby: <p>Hi</p>
<p>A bounding box label exists in the image in tiny image.(X Y H W)<br>
I want to make the bounding box area black. What can I do?</p>
<p>Thank you!</p>

ptrblck: <p>If I understand your use case correctly, you would like to use these coordinates to mask the area in an image?<br>
If so, you could index the input directly and assign the new value to it:</p>
<pre><code class="lang-python">X, Y, H, W, = 1, 2, 3, 3
x = torch.randn(2, 3, 5, 5)

x[:, :, Y:Y+H, X:X+W] = 0.
print(x)
</code></pre>
Weight decay in multi-label models trained with BCEWithLogitsLoss

josafatburmeister: <p>Hi,<br>
I have a question about weight decay on multi-label image classification models.</p>
<p>My training setting is the following: I’m training ResNet18 models on an image classification dataset, either as single-label or multi-label models.  For the single-label models, I use PyTorch’s CrossEntropyLoss as the loss function, and for the multi-label models, I use BCEWithLogitsLoss. I also use an Adam optimizer with weight decay.</p>
<p>For the single-label models, weight decay with coefficients around 0.5 works well and improves model performance. However, for the multi-label models, weight decay with coefficients around 0.5 results in a significant decrease in model performance compared to training runs without weight decay. With weight decay, the multi-label model learns to recognize only some classes and the F1-scores of all other classes are close to zero.</p>
<p>Since the single-label and multi-label models are trained on the same dataset, I am surprised by the different impact of weight decay in both cases. Is this due to the combination of BCEWithLogitsLoss and weight decay? Or does the weight decay coefficient simply need to be retuned for the multi-label case?</p>
<p>Thanks a lot!</p>

zetyquickly: <p>Hi <a class="mention" href="/u/josafatburmeister">@josafatburmeister</a></p>
<p>Could you please tell with a decay different from 0.5 does multi-label model show significantly better F1-scores?</p>

josafatburmeister: <p>Yes, without weight decay, the multi-label model achieves a macro-averaged F1-score of about 0.74, with F1-scores above 0.6 for all classes. As the weight decay coefficient is increased, the F1-scores decrease. Specifically, the F1 scores were as follows:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Weight decay coefficient</th>
<th>Macro-average F1-score</th>
<th>Minimum class-wise F1-score</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>0.74</td>
<td>0.63</td>
</tr>
<tr>
<td>0.1</td>
<td>0.67</td>
<td>0.5</td>
</tr>
<tr>
<td>0.2</td>
<td>0.6</td>
<td>0.29</td>
</tr>
<tr>
<td>0.3</td>
<td>0.55</td>
<td>0.23</td>
</tr>
<tr>
<td>0.4</td>
<td>0.49</td>
<td>0.12</td>
</tr>
<tr>
<td>0.5</td>
<td>0.41</td>
<td>0</td>
</tr>
</tbody>
</table>
</div>
Bug when train in soloV2

Sylvain_Ard: <p>Hello !<br>
I have this error :</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pastebin.com/F2UZDCJe">
  <header class="source">
      <img src="https://pastebin.com/favicon.ico" class="site-icon" width="16" height="16">

      <a href="https://pastebin.com/F2UZDCJe" target="_blank" rel="noopener nofollow ugc">Pastebin</a>
  </header>

  <article class="onebox-body">
    <img src="https://pastebin.com/i/facebook.png" class="thumbnail" width="" height="">

<h3><a href="https://pastebin.com/F2UZDCJe" target="_blank" rel="noopener nofollow ugc">C:\SOLO-master&gt;call activate solov2_2C:\SOLO-master&gt;conda.bat activate...</a></h3>

  <p>Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>when I train with solov2_r50_fpn_8gpu_3x.py in <a href="https://github.com/WXinlong/SOLO" class="inline-onebox" rel="noopener nofollow ugc">GitHub - WXinlong/SOLO: SOLO and SOLOv2 for instance segmentation, ECCV 2020 &amp; NeurIPS 2020.</a><br>
please help me !<br>
Best regards</p>
How to perform batch operations on tensor in parallel?

hhiim: <p>Suppose I have a tensor like [[1,2,3],[4,5,6],[7,8,9]]. I want to call a certain method for each tensor in it to process it. Is there a generic way to accomplish this?</p>
<p>I have defined a new “convolution” method, where the convolution kernel is moved over the image to get the features. But the computation is so complex that I couldn’t find a suitable native implementation of Pytorch and had to divide the image into small pieces and compute them one by one.</p>
<p>The usual way is to use a for loop, but the performance is low. Is it possible to parallelize this process? Compute multiple regions at the same time. GPUs are said to be good at parallelism …</p>

Torchbearer: <p><a class="mention" href="/u/hhiim">@hhiim</a>  did you find any solution for your problem</p>
How to globally prune deep neural network

aulakh83: <p>I am trying to prune my model.</p>
<p>class EncoderCNN(nn.Module):</p>
<pre><code>def __init__(self):
    super(EncoderCNN, self).__init__()
    cnn = models.vgg19(pretrained = False)
    modules = list(cnn.children())[:-2]
    self.cnn = nn.Sequential(*modules)
    self.enc_dim = list(cnn.features.children())[-3].weight.shape[0]
    self.avg_func = torch.nn.AvgPool2d(kernel_size=7, stride=1, padding=0)
    #self.tagClassifier=cnnModel
    #self.classifier_dim=list(cnnModel.features.children())[-1].weight.shape[0]
    
def forward(self, x):
    x = self.cnn(x)         
    avg_features = self.avg_func(x).squeeze()
    x = x.permute(0, 2, 3, 1)
 
    #y= self.tagClassifier(x)
    return x , avg_features
</code></pre>
<p>This is the code snippet of the model. for pruning i am trying to specify the parameters like<br>
parameters_to_prune = (<br>
(encoderCNN.cnn, ‘0.0.weight’),<br>
(encoderCNN.cnn, ‘0.2.weight’),<br>
)</p>
<p>And got an obvious error as AttributeError: ‘Sequential’ object has no attribute ‘0.0.weight’<br>
Error i do understand but how can i access these parameters that i am not sure and needs help for that</p>

aulakh83: <p>I got the solution by writing it as</p>
<p>parameters_to_prune = (<br>
(encoderCNN.cnn[0][0],‘weight’),<br>
(encoderCNN.cnn[0][2], ‘weight’),<br>
)</p>
How can I use batchwise mask for TransformerEncoder?

032june: <p>After reading the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html" rel="noopener nofollow ugc">official pytorch documentation for transformer</a>, I think that I fully understand how the src_mask and src_key_padding_mask work.<br>
It seems that the src_mask (S,S) decides which elements should be attended to per each element, and it is shared among all batches.<br>
And the src_key_padding_mask (N,S) aims to apply different masks for each batch.</p>
<p>However, how can I apply (N,S,S) dimension mask?<br>
I want to apply different masks for each element of each batch.<br>
Maybe it is not implemented in the official version of pytorch transformer…<br>
Any help will be greatly appreciated.</p>
Nv_peer_mem driver was not detected

gm_k: <p>When I run “/usr/bin/nvidia_entrypoint.sh”(ngc 21.03), the following NOTE is displayed. Even if nv_peer_mem is set separately, it continues to appear. Does it keep happening because it will be deprecated soon?</p>
<p>NOTE: MOFED driver was detected, but nv_peer_mem driver was not detected. Multi-node communication performance may be reduced.</p>

ptrblck: <p>Are you seeing any other errors/warning before the MOFED warning?<br>
Also, are you using multi-node training?</p>
Backward() fails after multiple forward passes when model has spectral_norm

draco: <p>I have 3 different inputs- I preform 3 forward passes and one backward pass.<br>
when my model has spectral_norm in it’s layers, the backward fails with the following message:<br>
one of the variables needed for gradient computation has been modified by an inplace operation<br>
once I remove spectral_norm from my model it passes.</p>
<p>is there any work around for this that will still let me perform 3 forward passes and one backward?</p>
<p>many thanks</p>

draco: <p>I think using spectral norm from TORCH.NN.UTILS.SPECTRAL_NORM<br>
instead of TORCH.NN.UTILS.PARAMETRIZATIONS.SPECTRAL_NORM has solved it for me</p>

jon_ryu: <p>I had the same issue with torch.nn.utils.parametrizations.spectral_norm and using torch.nn.utils.spectral_norm instead indeed resolves the problem. Is this a bug?</p>
how to define loss function in torch.optim.SGD()

Ronak_Shoghi: <p>Hi,</p>
<p>I am totally new to PyTorch and I have a question regarding the optimization using SGD.  I would like to distribute n points on the surface of a unit sphere in 5D space. for generating random points I use:<br>
points = torch.randn((npoints, dim), requires_grad=True)</p>
<p>I would like to optimize these points using torch.optim.sgd() and I need to define a proper loss function for this problem in order to be used for loss. backward().<br>
torch.optim.SGD([points], lr=0.001)<br>
for comparing the quality of optimization I calculate the k nearest neighbors and also the standard deviation for that array.</p>
<p>I would appreciate any help in this regard.</p>

stevenjin8: <p>Hi!</p>
<p>I’m a bit confused about what you want to do. From my understanding, optimizers interact with loss functions via gradients, so you would never really "define a loss function <strong>in</strong> <code>torch.optim.SGD</code>.</p>
<p>If you have a function <code>f</code> that quantifies how well n points are distributed on a unit sphere, then you could do something like</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD([points], lr=0.001)
for _ in range(num_iters):
    loss = f(points)  # calculate loss
    loss.backwards()  # back propagate
    optimizer.step()  # update parameters
    optimizer.zero_grad()  # reset gradients
</code></pre>
Interpretation of loss and accuracy

hossein: <p>Hello,<br>
I’m training a CNN network and these are my results for the first 14 epochs:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1156001b71f90383fb9e6ecb63bd48b99bbbb34.png" alt="pyt" data-base62-sha1="ry5WKuCeoO8Zvi1sICYafqTm0Je" width="438" height="254"></p>
<p>As you can see, despite that my train loss rose to 1.3 from epoch 2 to 3, my train accuracy increased, while it should be decreased! (there are other epochs like what I said. for example, between epoch 8 and 9, between epoch 10 and 11). Is this result natural??<br>
note: I applied <code>nn.CrossEntropyLoss</code> as my loss function.</p>

ptrblck: <p>This might be expected, as the loss and accuracy do not strictly depend on each other.<br>
I.e. the range of the logits also defines the loss, while only the <code>argmax</code> logit defines the prediction (and thus changes the accuracy) as seen in this small example:</p>
<pre><code class="lang-python"># setup
criterion = nn.CrossEntropyLoss()

# low loss, 0 acc
output = torch.tensor([[-1., 1.],
                       [-1., 1.]])
target = torch.tensor([0, 0])

loss = criterion(output, target)
print(loss)
&gt; tensor(2.1269)

acc = (torch.argmax(output, dim=1) == target).float().mean()
print(acc)
&gt; tensor(0.)

# high loss, acc=0.5
output = torch.tensor([[1., -1.],
                       [-5., 1.]])
target = torch.tensor([0, 0])
loss = criterion(output, target)
print(loss)
&gt; tensor(3.0647)

acc = (torch.argmax(output, dim=1) == target).float().mean()
print(acc)
&gt; tensor(0.5000)
</code></pre>
Defining random and sequiatial dataloaders - is there a data leak?

Ilya_Kotlov: <p>Hi, I defined in this way my two dataloaders, one random and one sequential and i’ve started to suspect there is a data leak (although not necessarily).</p>
<p>Could you give it a glimpse ? Do the dataloaders seem to be doing what I’ve intended without data leaks ?</p>
<p>Thank you in advance.</p>
<p>P.S I’ve adapted some existing code, so maybe it’s not the most short and elegant way to define dataloaders with this functionality, so inspecting this code is preferable, but suggestions to make it differently also will be accepted.</p>
<pre><code>    #class PascalVOCLoader(data.Dataset):
    self.dataset = PascalVOCLoader(
        root_dir,
        augmentations,
        output_dim=224,
        mode='classification')

    # Creating data indices for training and validation splits:
    dataset_size = len(self.dataset)
    validation_split = 1 - ds_split
    random_seed = 42
    indices = list(range(dataset_size))
    split = int(np.floor(validation_split * dataset_size))
    if shuffle:
        np.random.seed(random_seed)
        np.random.shuffle(indices)
    train_indices, val_indices = indices[split:], indices[:split]

    self.seq_train_subset = Subset(self.dataset, train_indices)
    self.seq_test_subset = Subset(self.dataset, val_indices)

    random_train_sampler = SubsetRandomSampler(train_indices)
    random_valid_sampler = SubsetRandomSampler(val_indices)

    random_train_loader = torch.utils.data.DataLoader(
        self.dataset,
        batch_size=self.batch_size['train'],
        num_workers=0,
        sampler=random_train_sampler,
        collate_fn=my_collate)
    random_validation_loader = torch.utils.data.DataLoader(
        self.dataset,
        num_workers=0,
        batch_size=self.batch_size['test'],
        sampler=random_valid_sampler)

    sequential_train_sampler = SequentialSampler(self.seq_train_subset)
    sequential_valid_sampler = SequentialSampler(self.seq_test_subset)

    sequetial_train_loader = torch.utils.data.DataLoader(
        self.seq_train_subset,
        batch_size=self.batch_size['train'],
        num_workers=0,
        sampler=sequential_train_sampler)
    sequetial_validation_loader = torch.utils.data.DataLoader(
        self.seq_test_subset,
        num_workers=0,
        batch_size=self.batch_size['test'],
        sampler=sequential_valid_sampler,
        collate_fn=my_collate)</code></pre>

ptrblck: <p>I don’t see any way this code would leak memory, so would assume it’s fine.</p>
Depthwise Separable Convolution is Slower than Standard Convolution

John_Watkins: <p>Hi,</p>
<p>I am trying to implement a depth-wise separable 1D-convolution in torch to read very long 1D-images to cut down parameter count and model latency. As I cannot seem to find an off-the-shelf implementation in torch, I have (following other posts) written my own:</p>
<pre><code>class depthwise_separable_conv(nn.Module):

    def __init__(self, nin, nout, depth_kernel_size, stride_size):
        super(depthwise_separable_conv, self).__init__()
        self.depthwise = nn.Conv1d(nin, nin, kernel_size=depth_kernel_size, stride=stride_size, padding=0, groups=nin)
        self.pointwise = nn.Conv1d(nin, nout, kernel_size=1)

    def forward(self, x):
        out = self.depthwise(x)
        out = self.pointwise(out)
        return out
</code></pre>
<p>However, I have noticed that the the separable convolution ended up taking noticeably longer than the standard 1D-convolution during training. I am wondering:</p>
<ol>
<li>Is there some error with my implementation?</li>
<li>If the implementation is correct, what might be the reason for the slowness of this implementation?</li>
<li>What do I do to speed it up?</li>
</ol>

walbermr: <p>I’m facing the same issue, did you reached somthing?</p>
TypeError: unsupported operand type(s) for *: 'Conv2d' and 'Tensor'

Mona_Jalal: <p>Could you please guide how to fix this error in PyTorch 1.9 and Pytho 3.8?</p>
<p><code>TypeError: unsupported operand type(s) for *: 'Conv2d' and 'Tensor'</code></p>

AlphaBetaGamma96: <p>Could you post the line that causes the error? What it seems is happening is you’re multiplying a <code>nn.Conv2d</code> object with a <code>Tensor</code> rather than calling it.</p>
<p>For example,</p>
<pre><code class="lang-auto">conv = nn.Conv2d(1,6,5)
input = torch.randn(1,1,32,32)

conv(input) #returns Tensor of shape: torch.Size([1, 6, 28, 28])
</code></pre>
<p>whereas <code>conv*input</code> returns an error,</p>
<pre><code class="lang-auto">conv*input
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
TypeError: unsupported operand type(s) for *: 'Conv2d' and 'Tensor'
</code></pre>
Loss surface landscape by GPU

Mounir_Abo_saleh: <p>Hello, I am trying to visualize my loss surface land-scape, I train the model on GPU colab</p>
<pre><code class="lang-auto">import loss_landscapes
import loss_landscapes.metrics
x, y = next(iter(testLoader))
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
x,y = x.to(device), y.to(device)
metric = loss_landscapes.metrics.Loss(criterion, x, y)
# compute loss data
loss_data = loss_landscapes.linear_interpolation(model_initial, model_final, metric, 40, deepcopy_model=True)
loss_data_fin = loss_landscapes.random_plane(model_final, metric,10, 40, normalization='filter', deepcopy_model=True)
</code></pre>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-42-1a4435ce1a5c&gt; in &lt;module&gt;()
----&gt; 1 loss_data_fin = loss_landscapes.random_plane(model_final, metric,10, 40, normalization='filter', deepcopy_model=True)

1 frames
/usr/local/lib/python3.7/dist-packages/loss_landscapes/model_interface/model_parameters.py in filter_normalize_(self, ref_point, order)
    224             # normalize one-dimensional bias vectors
    225             if len(self.parameters[l].size()) == 1:
--&gt; 226                 self.parameters[l] *= (ref_point.parameters[l].norm(order) / self.parameters[l].norm(order))
    227             # normalize two-dimensional weight vectors
    228             for f in range(len(self.parameters[l])):

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
</code></pre>
<p>it seem that it can’t work on GPU the last command:</p>
<pre><code class="lang-auto">oss_data_fin = loss_landscapes.random_plane(model_final, metric,10, 40, normalization='filter', deepcopy_model=True)
</code></pre>

ptrblck: <p>Did you push the input data as well as the model to the GPU? If so, could you check if <code>loss_landscapes</code> creates new tensors in internal methods without specifying a device (the default would be CPU, which could create this issue)?</p>

Mounir_Abo_saleh: <p>it solved without specifying a device</p>
<pre><code class="lang-auto">import loss_landscapes
import loss_landscapes.metrics

x, y = next(iter(testLoader))

# if haveCuda:
#   x,y = x.cuda(), y.cuda()

metric = loss_landscapes.metrics.Loss(criterion, x, y)

# compute loss data
loss_data = loss_landscapes.linear_interpolation(
    model_initial.cpu(), 
    model_final.cpu(), 
    metric, 
    40, 
    deepcopy_model=True
)
</code></pre>
Non-Scalar Gradient

simon-schaefer: <p>I want to compute the gradient of a tensor A of shape (batch_size, M) w.r.t. B of shape (batch_size, L) efficiently, so basically the Jacobian dA/dB of shape (batch_size, M, L) ? While the <code>torch.autograd.grad</code> function only accepts scalar outputs, iterating over each element of A and computing the gradient of a_ij w.r.t. B seems quite inefficient. Also, for my use case, finding an analytic solution seems to be hard.</p>
<p>The simplest case of the problem would be something like</p>
<pre><code class="lang-auto">import torch

B = torch.rand((4, 10), requires_grad=True)
W = torch.rand((20, 10))
A = torch.matmul(B, W.T)

torch.autograd.grad(A, B)
</code></pre>
<p>Is there a way to compute dA/dB without iteration?</p>

gphilip: <p>This should do the trick, I think (still learning my ropes around autograd):</p>
<pre><code class="lang-auto">B = torch.rand((4, 10), requires_grad=True)
W = torch.rand((20, 10))
A = torch.matmul(B, W.T)

A.backward(gradient=torch.ones_like(A))
B.grad
</code></pre>

simon-schaefer: <p>Thank you for your quick response!</p>
<p>I think <code>A.backward(gradient=torch.ones_like(A))</code>  is the same as <code>A.sum().backward()</code>. Consequently, you don’t compute the derivative of every element of A w.r.t. every element of B, but the derivative of the batch-wise sum of B w.r.t. A. Consequently, you get a gradient of shape (4, 10), not (4, 20, 10).</p>

gphilip: <p>This is something I don’t understand well enough to address, then! Let us wait for someone more knowledgeable to take a look.</p>