akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
[Torchtext] Delete rows with specific label

username: lipuck
cooked: <p>Hello,</p>
<p>I’m currently using the SNLI dataset with the corresponding split method from torchtext:</p>
<pre><code class="lang-auto">datasets.nli.NLIDataset.splits(...)
</code></pre>
<p>I would like to filter out examples which have a specific label. My idea was to use a Field object that filters during the preprocessing step, but did not find any similar approaches online. I’m unsure wether removing a complete row is even possible and if it is, how the label-Field object can notify the text-Field object.</p>
<p>Is this something torchtext is capable to do, or should I filter beforehand?</p>
<p>Thanks in advance!</p>

username: lipuck
cooked: <p>The pre-processing step only helped but not solve this problem. This specific splits-method returns NLIDatasets with a filter attribute, kicking out a specific kind of label ( ‘-’ ). The right way to do would possibly involve using TabularDataset and defininig fields similar to the NLIDatasets ones, but since ‘.jsonl’ is not supported and an easy hacky way seems to work, I will use this version for now.</p>
<pre><code class="lang-auto">class_labels = {'contradiction': 0, 'neutral': '-', 'entailment': 1, '-': '-'}
label_tokenizer = lambda x: class_labels[x]
label_field = data.Field(sequential=False, batch_first=True, preprocessing=label_tokenizer)
</code></pre>
<p><a href="https://pytorch.org/text/_modules/torchtext/datasets/nli.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/text/_modules/torchtext/datasets/nli.html</a></p>
How to combine different two losses?

username: Gwon
cooked: <p>Hi, I have a questions about DistributedDataparallel</p>
<p>I Implement codes like below in Single GPU(It combined two different losses)</p>
<pre><code class="lang-auto">def train(model, criterion, ...):
   model.train()
   outputs_a = model(inputs_a)
   loss_a = criterion(outputs_a, targets) 
   ...
   outputs_b = model(inputs_b)
   loss_b = criterion(outputs_b, targets)
   ...
   total_loss = (lam * loss_a) + ((1 - lam) * loss_b)
   total_loss.backward()
   ...
</code></pre>
<p>It runs well without any problems</p>
<p>After, when i try to using multiple-gpu with DistributedDataparallel it throws an error like this,</p>
<pre><code class="lang-auto">one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [512]] is at version 5; expected version 3 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.detect_anomaly(True).
</code></pre>
<p>I found this error caused by inplace operation,<br>
But i can’t found any inplace operation in my code!!</p>
<p>So I changed my code like below</p>
<pre><code class="lang-auto">def train(model, criterion, ...):
   model.train()
   outputs_a = model(inputs_a)
   loss_a = criterion(outputs_a, targets) 
   loss_a.backward()
   ...
   outputs_b = model(inputs_b)
   loss_b = criterion(outputs_b, targets)
   loss_b.backward()
   ...
   total_loss = loss_a + loss_b
   ...
</code></pre>
<p>Fortunately, it runs well without any error</p>
<p>However, i have a question</p>
<pre><code class="lang-auto">#1
total_loss = (loss_a + loss_b)
total_loss.backward()

#2
loss_a.backward()
loss_b.backward()
total_loss = loss_a + loss_b
</code></pre>
<p>Are these two operations the same?</p>
Problem exporting to onnx, input output not on same device

username: dangraf
cooked: <p>I’m working with yolov5 and trying to export it to onnx but so far without any success.<br>
So I have tried to narrow down the problem into a small code example to illustrate the problem.</p>
<p>torch version =1.7.1</p>
<pre><code>x = torch.rand([2, 3, 60, 80, 6]).to('cuda:0')
ag = torch.tensor([[[[[[ 10.,  13.]]],
      [[[ 16.,  30.]]],
      [[[ 33.,  23.]]]]],
    [[[[[ 30.,  61.]]],
      [[[ 62.,  45.]]],
      [[[ 59., 119.]]]]],
    [[[[[116.,  90.]]],
      [[[156., 198.]]],
      [[[373., 326.]]]]]])

class Test(nn.Module):

    def __init__(self, ancor):
        super().__init__()
        self.register_buffer('ancor',ancor)
    
    def forward(self, x):
        return x[...,2:4]*self.ancor[0]
</code></pre>
<p>this example is possible to run on the cuda device:</p>
<pre><code>model=Test(ag).to('cuda:0').eval()
x = x.to('cuda:0')
result = model(x)
</code></pre>
<p>but when I try to export it then it becomes a problem:</p>
<pre><code>fname = 'test.onnx'
torch.onnx.export(model, x, fname, verbose=False, opset_version=12)
</code></pre>
<p>part of the error message:<br>
/opt/conda/lib/python3.7/site-packages/torch/onnx/utils.py in _model_to_graph(model, args, verbose, input_names, output_names, operator_export_type, example_outputs, _retain_param_name, do_constant_folding, _disable_torch_constant_prop, fixed_batch_size, training, use_new_jit_passes, dynamic_axes)<br>
448     if do_constant_folding and _export_onnx_opset_version in torch.onnx.constant_folding_opset_versions:<br>
449         params_dict = torch._C._jit_pass_onnx_constant_fold(graph, params_dict,<br>
 → 450                                                             _export_onnx_opset_version)<br>
451         torch._C._jit_pass_dce_allow_deleting_nodes_with_side_effects(graph)<br>
452</p>
<p>RuntimeError: Input, output and indices must be on the current device</p>
<p>I do not understand the error message, both tensors are on the same device.<br>
Is this a bug or how do I avoid this error?</p>

username: ptrblck
cooked: <p>I don’t know, why this operation is failing, as I’m also unable to detect a device mismatch.<br>
Would you mind creating an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> so that we can track and check it?</p>

username: dangraf
cooked: <p>Thanks for checking. Im currently on a trip and are not able to run the script to show the system setup. If you have time, please send in the repport. Otherwize, I will do it tomorrow.(bug report: <a href="https://github.com/pytorch/pytorch/issues/52432" class="inline-onebox" rel="noopener nofollow ugc">Problem exporting to onnx with registered buffer · Issue #52432 · pytorch/pytorch · GitHub</a>)</p>
Given a 3D tensor M; how to construct an array of diagonal matrix in which elements from every diag(M[i])?

username: ElleryL
cooked: <p>Consider I have a 3D tensor</p>
<p><code>M = torch.randn(3,2,2)</code></p>
<p>I’m constructing another 3D tensor  <code>A</code> with shape <code>A.shape</code> is <code>(3,2,2)</code> such that</p>
<pre><code class="lang-auto">A[i] = torch.diag(torch.diag(M[i])) # for all i in len(M)
</code></pre>
<p>Well, my <code>M</code> here is on a computation graph (such as the output of neural network). So I’m looking for an efficient way of constructing <code>A</code> while not take it off from computation graph (still can backpropagate) because code like</p>
<pre><code class="lang-auto">A= []

for i in range(M.shape[0]):
   r.append(torch.diag(torch.diag(M[i])))
A = torch.stack(A)
</code></pre>
<p>It raises <code>RuntimeError: one of the variables needed for gradient</code></p>
<p>any ideas ? Thanks</p>

username: sven
cooked: <p>Didn’t see an answer here so:</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.diagonal.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.diagonal.html</a></p>
<p>You can specify the two dimensions over which you want to take the diagonal with <code>torch.diagonal</code>.</p>

username: Eta_C
cooked: <pre><code class="lang-python">assert M.ndim == 3
A = M * torch.eye(*M.shape[-2:]).repeat(M.shape[0], 1, 1)
</code></pre>
How can I save a tensor that's too big for torch.save?

username: ProGamerGov
cooked: <p>As an example, the following code will fail when you try to run it:</p>
<pre><code class="lang-auto">import torch
x = torch.zeros(2147483648) 
torch.save(x, 'fails_to_save.pt')
</code></pre>
<p>The error that the above code creates:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "C:\Users\PC\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 372, in save
    _save(obj, opened_zipfile, pickle_module, pickle_protocol)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 487, in _save
    zip_file.write_record(name, storage.data_ptr(), num_bytes)
TypeError: write_record(): incompatible function arguments. The following argument types are supported:
    1. (self: torch._C.PyTorchFileWriter, arg0: str, arg1: str, arg2: int) -&gt; None
    2. (self: torch._C.PyTorchFileWriter, arg0: str, arg1: int, arg2: int) -&gt; None

Invoked with: &lt;torch._C.PyTorchFileWriter object at 0x000001BDD3B083B0&gt;, 'data/1914873411776', 1915292139648, -8589934592
</code></pre>
<p>The specific tensor size that I got the error with was size: <code>(1281167, 1008, 2)</code>.</p>

username: ptrblck
cooked: <p>The issue is tracked <a href="https://github.com/pytorch/pytorch/issues/47201">here</a>.</p>
Access to Image Net for classification

username: Aryan_Asadian
cooked: <p>Hi,</p>
<p>I tried to download the ImageNet dataset for classification through torchvision, but I got an error message</p>
<p>The dataset is no longer publicly accessible</p>
<p>I have also submitted my request to <a href="http://image-net.org" rel="noopener nofollow ugc">image-net.org</a> to download the dataset, But I have not received any news from them. How can I  download ImageNet for classification?</p>

username: ptrblck
cooked: <p>Your workflow is correct and the dataset had to be removed due to the restrictions, so I think you would have to wait for the response.</p>
<p>In the meantime, you could check e.g. the <code>tiny-imagenet</code> hosted on <a href="https://www.kaggle.com/c/tiny-imagenet/data">Kaggle</a>.</p>
Can DNN only process some parts of image?

username: torch_fresh
cooked: <p>I want to process an image (not the whole image) like the figure below.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da27d53ce14df70b6f29761d1638290a86e6af84.jpeg" alt="fig1" data-base62-sha1="v7TqG08FFSRAgacSOyApXCaXZWY" width="640" height="480"></p>
<p>I don’t want the Faster-RCNN to process the part in the red area of the image. I know that I can change the pixels in the red area to black color. However, these black pixels still pass the Faster-RCNN and the inference time doesn’t decrease.</p>
<p>How can DNN only process the image outside the red area and skip the red area?</p>
<p>Thank you for any suggestion!</p>

username: ptrblck
cooked: <p>While you could theoretically avoid computing the tagged pixels and reduce the runtime, you would have to find fast algorithms, which are able to accept a random mask in the input image.<br>
E.g. you could try to write a custom convolution via <code>unfold</code> and remove the patches, which are tagged as invalid, and recreate the output. Once this is done you could compare the performance against a “standard” convolution via <code>unfold</code> and check the performance gains. Note that convolutions are implemented using optimized algorithms, which would most likely outperform the “masked unfold” approach even when they are using the complete image.</p>