akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to use AUTO_TUNE with pytorch?

username: Shubhankar
cooked: <p>How to use NVIDIA AUTO TUNE with pytorch.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I am not familiar with this? Could you explain what it is and what it does please?</p>

username: Shubhankar
cooked: <aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/tensorflow/tensorflow/issues/12871#issuecomment-370059148" target="_blank" rel="nofollow noopener">github.com/tensorflow/tensorflow</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/tensorflow/tensorflow/issues/12871#issuecomment-370059148" target="_blank" rel="nofollow noopener">About Deterministic Behaviour of GPU implementation of tensorflow</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2017-09-07" data-time="08:07:09" data-timezone="UTC">08:07AM - 07 Sep 17 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2018-09-21" data-time="18:57:17" data-timezone="UTC">06:57PM - 21 Sep 18 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/antares1987" target="_blank" rel="nofollow noopener">
          <img alt="antares1987" src="https://avatars0.githubusercontent.com/u/31724739?v=4" class="onebox-avatar-inline" width="20" height="20">
          antares1987
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Debian
TensorFlow version (use command below):('v1.3.0-rc2-20-g0787eee', '1.3.0')
CUDA/cuDNN version: 8.0
GPU model and memory: GeForce GTX...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">stat:awaiting response</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: ptrblck
cooked: <p>I’m not familiar with TensorFlow, but it seems their “auto tune” functionality corresponds to <code>torch.backends.cudnn.benchmark = True</code> mode.</p>
Adding a new output neuron part way during the training

username: rahulvigneswaran
cooked: <p>For example, initially, there are 10 classes / 10 output neuron in the model and I want to train them for 50 epochs. When I complete training for 50 epochs, I want to add one extra class/output neuron making the total class and output neuron count as 11 and then train for 50 more epochs. This is different from <a href="https://discuss.pytorch.org/t/possible-to-add-initialize-new-nodes-to-hidden-layer-partway-through-training/3809">this post</a> because this deals with the output layer neuron instead of the hidden layer.</p>

username: ptrblck
cooked: <p>The difference between a hidden and output layer should not make the suggested approaches invalid, should it?<br>
I think you could still add the parameter during the training. However, I’m currently unsure about dealing with the optimizer. E.g. would you like to reinitialize the optimizer after changing the layer?</p>
Does Pre-Training Work Internally in pytorch?

username: shashankMadan
cooked: <p>I am using the library called transformers by hugging face.<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/huggingface/transformers" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/25720743?s=400&amp;v=4" class="thumbnail onebox-avatar" width="175" height="175">

<h3><a href="https://github.com/huggingface/transformers" target="_blank" rel="nofollow noopener">huggingface/transformers</a></h3>

<p>🤗 Transformers: State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch. - huggingface/transformers</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
And It uses transfer learning extensively. Like using bert’s pretrained model in <code>from_pretrained</code> and coming across it’s fine tuning code we can save the new model weights and other hyper params in <code>save_pretrained</code>.<br>
My doubt is in there <code>modeling_bert</code> code there is no explicit code that takes the pre-trained weights in acount and then trains as it generally takes attention matrices and puts it in a feed forward network</p>
<pre><code class="lang-auto">self.decoder = nn.Linear(config.hidden_size,
                                 config.vocab_size,
                                 bias=False)
</code></pre>
<p>And here i do not see any kind of “inheritance” of the pre-trained weights…<br>
So is it internally handled by pytorch or am i missing something?</p>

username: ptrblck
cooked: <p>Could you link to the functions you have questions about, as the repository is quite large, which makes it hard to follow your questions.<br>
Generally, I think the Huggingface devs might have a faster and better answer, so maybe <a class="mention" href="/u/thomas_wolf">@Thomas_Wolf</a> finds some time to answer your question. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
I saved a dataparallel version of model using model.module.state_dict() but was able to recover the model as model.state_dict()

username: Shubhankar
cooked: <p>I saved my network as</p>
<pre><code class="lang-auto">checkpoint = {'epoch': epoch, 'loss': loss, 'model_state_dict': network.module.state_dict(), 
                       'optimizer_state_dict': optimizer.state_dict()}
                       torch.save(checkpoint, 'check-point.pth')
</code></pre>
<p>but was able to recover it using</p>
<pre><code class="lang-auto">with open('check-point.pth', 'rb') as f:
            checkpoint = torch.load(f)
            network.load_state_dict(checkpoint['model_state_dict'])
            optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
            epoch = checkpoint['epoch']
</code></pre>
<p>though I had to do <code>nn.DataParallel(network)</code> afterwards. Shouldn’t it through an error though since to recover the model <code>network.module.load_state_dict()</code> should have been used?</p>

username: ptrblck
cooked: <p>If <code>network</code> is a “standard” model, i.e. not wrapped in <code>nn.DataParallel</code>, the workflow should be fine.<br>
Once you wrap your model in <code>nn.DataParallel</code>, you would have to load the <code>state_dict</code> via <code>network.module.load_state_dict</code>.</p>
<p>I’m not sure I understand the question correctly, but why do you think it should thrown an error?</p>
Normalizing flow with 1 input

username: smu226
cooked: <p>Hello! I want to apply normalizing flow to some numerical data. Depending on the situation, the distribution I want to model can be a function of only one variable (for example a Lorentzian distribution) or of several (for example a multivariate gaussian). In the case of one number (say I have 10,000 numbers generated from a lorentzian distribution, with one variable), what is the best normalizing flow architecture I can use (I am aware that this might be an overkill, but I really need to use a normalizing flow)? The architectures I find on GitHub requires at least 2 numbers (for example Nice, or RealNVP), to be able to mix channels, but I have just one number as output, so I would need just one as input. Also, mathematically speaking, normalizing flows should work even for 1 number input/output. What should I use? Thank you!</p>
How many optimizers do I need?

username: 1414b35e42c77e0a57dd
cooked: <p>Hi, all.<br>
I’m wondering how many optimizers are needed to build some architectures.<br>
If my architectures are composed with 3-classifiers and two-encoders,</p>
<pre><code class="lang-auto">         -&gt; Classifer1
Encoder1 -&gt; Encoder2 -&gt; Classifier2
         -&gt; Classifier3
</code></pre>
<p>One optimizer can train this model well?<br>
When I use one optimizer, I sum up all losses.(In this case, three kinds of classification loss). So I’m not sure that three modules are well trained simultaneously.</p>

username: ptrblck
cooked: <p>The number of losses and optimizers do not depend on each other.<br>
E.g. you could use a single optimizer for the complete model to apply the same optimization method to all parameters.<br>
The loss(es) determine the gradient calculation. Calling <code>.backward()</code> on each loss separately or summing the losses and calling <code>backward</code> once will result in the same gradients.</p>
How to use submodule in onnx?

username: dalalaa
cooked: <p>I’m converting an ocr model to onnx through script-based onnx exporter like <a href="https://pytorch.org/docs/stable/onnx.html#tracing-vs-scripting" rel="nofollow noopener">https://pytorch.org/docs/stable/onnx.html#tracing-vs-scripting</a></p>
<p>But I met this error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "ocr.py", line 52, in &lt;module&gt;
    class OCR(nn.Module):
  File "ocr.py", line 59, in OCR
    @torch.jit.script
  File "/home/dai/py36env/lib/python3.6/site-packages/torch/jit/__init__.py", line 1226, in script
    fn = torch._C._jit_script_compile(qualified_name, ast, _rcb, get_default_args(obj))
RuntimeError:
Unknown builtin op: aten::east.
Here are some suggestions:
        aten::hash
        aten::wait
        aten::dist
        aten::list
        aten::cat

The original call is:
at ocr.py:75:16
        TGT_WIDTH = 800
        if w_raw &gt; TGT_WIDTH:
            ratio = w_raw / TGT_WIDTH
            img_tensor = torch.ops.my_ops.resize(img_tensor.float(),TGT_WIDTH, int(h_raw / ratio),3)
        img_tensor = resize_img(img_tensor)
        # self.show(img_tensor)

        img_tensor_ = (img_tensor - 0.5) / 0.5
        img_tensor_ = img_tensor_.permute(2,0,1).unsqueeze(0) # --&gt; n x c x h x w
        boxes = self.east(img_tensor_)
                ~~~~~~~~~ &lt;--- HERE
        # self.show(img_tensor,boxes)
        new_boxes = []
        pass_list:List[int] = []
        for i in range(boxes.size(0)):
            for j in range(i+1,boxes.size(0)):
                if i in pass_list or j in pass_list:
                    continue
                cx1,cy1,angle1 = box_analyse(boxes[i])
                cx2,cy2,angle2 = box_analyse(boxes[j])
</code></pre>
<p>self.east is a detection model, how can I export the whole ocr model(detection and recognition model) to onnx?</p>
Occasional Nan Loss using custom CUDA kernels

username: kevin_zhang
cooked: <p>Hi,</p>
<p>I implemented a RNN with custom fused kernel using CUDA. When I train it using manual seed, it would sometimes produce NaN loss from the very beginning (first batch). However, other times when I run the same program, the losses are real.</p>
<p>Also this only seems to be an issue when I use my custom CUDA kernel. When I use the Python version (using PyTorch functions), it doesn’t occur. Even if it’s running on GPU.</p>
<p>The stack trace is shared below. Any advice would be greatly appreciated.</p>
<p>/home/wz1232/anaconda3/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn’t retrieve source code for container of type CrossEntropyLoss. It won’t be checked for correctness upon loading.<br>
"type " + obj.<strong>name</strong> + ". It won’t be checked "<br>
/opt/conda/conda-bld/pytorch_1573049306803/work/torch/csrc/autograd/python_anomaly_mode.cpp:57: UserWarning: Traceback of forward call that caused the error:<br>
File “train_adam.py”, line 389, in <br>
global_step = train(global_step)<br>
File “train_adam.py”, line 329, in train<br>
loss = criterion(output.view(-1, ntokens), targets)<br>
File “/home/wz1232/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 541, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/wz1232/anaconda3/lib/python3.7/site-packages/torch/nn/modules/loss.py”, line 916, in forward    ignore_index=self.ignore_index, reduction=self.reduction)<br>
File “/home/wz1232/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py”, line 2009, in cross_entropy<br>
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)<br>
File “/home/wz1232/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py”, line 1317, in log_softmax<br>
ret = input.log_softmax(dim)</p>
<p>Traceback (most recent call last):<br>
File “train_adam.py”, line 389, in <br>
global_step = train(global_step)<br>
File “train_adam.py”, line 330, in train<br>
loss.backward()<br>
File “/home/wz1232/anaconda3/lib/python3.7/site-packages/torch/tensor.py”, line 166, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/wz1232/anaconda3/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 99, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: Function ‘LogSoftmaxBackward’ returned nan values in its 0th output.<br>
terminate called without an active exception<br>
Aborted (core dumped)</p>
Can someone explain, what is returned by the backward hook?

username: Umair_Javaid
cooked: <p>This is an example of backward hook applied on a Conv2d. Can someone explain what is being returned by this hook and how it is being calculated?</p>
<pre><code class="lang-auto">conv_model.conv2.register_backward_hook:  Conv2d(1, 4, kernel_size=(1, 1), stride=(1, 1)) (tensor([[[[0.0112, 0.0112, 0.0112, 0.0112, 0.0112],
          [0.0112, 0.0112, 0.0112, 0.0112, 0.0112],
          [0.0112, 0.0112, 0.0112, 0.0112, 0.0112],
          [0.0112, 0.0112, 0.0112, 0.0112, 0.0112],
          [0.0112, 0.0112, 0.0112, 0.0112, 0.0112]]]]), tensor([[[[-0.0459]]],


        [[[-0.0650]]],


        [[[-0.5905]]],


        [[[-0.1576]]]]), tensor([0.1069, 0.1513, 1.3748, 0.3670])) (tensor([[[[0.0043, 0.0043, 0.0043, 0.0043, 0.0043],
          [0.0043, 0.0043, 0.0043, 0.0043, 0.0043],
          [0.0043, 0.0043, 0.0043, 0.0043, 0.0043],
          [0.0043, 0.0043, 0.0043, 0.0043, 0.0043],
          [0.0043, 0.0043, 0.0043, 0.0043, 0.0043]],

         [[0.0061, 0.0061, 0.0061, 0.0061, 0.0061],
          [0.0061, 0.0061, 0.0061, 0.0061, 0.0061],
          [0.0061, 0.0061, 0.0061, 0.0061, 0.0061],
          [0.0061, 0.0061, 0.0061, 0.0061, 0.0061],
          [0.0061, 0.0061, 0.0061, 0.0061, 0.0061]],

         [[0.0550, 0.0550, 0.0550, 0.0550, 0.0550],
          [0.0550, 0.0550, 0.0550, 0.0550, 0.0550],
          [0.0550, 0.0550, 0.0550, 0.0550, 0.0550],
          [0.0550, 0.0550, 0.0550, 0.0550, 0.0550],
          [0.0550, 0.0550, 0.0550, 0.0550, 0.0550]],

         [[0.0147, 0.0147, 0.0147, 0.0147, 0.0147],
          [0.0147, 0.0147, 0.0147, 0.0147, 0.0147],
          [0.0147, 0.0147, 0.0147, 0.0147, 0.0147],
          [0.0147, 0.0147, 0.0147, 0.0147, 0.0147],
          [0.0147, 0.0147, 0.0147, 0.0147, 0.0147]]]]),)
</code></pre>
<p>Following is my model:</p>
<pre><code class="lang-auto">class try_conv_model(nn.Module):
  def __init__(self):
    super(try_conv_model, self).__init__()
    self.conv1 = nn.Conv2d(1, 1, kernel_size=1, bias=False)
    self.conv2 = nn.Conv2d(1, 4, kernel_size=1)
    self.gap = nn.AdaptiveAvgPool2d((1,1))
  
  def forward(self, x):
    print("-"*150)
    print("before conv1: ",x)
    x = self.conv1(x)
    print("before conv2: ",x)
    x = self.conv2(x)
    print("before AdaptiveAvgPool2D: ",x)
    x = self.gap(x)
    print("after AdaptiveAvgPool2D: ",x)
    x = x.view(-1, self.num_flat_features(x))
    return(x)

  def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features
  
conv_model = try_conv_model()
</code></pre>
user_suspended: yes

username: albanD
cooked: <p>As I mentioned in your other post. You want to check the warning in the doc <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.register_backward_hook">here</a></p>
Kernel dies creating SummaryWriter

username: funchansu
cooked: <p>Dear Community,</p>
<p>I just started using pytorch and was following the tutorial from this link: <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Visualizing Models, Data, and Training with TensorBoard — PyTorch Tutorials 2.2.0+cu121 documentation</a>. However, whenever I ran<br>
“writer = SummaryWriter(‘runs/fashion_mnist_experiment_1’)”<br>
jupyter notebook will report “the kernel appears to have died. It will restart automatically”. I have no idea why such a simple command will cause the kernel to die, since I’m not even training a model. Information is also shown in the screen shot below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/8/78dd83b85dfd315c07cb115efd4cccde92b0c159.png" data-download-href="https://discuss.pytorch.org/uploads/default/78dd83b85dfd315c07cb115efd4cccde92b0c159" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/78dd83b85dfd315c07cb115efd4cccde92b0c159_2_690x430.png" alt="image" data-base62-sha1="hfdT7TiGizTFSsCpap7LlySZQmZ" width="690" height="430" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/78dd83b85dfd315c07cb115efd4cccde92b0c159_2_690x430.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/78dd83b85dfd315c07cb115efd4cccde92b0c159_2_1035x645.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/78dd83b85dfd315c07cb115efd4cccde92b0c159_2_1380x860.png 2x" data-dominant-color="BCBDBD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2304×1436 268 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The server I’m working on doesn’t have a GPU. Could this have been the reason?</p>
<p>I would really appreciate it for any help and hints.<br>
Thank you so much</p>

username: albanD
cooked: <p>Hi,</p>
<p>This is quite surprising. Can you try to run the same command from a command line to see if you get a more useful error message?</p>
2d mask gradient issue

username: angadi
cooked: <p>I am changing vgg.py for my usage, I created a 2d mask of one of the filter which indicates top 10 elements but, i am doubtful if this affects gradients and training process. Any help will be appreciated</p>
<pre><code class="lang-auto">    def forward(self, x):
        x = self.features(x)
        val,ind = torch.topk(torch.flatten(x,1),10)
        mask_tensor = temp.permute(1,2,3,0) &gt;= val[:,-1]
        x = x*mask_tensor.permute(3,0,1,2)
        x = self.avgpool(x)
        x = torch.flatten(x, 1)
        x = self.classifier(x)
        return x
</code></pre>

username: albanD
cooked: <p>Since the <code>&gt;=</code> operation is not differentiable, no gradient will flow back towards the <code>mask_tensor</code>.<br>
And the gradient will be zeroed out for all the values where the mask is <code>0</code>.</p>
How can I get weak_intrusive_ptr from a raw pointer

username: MarisaKirisame
cooked: <p>Hi, I am trying to hack in the pytorch codebase, however I need to convert from raw pointer to weak_intrusive_ptr. uppon reading the intrusive_ptr.h file multiple time I cannot find anything that does what I want. Is it possible?</p>

username: albanD
cooked: <p>I think you want <a href="https://github.com/pytorch/pytorch/blob/489dd6cb9098f2c7ffd97b6f3ea17bbdbef12806/c10/util/intrusive_ptr.h#L385">c10::make_intrusive</a>.<br>
You can see it being used <a href="https://github.com/pytorch/pytorch/blob/bfbb3e0579d239713b70a47034938b69b751a204/aten/src/TH/generic/THTensor.cpp#L59">here</a> for example.</p>
Change 1x1x33 tensor to 1x1x34

username: Dylan_Yung
cooked: <p>So let’s say for a smaller example I have the tensor that’s 1x2<br>
[0,1]</p>
<p>I want to change it to<br>
[0,1,0]<br>
So append an extra column but leave everything else to same. Now I want to do this to a 3D tensor that is 1x1x33 to 1x1x34</p>

username: ptrblck
cooked: <p><code>torch.cat</code> should do exactly this:</p>
<pre><code class="lang-python">torch.cat((torch.randn(1, 1, 33), torch.randn(1, 1, 1)), dim=2)
</code></pre>

username: Eta_C
cooked: <pre><code class="lang-python">import torch
import torch.nn.functional as F

x = torch.rand(1, 1, 33)
y = F.pad(x, (0, 1, 0, 0), mode='constant', value=0)
print(x.shape)
print(y.shape)
</code></pre>
<p>also works but is not recommended.</p>
Why bce loss out put differing

username: Jaideep_Valani
cooked: <p>How pytorch calculates bce loss if output is bs,nc,h,w that is in case of segmentation<br>
I obtained  different results when I do<br>
X= -Mask <em>log( sigmoid (output) -(1-mask)</em> log(1-sigmoid(1-output) )<br>
X.mean(0).sum() ,<br>
here  it start from 3k ,ends in nan</p>
<p>And when I  use f.bce with logits range of loss 0.5 to 1<br>
Why is there a difference .m I missing any step for aggregating bce loss. There is no nan here .</p>

username: ptrblck
cooked: <p>Your formula might create Infs or NaNs, e.g. if you call <code>torch.log</code> on a zero or a negative input.<br>
What’s the reason you are reimplementing the bce loss manually?</p>

username: Jaideep_Valani
cooked: <p>i figured out the NaN reason ,i was using mixed FP 1e-12 was rounding to zero so i changed that to 1e-7. that got resolved.</p>
<p>For the difference part<br>
objective is to predict vehicle or object presence on road on certain spatial location coordinates from Driverless cars’ camera point of view.<br>
While creating label i created Mask of shape X,Y  whose non zero pixel  corresponds to 2d pixel position in camera space projected from its 3d position on road  .Basically model will detect car at position based on masks non zero pixel.<br>
Custom one gives values in thousands but comes down fast ,standard one gives 0 to 1 and stagnates after 0.01.</p>
P2P Cuda Aware MPI problem

username: seliad
cooked: <p>Hello,<br>
I’m trying to use Isends with cuda aware openmpi.</p>
<p>I found that I need to explicitly call torch.cuda.synchronize(device) before every Isend (otherwise training error collapses). I get that problem even when I stash the sent tensor (so it will have a reference and therefore won’t be freed and overwritten).</p>
<p>I have tried it with several different settings:</p>
<ol>
<li>with P2P enabled GPU (GTX1080)</li>
<li>and without P2P enabled GPUs (RTX2080ti). (in the latter case the sends must go through the host).</li>
</ol>
<p>I wonder, what could be happening there?<br>
(I am using a single thread with async operations)</p>
How to manually delete/free a tensor in ATen?

username: Charles-Xie
cooked: <p>I’m trying to build a custom C++ extension using the aten api.<br>
Does aten library provide a simple way to manually delete/free a tensor? Please take a look at the example below:</p>
<pre><code class="lang-cpp">// create a tensor
torch::Tensor tensor = torch::randn({3,4,5});
// manually delete this tensor
delete tensor;    // something like this
</code></pre>
<p><strong>The target is to free the memory of some large tensors in a function before the function ends, in order to save the total memory usage and avoid ‘CUDA out of memory’.</strong></p>
<p>I understand there might be some tricks like:</p>
<pre><code class="lang-cpp">// option 1
torch::Tensor * tensor = torch::Tensor new(torch::randn({3,4,5}));
delete tensor;
</code></pre>
<p>and</p>
<pre><code class="lang-cpp">// option 2
torch::Tensor tensor = torch::randn({3,4,5});
// free memory usage by setting it smaller
tensor.resize_(at::IntArrayRef{0});
</code></pre>
<p>But is there a simpler and more straightforward way to achieve this?</p>
<p>Note that I’ve already searched the forums and read this related question <a href="https://discuss.pytorch.org/t/how-to-free-gpu-memory-of-at-tensor-in-aten-c/18408" class="inline-onebox">How to free GPU memory of at::Tensor in ATen, C++?</a> but still got no answer.</p>
<p><strong>Another question</strong>:<br>
for an operation like</p>
<pre><code class="lang-cpp">torch::Tensor a = a.clone();
</code></pre>
<p>will tensor <code>a</code> be automatically freed? I’m not sure whether this will cause memory leak.</p>

username: tom
cooked: <p>So I also added it to the other thread:</p>
<p>When a Tensor (or all Tensors referring to a memory block (a Storage)) goes out of scope, the memory goes back to the cache PyTorch keeps. You can free the memory from the cache using</p>
<p><code>#include &lt;c10/cuda/CUDACachingAllocator.h&gt;</code></p>
<p>and then calling</p>
<p><code>c10::cuda::CUDACachingAllocator::emptyCache();</code></p>
<p>(of course, you could try using <code>torch::</code> instead of <code>c10::</code> and see if it is automatically imported somewhere).</p>
<p>Best regards</p>
<p>Thomas</p>

username: Charles-Xie
cooked: <p>Hi Thomas,<br>
Thanks a lot for you reply. I guess I might have not stated the problem clearly. My target is to free the memory of some large tensors manually in a function before these tensors are out of the function’s scope, in order to save the total memory usage and avoid “CUDA out of memory” error.<br>
I have edited the problem description. Can you take a look again? Thanks!</p>

username: tom
cooked: <p>Yeah well, I usually just introduce a local scope for that.</p>
<pre><code class="lang-auto">void myfunc() {
  some code here
  {
   torch::Tensor a = ... 
  }
  here a is gone
}
</code></pre>
<p>But if you must: Assigning the default-constructed undefined Tensor would likely work, too.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Charles-Xie
cooked: <p>That helps! Thanks a lot.</p>
Which one is correct to implement global adaptive pooling 3d?

username: John1231983
cooked: <p>I found three ways to implement global adaptive pooling for tensor size of <code>BxCxDxHxW</code>, or 5D tensor</p>
<p>Way 1</p>
<pre><code class="lang-auto">out = F.adaptive_avg_pool3d(input, kernel=(input.size(2),input.size(3), input.size(4)), stride =(1,1,1)
out = out.view(input(0), input(1))
</code></pre>
<p>Way 2:</p>
<pre><code class="lang-auto">self.avg_pool=nn.AdaptiveAvgPool3d(1)
out=self.avg_pool(input)
out = out.view(input(0), input(1))
</code></pre>
<p>Way3:</p>
<pre><code class="lang-auto">out= input_tensor.view(input.size(0), input.size(1), -1).mean(dim=2)
out = out.view(input(0), input(1))
</code></pre>
<p>If three ways are same, which one is recommended ?</p>

username: YFlag
cooked: <p>The three ways are same, but I’m kind of confused about implementation <span class="hashtag">#1:</span> shouldn’t you specify a <code>output_size</code> parameter either, like you do in implementation <span class="hashtag">#2</span>?</p>
When to use Module based loss functions vs functional loss functions

username: Sam_Pr
cooked: <p>When would we choose to use the <code>Module</code> forms of the loss functions rather than the functional forms? For example, <code>torch.nn.MSELoss</code> over <code>torch.nn.functional.mse_loss</code>. I just realised that I’ve never used the <code>Module</code> forms, and I’m wondering whether I’m missing a useful trick or something? It seems to me that loss is something we would almost always want to calculate outside of the model, so it doesn’t make intuitive sense for it to be a <code>Module</code>.</p>

username: ptrblck
cooked: <p>I’m usually creating the criterion as a module in case I want to store some internal states, e.g. <code>weight</code>, a specific reduction etc.<br>
Also, I would say it basically depends on your coding style and the use case you are working with.<br>
E.g. if you are reusing the criterion in multiple places (e.g. GAN training) and would like to experiment with different loss functions as well as reductions, a single instance might be more convenient to use instead of the functional approach.</p>
<p>EDIT: A bit unrelated to this exact question, but I’ve given my point of view on the usage of modules vs. functional API <a href="https://discuss.pytorch.org/t/beginner-should-relu-sigmoid-be-called-in-the-init-method/18689/6">in this post</a>,</p>

username: Sam_Pr
cooked: <p>Thanks, that’s in line with how I’m thinking about it now. It’s good to know I wasn’t missing anything obvious.</p>
How to call a function on the Dataset for every batch in Dataloader?

username: jwillette
cooked: <p>I am using a model where I want to initialize some random state in my Dataset that decides how many context points to add to the current instance (from this paper: <a href="https://arxiv.org/pdf/1807.01613.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1807.01613.pdf</a>)</p>
<p>I can’t figure out how to call a function on each batch. The only requirements I have is that</p>
<ol>
<li>It can be seen by the Dataset object so I can use it in its <code>__getitem__</code> method.</li>
<li>It will be called once for every batch.</li>
</ol>
<p>Is there a way to accomplish this?</p>

username: ptrblck
cooked: <p>You could probably create an internal attribute in your <code>Dataset</code>'s <code>__init__</code> and manipulate this attribute inside the training loop via: <code>loader.dataset.my_flag = my_value</code>.<br>
This <code>.my_flag</code> could be checked in <code>__getitem__</code>.</p>
<p>However, if I’m not mistaken, this will only work using a single process.<br>
For multiple workers in your <code>DataLoader</code>, a shared array could probably work as the flag as given in <a href="https://github.com/ptrblck/pytorch_misc/blob/master/shared_array.py">this small example</a>.</p>
Torchvision datasets access

username: Rocket
cooked: <p>Hi,</p>
<p>How to access datasets label?</p>
<pre><code class="lang-auto">import torchvision.datasets
fashion_train = torchvision.datasets.FashionMNIST(root = './data/torchvision/fashion_mnist', train = True, download = True)
</code></pre>
<p>data could be accessed by<br>
<code>fashion_train.data[0]</code></p>
<p>how to access labels?</p>
<p>is there attribute for checking keys?</p>

username: ptrblck
cooked: <p><code>fashion_train.targets</code> should work.</p>
Removing models from GPU in loop

username: stonecold
cooked: <p>Hello, I am currently trying to progressively “widen” my network in a loop. The reason I am using the loop is because I want to compare some properties of the output as I increase the size. I can’t think of anything simpler than the following. I create a network with width that can be configured from the beginning</p>
<pre><code class="lang-python">class onelayerNet(nn.Module):
    def __init__(self,scale):
        super(onelayerNet, self).__init__()
        self.fc1 = nn.Linear(10, 10*scale)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        return x
</code></pre>
<p>And then have the following loop</p>
<pre><code class="lang-python">Net = {}
for i in range(iters):
    with torch.no_grad():
        scale=10*(i+1)
        Net[i] = onelayerNet(scale=scale)

        if i&gt;0: #copy weights
            index =10*10*i #get shape of previous network
            Net[i].fc1.weight[0:index,:] = Net[i-1].fc1.weight[:,:] #copy weights
        
        Net[i].to(device)
        output = Net[i].forward(images);

        # do some stuff to the output

</code></pre>
<p>However, I can’t find anyway to remove the previous models from the GPU (as well as intermediate variables that change on each iteration like ‘output’). I’ve tried using del Net[i-1] after copying it’s weights but this doesn’t get removed from the GPU. I’ve tried ’ Net[i].to(‘cpu’)’ after computing the output, however this doesn’t help either.</p>
<p>Is there a smarter way to do this such that I won’t run into this issue? If not, is there a way I can remove unnecessary things from the GPU after they are used?</p>

username: ptrblck
cooked: <p>If you delete the model and no other references are pointing to it, it will be freed.<br>
Note that PyTorch uses a custom memory caching allocator, to avoid multiple memory allocations.<br>
<code>nvidia-smi</code> will thus still show the GPU memory as used by PyTorch.</p>
<p>I would recommend to <code>.clone()</code> the parameters just to make sure these parameters are not referencing each other.</p>
What are the difference between JIT and tensor_comprehensions for custom kernels?

username: Enamex
cooked: <p>Specifically, when writing TC-like loops in JIT-ed functions.</p>
<p>My issues are:</p>
<ol>
<li>I haven’t been able to get good performance out of <code>jit.script</code>. My use case might be a little too dynamic?</li>
<li>When JIT-ing, I have no control on any kinds of optimizations. I can’t nudge the jitter to fuse a particular sequence of operations, for example. So I can’t make use of the jitter to eliminate OOM errors.</li>
<li>Trying to do it with loops is generally slower than using existing maps and reductions with (memory-hungry) intermediates.</li>
<li>
<code>tensor_comprehensions</code> is not available on Windows, as far as I can see <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:">
</li>
</ol>
What's Optional[Tensor]?

username: dalalaa
cooked: <p>I got this error while calling tensor.size(0) in my project:</p>
<pre><code class="lang-auto">RuntimeError:
Arguments for call are not valid.
The following operator variants are available:

  aten::size.int(Tensor self, int dim) -&gt; (int):
  Expected a value of type 'Tensor' for argument 'self' but instead found type 'Optional[Tensor]'.

  aten::size(Tensor self) -&gt; (int[]):
  Expected a value of type 'Tensor' for argument 'self' but instead found type 'Optional[Tensor]'.

The original call is:
at ocr.py:78:23
        img_tensor = resize_img(img_tensor)
        # self.show(img_tensor)

        img_tensor_ = (img_tensor - 0.5) / 0.5
        img_tensor_ = img_tensor_.permute(2,0,1).unsqueeze(0) # --&gt; n x c x h x w
        boxes = self.east(img_tensor_)
        # self.show(img_tensor,boxes)
        new_boxes = []
        pass_list:List[int] = []
        for i in range(boxes.size(0)):
                       ~~~~~~~~~~ &lt;--- HERE
</code></pre>
<p>How can I get the first dimension size of a tensor in torchscript?</p>

username: huoge
cooked: <p><a href="https://pytorch.org/docs/stable/jit.html#optional-type-refinement" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/jit.html#optional-type-refinement</a><br>
here you can get information about Optional</p>
Select tensor with different dims

username: xingjici
cooked: <p>I have tow tensor :  Box[20,4] and Match[512] --&gt;(0,1,2,6,9,20,19…)<br>
I want to produce  a subset of Box given the Match resulting in MatchedBox[512,4]<br>
How to do that(with out for loop or a function)?<br>
Thank you for your time.</p>

username: Eta_C
cooked: <p>Just use it directly. Example:</p>
<pre><code class="lang-python">import torch

boxes = torch.rand(5, 4)
indices = torch.randint(0, 5, (10,))

print(boxes)
print(indices)
print(boxes[indices, :])
</code></pre>
Which function can do this?

username: dali_wang
cooked: <pre><code class="lang-auto">A_selected, idxs = torch.min(A,dim=1) #A.shape = [b,2,h,w]
</code></pre>
<p>If I have ‘A’ and ‘idxs’, What should I do to get ‘A_selected’  without min function?</p>

username: Eta_C
cooked: <p>A_selected = torch.min(A,dim=1)</p>

username: dali_wang
cooked: <p>I’m sorry, there’s something wrong with my question</p>

username: Eta_C
cooked: <pre><code class="lang-python">import torch
x = torch.rand(1, 2, 3, 4)
answer, index = torch.min(x, dim=1)

my_result = torch.gather(x, 1, index[:, None, ...]).squeeze(1)
print((my_result == answer).all())  # True
</code></pre>

username: dali_wang
cooked: <p>Problem solved, Thank you soooooo much~~! <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=9" title=":laughing:" class="emoji" alt=":laughing:"></p>
Get this error: WARNING:root:NaN or Inf found in input tensor

username: gebrahimi
cooked: <p>I am trying to run alexnet on cifar-10 and i get this warning and after that the loss values are NaN:</p>
<p>WARNING:root:NaN or Inf found in input tensor.</p>
<p>I check my input with the following command and it does not show if there is an inf or nan in the input:</p>
<pre><code>        assert torch.sum(torch.isnan(input)).item() == 0
        assert torch.sum(torch.isinf(input)).item() == 0
</code></pre>
<p>What can be wrong?</p>

username: Eta_C
cooked: <p>All feature maps are also not nan and inf?</p>
Argmax_cuda" not implemented for 'Bool

username: Muhammad_Haris
cooked: <p>I am getting this error when I run the PyTorch code, I try to change argmax to someother datatype but not working.<br>
please help</p>
<pre><code class="lang-auto"> def _mask(prev_generated_seq):
        prev_mask = torch.eq(prev_generated_seq, 1)
        lengths = torch.argmax(prev_mask,dim=1)
        max_len = prev_generated_seq.size(1)
        mask = []
        for i in range(prev_generated_seq.size(0)):
            if lengths[i] == 0:
                mask_line = [0] * max_len
            else:
                mask_line = [0] * lengths[i].item()
                mask_line.extend([1] * (max_len - lengths[i].item()))
            mask.append(mask_line)
        mask = torch.ByteTensor(mask)
        if args.cuda:
            mask = mask.cuda()
        return prev_generated_seq.data.masked_fill_(mask, 0)
</code></pre>
<pre><code class="lang-auto">**Error**

</code></pre>
<p>File “main.py”, line 179, in <br>
train_epoches(abstracts, model, config.epochs, teacher_forcing_ratio=1)<br>
File “main.py”, line 155, in train_epoches<br>
target_variables, model, teacher_forcing_ratio)<br>
File “main.py”, line 139, in train_batch<br>
prev_generated_seq = _mask(prev_generated_seq)<br>
File “main.py”, line 101, in _mask<br>
lengths = torch.argmax(prev_mask,dim=1)<br>
RuntimeError: “argmax_cuda” not implemented for ‘Bool’</p>
<pre><code class="lang-auto"></code></pre>

username: huoge
cooked: <p>there is a duplicate question:<a href="https://discuss.pytorch.org/t/runtimeerror-argmax-cuda-not-implemented-for-bool/58808" class="inline-onebox">RuntimeError: "argmax_cuda" not implemented for 'Bool'</a></p>

username: Muhammad_Haris
cooked: <p>yeah i already check that but didnt found answer which resolve my error</p>
How to perform several different transforms to the same batch image in an efficient way

username: neouyghur
cooked: <p>Hi, I am trying to get several different transformations of the same batch images?  I checked this git repository <a href="https://github.com/pytorch/vision/issues/157" rel="nofollow noopener">https://github.com/pytorch/vision/issues/157</a>, but I am not sure which one is the best way to do that?</p>
What is the most efficient way to get the closest pair of a set of vectors?

username: flyaway
cooked: <p>I have a matrix M in size of n*d, where n is the number of vectors and d is the dimension of each vector. Now, I want to find the closest pair of this n vectors. Be note, I only need the closest pair.<br>
I have two ways of doing this:</p>
<ol>
<li>Use the <code>PairwiseDistance</code> and <code>argmin</code> function. However, this method uses too much memory. Half of the returned matrix is useless.</li>
<li>
<code>functional.pdist</code> works well, it does not double computing. However, this function returns a condensed vector. I do not know how to recover the (i,j) index from this condensed vector.</li>
</ol>
<p>The input matrix M changes and need to recompute the closest pair in every iteration. So, I want the whole process to be as efficient as possible.</p>
<p>Any suggestions about solving this efficiently?</p>

username: flyaway
cooked: <p>I solved this by recovering the <code>(i,j)</code> index from the <code>k</code> index in the <code>functional.pdist</code>.<br>
The code:</p>
<pre><code class="lang-python">def get_index(k, n):
    p = n-1
    kk = k
    j = 1
    i = 0
    while kk &gt;= p:
        kk = kk-p
        p = p-1
        j += 1
        i += 1
    return i, j+kk

</code></pre>

username: KFrank
cooked: <p>Hi Fly!</p>
<aside class="quote no-group quote-modified" data-username="flyaway" data-post="1" data-topic="64649" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/flyaway/48/6551_2.png" class="avatar"> flyaway:</div>
<blockquote>
<p>I have a matrix M in size of n*d, where n is the number of vectors and d is the dimension of each vector. Now, I want to find the closest pair of this n vectors.</p>
<ol start="2">
<li><code>functional.pdist</code> works well, it does not double computing.</li>
</ol>
<p>Any suggestions about solving this efficiently?</p>
</blockquote>
</aside>
<p>In practice, for reasonable values of <code>n</code> and <code>d</code>, I expect that<br>
<code>pdist</code> will do about as well as you can.</p>
<p>However, the so-called computational cost of your <code>pdist</code><br>
solution is a suboptimal <code>O(d * n^2)</code>.  If you search on<br>
<em>closest-pair problem,</em> you will see that your problem can<br>
be solved in <code>O(d * n * log n)</code> time.  However, these<br>
“faster” algorithms don’t vectorize naturally, so you give<br>
up much of the benefit of using a gpu (and even pipelines<br>
in a cpu).  Thus your “naive” <code>pdist</code> solution may well run<br>
faster (even as it requires more operations).</p>
<p>Good luck.</p>
<p>K. Frank</p>
Fastest way to shuffle 3D tensor

username: pastormoser
cooked: <p>Hi, new to pytorch here so this might be very basic.<br>
I have a batch of volumes of the shape BxCxWxHxD, with W=H=D=160, batch B and channels C.<br>
What I’m trying to do is divide each volume into smaller blocks, shuffle them, and rebuild the volume. Depending on the size of the blocks, the performance using indices can be quite slow. Therefore I’ve been trying to find faster, native approaches.<br>
I found this thread <a href="https://discuss.pytorch.org/t/how-to-extract-smaller-image-patches-3d/16837" class="inline-onebox">How to extract smaller image patches (3D)?</a> and I managed to use it to divide the volumes very efficiently. This is my modification of <a class="mention" href="/u/ptrblck">@ptrblck</a>’s suggestion:</p>
<blockquote>
<p>W = 160 # width<br>
H = 160 # height<br>
D = 160 # depth<br>
batch_size = 2<br>
channel_size = 1</p>
<p>x = torch.randn(batch_size, channel_size, W, H, D)<br>
print(x.shape) # (2, 1, 160, 160, 160)<br>
size = 80 # patch size<br>
stride = 80 # patch stride<br>
patches = x.unfold(2, size, stride).unfold(3, size, stride).unfold(4, size, stride)<br>
patches = patches.reshape(2, 1, -1, 80, 80, 80)<br>
print(patches.shape) # (2, 1, 8, 80, 80, 80)</p>
</blockquote>
<p>For the shuffle I can just shuffle the 2nd dimension. However, no I don’t know how to reconstruct the volume to the original shape.<br>
Thanks in advance!</p>
Convert jax array to torch tensor

username: DasPantom
cooked: <p>Hello all,</p>
<p>is there some way to load a JAX array into a torch tensor? A naive way of doing this would be</p>
<pre><code class="lang-auto">import numpy as np

np_array = np.asarray(jax_array)
torch_ten = torch.from_numpy(np_array).cuda()
</code></pre>
<p>This would be slow as it would require me to move the jax array from the gpu to a cpu numpy array before loading it on the gpu again.</p>
<p>Just to be clear: I am not interested in any gradient information being preserved. Only the entries of the array need to be loaded.</p>
<p>Many thanks!</p>

username: albanD
cooked: <p>How is jax handling GPU arrays?<br>
If it uses things like cupy, you can check online how to convert it directly yo pytorch.</p>

username: DasPantom
cooked: <p>Unfortunately, this seems to be not implemented yet. See</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/google/jax/issues/1100" target="_blank" rel="nofollow noopener">github.com/google/jax</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/google/jax/issues/1100" target="_blank" rel="nofollow noopener">Support __cuda_array_interface__ on GPU </a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-08-02" data-time="19:30:50" data-timezone="UTC">07:30PM - 02 Aug 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/hawkinsp" target="_blank" rel="nofollow noopener">
          <img alt="hawkinsp" src="https://avatars0.githubusercontent.com/u/348932?v=4" class="onebox-avatar-inline" width="20" height="20">
          hawkinsp
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">https://numba.pydata.org/numba-doc/dev/cuda/cuda_array_interface.html
It would not be hard to make DeviceArray implement this interface on GPU.
It would be slightly harder to support wrapping a...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">enhancement</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Many thanks for your reply, Pan</p>
Tensor as index

username: Rocket
cooked: <p>When I run</p>
<p><code>print(train_y[0])</code></p>
<p>it returns</p>
<p>tensor([43], device=‘cuda:0’)</p>
<p>When i run</p>
<pre><code class="lang-auto">num = train_y[0]
da=data[num]
</code></pre>
<p>da is empty. why?</p>
<p>When I run</p>
<pre><code class="lang-auto">num=43
da=data[num]
</code></pre>
<p>da has data.</p>

username: ptrblck
cooked: <p>Could you recheck the value of <code>num = train_y[0]</code>, as this dummy code snippet is running fine?</p>
<pre><code class="lang-python">train_y = torch.tensor([[43], [0]]).cuda()
num = train_y[0]
print(num)

data = torch.randn(50)
print(data[num])
&gt; tensor([-0.2206])
print(data[43])
&gt; tensor(-0.2206)
</code></pre>

username: Rocket
cooked: <p>data exist,</p>
<p>but plot is empty</p>
<pre><code class="lang-auto">import matplotlib.pyplot as plt

train_y = torch.tensor([[43], [0]]).cuda()
num = train_y[0]
print(num)

data = torch.randn(50,3)
print(data[num])

plt.figure(0)
plt.plot(data[num])

print(data[43])

plt.figure(1)
plt.plot(data[43])
</code></pre>
<p>data[num][0]</p>
<p>THANKS</p>