akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
WARNING:root:Torch AMP is not available on this platform

username: liyz15
cooked: <p>I ran into <code>WARNING:root:Torch AMP is not available on this platform</code> consistently when running <a href="https://github.com/facebookresearch/SlowFast" rel="noopener nofollow ugc">SlowFast</a> on 8 V100s. And I’m not even enabling amp. Any idea where this warning comes from?</p>

username: ptrblck
cooked: <p>I haven’t seen this warning before and searching for it yields a result in <a href="https://github.com/pytorch/fairseq/issues/3491">fairseq</a>. The warning “namespace” also doesn’t seem to be created in PyTorch. Are you using <code>fairseq</code> or another library depending on PyTorch, which might raise this warning?</p>

username: liyz15
cooked: <p>I’ve found the source of the warning using trace and it turned out to be <a href="https://github.com/facebookresearch/fairscale/blob/99163d4f6945c736c177bfe295279d273d1ef81c/fairscale/optim/__init__.py#L22" rel="noopener nofollow ugc">fairscale</a>. I will look into it to find out why import failed, thanks a lot!</p>

username: origin_of_symmetry
cooked: <p>Met the same problem. Did you find out why?</p>

username: liyz15
cooked: <p>Not really, it turns out I don’t need fairscale at all so I simply uninstall it and everything works well.</p>

username: Alani
cooked: <p>My torch version is 1.8.1 and I met the same issue when using FairScale 0.4.4. The issue was fixed by rolling back to an old version of FairScale, e.g., 0.4.1.</p>

username: prclibo
cooked: <p>Thanks for this hint. Met the same problem and rolling back to 1.4.5+0.4.1 works.</p>
How can I optimize non-leaf tensors?

username: Andrea_Rosasco
cooked: <p>Hey there,<br>
I’m trying to implement an algorithm that should work as follows:</p>
<blockquote>
<p>inputA, inputB, network1, network2</p>
<ol>
<li>network1 takes inputA and generates network2 (a list of tensors used as weights and biases)</li>
<li>network2 takes inputB and we do gradient descent (only network 2 is updated here)</li>
<li>network2 takes inputB and we do gradient descent (only network 1 is updated here)</li>
</ol>
</blockquote>
<p>since PyTorch optimizers use <code>no_grad()</code> while updating the weights I was planning to use <a href="https://github.com/facebookresearch/higher" rel="noopener nofollow ugc">higher</a> to generate a differentiable optimizer starting from a standard PyTorch one.</p>
<p>The only problem is that I can’t create a PyTorch optimizer initialized with network2 weights as they are not leaf-tensors.</p>
<p>Would implementing sgd by hand solve the problem? Is there a simpler solution?</p>
<p>If I wasn’t clear enough in explaining the issue, please ask for further information</p>
<p>Thank you in advance.</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=12" title=":cry:" class="emoji only-emoji" alt=":cry:" loading="lazy" width="20" height="20"></p>
Libtorch cross compile on aarch64-linux-gnu-gcc include backward function

username: Jesse_Stone
cooked: <p>libtorch cross compile</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://fatalfeel.blogspot.com/2013/12/libtorch-cross-compile-on-aarch64-linux.html">
  <header class="source">
      <img src="https://fatalfeel.blogspot.com/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://fatalfeel.blogspot.com/2013/12/libtorch-cross-compile-on-aarch64-linux.html" target="_blank" rel="noopener nofollow ugc">fatalfeel.blogspot.com</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://fatalfeel.blogspot.com/2013/12/libtorch-cross-compile-on-aarch64-linux.html" target="_blank" rel="noopener nofollow ugc">libtorch cross compile on aarch64-linux-gnu-gcc include backward</a></h3>

  <p> Refer to: https://github.com/pytorch/pytorch#get-the-pytorch-source https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst https:...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>info share</p>
How can I call autograd.grad within an autograd.Function?

username: yangzh
cooked: <p>I need to implement my own function which inherits from autograd.Function, where I mannually calculate some gradient using autograd.grad. But I find this does not work (see below). I am a bit new here. Who can help me?</p>
<pre><code class="lang-auto">import torch
def simple_func():
    a = torch.tensor(3., requires_grad=True)
    c = a + 2
    grads = torch.autograd.grad(c, a, grad_outputs=torch.tensor(5.))
    print(grads)

class dummy_autograd_func(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        return x + 2

    @staticmethod
    def backward(ctx, dy):
        simple_func()
        return dy

# Running normally; it is fine
simple_func()

# Running within autograd.Function; it fails
x = torch.rand(3, 2, requires_grad=True)
y = dummy_autograd_func.apply(x)
s = y.sum()
s.backward()
</code></pre>
<p>The error message is</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/yangzh/work/PycharmProjects/rev_test/simple_test.py", line 29, in &lt;module&gt;
    s.backward()
  File "/home/yangzh/anaconda3/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/yangzh/anaconda3/lib/python3.8/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
  File "/home/yangzh/anaconda3/lib/python3.8/site-packages/torch/autograd/function.py", line 199, in apply
    return user_fn(self, *args)
  File "/home/yangzh/work/PycharmProjects/rev_test/simple_test.py", line 19, in backward
    simple_func()
  File "/home/yangzh/work/PycharmProjects/rev_test/simple_test.py", line 7, in simple_func
    grads = torch.autograd.grad(c, a, grad_outputs=torch.tensor(5.))
  File "/home/yangzh/anaconda3/lib/python3.8/site-packages/torch/autograd/__init__.py", line 234, in grad
    return Variable._execution_engine.run_backward(
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>

username: ptrblck
cooked: <p>By default the gradient computation would be disabled in your custom <code>autograd.Function</code>, so you would need to enable it via <code>torch.set_grad_enabled(True)</code> in the <code>backward</code>.</p>

username: yangzh
cooked: <p>Thank you. It works now!</p>
Gradient of `maximum` and `minimum` functions

username: Sia_Rezaei
cooked: <p>Hi,</p>
<p>This is regarding the behavior of <code>torch.maximum</code> and <code>torch.minimum</code> functions.<br>
Here is an example:<br>
Let <code>a</code> be and scalar.<br>
Currently when computing <code>torch.maximum(x, a)</code>, if x &gt; a then the gradient is 1, and if x &lt; a then the gradient is 0. BUT if x = a then the gradient is 0.5.<br>
The same is true for <code>torch.minimum</code>.<br>
Are the mathematical reasons for the 0.5 gradient when x = a? or is it for numerical stability issues?</p>
<p>Thanks,</p>

username: ptrblck
cooked: <p>I think this is expected behavior as described <a href="https://github.com/pytorch/pytorch/issues/56734">here</a> and implemented <a href="https://github.com/pytorch/pytorch/pull/59669">here</a>.<br>
CC <a class="mention" href="/u/alband">@albanD</a> to correct me</p>

username: Sia_Rezaei
cooked: <p>Ok, got it. So the reason for implementing them this way is for practical reasons rather than mathematical logic.</p>

username: albanD
cooked: <p>The mathematical logic is that, at this point, the function is not differentiable. but you can define some sub-diffentials. In this case, the convex hull of [(0,1), (1, 0)]. And to ensure we get a descent direction, we take the element in this set with the minimum norm: (0.5, 0.5).</p>
<p>Does that help?</p>

username: Sia_Rezaei
cooked: <p>Yes, makes sense, thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Convert tensorflow code snippet to pytorch

username: danielmanu93
cooked: <p>Hi All, I want to convert this tensorflow code snippet to pytorch. Especially the session.run() part is pretty confusion.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bde211c9bc460d84726d8de39ce1053e447768ec.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/bde211c9bc460d84726d8de39ce1053e447768ec" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/d/bde211c9bc460d84726d8de39ce1053e447768ec_2_690x331.jpeg" alt="1" data-base62-sha1="r5MANBXBLlglB5RgrB2lUQdcIs4" width="690" height="331" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/d/bde211c9bc460d84726d8de39ce1053e447768ec_2_690x331.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/b/d/bde211c9bc460d84726d8de39ce1053e447768ec.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/d/bde211c9bc460d84726d8de39ce1053e447768ec.jpeg 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">818×393 77 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>PyTorch doesn’t use a <code>session</code> and executes the code in “eager mode” by default, i.e. similar to numpy.<br>
You could thus just feed your data to your model and would get the output without creating sessions etc.<br>
Take a look at e.g. <a href="https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html">these tutorials</a> to see how PyTorch is used.</p>
Questions on cudatoolkit versions

username: CDhere
cooked: <p>Hi! I have some basic questions regarding versions of CUDA, <code>cudatoolkit</code>, and the driver:</p>
<ol>
<li>When selecting “Compute platform” when installing pytorch from the <a href="https://pytorch.org/" rel="noopener nofollow ugc">Pytorch website</a>, or choosing the docker image from<br>
<a href="https://hub.docker.com/r/pytorch/pytorch/tags" rel="noopener nofollow ugc">Pytorch official page on Docker Hub</a>, does it mean the CUDA version pytorch is pre-built with, or the version the target machine should have?</li>
<li>Since <code>conda</code> install seems to come with its own CUDA in <code>cudatoolkit</code>, can I install pre-built pytorch labeled by any CUDA version (for example, can I run a 11.3 pytorch docker image on a host machine with 11.0)?</li>
<li>If not, what determines if a pre-built pytorch version can run on my machine?</li>
<li>What’s the relation of the compatibility between CUDA version as “Compute platform”, CUDA version as on my machine, and driver version?</li>
</ol>
<p>I’ve been confused by these concepts for quite a while, and would really appreciate it if someone could help clarify! Thanks!</p>

username: ptrblck
cooked: <ol>
<li>
<p>The pip wheels and conda binaries ship with their own CUDA runtime (as specified during the installation), so you would only need a corresponding NVIDIA driver to run your code. Your local CUDA toolkit will be used if you are building PyTorch from source or a custom CUDA extension.<br>
The docker image would ship with a PyTorch source build which used the specified CUDA toolkit. The <code>-devel</code> container also comes with the CUDA compiler etc.</p>
</li>
<li>
<p>“can I install pre-built pytorch labeled by any CUDA version” - it depends on the GPU you are using (e.g. Amperes needs CUDA&gt;=11.0) as well as the driver (it has to be sufficiently new).</p>
</li>
<li>
<p>See 2.</p>
</li>
<li>
<p>Check the <a href="https://docs.nvidia.com/deploy/cuda-compatibility/">CUDA compatibility docs</a> for a detailed overview.</p>
</li>
</ol>

username: CDhere
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for the reply! So if I understand it correctly:</p>
<ol>
<li>if we just want to run a certain piece of code, we only need the CUDA runtime and match the right version between runtime, card arch and driver (and sometimes pytorch given its coverage)</li>
<li>whereas, we would need CUDA toolkit only when it comes to compiling some code, and the version of CUDA toolkit determines the runtime versions it can produce</li>
</ol>
<p>Please correct me if anything is wrong. Thanks again!</p>

username: ptrblck
cooked: <p>Yes, your understanding is correct. The driver requirements are also mentioned in the compatibility docs and you can also learn more about “enhanced compatibility” etc. which allows newer CUDA versions to use an older driver.</p>
ValueError: Expected target size (2, 256), got torch.Size([2, 256, 256])

username: shimaaholail
cooked: <p>Traceback (most recent call last):<br>
File “main_cd.py”, line 83, in <br>
train(args)<br>
File “main_cd.py”, line 15, in train<br>
model.train_models()<br>
File “D:\change detection\change detect\ChangeFormer\models\trainer.py”, line 330, in train_models<br>
self._backward_G()<br>
File “D:\change detection\change detect\ChangeFormer\models\trainer.py”, line 306, in _backward_G<br>
self.G_loss = self._pxl_loss(self.G_pred[-1], gt)<br>
File “D:\change detection\change detect\ChangeFormer\models\losses.py”, line 23, in cross_entropy<br>
ignore_index=ignore_index, reduction=reduction)<br>
File “C:\Users\2021276190015\Anaconda3\envs\BIT-CD\lib\site-packages\torch\nn\functional.py”, line 2422, in cross_entropy<br>
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)<br>
File “C:\Users\2021276190015\Anaconda3\envs\BIT-CD\lib\site-packages\torch\nn\functional.py”, line 2228, in nll_loss<br>
out_size, target.size()))<br>
ValueError: Expected target size (2, 256), got torch.Size([2, 256, 256])</p>
Replace torch.gather by other operator

username: John1231983
cooked: <p>Hello all, I have one script code, where <code>x1</code> and <code>x2</code> size of <code>1x68x8x8</code></p>
<pre><code class="lang-auto">tmp_batch, tmp_channel, tmp_height, tmp_width = x1.size()
 x1 = x1.view(tmp_batch*tmp_channel, -1)        
 max_ids = torch.argmax(x1, 1)            
 max_ids = max_ids.view(-1, 1)
            
 x2 = x2.view(tmp_batch*tmp_channel, -1)
 outputs_x_select = torch.gather(x2, 1, max_ids) # size of 68 x 1
</code></pre>
<p>As the above code, I have a trouble with <code>torch.gather</code> when I used old onnx. Hence, I would like to find an alternative solution that replace the <code>toch.gather</code> by other operators but gives same output with above code. Could you please give me some suggestion?</p>
Using PixelCNN over patches

username: Kinyugo
cooked: <p><strong>Is it possible to apply a PixelCNN over patches instead of individual pixels?</strong></p>
<p>As applying patch embedding to vision models is becoming more <a href="https://arxiv.org/abs/2201.03545" rel="noopener nofollow ugc">popular</a>, I am curious to know  whether the same strategy can be applied to autoregressive models making them faster. To the best of my knowledge it should be possible to do as long as non-overlapping patches are taken. Am I missing something? What are the possible caveats to this approach?</p>
OOM error using Java Pytorch

username: AlexandreLrz
cooked: <p>Hi,</p>
<p>I’m trying to create an image editor app for Android using Pytorch Java and I get this error using module.forward(…) with large images :</p>
<pre><code class="lang-auto">E/AndroidRuntime: FATAL EXCEPTION: main
    Process: com.alexlrz.photoeditor, PID: 25987
    java.lang.RuntimeException: Unable to start activity ComponentInfo{com.alexlrz.photoeditor/com.alexlrz.photoeditor.MainActivity}: com.facebook.jni.CppException: [enforce fail at CPUAllocator.cpp:79] data. DefaultCPUAllocator: not enough memory: you tried to allocate 1024000032 bytes.
      
      debug_handle:-1
      
    (no backtrace available)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2913)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3048)
        at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:78)
        at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:108)
        at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:68)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1808)
        at android.os.Handler.dispatchMessage(Handler.java:106)
        at android.os.Looper.loop(Looper.java:193)
        at android.app.ActivityThread.main(ActivityThread.java:6669)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858)
     Caused by: com.facebook.jni.CppException: [enforce fail at CPUAllocator.cpp:79] data. DefaultCPUAllocator: not enough memory: you tried to allocate 1024000032 bytes.
      
      debug_handle:-1
      
    (no backtrace available)
        at org.pytorch.LiteNativePeer.forward(Native Method)
        at org.pytorch.Module.forward(Module.java:52)
        at com.alexlrz.photoeditor.MainActivity.onCreate(MainActivity.java:146)
        at android.app.Activity.performCreate(Activity.java:7136)
        at android.app.Activity.performCreate(Activity.java:7127)
        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1271)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2893)
        	... 11 more
</code></pre>
<p>It works fine with small images (500x500) but I need to use larger images, at least 2K. Do you have any idea of how I could use it with large images ?</p>
<p>Thank you</p>
Help with LARS for Barlow Twins

username: dmandair
cooked: <p>Hi and thanks in advance for the help! This is more of a general question but I’m attempting to implement Barlow Twins using Pytorch Lightning. I’m unsure of how to use this LARS wrapper (<a href="https://pytorch-lightning-bolts.readthedocs.io/en/0.2.1/api/pl_bolts.optimizers.lars_scheduling.html" class="inline-onebox" rel="noopener nofollow ugc">pl_bolts.optimizers.lars_scheduling module — PyTorch-Lightning-Bolts 0.2.1 documentation</a>) with the configure_optimizer function in Pytorch lightening. Would I just wrap an Adams optimizer with this and that would be the sole item returned by the function? Additionally in Barlow twins, the LR had a warm-up scheduling - would this need to be implemented separately or is this taken care of in the implementation or the LARS wrapper?</p>
<p>Thanks again for the help!</p>
How to add namespaces to a tensorboard graph?

username: Nathan_Wood
cooked: <p>When displaying a graph using <code>torch.utils.tensorboard.SummaryWriter.add_graph</code>, only modules end up in boxes. I use a lot of ops around it, and it can be really, really hard to tell what’s going on. Is there something like tensorflow1.0’s <code>tf.namespace</code> (might be <code>variable_scope</code> actually), which would let me sort operations to more clearly see what’s happening? Thanks.</p>

username: promach
cooked: <p><a class="mention" href="/u/nathan_wood">@Nathan_Wood</a> Have you solved this issue ?</p>
<p>I have similar issue at <a href="https://discuss.pytorch.org/t/adding-node-and-cell-names-for-tensorboard-graph/142059" class="inline-onebox">Adding node and cell names for tensorboard graph</a></p>
Pytorch Lightning Loaded Model Gives Different Results Each Time

username: batuhansozer
cooked: <p>I have a U-Net architecture and when I train the model and print out some results without loading it, the model gives good results but when I load the model checkpoint file and try to make predictions it predicts random noise, even worse it gives different predictions for each run with the same test set</p>
<p>Here is my code<br>
model.py <a href="https://pastecode.io/s/nhjznzxc" class="inline-onebox" rel="noopener nofollow ugc">Untitled in python (nhjznzxc) - PasteCode.io</a><br>
train.py <a href="https://pastecode.io/s/umjqprb3" class="inline-onebox" rel="noopener nofollow ugc">Untitled in python (umjqprb3) - PasteCode.io</a><br>
pred.py pastecode .io/s/m9zkpwk5</p>
<p>Note: Since I’m a new user, I’m allowed to put only 2 links on my post so i added spaces to the link for pred.py.</p>
Looking for tips on optimizing cpu inference with torchscript models

username: jiyuan
cooked: <p>Hi,</p>
<p>I have some recommendation models trained and torchscripted, which I’d like to use for model serving.<br>
I’m using DJL which is a java library that loads torchscript models through JNI. Given I’m not doing much on the java side, I suppose the performance characteristics is largely dependently on the torchscript jit runtime.</p>
<p>Now my questions are:</p>
<ol>
<li>
<p><strong>How should one typically tune inter-op and intra-op threads to get best performance in terms of throughput and tail latency?</strong><br>
When I was working with tensorflow I used to set inter-op threads to the number of available cpus and intra-op threads to 1, which gives enough parallelism to handle a large number of requests. When I tried the same with torchscript model, unfortunately requests start queuing before even able to hit 50% cpu utilization. Is this expected?</p>
</li>
<li>
<p><strong>Are there any steps that should be taken to process the torchscript model before using it for inference?</strong><br>
For example, I noticed torch.jit.freeze, but haven’t really tried it. How much performance gain can I typically expect by applying it?</p>
</li>
<li>
<p><strong>Are there any other knobs to tune for cpu inference performance?</strong><br>
Apart from what’s mentioned above, what other knobs should one typically try to adjust for optimal performance?</p>
</li>
</ol>
<p>Thanks!</p>
Blocked multi-thread if use a tensor in main thread

username: MoriartyShan
cooked: <p>Code likes below.  It will block at creating <code>x1</code>. If you remove line <code>print("finish create\n", x)</code>, it works fine.<br>
This really bothers me.<br>
Thanks if anyone helps</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as multiprocessing

def run():
  print("run")
  device=torch.device("cpu")
  x1 = torch.rand(720, 1280, dtype=torch.float32, device=device)
  print('a', x1)
  x2 = x1.to(dtype=torch.float64)
  print('b', x2.dtype)

x = torch.rand(1080, 1920,device=torch.device("cpu"))
print("finish create\n", x)

thread = multiprocessing.Process(target=run)
thread.start()
print("wait")
thread.join()
print("finish")
</code></pre>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/moriartyshan">@MoriartyShan</a></p>
<p>The issue was with this line</p>
<pre><code class="lang-auto">print('a', x1)
</code></pre>
<p>If you convert this to</p>
<pre><code class="lang-auto">print('a', x1.detach().numpy())
</code></pre>
<p>I am not sure why this is happening. We need to get to the bottom of this. But printing of tensors within multiprocessing function caused your issue</p>

username: MoriartyShan
cooked: <p>Thanks for your advice.<br>
But after changing code as yours, it blocked at <code>x2 = x1.to(dtype=torch.float64)</code>. The line next it never executed.<br>
I think there may be some setting in pytorch causes it.<br>
And if change device in <code>run</code> to <code>'cuda:0'</code>, the problem disappears. But this really not what I want.</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/moriartyshan">@MoriartyShan</a> you are correc. This is weird behavior. For the time being, following is the workaround, but i will try to get to the bottom of this. It will be a good exercise</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as multiprocessing
import copy 

def run():
    print("run")
    device=torch.device("cpu")
    print(torch.__version__)
    x1 = torch.rand(720, 1280, dtype=torch.float32, device=device)
    x2 = torch.from_numpy(x1.detach().numpy().astype(np.float64))
    print('b', x2.dtype)

x = torch.rand(1080, 1920,device=torch.device("cpu"))
print("finish create\n", x)

thread = multiprocessing.Process(target=run)
thread.start()
print("wait")
thread.join()
print("finish")
</code></pre>

username: MoriartyShan
cooked: <p>Thanks.<br>
The example is a simplified code from my work. Changing tensor to numpy array can truly solve the problem above. Hope there is some way to work with pytorch in another thread.<br>
I will try figure it out also.</p>
"hybrid" mode of pytorch for cuda

username: evan0greenup
cooked: <p>For cuda version of pytorch, if there make model and optimizer <code>.to()</code> “cuda”, When run the code, the learning processing will completely take place in GPU, CPU will be unused.</p>
<p>It would be nice to add a “hybrid” option, which first use GPU, but if CPU is free, it will utilize CPU as much as possible.</p>
<p>I am wondering if this is possible?</p>

username: ptrblck
cooked: <p>You could manually push data to the CPU and let it perform some operations. Note however, that data transfers can be expensive and would most likely reduce the performance of your model. Also, the CPU is still executing the actual Python code and is launching the GPU workload, so it’s not completely idle,</p>
"Peaking" a distribution tensor

username: AreTor
cooked: <p>I have a tensor of probability distributions <code>t</code> of size <code>(n, c, h, w)</code>, such that <code>t.sum(dim=1) == 1</code>.</p>
<p>What’s the best way to retrieve low entropy predictions, and to “peak” them such that they become one-hot, preserving differentiability? Retrieving may be performed also using a threshold on the probabilities (e.g. <code>p</code> such that exists a <code>p[i] &gt; 0.9</code>).</p>
<p>For example, if I have <code>[0.05, 0.9, 0.05]</code> the peaked output would be <code>[0., 1., 0.]</code>.</p>
<p>How can I efficiently perform the peaking?</p>

username: GregorySech
cooked: <p>A somewhat naive way could be subtracting the threshold from <code>t</code>, then give this to a ReLU activation with intercept 0 and slope 1. Lastly divide by the sum of the result to go back to summing 1.</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/aretor">@AreTor</a> There are many flavours of combinations</p>
<p>e.g</p>
<p>a=[0.44, 0.55, 0.01]</p>
<p>What should be done in this case</p>
<p>A simplistic way excluding the above scenario is</p>
<p>I am assuming the last axis to be the ones having sum of 0</p>
<pre><code class="lang-auto">a = torch.rand(2, 3, 2, 3)

# Forcing two cases so that we can unit test
a[0,0,0,0]=0.99
a[0,0,0,1]=0.005
a[0,0,0,2]=0.005

a[0,2,1,0]=0.99
a[0,2,1,1]=0.005
a[0,2,1,2]=0.005

a1 = torch.ones(a.size())
print("A1 size {0}".format(a1.size()))
a = a/torch.sum(a, axis=3).view(2,3,2,1).repeat((1,1,1,3))
# Unit Test: Last axis sum is 0 
torch.sum(a, axis=3)

# Setting the threshold val
threshold=0.05
inv_threshold=1-threshold
a1[torch.logical_and(a &lt; threshold, a &gt; -threshold)]=2 # Ones which needs to be converted to 0
a1[torch.logical_or(a &gt; inv_threshold, a &lt; -inv_threshold)]=3 # Ones that need to be converted to 1

print("Before a was")
print(a)

a[torch.logical_and( (torch.sum(a1==1, axis=3)==0).view(2,3,2,1).repeat(1,1,1,3), a1==3)]=1
a[torch.logical_and( (torch.sum(a1==1, axis=3)==0).view(2,3,2,1).repeat(1,1,1,3), a1==2)]=0

print("After a is")
print(a)
</code></pre>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/gregorysech">@GregorySech</a> This should be tried, as it would be an awesome way to use activation functions to do intelligent approximations and rounding offs</p>

username: AreTor
cooked: <p>In the end, I solved with the following piece of code</p>
<pre><code class="lang-python">    # x: (n, c, h, w) tensor -- t: threshold
    mask = x &gt; t  # find values over threshold
    x[mask] = x[mask] / x[mask]  # peak these values to 1.
    x = x * (mask + ~mask.any(dim=1, keepdims=True))  # adjust to get prob again
</code></pre>
<p>that works for thresholds greater than 0.5.</p>
Jit with bool scalars

username: gune
cooked: <p>I am trying to run the following code</p>
<pre><code class="lang-auto">import torch

@torch.jit.script
def foo(x: torch.Tensor, y: bool):
    return x &amp; y

print(foo.graph)
</code></pre>
<p>But ran into an error</p>
<pre><code class="lang-auto">RuntimeError:
Arguments for call are not valid.
The following variants are available:

  aten::__and__.Scalar(Tensor self, Scalar other) -&gt; (Tensor):
  Expected a value of type 'number' for argument 'other' but instead found type 'bool'.

  aten::__and__.Tensor(Tensor self, Tensor other) -&gt; (Tensor):
  Expected a value of type 'Tensor' for argument 'other' but instead found type 'bool'.

  aten::__and__.bool(bool a, bool b) -&gt; (bool):
  Expected a value of type 'bool' for argument 'a' but instead found type 'Tensor'.

  aten::__and__.int(int a, int b) -&gt; (int):
  Expected a value of type 'int' for argument 'b' but instead found type 'bool'.

The original call is:
  File "bool_torchscript.py", line 5
@torch.jit.script
def foo(x: torch.Tensor, y: bool):
    return x &amp; y
           ~~~~~ &lt;--- HERE
</code></pre>
<p>Does this mean that scalar cannot be of bool type and has to be int/float only?</p>
Using pytorchvideo and torchvision transforms together for video action recognition

username: proxymate
cooked: <p>Hi everyone!</p>
<p>I am trying to finetune slowfast models on my custom dataset for video action recognition using pytorchvideo and torchvision. I am a bit unsure of how I have combined transforms from the two libraries together.</p>
<p>Here is how my transformation + augmentation scheme looks like</p>
<pre><code class="lang-auto">self.train_transform = ApplyTransformToKey(
      key="video",
      transform=Compose(
        [  
          UniformTemporalSubsample(32),
          Permute([1,0,2,3]),
          RandomAffine(degrees=20, translate=(0, 0.1), shear=(-15, 15, -15, 15)),
          GaussianBlur(kernel_size=3, sigma=(0.1, 1.5)),
          Permute([1,0,2,3]),
          Lambda(lambda x: x / 255.0),
          Normalize((0.45, 0.45, 0.45), (0.225, 0.225, 0.225)),
          RandomShortSideScale(
            min_size=256,
            max_size=320,
          ),
          RandomCrop(256),
          RandomHorizontalFlip(p=0.5),
          PackPathway(),
        ]
      ),
    )
</code></pre>
<p>The reason why I am a bit suspicious is that when I tried to visualize the videos produced after they were transformed they looked weird. Here are 2 different videos that I visualized after applying the above transforms : <a href="https://imgur.com/a/zcL1zeF" rel="noopener nofollow ugc">https://imgur.com/a/zcL1zeF</a></p>
<p>Would be really grateful if someone can please check this and let me know if this is wrong and point me how to correct the same.</p>
How to make a model predicting multivariate time series

username: meln1337
cooked: <p>Hello! I’m new to pytorch and solving problem on Kaggle “Tabular Playground Series - Jan 2022”.<br>
<a href="https://www.kaggle.com/c/tabular-playground-series-jan-2022/data" rel="noopener nofollow ugc">https://www.kaggle.com/c/tabular-playground-series-jan-2022/data</a><br>
The essence of this task is to figure out whether KaggleMart or KaggleRama should become the official Kaggle outlet. So there are 3 countries, 3 products and 2 markets, that’s why we have to predict num_sold for 18 cases. I know how to make 18 models for predicting 18 cases separately, but it’s very inconvenient. So how can i create 1 model for predicting 18 cases? By the way prediction I perform using LSTM</p>
AttributeError: 'Hardswish' object has no attribute 'activation_post_process'

username: xiuyangleiasp
cooked: <p>When I quantize a model with “Hardswish”, It’s got a Error with info:<br>
AttributeError: ‘Hardswish’ object has no attribute ‘activation_post_process’,<br>
And my coonfig is like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/2/2207601e936eb674f99b8a48711d6f52cbde25a8.png" data-download-href="https://discuss.pytorch.org/uploads/default/2207601e936eb674f99b8a48711d6f52cbde25a8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/2207601e936eb674f99b8a48711d6f52cbde25a8_2_690x142.png" alt="image" data-base62-sha1="4R22dQjFRo20sVkPIAZ1Gs8pq4g" width="690" height="142" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/2207601e936eb674f99b8a48711d6f52cbde25a8_2_690x142.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/2207601e936eb674f99b8a48711d6f52cbde25a8_2_1035x213.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/2207601e936eb674f99b8a48711d6f52cbde25a8_2_1380x284.png 2x" data-dominant-color="272928"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1745×361 47.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
How could I fix this error? thanks</p>

username: HDCharles
cooked: <p>you are using qat prepare but normal qconfig. Also in the mapping, nnq.Hardswish isn’t a qat module. If you are intending to do qat you should do something like</p>
<pre><code class="lang-auto">    import torch
    qconfig = torch.quantization.get_default_qat_qconfig("fbgemm")
    model = torch.nn.Sequential(torch.nn.modules.Hardswish(), torch.nn.modules.Linear(1,1))
    model.qconfig = qconfig
    model_prep = torch.quantization.prepare_qat(model)
    print(model_prep)
    model_prep(torch.randn(1,1))
</code></pre>
<p>if you are not intending to do qat you should do something like</p>
<pre><code class="lang-auto">    import torch
    qconfig = torch.quantization.get_default_qconfig("fbgemm")
    model = torch.nn.Sequential(torch.nn.modules.Hardswish(), torch.nn.modules.Linear(1,1))
    model.qconfig = qconfig
    model_prep = torch.quantization.prepare(model)
    print(model_prep)
    model_prep(torch.randn(1,1))
</code></pre>
Deterministic training when using mixed-precision

username: juliustao
cooked: <p>TL;DR: After using <code>torch.cuda.amp</code>, I have deterministic training, even though I set <code>torch.backends.cudnn</code> options as <code>deterministic=False</code>, <code>benchmark=False</code>, etc.</p>
<p>Environment</p>
<ul>
<li>2080Ti (CUDA 11.2, Driver 460.91.03)</li>
<li>PyTorch 1.11.0.dev20211127</li>
<li>Python 3.9.7</li>
</ul>
<p>I experimented with this <a href="https://github.com/pytorch/examples/issues/542" rel="noopener nofollow ugc">minimal MNIST example</a> and reproduced the nondeterminism across training runs (i.e. different epoch losses when I train from scratch multiple times). The source of nondeterminism is GPU operations since the random seeds are fixed.</p>
<p>If I set <code>torch.backends.cudnn.deterministic=True</code>, I see deterministic training like the original author.</p>
<p>However, if I use <code>amp</code> for mixed-precision training, I also see determinism even without <code>deterministic=True</code>.</p>
<p>Has anyone seen similar cases or have insights into how using <code>amp</code> could remove nondeterminism?</p>
<p><a href="https://juliustao.notion.site/Determinism-e9b2a1f6241e49e98b6fb20b0e846370" rel="noopener nofollow ugc">Here’s a writeup with further details on experiments and code.</a></p>
<p>Thanks!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="juliustao" data-post="1" data-topic="141979">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juliustao/48/45882_2.png" class="avatar"> juliustao:</div>
<blockquote>
<p>Has anyone seen similar cases or have insights into how using <code>amp</code> could remove nondeterminism?</p>
</blockquote>
</aside>
<p>Non-deterministic results could be caused by the actual kernel selection and <code>amp</code> could be using e.g. deterministic cuDNN kernels.</p>

username: juliustao
cooked: <p>Thanks for the quick response! How could I check whether the <code>amp</code> code is using deterministic cuDNN kernels? I know the non-determinism before using <code>amp</code> came from <code>Conv2d</code> layers preceding another <code>Conv2d</code> layer during backprop, if that narrows it down.</p>

username: ptrblck
cooked: <p>You could profile the kernels and see if you could determine the underlying algorithm from its name.<br>
Generally, it could be hard to guess it unless you are familiar with different algos and how they might be implemented internally. The general rule would be that kernels can be deterministic in the default setup and have to be deterministic if the right flags are set.</p>
Bert output sequence output vs pooled output

username: sandeep1
cooked: <p>The Bert outputs two things :-</p>
<p><code>last_hidden_state</code>: contains the hidden representations for each token in each sequence of the batch. So the size is <code>(batch_size, seq_len, hidden_size)</code>.</p>
<p><code>pooler_output</code> contains a “representation” of each sequence in the batch, and is of size <code>(batch_size, hidden_size)</code>. What it basically does is take the hidden representation of the [CLS] token of each sequence in the batch</p>
<p>So suppose:-</p>
<p><code>hidden,pooled=model(...)</code></p>
<p>Shouldn’t<br>
hidden[0] which is the embedding of 1st token ie: CLS be same as pooled as pooled is also the embedding of CLS?</p>
<p>While I have checked using code they both are not same. Why is it so? Am I missing something?</p>
<p>Thank you.</p>

username: tom
cooked: <p>BERT includes a linear + tanh layer as the pooler. I recently wrote <a href="https://github.com/MathInf/toroidal/blob/bff09f725627e4629d464008dc7c5f9d6322ebad/toroidal/models.py#L354-L403" rel="noopener nofollow ugc">a very compact implementation of BERT Base</a> that shows what is going on. L354 you have the pooler, below is the BERT model. The <code>last_hidden_state</code> is the output of the blocks, you can set <code>model.pooler</code> to <code>torch.nn.Identity()</code> to get these, as shown in the <a href="https://github.com/MathInf/toroidal/blob/bff09f725627e4629d464008dc7c5f9d6322ebad/test/test_bert.py#L7" rel="noopener nofollow ugc">test</a> which shows how to import BERT from the HF transformer library into this simple transformer implementation. (I also have a DistillBert version but not yet pushed to github. DistilBert drops the pooler. Interestingly, the models are coded quite differently internally, even though they use the same computation…)</p>
<p>Best regards</p>
<p>Thomas</p>

username: Tim5
cooked: <p>excuse me if I need to use BERT in the image captioning model instead of LSTM ? How can I make these changes?</p>
Wav2vec - 

 tokens

username: Brian_Hemmat
cooked: <p>The wav2vec2.0 base 960h model never seems to return a beginning of sentence or end of sentence token (or ’ or unknown so far). Is that expected? I can’t seem to find this discussed anywhere. Why are those tokens in the decoding dictionary? Why are those those options in the final emission matrix? Or am I just feeding in audio that is too difficult for the model to determine eos/bos? If so, can someone provide a counter-example?</p>
CrossEntropyLoss doesn't work on GPU with memory strided label input

username: ahmadh84
cooked: <p>I recently noticed that <code>nn.CrossEntropyLoss</code> doesn’t do correct back-prop on GPU when supplied labels is a slice of a tensor in some cases. It seems like the GPU implementation supposes that the label I provide is memory contiguous (no stride).  Here’s a minimal example:</p>
<pre><code class="lang-auto">import torch
from torch import nn

class BasicNetwork(nn.Module):
    def __init__(self, linear_n_feats: int, n_outputs: int):
        super(BasicNetwork, self).__init__()
        self.network = nn.Sequential(
            nn.Flatten(),
            nn.Linear(in_features=linear_n_feats, out_features=n_outputs)
        )

    def forward(self, X):
        return self.network(X)

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using {device} device")

batch_size = 64
in_channels = 3
temporal_depth = 128

model = BasicNetwork(linear_n_feats=in_channels * temporal_depth, n_outputs=7)
model = model.to(device)
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

losses = []

for batch_idx in range(300):
    # random input
    X = torch.rand((batch_size, in_channels, temporal_depth), device=device)
    # fixed label
    y = torch.arange(7, dtype=torch.int64, device=device)
    y = y.repeat(batch_size, 1)

    pred = model(X)

    # first column of y is the classification label
    curr_y = y[:, 0]  # .contiguous()  # .clone()
    loss = loss_fn(pred, curr_y)

    # back-propagation
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    losses.append(loss.item())

    if (batch_idx + 1) % 10 == 0:
        avg_loss = sum(losses[-10:]) / 10
        print(f"{batch_idx+1:&gt;8,d}: Loss = {avg_loss:.5f}")
</code></pre>
<p>You can see that the label is a <code>(64, 7)</code> array, and the label I use is the first column (<code>y[:,0]</code>). If you run it, you would see that the loss doesn’t go down much. But as soon as you either use <code>.contiguous()</code>, or <code>.clone()</code>, or just set <code>device = "cpu"</code>, the loss starts improving.</p>
<p>I haven’t experimented with other losses or slices of the prediction tensor - so can’t say if all loss GPU implementations expect contiguous tensors.</p>

username: ptrblck
cooked: <p>Thanks for reporting the issue! It was a <a href="https://github.com/pytorch/pytorch/issues/67167">known bug</a> and should be already fixed in the latest <code>1.10.1</code> release and the nightlies.</p>
How object detection training is done in batches?

username: Gabriel_Dornelles
cooked: <p>I’ve been trying to train object detection models, and my question is:</p>
<p>How this models are trained in batches given that images have a different input shape in datasets? I’m trying to fine tune DETR and it was trained in batches of 2, what am I missing? is it common to train this models in batches of 1?</p>
How to read a and create dataset from JSON files

username: Oussama_Bouldjedri
cooked: <p>I have many JSON files from which I have to create data sets, I can’t figure out how to handle this type of data in pythorch ?<br>
Thank you.</p>

username: thecho7
cooked: <p>It’d better to explain your json structure at least briefly.</p>

username: Oussama_Bouldjedri
cooked: <p>I have a list of files for every label, stored in a directory with the label name and every file contains this:</p>
<p>{“data”: [{<br>
“ID”:,<br>
“SensorName”: ,<br>
“DigitalValue”: ,<br>
“AnalogValue”: ,<br>
“SchematicName”:,<br>
“Pin”:,<br>
“Enable”:,<br>
“TimeStamp”:,<br>
…<br>
}]}</p>

username: thecho7
cooked: <ol>
<li>Choose data you want to use. For example, ID, DigitalValue, …</li>
<li>Convert them into number(integer, float, …)</li>
<li>Make a list of them</li>
<li>Convert that list into PyTorch Tensor</li>
</ol>
<pre><code class="lang-auto">data = []
for k, v in x['data'][0].items():
    if type(v) == str:
        # convert it into int or float
        # and append into data
    else:
        data.append(v)

torch_data = torch.Tensor(data)</code></pre>

username: Oussama_Bouldjedri
cooked: <p>thanks but, what about the labeling, how should I set the data to be sorted with its labels?</p>

username: thecho7
cooked: <p>Labels follow the same but</p>
<pre><code class="lang-auto">list_of_labels = [0, 1, 2, 0, 1, 3, ..., 4]
torch_labels = torch.LongTensor(list_of_labels) # for classifier
torch_labels = torch.FloatTensor(list_of_labels) # for regression
</code></pre>
<p>Wish my comment will help but the question is too broad.<br>
The keypoint is converting a list into tensor.</p>

username: Oussama_Bouldjedri
cooked: <p>Sorry for my too broad question, what I wanted to say is that every file has the structure I pasted in the previous answear, and for every label I have a list of file names so I was wondering how to read the data (you already answeared this thanks ), and add the label to this dataset ?</p>
Pytorch elastic and local set of machines

username: xandox
cooked: <p>Hi all!</p>
<p>I have set a machines in my local network with GPU (some have one GPU, some few, and there is 3 types of GPU models)<br>
I would like to turn this machines for somewhat cluster to use multi-node training for my team.<br>
And now I am considering different variants how to do it.<br>
One of this variant is <code>torch.distributed.elastic</code>. But I do not understand how it works.<br>
I found a couple examples but all of them for some cloud provider with kubernetes which is not my case.</p>
<p>So some question (let’s assume I do not want to install kubernetes cluster yet)</p>
<ol>
<li>What is <code>c10d</code>? I really unable to google it.</li>
<li>Quick start examples just pass <code>train.py</code> to <code>torchrun</code> but what about <code>virtualenv</code> or <code>conda env</code>. Is it possible to run docker image like this?</li>
<li>Is it not clear for me what is ElasticAgent and how to use it?</li>
</ol>

username: pritamdamania87
cooked: <p>cc <a class="mention" href="/u/aivanou">@aivanou</a> <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a></p>

username: aivanou
cooked: <p>Hello!<br>
Torchelastic(<code>torch.distributed.elastic</code>) provides you with ability to execute a single distributed job. For example, if you have 2 machines, and you need to execute a distributed job on them, you would need to log in to both machines and run the following command:</p>
<pre><code class="lang-auto">#machine 1
python -m torch.distributed.run --rdzv_id my_id --rdzv_backend c10d --rdzv_endpoint IP_MACHINE1:29400 --nnodes 2 --nproc_per_node 2 my_trainer.py

#machine 2
python -m torch.distributed.run --rdzv_id my_id --rdzv_backend c10d --rdzv_endpoint IP_MACHINE1:29400 --nnodes 2 --nproc_per_node 2 my_trainer.py
</code></pre>
<p>In general, if you want to create a cluster among N machines that can be used for many jobs, you would need to set up something like SLURM: <a href="https://slurm.schedmd.com/documentation.html" class="inline-onebox" rel="noopener nofollow ugc">Slurm Workload Manager - Documentation</a>.</p>
<ol>
<li>What is <code>c10d</code> ? &gt; it stands for <code>caffe2 and aten</code>, we probably need to remove it, since it is confusing for users.</li>
<li>The virtualenv you would need to handle by yourself, with torchX it is possible to use docker images.</li>
</ol>
<p>We are currently working on TorchX: <a href="https://pytorch.org/torchx/latest/" class="inline-onebox" rel="noopener nofollow ugc">TorchX — PyTorch/TorchX main documentation</a> that should allow you to execute your jobs easily.<br>
I can help you to set up the cluster and maybe you will find TorchX helpful. Write me in slack: csivanou at gmail dot com</p>