akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to create a c10::List<IValue> with libtorch API?

Bycob: <p>I posted about this problem last month and got no answer, but I still have not found a way to resolve it, hence the repost.<br>
I try to create a List from C++ API, but I keep getting a compilation error for code as simple as this one:</p>
<pre><code class="lang-auto">c10::List&lt;c10::IValue&gt; list;
</code></pre>
<p>I get this error:</p>
<pre><code class="lang-auto">In file included from /usr/local/include/ATen/core/List.h:472:0,
                 from /usr/local/include/ATen/core/ivalue_inl.h:13,
                 from /usr/local/include/ATen/core/ivalue.h:895,
                 from /usr/local/include/ATen/ATen.h:33,
                 from /usr/local/include/torch/csrc/api/include/torch/types.h:3,
                 from /usr/local/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /usr/local/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /usr/local/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /usr/local/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /usr/local/include/torch/csrc/api/include/torch/data.h:3,
                 from /usr/local/include/torch/csrc/api/include/torch/all.h:8,
                 from /usr/local/include/torch/csrc/api/include/torch/torch.h:3,
                 from /home/aaaa/bbbbb/test.cc:2:
/usr/local/include/ATen/core/List_inl.h: In instantiation of ‘c10::List&lt;T&gt;::List() [with T = c10::IValue]’:
/usr/local/include/ATen/core/ivalue_inl.h:976:62:   required from ‘c10::IValue::IValue(const std::vector&lt;T&gt;&amp;) [with T = c10::IValue; typename std::enable_if&lt;std::is_constructible&lt;c10::IValue, T&gt;::value, std::nullptr_t&gt;::type &lt;anonymous&gt; = nullptr]’
/home/aaaa/bbbbb/torch_exp/test.cc:89:49:   required from here
/usr/local/include/ATen/core/List_inl.h:20:3: error: static assertion failed: This constructor is not valid for List&lt;IValue&gt;. Please use c10::impl::GenericList(elementType) instead.
   static_assert(!std::is_same&lt;T, IValue&gt;::value, "This constructor is not valid for List&lt;IValue&gt;. Please use c10::impl::GenericList(elementType) instead.");
   ^~~~~~~~~~~~~
</code></pre>
<p>Am I trying to create a list the wrong way? In which case, what is the correct way of creating list of IValues? I want to pass them to a jit module that takes a List of Tensor as input</p>
<p>And I don’t understand why this assert is triggering, since <code>std::is_same&lt;T, IValue&gt;::value</code> should be true for T == c10::IValue ?</p>
Build failing on WSL2 Ubuntu 20.04

imaginary: <p>I’m not using ninja. Building from source always fails with a similar reason on WSL2 Ubuntu 20.04.<br>
On Ubuntu 20.04 (non-WSL, dual-boot OS), I had run into similar issues but I removed ninja &amp; then install succeeded. However, I haven’t had such luck on WSL2 Ubuntu 20.04.</p>
<pre><code class="lang-auto">[ 79%] Linking CXX shared library ../lib/libtorch_cpu.so
collect2: fatal error: ld terminated with signal 9 [Killed]
compilation terminated.
make[2]: *** [caffe2/CMakeFiles/torch_cpu.dir/build.make:19257: lib/libtorch_cpu.so] Error 1
make[2]: *** Deleting file 'lib/libtorch_cpu.so'
make[1]: *** [CMakeFiles/Makefile2:11850: caffe2/CMakeFiles/torch_cpu.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
Traceback (most recent call last):
  File "setup.py", line 818, in &lt;module&gt;
    build_deps()
  File "setup.py", line 315, in build_deps
    build_caffe2(version=version,
  File "/home/pytorch/pytorch/tools/build_pytorch_libs.py", line 58, in build_caffe2
    cmake.build(my_env)
  File "/home/pytorch/pytorch/tools/setup_helpers/cmake.py", line 345, in build
    self.run(build_args, my_env)
  File "/home/pytorch/pytorch/tools/setup_helpers/cmake.py", line 140, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Debug', '--', '-j', '8']' returned non-zero exit status 2.
</code></pre>

bogoman: <p>Hey! did you find a solution to this?</p>

imaginary: <p>For WSL2, the issue was resolved by setting up a <code>.wslconfig</code> file &amp; assigning specific amount of memory to WSL2. I allocated 8 GB &amp; it worked. I also set the <code>MAX_JOBS</code> environment variable to a low value.</p>
Cuda Out of Memory, even when I have enough free [SOLVED]

marcoramos: <p>EDIT: SOLVED - it was a number of workers problems, solved it by lowering them</p>
<p>I am using a 24GB Titan RTX and I am using it for an image segmentation Unet with Pytorch,</p>
<p>it is always throwing Cuda out of Memory at different batch sizes, plus I have more free memory than it states that I need, and by lowering batch sizes, it INCREASES the memory it tries to allocate which doesn’t make any sense.</p>
<p>here is what I tried:</p>
<p><strong>Image size = 448, batch size = 8</strong></p>
<ul>
<li>“RuntimeError: CUDA error: out of memory”</li>
</ul>
<p><strong>Image size = 448, batch size = 6</strong></p>
<ul>
<li>“RuntimeError: CUDA out of memory. Tried to allocate 3.12 GiB (GPU 0; 24.00 GiB total capacity; 2.06 GiB already allocated; 19.66 GiB free; 2.31 GiB reserved in total by PyTorch)”</li>
</ul>
<p>is says it tried to allocate 3.12GB and I have 19GB free and it throws an error??</p>
<p><strong>Image size = 224, batch size = 8</strong></p>
<ul>
<li>“RuntimeError: CUDA out of memory. Tried to allocate 28.00 MiB (GPU 0; 24.00 GiB total capacity; 2.78 GiB already allocated; 19.15 GiB free; 2.82 GiB reserved in total by PyTorch)”</li>
</ul>
<p><strong>Image size = 224, batch size = 6</strong></p>
<ul>
<li>“RuntimeError: CUDA out of memory. Tried to allocate 344.00 MiB (GPU 0; 24.00 GiB total capacity; 2.30 GiB already allocated; 19.38 GiB free; 2.59 GiB reserved in total by PyTorch)”</li>
</ul>
<p>reduced batch size but tried to allocate more ???</p>
<p><strong>Image size = 224, batch size = 4</strong></p>
<ul>
<li>“RuntimeError: CUDA out of memory. Tried to allocate 482.00 MiB (GPU 0; 24.00 GiB total capacity; 2.21 GiB already allocated; 19.48 GiB free; 2.50 GiB reserved in total by PyTorch)”</li>
</ul>
<p><strong>Image size = 224, batch size = 2</strong></p>
<ul>
<li>“RuntimeError: CUDA out of memory. Tried to allocate 1.12 GiB (GPU 0; 24.00 GiB total capacity; 1.44 GiB already allocated; 19.88 GiB free; 2.10 GiB reserved in total by PyTorch)”</li>
</ul>
<p><strong>Image size = 224, batch size = 1</strong></p>
<ul>
<li>“RuntimeError: CUDA out of memory. Tried to allocate 1.91 GiB (GPU 0; 24.00 GiB total capacity; 894.36 MiB already allocated; 20.94 GiB free; 1.03 GiB reserved in total by PyTorch)”</li>
</ul>
<p>Even with stupidly low image sizes and batch sizes…</p>
<p>EDIT: SOLVED - it was a number of workers problems, solved it by lowering them</p>

saandeep_aathreya: <p>I think there is a GPU memory leak. These are usually caused if you are not detaching your gradient information during inferring and saving your variables to a list. Can you post your source code here?</p>

marcoramos: <p>I think I solved the problem, it was a number of workers problem, lowered them and it seems ok now</p>

Manuel_Alejandro_Dia: <p>You can always run <code>nvidia-smi</code> to see if the processes that you launch are the only ones consuming GPU memory.</p>
<p>There may be other system processes that use the GPU, but they usually don’t use more than 100MB normally (On Ubuntu).</p>
Disabling gradient calculation in jit.script_method

vicki: <p>I am using scripting to convert a model to TorchScript module. How can I disable the grad-calculation in predict method of the module?<br>
I tried the standard <code>no_grad()</code> and <code>set_grad_enabled()</code> options. But it seems both are not supported in a <code>jit.script_method</code>.</p>

omerlux: <p>Did you find a solution? I’m facing the same problem and getting OOM error (I can see when I loop over <code>with torch.no_grad():</code> that the GPU memory increases)</p>
Best way to continue the pre-training of a BERT model

m-nlp-q: <p>Hello,</p>
<p>I am using the <code>PyTorch</code> version of Hugging Face library’s <code>BERT</code> model and I want to continue the pre-training of the model it in a domain specific dataset, before fine-tune it.</p>
<p>What is the best way to do it?</p>
<p>General tips about pre-training are much appreciated.</p>
RuntimeError: invalid multinomial distribution

Pranavk: <p>Hi I am receiving this error while sampling the reason for the error is the occurrence of  ’ nan ’ values but I am not sure why are those values there I am using softmax activation and I have also used torch.nn.utils.clip_grad but still getting same error.</p>
<p>Unhandled exception in thread started by &lt;function service at 0x7fc0d202b140&gt;<br>
Traceback (most recent call last):<br>
File “ppo_takeoff_8_actions.py”, line 352, in service<br>
action = drone.get_action(dist)<br>
File “ppo_takeoff_8_actions.py”, line 122, in get_action<br>
ac = dist.sample()<br>
File “/home/cuda/.local/lib/python2.7/site-packages/torch/distributions/categorical.py”, line 107, in sample<br>
sample_2d = torch.multinomial(probs_2d, 1, True)<br>
RuntimeError: invalid multinomial distribution (encountering probability entry &lt; 0)</p>
<p>Neural network definantion</p>
<blockquote>
<p>Blockquote<br>
self.actor = nn.Sequential(<br>
nn.Linear(in_features= 16 , out_features= 200),<br>
nn.ReLU(),<br>
nn.Linear(in_features= 200 , out_features= 500),<br>
nn.ReLU(),<br>
nn.Linear(in_features= 500 , out_features= 500),<br>
nn.ReLU(),<br>
nn.Linear(in_features= 500 , out_features= 100),<br>
nn.ReLU(),<br>
nn.Linear(in_features= 100 , out_features= 8),<br>
nn.Softmax(dim=-1)<br>
)</p>
</blockquote>
<p>back-propagation step</p>
<blockquote>
<p>Blockquote<br>
self.actor.optimizer.zero_grad()<br>
self.critic.optimizer.zero_grad()<br>
total_loss.backward()<br>
T.nn.utils.clip_grad_norm_(self.actor.parameters() ,5)<br>
T.nn.utils.clip_grad_norm_(self.critic.parameters() , 5)</p>
</blockquote>
<p>Thank you</p>

ptrblck: <p>NaN values can be created from various sources:</p>
<ul>
<li>your input to the model might contain invalid values (NaNs and/or Infs) and might thus “poison” your model</li>
<li>your training might diverge, the loss might explode and either directly create invalid loss values or invalid gradients during the backward pass</li>
<li>your gradients might explode due to an “unstable” operation in case you are using custom operations.</li>
</ul>
<p>You could start by checking the loss and e.g. lowering the learning rate in case you are seeing that the model diverges.</p>

Pranavk: <p>Ok thanks I will check all these things</p>

Pranavk: <p>Hi it was input data error there was one component that was going to infinity . Thank you</p>
Smooth continuation

antigol: <p>I would like to define the function</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bc3d8cac602b9eed3bf8b77826d6bbe0332d801.png" alt="image" data-base62-sha1="3XCutPhMDS6KOp1kHsqBn7fZ46l" width="341" height="81"></p>
<p>This function is infinitely differentiable. It’s derivatives in zero are all zero.</p>
<pre data-code-wrap="python"><code class="lang-python">def f(x):
    return (-1/x).exp() * (x &gt; 0)

x = torch.tensor(0.0, requires_grad=True)
y = f(x)
y.backward()
x.grad  # NaN
</code></pre>
<p>How can I make <code>torch.autograd</code> return the correct derivatives (all of them)?</p>
<p>note: I tried with <code>torch.autograd.Function</code> but I had a problem with the second derivative.</p>

albanD: <p>Hi,</p>
<p>This is expected I’m afraid. The chain rule that the autograd uses only works at points where the function is differentiable. But 1/x here is not, hence the nan.</p>
<p>You can indeed use a custom Function to achieve that.<br>
Can you share your issue with that?<br>
One thing to keep in mind for double backward is that you should <em>not</em> save intermediary Tensors and re-use them. Only input/output. You can make intermediary Tensors dummy outputs if needed here.</p>

antigol: <p>It worked! Thanks</p>
<pre data-code-wrap="python"><code class="lang-python">class SmoothStep(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        ctx.save_for_backward(x)
        y = torch.zeros_like(x)
        m = (x &gt; 0.0)
        y[m] = (-1/x[m]).exp()
        return y

    @staticmethod
    def backward(ctx, dy):
        x, = ctx.saved_tensors
        dx = torch.zeros_like(x)
        m = (x &gt; 0.0)
        xm = x[m]
        dx[m] = (-1/xm).exp() / xm.pow(2)
        return dx * dy
    
f = SmoothStep.apply
x = torch.linspace(-1, 1, 1000, requires_grad=True)

y0 = f(x)
y1, = torch.autograd.grad(y0.sum(), x, create_graph=True)
y2, = torch.autograd.grad(y1.sum(), x, create_graph=True)
y3, = torch.autograd.grad(y2.sum(), x, create_graph=True)

assert torch.isfinite(y0).all() and torch.isfinite(y1).all() and torch.isfinite(y2).all() and torch.isfinite(y3).all()
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74c996225738ab2ee56b7193d4b3378842667211.png" alt="image" data-base62-sha1="gF9hA3C89F3aV7CX98Ecrkqah57" width="380" height="252"></p>
Best way to downsample-batch image tensors

Hyung_Jin_Chung: <p>Say you have a gray image tensor of shape <code>(1, 1, 128, 128) </code>. What I would like to do here is to sample in each <code>h, w </code>dimension with <code>stride=2</code>, which would then make 4 sub-images of size <code>(1, 1, 64, 64)</code> depending on where the indexing starts. Then, I would like to batch them to finally form the tensor of size <code>(4, 1, 64, 64)</code>. For now, I’m using double for loops which is presumably inefficient. Is there an efficient way to vectorize this using functions implemented in pytorch?</p>
<p>The function I’m using right now:</p>
<pre><code class="lang-auto">def img_subsample(img, ratio=2):
    '''
    ex) Subsamples img of size 128 * 128,
    output is batched into 4 * 1 * 64 * 64
    Args:
        img: input of size ex) 128 * 128
        ratio(int): Subsampling ratio
    Returns: batched, subsampled img
    '''
    sz = img.shape[-1]
    batched_output = torch.zeros([ratio**2, 1, (sz//ratio), (sz//ratio)])
    for i in range(ratio):
        for j in range(ratio):
            batched_output[ratio*i+j, :, :, :] = img[i::ratio, j::ratio]
    return batched_output
</code></pre>

Naruto-Sasuke: <p>Refer to nn.PixelShuffle()</p>

Sunshine352: <p><code>torch.nn.functional.</code> <code>unfold</code> (<em>input</em>, <em>kernel_size</em>, <em>dilation=1</em>, <em>padding=0</em>, <em>stride=1</em> )<a href="https://pytorch.org/docs/1.0.0/_modules/torch/nn/functional.html#unfold" rel="noopener nofollow ugc">[SOURCE]</a></p>
<p>Extracts sliding local blocks from an batched input tensor.</p>
<p>WARNING</p>
<p>Currently, only 4-D input tensors (batched image-like tensors) are supported.</p>
<p>See <a href="https://pytorch.org/docs/1.0.0/nn.html?highlight=fold#torch.nn.Unfold" rel="noopener nofollow ugc"><code>torch.nn.Unfold</code></a> for details</p>
Concatante k linear layers results to single output tensor

davidsriker: <p>I have 14 classification blocks each one is: 1024 → 1.<br>
I wish to concatenate the 14 results to form a 1-D output as if I had 1024 → 14 classes.</p>
<pre><code class="lang-auto">prior_layers = []
    for l in LABELS:
         prior_layers.append(LinearAttentionPriorBlock(l, normalize_attention=normalize_attention))
</code></pre>
<p>then in the forward pass I would have something like:</p>
<pre><code class="lang-auto">res = []
    for l in LABELS:
         res.append(LinearAttentionPriorBlock(x))
</code></pre>
<p>and each entry int <code>res</code> will be batchX1 is it possible to get the desired results as if it was standard classification or the only way is to output 14 values from the forward pass.</p>

Dwight_Foster: <p>You could do something like</p>
<pre><code class="lang-auto">torch.stack(res, dim = 1)
</code></pre>
<p>that should output the thing you are looking for.</p>

davidsriker: <p>Great, that exactly what I was missing.</p>
Pytorch profiling in multi-gpu system

sangheonlee: <p>Hi,</p>
<p>My system is RTX 2080Ti * 8 and it was Turing architecture, So I have to use ncu instead of nvprof.</p>
<p>When I running the PyTorch with metric of ncu, If i just running the one GPU, they profile the kernel exactly what I want to. But if I running on the multi-GPU, it may be called ncclAllReduce, they cannot profile and stop before the start the PyTorch imagenet.</p>
<p>Can i ask why it cannot profile the imagent in multi-GPU, or recommend any else profiler…</p>
<p>I want to know how to profile in ncu or nvprof.</p>
<p>Below figure are screen shot about stopped imagenet in multi-GPU.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3bf992308ad3cdc7521bb8a454e78ff072dbdd7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a3bf992308ad3cdc7521bb8a454e78ff072dbdd7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3bf992308ad3cdc7521bb8a454e78ff072dbdd7_2_690x160.jpeg" alt="image" data-base62-sha1="nmAk8Y48O6UwvKIiXiUZ7TWwxdJ" width="690" height="160" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3bf992308ad3cdc7521bb8a454e78ff072dbdd7_2_690x160.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3bf992308ad3cdc7521bb8a454e78ff072dbdd7.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3bf992308ad3cdc7521bb8a454e78ff072dbdd7.jpeg 2x" data-dominant-color="191818"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">755×176 65.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks</p>

ptrblck: <p>You could use Nsight Systems as described <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59?u=ptrblck">here</a>.</p>

sangheonlee: <p>Oh <a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks.</p>
<p>Hmm,  If I used nsight, it shows the cache hit rate at multi-GPU?</p>
<p>And also, I hope collective communication also will be profiled.</p>
Collate function on libtorch

Monne: <p>Hi,</p>
<p>Is there some example of using collate func for dataloader with custom dataset in libtorch?</p>

ptrblck: <p><a href="https://github.com/pytorch/pytorch/blob/8c798e062216278673a75bac0848ea69a8bd3f03/test/cpp/api/dataloader.cpp#L1578-L1620">This</a> collate test might be useful.</p>

Monne: <p>Thanks, i will look into it</p>
Waring of named tensors in andriod phone

lzj: <p>Hi, when I run my pytorch model on an android phone, there was a warning as fellow:</p>
<p>Warning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (function operator()).</p>
<p>I initially thought that some operators in my model caused this problem, but when I use the model only of one conv layer on an android phone, it appeared again.</p>
<p>The model structure is shown in the figure below<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d81cb72d8446a2365a4fc1546ae95bbd2697fe6a.png" alt="图片" data-base62-sha1="uPOEH6j3HXbplJi685y1NGVRoro" width="273" height="290"></p>
<p>Why does this happen？my pytorch vision is v1.6.</p>

IvanKobzarev: <p>Named tensor is still in prototype phase.</p>
<p>The list of supported operators by named tensors is here:<br>
<a href="https://pytorch.org/docs/stable/name_inference.html#name-inference-reference-doc" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/name_inference.html#name-inference-reference-doc</a></p>
<p>convolution operator is not supported for named tensors.</p>

lzj: <p>Thank you for your answer!<br>
I know the convolution operator is not supported for named tensors. It is just because  it is unsupported, there is no named tensors operator in my model. Hence, it is normal that this warning should not appear? But the warning appeared.<br>
Or to say, is this warning bound to appear in the relatively recent version? But when I run the official demo app named HelloWorldApp, the warning doesn’t appear. Demo’s pytorch version and andriod library is 1.3.</p>
[Android] Pytorch converter for android application

111485: <p>Hi<br>
This is a student who conducts research related to images.<br>
I trained my custom data for object detection that estimates the bounding box using mobilenet.<br>
When I evaluated it on the desktop, it worked.<br>
So I wanted to install the app on my Android-based mobile phone and test it together.</p>
<p>However, in Android applications, when I load the model, the application crashes and the application keeps shutting down. Can I see why??</p>
<p>Android studio error.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff9982114a392f89d3b4e8c0e675f3f267edb36f.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff9982114a392f89d3b4e8c0e675f3f267edb36f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff9982114a392f89d3b4e8c0e675f3f267edb36f.png" alt="image" data-base62-sha1="At8DXsGq3MOR4DFy2Q7OK4CZu23" width="690" height="322" data-dominant-color="F8F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">899×420 15 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is the site I referenced.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/qfgaohao/pytorch-ssd">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/qfgaohao/pytorch-ssd" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/1967894e8a44c70a6a8ec79c3070d4445f94a574a4190f377803663640661538/qfgaohao/pytorch-ssd" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/qfgaohao/pytorch-ssd" target="_blank" rel="noopener nofollow ugc">GitHub - qfgaohao/pytorch-ssd: MobileNetV1, MobileNetV2, VGG based...</a></h3>

  <p>MobileNetV1, MobileNetV2, VGG based SSD/SSD-lite implementation in Pytorch 1.0 / Pytorch 0.4. Out-of-box support for retraining on Open Images dataset. ONNX and Caffe2 support. Experiment Ideas lik...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I referenced the code here, and tried converting it into a trace file and a script file, but all crashed.</p>
Torch.jit.load() not equivalent to torch.load() + torch.jit.trace(eval())

hidefromkgb: <p>I have succeeded adding my custom primitives to PyTorch via <code>RegisterOperators</code> and <code>RegisterPass</code> from within PyBind11. Models from TorchVision that I load and transform using <code>torch.jit.trace</code> definitely have them — I see their console output and the output tensor bears clear marks of their work.</p>
<p>However, when I try to <code>torch.jit.save</code> the traced model and <code>torch.jit.load</code> it back some other time, my prims are nowhere to be seen.</p>
<p>What am I doing wrong? How do I use my own primitives in a network loaded using <code>torch.jit.load</code>?</p>
Exclude loss from some processes in backward()

Leander_Thiele: <p>I have set up my model with DistributedDataParallel, which is working well (i.e. it runs). The problem is that during early epochs some processes compute infinite/nan losses. Can I simply omit the call to backward() in those cases or will this confuse the synchronization that’s happening under the hood?</p>

rvarm1: <p>All calls to <code>backward()</code> should be independent across successive calls, but not independent with respect to different processes on the <em>same</em> call.</p>
<p>In order to arbitrarily skip backwards, you can use the <code>no_sync</code> context manager:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html?highlight=no_sync#torch.nn.parallel.DistributedDataParallel.no_sync" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html?highlight=no_sync#torch.nn.parallel.DistributedDataParallel.no_sync</a><br>
However, be aware that this must be used across all ranks otherwise there will be synchronization issues. If you only need to skip backwards a few times early in the training, you could also consider having all processes agree to skip backward() or not:</p>
<pre><code class="lang-auto">should_skip_backwards = ... # each process computes this
should_skip_bwd_list = [None for _ in range(nranks)]
torch.distributed.all_gather_object(should_skip_bwd_list, should_skip_backwards)

globally_skip_bwd = any(should_skip_bwd_list) # all ranks agree
</code></pre>
Vectorized Jacobian Vector Products (in 1.8)

bchen: <p>Is it now possible with Pytorch 1.8 to compute vectorized Jacobian vector products? I know in the past it wasn’t (e.g., <a href="https://discuss.pytorch.org/t/jacobian-functional-api-batch-respecting-jacobian/84571" class="inline-onebox">Jacobian functional API batch-respecting Jacobian</a>), but I am wondering if that has changed with Pytorch 1.8 through some implementation of vmap.</p>
<p>In particular, I am referring to the following two use cases:</p>
<ol>
<li>Calculating the Jacobian vector products Jv_i for i=1,…, N, where J is the Jacobian of a function f at a point x and v_i are a set of vectors.</li>
</ol>
<p>I am wondering if this is how the new vectorized torch.autograd.functional.jacobian/hessian are implemented (with $v_i$ being the standard basis), and if so is there a way to do this for any general set of vectors $v_i$? Related in particular to <a href="https://github.com/pytorch/pytorch/pull/50915" class="inline-onebox" rel="noopener nofollow ugc">Add `vectorize` flag to torch.autograd.functional.{jacobian, hessian} by zou3519 · Pull Request #50915 · pytorch/pytorch · GitHub</a></p>
<ol start="2">
<li>Calculating the Jacobian vector products J_i v_i for i = 1, …, N, where J_i is the Jacobian of a function f at a point x_i (the difference vs. 1 is now also calculating the Jacobian over a batch of different inputs x_i)</li>
</ol>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<ol>
<li>We actually do that by doing <code>vJ</code> with a batch of <code>v</code> so you won’t be able to re-use it here.</li>
<li>That would be “element-wise” jacobian vector product. There are two cases here:<br>
a. if your function is already working with “batch” with no interaction between them. Then you can compute a regular Jv and it will give the same result.<br>
b. If there are interaction between the batches in the forward and you don’t want to see that in the Jacobian, you’ll have to do a for-loop around the Jv computation for now I’m afraid.</li>
</ol>

bchen: <p>Hi Alban,</p>
<p>Thanks! Could you give me a bit more information about 1)? A vJ product with a batch of v works just as well for me.</p>

albanD: <p>In that case, yes this is how it is implemented using the new vmap prototype.<br>
You can see the code here: <a href="https://github.com/pytorch/pytorch/blob/e91aeb0470f329556f599ac9feb21b0cff31e11c/torch/autograd/functional.py#L541-L550" class="inline-onebox">pytorch/functional.py at e91aeb0470f329556f599ac9feb21b0cff31e11c · pytorch/pytorch · GitHub</a></p>
3D daya to cobtibious (1,3) network?

Eren_Cerman: <p>I have 3D grayscale photo data of 6 slices and my input shape is therefore (6,1,100,100)</p>
<p>I want to estimate realtime 3 different characteristics and therefore (1,3) shaped output. The estimated values are all continious variables.</p>
<p>which network would be suitable dor estimation of continious variables?</p>

Dwight_Foster: <p>A CNN would probably be a good option. Just don’t use an activation function at the end so the output is continuous. You could squeeze the image to get rid of the 1 dimension and use 6 as your input channels. This would give you the correct output.</p>
How to provide input for the resnet50?

bucky: <p>Hi, I am trying to execute <code>./bin/resnet-training</code> and unable to figure out how to provide input for it.</p>
<p><code>./bin/resnet-training -help</code>   suggests<br>
<code>USAGE: resnet-training [options] --input-image-list-file Name of the file containing list of images (one image per line)</code><br>
however I dont understand what a " file containing list of images." is. is it a CSV file? or just a directory containing images?</p>
<p>I want to train resnet50 with imagenet data</p>

jfix: <p>I think it’s just a file with a list of file paths comma separated with the label for that image:</p>
<pre><code class="lang-auto">/path/to/cat.png,285
/path/to/dog.png,207
/path/to/zebra.png,340
</code></pre>
Identify a Matrix with PyTorch

spe2005: <p>Hi,</p>
<p>I am a beginner with PyTorch. I want to identify the matrix A in the ODE: dX/dt=A*X. So, I wrote a PyTorch code to do that. I only use a linear cell in the neural network.</p>
<p>Unfortunately for me, it doesn’t work at all. When I run the code, the loss is decreasing, but when I check the weights of the matrix, they are totally different from the coefficients of the matrix A.</p>
<p>Can someone help me?</p>
<p>The link of the notebook : <a href="https://colab.research.google.com/drive/1hngoL3R9G7jx3Yv-mTYMGRPFjbHiaaCC?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1hngoL3R9G7jx3Yv-mTYMGRPFjbHiaaCC?usp=sharing</a></p>
<p>Thanks,</p>
<p>Thomas</p>
How to combine multiple torch.load

Ripper346: <p>Hi, I am trying to make a custom Dataset with pytorch geometric. I need to replace the last row of <code>__init__</code> of the class in <a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html#creating-in-memory-datasets" class="inline-onebox" rel="noopener nofollow ugc">Creating Your Own Datasets — pytorch_geometric 1.6.3 documentation</a> to adapt it for multiple loads because I have several files to load in the same dataset.<br>
From what I understood (I can’t find in the docs), <code>torch.load(file_path)</code> returns a tuple with Data and slice. How can I combine these two for multiple files keeping the object structure? It is a little hard to explain it, I hope you understand my problem. Thank you</p>

patrickwilliams3: <p>It looks like you can pass an io.BytesIO() into torch.load. I would try using the io.BytesIO to read from multiple files and then load that into torch.load() with a singular buffer.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c180c92dfa586e9b413d60b0b83fca731933ff45.png" alt="Screen Shot 2021-03-15 at 11.50.02 AM" data-base62-sha1="rBO4BjM9tGlh4fS6LYVSLFz8lmd" width="670" height="146"></p>

Ripper346: <p>Ok, but I only moved the problem, how can I concatenate those BytesIO? I tried this</p>
<pre><code class="lang-auto">buffer = io.BytesIO()
for file in self.processed_paths:
    with open(file, 'rb') as f:
        buffer.write(f.read())
self.data, self.slices = torch.load(buffer)
</code></pre>
<p>but I get an EOFError when loading</p>

patrickwilliams3: <p>Try doing buffer.seek(0) before torch.load()</p>

Ripper346: <p>With seek changed the error. Now I have:</p>
<pre><code class="lang-auto">RuntimeError: [enforce fail at ..\caffe2\serialize\inline_container.cc:145] . PytorchStreamReader failed reading zip archive: invalid header or archive is corrupted
</code></pre>
<p>It kind makes sense of this error, I only concatenated some buffers. I don’t think that it is a well built structure for torch to load, isn’t it? The files are working independently, this isn’t a file divided in multiple files for some reason.<br>
Let’s say I have <code>a.pt</code> and <code>b.pt</code>, inside a.pt I have a dataset, inside b.pt I have another that I have to merge together. If I run <code>torch.load('a.pt')</code> it works, same thing for b.pt.</p>
Validation of the dataset in GCN

Saguaro: <p>Hello!</p>
<p>I am trying to do the validation on my dataset to obtain optimal number of channels in my neural network. I have the following code:</p>
<pre><code class="lang-auto">def train_during_validation():
    for epoch in range (1, 201):
        model.train()
        optimizer.zero_grad()
        out = model(data.x, data.edge_index)
        loss = criterion(out[data.val_mask], data.y[data.val_mask])
        loss.backward()
        optimizer.step()
    return loss

def validation():
    loss_val = np.zeros(50, dtype = float)
    model = GCN(hidden_channels = 1)
    loss_val = train_during_validation()
    print(loss_val)
        
validation()

</code></pre>
<p>In the code above I train previously defined model with 16 channels and I obtain a loss of 0.33. But as soon as I start doing validation on <code>hidden_channel</code> (see code below), my loss does not go down (it remains on 1.95). I do not understand why. Can somebody explain?</p>
<pre><code class="lang-auto">def train_during_validation(model):
    print(f'Model:{model}')
    for epoch in range (1, 201):  
        model.train()
        optimizer.zero_grad()
        out = model(data.x, data.edge_index)
        loss = criterion(out[data.val_mask], data.y[data.val_mask])
        loss.backward()
        optimizer.step()
    return loss

def validation():
    loss_val = np.zeros(50, dtype = float)
    model = GCN(hidden_channels = 1)
    for i in range (50):
        model = GCN(hidden_channels = i)
        #print(model)
        loss_val[i] = train_during_validation(model)
        print(loss_val[i])
        
validation()
</code></pre>
<p>Thank you in advance!</p>
<p>Best regards</p>
3D Deep Learning Model on Android

NAVEEN_PALURU: <p>Hi All,</p>
<p>Is there a way for deploying a 3D deep learning model in android? I mean my input is of dimension (batch,channel,depth,height,width) = (1,1,64,64,64) and my output is also of dimension (1,1,64,64,64) ?</p>

IvanKobzarev: <p>Yes, there is no limitations on dimension of tensors for pytorch mobile.</p>
What's CUDA StreamGuard?

111480: <p>What’s role CUDA Stream Guard??</p>
<p>What can I use it for?</p>
<p>The official document lacks explanation, where can I see the detailed explanation?</p>

ptrblck: <p>The <a href="https://pytorch.org/cppdocs/notes/tensor_cuda_stream.html">libtorch docs</a> explain its usage and also the <a href="https://github.com/pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/c10/core/StreamGuard.h#L7-L21">inline docs</a> might be helpful.</p>

111480: <p>Front and back of guard “{}”  What is the role?<br>
Is it vital?<br>
I want a C++ multi-threading multi-stream environment. Is it right to use Guard?</p>
Why is there such a big difference in accuracy when using ReLU verus Sigmoid?

jpj: <p>I am using two hidden layers in a feed forward neural network, with SGD optimizer. Using CrossEntropyLoss for the loss.</p>
<p>The dataset has 4 classes and the majority class is 67% of the dataset.</p>
<p>When I use ReLU in the hidden layers, I get an extremely good accuracy(and precision as well).<br>
But when I use sigmoid, it does not perform well and mostly only predicts the majority class.</p>
<p>Why is there such a big contrast?</p>
Create multi-dimensional sparse tensor

Graviton: <p>I try to create a multi-dimensional sparse tensor in my implemention of sparse GAT, but I get the following error.</p>
<pre data-code-wrap="python"><code class="lang-python">class SpecialSpmmFunction(torch.autograd.Function):
    """Special function for only sparse region backpropataion layer."""
    @staticmethod
    def forward(ctx, indices, values, shape, b):
        assert indices.requires_grad == False
        a = torch.sparse_coo_tensor(indices, values, shape) #  This error occurred
        ctx.save_for_backward(a, b)
        ctx.N = shape[0]
        return torch.matmul(a, b)

    @staticmethod
    def backward(ctx, grad_output):
       # ............
class SpecialSpmm(nn.Module):
    def forward(self, indices, values, shape, b):
        return SpecialSpmmFunction.apply(indices, values, shape, b)
#.............
 e_rowsum = self.special_spmm(edge.unsqueeze(0).expand(nbatches, -1, -1), edge_e, torch.Size([nbatches, N, N]), torch.ones(size=(N, 1), device=dv))
</code></pre>
<blockquote>
<p>a = torch.sparse_coo_tensor(indices, values, shape)<br>
RuntimeError: indices must be sparse_dim x nnz, but got: [161, 2, 3025]</p>
</blockquote>
<p><code>torch.sparse_coo_tensor</code> seems like that cannot create multi-dimensional sparse tensor</p>
Question about how to implement scatter a tensor

x-zho14: <p>Hi, I have would like to implement such a requirement: a is a tensor(3,4), b is a tensor(5,4),  mask is a tensor = [1, 0, 1, 0, 1]. How to put a to b according to the mask where is 1? That is b[0] = a[0], b[2] = a[1], b[3] = a[2]. Thank you very much!</p>

Napam: <p>You can achieve this by using a boolean mask rather than your numeric mask (PyTorch, NumPy, etc implements different behaviors depending on the data type in the arrays). For your example you would have to do</p>
<pre><code class="lang-auto">import torch 

a = torch.arange(3*4).reshape(3,4)
b = torch.zeros(5,4, dtype=torch.long)
mask = torch.tensor([1,0,1,0,1])
b[mask.bool()] = a
print(b)
# tensor([[ 0,  1,  2,  3],
#         [ 0,  0,  0,  0],
#         [ 4,  5,  6,  7],
#         [ 0,  0,  0,  0],
#         [ 8,  9, 10, 11]])
</code></pre>
RuntimeError('dot : expected both vectors to have same dtype, but found Double and Float

sugh7020: <p>I am trying to perform this operation:</p>
<pre><code class="lang-auto">L_c += 1 - torch.abs(torch.dot(nuNormalized, Nu))
</code></pre>
<p>where:</p>
<blockquote>
<p>nuNormalized → tensor([ 0.3790, -0.4208,  0.8242], dtype=torch.float64)<br>
Nu → tensor([-0.9961, -0.9961, -0.9961], device=‘cuda:0’)</p>
</blockquote>
<p>I am getting this error:</p>
<blockquote>
<p>(&lt;class ‘RuntimeError’&gt;, RuntimeError(‘dot : expected both vectors to have same dtype, but found Double and Float’,), &lt;traceback object at 0x7f4d276155c8&gt;)</p>
</blockquote>
<p>Any suggestions please? Thanks!</p>

Dwight_Foster: <p>You could try converting Nu to float using this:</p>
<pre><code class="lang-auto">L_c += 1 - torch.abs(torch.dot(nuNormalized, Nu.float()))
</code></pre>

sugh7020: <p>Hi <a class="mention" href="/u/dwight_foster">@Dwight_Foster</a> Thanks for helping.<br>
Were you suggesting nuNormalized.float()?</p>

Dwight_Foster: <p>No it says nuNormalized is already a float. I was talking about just Nu. Looking at it now though what is the type of L_c that might be the problem. If that is a double then you may have to do</p>
<pre><code class="lang-auto">L_c += 1 - torch.abs(torch.dot(nuNormalized, Nu)).double()
</code></pre>

sugh7020: <p><a class="mention" href="/u/dwight_foster">@Dwight_Foster</a> , I tried:</p>
<pre><code class="lang-auto">L_c += 1 - torch.abs(torch.dot(nuNormalized, Nu)).double()
</code></pre>
<p>I am getting:</p>
<blockquote>
<p>RuntimeError: dot : expected both vectors to have same dtype, but found Double and Float</p>
</blockquote>

Dwight_Foster: <p>Try this:</p>
<pre><code class="lang-auto">L_c += 1 - torch.abs(torch.dot(nuNormalized, Nu.float()))

</code></pre>
<p>If that does not work can you print out the type of both of those just doing this</p>
<pre><code class="lang-auto">print(nuNormalized.type(), Nu.type())</code></pre>
Applying softmax function on logins for computing mean IOU

887574002: <p>Hi, I have a minor question about computing mean IOU in segmentation task.<br>
Should we apply an activation function on logins before computing mean IOU?<br>
I tried both ways, computing the mean IOU with and without applying a softmax function on logins. The scores that I obtained for Dice Loss as well as Dice score are different.<br>
So, what is the correct one. I want to report my results on a paper and I need to know which one should I rely.</p>
<p>I will appreciate any help.</p>
Copy Encoder with same key name

Praveen_K: <p>I have a CustomModel built in a following way:</p>
<pre><code class="lang-auto">class CustomModel(nn.Module):
    def __init__(self):
        super(CustomModel, self).__init__()
        self.encoder = nn.Sequential(nn.Linear(48 * 48 * 3, 2646), nn.ReLU())
        self.decoder = nn.Sequential(nn.Linear(2646, 48 * 48 * 3), nn.ReLU())


    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x
</code></pre>
<p>Create the model for training.</p>
<pre><code class="lang-auto">net = CustomModel()
print(net)
</code></pre>
<p>The Output of <code>net</code></p>
<pre><code class="lang-auto">CustomModel(
  (encoder): Sequential(
    (0): Linear(in_features=6912, out_features=2646, bias=True)
    (1): ReLU()
  )
  (decoder): Sequential(
    (0): Linear(in_features=2646, out_features=6912, bias=True)
    (1): ReLU()
  )
)
</code></pre>
<p>After training <code>autoencoder</code>, I would like to remove <code>decoder</code> part and attach <code>classifier</code> layer.<br>
So I’m doing the following:</p>
<pre><code class="lang-auto">new_classifier = nn.Sequential(*list(net.children())[:-1])
net = new_classifier
net.add_module('classifier', nn.Sequential(nn.Linear(2646, 850), nn.LogSoftmax(dim=1)))
print(net)
</code></pre>
<p>The output of this <code>Encoder-Classifier</code></p>
<pre><code class="lang-auto">Sequential(
  (0): Sequential(
    (0): Linear(in_features=6912, out_features=2646, bias=True)
    (1): ReLU()
  )
  (classifier): Sequential(
    (0): Linear(in_features=2646, out_features=850, bias=True)
    (1): LogSoftmax()
  )
)
</code></pre>
<p>If you can see the <code>net</code> after adding <code>classifier</code>, the key name of <strong>encoder</strong> has changed to <strong>0</strong>. How can i copy the encoder with same key name as my base class.</p>
<p>Can anyone please help me with this !</p>

Napam: <p>You can simply pass an <code>OrderedDict</code> with <code>net.named_children()</code> to <code>nn.Sequential</code> like this:</p>
<pre><code class="lang-python">new_classifier = nn.Sequential(OrderedDict(list(net.named_children())[:-1]))
</code></pre>
<p>:^)</p>

Praveen_K: <p>Great. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Adding Dropout in Pre-trained Vgg16

ageryw: <p>Pre-trained  VGG16 architecture does not contain a dropout layer except in the last classifier layer. How can I add dropout layer after every convolution layer in VGG 16 if I want to keep the value of VGG16 parameters (weights/Biases)? (FYI, I wanted to add dropout layer between the convolutional layers in order to quantify MC-Dropout uncertainty during prediction).</p>

aguennecjacq: <p>To do so, you need to modify VGG.features. It is a nn.Sequential object and so all you need to do is modify the layers this specific object, you don’t need to modify the forward. Something of the following:</p>
<pre><code class="lang-auto">model = torchvision.model.vgg19(pretrained=True)
feats_list = list(model.features)
new_feats_list = []
for feat in feats_list:
    new_feats_list.append(feat)
    if isinstance(feat, nn.Conv2d):
        new_feats_list.append(nn.Dropout(p=0.5, inplace=True))

# modify convolution layers
model.features = nn.Sequential(*new_feats_list)
</code></pre>

ageryw: <p><a class="mention" href="/u/aguennecjacq">@aguennecjacq</a>  Oh thank you so much.   This is so helpful. It worked!</p>
How to broadcast a 1D tensor with a 4D tensor?

671fc6655b8b9292a04b: <p>Suppose the size of a is (128,) and the size of b is (128, 3, 64, 64). How to do broadcast multiplication with these two tensors?</p>

kelam_goutam: <p>I guess u need to convert tensor A to size (128,1) and then multiply with tensor B.</p>
<pre><code class="lang-auto">output  = A. unsqueeze(1) * B 
</code></pre>

671fc6655b8b9292a04b: <p><a class="mention" href="/u/kelam_goutam">@kelam_goutam</a> Thanks for your reply. It works if the size of b is (128,3), i.e, b is 2-dimensional tensor, but doesn’t work when the dimension of b is more than 2 dimensions.</p>

justusschock: <p>the most generic way would be to do something like this (which is not broadcasting, but explicit expanding):</p>
<pre><code class="lang-auto">output = a.expand_as(b)*b
</code></pre>
<p>Edit:<br>
if you want to do it with broadcasting, you can simply do</p>
<pre><code class="lang-auto">output = a * b
</code></pre>
<p>if the shapes cannot be broadcasted, an error will be raised, but you will also receive this error if you use the explicit approach</p>

671fc6655b8b9292a04b: <p>Thanks for your apply. I tried the solution but it does not work as I expected.<br>
A related question. Suppose I have a = torch.tensor([1,2,3]), how to expand it to</p>
<p>tensor([[[ 1,  1,  1],<br>
[ 1,  1,  1]],</p>
<pre><code>       [[ 2,  2,  2],
       [ 2,  2,  2]],

       [[ 3,  3,  3],
       [ 3,  3,  3]]])
</code></pre>
<p>Thanks!</p>

kelam_goutam: <p>A crude way to solve your problem could be the extension of my previous solution.</p>
<pre><code class="lang-auto">output = A.unsqueeze(1).unsqueeze(2).unsqueeze(3) * B
</code></pre>
<p>This basically makes A a 4D tensor.</p>

671fc6655b8b9292a04b: <p><a class="mention" href="/u/kelam_goutam">@kelam_goutam</a> Thanks!</p>

colesbury: <p>These also work:</p>
<pre><code class="lang-auto">output = a.reshape(128, 1, 1, 1) * b
</code></pre>
<p>or</p>
<pre><code class="lang-auto">output = a.reshape(-1, 1, 1, 1) * b
</code></pre>

iacob: <p>You can broadcast a vector to a higher dimensional tensor <a href="https://stackoverflow.com/a/66636211/9067615" rel="noopener nofollow ugc">like so</a>:</p>
<pre><code class="lang-auto">def row_mult(t,vec):
    extra_dims = (1,)*(t.dim()-1)
    return t * vec.view(-1, *extra_dims)
</code></pre>