akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Benchmark M1 GPU VS 3080 (or other). Is it reasonable to buy / use M1 GPU?

paantya: <p>Tell me please, did anyone compare the performance of the m1 GPU with other video cards, like 3080?<br>
To evaluate how well they perform for the tasks of learning fully connected, convolutional, recurrent layers.</p>
<p>And does it make sense now to think about using the m1 GPU</p>

philipturner: <p>Take a look at some of the posts on this thread. One compares Nvidia and Apple GPUs. Although both have hardware acceleration, Apple’s hardware acceleration only increases performance by a tiny amount (well, not really “tiny”, but that’s for another discussion). On Nvidia, you can reach 100 TFLOPS of processing power or more because of tensor cores.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://forums.fast.ai/t/mac-m1-gpus/96428/2">
  <header class="source">
      <img src="https://forums.fast.ai/uploads/default/optimized/3X/b/3/b395de7a2ba00b82865031c97a8cadf3d80e71e5_2_32x32.png" class="site-icon" width="32" height="32">

      <a href="https://forums.fast.ai/t/mac-m1-gpus/96428/2" target="_blank" rel="noopener nofollow ugc" title="01:09PM - 12 May 2022">Deep Learning Course Forums – 12 May 22</a>
  </header>

  <article class="onebox-body">
    <img src="https://forums.fast.ai/uploads/default/original/3X/b/3/b395de7a2ba00b82865031c97a8cadf3d80e71e5.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://forums.fast.ai/t/mac-m1-gpus/96428/2" target="_blank" rel="noopener nofollow ugc">MAC M1 GPUs</a></h3>

  <p>As I understand, for fastai to make use of these GPUs, the underlying pytorch framework would need to work with it. Pytorch team seems to be working on it, but I haven’t heard any pytorch builds that can leverage the M1 architecture (yet.).  EDIT:...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


paantya: <p>Tnx</p>
<p>Max gpu suppurt pytorch:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/388;"><img src="https://pytorch.org//assets/images/METAPT-002-BarGraph-02-static.png" class="thumbnail" width="690" height="388"></div>

<h3><a href="https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/" target="_blank" rel="noopener nofollow ugc">Introducing Accelerated PyTorch Training on Mac</a></h3>

  <p>In collaboration with the Metal engineering team at Apple, we are excited to announce support for GPU-accelerated PyTorch training on Mac. Until now, PyTorch training on Mac only leveraged the CPU, but with the upcoming  PyTorch v1.12 release,...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Can't use pytorch operator: max_pool2d_with_indices in coremltools

pboudoin: <p>Nowadays, a lot of PyTorch models use <a href="https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html" rel="noopener nofollow ugc">MaxPool2d</a> operator with the option return_indices=True</p>
<p>I need to convert some PyTorch models into CoreML. For that I use coremltools. But coremltools doesn’t support yet this operator with return_indices=True. And it is confirmed in a GitHub <a href="https://github.com/apple/coremltools/issues/1081" rel="noopener nofollow ugc">issue</a> on their repo</p>
<p>When you try using this code:</p>
<pre><code>traced_model = torch.jit.trace(model, input_t)
mlmodel = ct.convert(
    traced_model,
    inputs=[ct.TensorType(shape=input_t.shape)]
)
</code></pre>
<p>You’ve got this runtime error:</p>
<pre><code>RuntimeError: PyTorch convert function for op 'max_pool2d_with_indices' not implemented
</code></pre>
<p>After reading the doc of coremltools, you’ve got 2 choices</p>
<ul>
<li>Implementing a <a href="https://coremltools.readme.io/docs/custom-operators" rel="noopener nofollow ugc">custom operator</a> in coremltools: Not as easy as it seems</li>
<li>Decompose the PyTorch model causing the issue into coremltools supported operators: That’s what I try to do, but I’m debutant in PyTorch.</li>
</ul>
<p>So I need help in order to decompose this:</p>
<pre><code>pool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1, return_indices=True)
output, indices = pool(input)
</code></pre>
<p>Into something like</p>
<pre><code>pool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1, return_indices=False) # &lt;-- notice the return_indices=False here
output = pool(input)
indices = some code here...
</code></pre>
<p>Please could someone help me?</p>
Splitting 3D tensor into multiple 3D tensors based on condition

svenac20: <p>Hi,<br>
I have a question about splititng a 3D tensor into multiple smaller 3D tensors.<br>
I have a tensor of size Nx3x3 lets name it R. I want to split this Nx3x3 into 3 smaller tensors (so these smaller tensors would be of size Ax3x3, Bx3x3, Cx3x3 where A,B,C &lt;= N)<br>
The problem is i want to split it by the conditions<br>
So basic flow goes like this<br>
if (R[:, 2, 2] &gt; 0)<br>
do something…<br>
else if (R[:, 1, 1] &gt; 0)<br>
do something…<br>
else<br>
do something<br>
My problem is now that some 3x3 tensor can satisfy both the conditions (it can have [2][2] &gt; 0 and [1][1] &gt; 0). How can i remove the tensor that satisfies the first condition and second condition from tensors that satisfy only the second condition?</p>
Upsample for 5D tensor is not implemented?

Shunsuke_Saito: <p>Does PyTorch 0.3.0 support Upsample for volumetric input (BatchSize, C, D, H, W)?<br>
Though the docs say it support 3-D, 4-D or 5-D tensor, I got the following error:</p>
<blockquote>
<blockquote>
<blockquote>
<p>A = torch.ones(4,5,10,10,10)<br>
up = torch.nn.Upsample(scale_factor=2, mode=‘bilinear’)<br>
B = up(A)<br>
NotImplementedError: Got 5D input, but bilinear mode needs 4D input</p>
</blockquote>
</blockquote>
</blockquote>

sunformoon: <p>Using the PyTorch 0.3.0 as well. Got the same error.<br>
<a class="mention" href="/u/shunsuke_saito">@Shunsuke_Saito</a> Have you figured out why the error comes out here?</p>
<p>Thanks!</p>

sunformoon: <p>Just figured it out.<br>
For 5D, we need the upsampling strategy to be ‘trilinear’ rather than the ‘bilinear’.</p>
<p>Good luck!</p>

kabbas570: <p>Thanks, this is the correct answer.</p>
Faster-RCNN scores

josh_diller: <p>The Pytorch Faster-RCNN model returns bounding boxes, labels and scores.<br>
I’m not sure, are the scores represent how likely is the bounding box to contain an object (objectness score) or classification score?</p>

Aviv_Shamsian: <p>In the forward pass of the RPN module (which is part of FCRNN module) there is an objectness variable that holds the objectness score but it’s not stored as part of the class. I’m not sure if those are the same scores you get when inference the model.<br>
Maybe <a class="mention" href="/u/ptrblck">@ptrblck</a>  can help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> .</p>

soltius: <p>By looking at the code of the PyTorch implementation of Faster RCNN and especially roi_heads.py, we can find the line<br>
<code>        pred_scores = F.softmax(class_logits, -1)</code><br>
in the function <code>postprocess_detections</code> which returns boxes, labels and scores,</p>
<p>which means scores are the classification scores.</p>
LSTM and gradient cliping

Andre_Amaral_IST: <p>Hey,</p>
<p>What is the best configuration for the max_norm of the gradient clipping?<br>
From what I saw people use 1, 3 or 10 usually.<br>
I also saw on a website that for the LSTM language model a max_norm of 0.25 gives better results.<br>
In my case, I am applying an LSTM to time series, what could be the best value?</p>
<p>Regards<br>
André</p>

Andre_Amaral_IST: <p>Anyone could give an help?<br>
Thanks in advance</p>
How to build with custom utility source files?

Sehun0819: <p>I’m struggling with building pytorch, with additional c++ source code. Because it is about code instrumentation, I’m not considering c++ extensions.<br>
So I tried to locate my .h and .cpp files in <code>torch/csrc/utils/</code>, but it seems like additional treatment is needed to build properly with additional source files. If it is true, where is the place that I should add the information about my additional sources?</p>
<p>Current state:</p>
<p>torch/csrc/utils/instrument.h:</p>
<pre><code class="lang-auto">int foo();
</code></pre>
<p>torch/csrc/utils/instrument.cpp:</p>
<pre><code class="lang-auto">#include &lt;torch/csrc/utils/instrument.h&gt;

int foo() {
  return 42;
}
</code></pre>
<p>When <code>foo()</code> is invoked by other (existing) source code, pytorch is built without error, but fails to be imported in python with undefined symbol error.</p>

Sehun0819: <p>Solved.</p>
<p>By entirely reinstalling pytorch,(that is, <code>python setup.py clean</code> followed by <code>python setup.py install</code>) it works. Guess compile commands for new source files are added by cmake, which remains unchanged when only <code>python setup.py install</code> invoked <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>
Backward() of 2 losses (explained)

anonymous007: <p>Hello, this is quite a simple question but I didn’t find an answer.</p>
<p>When you have 2 losses the approach I read is to sum up the losses and then apply .backward()</p>
<p>My question is about understanding this.</p>
<p>To get Loss1 I had some inputs (x1’s)  and the output (y1)<br>
To get Loss2 I had some inputs (x2’s) and the output  (y2)</p>
<p>If I would have Loss1 only then backward() will give me the gradient with respect to my parameters. But note that I used x1’s as input so I will have Z1= wx+b and A1 (Activation) values.</p>
<p>In contrast, for Loss2 I would have Z2 and A2.<br>
Z1,A1,Z2,and A2 are different because different inputs were used.</p>
<p>Now you tell me that I just need to sum up L1+L2 and use backward(). But to compute the gradient of a particular parameter, I will have  da/dZ and/or dZ/dA in the chain (chain rule differentiation). So which one is taken, Z1 or Z2? A1 or A2? How exactly differentiation takes place?</p>
<p>Elsewhere I read that summing up the losses is similar to:<br>
L1.backward()<br>
L2.backward()</p>
<p>Is this correct? If so, this second case looks more clear, because L1 is backpropagated with Z1 and A1, and L2 with Z2 and A2, but then I’m back to the same question. If I manage to get dL1/dW and dL2/dW, how do I update my  parameter…? Would it be:<br>
W = W_prev - learning_rate * dL1/dW - learning_rate * dL2/dW<br>
so in the end I sum up for each gradient obtained separately?</p>
<p>Would you explain me how it goes because I’m confused.<br>
Regards</p>

ptrblck: <p>Assuming both computation graphs are independent (as indicated by ZX, AX) then the gradient will be accumulated and independently calculated as seen here:</p>
<pre><code class="lang-python"># setup
model = nn.Linear(1, 1, bias=False)

x1 = torch.randn(1, 1)
x2 = torch.randn(1, 1)

# sum
out1 = model(x1)
out2 = model(x2)

loss = out1 + out2
loss.backward()

print(model.weight.grad)
# tensor([[0.8689]])

# separate
model.zero_grad()

out1 = model(x1)
out2 = model(x2)

out1.backward()
print(model.weight.grad)
# tensor([[-0.6130]])

out2.backward()
print(model.weight.grad)
# tensor([[0.8689]])
</code></pre>

anonymous007: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>  Thanks a lot I finally understood it. Your example was very clear</p>
Can any one help me in a way to adopt attention mechanism in the the following Residual encoder decoder CNN model please

Tina: <p>‘’’<br>
import torch</p>
<p>import torch.nn as nn</p>
<p>import torch.nn.functional as F</p>
<p>class Generator(nn.Module):</p>
<pre><code>def __init__(self, in_channels=1, out_channels=96, num_layers=10, kernel_size=5, padding=0):

    super(Generator, self).__init__()

    encoder = [

        nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, stride=1, padding=padding)

    ]

    decoder = [

        nn.ConvTranspose2d(out_channels, in_channels, kernel_size=kernel_size, stride=1, padding=padding)

    ]

    for _ in range(num_layers):

        encoder.append(

            nn.Conv2d(out_channels, out_channels, kernel_size=kernel_size, stride=1, padding=padding)

        )

        decoder.append(

            nn.ConvTranspose2d(out_channels, out_channels, kernel_size=kernel_size, stride=1, padding=padding)

        )

    self.encoder = nn.ModuleList(encoder)

    self.decoder = nn.ModuleList(decoder)

    self.__init_weights()

def __init_weights(self):

    for m in self.modules():

        classname = m.__class__.__name__

        if classname.find('Conv') != -1:

            m.weight.data.normal_(0, 0.01)

            if hasattr(m.bias, 'data'):

                m.bias.data.fill_(0)

def forward(self, x: torch.Tensor):

    residuals = []

    for block in self.encoder:

        residuals.append(x)

        x = F.relu(block(x), inplace=True)

    for residual, block in zip(residuals[::-1], self.decoder[::-1]):

        x = F.relu(block(x) + residual, inplace=True)

    return x
</code></pre>
<p>‘’’</p>
What is the correct way to use NaNLabelEncoder?

Rohit: <p>I am training a forecasting model using the TFT implementation from pytorch_forecasting library. I am preparing my dataset using the TimeSeriesDataset class from the same library. The class has a parameter which accepts to provide a dictionary of column name-label encoder mapping for label encoding the categorical variables (categorical_encoders). I am creating a dictionary as below and assigning it to this parameter in the following manner.</p>
<p><code>categorical_encoders = { ..., 'mycatvar' : NaNLabelEncoder(add_nan=True, warn=True), ... }</code></p>
<p><code>dataset = TimeSeriesDataSet(..., categorical_encoders=categorical_encoders, ...)</code></p>
<p>I have two questions:</p>
<ol>
<li>Considering I am preparing my test data separately in a different notebook, how do I ensure that the test data is encoded using the same label mapping?</li>
<li>When should I use <code>NaNLabelEncoder(..).fit()</code> and <code>NaNLabelEncoder(..).transform()</code> ? I am assuming it is only when I use NaNLabelEncoder explicitly, without expecting TimeSeriesDataset to do it for me.</li>
</ol>
Can you take a look at the LibTorch model?

21fa417e3fb06e56040c: <p>Hi everyone , I made a toy project. The project is a node JS web server using Libtorch.<br>
I used a pre-trained model and the classification results are always constant. What could be the problem?</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/kerry-Cho/nodeJS-addon-example/blob/14b9a5702f6ad5d51eed6d0ed04a87303227e12d/src/classifier/classifier.cpp#L69">
  <header class="source">

      <a href="https://github.com/kerry-Cho/nodeJS-addon-example/blob/14b9a5702f6ad5d51eed6d0ed04a87303227e12d/src/classifier/classifier.cpp#L69" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/kerry-Cho/nodeJS-addon-example/blob/14b9a5702f6ad5d51eed6d0ed04a87303227e12d/src/classifier/classifier.cpp#L69" target="_blank" rel="noopener nofollow ugc">kerry-Cho/nodeJS-addon-example/blob/14b9a5702f6ad5d51eed6d0ed04a87303227e12d/src/classifier/classifier.cpp#L69</a></h4>



    <pre class="onebox">      <code class="lang-cpp">
        <ol class="start lines" start="59" style="counter-reset: li-counter 58 ;">
            <li>        torch::Tensor out = torch::softmax(model-&gt;forward(img_tensor), 1);</li>
            <li>
            </li>
<li>        std::tuple&lt;torch::Tensor, torch::Tensor&gt; result = torch::max(out, 1);</li>
            <li>
            </li>
<li>        torch::Tensor prob = std::get&lt;0&gt;(result);</li>
            <li>        torch::Tensor index = std::get&lt;1&gt;(result);</li>
            <li>
            </li>
<li>        auto probability = prob.accessor&lt;float,1&gt;();</li>
            <li>        auto idx = index.accessor&lt;int64_t, 1&gt;();       </li>
            <li>        </li>
            <li class="selected">        return Napi::String::New(env, string_format("{ \"Class\": \"%d\", \"Probability\": \"%.3f\" }",</li>
            <li>                idx[0], probability[0]));</li>
            <li>
            </li>
<li>    } catch( std::exception&amp; ex) {</li>
            <li>        return Napi::String::New(env, ex.what());</li>
            <li>    }</li>
            <li>}</li>
        </ol>
      </code>
    </pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Does PyTorch fall back to non-accelerated code when NNPACK shows "Unsupported hardware"?

KoenTanghe: <p>Context: I inherited a project using Python3 and PyTorch 1.11.0, and I was running this on Ubuntu 20.04 on an (old) desktop machine.</p>
<p>When I run the code, I’m seeing this message (it’s not clear actually if this is an error or a warning):</p>
<blockquote>
<p>[W NNPACK.cpp:51] Could not initialize NNPACK! Reason: Unsupported hardware.</p>
</blockquote>
<p>I checked /proc/cpuinfo and the machine doesn’t support AVX2 (it’s an Intel(R) Core™2 Quad CPU Q8200 @ 2.33GHz), so that could be the reason for this message.</p>
<p>The code does appear to continue to run though, and I also get a nice confusion matrix of the classification task it’s running at the end, which appears reasonable compared to previously reported results with that code.</p>
<p>So, my question is:<br>
<strong>What happens when NNPACK detects there is no AVX2 support (I presume) on the machine it’s running on?</strong><br>
<strong>Does PyTorch fall back to non-accelerated code in that case (so, still running correctly, but slower)?</strong></p>
<p>I asked on the NNPACK GitHub issues board, and there the answer was: <em>“NNPACK doesn’t have a fall-back for unsupported machines. Higher-level frameworks integrating NNPACK may have one.”</em><br>
So, it depends on how PyTorch handles this then.</p>
<p>Thanks for your feedback!</p>
<p>Koen</p>

ptrblck: <p>PyTorch should use a fallback if NNPACK cannot be initialized.<br>
The warning is raised from <a href="https://github.com/pytorch/pytorch/blob/c1cdb1216b97970d903a6d6e9e7d0e2b4ffaef46/aten/src/ATen/native/NNPACK.cpp#L39-L59"><code>init_nnpack</code></a>, which is called by <code>_nnpack_available()</code> and used in <a href="https://github.com/pytorch/pytorch/blob/e011a8e18bf469a6a612fd1e7647159c353730a9/aten/src/ATen/native/Convolution.cpp#L272-L288"><code>use_nnpack</code></a> to select the backend <a href="https://github.com/pytorch/pytorch/blob/e011a8e18bf469a6a612fd1e7647159c353730a9/aten/src/ATen/native/Convolution.cpp#L1154-L1159">here</a> as <code>ConvBackend::NnpackSpatial</code> or <code>ConvBackend::Slow2d</code>.</p>

KoenTanghe: <p>Thank you for looking this up in the source and confirming the fallback mechanism.<br>
This also confirms that the code indeed continues to run (and doesn’t crash) and that this is merely a warning message.</p>
Handle varying data length/Effect of zero padding input

szahan: <p>I am trying to train some video data that has a highly variable sequence length, which could be anything from 500 to 3460 frames. My model includes GCN, CNN, and Linear/GRU layers. I am confused about which is a better approach</p>
<pre><code class="lang-auto">- Zero pad to the maximum frame length (say, 3500). Does it matter if it is in the front or back?
- Repeat frames to maximum frame length
</code></pre>
<p>I am working on action quality assessment so I am assuming repeating is not a good idea as samples with a lower number of frames will have more repeats of the same movement than there actually were which will change the nature of the action. But if I am zero-padding how will it affect the quality prediction. Like some very good quality action could be very small/large in length and so can be very poor ones.</p>
<p>Or</p>
<pre><code class="lang-auto">- Figure out a way to work with mismatched length
</code></pre>
<p>Besides, as you can see sequences are very long, what could be the best course of action to better capture temporal relationships.<br>
Thanks for any suggestions</p>
Process killed when loading cpp extensions

arnoldzhu: <p>I was trying to load a cpp extension with <code>torch.utils.cpp_extension.load()</code> on my Jetson TX2 device with CUDA-10.2.</p>
<pre><code class="lang-auto">from torch.utils.cpp_extension import load
matrix_op_cuda = load(
    'matrix_op_cuda', ['./matrix_op/matrix_op_cuda.cpp', './matrix_op/matrix_op_cuda_kernel.cu'], verbose=True)
</code></pre>
<p>The compilation process with ninja went smoothly, but when loading the extension module, the process was killed and the log showed no more info.</p>
<p>I tried to load the extension again but the same thing happened.</p>
<pre><code class="lang-auto">Building extension module matrix_op_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module matrix_op_cuda...
Killed
</code></pre>
<p>Wonder what could be the problem?</p>

ptrblck: <p><code>Killed</code> is usually raised if the OS sends a <code>SIGKILL</code> to the process if you are running out of memory. Based on the output it seems that the compilation succeeds but the loading of the kernel is raising the error? Try to check the memory usage during the call and see if you are indeed running OOM.</p>
How could I share dataset among processes?

coincheung: <p>Hi,</p>
<p>I use distributed training mode to train my model.</p>
<p>My dataset is not large so I can load it to memory such that I do not need to read them from disk and decode them.</p>
<p>I found this example: <a href="https://github.com/ptrblck/pytorch_misc/blob/master/shared_array.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch_misc/shared_array.py at master · ptrblck/pytorch_misc · GitHub</a></p>
<p>It is inspiring but it only supports single gpu training rather than ddp. Besides, this only supports the senario where all input images have identical sizes. What if my dataset contains images of various sizes?</p>

ptrblck: <aside class="quote no-group" data-username="coincheung" data-post="1" data-topic="151184">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ad7895/48.png" class="avatar"> coincheung:</div>
<blockquote>
<p>What if my dataset contains images of various sizes?</p>
</blockquote>
</aside>
<p>Maybe check the <a href="https://github.com/ptrblck/pytorch_misc/blob/master/shared_dict.py"><code>shared_dict</code></a> example to add the tensors into a <code>dict</code> which would support various shapes.</p>

coincheung: <p>Hi,</p>
<p>Thanks for replying!!! Does this support sharing amoung different gpu? I mean I am not only using multi-worker for dataloader but also using distributed training mode. Can I simply keeping only one piece of copy in the memory for all processes from dataloader and gpus?</p>

ptrblck: <p>I don’t know as I haven’t tried this use case, but usually you would use a <code>DistributedSampler</code> in a DDP setting which would make sure that each process loads only the corresponding chunk of the data used on the corresponding device.</p>
Saving feature extractor model

Ambrozy: <p>When I try to do this:</p>
<pre><code class="lang-auto">effnet = models.efficientnet_b0(pretrained=True)
effnet = create_feature_extractor(effnet, return_nodes=['features.7.0.block.3.1'])
torch.save(effnet, 'effnet.pth')
</code></pre>
<p>I get <code>RecursionError: maximum recursion depth exceeded</code></p>
<p>It’s a bug or I shouldn’t do that?</p>

ptrblck: <p>I don’t know if saving a feature extractor is supported, but would assume to see a proper error message instead of a <code>RecursionError</code>. Could you create an issue on GitHub so that we could track it there, please?</p>
Train/val/test transforms with torchvision datasets

danielkorth: <p>Hey, so I was wondering what the best way is to apply separate train/test transforms on a torchvision dataset. What I am trying to do is have separate transforms for training, validation and test set (where validation and test transforms is similar)</p>
<p>I currently have everything setup like this (oversimplified):</p>
<pre><code class="lang-auto">train_transforms = T.Compose([
     T.RandomCrop(32, padding=4),
     T.RandomHorizontalFlip(),
     T.ToTensor(),
     T.Normalize(mean, std)
])
test_transforms = T.Compose([
      T.ToTensor(),
      T.Normalize(mean, std)
])


train_dataset = CIFAR10(root=PATH, train=True, transform=train_transforms)

targets_ = rain_dataset.targets
train_idx, val_idx = sklearn.model_selection.train_test_split(np.arange(len(targets_)), test_size=0.2, stratify=targets_)
train_sampler = SubsetRandomSampler(train_idx)
val_sampler = SubsetRandomSampler(val_idx)

test_dataset = CIFAR10(root=PATH, train=False, transform=test_transforms)

train_loader = DataLoader(train_dataset, sampler=train_sampler)
val_loader = DataLoader(train_dataset, sampler=val_sampler)
test_loader = DataLoader(test_dataset)
</code></pre>
<p>However, doing it this way does also apply the train_transforms to the val_loader, but for validation I would like to not have any augmentation (i.e. I want <code>test_transforms</code> for <code>val_loader</code>)<br>
Is there any straightforward way to apply another transforms to the validation loader?<br>
Like creating a separate Dataset that only contains validation data?</p>

ptrblck: <p>You could create another <code>CUFAR10</code> dataset using the <code>val_transform</code> and pass it to the <code>DataLoader</code> with the <code>val_sampler</code> to create <code>val_loader</code>.</p>
Enabling cuDNN slows down training process

zachnguyen03: <p>Hi! I am trying to train a model using Pytorch and I found that enabling cuDNN would speed up the training process. However, training got significantly slowed down instead. Can anyone help me with this issue? I performed some minor tests and got this result:</p>
<pre><code class="lang-auto">Enable Cudnn
Conv:  0.5449938774108887
Conv backward:  0.1148219108581543
---------------------------
Disable Cudnn
Conv:  0.01213216781616211
Conv backward:  0.13735365867614746
---------------------------
</code></pre>
<p>Here is the code snippet that I used for testing:</p>
<pre><code class="lang-python">import torch
import time
def run():
    in_c = 10
    out_c = 15
    kernel = 3
    padding = 1
    inp = torch.rand(512, in_c, 128, 128, requires_grad=True).cuda()
    conv = torch.nn.Conv2d(in_c, out_c, kernel, padding=padding, bias=False).cuda()
    torch.cuda.synchronize()
    start = time.time()
    out = conv(inp)
    torch.cuda.synchronize()
    print("Conv: ", time.time() - start)
    start = time.time()
    out.sum().backward()
    torch.cuda.synchronize()
    print("Conv backward: ", time.time() - start)
    print('---------------------------')
if __name__=='__main__':
    torch.backends.cudnn.enabled=True
    print('Enable Cudnn')
    run()
    torch.backends.cudnn.enabled=False
    print('Disable Cudnn')
    run()
</code></pre>

ptrblck: <p>I cannot reproduce the issue using warmup iterations and a mean over multiple profiling iterations.<br>
I would also recommend to use <code>torch.utils.benchmark</code> to profile CUDA workloads, as it would add warmup iterations, add synchronizations, and execute the workload until a time threshold is met.</p>
<p>Results in a 3090 using the latest nightly:</p>
<pre><code class="lang-python">Enable Cudnn
Conv:  0.35583066940307617
Conv backward:  14.73327088356018
---------------------------
Disable Cudnn
Conv:  1.5492074489593506
Conv backward:  16.762986421585083
---------------------------
</code></pre>
Multiple Linear Regression results in Nan even when reducing LR and increasing epochs

Cpugneandenoro2021: <p>Hello, I am new to machine learning and pytorch so I am starting with linear regression.</p>
<p>This is my code:</p>
<pre><code class="lang-auto">class LinearRegressionModel(torch.nn.Module):
    
    def __init__(self):
        super(LinearRegressionModel, self).__init__()
        self.linear = torch.nn.Linear(4, 1)  # four input and one output
    
    def forward(self, x):
        y_pred = self.linear(x)
        return y_pred
    
our_model = LinearRegressionModel()

#changed size_average = False to reduction = 'sum' because of depracated value
criterion = torch.nn.MSELoss(reduction = 'sum')
optimizer = torch.optim.SGD(our_model.parameters(), lr = 0.01)


#running into issue with data being either double or float
#converted all data into float
for epoch in range(500):
 
    pred_y = our_model(X)
    
    loss = criterion(pred_y, ynew)
    
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print('epoch {}, loss {}'.format(epoch, loss.item()))
</code></pre>
<p>I keep getting returned Nan even when the LR is reduced to increased or when the number of epochs is increased.</p>

ptrblck: <p>You could check the <code>.grad</code> attributes of the parameters and make see if they are blowing up.<br>
If so, reduce the learning rate further and use a <code>mean</code> reduction in the criterion.</p>
DDP + fp16 + gradient accumulation

Zeeky: <p>Hi,<br>
I face a problem related to ddp + fp16 + gradient accumulation.<br>
The following is my code. <code>train_loader</code> spit a mini-batch each time and I use gradient accumulation to reach the actual batch size.<br>
The problem is that the model trained using this DDP training script is worst than the model trained by a single GPU. Any suggestions?</p>
<p>Code for training:</p>
<pre><code class="lang-auto">for epoch in range(num_train_epochs):
    for data, labels in tqdm(train_loader, desc=f'training-{epoch}'):
        steps += 1
        if steps % gradient_accumulation_steps == 0:
            with amp.context():
                output = model(data)
                loss = loss_fct(output, labels)
            amp.backward(loss)
            amp.step(model, optim)
        else:
            with model.no_sync():
                with amp.context():
                    output = model(data)
                    loss = loss_fct(output, labels)
                amp.backward(loss)
        train_total_loss += loss.item()
</code></pre>

mrshenli: <p>Usually, when switch from local training to distributed training, you might need to re-tune things like learning rate and other hyper-parameters, because 1) the global batch size might be different after introducing more data parallel processes 2) the gradient averaging method used by DDP might or might not offer mathematical equivalence to local training; it depends on the loss function.</p>
<p>Regarding the difference in model quality, does anything change if you disable AMP?</p>
<p>cc <a class="mention" href="/u/mcarilli">@mcarilli</a> <a class="mention" href="/u/ngimel">@ngimel</a></p>

Zeeky: <p>Yes, when I disable fp16 on single GPU training, the model performs better in the first few logging steps. Eventually, with or without fp16, there is no significant difference. But the speed is very slow without fp16.</p>

ptrblck: <aside class="quote no-group" data-username="Zeeky" data-post="3" data-topic="150981">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zeeky/48/13133_2.png" class="avatar"> Zeeky:</div>
<blockquote>
<p>Yes, when I disable fp16 on single GPU training, the model performs better in the first few logging steps. Eventually, with or without fp16, there is no significant difference.</p>
</blockquote>
</aside>
<p>I’m not sure if I understand this description correctly, but it seems you are hitting numerical differences early in the training, but the training converges fine?<br>
Also, which <code>amp</code> implementation are you using, as <code>amp.context()</code> and <code>amp.backward()</code> etc. are neither defined in the native amp util. via <code>torch.cuda.amp</code> nor in the deprecated <code>apex.amp</code> implementation.</p>
Why doesn't consider FloatTensor for embedding layer?

FaezBarghasa: <p>We’ve created a model for predicting time based data (our data type is float). Beside that, we’ve used scaler for normalising parameters. We want to use the embedding layer for tracking similarities. But, we got this error:</p>
<pre><code class="lang-auto">return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.FloatTensor instead (while checking arguments for embedding)

</code></pre>
<p>We got that it only accepts long (embedding layer only accepts long). Please help me. Or, prepare a solution in further versions for users who their input is type of float.</p>

ptrblck: <p>Embedding layers are treated as “lookup” layers which are indexed by the input and thus require integer types. Could you explain your use case a bit more and how floating point types should work in embeddings?</p>
NNAPI Serializer & PixelShuffle

thelytch: <p>PyTorch Version: 1.8.1+cu102</p>
<p>Using this tutorial for NNAPI (<a href="https://pytorch.org/tutorials/prototype/nnapi_mobilenetv2.html" class="inline-onebox" rel="noopener nofollow ugc">(Prototype) Convert MobileNetV2 to NNAPI — PyTorch Tutorials 1.8.1+cu102 documentation</a>), I’m trying to convert my QAT trained model with qnnpack backend, using <code>torch.backends._nnapi.prepare.convert_model_to_nnapi(traced, input_tensor)</code></p>
<p>However in my model I make use of <code>torch.nn.functional.pixel_shuffle</code>, and when I run the conversion function I receive the following error:</p>
<pre><code class="lang-auto">File ".../pytorch/lib/python3.8/site-packages/torch/backends/_nnapi/serializer.py", line 641, in add_node                                                                                                                             
    raise Exception("Unsupported node kind (%r) in node %r" % (node.kind(), node))                                                                                                                                                                                                                             
Exception: Unsupported node kind ('aten::pixel_shuffle') in node %99 : Tensor = aten::pixel_shuffle(%x_hat, %98)
</code></pre>
<p>It seems like <code>PixelShuffle</code> is not supported; is that correct?</p>
<p>However I can see here <code>DEPTH_TO_SPACE</code> and <code>SPACE_TO_DEPTH</code> are NNAPI operation codes: <a href="https://github.com/pytorch/pytorch/blob/cf7d56d8f223d4b10a624a5f7af886a2b67f0955/torch/backends/_nnapi/serializer.py#L44" class="inline-onebox" rel="noopener nofollow ugc">pytorch/serializer.py at cf7d56d8f223d4b10a624a5f7af886a2b67f0955 · pytorch/pytorch · GitHub</a></p>
<p>Is there any way to use <code>PixelShuffle</code> like behaviour with the NNAPI?</p>

axitkhurana: <p>Apologies PixelShuffle is not currently supported by the NNAPI converter.</p>
<p>You can see list of supported ops at this time here: <a href="https://github.com/pytorch/pytorch/blob/master/torch/backends/_nnapi/serializer.py#L726" class="inline-onebox" rel="noopener nofollow ugc">pytorch/serializer.py at master · pytorch/pytorch · GitHub</a></p>
<p>We’re adding more ops soon!</p>

reintseri: <p>Waking up this question 1 year later. Is it so that PixelShuffle is still unsupported by the NNAPI converter? I also tried to replace PixelShuffle with an implementation using .view operator but I guess that operator is also unsupported? Is there any way to bypass this?</p>

axitkhurana: <p>Hi <a class="mention" href="/u/reintseri">@reintseri</a>, apologies, it is still unsupported. We’re open to contributions to the serializer if you’d like to help add it.</p>
<p>Thanks!</p>
How to transfer the pretrained weights for a standard ResNet50 to a 4-channel

snoow: <p>In Remote sensing, the image usually has more than three channels. For example, the image has NIR ,R ,G and B. I want to leveraged on the pretrained weights for a standard ResNet50 and transferred them to a 4-channel input version by copying RGB weights + the NIR weight as equivalent to the red channel.How to solve it?</p>

ptrblck: <p>You could replace the first conv layer with a new one using 4 input channels:</p>
<pre><code class="lang-python">model = models.resnet50(pretrained=True)
weight = model.conv1.weight.clone()
model.conv1 = nn.Conv2d(4, 64, kernel_size=7, stride=2, padding=3, bias=False)
with torch.no_grad():
    model.conv1.weight[:, :3] = weight
    model.conv1.weight[:, 3] = model.conv1.weight[:, 0]
    
x = torch.randn(10, 4, 224, 224)
output = model(x)
</code></pre>

snoow: <p>Thank you for your suggestion.it works.</p>

mohammed_guermal: <p>HI,</p>
<p>why the use of torch.no_grad(), and does affect the training if i want to fine tune my model?</p>
<p>Thanks</p>

Hasith_Karunasekera: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> why did you use <em>with torch.no_grad()</em> ? Is it because here you use if for inference ? If I want to re-train the network after initializing the weights like this, can I just assign the weight without using  <em>with torch.no_grad()</em> ?</p>

Hasith_Karunasekera: <p>I think it is to avoid a runtime error.<br>
RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation</p>

ptrblck: <p>Yes, you would need to warp the assignment into the <code>no_grad()</code> guard to avoid Autograd tracking this operation.</p>
Convert a tuple into tensor

costa98: <p>Hello, I have a NN that has a gru and a resnet. The problem is that the output of the gru is a &lt;class ‘tuple’&gt; but the resnet needs a tensor as an input, how can I overcome this problem?<br>
Thanks</p>

chetan06: <p>Use numpy…asarray(tuple) to convert it to a numpy array and then convert it into tensor</p>

Deeply: <p>Sorry to open this again, but this might be useful to some people. Going to <code>numpy</code> and then back to tensor might not be a good idea, especially if the tensor is placed on the GPU. Assuming the tuple is called xx, here’s how I did it when I bumped into it today:</p>
<pre><code class="lang-auto">xx  = torch.stack(list(xx), dim=0)
</code></pre>
<p>Writing this as a function:</p>
<pre><code class="lang-auto">def tuple_of_tensors_to_tensor(tuple_of_tensors):
    return  torch.stack(list(tuple_of_tensors), dim=0)
</code></pre>

dato_nefaridze: <p>what if those arrays in tuples have different dimensions? like 2x5 and 5x7 ?</p>

mayuimathurlens: <p>I had the same issue after using gru, the tuple contains tensor, you can access the tensor simply by indexing the tuple as well.</p>

shahidedu7: <p>Just use torch.tensor(tuple) to convert tuple to PyTorch Tensor.</p>

HashRocketSyntax: <p>because that returns<br>
<code>ValueError: only one element tensors can be converted to Python scalars</code></p>

HashRocketSyntax: <p>fyi this requires all tensors to be the same shape,<br>
<code>RuntimeError: stack expects each tensor to be equal size, but got [5, 4] at entry 0 and [2, 4] at entry 20</code></p>
<p>mine weren’t because the final batch size is a remainder</p>

Deeply: <p>fmi, I would convert all except the last one (the remainder), then, append the last element. It would be nice to tweak the above function to handle this case.</p>

AlphaBetaGamma96: <p>Doing <code>torch.tensor(tuple)</code> will break the flow of gradients as you’re re-wrapping your Tuple object. The best way to convert from a tuple to a tensor is to use the <code>torch.stack</code> or <code>torch.cat</code>. Although as previously mentioned in this thread, all the tensors must be of ‘equal size’. So, it’s something you’ll have to implement on a case-by-case basis.</p>
Cuda kernel image error

MED-YAHYAOUI: <p>i have torch ‘1.11.0+cu102’ with cuda version 10.2<br>
and i have NVIDIA GeForce RTX 3060 Ti<br>
nvidia driver is 470.74</p>
<p>here is the Nvidia-Smi result<br>
NVIDIA-SMI 470.74       Driver Version: 470.74       CUDA Version: 11.4<br>
GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
MIG M. |<br>
|===============================+======================+===== |<br>
|   0  NVIDIA GeForce …  Off  | 00000000:0B:00.0 Off |                  N/A |<br>
|  0%   48C    P0     1W / 240W |      0MiB /  7982MiB |      0%      Default |<br>
|                               |                      |                  N/A |<br>
±------------------------------±---------------------±---------------------+</p>
<p>and i am trung to run the training of pix2pix model,<br>
and i get this error message:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/094b82be43dd84097177a7ff6a56136f908ae033.png" data-download-href="https://discuss.pytorch.org/uploads/default/094b82be43dd84097177a7ff6a56136f908ae033" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/094b82be43dd84097177a7ff6a56136f908ae033.png" alt="image" data-base62-sha1="1ke4QAnjUYMG3B0Xb4Zy7ngGsKf" width="690" height="235" data-dominant-color="EEE8D6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1269×433 29.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>alternatively i have the same envirement at collab or Kaggle and it works verry well,</p>
<p>how can i make it work on my card please?<br>
PS: im working on vertual machine connected using ssh so i don’t have access to the UI of systeme,<br>
i have centos as OS.</p>

InnovArul: <p>Can you trying installing torch with cuda 11 and see if it solves your issue? If I remember correctly from one of <a class="mention" href="/u/ptrblck">@ptrblck</a>’s answers,  this could be the solution for the issue you are having.</p>

MED-YAHYAOUI: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  <a class="mention" href="/u/innovarul">@InnovArul</a> thnx it workt now</p>
Anyone knows about the exectly spline type used in "F.grid_sample" with "bicubic" mode?

MSKTHU: <p>To my knowledge, there are B-spline and catmull-rom spline which can be used for bicubic sampling. My question is which type of spline excetly used in the F.grid_sample API with “bicubic” mode?</p>

Matias_Vasquez: <p>In the <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.grid_sample.html" rel="noopener nofollow ugc">documentation</a> for <code>grid_sample</code> there is a note at then end saying how <code>bicubic</code> is implemented.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8cab97acb2172b5e420efc7d59d01f171ce87be.png" data-download-href="https://discuss.pytorch.org/uploads/default/a8cab97acb2172b5e420efc7d59d01f171ce87be" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8cab97acb2172b5e420efc7d59d01f171ce87be.png" alt="image" data-base62-sha1="o5cy7uSvFIucxi5XPOHdtMojbc2" width="690" height="160" data-dominant-color="CAD8DF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">803×187 21.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>They also link to <a href="https://en.wikipedia.org/wiki/Bicubic_interpolation#Bicubic_convolution_algorithm" rel="noopener nofollow ugc">this wikipedia entry</a>, where <strong>bicubic convolution algorithm</strong> is explained.</p>
<p>I do not know much about this topic, but in <a href="https://ijarcce.com/wp-content/uploads/2016/02/IJARCCE-7.pdf" rel="noopener nofollow ugc">this paper</a> they compare these methods and it seems that they are different (meaning it is neither of the methods you mentioned, but another).</p>
<p>I hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

MSKTHU: <p>Yeah, you are right. I checked the links and reproduced the result. Thank you. In my point of view, bicubic refers to all the interpolation method that use cubic splines. Then from the link I realize that BICUBIC is also a specific method with varying alpha.</p>
Positional Encoding of Time-Series features

Ouilliam: <p>Hello,</p>
<p>I’m trying to use a Transformer Encoder I coded with weather feature vectors which are basically 11 features about the weather.<br>
I have a data point per day, so this is a time-series but there are no evidences of the time in my vectors and I know I must use positional encoding to tell my network “this vector is from the 21/05/2021, this one is from the 20/04/2019, etc”.</p>
<p>The usual positional encoding with sin &amp; cos used in NLP doesn’t seem to fit my problem as it encodes the position relative to other words in the sentence and my features are independant values (the temperature of the day doesn’t come after the amount of rain for instance).</p>
<p>I don’t really know how to do encode my feature vectors for my transformer encoder to get something like <code>positional_encoded = original_vector + date_encoding</code> which I think could be great.</p>
<p>What would be the best way to do so ?</p>
<p>Thanks a lot for your answers and help!</p>
Google Summer of Code opportunity with BeagleBoard and Bela

victor-shepardson: <p>BeagleBoard are looking for students interested in bringing PyTorch to the Bela (<a href="https://bela.io/" rel="noopener nofollow ugc">https://bela.io/</a>) embedded platform for musical and interactive computing as part of this year’s Google Summer of Code.</p>
<blockquote>
<p>Bela is an embedded computing platform for creating beautifully responsive interactive projects. Bela provides ultra-low latency, high quality audio, analog and digital I/O in a tiny self-contained package that can be easily embedded into a huge range of applications. Built on the <a href="https://beagleboard.org/" rel="noopener nofollow ugc">BeagleBone</a> family of open-source embedded computers, Bela combines the processing power of an embedded computer with the timing precision and connectivity of a microcontroller.</p>
</blockquote>
<blockquote>
<p>This project would involve developing tools and workflows for combining PyTorch with Bela, beginning with streamlining the compilation and distribution of <code>libtorch</code> for Bela, and investigating its performance through creating benchmarks and comparisons. Further objectives would then include creating a small examples library including Bela web GUIs, and creating project templates both in Python and for Bela.</p>
</blockquote>
<p>more information:</p>
<p><a href="https://elinux.org/BeagleBoard/GSoC/Ideas-2022#Running_Machine_Learning_Models_on_Bela_via_PyTorch" class="onebox" target="_blank" rel="noopener nofollow ugc">https://elinux.org/BeagleBoard/GSoC/Ideas-2022#Running_Machine_Learning_Models_on_Bela_via_PyTorch</a></p>

copythatpasta: <p>Not a student, but challenge accepted. Just bought the Bela starter kit</p>

peacheym: <p>I’m a current graduate student working with Bela and would love to have libtorch running properly on that system. I’ve made several attempts at installation via different mechanisms, but will follow this thread to hopefully see some successful updates!</p>