akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Memory problem with bottleneck profiler

fiorenza2: <p>Hi there,</p>
<p>I have been trying to profile my code using <code>torch.utils.bottleneck</code> and received the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code                                                                                  
    exec(code, run_globals)
  File "/home/phil/.envs/rp1env/lib/python3.6/site-packages/torch/utils/bottleneck/__main__.py", line 231, in &lt;module&gt;
    main()
  File "/home/phil/.envs/rp1env/lib/python3.6/site-packages/torch/utils/bottleneck/__main__.py", line 210, in main
    autograd_prof_cpu, autograd_prof_cuda = run_autograd_prof(code, globs)
  File "/home/phil/.envs/rp1env/lib/python3.6/site-packages/torch/utils/bottleneck/__main__.py", line 102, in run_autograd_prof
    result = [run_prof(use_cuda=False)]
  File "/home/phil/.envs/rp1env/lib/python3.6/site-packages/torch/utils/bottleneck/__main__.py", line 98, in run_prof
    exec(code, globs, None)
  File "/home/phil/.envs/rp1env/lib/python3.6/site-packages/torch/autograd/profiler.py", line 262, in __exit__
    records = torch.autograd._disable_profiler()
MemoryError: std::bad_alloc
</code></pre>
<p>Furthermore, it appears to run my script two times (I imagine for CUDA and CPU respectively), and upon completing the second run (successfully) this error gets printed.</p>

ptrblck: <p>Could you post a minimal code snippet to reproduce this error, please?</p>

fiorenza2: <p>Turns out that the code I was debugging was simply too complex for the profiler; I scaled it back considerably and it works fine.</p>

RylanSchaeffer: <p>I have this problem. <a class="mention" href="/u/fiorenza2">@fiorenza2</a> , what do you mean the code was too complex for the profiler? Could you clarify what you meant “scaled it back”?</p>

fiorenza2: <p>Hi there,</p>
<p>It’s been a while, but basically the code was loading very large tensors into memory (I believe this was model-based RL, so very large trajectory data). What believe I did was create a scaled down version which used much smaller hyperparameters (e.g., rollout horizon) that would reduce the memory usage. This meant that I was able to get meaningful breakdowns in where time was being taken up.</p>
_MultiProcessingDataLoaderIter error when num_workers>0 in Debug mode

aiboys: <p><strong>’_MultiProcessingDataLoaderIter’ erro when num_workers&gt;0</strong></p>
<p>environment： remote Linux+ Pycharm IDE + PyTorch 1.8.0</p>
<p>When I tried to debug my code with Pycharm IDE, there was a bug showing that <code>_MultiProcessingDataLoaderIter</code> error.<br>
Then I found that when I set ‘num_workers_per_gpu’ &gt; 0, such bug would happen.<br>
There is no bug in my code, cause when I directly ran it, there is no any error.<br>
<strong>Such bug just happened  when only Debug mode. And when I set num_workers_per_gpu =0, there was no bug in any mode.</strong> It looks like so strange:<br>
(1) num_workers_per_gpu =0,<br>
debug mode <img src="https://discuss.pytorch.org/images/emoji/apple/white_check_mark.png?v=10" title=":white_check_mark:" class="emoji" alt=":white_check_mark:"><br>
run mode: <img src="https://discuss.pytorch.org/images/emoji/apple/white_check_mark.png?v=10" title=":white_check_mark:" class="emoji" alt=":white_check_mark:"><br>
(2) num_workers_per_gpu &gt;0,<br>
debug mode: <img src="https://discuss.pytorch.org/images/emoji/apple/x.png?v=10" title=":x:" class="emoji" alt=":x:"><br>
run mode: <img src="https://discuss.pytorch.org/images/emoji/apple/white_check_mark.png?v=10" title=":white_check_mark:" class="emoji" alt=":white_check_mark:"></p>
<p>It seems  when Debug mode, the following loop would break uncorrectly (the condition of while loop here would set false):</p>
<pre><code class="lang-auto">    while self._rcvd_idx &lt; self._send_idx:
                info = self._task_info[self._rcvd_idx]
                worker_id = info[0]
                if len(info) == 2 or self._workers_status[worker_id]:  # has data or is still active
                    break
                del self._task_info[self._rcvd_idx]
                self._rcvd_idx += 1
    else:
                # no valid `self._rcvd_idx` is found (i.e., didn't break)
                if not self._persistent_workers:
                    self._shutdown_workers()
                raise StopIteration
</code></pre>
<p>I do not know how to solve it. And I checked torch.batchSampler, the reason that conditon becomes false could be that my sampler catches no data yet. But it just my own guess.</p>

gcramer23: <p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> for data loader questions</p>

VitalyFedyunin: <p>Hi! Do you have actual error trace?</p>
[Beginner] Error implementing FC layer after Dropout layer

machina: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb0e12e2aeb2fc99de552f14bd4788fccc9b8914.png" alt="image" data-base62-sha1="qGLrgW8okZyotrdHv43k6BLkGpK" width="221" height="226"><br>
I tried implementing the above model word for word like this:</p>
<pre><code class="lang-auto">class TargetA(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(64, 5, 5)
        self.conv2 = nn.Conv2d(64, 5, 5)
        self.drop1 = nn.Dropout(0.25)
        self.fc1 = nn.Linear(128)
        self.drop2 = nn.Dropout(0.5)
        self.fc2 = nn.Linear()

    def forward(self, x):
        x = self.conv1(x)
        x = nn.ReLU(x)
        x = self.conv2(x)
        x = nn.ReLU(x)
        x = nn.Dropout(0.25)(x)
        x = self.fc1(x)
        x = nn.ReLU(x)
        x = nn.Dropout(0.5)(x)
        x = self.fc2(x)
        x = nn.Softmax(x)
        return x

</code></pre>
<p>line 51, in <strong>init</strong><br>
self.fc1 = nn.Linear()<br>
TypeError: <strong>init</strong>() missing 2 required positional arguments: ‘in_features’ and ‘out_features’</p>
<p>It doesn’t like how I’m doing the Linear layers.  Any help on how to implement the above photo?</p>

JuanFMontesinos: <p>Hi, the linear layer requires you to specify how many channels do you need as input and as output.<br>
In your case the convolutional layers are wrong.<br>
You need to set<br>
<code>self.conv1= nn.Conv2d(in_channels,out_channels,kernel_size)</code><br>
Therefore the output of conv1 will have 5 channels which mismatch the input of conv2.</p>
<p>The fully connected layer (or linear layer)<br>
requires as input<br>
<code>Linear(in_channels,out_channels)</code><br>
In channels should match conv2’s out channels</p>

machina: <p>Thanks for the response,  how would you interpret the first two layers based on the top picture? Is 64 the input and output size? is 5x5 the kernal size?<br>
My guess so far would be:</p>
<pre><code class="lang-auto">class TargetA(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(? , ?, ?)
        self.conv2 = nn.Conv2d(?, ?, ?)
        self.drop1 = nn.Dropout(0.25)
        self.fc1 = nn.Linear(,128)
        self.drop2 = nn.Dropout(0.5)
        self.fc2 = nn.Linear(128,10)
</code></pre>

JuanFMontesinos: <p>Well, from the picture I would say the kernel size is 5x5.<br>
The rest is not really defined. The 2nd convolution should be 64,128 unless they concatanate anything.<br>
The first one should be something to 64</p>
Is MPI_THREAD_MULTIPLE compatible with pytorch.distributed?

Pengfei_Xu: <p>I wonder if there is a particular reason why the MPI initialize of pytorch.distributed only support MPI_THREAD_SERIALIZED rather than MPI_THREAD_MULTIPLE?</p>
<p>I tried to modify it to MPI_THREAD_MULTIPLE and can successfully build pytorch from source. Are there particular cases where MPI_THREAD_MULTIPLE fail for pytorch?</p>
PytorchVideo - AvgPool3d has reduced the sequence length (From 16 to 13)

jpainam: <p>Hi, I’m working with the new library PytorchVideo. And I’m using ResNet3D<br>
My input sequence for a single batch size is <code>torch.Size([1, 3, 16, 224, 224])</code> and I was expecting to get after the average pool layer <code>torch.Size([1, 2048, 16, 1, 1])</code> but I got <code>torch.Size([1, 2048, 13, 1, 1])</code>. I can’t figure out how the sequence length changed from <code>16</code> to <code>13</code>.</p>
<p>Any helped?<br>
I verified the inputs of that layer is <code>torch.Size([1, 2048, 16, 7, 7])</code> which is expected and</p>
<pre><code class="lang-python">(5): ResNetBasicHead(
        (pool): AvgPool3d(kernel_size=(4, 7, 7), stride=(1, 1, 1), padding=(0, 0, 0))
        (dropout): Dropout(p=0.5, inplace=False)
        (proj): Linear(in_features=2048, out_features=25, bias=True)
        (output_pool): AdaptiveAvgPool3d(output_size=1)
      )
</code></pre>

ptrblck: <p>I’m not sure which pooling layer you are referring to, but neither would change the number of channels.<br>
Also, your current module would raise a shape mismatch, so I assume the <code>forward</code> method reshapes the activation etc.<br>
Using the layers manually yield the expected shapes:</p>
<pre><code class="lang-python">x = torch.randn(1, 3, 16, 224, 224)
pool = nn.AvgPool3d(kernel_size=(4, 7, 7), stride=(1, 1, 1), padding=(0, 0, 0))
out = pool(x)
print(out.shape)
&gt; torch.Size([1, 3, 13, 218, 218])

lin = nn.Linear(in_features=2048, out_features=25, bias=True)
out = lin(out)
&gt; RuntimeError: mat1 and mat2 shapes cannot be multiplied (8502x218 and 2048x25)

adaptive_pool = nn.AdaptiveAvgPool3d(output_size=1)
out = adaptive_pool(x)
print(out.shape)
&gt; torch.Size([1, 3, 1, 1, 1])
</code></pre>

jpainam: <p><code>(1, 3, 16, 224, 224)</code> is the input to the model. for the Avg pool, I have</p>
<pre><code class="lang-python">x = torch.randn(1, 2048, 16, 7, 7)
pool = nn.AvgPool3d(kernel_size=(4, 7, 7), stride=(1, 1, 1), padding=(0, 0, 0))
out = pool(x)
print(out.shape)
&gt; torch.Size([1, 2048, 13, 1, 1])
</code></pre>
<p>I was expecting the same sequence length <code>16</code> and not <code>13</code>. My understanding of pooling (2D or 3D) is that it only works on the spatial dimension.</p>

ptrblck: <aside class="quote no-group" data-username="jpainam" data-post="3" data-topic="130804">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpainam/48/5870_2.png" class="avatar"> jpainam:</div>
<blockquote>
<p>My understanding of pooling (2D or 3D) is that it only works on the spatial dimension.</p>
</blockquote>
</aside>
<p>No, that’s not the case and is the difference between the 2D and 3D layers.<br>
While the 2D layers apply the kernel on the <code>H</code> and <code>W</code> dimension for an input of <code>[N, C, H, W]</code>, 3D layers will use a 3D kernel applied on <code>D</code>, <code>H</code>, <code>W</code> for an input of <code>[N, C, D, H, W]</code>.</p>
Namescope of parameters in pytorch

Hengck: <p>is there a way that we can control the name of the parameters?<br>
I suppose this is known as the key of the dict_state in pytorch?</p>
<p>in tensorflow, it is done by:</p>
<pre><code class="lang-auto">with tf.name_scope("my_namescope"): 
    v1 = tf.Variable(1, name="var1", dtype=tf.float32)
    ...

print(v1.name)  #"my_namescope/var1:0
</code></pre>

fmassa: <p>There are not yet scopes in pytorch, so you can’t attach names to Variables.<br>
There is one exception though: inside a <code>nn.Module</code>, you can retrieve a name of Variables assigned to the module via it’s name (this is what you get from <code>model.named_parameters()</code>.</p>

rslz: <p>Is there any news on this? This feature would be really nice in order to create orderly graphs in tensorboard. The autogenerated graphs can be really, really messy.</p>
L1 loss / reconstruction loss turns 0

banikr: <p>I am using simple L1 loss for a channel for 3D volumetric segmentation. The target(one-hot encoded) size is:<br>
<code>2, 9, 64, 96, 96</code><br>
I am using the following codes for that:<br>
(<code>identify_axis</code> function returns <code>[2, 3, 4]</code> for 3D images.)</p>
<pre><code class="lang-auto">def channelwiseL1(pred, trgt):
    axis = identify_axis(pred.shape) 
    pred = torch.mean(torch.mean(pred, axis), 0)
    trgt = torch.mean(torch.mean(trgt, axis), 0)
    # return torch.mean(torch.mean(torch.abs(pred, trgt), axis), 0)
    return abs(pred - trgt)[7]
</code></pre>
<p>I wanted to implement the loss only on channel 7.<br>
Unfortunately, the L1 loss turns 9 after 3/4 epochs.<br>
I am using the following optimizer:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, betas=(0.9, 0.999))
</code></pre>
Can someone recommend for a tutorial about speech recognition in pytorch?

hananelroe: <p>I made a neural network with the urban sound dataset according to a tutorial, but now I wanna create my own dataset and network, which will recognize a wake word for sound assistant (hopefully). where should I start?</p>

tom: <p>There is a voice activity detection example in torchaudio. I used that for inspiration when doing my raspberry lightbulb control.</p>
<p>Best regards</p>
<p>Thomas</p>

hananelroe: <aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/hub/snakers4_silero-vad_vad/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/hub/snakers4_silero-vad_vad/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://pytorch.org/hub/snakers4_silero-vad_vad/" target="_blank" rel="noopener nofollow ugc">PyTorch</a></h3>

  <p>An open source machine learning framework that accelerates the path from research prototyping to production deployment.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Is it this one?</p>

tom: <p>I had been thinking of <a href="https://github.com/pytorch/audio/blob/main/examples/interactive_asr/vad.py" class="inline-onebox" rel="noopener nofollow ugc">audio/vad.py at main · pytorch/audio · GitHub</a></p>
Bounded buffer using torch.multiprocessing

abhisek: <p>I am trying to implement a bounded buffer like approach where data generator and the model work as asynchronous processes. The data generator preprocess the data and stores in a shared queue (with predefined max size to limit the memory usage). The model on the other hand consumes data from this queue at its own pace until the queue is empty. Below is the snippet of my implementation.</p>
<pre><code class="lang-python">    '''
        self._buffer is an object of multiprocessing.Queue
    '''
    def produce(self):
        for obj in self._generator:
            self._buffer.put(obj=obj, block=True, timeout=None)
        self._buffer.put(obj=None)

    def consume(self):
        while True:
            dat = self._buffer.get(block=True, timeout=None)
            if dat is None:
                break
            else:
                # Train model on `dat`

        def run(self):
            pt = multiprocessing.Process(target=self.produce)
            ct = multiprocessing.Process(target=self.consume)
            pt.start()
            ct.start()
            pt.join()
            ct.join()
</code></pre>
<p>I also set <code>multiprocessing.set_start_method('spawn')</code> but I get an error <code>generator object cannot be pickled</code>.</p>
<p><strong>What is the correct way to achieve this?</strong></p>
Constant/Variable Q Transform

lewiswolf: <p>Hi,</p>
<p>I’ve been looking into using a Constant Q Transform in my pipeline, which I’m currently doing with librosa. I would like to rewrite this function, so that I only need to use pytorch/torchaudio for my application, and also so that it can be written in c++ like torch.stft. I am however unsure on how to get started. Where is the c++ part of torch.stft defined, so that I can get a sense of how to proceed with writing a VQT function. (VQT and CQT are essentially the same).</p>
<p>Thanks!</p>

nateanl: <p>Hi <a class="mention" href="/u/lewiswolf">@lewiswolf</a>! Thanks for the proposal. It’ll be great to add the VQT and CQT functions. Here is the reference for <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/SpectralOps.cpp#L624" rel="noopener nofollow ugc">torch.stft</a>.<br>
I also recommend adding the functions to <a href="https://github.com/pytorch/audio" rel="noopener nofollow ugc">torchaudio repo</a>, it already supports several audio processing functions like MFCC, PitchShift, and so on. Feel free to create an issue for feature proposal and add your PR there <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

yoyololicon: <p><a class="mention" href="/u/lewiswolf">@lewiswolf</a><br>
You can try <a href="https://github.com/KinWaiCheuk/nnAudio" rel="noopener nofollow ugc">nnAudio</a>, which is built upon pytorch and would probably fit your needs.</p>

nateanl: <p><a class="mention" href="/u/yoyololicon">@yoyololicon</a> I checked the implementation in nnAudio. I think it can be refactored by using the native torch complex dtype. Do you think it’s worthy implementing it in C++ like torch.stft, or it’s good enough to implement in Python, like LFCC/MFCC?</p>

lewiswolf: <p>I’d argue that a C++ implementation would be better, just based on the overall performance of these algorithms. Thanks for sharing though everyone. I’ll take a crack at it shortly!</p>

nateanl: <p>Sounds good! Looking forward to your implementation🙂 Also I realized you can add your C++ implementation to torchaudio, as we have several C++ functions already such as <a href="https://github.com/pytorch/audio/blob/main/torchaudio/csrc/lfilter.cpp" class="inline-onebox" rel="noopener nofollow ugc">audio/lfilter.cpp at main · pytorch/audio · GitHub</a>. It will attract more researchers specially in audio and speech field.</p>

yoyololicon: <p>Seems like there’s already a CQT proposal at  <a href="https://github.com/pytorch/audio/issues/588" rel="noopener nofollow ugc">torchaudio</a>.<br>
<a class="mention" href="/u/nateanl">@nateanl</a> Yeah I think implementing CQT in Python is efficient enough cuz most of the operation can be done as matrix operation.</p>
How to save one memory copy in `reshape` and `permute`

leslie-fang-intel: <p>I have a tensor with channel_last format. I need to do some transformation as below:</p>
<pre><code class="lang-auto">a = torch.empty((10, 3, 32, 32), memory_format=torch.channels_last)
a.reshape(10, -1, 16).permute(0, 2, 1).contiguous()
</code></pre>
<p>However, in this way, I need to do 2 memory copy in <code>reshape</code> and <code>contiguous</code> specifically. Is it possible to save one memory copy?</p>
How to check if all values of a torch.tensor is False using torch.all or another utility?

Mona_Jalal: <p>is there an equivalent of torch.all <a href="https://pytorch.org/docs/stable/generated/torch.all.html" class="inline-onebox" rel="noopener nofollow ugc">torch.all — PyTorch 1.8.1 documentation</a> but for checking to see if all of values of a torch.tensor is false?</p>
<pre><code class="lang-auto">
&gt;&gt;&gt; a
tensor([[ True,  True],
        [ True, False],
        [ True,  True],
        [ True,  True]])
&gt;&gt;&gt; torch.all(a)
tensor(False)
</code></pre>
<p>Here, torch.all checks to see if all values are True or not.</p>

Mona_Jalal: <pre><code class="lang-auto">&gt;&gt;&gt; b = torch.tensor([[False, False], [False, False], [False, False]])
&gt;&gt;&gt; b
tensor([[False, False],
        [False, False],
        [False, False]])
&gt;&gt;&gt; torch.all(b == False)
tensor(True)
</code></pre>

Jonas_De_Schouwer: <pre><code class="lang-python">not torch.any(a)
</code></pre>
<p>also does the trick</p>
Video frames as conv2d channels or 1 channel conv3d

Saad: <p>Hi, I have a greyscale video input that I’m trying to do a classification problem on. There are ~62 frames in the video and each individual frame is 64x64 pixels. I was wondering if there is any difference between using a conv2d with 62 channels and using a conv3d with 62 depth and 1 channel. Conv3d seems to make more sense in context but since it’s just 1 channel I’m wondering if it actually makes any difference in practice?</p>
<p>Thanks in advance for any help</p>

JuanFMontesinos: <p>Well it does.<br>
The conv2d with 64 channels takes into account all the frames when computing the outputs.<br>
Meanwhile a conv3d with temporal kernel=1 is like applying independenty 2d convolutions to each frame. Therefore their features will depend in each frame only.</p>
Got an error, does anybody know why?

hananelroe: <p>the error:</p>
<p>File “C:\Users\hagit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\torch\nn\modules\module.py”, line 201, in _forward_unimplemented<br>
raise NotImplementedError<br>
NotImplementedError</p>

zetyquickly: <p>Hi <a class="mention" href="/u/hananelroe">@hananelroe</a></p>
<p>Look at the file within this path, it’s highly likely that you’ll find <code>pass</code> statement in <code>forward</code> method.</p>
<p>If so, you need to put some calculations in it. In order to make it work</p>

hananelroe: <p>But this is not my code, it’s part of the library…</p>

ptrblck: <p>The error is raised, if an <code>nn.Module</code> is missing the <code>forward</code> definition either because it was never provided or often because of a typo as seen here:</p>
<pre><code class="lang-python"># missing forward
class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.lin = nn.Linear(1, 1)

x = torch.randn(1, 1)
model = MyModel()    
out = model(x)
&gt; NotImplementedError

# typo
class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.lin = nn.Linear(1, 1)
        
    def forwrad(self, x): # NOTE THE TYPO!!!
        return self.lin(x)

model = MyModel()    
out = model(x)
&gt; NotImplementedError

# works
class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.lin = nn.Linear(1, 1)
        
    def forward(self, x):
        return self.lin(x)
    
model = MyModel()    
out = model(x) # works
</code></pre>

hananelroe: <p>I couldn’t find the typo in your example text… Could you tell me what was the typo?</p>

ptrblck: <p><code>forward</code> → <code>forwrad</code> was the typo in my example.</p>

hananelroe: <p>okay so what happened is that one of my files was accidentally deleted, and the recovered file didnt have the forward method. oopsie.</p>
How to get parameters’ gradients with "torch.nn.utils.prune"?

chenwydj: <p>May I ask how to get parameters’ gradients after applying pruning masks?</p>
<p>I applied <code>prune.global_unstructured</code> and <code>pruning_method=prune.RandomUnstructured</code> to ResNet50.</p>
<p>I got the following observations. It seems pruned <code>nn.Parameter</code> are converted into plain <code>nn.Tensor</code>, and I can no longer get their gradients.</p>
<pre><code class="lang-bash">* remain weight ratio =  10.01393285846195 %
-&gt; optimizer.step() # after loss.backward(), before gradient descent step
(Pdb) model.conv1.weight.grad.shape # I intentionally avoid pruning conv1
torch.Size([64, 3, 3, 3])
(Pdb) model.layer1[0].conv1.weight.grad.shape # I prune all other weights
main_train.py:1: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradien
t for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 fo
r more informations.
  '''
*** AttributeError: 'NoneType' object has no attribute 'shape'
(Pdb) model.layer1[0].conv1.weight.shape
torch.Size([64, 64, 3, 3])
(Pdb) type(model.layer1[0].conv1.weight)
&lt;class 'torch.Tensor'&gt;
(Pdb) type(model.conv1.weight)
&lt;class 'torch.nn.parameter.Parameter'&gt;
</code></pre>
Access python module used in torch.hub model?

turian: <p>I am using the <a href="https://github.com/harritaylor/torchvggish" rel="noopener nofollow ugc">torchvggish</a> torch.hub model. This package has no pypi module.</p>
<p>I would like to modify global variable <code>torchvggish.vggish_params.EXAMPLE_HOP_SECONDS</code> after loading the model. However, I cannot import <code>torchvggish.vggish_params</code> because I don’t have it installed on my system and it’s tricky to install.</p>
<p>What would be the simplest way to modify <code>torchvggish.vggish_params.EXAMPLE_HOP_SECONDS</code>?</p>

ptrblck: <p>I think you could create a fork, change the repository as you wish, and point <code>torch.hub</code> to your repo.</p>
Cross-validation in pytorch

DanteTemplar: <p>How to beautifully implement cross-validation in pytorch, while using different transformers for validation and training selection?<br>
Are there solutions that only change the dataloader iterator, that is, do not require many changes in the fitting code?</p>

ptrblck: <p><a href="https://github.com/skorch-dev/skorch"><code>skorch</code></a> is a sikit-learn compatible PyTorch wrapper, which should allow you to apply the scikit-learn cross-validation utilities directly.</p>
How does JIT torch script find type of the variable

Kitrak_rev: <p>I am a beginner in open-source communities. And I have been trying to track down a bug in JIT for identifying type of a variable, but i am not able to point to the exact file where it happens. I have been looking into ir.h, parser .h, and the Scalar type of IR isnt defined anywhere. I am specifically working on ceil function. Any info regarding this  is welcome.</p>

ptrblck: <p>I’m not sure what exactly you are looking for, but in case it’s a specific C++ type you might want to <code>grep -r</code> your PyTorch repository to check where it’s used and defined.</p>
Where are the definitions of those __device__ 'std::pow' like functions?

BruceDai003: <p>As I’m tracking down the source code in aten/src/ATen/native/cuda/PowKernel.cu file, as you see there is a lot of host and device functions that would call some <code>std::pow()</code>  and also global ‘::pow()’ functions. But I’m not sure where are those functions defined? I mean in <code>&lt;cmath&gt;</code> there is std::pow on the host side, if I remember that correctly. But where are the device side definition? I think it must be overloaded somewhere, right?<br>
So there must be device side explicitly defined and specialized std::pow(). Is it defined inside pytorch or from the compilers? Could you tell where that could be?<br>
Here is the code from PowKernel.cu file:</p>
<pre><code class="lang-auto">#ifdef _MSC_VER

// Functions for pow

// pow for at::Half

static inline __host__ __device__ at::Half pow_(at::Half base, at::Half exp) {

  return static_cast&lt;at::Half&gt;(std::pow(static_cast&lt;float&gt;(base), static_cast&lt;float&gt;(exp)));

}
...
#else

template &lt;typename Base_type, typename Exp_type&gt;

static inline __host__ __device__ Base_type pow_(Base_type base, Exp_type exp) {

  return ::pow(base, exp);

}
</code></pre>
<p>Question: where can I find the <code>std::pow()</code> function and <code>::pow()</code> 's definitions?</p>

tom: <p>On Linux this is in <code>include/crt/math_functions.hpp</code>, but there the code snippet actually uses <code>::pow</code>, which comes from <a href="https://docs.nvidia.com/cuda/cuda-math-api/group__CUDA__MATH.html#group__CUDA__MATH" rel="noopener nofollow ugc">the CUDA Math API</a>.<br>
As far as I understand, much of the trickery is needed to support half datatypes on Windows due to templating limitations (as the comment suggests) in the compiler.</p>
<p>Best regards</p>
<p>Thomas</p>
Cant reproduce QAT precision

alanzhai219: <p>I follow the tutorial of <a href="https://github.com/pytorch/tutorials/blob/master/advanced_source/static_quantization_tutorial.rst" class="inline-onebox" rel="noopener nofollow ugc">tutorials/static_quantization_tutorial.rst at master · pytorch/tutorials · GitHub</a><br>
MobileNet-V2 baseline and PTQ work as expected but QAT top1 is only 67.88 after 8 epoch.<br>
log here <a href="https://mclo.gs/TRHsVfl" class="inline-onebox" rel="noopener nofollow ugc">Unknown server log [#TRHsVfl] - mclo.gs</a></p>
<p>Thanks.</p>

raghuramank100: <p>The tutorial shows an easy to run example, where you are using only a few batches of data for training (num_train_batches = 20). Also, the result of 71.5% is after training over 30 epochs on the full training dataset. If you are training on the full imagenet dataset, your results look ok. A full script to reproduce the training numbers is available at: <a href="https://github.com/pytorch/vision/blob/master/references/classification/train_quantization.py" class="inline-onebox" rel="noopener nofollow ugc">vision/train_quantization.py at master · pytorch/vision · GitHub</a></p>

alanzhai219: <p><a class="mention" href="/u/raghuramank100">@raghuramank100</a> Thanks for your kindly reply. Your answer is right. I am reproducing the accuracy as your suggested and result looks good.</p>
<ol>
<li>
<p>Bug <a href="https://github.com/pytorch/pytorch/issues/30125" rel="noopener nofollow ugc">30125</a> and comments provide some useful info. Believe it or not, QAT of <a href="https://github.com/pytorch/tutorials/blob/master/advanced_source/static_quantization_tutorial.rst" rel="noopener nofollow ugc">static_quantization_tutorial.rst</a> is NOT a good guide because it cant reproduce 71.5%.</p>
</li>
<li>
<p>I find some differences between <a href="https://github.com/pytorch/tutorials/blob/master/advanced_source/static_quantization_tutorial.rst" rel="noopener nofollow ugc">static_quantization_tutorial.rst</a> and <a href="https://github.com/pytorch/vision/blob/master/references/classification/train_quantization.py" rel="noopener nofollow ugc">train_quantization.py</a> which seems to cause different accuracy. Actually, hyperparameter and training tricks make sense, right?</p>
</li>
<li>
<p>BTW, when and why to insert hyperparameter/freeze-operation/disable-operation is not easy during training. Any QAT trick or routine for other networks instead of mobilenet-v2, such as more classification or detection or segmentation?  That is much helpful.</p>
</li>
</ol>
<p>Thanks.</p>
Slicing over 2 dimensions on a 3D Tensor

ravijoshi: <p>I have a 3 dimensional tensor that contains single channel images. For each image, I also have a pixel location (y, x).  I want to grab a patch (say a patch of size 3x3) centered around the pixel location in this 3  dimensional tensor.</p>
<p>Please see my approach:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; d = torch.rand(4, 64, 64)
&gt;&gt;&gt; yx = torch.tensor([(15, 21), (30, 59), (40, 5), (20, 25)])
&gt;&gt;&gt; y, x = yx.T
&gt;&gt;&gt; m = 1  # a patch of 3x3 centered around given pixel location

# loop through each image and stack the result
&gt;&gt;&gt; out = torch.stack([di[yi - m : yi + m + 1, xi - m : xi + m + 1] \
                                     for di, yi, xi in zip(d, y, x)])
&gt;&gt;&gt; out.size()
torch.Size([4, 3, 3])
</code></pre>
<p>Well, if I reduce the patch to size 1x1 pixel, which mean only the pixel and no neighbors pixels, then I could simply use the following:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; loc = (range(4), y, x)
&gt;&gt;&gt; out = d[loc]
&gt;&gt;&gt; out.size()
torch.Size([4])
</code></pre>
<p>However, I actually need a patch of size more than 1x1. I tried using the same approach but no success:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x_slice = slice(x - m, x + m + 1)
&gt;&gt;&gt; y_slice = slice(y - m, y + m + 1)
&gt;&gt;&gt; d[(range(4), y_slice, x_slice)]
TypeError: only integer tensors of a single element can be converted to an index
</code></pre>
<p><strong>My approach applying <code>torch.stack( ... )</code>  uses a <code>for</code> loop. I am wondering if there is a better PyTorch way of doing the same.</strong></p>
<p><em>PS: In this case, please assume that input pixel locations are never on the edges. In other words, it is always possible to grab a patch of a given size!</em></p>
Error when using string variable for device

Podidiving: <p>Hello. I’m facing quite weird behaviour of torch. Here is my code:</p>
<pre><code class="lang-auto">image = ... # some numpy image
device = "cuda:0"
model = torch.jit.load("model.pt", map_location=device)
result = model(torch.from_numpy(image)[None].to(device))
</code></pre>
<p>This code throws an error</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-42-ea97506efa7e&gt; in &lt;module&gt;
----&gt; 1 result = model(torch.from_numpy(image)[None].to(device))

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

RuntimeError: MALFORMED INPUT: bad dtype in CompareSelect
</code></pre>
<p>However, if I write:</p>
<pre><code class="lang-auto">image = ... # some numpy image
model = torch.jit.load("model.pt", map_location="cuda:0")
result = model(torch.from_numpy(image)[None].to("cuda:0"))
</code></pre>
<p>It works fine.</p>
<p>My pytorch version is `‘1.9.0+cu102’</p>
<p>Any ideas, why it is happening?</p>

ptrblck: <p>Could you post the model definition (in case that’s not possible, could you try to use a proxy model) to reproduce the issue? I also guess you’ve used <code>torch.jit.trace</code> to store the model?</p>
The impact of 'del' during traing

Ford: <p>Due to the limited GPU space, I want to to use ‘del’ to release the space storing the variable, to further avoid the error of ‘cuda out of memory’. I wonder whether this operation would cause some undesired consequence, like  stopping the back propagation of the corresponding variable?</p>

ptrblck: <p>I would claim it depends what exactly you are deleting, as e.g. deleting a needed layer would of course raise an error:</p>
<pre><code class="lang-python">model = models.resnet152()
x = torch.randn(1, 3, 224, 224)
out = model(x)
out.mean().backward()

del model.layer4[0].bn1

out = model(x)
&gt; AttributeError: 'Bottleneck' object has no attribute 'bn1'
</code></pre>
<p>Generally, I would recommend to utilize the function scoping of Python, i.e. to use method to <code>train()</code> and <code>evaluate()</code> the model since all intermediate objects will be freed once the method is left.<br>
Often additional memory is used if you are using a validation loop in the same method (e.g. <code>main()</code>) as the training loop and e.g. keep the training input data as well as the validation data alive.</p>
Are there any cool or good extension in VScode for debugging PyTorch code?

Mona_Jalal: <p>I am wondering if there is any great extension in VSCode for debugging PyTorch code?<br>
If not, what are some other tools, pieces of code, that you would suggest for a beginner-level debugging of PyTorch code?</p>
<p>I am using VSCode in OSX for remotely running some PyTorch code in a Python Virt Env in CentOS 7.</p>
Different latency to use repeat_interleave on CPU

Wilson: <p>I find that repeat_interleave can generate different latency on the cpu.<br>
For example</p>
<blockquote>
<p>import torch<br>
a = torch.rand(1,1024,256,100)<br>
torch.repeat_interleave(a,4,dim=0)<br>
torch.repeat_interleave(a,4,dim=1)<br>
torch.repeat_interleave(a,4,dim=2)<br>
torch.repeat_interleave(a,4,dim=3)</p>
</blockquote>
<p>They cost 0.22s, 0.27s, 0.34s and 2.25s individually.<br>
I realize that it should be related to the allocation on memory. Seems that other functions such as <code>cat</code> may also have similar problems if we perform over dimensions arbitrary. Is there any methodology to avoid such a huge latency? Thanks.</p>
Pytorch extensions - selecting specific MSVC cl.exe for distutils, setuptools

SpaceCowboy850: <p>So…MS broke cuda compiles with their latest update to VS2019 (a break in xutility).</p>
<p>It appears that an older version of VS2019 still works.  As luck would have it, I have a community version that is on the broken new version, but a VS2019 professional version that should have a version that still works.</p>
<p>However, I’m finding it unclear how I can set this.  It appears that the cl.exe in git-bash is picking up the community directory and I don’t know how to change this to the professional directory.</p>
<p>Has anyone had any experience redirecting what VS setuptools is discovering?</p>
Understanding a simple attention code snippet for images

Mona_Jalal: <p>I am trying to understand why the following simple code module does act as an attention layer? Most of what I am finding in blogs/papers is quite more complex. Is there a mathematical or intuitive way of understanding the following as an attention layer?</p>
<pre><code class="lang-auto">  self.layer_attend1 =  nn.Sequential(nn.Conv2d(layers[0], layers[0], stride=2, padding=1, kernel_size=3),
                                            nn.AdaptiveAvgPool2d(1),
                                            nn.Softmax(1))
</code></pre>
RuntimeError: ONNX export failed: Couldn’t export Python operator qfn

111565: <p>Hi,<br>
I’m trying to convert a pytorch model to onnx, but the conversion gives the following error:</p>
<p>RuntimeError: ONNX export failed: Couldn’t export Python operator qfn</p>
<p>The definition for qfn is:</p>
<p>class qfn(torch.autograd.Function):<br>
<a href="https://github.com/staticmethod" rel="noopener nofollow ugc">@staticmethod</a><br>
def forward(ctx, input, k):<br>
n = float(2**k - 1)<br>
out = torch.round(input * n) / n<br>
return out</p>
<pre><code class="lang-auto">@staticmethod
def backward(ctx, grad_output):
    grad_input = grad_output.clone()
    return grad_input, None
</code></pre>
<p>qfn_tmp = qfn.apply</p>
Fully connected input size

Alexandra_Lorenzo: <p>Hello,<br>
I have the following error, that seems related to my model initialization.</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)
</code></pre>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 1, kernel_size=7, padding=3)
        self.conv2 = nn.Conv2d(1, 8, kernel_size=7, padding=3)
        self.conv3 = nn.Conv2d(8, 8, kernel_size=7, padding=3, stride=2)
        self.conv4 = nn.Conv2d(8, 8, kernel_size=7, padding=3, stride=2)
        self.conv5 = nn.Conv2d(8, 16, kernel_size=7, padding=3, stride=2)
        self.conv6 = nn.Conv2d(16, 32, kernel_size=7, padding=3, stride=2)
        self.conv7 = nn.Conv2d(32, 32, kernel_size=7, padding=3, stride=2)
        self.fc1 = nn.Linear(32*16*16,256) #Error : size ? 
        self.fc2 = nn.Linear(256,48)


    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = self.conv3(x)
        x = self.conv4(x)
        x = self.conv5(x)
        x = self.conv6(x)
        x = self.conv7(x)
        print('size',x.size())
        x = self.fc1(x)
        x = self.fc2(x)
        
        return x

device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # PyTorch v0.4.0
model = Net().to(device)
summary(model, (1, 512, 512))
</code></pre>
<p>But it seems that my fully connected 1 is incorrect.<br>
Anyone knows why?</p>

zetyquickly: <p>Hello <a class="mention" href="/u/alexandra_lorenzo">@Alexandra_Lorenzo</a></p>
<p>That error looks very uninformative, I think this is because of quite old version used (<code># PyTorch v0.4.0</code>).</p>
<p>What I can see from you code that <code>Linear</code> takes output from <code>Conv2d</code> without any modifications.</p>
<pre><code class="lang-auto">...
x = self.conv7(x)
print('size',x.size())
x = self.fc1(x)
...
</code></pre>
<p>This is definitely cause shape mismatch error. <code>Linear</code> takes 3d input, while <code>Conv2d</code> outputs 4d tensors. Please consider using poolings or flatening before dense layer</p>
Fail to download LibTorch 1.8.2 for windows

lin_morty: <p>I tried to download this link from the official website.<br>
<a href="https://download.pytorch.org/libtorch/cu111/libtorch-win-shared-with-deps-1.8.2%2Bcu111.zip" class="onebox" target="_blank" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cu111/libtorch-win-shared-with-deps-1.8.2%2Bcu111.zip</a></p>
<p>But I got Access Denied, please help.</p>
<p>The full message is:<br>
</p>
<p><code>AccessDenied</code></p>
<p>Access Denied</p>
<p>5PN39BYB7FAMTEF4</p>
<p>EMCyED2fW54u7BEqOJI9xq7xnxi5FwWNe/OySd1kA4kldglzp9IoGAQro6tZP4Jw6GXDiYHTVdY=</p>

ptrblck: <p>Your link is invalid, as PyTorch <code>1.8.2</code> doesn’t exist.<br>
Change the link to the <code>1.8.1</code> or <code>1.9.0</code> version and it should work:<br>
<a href="https://download.pytorch.org/libtorch/cu111/libtorch-win-shared-with-deps-1.8.1%2Bcu111.zip">1.8.1</a><br>
<a href="https://download.pytorch.org/libtorch/cu111/libtorch-win-shared-with-deps-1.9.0%2Bcu111.zip">1.9.0</a></p>

lin_morty: <p>Thanks a lot. This solves my problem.</p>
<p>Just for you information, I ask this because “<a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a>” show only LTS(1.8.2).</p>

ptrblck: <p>OK, I see that the link is indeed missing. CC <a class="mention" href="/u/seemethere">@seemethere</a></p>

mszhanyi: <p>It has been  fixed now</p>

seemethere: <p>I had totally written a response to this but had neglected to post it, this should be resolved with <a href="https://github.com/pytorch/pytorch.github.io/pull/816" class="inline-onebox" rel="noopener nofollow ugc">quick-start: Update links for lts/1.8 libtorch by seemethere · Pull Request #816 · pytorch/pytorch.github.io ·</a></p>