akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Tensor.clone.detach() vs tensor.detach()?

Ge0rges: <p>What’s the difference between <code>Tensor.clone.detach()</code> and <code>tensor.detach()</code>? Since detach returns the a detached version of tensor, what is the point of cloning?</p>

russellizadi: <p>When the <code>clone</code> method is used, torch allocates a new memory to the returned variable but using the <code>detach</code> method, the same memory address is used.</p>
<p>Compare the following code:</p>
<pre><code class="lang-auto">import torch
device = torch.device("cuda")
a = torch.randn([10000, 10000])
a = a.to(device)
print(round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
b = a.detach()
print(round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
c = a.clone().detach()
print(round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
</code></pre>
<pre><code class="lang-auto">0.4 GB
0.4 GB
0.7 GB
</code></pre>
<p>to this code:</p>
<pre><code class="lang-auto">import torch
device = torch.device("cuda")
a = torch.randn([10000, 10000])
a = a.to(device)
print(round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
b = a.clone().detach()
print(round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
c = a.detach()
print(round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
</code></pre>
<pre><code class="lang-auto">0.4 GB
0.7 GB
0.7 GB
</code></pre>

Ge0rges: <p>Does that mean that any modifications made to the detached tensor also occur to the attached version?</p>

russellizadi: <p>That’s true. Following the same example:</p>
<pre><code class="lang-auto">import torch
device = torch.device("cuda")
a = torch.randn([2])
a = a.to(device)
print(a)
b = a.detach()
print(b)
c = a.clone().detach()
print(c)
b[0] = 1.
print(a)
print(c)
</code></pre>
<pre><code class="lang-auto">tensor([ 0.2042, -1.8436], device='cuda:0')
tensor([ 0.2042, -1.8436], device='cuda:0')
tensor([ 0.2042, -1.8436], device='cuda:0')
tensor([ 1.0000, -1.8436], device='cuda:0')
tensor([ 0.2042, -1.8436], device='cuda:0')
</code></pre>

kwea123: <p>Apart from the difference in memory, do they share the same properties such as the value, requires_grad=False, and all other properties?</p>
CUDA out of memory but says error suggests there is free memory

neuralpat: <p>Hi,</p>
<p>I’m trying to fine-tune gpt2 and while training (with a batch size of 1) I get</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "H:/PycharmProjects/pythonProject/DecisionSummariesLM.py", line 102, in &lt;module&gt;
    outputs = model(input_ids,
  File "C:\ProgramData\Anaconda3\envs\pythonProject\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\pythonProject\lib\site-packages\transformers\models\gpt2\modeling_gpt2.py", line 888, in forward
    transformer_outputs = self.transformer(
  File "C:\ProgramData\Anaconda3\envs\pythonProject\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\pythonProject\lib\site-packages\transformers\models\gpt2\modeling_gpt2.py", line 738, in forward
    outputs = block(
  File "C:\ProgramData\Anaconda3\envs\pythonProject\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\pythonProject\lib\site-packages\transformers\models\gpt2\modeling_gpt2.py", line 319, in forward
    feed_forward_hidden_states = self.mlp(self.ln_2(hidden_states))
  File "C:\ProgramData\Anaconda3\envs\pythonProject\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\pythonProject\lib\site-packages\transformers\models\gpt2\modeling_gpt2.py", line 261, in forward
    return self.dropout(h2)
  File "C:\ProgramData\Anaconda3\envs\pythonProject\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\pythonProject\lib\site-packages\torch\nn\modules\dropout.py", line 58, in forward
    return F.dropout(input, self.p, self.training, self.inplace)
  File "C:\ProgramData\Anaconda3\envs\pythonProject\lib\site-packages\torch\nn\functional.py", line 983, in dropout
    else _VF.dropout(input, p, training))
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 8.00 GiB total capacity; 3.19 GiB already allocated; 3.52 GiB free; 3.20 GiB reserved in total by PyTorch)
</code></pre>
<p>I don’t really understand why I’m getting this as it’s only 20 MiB and the error seems to suggest that there are 3.20 GiB allocated for pytorch.</p>
<p>I have fine-tuned the same model on the same GPU before with different data but 20 MiB ?</p>
What's the difference between scheduler.step() and scheduler.step(epoch)?

Liang: <p>I’m using 1.4.0. What’s the difference between scheduler.step() and scheduler.step(epoch)? What does<br>
scheduler.step(epoch) do?</p>
<pre><code class="lang-auto">import torch

optimizer1 = torch.optim.SGD([torch.randn(1, requires_grad=True)], lr=1e-3)
exp_lr_scheduler1 = torch.optim.lr_scheduler.MultiStepLR(optimizer1,
                                             milestones=[5, 10], gamma=0.1)

optimizer2 = torch.optim.SGD([torch.randn(1, requires_grad=True)], lr=1e-3)
exp_lr_scheduler2 = torch.optim.lr_scheduler.MultiStepLR(optimizer2,
                                             milestones=[5, 10], gamma=0.1)

for epoch in range(1, 15):
    exp_lr_scheduler1.step()
    exp_lr_scheduler2.step(epoch)
    print('Epoch {}, lr1 {}, lr2 {}'.format(epoch,
        optimizer1.param_groups[0]['lr'],
        optimizer2.param_groups[0]['lr']))

</code></pre>
<p>The result is like follows:</p>
<pre><code class="lang-auto">Epoch 1, lr1 0.001, lr2 1.0000000000000003e-05
Epoch 2, lr1 0.001, lr2 1.0000000000000003e-05
Epoch 3, lr1 0.001, lr2 1.0000000000000003e-05
Epoch 4, lr1 0.001, lr2 1.0000000000000003e-05
Epoch 5, lr1 0.0001, lr2 1.0000000000000003e-05
Epoch 6, lr1 0.0001, lr2 1.0000000000000003e-05
Epoch 7, lr1 0.0001, lr2 1.0000000000000003e-05
Epoch 8, lr1 0.0001, lr2 1.0000000000000003e-05
Epoch 9, lr1 0.0001, lr2 1.0000000000000003e-05
Epoch 10, lr1 1e-05, lr2 1.0000000000000003e-05
Epoch 11, lr1 1e-05, lr2 1.0000000000000003e-05
Epoch 12, lr1 1e-05, lr2 1.0000000000000003e-05
Epoch 13, lr1 1e-05, lr2 1.0000000000000003e-05
Epoch 14, lr1 1e-05, lr2 1.0000000000000003e-05
</code></pre>
<p>They have different behavious when using <code>MultiStepLR</code>.</p>

wnma3mz: <p>Thanks, it worked for me, and I also tried it in torch 1.4.0. The results are just like your results.</p>
<p>But it’s worth noting that when this code was run in torch 1.2.0, the results were as follows</p>
<pre><code class="lang-auto">Epoch 1, lr1 0.001, lr2 0.001
Epoch 2, lr1 0.001, lr2 0.001
Epoch 3, lr1 0.001, lr2 0.001
Epoch 4, lr1 0.001, lr2 0.001
Epoch 5, lr1 0.0001, lr2 0.0001
Epoch 6, lr1 0.0001, lr2 0.0001
Epoch 7, lr1 0.0001, lr2 0.0001
Epoch 8, lr1 0.0001, lr2 0.0001
Epoch 9, lr1 0.0001, lr2 0.0001
Epoch 10, lr1 1.0000000000000003e-05, lr2 1.0000000000000003e-05
Epoch 11, lr1 1.0000000000000003e-05, lr2 1.0000000000000003e-05
Epoch 12, lr1 1.0000000000000003e-05, lr2 1.0000000000000003e-05
Epoch 13, lr1 1.0000000000000003e-05, lr2 1.0000000000000003e-05
Epoch 14, lr1 1.0000000000000003e-05, lr2 1.0000000000000003e-05
</code></pre>
Question about requires_grad = True

morphism: <p>Hello, everyone</p>
<p>As far as I know, model parameters (W and b) have settting requires_grad = True, by default<br>
when they are intialized</p>
<p>Then, how about data matrix and label ??<br>
Should I set requires_grad = True when I initialize data matrix (input data or batch data) as torch.tensor??</p>
<p>Also, should I set requires_grad = True when I input label vector into training as torch.tensor??</p>
<p>Thank you and have a nice day!</p>

albanD: <p>Hi,</p>
<p>You only need to set <code>requires_grad=True</code> if you need to get gradients for that Tensor.<br>
That is why parameters are created this way by default.<br>
You should create your inputs with this only if you plan on asking for gradients wrt to the inputs (which people usually don’t). So in general you don’t need to set it no.</p>

morphism: <p>Dear alban<br>
Thank you for your reply.<br>
I understand what you mean</p>
<p>May I ask one more question?<br>
What will happen if I wrongly set requires_grad = True for data matrix and label??</p>

albanD: <p>It won’t break anything, the things that can happen are:</p>
<ul>
<li>you will compute gradients for some Tensors and don’t use it. Slowing down your code slightly for no reason.</li>
<li>If your label is of a non-contiguous dtype (like long), it will raise an error if you try to set requires_grad=True. Because that is only supported for contiguous dtypes.</li>
</ul>

morphism: <p>Thank you, alban<br>
You helped me a lot</p>
<p>Have a nice week and see you again.</p>
<p>Thanks</p>
Storing tensors for later computation out of memory error

Matthew_Macfarlane: <p>Hello</p>
<p>I have a task where at each step I need to encode a Graph Neural Network. Due to the structure of my graph I know that at each step only certain nodes will need re-encoded and in principle I can store the encodings of the other nodes from previous computations for use in this prediction. Once the graph is encoded I run a simple feedforward neural network to output a decision on the Graph Nodes. When I try to store tensors from my past evaluations and update only certain nodes very quickly I run out of memory (64GB) . I feel this problem is somehow due to many computation graphs being stored but I am not sure exactly what is going on. I have also tried to just re-encode from scratch the graph each time. This doesn’t cause any oom issues however its major downside is the time it takes and I want to take advantage of the fact that I already know the encodings for some nodes from past steps.</p>
<p>I am looking for some advice as to what is going on here.</p>
Freeze when using garbage collector for debugging

vkubicki: <p>I am using a modified version of the SAC algorithm presented at: <a href="https://github.com/openai/spinningup/tree/master/spinup/algos/pytorch/sac" class="inline-onebox" rel="noopener nofollow ugc">spinningup/spinup/algos/pytorch/sac at master · openai/spinningup · GitHub</a> . The memory consumption slowly increase until the process is killed. Following recommendations from <a href="https://discuss.pytorch.org/t/how-to-debug-causes-of-gpu-memory-leaks/6741" class="inline-onebox">How to debug causes of GPU memory leaks?</a>, I created this function:</p>
<pre><code class="lang-auto">def gc_n():
    n = 0
    for obj in gc.get_objects():
        try:
            if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
                n += 1
        except:
            pass
    return n
</code></pre>
<p>When I run it, I am capable of printing <code>n</code> to a file, however, the code seems to freeze afterward, with no possibility to kill via <code>CRTL+C</code>, and the message:</p>
<pre><code class="lang-auto">/home/kubicki/.local/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:126: UserWarning: torch.distributed.reduce_op is deprecated, please use torch.distributed.ReduceOp instead
  warnings.warn("torch.distributed.reduce_op is deprecated, please use "
</code></pre>
<p>If I use a simpler function:</p>
<pre><code class="lang-auto">def gc_simple_n():
    return len(gc.get_objects())
</code></pre>
<p>The run proceeds normally, but since every Python object is counted, the result is much harder to interpret.</p>
<p>Could you help me understand the error and get the correct number of tensors ?</p>

vkubicki: <p>Pytorch was installed using <code>pip install torch torchvision</code>, but I did not use the GPU. I reinstalled it using <code>pip install torch==1.7.1+cpu torchvision==0.8.2+cpu torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html</code>, and <code>gc</code> works without freeze. I still get the warning, though.</p>
Custom activation function inside nn.module

cbd: <p>I want to create custom activation function inside “class gen(nn.Module):”. I want to apply this activation function after layers define by “nn.sequential” such that loss calculated from output of custom activation function will be backpropagate.</p>
<p>Activation function code is  “z=torch.where(x &gt; 0.1, x, zero_tensor)”</p>
<p>How should i define custom activation function inside “class gen(nn.Module):” such that loss will be backpropagate? I need many layers of such activation function.</p>
<p>Should i need to use “forward” function instead of “my_act” or something else.</p>
<p>Independent running code of activation function is as below:</p>
<p>class act():</p>
<p>def my_act(x):<br>
zero_tensor = torch.zeros(x.size())<br>
z=torch.where(x &gt; 0.1, x, zero_tensor)<br>
return z</p>
How to assign a tensor with another tensor?

jerly-hjzhou: <pre><code class="lang-auto">import torch
from LeNet5 import LeNet5

model = torch.load('LeNet5.pth')  # 加载模型
layerWeight = model.state_dict()['conv1.0.bias'].clone()
print("----layerWeight is ",layerWeight)
print("----model is ", model.state_dict()['conv1.0.bias'])

model.state_dict()['conv1.0.bias'][0] = 1

print("++++layerWeight is", layerWeight)
print("++++model is", model.state_dict()['conv1.0.bias'])

print("layerWeight type is ",type(layerWeight))
print("model type is",type(model.state_dict()['conv1.0.bias']))

model.state_dict['conv1.0.bias'] = layerWeight


print("****layerWeight is",layerWeight)
print("****model is", model.state_dict()['conv1.0.bias'])
</code></pre>
<p>Above the code, I try <strong>model.state_dict[‘conv1.0.bias’] = layerWeight</strong>,(model.state_dict[‘conv1.0.bias’]  and layerWeight are both the type of tensor)but get the error in the picture below.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55b99882abede30470d5e24e99c8aa23ff86f7a5.png" data-download-href="https://discuss.pytorch.org/uploads/default/55b99882abede30470d5e24e99c8aa23ff86f7a5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55b99882abede30470d5e24e99c8aa23ff86f7a5_2_690x175.png" alt="image" data-base62-sha1="cemdLvidznSpbb9Fu4M3VEr2G6F" width="690" height="175" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55b99882abede30470d5e24e99c8aa23ff86f7a5_2_690x175.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55b99882abede30470d5e24e99c8aa23ff86f7a5_2_1035x262.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55b99882abede30470d5e24e99c8aa23ff86f7a5_2_1380x350.png 2x" data-dominant-color="F5F4F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1380×351 52.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Please hope you could tell me how to solve it.</p>

ptrblck: <p>The error is raised, since you are not calling the <code>model.state_dict()</code> method.<br>
That being said, a direct assignment will most likely not work and you could use:</p>
<pre><code class="lang-python">sd = model.state_dict()
sd[key] = layerWeight
model.load_state_dict(sd)
</code></pre>
<p>instead.</p>

jerly-hjzhou: <p>Why are you so experienced! It’s an amazing thing for me because I could find nothing about that in pytorch document.</p>
why these results are different? please help me....

seokilee: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/0/8008935e581637a738909678d99bd7271b4c6edf.png" data-download-href="https://discuss.pytorch.org/uploads/default/8008935e581637a738909678d99bd7271b4c6edf" title="스크린샷 2021-02-02 오후 5.38.13"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/8008935e581637a738909678d99bd7271b4c6edf_2_482x500.png" alt="스크린샷 2021-02-02 오후 5.38.13" data-base62-sha1="igDulyXo9yEQy6buo7n6OGGi1av" width="482" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/8008935e581637a738909678d99bd7271b4c6edf_2_482x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/8008935e581637a738909678d99bd7271b4c6edf_2_723x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/8008935e581637a738909678d99bd7271b4c6edf_2_964x1000.png 2x" data-dominant-color="3D3E3E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">스크린샷 2021-02-02 오후 5.38.13</span><span class="informations">1619×1677 381 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c9fde52ae68e0e6047d2df8240564e83e91be99.png" data-download-href="https://discuss.pytorch.org/uploads/default/4c9fde52ae68e0e6047d2df8240564e83e91be99" title="스크린샷 2021-02-02 오후 5.36.29"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c9fde52ae68e0e6047d2df8240564e83e91be99_2_541x500.png" alt="스크린샷 2021-02-02 오후 5.36.29" data-base62-sha1="aVQNRPzqC75OJ5SSnRi6vJGrOop" width="541" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c9fde52ae68e0e6047d2df8240564e83e91be99_2_541x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c9fde52ae68e0e6047d2df8240564e83e91be99_2_811x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c9fde52ae68e0e6047d2df8240564e83e91be99_2_1082x1000.png 2x" data-dominant-color="4D4D4D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">스크린샷 2021-02-02 오후 5.36.29</span><span class="informations">1484×1371 222 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Access Dataset object inside DataLoader

dazzle-me: <p>Hello everyone, I want to know if we can access the Dataset object after creation of DataLoader object.</p>
<p>Motivation : suppose we have a simple Dataset, -</p>
<pre><code class="lang-auto">class ExampleDataset(torch.utils.data.Dataset):
    def __init__(self, data):
        self.data = data

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return len(self.data)
</code></pre>
<p>There is nothing tricky, - we can’t really modify any parameters here which can affect your training.<br>
What I want to do is, -</p>
<pre><code class="lang-auto">class TrickyDataset(torch.utils.data.Dataset):
    def __init__(self, data, aug_functions):
        self.data = data
        self.current_epoch = 0
        self.light_aug = aug_functions[0]
        self.medium_aug = aug_functions[1]
        self.hard_aug = aug_function[2]

    def __getitem__(self, index):
        if self.current_epoch &lt; 10:
           return self.light_aug(self.data[index])
        ... etc

    def update_epoch(self, epoch):
          self.current_epoch = epoch

    def __len__(self):
        return len(self.data)
</code></pre>
<p>So, if we have some</p>
<pre><code class="lang-auto">TrickyLoader = torch.utils.data.DataLoader(TrickyDataset(*params))
</code></pre>
<p>Can we modify Dataset inside during training (because I want to change the way data is being augmented during training)?</p>

Dwight_Foster: <p>I don’t think you can although you could just create a new dataloader inside your training loop and use it.</p>

dazzle-me: <p>Actually, I tried to look in the wrong place, - I looked into the <a href="https://pytorch.org/docs/stable/data.html#map-style-datasets" rel="noopener nofollow ugc">docs</a> and found no information about my question.</p>
<p>But when I looked into the <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/dataloader.html#DataLoader" rel="noopener nofollow ugc">source code</a> of DataLoader, - I found out, that you can easily do what I have described above by just directly calling</p>
<pre><code class="lang-auto">dataloader = torch.utils.data.DataLoader(TrickyDataset(*params))
dataloader.dataset.update_epoch(epoch)
</code></pre>
<p>And now it works (I hope so, haven’t visualized it yet).</p>
How to re-use same gpu memory for each forward pass?

blackyang: <p>In torch, we can declare an empty torch.Tensor() in a module’s init function, then in the forward function, just resize this torch tensor. I think in this way we can save some time on mallocing new memories.</p>
<p>In pytorch, do we even need to do this? If yes, then should it be a tensor or an autograd.Variable?</p>
<p>Thanks!</p>

chenyuntc: <p>I don’t think it can save much time/memory, I usually create a  new variable at every iteration.</p>

blackyang: <p>Oh forgot to mention, my buffer tensor remains (almost) the same for every forward pass, so in this way, I can save some time on assigning values</p>

chenyuntc: <p>AFAIK, the most time is spent in copying data from CPU to GPU, allocating memory won’t cost much time.</p>

blackyang: <p>Yeah I know, but in my case, my tensor <code>t</code> is like a helper tensor that is independent to input. So if I create a new one at each forward pass, then I also need to run function <code>assign_values(t)</code> each time, which will take some time.</p>
<p>On the other hand, if I am able to re-use gpu memory, then I only need to call <code>assig_value(t)</code> once</p>

blackyang: <p>Anyway, I figured out how to do this in a way similar to torch. Just create a buffer tensor <code>t</code> in init(), then in forward pass, wrap it with <code>Variable(t)</code></p>

chenyuntc: <p>You’re right, <code>register_buffer</code> works.</p>

blackyang: <p>Actually in my case <code>register_buffer</code> is not necessarily needed, since my buffer tensor is not persistent.</p>
<p>Anyway, thanks for your time:)</p>

colesbury: <p>PyTorch comes with built-in caching allocator for CUDA, so memory allocations are automatically re-used internally without any extra work on your part.</p>

blackyang: <p>Great! Good to know, thanks</p>

Annie-Sihan-Chen: <p>Can you explain in more detail(C++) about how does the automatically re-used work?</p>
Effective way to change pixel intensity value (pixel-shuffling)?

Yun_Lee: <p>Hi,<br>
I want to change pixel value in random way! (pixel-shuffling)</p>
<p>For example,<br>
if I use CIFAR10 data,<br>
then I could get [3,32,32] pixel tensors, and each pixel can get integer value between [0~255]<br>
What I want to do is<br>
I want to make new mapping! for 256 values. This can be named pixel-shuffling.<br>
for example, [0,1,2,255] → randint(255) =&gt;  [244,3,0,255]</p>
<p>I want to ask you if there is better way to do pixel-shuffling.<br>
I only can think for-loop to change each pixel value like below…</p>
<p>(X is given input [batch_size, 3,32,32])<br>
(key is mapping dictionary {0:244, 1:3, , previous_value: result_value,}</p>
<pre><code class="lang-auto">    result_X  = X
    for batch_i in range(X.shape[0]):
        for channel_j in range(X.shape[1]):
            for row_k in range(X.shape[2]):
                for col_l in range(X.shape[3]):
                    previous = X[batch_i][channel_j][row_k][col_l]
                    previous = previous.item()
                    result_X[batch_i][channel_j][row_k][col_l] = key[previous]
</code></pre>
<p>is there effective ways to do pixel-shuffling ?</p>

ptrblck: <p>You could probably directly index the mapping as seen here:</p>
<pre><code class="lang-python"># create input
x = torch.arange(2*3*32*32).to(torch.uint8)
x = x.view(2, 3, 32, 32)
print(x)

# create mapping by shifting it by 1
mapping = torch.arange(0, 256, dtype=torch.uint8) + 1
# mapping[index] would contain the new value

# index into mapping directly and assign to the result tensor
result = mapping[x.view(-1).long()]
# reshape back
result = result.view(2, 3, 32, 32)
print(result)

# your approach
X = x.clone()
result_X  = X
key = mapping
for batch_i in range(X.shape[0]):
    for channel_j in range(X.shape[1]):
        for row_k in range(X.shape[2]):
            for col_l in range(X.shape[3]):
                previous = X[batch_i][channel_j][row_k][col_l]
                previous = previous.item()
                result_X[batch_i][channel_j][row_k][col_l] = key[previous]

# compare
print((result == result_X).all())
&gt; tensor(True)
</code></pre>

Yun_Lee: <p>Thank you so much for sharing your answer!!<br>
your way is way more effective!</p>
Question in pytorch

randinoo: <p>how I create the final result after the SOFTMAX function of the CNN network in binary image form (0,1)?. With the entry of this network is a single image and from this image we reserve (80% train, 20% test).</p>
<p>Help me plzzzz</p>

ptrblck: <p>You can use <code>torch.argmax</code> and either pass the logits or the probabilities to it in case you are working on a multi-class classification.</p>

randinoo: <p>Yes how ? plzz <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I want to display the output image</p>

ptrblck: <p>Assuming your model outputs logits in the shape <code>[batch_size, nb_classes ,*]</code>, where <code>*</code> denotes additional dimensions, this would work:</p>
<pre><code class="lang-python">output = model(input)
pred = torch.argmax(output, dim=1)
</code></pre>
Accessing optimizer parameters for each layer parameter

pkadambi: <p>I want to get optimizer tensors for each parameter in my network.<br>
I can do something like this to extract the optimizer statistics</p>
<pre><code class="lang-auto">parameters = [prm for prm in optimizer.param_groups['params']]
param_ = parameters[0]
state = optimizer.state
statistic1 = state[param_]['exp_avg']
statistic2 = state[param_]['exp_avg_sq']
</code></pre>
<p>Now which parameter does p actually correspond to? I have a transformer network where many of the encoder layers have the same shape, so I won’t be able to rely on shape information alone.<br>
I know that toch.optim and torch.module are independent so this might by tricky.</p>
<p>If I look at the optimizer state dict</p>
<pre><code class="lang-auto">optimizer_state_dict['param_groups']['params']
</code></pre>
<p>is just a list of numbers [0, 1, 2, …]<br>
For example, how would I get the <code>exp_avg</code> for <code>layer1.attention.key.dense.weight</code>?</p>
Conv2D: Forward way slower than Backward

eduardo4jesus: <p>Dear fellows,</p>
<p>I am wondering about the executing time for the forward and backward stages of the convolution 2d operation.</p>
<p><a href="https://colab.research.google.com/gist/ecoelhor/a698f88e352b80e163fe591470e2e363/pytorch-1-7-0-profiling-conv2d.ipynb" rel="noopener nofollow ugc">Check this notebook</a></p>
<p>As far as I know, the forward pass is composed by a single convolution operations. When doing the back propagation, the derivatives w.r.t each of the inputs would be computed with a convolution operation each.  Thus, I was expecting the backward pass to be slower than the forward.</p>
<p>But, according to the tests I did on the notebook (linked above) the backward pass is just a very small fraction of the forward pass.</p>
<p>From the notebook, I have:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/d/ddad2b9135e6c9a5fa41cbbd9ece5a7775bf5768.png" data-download-href="https://discuss.pytorch.org/uploads/default/ddad2b9135e6c9a5fa41cbbd9ece5a7775bf5768" title="Screen Shot 2021-01-30 at 6.08.50 PM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/d/ddad2b9135e6c9a5fa41cbbd9ece5a7775bf5768.png" alt="Screen Shot 2021-01-30 at 6.08.50 PM" data-base62-sha1="vD2xfPZsqoU1ZvKtjiiDROvyak8" width="690" height="115" data-dominant-color="E7E7E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-01-30 at 6.08.50 PM</span><span class="informations">814×136 8.68 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Could someone shed some light on this?</p>

albanD: <p>Hi,</p>
<p>I think you’re missing the following line <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<pre><code class="lang-auto">    backward1 *= 1e3 # From seconds to miliseconds
</code></pre>
<p>Then all the timings will be what you expect.</p>

eduardo4jesus: <p>Thank you so much. I can’t believe I missed that. It’s very embarrassing, what a shame.</p>

albanD: <p>No worries! Took me a while to find it as well <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Pytorch XLA is not at all working on colab or kaggle kernels

SpaceY: <p>Hi Everyone,<br>
I am new to PyTorch and started using it recently for training on GPUs and TPUs.<br>
I am not able to run any of the pytorch NN models on kaggle kernels and Google Colab.<br>
Even the Notebook given on the Pytorch XLA Documentation doesnt work on Colab it just keeps on running for ever<br>
and never shows an result.<br>
<a href="https://colab.research.google.com/github/pytorch/xla/blob/master/contrib/colab/multi-core-alexnet-fashion-mnist.ipynb" rel="noopener nofollow ugc">Notebook: AlexNet on MNIST multicore </a><br>
I even tried to Run on Single Core but speed was unbearable.<br>
Even After This I tried using Pytorch Lightning Same Result It keeps on Running and never stops.<br>
Without Any Error.<br>
Can Anyone help or Guide What am I doing Wrong.<br>
Thanks.</p>
How to add a L1 or L2 regularization to weights in pytorch

zezo: <p>In tensorflow, we can add a L1 or L2 regularizations in the sequential model. I couldn’t find equivalent approach in pytorch. How can we add regularizations to weights in pytorch in the definition of the net:</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(n_feature, n_hidden)   # hidden layer
        """ How to add a L1 regularization after a certain hidden layer?? """
        """ OR How to add a L1 regularization after a certain hidden layer?? """
        self.predict = torch.nn.Linear(n_hidden, n_output)   # output layer

    def forward(self, x):
        x = F.relu(self.hidden(x))      # activation function for hidden layer
        x = self.predict(x)             # linear output
        return x

net = Net(n_feature=1, n_hidden=10, n_output=1)     # define the network
# print(net)  # net architecture
optimizer = torch.optim.SGD(net.parameters(), lr=0.2)
loss_func = torch.nn.MSELoss()  # this is for regression mean squared loss
</code></pre>

Dwight_Foster: <p>There are no l2 or l1 layers in pytorch. Instead you can use the adagrad optimizer found <a href="http://pytorch.org/docs/optim.html#torch.optim.Adagrad" rel="noopener nofollow ugc">here</a>. The weight decay parameter will add a l2 weight decay to your model.</p>
I wanna do A=B.mm(B.t()) to get the upper triangular matrix since the result is symmetric

z_tim: <p>hello! I wanna do this in pytorch A=B.mm(B.t()) to get the gram matrix, in fact the result is a symmetric matrix, which mean I actually can reduce the operations by half to get only the upper triangular matrix, how to do it without using loops? cuz using for loop is extremly slow in torch</p>

z_tim: <p>using triu_indices/tril_indices is extremely slow too(e.g. (64,64,300,300)), please give an answer without using indices and for loop, thanks!</p>

googlebot: <p>Your only hope of beating mm would be to use a specialized low level function - that’s herk from BLAS (cublas or cpu version) I believe. You’d need to write a C++ extension to have python interface &amp; gradients.</p>
<p>Even if you go that route, don’t expect 2x speedup, as arithmetic instructions themselves are cheap, while memory latency and other overheads won’t change.</p>

z_tim: <p>Thanks for your answer!<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Where can I find the source code for mm? Maybe I can modify there directly if possible, just wanna change<br>
for i in range(100):<br>
for j in range(100):</p>
<p>to</p>
<p>for i in range(100):<br>
for j in range(i):</p>

googlebot: <p>No, you won’t be able to modify mm if you think it is about such loops. I’ll point you to sources, but this is just to give you an idea</p>
<p><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/LinearAlgebra.cu#L65" rel="noopener nofollow ugc">addmm_out_cuda_impl</a><br>
<a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/LinearAlgebra.cpp#L455" rel="noopener nofollow ugc">addmm_impl_cpu_</a></p>
<p>note that there are like 5-10 wrappers above these routines in ATen (and mm dispatches to addmm there), and they still dispatch to an external blas library (that will process avx/cuda blocks, instead of a loop like that)</p>
<p>PS if you’re using CUDA, amortization from parallelism may make your efforts have only a minor effect.</p>
Optimization vs architecture

copythatpasta: <p>Which factor is bigger when determining SOTA results with object detectors, optimization or architecture. For example, detectors like YOLO and efficientdet have achieved results with their respective optimizations used in the paper. Could better results be achieved solely by modifying optimization using the same architecture?</p>
<p>I am wondering if research in that area is worth the effort. My intuition tells me that it all comes down to an optimization problem but i would like outside input</p>
Confusing situation in model training

Liana_Baghdasaryan: <p>Although batch_norm is deactivated, lr_rate and momentum is 0, model loss and accuracy changes.What can cause this?</p>

ptrblck: <p>Could you explain the use case you are working on and when exactly the accuracy and loss change?<br>
Are you seeing different losses for the same input batch while the model is in evaluation mode via <code>model.eval()</code>?</p>

Liana_Baghdasaryan: <p>The problem is now solved.<br>
When I save model and load it, model state_dict’s keys start with “module.” word.<br>
And when I wanted to replace my model params with the saved model params, there was a key mismatch. But I informed about it from jupyter notebook, not from actual script. The script didn’t showed me any warnings:)</p>
Split point cloud input into tiles

Eyal: <p>Hello,<br>
I’m currently working on a semantic segmentation task on point clouds, and I’ve bumped onto a memory problem. I have about 30 samples of point clouds that I want to train on, and it crashes with:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/3/833673b4b656e807bb87060a8e844f402ce987f5.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/833673b4b656e807bb87060a8e844f402ce987f5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/833673b4b656e807bb87060a8e844f402ce987f5_2_690x101.jpeg" alt="image" data-base62-sha1="iILd6OpaI5TCkjkSMBZscdwA8zr" width="690" height="101" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/833673b4b656e807bb87060a8e844f402ce987f5_2_690x101.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/833673b4b656e807bb87060a8e844f402ce987f5_2_1035x151.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/3/833673b4b656e807bb87060a8e844f402ce987f5.jpeg 2x" data-dominant-color="2A2A2A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1239×182 69.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Even when I start training just on one sample.</p>
<p>So I’ve figured I need to split the data onto tiles (with overlap) in the CPU and send them one by one to the GPU. I’ve seen some examples of doing so with regular images and pretty confused on how to do so with point clouds.<br>
Are there any elegant methods of doing so in the Data Loader?</p>
<p>Thanks!</p>
RuntimeError: Checkpointing is not compatible with .grad(), please use .backward() if possible

LouisD: <p>Hello everyone,</p>
<p>I ran into a “RuntimeError: Checkpointing is not compatible with .grad(), please use .backward() if possible” in the following codes:</p>
<pre><code class="lang-auto">W = model.layers[0].conv.mlp[-1].weight
train_loss = [loss1, loss2]

for i, t in enumerate(tasks):
    gygw = torch.autograd.grad(train_loss[i], W, retain_graph=True)
    norms.append(torch.norm(torch.mul(Weights.weights[i], gygw[0])))
</code></pre>
<p>Was trying to calculate the gradient of W w r t loss as in <a href="https://github.com/brianlan/pytorch-grad-norm/blob/master/train.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch-grad-norm/train.py at master · brianlan/pytorch-grad-norm · GitHub</a></p>
<p>Thank you very much for any help! Thanks!</p>

LouisD: <p>My model utilized gradient checkpointing in some layers. Close this topic…</p>
Problem in using a code that makes use of PyTorch Encoding package

Mona_Jalal: <p>Please also check here:<br>
After I install the repo using:</p>
<p><code>$ pip install .</code></p>
<p>and run my code, I get the following error:</p>
<pre><code class="lang-auto">    Running setup.py install for torch-encoding ... done
Successfully installed Pillow-8.1.0 certifi-2020.12.5 chardet-4.0.0 idna-2.10 nose-1.3.7 portalocker-2.2.0 requests-2.25.1 scipy-1.6.0 torch-encoding-1.2.2b20210129 torchvision-0.8.2 tqdm-4.56.0 urllib3-1.26.3
(torchenc) mona@goku:~/research/code/PyTorch-Encoding$ python test_demo.py 
Traceback (most recent call last):
  File "test_demo.py", line 2, in &lt;module&gt;
    import encoding
  File "/home/mona/research/code/PyTorch-Encoding/encoding/__init__.py", line 12, in &lt;module&gt;
    from .version import __version__
ModuleNotFoundError: No module named 'encoding.version'



</code></pre>
<p>This is the <code>test_demo.py</code> code:</p>
<pre><code class="lang-auto">import torch
import encoding

# Get the model
model = encoding.models.get_model('DeepLab_ResNeSt269_PContext', pretrained=True).cuda()
model.eval()

# Prepare the image
url = 'https://github.com/zhanghang1989/image-data/blob/master/' + \
      'encoding/segmentation/pcontext/2010_001829_org.jpg?raw=true'
filename = 'example.jpg'
img = encoding.utils.load_image(encoding.utils.download(url, filename)).cuda().unsqueeze(0)

# Make prediction
output = model.evaluate(img)
predict = torch.max(output, 1)[1].cpu().numpy() + 1

# Get color pallete for visualization
mask = encoding.utils.get_mask_pallete(predict, 'pascal_voc')
mask.save('output.png')

</code></pre>
<p>Can you please guide how to fix it?</p>
<p>Here are my pip freeze results. I am aware of <a href="https://github.com/zhanghang1989/PyTorch-Encoding/issues/162" class="inline-onebox" rel="noopener nofollow ugc">ninja: build stopped: subcommand failed. · Issue #162 · zhanghang1989/PyTorch-Encoding · GitHub</a> but I am hoping we can make this work for the newer versions of packages.</p>
<pre><code class="lang-auto">$ pip freeze
certifi==2020.12.5
chardet==4.0.0
idna==2.10
nose==1.3.7
numpy==1.19.5
Pillow==8.1.0
portalocker==2.2.0
requests==2.25.1
scipy==1.6.0
torch==1.7.1
torch-encoding==1.2.2b20210129
torchvision==0.8.2
tqdm==4.56.0
typing-extensions==3.7.4.3
urllib3==1.26.3

</code></pre>
<pre><code class="lang-auto">mona@goku:~$ python
Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0] on linux
$ pip --version
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)

$ lsb_release -a
LSB Version:	core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal
</code></pre>

ptrblck: <p>The error is raised by <code>import encoding</code>, which seems to be a custom script in your repository, so you would have to make sure it’s able to import all modules in its <code>__init__.py</code>.</p>

Mona_Jalal: <p>Thank you. I moved it to home dir and worked.</p>
PyTorch on Linux: can I use a desktop environment while training on GPU?

alanfranz: <p>Hello everybody,<br>
It’s my first post here; maybe it’s a stupid question, but I’ve always used PyTorch on cloud gpu instances. Recently I’m experiencing some issues with spot instances (they get preempted quite quickly) and, since I’d like to tinker a bit more with some models, I’m considering buying a physical GPU (not an extra-easy thing to do, but I think I can get a 3070 or 3080).</p>
<p>The big question is: can I use PyTorch while a run a desktop environment with a compositor on the same GPU (OS is Ubuntu Linux 20.04), or do I need to setup a separate, headless computer just with the GPU? Will the desktop environment take up a lot of available memory?</p>
<p>(of course I won’t play games or run 3D intensive software while I’m traing, but some basic desktop app will be there).</p>
<p>I’ve found some old threads where they recommend to leave the Nvidia GPU for CUDA “all by herself”, is that still the way to go?</p>

Dwight_Foster: <p>I run ubuntu with a cuda gpu will training deep learning models and it doesn’t seem to hurt my performance a lot. I have a 2080 so a 3080 will probably be able to handle it even better. As long as your not doing a lot in the background the os shouldn’t take that much gpu power.</p>
Does transforms.Compose applied to all Images?

Praveen_K: <p>I have Image data and I have custom augmentation classes for transforms.</p>
<pre><code class="lang-auto">transform_custom = [Rescale(256), RandomCrop(224), Resize(224)]
</code></pre>
<p>These classes are passed to  <code>transform</code> and its output to <code>DataLoader</code></p>
<pre><code class="lang-auto">transformed_dataset = Classification(df, transform=transforms.Compose(transform_custom))
dataloader = DataLoader(transformed_dataset, shuffle=True, num_workers=0, batch_size=4).
</code></pre>
<blockquote>
<p>Question:</p>
</blockquote>
<p>Does all Images undergo .all mentioned <code>rescale</code>,  <code>random_crop</code> and  <code>resize</code> transformations or Images and transformations are randomly selected to undergo transformations?</p>
<p>If all the Images undergo all transformations, How can I make selection of Images and selection of transformation random?</p>
<p>As per my observation, all images undergo all transformations from left to right! but the pytorch library undergoes transformation randomly. how can i implement this feature in my script</p>

Dwight_Foster: <p>Yes the transformations you applied to the images are applied to all of them. If you want to have it applied randomly you could use different transformations like randomresizedcrop or something else. Just go into the <a href="https://pytorch.org/docs/stable/torchvision/transforms.html" rel="noopener nofollow ugc">docs</a> and look for transforms with a probability argument those will apply to random images. You can also try to use the</p>
<pre><code class="lang-auto">torchvision.transforms.RandomApply(transforms, p=0.5)
</code></pre>
<p>transform also found in the docs to apply transformations randomly.</p>
Torchserve Model Deployment

Enoch_Sarku1: <p>Hello all,<br>
I am new to torchserve and tried to deploy my own model through the platform. Am working on deploying a traffic sign to an autonomous car.<br>
But when I predict at curl, I get an error that says<br>
code 404<br>
type ModelNotFoundException</p>
<p>Am also not sure all the files I need to make sure it works since my model is custom<br>
I need help. Thanks</p>
Multiple references to nn.Parameter in different modules

jmaronas: <p>Hello.</p>
<p>I would like to know how can I keep a reference to a nn.Parameter in a module and a submodule, as both of them use this parameter and I don’t want to pass them as argument.</p>
<p>For example:</p>
<pre><code class="lang-auto">class SubModule(nn.Module):
    def __init__(self,X):
            super(SubModule, self).__init__()
            self.X = X

class BigModule(nn.Module):
     def __init__(self):
           super(BigModule, self).__init__()
           self.X = nn.Parameter(torch.ones(10,10))
           self.submodules = nn.ModuleList([SubModule(self.X),SubModule2(self.X)])


</code></pre>

googlebot: <p>Python offers no elegant solutions for this, in addition your snippet would create a second X parameter (because <code>nn.Module.__setattr__</code> intercepts nn.Parameter assignments)</p>
<p>I use state object pattern (argument to forward()) for this, this has two pros: 1)works with JIT 2)tensors and parameters are interchangeable. Downsides: 1)state argument must be added to submodules 2)some boilerplate to initialize the state object.</p>
<p>For simpler scripts, I’d consider global variables:</p>
<pre><code class="lang-auto">X = None
class BigModule(nn.Module):
     def __init__(self):
           global X
           super(BigModule, self).__init__()
           self.X = X = nn.Parameter(torch.ones(10,10))
</code></pre>

jmaronas: <p>Thank you. Just wanted to confirm my guess, which seems to match yours.</p>
What does grad_outputs do in autograd.grad?

Kimmo_Ojala: <p>I noticed that when I leave grad_outputs as None in autograd.grad I seem to get back the same gradients as when I set it as a sequence of ones (just 1 x 1 in my  case). But when I compare the resulting gradient tensors with ==, the results are mostly 0 but sometimes 1 although the numbers seem to be exactly the same.</p>
<p>What does grad_outputs actually do in autograd.grad?</p>

alexis-jacq: <p>When I do it:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.rand(5, requires_grad=True)
&gt;&gt;&gt; y = (x**2).sum()
&gt;&gt;&gt; g1 = torch.autograd.grad(y,x,torch.ones(5), retain_graph=True)
&gt;&gt;&gt; g2 = torch.autograd.grad(y,x,None, retain_graph=True)
&gt;&gt;&gt; g1[0]==g2[0]
tensor([ 1,  1,  1,  1,  1], dtype=torch.uint8)
</code></pre>
<p>Can you show your code?</p>
<p>In autograd.grad, if you pass <code>grad_output=None</code>, it will change it into a tensor of ones of the same size than <code>output</code> with the line:</p>
<pre><code class="lang-python">new_grads.append(torch.ones_like(out))
</code></pre>
<p>(that’s here : <a href="https://github.com/pytorch/pytorch/blob/master/torch/autograd/">https://github.com/pytorch/pytorch/blob/master/torch/autograd/</a><code>__</code>init__.py)</p>

Krishna_Garg: <p><a class="mention" href="/u/alexis-jacq">@alexis-jacq</a> Your answer still doesn’t explain the use of grad_output. Can you please explain the case when the grad_output is not ones?</p>

Zayd: <p><a class="mention" href="/u/krishna_garg">@Krishna_Garg</a> While this answer is by no means comprehensive, I have seen <code>grad_outputs</code> used when calculating higher-order derivative, vector products, e.g., Hessian-vector products.</p>
Which dimensions are the real and imaginary parts of an old complex tensor in?

Mahmoud_Abdelkhalek: <p>I’m currently trying to use <code>torchaudio.transforms.Spectrogram</code> with <code>power = None</code> to return the complex spectrogram. The <code>forward</code> method of this class returns a tensor with a shape of <code>(...,2)</code>. I am aware that the last two dimensions represent the real and imaginary parts, but it is not clear from the documentation which one is which and how they represent a complex tensor.</p>
<p>For example, if <code>torchaudio.transforms.Spectrogram</code> returns <code>x</code> with shape <code>(256,256,2)</code>, is <code>x[...,0]</code> the real part or the imaginary part? Also, if <code>x</code> was a complex tensor, then would it be expressed as <code>x[...,0] + x[...,1]j</code> or <code>x[...,1] + x[...,0]j</code>?</p>

yoyololicon: <p>Yes, the first is real and the second is imaginary.</p>

Mahmoud_Abdelkhalek: <p>So it’s <code>x[...,0] + x[...,1]j</code>?</p>