akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to create a linear layer and initialize it with specific weight?

nikiguo93: <p>Hi all. I want to create a linear layer and then initialize it with specific weights(I have a .sav file). Can anybody tell me how to realize it with pytorch?<br>
after doing this I should discard some layers of pretrained alexnet and add this linear layer to the alexnet. Can anybody tell me how to realize that with pytorch?</p>

ptrblck: <p>You can load parameters via:</p>
<pre><code class="lang-python">with torch.no_grad():
    model.layer.weight.copy_(custom_weight_tensor)
</code></pre>
<p>I don’t know exactly what “discard” layers means, but in case you want to remove them you could write a custom model and reuse other layers or replace them with <code>nn.Identity</code> assuming that the shapes of the activations would still match.</p>

nikiguo93: <p>Hi, thank you for your information. I trained a linear regression model and now I want to use these weights and bias from linear regression model to initialize the linear layer. I hope that the linear layer can behave like the linear regression model. how to realize that?</p>

ptrblck: <p>I don’t know how the architecture of the linear regression model looks like, but in case it’s using non-linear activation functions you wouldn’t be able to squeeze this model into a single linear layer.<br>
Otherwise I would assume that training a single linear layer would also work for your use case.</p>
Load multiple models with DistributedDataParallel

thecho7: <p>I found out the imbalance of GPU usages on my implementation.</p>
<pre><code>def train(device, args):
    torch.distributed.init_process_group(backend='nccl', rank=device, world_size=torch.cuda_device_count())
    model_A = A()
    model_B = B()
    
    # Not gonna update A
    model_A.to(device)
    ckpt = torch.load(path)
    model_A.load_state_dict(ckpt['model_state_dict '])
    model_A = torch.nn.parallel.DIstributedDataParallel(model_A, device_ids=[device])
    model_A.eval()

    # Gonna update B
    model_B.to(device)
    model_B = torch.nn.parallel.DistributedDataParallel(model_B, device_ids=[device])
    model_B.train()        

if __name__ == '__main__':
    args = argparse()
    torch.multiprocessing.spawn(train, nprocs=torch.cuda.device_count(), args=(args, ))
</code></pre>
<p>I intend to load both models A and B on GPUs however,<br>
GPU usage tells me only model_A is allocated on GPU:0 and not allocated on the other GPUs.<br>
like below,</p>
<p>GPU 0: 7000MiB / 11019MiB (model A, B)<br>
GPU 1: 4000MiB / 11019MiB (model B)<br>
GPU 2: 4000MiB / 11019MiB (model B)<br>
GPU 3: 4000MiB / 11019MiB (model B)</p>
<p>Please ask me free if you have any unclears.</p>

mrshenli: <p>It could be all processes unintentionally created CUDA context on the default GPU (<code>cuda:0</code>). To avoid this situation, can you try setting <code>CUDA_VISIBLE_DEVICES</code> env var to a different device for each process, so that each process would only see one GPU?</p>

thecho7: <p>I solved this issue by <strong>map_location='cpu’</strong><br>
Loading pretrained model</p>
<pre><code>ckpt = torch.load(path)
</code></pre>
<p>automatically allocates the parameters to GPU:0.<br>
I cannot understand why until now, but it is solved by using map_location while I load the model.</p>
<pre><code>ckpt = torch.load(path, map_location='cpu')
model_A.load_state_dict(ckpt['model_state_dict '])
model_A.to(device)</code></pre>

mrshenli: <blockquote>
<p>I cannot understand why until now, but it is solved by using map_location while I load the model.</p>
</blockquote>
<p>I think this is because, all processes are trying to load the model to <code>cuda:0</code> by default if you don’t set <code>map_location</code> or <code>CUDA_VISIBLE_DEVICES</code>. BTW, does directly setting <code>map_location</code> to <code>device</code> work for you?</p>
Best way to output a modular value?

ZimoNitrome: <p>I am trying to train a model to output the correct angle. I am currently restricting my outputs to [0, 2pi]:</p>
<p><code>2 * torch.pi * torch.sigmoid(logits)</code></p>
<p>My question is: What if the prediction is something like 1.9pi, but the target is 0.1pi. The loss should in theory be something like 0.2pi but in my current situation it will be 1.8pi.</p>
<p>I can not have a modular loss function since the angle is not the actual end product of my targets, but is rather an intermediate step towards computing the final output. Can a model instead output a modular value? I am afraid that performing <code>logits % (2 * torch.pi)</code> will lead to logits spinning out of control.</p>

KFrank: <p>Hi Zimo!</p>
<aside class="quote no-group" data-username="ZimoNitrome" data-post="1" data-topic="135867" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zimonitrome/48/27591_2.png" class="avatar"> ZimoNitrome:</div>
<blockquote>
<p>Can a model instead output a modular value?</p>
</blockquote>
</aside>
<p>Rather than outputting a single value that is your angle, I think it makes<br>
more geometric sense to output two values, <code>x</code> and <code>y</code>, on the ray that<br>
defines your angle.  Then regulate your <code>x</code> and <code>y</code> by pushing them<br>
towards (but not constraining them to lie on) the unit circle with a term<br>
like <code>loss_reg = (1.0 - (x**2 + y**2))**2</code>.</p>
<p>You do now have the redundancy of using two values to represent a<br>
single angle, but geometrically this representation is “unbiased,” so<br>
I think this outweighs any disadvantage of the redundancy.</p>
<p>If, at some point, you need an actual angle, you can use<br>
<code>theta = torch.atan2 (y, x)</code>, but I would avoid converting<br>
<code>x</code> and <code>y</code> to an angle until you actually need to.</p>
<p>Best.</p>
<p>K. Frank</p>
Please tell me what detail part I should study of pytorch to get the requirement

markleo: <p>Hi, guys.</p>
<p>I am new to PyTorch, I have no idea to start with pytorch.<br>
I am a college student, and doing a pytorch homework in limit time, so I can not learn all the knowledge of pytorch.</p>
<p>Let me tell you a little bit about the title:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d6c2deca1e5f6f3361a3b153700d4cee49568d8.png" data-download-href="https://discuss.pytorch.org/uploads/default/0d6c2deca1e5f6f3361a3b153700d4cee49568d8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d6c2deca1e5f6f3361a3b153700d4cee49568d8_2_581x499.png" alt="image" data-base62-sha1="1UJYMDokXPlQwec1T4lvC7teQas" width="581" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d6c2deca1e5f6f3361a3b153700d4cee49568d8_2_581x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d6c2deca1e5f6f3361a3b153700d4cee49568d8_2_871x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d6c2deca1e5f6f3361a3b153700d4cee49568d8.png 2x" data-dominant-color="8ACEF7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1156×994 22 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>the icon1 and icon2 are fixed image (I will provide the fixed icon image in the project, so pytorch can recognize it and record the location sequence) ,</p>
<p>this is a video records the icons’ moving trail,</p>
<p>now I should get the location data of them like this:</p>
<pre><code class="lang-auto">[
   "0s": {
    "icon1": (91.21, 87.77),
    "icon2": (64.32, 67.15)
  },
   "1s": {
    "icon1": (91.09, 85.73),
    "icon2": (62.11, 68.85)
  }
  ...
   "60s": {
    "icon1": (25.54, 80.76),
    "icon2": (13.24, 27.11)
  }
]
</code></pre>
<p>====</p>
<p>Please tell me what detail part I should study of pytorch to get the requirement.</p>
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

Elton_Lobo: <p>I am trying to build a simple Faster RCNN model, I have tested till rpn section, everything works fine until the forward pass of Final_net, I am clueless on what is causing the error. Here is the link to the kaggle notebook, detailed error in the 3rd last cell: <a href="https://www.kaggle.com/el0700/final-imd" class="inline-onebox" rel="noopener nofollow ugc">Final_IMD | Kaggle</a></p>

ptrblck: <p>Which PyTorch version are you using? If an older one, could you update to <code>1.9.1</code> or the nightly and rerun your script? We were hitting a missing shape check when calling into cublas via <code>matmul</code> ops, which might cause this downstream error.</p>

Elton_Lobo: <p>Updated pytorch, still the same error.</p>

ptrblck: <p>Could you post a minimal, executable code snippet to reproduce the issue as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>
How to use only the decoder part from an autoencoder?

ZHAOBO_XU: <p>Hi there! I am using an autoencoder structure in my project. After I trained the net, how can I use the trained decoder part to generate some reconstructed images? Is it possible to create two ‘forward’ function within the network class, like one for the whole and another for the decoder alone?</p>

ptrblck: <p>Yes, it would be possible to split the forward passes and call them separately:</p>
<pre><code class="lang-python"># original forward
def forward(self, x):
    out = self.forward_encoder(x)
    out = self.forward_decoder(out)
    return out
</code></pre>
<p>and later call <code>model.forward_decoder(input)</code> to call the decoder only.<br>
However, since you are skipping the <code>__call__</code> method, no hooks would be called and your use case might thus be limited to the forward/backward pass only (i.e. other tools depending on hooks might not work).<br>
The better approach would be to write the encoder and decoder in separate submodules and call them as:</p>
<pre><code class="lang-python">def forward(self, x):
    out = self.encoder(x)
    out = self.decoder(out)
    return out
</code></pre>
<p>where <code>self.encoder</code> and <code>self.decoder</code> as <code>nn.Module</code>s. Later you could then use <code>out = model.decoder(input)</code>.</p>

ZHAOBO_XU: <p>Thank you so much! It is so helpful <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=10" title=":grinning:" class="emoji" alt=":grinning:"></p>
What are the advantages of using torch.fx vs hooks for feature extractors?

Brando_Miranda: <p>I saw:</p>
<p><a href="https://pytorch.org/docs/stable/fx.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/fx.html</a></p>
<p>I am using hooks to get features for CCA comparison on NNs (<a href="https://github.com/brando90/ultimate-anatome?organization=brando90&amp;organization=brando90" class="inline-onebox" rel="noopener nofollow ugc">GitHub - brando90/ultimate-anatome: Ἀνατομή is a PyTorch library to analyze representation of neural networks</a>). Is there an advantage of moving to migrating to <code>torch.fx</code>? if yes what are they?</p>
<p>Thanks in advance!</p>
<hr>
<p>context:</p>
<ul>
<li>
<a href="https://github.com/moskomule/anatome/issues/25" class="inline-onebox" rel="noopener nofollow ugc">Use FX instead of hooks · Issue #25 · moskomule/anatome · GitHub</a> and</li>
<li>
<a href="https://github.com/brando90/ultimate-anatome?organization=brando90&amp;organization=brando90" class="inline-onebox" rel="noopener nofollow ugc">GitHub - brando90/ultimate-anatome: Ἀνατομή is a PyTorch library to analyze representation of neural networks</a>  and</li>
<li><a href="https://github.com/brando90/ultimate-anatome/issues/9" class="inline-onebox" rel="noopener nofollow ugc">torch.fx · Issue #9 · brando90/ultimate-anatome · GitHub</a></li>
</ul>
Manually set number of batches in DataLoader

nobodykid: <p>Hello, I’m quite new in Pytorch and I have a question</p>
<p>In <code>DataLoader</code>, there’s <code>batch_size</code> args that will determine size of data per batch and number of batches based on total data in the set. However, I’d like to manually control number of batches I’d like to run per epoch. Is there a way I could do this while still using DataLoader?</p>
<p>Thank you</p>

ptrblck: <p>A simple approach would be to use the index of the training loop and call <code>break</code> once you sampled the desired number of batches.<br>
Alternatively, you could also split the <code>Dataset</code> using <code>random_split</code> and the desired length or use a sampler, but I think the first approach would be the easiest, if you just want to change the number of batches.</p>

nobodykid: <p>So, like this?</p>
<pre><code class="lang-auto">for i, x in enumerate(train_dataloader):

    # load x and process

    if i == n_batch:
        break
</code></pre>

ptrblck: <p>Yes, that should work.</p>

nobodykid: <p>It fixed my problem. Thank you very much for the help</p>

mjdmahsneh: <p>See also <a href="https://discuss.pytorch.org/t/how-to-set-the-steps-per-epoch-using-the-dataloader/98841/4" class="inline-onebox">How to set the steps per epoch using the dataloader? - #4 by mjdmahsneh</a></p>
Gather with batch size

javirk: <p>Hello, say I have a tensor <code>c</code> of 3D coordinates with shape (B x P x 3), where B is the batch size, and another one with indexes <code>idx</code> and shape (B x I x 2):</p>
<pre><code class="lang-auto">c = torch.tensor([[[1,1,2],[5,2,3],[6,1,8]]])
idx = torch.tensor([[[0,1],[1,2]]])
</code></pre>
<p>(Here the batch size is 1, just for simplicity).<br>
I want to slice the coordinates according to the idx tensor, so the result should be a tensor with shape (B x I x P x 3) and values</p>
<pre><code class="lang-auto">result[:,0] = torch.tensor([[1,1,2],[5,2,3]])
result[:,1] = torch.tensor([[5,2,3],[6,1,8]])
</code></pre>
<p>I have tried using slicing <code>c[:,idx]</code>, but that only works apparently when B=1, although it adds dimensions I don’t want. I also tried with <code>torch.gather(c, dim=1, index=idx)</code>. What is the way to do this?</p>
<p>Thanks!</p>
How to confirm yolov5 .pt in python into Torch Script

zhangkun: <p>dear all:</p>
<p>i have use the following code to confirm model in python into Torch Script. but, when i transfer a yolov5 model, there is a error “Could not get name of python class object”. what is the problem?</p>
<p><em>import torch</em><br>
<em>import torchvision</em></p>
<p><em># An instance of your model.</em><br>
<em>model = torchvision.models.resnet18()</em></p>
<p><em># An example input you would normally provide to your model’s forward() method.</em><br>
<em>example = torch.rand(1, 3, 224, 224)</em></p>
<p><em># Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.</em><br>
<em>traced_script_module = torch.jit.trace(model, example)</em></p>
<p><em>traced_script_module.save(“traced_resnet_model.pt”)</em></p>

ptrblck: <p>I think you might be passing an unsupported object to <code>torch.jit</code> as described e.g. in <a href="https://github.com/pytorch/pytorch/issues/52920">this issue</a>.</p>

zhangkun: <p>thank you. i have found that in the yolo folder, there is a py to transfer the pt model.</p>

zhangkun: <p>i have found that in the yolo folder, there is a py to transfer the pt model.</p>
<p>thank you.</p>
Batch conv2d with multiple kernels of batch size

Paul_Hongsuck_Seo: <p>Hi, I wonder if there is a way to apply conv2d to input batch with multiple kernels.<br>
I want to apply conv2d operation for each input of the minibatch with its corresponding kernel where there are B input feature maps and B kernels within each minibatch.</p>
<p>This is similar to torch.bmm operation where each matrix in a batch is multiplied by a different weight matrix. I found this backend function ‘THCudaTensor_conv2Dmm’. Is this function related to what I want? If so, how can I access it in my python code?</p>
<p>Thanks,<br>
Paul</p>

ptrblck: <p>You can use the <code>groups</code> parameter in <code>nn.Conv2d</code> to achieve this. Try to set <code>groups=in_channels</code> and each input channel will have its own set of filters.<br>
Note that both <code>in_channels</code> and <code>out_channels</code> have to be divisible by <code>groups</code>.</p>

Paul_Hongsuck_Seo: <p>This is cool and I think it resolves my issue. I will try that. Thank you!</p>

paganpasta: <p>Can you please explain how I can group such that the sample code works?</p>
<pre><code class="lang-auto">A = torch.randn(5, 512, 7, 7) # minibatch, in_channels, iH, iW
B = torch.randn(5, 512, 1, 1) # minibatch, filter_channels, h, w 

# Aim is to run conv2d at batch level
out = F.conv2d(A, B)
# However, It applies all the filters to all the batch samples of A
# Desired output shape = [5, 1, 7, 7]
# Obtained output shape = [5, 5, 7, 7]
</code></pre>
<p>Using a groups = <code>512</code> or <code>5</code> raises RuntimeError due to shape mismatch.</p>

paganpasta: <p>I’ll answer my own query</p>
<pre><code class="lang-auto">A1 = A.view(A.shape[0]*A.shape[1], A.shape[2], A.shape[3]).unsqueeze(0)

out = F.conv2d(A1, B, groups=B.shape[0]).permute(1, 0, 2 ,3)
print(out.shape)

# Result: [5, 1, 7, 7]
</code></pre>
<p>Verified by computing in <code>batch</code> and <code>individually</code> for each of the <code>5</code> entries in the batch and comparing the results.</p>

dansontong: <p>the filter should be in shape ( filter_channels, in_channels, h, w)</p>
Torch.matmul problem

hcleung3325: <p>I have two tensor torch.randn(3, 10, 4) and torch.randn(4, 10, 3)<br>
I want to multiply them to become a (10,10) tensor</p>
<p>However, it has error when I use matmul, may I know which part is wrong?<br>
tensor1 = torch.randn(3, 10, 4)<br>
tensor2 = torch.randn(4, 10, 3)<br>
print(torch.matmul(tensor1, tensor2).size())</p>

KFrank: <p>Hi Hcleung!</p>
<aside class="quote no-group" data-username="hcleung3325" data-post="1" data-topic="135794" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hcleung3325/48/38499_2.png" class="avatar"> hcleung3325:</div>
<blockquote>
<p>I have two tensor torch.randn(3, 10, 4) and torch.randn(4, 10, 3)<br>
I want to multiply them to become a (10,10) tensor</p>
</blockquote>
</aside>
<p>You may use the Swiss-army knife of generalized tensor-multiplication<br>
operators, <code>torch.einsum()</code>, to “contact” the desired indices:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; tensor1 = torch.randn(3, 10, 4)
&gt;&gt;&gt; tensor2 = torch.randn(4, 10, 3)
&gt;&gt;&gt; torch.einsum ('kil,ljk -&gt; ij', tensor1, tensor2).shape
torch.Size([10, 10])
</code></pre>
<p>However, you haven’t told us what sort of “multiplication” you want<br>
to perform, so you should check that <code>einsum()</code> is doing what you<br>
want and expect.</p>
<p>Best.</p>
<p>K. Frank</p>
T_max and T_cur in CosineAnnealingLR

sigma_x: <p>In pytorch <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CosineAnnealingLR.html#torch.optim.lr_scheduler.CosineAnnealingLR" rel="noopener nofollow ugc">documentation</a> T_cur is ‘current epoch’, and T_max is the maximum number of iterations. I found these definitions a bit confusing, because typically ‘iterations’ means (dataset_size/batch_size). So for example, if I have batch_size=1, dataset_size=100, does it mean that T_max=100 should I choose to keep the learning rate decaying until the termination of the algorithm?</p>
There is no sampling occuring when I apply offset and n_frames in the load function

ThiruRJST: <h1><a name="is-n_frames-equivalent-to-librosas-duration-1" class="anchor" href="#is-n_frames-equivalent-to-librosas-duration-1"></a>Is n_frames equivalent to librosa’s duration?</h1>
<p>pytorch_version: 1.9.0+cu111<br>
torchaudio_version: 0.9.0</p>
<p>I’m currently working in audio classification task where I found that if the audio file is provided entirely in <strong>load_audio()</strong> function, it provides the stereo output of the sampled version with default sampling_rate</p>
<p>But, if i provide offset and n_frames into the function, it doesn’t resample it.</p>
<p>Actually I have a doubt that is n_frames is equivalent to librosa’s duration.</p>
<p>In my case the offset is 30 and duration is 10 seconds</p>
<p><strong>reading the audio file with offset and n_frames</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1d9dfddcfb0253ce466eaf784c78a42a3fd136c.png" data-download-href="https://discuss.pytorch.org/uploads/default/f1d9dfddcfb0253ce466eaf784c78a42a3fd136c" title="Screenshot from 2021-10-30 16-09-33"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f1d9dfddcfb0253ce466eaf784c78a42a3fd136c_2_690x388.png" alt="Screenshot from 2021-10-30 16-09-33" data-base62-sha1="yvvRAlKcTjDtFFuQ3CWpNiMfpRG" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f1d9dfddcfb0253ce466eaf784c78a42a3fd136c_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f1d9dfddcfb0253ce466eaf784c78a42a3fd136c_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f1d9dfddcfb0253ce466eaf784c78a42a3fd136c_2_1380x776.png 2x" data-dominant-color="2C2D28"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-10-30 16-09-33</span><span class="informations">1920×1080 202 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

nateanl: <p>Hi <a class="mention" href="/u/thirurjst">@ThiruRJST</a>, thanks for sharing it. In torchaudio, the <code>frame_offset</code> and <code>num_frames</code> are in frames or samples, thus it’s different from librosa’s duration argument.</p>
<p>If you want to extract 10 seconds of audio from the 30th second, you can call the method <code>torchaudio.load(file_path, frame_offset=30*sample_rate, num_frames=10*sample_rate)</code>.</p>
Fine-tune a customized pretrained language model

valgi0: <p>Hi there!</p>
<p>I am here because I am struggling with this problem: how to best fine-tune a pretrained language model such as BERT, BART, RoBERTA, and so on, but with architectural or data flow customization.</p>
<p>I would like to have your opinions if you have experience creating a kind discussion on that topic.</p>
<p>I try to better explain the problem. I have a pretrained model called BART that is a model for summarization (and text generation). I want to alter its inner structure in different ways to study its behavior:</p>
<ul>
<li>Transforming the data flow without changing the weights. Full match with the checkpoint. I guess the weights now should be fine-tuned to work with this new data flow. For example, I want the encoder to process more inputs before calling the decoder instead of calling it after each input.</li>
<li>Adding new custom layers with new weights to train. For example, I want to add a linear projection (<code>nn.Linear()</code>) after the encoder. In this case, I have a new matrix of fresh weights that needs to be trained from scratch.</li>
<li>Finally, I can alter data flow and add new custom layers at the same time.</li>
</ul>
<p>In all of these scenarios, I need an efficient strategy to fine-tune the model. If the learning rate is too high or small model does not converge.</p>
<p>I thought it could be useful to use a different learning rate for the custom layer, maybe a different scheduler, or just freeze some layers. The possibilities are almost infinite.</p>
<p>Do you have any experience to share on these topics? It could be helpful for me and everyone in this community <img src="https://discuss.pytorch.org/images/emoji/apple/relaxed.png?v=10" title=":relaxed:" class="emoji" alt=":relaxed:"></p>
<p>Thank you. Have a nice day <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Why does the shuffle operation take a lot of time?

QuanHei: <p>I have millions of 16x16 pixels images and when I set shuffle=False, the training is fast.<br>
But…When I set shuffle=True,  so slow…<br>
How to speed up?</p>

JuanFMontesinos: <p>It’s a matter of how do you save images rather than dataloader itself.<br>
Shuffling just shuffles a list of indices. It seems your system is slower accessing non-sequential lists of indices.<br>
How do you do so?</p>

albanD: <p>Hi,</p>
<p>Do you mean slow in runtime? Or slow to learn?</p>

QuanHei: <p>Thanks for you reply. I mean slow in run time.</p>

QuanHei: <p>Thanks for your reply, I have solved the problem. Thank you</p>

Lizhen_Ji: <p><a class="mention" href="/u/quanhei">@QuanHei</a> Hi, same problem here, how do you solve the problem?<br>
I have a very large dataset(1200000 images) and training with shuffle=False is fast while training with shuffle=True is slower and slower.</p>
<p>Any suggesstion or help would be appreciated, thanks!</p>
How to free up vision::model's memory when I don't use it?

x1290148: <p>When I finished predicting, I found out that my program took up 8G. How do I free it up if I don’t make a prediction next? I tried using detach delete c10::cuda::CUDACachingAllocator::emptyCache().<br>
But it doesn’t seem to help with memory.</p>
<pre><code class="lang-auto">    vision::models::WideResNet50_2 module_wideresnet_50_;//https://github.com/pytorch/vision
	auto anomaly_features = torch::jit::load("D:\\patchcore_features.pt");
	auto ap = anomaly_features.attr("feature").toTensor().to(torch::kCUDA);
	torch::load(module_wideresnet_50_, "D:\\patchcore_model.pt");
	module_wideresnet_50_-&gt;eval();
	module_wideresnet_50_-&gt;to(torch::kCUDA);
</code></pre>
Why torch.nn.DataParallel causes unexpected CUDA OOM?

Olivier-CR: <p>Hi,</p>
<p>I’m using <code>torch.nn.DataParallel</code> to do single-node data parallelism , and I’m wondering the following: how should the DataLoader batch be scaled?</p>
<p>I’m asking since I have a code running fine with batch 16 on a T4 GPU, but doing CUDA OOM with batch 4<em>16 = 64 (and even with 48!) with <code>torch.nn.DataParallel</code> over 4x T4 GPUs. Is torch.nn.DataParallel doing anything weird with the memory, so that it has less memory available than N</em>1 GPU memory? or is <code>torch.nn.DataParallel</code> already applying a scaling rule so that the dataloader batch is the per-GPU batch and not the SGD-level batch? (I don’t think that’s the case as the doc says it “splits the input across the specified devices by chunking in the batch dimension”)</p>
<p><em><strong>note</strong>:I know <a href="https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html" rel="noopener nofollow ugc">PyTorch recommends DDP even for single-node data parallel</a>, but honestly I’m not smart enough to figure out how to use all those torchrun/torch.distributed/launch.py tool, MPI, local_rank things and <a href="https://discuss.pytorch.org/u/olivier-cr/activity/topics">couldn’t make DDP work after a week and 7 issues opened</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></em></p>

techkang: <p><code>nn.DataParallel</code> will transfer the data to GPU0 and scatter latter. So it is easy to cause OOM problems. You can refer to <a href="https://kangsheng.xyz/2020/08/13/pytorch%E4%B8%AD%E5%8D%95%E6%9C%BA%E5%A4%9A%E5%8D%A1%E8%AE%AD%E7%BB%83/" rel="noopener nofollow ugc">pytorch ddp</a> to understand how to leverage <code>DDP</code> if you can read Chinese.</p>

Olivier-CR: <p>ok so if I understand correctly, in nn.DataParallel, all the cluster data must be able to fit in one GPU? That does not make any sense right? Because if all the data fits in one GPU, people would not use data parallelism in the first place <img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=10" title=":upside_down_face:" class="emoji" alt=":upside_down_face:"> <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_raised_eyebrow.png?v=10" title=":face_with_raised_eyebrow:" class="emoji" alt=":face_with_raised_eyebrow:"></p>

ptrblck: <p>The data would still be split in its batch dimension and the forward activations would thus be shared between all devices, which are usually much larger than the input data.<br>
However, <a class="mention" href="/u/techkang">@techkang</a> is right and the scatter/gather ops from the default device create a memory imbalance which is why we recommend the usage of DDP (besides DDP also being faster).</p>
How to select particular elements of a 3D tensor based on indices of last dimension in torch

jack_feinmann: <p>I have a tensor of size 3 x 2 x 3 like<br>
[[[0, 11, 12],<br>
[3, 41, 51]]</p>
<p>[[10, 1, 21],<br>
[31, 4, 51]]</p>
<p>[[40, 14, 2],<br>
[34, 44, 5]] ]</p>
<p>I want to select elements of [0, 1, 2] applied to the last  dimension making it<br>
[[0, 3] , [1, 4], [2, 5]]</p>
<p>could you assist me please how to do it? thanks</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">x = torch.tensor([[[0, 11, 12],
                   [3, 41, 51]],
                  [[10, 1, 21],
                   [31, 4, 51]],
                  [[40, 14, 2],
                   [34, 44, 5]]])

out = x[torch.arange(x.size(0)), :, [0, 1, 2]]
print(out)
&gt; tensor([[0, 3],
          [1, 4],
          [2, 5]])
</code></pre>
How to create a mask tensor

ck172: <p>I have a list that contains different sizes of tensors so I used torch.nn.utils.rnn.pad_sequence to the list. For example, the original data is<br>
grp_pt_data_list = [<br>
[[1,2,1],[1,1,5],[2,1,3],[1,2,1],[1,1,1],],<br>
[[1,2,1],[2,3,2],[1,3,2]],<br>
[[1,3,1],[2,4,2],[1,1,2]]]<br>
after padding the tensor is<br>
tensor([[[[1., 1., 2., 1., 1.],<br>
[1., 2., 1., 0., 0.],<br>
[1., 2., 1., 0., 0.]],<br>
[[2., 1., 1., 2., 1.],<br>
[2., 3., 3., 0., 0.],<br>
[3., 4., 1., 0., 0.]],<br>
[[1., 5., 3., 1., 1.],<br>
[1., 2., 2., 0., 0.],<br>
[1., 2., 2., 0., 0.]]]], device=‘cuda:0’)<br>
The length becomes maximum value 5. I wanted to apply a mask to the result of a nn.Conv2d(fea_dim, 4, kernel_size=(1,1)), so that the padded zeros could be reset to 0, and the other values will remain. Expected mask will be tensor([[ True,  True,  True,  True,  True],<br>
[ True,  True,  True, False, False],<br>
[ True,  True,  True, False, False]])<br>
How should I create the mask tensor and apply the mask to conv2d results without a for loop? Or is there any other better way to not calculating the padded values?</p>

my3bikaht: <p>it’s hard to understand what your padding does, but as for mask, you can just do mask with 1 and 0, then element-wise multiplication. Yet, why can’t you skip padding and mask if you have kernel size 1?</p>

ck172: <p>it’s because the input list have irregular sizes so I need to pad them to form a tensor</p>
Conversion of (16,7,2,16,768) to (16,7,2,768) using CNN

sandeep1: <p>Hi, i am trying to Convert tensor of (16,7,2,16,768) to (16,7,2,768) using CNN. How can i apply CNN on only last two layers so that the value of the other dimensions are untouched.</p>
<p>Thank you.</p>

ptrblck: <p>Could you explain a bit what “untouched” would mean in this context?<br>
If you are concerned about the size of the other dimensions, you could use e.g. <code>nn.Conv3d</code> with a kernel size of <code>16</code> for the second dimension of the kernel and <code>squeeze</code> this dimension afterwards:</p>
<pre><code class="lang-python">x = torch.randn(16, 7, 2, 16, 768)
conv = nn.Conv3d(7, 7, kernel_size=(1, 16, 1))
out = conv(x)
print(out.shape)
&gt; torch.Size([16, 7, 2, 1, 768])
out = out.squeeze(3)
print(out.shape)
&gt; torch.Size([16, 7, 2, 768])
</code></pre>
<p>However, this would of course use all provided values of the input tensor.</p>
How to get around module having parameters that were not used in producing loss error?

JimW: <p>There are 2 blocks (A and B) in my network architecture, and I defined an optimizer_A and loss_A to update module_A, and also defined optmizer_B and loss_B to update module_B.</p>
<p>I hope to disable any updating of module_B in the first 3 epochs, while allowing both modules to be updated after the 3rd epoch.  What I am currently doing is:</p>
<pre><code class="lang-auto">forward(x):
   if self.train and self.epoch&gt;3:
         x=x+module_B(x)
   prediction = module_A(x)
   return prediction
...

loss_A = calculate_loss_A(...)
if epoch&gt;3:
     loss_B = calculate_loss_B(...)
...
if epoch&gt;3:
    loss_A.backward(retain_graph=True)
    loss_B.backward()
else:
    loss_A.backward()
if epoch&gt;3:
    optimizer_A.step()
    optimizer_B.step()
else:
    optimizer_A.step()
</code></pre>
<p>I think the above code was correct to illustrate what I wanted to do. Before adding the “epoch&gt;3” condition, my code worked fine. but after adding it, I got this error message:<br>
<code>RuntimeError:  Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument 'find_unused_parameters=True' to 'torch.nn.parallel.DistributedDataParallel', and by making sure all 'forward' function outputs participate in calculating loss.  If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's 'forward' function.</code></p>
<p>Could someone provide some guidance which part is wrong in my correct implementation? How to get around this error?</p>
Jit.script Tacotron2

kots: <p>Hello, i have successfully managed to convert NVIDIA Tacotron2 using torch.jit.script. However, when i try to run it on an Android Device i get a <strong>RuntimeError: NNPACK SpatialConvolution_updateOutput failed</strong><br>
Below is the exact error:<br>
2020-05-29 16:25:58.166 24263-24427…/AndroidRuntime:   File “…/python3.6/site-packages/torch/nn/modules/conv.py”, line 207, in forward<br>
self.weight, self.bias, self.stride,<br>
_single(0), self.dilation, self.groups)<br>
return F.conv1d(input, self.weight, self.bias, self.stride,<br>
~~~~~~~~ &lt;— HERE<br>
self.padding, self.dilation, self.groups)<br>
RuntimeError: NNPACK SpatialConvolution_updateOutput failed</p>
<p>Jit is created with PyTorch 1.5 and python 3.6.10<br>
The Android version is 9.<br>
Any idea on how to resolve this problem?</p>

smth: <p>is it possible that it’s running out of memory, or something?</p>

shenqiti: <p>that’s true, thank you for your suggestion!</p>
Fast operation for separating values of a tensor into two tensors

Sia_Rezaei: <p>I’m looking for a fast operation than can separate the positive and negative values of a tensor X to two tensors of the same size.<br>
One way to do this is using <code>maximum</code>  and <code>minimum</code> like below:</p>
<pre><code class="lang-auto">neg = torch.minimum(X, 0)
pos = torch.maximum(X, 0)
</code></pre>
<p>In the above solution, each element of X gets compared to 0 twice. But really the comparison only needs to happen once, because if you know X &lt; 0 is true then you already know X &gt; 0 is False.<br>
So you can cut the computation in half.<br>
If we call that operation <code>sieve</code>, then it would look like this:<br>
<code>pos, neg = torch.sieve(X, condition=(X &gt; 0))</code></p>
<p>Is there an operation in torch that can achieve that?</p>

eqy: <p>Would <code>torch.where</code> work for you? <a href="https://pytorch.org/docs/stable/generated/torch.where.html" rel="noopener nofollow ugc">torch.where — PyTorch 1.10.0 documentation</a></p>
<p>However, I’m not familiar with the implementation details/potential performance tradeoffs, so you might want to check the final speed of whatever approach you go with (e.g., just because one approach only does a single read of data/comparison doesn’t mean it is <em>guaranteed</em> to be faster).</p>

Sia_Rezaei: <p><code>where</code> kinda does the inverse of what I’m looking for.<br>
My function takes one tensor as input and returns two tensors. <code>where</code> does the opposite.</p>

eqy: <p>Good catch, in that case I’m not sure there is a clear alternative other than computing a mask and then reusing it for indexing (which would require more reads than the min/max approach anyway).</p>
<p>Is this a performance bottleneck in your application?</p>

Sia_Rezaei: <p>Yes, it is an operation that is performed on feature maps of each layer. Like an activation function.<br>
It would make my runs 10~15% faster overall.</p>

Sia_Rezaei: <p>Any idea where to start if I were to write a custom CPU and kernel code for this?</p>

Sia_Rezaei: <p>Created a feature request for it here: <a href="https://github.com/pytorch/pytorch/issues/67745" class="inline-onebox" rel="noopener nofollow ugc">A `sieve` operation for separating values of a tensor into two tensors based on a condition. · Issue #67745 · pytorch/pytorch · GitHub</a></p>
Fine-tuning detection network problem

Yangmin: <p>Hello.</p>
<p>I am trying to finetune the object detection task.</p>
<p>I am using the pre-trained Faster R-CNN using ResNet50 as a backbone.</p>
<p>Usually, the object detection network is trained with the big scene images.<br>
But when I feed in the cropped person images in the network backbone(ResNet50) while fine-tuning<br>
the Faster R-CNN network with scene images, I figured out performance gradually drops.</p>
<p>Even though I freezed all the layers, the performance drops.</p>
<p>I think it is because Faster R-CNN network layers(conv, fc, BN layer) is trained with the scene images<br>
and also BN layer(running_mean,running_var) is calculated with the scene images, so the values are fit for scene images.<br>
But if I feed in the cropped person images during training, even if I freeze all layers,<br>
the BN layer(running_mean,running_var) is calculated according to mini-batches(scene + cropped person in this case) which are going to be used in inference.<br>
So, I think the BN layer(running_mean,running_var) is corrupted?? and finally, the network inference drops.</p>
<p>Is this correct??</p>
Torchloader: why I delete some prepocessing transforms then code can't work?

Igo312: <p>As title said my transform list is as following</p>
<pre><code class="lang-auto">transforms_list += [
            random_flip,
            random_resize,
            random_crop,
            filter_hv,
            anchor_target,
                   ]
</code></pre>
<p>and <code>anchor_target</code> is a assign function and I want to delete it such as</p>
<pre><code class="lang-auto">transforms_list += [
            random_flip,
            random_resize,
            random_crop,
            filter_hv,
            #anchor_target,
                   ]
</code></pre>
<p>Of course I define my own <code>collate_fn</code> and the return is a <code>dict</code>. And when I set <code>num_workers=0</code> code works fine, but one I set <code>num_workers&gt;0</code>. It will cause error that</p>
<blockquote>
<p>dataloader worker pid …， …exited unexpectedly</p>
</blockquote>
<p>that’s really weird.</p>
<p>PS: the return dict’s key is <code>string</code>, it’s value has <code>np.ndarray</code>, <code>string</code>, <code>bool</code> types. Do I return some wrong type in <code>collate_fn</code>?</p>
[Documentation] FX module is for Pass Writers

anjali-chadha: <p>I am going through the documentation of recently released FX module to Pytorch release 1.10.</p>
<p>Can someone explain what does “pass writers” mean in this context?</p>
<blockquote>
<p>FX is a toolkit for pass writers to facilitate Python-to-Python transformation of <code>nn.Module</code> instances.</p>
</blockquote>
<p>I tried looking up the term “pass writers” with no relevant result that can tie it back to FX module.</p>
<p><a href="https://github.com/pytorch/pytorch/blob/master/torch/fx/OVERVIEW.md" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/fx/OVERVIEW.md</a></p>
<p>Thanks!</p>

jfix: <p>It’s just meant to reference a person (a “writer”) who writes some transformation (a “pass”) for the FX IR. E.g. relevant wiki page on a compiler that does multiple passes:</p>
<aside class="onebox wikipedia" data-onebox-src="https://en.wikipedia.org/wiki/Multi-pass_compiler">
  <header class="source">

      <a href="https://en.wikipedia.org/wiki/Multi-pass_compiler" target="_blank" rel="noopener nofollow ugc">en.wikipedia.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:300/304;"><img src="//upload.wikimedia.org/wikipedia/commons/thumb/7/7f/Multi-passcompiler.png/300px-Multi-passcompiler.png" class="thumbnail" width="300" height="304"></div>

<h3><a href="https://en.wikipedia.org/wiki/Multi-pass_compiler" target="_blank" rel="noopener nofollow ugc">Multi-pass compiler</a></h3>

<p>A multi-pass compiler is a type of compiler that processes the source code or abstract syntax tree of a program several times. This is in contrast to a one-pass compiler, which traverses the program only once. Each pass takes the result of the previous pass as the input, and creates an intermediate output. In this way, the (intermediate) code is improved pass by pass, until the final pass produces the final code.
 Multi-pass compilers are sometimes called wide compilers,[citation needed] referrin...</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


anjali-chadha: <p>Appreciate your ELI5 explanation, <a class="mention" href="/u/jfix">@jfix</a>! Thank you.</p>
Functional Derivative Discontinuity

cw-tan: <p>Hi, I tried using Pytorch’s autograd to calculate functional derivatives. For the case of a periodic function, a=cos^2(x), I considered an integral A=\int a^6, which should have a functional derivative of dA/da = 6a^5. The code to perform it is as follows.</p>
<pre><code class="lang-auto">a = torch.tensor(np.cos(np.linspace(0, 2*np.pi, 10000, endpoint=False))**2, requires_grad=True).float()
A = torch.trapz(a**6)
b = torch.autograd.grad(A, a)[0] 
plt.plot(b, 'b', label = 'autograd')
plt.plot(6*a.detach().numpy()**5, '--r', label = 'expected')
plt.legend()
plt.show()
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/8/78cf25936a562468a7d74135be8c25019d9e19ce.png" alt="discontinuity" data-base62-sha1="heJ6AWpXtk3lW64PeYIMVaUAyN8" width="342" height="234"></p>
<p>The expected result agrees with the autograd result for the most part, but there’s some strange behavior at the boundaries. Increasing the number of grid points for the integration doesn’t seem to make this problem go away (but I haven’t tested this extensively). After testing a few examples, I noticed that it went down to about half of the expected value at the boundaries. Does anybody have an idea of how this problem arises and how it can be fixed? Thank you!</p>

cw-tan: <p>It seems to be fixed by replacing the <code>torch.trapz()</code> with <code>torch.sum()</code>. The discontinuity goes away when the second line is <code>A = torch.sum(a**6)</code> instead. I’m still not sure why this is the case though.</p>

albanD: <p>Isn’t this due to the border condition of the trapz function?</p>

cw-tan: <p>Thanks, that’s probably the case. However, I’m still somewhat puzzled as to how I should think about it since both methods (sum and trapz) are numerical approximations to what theoretically is a continuous integral. I’m guessing this wouldn’t be a problem if I had used periodic boundary conditions and an auto-differentiable implementation of trapz that handled the boundaries correctly?</p>
How do I find Framework Version of .pt/.pth Model File?

sheteng: <p>Hi, I tried</p>
<pre><code class="lang-auto">model=torch.load("densenet121.pth")
model._version
</code></pre>
<p>which gives 1.<br>
First, I am not sure if this 1 means the major version of the torch or the version of the model.<br>
Even if it is the major version of the torch, the number is not useful.</p>
<p>Is there a way to find out which 1.x the model file is using?</p>
<p>Thanks</p>

tom: <p>No, this is very likely not the PyTorch version, but an internal version of the model.<br>
The recommendation is to <a href="https://pytorch.org/docs/stable/notes/serialization.html#saving-and-loading-torch-nn-modules" rel="noopener nofollow ugc">not serialize models but only state dicts</a> for somewhat related reasons (Python takes the class from the source code and the object from the serialization and this gives rather not-nice-to-debug failure modes).</p>
<p>It’s probably not that useful to you now, but if you save state dicts instead of models, the backward compatibility between PyTorch versions is quite reasonable.</p>
<p>Best regards</p>
<p>Thomas</p>
How to accumulate gradients with Opacus

deponce: <p>Hey everyone, I attempt to accumulate gradients in training to save GPU memory.</p>
<p>The training loop works quite well without privacy_engine</p>
<pre><code class="lang-auto">opt.zero_grad()
for i, (input, target) in enumerate(dataset):
    pred = net(input)
    loss = crit(pred, target)
    # one graph is created here
    loss.backward()
    # graph is cleared here
    if (i+1)%10 == 0:
        # every 10 iterations of batches of size 10
        opt.step()
        opt.zero_grad()
</code></pre>
<p>However, when I attach the  privacy_engine to the optimizer, CUDA is out of memory.<br>
Do anyone know how to solve this problem? Thanks in advance.</p>

ffuuugor: <p>Hi!<br>
It is expected that Opacus has a certain memory overhear. At the very least, we have to store per-sample gradients for all model parameters - that alone increases the memory required to store gradient by the factor of batch size.</p>
<p>In order to address that I suggest using <code>virtual_step()</code> method in the optimizer.<br>
It does gradient clipping and accumulation (thus saving memory), but doesn’t do the actual optimizer step.<br>
Your code would look smth like this:</p>
<pre><code class="lang-auto">opt.zero_grad()
for i, (input, target) in enumerate(dataset):
    pred = net(input)
    loss = crit(pred, target)
    # one graph is created here
    loss.backward()
    # graph is cleared here
    if (i+1)%10 == 0:
        # every 10 iterations of batches of size 10
        opt.step()
        opt.zero_grad()
    else:
        opt.virtual_step()
</code></pre>
<p>For more examples using virtual_step see our Cifar10 tutorial: <a href="https://github.com/pytorch/opacus/blob/main/examples/cifar10.py" class="inline-onebox" rel="noopener nofollow ugc">opacus/cifar10.py at main · pytorch/opacus · GitHub</a></p>