akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
MobileNet-V2 PyTorch implementation

Aryan_Asadian: <p>I was trying to re-implement MobileNet-V2 according to the <a href="https://openaccess.thecvf.com/content_cvpr_2018/papers/Sandler_MobileNetV2_Inverted_Residuals_CVPR_2018_paper.pdf" rel="noopener nofollow ugc">paper</a>. Based on Table 2(Page 5) of the paper,  after the sequence of Inverted residual blocks, there should be 2 other regular convolutional layers. But in <a href="https://github.com/pytorch/vision/blob/62e185c7ee6df9938809bc0aec353bbcad1f1223/torchvision/models/mobilenetv2.py#L31" rel="noopener nofollow ugc">Pytorch’s implementation</a>, the second regular convolutional layer(last row in table 2) has been removed. What is the reason for this modification?</p>

ksz: <p>I am not sure, but it seems that everything is correct. Last row: <code>conv2d 1x1</code> is just a classifier. You can use the convolution layer with filter 1x1 instead of the Linear layer. The output will be the same - for a mathematical point of view, they are the same operations. I suppose that authors use that kind of format in the table to keep the same conventions - without introducing a new type of layer.</p>
<p>See example:</p>
<pre><code class="lang-auto">
number_of_classes = 1000
batch_size = 512
i = torch.rand(batch_size, 1280, 7, 7)

class NetworkA(torch.nn.Module):
    def __init__(self):
        super(NetworkA, self).__init__()
        self.linear = torch.nn.Linear(1280, number_of_classes)
        
    def forward(self, x):
        x = torch.nn.functional.adaptive_avg_pool2d(x, (1, 1)).reshape(x.shape[0], -1)
        return self.linear(x)
    
class NetworkB(torch.nn.Module):
    def __init__(self):
        super(NetworkB, self).__init__()
        self.conv = torch.nn.Conv2d(1280, number_of_classes, kernel_size=1)
        
    def forward(self, x):
        x = torch.nn.functional.adaptive_avg_pool2d(x, (1, 1))
        return self.conv(x).reshape(x.shape[0], -1)    
    
netA = NetworkA()
print(netA(i).shape)

netB = NetworkB()
print(netB(i).shape)
</code></pre>
<p>The result:</p>
<pre><code class="lang-auto">torch.Size([512, 1000])
torch.Size([512, 1000])
</code></pre>
MultiHeadAttention not working in Sentiment Classifier

Michael_Ringer: <p>Hey,</p>
<p>i currently want to implement a imdb sentiment classifier with a transformer (encoder block). When i remove the encoder blocks from the network i get better results as with them.</p>
<p>And when i plot the attention matrix which as far as i know should show a vertical line just shows some straight lines.</p>
<p>Colab link:</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20210219-080406-RC00_358389976" class="site-icon" width="16" height="16">
      <a href="https://colab.research.google.com/drive/1yCtnhyudbdnErkgv5I5AyDz32N_C8c_h?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://colab.research.google.com/drive/1yCtnhyudbdnErkgv5I5AyDz32N_C8c_h?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Recommended practice of using tensorboard in multi-gpu training

CDhere: <p>I’ve been using DDP for all my distributed training and now would like to use tensorboard for my visualization/logging. The only solution I can think of is to use “gather” in the rank 0 process each time I want to log an item to the board, since each process/GPU only has a subset of the data and statistics. Apparently this is somewhat cumbersome and I’m not sure if this hurts distributed efficiency. I wonder:</p>
<ol>
<li>
<p>if doing so indeed hurts distributed efficiency</p>
</li>
<li>
<p>if there are recommended practices when it comes to use tensorboard in the DDP setting? If so, what are they?</p>
</li>
</ol>
<p>Thanks!</p>

Yanli_Zhao: <p><a class="mention" href="/u/cdhere">@CDhere</a> 1. all_gather involves communication sync among all ranks, so it has overhead. Depending on your application latency, you can measure the ratio btw the tensor board all gather overhead and total application latency. Based on the measured overhead, you can decide how frequently you want to dump data to tensor board. 2. as far as I know Tensorboard does not natively all gather info from multiple ranks. All gather by application should be a good way for now.</p>
Loading large sized data

banikr: <p>Hello,<br>
I am working on 114 3D images each having 3D segmentation labels. For each image, I can generate 96 or 192 3D patches. I want to know how to load the images in Dataloader since the patch from all the images can’t be loaded together.</p>

Eta_C: <ol>
<li>custom subclass of <code>Dataset</code>
</li>
<li>save file path in <code>__init__</code>
</li>
<li>load each image and generate its patches in <code>__getitem__</code>
</li>
</ol>
LSTM and LSTMCell ？

bamtercelboo: <p>I want to know the difference between the LSTM and LSTMCell in the pytorch document？And how to use it correctly.</p>

tlaurent: <p>LSTMCell takes ONE input x_t. You need to make a loop in order to do one pass of backprop through time.</p>
<p>LSTM takes a SEQUENCE of inputs x_1,x_2,…,x_T. No need to write a loop to do one pass of backprop through time.</p>

bamtercelboo: <p>ok, Thanks for your answer, Is there any other difference?</p>

bamtercelboo: <p><a class="mention" href="/u/tlaurent">@tlaurent</a>  ok, Thanks for your answer, Is there any other difference?</p>

tlaurent: <p>I would guess that using LSTM() is faster than writing your own loop with LSTMCell (because it is optimized by cudnn).</p>

bamtercelboo: <p><a class="mention" href="/u/tlaurent">@tlaurent</a> ok thanks gor your answer, I will try it.</p>

nibor: <p>LSTMCell() is the building block of the LSTM(). In LSTM(), there are multiple LSTMCell() in one layer as equal as sequence length or unrolling length of network.</p>

Carlos_Jose_Gonzalez: <p>I made this example, I hope it helps <a href="https://github.com/CarlosJose126/RNNs/blob/main/LSTMCellvsLSTM.ipynb" class="inline-onebox" rel="noopener nofollow ugc">RNNs/LSTMCellvsLSTM.ipynb at main · CarlosJose126/RNNs · GitHub</a></p>
Error in SWA Blogpost

dbhuangal1204: <p>In the blogpost titled “PyTorch 1.6 now includes Stochastic Weight Averaging” there is a section of code that does not match up with its description.</p>
<p>The blogpost can be found <a href="https://pytorch.org/blog/pytorch-1.6-now-includes-stochastic-weight-averaging/" rel="noopener nofollow ugc">HERE</a>.</p>
<p>Under the section “HOW TO USE SWA IN PYTORCH?” there is a description: “In the example below, <code>swa_model</code> is the SWA model that accumulates the averages of the weights. We train the model for a total of 300 epochs, and we switch to the SWA learning rate schedule and start to collect SWA averages of the parameters at epoch 160.”</p>
<p>However, the example code seems to train only for 100 epochs and starts collecting SWA averages after epoch 5. I wanted to see if this was an error or if I am misunderstanding the code.</p>
<p>Code shown in the blogpost:</p>
<pre><code class="lang-auto">from torch.optim.swa_utils import AveragedModel, SWALR
from torch.optim.lr_scheduler import CosineAnnealingLR

loader, optimizer, model, loss_fn = ...
swa_model = AveragedModel(model)
scheduler = CosineAnnealingLR(optimizer, T_max=100)
swa_start = 5
swa_scheduler = SWALR(optimizer, swa_lr=0.05)

for epoch in range(100):
      for input, target in loader:
          optimizer.zero_grad()
          loss_fn(model(input), target).backward()
          optimizer.step()
      if epoch &gt; swa_start:
          swa_model.update_parameters(model)
          swa_scheduler.step()
      else:
          scheduler.step()

# Update bn statistics for the swa_model at the end
torch.optim.swa_utils.update_bn(loader, swa_model)
# Use swa_model to make predictions on test data 
preds = swa_model(test_input)
</code></pre>

ptrblck: <p>This looks indeed not right. CC <a class="mention" href="/u/vincentqb">@vincentqb</a></p>

vincentqb: <p>Thanks for catching this! Yes, this is a typo in the example of the blog, see <a href="https://pytorch.org/docs/stable/optim.html#putting-it-all-together" rel="noopener nofollow ugc">documentation</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I’ll look into getting the post updated.</p>
ZFNet visualization

Darek554: <p>Hi.<br>
I’m trying to implement ZFNet, i also want to make the same visualizations from <a href="https://arxiv.org/pdf/1311.2901.pdf" rel="noopener nofollow ugc">this</a> paper, but i don’t know how to do deconvolution.<br>
There is a module ConvTranspose2d, but i don’t think this is something i need, because this module has a learnable parameters.<br>
Any advice? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Dwight_Foster: <p>I think your only option is a ConvTranspose2d but you could look at <a href="https://discuss.pytorch.org/t/true-deconvolution-layer-not-transposed-convolution/27776">this</a> forum post see if it helps.</p>
Upscaling the numpy array using Deep learning

Nikhil_Chowdary_Gutl: <p>Hi Everyone,</p>
<p>Suppose I have an vector matrix of size 512 x 1 or 512 x 2, and I need to upscale this matrix to 3024 x 3024 using pytorch.</p>
<p>Can I know which algorithm/ technique I need to apply to achieve this? Any help would be beneficial to me.</p>
<p>Here I am not focusing on my output accuracy.</p>
<p>Thank you.</p>
Perform element-wise multiplication along the columns

siyuchen95: <p>Hi, I am trying to multiply two tensors of size (N, d) and (N, d) in the sense that each row is multiplied element-wise and summed over. What is the best practice?</p>
<p>More specifically, suppose that the two tensors are (e.g., N = 2, d = 4)</p>
<pre><code class="lang-auto">A = torch.Tensor([[1, 2, 3, 4], [5, 6, 7, 8]])
B = torch.Tensor([[4, 3, 2, 1], [8, 7, 6, 5]])
</code></pre>
<p>and I want to get the following result,</p>
<pre><code class="lang-auto">torch.Tensor([[20], [164]])
</code></pre>
<p>Thanks!</p>

Laura_Montalvo: <p>c = torch.mul(A,B)<br>
c = [c[0].sum(), c[1].sum()]<br>
c = [tensor(20.), tensor(164.)]<br>
however c becomes a list</p>

siyuchen95: <p>Thanks Laura! I tried also something similar</p>
<pre><code class="lang-auto">torch.diagonal(torch.mm(A, B))
</code></pre>
<p>but it takes too much space, especially when the two tensors are big.</p>
<p>Do you think there is a walk-around?</p>

Eta_C: <pre><code class="lang-python">C = (A * B).sum(dim=1, keepdims=True)
</code></pre>

siyuchen95: <p>Thank you! It works perfectly.</p>
How to i print Intersection Of Union while training

Valerien: <p>Hi, I’m trying to use yolov4 with pytorch, I am using that with this code (link on below)</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/Tianxiaomo/pytorch-YOLOv4" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars.githubusercontent.com/u/31844343?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="60" height="60">

<h3><a href="https://github.com/Tianxiaomo/pytorch-YOLOv4" target="_blank" rel="noopener nofollow ugc">Tianxiaomo/pytorch-YOLOv4</a></h3>


  <p><span class="label1">PyTorch ,ONNX and TensorRT implementation of YOLOv4</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I want to see IoU(Intersection of Union) while training.How can i print that ?</p>
<p>Thanks in advance</p>
torch::Generator seeding segfault (at::Generator mutex)

vdg: <p>Hi,</p>
<p>I would like to create a RNG generator which can be seeded individually (as opposed to the global <code>torch::manual_seed(123)</code>.</p>
<p>However, I keep running into segmentation faults. I understood the at::Generator is not thread-safe, and therefore a mutex is necessary, but even just creating the mutex gives me a segfault:</p>
<pre><code class="lang-auto">auto gen_cpu = torch::Generator();
std::lock_guard&lt;std::mutex&gt; lock(gen_cpu.mutex());
//Results in segfault
</code></pre>
<pre><code class="lang-auto">auto gen_cpu = torch::Generator();
gen_cpu.set_current_seed(0);
//Also results in segfault
</code></pre>
<p>Any help with how the Generator should be created and seeded without segfaults?</p>
<p>Thanks!</p>

vdg: <p>It seems the default constructor <code>torch::Generator()</code> leaves the pointer to <code>GeneratorImpl</code> a nullptr.<br>
I took another route and fixed it using:</p>
<pre><code class="lang-auto">auto cpu_gen = at::detail::createCPUGenerator();
{
    std::lock_guard&lt;std::mutex&gt; lock(cpu_gen.mutex());
    cpu_gen.set_current_seed(0);
}
std::cout&lt;&lt;"gen_cpu"&lt;&lt;torch::rand(5,cpu_gen)&lt;&lt;std::endl;
</code></pre>
Error when using CUDA

111469: <p>I got the following error when I’m trying to use CUDA to train a LSTM.</p>
<pre><code class="lang-auto">RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)
</code></pre>
<p>The full code which shows the location:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "---/train.py", line 228, in &lt;module&gt;
    main(args)
  File "---/train.py", line 132, in main
    output, _ = model(sample['src_tokens'], sample['src_lengths'], sample['tgt_inputs'])
  File "---\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "---\seq2seq\models\model.py", line 24, in forward
    decoder_out = self.decoder(tgt_inputs, encoder_out)
  File "---\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "---\seq2seq\models\lstm.py", line 303, in forward
    rnn_layer(lstm_input, (tgt_hidden_states[layer_id], tgt_cell_states[layer_id]))
  File "---\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "---\nn\modules\rnn.py", line 974, in forward
    self.bias_ih, self.bias_hh,
RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)
</code></pre>
<p>I’m using</p>
<pre><code class="lang-auto">model.to(device)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">sample = {key: value.to(torch.device("cuda:0")) if type(value) != int else value for key, value in sample.items()}
</code></pre>
<p>to make sure the input and the model are devided into the CUDA.</p>
<p>The error happened at the LSTM Decoder:</p>
<pre><code class="lang-auto">        self.layers = nn.ModuleList([nn.LSTMCell(
            input_size=hidden_size + embed_dim if layer == 0 else hidden_size,
            hidden_size=hidden_size)
            for layer in range(num_layers)])
</code></pre>
<p>Version: 1.7.1+cu101, in Windows 10. <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_head_bandage.png?v=9" title=":face_with_head_bandage:" class="emoji" alt=":face_with_head_bandage:"></p>

ptrblck: <p>The error seems to be raised in:</p>
<pre><code class="lang-python">rnn_layer(lstm_input, (tgt_hidden_states[layer_id], tgt_cell_states[layer_id]))
</code></pre>
<p>Could you check the <code>.device</code> attribute of all inputs as well as the parameters of <code>rnn_layer</code> and make sure they are on the GPU?<br>
In case you get stuck, please post an executable code snippet, so that we could reproduce and debug the issue.</p>

111469: <p>Thanks a lot! I found the error, there were two tensors defined without specifying the devices. <img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=9" title=":innocent:" class="emoji" alt=":innocent:"></p>
Why is forward pass generating nan values?

jpj: <pre><code class="lang-auto">def forward(self, x):

        hidden_inputs = self.hidden(x)

        hidden_outputs = self.sigmoid(hidden_inputs)

        final_inputs = self.output(hidden_outputs)

        final_outputs = self.softmax(final_inputs)

        return final_outputs

</code></pre>
<pre><code class="lang-auto">torch.isnan(self.features).any() # false
label_predict = self.forward(self.features)
torch.isnan(label_predict).any() # true
</code></pre>
<pre><code class="lang-auto"></code></pre>
<p>I checked, and it’s the nn.linear causing the output to be nan. Why is that and how can I fix that?</p>

ptrblck: <p>Could you check the stats of the input tensor as well as the parameters of the linear layer, which is causing this issue?<br>
E.g. if your input contains <code>Infs</code> or very large values in their magnitude, the result might overflow and could be set to NaN in further operations.</p>

jpj: <p>Thanks. I had checked for NaN values in preprocessing but not for Inf values.</p>
RTX 3090 high VRAM usage

aguennecjacq: <p>Hello everyone,</p>
<p>Recently my lab has received two new rtx3090 gpus and I decided to try them out with a resnet50 model I’m using these days. Curiously I found that there was nearly a 50% increase in VRAM compared to when I used  two RTX 2080Ti with a Titan rtx nvlink bridge.<br>
The project setup: my dataset is comprised of 640*640 images, training batch size: 20 and testing batch size: 120. I’m using the standard torchvision resnet50 model and wrapping it with nn.DataParallel (I haven’t started setting up DistributedDataParrallel yet).<br>
I’m also using torch.backends.cudnn.benchmark = True and the optimizer is the classic optim.SGD.<br>
When I was using two rtx 2080Ti, I would use around 10GiB on each graphic card, whereas now with the  two rtx3090 I have 19.7 GiB on the first gpu and 11GiB on the second. Here is a screenshot of the nvidia-smi command:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc5572406f521defe821a35c949dffa96745175b.png" data-download-href="https://discuss.pytorch.org/uploads/default/dc5572406f521defe821a35c949dffa96745175b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc5572406f521defe821a35c949dffa96745175b.png" alt="image" data-base62-sha1="vra6U09W4B6jbTOHghS55uSfNzR" width="659" height="500" data-dominant-color="411834"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">716×543 57.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Why do we have this increase in VRAM usage with the rtx3090?<br>
Thanks in advance</p>

ptrblck: <p><code>nvidia-smi</code> shows all allocated memory, which is the one used by PyTorch, the cache, as well as the CUDA context etc.<br>
You could check the used memory via: <code>print(torch.cuda.memory_allocated())</code> or <code>print(torch.cuda.memory_summary())</code>.</p>

aguennecjacq: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="112373">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>torch.cuda.memory_summary()</p>
</blockquote>
</aside>
<p>Thanks for the help ! I deleted the python cache of my project  (I forgot to delete it when I changed gpu) and the problem seems to have gone more or less, I only have a 1GiB difference during training after the first two epochs compared to with the two rtx2080ti. Using <code>torch.cuda.memory_summary()</code>, we can still observe some memory usage higher than what the rtx2080ti could handle (a peak memory usage of 22Ggib for example), but I expect this is just the optimizer enjoying itself with the extra memory.</p>
Multiprocessing on CPU

fs4ss1: <p>Hi!</p>
<p>Does multiprocessing works in CPU?</p>
<p>I need to predict 200 images (200x200) with CPU, but when I use multiprocessing to  create some process the execution time don’t change (or the execution time increase).</p>
<p>Thanks!</p>

tom: <p>PyTorch uses multiple threads in the default configuration, if you want PyTorch to only use one thread per process, you would want to disable that:</p>
<p><a href="https://pytorch.org/docs/stable/torch.html#parallelism" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/torch.html#parallelism</a></p>
Matmul boradcasting inconsistent results

liqi17thu: <p><strong>Hi there!</strong></p>
<p>I have two matrices <em>proj_mat</em> and <em>cam_coords</em> which shape as (B, 4, 4) and (B, 4, N). The matrix <em>cam_coords</em> is just copies of matrix shape as (4, N), i.e., cam_coords[0] == cam_coords[1] == … == cam_coords[B-1]. So according to the broadcast rules I think we should have</p>
<p><code>proj_mat @ cam_coords == proj_mat @ cam_coords[0]</code>.</p>
<p>And this holds true if I run it using CPU. However, when I switch to GPU the two results no longer matched. So does this happen because of the internal algorithm for Cuda? And how could I fix this problem?</p>

tom: <p>Likely the results are equal up to numerical precision.</p>

liqi17thu: <p>However, the results can have about ±5 difference and have a real impact on my model training.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/6/1636401b927d9901f45e24501aac1e2babb314a8.png" data-download-href="https://discuss.pytorch.org/uploads/default/1636401b927d9901f45e24501aac1e2babb314a8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/1636401b927d9901f45e24501aac1e2babb314a8_2_690x80.png" alt="image" data-base62-sha1="3auJAEa86bU4nvxGBmu8AVlUcBi" width="690" height="80" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/1636401b927d9901f45e24501aac1e2babb314a8_2_690x80.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/1636401b927d9901f45e24501aac1e2babb314a8_2_1035x120.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/6/1636401b927d9901f45e24501aac1e2babb314a8.png 2x" data-dominant-color="18343E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1238×144 37.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

tom: <p>Yes, you typically have approximately fixed relative error (given tensor shapes). If you have 1e-5ish relative error with 1e0ish and now have operands are 1e5ish then the error can get 1e0ish.<br>
You could compare to the same computation with .double() to see which is closer, but in the end there isn’t that much you can do.</p>
<p>Best regards</p>
<p>Thomas</p>
Min / Max across multiple axes

gokul_uf: <p>Hi Everyone,<br>
I’m trying to get the min and max of each image in a batch of images (<code>NCHW</code>) format.<br>
From what I could find, <code>torch.min</code>'s <code>dim</code> argument takes only one dimension.</p>
<p>Any way to make it take multiple axes?</p>

polarbaryon: <p>I am also interested in a function like this.  Functions like mean(), median(), and mode() can take in a list of dimensions to reduce; is functionality for other functions like min() and max() possible?</p>
<p>My use case is that I want to quantize a weight tensor; if I am interested in quantizing using constants from the whole tensor, a step along the way is to calculate the min and max of the whole tensor, which min() and max() do accomplish.  However, I want to compare this against channel-wise quantization (with different constants for zero point, scale, etc), and it would be great if min() and max() produced the right outputs given a list of dimensions to reduce.</p>

kmbae: <p>I just manage to handle the problem by using two min functions like this!</p>
<pre><code class="lang-auto">In [1]: x = torch.rand(2,3,4,4)
Out[1]:
tensor([[[[0.6575, 0.9134, 0.7617, 0.1276],
          [0.3552, 0.0012, 0.0331, 0.4121],
          [0.4304, 0.8518, 0.0515, 0.4528],
          [0.7224, 0.2117, 0.7193, 0.9164]],

         [[0.9737, 0.2632, 0.4182, 0.8434],
          [0.7079, 0.3799, 0.3052, 0.8838],
          [0.9668, 0.6371, 0.7326, 0.7869],
          [0.9386, 0.0949, 0.7356, 0.3458]],

         [[0.3471, 0.2736, 0.3536, 0.4553],
          [0.4408, 0.3221, 0.1848, 0.0352],
          [0.5171, 0.1303, 0.2172, 0.4764],
          [0.3724, 0.6341, 0.7000, 0.4547]]],


        [[[0.5591, 0.5444, 0.0419, 0.1762],
          [0.9670, 0.7052, 0.4980, 0.7963],
          [0.3407, 0.5290, 0.1264, 0.7158],
          [0.9510, 0.1153, 0.8766, 0.1305]],

         [[0.9847, 0.3020, 0.2368, 0.5934],
          [0.6701, 0.9279, 0.1959, 0.8628],
          [0.9260, 0.6380, 0.1120, 0.3395],
          [0.0412, 0.0786, 0.2994, 0.5483]],

         [[0.7809, 0.1019, 0.6274, 0.9994],
          [0.9357, 0.2192, 0.6914, 0.3103],
          [0.3666, 0.3832, 0.2777, 0.1814],
          [0.4448, 0.6855, 0.3532, 0.9655]]]])
In [2]: x.min(2)[0].min(2)[0]
Out[2]: 
tensor([[0.0012, 0.0949, 0.0352],
        [0.0419, 0.0412, 0.1019]])
</code></pre>
Use model predictions on a heldout dataset in custom loss function

akv12: <p>I want to design a custom loss function L = L_primary + (alpha)*(new_loss).</p>
<p>Besides, the primary <code>torch.nn.CrossEntropyLoss()</code> component, I want to add another <code>new_loss</code> component that utilizes the model prediction on a held-out dataset that is different from the training dataset used to compute <code>torch.nn.CrossEntropyLoss()</code>.</p>
<p>While designing the <code>new_loss</code> component, I am not sure if I should use <code>with torch.set_grad_enabled(False):</code> to predict the output on the held-out dataset.</p>
<p>Here is a code snippet of my training:</p>
<blockquote>
<p>for epoch in range(NUM_EPOCHS):<br>
model.train()<br>
for batch_idx, (features, targets) in enumerate(train_loader):</p>
<pre><code>    features = features.to(DEVICE)
    targets = targets.to(DEVICE)
        
    ### FORWARD AND BACK PROP
    logits = model(features)
    probas = F.softmax(logits, dim=1)
    
    L_primary = cost_fn(logits, targets)
  
     model.eval()
    new_loss = NEW_LOSS(model, data_loader)  
     model.train()

    loss = L_primary + (alpha)*new_loss
        
    optimizer.zero_grad()
    
    cost.backward()
    
    ### UPDATE MODEL PARAMETERS
    optimizer.step()
</code></pre>
</blockquote>
<p>I want to backpropagate on sum of both the losses to update the model parameters. Please let me know if the following is correct to train the model but not run into memory issues while storing outputs.</p>
<blockquote>
<p>def NEW_LOSS(model, data_loader):</p>
<pre><code>### Initialize the prediction and label lists(tensors)
predlist=torch.zeros(0,dtype=torch.float32, device=DEVICE)

with torch.set_grad_enabled(False):
    for i, (features, targets) in enumerate(data_loader):
        features = features.to(DEVICE)
        targets = targets.to(DEVICE)
        logits = model(features)
        probas = F.softmax(logits, dim=1)
    
        # Append batch prediction results
        predlist=torch.cat([predlist,probas[:,0].view(-1).type(torch.float32)])

### A function that computes loss using predlist
new_loss = Compute_Loss(predlist)
return new_loss
</code></pre>
</blockquote>
<p>If I don’t use <code>with torch.set_grad_enabled(False):</code>, then I am running into memory issues. Also, any other suggestions to address my problem are welcome. Help is greatly appreciated. Thank you.</p>
For 3D conv net, what can I try to do to get a MSEloss closer to 0?

Christopher: <p>My network takes in a minibatch of 3D MRI image as numpy and should output 1 integer for each image. I currently have n=12 x,y training pairs and n=2 x,y validation pairs.<br>
I was able to get the lowest training MSEloss of 1.4 at epoch 464 with a learning rate of 0.01 and a loss of 1.7 with a learning rate of 0.1  at epoch 90. I’m trying to get a training loss as close to 0 as possible. I tried going from 2 conv layer set to 3 conv layer set. What can I try to do to get a loss closer to 0?</p>
<p>Here’s the model:</p>
<pre><code class="lang-auto">class CNNModel(nn.Module):
    def __init__(self):
        super(CNNModel, self).__init__()
        
        self.conv_layer1 = self._conv_layer_set(1, 32)
        self.conv_layer2 = self._conv_layer_set(32,32)
        self.conv_layer3 = self._conv_layer_set(32, 64)

        self.ad_avg_pol=nn.AdaptiveAvgPool3d((1,1,1))
        
        self.fc1=nn.Linear(64,1)
               
        
    def _conv_layer_set(self, in_channels, out_channels):
        conv_layer = nn.Sequential(
        nn.Conv3d(in_channels,    out_channels, kernel_size=(3, 3, 3),stride=1, padding=0),
        nn.LeakyReLU(),
        nn.MaxPool3d((2, 2, 2)) 
        )
        return conv_layer
    
    def forward(self, x):
        x=torch.unsqueeze(x,1) #add dimension for channel
        x = self.conv_layer1(x)
        x = self.conv_layer2(x)
        x = self.conv_layer3(x)

        x= self.ad_avg_pol(x)
        x=x.reshape(x.size(0),-1)
        x=self.fc1(x)
        
        return x
</code></pre>
How to get memory occupied by tensors when using CPU?

kenenbek: <p>There is a function for <code>cuda</code> device.  <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.memory_allocated" class="inline-onebox" rel="noopener nofollow ugc">torch.cuda — PyTorch 1.7.1 documentation</a></p>
<p>How can I do the same when I train a model on CPU?</p>
<p>Or maybe some workaround exists?</p>
Difference between model.parmaters() and model.named_parameter()?

aswamy: <p>What is the difference between model.parmaters() and model.named_parameter()?</p>

ptrblck: <p>The former returns a <code>list</code> of all parameters, the latter a <code>dict</code> with the names as well as the parameters.<br>
The <a href="https://pytorch.org/docs/stable/index.html">docs</a> explain both methods with more detail.</p>
Pytorch multi task

OuisYasser: <p>Hello mates,<br>
I have a question, let’s say I have an input image which I would then create many versions of it using autoencoder and multi task learning<br>
Now for testing, I want the user to give let’s say a task number so the output would be only that version of image. Is it possible to do it with pytorch ?<br>
Thanks in advance</p>
Manually specifying gradients in optimizer

garland: <p>For illustration, here’s a toy model:</p>
<pre><code class="lang-auto">input = torch.distributions.normal.Normal(loc = 0, scale = 1).sample([50,5])
target = torch.ones([50, 1])
simple_model = torch.nn.Linear(5,1)
output = simple_model(input)
</code></pre>
<p>which I could optimise through standard procedures:</p>
<pre><code class="lang-auto">optimizer = optim.Adam(simple_model.parameters(), lr=1e-3)
criterion = torch.nn.MSELoss()
loss = criterion(output, target)
loss.backward()
optimizer.step()
</code></pre>
<p>However, my actual application is more complicated, and I deal with explicit gradients, not a loss. For example I only have access to the gradients returned by <code>torch.autograd.grad(_, simple_model.parameters())</code>. Is there anyway I can use these gradients to directly udpate the parameters of <code>simple_model</code> without writing my own optimiser?</p>

ptrblck: <p>Have a look at <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html" rel="nofollow noopener">this tutorial</a>, where a manual update is described.<br>
If It’s possible to assign your custom gradients to the <code>.grad</code> attribute of all parameters, you could still use an optimizer directly.</p>

Brando_Miranda: <p>Is the pseudocode something like this:</p>
<pre><code class="lang-auto">loss = criterion(mdl(x), y)
loss.backward()
for name, w in mdl.named_parameters():
    w.grad = w.grad + 1.0
</code></pre>
<hr>
<p>note:  I wanted to do something like the above but collect the gradients from an rpc (hence my curiosity for actual code example but the link didn’t seem super useful, I’d probably do something like <code>w.grad = rpc_grads</code> in my real example)</p>

ptrblck: <p>Your code looks fine for a single model, but I’m not sure how you are using it in the RPC use case, so you might need to gather the gradients first etc.</p>
Testing my GRU layer

miturian: <p>I am building a network which includes a bidirectional GRU layer. After getting all my dimensions to fit, I am now trying to use simple inputs to check that the network is actually doing as intended.  However, I can not seem to get the expected output from my GRU layer?</p>
<p>My code looks like this:</p>
<pre><code class="lang-auto">

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.nHidden=30
        self.nFilter=20
        self.conv1 = nn.Conv2d(1,self.nFilter,(5, 5), stride=(1, 1), padding=(2, 2))
        self.birnn = nn.GRU(self.nFilter*129,self.nHidden,1,bidirectional=True,batch_first=True)

        
        #debugging:
        self.conv1.bias[:]=0
        self.birnn.bias_ih_l=0
        self.birnn.bias_hh_l=0
        # self.birnn.weight_ih_l=0
        # self.birnn.weight_hh_l=0
        
    def forward(self, x):
        
        #filtering:
        x = F.relu(self.conv1(x))

        #biGRU:
        x=x.permute((0,2,1,3))

        x=x.reshape((-1,29,20*129))
        print(x.size())
        print(torch.sum(x,(1,2)))
        
        x,hn = self.birnn(x)
        print(torch.sum(x,(1,2)))

        return x
    
net=Net()
x=torch.zeros(3,1,29,129)
x[1,:,2,:]=1
output=net(x)
</code></pre>
<p>and running it gives the following:</p>
<pre><code class="lang-auto">torch.Size([3, 29, 2580])
tensor([   0.0000, 1551.1537,    0.0000], grad_fn=&lt;SumBackward1&gt;)
tensor([-6.9812,  5.4175, -6.9812], grad_fn=&lt;SumBackward1&gt;)
</code></pre>
<p>my intention is that one data point is all 1’s, and the rest are zeros. As I understand nn.GRU, if inputs are 0 and bias’s are zero, shouldn’t outputs be zero too? So, my expectation was to have an output that was [0,K,0] (like the input was). Why is that not happening?</p>
<p>Thank you for reading <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

miturian: <p>I found the problem myself.<br>
I was not changing the correct variables in the GRU layer. I had not understood the variables-names properly, as they are described in <a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html?highlight=nn%20gru#torch.nn.GRU" class="inline-onebox" rel="noopener nofollow ugc">GRU — PyTorch 1.7.1 documentation</a>.</p>
<p>Changing to</p>
<pre><code class="lang-auto">gru.bias_ih_l0[:]=0
gru.bias_ih_l0_reverse[:]=0
gru.bias_hh_l0[:]=0
gru.bias_hh_l0_reverse[:]=0
</code></pre>
<p>solves it.</p>
<p>(I discovered it by inspecting a gru object directly)</p>
Automatic differentation for pytorch einsum

kevs: <p>Hi, I am trying to figure how the tensor gradient is calculated when using the einsum operator. Can someone point me towards documentation for this or alteast the internal implementation in source code?</p>

tom: <p>Good question, actually one of the exercises in my advanced autograd class.<br>
The answer lies in the autograd graph:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(5,5, requires_grad=True)
y = torch.randn(5,5, requires_grad=True)
a = torch.einsum('ik,kj-&gt;ij', x, y)
a
</code></pre>
<p>will show you that the <code>grad_fn</code> of <code>a</code> it is some <code>view</code> backward. And indeed, the graph looks like this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b5a0f43e0158d8ca76051647fe77e1285331919d.png" data-download-href="https://discuss.pytorch.org/uploads/default/b5a0f43e0158d8ca76051647fe77e1285331919d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b5a0f43e0158d8ca76051647fe77e1285331919d_2_387x500.png" alt="image" data-base62-sha1="pULgpltum6ZZGzWQBsXnybb1yAl" width="387" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b5a0f43e0158d8ca76051647fe77e1285331919d_2_387x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b5a0f43e0158d8ca76051647fe77e1285331919d_2_580x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/5/b5a0f43e0158d8ca76051647fe77e1285331919d.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">641×827 39.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><code>einsum</code> reduces to reshaping operations and batch matrix multiplication in <code>bmm</code>.<br>
Like when you write your own computation in Python, PyTorch actually keeps track of the calls to “explicitly differentiable functions” and then computes the backward piece by piece.</p>
<p>The reduction is actually the other way round as in NumPy where <code>bmm</code> reduces to <code>einsum</code>.<br>
If you look at the <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Linear.cpp" rel="noopener nofollow ugc">PyTorch <code>einsum</code> implementation</a> you see that (save some optimizations which would have been better put in sumproduct_pair IMHO) it rearranges some things and then calls <code>sumproduct_pair</code>. That in turn could have a very simple explicit derivative with expand + sumproduct_pair.</p>
<p>Best regards</p>
<p>Thomas</p>
SummaryWriter().add_histogram() does not log

aswamy: <p>Hello,</p>
<p>I am trying to visualize neural network weights by adding weights to the histograms of tensorboard.</p>
<p>for name, weight in self.net.named_parameters():<br>
self.writer.add_histogram(name, weight, global_step=step_id)</p>
<p>code itself runs without any errors, but in tensorboard I do not see the histograms(i don’t see any log).<br>
I printed the name and weights to confirm they are not empty tensor.</p>
<p>Any help to resolve this is appreciated.</p>
<p>Thank you!</p>

googlebot: <p>Do you use flush()? Generally, file size increase would indicate success on pytorch side. On browser side, chrome-based browser may be needed, and density plots are on separate page.</p>

aswamy: <p>No, I did not quite understand what you are saying. Can you please provide more information?<br>
Thank you!</p>

googlebot: <p>below works for me:</p>
<pre><code class="lang-auto">from torch import nn
from torch.utils.tensorboard import SummaryWriter

net = nn.Linear(8,8)
writer = SummaryWriter(DIR) #,flush_secs=120
for name, weight in net.named_parameters():
	writer.add_histogram(name, weight, global_step=1)
writer.flush() #to avoid delayed disk writes when inside a training loop
</code></pre>