akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why not type annotate `Module` to be Tensor->Tensor?

Daniel-Chin: <p>Say I have</p>
<pre><code class="lang-python">a = torch.zeros((3, ))
fc = nn.Linear(3, 6)
b = fc(a)
</code></pre>
<p>My IDE thinks <code>a</code> is a <code>torch.Tensor</code> but <code>b</code> is just <code>Any</code>. This forces me to do things like</p>
<pre><code class="lang-python">b: torch.Tensor = fc(a)
</code></pre>
<p>a lot im my code.</p>
<p>Why didn’t pytorch annotate <code>Module.__call__</code> to be <code> -&gt; torch.Tensor</code>?</p>

ConvolutionalAtom: <p>I am also interested in this.<br>
The line responsible for this behaviour seems to be <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py#L1558" rel="noopener nofollow ugc">here source code</a></p>
<p>I think it seems feasible to get type hints of function via <a href="https://docs.python.org/3/library/typing.html#typing.get_type_hints" rel="noopener nofollow ugc">get_type_hints</a> from typing library.</p>
<p>One can use this to get return type of forward function defined by the users.</p>
<p>Maybe there is a way add this feature to pytorch without any problems ???</p>

ConvolutionalAtom: <p>Just discovered this related issue: <a href="https://github.com/pytorch/pytorch/issues/74746" class="inline-onebox" rel="noopener nofollow ugc">Change the type hint for nn.Module.__call__ to be friendly to overrides. · Issue #74746 · pytorch/pytorch · GitHub</a></p>

Daniel-Chin: <p>Thanks for the reference to <span class="hashtag">#74746</span>.</p>
<p>To add to that, I would hope the built-in objects such as <code>nn.Linear</code> to come with <code>Tensor-&gt;Tensor</code> annotation as well, not just user-defined classes.</p>

ConvolutionalAtom: <p>I tested with mypy, it detects nn.Linear just like the way you descibed, i.e. <code>Tensor-&gt;Tensor</code> since source code for built-in modules uses type hinting as well. Again, the problem/feature is the way IDEs recognize the types.</p>
<p>Relevant source code: <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/linear.py#L113" rel="noopener nofollow ugc">here</a></p>

ConvolutionalAtom: <p>For <code>nn.Linear</code>, reveal_type returns <code>Any</code>, instead of <code>torch.Tensor</code>. To replicate this, run the code below</p>
<pre><code class="lang-auto">pytorch=1.13
mypy=0.991
</code></pre>
<pre><code class="lang-auto"># main.py
import torch
from torch import nn


class LinWrapper(nn.Module):

    def __init__(self) -&gt; None:
        super().__init__()
        self.lin = nn.Linear(12,2)

    def forward(
        self, x: torch.Tensor
    ) -&gt; torch.Tensor:
        out = self.lin(x)
        reveal_type(out)
        return out
</code></pre>
<p>Bash command to run</p>
<pre><code class="lang-auto">mypy main.py
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">main.py:15: note: Revealed type is "Any"
</code></pre>
<p>Note when tested with custom module (with type hints), it reveals to be <code>torch.Tensor</code>.</p>
Unable to use torch on vmware docker instance

murgado: <p>Hello there!<br>
I’m running into the following issue here:</p>
<p>I have a FastAPI server with a call connected to OpenCV and EasyOCR to read an image text, however the executions crashes whenever EasyOCR reader is called (which internally, uses a torch model). The last statement that I receive would be:</p>
<p>[Could not initialize NNPACK! Reason: Unsupported hardware]</p>
<p>I’ve read that this message is nothing more than a warning but still, it is the last message that I receive from the server machine (again, a docker container within vmware without GPU).</p>
<p>Can someone provide some light?<br>
I’m using torch cpu-only libraries and I have also tried to downgrade Python to 3.8, which is currently my current version.</p>
Exact use of task_info queue?

Vinayaka_Hegde: <p>Hi, whats the exact use of the <code>task_info</code> queue? Since we already have <code>num_workers</code> number of index queues - one per worker to put the next index to be fetched from the sampler by that worker. The <code>task-info</code> queue is being populated when we put indices from the sampler to the workers index queue.<br>
And, it is being popped when we need to return the next batch from the data loader.</p>
<p>populated :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8fce1051a4c84a72be5d8e20641e71d728170eb.png" data-download-href="https://discuss.pytorch.org/uploads/default/f8fce1051a4c84a72be5d8e20641e71d728170eb" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8fce1051a4c84a72be5d8e20641e71d728170eb.png" alt="image" data-base62-sha1="zwEcBQZi3KXrQVQ1soRFjIJkM9d" width="690" height="309" data-dominant-color="FBFBF9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1297×581 25.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>popped :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a150407f0a6f1915d09b09d52809a10103fb000.png" data-download-href="https://discuss.pytorch.org/uploads/default/0a150407f0a6f1915d09b09d52809a10103fb000" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a150407f0a6f1915d09b09d52809a10103fb000.png" alt="image" data-base62-sha1="1rbNE15qinI97BNtV9IH25EW43C" width="690" height="133" data-dominant-color="FCFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1163×225 6.35 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So, Is it just used to store the out of order samples as hinted above right? And is not used for any other thing ?</p>
Difference b/w index and send-idx?

Vinayaka_Hegde: <p>Hi, I’m trying to understand the dataloader.py file and I’m not able to get the difference b/w send_idx and index ? (Pls refer screenshot)</p>
<p><code>send_idx</code> starts from zero and is incremented every time a new index is fetched from the sampler. So is <code>index</code>! since <code>index</code> starts from zero(which is the start value in a sequential sampler) and goes till <code>len(dataset)-1</code>. And is incremented every time when we call <code>_next_index</code></p>
<p>SO, shouldnt <code>index</code> be equal to <code>send-idx</code> every time ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/683e15abdb11f7799e2a67218bd4da065a361255.png" data-download-href="https://discuss.pytorch.org/uploads/default/683e15abdb11f7799e2a67218bd4da065a361255" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/683e15abdb11f7799e2a67218bd4da065a361255_2_690x280.png" alt="image" data-base62-sha1="eSaFHNjjl8Q3yXj5pYjVHc1VwQR" width="690" height="280" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/683e15abdb11f7799e2a67218bd4da065a361255_2_690x280.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/683e15abdb11f7799e2a67218bd4da065a361255_2_1035x420.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/683e15abdb11f7799e2a67218bd4da065a361255_2_1380x560.png 2x" data-dominant-color="FCFBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1418×576 26.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
What do the below counts mean?

Vinayaka_Hegde: <p>Hi, I’m digging at the pytorch dataloder.py file and wanted to understand what send_idx and rcvd_idx mean ?</p>
<p>WHat I understand is <code>send_idx</code> is the index of the next item from the sampler to be sent to the workers so that a worker can fetch this particular index from dataset and write the data into the data queue / worker_result_queue.</p>
<p>For <code>rcvd_idx</code>, what I understand is that this is the index of the item that the worker has just fetched and this indicates the index that is returned from the <code>__next__</code> method of the dataloader class.</p>
<p>Therefore, <code>send_idx</code> is incremented for every index that the worker fetches from the sampler and adds to its index queue to process.<br>
<code>rcvd_idx </code> is incremented when the <code>next_data</code> method is called from the <code>__next__</code> method where in the worker is returning this as the next batch.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e2677658eb078b11b98de5728261c508a6fa54a.png" data-download-href="https://discuss.pytorch.org/uploads/default/6e2677658eb078b11b98de5728261c508a6fa54a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e2677658eb078b11b98de5728261c508a6fa54a.png" alt="image" data-base62-sha1="fIqW0vLo60qfzxvepUhmFowmSoi" width="690" height="70" data-dominant-color="F8F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1164×119 6.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any help would be appreciated !</p>
Leave no GPU empty when pushing the data to device

davids: <p>I am training a ResNet on CIFAR10 dataset using 4 GPUs. When pushing the batches to GPUs, the data can be weirdly distributed, leaving one of the GPUs empty, resulting in <code>TypeError: forward() missing 1 required positional argument: 'out'</code> error. Specifically, if I select a batch size to be 4, 1 instance is pushed to each of the GPUs (no error). If I select a batch size of 5, data is distributed as 2, 2, 1, 0 (causes error). Batch size of 6 results in 2, 2, 2, 0 (causes error). Batch size of 7 results in 2, 2, 2, 1 distribution across GPUs (again no error). Why is this happening and how to force the data to be distributed in a way that no GPU is empty?</p>
How to check if PyTorch is using TCmalloc?

rinkujadhav2013: <p>I’m trying to follow tips from <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html#switch-memory-allocator" rel="noopener nofollow ugc">PyTorch’s performance tuning guide</a>. One of them is that one could try using tcmalloc from google.<br>
Since I don’t have root permission on the server, I installed it locally and added path to the <code>tcmalloc.so*</code> files to <code>LD_LIBRARY_PATH</code> and path to include files to <code>CPATH</code>. I then added the <code># export LD_PRELOAD=&lt;tcmalloc.so&gt;:$LD_PRELOAD</code> comment right below the import statements in my python code. But I don’t know if PyTorch is actually making use of tcmalloc or not. Is there any way I can check/verify that?<br>
P.S. I’m using host memory heavily so I thought that tcmalloc could help with that.</p>

ptrblck: <aside class="quote no-group" data-username="rinkujadhav2013" data-post="1" data-topic="170672">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rinkujadhav2013/48/54900_2.png" class="avatar"> rinkujadhav2013:</div>
<blockquote>
<p>I then added the <code># export LD_PRELOAD=&lt;tcmalloc.so&gt;:$LD_PRELOAD</code> comment right below the import statements in my python code.</p>
</blockquote>
</aside>
<p><code>export LD_PRELOAD</code> should not be a comment in your Python code but should be exported in your current terminal.<br>
Once this is done you could use <code>LD_DEBUG=libs</code> to check if the linker is indeed preloading the right library.</p>
Concatinate more imput Images in a Block

Raphael_Olipitz: <p>Hey Pytorch-Masters,</p>
<p>I want to feed more than one picture in one ConvBlock, so you can see it as one more than one shortcut in a Resnet. But I got an Exception like :</p>
<pre><code>if in_channels % groups != 0:
</code></pre>
<p>RuntimeError: Boolean value of Tensor with more than one value is ambiguous</p>
<p>Thats my Block I am testing:</p>
<p>class three_ConvBlock(nn.Module):</p>
<pre><code>def __init__(self, in_ch, out_ch, conv_index, customized_shortcut):
    super(three_ConvBlock, self).__init__()

    conv_index = nn.Parameter(conv_index)
    customized_shortcut = nn.Parameter(customized_shortcut)

    self.layer_1 = nn.Conv3d(in_ch,out_ch,3,stride=(1,1,1), padding=0)
    self.layer_2 = nn.Conv3d(in_ch,out_ch,3, stride=(1,1,1), padding=0)

def forward(self,x, conv_index, customized_shortcut):

    x_1 = self.layer_1(x)
    conv_index_out = x_1

    x_2 = torch.sub(x_1,customized_shortcut)
    x_2 = torch.cat((x_2,conv_index))
    x_3 = self.layer_2(x_1)        

    #print(conv_index, customized_shortcut)
    return x_3 , conv_index_out
</code></pre>
<p>Image = torch.randint(10, (3,3,3),requires_grad=True,dtype=torch.float64)</p>
<p>a = torch.randint(5, (3,3,3),requires_grad=True,dtype=torch.float64)</p>
<p>b = torch.randint(4, (3,3,3),requires_grad=True,dtype=torch.float64)</p>
<p>test_layer = three_ConvBlock(Image,Image,conv_index=a,customized_shortcut=b)</p>
<p>result = test_layer(Image,conv_index=a,customized_shortcut=b)</p>
<p><span class="hashtag">#print</span>(model.parameters)</p>
<p>print(result)</p>
<p>Thanks for any advice!!!</p>

ptrblck: <p>You are trying to pass the image in a shape of <code>[3, 3, 3]</code> into the <code>in_channels</code> and <code>out_channels</code> arguments while ints or tuples or ints are expected.<br>
Use the actual channel dimension and it should work:</p>
<pre><code class="lang-python">test_layer = three_ConvBlock(Image.size(0),Image.size(1),conv_index=a,customized_shortcut=b)
</code></pre>
<p>Afterwards you will run into a shape mismatch error since <code>nn.Conv3d</code> layers expect an input with 5 dimensions representing <code>[batch_size, channels, depth, height, width]</code>.</p>
CUDA out of memory error if I don't execute optimizer.step in SWA

Anirban_Nath: <p>I have a program in which I am applying Stochastic Weighted Averaging (SWA) as the optimizer. In this code, if I perform forward prop + back prop, then everything seems to work fine. However, SWA optimizer has a function called “bn_update” that does: -</p>
<blockquote>
<p>Updates BatchNorm running_mean, running_var buffers in the model.</p>
<p>It performs one pass over data in <code>loader</code> to estimate the activation<br>
statistics for BatchNorm layers in the model.</p>
</blockquote>
<p>This function essentially requires me to ONLY perform forward propagation, while some momentum is calculated. However, in about 2 batches, I get a CUDA out-of-memory error. I know this isn’t an issue of batch size or any of the other standard reasons why out-of-memory error is thrown, because my code requires me to do entire epochs of training prior to running this “bn_update” function using the same dataloader and they execute flawlessly.</p>
<p>So why is this that only forward prop seems to upset GPU memory so much?</p>

ptrblck: <p>Based on your description it seems as if SWA is using an unexpected amount or memory or seems to be increasing it.<br>
Could you post a minimal and executable code snippet showing this behavior, please?</p>
Commercial licensing of pytorch with CUDA

Jaap: <p>Not sure if this is the right place to ask but I guess experience is around. Is there any problem in packaging a commercial application with pytorch + CUDA for GPU support as long as everything is correctly cited and documented ? I have search online but couldn’t find any clear answer to this.</p>

pvicentejr: <p>Hi <a class="mention" href="/u/jaap">@Jaap</a> , did you find an answer for this. I am wondering the same thing due to the cuda cudnn dependencies.</p>

J_Johnson: <aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/master/LICENSE">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/LICENSE" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/LICENSE" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/LICENSE</a></h4>


      <pre><code class="lang-">From PyTorch:

Copyright (c) 2016-     Facebook, Inc            (Adam Paszke)
Copyright (c) 2014-     Facebook, Inc            (Soumith Chintala)
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)

From Caffe2:

Copyright (c) 2016-present, Facebook Inc. All rights reserved.

All contributions by Facebook:
Copyright (c) 2016 Facebook Inc.

All contributions by Google:
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/LICENSE" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Cannot use datasets with tokenizer in torchscript

Harsh_Sharma: <p>I was trying to use xlm-roberta for label classification task and set the flag torchscript=True in both model.pretrained and tokenizer.frompretrained but for some reason, is consuming a lot of memory<br>
I tried to set return tensors =‘pt’ in tokenizer - but datasets object expects list of nump array and hence getting an error</p>
Using deep learning

Kennedy_Bwire: <p>what are the key areas to do a research</p>

ptrblck: <p>Your question might be a bit too general for this PyTorch discussion board, as we are mostly focused on the framework itself.</p>

akuysal: <p>You need to specify an application domain (nlp, image classification, etc.). Then, you may investigate for the suitable deep learning model for the specified application domain. After finding the right neural network model, you can try to implement it with a framework (PyTorch, Tensorflow, etc).</p>

Kennedy_Bwire: <p>Thank you for the quick response and insights as well. I’m working on my master’s thesis, and my interests include machine learning, deep learning, and natural language processing. I would like to have some intellectual advice on which areas within my areas of expertise I can dwell on to contribute significantly to the body of knowledge.</p>
No difference in gradient of Gaussian Kernel derivative (Cross Covariance matrix) wrt inputs

lubaroli: <p>Hello, I’m trying to understand why the gradient returned by autograd does not change w.r.t. the input in a Gaussian kernel. More specifically, suppose we have the following naive implementation of the RBF kernel:</p>
<pre><code class="lang-auto">def naive_rbf(X, Y, h):
    batch, _ = X.shape
    Kxy = torch.empty(batch, batch)
    dx_Kxy = torch.zeros_like(X)
    dy_Kxy = torch.zeros_like(X)
    for i in range(batch):
        for j in range(batch):
            # using row vectors
            diff = X[i] - Y[j]
            norm = diff @ diff.T
            Kxy[i, j] = (-0.5 / h ** 2 * norm).exp()
            dx_Kxy[i] += -diff * Kxy[i, j] / h ** 2
            dy_Kxy[j] += diff * Kxy[i, j] / h ** 2
    return Kxy, dx_Kxy, dy_Kxy
</code></pre>
<p>So, given that:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/4/1476deae5ffae547be471ff4eb63904db22d8f69.png" data-download-href="https://discuss.pytorch.org/uploads/default/1476deae5ffae547be471ff4eb63904db22d8f69" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/1476deae5ffae547be471ff4eb63904db22d8f69_2_690x265.png" alt="image" data-base62-sha1="2V2e48SBDGStl9MSUOLr9Ogpiid" width="690" height="265" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/1476deae5ffae547be471ff4eb63904db22d8f69_2_690x265.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/1476deae5ffae547be471ff4eb63904db22d8f69_2_1035x397.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/1476deae5ffae547be471ff4eb63904db22d8f69_2_1380x530.png 2x" data-dominant-color="000000"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1853×712 55.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’d expect the gradients w.r.t. different inputs to have opposite signs.</p>
<p>However, if I run:</p>
<pre><code class="lang-auto">BATCH, DIM = [20, 5]
X = torch.randn(BATCH, DIM)
h = torch.tensor(1.0)

Xg = X.clone().requires_grad_()
K = naive(Xg, X, h)[0]
dx_K = grad(K.sum(), Xg, h)[0]

Xg = X.clone().requires_grad_()
K = naive(X, Xg, h)[0]
dy_K = grad(K.sum(), Xg, h)[0]
torch.allclose(dx_K, dy_K)  # True
</code></pre>
<p>Can anyone shed some light on why this happens?</p>
How loss of dimension 1 can work with output of n dimension when backward

okay_All_Right: <p>inp = torch.tensor([[1,1,1]], dtype=torch.float)</p>
<p>i = torch.tensor([[1]], dtype=torch.long).view(-1, 1)</p>
<p>outp = self.eval_net(inp).gather(1, i)</p>
<p>self.optimizer.zero_grad()</p>
<p>loss = F.smooth_l1_loss(q_eval, 0*q_eval)</p>
<p>loss.backward()</p>
<p>so how the backward work if(eval_net) has multiple outputs ?</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.fc = nn.Linear(3, 1)<br>
self.a_head = nn.Linear(1, 2) <span class="hashtag">#in</span> this case two outputs</p>

okay_All_Right: <p>come on you people, I don’t belive that nobody helps me ! <img src="https://discuss.pytorch.org/images/emoji/apple/worried.png?v=9" title=":worried:" class="emoji" alt=":worried:"></p>

Ruy_Diaz: <p>It seems I have a similar question.</p>
<p>I’ve found <a href="https://stackoverflow.com/questions/61506961/why-do-the-keras-loss-functions-reduce-the-dimensionality-by-one" rel="noopener nofollow ugc">this</a> useful.</p>
<p>Best regards!</p>
Need written permission to use PyTorch logo on our website

gavinminnis: <p>Hello,</p>
<p>I am trying to get written approval to use the PyTorch logo on the <a href="http://TI.com" rel="noopener nofollow ugc">TI.com</a> website. Can someone help me with making the right connection so I can get the appropriate approval.</p>
<p>Thanks,<br>
Gavin</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/can-i-use-pythorch-logo-website-social-media/113144/4">here</a>.</p>
Backward result in pytorch

Mirocos: <p>I am trying to calculate Jacobian matrix of high-dimension tensor using backward function in Pytorch.<br>
I have my test code below:</p>
<pre><code class="lang-python">    t1 = torch.tensor([1.], requires_grad=True)
    t2 = torch.tensor([
        [0.1, 0.2, 0.3],
        ], requires_grad=True)

    tr = t1 + t2
    tr.backward(torch.tensor([
        [1., 0., 0.],
    ]))

    print(t1.grad.data)
    print("=======================")
    print(t2.grad.data)
</code></pre>
<p>And the result is here:</p>
<pre><code class="lang-bash">tensor([1.])
=======================
tensor([[1., 0., 0.]])
</code></pre>
<p>Why the shape of dtr/dt1 is not (3,1)? Why its shape is (1)?</p>

rinkujadhav2013: <p><code>t1.grad</code> will have the shape same as <code>t1</code>. That’s true for any tensor in general, its <code>grad</code>’s shape will be same as its own shape.</p>

Mirocos: <p>And gradient will be sumed to <code>grad</code> of leaf node?<br>
For example now <code>t2</code>’s shape is (2,3)， and <code>grad</code> of <code>t1</code> become <code>[2.]</code></p>
<pre><code class="lang-python">    t1 = torch.tensor([1.], requires_grad=True)
    t2 = torch.tensor([
        [0.1, 0.2, 0.3],
        [0.4, 0.5, 0.6]
        ], requires_grad=True)

    tr = t1 + t2
    tr.backward(torch.tensor([
        [1., 0., 0.],
        [1., 0., 0.]
    ]))

    print(t1.grad.data)
    print("=======================")
    print(t2.grad.data)
</code></pre>
<pre><code class="lang-bash">tensor([2.])
=======================
tensor([[1., 0., 0.],
        [1., 0., 0.]])
</code></pre>

srishti-git1110: <p>Hi, yes.<br>
Unless you use <code>zero_grad</code>, gradients in PyTorch are accumulated by default.</p>

Mirocos: <p>Did you mean interface like <code>optimizer.zero_grad()</code> or <code>t.grad().zero_()</code>? But What if my backward is an one pass operation?<br>
I mean for code below, it’s better that the gradient of <code>t1</code> is ([1.]), though <code>t1</code> actually contributes to two<br>
independent  elements in <code>t2</code>. Because in many cases, different index of tensor are independent or parallel for calculation.</p>
<pre><code class="lang-python">    t1 = torch.tensor([1.], requires_grad=True)
    t2 = torch.tensor([
        [0.1, 0.2, 0.3],
        [0.4, 0.5, 0.6]
        ], requires_grad=True)

    tr = t1 + t2
    tr.backward(torch.tensor([
        [1., 0., 0.],
        [1., 0., 0.]
    ]))

    print(t1.grad.data)
    print("=======================")
    print(t2.grad.data)
</code></pre>

rinkujadhav2013: <p>Both <code>t1.grad.zero_()</code> and <code>optimizer.zero_grad()</code> will zero out the gradient.<br>
If you don’t call any of those then t1.grad would accumulate over backward passes. If you have just 1 backward pass then t1.grad would be just the gradient for that 1 backward pass. If you call backward again (without zeroing out the grad) then t1.grad would be the old value plus the new value and so on.</p>
LSTM: Encoder/Decoder architecture for different input/output sequence length

Planet_Boost: <p>So I have input data which consists of 9 variables with a sequence length of 92. I want to predict a sequence of 7 other variables, however, this one has a sequence length of 4. Input shapes into my model would be the following: input X: [batch size, 92, 9] and target Y: [batch size, 4, 7].</p>
<p>Further information is that both sequences (the X sequence, and the Y sequence) co-occur, for which I mean that the sequence of 4 is thus not an extension of the sequence of 92.</p>
<p>Initially I thought that this is no problem for an LSTM model, however, my initial model would predict a sequence of 92, not 4, simply because an input of 92 is inputted. So I came across that one could achieve a different output sequence length by making use of an encoder/decoder architecture, however I think I am currently only pretending it is:</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(Encoder, self).__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, batch_first=True)
        
    def forward(self, x):
         # output hidden[0] has shape [1, batch size, hidden size]
        _, hidden = self.lstm(x)
        return hidden

class Decoder(nn.Module):
    def __init__(self, hidden_size, input_size, output_size):
        super(Decoder, self).__init__()
        self.hidden_size = hidden_size

        self.lstm = nn.LSTM(int(hidden_size/4), hidden_size, batch_first=True)
        self.linear = nn.Linear(hidden_size, output_size) 
        
    def forward(self, hidden):
        # output has shape [batch size, 4, output size]
        hidden = torch.flatten(hidden[0], end_dim=1)
        hidden = hidden.unsqueeze(1)
        hidden = hidden.reshape(-1, 4, int(self.hidden_size/4))
        output, _ = self.lstm(hidden)
        output = self.linear(output)
        return output

class seq2seq(nn.Module):
    
    def __init__(self, input_size, output_size, hidden_size):

        super(seq2seq, self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size

        self.encoder = Encoder(input_size, hidden_size)
        self.decoder = Decoder(hidden_size, input_size, output_size)
</code></pre>
<p>My understanding is that the encoder wraps the input sequence into a code, and the decoder sort of unwraps it into the output sequence.</p>
<p>What I currently did, is using the encoder class to obtain the last hidden state, which I then want to unwrap in the decoder class. But the reason I use this architecture in the first place was to obtain an output sequence of 4, and I don’t think I have figured it out yet. What I currently do is practically manually shaping the ‘hidden’ tensor into a shape of [batch size, hidden size] and then adding a new dimension (unsqueeze), and then reshaping it into [batch size, 4, hidden size/4]. I tried something with ‘teacher forcing’ but I just failed to understand it and ran into the same problem eventually.</p>
<p>What am I missing in my architecture? Or is this ok?</p>
Finding out dataset format for pytorch model Hybridnets

ARK3r: <p>Hi everyone!</p>
<p>I’m trying to train my own dataset on the <a href="https://github.com/datvuthanh/HybridNets" rel="noopener nofollow ugc">HybridNets</a> architecture and I tried looking at the links provided for the dataset to find out the structure of the data but the links were broken.</p>
<p>I looked at some other segmentation models and they seemed to be using two image files each in train_images and train_masks and for object detection, it’s a file with a set of points that make up a polygon. Based on my understanding, for object detection, there are multiple formats to display this encoding.</p>
<p>So now my question is, how should I set up my dataset to train for HybridNets on it?</p>
<p>And how did you find out the answer to this? (so that I can figure it out myself next time)</p>
Fuse these operator for faster inference speed?

luvwinnie: <p>I wish to fuse these operator  when export to onnx, is it possible to fuse these operator so my model inference speed can be improve?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/b/cbe4b5b0159881c6b9bf38d7fbb11c195233476e.png" data-download-href="https://discuss.pytorch.org/uploads/default/cbe4b5b0159881c6b9bf38d7fbb11c195233476e" title="Screenshot 2023-01-19 at 18.42.30"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cbe4b5b0159881c6b9bf38d7fbb11c195233476e_2_278x500.png" alt="Screenshot 2023-01-19 at 18.42.30" data-base62-sha1="t5IVXpW22p2Cg2Z0Q3klz0kGVm6" width="278" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cbe4b5b0159881c6b9bf38d7fbb11c195233476e_2_278x500.png, https://discuss.pytorch.org/uploads/default/original/3X/c/b/cbe4b5b0159881c6b9bf38d7fbb11c195233476e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/b/cbe4b5b0159881c6b9bf38d7fbb11c195233476e.png 2x" data-dominant-color="404041"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-01-19 at 18.42.30</span><span class="informations">374×672 15.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

AlphaBetaGamma96: <p>Have you looked into <code>torch.compile</code>? <a href="https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">torch.compile Tutorial — PyTorch Tutorials 1.13.1+cu117 documentation</a></p>

luvwinnie: <p>I have tried with torch.compile, but it shows the below errors.</p>
<pre><code class="lang-auto">File /misc/home/usr16/cheesiang_leow/venv_31_11.5/lib/python3.8/site-packages/torch/_dynamo/utils.py:1068, in get_fake_value(node, tx)
   1064 elif isinstance(
   1065     cause, torch._subclasses.fake_tensor.DynamicOutputShapeException
   1066 ):
   1067     unimplemented(f"dynamic shape operator: {cause.func}")
-&gt; 1068 raise TorchRuntimeError() from e

TorchRuntimeError: 

from user code:
   File "/misc/DataStorage2/cheesiang_leow/artibrains/single_line_v2/modules/feature_extraction.py", line 253, in forward
    return self.ConvNet(input).permute(0, 3, 1, 2).mean(dim=(-1))
  File "/misc/DataStorage2/cheesiang_leow/artibrains/single_line_v2/modules/feature_extraction.py", line 210, in forward
    x = self.bn0_1(x)

Set torch._dynamo.config.verbose=True for more information


You can suppress this exception and fall back to eager by setting:
    torch._dynamo.config.suppress_errors = True
</code></pre>

AlphaBetaGamma96: <p>Do you have a minimal reproducible example?</p>
Why does this Adam implementation cursed?

neva: <p>when runing this code I get nan loss which makes no since.<br>
i ran identical code with the stock wadam and it worked fine<br>
whats even weirder is that even the first loss is nan</p>
<p>class Adam():<br>
def <strong>init</strong>(self,parametrs,lr=0.01,betas=(0.9,0.999),eps=1e-8,decay= 0.):<br>
self.groups=[{‘params’:p,<br>
‘mom1’:torch.zeros(p.shape).to(p.device),<br>
‘mom2’:torch.ones(p.shape).to(p.device),<br>
‘max_mom2’:torch.ones(p.shape).to(p.device)} for p in parametrs]<br>
self.lr=lr<br>
self.betas=betas<br>
self.eps=eps<br>
self.decay=decay<br>
self.step=0<br>
def update(self):<br>
self.step+=1<br>
with torch.no_grad():<br>
for p in self.groups:<br>
grad=p[‘params’].grad</p>
<pre><code>    p['mom1']=grad*(1-self.betas[0])+p['mom1']*self.betas[0]
    p['mom2']=grad*(1-self.betas[1])+p['mom2']*self.betas[1]
    #p['max_mom2']=torch.maximum(p['mom2'],p['max_mom2'])

    m=p['mom1']/(1-self.betas[0]**self.step)
    v=p['mom2']/(1-self.betas[1]**self.step)
    #v=p['max_mom2']/(1-self.betas[1]**self.step)
    v=torch.sqrt()

    p['params'].data-=self.lr*m/(v+self.eps)
    p['params'].data-=self.decay*p['params']
</code></pre>
<p>opt=Adam(model2.parameters())</p>

ordalian: <p><code>v=torch.sqrt()</code> is incorrect, sqrt is a function not a class. I think you might have wanted to do <code>v = torch.sqrt(v)</code>, or you can use the class method <code>v=v.sqrt()</code>.</p>

neva: <p>bug is so obvious its mind numbing. there is no **2 on the second moment</p>

neva: <p>no thats not the issue its honestly idiotic by me.<br>
i didnt square the second moment</p>
Batching with graph as input and sentences as output

RishabhSheoran: <p>Hi! I have a problem where my input is graph and the output is sentences. I’m using GCN as an encoder and then  pool the features of all nodes. I’m using LSTM as a decoder. My question is what is the most efficient way to do batching in this case as the graphs have different number of nodes and the sentences are of variable lengths too. Each graph correspond to one sentence in target.</p>
<p>My approach:<br>
I’m thinking of padding the sentences with the max length of a sentence in a batch. For the graphs, I have node embeddings as stacked word embeddings of all nodes of a graph (each node has one word) and I have an adjacency matrix for the graph. I’m thinking of padding the node embeddings with the zeros according to the maximum number of nodes for graphs in a batch and use padding again for the adjacency matrix.</p>

ArchieGertsman: <p>If you haven’t already looked at <a href="https://pytorch-geometric.readthedocs.io/en/stable/" rel="noopener nofollow ugc">PyTorch Geometric</a>, it’s a great tool for graph neural networks. It allows you to seamlessly batch data of varying sizes.</p>
Using Pytorch SentencePiece to go from Indices to Text

J_Johnson: <p>Hello! I’m using Pytorch’s <a href="https://pytorch.org/text/stable/data_functional.html" rel="noopener nofollow ugc">SentencePiece model</a> again, which is far more efficient than any other tokenizer system I’ve come across. However, I’m having difficulty getting back from indices to tokens(text) using the model. Here is a wish list, which, any one of these would solve the issue for me:</p>
<ol>
<li>A built-in command to convert from indices to tokens, similar to <code>vocab.get_itos[index]</code> (<a href="https://pytorch.org/text/stable/vocab.html#torchtext.vocab.Vocab.get_itos" rel="noopener nofollow ugc">see here</a>);</li>
<li>A reverse method of <code>sentencepiece_numericalizer</code>;</li>
<li>A method to extract from the saved SP model a dictionary containing the tokens: indices, or vice versa.</li>
</ol>
<p>TIA</p>

J_Johnson: <p>The <code>generate_sp_model</code> creates two files, one ending in <code>.model</code> and the other ending in <code>.vocab</code>. Using the following workaround for now:</p>
<pre><code class="lang-auto">from torchtext.vocab import build_vocab_from_iterator

filestr=open("m_user.vocab", "r", encoding="utf-8")

x=filestr.read().splitlines()

for k in range(len(x)):
    x[k]=x[k][:x[k].find("\t")].replace("\t", "")

def yield_tokens(tokenlist):
    for token in tokenlist:
        yield token.strip().split()

vocab=build_vocab_from_iterator(yield_tokens(x))
</code></pre>
<p>And then I can call <code>vocab.get_itos(index)</code> or <code>vocab.get_stoi(text)</code> accordingly. Still looking for an optimal solution, but this gets the job done.</p>
Manually computing the euclidean norm of the difference between two MNIST images with PyTorch

trmpkm: <p>I’ve been playing around with the MNIST dataset and PyTorch and have loaded my dataset like so:</p>
<pre><code class="lang-auto">#Loading the MNIST training and test data:

train_data = datasets.MNIST(
    root = 'data',
    train = True,                         
    transform = ToTensor(), 
    download = True,            
)
test_data = datasets.MNIST(
    root = 'data', 
    train = False, 
    transform = ToTensor()
)
</code></pre>
<p>what’s interesting to me is that I am thinking of these images as 28 by 28 matrices with each entry representing the shade of the pixel. So I wanted to compute the Euclidean distance between two images after flattening them, in particular, I defined the following function:</p>
<pre><code class="lang-auto">def wtrain(i,j):
   s = train_data.data[i]
   t = train_data.data[j]
   s = torch.flatten(s)
   t = torch.flatten(t)
   d = (s-t)/1000
   d = (torch.norm(d))**2
   return d
</code></pre>
<p><strong>What’s weird is that wtrain[0,1] does not equal wtrain[1,0]?</strong></p>
<p>I was wondering if anyone could see why or offer an alternative way to compute this distance.</p>

ptrblck: <p>Both output tensors are using their raw <code>uint8</code> <code>dtype</code> since you are indexing the internal <code>.data</code> attribute in <code>train_data.data[index]</code> and the <code>ToTensor</code> transformation is thus skipped.</p>
<p>The subtraction in <code>(s - t)</code> will thus under/overflow and will create different outputs based on the order.<br>
You could either transform <code>s</code> and <code>t</code> to <code>float32</code> before applying the subtraction:</p>
<pre><code class="lang-python">s = torch.flatten(s).float()
t = torch.flatten(t).float()
</code></pre>
<p>or you could index the <code>Dataset</code> which will normalize  and cast the image tensors to <code>float32</code> by applying <code>ToTensor</code> on them:</p>
<pre><code class="lang-python">s = train_data[i][0]
t = train_data[j][0]
</code></pre>
Questions about choosing metrics

ChineseBest: <p>Hi, for a specific machine learning problem, is it possible to me to choose a function, like f(a) to evaluate a, while my loss function is actually generated from g(f(a))? If yes, are there any examples? If not, why I cannot use it? Thanks a lot.</p>

ptrblck: <p>I’m not sure if I have ever seen a use case of an evaluation function <code>f(a)</code> and a loss <code>g(f(a))</code>, but I don’t see why it couldn’t work assuming both methods are differentiable.<br>
Note that you are often using a metric function <code>f(a)</code> and a loss <code>g(a)</code> such as the accuracy as the metric function and e.g. cross-entropy loss as the loss function.</p>
Details on how workers prefetch data?

Vinayaka_Hegde: <p>Hi, lets assume that <code>num-workers</code> = 8 and <code>batch_size</code>=2.<br>
So, 2*8 = 16 number of batches have to be pre loaded .</p>
<p>Now, when the current batch is consumed by a worker and <code>next</code> is called on the data loader, how many new batches does it fetch? As I understand, 1 new batch is fetched for every consumed batch to maintain the constant count of 16 batches each time. Is this correct ?</p>
<p>Do the number of batches always remain 16 and for every batch consumed, a new batch would be fetched ?</p>

ptrblck: <aside class="quote no-group" data-username="Vinayaka_Hegde" data-post="1" data-topic="170619">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vinayaka_hegde/48/56332_2.png" class="avatar"> Vinayaka_Hegde:</div>
<blockquote>
<p>So, 2*8 = 16 number of batches have to be pre loaded .</p>
</blockquote>
</aside>
<p>This would be the case if the default <code>prefetch_factor</code> of <code>2</code> is used but note that the <code>2</code> is not coming from the <code>batch_size</code>.<br>
You can take a look at <a href="https://discuss.pytorch.org/t/when-does-a-pytorch-dataset-worker-pull-new-data/153286/2">this code snippet</a> to see how the <code>prefetch_factor</code> works.</p>
Audio pronounciation analyzer

ammar_siddiqui: <p>I have audio files as a dataset and i want to create a model which will rate the user provided audio as good, bad or moderate with respect to the pronounciation. Which approach can I use to make a model of this kind?</p>

tkuser: <p>If you have sequential data, one of the classical approaches would be to use recurrent neural network or LSTM (however I have used 1D CNNs at it also worked out perfectly). Another newer approach would be the use of transformer architecture for this kind of task which in my case has shown to get good performance and was associated with working better to out of distribution samples (see for instance [1].</p>
<p>[1] <a href="https://proceedings.neurips.cc/paper/2021/hash/3941c4358616274ac2436eacf67fae05-Abstract.html" class="inline-onebox" rel="noopener nofollow ugc">Exploring the Limits of Out-of-Distribution Detection</a></p>

ammar_siddiqui: <p>Will look into it. Thanks</p>
Tracking a dataloader when loading large numpy arrays

aquorio15: <p>Hi everyone</p>
<p>I am loading large numpy arrays of over &gt;1GB in size. In total i have around 1,00,000 such files. But the issue is when i am trying to load them in the dataloader i am not getting a progress bar. I used tqdm on the dataloader but it is not showing the progress bar.<br>
This is what i have done</p>
<blockquote>
<p>class CustomDataset(Dataset):</p>
</blockquote>
<blockquote>
<pre><code>def __init__(self, source_path, target_path, transform=None):
    self.source_data = [np.load(path, allow_pickle=True, mmap_mode='r') for path in source_path]
    self.target_data = [np.load(path, allow_pickle=True, mmap_mode='r') for path in target_path]
    self.start_indices = [0] * len(source_path)
    self.data_count_source = 0
    self.data_count_target = 0
    for index, memmap in enumerate(self.source_data):
        self.start_indices[index] = self.data_count_source
        self.data_count += memmap.shape[0]
    
    for index, memmap in enumerate(self.target_data):
        self.start_indices[index] = self.data_count_target
        self.data_count += memmap.shape[0]

def __len__(self):
    return self.data_count_source

def __getitem__(self, index):
    memmap_index_source = bisect(self.start_indices, index) - 1
    index_in_memmap_source = index - self.start_indices[memmap_index_source]
    source_data = self.source_data[memmap_index_source][index_in_memmap_source]
    memmap_index_target = bisect(self.start_indices, index) - 1
    index_in_memmap_target = index - self.start_indices[memmap_index_target]
    target_data = self.target_data[memmap_index_target][index_in_memmap_target]
    return source_data, target_data
</code></pre>
</blockquote>
<blockquote>
<p>root_dir = ‘en-hi/’<br>
source_path, target_path = os.path.join(root_dir, ‘train’, ‘source’), os.path.join(root_dir, ‘train’, ‘target’)<br>
train_source_file = glob.glob(source_path + ‘/<em>.pt.npy’)<br>
train_target_file = glob.glob(target_path + '/</em>.pt.npy’)<br>
train_dataset = CustomDataset(train_source_file, train_target_file)<br>
train_dataloader = DataLoader(train_dataset, batch_size=1, drop_last=False, num_workers=0)<br>
pbar = tqdm(train_dataloader)<br>
for d in pbar:<br>
pass</p>
</blockquote>
<p>Is there any way to track the update of loading the numpy arrays? Thank you for any kind of help</p>
Profiling newly created threads during a run

taehyunzzz: <p>Hi, my program uses threading library to run some code in parallel to the main thread.</p>
<p>I am doing something that looks like :<br>
(during backward pass)<br>
t1 = threading.Thread(my_function_to_run_in_parallel)<br>
t1.start()</p>
<p>(at the end of backward pass)<br>
t1.join()</p>
<p>torch profiler shows the following result as trace<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55eb18b231c875c15a7bc63e0ced248ecf08d986.png" data-download-href="https://discuss.pytorch.org/uploads/default/55eb18b231c875c15a7bc63e0ced248ecf08d986" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55eb18b231c875c15a7bc63e0ced248ecf08d986_2_690x424.png" alt="image" data-base62-sha1="cg4had7CUTaq4MEku3dhxhNmB3U" width="690" height="424" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55eb18b231c875c15a7bc63e0ced248ecf08d986_2_690x424.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55eb18b231c875c15a7bc63e0ced248ecf08d986_2_1035x636.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55eb18b231c875c15a7bc63e0ced248ecf08d986_2_1380x848.png 2x" data-dominant-color="EFF0F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1589×978 60.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there any way to profile the contents of my_function_to_run_in_parallel?<br>
Inserting record_functions inside this function creates segfaults <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
How does Conv2D down sampling work behind the scenes?

setsailforfail: <p>From my current understanding, the following operation:</p>
<pre><code class="lang-auto">input = torch.rand(size=(3, 5, 10, 10))
conv = nn.Conv2D(input_channels=5, output_channels=1, kernel_size=(3,3))
output = conv(input)
</code></pre>
<p>produces a tensor of size <code>(3, 1, 8, 8)</code>. To get from 5 channels to 1 channel, PyTorch uses a kernel of the shape <code>(5, 3, 3)</code>. Is this understanding correct so far?</p>
<p>If so, what if we apply the following operation:</p>
<pre><code class="lang-auto">input = torch.rand(size=(3, 5, 10, 10))
conv = nn.Conv2D(input_channels=5, output_channels=3, kernel_size=(3,3))
output = conv(input)
</code></pre>
<p>Do we use simply 3 different kernels of the size <code>(5, 3, 3)</code>?</p>

ptrblck: <p>Yes, each output channel is created by a separate kernel in the default setup and the actual <code>.weight</code> shape is defined as <code>[out_channels, in_channels, height, width]</code>.</p>