akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Use some of the input channel of the convolution kernel

username: FeHng
cooked: <p>I have a conv layer with input_channel=10 and output_channel=20. Usually I need take  feature maps with shape [B,H,W,C=10] as input. But I want use different channel at different time, the number of channels will not be greater than 10. For example, I want take feature maps with shape [B,H,W,C=5] as input. That means, in theory, only half of the kernel will be used, which makes a speed up.<br>
The easiest way is to pad the feature maps to full shape,but I don’t want do this. Because this means I can’t get acceleration.</p>

username: ptrblck
cooked: <p>You could slice the conv kernel and use the functional API via <code>F.conv2d</code>.</p>
Object detection labeling tactics

username: blackbirdbarber
cooked: <p>I am having this image and labeling it.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88c1795d312030d4135cf535881e50586ff738e7.jpeg" alt="image" data-base62-sha1="jvNs2zsyqnh6QyueqDIikHdpHvh" width="370" height="224"><br>
Should I label every apple, or just the whole bulk of apples?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f68ab096273091a42e078139635ba320885e7239.jpeg" alt="image" data-base62-sha1="zb0B9t6jusY5v0r8JkyoLeqNNKV" width="367" height="227"></p>
<p>The problem I am having classes like biological, metal, paper, cardbox, etc.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5ba7bbe4b21698ca6690d860e9ec43192b32c119.jpeg" alt="image" data-base62-sha1="d4OOJ0gq8s4rzZ9fyNHSTSKS17P" width="359" height="224"><br>
In this case all these objects are biological.</p>
Outputs of F.conv2d are different when input is 5D tensor

username: garyyyccc
cooked: <p>I’m trying to do convolution for 5D tensors while I got the different outputs for the below two methods. The problem is that when I use the different batch size, sometimes the outputs are the same but sometimes not in CUDA mode. However, The two methods get the same outputs when I use CPU.</p>
<p>I think this may caused by CUDA memory. Anyone knows how to fix this problem?</p>
<pre><code class="lang-auto">x = torch.randn([batch_size, N, in_channels, in_width, in_height]).to("cuda")
w = torch.randn([out_channels, in_channels, k, k]).to("cuda")

# the first method
y_shape = [x.shape[0], x.shape[1]]
# merge the first two dimensions 
x_flat = x.flatten(0, 1).contiguous()  
y1 = F.conv2d(x_flat, w, None)
y_shape.extend(y1.shape[1:])
# reshape to [batch_size, N, out_channels, out_w, out_h]
y1 = y1.view(y_shape)

# the second method
for n in range(N):
    y2 = F.conv2d(x[:,n,...], w, None)
    # compare
    print(torch.equal(y1[:, n, ...], y2))
</code></pre>
<p>I’m using NVIDIA GTX 2080s with CUDA 11.6.<br>
My Pytorch version is 1.9.0.</p>
<p>When I set the batch size to 256, they are all different. But when the batch size to 128, the outputs are all the same.<br>
In my example,</p>
<pre><code class="lang-auto">batch_size = 256 #128
N = 16
in_channels = 3
in_width = 32
in_height = 32
out_channels = 256 
k = 3
</code></pre>
<p>Thanks for any help!</p>

username: KFrank
cooked: <p>Hi Gary!</p>
<aside class="quote no-group" data-username="garyyyccc" data-post="1" data-topic="145675" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/f07891/48.png" class="avatar"> garyyyccc:</div>
<blockquote>
<p>The problem is that when I use the different batch size, sometimes the outputs are the same but sometimes not in CUDA mode.</p>
</blockquote>
</aside>
<p>Check whether the discrepancy is consistent with floating-point round-off<br>
error.  (An easy way to do this is to repeat the computation after converting<br>
your tensors to <code>.double()</code> which should cause round-off error to drop by<br>
several orders of magnitude.)</p>
<p>Round-off errors are to be expected because changing the size of your<br>
tensors can cause pytorch to change the specific order in which the<br>
computations are performed, leading to results that would have been<br>
mathematically the same but differ by round-off error.</p>
<p>Best.</p>
<p>K. Frank</p>
Autograd package in no_grad Environment

username: Ilker_GURCAN
cooked: <p>Hello Pytorch devs and users,</p>
<p>My question may look a bit awkward at a first glance; but it turns out to be useful in some occasions. For the model I have been working on, I need to take the derivative of a function and then, I should create its corresponding graph to have it in my cost function. Furthermore, this derivative operator should also operate during validation/testing phase, b/c it is a derivative w.r.t. <strong>input</strong> rather than <strong>parameters</strong>.</p>
<p>However, the problem is that when I employ <code>with torch.no_grad()</code> to do validation (likewise testing), autograd package <em>naturally</em> starts to complain. For the time being, I use the following two lines of code to do my validation; though I face <strong>deadlocks</strong> when I run my model using <code>DistributedDataParallel</code> to do distributed training in a single node/multi gpu environment:</p>
<pre><code class="lang-auto">myModel.eval()
myModel.requires_grad_(False)
</code></pre>
<p>Is there a way to use autograd package to take derivative of a function w.r.t. inputs in <code>with torch.no_grad()</code> environment?</p>

username: Dazitu616
cooked: <p>Hi, I’ve also encountered this problem recently, because my network requires some gradient as input (like an error signal). <strong>I searched and asked someone and there seems to be impossible to calculate gradient in <code>torch.no_grad()</code></strong>. So I have to do some hacks by only applying <code>torch.no_grad()</code> to the pipelines that don’t require grad, while manually cleaning the grad after calculating the derivative I need</p>

username: Ilker_GURCAN
cooked: <p>Oh. I see. I thought so. Well… <strong>Hope springs eternal</strong> <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"> like they say… Anyway, the cumbersome issue with my model is that the autograd is required deep beneath within one of my modules; hence there is no even a slim chance for me to use <code>torch.no_grad()</code> during validation. Probably, I will continue with the workaround which disables autograd on weights (while keeping it functional for the input) during validation phase:</p>
<pre><code class="lang-auto">myDDP.module.eval()
myDDP.module.requires_grad_(False)
</code></pre>
<p>Then, I will put it back in training mode:</p>
<pre><code class="lang-auto">myDDP.module.train()
myDDP.module.requires_grad_(True)
</code></pre>
<p>It seems this is the only option for me to both avoid deadlock caused by DDP at single node/multi-GPU training and keep autograd package functional on inputs during validation phase.</p>
<p>Thanks for the reply, <a class="mention" href="/u/dazitu616">@Dazitu616</a></p>
Dataloader stops after first epoch

username: ali-ism
cooked: <p>Hello,</p>
<p>I have implemented the Bucket Iterator using the DataLoader as done in the <a href="https://github.com/pytorch/text/blob/master/examples/legacy_tutorial/migration_tutorial.ipynb" rel="noopener nofollow ugc">torchtext legacy migration guide</a>.</p>
<p>However, this implementation results in the bucket DataLoader expiring after the first epoch. The only way I’ve been able to overcome this is by instantiating a new DataLoader inside the for loop going over the epochs.</p>
<p>Why is this DataLoader not resetting like it usually should?</p>
<p>Below is a snippet of my code for reference.</p>
<pre><code class="lang-auto">dataset = EnFRDataset() #map_style Dataset, each sample is a pair of english-french sentences
splits = [int(0.8*len(dataset))+2, int(0.1*len(dataset)), int(0.1*len(dataset))]
train, val, test = random_split(dataset, splits, generator=torch.Generator().manual_seed(41))

def collate_batch(batch):
    src_list, trg_list = [], []
    for data in batch:
        src_list.append(data[0])
        trg_list.append(data[1])
    src = pad_sequence(src_list, padding_value=dataset.en_vocab['&lt;PAD&gt;'], batch_first=True)
    trg = pad_sequence(trg_list, padding_value=dataset.fr_vocab['&lt;PAD&gt;'], batch_first=True)
    return src, trg

def batch_sampler(subset):
    indices = [(i, len(data[0])) for i, data in enumerate(subset)]
    random.shuffle(indices)
    pooled_indices = []
    # create pool of indices with similar lengths 
    for i in range(0, len(indices), batch_size * 100):
        pooled_indices.extend(sorted(indices[i:i + batch_size * 100], key=lambda x: x[1]))
    pooled_indices = [x[0] for x in pooled_indices]
    # yield indices for current batch
    for i in range(0, len(pooled_indices), batch_size):
        yield pooled_indices[i:i + batch_size]

batch_size = 10

for epoch in range(5):
    #without this line the Dataloader will expire after the first epoch
    bucket_dataloader = DataLoader(train, batch_sampler=batch_sampler(train), collate_fn=collate_batch)
    for _, (X, y) in enumerate(bucket_dataloader):
        #train model
</code></pre>
<p>This issue was also raised in <a href="https://discuss.pytorch.org/t/iterable-style-dataset-exhausts-after-1-epoch/124368">another post</a> that was doing the same thing but never got a response.</p>
<p>Thanks.</p>
How to implement a custom learning rate scheduler?

username: ShaneTian
cooked: <p>I want to implement <code>NoamDecay</code> in PyTorch.</p>
<p>I see that the official LR scheduler contains two methods: <code>get_lr</code> and <code>_get_closed_form_lr</code>, such as <a href="https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#LinearLR" rel="noopener nofollow ugc">LinearLR</a>. What is the difference of them?</p>
<p>This is my code. It does NOT work. I guess because of <code>_get_closed_form_lr</code> or <code>self.last_epoch</code>?</p>
<pre data-code-wrap="py"><code class="lang-nohighlight">class NoamLR(_LRScheduler):
    """Implements the Noam Learning rate schedule. This corresponds to increasing the learning rate
    linearly for the first ``warmup_steps`` training steps, and decreasing it thereafter proportionally
    to the inverse square root of the step number.
    
    Args:
        optimizer (Optimizer): Wrapped optimizer.
        warmup_steps (int): The number of steps to linearly increase the learning rate. Default: 0.
        last_epoch (int): The index of the last epoch. Default: -1.
        verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``.
    """

    def __init__(self, optimizer, warmup_steps=0, last_epoch=-1, verbose=False):
        self.warmup_steps = warmup_steps
        super(NoamLR, self).__init__(optimizer, last_epoch, verbose)

    def get_lr(self):
        """Compute learning rate using the scheduler.

        Returns:
            list: The learning rate list of each parameter group.
        """
        if self.last_epoch == 0:
            return [group["lr"] for group in self.optimizer.param_groups]

        return [group["lr"] * (self.warmup_steps ** 0.5) *
                min(self.last_epoch ** -0.5, self.last_epoch * (self.warmup_steps ** -1.5))
                for group in self.optimizer.param_groups]
</code></pre>
Which of these learning rate schedulers is better suited for adam optimizer and 100 epochs?

username: Mona_Jalal
cooked: <p>I am using adam optimizer and 100 epochs of training for my problem. I am wondering which of the following two learning rate schedulers sound better?</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(model.parameters(
), lr=learning_rate, weight_decay=5e-4)       # best:5e-4, 4e-3
##exp_lr_scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[20, 40, 90], gamma=0.1)  # gamma=0.3  # 30,90,130 # 20,90,130 -&gt; 150
# or
# Decay LR by a factor of 0.1 every epoch
exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.1)
</code></pre>

username: Dazitu616
cooked: <p>The <code>MultiStepLR</code> one. People often decay lr by 2-3 times throughout training.</p>
Multiprocessing import error

username: sbelharbi
cooked: <p>– EDIT: it seems a python issue or related. dont know how. <code>from multiprocessing.util import register_after_fork</code> is a python module that is being imported in <code>torch.multiprocessing.reductions.py</code>. not sure why this issue raises because <code>from multiprocessing.util import register_after_fork</code> works fine in python. almost certain that it has something to do with the installation. tried code on different virtual env. torch multiprocessing works fine… creating a new new virtual env. and installing pytorch in it solved the issue.</p>
<p>hi,<br>
code was working fine.<br>
did a script where i used <code>import torch.multiprocessing as mp</code>. running the script throws the error bellow.<br>
now, running any script, without even multiprocessing, throws the same error.<br>
simply <code>import torch</code> throws the error.<br>
doing <code>import torch</code> in python or ipython interpreter is fine.<br>
any idea?<br>
<code>pytorch 1.9.0</code><br>
<code>Python 3.7.10 (default, Feb 26 2021, 18:47:35) [GCC 7.3.0] :: Anaconda, Inc. on linux</code><br>
thanks</p>
<p>error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "multips.py", line 3, in &lt;module&gt;
    import torch
  File "x/lib/python3.7/site-packages/torch/__init__.py", line 688, in &lt;module&gt;
    from torch import multiprocessing as multiprocessing
  File "x/lib/python3.7/site-packages/torch/multiprocessing/__init__.py", line 18, in &lt;module&gt;
    from .reductions import init_reductions
  File "x/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 7, in &lt;module&gt;
    from multiprocessing.util import register_after_fork
ModuleNotFoundError: No module named 'multiprocessing.util'; 'multiprocessing' is not a package
</code></pre>

username: gcramer23
cooked: <aside class="quote no-group" data-username="sbelharbi" data-post="1" data-topic="130189">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/f475e1/48.png" class="avatar"> sbelharbi:</div>
<blockquote>
<pre><code class="lang-auto">ModuleNotFoundError: No module named 'multiprocessing.util'; 'multiprocessing' is not a package
</code></pre>
</blockquote>
</aside>
<p><a class="mention" href="/u/sbelharbi">@sbelharbi</a> The module can not be found. I would try a clean install.</p>

username: sbelharbi
cooked: <p>yeah. i am making a new install…<br>
installing pytroch in a fresh virtual env. solved the issue.</p>

username: stonelazy
cooked: <p>Had the same issue, I deleted the $HOME.cache/torch location from my Mac and it solved the issue.</p>
<blockquote>
<p>rm -rf .cache/torch</p>
</blockquote>
Question about modifying graph model

username: ChrisLiu2
cooked: <p>Having some pretrained CNNs, I plan to add RNN’s between certain layers of the CNN. A way to do it is to parse the original model and rewrite code to map old parameters to those in a new network, but that would require a lot of redundant code and also infeasible for all of the models I have.</p>
<p>Another way to do it seems to be by using torch.fx. Conceptually, I should be able to just convert pretrained CNN’s to graph models, add the RNN’s as new nodes in the graph and recompile. But after reading the tutorial from pytorch I’m still not sure how to do this. I’m not sure if it’s possible to add new submodules to a module. Can someone give a pointer to whether this is possible and how this could be achieved?</p>
'STL10' object has no attribute 'targets'

username: Ashley_Gao
cooked: <p>I’m trying to access the targets attribute of an STL 10 object (after loading the dataset successfully). My code is</p>
<pre><code class="lang-auto">dataset_tgt = datasets.STL10(root='./data',
                              split='train',
                              transform=transform,
                              download=True)

idx = dataset_tgt.targets == label
dataset_tgt.targets = dataset_tgt.targets[idx]
dataset_tgt.data = dataset_tgt.data[idx]
</code></pre>
<p>And I get the error ‘STL10’ object has no attribute ‘targets’. I tried to replace targets with train_labels but the error persists. Any help would be great! Thank you guys so much!!</p>

username: Ashley_Gao
cooked: <p>Problem solved by changing targets to labels.</p>
Why gradient becomes 10 times smaller？

username: cvlearn913
cooked: <p>When I tested the automatic differential, the gradient didn’t look the same as what I calculated manully</p>
<pre><code class="lang-auto">import numpy as np
import  torch

data = torch.from_numpy(np.array([[0.5]],dtype=np.float32))
target  =torch.from_numpy(np.array([[5.5],[7.5]],dtype=np.float32))
W = torch.tensor([[2.0],[4.0]],requires_grad=True)
out_put =data*W
print("out_put",out_put)
loss = target-out_put
print("loss",loss)
loss.mean().backward()
print("loss.mean",loss.mean())
print("W.grad",W.grad)
</code></pre>
<pre><code class="lang-auto">out_put tensor([[1.],
        [2.]], grad_fn=&lt;MulBackward0&gt;)
loss tensor([[4.5000],
        [5.5000]], grad_fn=&lt;SubBackward0&gt;)
loss.mean tensor(5., grad_fn=&lt;MeanBackward0&gt;)
W.grad tensor([[-0.2500],
        [-0.2500]])

</code></pre>
<p>I think it will be -2.5,but get -0.25.Why does this happen？</p>

username: mMagmer
cooked: <p>your objective is</p>
<pre><code class="lang-auto">J = (.5*w-target)/2 #divicion is because of meam
dJ/dw = d(.25*w+target/2)/dw = .25
</code></pre>
<p>see, no problem.</p>
It seems BCELOSS does not do range check in torch-1.10.2 as torch-1.10.0+cu111

username: songyuc
cooked: <p>Hi, guys!<br>
I find that <code>nn.BCELoss</code> does not do range check in torch-1.10.2 as  torch-1.10.0+cu111, which may trigger error like " /…/Loss.cu: … [59,0,0] Assertion input_val ＞= zero &amp;&amp; input_val ＜= one failed.".<br>
So, I want to make it sure whether range check (from 0 to 1) is the official behavior of <code>nn.BCELoss</code>?</p>
<p>Your answer will be appreciated!</p>

username: ptrblck
cooked: <p>I cannot reproduce the issue using <code>1.10.2</code>:</p>
<pre><code class="lang-python">&gt;&gt;&gt; criterion = nn.BCELoss()
&gt;&gt;&gt; x = torch.randn(10, 1)
&gt;&gt;&gt; y = torch.empty(10, 1).uniform_(0, 1)
&gt;&gt;&gt; loss = criterion(x, y)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/miniforge3/envs/1.10.2_cu113/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/miniforge3/envs/1.10.2_cu113/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 603, in forward
    return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
  File "/miniforge3/envs/1.10.2_cu113/lib/python3.8/site-packages/torch/nn/functional.py", line 2915, in binary_cross_entropy
    return torch._C._nn.binary_cross_entropy(input, target, weight, reduction_enum)
RuntimeError: all elements of input should be between 0 and 1
&gt;&gt;&gt; loss = criterion(x.cuda(), y.cuda())
&gt;&gt;&gt; /pytorch/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [1,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [2,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [5,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [8,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:115: operator(): block: [0,0,0], thread: [9,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.

&gt;&gt;&gt; print(torch.__version__)
1.10.2+cu113
</code></pre>
<p>Both, the CPU and CUDA ops show the expected error.</p>

username: songyuc
cooked: <p>Thanks for your response. I will try how to reproduce my error case then.</p>
A6000 uses more memory than Titan

username: John_Grabner
cooked: <p>I just upgraded my HW to A6000 from Titan and noticed something unexpected. On Titan, smi would report GPU memory usage around 21G - 23G (can’t remember the exact number but Titan has 24G so had to be less than that). Now that same code is using 38G on A6000.</p>
<p>My dataloader does not know the HW I am using, so its parameters are fixed.</p>
<p>The only other thing I changed was to upgrade Dockerfile from  pytorch/pytorch:1.6.0-cuda10.1-cudnn7-runtime to  pytorch/pytorch:1.7.1-cuda11.0-cudnn8-runtime since A600 was not happy with the original.</p>
<p>Is this PyTorch just leaving stuff in memory since there is room or is there a default data structure that changed size in PyTorch?</p>

username: ptrblck
cooked: <p>You could check the memory usage via <code>print(torch.cuda.memory_summary())</code> to check how much memory is allocated and how much is reserved, i.e. is in the cache and can be reused.<br>
I would assume the cache is filling up and if you want to free this memory so that other applications could use it, you could call <code>torch.cuda.empty_cache()</code>.</p>

username: John_Grabner
cooked: <p>Thank you.  I’m using pytorch-lightining so they handle all things CUDA.<br>
I will however add print(torch.cuda.memory_summary()) as you suggested<br>
so that I can track actual usage.</p>
Verilog / Hardware acceleration for nn.conv2D

username: Johnkhk
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/5/65d5a80c08353dd0d5de8af9e7d4fb26cc7ff352.png" alt="image" data-base62-sha1="ewRYQapg1wSaX4ajsfa2rs4B23U" width="505" height="452"><br>
Was wondering if it is possible to replace the nn.Conv2d with a Verilog implemented function for hardware acceleration. I’m trying to run this on a Windows PC via simulation. I already have a verilog file convolver.v written. Also, if verilog is not an option, is there another FPGA or hardware accelerated implemetation to replace nn.Conv2d or nn.ReLU to increase inference time running on Windows 10 without fpga or boards/hardware?</p>
Applying the chain rule

username: sofiane
cooked: <p>I would like to compute the gradient of “critic” with respect to “actor” 's weights.<br>
(Trying to apply “deterministic policy gradient” equation (6) of the following paper <a href="http://proceedings.mlr.press/v32/silver14.pdf" rel="noopener nofollow ugc">http://proceedings.mlr.press/v32/silver14.pdf</a>)<br>
“actor” and “critic” are two neural networks, consider critic as Q(s, a) state-action value that returns the reward. Therefore I had to concatenate the “states” vector and the action vector “actor_actions” to feed it to the input of the critic. Then I have to compute the gradient of critic with respect to “states” by setting the requires_grad of “states” to True.</p>
<pre><code class="lang-auto">optimizer_actor.zero_grad()
actor_actions = actor(states)
critic_values = - critic(torch.cat((states, actor_actions), dim = 2)).sum()/batch_size # notice the minus to apply gradient ascent instead of descent
critic_values.backward()
optimizer_actor.step()
</code></pre>
<p>Does the previous code snippet compute the gradient with respect to “actor” weights?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="sofiane" data-post="1" data-topic="145560">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e19adc/48.png" class="avatar"> sofiane:</div>
<blockquote>
<p>Does the previous code snippet compute the gradient with respect to “actor” weights?</p>
</blockquote>
</aside>
<p>Yes, it should unless you have detached a tensor somewhere internally.<br>
The <code>actor_actions</code> tensor seems to be created by the <code>actor</code> so I assume its parameters were used to create the output. If so, then passing the concatenated tensor to the <code>critic</code> and calling <code>backward</code> on its output should calculate the gradients of both models.<br>
You can verify it by printing the <code>.grad</code> attributes of the parameters of both models after the <code>backward</code> call.</p>
pin_memory=True and num_worker >1 . Occur CUDA Driver Error

username: jinyuan.feng
cooked: <ol>
<li>use torch.spawn</li>
<li>dataloader: pin_memory=True.</li>
<li>dataloader: num_worker&gt;1.<br>
ERROR:  <code>RuntimeError: CUDA driver error: initialization error</code>
</li>
</ol>

username: ptrblck
cooked: <p>I guess you are trying to re-initialize the CUDA context in each new process, which will fail.</p>
Linear layer throws dimension error

username: artuntun
cooked: <p>Hi folks <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=12" title=":wave:" class="emoji" alt=":wave:" loading="lazy" width="20" height="20"></p>
<p>I have the following linear layer:</p>
<pre><code class="lang-python">self.decoder = nn.Linear(
    in_features=192,
    out_features=4,
    bias=False
)
self.bias = nn.Parameter(torch.zeros(4))
self.decoder.bias = self.bias
</code></pre>
<p>But when I run</p>
<pre><code class="lang-python">self.decoder(torch.randn(64, 20, 192))
</code></pre>
<p>I get a dimensions error:</p>
<pre><code class="lang-auto">Exception has occurred: RuntimeError
mat1 and mat2 shapes cannot be multiplied (1280x192 and 4x192)
</code></pre>
<p>Why is it that the linear layer is considering the input_features to be 4? The most weird thing is that if I run in a python terminal:</p>
<pre><code class="lang-auto">linear = nn.Linear(192, 4)
linear(torch.randn(64, 20, 192))
</code></pre>
<p>Then it works. Those two layers are exactly the same but within my model it complains</p>

username: ptrblck
cooked: <p>I cannot reproduce the issue given your code:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.decoder = nn.Linear(
            in_features=192,
            out_features=4,
            bias=False
        )
        self.bias = nn.Parameter(torch.zeros(4))
        self.decoder.bias = self.bias
        
    def forward(self, x):
        x = self.decoder(x)
        return x
    
model = MyModel()
x = torch.randn(64, 20, 192)
out = model(x)
print(out.shape)
# &gt; torch.Size([64, 20, 4])
</code></pre>
How to compute grad_sample for large model

username: doudeimouyi
cooked: <p>Hi,<br>
I am trying to Opacus for Vision Transformer model. Here is what ViT look like:</p>
<pre><code class="lang-auto">class ViT(nn.Module):
    def __init__(self, ...):
        .......
       self.class_token = nn.Parameter(torch.zeros(1, 1, dim))
       ........

    def forward(self, x):
        ......
        x = torch.cat((self.class_token.expand(b, -1, -1), x), dim=1)  
       ......
        return x    
</code></pre>
<p>Using Opacus for this model will raise an error because of the nn.Parameter here <code>self.class_token = nn.Parameter(torch.zeros(1, 1, dim))</code>. Unless we compute grad_sample for the whole ViT model:</p>
<pre><code class="lang-auto">@register_grad_sampler(ViT)
def compute_vit_grad_sample():
   ......
</code></pre>
<p>But if I re-write ViT like this:</p>
<pre><code class="lang-auto">class cls_token(nn.Module):
    def __init__(self, dim):
       self.class_token = nn.Parameter(torch.zeros(1, 1, dim))
           
    def forward(self, x):
        x = torch.cat((self.class_token.expand(b, -1, -1), x), dim=1) 
        return x    


class ViT(nn.Module):
    def __init__(self, ...):
        .......
       self.class_token = cls_token(dim)
           
    def forward(self, x):
        ......
        x = cls_token(x)
       ......
        return x    
</code></pre>
<p>In this way, it is true that now I only need to compute grad_sample for the <code>cls_token</code> instead of the whole <code>ViT</code> module?</p>
<pre><code class="lang-auto">@register_grad_sampler(cls_token)
def compute_cls_grad_sample():
   ......
</code></pre>
<p>Thanks!</p>

username: pstock
cooked: <p>Hi doudeimouyi,</p>
<p>Thanks for your interest! The second approach (wrapping the <code>cls_token</code> in a <code>nn.Module</code> and only implementing the <code>grad_sampler</code> for this module) would be correct.</p>
<p>Indeed, in this approach, you are calling the <code>forward</code> method of the module <code>cls_token</code>, hence Opacus is able to correctly compute the grad samples.</p>
<p>Note however that you need to replicate your approach (encapsulate <code>nn.Parameters</code> in a <code>nn.Module</code> for which you define a custom grad sampler) every time you directly use a parameter in a forward pass instead of relying on a <code>forward</code> method of a child module.</p>
<p>Do not hesitate if you have further questions.</p>
Misaligned address issue when using half precision

username: JizeCao
cooked: <p>I simply call <code>model.half()</code> and change my input data from float32 to float16. However, the model will throw <code>RuntimeError: cuda runtime error (74) : misaligned address</code> error at /opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/ATen/native/cuda/SoftMax.cu:545. The exact error address will change at each time I retrain the model.</p>

username: ptrblck
cooked: <p>Do you have a code snippet to reproduce this issue?<br>
We’ve seen a similar issue recently in <a href="https://github.com/NVIDIA/apex/issues/427#issuecomment-520675492" rel="nofollow noopener">apex/amp</a>, but could not reproduce it due to lack of information.</p>

username: JizeCao
cooked: <p>I guess the issue happens between the dataParallel wrapper and the half precision. I have tried apex/amp and the same issue exists except the <code>O1</code> optimization. The issue may be reproduced simply with a linear projection. The error’s position is not fixed, sometimes happens at the very first linear layer, I guess you can replicate it with simply a linear layer, &gt; 1 GPUs and DataParallel wrapper.</p>
<pre><code class="lang-auto">model, optimizer = amp.initialize(model, optimizer,
                                  opt_level='O2'
                                  )
model = torch.nn.DataParallel(model, device_ids=[0,1,2])
</code></pre>

username: ptrblck
cooked: <p>Could you rerun the code with <a href="https://github.com/NVIDIA/apex/tree/master/examples/simple/distributed" rel="nofollow noopener">DistributedDataParallel</a>?</p>

username: Miquel_Espinosa
cooked: <p><a class="mention" href="/u/jizecao">@JizeCao</a> <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I am encountering the same problem. Have you found any solution to the problem?</p>
<p>The error <code>RuntimeError: CUDA error: misaligned address</code> is thrown when float16 is used together with multiple GPUs.<br>
Specifically, I am using nn.DataParallel() with torch.cuda.amp.GradScaler().</p>
<p>There is this related <a href="https://github.com/pytorch/pytorch/issues/31935" rel="noopener nofollow ugc">issue</a> in GitHub.</p>
<p>Thanks in advance,</p>

username: ptrblck
cooked: <p>As described in the linked issue, could you create a new issue with steps how to reproduce it and tag me there, please?</p>

username: Miquel_Espinosa
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Sorry for the late reply</p>
<p>I tried to reproduce the error with a simple vanilla network, but the error did not arise in such case.<br>
So I suspected that it must have been thrown from a code error or some hardware specific thing.</p>
<p>But later on, I decreased the batch size and the error does not appear now, and I’m able to train correctly the model.<br>
Therefore, I think it was probably an issue with the memory, that it run out of memory or something like that.</p>
<p>Thanks for the support!</p>
Tensors appear to be on same device yet I'm still getting an error

username: EvanZ
cooked: <p>Just showing a few lines in my validation step:</p>
<pre><code class="lang-auto">        td = TweedieDevianceScore(power=self.power)
        x = self.base_model(x, mask)
        preds = torch.exp(self.linear(x).mean(axis=1).squeeze(1) + torch.log(exposure))
        print(f'preds device: {preds.device} y device: {y.device}')
        loss = td(preds=preds, targets=y)
</code></pre>
<p>The result of the print statement seems to indicate those tensors are all on the gpus:</p>
<blockquote>
<p>preds device: cuda:0 y device: cuda:0<br>
preds device: cuda:1 y device: cuda:1<br>
preds device: cuda:2 y device: cuda:2<br>
preds device: cuda:3 y device: cuda:3</p>
</blockquote>
<p>And yet I am seeing the following error:</p>
<pre><code class="lang-auto">Exception has occurred: RuntimeError       (note: full exception trace is shown but execution is paused at: validation_step)
Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
  File "/home/ubuntu/deep-behavior-embedding/src/model/lightning_model.py", line 375, in validation_step (Current frame)
    loss = td(preds=preds, targets=y)
</code></pre>
<p>If <code>preds</code> and <code>y</code> are on the gpu, what else could be going wrong here?</p>

username: EvanZ
cooked: <p>The answer to this was the <code>TweedieDevianceScore</code> needed to be moved to the same device. Very simple fix.</p>
Deformable RoIPooling causing error during multi-gpu training

username: Ashwin_Raju
cooked: <p>Hello,</p>
<p>I am using Deformable RoiPooling and FPN with multiple scales which uses trainable offsets. I am also training using DistributedDataParallel which requires all parameters to used to compute the loss.</p>
<p>Deformable RoIPooling doesn’t have to necessarily use all the scales for an batch. May be some scale may not be used in a given batch because of the level calculated using the RoiPool formula. At these cases some Deformable RoiPooling remains not used at that batch which leads to the error:</p>
<p><code>RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument </code>find_unused_parameters=True<code>to</code>torch.nn.parallel.DistributedDataParallel<code>; (2) making sure all </code>forward<code>function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's</code>forward<code>function. Please include the loss function and the structure of the return value of</code>forward<code> of your module when reporting this issue (e.g. list, dict, iterable).</code></p>
<p>How do I overcome this issue ? Any suggestions ?</p>
How to deal with spectrograms of different sizes

username: Luiz_Souza
cooked: <p>So, I’m building a variatoinal autoencoder but I don’t know how to define the input shape of my network, since each spectrogram of my dataset has a different shape.</p>
<p>How do i deal with that? I think i can’t simply resize it, since it’s a spectrogram and the audio will suffer from changes.</p>
Running into None gradients when optimizing function

username: treadmill
cooked: <p>Trying to optimize a custom defined function “copt2” in pytorch using standard optimizers. I got an error when trying to use Rprop so I switched to Adam. Adam optimizer will run but the objective is not decreasing and after checking the gradient is not being defined it seems. What is the cause of this? Is it the max function in the definition of copt2, which used to shift the input of the exp function so it doesnt blow up?</p>
<p>And if so, can I write the max function outside the copt2 definition and optimize properly?</p>
<pre><code class="lang-auto">x0=torch.randn((18,6),requires_grad=True)
epsilon=1e-3 
def copt2(x):
   N = x.size()[0]
   dimen = int(x.size()[1]/2)
   norm = LA.vector_norm(x, dim=1, keepdim=True)
   x = torch.tensor(x/norm, requires_grad=True)
   x1=x[...,:dimen]
   x2=x[...,dimen:]
   xxt1=torch.matmul(x1,x1.T)
   xxt2=torch.matmul(x2,x2.T)
   xxt3=torch.matmul(x2,x1.T)
   xxt4=torch.matmul(x1,x2.T)
   xxt=(xxt1+xxt2)**2+(xxt3-xxt4)**2
   xxt=torch.triu(xxt,1)
   s=torch.max(xxt)
   expxxt=torch.exp((xxt-s)/epsilon)
   u=torch.triu(expxxt,1).sum()
   f=s+epsilon*torch.log(u)
   return f
y=copt2(x0)
# initialize
print(y)
en_opt = [y]
torch.autograd.set_detect_anomaly(True)
optimizer = torch.optim.Adam([x0], lr=0.0001)
for i in range(20):
    optimizer.zero_grad()
    y = copt2(x0)
    y.backward(retain_graph=True)
    optimizer.step()
    en_opt.append(y.item())

    if (i + 1) % 20 == 0:
        print(x0.grad)
        print(i + 1, y)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">tensor(0.873624663431270431779296, grad_fn=&lt;AddBackward0&gt;)
None
20 tensor(0.873624663431270431779296, grad_fn=&lt;AddBackward0&gt;)
</code></pre>

username: albanD
cooked: <p>Why do you do <code>x = torch.tensor(x/norm, requires_grad=True)</code> ? This will create a brand new tensor that is unrelated to the given input.<br>
That would be the part that breaks the gradient propagation. You should just do <code>x = x / norm</code>.</p>

username: treadmill
cooked: <p><a class="mention" href="/u/alband">@albanD</a> Yeah, I just figured out that was the problem. Was about to delete question when I saw your answer. Thanks.</p>
LibTorch Tensor Masking

username: Orcun_Deniz
cooked: <p>Hey,</p>
<p>I have a tensor with size (1, 500, 9) and I want to get rows which satisfies condition (first_row_elem &gt;= 0.3) of this tensor, yielding a tensor shape (1, N, 9 | N &lt; 500).</p>
<p>In python I would do;</p>
<pre><code class="lang-auto">mask = tensor[:,:,0] &gt;= 0.3
filtered_tensor = tensor[mask]
</code></pre>
<p>But using LibTorch C++ i did;</p>
<pre><code class="lang-auto">at::Tensor mask = boxes3d.slice(2, 0, 1) &gt;= 0.3;
at::Tensor filtered_boxes = boxes3d.index({mask.squeeze(2)});
</code></pre>
<p>I was wondering if there were any better ways to do this in C++ LibTorch?</p>
ONNX vs Torch Output Mismatch

username: Orcun_Deniz
cooked: <p>I have a 2dConvNet that i export from PyTorch to ONNX and TorchScript. However, while TorchScript output matches the native PyTorch code’s output, ONNX model that i do inference with onnxruntime-gpu, does not match the source code’s output.<br>
I compare the outputs with real-life data and with <code>torch.isclose(atol=rtol=1e-5)</code>. When i use <code>torch.zeros</code>, <code>torch.ones</code>, <code>torch.rand</code> as input, it passes the test but input with higher dynamic range makes outputs differ.</p>
<p>Here are script, onnx and sample input data i’ve been using. <a href="https://drive.google.com/drive/folders/1bYdpwfZWhWqv35lk4dnY2v0yw2LP9nAo?usp=sharing" rel="noopener nofollow ugc"> link</a></p>

username: marksaroufim
cooked: <p>ONNX outputs on GPU are not necessarily deterministic <a href="https://github.com/microsoft/onnxruntime/issues/4611" class="inline-onebox" rel="noopener nofollow ugc">Inference on GPU is not deterministic · Issue #4611 · microsoft/onnxruntime · GitHub</a></p>

username: Orcun_Deniz
cooked: <p>I finally completed the production code and deployed the model. Saw that predictions from ONNX-&gt; Tensorrt model is still accurate.</p>
How can i average subword embedding?

username: Sammo1
cooked: <p>how can i average subword embedding vectors to generate an approximate vector for the original word as i get the embedding using this function</p>
<pre><code class="lang-auto">def get_bert_embed_matrix(bert):
    bert_embeddings = list(bert.children())[0]
    bert_word_embeddings = list(bert_embeddings.children())[0]
    mat = bert_word_embeddings.weight.data.numpy()
    return mat
</code></pre>
<p>and used tokenizer for BERT</p>
Matrix-Vector Multiplication over Batch

username: jpbohl
cooked: <p>Hi everyone,<br>
to compute a custom loss function I want to multiply a matrix of size (k x n), with n being the number of rows in one feature vector, with a batched feature vector of length (n * batch_size).<br>
Is there an efficient way to achieve this?</p>

username: AlphaBetaGamma96
cooked: <p>Can you post a minimal reproducible example or pseudocode of what you want to do?</p>
Questions about optimizers

username: vojkan7
cooked: <p>Hi everyone</p>
<p>Its about split learning</p>
<p>1.I have one server and 10 clients. The server has one optimizer for his part/layers. Does each client need to have an individual/its own optimizer?</p>
<p>2.Previosly I had for each client one optimizer</p>
<pre>client_model_list = [ResNet18Client(config).to(device) for client_num in range
(total_client_num)]


client_optimizer_list = [optim.SGD(client_model_list[client_num].parameters(), 
lr=0.05, momentum=0.9) for client_num in range(total_client_num)]</pre>
<p>Now I have my own parameters and when I create one optimizer, everything is good with training.</p>
<pre>for i, l in enumerate(p.children()):
    for m,n in enumerate(l.children()):
        if(m==8):
           ft1=n
        if(m==9):
           ft2=n
   params1 = list(ft1.parameters()) + list(ft2.parameters())
   frst_opt=torch.optim.SGD(params1, lr=0.05, momentum=0.9)</pre>
<p>But when I make for each client one optimizer, the loss is increasing 2,5…2,7…3,5… etc</p>
<pre>first_lst=[]
for p in client_model_list:
  u=0
  for i, l in enumerate(p.children()):
    for m,n in enumerate(l.children()):
      if(m==8):
        ft1=n
      if(m==9):
        ft2=n
    
    params1 = list(ft1.parameters()) + list(ft2.parameters())

  frst_opt=torch.optim.SGD(params1, lr=0.05, momentum=0.9)
  first_lst.append( torch.optim.SGD(params1, lr=0.05, momentum=0.9))</pre>
<p>Where I made a mistake/ any help?</p>
When does Pytorch initialize parameters?

username: Danil_Hu
cooked: <p>Hi, I’m now writing my own network with Pytorch. And I want to use a pretrained model in my net. Here is my overwriting <code>__init__()</code> code:</p>
<pre><code class="lang-python">class Generator(nn.Module):
    def __init__(self) -&gt; None:
        super(Generator, self).__init__()
        model_path = "somedir"
        chekpoint = torch.load(model_path)
        h_model = H_model()
        h_model.load_state_dict(chekpoint['model'])
        # 设置为测试模式
        h_model.eval()
        self.H_model = h_model
        self.unet = UNet(enc_chs=(9,64,128,256,512), dec_chs=(512, 256, 128, 64), num_class=3, retain_dim=False, out_sz=(304, 304))
</code></pre>
<p>Here, the h_model is loaded from checkpoint which I’ve trained it well. My question is that after the initialization, will the parameter in h_model changed? And why(I mean how does Pytorch treat self-defined layer when it initializes parameters? And when does Pytorch initialize parameters?)</p>

username: tom
cooked: <p>The <code>nn.Module</code> subclasses initialize their parameters in the <code>__init__</code>. For many modules in PyTorch itself, this is typically done by calling a method <code>reset_parameters</code>. So your code snippet should train starting from the checkpoint.<br>
Note that the standard initialization of many standard modules can be considered to <em>not</em> be state of the art. Thus applications very likely override the initialization either in the higher-level <code>__init__</code> or some other place in the code between instantiation and training start.</p>
<p>Best regards</p>
<p>Thomas</p>
Serialization RetinaNet

username: _jk
cooked: <p>What is wrong with this code?</p>
<pre><code class="lang-auto">model = torchvision.models.detection.retinanet.retinanet_resnet50_fpn(pretrained=True)
model.eval()
input_tensor = torch.rand(1,3, 224, 224)
script_model = torch.jit.trace(model, input_tensor)
script_model.save("test.pt")
</code></pre>
<p>It raises an error: “Only tensors, lists, tuples of tensors, or dictionary of tensors can be output from traced functions”</p>
Modify the softmax values without causing errors

username: Ashwin_Raju
cooked: <p>I am trying to do something similar to Instance Mapping projection, where I project the instance segmentation results (output from sigmoid function) to a softmax probability map. However, I get error saying that the  gradient computation has been modified by an inplace operation. How do I solve this ? any suggestions.</p>
<p>Here is the code</p>
<pre><code class="lang-auto">feature_probs = F.softmax(feature_probs,1) [ 1xCxHxW]
masks = masks_logits.sigmoid() [Nx1x28x28]
boxes = [Nx4]
classes = [Nx1] 
for  mask, box, pred_class in zip(masks, boxes, classes):
     w = box[3] - box[1]
     h = box[2] - box[0]
     resize_mask = interpolate(mask, [w,h])
     feature_prob[0][pred_class+1: box[1]:box[3],box[0]:box[2] ] = resize_mask

feature_prob_normalize = F.normalize(feature_prob, p=1 ,dim=0)
loss = calculate_loss(feature_prob_normalize, target)
</code></pre>
<p>Error : RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 7, 256, 256]], which is output 0 of SoftmaxBackward. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later.</p>

username: tom
cooked: <p>One easy thing you could try is to use <code>feature_probs = F.softmax(feature_probs, 1).clone()</code> instead. Then only the clone of the output tensor is modified, not the output tensor itself.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Ashwin_Raju
cooked: <p>Thanks !! It works <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Using a target size (torch.Size([1, 55, 46, 46])) that is different to the input size (torch.Size([1, 55, 47, 47])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size

username: specialized_boy
cooked: <p>I am getting error <code>/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:47: UserWarning: Using a target size (torch.Size([1, 55, 46, 46])) that is different to the input size (torch.Size([1, 55, 47, 47])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.</code> . My input size is <code>(1,55,46,46)</code> but i dont know why i am getting <code>[1, 55, 47, 47]</code> and unable to change it to <code>(1,55,46,46)</code>?</p>
<pre><code class="lang-python">class AutoEncoder(nn.Module):
  def __init__(self):
    super(AutoEncoder, self).__init__()
    self.encoder = nn.Sequential(
            nn.Conv2d(55, 16, 3, stride=1, padding=1),  # b, 16, 10, 10
            nn.ReLU(True),
            nn.MaxPool2d(2, stride=1),  # b, 16, 5, 5
            nn.Conv2d(16, 8, 3, stride=1, padding=1),  # b, 8, 3, 3
            nn.ReLU(True),
            nn.MaxPool2d(2, stride=1)  # b, 8, 2, 2
        )
    self.decoder = nn.Sequential(
            nn.ConvTranspose2d(8, 16, 3, stride=1),  # b, 16, 5, 5
            nn.ReLU(True),
            nn.ConvTranspose2d(16, 8, 5, stride=1, padding=1),  # b, 8, 15, 15
            nn.ReLU(True),
            nn.ConvTranspose2d(8, 55, 2, stride=1, padding=1),  # b, 1, 28, 28
            nn.Tanh()
        )
  def forward(self, x):
    x = self.encoder(x)
    print(x.shape)
    x = self.decoder(x)
    print(x.shape)
    return x
</code></pre>

username: aiGeekProgrammer
cooked: <aside class="quote no-group" data-username="specialized_boy" data-post="1" data-topic="145591">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/specialized_boy/48/14787_2.png" class="avatar"> specialized_boy:</div>
<blockquote>
<pre><code class="lang-auto">self.encoder = nn.Sequential(
            nn.Conv2d(55, 16, 3, stride=1, padding=1),  # b, 16, 10, 10
</code></pre>
</blockquote>
</aside>
<p>How did you calculated those values: # b, 16, 10, 10</p>
<p>If your input is:</p>
<pre><code class="lang-auto">input = torch.rand((1,55,46,46))
</code></pre>
<p>and you process this input through encoder you will get different shapes than those in your comments</p>
<pre><code class="lang-auto">conv_1 = Conv2d(55, 16, 3, stride=1, padding=1)  
t1 = conv_1(input)
t1.size()
&gt;&gt;&gt;torch.Size([1, 16, 46, 46])
</code></pre>