akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Benefit of nn.DataParallel without batched input

pmeier: <p>Consider applications, e. g. <a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html" rel="nofollow noopener">Neural Style Transfer</a>, where the input is not batched but rather a single image. Can an application like this benefit from multiple GPUs if the <code>model</code> is wrapped in a <code>nn.DataParallel</code>?</p>
<p>I’m asking, because I want to be sure before I buy new hardware. If I understand the <code>multiprocessing</code> correctly, for such applications multiple GPUs are useless and thus a single GPU with high memory would be the better choice.</p>

alex.veuthey: <p>Indeed, <code>nn.DataParallel</code> works by splitting batches on the available GPUs. <code>multiprocessing</code> however is used generally for loading data with multiple workers (CPU based) onto the GPU, therefore speeding training considerably.</p>
<p>I’m not familiar with style transfer training, but I don’t see a reason why training would happen with single image batches <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:">. Even for inference, you might want to process multiple images at once…</p>

pmeier: <p>In style transfer you do not train a model, but use the output of the intermediate layers of a pretrained model to optimize / train a single image. Thus, during the optimization nothing is loaded from the drive.</p>

alex.veuthey: <p>Interesting, thanks for the clarification! In that case, I think using DataParallel will add useless overhead <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

pmeier: <p>Thanks for the input, but since you ‘only’ <em>think</em> that multiple GPUs would be useless, I will keep this thread open for a little longer hoping someone who <em>knows</em> weighs in <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

ptrblck: <p>As <a class="mention" href="/u/alex.veuthey">@alex.veuthey</a> said, if you are only using a single image, the data parallel approach won’t speed up anything.</p>

rwightman: <p>I see no reason why you couldn’t add a batch dimension and process multiple images at a time and thus be able to leverage DataParallel. Just make sure you don’t reduce any of the losses across the batch dim.</p>

pmeier: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for the clarification.</p>

pmeier: <p><a class="mention" href="/u/rwightman">@rwightman</a> As explained above, during the whole training (maybe optimisation is a better fit here) you only process a single image. You do not train a model, but rather train (optimise) the pixel values of a single image to fit a given loss function.</p>
How to detect eclipse in image?

thisthuan: <p>I have some image as below.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/62a0149c30f0e45598dbb1cbb92343003d0f32d2.png" data-download-href="https://discuss.pytorch.org/uploads/default/62a0149c30f0e45598dbb1cbb92343003d0f32d2" title="button-8"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/62a0149c30f0e45598dbb1cbb92343003d0f32d2_2_83x500.png" alt="button-8" data-base62-sha1="e4tLj0WB8jPuQ8DtodShYPBW8gO" width="83" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/62a0149c30f0e45598dbb1cbb92343003d0f32d2_2_83x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/6/62a0149c30f0e45598dbb1cbb92343003d0f32d2_2_124x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/6/62a0149c30f0e45598dbb1cbb92343003d0f32d2_2_166x1000.png 2x" data-dominant-color="1A2938"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">button-8</span><span class="informations">185×1103 255 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/24e983fdec2d621ac4eb0e7b5646fd898386968a.png" data-download-href="https://discuss.pytorch.org/uploads/default/24e983fdec2d621ac4eb0e7b5646fd898386968a" title="button-20"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/24e983fdec2d621ac4eb0e7b5646fd898386968a_2_100x500.png" alt="button-20" data-base62-sha1="5gxuC7n0m48QUprGBoTpj2i4Tjk" width="100" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/24e983fdec2d621ac4eb0e7b5646fd898386968a_2_100x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/2/24e983fdec2d621ac4eb0e7b5646fd898386968a_2_150x750.png 1.5x" data-dominant-color="263540"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">button-20</span><span class="informations">185×925 215 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>please help ideas to detect eclipse area. Thank you.</p>
[Libtorch]:how to transfer data from tensor to vector with libtorch

menguangwen: <p>I transfer data from  tensor to vector as follow:</p>
<p>vector box_vec;<br>
torch::Tensor out1 = outputs-&gt;elements()[0].toTensor();<br>
auto box_temp = out1.view({ 1 ,21708*4 }).to(torch::kCPU);<br>
for (int i = 0; i &lt; 21708 * 4; i++) { box_vec.push_back(box_temp[0][i].item().toFloat());</p>
<p>How to transfer data from  tensor to vector with libtorch directly?</p>

tom: <p><code>box_temp.data&lt;float&gt;()</code> will give you a <code>float*</code> which you can liberally use. You do have to deal with strides yourself. An alternative could be to use <code>box_vec.reserve(...)</code>, create a tensor view on that with <a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/templates/variable_factories.h#L61" rel="nofollow noopener"><code>auto box_vec_view = torch::from_blob</code></a> (but I don’t know if it should be <code>at::from_blob</code> or <code>torch::</code>) and <code>box_vec_view.copy_(out1)</code> the tensor there.</p>
<p>Note that from_blob does <strong>not</strong> take ownership of the memory, I use that because it’s impossible to create a <code>std::vector</code> with memory from elsewhere.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: The preferred way is to use the C++ category rather than a <code>[Libtorch]</code> prefix.</p>

menguangwen: <p>Great .Thanks a lot.</p>
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time. WHY?

Sergio_Calle_Jimenez: <p>I am getting the next problem:<br>
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.<br>
I could fix it using: error.backward(retain_graph=True)</p>
<p>But why does happen? I did not call twice inside of the loop.<br>
I have to say that in each iteration I am using the same variables: value_path and value_body</p>
<pre><code>	for i_emb in range(emb_times):
		optimizer_s.zero_grad()
		out_data = net(value_path,value_body)
		out_constant = (out_data.div(out_data.norm(p=2, dim=1, keepdim=True))).detach()
		error = ((out_data - out_constant).pow(2)).sum()
		error.backward(retain_graph=True****************)
		optimizer_s.step()
</code></pre>
<p>I do not understand why that happens.</p>

albanD: <p>Hi,</p>
<p>Where does “value_path” and “value_body” come from? Do they require gradients? If they do, whatever created them is a part of the gradient that will be common to all the iterations of the loop, hence the error.</p>

Sergio_Calle_Jimenez: <p>Hi AlbanD</p>
<p>Both of them  (value_path, value_body) are a coming from the raw data, there is not a graph calculation.<br>
Anyway I just modify the initial definition.</p>
<p>value_body = torch.zeros((batch, 1, size_body, size_emb_body), device = ‘cuda’, requires_grad=False)<br>
value_path = torch.zeros((batch, 1, size_path, size_emb_path), device = ‘cuda’,requires_grad=False)</p>
<p>And I still need to define “error.backward(retain_graph=True)” if not, the ERROR shows up. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
Best practices for Dataloader

vishalagarwal: <p>Hey, is there a best practices for creating efficient dataloader dealing with images?</p>

ptrblck: <p>Multiple workers usually help. If you set <code>num_workers</code> to a very high number, the performance might drop again, so you should play around a bit and find a sweet spot for your system.<br>
Pinned memory and <code>.to('cuda', non_blocking=True)</code> might also hide the latency of the host to device copy.</p>
<p>Did you encounter any issues/bottlenecks?</p>
The output image of UNet

ykeivn: <p>I want to use the UNet output image, but I get 2 different images, can tell me the reason?<br>
first, use this code ‘torchvision.utils.save_image(img, os.path.join(result_path, str(i + 1) + ‘image.bmp’))’<br>
second,use this code ’    im = torch.squeeze(im)<br>
im = transforms.ToPILImage()(im)<br>
im_resized = im.resize((100, 100))<img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a575cc2589a9b9a1f7651a5c000dda671ceca722.png" alt="1SR" data-base62-sha1="nBJa4CoodxqKk8UeQOwdBC4wI4W" width="100" height="100"> !<br>
im_resized.save(os.path.join(result_path, str(i+1) + ‘SR.bmp’)) ’</p>

ptrblck: <p>What is the difference in both images?<br>
Do they look completely different?<br>
Did you try to load both after saving and compare their values?<br>
How large is the difference?</p>
Creating custom loss function

Praveen_raja_sekar: <p>class CrossEntropyLoss(nn.Module):</p>
<pre><code>def __init__(self):
    super(CrossEntropyLoss,self).__init__()

def forward(self, logits, target):
    log_probabilities = F.log_softmax(logits, dim=-1)
    totloss = torch.mean(torch.sum(- target * log_probabilities, 1))
    return totloss
</code></pre>
<p>TypeError: <strong>init</strong>() takes 1 positional argument but 3 were given</p>
<p>Can anyone please please help me to fix this</p>

el_samou_samou: <p>Hi, can you put the line where you instanciate CrossEntropyLoss module in your code? thanks</p>
Speeding up GPU training on a CNN

harizas: <p>Hi<br>
Im training a semantic segmentation model and find that my training speed is very slow. Im using a custom dataloader, and my images are about 350* 400 pixels in size. Each training instance is taking about 1.3 sec, and Im using a TeslaP100 on google Cloud.<br>
I used time.time and profiled my training code. I found that the time taken to transfer the image data was the one taking the most amount of time, almost 1 sec.<br>
Can someone help me with ideas on how to speed it up? should I use pytorchs dataloader and pin_memory?<br>
Ive attached the training function below. The lines taking a long time are marked</p>
<pre><code>def train(trainable_model, train_data, optimizer, epoch, criterion):
    total_train_data = train_data.__len__()
    batch_indices = np.array_split(np.random.permutation(total_train_data),
                               math.ceil(total_train_data / settings.opt['batch_size']))

   trainable_model.train()
   total_train_loss = 0

   trainable_model = trainable_model.cuda()
   criterion = criterion.cuda()

   for batch_index, indices in enumerate(tqdm(batch_indices)):
        optimizer.zero_grad()
        for idx,index in enumerate(tqdm(indices)):
           rgb, mask, filename,humanImg = train_data[index]
           var_rgb = Variable(rgb.unsqueeze(0).float())
           var_mask = Variable(mask.float())
           var_mask = var_mask.cuda()   &lt;--- taking 5 ms
           var_rgb = var_rgb.cuda()     &lt;--- taking 1 sec         
            
           output = trainable_model(var_rgb)
           loss = ((criterion(output, var_mask.unsqueeze(0).long()) / (len(indices))) 
           total_train_loss += loss.data[0]
           loss.backward()
                   
    del var_mask, var_rgb # after every batch delete reference to any variables
    optimizer.step()</code></pre>

albanD: <p>Hi,</p>
<p>Have use use <code>torch.cuda.synchronize()</code> properly when measuring times? The cuda api is asynchronous and so the times are not accurate unless you synchronize explicitly.</p>

harizas: <p>Hi albanD<br>
I used cuda.synchronize and found that the function that was actually taking the most time was the actual model call, not the loading into gpu calls.<br>
I did</p>
<pre><code> torch.cuda.synchronize()
 a = time.perf_counter()
 var_mask = var_mask.cuda()
 torch.cuda.synchronize()
 b  = time.perf_counter()
 print(b-a)
 torch.cuda.synchronize()
 a = time.perf_counter()
 var_rgb = var_rgb.cuda()
 torch.cuda.synchronize()
 b = time.perf_counter()
 print(b - a)
 torch.cuda.synchronize()
 a = time.perf_counter()
 output = trainable_model(var_rgb)
 torch.cuda.synchronize()
 b  = time.perf_counter()
 print(b - a)
</code></pre>
<p>and got<br>
0.00036394898779690266<br>
0.0005174019897822291<br>
0.8291985339892562</p>
<p>so it looks like just evaluating the model is taking most of the time. is it possible to speed this up? I have 4 GPUs on this computer so running it in parallel should make it faster right?</p>
<p>thanks</p>
Optimizing and Satisficing Metrics

huyvnphan: <p>I have 2 losses: <code>loss_a</code> and <code>loss_b</code></p>
<ul>
<li>
<code>loss_a</code> is a satisficing metrics. I just need l<code>oss_a &lt; threshold</code>.</li>
<li>
<code>loss_b</code> is an optimizing metrics. I want it to be as low as possible.</li>
</ul>
<p>Right now this what I am doing:</p>
<pre><code class="lang-auto">total_loss = alpha*loss_a + loss_b
total_loss.backward()
</code></pre>
<p>By adjusting <code>alpha</code>, I can adjust <code>threshold</code> for <code>loss_a</code>. Is there a better way to do this?</p>

tom: <p>If you use <code>lossa.clamp(min=threshold)</code> it will only get a gradient when it’s above the threshold. That’s cool, but in reality you may have the other losses gradient pointing towards a direction of increasing lossa. How much this matters depends on your specific situation.</p>
<p>Best regards</p>
<p>Thomas</p>

huyvnphan: <p>This works pretty well for my application. Thank you</p>
Generate large Gaussian distributed vector

LMA: <p>I need to generate a large random vector, whose entries are <strong>independent but not identically distributed</strong>.  Each entry has its own mean and variance.  We could loop over the entries and sample a scalar Gaussian distribution, but that would need many function calls, slowing down the speed.</p>
<p>Alternatively, I could use the ``<a>torch.distributions.multivariate_normal()</a>`’, but the co-variance matrix was large enough to cause an <strong>out-of-memory</strong> exception.</p>
<p>Thanks.</p>

KFrank: <p>Hello Liangping!</p>
<aside class="quote no-group quote-modified" data-username="LMA" data-post="1" data-topic="49689" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/aca169/48.png" class="avatar"> LMA:</div>
<blockquote>
<p>I need to generate a large random vector, whose entries are <strong>independent but not identically distributed</strong>.  Each entry has its own mean and variance.  We could loop over the entries and sample a scalar Gaussian distribution, but that would need many functions, slowing down the speed.</p>
</blockquote>
</aside>
<p>Let’s assume your large random vector is to be of length <code>N</code>, and<br>
that you have your desired per-element means and standard<br>
deviations (standard deviation = sqrt (variance)) in the existing<br>
length-<code>N</code> vectors <code>means</code> and <code>stdevs</code>.</p>
<p>Then:</p>
<pre data-code-wrap="python"><code class="lang-python">rvec = torch.randn (N)
rvec.mul_ (stdevs)
rvec.add_ (means)
</code></pre>
<p><code>torch.randn()</code> gives you a vector with normally (mean = 0,<br>
stdev  = 1), independently and identically distributed elements.<br>
We then multiply <code>rvec</code>, element-wise, by <code>stdevs</code> to give each<br>
of its elements its own desired standard deviation (and, hence,<br>
variance).  Lastly, we add, element-wise, to give each element<br>
its desired mean.</p>
<p>Good luck.</p>
<p>K. Frank</p>

LMA: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="49689">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>torch.randn</p>
</blockquote>
</aside>
<p>Hi Frank,</p>
<p>That is an elegant idea! I tried it out and it sped up by a factor of more than 5000!</p>
<p>Thanks!</p>
<p>Liangping</p>
Weird OOM when I multiply a tensor with 1

lliuz: <p>I ran into a strange problem. When I run the following code and commented out <code>* 1</code>, the code works well, consuming about 2GB memory, but when <code>* 1</code> is retained, the code becomes OOM. Even though I run it on a machine with 12GB GPU memory.</p>
<pre><code class="lang-python">import torch
import torch.nn.functional as F


def func(x1, x2):
    _, _, H, W = x1.size()
    x2 = F.pad(x2, [4] * 4)
    cv = []
    for i in range(9):
        for j in range(9):
            cost = x1 * x2[:, :, i:(i + H), j:(j + W)] # * 1
            cv.append(torch.mean(cost, 1, keepdim=True))
    return torch.cat(cv, 1)


if __name__ == '__main__':

    x1 = torch.randn(4, 256, 128, 256, requires_grad=True).to(torch.device("cuda"))
    x2 = torch.randn(4, 256, 128, 256, requires_grad=True).to(torch.device("cuda"))

    y = func(x1, x2)
    y.sum().backward()
</code></pre>

lliuz: <p>It seems a bug fixed in PyTorch 1.1, but currently I have taken many experiments on PyTorch 1.0, can I walk around this problem without upgrading…</p>

albanD: <p>Hi,</p>
<p>I would try to use inplace operations if possible to reduce the memory overhead if you multiply by a constant.</p>
Show all backward functions in a model

cherepanovic: <p>Hello community!</p>
<p>Is there a possibility to illustrate all differentiable backward functions which will be applied on the model?</p>
<p>thanks!</p>

albanD: <p>The pytorchviz utility from Sergey available <a href="https://github.com/szagoruyko/pytorchviz" rel="nofollow noopener">here</a> will do that.</p>
Given an output; how do I know if the output depends on some parameters?

ElleryL: <p>Consider I have following codes</p>
<pre><code class="lang-auto">w1 = nn.Parameter(torch.log(torch.Tensor([1.]))).to(device)
w2 = nn.Parameter(torch.log(torch.Tensor([1.]))).to(device)

y = w2 * x
&gt;&gt;&gt; y 
tensor([0.], grad_fn=&lt;MulBackward0&gt;)
</code></pre>
<p>Clearly <code>y</code> only depends on <code>w2</code> not <code>w1</code>. How do I know when the network is much more complicated?</p>

albanD: <p>I would suggest to set all the gradients to 0 then run a backward on your output. If it’s a Tensor, you can backward a tensor full of ones of the same size as the output. Then check the value of each parameters’ gradients. The ones that are non zero have an impact on the output.</p>
Problem to convert the return of torch::argmax to an int, in pytorch c++

ivan_rodriguez: <p>Hello,</p>
<p>I am starting to use pytorch and I have a problem when I try to convert the return of torch::argmax to an int in pytorch c++.</p>
<p>For instance:</p>
<pre><code>auto arrayCpu = torch::randint(1, 10, 10);

auto max = torch::argmax(arrayCpu);

auto maxNumber = *max.data&lt;int&gt;();
</code></pre>
<p>Error:</p>
<p>terminate called after throwing an instance of ‘c10::Error’<br>
what():  expected scalar type Int but found Long (data at …</p>
<p>Thanks for any help!</p>

tom: <pre><code class="lang-auto">auto maxNumber = max.data&lt;int64_t&gt;();
</code></pre>
<p>Usually, PyTorch uses int64_t / scalar type kLong for indices and indexing tensors.</p>
<p>Best regards</p>
<p>Thomas</p>

ivan_rodriguez: <p>Thanks for your quick reply! It works perfectly now!</p>
<p>Best regards,</p>
<p>Ivan</p>
Does pytorch has potential for efficient inference of fully binirized (XNOR) conv nets?

Marat: <p>I understand that using some hacks I could train xnor net, but what about efficient inference? Does pytorch support fair bitwise xnor-conv operations?</p>

albanD: <p>Hi,</p>
<p>No bitwise convolutions are not supported.<br>
But the ones in the paper have the input as floats and the weights as binary values?</p>
<p>Most element-wise bitwise operations are implemented for <code>ByteTensor</code>s such as and, or, nor, xor… You can do <code>dir(torch.ByteTensor())</code> to see what is there, most like <code>__xor__</code>.</p>

Marat: <p>This is not what I want</p>

Marat: <p>Can I do Conv2d with torch.ByteTensor or torch.CharTensor?</p>

albanD: <p>Hi,</p>
<p>No only float convolutions are supported.</p>
Torch.Le() gradient

shubh809: <p>Can some tell that whether torch calculates gradient for Le operation and how to test that?</p>

albanD: <p>The problem is that the gradient of this function is 0 almost everywhere, so they are quite useless.</p>

shubh809: <p>But don’t you think we can’t really find gradient for a&lt;b. It should be none.</p>

shubh809: <p>Ceil also have gradient zero for which I can access it’s gradient using param.grad but in case of le it returns none.Should I assume that pytorch give none value for gradeint. Also I have found this       <a href="https://github.com/pytorch/pytorch/blob/7707dee76169a2c2d6637f80af5ed59eeb32a997/torch/csrc/jit/symbolic_script.cpp#L825" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/7707dee76169a2c2d6637f80af5ed59eeb32a997/torch/csrc/jit/symbolic_script.cpp#L825</a> .</p>
<p>What’s your call on this?</p>

albanD: <p>In the autograd semantic, <code>None</code> and a Tensor full of <code>0</code>s are the same thing. None is just more memory efficient and a Tensor full of 0 is needed if some part of the tensor might have gradients.</p>
Tensor using where to get indexes in C++

jinfagang: <p>Hi, does anyone know how to do where operation to filter out index of a tensor?</p>
<p>in numpy we have:</p>
<pre><code class="lang-auto">            inds = np.where(scores[:, j] &gt; 0.29)[0]
</code></pre>
<p>So we can get index where the j columns score bigger than 0.29. How to do this operation in C++ tensor??</p>

tom: <p>As the <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html" rel="nofollow noopener">numpy documentation</a> suggests, that is <code>torch.nonzero</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

jinfagang: <p>Just find it out!!!</p>
<p>Now, I got another issue!!</p>
<p>How to get specific row of a 2 dim tensor by indcies??</p>

tom: <p><code>func: index(Tensor self, Tensor?[] indices) -&gt; Tensor</code>?</p>
Need to get gradients w.r.t 1 variable that I use to calculate an affine transformation matrix

andrster: <p>Hello, I have the following snippet:</p>
<pre><code>def param2theta(param, w=25, h=1):

   param = torch.pinverse(param)
   theta = torch.zeros([2,3])
   theta[0,0] = param[0,0]
   theta[0,1] = param[0,1]*h/w
   theta[0,2] = param[0,2]*2/w + theta[0,0] + theta[0,1] - 1
   theta[1,0] = param[1,0]*w/h
   theta[1,1] = param[1,1]
   theta[1,2] = param[1,2]*2/h + theta[1,0] + theta[1,1] - 1

   return theta.unsqueeze(0)


mat = torch.ones(1,26).view(1,1,1,-1).float()


##scale##
y =  Variable(torch.tensor([.2]), requires_grad = True)
s = y*(torch.tensor([[ [1.,0.,0],[0,0.,0], [0,0,0] ]]).reshape( 1,3,3) ).requires_grad_() + \
(torch.tensor([[ [0,0.,0],[0,1.,0], [0,0,1] ]]).reshape( 1,3,3) )

##translation##
z =  Variable(torch.tensor([ y* mat.size(-1) ]), requires_grad = True)
t = z*(torch.tensor([[ [0,0.,1.],[0,0,0.], [0,0,0] ]]).reshape( 1,3,3) ).requires_grad_() + \
(torch.tensor([[ [1,0.,0],[0,1,0.], [0,0,1] ]]).reshape( 1,3,3) )

f=torch.bmm(t,s)
theta = param2theta(f.squeeze())
grid = F.affine_grid(theta, mat.size())

mat = F.grid_sample(mat, grid, mode='nearest')

print('after grid_sample',mat,'\n \n \n')

mat.sum().backward()


print(y.grad)
print(z.grad)
</code></pre>
<p>but when calling mat.sum().backward() it says that y.grad and z.grad are 0.</p>
<p>What can I do in order to get the gradients w.r.t y  ?</p>

el_samou_samou: <p>Hi,</p>
<p>Without looking too much into detail I can tell you that the gradient of <code>mat = F.grid_sample(mat, grid, mode='nearest')</code> is 0 because of <code>mode=nearest</code>. Try with any other option and you should have a non-zero gradient.</p>
<p>Hope this helps.</p>

el_samou_samou: <p>(To be as clear as possible, gradient is 0 in that case because your function is piece-wise constant with <code>mode="nearest"</code>.)</p>

andrster: <p>Hello and thank you for your time.<br>
Just changed mode=‘nearest’ to mode=‘bilinear’ and still it outputs 0. gradients <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"><br>
Any other suggestions?</p>

andrster: <p>Actually, I don’t know why, but I think colab had lagged on me and that’s why it didn’t change anything, even after I changed the mode, as you suggested. After refreshing, with mode=‘bilinear’, it works as expected! thank you!</p>
Batch Outer Product

ArmenAg: <p>Title says it all. Is there a way to compute a batch outer product. I noticed that pytorch conveniently has <code>torch.ger</code> which takes in two one-dimensional vectors and outputs there outer-product: <code>(1, n) * (1, m) -&gt; (n, m)</code></p>
<p>Is there a batch version of this operation? <code>(b, n) * (b, m) -&gt; (b, n, m)</code></p>

fmassa: <p>No, there is no batch version for <code>ger</code>. But you can obtain the same results by appending 1 to the tensors.</p>
<pre><code class="lang-auto">x = torch.rand(10, 3)
y = torch.rand(10, 3)

res = torch.bmm(x.unsqueeze(2), y.unsqueeze(1))
</code></pre>

Yin_Tang: <p>I guess the following line may help:</p>
<p>reduce(lambda a, b: torch.ger(a,b), list_of_vectors)</p>

pinouchon: <p>Looks like you can also do it with <a href="https://pytorch.org/docs/stable/torch.html?highlight=einsum#torch.einsum" rel="nofollow noopener">einsum</a>:</p>
<p><code>einsum = torch.einsum('bi,bj-&gt;bij', (x, y))</code></p>
<p>Full example:</p>
<pre><code class="lang-auto">import torch
x = torch.ones(4, 3)
y = torch.rand(4, 3)
bmm = torch.bmm(x.unsqueeze(2), y.unsqueeze(1)) # Method of @fmassa
einsum = torch.einsum('bi,bj-&gt;bij', (x, y)) # Method with einsum
bmm==einsum # gives a whole bunch of ones
</code></pre>
PyTorch Translation with a Sequence to Sequence Network and Attention

deathcrush: <p>Hi,</p>
<p>I was going through the (amazing!) tutorial <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="nofollow noopener">here</a> and I was a little nonplussed about an implementation detail in the <code>EncoderRNN</code> class. After retrieving the input from the embedding look-up table, the variable <code>output</code> is bound to it, so operations that mutate <code>output</code> will mutate the results in the embedding table as well. That is to say, the embedding table parameters are updated using the gradients of the GRU outputs.</p>
<p>Can anyone explain the mechanics there in a bit more detail/provide comments on my statements above?</p>
<p>Insights welcome and cherished <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to know that if I am using mkldnn

Yang_Yang2: <p>I am trying to use mkldnn to speed up training. My model is fasttext. I have followed the instruction of pytorch document (from source). But it seems that sometimes the training is still too slow. And also I want to know how many benefits I can gain by using mkldnn.</p>

LeviViana: <p>Try this:</p>
<pre><code class="lang-python">import torch
print(*torch.__config__.show().split("\n"), sep="\n")
</code></pre>
<p>You will get the build information, if <code>USE_MKLDNN=ON</code> then you are good to go.</p>
<p>PS: this works for PyTorch 1.1 or earlier.</p>
How to train two networks at once in PyTorch?

etekiller: <p>I am trying to train two networks at once and calling two backward passes on two different losses but that breaks everything:</p>
<pre><code class="lang-auto">Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>I do it this way:</p>
<pre><code class="lang-auto">        optimizer_d.zero_grad()

        npArray = randomNpArray(1, 3, 4, 11)
        randomTensor = numpyToTensor(npArray).cuda().float()        

        guess_real = discriminator(image)
        loss_d_real = criterion(guess_real, Tensor([1,0]).cuda() )
        realLossesArray.append(loss_d_real.item())
        
        generatedImage = generator(randomTensor)  

        guess_fake = discriminator(generatedImage)
        loss_d_fake = criterion(guess_fake, Tensor([0,1]).cuda() )
        fakeLossesArray.append(loss_d_fake.item())        

        loss_d = loss_d_real + loss_d_fake
        loss_d.backward()
        optimizer_d.step()

        # -------------------
        optimizer_d.zero_grad()
        optimizer_g.zero_grad()

        loss_g = criterion( guess_fake, Tensor([1,0]).cuda() )

        generatorLossesArray.append(loss_g.item())
        
        loss_g.backward()
        optimizer_g.step()
</code></pre>
<p>I don’t understand how to make PyTorch know that the second calculation of the gradient is completely irrelevant to the first one.</p>
<p>I’d appreciate any help.</p>

Prerna_Dhareshwar: <p>So when you call backward the first time it calculated all the gradients and gets rid of the computational graph from memory, so since you need <code>guess_fake</code> for <code>loss_g</code>, there is no computational graph for it, so I suggest recalculating <code>guess_fake</code> after one step of discriminator.</p>
<p>You could also do <code>retain_graph = True</code> but then all the graphs are retained in memory and this could cause CUDA out of memory error. So just re calculate whatever you need for the second backward()after the first backward().</p>

etekiller: <p>Alright, awesome, thank you very much for your help. Now I understand it and it works :).</p>
Requires_grad or two different optimizers

Ivamcoder: <p>Hello,</p>
<p>I have a network which is composed of two sequential subnetworks. I wanted to perform partial training on each of these subnetworks (i.e updating the parameters of the first subnetwork using first part of data while freezing the second subnetwork and the converse for the second subnetwork and second part of data). As far as I know, there are two options for updating the parameters:</p>
<p>1- making the requires_grad of the respective subnetwork parameters to False and update the other<br>
2- create two different optimizers for each subnetworks’ parameters and perform optimization on the respective optimizer.</p>
<p>I wanted to know if any of these two approaches are possible in pytorch? and which one do you suggest for this task and what’s the difference between them?</p>

ecdrid: <p>I would go with option1 , freeze what you don’t wanna update with this</p>
<pre><code class="lang-auto">for param in child.parameters():
    param.requires_grad = False
</code></pre>
<p>and then</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, whole_model.parameters()), lr=opt.lr, amsgrad=True)
</code></pre>
Dicom files in pytorch?

ba3wossen: <p>I ran my program using the Pytorch <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">transfer learning example</a>, with dicom files, but this error was thrown<br>
“RuntimeError: Found 0 files in subfolders of: …/data2/train<br>
Supported extensions are: .jpg,.jpeg,.png,.ppm,.bmp,.pgm,.tif,.tiff,.webp”</p>
<p>Is there any way to continue using dicom files and not convert to png?</p>

JuanFMontesinos: <p>You can create a custom dataset class which opens your files with whichever library you need. Think that PIL is chosen  by default but it’s not mandatory. The Drawback is you would lose transformations</p>

ba3wossen: <p>How would I go about writing a custom dataset class… Any tutorials that you know of that explains how to?</p>

JuanFMontesinos: <p>You need to subclass a Datasetclass as follows:</p>
<pre><code class="lang-auto">from torch.utils import data
class custom_dataset(data.Dataset):
    def __init__(self,*args,**kwargs):
    """
    Custom arguments, up to you
    """
    self.data = list_of_paths
    def __getitem__(self, index):
        """
        Args:
            index (int): Index
        Returns:
            whatever you need
        """
        #  Here you have to code workload to open files or to do any kind of preprocessing.This function is submitted to multiprocessing.
       return load(self.data[index])
    def __len__(self):
        # Return amount of samples of your dataset.
        return len(self.data)
</code></pre>
<p>You need to define a class like the previous one, with init, getitem and len.<br>
Len should return the amount of samples of your dataset, images in your case.<br>
In init you define a list of paths, files, or whatever setup you need.</p>
<p>In getitem you have to load those files prviously enlisted and put all the workload there. This is, any preprocessing operation you have to run on the fly as this function is submitted to multiprocessing</p>

ba3wossen: <p>Thank you. Can I still use “torch.utils.data.DataLoader” for the multiprocessing?</p>

ecdrid: <p>Yes you can use that…</p>
<pre><code class="lang-auto"># Convert DICOM to JPG/PNG via openCV
def convert_images(filename, img_type='jpg'):
    """Reads a dcm file and saves the files as png/jpg
    
    Args:
        filename: path to the dcm file
        img_type: format of the processed file (jpg or png)
        
    """
    # extract the name of the file
    name = filename.parts[-1]
    
    # read the dcm file
    ds = pydicom.read_file(str(filename)) 
    img = ds.pixel_array
    
    # save the image as jpg/png
    if img_type=="jpg":
        cv2.imwrite(outdir + name.replace('.dcm','.jpg'), img)
    else:
        cv2.imwrite(outdir + name.replace('.dcm','.png'), img)

# Using dask 
all_images = [dd.delayed(convert_images)(all_files[x]) for x in range(len(all_files))]
dd.compute(all_images)
</code></pre>
Strange! One tensor can slice while same size another can not!

jinfagang: <p>I got some weired things in C++ pytorch. Simply I got 2 tensor with same size, but one can slice, the other can not.</p>
<p>I got a tensor of <code>priors</code> which using in SSD, and a tensor of <code>loc</code> which is prediction of network. They are same size, and program checked exactly same size:</p>
<pre><code class="lang-auto">origin prior: [2976, 4] origin loc: [2976, 4]
loc slice: [2976, 2]

</code></pre>
<p>As you can see, the loc can slice using:</p>
<pre><code class="lang-auto">loc.slice(1, 0, 2)
</code></pre>
<p>which gets the first 2 columns.</p>
<p>But when slice pirors, it was throwing an error:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'std::runtime_error'
  what():  invalid argument 0: Sizes of tensors must match except in dimension 0. Got 1 and 2 in dimension 1 at /media/jintain/sg/ai/libs/pytorch/aten/src/THC/generic/THCTensorMath.cu:71

</code></pre>
<p>Both tensor are contiguous, and I checked it. Why they are not same behaviour?</p>

tom: <p>Are you sure that you’re getting the error from slice? That isn’t an error slice would throw…</p>
<p>Best regards</p>
<p>Thomas</p>

jinfagang: <p>Sorry, I have figure it out.<br>
this works exactly.</p>
Assigning value at given target indices

Dheeru_Dua: <p>Hi,</p>
<p>Sorry if this question has been asked already. I couldn’t find a solution anywhere.</p>
<p>I want to assign a value to a tensor at particular column indices.</p>
<p>For instance, given a tensor t1 = [[0.3,0.4,0.5], [0.1,0.2, 0.8]] , indices = [1,2] and value = 1. I want to compute t2 such that t2 = [[0.3,1, 0.5], [0.1, 0.2, 1]].</p>
<p>Basically, I would like to do below with pytorch tensor indexing</p>
<p>for i in range(t1.size(0)):<br>
t2[i][indices[i]] = value</p>
<p>Thanks!</p>
<p>P.S. I tried scatter and index_select but couldn’t figure out how to use them for above problem.</p>

lugiavn: <p>You can look up how they convert class label to 1 hot coding vector, then use it as a mask, for example t2 = mask * value + (1 - mask) * t1</p>
C++ tensor operation

jinfagang: <p>I have a tensor which contains boxes predict by detection. and it’s format like:</p>
<p>cx, cy, w, h</p>
<p>I want convert this tensor to:</p>
<p>x1, y1, w, h</p>
<p>So I can do this in python:</p>
<pre><code class="lang-auto">boxes[:, :2] -= boxes[:, 2:] / 2
    boxes[:, 2:] += boxes[:, :2]
</code></pre>
<p>But how to do this in libtorch???</p>
<p>support I have</p>
<pre><code class="lang-auto">torch::Tensor boxes
</code></pre>

tom: <p>Are you looking for <code>boxes.slice</code> or <code>boxes.narrow</code>?</p>
<p>Best regards</p>
<p>Thomas</p>

jinfagang: <p>thank u for your reply Tomas.</p>
<p>Can you elibrate your answer? I am not quit understand of the <code>narrow</code> usage. I know there is a slice method of tensor.</p>
<p>I got trouble in slice.</p>
<p>My tensor is size as : [2976, 4], which is 2976 rows and 4 columns.<br>
I want get the first 2 columns:</p>
<pre><code class="lang-auto">priors.slice(1, 0, 2)
</code></pre>
<p>Got error:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'std::runtime_error'
  what():  invalid argument 0: Sizes of tensors must match except in dimension 0. Got 1 and 2 in dimension 1 at /media/jintain/sg/ai/libs/pytorch/aten/src/THC/generic/THCTensorMath.cu:71

</code></pre>
<p>Did I slice it wrong?</p>
How to use PyTorch hub?

Quebradawill: <p>Pytorch Hub is a pre-trained model repository designed to facilitate research reproducibility.</p>
<p>When I use pytorch hub, there are many errors, how to import pre-trained models?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/71f3cdeb145de7be85f45b39777dd6f151172a4e.png" data-download-href="https://discuss.pytorch.org/uploads/default/71f3cdeb145de7be85f45b39777dd6f151172a4e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/71f3cdeb145de7be85f45b39777dd6f151172a4e_2_416x500.png" alt="image" data-base62-sha1="gg4iLeBUTy45jLTFFOtKZvTOirY" width="416" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/71f3cdeb145de7be85f45b39777dd6f151172a4e_2_416x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/7/71f3cdeb145de7be85f45b39777dd6f151172a4e_2_624x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/7/71f3cdeb145de7be85f45b39777dd6f151172a4e_2_832x1000.png 2x" data-dominant-color="FBEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">840×1008 78.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Does changing the hub directly help as suggested in <a href="https://discuss.pytorch.org/t/pytorch-hub-permissionerror/49266/2">the other topic</a>?</p>

ailzhang: <p>Hi this is a known issue that was fixed by <a href="https://github.com/pytorch/pytorch/pull/20782" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/20782</a>. It’s should be available in pytorch nightly.</p>

Quebradawill: <p>Thanks for your reply, I will wait for the next version.</p>
Generate samples from distribution parameterized by a trainable parameters; samples has no grad

ElleryL: <p>Hi; I have a trainable parameters that parameterized exponential distribution; however, when I generate samples from the distribution, the generated samples has no grad_fn. Hence, the backward can’t bakcpropage through my samples.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x
tensor([1.], grad_fn=&lt;ExpBackward&gt;) # it's a trainable parameters can be back-propagated 
&gt;&gt;&gt; torch.distributions.exponential.Exponential(x).sample()
tensor([0.3613]) # lost its grad_fn
</code></pre>
<p>is there any way to fix it ?</p>
Few shot learning for image classification

Mona_Jalal: <p>I am totally new to the concept of few shot learnig. I am looking for a tested and well-designed tutorial that can walk me through the concept (e.g., similar to transfer learning tutorial). Could you please share a link with me?</p>
<p>I found this: <a href="https://github.com/orobix/Prototypical-Networks-for-Few-shot-Learning-PyTorch" rel="nofollow noopener">https://github.com/orobix/Prototypical-Networks-for-Few-shot-Learning-PyTorch</a></p>

Manav_Madan: <p>hi, were you able to find some tutorials ?<br>
I have to implement few shot learning for image classification and I have absolutely no idea how to start.<br>
Thanks</p>
Autoencoding a single real number

evanthebouncy: <p>I am trying to model a real-valued function with real-valued input and outputs.</p>
<p>however, before that I thought it’s a good idea to understand how to model the simplest function, the “constant” function. The architecture is simply an input y, some non-linearity, then back to a single output y_pred. hence the title “autoencoding a single real number”</p>
<p>here is what I came up with so far</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/evanthebouncy/acf1bacd277ac98fdd588ee9d8ee9a57" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/evanthebouncy/acf1bacd277ac98fdd588ee9d8ee9a57" target="_blank" rel="nofollow noopener">https://gist.github.com/evanthebouncy/acf1bacd277ac98fdd588ee9d8ee9a57</a></h4>
<h5>auto_enc_real.py</h5>
<pre><code class="Python">import torch
import torch.nn as nn
from torch.autograd import Variable
import numpy as np
import torch.nn.functional as F
import random
from tqdm import tqdm

if torch.cuda.is_available():
    def to_torch(x, dtype, req = False):</code></pre>
This file has been truncated. <a href="https://gist.github.com/evanthebouncy/acf1bacd277ac98fdd588ee9d8ee9a57" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I could not get it to converge. My value of y is between 0 and 1, so my model happily learned that it should output 0.5 all the time (kind of a common issue). I tried changing the loss function to L1 loss, L2, nothing worked.</p>
<p>anything helps. I implemented a transformer network for some fancy task and had to tear it all down to realise this was the core problem of my model not converging . . .</p>
<p>edit: solved.</p>
<p>solution: <a href="https://www.facebook.com/profile.php?id=1244039" rel="nofollow noopener">David Qiu</a> torch.sum((y - y_pred) ** 2): dimensions don’t match and it is doing some broadcasting to mess you up</p>
Using the last hidden states of an LSTM? How to reshape?

josmi9966: <p>I have (for illustration) an LSTM(insize, hiddensize, num_layers=3, bidirectional=True, batch_first=True) and I want to use the last hidden state of each instance in a batch as an input to a Linear layer.</p>
<p>The Linear layer requires a tensor that has the batch instances in the first dimension, but the LSTM returns the last hidden state as shape (num_layers*isbidirectional), batchsize, hiddensize, (where isbidirectional is 2 if bidirectional, otherwise 1) even if batch_first=True</p>
<p>Maybe I am missing something here, but would it not be much more convenient if the hidden state would be of shape batchsize, (num_layers*isbidirectional), hiddensize?</p>
<p>What is the best way to create the necessary tensor that can be used as an input to the linear layer so that the batches are in the first dimension?</p>
<p>One thing I want to achieve is to just create a 2-dimensional tensor of shape batchsize, (num_layers<em>isbidirectional</em>hiddensize) so that the 2nd dimensions has all the hidden vectors that belong to each element in the batch dimension concatenated. How would one do this ideally?</p>

johann-petrak: <p>Maybe use</p>
<pre><code class="lang-auto">h_n.transpose(0,1),contiguous(),view(batchsize, -1)
</code></pre>
<p>The <code>contiguous()</code> call is needed because <code>transpose()</code> copies data, but I think this cannot be done without copying data, or does anyone else know a better solution?</p>

JohnCwok: <aside class="quote no-group" data-username="josmi9966" data-post="1" data-topic="24412">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/josmi9966/48/2272_2.png" class="avatar"> josmi9966:</div>
<blockquote>
<p>The Linear layer requires a tensor that has the batch instances in the first dimension, but the LSTM returns the last hidden state as shape (num_layers*isbidirectional), batchsize, hiddensize, (where isbidirectional is 2 if bidirectional, otherwise 1) even if batch_first=True</p>
</blockquote>
</aside>
<p>I’m replying because I run into the same problem and I can’t really understand that one would be forced to use things like <code>h_n.transpose(0,1),contiguous(),view(batchsize, -1)</code> or <code>input_to_cl = out.permute(1, 0, 2).view(-1, self.hidden_dim*len(x))` </code> (suggested in another post) which are quite obscure to me. It also sounds like super cumbersome for a framework such as pytorch, so I did a test :</p>
<pre><code class="lang-auto">test = torch.tensor([[[1,2,3]],[[1,2,3]]]).type(torch.FloatTensor)
#tensor of size ([2,1,6]), 2 beeing batch dimension.
lstm = nn.LSTM(3,6,batch_first = True)
lstm(test)[0].size() 
</code></pre>
<p>outputs <code>torch.Size([2, 1, 6])</code>, which means that using <code>batch_first = True</code> in the input of the LSTM instance makes it return an output also with batch first, which solves the issue.</p>
Minibatch for FC layer

slavavs: <p>I can’t figure out how to make a minibatch for FC layer. Input FC does not accept batch_size.<br>
Can you help figure it out? I have a large dataset and count each sample for a very long time.</p>
<p>We set batch_size at the network training stage. In fact, is it just the dimension of the array?</p>

ptrblck: <p><code>nn.Linear</code> will accept input batches with the shape <code>[batch_size, *, nb_features]</code>.<br>
The standard use case would be just <code>[batch_size, nb_features]</code>, where <code>nb_features</code> defines the number of features in each sample.<br>
E.g. if your data only has a single feature, you should pass the input as <code>[batch_size, 1]</code>.</p>
CNN back propagation

Rajamannar_Krishnamo: <p>Hi I am new to D.L . I know how back propagation works but in CNN how does back propagation works and how does the kernel values get initialized when we set only the filter size</p>
<pre><code class="lang-auto"># Here we are only giving the kernel_size =5
torch.nn.Conv2d(3, 16, 5, stride=1, padding=0)
</code></pre>

ptrblck: <p>Have a look at <a href="https://arxiv.org/pdf/1603.07285.pdf" rel="nofollow noopener">this paper - chapter 4.1</a> to see that transposed convolutions are used for backpropagation.<br>
You can find the init method for conv modules <a href="https://github.com/pytorch/pytorch/blob/d9e15bccb0a127314850567aaf0fea0a6364c3eb/torch/nn/modules/conv.py#L49" rel="nofollow noopener">here</a>.</p>