akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to choose learning rate when using Mixed Precision Training

username: numbpy
cooked: <p>I’m training ImageNet on ResNet-50 architecture using Fastai v1 1.0.60dev (Pytorch 1.3.0). There are substantial speed gains using Mixed Precision Training since I can effectively use 4 times the batch size (= 512) thanks to reduction in VRAM consumption and using smaller size of 224. The problem is I am unable to select a good learning rate. I am using fastai’s lr_finder with SGD but the suggested lr causes huge overfitting. Diving the suggested lr by 100 just overfits a bit while lr divided by 512 seems to be okish but slow.</p>
<p>While these guesses work, I’m not sure how to choose a good learning rate in general. I thought about using APEX but the dynamic loss scaling seems to be integrated in the learn.to_fp16(). Training is done using learn.fit_one_cycle() ie 1-cycle policy. I think everything else is working fine.<br>
Below is the image for lr divided by 100<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e40d7b7085a9dbd78575a9772808951a9cd01458.png" data-download-href="https://discuss.pytorch.org/uploads/default/e40d7b7085a9dbd78575a9772808951a9cd01458" title="Screenshot%20at%202019-11-28%2008-55-56"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e40d7b7085a9dbd78575a9772808951a9cd01458_2_690x382.png" alt="Screenshot%20at%202019-11-28%2008-55-56" data-base62-sha1="wxrKq32xekJjCBC5RiP9CsDBCkw" width="690" height="382" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e40d7b7085a9dbd78575a9772808951a9cd01458_2_690x382.png, https://discuss.pytorch.org/uploads/default/original/3X/e/4/e40d7b7085a9dbd78575a9772808951a9cd01458.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/4/e40d7b7085a9dbd78575a9772808951a9cd01458.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20at%202019-11-28%2008-55-56</span><span class="informations">955×529 25.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And this is for lr divided by 512<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f72443eb5fcd41ca29d0c7d39a71b00d24648c7f.png" alt="Screenshot%20at%202019-11-28%2008-57-26" data-base62-sha1="zgjDghVXMws9mg1iTyfRFQd4Lo3" width="669" height="366"></p>

username: rvarm1
cooked: <p>Could you potentially try a learning rate in between <code>lr/100</code> and <code>lr/512</code>, or potentially stick with <code>lr/100</code> and decay the learning rate over time? Also, this might be a better question for the fastai forums, since it uses fastai’s library: <a href="https://forums.fast.ai/c/fastai-users" rel="nofollow noopener">https://forums.fast.ai/c/fastai-users</a></p>

username: numbpy
cooked: <p>I posted the question there but haven’t gotten a reply. Will try varying the learning rates as suggested.</p>
Load torchscript module error!

username: NickKao
cooked: <p>I transfer pth pretrain model to pt(by torchscript)</p>
<p>I can load in python.<br>
<code>model=torch.jit.load('./moran_v2.pt')</code></p>
<p>But can not be load in libtorch and pytorch andorid ?</p>
<pre><code class="lang-auto">E/PytorchHelloWorld: Error reading assets
                     com.facebook.jni.CppException: false CHECK FAILED at ../torch/csrc/jit/import.cpp (deserialize at ../torch/csrc/jit/import.cpp:178)
                     (no backtrace available)
                         at org.pytorch.Module$NativePeer.initHybrid(Native Method)
                         at org.pytorch.Module$NativePeer.&lt;init&gt;(Module.java:70)
                         at org.pytorch.Module.&lt;init&gt;(Module.java:25)
                         at org.pytorch.Module.load(Module.java:21)
</code></pre>
<p>You can get model from below link:<br>
<aside class="onebox googledrive">
  <header class="source">
      <a href="https://drive.google.com/file/d/1uxlZi6c0r-IzjCht04iGlc910NYWZvE5/view?usp=drive_open" target="_blank" rel="nofollow noopener">drive.google.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://drive.google.com/file/d/1uxlZi6c0r-IzjCht04iGlc910NYWZvE5/view?usp=drive_open" target="_blank" rel="nofollow noopener"><span class="googledocs-onebox-logo g-drive-logo"></span></a>



<h3><a href="https://drive.google.com/file/d/1uxlZi6c0r-IzjCht04iGlc910NYWZvE5/view?usp=drive_open" target="_blank" rel="nofollow noopener">moran_v2.pt</a></h3>

<p>Google Drive file.</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Help me please…</p>
Conv2d and the value of padding

username: abder
cooked: <p>Hello,</p>
<p>I would like to ask about the value of padding we set in the <a href="https://pytorch.org/docs/stable/nn.html" rel="nofollow noopener">Conv2d</a> function.</p>
<p>I know what zero-padding is. However, what does it mean that the padding is 0 for instance, or 1,2, or 3. What do these values mean? Do they represent the number of columns and rows that will be filled with zeros?</p>
<p>Thanks a lot.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="abder" data-post="1" data-topic="62699">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abder/48/12163_2.png" class="avatar"> abder:</div>
<blockquote>
<p>Do they represent the number of columns and rows that will be filled with zeros?</p>
</blockquote>
</aside>
<p>Yes, the <code>padding</code> argument specifies how many rows and columns on each border will be added.<br>
E.g. <code>padding=1</code> will add one column on the left, right, upper, and lower border.</p>
Libtorch Download site

username: eraserpencil
cooked: <p>Hi! I am looking for a place to download versions of libtorch with cuda9.0 that i can use or find to use. Hoping to have a link that would be similar to<br>
<a href="https://download.pytorch.org/whl/cu90/torch_stable.html" class="onebox" target="_blank" rel="nofollow noopener">https://download.pytorch.org/whl/cu90/torch_stable.html</a><br>
but for libtorch.</p>
<p>I’m trying to do a C++ port of a repo I found online, but the pytorch version it uses and the libtorch version i build is not compatible and torchscript complained</p>
Extension cpp (how to debug cuda code)

username: paul_c
cooked: <p>Reference the <a href="https://github.com/pytorch/extension-cpp" rel="nofollow noopener">c++ extension</a>, I write a c++ extension of roi align and roi pool in pytorch: <a href="https://github.com/AceCoooool/RoIAlign-RoIPool-pytorch" rel="nofollow noopener">roi align&amp;roi pool</a>.I find it’s easy to debug cpp code (pure c++ code), However, the <code>.cu</code> code need to run <code>python setup.py install</code> again and again to find error. (use nvcc will cause an error about cannot find <code>ATen</code> library).</p>
<p>My Question is: Is there an easy way to debug <code>.cu</code> code？</p>
<p>thank you~ <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=5" title=":heart:" class="emoji" alt=":heart:"></p>

username: xiaodongww
cooked: <p>Hi, I am new to writing cpp extensions and  met the same problem. Do you find out how to debug cpp extensions?  For now, I just add <code>prinf</code> in cpp scripts and run <code>python setup.py install</code> again and again to find out where is the bug.</p>
<p>Any suggestions for how to debug c++ extensions? Or any keyword for how to search such problems?<br>
Many thanks!</p>
How to convert CNN to FCN?

username: Swisher
cooked: <p>To train a classifier in CIFAR100, I have already trained a classifier via VGGnets, but how can I convert my VGGnets to FCN?</p>
<p>My VGGnets code like below:</p>
<pre><code class="lang-auto">class Unit(nn.Module):
    def __init__(self,in_channels,out_channels):
        super(Unit,self).__init__()
 
        self.conv = nn.Conv2d(in_channels=in_channels,kernel_size=3,out_channels=out_channels,stride=1,padding=1)
        self.bn = nn.BatchNorm2d(num_features=out_channels)
        self.relu = nn.ReLU()
 
    def forward(self,input):
        output = self.conv(input)
        output = self.bn(output)
        output = self.relu(output)
 
        return output

class Model(nn.Module):
    def __init__(self,num_classes=100):
        super(Model,self).__init__()
 
        self.net = nn.Sequential(
            Unit(in_channels=3,out_channels=32),
            Unit(in_channels=32, out_channels=32),
            Unit(in_channels=32, out_channels=32),

            nn.MaxPool2d(kernel_size=2),

            Unit(in_channels=32, out_channels=64),
            Unit(in_channels=64, out_channels=64),
            Unit(in_channels=64, out_channels=64),
            Unit(in_channels=64, out_channels=64),

            nn.MaxPool2d(kernel_size=2),

            Unit(in_channels=64, out_channels=128),
            Unit(in_channels=128, out_channels=128),
            Unit(in_channels=128, out_channels=128),
            Unit(in_channels=128, out_channels=128),

            nn.MaxPool2d(kernel_size=2),

            Unit(in_channels=128, out_channels=128),
            Unit(in_channels=128, out_channels=128),
            Unit(in_channels=128, out_channels=128),

            nn.MaxPool2d(kernel_size=2))       
 
        self.dense = nn.Sequential(
            nn.Linear(in_features=2*2*128,out_features=num_classes))

    def forward(self, input):
        output = self.net(input)
        output = output.view(-1,2*2*128)
        output = self.dense(output)
        return output

</code></pre>
<p>Sincerely waiting for your help!</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m not sure what you mean by converting it to an FCN?<br>
Fully connected networks and convolutional ones have different weight structure so there isn’t a one to one mapping between them when you train.<br>
Could you give more context of why you want to do that?</p>

username: Swisher
cooked: <p>Just a task and it requires me to train a FCN in CIFAR100.<br>
Can I convert linear layer to conv layer and add convtranspose layer as the last layer?<br>
Thank you for your help!</p>

username: albanD
cooked: <blockquote>
<p>Just a task and it requires me to train a FCN in CIFAR100.</p>
</blockquote>
<p>Then just linearize the input as a batch of 1D Tensors and apply an FCN to it. No need to consider CNN architectures.</p>

username: Swisher
cooked: <p>Okay! I will have a try!<br>
Thanks a lot!</p>
Is model(input) same as model.__call__(input)

username: ASHUTOSH_CHANDRA
cooked: <p>Let’s say I defined the a model class. Now I want to implement a train function inside the class for which i need to call the model. How should i do that.</p>
<p>Which one is the right way to do this :<br>
def train(self,x):<br>
out=self.<strong>call</strong>(x)</p>
<p>or</p>
<p>def train(self,x):<br>
out=self.forward(x)</p>

username: albanD
cooked: <p>Yes <code>model(input)</code> and <code>model.__call__(input)</code> are the same.<br>
It is the python convention to have the <code>__call__</code> method being called when you use <code>foo()</code>.<br>
In this case, you can do <code>self.__call__(x)</code> (or <code>self(x)</code> if that works but I’m not sure it does).</p>
<p><a class="mention" href="/u/shagunsodhani">@shagunsodhani</a> No this is not the right way !!  You should <strong>not</strong> call <code>.forward()</code> directly ! If you do this, many of the <code>nn.Module</code> functionality won’t work (hooks for example).</p>

username: shagunsodhani
cooked: <p><a class="mention" href="/u/alband">@albanD</a> you are right. I confused it with something else <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Batch index_select

username: zcajiayin
cooked: <p>I have a batch data:</p>
<p>tensor A :</p>
<pre><code class="lang-auto">[[[66.,  0.,  0.],
  [77.,  0.,  0.],
  [54.,  0.,  0.],
  [33.,  0.,  0.]],

 [[74.,  0.,  0.],
  [31.,  0.,  0.],
  [43.,  0.,  0.],
  [53.,  0.,  0.]]]
</code></pre>
<p>I have a index<br>
tensor B:</p>
<pre><code class="lang-auto">[[ 0,  2],
 [ 1,  3]]
</code></pre>
<p>How do I extract <code>0</code>, <code>2</code> lines from first sample, and <code>1</code>, <code>3</code> from the second sample to get:</p>
<pre><code class="lang-auto">[[[66.,  0.,  0.],
  [54.,  0.,  0.]],

 [[31.,  0.,  0.]
  [53.,  0.,  0.]]]
</code></pre>

username: zcajiayin
cooked: <p>I found a function called <code>[index_select](https://pytorch.org/docs/stable/torch.html#torch.index_select)</code>, but it doesn’t work for batch, because the argument <code>index</code> is a 1-D tensor. How can I do this for batch?</p>

username: zcajiayin
cooked: <p>Just find a way to do this using <code>gather</code>:</p>
<pre><code class="lang-auto">dummy = B.unsqueeze(2).expand(B.size(0), B.size(1), A.size(2))
out = torch.gather(A, 1, dummy)
</code></pre>

username: albanD
cooked: <p>Yes, gather is the way to go here !</p>
How weights and bias are handled in the core and why?

username: marcindulak
cooked: <p>I’m trying to find the answer to the question asked at<br>
" <a href="https://stackoverflow.com/questions/51110604/proper-way-to-implement-biases-in-neural-networks" rel="nofollow noopener">Proper way to implement biases in Neural Networks</a>"</p>
<p>Is the separation between the weights and biases due to: 1. readability of the implementation, 2. performance considerations, 3. flexibility/modularity, 4… others?</p>
<p>Please link to the relevant code if possible.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I have the same opinion as most people on the link you sent:</p>
<ul>
<li>It is easier to implement as you only do x*w +b. You don’t need to do concatenation of 1s to x.</li>
<li>The concatenation can be bad for performance.</li>
<li>You can easily get more flexibility in how to initialize or regularize them differently.</li>
</ul>
How is the pytorch Tensor data structure implemented?

username: varun-invent
cooked: <p>Recently I found out that after creating a pytorch tensor like following,<br>
w = torch.randn(5,10)<br>
I can initialize any number of arbitrary number of variables inside the tensor <code>w</code> such as,<br>
w.a = 1<br>
w.b = 1.5 etc.</p>
<p>Can anyone point me to the use of such feature? Also if anyone can point me to the code snippet where the data structure of pytorch Tensor is defined, that will be really great! I wish to know more about it.</p>
<p>Thanks!<br>
Varun</p>

username: albanD
cooked: <p>Hi,</p>
<p>There is no good use case for this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
This is a historical thing that probably shouldn’t be allowed.<br>
Unfortunately, we cannot change this now because that would break existing code quite badly.</p>
Create a dataset from a GAN

username: jsonnet
cooked: <p>Hi everyone,</p>
<p>I am currently playing around with GANs and build a simple one for CIFAR10.<br>
For further use, e.g. as an evaluation set for a CNN, I wanted to save some generated images, 
but I couldn’t figure out how I would create a dataset and save them for later loading into a DataLoader.
 <em>Already gave the answer myself</em> <img src="https://discuss.pytorch.org/images/emoji/apple/man_facepalming.png?v=9" title=":man_facepalming:" class="emoji" alt=":man_facepalming:"></p>
<p>For simplicity let’s say I know in advance which class is generated (0-9) and tell my generator to generate 100 airplane images which I want to save.</p>
<p>So im my example the generator would take the random noise and generate the specified amount of images. 
But at that point I am stuck, I though about something like below, but this didn’t seem to work as I got some size mismatch error.
 <em>Derp, see below</em> <img src="https://discuss.pytorch.org/images/emoji/apple/man_facepalming.png?v=9" title=":man_facepalming:" class="emoji" alt=":man_facepalming:"></p>
<pre><code class="lang-auto">gen_imgs = generator(Variable(torch.randn(100, 100, 1, 1)))
gen_data = data_utils.TensorDataset(gen_imgs, torch.from_numpy(np.array([0]*100)))
</code></pre>
<p>
I would be happy, if someone could fill me in, on how I could realise something like that.
<br>
Thanks!</p>

username: stante
cooked: <p>Hello,</p>
<p>I am not sure if I completely understood your question because more or less you wrote the solution. I don’t know the size of your gen_imgs. But for the sake of simplicity assume I assume the following:</p>
<pre><code class="lang-python"># Assume a batch dimension of 128, 3 channels and 32 pixel width and height
gen_imgs = torch.randn(128, 3, 32, 32)
# You Y has to match the batch dimension
dataset = data_utils.TensorDataset(gen_imgs, torch.from_numpy(np.array([0]*128)))
dataloader = data_utils.DataLoader(dataset, batch_size=8)

for x, y in dataloader:
    # do something
    pass
</code></pre>
<p>Or do you want to know how to write the result of your generator on disk?</p>

username: jsonnet
cooked: <p>You are totally right, and it was 100% my fault for not reading the error message right, I though I got some way different tensor, but actually it was the right one only with the fact, that I resized the CIFAR images for the GAN to 64x64 thus resulting a [100 x 16384] vector and my CNN was expecting 4096 (for 32x32).<br>
So a quick change to <code>TensorDataset(F.interpolate(gen_imgs, 32), torch.from_numpy(np.array([0] * 100)))</code> and everything was working.<br>
So thanks for pointing that out!</p>
How can I visualize a 3D convolution filters?

username: isalirezag
cooked: <p><a href="https://discuss.pytorch.org/t/understanding-deep-network-visualize-weights/2060/33">Here</a> I learn to visualize 3d convolusion filters, How I can visualize the convolusion filters for a model with 3D convolusion?</p>

username: ahkarami
cooked: <p><a class="mention" href="/u/isalirezag">@isalirezag</a><br>
There is a great reference for understanding different kinds of Convolution Operators (<strong>3D Convolution</strong>, <em>Spatially separable convolution</em>, <em>depthwise separable convolution</em>, etc.):<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://d3g6anj9jkury9.cloudfront.net/assets/favicons/favicon-16x16-c1207cd2f3a20fd50de0e585b4b307a3.png" class="site-icon" width="16" height="16">
      <a href="https://leanpub.com/convolutions-for-deep-learning" target="_blank" rel="nofollow noopener">Leanpub</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:95/124;"><img src="https://d2sofvawe08yqg.cloudfront.net/convolutions-for-deep-learning/medium?1575064090" class="thumbnail"></div>

<h3><a href="https://leanpub.com/convolutions-for-deep-learning" target="_blank" rel="nofollow noopener">Deep Dive into Different Types of Convolutions for Deep Learning</a></h3>

<p>This book teaches the different types of convolution operators to design Deep Neural Networks with a variety of illustrative figures and examples. The basic concepts of the convolution operator to its advanced types are described with full details in...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Help understanding 3D Convolution

username: Manuel_Alejandro_Dia
cooked: <p>I have a image composed of M channels of H height and W width and I want to apply a channel-wise convolution, so I thought of using the <a href="https://pytorch.org/docs/stable/nn.html#conv3d" rel="nofollow noopener">Conv3d</a> class. Currently, my image has shape (M, H, W)</p>
<p>But in the docs they specify that the input must be (N, Cin, D, H, W). What I know is that N is the minibatch size, H is the height and W is the width. But I am getting confused about Cin and D.</p>
<p>From what I understand, Cin is the number of channels of the image, but what does D mean? To do the convolution on my image should I pass it like (N, 1, M, H, W) or like (N, M, 1, H, W)?</p>
<p>Thanks!</p>

username: JuanFMontesinos
cooked: <p>3D convolutions are supposed to deal with temporal structures, in short, a video.<br>
D is in this case is something amount of images</p>

username: Manuel_Alejandro_Dia
cooked: <p>Thanks man! Seems like I’ll be using Conv2D then.</p>

username: ahkarami
cooked: <p><a class="mention" href="/u/manuel_alejandro_dia">@Manuel_Alejandro_Dia</a> &amp; <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a><br>
There is a great reference for understanding Different kinds of Convolution Operators (<strong>3D Convolution</strong>, <em>Spatially separable convolution</em>, <em>depthwise separable convolution</em>, etc.):<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://d3g6anj9jkury9.cloudfront.net/assets/favicons/favicon-16x16-c1207cd2f3a20fd50de0e585b4b307a3.png" class="site-icon" width="16" height="16">
      <a href="https://leanpub.com/convolutions-for-deep-learning" target="_blank" rel="nofollow noopener">Leanpub</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:95/124;"><img src="https://d2sofvawe08yqg.cloudfront.net/convolutions-for-deep-learning/medium?1575064090" class="thumbnail"></div>

<h3><a href="https://leanpub.com/convolutions-for-deep-learning" target="_blank" rel="nofollow noopener">Deep Dive into Different Types of Convolutions for Deep Learning</a></h3>

<p>This book teaches the different types of convolution operators to design Deep Neural Networks with a variety of illustrative figures and examples. The basic concepts of the convolution operator to its advanced types are described with full details in...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
How to implement customized convolution operation

username: Marsrocky
cooked: <p>I am implementing a convolutional layer with customized convolution operator (such as deformable convolution). I find that I have to extend Pytorch by C and now I want to find the source code of convolution. Or is there any other simpler ways to do it?</p>
<p>Thank you!</p>

username: ginobilinie
cooked: <p>I have also tried to do so. Wait for better suggestions.</p>

username: shirui-japina
cooked: <p>Waiting, too + 1 <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>

username: ptrblck
cooked: <p>You could try to use <code>unfold/fold</code> for a custom convolution. Note that it will be most likely slower than other optimized methods.</p>

username: Pranaymodukuru
cooked: <p>I would like to implement a custom convolution layer, where in I actually modify each individual part of the input with other external filter which also requires gradient.</p>
<p>I am succesful in implementing that in python, but it is very slow as expected.</p>
<p>Now to speed it up, I would like to implement the convolution in C++ front end. As shown here <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_extension.html</a></p>
<p>But want to use the backward pass of the inbuilt convolution. Also the extra parameter which is a part of the layer should get the gradients as well.</p>
<p>Before starting it, I wanted to know whether I need to write my own backward pass, or does the c++ end take care of it.</p>
<p>Or is there any other simpler way, please suggest.<br>
Thank you</p>

username: ahkarami
cooked: <p>There is a great reference for understanding Different kinds of Convolution Operators:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://d3g6anj9jkury9.cloudfront.net/assets/favicons/favicon-16x16-c1207cd2f3a20fd50de0e585b4b307a3.png" class="site-icon" width="16" height="16">
      <a href="https://leanpub.com/convolutions-for-deep-learning" target="_blank" rel="nofollow noopener">Leanpub</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:95/124;"><img src="https://d2sofvawe08yqg.cloudfront.net/convolutions-for-deep-learning/medium?1575064090" class="thumbnail"></div>

<h3><a href="https://leanpub.com/convolutions-for-deep-learning" target="_blank" rel="nofollow noopener">Deep Dive into Different Types of Convolutions for Deep Learning</a></h3>

<p>This book teaches the different types of convolution operators to design Deep Neural Networks with a variety of illustrative figures and examples. The basic concepts of the convolution operator to its advanced types are described with full details in...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Convolution Transpose Padding

username: willy
cooked: <p>Is there any good resources that I refer to for understanding convolution transposes? I am currently having trouble understanding how exactly do the padding and output padding works on convolution transpose.</p>

username: ptrblck
cooked: <p>I like the <a href="http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html#transposed-convolution-arithmetic" rel="nofollow noopener">Theano conv arithmetic tutorial</a>.</p>

username: ahkarami
cooked: <p>There is a great reference for understanding Transposed Convolution Operator:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://d3g6anj9jkury9.cloudfront.net/assets/favicons/favicon-16x16-c1207cd2f3a20fd50de0e585b4b307a3.png" class="site-icon" width="16" height="16">
      <a href="https://leanpub.com/convolutions-for-deep-learning" target="_blank" rel="nofollow noopener">Leanpub</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:95/124;"><img src="https://d2sofvawe08yqg.cloudfront.net/convolutions-for-deep-learning/medium?1575064090" class="thumbnail"></div>

<h3><a href="https://leanpub.com/convolutions-for-deep-learning" target="_blank" rel="nofollow noopener">Deep Dive into Different Types of Convolutions for Deep Learning</a></h3>

<p>This book teaches the different types of convolution operators to design Deep Neural Networks with a variety of illustrative figures and examples. The basic concepts of the convolution operator to its advanced types are described with full details in...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
DQN - exploding loss problem

username: soulless
cooked: <p>I am trying to solve an image localization problem similar to the paper below. In short, I am trying to train an agent with dqn to control a bounding box to localize an object. I pretrained a resnet on image patches that contain and do not contain the object of interest. Then I add Linear layers to the resnet to form the DQN.<br>
The DQN has several actions like translation and scaling.</p>
<p>The problem I am facing right now is an exploding loss problem. The loss keeps on increasing as I train it. With an Adam optimizer, I have tried learning rate ranging from 1e-3 to 1e-12 with batch size 50, 100 and 200. I also tried techniques like double dqn and prioritized experience replay. However, the exploding loss problem still cannot be alleviated. Therefore, I am writing to seek advice and suggestions on any possible reasons for this happening.</p>
<p><a href="http://slazebni.cs.illinois.edu/publications/iccv15_active.pdf" class="onebox" target="_blank" rel="nofollow noopener">http://slazebni.cs.illinois.edu/publications/iccv15_active.pdf</a></p>
<p>Thank you very much</p>

username: timesler
cooked: <p>What are the activations on the outputs of your linear layers? Are you able to show the pytorch model code?</p>
Stack python lists to tensor for following computations

username: Knievel
cooked: <p>I’m trying to remove an ugly and inefficient nested for-loop in my forward pass with matrix operations, but the result I came (stacking my lists to one tensor) to, makes my network not learn anymore. (Simplified versions of my forward pass are below.)</p>
<p>For the output I need to combine 4 states for each pixel of an 2D image. In the final for loop I iterate over all points and concat the 4 corresponding states to one tensor (batch_size x 4<em>hidden_size) to pass them through a linear layer with dimensions (4</em> hidden_size, num_classes).</p>
<p>To be more efficient I tried to convert the lists to a tensor that holds all states and reshape it, so the last dimension contains all 4 hidden states (i.e. height x width x batch_size x 4* hidden_size). Then I pass the whole tensor through the same linear layer.</p>
<p>My linear and softmax layer are created exactly the same way in both versions, but from my understanding this should work if the last dimension of input to them is the same.<br>
All the resulting shapes match up, however just computing the error on the newly initialized network is more than doubled from version 1 to 2 (same seeds, multiple tries) and version 2 is not able to learn anymore.</p>
<p>Version 1:</p>
<pre><code class="lang-auto">result = torch.zeros(height, width, batch_size, num_classes)
states = []

for d in directions:  # always 4 in my case    
    h_states = []
    for h in range(height):
        w_states = []
        for w in range(width):
            state = compute_hidden_state(input[h, w], last_h_state, last_w_state)
            w_states.append(state)
        h_states.append(w_states)
    states.append(h_states)

# states is now a list of dimensions 4 x height x width x batch_size x hidden_size

for h in range(height):
    for w in range(width):
        current_states = torch.cat((states[0][h][w], states[1][h][w], states[2][h][w], states[3][h][w]), 1)
        output = linear_layer(current_states)
        result[h, w] = softmax_layer(output)
</code></pre>
<p>Version 2:</p>
<pre><code class="lang-auto">states = []

for d in directions:  # always 4 in my case    
    h_states = []
    for h in range(height):
        w_states = []
        for w in range(width):
            state = compute_hidden_state(input[h, w], last_h_state, last_w_state)
            w_states.append(state)
        h_states.append(torch.stack(w_states))
    states.append(torch.stack(h_states))

states_tensor = torch.stack(states)

# states is now a tensor of dimensions 4 x height x width x batch_size x hidden_size

states_tensor = states_tensor.permute(1, 2, 3, 0, 4).contiguous().view(height, width, batch_size, -1)
output = linear_layer(states_tensor)
result = softmax_layer(output)
</code></pre>
<p>(Not sure why my code blocks format differently)</p>

username: Knievel
cooked: <p>As always, my mistake was somewhere completely different, sorry for wasting your time if you read this.</p>
How to get the minimal subset of libtorch for inferencing in C++?

username: Wenlong_Shi
cooked: <p>If I just want to use libtorch for inferencing with a saved model in C++, how to get the minimal subset of libtorch for the task? The whole libtorch is so large.</p>

username: albanD
cooked: <p>Hi,</p>
<p>To the best of my knowledge, this is not possible at the moment. This is mainly because we have a dynamic dispatcher and so which part of the lib will be used is not known at compilation time.<br>
That being said, I think there is work towards this as this would be very useful for the mobile build as well. But I don’t think there is any solution yet.</p>

username: nbubis
cooked: <p>The documentation seems to suggest there will be a way in the future:</p>
<p>“As of PyTorch 1.3, PyTorch supports an end-to-end workflow from Python to deployment on iOS and Android. This is an early, experimental release that we will be building on in several areas over the coming months: … Build level optimization and selective compilation depending on the operators needed for user applications (i.e., you pay binary size for only the operators you need)”</p>
<p>However, it doesn’t look like there’s an easy way to actually do so right now, I’d love to hear from anyone who’s succeeded. Following the android tutorial gives an app size (sans model) of around 70MB, clearly too large for most applications.</p>
Resetting Dataparallel after initialization, and using it in a dictionary

username: pytorcher
cooked: <p>Two questions, first is that I have a network that I would like to train for a little while using dataparallel and then modify the network and continue to train with dataparallel.  What is the correct way to reinitialize?  Should I remove the dataparallel in some way or just call it again on net.module(which runs but doesn’t appear to work as expected)?<br>
Second question is continued from here: <a href="https://discuss.pytorch.org/t/how-to-transition-to-functions-not-being-allowed-to-have-member-variables/62170/2" class="inline-onebox">How to transition to functions not being allowed to have member variables</a>.  Now that I am using a dictionary it appears the tensors in that dictionary are not split across GPUs.  What’s the best way to include them in the dataparallel?</p>
Why do I use net.zero_grad() and then go backward on loss. The gradient before and after execution is the same

username: cclover
cooked: <p>Why do I use net.zero_grad() and then go backward on loss. The gradient before and after execution is the same</p>
<pre><code class="lang-auto">print(out)
target = Variable(t.arange(0,10)).float()
criterion = nn.MSELoss()
loss = criterion(out, target)
print(loss)
#梯度清0
net.zero_grad()
print("反向传播之前的conv1.bias的梯度")
print(net.conv1.bias.backward)
loss.backward(retain_graph=True)
print("反向传播之后的梯度")
print(net.conv1.bias.backward)

</code></pre>
<p><strong>反向传播之前的conv1.bias的梯度</strong><br>
&lt;bound method Tensor.backward of Parameter containing:<br>
tensor([ 0.0900,  0.1849, -0.1985, -0.0836, -0.1590,  0.0689],<br>
requires_grad=True)&gt;<br>
<strong>反向传播之后的梯度</strong><br>
&lt;bound method Tensor.backward of Parameter containing:<br>
tensor([ 0.0900,  0.1849, -0.1985, -0.0836, -0.1590,  0.0689],<br>
requires_grad=True)&gt;</p>

username: mailcorahul
cooked: <ol>
<li>
<code>net.zero_grad()</code> zeroes out <code>grad</code> attribute of every parameter in the <code>net</code>.</li>
<li>
<code>loss.backward()</code> computes gradients of every parameter w.r.t <code>loss</code> tensor.</li>
</ol>
<p>In order to see the gradient of a parameter, you can do <code>print(net.conv1.bias.grad)</code> instead of <code>print(net.conv1.bias.backward)</code>.<br>
<code>backward</code> as you see is a method and it is showing the actual parameters and not the gradients.</p>
<p>The fact that the parameters remain the same before and after <code>backward()</code> call is because you need to do <code>optimizer.step()</code> in order to update the parameters.</p>

username: cclover
cooked: <p>you are right！<br>
Thanks</p>
ValueError: Expected input batch_size (1) to match target batch_size (2)

username: whm200101218
cooked: <p>It is from a piece of simple code</p>
<p>x = torch.randn(1, 100)</p>
<p>criterion = nn.NLLLoss()</p>
<p>y = np.array([1,0])</p>
<p>y = torch.from_numpy(y)</p>
<p>y=y.long()</p>
<p>f1=nn.Sequential(nn.Linear(100,10),nn.ReLU(),nn.Linear(10,2),nn.LogSoftmax(dim=1))</p>
<p>predict=f1(x)</p>
<p>predict=predict.squeeze()</p>
<p>loss = criterion(input1,y) <span class="hashtag">#error</span> from here</p>
<p>print(loss)</p>

username: Ted
cooked: <pre><code class="lang-python">
x = torch.randn(1, 100)

criterion = nn.NLLLoss()
# binary label
y = np.array([1])
# if u have K classes the target label will be made out of interegers between 0 and K-1
# in your case since you have 2 classes lables will be 0 and 1

y = torch.from_numpy(y)
y=y.long()

f1=nn.Sequential(nn.Linear(100,10),nn.ReLU(),nn.Linear(10,2),nn.LogSoftmax(dim=1))

predict=f1(x)

loss = criterion(predict,y)

print(loss)

</code></pre>
Intercepting convolutional kernel inputs

username: guitarman
cooked: <p>I was wondering if it is possible to somehow grab the inputs NxN that are processed for each of the kernel convolution on each pixel of the image?</p>
<p>I’m trying to analyse each of the kernel data before and somehow connect it with a specific output neuron.</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>I’m not sure, if I understand your use case correctly, but if you want to compute the convolution manually, you could use <code>unfold</code>. Have a look at <a href="https://discuss.pytorch.org/t/convolution-that-only-take-channel-wise-summation/21240/4">this post</a>, which might be a good starter.</p>

username: guitarman
cooked: <p>Thank you very much for the useful info!</p>
What is a soft label? Is soft label different from general CE in the Teacher model?What is a soft label? Is soft label different from general CE in the Teacher model?

username: 439327bcc62cd8e45e96
cooked: <p>What is a soft label? Is soft label different from general CE in the Teacher model?</p>

username: mailcorahul
cooked: <p>I suppose you are referring to Knowledge Distillation/Model compression.</p>
<p>A model’s output(called teacher model in this context) is what generally referred to as <code>soft label</code>. It is called soft because the output may not be strictly something like <code>[1, 0, 0]</code> for a 3-class classification task, instead it might something like <code>[0.85, 0.1, 0.05]</code>.</p>
<p>This soft label is used to train a much smaller network(called student model) instead of using the hard targets.</p>
<p>Hope this makes sense. Thanks.</p>

username: 439327bcc62cd8e45e96
cooked: <p>Thank you for your kind reply.</p>
Fine-tuning Reset50

username: gss
cooked: <p>I am new to Pytorch and CNN and am currently working on this network but as I increase the batch size or change the learning rate it doesn’t seem to improve. I have been reading about changing the last layers. I am not sure how to implement that in my current model, though. Any ideas on how I could do that? Thanks.</p>
<pre><code class="lang-auto">transformations = transforms.Compose([
    #transforms.Resize((800,600),interpolation=Image.BICUBIC),
    transforms.Resize(256),
    transforms.RandomHorizontalFlip(),
    transforms.RandomVerticalFlip(),
    transforms.ColorJitter(brightness=0.1,saturation=10,contrast=30,hue=0.1),
    #transforms.CenterCrop(224),
    transforms.ToTensor()])

resnet50=models.resnet50()

# Change the last layer
num_ftrs = resnet50.fc.in_features
resnet50.fc = nn.Linear(num_ftrs, 17)

model =resnet50.to(device)
print(model)
optimizer = torch.optim.SGD(model.parameters(), lr=LR)
criterion = nn.BCEWithLogitsLoss()
</code></pre>

username: gss
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Any guide you can give me? Please <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"></p>

username: mailcorahul
cooked: <p>Are you finetuning Resnet50 on any custom dataset? Can you give more info about the loss/accuracy values?</p>

username: mailcorahul
cooked: <p>Is this a multi-label classification problem? Also make sure there is no <code>nn.Sigmoid</code> module at the end of the network output since you are using <code>nn.BCEWithLogitsLoss()</code>.</p>
Given groups=1, weight of size 32 1 5 5, expected input[2, 3, 96, 96] to have 1 channels, but got 3 channels instead

username: liang_chen
cooked: <p>Why the expected channel is 1 rather than 3, I have written in conv1 that its in_channels = 3.<br>
There might be some problem in the Model, I can’t figure it out, can somebody help me?<br>
Thank you.</p>
<pre><code class="lang-auto">class CNN_Discriminator(nn.Module):
    def __init__(self):
        super(CNN_Discriminator, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(in_channels=3, out_channels=32, kernel_size=5, padding=2),  # batch, 32, 96，96,
            nn.LeakyReLU(0.2, True),
            nn.AvgPool2d(2, stride=2),  # batch, 32, 48, 48
        )
        self.conv2 = nn.Sequential(
            nn.Conv2d(32, 64, 5, padding=2),  # batch, 64, 48, 48
            nn.LeakyReLU(0.2, True),
            nn.AvgPool2d(2, stride=3)  # batch, 64, 16, 16
        )
        self.fc = nn.Sequential(
            nn.Linear(64 * 16 * 16, 1024),
            nn.LeakyReLU(0.2, True),
            nn.Linear(1024, 1),
            nn.Sigmoid()
        )

    def forward(self, x):
        '''
        x: batch, width, height, channel=3
        '''
        x = self.conv1(x)
        x = self.conv2(x)
        x = x.view(x.size(0), -1)
        x = self.fc(x)
        return x
</code></pre>

username: ptrblck
cooked: <p>Your model works fine with this input:</p>
<pre><code class="lang-python">model = CNN_Discriminator()
x = torch.randn(2, 3, 96, 96)
out = model(x)
</code></pre>
<p>Are you sure you are using this model definition?</p>

username: liang_chen
cooked: <p>Thank you I find out that that I import a wrong local model <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"><img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"><img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"></p>
Could someone explain the intuition of using trained models of data with a completely different vocabulary?

username: NoreOxford
cooked: <p>Something occurred to me as I was trying to apply my trained models on new datasets, when I tokenised the new dataset, some of the words were obviously assigned a different numeric value than the same word in the original dataset that was used for the training/validation. How can the model make predictions when the datasets have different vocabularies and thus different numeric values for the same words? I am guessing it has something to do with models learning embedding layers, is that correct? Does this apply to non-deep learning models like the ones in sklearn (Forests, SVMs, NB, etc)? I’m just trying to understand the intuition for how it works. Thanks!</p>

username: mobius
cooked: <p>I’m not sure I understood 100% your question but I think the language word2idx/ embedding mapping should not change if you want to make use of the knowledge contained in your word embeddings.</p>
<p>Having words map to a different word vector space is like initializing the embeddings randomly. Your model will probably converge if you allow the gradients to propagate to the embeddings and it will move the word to a vector space that makes sense for the problem you are solving but you are not taking advantage of any information the word embedding has to offer.</p>
Stacked two LSTMs with different hidden layers

username: chan_man
cooked: <p>Hi,</p>
<p>I would like to create LSTM layers which contain different hidden layers to predict time series data, for the 1st layer of LSTM_1 contains 10 hidden layers,  LSTM_2 contains 1 hidden layer, the proposed neural network architecture is illustrated following<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6bbbeb43b658cb5b21cd956f2e1c11a416c98b5.png" alt="f1" data-base62-sha1="uDCq3P8ojyqdA5gMQ9bToj54mRT" width="433" height="294"></p>
<pre><code>def __init__(self, nb_features=1, hidden_size_1=100, hidden_size_2=100, nb_layers_1 =10, nb_layers_2 = 1, dropout=0.5): #(self, nb_features=1, hidden_size=100, nb_layers=10, dropout=0.5):      
    super(Sequence, self).__init__()
    self.nb_features = nb_features
    self.hidden_size_1 = hidden_size_1
    self.hidden_size_2 = hidden_size_2
    self.nb_layers_1 =nb_layers_1
    self.nb_layers_2 = nb_layers_2
    self.lstm_1 = nn.LSTM(self.nb_features, self.hidden_size_1, self.nb_layers_1, dropout=dropout)  
    self.lstm_2 = nn.LSTM(self.hidden_size_1, self.hidden_size_2, self.nb_layers_2, dropout=dropout)
    self.lin = nn.Linear(self.hidden_size_2, 1)



def forward(self, input):  
    h_t1 = Variable(torch.zeros(self.nb_layers_1, input.size()[1], self.hidden_size_1))
    c_t1 = Variable(torch.zeros(self.nb_layers_1, input.size()[1], self.hidden_size_1))
    h_t2 = Variable(torch.zeros(self.nb_layers_2, input.size()[1], self.hidden_size_2))
    c_t2 = Variable(torch.zeros(self.nb_layers_2, input.size()[1], self.hidden_size_2))
    outputs = []

    for i, input_t in enumerate(input.chunk(input.size(1))):
        h_t1, c_t1 = self.lstm_1(input_t, (h_t1, c_t1))
        h_t2, _ = self.lstm_2(h_t1, (h_t2, c_t2))
        output = self.lin(h_t2[-1])
        outputs += [output]

    outputs = torch.stack(outputs, 1).squeeze(2)

    return outputs
</code></pre>
<p>However, I got runtime error for the above code “RuntimeError: Expected hidden[0] size (3, 60, 100), got (1, 60, 100)”. is it correct for the above code, Could you please give some suggestions? Many thanks</p>

username: mobius
cooked: <p>Quick reply, I didn’t try your code but it seems that you are ignoring the lstm output</p>
<p>i.e. these two lines:</p>
<pre><code class="lang-auto">h_t1, c_t1 = self.lstm_1(input_t, (h_t1, c_t1))
h_t2, _ = self.lstm_2(h_t1, (h_t2, c_t2))
</code></pre>
<p>should read:</p>
<pre><code class="lang-auto">_, (h_t1, c_t1) = self.lstm_1(input_t, (h_t1, c_t1))
_, (h_t2, _) = self.lstm_2(h_t1, (h_t2, c_t2))
</code></pre>
<p>if you are not interested in the lstm output at all and just want to use the hidden states.</p>
<p>Check the LSTM output here <a href="https://pytorch.org/docs/stable/nn.html#lstm" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#lstm</a></p>
Take second gradient

username: pomonam
cooked: <p>Hello,</p>
<p>I have a function f which takes in two variables x and y. I would like to minimize:<br>
x = x - alpha * \frac{\partial f}{\partial x} for x and<br>
y = y - alpha * \frac{\partial^2 f}{\partial x \partial y} for y.</p>
<p>I have two optimizers that take minimize x and y respectfully. The gradient step for x is simple, but I am not sure how to construct a gradient step for y. Any pointers would be very helpful! Thank you.</p>

username: vainaijr
cooked: <p>do you mean something like this,</p>
<pre><code class="lang-auto">import torch

x = torch.tensor([1.], requires_grad=True)
y = torch.tensor([3.], requires_grad=True)

f = (x*y)**2

grads_x = torch.autograd.grad(f, x, create_graph=True)

grads_x # 2*x * y**2 = 2*1 * 3**2 = 18
</code></pre>
<p>(tensor([18.], grad_fn=),)</p>
<pre><code class="lang-auto">grads_x_y = torch.autograd.grad(grads_x, y)
grads_x_y # 2*x * 2*y = 2*1 * 2*3 = 12
</code></pre>
<p>(tensor([12.], grad_fn=),)</p>
<p>or</p>
<pre><code class="lang-auto">f = x**2 + y**2
grads_x = torch.autograd.grad(f, x, create_graph=True)
grads_x # 2*x = 2*1 = 2
</code></pre>
<p>(tensor([2.], grad_fn=),)</p>
<pre><code class="lang-auto">grads_x_y = torch.autograd.grad(grads_x, y, allow_unused=True)
grads_x_y
</code></pre>
<p>(None,)</p>