akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
View size is not compatible with input tensor's - But it Does!

barakb: <p>I’m trying to do a simple line:</p>
<pre><code class="lang-auto">        w = w.view(w.shape[0] * w.shape[2], w.shape[1], 1,1,1)

</code></pre>
<p>And I’m getting an error:</p>
<p>view size is not compatible with input tensor’s</p>
<p>Not only I checked a million times the sizes and that they are compatible, but it’s easy to see that the operation must be the same.</p>
<p>Is it because “view” is just looking at the memory differently? and maybe here he can’t? any Idea??</p>
<p>Thanks.</p>

ptrblck: <p>Could you post a code snippet to reproduce this error?</p>

barakb: <p>It’s hard it’s a big program, and it used to work before.<br>
But I solved it using reshape.<br>
Sorry for bothering.</p>
RuntimeError: cublas runtime error : library not initialized at

yang2: <p>After I tried the solution from  <a href="https://discuss.pytorch.org/t/cublas-runtime-error-library-not-initialized-at-data-users-soumith-builder-wheel-pytorch-src-torch-lib-thc-thcgeneral-c-383/1375">issue 1375</a>, i.e. <code>sudo rm -r ~/.nv</code>, I still got the same error like before.<br>
But the problem happened when I called the model’s <code>forward()</code> function instead of the steps of placing the input tensors on GPU.<br>
The version of python I use now is 3.5, pytorch 0.3.1, cudann 7.0.5. I got two GPUs on my computer and used the below snippet for specifying GPU:</p>
<pre><code class="lang-auto">opt_gpu = 0 
os.environ["CUDA_VISIBLE_DEVICES"] = opt_gpu
</code></pre>
<p>The detailed errors are:</p>
<pre><code class="lang-auto">RuntimeError: cublas runtime error : library not initialized at /opt/conda/conda-bld/pytorch_1518241081361/work/torch/lib/THC/THCGeneral.c:405
/opt/conda/conda-bld/pytorch_1518241081361/work/torch/lib/THC/THCTensorScatterGather.cu:176: void THCudaTensor_scatterFillKernel(TensorInfo&lt;Real, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [0,0,0], thread: [106,0,0] Assertion `indexValue &gt;= 0 &amp;&amp; indexValue &lt; tensor.sizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1518241081361/work/torch/lib/THC/THCTensorScatterGather.cu:176: void THCudaTensor_scatterFillKernel(TensorInfo&lt;Real, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [0,0,0], thread: [360,0,0] Assertion `indexValue &gt;= 0 &amp;&amp; indexValue &lt; tensor.sizes[dim]` failed.
/opt/conda/conda-bld/pytorch_1518241081361/work/torch/lib/THC/THCTensorScatterGather.cu:176: void THCudaTensor_scatterFillKernel(TensorInfo&lt;Real, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [1,0,0], thread: [80,0,0] Assertion `indexValue &gt;= 0 &amp;&amp; indexValue &lt; tensor.sizes[dim]` failed.
</code></pre>
<p>Any advices, thanks.</p>

zzuczy: <p>I got same error as yours. Can yu tell me how to fit it?</p>
No CUDA-capable device is detected

ibtingzon: <p>When I run</p>
<pre><code class="lang-auto">import torch
torch.cuda.FloatTensor([1.])
</code></pre>
<p>I seem to be getting the error: <code>RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:50</code>.</p>
<p>I’m using Debian (Stretch). Checking the nvidia driveron the console:</p>
<pre><code class="lang-auto">/sbin/modinfo nvidia_current
</code></pre>
<p>returns:</p>
<pre><code class="lang-auto">filename:       /lib/modules/4.9.0-11-amd64/updates/dkms/nvidia-current.ko
alias:          char-major-195-*
version:        418.74
supported:      external
license:        NVIDIA
srcversion:     AB4044DE27C9CA55579A110
</code></pre>
<p>when I do <code>nvcc --version</code>, I get:</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Tue_Jun_12_23:07:04_CDT_2018
Cuda compilation tools, release 9.2, V9.2.148
</code></pre>
<p>On python, when I do</p>
<pre><code class="lang-auto">import torch
print(torch.version.cuda)  
print(torch.cuda.device_count())
print(torch.cuda.is_available())
</code></pre>
<p>I get:</p>
<pre><code class="lang-auto">9.2.148
0
False
</code></pre>
<p>I’ve tried doing <code>os.environ["CUDA_VISIBLE_DEVICES"] = '0</code>, but it still doesn’t work.</p>
<p>Any help would be much appreciated!</p>

ptrblck: <p>Did you update the NVIDIA drivers or CUDA recently? If so, I would recommend to restart the machine, as we’ve seen some similar errors in the past.</p>
Do not call optimizer.zero_grad()

Wendell_Philips: <p>I want to use gradient accumulation like below:</p>
<pre><code class="lang-auto">accum=8
optimizer.zero_grad():
for step in train_loader:
    out=model(input)
    loss=criterion(out,target)
    loss.backwad()
   if step % accum==0:
        optimizer.step()
</code></pre>
<p>I  don’t call optimizer.zero_grad() again. What will be? Since my training looks very well.</p>

ptrblck: <p>The gradients will be accumulated for all batches in your <code>train_loader</code>, while the optimizer performs a step after <code>accum</code> steps.<br>
Usually you would zero out the gradients after calling <code>optimizer.step()</code>.<br>
It’s interesting to hear it’s working. Did you compare it with other approaches (e.g. zeroing out the gradients after <code>optimizer.step()</code>)?</p>
What is a hidden state in an RNN?

Anandh_Perumal_Konar: <p>Let’s say  I have a input text of sequence 5.</p>
<p>And I applied a non-Linear layer of size 20 then an LSTM layer of hidden state 40.<br>
So what does it mean here? Each hidden node in lstm is connected by all the linear layer node 20 ?<br>
If yes then for particular sequence let’s say only 10 nodes were fired, then how does the LSTM know the time step of what node need is first and what is next?</p>

phan_phan: <p>In the simplest RNN, there is only one <em>non-linear</em>, <em>fully connected</em> layer which, at each step <code>t</code>, takes as input :</p>
<ul>
<li>The vector positioned at <code>t</code> in the sequence : <code>x_t</code> (in your case, its size is 20)</li>
<li>Its own vector output from the previous step : <code>h_(t-1)</code> (in your case, its size is 40)</li>
</ul>
<p>to output the new vector <code>h_t</code> (in your case, size 40 also).<br>
This vector is the hidden state at step <code>t</code>.</p>
<p>A LSTM is much more complex (<a href="https://pytorch.org/docs/stable/nn.html#lstm" rel="nofollow noopener">see the docs</a> to see how everything is computed), but the global idea is the same.</p>
How to access parameters in a certain layer of pretrained network?

pinata1337: <p>Hi, i am recently trying to try out some style transfer codes facilitating activations of pretrained network in torchvision.<br>
Reference: <a href="https://discuss.pytorch.org/t/accessing-intermediate-layers-of-a-pretrained-network-forward/12113" class="inline-onebox">Accessing intermediate layers of a pretrained network forward?</a></p>
<p>The issue is that I wish to try using an object detection network such as Faster R-CNN, in which case the definition of network is kind of of different e.g. there is no more ‘.features’ attribute.<br>
How to access parameters (weight and activation in multiple layers if possible) in such case?</p>

spanev: <p>Hi <a class="mention" href="/u/pinata1337">@pinata1337</a>,</p>
<p>An easy way to find out it is to print your model:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; print(frcnn)
FasterRCNN(
  (transform): GeneralizedRCNNTransform()
  (backbone): Sequential(
    (0): ConvBNReLU(
      (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU6(inplace=True)
    )
[...]
</code></pre>
<p>You can always access the submodules of a module by it’s named attribute (or index when the Module is a <code>Sequential</code>). So, in the case of FasterR-CNN, let’s say you want to access the first <code>Conv2D</code> in the first <code>ConvBNReLU</code> in <code>backbone</code>:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; frcnn.backbone[0][0]
Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
</code></pre>

pinata1337: <p>Much obliged for the help! Now I know how to access the layers in that network.</p>
<p>However I still encountered other obstacles while trying to get activations for certain data, at a certain layer of the network(e.g. the output) .<br>
When I tried to obtain activation of a layer of the network with the code:</p>
<p>net = models.detection.fasterrcnn_resnet50_fpn(pretrained=True).eval()<br>
for inputs, targets in trainloader:<br>
aaaa = net(inputs)<br>
bbbb = net(targets)<br>
loss = nn.MSELoss(aaaa, bbbb)</p>
<p>Errors pop out:<br>
RuntimeError: The expanded size of the tensor (128) must match the existing size (800) at non-singleton dimension 2.  Target sizes: [3, 128, 128].  Tensor sizes: [3, 800, 800]</p>
<p>I do not fully understand the meaning of this error because cnn should work for any size of image(128 in my case) as long as they’re consistent…</p>
<p>I think I might need to read some kind of manual about how to access parameters of the network <em>given input data and location of the wanted parameters</em>, still looking just yet.<br>
Is there any insights you could share on this issue?</p>

mailcorahul: <p>That’s because the tensor(or images in your case) shapes passed on to MSELoss is different.<br>
One tensor is of shape (3, 128, 128) and the other is (3, 800, 800). Make sure ‘aaaa’ and ‘bbbb’ shapes’ match.</p>
How to repeat a matrix into a bigger one along both dimensions?

zfzhang: <p>What is the simplest syntax to transform 2D tensor</p>
<pre><code class="lang-auto">A   B
C   D
</code></pre>
<p>into</p>
<pre><code class="lang-auto">A  A  B  B
A  A  B  B
C  C  D  D
C  C  D  D
</code></pre>
<p>Note they are parameter tensors, so I need autograd to back propagate gradient from latter into former.<br>
Thanks!</p>

zfzhang: <p>I found a numpy.repeat()-like function in latest pytorch (1.1), but it is needed to be called twice:<br>
<code>z = x.repeat_interleave(2,dim=0).repeat_interleave(2,dim=1)</code></p>

Balamurali_M: <p>mat1 = np.random.rand(3,3)<br>
mat2 = np.ones(3,3)<br>
out = np.kron(mat1,mat2)</p>

zfzhang: <p>Thanks for the hint! Kronecker product seems to be exactly what I need. But there seems no such corresponding function in Pytorch <img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=9" title=":flushed:" class="emoji" alt=":flushed:"></p>

I-Love-U: <aside class="quote no-group" data-username="zfzhang" data-post="4" data-topic="47461">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/ecc23a/48.png" class="avatar"> zfzhang:</div>
<blockquote>
<p>But there seems no such corresponding function in Pytorch</p>
</blockquote>
</aside>
<aside class="quote" data-post="1" data-topic="3919">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vitchyr/48/704_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/kronecker-product/3919">Kronecker Product</a> 
  </div>
  <blockquote>
    Is there a way to perform a Kronecker product between two matrices? I know there’s <a href="http://pytorch.org/docs/torch.html#torch.ger" rel="noopener nofollow ugc">outer products for vectors</a>, but is there something like that for 2D Tensors?
  </blockquote>
</aside>
In Windows 10, libtorch-gpu should work in coordination with CUDA 10.0?

111149: <p>We are planning migrate our project from python to C++. With Pytorch, as it include cuda/cudnn environment itself, so we can avoid unnecessery cuda configration work. However, with Libtorch, some tutorials show it need to install CUDA 10.0 otherwise the cmake can’t find the cuda library. Is it right? Or libtorch can be linked with python cudatoolkit 10.0.130 package?</p>

peterjc123: <p>When building the library, you need to install CUDA 10.0. As a user of your library, you don’t need to have CUDA installed, only the cudatoolkit 10.0.130 package is okay.</p>
The conda install downloaded 1.1 instead of 1.2

wine_lover: <p>Hello, I installed the pytorch using ```<br>
conda install pytorch torchvision cudatoolkit=9.0 -c pytorch</p>
<pre><code class="lang-auto">However, it downloaded pytorch-1.1.0 instead of the latest version 1.2. How does that happen?</code></pre>

ptrblck: <p>The PyTorch <code>1.2.0</code> binaries ship with CUDA9.2 and CUDA10.0, so you would have to change the <code>cudatoolkit</code> version to one of these.</p>

wine_lover: <p>Hi Ptrbick, thanks, do i have to update my cuda version? on my windows system, after typing nvcc --version, the information is Cuda compilation tools, release 9.0, V9.0.176</p>

ptrblck: <p>The binaries ship their CUDA and cudnn versions, so you don’t need to install CUDA locally (unless you would like to build from source). You need to install the NVIDIA drivers, though.</p>

peterjc123: <p>The reason is that we didn’t build PyTorch 1.2 binaries with CUDA 9.0, so you should either install CUDA 9.2 or CUDA 10.0. Please see the install commands in <a href="https://pytorch.org" rel="nofollow noopener">https://pytorch.org</a>.</p>
Cannot find transformer module

Mohamed_Ragab: <p>I cannot find transformer module with pytorch version (1.1.0), please can someone help me  to get the latest libraries from PyTorch</p>

ptrblck: <p><code>nn.Transformer</code> was added in PyTorch 1.2.0, so you should update to the latest stable version.<br>
Have a look at the install instructions for the binaries: <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">link</a>.</p>

Mohamed_Ragab: <p>Thank you so much <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
How to create a directional conv filter

sakura: <p>I’d like to create some conv filters with orientation so that they can capture directional informations. The vertical and horizontal filters are easy to implement by changing aspect ratio. But how to create directional filters, such as diagonal？</p>

phan_phan: <p>Do you mean, forcing some conv filters to be only diagonal ?</p>

sakura: <p>Yes. The alternative solution is rotating feature map manually. I wonder if there are some other simple solutions?</p>

phan_phan: <p>You can create a <a href="https://pytorch.org/docs/stable/nn.html#parameters" rel="nofollow noopener"><code>nn.Parameter</code></a> <code>w</code> with a size <code>[out_channels, in_channels, kernel_size]</code>.<br>
Then <a href="https://pytorch.org/docs/stable/torch.html#torch.diag_embed" rel="nofollow noopener"><code>torch.diag_embed(w)</code></a> will have a size <code>[out_channels, in_channels, kernel_size, kernel_size]</code>, in which each squared kernel is diagonal.<br>
So you can use <a href="https://pytorch.org/docs/stable/nn.functional.html#conv2d" rel="nofollow noopener"><code>nn.functional.conv2d(x, torch.diag_embed(w))</code></a>.</p>

sakura: <p>Thanks for your reply and I have another questions. In the forward, the filter does collect the directional information. But the backward still compute the gradient which change the zeros in original filter. This solution cannot keep the shape of filter?</p>

phan_phan: <p>That is why (in the diagonal case), you can create a parameter <code>w</code> with <strong>only</strong> the size:<br>
<code>[out_channels, in_channels, kernel_size]</code> (for example : <code>[16, 32, 3]</code>).<br>
This parameter will only contain the useful diagonal information, without the useless zeros. So at the optimization step, only these floats will be updated.</p>
<p>It is just the tensor <code>torch.diag_embed(w)</code> which contains also the zeros. This one has the size:<br>
<code>[out_channels, in_channels, kernel_size, kernel_size]</code> (for example : <code>[16, 32, 3, 3]</code>).</p>
<p>And in the general case, I’m sure you can always find a way to create a parameter with only the useful floats (that need to be updated), and <strong>then</strong> transform it to the form you need.</p>

sakura: <p>Yes thats a general solution! Thanks for your help!~</p>
Convolution on non-overlapping blocks of a 3D image then remapping them

darrenjkt: <p>Hi I have a 3D MRI image of size (128,128,128) as input to my model. When it enters the model it has the shape (8, 4, 128, 128, 128) which is (Batch, Channels, H, W, D).</p>
<p>I would like to separate the channels and perform a convolution on blocks of (32,32,32) for this (128,128,128) input. Then I wish to take the conv weights and multiply it with the input values to the conv and remap them to a (128,128,128) block.</p>
<p>My current inefficient solution (using many for loops and scikit-image) is below however it takes too long and requires too much memory. What’s the best way to do this?</p>
<pre><code class="lang-auto">from skimage.util.shape import view_as_blocks

class LFBlock(nn.Module):
    def __init__(self, input_shape=(128,128,128), kernel_size=(1,1,1), blk_div=4):
        super(LFBlock,self).__init__()      
        
        # Divides the (128,128,128)//4 -&gt; (32,32,32)
        self.block_shape = (input_shape[0]//blk_div, input_shape[1]//blk_div, input_shape[2]//blk_div)
        
        self.num_blocks = (input_shape[0]//self.block_shape[0])*(input_shape[0]//self.block_shape[0])*\
        (input_shape[0]//self.block_shape[0])
        
        conv_list = []
        for n in range(self.num_blocks):
            conv_list.append(nn.Conv3d(1,1, kernel_size=kernel_size, stride=1, padding=0, bias=True))
            
        self.conv1x1s = nn.ModuleList(conv_list)        
                
    def forward(self, lf_in):
        # Batch
        for i in range(lf_in.shape[0]):
            # Modality
            for ch in range(lf_in.shape[1]):
                x_lf = lf_in[i,ch,:]
                lf_blocks = view_as_blocks(x_lf.cpu().numpy(), block_shape=self.block_shape)
                
                # Do Conv3d on each block 
                for x in range(len(lf_blocks)):
                    for y in range(len(lf_blocks)):
                        for z in range(len(lf_blocks)):
                            
                            conv_idx = x*len(lf_blocks) + y*len(lf_blocks) + z
                            
                            # Convolve the block, then multiply with the weight of the block. 
                            tensor_img = torch.from_numpy(lf_blocks[x,y,z])[None, None,:]
                            conv = self.conv1x1s[conv_idx](tensor_img.cuda())
                            
                            # w * x. 
                            # view_as_blocks returns a view so modifications are done in-place

                            lf_blocks[x,y,z] = tensor_img.cpu()*self.conv1x1s[conv_idx].weight.data.cpu()
        
        # Linearly sum the modalities together
        # out = w0*x0 + w1*x1 + w2*x2 + w3*x3 
        out = (lf_in[:,0]+lf_in[:,1]+lf_in[:,2]+lf_in[:,3])[:,None]
        
        return out
</code></pre>
<p>Any help is appreciated. Thank you!</p>
Which one should I prefer?

blackbirdbarber: <pre><code class="lang-auto">k=torch.tensor([2,1])
print(k.size())
k= k.contiguous().view(1,2) #1
# or
k= k.reshape(1,2) #2
</code></pre>

albanD: <p>The one you prefer <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
<code>.reshape()</code> has been added recently to help users more used to numpy.</p>
Why there is no inplace version of view?

blackbirdbarber: <pre><code class="lang-auto">k=torch.tensor([2,1])
k= k.view(1,2)
</code></pre>
<p>Something like <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p><code>k.view_(1,2)</code></p>

albanD: <p>I think this is mostly historical as <code>.view()</code> is so cheap anyway that you don’t really need to do it inplace.</p>
Is Clone/Detach correct for combined loss?

John1231983: <p>Hello all, I have two variables <code>a,b</code> with requires_grad=True. The <code>loss1</code> will take the <code>a</code> and <code>b</code> as inputs, while the <code>loss2</code> will take the normalization of <code>a</code> and <code>b</code> as input. So, should I use a clone in this case? This is my implementation</p>
<pre><code class="lang-auto">loss1= loss1(a,b)
a_norm = a / 255
b_norm = b/255
loss2 = loss2(a_norm, b_norm)

loss_total = loss1+loss2
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5b7442bf01489cfc093af6f97395133de5a439b5.png" data-download-href="https://discuss.pytorch.org/uploads/default/5b7442bf01489cfc093af6f97395133de5a439b5" title="fig"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5b7442bf01489cfc093af6f97395133de5a439b5_2_690x344.png" alt="fig" data-base62-sha1="d32xjgpvP4wY8lkjGGR1jq3NJ9H" width="690" height="344" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5b7442bf01489cfc093af6f97395133de5a439b5_2_690x344.png, https://discuss.pytorch.org/uploads/default/original/3X/5/b/5b7442bf01489cfc093af6f97395133de5a439b5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/b/5b7442bf01489cfc093af6f97395133de5a439b5.png 2x" data-dominant-color="F1F2EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">fig</span><span class="informations">935×467 19.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Hi,</p>
<p>You don’t need to clone or detach here. The code you have will work well.</p>

John1231983: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="56388">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>t need to clone or detach her</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/alband">@albanD</a>: Thanks. But do you think the loss 1 will be received a normalization value after gradient update because we use a_norm= a/ 255 in forward</p>

albanD: <p>When you do <code>a_norm = a / 255</code>, you do not modify <code>a</code> in any way.</p>
RuntimeError: output with device cpu and dtype Long doesn't match the desired device cpu and dtype Float

aditi: <p>I have copied a program from github and tried to run the code. The above error message is coming. Please help to resolve the issue.</p>

albanD: <p>Hi,</p>
<p>The problem is that you provide a Tensor of type Long to a function that expects a Tensor of type Float.<br>
You will need to look at the stack trace to know which function it is. Then you can fix this most likely by just converting the LongTensor to a FloatTensor by using <code>.float()</code>.</p>

aditi: <p>Thanks!! Problem resolved.</p>
Padding_mode argument for convolutional layers, only zeros and circular? How to do reflection padding?

jmiano: <p>I have been looking into how to do reflection/mirror padding of some images and activation maps rather than the default zero padding. I have 3D images so I cannot use torch.nn.ReflectionPad3d (does not exist). It seems like I could use torch.nn.functional for this, but I would prefer to keep something that fits into my torch.nn.Sequential blocks in __ init __ more cohesively, and I was unsure of how to do it with functional.</p>
<p>Then I saw something called “padding_mode” in torch.nn.Conv3d, which I think could be what I am looking for. However, the documentation only says “zeros” and “circular” as the padding_mode options, and does not have listed “replicate” or “reflect”. How can I use the padding_mode attribute to do reflection padding? Here is the documentation where I saw “zeros” and “circular” listed:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c19f02a93a2b64509a08212b8619715470eff156.png" data-download-href="https://discuss.pytorch.org/uploads/default/c19f02a93a2b64509a08212b8619715470eff156" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c19f02a93a2b64509a08212b8619715470eff156_2_690x38.png" alt="image" data-base62-sha1="rCQPsvjDNlT526GcZt8gdsjoq34" width="690" height="38" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c19f02a93a2b64509a08212b8619715470eff156_2_690x38.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c19f02a93a2b64509a08212b8619715470eff156_2_1035x57.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c19f02a93a2b64509a08212b8619715470eff156_2_1380x76.png 2x" data-dominant-color="F2E9EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1510×85 8.28 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks</p>
Calling backward doubles memory

LeanderK: <p>I am currently profiling the memory-consumption of my model. My code is essentially:</p>
<pre><code class="lang-auto">log_probs = model.log_prob(inp)
loss = (-1)*log_probs.mean()
loss.backward()
</code></pre>
<p>I have ~4 gig of GPU memory (3980MB) before calling backward and ~8 gigs (8882MB) after, but this is suprising to me. Pytorch maintains the whole computational graph before calling backward, which it should be able to free while computing it. I have <code>2 281 048</code> parameters, where for each parameter a gradient is computed. I am not sure why I now occupy ~8 gigs of GPU memory?</p>
<p>If I am correct, this means:<br>
<code>(8 gigabytes) / (4 bytes) = 2 000 000 000 floats</code><br>
which are a lot of floats and more than I would expect.</p>
<p>If this is not expected, is there any way in which I could figure out what’s going on? For example, get the memory consumption per module?</p>
<p>EDIT: The reason for my investigation is that I get an OOM when scaling to a higher-dimensional dataset and I am currently looking into reducing the memory-requirement.</p>

albanD: <p>Hi,</p>
<p>How do you measure the gpu memory usage? You should be using <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.memory_allocated" rel="nofollow noopener"><code>torch.cuda.memory_allocated</code></a> to get the memory actually used by Tensors. Note that this number only counts Tensors, and not the memory required by the cuda driver at initialization.<br>
If you check via nvidia-smi, you will see a larger number because we have a special allocator for speed reasons that do not return the memory back to the driver when Tensors are freed.</p>

LeanderK: <p>Ok, thanks I have not looked at <code>torch.cuda.memory_allocated</code>.</p>
<p>This is off-topic, but do you by chance know what happens when cublas fails to allocate memory (or whether it uses pytorchs allocator)? I suspect that the caching allocator consumes all the memory and then cublas fails when i use it in <code>optimizer.step()</code> because the memory requirement after <code>backward.step()</code> is indeed very small.</p>

albanD: <p>I don’t think cublas manages any memory, it takes pointers to inputs. Do you think a of a particular function in there?</p>

LeanderK: <p>Hmm, I unfortunately didn’t save the error. It might have been magma? It was an assert error and not an pytorch OOM error. I will come back or open a new issue if I manage to reproduce it (or when torch.cuda.memory_allocated before optmizier.step fixes the issue).</p>
DataLoader with h5py

yikang-li: <p>Hi,</p>
<p>Recently, I encounter some problem with data loading. That is, when I utilizing DataLoader with more than 1 workers  to load HDF5 file with h5py, the retrieved data  usually appears random-like.  I have turn off the <code>shuffle</code> and use the default <code>sampler</code> and <code>collate_fn</code>.  The problem may come from <code>h5py</code>.</p>
<p>I wonder if there is any solution to such problem.</p>
<p>Best,<br>
Yikang</p>

QuantScientist: <p>Where is your code …?</p>

Jesse_Wang: <p>Same problem. <img src="https://discuss.pytorch.org/images/emoji/apple/cold_sweat.png?v=9" title=":cold_sweat:" class="emoji" alt=":cold_sweat:"></p>

platero: <p>See <a href="https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643/16" class="inline-onebox">DataLoader, when num_worker &gt;0, there is bug</a></p>
Is it recommended to convert a model into a traced or scripted one and then train it?

justanhduc: <p>My question is as the title. From what I read, there is some speed-up benefit when converting a model to static graph so I suppose training a traced or scripted model is faster. Do you have any idea about this?</p>

Michael_Suo: <p>The JIT can bring speed benefits for some patterns (especially if your code has lots of unfused pointwise operations). See <a href="https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/" rel="nofollow noopener">this post</a> as an example.</p>
Help in back propagating the loss

harshj94: <p>I am working on a model which looks like below fig:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/4/41d14f9ebc6a761c2d05c4d374dc28624542ee77.png" alt="Model%20Architecture" data-base62-sha1="9oftUctZ46uGN56ThHYTmqbGxAr" width="619" height="137"></p>
<p>Input image (I) is fed to the system which goes through some convolution layers and produces feature vector X. The feature vector X is then connected to two fully connected layers U and V. U and V are then connected to a series of deconvolution layers that generate some image. I have different losses and ground truths at the end of U and V.</p>
<p>The problem is that I am not able to understand how to perform the backpropagation in such a model. How would I be able to combine the two gradients when they backpropagate from U-&gt;X and V-&gt;X? The idea is to simply add the two gradients, but I am not able to understand how to achieve that.</p>
<p>This is my first question in PyTorch discussion. Let me know if I am unclear and if you need more details.</p>

spanev: <p>Hi <a class="mention" href="/u/harshj94">@harshj94</a>,</p>
<p>In PyTorch, gradients are accumulating by default.<br>
So if you simply have this before <code>optimizer.step</code>:</p>
<pre><code class="lang-auto">Loss1.backward()
Loss2.backward()
</code></pre>
<p>with no <code>optimizer.zero_grad()</code>  in between,<br>
you will end up with <code>x.grad</code> containing <code>dLoss1/dx + dLoss2/dx</code>.</p>
<p>Also when you create the optimzer you have to give it the params of the 3 models:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(list(model1.parameters()) + list(model2.parameters()) + list(model3.parameters()))
</code></pre>

harshj94: <p>This is working.<br>
Just that the Loss1 had to be written like<br>
Loss1.backward(retain_graph = True)</p>
Issue with tensorboard in pytorch

Subho: <p>I am trying to use tensorboard with pytorch (1.2.0+cu92). When calling <code>writer = SummaryWriter()</code>, I get the following error:<br>
<code>AttributeError: module 'tensorflow.io' has no attribute 'gfile'</code></p>
<p>How can I fix this? The following few lines would reproduce the error.</p>
<pre><code class="lang-auto">import torch
from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter()
</code></pre>
Torch.manual_seed is need to be set everytime

gouthamvgk: <p>I am using torch.manual_seed to fix the generator but it is giving different values at different runs unless I manually declare it before every run. One of the example run is,</p>
<blockquote>
<p>torch.manual_seed(10)<br>
d = torch.randn(4)<br>
e = torch.randn(4)<br>
print(d,e)<br>
torch.manual_seed(10)<br>
e = torch.randn(4)<br>
print(d,e)</p>
</blockquote>
<p>The output is,<br>
tensor([-0.6014, -1.0122, -0.3023, -1.2277]) tensor([ 0.9198, -0.3485, -0.8692, -0.9582])<br>
tensor([-0.6014, -1.0122, -0.3023, -1.2277]) tensor([-0.6014, -1.0122, -0.3023, -1.2277])</p>

ptrblck: <p>A seed in a random number generator makes sure to return the same sequence of random numbers.<br>
As you can see in your example, the second <code>e</code> gets the same values as <code>d</code> after resetting the seed.</p>
<p>I’m not sure I understand the issue correctly, but would you like to get the exactly same values for each <code>torch.randn</code> call? This would be an unusual use case.</p>

kirk86: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="56362">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>would you like to get the exactly same values for each <code>torch.randn</code> call?</p>
</blockquote>
</aside>
<p>I believe that was the intention inferring from the context of the thread. Additionally, if we replace the above example with numpy we get the same behaviour.</p>
<pre><code class="lang-auto">import random
import numpy as np

random.seed(2019)
np.random.seed(2019)
rng = np.random.RandomState(2019)

d = rng.randn(4)
e = rng.randn(4)
c = rng.randn(4)
print(d, e, c)
</code></pre>
<p>every call to <code>rand</code> will produce a different result but the running the above code in a script over and over again will yield three tensors with different random values due to the call to <code>randn</code> but those values will be consistent across different runs of the same script.</p>

gouthamvgk: <p>Yeah, my usecase was to produce the same random numbers at two different parts in a script. I had a wrong understanding of how manual_seed works. Thank You for clearing it out!</p>
Batched shuffling of feature vectors

apsod: <p>Hi, Is there an idiomatic way to do a batched shuffle of feature vectors?</p>
<p>I want to generate K negative samples per batch of data. i.e, given a tensor with shape (Batch x Features), I want to generate noise with the shape (K x Batch x Features), where the features in the noise are shuffled versions of the original data.  e.g. given a batch  like <code>[ [ a b c ] [ a d e ] [ b d e ] ]</code>, I want to generate two negative examples per datum like so:</p>
<pre><code class="lang-auto"> a b c     a c b | c b a
 a d e -&gt;  a e d | d e a 
 b d e     b d e | e b d
</code></pre>
<p>I’ve tried doing this using randperm, but it is prohibitively slow, especially on the GPU, and requires a lot of logistics and scaffolding.</p>

ptrblck: <p>I’m not sure, how exactly you are shuffling your input. Is there any specific logic or are you just randomly sampling shuffle indices?<br>
As far as I understand, <code>randperm</code> seems to work but is too slow?</p>

apsod: <p>I want to generate shuffles on the feature level over a batch.<br>
So given a datum like <code>[a b c]</code> i want to generate k shuffles of it, e.g. two shuffled versions: <code>[ [ a c b ] [c b a ] ]</code>, with this generalized over batched data.</p>
<p>The issue is that the call to randperm only generates one permutation, so I need to make (K x Batch) calls to randperm (which is <strong>very</strong> slow) and then index out the shuffles from the original data.</p>
<p>Now that i think about it, it wouldn’t be a problem if randperm could return multiple permutations.</p>

zh217: <p>I recently wanted to do the same thing. In case anyone is interested, the following is my solution:</p>
<pre><code class="lang-python">import torch


n_batch = 8
n_feat = 11

rand = torch.rand(n_batch, n_feat)
batch_rand_perm = rand.argsort(dim=1)
print(batch_rand_perm)
</code></pre>
<pre><code class="lang-auto">tensor([[ 1,  2,  4,  6,  0,  9,  3,  7,  8,  5, 10],
        [10,  6,  9,  2,  7,  4,  3,  5,  1,  0,  8],
        [ 8,  7,  4,  6,  0,  1,  9, 10,  2,  5,  3],
        [ 1,  3,  5,  4,  6,  8,  2,  9,  0,  7, 10],
        [ 3, 10,  7,  1,  4,  5,  0,  8,  9,  6,  2],
        [ 2, 10,  0,  1,  6,  8,  7,  9,  5,  3,  4],
        [ 5,  9,  8, 10,  0,  1,  6,  7,  4,  2,  3],
        [ 0,  9,  8,  4,  1,  3,  2,  7,  5, 10,  6]])
</code></pre>
Https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly

Carsten_Ditzel: <p>I am suspecting NaN values in my script so I would like to use the <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly" rel="nofollow noopener">anomaly detector</a> of pytorch. However, I am confused as to how exactly to use it.</p>
<p>My dataset class uses h5py so I am wondering where I have to include the context manager in in order for it to work.</p>

albanD: <p>Hi,</p>
<p>You need to wrap your forward and backward pass with it, so wherever you do that.<br>
If you want it for the whole script, you can also add <code>torch.autograd.set_detect_anomaly(True)</code> at the beginning of your script and it will stay on !</p>

Carsten_Ditzel: <p>I dont have an explicit backward pass, only the forward method of my model, or do you mean the backward invocation in my training loop?</p>
<p>Will the inclusion of this line at the beginning of my file then also apply to the crucial part in which I load in hdf5 files and convert them into tensors? Because this is where I suspect NaN values to occur…</p>

albanD: <p>I meant the  backward invocation in your training loop.</p>
<p>At the moment, the anomaly detection mode only detect nans that appear during the <code>.backward()</code> call as it is hard for the end user to poke around in what happens there.<br>
I won’t detect nans that appear outside of the backward invocation.</p>
<p>If nans appear in your own code, You can add some check at few places in your code to find where they appear. To check if a tensor contains nans, you can check: <code>if your_tensor.ne(your_tensor).any():</code></p>

Carsten_Ditzel: <p>since NaN != NaN, I get it =)</p>
<p>thanks a lot!</p>

Oli: <p>How expensive is the detect_anomaly to run during training? Unsignificant right :D?</p>

ptrblck: <p>It might be expensive, so use it for debugging only.</p>

albanD: <p>Actually very significant <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> It does a lot of bookkeeping and checks the return values of each low level function for nans.<br>
As mentioned above, you should only use it for debugging.</p>

Oli: <p>Great, thanks. I’ll keep to checking the loss then</p>
<p><code>assert not torch.isnan(loss), 'nan err msg'</code></p>
Input normalization inside the model

mkocabas: <p>I tried to normalize the input during the forward pass of the model doing this:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        mean = torch.as_tensor([0.485, 0.456, 0.406])[None, :, None, None]
        std = torch.as_tensor([0.229, 0.224, 0.225])[None, :, None, None]
        self.register_buffer('mean', mean)
        self.register_buffer('std', std)
        ...


    def forward(self, inputs):
        # Input size [batch, channel, width, height]
        # Normalize inside the model
        inputs = inputs.sub(self.mean).div(self.std)
        ...
        return output
</code></pre>
<p>During training everything is fine and working but when I switch to <code>eval()</code> mode, model starts to give random outputs. Disabling <code>eval()</code> helps to get meaningful outputs during validation, but I need <code>eval()</code> mode since I use dropout and batchnorm in the model. Any idea what causes this weird behavior?</p>

ptrblck: <p>Do your training and validation images have the same distributions or are you processing them differently?<br>
Are you seeing the bad results also after calling <code>eval()</code> and using your training images?<br>
If so, this problem might be unrelated to the processing inside the model, but might come from a small batch size and thus skewed running estimates in the batch norm layers.</p>

mkocabas: <p>They have the same distribution and processing is identical for both sets.</p>
<p>When I disable normalization inside the model and perform it using torchvision library things get back to normal. I suppose the weird performance drop is purely from normalization inside the model.</p>
About dataLoader

CHENCHAO0526: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/2/726a7ff0263b92d98ba74bed398915626732ab17.png" data-download-href="https://discuss.pytorch.org/uploads/default/726a7ff0263b92d98ba74bed398915626732ab17" title="Screenshot%20from%202019-09-19%2016-08-41"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/726a7ff0263b92d98ba74bed398915626732ab17_2_690x388.png" alt="Screenshot%20from%202019-09-19%2016-08-41" data-base62-sha1="gkaBAEOdMjDzu9iFKzsrEqxi6Gj" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/726a7ff0263b92d98ba74bed398915626732ab17_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/726a7ff0263b92d98ba74bed398915626732ab17_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/726a7ff0263b92d98ba74bed398915626732ab17_2_1380x776.png 2x" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-09-19%2016-08-41</span><span class="informations">1920×1080 314 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I got two Dataloader, in the for loop of dataloader, one function in dataset go twice, cause some bugs , I really don’t know why. what’s more weird is that , where I delete valloader , the codes goes right。</p>

ptrblck: <p>What kind of error are you seeing?<br>
Could you post the error message by wrapping it in three backticks ``` and update the topic title, please? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to Multiply 2 feature maps from 2 different cnn layers

MyTechway: <p>Hi,</p>
<p>I am trying to get a feature map from VGG11 conv5 layer as (14 X 14 X 512) and also getting another feature map from another different layer as (1 X 1 X 512). I want to multiply these maps and get the output as (14 X 14 X 512) and pass it to FC layer. Is there are any layer that can do multiplication like this?</p>

InnovArul: <p>just <code>torch.mul(x1, x2)</code> will do, where <code>x1, x2</code> are those two features.</p>

Jarvis: <p>Does it support backprop?</p>

InnovArul: <p>yes it supports backprop.</p>
How to set plt figure's title?

wonchulSon: <pre><code class="lang-auto">def visfeaturemap(feature_maps, filename='default.png', save=False):
    # plot all 64 maps in an 8x8 squares
    square = 8
    ix = 1
    for _ in range(square):
        for _ in range(square):
            # specify subplot and turn of axis
            ax = plt.subplot(square, square, ix)
            ax.set_xticks([])
            ax.set_yticks([])
            # plot filter channel in grayscale
            plt.title(filename)
            plt.imshow(feature_maps[0, ix - 1, :, :].cpu().numpy(), cmap='gray')
            ix += 1

    plt.title(filename)
    plt.show()

    if save:
        plt.savefig(filename)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fad98f988895948297d00476a7f84f97b39a83c4.png" alt="image" data-base62-sha1="zN7umxRwHHgfYZgsxB8Coo7GhTe" width="440" height="298"></p>
<p>I want just one total figure title.<br>
Plz help me^^.<br>
Thank you.</p>

Swarchal: <p>I think you might be looking for <code>plt.suptitle()</code></p>

radeeplearning: <p>If you just want one title,why don’t you remove the plt.title(filename) within your loops?</p>
How to Freeze weights of pretrained network while training 2 nets

Or_Rimoch: <p>Hi!</p>
<p>I have 2 networks: Encoder (already trained) + Decoder.<br>
I’d like to train both nets End2End.<br>
But I’d like to keep The Decoder’s parameters “freeze”  - no updates while training!</p>
<ol>
<li>
<p>I’ve tried to set all the parameters of the Decoder with:<br>
“require_grads = False” + “decoder.eval()” before training.</p>
</li>
<li>
<p>Also, I optimize over the Encoder’s parameters: so I’ve put them inside my optimizer object.<br>
In the train / test process I set : “encoder.train()” &amp; “encoder.eval()” respectively.</p>
</li>
</ol>
<p>When I compare the Decoder’s parameters before &amp; after training I found out that values were changed.</p>
<p>Please help me to understand what might went wrong.</p>
<p>Thanks,<br>
Or Rimoch</p>

spanev: <p>Hi <a class="mention" href="/u/or_rimoch">@Or_Rimoch</a>,</p>
<aside class="quote no-group" data-username="Or_Rimoch" data-post="1" data-topic="56335">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/or_rimoch/48/2744_2.png" class="avatar"> Or_Rimoch:</div>
<blockquote>
<p>I have 2 networks: Encoder (already trained) + Decoder.</p>
</blockquote>
</aside>
<p>I assume that you mean:</p>
<blockquote>
<p>Encoder + Decoder (already trained).</p>
</blockquote>
<p>If you set <code>Decoder</code>’ parameters’ <code>requires_grad</code> to <code>False</code> and you don’t pass them to the optimizer, there shouldn’t be any way for them to get modified.<br>
Do you have a link to the code or a simple repro?</p>
Modify the feature(activation) maps

zeke_wang: <p>Is it possible to modify the activation maps when our model is in evaluation mode? e.g. set value of certain activation map to 0 but others keep unchanged. I only know we can indirectly turn off an activation by setting the corresponding weight matrix to zero, but if we want to make detailed modification on the activation map, it is impossible in this way.</p>

spanev: <p>Hi <a class="mention" href="/u/zeke_wang">@zeke_wang</a><br>
You should be able to do it with <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.register_forward_pre_hook" rel="nofollow noopener">forward pre hooks</a>.</p>
<p>Let’s say you have the following model:</p>
<pre><code class="lang-auto">net = nn.Sequential(nn.Linear(16,4),
                    nn.Linear(4, 16))
</code></pre>
<p>If you want to modify the activation maps that goes to the second Linear, and only in eval mode, your hook will look like this:</p>
<pre><code class="lang-auto">def hook_fn(module, input)
    if not module.training:
        for i in input:
            i[mask] = 0
</code></pre>
<p>and to register it:</p>
<pre><code class="lang-auto">hook = net[1].register_forward_pre_hook(hook_fn)
</code></pre>

zeke_wang: <p>Thanks for your help. It is so useful to get familiar with hook in pytorch. I only treated it as a intermediate output reader in the past.<br>
I consider the complete version will be:</p>
<pre><code class="lang-auto">def modify_input(idx, mask):
    def hook(module, input):
        if not module.training:
            input[idx] = mask
    return hook
hook = net[1].register_forward_pre_hook(modify_input(idx, mask))
</code></pre>
Helping regression task with a discriminator

watchdog: <p>Hello everyone. I would like to add a discriminator as an extra loss, in order to enhance my current network that is trained with a regression task. Currently i have 4 losses(2 L1 smooth, 1 MSE and one SSIM). Though, until now I haven’t seen any progress. Has anyone such an experience or could help me with any idea? I use weight=0.05 for the discriminator loss.</p>