akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Good pretrained pytorch background removal model

kq3rHCt9gK: <aside class="quote quote-modified" data-post="1" data-topic="38024">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xxaemaethxx/48/9470_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/removing-background-of-image-pytorch-python/38024">Removing background of image (pytorch, python)</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Link to the jupyter notebook containing my code is here: <a href="https://github.com/alpizano/machine-learning-neural-network/blob/master/image_processing.ipynb" rel="noopener nofollow ugc">https://github.com/alpizano/machine-learning-neural-network/blob/master/image_processing.ipynb</a> 
So I am starting a DL neural network project and my Professor assigned that we FIRST take an image in python (+pytorch) and convert 3d matrix (rows, cols, RGB) to 2d matrix (rows,cols) by averaging (R+G+B)/3 channels to turn into grayscale, THEN to remove the background of the image by picking an intensity of a pixel, say 200, and looping/iterat…
  </blockquote>
</aside>

<p>It seems that robust background removal was still challenging.</p>
<p>But it seems the service offered by this website is OK. Is there a pretrained background removal model that has a similar performance as this website.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.remove.bg">
  <header class="source">
      <img src="https://www.remove.bg/favicon-32x32.png?v=YAXaAv7pao" class="site-icon" width="" height="">

      <a href="https://www.remove.bg" target="_blank" rel="noopener nofollow ugc">remove.bg</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://sb.kaleidousercontent.com/67418/600x315/48eb30a742/emilia-og-image.jpg" class="thumbnail">

<h3><a href="https://www.remove.bg" target="_blank" rel="noopener nofollow ugc">Remove Background from Image for Free – remove.bg</a></h3>

  <p>Remove image backgrounds automatically in 5 seconds with just one click. Don't spend hours manually picking pixels. Upload your photo now &amp; see the magic.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Custom Convolution

masterpradip: <p>I want to create a custom convolution operation by overriding, if possible, the torch.nn.functional.conv2d(). For example, I want to “<strong>add</strong>” the network weights instead of “<strong>multiplying</strong>” them, <strong>just for example</strong>.<br>
I don’t want to write the forward propagation from scratch as that won’t allow me to use the backward(). In that case I’d have to calculate the gradients and all those stuffs by myself.</p>

urw7rs: <p>you can write a new nn.module like this</p>
<pre><code class="lang-auto">class CustomConv(nn.module):
    def __init__(self):
        weights = nn.Parameter(torch.randn(3, 1))
    # initialize module

    def forward(self, input):
    # custom convolution here
</code></pre>
<p>and use it like this</p>
<pre><code class="lang-auto">conv = CustomConv()
features = conv(input)
</code></pre>
<p>and it will work with .backward()<br>
but it will work like torch.nn.conv2d rather than torch.nn.functional.conv2d()<br>
and you need to use nn.Parameter() for weights</p>

masterpradip: <p>First of all thanks for your response.<br>
If I do it the way you said, where should I save all the filters and the activation maps so that it could be used in backward()?</p>

urw7rs: <p>you save the filters like this</p>
<pre><code class="lang-auto">class CustomConv(nn.module):
    def __init__(self):
        filters = nn.Parameter(# filter tensor)
</code></pre>
<p>so autograd registers the filter tensor as a parameter that needs to be optimized.</p>
<p>And you should return the activation map in the forward method like this</p>
<pre><code class="lang-auto">    def forward(self, input):
    # custom convolutions
    return activation_map
</code></pre>
Pytorch gradient accumulation

Kalyan_Katikapalli: <p>This article <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255" rel="nofollow noopener">https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255</a> suggest to do gradient accumulation in the following way</p>
<pre><code class="lang-auto">model.zero_grad()                                   # Reset gradients tensors
for i, (inputs, labels) in enumerate(training_set):
    predictions = model(inputs)                     # Forward pass
    loss = loss_function(predictions, labels)       # Compute loss function
    loss = loss / accumulation_steps                # Normalize our loss (if averaged)
    loss.backward()                                 # Backward pass
    if (i+1) % accumulation_steps == 0:             # Wait for several backward steps
        optimizer.step()                            # Now we can do an optimizer step
        model.zero_grad()                           # Reset gradients tensors
</code></pre>
<p>I have two doubts.</p>
<ol>
<li>What is the need of this step , <strong>loss=loss/accumulation_steps</strong>?</li>
<li>Suppose, there are 960 training instances and the memory can accommodate a maximum of 64.  So, number of batches =15. So, if i choose accumulation_steps=2, the parameters are not updated for the last batch. Doesn’t  it affect the performance of the model?</li>
</ol>

urw7rs: <ol>
<li>
<p>loss gradients are added(accumulated) by loss.backward() and loss / accumulation_steps divides the loss in advance to average the accumulated loss gradients.</p>
</li>
<li>
<p>First, because batches that aren’t accumulated are wasted, you should make sure batches are divisible by accumulation_steps. Second, the last batch actually gets accumulated since the first batch gets accumulated. And I think (i + 1) should be I because of this.</p>
</li>
</ol>
How to avoid for loops in pytorch?

tomb88: <p>I’m working on a task where the input is a sequence of images (5 in my case) and the output should be a set of sentences, one per image. I’m using an encoder-decoder architecture where I would like to use the sentences I already generated as input for generating the next one but this requires me to loop for each instance (for loop) inside the forward pass.<br>
As already discussed here, each call inside the loop duplicates the computation graph therefore increasing the memory utilization.<br>
Is there any way to get around the need to use the loop? Or alternatively, is there an explicit way to avoid duplicating the graph?</p>
<p>Thanks</p>

urw7rs: <p>you could use .detach() to avoid duplicating the graph</p>
TypeError: __init__() missing 1 required positional argument: 'size'

ywr461903285: <p>could anyone help me solve this problem,<br>
code below:<br>
import torch<br>
from torch import nn, optim<br>
from torch.autograd import Variable<br>
from torch.utils.data import DataLoader<br>
import torchvision.models as models<br>
from torchvision import datasets, transforms<br>
class CNN(nn.Module):<br>
def <strong>init</strong>(self,size):</p>
<pre><code>    super(CNN, self).__init__()
    resnet = models.resnet152(pretrained=True)
    modules = list(resnet.children())[:-1]      # delete the last fc layer.
    self.resnet = nn.Sequential(*modules)
    self.linear = nn.Linear(resnet.fc.in_features, size)
    self.bn = nn.BatchNorm1d(size, momentum=0.01)
            
def forward(self, images):
    """Extract feature vectors from input images."""
    with torch.no_grad():
        features = self.resnet(images)
    features = features.reshape(features.size(0), -1)
    features = self.bn(self.linear(features))
    return features
</code></pre>
<p>device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)</p>
<p>NET = CNN().cuda()</p>
<p>criterion = nn.MSELoss()<br>
optimizer = torch.optim.SGD(NET.parameters(), lr=0.01)</p>
<p>for epoch in range (10):<br>
for img, rst1, rst2 in train_dataloader:</p>
<pre><code>    img=img.to(device)
    rst1 = rst1.to(device)
    rst2 = rst2.to(device)
    outputs = NET(img)
    loss = criterion(outputs, (rst1,rst2))
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()</code></pre>

Sunshine352: <aside class="quote no-group" data-username="ywr461903285" data-post="1" data-topic="55940">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/48db29/48.png" class="avatar"> ywr461903285:</div>
<blockquote>
<p>NET = CNN().cuda()</p>
</blockquote>
</aside>
<p>To instantiate CNN, you should include parameter-‘size’， that is,<br>
NET = CNN(size=xxx).cuda()</p>

ywr461903285: <p>thank you very much ! it help me o lot</p>
Can't get lower loss

smu226: <p>Hello! I have a NN with a single linear layer, no activation function, just a matrix multiplication and bias (I need to do some tests and I came across this issue). So the input is 4D and the output is 2D and the relation between them is like this: [x,y,x+vx,y+vy] -&gt; [x+2vx,y+2vy] so for example [10,20,40,100] -&gt; [70,180]. My training data has no noise i.e. ideally, for a given input you should get the exactly right output and it can be shown that the matrix that the NN should learn, in the perfect case is this:<br>
[[-1,  0,  2, 0],<br>
[ 0, -1, 0,  2.]]<br>
with a bias of zero for both output nodes. My training data has 512 examples and after the training, this is the matrix learnt by the NN:<br>
[[-9.9997e-01,  2.6156e-05,  2.0000e+00, -2.6044e-05],<br>
[ 2.6031e-05, -9.9996e-01, -2.5983e-05,  2.0000e+00]]<br>
and the bias is:<br>
[0.0003, 0.0003].<br>
As you can see the result is very close to the right answer, but I am not sure why it doesn’t go lower than that. Given that I have a single linear layer, it should be just one minima, so the algorithm shouldn’t get stuck and given that I have no noise, the loss should simply go to zero, but it seems to get stuck somewhere around 1e-5. I am using adam optimizer. I do the training by starting with a LR of 1e-3 and I train until there is no significant improvement over several epoch, then I go to 1e-4 and so on until around 1e-8. No other tricks beside this, just the linear layer. Can someone tell me if this is normal? I am not sure what prevents the NN to get zero loss. Thank you!</p>

ptrblck: <p>Since the <code>weight</code> and <code>bias</code> are almost perfectly fit, the loss value should be quite small at the end of the training. If you lower the learning rate further (to <code>1e-8</code>), the parameter updates will be very small and might even get lost due to floating point precision.</p>
<p>You could try to increase the learning rate a bit or skip some reductions and see, if that would lower the loss. However, since your model is almost perfectly fit, I’m not sure if it’s worth a lot of effort to get the “perfect” numbers.</p>
How to create inverse function of neural net?

dejanbatanjac: <p>Neural net (NN) is a function, right.</p>
<p>The realm what it do may be converting the image into another image (segmentation tasks).<br>
NN:  I =&gt; I</p>
<p>Can you provide some hints or papers how based on NN we can create inverse neural net INN?</p>

KFrank: <p>Hi Dejan!</p>
<aside class="quote no-group" data-username="dejanbatanjac" data-post="1" data-topic="55895" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/77aa72/48.png" class="avatar"> dejanbatanjac:</div>
<blockquote>
<p>Neural net (NN) is a function, right.</p>
</blockquote>
</aside>
<p>Yes, but not necessarily an invertible function.</p>
<aside class="quote no-group">
<blockquote>
<p>Can you provide some hints or papers how based on NN we can create inverse neural net INN?</p>
</blockquote>
</aside>
<p>In general, you cannot invert a neural network.  And this is true<br>
not just for unusual edge cases – most typical neural networks<br>
won’t be invertible.</p>
<p>Consider (among others) these two points:</p>
<p>Rectified linear units (f (x) = max (0, x)) are not invertible.</p>
<p>Any linear layer with fewer outputs than inputs in not invertible.</p>
<p>Best.</p>
<p>K. Frank</p>

spanev: <p>Hi <a class="mention" href="/u/dejanbatanjac">@dejanbatanjac</a>,</p>
<p>You can take a look at this paper <a href="https://openreview.net/pdf?id=HJsjkMb0Z" rel="nofollow noopener">Deep Invertible Networks</a> and its (PyTorch) <a href="https://github.com/jhjacobsen/pytorch-i-revnet" rel="nofollow noopener">official implementation</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
<p>It propose two fully invertible architectures (section 3.1), the first one being only injective and the second one bijective. The two are actually made of two sub-models: a mapping Φ and it’s pseudo-inverse Φ-1, the latter being built with inverse and pseudo-inverse blocks/layers from Φ.</p>
<p>The authors focus classification, so I’m not sure how (good) you could reuse this work for segmentation task</p>
How Tensor gets the index of specific elements?

Audrey: <p>I have 2 Tensors named  <strong>x</strong>  and  <strong>list</strong>  and their definitions are below:</p>
<pre><code class="lang-auto">x = torch.tensor(3)
list = torch.tensor([1,2,3,4,5])
</code></pre>
<p>Now I want to get the index of element  <strong>x</strong>  from  <strong>list</strong> . The expected output is an Integer:</p>
<pre><code class="lang-auto">2
</code></pre>
<p>How can I do in an easy way?</p>

spanev: <p>Hi <a class="mention" href="/u/audrey">@Audrey</a>,</p>
<p>You can do this:</p>
<pre><code class="lang-auto">(list == x).nonzero()
</code></pre>

Audrey: <p>Thanks for your help, your answer works! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Does parameter adjustment scheme from 'Accurate, Large Minibatch SGD' work with PyTorch optim.SGD?

Andrew_Paint: <p>I want to expand a 4-GPU training scheme to 8 GPUs. I’m wondering if I can use the adjustment rules from the paper “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”. The scheme proposed in the paper is for Distributed synchronous SGD. Even though the optim.SGD in PyTorch is not distributed version, but I’m assuming it is a synchronous one when it comes to multi-GPU training. I’m not sure if it’s the right assumption.</p>
Lu_unpack source code

Kiuhnm_Mnhuik: <p>Is lu_unpack (<a href="https://pytorch.org/docs/stable/_modules/torch/functional.html#lu_unpack" rel="nofollow noopener">src</a>) as efficient as possible? For instance let’s have a look at this part:</p>
<pre><code class="lang-auto">P = torch.eye(sz, device=LU_data.device, dtype=LU_data.dtype)
final_order = list(range(sz))
for k, j, in enumerate(LU_pivots_zero_idx):
    final_order[k], final_order[j] = final_order[j], final_order[k]
P = P.index_select(1, torch.as_tensor(final_order, device=LU_pivots.device))
</code></pre>
<p>Why isn’t <code>final_order</code> allocated on <code>LU_pivots.device</code> from the start? Doesn’t <code>final_order[j]</code> forces a sync if <code>LU_pivots</code> is on a GPU?</p>

Kiuhnm_Mnhuik: <p>I can’t edit the text of my question anymore. Maybe I sounded too critical. I was just wondering whether there was a particular reason the author chose to use a Python list for <code>final_order</code> instead of a tensor allocated on the same device as <code>LU_pivots</code>.<br>
It’s my understanding that most(?) of PyTorch commands are asynchronous, that is, the command returns right away and doesn’t wait for the operation to complete. If we use the computed value on the Python-side, though, the Python code stops and waits for the value to be available before proceeding to the next statement.<br>
So I think that if <code>final_order</code> were a PyTorch tensor then the whole piece of code above would be executed asynchronously without any slowdown. Also, the last line wouldn’t need to transfer data between devices.</p>
Avoiding allocations and memory transfers

Kiuhnm_Mnhuik: <p><strong>Q1:</strong><br>
Let’s say <code>D</code> is a matrix with k <em>very</em> long columns and I want to compute <code>D + d.ger(x)</code>. Also, <em>I don’t need to backpropagate through it</em>. I think the best approach is a simple for loop:</p>
<pre><code class="lang-auto">for i in range(k):
    D[:,i] = x[i]*d
</code></pre>
<p>Does <code>x[i]*d</code> create a temporary tensor?<br>
Maybe I should use</p>
<pre><code class="lang-auto">for i in range(k):
    D[:,i].add_(d, alpha=x[i])
</code></pre>
<p>Is this the most memory-efficient version?</p>
<p><strong>Q2:</strong><br>
What happens when I use python scalars as in <code>some_tensor * 2.4</code>? Is 2.4 moved from the cpu to the gpu as usual? Should I pre-initialize known constants (<code>T.scalar_tensor(2.4)</code>) as much as possible to avoid slowdowns? Even small integers such as 2 and 3? Or maybe the transfer is asynchronous and thus irrelevant as long as the GPU has still some work to complete?</p>
<p><strong>Q3:</strong><br>
Is there a way to make absolutely sure a piece of code is not creating temporary buffers or doing cpu-&gt;gpu transfers? Maybe some context managers which cause the code to throw?</p>

Kiuhnm_Mnhuik: <p><strong>Q1:</strong> The function <code>torch.addr_</code> is what I was looking for because it adds the outer product of two vectors completely in-place without allocating any extra memory.<br>
As a side note, since the matrices are in row-major order in PyTorch, accessing the columns the way I do in my question above is slower than accessing the rows.</p>
Torch.utils.bottleneck says that `to` takes the most GPU time. What does this mean?

noahtren: <p>Hi all,</p>
<p>I’m trying to optimize my network and have been using <code>torch.utils.bottleneck</code>. I’m now at the point in developing this that the profiler says my largest bottleneck is <code>to</code>.</p>
<p>I haven’t been able to find any documentation for this. What does this mean, exactly? And is this a good or bad sign in terms of optimization/GPU utilization?</p>

karmus89: <p>Hi <a class="mention" href="/u/noahtren">@noahtren</a>!</p>
<p>The <code>to</code> seems to me to point towards moving tensors from CPU to GPU, i.e. <code>a_cpu_tensor.to('cuda')</code>. Are there a lot of device-related tensor instructions back and forth in your code?</p>

ptrblck: <p>Additionally to <a class="mention" href="/u/karmus89">@karmus89</a> answer:<br>
your code might run asynchronous CUDA operations and the <code>.to</code> operation might create a synchronization point, so that the actual kernel times will be accumulated in the <code>.to</code> call.</p>

noahtren: <p><a class="mention" href="/u/karmus89">@karmus89</a> my code doesn’t have any device-specific instructions. All relevant tensors are stored on a single CUDA device.</p>
<p>If there is a slow-down in my code, it’s likely due to either using a lot of indexing, or calling <code>clone()</code>. Any chance cloning could show up as <code>to</code> in the profiler?</p>
<p>Thanks.</p>
<p>Note: I’m not explicitly calling <code>to</code> anywhere in my code</p>
Distributed training prints log messages twice

kirk86: <p>How can I prevent my log messages to not get printed multiple times when I use distributed training?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/4/40a473ce455cbf84b6a026fd493911ac39cb250c.png" alt="image" data-base62-sha1="9dQTwZedV1qskaljP1EzZ2d4M0s" width="450" height="120"><br>
Any ideas how to resolve this or where to took at?</p>
<p>Also, I keep getting an error every time I set <code>OMP_NUM_THREADS &gt; 1</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/b186211275a78d4339aa67fd40450c112c71677d.png" data-download-href="https://discuss.pytorch.org/uploads/default/b186211275a78d4339aa67fd40450c112c71677d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b186211275a78d4339aa67fd40450c112c71677d_2_690x244.png" alt="image" data-base62-sha1="pkrSJd6d8csfpxmLB7PaknD7ThH" width="690" height="244" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b186211275a78d4339aa67fd40450c112c71677d_2_690x244.png, https://discuss.pytorch.org/uploads/default/optimized/2X/b/b186211275a78d4339aa67fd40450c112c71677d_2_1035x366.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/b/b186211275a78d4339aa67fd40450c112c71677d.png 2x" data-dominant-color="242724"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1147×407 134 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any thoughts into what might have gone wrong here?</p>

kirk86: <p>I managed to solve the error I was getting when using <code>OMP_NUM_THREADS &gt; 1</code><br>
Basically looking in my script I had to add <code>init_method="env://"</code> in the call to the <code>process_group</code></p>
<p><code>torch.distributed.init_process_group(backend='nccl', init_method='env://')</code></p>
<p>The other thing that I was missing is that when calling the <code>launch</code> utility you have to pass a random port otherwise I would get the above error</p>
<p><code>python -m torch.distributed.launch --nproc_per_node=number of gpus --master_port=some random high number port main.py</code></p>
Any way to export Densenet with memoryefficient=true to onnx?

Tomas_Batrla: <p>Hi,<br>
I am trying to export Densenet121 to ONNX format, but I am stuck with<br>
the process, getting exception :</p>
<p><strong>ONNX export failed: Couldn’t export Python operator CheckpointFunction</strong></p>
<p>Is there any way to go around the exception?<br>
My code for export is:</p>
<pre><code class="lang-auto">
 
 def export_model(model):
     sample_batch_size = 1
     channel = 3
     height = 224
     width = 224
     dummy_input = torch.randn(sample_batch_size, channel, height, width)
 
     torch.onnx.export(model, dummy_input, "onnx_model_name.onnx", input_names=['input'], output_names=['output'])
 
 model = torch.hub.load('pytorch/vision', 'densenet121', pretrained=True, memory_efficient=True)
 export_model(model)
</code></pre>

ptrblck: <p>The <code>memory_efficient=True</code> option uses internally <code>torch.utils.checkpoint</code> to trade compute for memory.<br>
This operator is not defined in ONNX, which raises this error.<br>
However, if you don’t necessarily need checkpointing, you could just set this argument to False and the export should work.</p>
Cannot pass custom weights to F.instance_norm

Ierezell: <p>Hi,</p>
<p>When I use F.instance_norm with batch_size = 1 everything runs fine.<br>
But with higher batch size I got some size errors.</p>
<p>Each element of my batch is a person so I want to gives weights to normalize per persons per channels e.g : batch of 3 persons 64 channels and wathever 2D size =&gt; weights of size 3,64.</p>
<p>If I get it right instanceNorm is perfect for that but I cannot pass other thing that 64 element to F.instance_norm(weights=…)</p>
<p>I tried (3,64), 192 (3*64), but it only accept 64 elements (which is wrong because I want parameters per channels per batch)</p>
<p>Does anyone know how to do it ?</p>
<p>Thanks in advance,<br>
Pierre</p>

SimonW: <p>Instance norm does not use per-element weight. It doesn’t make sense as a network layer. You can just do the affine transform yourself after instance norm…</p>

Ierezell: <p>Thanks for the answer,</p>
<p>I misunderstood some concept but after reading again it’s more clear <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>You’re solution is working great, I’m just doing the “Adaptative”  layer on my own after a classic instance norm.</p>
<p>Thanks again</p>

Ierezell: <p>For those who can have the same question :</p>
<p>I’m just doing a classical python instance norm and then I multiply by my own weights to get the adaptive behaviour</p>
<p>out = nn.InstanceNorm(in)<br>
weighted_out = out * w_custom +b_custom</p>
Accessing the computation graph

Rishabh_Dahale: <p>Is there any way to access the computation graph created by Pytorch during forward pass and shift that to CPU from GPU? I want to free the GPU so that I can load the next part of the model on it and continue the training.</p>

spanev: <p>Hi <a class="mention" href="/u/rishabh_dahale">@Rishabh_Dahale</a>,</p>
<p>I don’t think it is possible.</p>
<p>But if your problem is that the computation graph is too big to fit in your GPU memory, gradient checkpointing might be able to help you here.</p>
<p>Please take  a look at the <a href="https://pytorch.org/docs/stable/checkpoint.html" rel="nofollow noopener">official doc</a> and at <a href="https://github.com/prigoyal/pytorch_memonger/blob/master/tutorial/Checkpointing_for_PyTorch_models.ipynb" rel="nofollow noopener">this tutorial</a>. Even though it is using the <code>Variable</code> API, the checkpointing is still relevant.</p>
Install pytorch problem :libtorch.so.1: undefined symbol:

qiuzhongwei-USTB: <p>Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)<br>
[GCC 7.3.0] on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/var/storage/shared/titanv/sys/jobs/application_1545092406025_20974/anaconda3/envs/python36/lib/python3.6/site-packages/torch/<strong>init</strong>.py”, line 84, in <br>
from torch._C import *<br>
ImportError: /var/storage/shared/titanv/sys/jobs/application_1545092406025_20974/anaconda3/envs/python36/lib/python3.6/site-packages/torch/lib/libtorch.so.1: undefined symbol: _ZNK2at11TypeDefault19tensorWithAllocatorEN3c108ArrayRefIlEEPNS1_9AllocatorE</p>
</blockquote>
</blockquote>
</blockquote>
<p>##########################################################################<br>
I install pytorch in a new conda env by conda. I meet this problem when I import torch in python, as above. How can i solve this problem? Thanks</p>

Alexios: <p>They recommend using pip to install it instead of conda and even if you’re in a conda environment. Might be related to that. I was trying to understand why that’s recommendation when I hit your question.</p>
<p>I’m also wondering if it’s normal for <strong>libtorch.so</strong> to be about 1.3 GB? If so, why?</p>
Assertion `jcp.ur_w * (jcp.nb_oc_blocking + 1) <= num_avail_regs' failed

Kenzo: <p>Hello everyone,</p>
<p>I’ve updated my pyTorch from 1.0.0 to the newest version and now when I want to execute my code I’m facing this error. Any idea how can I solve the error??</p>
<blockquote>
<p>**python3: /pytorch/third_party/ideep/mkl-dnn/src/cpu/jit_avx2_conv_kernel_f32.cpp:567: static mkldnn::impl::status_t mkldnn::impl::cpu::jit_avx2_conv_fwd_kernel_f32::init_conf(mkldnn::impl::cpu::jit_conv_conf_t&amp;, const convolution_desc_t&amp;, const mkldnn::impl::memory_desc_wrapper&amp;, const mkldnn::impl::memory_desc_wrapper&amp;, const mkldnn::impl::memory_desc_wrapper&amp;, const primitive_attr_t&amp;): Assertion `jcp.ur_w * (jcp.nb_oc_blocking + 1) &lt;= num_avail_regs’ failed.<br>
Aborted (core dumped)<br>
**</p>
</blockquote>
About visualizing data

Omar2017: <p>Hi every one<br>
I need help to visualize a frames in data set when applying each transformation operation?</p>
<pre><code class="lang-auto">    mean=[0.485, 0.456, 0.406]
    std=[0.229, 0.224, 0.225]
    normalize = Normalize(mean=mean, std=std)
    spatial_transform = transforms.Compose([transforms.RandomRotation(20),
                                            transforms.RandomResizedCrop(224),
                                            transforms.RandomHorizontalFlip(),
                                            transforms.ColorJitter(hue=.05, saturation=.05),
                                            transforms.ToTensor(),
                                            transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])])


  
    vidSeqTrain = makeDataset(trainDataset, trainLabels, spatial_transform=spatial_transform, seqLen=seqLen)
    
        
    vidSeqTest = makeDataset(data2, label2, seqLen=seqLen, spatial_transform= spatial_transform)
    
 
    testLoader = torch.utils.data.DataLoader(vidSeqTest, batch_size=trainBatchSize,
                            shuffle=True, num_workers=int(numWorkers/2), pin_memory=True)
    
    # torch iterator to give data in batches of specified size
    trainLoader = torch.utils.data.DataLoader(vidSeqTrain, batch_size=trainBatchSize,
                            shuffle=True, num_workers=numWorkers, pin_memory=True, drop_last=True)
</code></pre>

ptrblck: <p>If you want to visualize the <code>PIL.Image</code> after each transformation, you could create a custom transformation and call e.g. <code>img.show()</code> (or use any other library to plot the image):</p>
<pre><code class="lang-python">class Visualize(object):
    def __call__(self, img):
        img.show()
        return img
        
spatial_transform = transforms.Compose([transforms.RandomRotation(20),
                                        Visualize(),
                                        transforms.RandomResizedCrop(224),
                                        Visualize(),
                                        transforms.RandomHorizontalFlip(),
                                        Visualize(),
                                        transforms.ColorJitter(hue=.05, saturation=.05),
                                        Visualize(),
                                        transforms.ToTensor(),
                                        transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])])


to_pil_image = transforms.ToPILImage()
x = torch.randn(3, 256, 256)
img = to_pil_image(x)
out = spatial_transform(img)
</code></pre>
What is the difference between these network architectures?

vainaijr: <pre><code class="lang-auto">class A(nn.Module):
  def __init__(self):
    super().__init__()
    self.conv = nn.Conv2d(in_channels=256, out_channels=32, 
                      kernel_size=9, stride=2, padding=0)
  def forward(self, x):
    return self.conv(x)
</code></pre>
<pre><code class="lang-auto">class B(nn.Module):
  def __init__(self):
    super().__init__()
    self.capsules = nn.ModuleList([
            nn.Conv2d(in_channels=256, out_channels=32, 
                      kernel_size=9, stride=2, padding=0)
            for _ in range(8)])    
  def forward(self, x):
    ...
</code></pre>
<pre><code class="lang-auto">class C(nn.Module):
  def __init__(self):
    super().__init__()
    self.capsules = nn.ModuleList([A() for _ in range(8)])    
  def forward(self, x):
     ...
</code></pre>

spanev: <p>Hi <a class="mention" href="/u/vainaijr">@vainaijr</a>,</p>
<p>The only difference is when you try to access the convolutions.</p>
<p>Let’s say you want to access the first conv.<br>
In C case you have to do:</p>
<pre><code class="lang-auto">c.capsules[0].conv
</code></pre>
<p>and in B case you can simply do:</p>
<pre><code class="lang-auto">b.capsules[0]
</code></pre>
<p>Except from that, running them would give you exactly the same result. A is just a wrapper forwarding <code>x</code> to Conv2d.</p>
Tutorial for nn.Transformer module

zhangguanheng66: <p>PyTorch team just releases a new tutorial to apply <code>nn.Tranformer</code> module for word language task (<a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="nofollow noopener">link</a>). Feel free to take a look and give some feedbacks.</p>

ptrblck: <p>Thanks for sharing! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Limit process to single GPU

besterma: <p>Hi all,</p>
<p>I have a setup with 4 GPUs. When now working with multiple processes in PyTorch, Is there a way to enforce that a process only accesses a given, single gpu, therefore limiting the CUDA driver context to be present only once per process?</p>
<p>Thanks in advance for your help,</p>
<p>Benjamin</p>

spanev: <p>Hi <a class="mention" href="/u/besterma">@besterma</a>,</p>
<p>Sure, you can do it with the env variable <code>CUDA_VISIBLE_DEVICES</code>.</p>
<p>E.g. to use GPU 0 and 2:</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=0,2 python pytorch_script.py
</code></pre>
<p>and in your case you have to give a different env variable to each process.</p>

besterma: <p>Thank you <a class="mention" href="/u/spanev">@spanev</a>!</p>
<p>In case anyone is wondering, here is how to set process specific env variables:</p>
<pre><code class="lang-python">import torch.multiprocessing as _mp
import torch
import os

mp = _mp.get_context('fork')


class Process(mp.Process):
    def __init__(self):
        super().__init__()
        print("Init Process")
        return

    def run(self):
        print("Hello World!")
        os.environ['CUDA_VISIBLE_DEVICES'] = '1'
        print(torch.cuda.device_count())
        print(torch.cuda.current_device())

if __name__ == "__main__":
    num_processes = 1
    os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2,3'
    processes = [Process() for i in range(num_processes)]
    [p.start() for p in processes]
    print("main: " + os.environ['CUDA_VISIBLE_DEVICES'])
    [p.join() for p in processes]
</code></pre>
<p>It is important to set it in the run method of the process, as the <strong>init</strong> method is still called in the main process, therefore setting the env vars of the main process when set there.</p>
How to dilate a mask (making mask bigger by n pixels)? I am currently using `torch.unfold` but it runs out of memory easily

yxchng: <p>The following is my method using unfold.</p>
<blockquote>
<p>boundary_width = 9<br>
mask = mask.unfold(2, boundary_width, 1).unfold(3, boundary_width, 1)<br>
mask = mask.contiguous().view(*mask.size()[:-2], -1)<br>
mask = (mask.max(4)[0] &gt; 0).float() * 1</p>
</blockquote>
<p>However, it runs out of memory very quickly if the boundary_width is large (say 49 pixels). Is there any other way to do this? Or is there any way to reduce <code>unfold</code> memory usage?</p>
<p>I am hoping to be able to use it with ~49 pixels.</p>
Place different but correlated images in multiple channels in a CNN

mru4913: <p>Let’s say we are building a CNN and we have a multi-class classification problem. For each instance/example/object, it has 4 images captured by a sequence of 4 detectors. The resolutions of four detectors are different from each other (so they are correlated but the size of images is varied). The images are simple and grey (single channel NO RGB)</p>
<p>object : | image1 | image2 | image3 | image4 | label |</p>
<pre><code class="lang-auto"> [object]   -&gt; |image 1 detector| -&gt; |image 2 detector| -&gt;  ... | image 4 detector|

</code></pre>
<p>We know the number of channels in the input layer is usually 3, standing for <code>RGB</code>. I have tried rescaling each type of image to the same size so that we can treat each image as a channel. We have four channels in the input layer. Aer there any better ideas to implement this model? any recent research papers related to this?</p>
<p>Thank you.</p>
Per-process backward() call

johnwl: <p>I have a question about independent backward passes in each process in Pytorch’s distributed framework.</p>
<p>Suppose I spawn n worker processes with mp.spawn(), and then in each worker process, I call dist.init_process_group() with its rank and a URL (e.g. tcp://127.0.0.1:1111 to initialize it to a port on the local host).</p>
<p>I’m parallelizing the model either with<br>
apex.parallel.DistributedDataParallel(model)<br>
or<br>
torch.nn.parallel.DistributedDataParallel(model.cuda(), device_ids=[rank])</p>
<p>When I call loss.backward() on my network’s loss function, I believe the gradients are being averaged from all gpus in the process group (all_reduce sort of operation, so each gpu gets the result).</p>
<p>Suppose I wish for each process to ignore all other gpus and only call backward() with respect to its data subset. I wish to use these process-dependent gradients then to update my main, shared model. Does Pytorch’s current framework support this, and if so, how?</p>
<p>Presumably two ways to do it would be (1) create a new process group right before the backward call, and then rejoin the joint process group immediately afterwards (2) do not parallelize the model under DistributedDataParallel, but keep a separate model at each process, and do the send/recv/reduce operations myself. But this wouldn’t support synchronized BatchNorm, most likely.</p>
<p>Thanks!</p>
Am i training my GAN wrong?

makslevental: <p>i guess i’m pretty noob. here is my train loop body</p>
<pre><code class="lang-python">    for i, (lr_image, hr_image) in enumerate(train_bar):
        start = time.time()
        batch_size = lr_image.shape[0]
        if torch.cuda.is_available():
            lr_image = lr_image.cuda()
            hr_image = hr_image.cuda()

        d_lr = adjust_learning_rate(
            l.optimizerD, epoch, i, l.train_loader.size, config.d_lr
        )
        g_lr = adjust_learning_rate(
            l.optimizerG, epoch, i, l.train_loader.size, config.g_lr
        )

        if config.prof and i &gt; 10:
            break

        ############################
        # (1) Update D network
        ##########################
        l.netD.zero_grad()
        fake_img = l.netG(lr_image)
        real_out = l.netD(hr_image).mean()
        fake_out = l.netD(fake_img).mean()
        d_loss = 1 - real_out + fake_out
        d_loss.backward(retain_graph=True)
        l.optimizerD.step()

        ############################
        # (2) Update G network
        ###########################
        l.netG.zero_grad()
        g_loss = l.generator_loss(fake_out, fake_img, hr_image)
        g_loss.backward()
        l.optimizerG.step()
</code></pre>
<p>i got it from <a href="https://github.com/leftthomas/SRGAN/blob/master/train.py" rel="nofollow noopener">https://github.com/leftthomas/SRGAN/blob/master/train.py</a>. I realize that this might be wrong because <code>fake_img</code> isn’t detached on being passed to the discriminator. according to this post (i think) <a href="https://discuss.pytorch.org/t/when-training-gan-why-do-we-not-need-to-zero-grad-discriminator/20439" class="inline-onebox">When training GAN why do we not need to zero_grad discriminator?</a> i might be doing the wrong thing. also the loss function for the discriminator is a little wacky but it makes sense to me in a way (if real_out == 1 then low penalty, if fake_out == 1 then high penalty).</p>
Summation of output of previous layer

zahra: <p>Hi,</p>
<p>I want to do summation of outputs of several convolution layer and send generated output to the next layer. Is there sum layer (sth like pooling layer, convolution layer, …)?!<br>
How could I do it?</p>
<p>Thanks</p>

zahra: <p>Maybe linear layer with bias = 0 do the same. Although activation function changes it to non-linear value. Maybe there are not any way, isn’t it? Unless, I customize what I want…</p>

ptrblck: <p>How would you like to sum the conv layer outputs?<br>
If you would like to sum over a specific dimension, you could simply use:</p>
<pre><code class="lang-python">output = conv(input)
output = output.sum(dim=1)  # change the dim to your use case
</code></pre>
<p>On the other hand, if you would like to sum patches similar to a conv layer, you could define a convolution kernel with all ones, and apply it using the functional API:</p>
<pre><code class="lang-python">output = torch.randn(1, 2, 8, 8) # comes from a preceding conv layer

sum_kernel = torch.ones(1, 2, 3, 3)
output_sum = F.conv2d(output, sum_kernel, stride=1, padding=1)
</code></pre>

zahra: <p>Dear Ptrblck,</p>
<p>Thanks a lot. Like every time your answer gave me an idea to solve my problem <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
<p>Best Regards</p>
Solving sparse linear systems on the GPU

mzhang: <p>I believe the sparse matrix features in pytorch aren’t as developed as the dense features yet, but at least we can create sparse matrices with torch.sparse.Tensor. If I wanted to do a sparse linear solve on the GPU by giving it a sparse matrix and a dense vector, what would be the best way to do it? Should I look into external packages like cupy? Does anybody have experience calling packages like suitesparse from python? I’d still prefer to have my tensors in pytorch since I’m using many pytorch features, but at some point I need to do the sparse solve so even if I’m using a external library I’d like it to have decent interop with pytorch at the GPU level.</p>

Yurong_You: <p>Hi I am also insterested in this! Have you found any solution?</p>
Clarifying discrepancies between DataParallel and DistributedDataParallel

kirk86: <p>Hi folks,<br>
I’ve seen in the example <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="nofollow noopener">here</a> that for <code>DataParallel</code> one has to follow the following steps:</p>
<ol>
<li>Create model</li>
<li>Wrap the model with <code>DataParallel</code>
</li>
<li>Send to device</li>
</ol>
<p>These lead to my first confusion when I tried <code>DistributedDataParallel</code> following the pytorch imagenet example which actually has steps 2 and 3 in reverse order and without that order it throws an error about dense cuda tensors.</p>
<p>My second confusion comes again from following the pytorch imagenet example of <code>DistributedDataParallel</code> usage which utilises <code>torch.utils.data.distributed.DistributedSampler</code> to wrap the <code>train_set</code> but not the <code>valid</code> and <code>test</code> set. Why is that?</p>
<p>My final confusion arises from the fact that running <code>DistributedDataParallel</code> I observe the gpu volatility going up down like it’s the stock market all the way from 0% to 99%, anyone can shed some light on this?</p>
Is there a way to know the arguments of the loaded layer?

439327bcc62cd8e45e96: <p>I want to know the number of arguments in the picture (160, 16, 64 …)</p>
<p>Please help me.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e27ce629cd0d26508f6bcba00692a20b2dc523eb.png" data-download-href="https://discuss.pytorch.org/uploads/default/e27ce629cd0d26508f6bcba00692a20b2dc523eb" title="sadasdasd"><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e27ce629cd0d26508f6bcba00692a20b2dc523eb.png" alt="sadasdasd" data-base62-sha1="wjBvcTT5g8eqlCweyHT5pOYEwH1" width="348" height="500" data-dominant-color="2A2A2A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">sadasdasd</span><span class="informations">586×840 2.38 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You can access the internal attributes using the argument name, e.g. <code>print(bn.num_features)</code>.</p>

439327bcc62cd8e45e96: <p>Thank you for your kind reply.</p>
I want to modify batch norm to group norm. Do I still need fine tune?

439327bcc62cd8e45e96: <p>I’m wondering how to fine tune a complex structure as shown in the picture. Using Modules. () Or chiled () doesn’t work. I want to change all batch norm to group norm.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/aaa54f66fb08008673cac3cd6943e007c9cb6786.png" data-download-href="https://discuss.pytorch.org/uploads/default/aaa54f66fb08008673cac3cd6943e007c9cb6786" title="sakldkasld"><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/aaa54f66fb08008673cac3cd6943e007c9cb6786.png" alt="sakldkasld" data-base62-sha1="olBlpBJu1l8J2dUap3KwfGplhVc" width="690" height="439" data-dominant-color="161616"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">sakldkasld</span><span class="informations">1319×841 10.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

LeviViana: <p>Yes, you need to refit the model. Group Norm and Batch Norm don’t normalize the same distribution, so I don’t think there is even a way of doing the bridge with a mathematical formula.</p>
Detach() cause OOM and batch size need to be dramatically reduced

DLiang: <p>Hi guys, I met a problem while using detach() to block the backpropagation yet cause OOM problem, the situation is like:<br>
y = netA(x)<br>
z = netB(y.detach())</p>
<p>netA is a quite huge network, and netB is just a Linear network. I want to use z as loss while only updating the trainable weights in netB. However, when I remove ‘.detach()’, the network could be trained at a batch size of 32, yet when with ‘.detach()’, the batch size need to be dramatically reduced to 18 or lower to avoid OOM issue.</p>
<p>The model is trained using two GPU with nn.DataParallel() .</p>
<p>I think this is quite irrational, since detach() actually use the memory of source tensor and do not create new tensors require large amount of memory. Anyone met this kind of problem?</p>
<p>Or do you have any other way of blocking gradient backpropagation? Thank you!</p>