akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Model testing gives different result when trained again

copperwiring: <p>My training code can be seen here:</p>
<aside class="quote quote-modified" data-post="1" data-topic="72343">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ad7895/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/model-accuracy-stuck-at-0-5-though-loss-is-consistently-decreasing/72343">Model Accuracy stuck at 0.5 though Loss is consistently decreasing</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    This is using PyTorch 
I have been trying to implement UNet model on my images, however, my model accuracy is always exact 0.5. Loss does decrease. 
I have also checked for class imbalance. I have also tried playing with learning rate. Learning rate affects loss but not the accuracy. 
My architecture below ( from <a>here</a> ) 
""" `UNet` class is based on https://arxiv.org/abs/1505.04597

The U-Net is a convolutional encoder-decoder neural network.
Contextual spatial information (from the decoding,
ex…
  </blockquote>
</aside>

<p>I am able to train the model now.<br>
However, when I test it on validation dataset (15 epocs), it gives me different results everytime I train it again for same input data (images). How can I get a stable good value?</p>
<pre><code class="lang-auto">    model = torch.load(model_path)
    model.eval()
    total_acc = 0
    
    for idx in range(len(data_test)):        
        with torch.no_grad():
#             chips_x, chips_y = data_test.get_item_xarray(idx)
            X, y = data_test[idx]  

            image_t = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
            input_cloud = Variable(torch.from_numpy(y)).to(device) # [batch, channel, H, W]

            output_t = model(image_t)
            preds = output_t &gt; 0
#             print("preds in loop:", preds)
            acc = accuracy_check_for_batch(input_cloud, preds, image_t.size()[0])
            total_acc = total_acc + acc
</code></pre>
<p>P.S. I am saving the entire model and not just the weights.</p>
What is the reason for pytorch's GPU memory "floor"?

jpopham91: <p>Not a blocking issue or anything, but I was curious why there seems to be a minimum amount of gpu memory used by pytorch.</p>
<p>For example, if I run <code>x = th.randn(1).cuda(1)</code> my gpu memory usage for that process goes from zero to 723 MiB, when that tensor should only require 32 bits of memory.</p>

ptrblck: <p>The first CUDA call creates the CUDA context, which uses the memory on the device.</p>
Visualise model layer outputs (4D tensors) pytorch

Sharut_Gupta: <p>I wanted to visualise my layer outputs. Say the output is [32X512X7X7] where 32 is batch size and 512 is the number of channels. How am I supposed to visualise a plot of this sort.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bfa64610f9831326c2bb5ea3e98021b7e2b294bc.png" data-download-href="https://discuss.pytorch.org/uploads/default/bfa64610f9831326c2bb5ea3e98021b7e2b294bc" title="Screen Shot 2020-03-11 at 6.17.20 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bfa64610f9831326c2bb5ea3e98021b7e2b294bc_2_690x232.png" alt="Screen Shot 2020-03-11 at 6.17.20 PM" data-base62-sha1="rlpr4jOpz2mERoNex1TMXFciHGk" width="690" height="232" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bfa64610f9831326c2bb5ea3e98021b7e2b294bc_2_690x232.png, https://discuss.pytorch.org/uploads/default/original/3X/b/f/bfa64610f9831326c2bb5ea3e98021b7e2b294bc.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/f/bfa64610f9831326c2bb5ea3e98021b7e2b294bc.png 2x" data-dominant-color="E5E9EB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-03-11 at 6.17.20 PM</span><span class="informations">825×278 133 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You could use the <a href="https://pytorch.org/docs/stable/tensorboard.html#torch.utils.tensorboard.writer.SummaryWriter.add_histogram">add_histogram</a> method to create this visualization.</p>
<p>If I’m not mistaken, the way to do so would be to get the forward activations (e.g. using forward hooks), flatten them, and use the aforementioned method.</p>

Sharut_Gupta: <p>Are you sure that we have to flatten them? I mean, you will get a 32X512X7X7 vector. How shall I add that to this function?</p>

ptrblck: <p>Not 100% sure, but I would assume you would like to visualize the histogram from all values in the batch, right?</p>
What happens when scheduler.step() is called on Adam optimizer

blackberry: <p>Adam optimizer is said to use adaptive learning rates for each parameter. We provide an initial learning rate when instantiating it.</p>
<pre><code class="lang-auto">optimizer = optim.Adam(params, lr=4e-6)
scheduler = lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.5)

for epoch in range(100):
    # ...
    scheduler.step()
</code></pre>
<p>I wonder what exactly happens when we call scheduler step. I “guess”, for the above example, it reduces the learning rates of each and every parameter by half. Is this right?</p>
<p>Alternative would be to halve the global learning rate (4e-6), resetting per parameter learning rates and re-estimating them.</p>
<p>Which one is used in PyTorch?</p>
<p>I usually observe a drop in validation accuracy right after scheduler.step takes affect, when the network is close to convergence (it may also be happening at the beginning, but not noticeable due to large leaps in accuracy).</p>
<p>Thanks.</p>
What are the requirements for the log_dir to work in tensorboard + Pytorch?

pinocchio: <p>I had:</p>
<pre><code class="lang-auto">    tb = SummaryWriter()
    args.tb = tb
</code></pre>
<p>but nothing appeared when I had</p>
<pre><code class="lang-auto">tb.add_scalar('loss', loss, i)
</code></pre>
<p>Why?</p>
<p>Do I have to have the folder defined before hand? Or does tensorboard create the folders by itself if they need to be created?</p>
<hr>
<p>Follow up, where is the dir?</p>
<p>Ok the docs say:</p>
<pre><code class="lang-auto">log_dir (string) – Save directory location. Default is runs/CURRENT_DATETIME_HOSTNAME, which changes after each run. Use hierarchical folder structure to compare between runs easily. e.g. pass in ‘runs/exp1’, ‘runs/exp2’, etc. for each new experiment to compare across them
</code></pre>
<p>but its not here:</p>
<pre><code class="lang-auto">cd /runs
cd: no such file or directory: /runs
</code></pre>

pvardanis: <p>Did you start tensorboard backend? Try to open cmd in the same directory as your project and type:</p>
<pre><code class="lang-auto">tensorboard --logdir=runs
</code></pre>
<p>Or in Jupyter you can run a cell with the following command:</p>
<pre><code class="lang-auto">%load_ext tensorboard # this loads tensorboard backend
%tensorboard --logdir runs
</code></pre>
<p>Also you don’t need  <code>args.tb = tb</code>  in your code.</p>
RuntimeError: element 11 of tensors does not require grad and does not have a grad_fn

Vijit_Mehrotra: <p>I am trying  running a GAN neural Netowrk  but with WGANLOSS for my generated Mixture Model. it is not giving any output only throwing exception as above.</p>
<p>How can i tackle the problem and the get the desired output.</p>
<p>Hope if someone could help me solve this problem.</p>
In Detectron2, What is a proposal?

alx: <p>Hi all, I’ve been using detectron2 and I was wondering what a ‘proposal’ was. Here’s where they use the term:</p>
<p>batch_size_per_image (int): number of proposals to sample when training</p>
<pre><code class="lang-auto">cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.00025  # pick a good LR
cfg.SOLVER.MAX_ITER = 300    # 300 iterations seems good enough for this toy dataset; you may need to train longer for a practical dataset
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128   # faster, and good enough for this toy dataset (default: 512)
</code></pre>

RicCu: <p>Many object detection algorithms (sometimes known as 2 phase detection) extract multiple patches that might contain a relevant entity. These <strong>proposals</strong> are then refined to identify the ones that actually contain an object and label that patch, etc.<br>
I haven’t seen the code and usage within Detectron2 specifically, but it’s almost certainly that.</p>
NN layers and autograd.function

nirandaperera: <p>Hi,</p>
<p>I was wondering if all the nn layers have a corresponding <code>torch.autograd.Function</code> associated with them (in python side)?</p>
<p>Best</p>

albanD: <p>Hi,</p>
<p>No they don’t. Even at the cpp level, some of them are implemented with a single elementary Function but many of them have multiple.</p>

nirandaperera: <p>Thanks <a class="mention" href="/u/alband">@albanD</a>.</p>
<p>BTW I have been trying to figure out how exactly pytorch constructs the backward graph?<br>
Is it appends to a tree data structure while going through the forward pass?<br>
or does each tensor holds meta data about how they were originated (I guessing this could be <code>grad_func</code>) and from which tensors and backtracks until reaching a source node?</p>

albanD: <p>Yes it’s a tree data structure <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
You can check this graph using this package: <a href="https://github.com/szagoruyko/pytorchviz/">https://github.com/szagoruyko/pytorchviz/</a></p>

nirandaperera: <p>Is it though? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Because when I checked the code (<a href="https://github.com/szagoruyko/pytorchviz/blob/46add7f2c071b6d29fc3d56e9d2d21e1c0a3af1d/torchviz/dot.py#L56" rel="nofollow noopener">https://github.com/szagoruyko/pytorchviz/blob/46add7f2c071b6d29fc3d56e9d2d21e1c0a3af1d/torchviz/dot.py#L56</a>), It felt like, there isn’t a data structure per se, but each tensor keeps a list of functions or something like that? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"> Isn’t that how this graph is built in the pytorchviz package?</p>

albanD: <p>It is an implicit tree <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> We don’t have a centralized structure for it.</p>

nirandaperera: <p>Exactly! thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

nirandaperera: <p><a class="mention" href="/u/alband">@albanD</a> I have another follow up question.</p>
<p>I was wondering if the forward activations are exposed in the <code>nn.Module</code> level?<br>
I have seen activations need to be explicitly saved to the <code>ctx</code> if we are using the <code>autograd.Function</code>. But I am wondering how PT would keep the activations when someone’s using the <code>nn.Module</code> API?</p>

albanD: <p>The autograd engine works below the nn.Module level.<br>
So if you use regular python function or nn.Module, the autograd does not see any difference and will save the required values properly.</p>
Randomly removing elements from a binary tensor

Kirty_Vedula: <p>I am puzzled about how I can implement a probabilistic deletion in a tensor from binary trainable layer in PyTorch. A single binary deletion looks like: 10110 -&gt; 1010 where we don’t know if the 3rd or 4th bit is deleted.</p>
<p>To give some context, I am trying to solve the problem where the decoder receives the message 1010 and needs to recover the original message. This can mean a bit was deleted either in position 3 or 4. We cannot say for sure what position the deletion occurred. <a href="https://simons.berkeley.edu/sites/default/files/docs/2789/slideswang.pdf" rel="nofollow noopener">More on this here.</a></p>
<p>This is related to binary erasure channel where a single erasure looks like: 10110 -&gt; 10e10. I was able to implement erasures using this.</p>
<pre><code class="lang-auto">bec_p_enc = 0.6
noise_shape = [100,7]
x = torch.from_numpy(np.random.choice([0.0, 1.0], noise_shape, p=[bec_p_enc,  1 - bec_p_enc])).type(torch.FloatTensor)
</code></pre>
<ol>
<li>
<p>Is there a way to remove an element from a tensor, say  based on how probable the outcome is going to be? I tried <code>random.sample</code> and other commands but they didn’t help.</p>
</li>
<li>
<p>Can this layer be trainable using <a href="https://discuss.pytorch.org/t/custom-binarization-layer-with-straight-through-estimator-gives-error/4539/5">Straight-through estimation</a>?</p>
</li>
</ol>
Does torch.load reads state_dict in memory?

kirk86: <p>I was wondering if <code>torch.load</code> which returns the <code>state_dict</code> reads and loads it directly in memory or simply reads the saved <code>state_dict</code> and returns a copy of it?</p>
<p>Basically I want to load the weights of a saved model and modify them without messing with the original weights.</p>
<p>I understand that I could create a replicate of the weights by copying them from <code>state_dict</code> modifying and then loading it to the model.</p>
<p>But, is this the recommended way?</p>
<p>Is there any chance that <code>torch.load</code> which returns <code>state_dict</code> operates like <code>.detach()</code> where it returns a reference to the weights and whatever changes we make there will be automatically reflected in the stored <code>state_dict</code>?</p>
<p>For instance,</p>
<pre><code class="lang-auto">chkpt = torch.load(path)
model = vgg()
replica_weights = chkpt['state_dict']
replica_weights += 1.24
model.load_state_dict(replica_weights)
</code></pre>
<p>In the above example will the changes in <code>replica_weights</code>show up in <code>chkpt['state_dict']</code>, in other words do they share the same memory space? Is <code>replica_weights</code> a copy of <code>chkpt['state_dict']</code> or a reference pointing in the same memory space?</p>

kirk86: <p>As suspected<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7a78b853220080d59f8e79c16335176f0e89632.png" data-download-href="https://discuss.pytorch.org/uploads/default/d7a78b853220080d59f8e79c16335176f0e89632" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d7a78b853220080d59f8e79c16335176f0e89632_2_482x499.png" alt="image" data-base62-sha1="uLLCpXnis1VI9oClmhN4DAOAmLU" width="482" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d7a78b853220080d59f8e79c16335176f0e89632_2_482x499.png, https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7a78b853220080d59f8e79c16335176f0e89632.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7a78b853220080d59f8e79c16335176f0e89632.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">685×710 93.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As it turns out whenever we mess the weights we can still run <code>torch.load</code> to load the unmodified weights.</p>
Calling forward() of ConvTranspose2d after putting the module into Sequential gave an exception

ldw: <p>Hello,</p>
<p>I created ConvTranspose2d and put it in Sequential. Calling forward() afterwards gave me an exception. It seems like asserting the argument has an issue with that since ConvTranspose2d.forward() requires two arguments while I put just a tensor in the Sequential.forward().</p>
<p>Would you please take a look at this and let me know what I should do about this?</p>

mailcorahul: <p>Can you post your code along with the exception?</p>

yf225: <p><a class="mention" href="/u/ldw">@ldw</a> This is a known issue in libtorch 1.4 and will be fixed in 1.5 (which is slated to be released in April) <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>Related PR: <a href="https://github.com/pytorch/pytorch/pull/33027" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/33027</a></p>
Evaluation metric when loss is MSE

farhanrahman: <p>If I have 3 channel images (RGB) as inputs and say the network is prediction 5 channels of values where all the values range from 0.0-1.0, I believe a good choice of loss would be MSE since its a regression problem.</p>
<p>However, I am having a hard time on deciding a good evaluation metric for this type of a task. What can be a good choice? From practice I believe using the same function (MSE) will not be a good way to evaluate right?</p>
Class Activation Map (CAM) in c++ front end?

Ziri_Ziri: <p>Hi everyone ,</p>
<p>I saw many example of visualizing CAM with python using “register_backward_hook” but can’t find any way to do the same in C++ frontend.</p>
<p>Is there away to visualize Conv layers in c++ frontend ?</p>
<p>Thank you in advance.</p>

Edwinzero: <p>Hello <a class="mention" href="/u/ziri_ziri">@Ziri_Ziri</a>, I have the same problem here, do you have an available solution to this problem?</p>
Model structure problem

111264: <p>Hi,<br>
I have to compare and change values of two models,</p>
<p>model 1 names block_model and the other is new_model.</p>
<p>I made this models with other methods, so</p>
<p>model 2 has structure like<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4840c844d4649e1077ccaef5daac540eff359fd.png" alt="image" data-base62-sha1="ntndyjoxLi9NSAKExumPg4N3NWR" width="579" height="292"> ,</p>
<p>and model 1 has same structure like<br>
MSRN(<br>
(sub_mean): MeanShift(3, 3, kernel_size=(1, 1), stride=(1, 1))<br>
(add_mean): MeanShift(3, 3, kernel_size=(1, 1), stride=(1, 1))<br>
(head): Sequential(<br>
(0): DSConv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
)<br>
(body): Sequential(<br>
(0): MSRB(<br>
(conv_3_1): DSConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(conv_3_2): DSConv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(conv_5_1): DSConv2d(64, 64, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), bias=False)<br>
(conv_5_2): DSConv2d(128, 128, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), bias=False)<br>
(confusion): DSConv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)<br>
(relu): ReLU(inplace=True)<br>
)<br>
(1): MSRB(<br>
(conv_3_1): DSConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(conv_3_2): DSConv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(conv_5_1): DSConv2d(64, 64, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), bias=False)<br>
(conv_5_2): DSConv2d(128, 128, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), bias=False)<br>
(confusion): DSConv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)<br>
(relu): ReLU(inplace=True)<br>
)<br>
…</p>
<p>as you can see, model 2 has “Model( (model) : ~)” at first.</p>
<p>I want to compare and change values of each convolution.<br>
anyone know how to access to the same layer of both model?</p>
<p>now i’m using zip(block_model, newmodel), but it’s sequence is different because of “Model( (model) : ~)”, but I don’t know how to get the same layer of both model at one time.</p>
<p>Please help me !</p>
<p>Thanks.</p>

yash1994: <p>By looking at architecture posted in the screenshot. You can simply do <code>zip(block_model, newmodel.model)</code> to compare layers.</p>

111264: <p>thanks for reply !</p>
<p>it works… thankyou !!!</p>
Loading model to another GPU

Franklin_V: <p>I am trying to load a model with trained parameters. I trained the model on ‘cuda:1’ and now I want to load them to ‘cuda:0’. So I tried to using -</p>
<p><code>model.load_state_dict(torch.load('model_trained.ckpt'), map_location={'cuda:1': 'cuda:0'})</code></p>
<p>However I encounter an error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main_with_noise.py", line 176, in &lt;module&gt;
    model.load_state_dict(torch.load('model_trained.ckpt'), map_location={'cuda:1': 'cuda:0'})
  File "/home/hetro/v3/lib/python3.5/site-packages/torch/serialization.py", line 387, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "/home/hetro/v3/lib/python3.5/site-packages/torch/serialization.py", line 574, in _load
    result = unpickler.load()
  File "/home/hetro/v3/lib/python3.5/site-packages/torch/serialization.py", line 537, in persistent_load
    deserialized_objects[root_key] = restore_location(obj, location)
  File "/home/hetro/v3/lib/python3.5/site-packages/torch/serialization.py", line 119, in default_restore_location
    result = fn(storage, location)
  File "/home/hetro/v3/lib/python3.5/site-packages/torch/serialization.py", line 99, in _cuda_deserialize
    return storage_type(obj.size())
  File "/home/hetro/v3/lib/python3.5/site-packages/torch/cuda/__init__.py", line 599, in _lazy_new
    return super(_CudaBase, cls).__new__(cls, *args, **kwargs)
RuntimeError: CUDA error: out of memory

</code></pre>
<p>As its trying to use ‘cuda:1’ which at the moment is already full. I dont understand why its trying to use ‘cuda:1’ When I have specified it to map to ‘cuda:0’ and then load.</p>

mailcorahul: <p>I think you can simply use the environment variable <code>CUDA_VISIBLE_DEVICES=0</code> prefixing your python command  to make your process use only <code>gpu 0</code></p>

Franklin_V: <p>I found the cause of the error. I was passing map_location as an argument to model.load_state_dict() whereas  the correct usage is passing it as an argument to torch.load() like this -</p>
<pre><code class="lang-auto">model.load_state_dict(torch.load('model_trained.ckpt', map_location={'cuda:1': 'cuda:0'}))
</code></pre>
<p>On a side note, here</p>
<pre><code class="lang-auto">map_location={'cuda:1': 'cuda:0'}
</code></pre>
<p>means to take state_dict saved in cuda:1 format and transform it to cuda:0 format right?</p>
Autograd with non-scalar

Ilia_Karmanov: <p>At the moment if I want to view the gradients of a specific layer w.r.t each one of my inputs in the batch I loop through the gradient_weights in the loss.backward() method like so:</p>
<pre><code class="lang-auto">
import torch
import torchvision
x = torch.rand((2,3,224,224))
y = torch.ones(2, dtype=torch.long)
m = torchvision.models.resnet18()
criterion_vec = torch.nn.CrossEntropyLoss(reduction='none')
optimizer = torch.optim.SGD(m.parameters(), 0.001)
m.train()
out = m(x)

b = out.shape[0]
grads = []
for i in range(b):
    
    idx = torch.zeros(b)
    idx[i] = 1

    loss = criterion_vec(out, y)
    optimizer.zero_grad()
    loss.backward(torch.FloatTensor(idx), retain_graph=True)
    g = m.conv1.weight.grad[0][0][0]
    print(g)
    m.conv1.weight.grad.data.zero_()
</code></pre>
<p>I was wondering if there was an easier way to get the gradients w.r.t each one of the inputs in the batch. Or is this reduction on the batch-axis forced by cuDNN?</p>

albanD: <p>Yes, unfortunately, the cudnn ops always do the reduction.</p>
<p>You can check libraries like <a href="https://github.com/cybertronai/autograd-hacks">https://github.com/cybertronai/autograd-hacks</a> that implement custom backwards to replace these ops and give per-sample gradients efficiently.</p>

Ilia_Karmanov: <p>Thanks very much! I tried it and it matches</p>
<pre><code class="lang-auto">model.conv1.weight.grad1.shape  #torch.Size([4, 20, 1, 5, 5])
mean_grads = model.conv1.weight.grad1.mean(axis=0)
pytorch_grads = model.conv1.weight.grad
</code></pre>
Create ramifications recursively to multilabel/multitask network

Cristian_C: <p>Hi</p>
<p>I’m trying to train a multilabel network to extract tags from images. The number of tags is about to 250. In order to increase the accuracy I tough to use a base network as resnet familiy and then add different ramifications from the output of the resnet network, one for each tag. I know how to do it manually. The problem is do that for that huge amount of tags. I would like to find the way to do it recursively. I tried using lists and dictionaries, but didn’t work.</p>
<p>This is my code.</p>
<pre><code class="lang-python">
class MultiTaskNet(nn.Module):
    # n_tags = number of tags in the dataset
	def __init__(self, n_tags):
		super(MultiTaskNet, self).__init__()
		self.base_net = torchvision.models.resnet18(pretrained=True)
		num_ftrs = self.base_net.fc.in_features
		self.base_net.fc = nn.Linear(num_ftrs, 1024)

        #network defined assembled at the output of the base_net metwork
		self.tag_net = nn.Sequential(nn.Linear(1024, 1024),nn.PReLU(), nn.Linear(1024, 512), nn.PReLU(), nn.Linear(512, 256), nn.PReLU(), nn.Linear(256, 1) )

		self.tag_net_list = []
		for i in range (0,n_tags):
			self.tag_net_list.append(self.tag_net)
	
    def forward(self, x):
		response = []
		x = self.base_net(x)
		for tag_net in self.tag_net_list:
			x2 = tag_net(x)
			response.append(x2)
		return response
</code></pre>
<p>I think that the loss is not capable to calculate how improve the ramification weight on this structure.</p>
<p>Could you help me to do it in a proper way?</p>
<p>Thank you!</p>

yash1994: <p>Hi,</p>
<p>By looking at your architecture, first question: Is there any specific reason to branch <code>tag_net</code> for every tag? Your <code>tag_net</code> takes input tensor of 1024 values and gives output tensor of only 1 value so instead of branching <code>tag_net</code> for every tag, you can just feed the output of resnet18 to a single <code>tag_net</code> and get output tensor of same values as your number of tags. I am guessing your image contains multiple tags so you have to use <code>nn.BCEWithLogitsLoss</code> function for calculating loss and pass hot-encoded vector as your target.</p>
<p>Take a look at this: <a href="https://gombru.github.io/2018/05/23/cross_entropy_loss/" rel="nofollow noopener">https://gombru.github.io/2018/05/23/cross_entropy_loss/</a></p>
<p>Hope this helps.</p>
Forward function AttributeError: 'NoneType' object has no attribute 'data'

Javier_Rodriguez_Pui: <p>I am training a U-Net like and I am getting this error after backpropagation.</p>
<pre><code class="lang-auto">    grads = [p.grad.data for p in params]
AttributeError: 'NoneType' object has no attribute 'data'
</code></pre>
<p>I read that the problem can be that in the forward function not every layer is been called. How can I easily detect which one is causing the problem? I debugged the forward function and it seems ok to me.</p>

JuanFMontesinos: <p>That means some parameters weren’t updated (code bug or broken computational graph). param.grad is None by default and NoneType has no object data</p>
Stochastic Gradient Descent with Warm Restarts

DanielTudosiu: <p>I am in the middle of porting my research to PyTorch (the right DL framework), but I am using <a href="https://arxiv.org/abs/1608.03983" rel="nofollow noopener">the</a> Stochastic Gradient Descent with Warm Restarts and PyTorch does not have a full implementation of it.</p>
<p>So I tried to port <a href="https://github.com/tensorflow/tensorflow/blob/25c197e02393bd44f50079945409009dd4d434f8/tensorflow/python/training/learning_rate_decay.py#L514" rel="nofollow noopener">this</a> TensorFlow implementation. Could anybody please check if it respects the behavior of a learning rate scheduler in Pytorch, because I “winged” it in the sense that I do not know how the learning rate scheduler and the optimizer interact behind the hood, especially with something like Adam where the learning rate scheduler should only provide an upper bound from my understadning.</p>
<p>You can find the port bellow:</p>
<pre><code class="lang-auto">import math
import warnings

from torch.optim.lr_scheduler import _LRScheduler


class CosineDecayRestarts(_LRScheduler):
    def __init__(
        self,
        optimizer,
        first_decay_steps,
        t_mul=2.0,
        m_mul=1.0,
        alpha=0.0,
        last_epoch=-1,
    ):
        self.first_decay_steps = first_decay_steps
        self.t_mul = t_mul
        self.m_mul = m_mul
        self.alpha = alpha

        super(CosineDecayRestarts, self).__init__(optimizer, last_epoch)

    def get_lr(self):
        if not self._get_lr_called_within_step:
            warnings.warn("To get the last learning rate computed by the scheduler, "
                          "please use `get_last_lr()`.", DeprecationWarning)

        if self.last_epoch == 0:
            return self.base_lrs

        return [self._calculate_decayed_lr(group['lr']) for group in self.optimizer.param_groups]

    def _calculate_decayed_lr(self, group_lr):
        completed_fraction = self._step_count / self.first_decay_steps

        if not self.t_mul == 1.0:
            i_restart = math.floor(
                math.log(1 - completed_fraction * (1 - self.t_mul)) / math.log(self.t_mul)
            )
            sum_r = (1.0 - self.t_mul ** i_restart) / (1.0 - self.t_mul)
            completed_fraction = (completed_fraction - sum_r) / self.t_mul ** i_restart
        else:
            i_restart = math.floor(completed_fraction)
            completed_fraction = completed_fraction - i_restart

        m_fac = self.m_mul ** i_restart
        cosine_decayed = 0.5 * m_fac * (1.0 + math.cos(math.pi * completed_fraction))
        decayed = (1 - self.alpha) * cosine_decayed + self.alpha

        return group_lr * decayed
</code></pre>
Convert pre-trained model from Tensorflow to libtorch for inference

jb892: <p>Hi,</p>
<p>I’m working on a project which converts a pre-trained model from TF to libtorch. But I have experienced some issue with the final performance. It’s a huge drop. After some debugging work, I found the major problem is caused by the Conv2d layers in the network. For example, the output of the layer before the first Conv2d layer is the same as in Tensorflow. However, the output tensor of Conv2d is completely different from that in TF. I’m so confused. I have put some example code below.</p>
<pre><code class="lang-auto">Tensor loadTensorWeightFromH5(std::string name)
{
    Tensor output = readH5ToTensor(name);
    if (output.dim() == 4)
        // Convert TF weight [kernel_h, kernel_w, channel_in, channel_out] 
        // to [channel_out, channel_in, kernel_h, kernel_w]
        output = output.permute({ 3, 2, 0, 1 }); 
    return output;
}

struct Net : nn::Module
{
    Net()
        :mConv0(nn::Conv2dOptions(3, 32, 1).padding(0).stride(1)),
        mBN0(nn::BatchNormOptions(32))
    {
        register_module("Conv0", mConv0);
        register_module("BN0", mBN0);

        mConv0-&gt;weight = loadTensorWeightFromH5("Conv0_weights:0");
        mConv0-&gt;bias = loadTensorWeightFromH5("Conv0_bias:0");
        mBN0-&gt;weight = loadTensorWeightFromH5("BN0_gamma:0");
        mBN0-&gt;bias = loadTensorWeightFromH5("BN0_beta:0");
        mBN0-&gt;running_mean = loadTensorWeightFromH5("BN0_moving_mean:0");
        mBN0-&gt;running_var = loadTensorWeightFromH5("BN0_moving_variance:0");
    }

    Tensor forword(Tensor &amp; input)
    {
        // Convert NHWC to NCHW
        auto x = input.permute({ 0, 3, 1, 2 });

        x = mConv0-&gt;forward(x);

        x = mBN0-&gt;forward(x);

        x = torch::relu(x);

        // Convert NCHW to NHWC
        x = x.permute({ 0, 2, 3, 1 });

        return x;
    }

    nn::Conv2d mConv0;
    nn::BatchNorm mBN0;
}

</code></pre>
<p>Could anyone help?</p>

yf225: <p>If we run the same code in PyTorch Python API, does it give the same output as libtorch?</p>

jb892: <p>Nope. That’s a good point! I will give it a try, thx!</p>

jb892: <p>Hi after some research, I finally found what I did wrong here. It’s nothing to do with the Conv2D layer. It’s the batch norm layer cause the problem. Since the only thing, I wanna do here is inferencing. I need to set batch norm layers to evaluation mode for testing. Which looks like <code>bn-&gt;eval()</code>. It solved my problem.</p>
Why randomly apply torchvision.transforms.functional.resize won't cause an error?

crissallan: <p>Hi, I was rencently training a instance segmentation CNN and randomly apply “torchvision.transforms.functional.resize()” in the “<strong>getitem</strong>” method of the “Dataset” object(for data augmentation). However, I remembered that for a batch of data, the height and width of each images in the same batch should be the same, otherwise it will raise an error.<br>
But my traning scripts runs succesfully? why?<br>
many thanks!</p>

yash1994: <p>It depends on your CNN architecture. Convolution operations, pooling, and non-linearities will work for any input dimensions. But when you use CNN for the image classification task it involves a fully-connected layer like MLP which does require the fixed-size inputs. Generally, object segmentation CNN (this type of model is called fully-convolutional) doesn’t involve any fully-connected layer so that’s why your training script is running successfully.</p>
How to get a full coverage report of tests?

talos: <p>I’m pretty new to contributing to open source projects and am trying to get some coverage reports so I can find out what needs more / better testing. However, I am having trouble getting the full coverage of a test.</p>
<p>For example, lets say I want to get the coverage report of test_indexing_py.</p>
<p>I run the command:</p>
<blockquote>
<p>coverage run -m unittest test_indexing.py</p>
</blockquote>
<p>Resulting in this:</p>
<blockquote>
<p>…Warning: indexing with dtype torch.uint8 is now deprecated, please use a dtype torch.bool instead. (expandTensors at C:\Projects\pytorch\aten\src\ATen/native/IndexingUtils.h:20)<br>
Warning: indexing with dtype torch.uint8 is now deprecated, please use a dtype torch.bool instead. (expandTensors at C:\Projects\pytorch\aten\src\ATen/native/IndexingUtils.h:20)<br>
…s…<br>
----------------------------------------------------------------------<br>
Ran 62 tests in 0.395s</p>
<p>OK (skipped=1)</p>
</blockquote>
<p>Ok, looks like the tests ran. Now when I check the coverage report:</p>
<blockquote>
<p>coverage report -m</p>
</blockquote>
<p>I only get the coverage for the test file and not for the classes tested:</p>
<blockquote>
<p>Name               Stmts   Miss  Cover   Missing<br>
------------------------------------------------<br>
test_indexing.py     441      8    98%   370, 376-381, 620, 648</p>
</blockquote>
<p>How do I get the full coverage report including the classes tested?</p>
<p>Any guidance will be greatly appreciated.</p>
Out of CUDA memory while tracing a model with jit.trace()

ldw: <p>I got “out of memory” when I tried to trace a model with jit.<br>
What is interesting is that when I run the model in the test mode it works.</p>
<p>What is the difference between testing and tracing?</p>

Michael_Suo: <p>When you say “test mode”, do you mean calling <code>eval()</code> and then <code>forward()</code>? There are minor differences; certain <code>Module</code>s have different behavior in <code>eval</code> vs. <code>training</code> mode. In addition, if during testing you aren’t recording gradients (e.g. you set <code>requires_grad</code> to off), you don’t have to build up an autograd graph and keep the bookkeeping structures around.</p>
<p>Keep in mind that tracing will actually execute your model in order to trace it. When you run the model forward on the example inputs manually, does it still oom?</p>
<p>It would be helpful to have a script that we can use to reproduce the issue, otherwise there isn’t a ton of information to go off of.</p>

ldw: <p>With respect to “test mode”, I meant what you said. And I feel like there seems to be a memory leak in my code then.</p>
<p>Thanks for your reply.</p>
For loop alternative in GPU tensor comprehension

mirbagheri: <p>Hi<br>
I’m dealing with a loss function that requires computation of pairwise inner product of samples of each class in my dataset. Since my classes have different number of samples I cannot store all my data in one tensor and use pytorch native operations to compute loss. Therefore I have to loop over class labels and compute the loss for each of the classes separately before averaging like below:</p>
<pre><code class="lang-auto">def loss(output, label):
    unq_idx, rev_idx = torch.unique(labels, return_inverse=True)
    cdot = torch.zeros_like(unq_idx)
    for i in range(len(unq_idx)):
        output_i = output[rev_idx == i]
        cdot[i] = output_i.matmul(output_i.transpose(-1, -2)).mean((-1, -2))
    return cdot.mean()
</code></pre>
<p>The loss function is computed on the GPU however the speed is much lower compared to when I use native pytorch operations (no for loop) to calculate the same loss for a balanced dataset (equal size and average class size). I was wondering if there is any pytorch function (or combination of them) that allows me to comprehend my loss tensor with each element requiring slices of output with different lengths.</p>
Confusing about torch.Tensor and torch.tensor!

Liangzhi_Tang: <p><code>torch.Tensor(2)</code> returns <code>tensor([-4.9389e+23,  4.5758e-41])</code>, while <code>torch.tensor(2)</code> returns <code>tensor(2)</code>.  I coud’t find the manual for torch.Tensor, and it’s very confusing of these two object constructor. What’s the reason to define ‘torch.Tensor’ when ‘torch.tensor’ is already defined?</p>

ptrblck: <p><code>torch.Tensor</code> uses uninitialized memory, so you shouldn’t use it.<br>
<code>torch.tensor</code> creates a new tensor using the passed values.</p>

Liangzhi_Tang: <p>I found the usage in <code>torch.nn.batchnorm</code></p>
<blockquote>
<pre><code>        self.weight = Parameter(torch.Tensor(num_features))
</code></pre>
</blockquote>
<p>So <code>torch.Tensor</code> is not recommended?</p>

ptrblck: <p>Yes, don’t use it.<br>
<code>nn.BatchNorm</code> calls <a href="https://github.com/pytorch/pytorch/blob/cf8b728255470bafb240db54ce7680cd5fd01b25/torch/nn/modules/batchnorm.py#L39"><code>reset_parameters()</code></a> afterwards, which will initialize these tensors.</p>

Liangzhi_Tang: <p>Ok, thanks for the reply!</p>
Tips for building pytorch for contributing

kl_divergence: <p>I’ve been contributing to Pytorch and had few queries regarding building it. Building Pytorch is resource intensive.<br>
1, Do we need to build Pytorch everytime when we do git pull ? I noticed that even though the compiled binaries are still there, using the build command again starts the compilation again (but not all).</p>
<pre><code class="lang-auto">DEBUG=1 USE_DISTRIBUTED=0 USE_MKLDNN=0 USE_CUDA=0 BUILD_TEST=0 USE_FBGEMM=0 USE_NNPACK=0 USE_QNNPACK=0 USE_XNNPACK=0 python setup.py develop
</code></pre>
<ol start="2">
<li>Are there more flags that like the ones above that can reduce the compilation time and resource required ? I am using <code>ninja</code> as mentioned in the README,</li>
</ol>

ptrblck: <ol>
<li>
<p>If code changed, you would have to rebuild at least the changed functions and whatever is depending on it.</p>
</li>
<li>
<p><a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md">The contributing docs</a> have some information about e.g. <code>ccache</code> etc., which are useful for contributing to PyTorch. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
</li>
</ol>

kl_divergence: <p>How do we rebuild the changed functions only ? I am using ccache also.</p>

ptrblck: <p>I’m using <code>python setup.py develop</code> to only build what is needed after setting up ccache and ninja.</p>

kl_divergence: <p><code>python setup.py develop</code> builds everything as per <code>CONTRIBUTING.MD</code>. I’m currently using this</p>
<pre><code class="lang-auto">DEBUG=1 USE_DISTRIBUTED=0 USE_MKLDNN=0 USE_CUDA=0 BUILD_TEST=0 USE_FBGEMM=0 USE_NNPACK=0 USE_QNNPACK=0 USE_XNNPACK=0 python setup.py develop
</code></pre>
<p>My query is for example if a breaking change is introduced in Pytorch, for example here,</p>
<pre><code class="lang-auto">AttributeError: module 'torch' has no attribute 'true_divide'
</code></pre>
<p><code>true_divide</code> will become part of 1.5. But now after importing torch, it throws this error. So instead of building again, how can I only build the change that has been recently introduced. So now when I run <code>develop</code> command, it compiles <code>[124/620]</code> 620 other binaries also which means some things are getting compiled again even though they have been compiled.</p>

ptrblck: <p>If <code>true_divide</code> was used in other methods (and files), they need to be rebuilt as well. Same goes for other depending files thereof.</p>

kl_divergence: <p>That’s my question. What do I do to rebuild only the changes ? Is there some automatic way of compiling only the changes ?</p>

ptrblck: <p>Yes, as I said <code>python setup.py develop</code> should do the job. At least it’s working every time for me.</p>

kl_divergence: <p>Okay, thanks will try.<br>
I’m doing with this command</p>
<p><code>DEBUG=1 USE_DISTRIBUTED=0 USE_MKLDNN=0 USE_CUDA=0 BUILD_TEST=0 USE_FBGEMM=0 USE_NNPACK=0 USE_QNNPACK=0 USE_XNNPACK=0 python setup.py develop </code><br>
It builds other stuff also, but this is a subset of what you shared.</p>
Same batch size in evaluation trigger "RuntimeError: CUDA out of memory."

songyuc: <p>Hi, guys,<br>
I am learning about DeepLabV3+ model these days.<br>
And I meet a strange phenomenon that using the same batch size in evaluation trigger “RuntimeError: CUDA out of memory.”, which is normal in training.<br>
But the inference speed seems quite faster than the training.</p>
<p>Any idea or answer will be appreciated!</p>

ptrblck: <p>Are you seeing the OOM error after a few iterations or how were you able to see the validation step being faster?<br>
Since Python uses function scoping, you might want to wrap some code in specific functions so that they’ll be cleared as explained <a href="https://discuss.pytorch.org/t/spliting-my-model-on-4-gpu-and-cuda-out-of-memory-problem/37311/3">here</a>.</p>

songyuc: <p>Hi, I saw this OOM error as soon as the program started.</p>

ptrblck: <p>What is the largest batch size you can run in training and evaluation without running into the OOM error?</p>

songyuc: <p>The largest batch size in training and evaluation is 4 and 2.</p>

ptrblck: <p>Could you post a code snippet to reproduce this issue?<br>
Instead of your real data, you could initialize the input and target using random tensors, so that we could debug this issue.</p>

songyuc: <p>I think should check the problem myself first,<br>
and I want to know if there is way to show the GPU memory usage of a PyTorch model?</p>

ptrblck: <p>You could use <code>torch.cuda.memory_allocated()</code>, <code>torch.cuda.memory_cached()</code> etc. in your script to check the memory. Also, <code>nvidia-smi</code> will give you the overall memory usage (including the CUDA context).</p>

songyuc: <p>Thanks for your answer.<br>
I will try it next time.</p>
PyTorch Performance Profiling

mortazavi: <p>What’s the standard tool for PyTorch profiling these days?<br>
Is there anything that would also work with torch.multiprocessing?<br>
Setting up timers and counters of one’s own at different parts of code doesn’t seem the most effective way to do this and spend one’s time?<br>
(TB’s profiling probably has hooks for this but would only work with TF.)</p>

albanD: <p>I would suggest the builtin profiler: <a href="https://pytorch.org/docs/stable/autograd.html#profiler">https://pytorch.org/docs/stable/autograd.html#profiler</a></p>
<p>The support for distributed backward is work in progress though. But the rest should be fairly stable.</p>

mortazavi: <p>I like it for its simplicity, and I’ll give it a try.</p>
<p>Hopefully, more documentation . . . is on the way . . .  and the distributed release, too . . .</p>
<p>(Is the latter really always necessary when the user can already directly use it in multiprocessing’s process “target”'s ? Or is that not allowed in torch.multiprocessing? I guess, I will find out, and if I learn anything, I’ll try to remember to post it here.)</p>
Is it a good way to use checkpoint to improve efficiency of GPU memory

songyuc: <p>Hi guys,<br>
I just read some materials from Internet that the checkpoint in PyTorch could be utilized to decrease the usage of GPU memory, and I want to know if that is true?</p>
<p>Any answer or idea will be appreciated!</p>

albanD: <p>Hi,</p>
<p>This is what this module was built for. So yes it reduces memory usage at the cost of using more compute.</p>

songyuc: <p>Hi, Alban,<br>
could you please share some tutorial of how to use checkpoint to reduce memory usage?</p>

albanD: <p>I don’t think there is much more than just the doc: <a href="https://pytorch.org/docs/stable/checkpoint.html#torch.utils.checkpoint.checkpoint">https://pytorch.org/docs/stable/checkpoint.html#torch.utils.checkpoint.checkpoint</a></p>
<p>It should be as simple as calling groups of layers from your network with the module.<br>
You can find an advanced example in torchvision: <a href="https://github.com/pytorch/vision/blob/216035315185edec747dca8879d7197e7fb22c7d/torchvision/models/densenet.py">https://github.com/pytorch/vision/blob/216035315185edec747dca8879d7197e7fb22c7d/torchvision/models/densenet.py</a></p>

songyuc: <p>I am really grateful for your answer.<br>
After I read some of your provided materials, I find that it seems using checkpoint to trade compute for memory only applies to ReLU functions, am I right?</p>
<p>Looking forward to your reply!</p>

albanD: <p>Hi,</p>
<p>No it applies to any module.<br>
It allows you to reduce the number of intermediary results between ops. So it will only be useful if you checkpoint a function that contains multiple ops.</p>

songyuc: <p>Thanks sincerely for your answer!</p>