akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Semantic Segmentation dataloader and input format problem

mertakcay: <p>Hi everyone, i have 6 class for semantic segmentation with deeplabv3.i’m using pytorch segmentation model for training.As I remember,the each layer of input must represent one class to train but I notice that some colormaps on image are not be same with annot. tool. How can I fix it and what is this the problem ?</p>
<p><strong>DATALOADER</strong></p>
<pre><code class="lang-auto"># read data
        image = cv2.imread(self.images_fps[i])
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
        mask = cv2.imread(self.masks_fps[i])
        mask = cv2.cvtColor(mask, cv2.COLOR_BGR2RGB)
        print("Output Mask",mask.shape)
        print("1:",np.unique(mask[:,:,0]))
        print("2:",np.unique(mask[:,:,1]))
        print("3:",np.unique(mask[:,:,2]))
        
        
        temp = np.zeros(shape=(mask.shape[0],mask.shape[1],len(CLASSES)), dtype=np.uint8)
        #print for logging 
        for index,color in enumerate(self.COLORS):
          print(f"Color: {color} Code: {CLASSES[index]}")
          for i in range(mask.shape[0]):
            for k in range(mask.shape[1]):
              if(mask[i,k] == color).all():
                #print("mask[i,k]: ",mask[i,k])
                temp[i,k,index] = 1
          print(f"Mask {index}",np.unique(temp[:,:,index]))
          print(f"Mask --{index}",np.unique(temp[:,:,index]))
        mask = temp       
</code></pre>
<p>TRANING MODEL LOADER</p>
<pre><code class="lang-auto">model = smp.DeepLabV3(
    encoder_name=ENCODER,        
    encoder_weights=ENCODER_WEIGHTS,                       
    classes=len(CLASSES),
    activation=ACTIVATION 

CLASSES  = ['background','car','crack', 'lane', 'pothole', 'road']
ENCODER = 'timm-mobilenetv3_small_075'
ENCODER_WEIGHTS = 'imagenet'
ACTIVATION = 'softmax2d'
DEVICE = 'cuda                   
)


</code></pre>
Build-version about libtorch-win-shared-with-deps-debug-1.10+cu113

zk_huang: <p>I think words in build-version in  libtorch-win-shared-with-deps-debug-1.10+cu113  seems wrong.<br>
It should be 1.10.0+cu113  but it reads "1.10.1+cu113. "</p>

ptrblck: <p>Could you explain a bit more why the <code>1.10.1</code> version tag is wrong?</p>

zk_huang: <p>I want to download  libtorch-win-shared-with-deps-debug-1.10.0, so I change<br>
<a href="https://download.pytorch.org/libtorch/cu113/libtorch-win-shared-with-deps-debug-1.10.1%2Bcu113.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cu113/libtorch-win-shared-with-deps-debug-1.10.1%2Bcu113.zip</a> to<br>
<a href="https://download.pytorch.org/libtorch/cu113/libtorch-win-shared-with-deps-debug-1.10.0%2Bcu113.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cu113/libtorch-win-shared-with-deps-debug-1.10.0%2Bcu113.zip</a></p>
<p>After I download  libtorch-win-shared-with-deps-debug-1.10.0 , I found the tag is 1.10.1+cu113 .  Meanwhile,   I download  libtorch-win-shared-with-deps-debug-1.10.1 , the tag is also 1.10.1+cu113 .</p>
<p>libtorch-win-shared-with-deps-debug-1.10.0  and   libtorch-win-shared-with-deps-debug-1.10.1 ara the same tag “1.10.1+cu113”</p>

ptrblck: <aside class="quote no-group" data-username="zk_huang" data-post="3" data-topic="140490">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zk_huang/48/45341_2.png" class="avatar"> zk_huang:</div>
<blockquote>
<p>After I download libtorch-win-shared-with-deps-debug-1.10.0 , I found the tag is 1.10.1+cu113 .</p>
</blockquote>
</aside>
<p>Where do you get this tag from?<br>
After downloading the <code>1.10.0</code> zip file, I see that <code>build-version</code> defines it at <code>1.10.0+cu113</code> while <code>CAFFE2_VERSION</code>, <code>PACKAGE_VERSION</code>, and <code>TORCH_VERISON</code> are defined as <code>1.10.0</code>, which is expected.</p>

zk_huang: <p>I check it again. Build-version in libtorch-win-shared-with-deps-debug-1.10.0+cu113 is right.    You are right.  Maybe I got some mistakes.   Thank you for your reply.</p>
Torch.autograd.grad for op-by-op backward

ralzq01: <p>Hi,</p>
<p>I’m trying to use <code>torch.autograd.grad</code> to write backward procedure operator-by-operator. It works fine when there is no same tensor to be used by multiple operators. But I don’t know how to deal with residual connections shown in below (tensor <code>a</code>). Is there any way that I can do backward operator-by-operator (and at last accumulate grad of  tensor <code>a</code> by manual code)?</p>
<pre><code class="lang-auto">import torch

a = torch.tensor([1,2,3,], dtype=torch.float).requires_grad_()
b = torch.tensor([1,2,3,], dtype=torch.float).requires_grad_()
c = torch.tensor([2,3,4], dtype=torch.float).requires_grad_()
d = c + a
e = d * b
f = e + a
# f = (c + a) * b + a
loss = torch.sum(f)
# loss.backward()
# print(a.grad)
(grad_e, grad_a1) = torch.autograd.grad([loss], [e, a])
print(grad_e, grad_a1)
(grad_d, grad_b) = torch.autograd.grad([e], [d, b], [grad_e])
print(grad_d, grad_b)
(grad_c, grad_a2) = torch.autograd.grad([d], [c, a], [grad_d])
print(grad_c, grad_a2)
grad_a = grad_a1 + grad_a2
</code></pre>
<p>Will encounter</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time (or directly access saved variables after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved variables after calling backward.
</code></pre>
Averaged Euclidean Distance

J_Johnson: <p>Suppose we have a matrix A composed of m vectors with n dimensions.</p>
<p>And suppose we want to get the averaged Euclidean distance between all of those vectors. I know how to do this in for loops but I am not quite as clear for matrix operations.</p>
<p>I see someone posed the question here and received an answer:</p>
<p><a href="https://stats.stackexchange.com/questions/192423/average-distance-in-distance-matrix" class="inline-onebox" rel="noopener nofollow ugc">Average distance in distance matrix - Cross Validated</a></p>
<p>How would this be implemented in Pytorch as a matrix operation?</p>
<p>Additionally, how would it need to be modified if the matrix is complex(i.e. torch.cfloat)?</p>

J_Johnson: <p>So I found  <a href="https://pytorch.org/docs/stable/generated/torch.cdist.html" rel="noopener nofollow ugc">torch.cdist</a> creates a <a href="https://en.m.wikipedia.org/wiki/Euclidean_distance_matrix" rel="noopener nofollow ugc">Euclidean distance matrix</a>.</p>
<p>So getting the average shouldn’t be an issue.</p>
<p>I noticed cdist doesn’t support complex matrices. But that’s fine. I can just do this:</p>
<pre><code class="lang-auto">vector_dims=10
n_vectors=100

A = torch.rand(n_vectors, vector_dims, dtype=torch.cfloat)

B = torch.cdist(A.real, A.real)

C = torch.cdist(A.imag, A.imag)

E_dist = (B**2 + C**2)**0.5

Avg_dist = torch.sum(E_dist) / (n_vectors*(n_vectors-1)))
</code></pre>
How to slice multiple spans from a big 1D tensor parallelly on GPU?

Alani: <p>I have a big 1D tensor <code>A</code>, which contains around 20M elements. I also have some spans with unequal lengths, i.e.,  <code>B=[(s_1, e_1), (s_2, e_2), ..., (s_n, e_n)]</code>, where <code>n</code> may be more than 8K. The one-time slicing <code>A[s:e]</code>  is very fast, but slicing for all spans in <code>B</code> by <code>for loop</code> is very time consuming. Is there any way to slice parallelly on gpu? My torch version is 1.8.1, and some operations like <code>map_()</code> and <code>apply_()</code> are only available on CPU.</p>
<p>For example:</p>
<pre><code class="lang-python">A = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
B = torch.tensor([[0, 3], [6, 8]])
C = UnkownFuction(A, B)
</code></pre>
<p><code>C</code> is also a 1D tensor tensor <code>[1, 2, 3, 4, 7, 8, 9]</code></p>
<p>Thanks for your kind help in advance!</p>
After pytorch fixes the random seed, why can the results be repeated only in the first few epochs

C_jojo: <p>After pytorch fixes the random seed, only the results of the first few epochs can be repeated, and the results of the next few epochs cannot be repeated.<br>
Here is my seed setup:<br>
jj=0<br>
np.random.seed(jj)<br>
random.seed(jj)<br>
torch.manual_seed(jj)<br>
torch.cuda.manual_seed_all(jj)<br>
torch.cuda.manual_seed(jj)<br>
# torch.backends.cudnn.benchmark = False<br>
torch.backends.cudnn.deterministic = True<br>
# torch.use_deterministic_algorithms(True)<br>
torch.backends.cudnn.enabled = False<br>
torch.backends.cudnn.benchmark = False<br>
os.environ[‘PYTHONHASHSEED’] = str(jj)<br>
os.environ[‘CUBLAS_WORKSPACE_CONFIG’] = ‘:4096:8’<br>
Furthermore，the Dataloader:<br>
def seed_worker(worker_id):<br>
worker_seed = torch.initial_seed() % 2 ** 32<br>
np.random.seed(worker_seed)<br>
random.seed(worker_seed)<br>
data_loader= DataLoader(train_sample, batch_size=20,num_workers=0,worker_init_fn=seed_worker)</p>
<p>When I set epochs=10, the results of the experiment are the same no matter how many times it is repeated. When I set epochs=200, the experimental results have no deterministic characteristics.</p>
Improving doc for is_leaf

Zhihan_Yang: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/884a830976f654c6beef4187e1c730e622e43505.png" data-download-href="https://discuss.pytorch.org/uploads/default/884a830976f654c6beef4187e1c730e622e43505" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/884a830976f654c6beef4187e1c730e622e43505_2_690x193.png" alt="image" data-base62-sha1="jrGzLEctBTOQXkYSDDcFv7PblmB" width="690" height="193" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/884a830976f654c6beef4187e1c730e622e43505_2_690x193.png, https://discuss.pytorch.org/uploads/default/original/3X/8/8/884a830976f654c6beef4187e1c730e622e43505.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/8/884a830976f654c6beef4187e1c730e622e43505.png 2x" data-dominant-color="EFEFF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">924×259 36.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The documentation writes that “only leaf tensors will have their grad populated during a call to backward”. Maybe we should add that there’s one additional condition: in order for their grad to be populated, the leaf tensors must also require grad?</p>
<p>Without it, the page is slightly confusing, since at the top it says “all tensors that have requires_grad which is False will be leaf Tensors by convention”.</p>
How to get one sample of dataset to train and to test

hitbuyi: <pre><code>dataloader = datasets.CIFAR10
trainset = dataloader(root='./data', train=True, download=True, 
                                     transform=transform_train)
</code></pre>
<p>Now I want to get one sample of trainset to train for debugging my code ,how to  write the code?</p>

nivek: <p>Based on your code, you should be able to get the first element by running this:</p>
<pre><code class="lang-auto">print(next(iter(trainset)))
</code></pre>
<p>You may find the <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#preparing-your-data-for-training-with-dataloaders" rel="noopener nofollow ugc">examples on this page</a> helpful.</p>

hitbuyi: <p>Thanks a lot, maybe</p>
<pre><code class="lang-auto">torch.utils.data.Subset(testset, subset_indices)
</code></pre>
<p>works</p>
Print model layer from which input is passed

cbd: <p>In below code, input is passed from layer “self.linear1” in forward pass. I want to print the layers from which input is passed though other layer like “self.linear2” is initialise. It should be print only “linear1”.</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.nn.functional as F
import torch
from torchsummary import summary

class TwoLayerNet(nn.Module):
    def __init__(self, D_in, H, D_out):
        super(TwoLayerNet, self).__init__()
        self.linear1 = nn.Linear(D_in, H) 
        self.linear2 = nn.Linear(H, D_out)

    def forward(self, x):
        h_relu = F.relu(self.linear1(x))
        #y_pred = self.linear2(h_relu)
        return h_relu

N, D_in, H, D_out = 32, 100, 50, 10

x = torch.randn(N, D_in)  
model = TwoLayerNet(D_in, H, D_out)
print("model is ",model)
y_pred = model(x)</code></pre>

ecdrid: <p>A quick workaround will be using a wrapper layer over the pytorch’s linear layer where you print if input is flowing throw it.</p>
<p>Otherwise, you can simply add print statements or logging.</p>

cbd: <p>I just need the summary like printing model but the only layers through which input pass.</p>

ptrblck: <p>Maybe printing the graph in e.g TensorBoard as given in <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html#inspect-the-model-using-tensorboard">this tutorial</a> might be helpful.</p>

Ludwig96: <p>I have also same issue any suggestion for this so please reply. Thanks in advance<a href="https://www.mcdvoice.tips/" rel="noopener nofollow ugc">.</a></p>

cbd: <p>Above solution will help.</p>
What does the last_epoch parameter of SGDR mean?

JamesDickens: <p>I’m reading the docs <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CosineAnnealingWarmRestarts.html#torch.optim.lr_scheduler.CosineAnnealingWarmRestarts" rel="noopener nofollow ugc">here</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97af13f6cf52525b8cefbe42fc216e3f7f7d5e48.png" data-download-href="https://discuss.pytorch.org/uploads/default/97af13f6cf52525b8cefbe42fc216e3f7f7d5e48" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97af13f6cf52525b8cefbe42fc216e3f7f7d5e48_2_660x500.png" alt="image" data-base62-sha1="lDRcsdLnUcpVtiGEC6jd0GAvMgw" width="660" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97af13f6cf52525b8cefbe42fc216e3f7f7d5e48_2_660x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97af13f6cf52525b8cefbe42fc216e3f7f7d5e48_2_990x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/7/97af13f6cf52525b8cefbe42fc216e3f7f7d5e48.png 2x" data-dominant-color="F5F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1141×864 88.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m confused what the last_epoch parameter does when it’s not set to -1, for example when the optimizer is regular SGD.  Any insights appreciated.</p>
Can I manually apply loss after loosing gradient?

pumplerod: <p>I’ll try my best to simplify what I am trying to accomplish.  In trying to create a decision tree through pyTorch my gradient is lost.  I’d like to know if there is a way I can still calculate my loss and manually apply a gradient to earlier layers.</p>
<p>I first pass the features of my data set through a Linear Layer.  I have a Variable (requires_grad=True) which is used to determine a “split_value”.  Using this value I split the incoming tensor using .ge() and .lt().  This is where I loose my gradient.  After reassembling the output tensor I can calculate loss but I’m not sure how to reach back into the chain to apply it to the “split_value” or the LinearLayer.</p>
<p>simple example code (This is using random values for the example so I realize this example would never converge):</p>
<pre><code class="lang-auto">class_values = [ 0.0, 0.5, 1.0]

feature_data = torch.rand( (20,10))
target_data = torch.tensor( np.random.choice( class_values, (20)))

split_value = Variable( torch.rand(1), requires_grad=True)

mix_feature_layer = torch.nn.Linear( 10, 5)

#
# Here I am splitting based on the 0th index feature output from the Linear layer
#   This is where I loose access to the gradient
true_labels = target_data[ mix_feature_layer( feature_data)[ :, 0].ge( split_value)]
false_labels = target_data[ mix_feature_layer( feature_data)[ :, 0].lt( split_value)]

#
# Let the loss be the combined amount of variance from both the true and false branch of the split.
#   Ideally the split value should be trained to group the labels so that each branch has little to no variance.
loss = 0
loss += torch.var( true_labels) if true_labels.size(0)&gt;1 else 0
loss += torch.var( false_labels) if false_labels.size(0)&gt;1 else 0
 
</code></pre>
<p>The question becomes, how to apply this loss back to affect the split_value and weights of the Linear Layer?</p>
How to remove all torch in GPU

Rami_Ismael: <p>I was hoping there is a quick method to remove all pytorch from the GPU?</p>

ptrblck: <p>You could delete all tensors, parameters, models etc. and call <code>empty_cache()</code> afterwards to remove all allocations created by PyTorch. To also remove the CUDA context, you would have to shut down the Python session.</p>

Rami_Ismael: <p>Is there an approach to have a list of all tensor that exist on the gpu?</p>

ptrblck: <p>No, I don’t think there is a way to list <em>all</em> tensors. You could try to use <a href="https://discuss.pytorch.org/t/how-to-debug-causes-of-gpu-memory-leaks/6741/19">this approach</a>, but as discussed in the topic tensors kept alive in the C++ backend won’t be returned by this approach.</p>

Rami_Ismael: <p>I see the how to get the list of all tensor. However, I have hard time moving the tensor to cpu and then remove from from gpu.</p>
Why do I fail to implement the backward propagation with MADDPG?

ntuce002: <p>I began to train my MADDPG model, but there’s something wrong while calculating the backward.<br>
<a href="https://i.stack.imgur.com/xbFSo.jpg" rel="noopener nofollow ugc">Here’s the detail of the error</a></p>
<p>And here’s the code of the training loop:</p>
<pre><code class="lang-auto">       for idx in range(1, len(self.agents) + 1):
            with torch.autograd.set_detect_anomaly(True):
                critic_value_ = \
                    self.agents[idx - 1].target_critic.forward(new_pos_state, new_phase_state, new_actions).flatten()

                critic_value_[dones[:, 0]] = 0.0
                critic_value = self.agents[idx - 1].critic.forward(pos_state, phase_state, old_actions).flatten()

                target = rewards[:, idx - 1] + self.agents[idx - 1].gamma * critic_value_
                critic_loss = F.mse_loss(target, critic_value)
                self.agents[idx - 1].critic.optimizer.zero_grad()
                critic_loss.backward(retain_graph=True)
                self.agents[idx - 1].critic.optimizer.step()

                actor_loss = self.agents[idx - 1].critic.forward(pos_state, phase_state, mu).flatten()
                actor_loss = -torch.mean(actor_loss)
                self.agents[idx - 1].actor.optimizer.zero_grad()
                actor_loss.backward(retain_graph=True)
                self.agents[idx - 1].actor.optimizer.step()

                self.agents[idx - 1].update_network_parameters()
</code></pre>
<p>I think the line <code>critic_value_[dones[:, 0]] = 0.0</code> may be problematic. However, I have no idea how to fix it well.<br>
<a href="https://gist.github.com/chhuang17" rel="noopener nofollow ugc">The other relative codes</a> have been uploaded to my Github. Also, I can provide more other codes if necessary.</p>
<p>Hope someone can help me, thanks!</p>
`model.eval()` returns the same results for all inputs data

marsggbo: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52dd5604911511faa75c3e781f15edf712a7d05b.png" alt="33a951b46af6be2c3bc2cfc4710df11" data-base62-sha1="bP3muxRFV1xhs57BLr6HcQaggkz" width="582" height="462"></p>
<p><code>self._logits_and_loss</code> just calls <code>self.network(X)</code> function.</p>
<p>We can see that given the same input data <code>X</code>, <code>eval</code> mode will return nearly the same results for all data. But if I change to <code>train</code> mode, the results seem relatively correct.</p>
<p>My model is based on my own framework and it is kind of complicated, so I cannot give the code of the model here.</p>
<p>I guess there is something wrong with the BN layer, so I tried to print the data of the last BN layer after training several epochs</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5aeacbe2f0c0ac512b19a943fb2fe726fc811a6e.png" alt="image" data-base62-sha1="cYi1kagW9CY5ZxkvK0q2IJhO75Y" width="588" height="245"></p>
<p>From the above figure, it seems that everything is OK with the last BN layer. I wonder what are the possible reasons? Are any other suggestions to debug this problem? Thanks!</p>
Why does torch.add function increase the memory usage of GPU?

Ardeal: <p>Hi,<br>
For the following function,<br>
the shape of l and r are both (1, 64, 640, 360), and the device are both ‘cuda:0’ and the datatype are both torch.float32.</p>
<p>the problem is: once the function is executed, the used memory of GPU increased by 58MB.<br>
why does torch.add function increase the memory usage of GPU?</p>
<p>I tested it many times, and 58MB more memory was increased every time.</p>
<p>even if I use <code>l+r</code> instead of torch.add(l, r), the memory used increased 58MB. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=10" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<pre><code class="lang-auto">def _add(self, *inputs):
        l, r = inputs
        return torch.add(l, r)

</code></pre>

mMagmer: <p>hi,<br>
maybe it’s because of retaining input for computing grad, if you don’t need to call backward, you can use</p>
<pre><code class="lang-auto">@torch.no_grad()
def _add(self, *inputs):
        l, r = inputs
        return torch.add(l, r)
</code></pre>
<p>i’m not sure about it.</p>

Ardeal: <p>I tried <code>@torch.no_grad()</code>. It is not that reason.<br>
I tested the function for forward, but not backward.</p>

mMagmer: <p>then maybe you can use inplace operation.</p>
<pre><code class="lang-auto">l.add_(r)
</code></pre>

Ardeal: <p>Nice!!!<br>
the inplace operation works. GPU memory doesn’t increase any longer.<br>
during forward, the memory of add operation doesn’t increse any longer.<br>
however, during backward, I experienced another issue:</p>
<p><code>Exception has occurred: RuntimeError one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 64, 640, 360]], which is output 0 of AsStridedBackward, is at version 9; expected version 8 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</code></p>

mMagmer: <p>you can’t use backward when using In-place operation.</p>

Ardeal: <aside class="quote no-group quote-modified" data-username="mMagmer" data-post="6" data-topic="140492" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmagmer/48/43283_2.png" class="avatar"> mMagmer:</div>
<blockquote>
<p>you can’t use backward when using In-place operation.<br>
i think a workaround is to write a custom module with a set backward method if it’s possible.</p>
</blockquote>
</aside>
<p>I am considering why torch.add will increase 58MB memory usage?<br>
how about some other operation, such as, torch.divide, torch.mul and etc.?</p>
Understanding chain rule for matrices

Zhihan_Yang: <p>Suppose the gradient of a scalar loss with respect to a matrix A has shape (M, N), which is just the dimension of matrix A. Now suppose we want to compute the derivative of the same scalar loss with respect to B, which is defined as A=f(B) for some function f and has shape (M’, N’).</p>
<p>Then,</p>
<ul>
<li>dA/dB has shape <code>(M, N, M', N')</code> in PyTorch (if you try using <code>torch.autograd.functional.jacobian</code>, this is the shape that you get)</li>
<li>dL/dA has shape <code>(M, N)</code> in PyTorch</li>
</ul>
<p>We clearly know that dL/dB has shape <code>(M', N')</code>, but dL/dA @ dA/dB is invalid in PyTorch (but valid mathematically as chain rule).</p>
<p>How should I understand this issue? How should I compute dL/dB given dL/dA and dA/dB?</p>
<p>Example:</p>
<pre><code class="lang-python">torch.matmul(torch.randn(4, 5), torch.randn(4, 5, 6, 7)
# RuntimeError: mat1 and mat2 shapes cannot be multiplied (140x6 and 5x4)
</code></pre>
How to flow tensor data data from a folder?

mohammadreza_Aalaei: <p>I have saved 80000 PyTorch tensors of shape (64, 64) in some folder in pt format. because of the memory limitation, I cant load all of these files at once so I want to use this folder as the root and load the data in batches like what is done with images dataset using torchvision.datasets.ImageFolder, Is there some similar PyTorch function that does this for me?</p>

ptrblck: <p>If the structure of your data is equal to what <code>ImageFolder</code> expects (i.e. samples for classes are located in their corresponding folder), you could use <a href="https://github.com/pytorch/vision/blob/922db3086e654871c35cd80c2c01eabb65d78475/torchvision/datasets/folder.py#L107"><code>torchvision.datasets.DatasetFolder</code></a> and define a custom <code>loader</code> method, which could be as simple as <code>lambda path: torch.load(path)</code>. You can check the default loaders <a href="https://github.com/pytorch/vision/blob/922db3086e654871c35cd80c2c01eabb65d78475/torchvision/datasets/folder.py#L244-L268">here</a>.</p>
<p>If that’s not the case and your data is located in a single folder, you could write a custom <code>Dataset</code> instead as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a>.</p>
Which one makes training faster (or more efficient), detach() or requires_grad = False

yoshitomo-matsubara: <p>Hello,</p>
<p>When training only specific layers in a model, which one should make training faster, <code>detach()</code> or <code>requires_grad = False</code>? Or no difference?</p>
<p>Assuming you have some pretrained model and want to fine-tune some of its layers while freezing the other layers and your optimizer contains updatable parameters only (i.e., those with <code>requires_grad = False</code> are not passed to optimizer)</p>
<p>Approach 1: ResNet with frozen <code>conv1</code>, <code>bn1</code>, <code>layer1</code>, and <code>layer2</code></p>
<pre><code class="lang-auto">    def forward(self, x):
        x = self.conv1(x) # its parameters have requires_grad = False
        x = self.bn1(x) # its parameters have requires_grad = False
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x) # its parameters have requires_grad = False
        x = self.layer2(x) # its parameters have requires_grad = False
        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)
        x = torch.flatten(x, 1)
        x = self.fc(x)

        return x
</code></pre>
<p>vs.</p>
<p>Approach 2: ResNet with detach() after <code>layer2</code></p>
<pre><code class="lang-auto">    def forward(self, x):
        x = self.conv1(x) 
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)

        x = x.detach()

        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)
        x = torch.flatten(x, 1)
        x = self.fc(x)

        return x
</code></pre>
<p>Also, would the answer change if optimizer contains all the parameters in a model (without <code>requires_grad = False</code>) but with <code>detach()</code> after <code>layer2</code>?</p>
<p>Thank you! (and Happy Holidays!)</p>

ptrblck: <p>I think both approaches would yield the same execution as Autograd should be smart enough to stop the backpropagation where no gradients are needed in previous layers (approach 1).<br>
The difference of passing all parameters or only the subset which requires gradients would be in <a href="https://github.com/pytorch/pytorch/blob/fb78a31916d471ebdfb0466dc53446f27f4f41b2/torch/optim/adam.py#L110-L111">this check</a>, which checks if a valid gradient attribute is set for the current parameter and skips it if not. I also doubt you would see a significant performance difference and would thus claim the used approach depends on your coding style and how explicit you want to write your code.<br>
Personally, I would avoid passing unused parameters to the optimizer just to avoid any side effects.</p>

yoshitomo-matsubara: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for the answer! Good to hear that there would not be a significant difference in terms of performance.</p>
<p>I also agree with you and would avoid passing unused parameter to the optimizer.</p>
CUDA illegal memory error when using nn.Conv1d and torch.backends.cudnn.benchmark = True

timmypi: <p>Hi,</p>
<p>I encountered a CUDA runtime error: illegal memory: an illegal memory access was encountered. This happens when calling model.to(‘cuda’), and the stack trace points to this line in the convert(t) method (<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py" rel="noopener nofollow ugc">line 903/905</a>):</p>
<p>return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)</p>
<p>This happens when I build the model with nn.Conv1d, but is fine when I build the same model using nn.Conv2d. Does anybody have a clue where the problem might lie?</p>
<p>Thanks a lot and happy holidays!</p>
<p>Edit: So in my code I also set torch.backends.cudnn.benchmark to True. I found out that removing this resolves my illegal memory error (in the version that uses Conv2d, having the value as True causes no problem). I haven’t managed to understand yet why setting the benchmark to True causes a problem in the version that uses Conv1d</p>

ptrblck: <p>Could you post the output of <code>python -m torch.utils.collect_env</code> here, please?</p>
How to create a custom dataloader in c++

mas: <p>Hi all follow developers,</p>
<p>I have been struggling to create a custom dataloader in c++ torch. I think I am missing some key concepts. Could someone help guide me to the right path?</p>
<p>I have both input and target. However, I am struggling to create a dataset to run: <code>torch::data::make_data_loader</code></p>
<p>Thank you.</p>

ptrblck: <p>What kind of issues are you seeing? <a href="https://github.com/pytorch/pytorch/blob/afb742382afc9002e15954abfb1f7616d2fa07a0/test/cpp/api/dataloader.cpp#L162-L183">This code snippet</a> shows a minimal example how to use <code>make_data_loader</code>.</p>
Freeze layers in torch C++

MakGulati: <p>how can I freeze subset of layers of NN from training in C++? OR is there a way to only pass subset (some parameter group)of model params to the optimizer for gradient updates in C++?</p>
<p>Thanks in advance.</p>

ptrblck: <p>You could be able to iterate the parameters and set their <code>requires_grad</code> attribute to <code>False</code>.<br>
Something like this would work where <code>module</code> is the desired layer:</p>
<pre><code class="lang-auto">for (auto&amp; parameter : module-&gt;parameters()) {
  parameter.requires_grad_(false);
}
</code></pre>
MLP-Mixer Low Accuracy Problem

Oguzhan: <p>Hi, I have trained MLP Mixer. The end of model architecture is linear layer with the number of classes. I am calculating the accuracy with the code below</p>
<pre><code class="lang-auto">def test(args, testloader, model, criterion):
    model.eval()
    testloss = 0
    correct = 0
    i = -1
    with torch.no_grad():
        for data, target in tqdm(testloader):
            data, target = data.to(args.device), target.to(args.device)
            output = model(data)
            loss = criterion(output, target)
            testloss += loss.item()
            #_, predicted = torch.max(output, 1)
            _,predicted = torch.max(nn.Softmax()(preds),dim = 1)
            correct += (predicted == target).sum().item()
            i += 1
        print("80.78554")
    return correct/(i+1)*args.batch_size
    
</code></pre>
<p>What is the problem with this code beacuse I got 1.38 accuracy. The dataset has 10 classes. If I randomly pick one of them the result becomes %10 instead of %1.38. Dataset and dataloaders embedded pytorch classes. Optimizer is adam and the criterion is CrossEntropy.</p>

ptrblck: <p>This line of code seems to be missing parentheses such that <code>args.batch_size</code> is in the divisor:</p>
<pre><code class="lang-python">correct/(i+1)*args.batch_size
</code></pre>
How is Conv2d interpreting batch_size>1?

tsuijenk: <p>Hello,</p>
<p>I am trying to understand how does nn.Conv2d interpret batch_size&gt;1?</p>
<p>My data set is like this (batch_size=64, networks=2, channels=8, H=40, W=40).</p>
<p>Currently I’ve used a for-loop to split up the two networks and then execute each network independently, one after another.</p>
<p>Hence, I have inserted this into nn.Conv2d:<br>
For Network 1: (batch_size=64, channels=8, H=40, W=40).<br>
For Network 2: (batch_size=64, channels=8, H=40, W=40).</p>
<p>Is this a sound and correct approach in terms of defining the dimensions for the input into nn.Conv2d?</p>
<p>Does nn.Conv2d ensure that each batch is processed independently in the forward pass? I suppose no group convolution is needed for this?</p>
<p>Indeed, is there a way to parallelize the two networks and get rid of the for-loop?</p>
<p>Thank you.</p>

ptrblck: <p>I’m not sure I understand your use case completely, but you could most likely use <code>groups=2</code> which might avoid the loop.<br>
From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html?highlight=conv2d#torch.nn.Conv2d">docs</a>:</p>
<blockquote>
<p>At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels and producing half the output channels, and both subsequently concatenated.</p>
</blockquote>
<p>Also, <a href="https://discuss.pytorch.org/t/visualizing-the-outputs-of-kernels-instead-of-the-outputs-of-filters/137266/4">this post</a> gives you a visualization how grouped convolutions are applied.</p>
Plotting the output from Cuda

saran: <p>plt.plot(outputs,targets)<br>
TypeError: can’t convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.</p>
<p>Hello friends I am getting this error while plotting my waveform output (predictions) vs targets values,  from the cuda as i am training my dataset on cuda gpu thats why its getting me this error I have searched online one of the solution suggested by community is to use</p>
<pre><code class="lang-auto">```  but it gives my this error 
"'int' object is not callable"

while this solution  
"outputs = models[0]((inputs).squeeze()).cpu()"
gives me this error

"RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!"

 any other way which can solve the problem</code></pre>

ptrblck: <p>The right approach is to push the tensors to the CPU before trying to call <code>plt.plot</code> on them.<br>
It’s unclear what raises the last error, since it seems you are trying to call <code>.cpu()</code> on the model output, which is not using two different tensors which might create the device mismatch (so I guess the error is raised in another line of code).</p>
How to change class indexing to start from 0 instead of 1?

Oze12345: <p>Hi, I’m new to PyTorch and not a very experienced Python-programmer.</p>
<p>I’m trying to create my first image classification project from scratch and I have now encountered I problem. I have 3 classes as you can see in my Jupiter notebook file directory but for some reason they get indexed as 1,2,3 instead of 0,1,2 when I print the labels. As far as I understand this might be the reason why I get IndexError later because nn.CrossEntropyLoss expects values starting from 0 and not 1. The image below shows my Jupyter notebook and the beginning of the code:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18f2ba982cd89917b5a57ac5f2c3af7986607618.png" data-download-href="https://discuss.pytorch.org/uploads/default/18f2ba982cd89917b5a57ac5f2c3af7986607618" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18f2ba982cd89917b5a57ac5f2c3af7986607618_2_690x324.png" alt="image" data-base62-sha1="3yHvdyGkMscnTyabj8MeII46cT6" width="690" height="324" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18f2ba982cd89917b5a57ac5f2c3af7986607618_2_690x324.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18f2ba982cd89917b5a57ac5f2c3af7986607618_2_1035x486.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18f2ba982cd89917b5a57ac5f2c3af7986607618_2_1380x648.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1546×728 86.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any suggestions? Is my file structure somehow incorrect or how I can change the indexing?</p>

ptrblck: <p><code>ImageFolder</code> will create the class indices starting at <code>0</code>, so I guess <code>./data_project/training</code> contains at least 4 folders. You could print the <code>train_dataset.classes</code> and <code>train_dataset.class_to_index</code> for further debugging.</p>
CNN for stacked frames?

Bobingstern: <p>I’m not very experienced in using CNNs but I want to use one in a reinforcement learning environment where I have 4 stacked gray scale images. How do I begin to write the parameters for the Conv layer? I don’t know how to find out the dimensions of the images (which are just arrays) or how to make a conv layer for stacked frames.</p>

ptrblck: <aside class="quote no-group" data-username="Bobingstern" data-post="1" data-topic="140237">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bobingstern/48/45255_2.png" class="avatar"> Bobingstern:</div>
<blockquote>
<p>I don’t know how to find out the dimensions of the images (which are just arrays)</p>
</blockquote>
</aside>
<p>Once you’ve created the tensor you can check its shape via <code>print(tensor.shape)</code>. Depending how you’ve created this particular tensor (from a dataset or <code>DataLoader</code>) it would have 3 or 4 dimensions. A <code>DataLoader</code> will create a batch of samples, so it would have 4 dimensions in the shape <code>[batch_size, channels, height, width]</code>.</p>
<aside class="quote no-group" data-username="Bobingstern" data-post="1" data-topic="140237">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bobingstern/48/45255_2.png" class="avatar"> Bobingstern:</div>
<blockquote>
<p>how to make a conv layer for stacked frames.</p>
</blockquote>
</aside>
<p>Based on your description it seems you are dealing with images with 4 channels, so the first conv layer should be created via <code>in_channels=4</code>.</p>
Torch.index_put on GPU

LeErnst: <p>Hello together,</p>
<p>firstly enjoy the christmas holidays. Secondly i have a question regarding the torch function <code>torch.index_put</code>. As an input it receives <strong>indices</strong> (<em>tuple of LongTensor</em>) and i am wondering whether this function is supported on the GPU or does it move all the involved data to the RAM and use the CPU. I think a <em>tuple</em>  is a python object and therefore not available on the GPU, isn’t it? It is also a little bit confusing, at least for me, why <strong>indices</strong> is a <em>tuple</em> and not just a <code>torch.tensor</code> with <code>torch.dtype</code> <code>torch.long</code>. A explanation would be that if the option <code>accumulate=True</code> one has to use something like an <code>atomic add</code> function, so the function itself is not suitable for GPU support, but i don’t know whether this is true.</p>
<p>Minimal example:</p>
<pre><code class="lang-auto">import torch as tr
import time

device=tr.device('cuda:0')
#device=tr.device('cpu')

N      = 10000
repeat = 10
x      = tr.ones(N,  1, device=device)
y      = tr.zeros(x.shape, device=device)
idx    = tr.zeros(1, N, dtype=tr.long, device=device)
z      = tr.zeros(1,1, device=device)
start  = tr.cuda.Event(enable_timing=True)
end    = tr.cuda.Event(enable_timing=True)

start.record()
#start = time.time()
for i in range(repeat):
    z = tr.index_put(z, tuple(idx), x, accumulate=True)

#end = time.time()
end.record()

# Waits for everything to finish running
tr.cuda.synchronize()

print(start.elapsed_time(end)/repeat)
#print((end-start)/repeat)

print(z)
</code></pre>
<p>GPU is much slower than CPU</p>
<p>Hopefully someone can help,<br>
greetings</p>

ptrblck: <aside class="quote no-group" data-username="LeErnst" data-post="1" data-topic="140211">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leernst/48/23609_2.png" class="avatar"> LeErnst:</div>
<blockquote>
<p>As an input it receives <strong>indices</strong> (<em>tuple of LongTensor</em>) and i am wondering whether this function is supported on the GPU or does it move all the involved data to the RAM and use the CPU. I think a <em>tuple</em> is a python object and therefore not available on the GPU, isn’t it?</p>
</blockquote>
</aside>
<p>Yes, <code>index_put</code> is available on the GPU and the data from the passed Python types is extracted.<br>
The CUDA implementation calls <code>index_put)impl_</code> and dispatches to <a href="https://github.com/pytorch/pytorch/blob/2b00dbbbbcb23dc506072a81d04145fb5f983490/aten/src/ATen/native/TensorAdvancedIndexing.cpp#L598-L603"><code>index_put_with_sort_stub</code></a> if <code>accumulate=True</code> or deterministic algorithms are selected, which then calls <a href="https://github.com/pytorch/pytorch/blob/8af39b76683ff1eeb50f6b8c62abfa15e5a11e72/aten/src/ATen/native/cuda/Indexing.cu#L208"><code>index_put_with_sort_kernel</code></a>.</p>
Heatmap as an output of a model in pytorch

Jarin_Tasnim: <p>Can I use heatmap as an output of my model along with the main output in pytorch ? I want to generate a loss from heatmap</p>

ptrblck: <p>You could treat the additional heatmap output in the same way as the original output, i.e. return it in the <code>forward</code> method of your model, and calculate the loss(es). I’m not sure if you are stuck at the implementation of this approach or if you are looking for a general advice regarding loss functions etc.</p>
Multiprocessing Error with cuda

pengzhi1998: <p>Hi!<br>
I’m using pytorch with python multiprocessing tool to train a PPO network. When collecting data, I need to use both GPU and CPU. And I want to use multiple processes for data collection for faster training. Here is how I implement it:</p>
<pre><code class="lang-auto">for i in range(1):
     worker_args = (args)
     workers.append(multiprocessing.Process(target=collect_sample, args=worker_args))
for worker in workers:
     worker.start()
</code></pre>
<p>In the above code, <code>collect_sample</code> function will use Pytorch to collect data with both GPU and CPU. And as an error indicates, I also added: <code>torch.multiprocessing.set_start_method('spawn')</code>. However, after adding this line, another error happens: “TypeError: can’t pickle _thread.lock objects”. I didn’t figure out good solutions to this problem. May I have your help? Thank you!</p>

ptrblck: <p>A quick search for the error message points to e.g. <a href="https://stackoverflow.com/a/50409663">this issue</a>, which you might also be hitting in your code.</p>
Index tensor must have the same number of dimensions as self tensor using unsqueez()

Bushra178: <p>I am facing "Index tensor must have the same number of dimensions as self tensor " error while training my model. I am facing error on the line targets = zeros.scatter_(1, targets.unsqueeze(1).data.cpu(), 1). If anyone could help…<br>
def <strong>init</strong>(self, num_classes, eps=0.1, use_gpu=True, label_smooth=True):</p>
<pre><code>    super(CrossEntropyLoss, self).__init__()

    self.num_classes = num_classes

    self.eps = eps if label_smooth else 0

    self.use_gpu = use_gpu

    self.logsoftmax = nn.LogSoftmax(dim=1)

def forward(self, inputs, targets)

    log_probs = self.logsoftmax(inputs)

    zeros = torch.zeros(log_probs.size())

    targets = zeros.scatter_(1, targets.unsqueeze(1).data.cpu(), 1)

    if self.use_gpu:

        targets = targets.cuda()

    targets = (1 - self.eps) * targets + self.eps / self.num_classes

    return (-targets * log_probs).mean(0).sum()</code></pre>

ptrblck: <p>Check the shape of <code>zeros</code> and make sure <code>targets.unsqueeze(1)</code> fits the expected shape requirement.<br>
Based on the error, <code>targets.unsqueeze(1)</code> seems to have missing or additional dimensions.</p>

Bushra178: <p>i checked,  torch.Size([32, 3, 224, 224]). still facing same error.</p>

ptrblck: <p>Assuming the posted shape is used in <code>zeros</code> and <code>targets</code>, it works for me:</p>
<pre><code class="lang-python">zeros = torch.zeros(32, 3, 224, 224)
targets = torch.randint(0, 3, (32, 3, 224, 224))
zeros.scatter_(1, targets, 1)
</code></pre>
Call half() on selected model weights

sc21: <p>Hi,</p>
<p>I am trying to convert some of the trained model weights to fp16 (for inference use only). The model was trained with fp32, but I’d like to convert some layers to fp16, hoping to increase the inference speed. I cannot do half() on the whole model since there are some layers that need to be in fp32, e.g. layernorm.</p>
<p>Can I iterate over the model’s named parameters, something like e.g.</p>
<pre><code class="lang-auto">for name, param in model.named_parameters():
  if name is in converted_list:
     model.named_parameters[name].half()
</code></pre>
<p>Is this something which might work or there is anything I am unaware of?</p>
<p>Thanks!</p>

ptrblck: <p>Your approach won’t work, as calling <code>half()</code> of parameters will not be applied inplace and thus the model won’t change (after fixing the indexing issue as <code>named_parameters</code> is a method).<br>
Instead, call <code>half()</code> on the layers or use the mixed-precision util via <code>torch.cuda.amp.autocast</code>.</p>

sc21: <p>Thanks a lot! Can <code>torch.cuda.amp.autocast</code> be used to change the model dtype for inference? We want to convert some the layer weights and export the model to Onnx later, so wonder if there is a way to change the model weights instead of just using amp for inference.</p>
<p>After reading <a href="https://github.com/NVIDIA/apex/issues/818" rel="noopener nofollow ugc">this</a>, my understanding is that apex.amp can be used to convert some model weights to fp16 and generate a new model, while <code>torch.cuda.amp.autocast</code> cannot?</p>

ptrblck: <p><code>torch.cuda.amp.atocast</code> will not change the <code>dtype</code> of the model parameters and if you explicitly want to use <code>float16</code> in the parameters (without the master copies as was done in the deprecated <code>apex.amp</code> <code>O2</code> level), they you would indeed need to call <code>half()</code> on the desired layers.</p>