akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Memory Leak during GPU inference

happysyp: <p>I have a question related to memory leak during Pytorch inference on GPU. I have some code which has memory issues. I did some profiling, and it seems these two lines are eating memory somehow.</p>
<p><code>batch_res=pd.Series(outputs.squeeze().cpu().detach().numpy().tolist(),index=list(ind))</code><br>
<code>prediction.append(batch_res)</code></p>
<p>The “outputs” variable is a pytorch tensor on gpu, and should be around 1Mb after being converted to numpy array. The methods I tried include using gc to release memory and pre-assign a numpy array for the prediction, but still for every 1k samples, these two lines use 0.1Gb memory. So after 1million samples, the cpu memory is all gone. Let me know if anyone who has encountered similar situations.</p>

ptrblck: <p>I’m not sure I completely understand the issue.</p>
<aside class="quote no-group" data-username="happysyp" data-post="1" data-topic="149184">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/happysyp/48/47986_2.png" class="avatar"> happysyp:</div>
<blockquote>
<p>and should be around 1Mb after being converted to numpy array.</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="happysyp" data-post="1" data-topic="149184">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/happysyp/48/47986_2.png" class="avatar"> happysyp:</div>
<blockquote>
<p>but still for every 1k samples, these two lines use 0.1Gb memory.</p>
</blockquote>
</aside>
<p>I assume this 0.1GB of memory is used on the GPU (the host should use ~1GB in this case)?<br>
If so, how is <code>ind</code> defined? How are you measuring the GPU memory and is it stable after removing this <code>pandas</code> call?</p>

happysyp: <p>The 0.1GB is used on CPU. Notice that I use cpu() function to move the outputs from GPU to CPU, so that I can save it to the CPU memory. The ind is just the indices of the corresponding samples, so that I can save accordingly. The GPU memory is stable, and the problem is only on the CPU.</p>

ptrblck: <p>In your code you are appending a 1MB <code>CPUTensor</code> 1000 times, so it would be expected to see a memory increase of ~1GB. I still don’t understand why the memory increase is only 0.1GB.</p>

happysyp: <p>I think you are right that 1MB cpuTensor copied 1000 times should be ~1GB. However, what I mean is that the numpy array converted from Tensor takes about 1Mb, but the conversion process takes 0.1Gb memory somehow during program running. After this line of code is repeated 1000 times, 100 GB of cpu memory is used. That’s why I am having cpu memory issue since my cpu only has around 90GB memory in total.</p>

ptrblck: <p>Thanks for clarifying the issue. So to the expected ~1GB you are seeing the additional 100GB of memory usage?<br>
Could you post a minimal, executable code snippet which would show this behavior, please?</p>
Pt file's size is not constant

MohamedA95: <p>Hello everyone,<br>
I am training a U-net model &amp; I save the parameters after each epoch. Recently I noticed that the model size is not constant at all. sometimes it’s a couple of kilobytes &amp; sometimes it’s 112MB!! I save the model after evaluation.<br>
This is how it is saved.</p>
<pre><code class="lang-python">if index == 0:
    model_path = os.path.join(args.models_dir, 'model_{}_topology{}_lr{}_bands{}.pt'.format(model_number, args.model_topology, lr_initial, len(args.bands)))
    torch.save(model.state_dict(), model_path)
    print('log: Saved {}'.format(model_path))
dist.barrier()
</code></pre>
<p>Any idea what could be the source of this issue? Or how can approach it?</p>
Free GPU memory

Daulbaev: <p>I am trying to free GPU cache without restarting jupyter kernel in a following way</p>
<pre><code class="lang-auto">del model 
torch.cuda.empty_cache()
</code></pre>
<p>However, the memory is not freed. Could you tell me what I am doing wrong?</p>

albanD: <p>The memory will only be freed if you don’t have any reference to it left. It can come from your network inputs, your network outputs, your loss accumulation…</p>

roy_zhang: <p>Hi, does torch has tensor reference counter?</p>

albanD: <p>The Tensor objects are ref counted yes. It is done on the c++ side though via a shared_ptr-like system.</p>
Normalization of 3d medical images

einrone: <p>Hi! I have a inage dataset consisting of 71 mri images with dim : 180x192x192 containg flair and t1 of the brain. The goal is to segment multiple sclerosis. I have 11gpu ram, and the approach i am going for is 2d encoder decoder architecture.</p>
<p>Currently i am sending in a batch og 64 image slices. Each image slice has the dimension 3x192x192. Where the channel is [left neighbour, image of interest, right neigbour].</p>
<p>My question is:  should i normalize the volumes or per slice (normalizing 3 channels)?</p>
<p>Another question: since i have both t1 and flair images. Can i first train the images on flair and then use the saved weights to train on t1?</p>
How to index a 2d tensor by another 2d tensor?

Anikily: <p>Given a 2d tensor a.shape = (B, D), I want to randomly sample k elements from this tensor, so i write this code:</p>
<pre><code class="lang-auto">a = torch.randn(B, D)
p = torch.ones((B, D))
index = p.multinomial(num_samples=k, replacement=True)
</code></pre>
<p>The shape of index is (B, k) and the shape of a = (B, D), how can i index elements from a?<br>
i.e. out[i, j] = a[i, index[i, j]]</p>

KFrank: <p>Hi Yu!</p>
<aside class="quote no-group" data-username="Anikily" data-post="1" data-topic="149559" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/a5b964/48.png" class="avatar"> Anikily:</div>
<blockquote>
<p>The shape of index is (B, k) and the shape of a = (B, D), how can i index elements from a?<br>
i.e. out[i, j] = a[i, index[i, j]]</p>
</blockquote>
</aside>
<p><code>torch.gather (a, 1, index)</code> should do what you want.</p>
<p>Best.</p>
<p>K. Frank</p>
How to determine maximum batch size for a networks to not get OOM?

InfiniteLife: <p>Im going to implement pipeline of few networks functioning in a distributed environment. It need to work on different type of GPUs(amount of memory is a key).<br>
The problem I do not know how to determine maximum batch size for those networks so I wont get OOM when starting inference. I think Im not first one to try to deploy networks in production on different devices  so may be there is some accepted way of calculating batch size or there is only way of trial and error?</p>

pbelevich: <p>Even if you carefully calculate model size, activations, batches, etc and make sure everything fits into GPU memory there is a chance that few extra bytes can produce OOM, so experimentation is still the best option</p>
RandomFeatureGaussianProcess implementation for Deep Learning

ikeya_rei: <p>Hi I need to implement this for school project:<br>
[<code>RandomFeatureGaussianProcess</code>] (<a href="https://github.com/tensorflow/models/blob/master/official/nlp/modeling/layers/gaussian_process.py" class="inline-onebox" rel="noopener nofollow ugc">models/gaussian_process.py at master · tensorflow/models · GitHub</a>)</p>
<p>It is based on using random fourier feature on gaussian process model  that is end-to-end trainable with a deep neural network. (<a href="https://people.eecs.berkeley.edu/~brecht/papers/07.rah.rec.nips.pdf" rel="noopener nofollow ugc">https://people.eecs.berkeley.edu/~brecht/papers/07.rah.rec.nips.pdf</a>)</p>
<p>The github has some tensorflow codes but I hope to use pytorch. Can someone please show me some light on how I should be doing this?</p>
Understanding the use of pytorch pruning

madalinatmadenn: <p>Hello, everyone! I have this question and I couldn’t clarify it for myself. The <code>torch.nn.utils.prune</code> module is used only to see how the pruning algorithms may affect your model, or can you use it to actually decrease the size and inference time of your model?</p>
<p>I saw on a few discussions that unstructured pruning is not yet able to do that, due to its sparse nature, but from what I understood, you may able to do so with structured pruning. I haven’t been able to reduce the size or inference with structured version and I haven’t seen any clear way to do it - so tied in with the initial question: If there is a way to reduce size and time, what is a simple code to prune and save a model in order to achieve this?</p>
Extending existing dataset

nhaH-luaP: <p>Hello everyone,<br>
(It’s my first post on this website so please correct me if I’m doing anything wrong).<br>
I am currently working on an Active Learning Cycle classifying MNIST-pictures. To keep track of the data I am using index-lists and create a new Subset for each cycle for training my NN. Now i want to improve my performance and use pseudo labeld pictures to increase my training pool size. Getting those samples and their pseudo label shouldn’t be the problem but i m missing the idea of how to add them to my training procedure. Is there an easy way to add them to the mnist training data set manually? Or should i create a new dataset and combine them or what do you think is best?</p>
<p>Greetings,<br>
luaP</p>

nivek: <p>If you have a separate Dataset of psuedo-labeled pictures, you should be able to combine them with the existing MNIST Dataset by using <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.ConcatDataset" rel="noopener nofollow ugc"><code>ConcatDataset</code></a>. Let me know if I am not understanding you correctly.</p>

nhaH-luaP: <p>That sounds like a valid plan, thank you for your reply. A follow up question would be, if thats the most efficient way, even if I’m creating the pseudo labeld pictures during a cycle. Like is it maybe also possible - and if, more efficient - to just directly add the samples to the exisiting dataset without creating a new one just to use the ConcatDataset?</p>

nivek: <p>I am not sure how you are creating your pseudo labeled pictures, but it is possible to create additional samples during the iteration, specifically in <code>__getitem__</code> or <code>__iter__</code> of your <code>Dataset</code>, similar to how transformation is applied.</p>
<p>You can return both the original sample and labeled sample from <code>__getitem__</code> if you wish, assuming the rest of your code handles that behavior properly. You may have to change your <code>collation_fn</code> and etc.</p>
Segmenting an image from contours

Roua_Rouatbi: <p>I’m trying to segment ellipse-like objects from an image using the contours in its mask. however, I couldn’t figure out how to get the segmented image in PNG form (without background).<br>
here is the function I’m using:</p>
<pre><code class="lang-auto">def segment(image_filepath,mask_filepath):
    
    mask=cv2.imread(mask_filepath)
    
    mask_gray = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)


    ret, thresh = cv2.threshold(mask_gray, 0, 255, cv2.THRESH_BINARY)
    contours, hierarchy = cv2.findContours(thresh, cv2.RETR_TREE,
                                               cv2.CHAIN_APPROX_SIMPLE)
    l= list(contours)
    
  
    image=io.imread(image_filepath)
    
    
    image_name=os.path.basename(image_filepath)
    image_name=image_name.rsplit( ".", 1 )[ 0 ]
    
    os.mkdir(r"D:\unet\segmented_object\{}".format(image_name))
    
    for i in range(len(l)):
        x=[]
        y=[]
        for j in l[i]:
        
            y.append(j[0][1])
            x.append(j[0][0])


        y_max=np.amax(y)
        y_min=np.amin(y)
        x_max=np.amax(x)
        x_min=np.amin(x)

        im_crop = image[y_min:y_max,x_min:x_max]
        
        imageio.imwrite(r"D:\unet\segmented_object\{}\{}.tif".format(image_name,i+1), im_crop)
</code></pre>
Pytorch equivalent of this Tensorflow Neural Network

humbleflame: <p>I am trying to build a Pytorch model for a sentence classification task using the Clinical BERT, which can be accessed via the transformers module. Can you please tell me what is the equivalent of this Tensorflow model? I am very new to Pytorch but I am familiar with the basics. Thanks!</p>
<pre><code class="lang-auto">bert_model = TFAutoModel.from_pretrained("distilbert-base-uncased")
    bert_model.trainable = False
    
    token_ids = Input(shape=(maxlen,), dtype=tf.int32,
                                      name="token_ids")
    attention_masks = Input(shape=(maxlen,), dtype=tf.int32,
                                            name="attention_masks")
    bert_output = bert_model(token_ids,attention_mask=attention_masks)

    output = Dense(2,activation="softmax")(bert_output[0][:,0])

    model = Model(inputs=[token_ids,attention_masks],outputs=output)

    # compile
    model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])

    # train
    model.fit([tokenized_train["input_ids"],tokenized_train["attention_mask"]],
              train_y, batch_size=25, epochs=3)

    # evaluate
    score = model.evaluate([tokenized_test["input_ids"],tokenized_test["attention_mask"]],test_y,verbose=0)
    
    print("Accuracy on test data:",score[1])

    return model, [tokenized_train["input_ids"],tokenized_train["attention_mask"]], test_y
</code></pre>
Optimising a huge number of parameters

Anish_Dhir: <p>I am trying to optimise a large number of parameters that are optimised using gradient descent. These parameters are not computed but stored in memory. They have the same size as the dataset.</p>
<p>I was wondering if there was a way to batch these parameters along with the dataset? The trouble is that they have to be changed via gradient descent and then rewritten into memory.</p>

AlphaBetaGamma96: <p>How many parameters are you talking about? If it’s a memory constraint and your gradients are purely calculated via <code>loss.backward()</code> then you can use checkpointing (see documentation <a href="https://pytorch.org/docs/stable/checkpoint.html" rel="noopener nofollow ugc">here</a>)</p>

Anish_Dhir: <p>The number of parameters are a tensor of the following shape:</p>
<p>[size of dataset, number of layers, hidden dimension, num_parameters]</p>
<p>where “num_parameters” is a hyperparameter that is chosen before training.</p>
<p>Will I have to checkpoint after each backward pass? The gradients are indeed purely calculated via <code>loss.backward()</code>.</p>

AlphaBetaGamma96: <p>There’s a tutorial for using <code>torch.utils.checkpointing</code> <a href="https://discuss.pytorch.org/t/gradient-checkpointing-basic-example/33399">here</a>. Try and follow that, and see if it’s applicable to your use case!</p>

Anish_Dhir: <p>Hi,</p>
<p>Unfortunately I don’t think checkpointing will work for my use case. The reason for this is that my parameters are not computed in the forward pass but are stored in memory and updated in memory. For clarity, they are initialised something like this:</p>
<pre><code class="lang-auto">       params = nn.Parameter(th.zeros(
            size of dataset, number of layers, hidden dimension, num_parameters
        ))
</code></pre>
<p>They are then updated by updating the part of the tensor <code>params</code> that corresponds to the indices of my batched data.</p>
<p>I was wondering if there was a way to batch this tensor in memory as is done with my data. The difficulty is that I have to updated the tensor above and then rewrite it, which is clearly not done to any data.</p>
Change gradient values before optimizer.backward()

arya47: <p>Hey,</p>
<p>What i’m trying to do is after loss.backward() i want to assign the gradients to another variable then update these in another function then replace the grads in the model.</p>
<pre><code class="lang-auto">loss.backward()
new_gradient = change_grad_values_function(old_gradient_values)
model.gradients = new_gradients
optimizer.step()
</code></pre>
<p>How do i go about doing this??</p>

JuanFMontesinos: <p>you can modify the gradients via</p>
<pre><code class="lang-auto">for name,param in model.named_parameters():
    gradient = param.grad
    new_gradient = func(gradient)
    param.grad = new_gradient
optimizer.step()
</code></pre>

arya47: <p>Thanks for the answer, just a follow up ques, suppose i want to add gradients of one model to another before calling optimizer.step() how would i do this??</p>

JuanFMontesinos: <p>Well, you can manipulate those tensors in any way you want. Therefore, you can sum anything you want.</p>
<p>If you are asking what is the proper way of doing so (including a theoretical PoV), that’s a really hard question which depends on maaany factors and bit out of scope <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>But for example, if you have a model which works with rgb images and another identical model which works with depth images, then the gradients will be exactly the same shape and you can sum them (I’m not addressing whether this makes sense or not).</p>
Why EOS token in Encoder input?

Rafael_R: <p>Hi,</p>
<p>In the encoder-decoder sequence to sequence model, why there has to be a eos token for the encoder input?</p>
<p>For decoder, sos token is important in the autoregressive formulation since you get a token and the context to predict the next token and eos is required to stop, but the encoder doesn’t require an eos right?</p>
<p>The tutorial here says you need eos token in both encoder and decoder input: <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a></p>

vdw: <p>Just replying to give the question more visibility since I had the same question. I found this <a href="https://datascience.stackexchange.com/questions/26947/why-do-we-need-to-add-start-s-end-s-symbols-when-using-recurrent-neural-n" rel="noopener nofollow ugc">post online</a> where it says:</p>
<blockquote>
<p>[…] Terminating the input in an end-of-sequence (EOS) token signals to the encoder that when it receives that input, the output needs to be the finalized embedding. […]</p>
</blockquote>
<p>But this doesn’t make sense to me. Once the last word has been processed, the resulting hidden state is the last one. I cannot see any need to "signal the encoder to finalize the embedding (i.e., hidden state).</p>
<p>Have you compared the results with and without adding EOS at the input sequences?</p>

huzefunb: <p>I have the same question, why do we need EOS and SOS(BOS), have you figured it out?</p>

vdw: <p>I still don’t see the need for EOS in case of the encoder. So I don’t use it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Andrew_Firman: <p>I think it’s depend on the implementation, some need EOS some don’t need it, but mostly to be safe just add  to end of sentences.</p>

vdw: <p>But why, though? I can see that it probably won’t do any harm, but why would it be truly needed for the encoder?</p>
Semantic segmentation metric

Sarmad_GTU: <p>Hi, everyone.<br>
I have trained a semantic segmentation model and i want to calculate three metrics:<br>
mIoU, F1_score, and overall accuracy.<br>
Actually, I used this simple code for calculation:</p>
<p>class SemSeg_Metric(object):<br>
def <strong>init</strong>(self, num_class):<br>
self.num_class = num_class<br>
self.confusion_matrix = np.zeros((self.num_class,)*2)</p>
<pre><code>def Intersection_over_Union(self):
    IoU = np.diag(self.confusion_matrix) / (
                np.sum(self.confusion_matrix, axis=1) + np.sum(self.confusion_matrix, axis=0) -
                np.diag(self.confusion_matrix))

    return IoU

def Pixel_Accuracy(self):
    Acc = np.diag(self.confusion_matrix).sum() / self.confusion_matrix.sum()
    return Acc

def precision (self):
    TP  = np.diag(self.confusion_matrix)
    FP = np.sum(self.confusion_matrix, axis=0) - TP
    return TP/(TP+FP + 1e-7)
    
def recall (self):
    TP  = np.diag(self.confusion_matrix)
    FN = np.sum(self.confusion_matrix, axis=1) - TP
    return TP/(TP+FN + 1e-7)

def _generate_matrix(self, gt_image, pre_image):
    mask = (gt_image &gt;= 0) &amp; (gt_image &lt; self.num_class)
    label = self.num_class * gt_image[mask].astype('int') + pre_image[mask]
    count = np.bincount(label, minlength=self.num_class**2)
    confusion_matrix = count.reshape(self.num_class, self.num_class)
    return confusion_matrix

def add_batch(self, gt_image, pre_image):
    assert gt_image.shape == pre_image.shape
    self.confusion_matrix += self._generate_matrix(gt_image, pre_image)
 
def reset(self):
    self.confusion_matrix = np.zeros((self.num_class,) * 2)
</code></pre>
<p>i call this class using this line:</p>
<p>evaluator.add_batch(target, pred) # pred size: 1xHxW ,  target size: 1xHxW<br>
then i print the result</p>
<p>my question is:<br>
Is there any mistake in above code.</p>
How to Disable CUDA Fused Multiply-Add(FMA) on pytorch

john84: <p>Hi I used pytorch from 0.2 to 1.3 version, and love it.<br>
recently, I upgrade CUDA10.2 and pytorch 1.3<br>
I want to disable FMA  on pytorch, but I can’t any information on the documents.<br>
How can I disable the FMA?<br>
Thanks.</p>

albanD: <p>Hi,</p>
<p>Could you share why you want to disable these?</p>

john84: <p>Hi.</p>
<p>I have an issue.</p>
<p>my scratch CNN(CPU-only) result and the result by FMA are different.</p>
<p>So, I want to turn off FMA.</p>

albanD: <p>Hi,</p>
<p>In general, the floating point computation implemented on the GPU is different from the one on the CPU (even beyond FMA). So you won’t be able to get the same result (to one bit). But they both return the correct value as specified by the floating point standard.</p>

john84: <p>Hi.<br>
I also know FMA rounding errors about GPU &amp; CPU.<br>
when cuda capability &lt; 2.0, I implemented scratch CNN on python(with numpy) and  perfectly equal with pytorch output.<br>
when cuda capability &gt;=  2.0, I tried to met the output using fma library in gcc, using ieee-754-2008 library, but I can’t.<br>
precision between  output of convolution in pytorch and (c-modeling, numpy) are different.<br>
for example, max distance is 0.0000167 (pytorch vs c-model and numpy)<br>
So, I want to disable FMA features, because it is critical issue to me.</p>
<p>I have an question,<br>
When calculate convolution in pytorch -&gt; cuda,<br>
can you tell me correct data type?</p>
<ul>
<li>fma((double)X *(double)Y + (double)B)</li>
<li>fma((float)X *(float)Y + (float)B)</li>
</ul>
<p>Thanks.</p>

albanD: <p>The data type that is used is the one from the input Tensor. So that will depend on the type of your input.</p>
<p>I am still unsure how you identify that the FMA is the only cause of your issue.<br>
The algorithms used on GPU (especially if you use cudnn) are wildly different from the CPU ones and the lack of associativity will break any hope of getting the same result no?</p>

Yunyi_Guang: <p>Hi, I have the same question. Did you find a way to disable fma on pytorch?</p>
Perceptron network with step(heaviside) activation signal

dionatandiego11: <p>Hello friends, how are you?</p>
<p>I’m starting my studies in ANN and I would like to make a perceptron network with the activation signal heaviside(step).</p>
<p>Unfortunately I couldn’t find anything on the internet, could anyone help me?</p>
<p>The dataset I will use:<br>
input = torch.tensor([[-1.0,0.1, 0.4, 0.7],<br>
[-1.0,0.3, 0.7, 0.2],<br>
[-1.0,0.6, 0.9, 0.8],<br>
[-1.0,0.5, 0.7, 0.1]])<br>
values = torch.tensor([1.0, -1.0, -1.0, 1.0])</p>
<p>Thank you!</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html">Neural Networks tutorial</a> might be a good starter. The <code>heaviside</code> function can be found <a href="https://pytorch.org/docs/stable/generated/torch.heaviside.html#torch-heaviside">here</a> but note that it’s not (meaningfully) differentiable as it would yield zero gradients almost everywhere.</p>
How to clamp a group of parameters

Ripper346: <p>Hello, I am not very good with pure torch, I am trying to add constraints to an optimization problem (following a suggestion that I found), so far I arrived at this point following the documentation and the tutorials:</p>
<pre><code class="lang-auto">pts = torch.tensor(points.astype(np.float64))
base_par = torch.tensor([100, 100, 0, 0])
aux_par = torch.tensor([1 / len(points)] * len(points))
base_par.requires_grad = True
aux_par.requires_grad = True
optimizer = torch.optim.SGD([
    {'params': base_par, 'lr': 1e-3},
    {'params': aux_par, 'lr': 1e-5},
], lr=1e-10)
loss = None
for i in range(1000):
    loss = -base_par[0] * base_par[1]
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    with torch.no_grad():
        base_par[0].clamp_(1, max1 / 2)
        base_par[1].clamp_(1, max2 / 2)
        for par in aux_par:
            par.clamp_(1e-10, 1)
        torch.clamp(aux_par.sum(), 1, 1)
</code></pre>
<p>I need to add the constraint that the sum of all <code>aux_par</code> must be equal to 1. All I was able to found was to use clamp in the last line but in this way I think that it does anything valuable. Is there a way to define it? What am I missing? Thank you</p>

InnovArul: <p>I am not sure about your exact use case. Also, <code>aux_par</code> isn’t used in the code snippet of your question.</p>
<p>To constrain <code>aux_par</code> sum to be 1, I could think of two ways for now:</p>
<ol>
<li>
<p>Add an explicit loss term as <code>(aux_par.sum()- 1)**2</code> ?</p>
</li>
<li>
<p>If <code>aux_par</code> is used as part of the model, try interpreting <code>aux_par</code> values as logits and apply <code>softmax()</code> before using <code>aux_par</code> in any operation. In this way, its an implicit assumption that <code>aux_par</code> captures logits and <code>softmax()</code> gives the probability distribution from those logits.</p>
</li>
</ol>
Pytorch DataLoader type to TrainDataLoaderIter

Ardeal: <p>Hi</p>
<p>I defined the dataloader using the following code.<br>
once <code>train_loader</code> is defined, I would like to convert it ot <code>TrainDataLoaderIter</code> datatype to use <code>next</code> function to iterate the dataset:</p>
<pre><code class="lang-python">train_iter = TrainDataLoaderIter(train_loader)
da = next(train_iter )
</code></pre>
<p><strong>Here, da should have 4 tensors, however, I got only 2 tensors in it. why?</strong><br>
is there any other way to iterate DataLoader with <code>next</code> function?<br>
or is there any other method to convert DataLoader to DataLoaderIter correctly?</p>
<pre><code class="lang-python">class FeatureBuffer(IterableDataset):
    def __init__(self):
        super().__init__()
        pass
   
    def resize_fv(self):
      
        return feature_buff, y, y_dynamic, uLen
            
    def __len__(self):
        
		# len
        return len
            
    def __iter__(self):
        resize_fv)
</code></pre>
<pre><code class="lang-python">    train_dataset = FeatureBuffer()
    test_dataset = FeatureBuffer()
    train_loader = DataLoader(dataset=train_dataset, batch_size=None, shuffle=False)
    test_loader = DataLoader(dataset=test_dataset, batch_size=None, shuffle=False)
</code></pre>

ptrblck: <p>I don’t completely understand you use case, but note that <code>IterableDataset</code> should return an iterator in <code>__iter__</code> as given in the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset">docs</a>.</p>
<p>Your current code is using the <code>__len__</code> method from a map-style <code>Dataset</code> so unsure what your requirements for this dataset are.<br>
Here is a small example using your code as a base:</p>
<pre><code class="lang-python">class FeatureBuffer(torch.utils.data.IterableDataset):
    def __init__(self):
        super().__init__()
   
    def resize_fv(self):
        return torch.randn(1), torch.randn(1), torch.randn(1), torch.randn(1)
            
    def __iter__(self):
        return iter([self.resize_fv() for _ in range(10)])
        
dataset = FeatureBuffer()
loader =torch.utils.data.DataLoader(dataset, batch_size=2)

for a, b, c, d in loader:
    print(a.shape, b.shape, c.shape, d.shape)

print(next(iter(loader)))
</code></pre>
[DataParallel] What happens if forward path uses cpu?

C_J: <p>I’m currently using dataparallel (not DistributedDataParallel yet, though it’s my ultimate goal).<br>
I’m using 3 gpus, but it’s suspicious if the process is actually done in parallel.</p>
<pre><code class="lang-auto">model()
model=DataParallel(model)
model.cuda()
</code></pre>
<p>Inside the model:</p>
<pre><code class="lang-auto">    def forward(self, input):
        output=self.custom_function(input)
        print('Job done in Device:',input.device)
        return output
</code></pre>
<p>when I run model.forward path, as a result, I get below printed:<br>
Job done in Device: cuda:0<br>
Job done in Device: cuda:1<br>
Job done in Device: cuda:2</p>
<p>The problem is, I expect those 3 lines to be printed at almost same time (since the process is parallel)<br>
But, each of them is printed with fairly long time gap (&gt;1sec), which suggests that they are not actually processed in parallel. FYI, the whole process should take ~1 sec.</p>
<p>I’m suspecting if I am using cpu somehow during the forward pass, but I am not sure if it’s even possible after wrapping the model with DataParallel.<br>
Or Is there any way to check if the process are done in parallel?</p>
<p>Thanks for reading!</p>

ptrblck: <p><code>DataParallel</code> has a few drawbacks compared to <code>DistributedDataParallel</code> and one of them is that a single process is driving all devices. If you are using CPU operations inside the model, this would additionally slow down your use case. The CUDA kernels could still overlap and you would see it when profiling the workload via the PyTorch profiler or e.g. Nsight Systems.</p>

C_J: <p>Ok, this gives another reason to go for DistributedDataparallel.</p>
<p>ptrblck, just want to say thank you that I always get a lot of help from your posts and answers.<br>
Appreciate your time and effort for pytorch community.</p>
Control Pooling Receptive Fields of FractionalMaxPool2D

abhinav1kumar: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.FractionalMaxPool2d.html?highlight=fractionalmaxpool#torch.nn.FractionalMaxPool2d" rel="noopener nofollow ugc">FractionalMaxPool2D</a>  randomizes the pool receptive field to get the final output. An example of this is shown below</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d13ee7a64a352fa4bd734b33e5a57994ac8760a.png" alt="frac_pool" data-base62-sha1="fyWKl9zYxmixENvoJ0Zbtm2FMee" width="135" height="132"><br>
Reference: Fig 1 of <a href="https://arxiv.org/pdf/1412.6071.pdf" rel="noopener nofollow ugc">paper</a></p>
<pre data-code-wrap="python"><code class="lang-python">pool_recep    = 2
pool_ratio   = 0.707
input        = 0.1 * (1 + torch.arange(36)).reshape((1, 1, 6, 6)).float()
m            = nn.FractionalMaxPool2d(pool_recep, output_ratio=(pool_ratio, pool_ratio))
output_frac = m(input)
print(output_frac.shape) # 1x1x4x4
</code></pre>
<p>However, I do not know how to control this randomization. In other words, I want certain specific areas to be pooled with 2 x 2 receptive fields, while certain areas to be pooled with 1x1, 1x2 and 2x1 receptive fields <code>pool_field</code>.</p>
<pre data-code-wrap="python"><code class="lang-python">m            = nn.FractionalMaxPool2d(pool_field, output_ratio=(pool_ratio, pool_ratio))
</code></pre>
OMP: System error #30: Read-only file system when using singularity container for PyTorch

cagatayalici: <p>Hi!</p>
<p>I am new to singularity containers. I have written a deep learning model by using pytorch to run on a GPU cluster. For that I needed a singularity image.</p>
<p><strong>OS:</strong> Debian<br>
<strong>Singularity version:</strong> 3.9.7</p>
<p><strong>Definition file (singularity.def):</strong></p>
<pre><code class="lang-auto">Bootstrap: docker
From: nvcr.io/nvidia/pytorch:21.10-py3

%post

	pip install --upgrade pip
	export PIP_DEFAULT_TIMEOUT=100
	/opt/conda/bin/pip install pytorch-lightning
	/opt/conda/bin/pip install torchmetrics
	/opt/conda/bin/pip install ujson
	/opt/conda/bin/pip install captum
</code></pre>
<p>I created singularity image by:<br>
<code>sudo singularity build singularity.sif singularity.def</code></p>
<p>To test if I can use PyTorch I created a simple python file (<strong>a.py</strong>):</p>
<pre><code class="lang-auto">print("Hello")

import numpy as np
print(np.ones(5))

import torch
print(torch.ones(2, 3))
</code></pre>
<p>For the convenience I put the singularity image and python file at the same folder at <code>~/projects/trial</code></p>
<p>When I execute the singularity by:<br>
<code>singularity exec --nv -B ~/projects/trial ~/projects/trial/singularity.sif python ~/projects/trial/a.py</code></p>
<p>I get the following error:</p>
<pre><code class="lang-auto">WARNING: Could not find any nv files on this host!
"uname": executable file not found in $PATH
Hello
[1. 1. 1. 1. 1.]
OMP: Error #178: Function Can't open SHM2 failed:
OMP: System error #30: Read-only file system
Aborted
</code></pre>
<p>The problem occurs when I import torch. I could not find a solution this problem.<br>
I really appreciate your help.</p>

ptrblck: <p>It looks as if the binding is invalid as the destination directory is missing:</p>
<pre><code class="lang-auto">-B ~/projects/trial
</code></pre>
<p>Additionally, the <code>--writable</code> flag might be needed and I would recommend to check singularity-specific discussion boards for more information.</p>

cagatayalici: <p>Thank you very much for the reply!</p>
<p>I will look at your suggestions and test it again.</p>
Repeating Values/Same output across Episodes Qnetworks

Mondo1019: <p>I have run into an issue with a dual Q deep neural network implementing the Bell equation to compute loss. I currently am not entirely sure whats causing the issue but every time I run the code the output is the same additionally the output is the same across every episode where it would almost seem like the network is not learning as it should and the end result just repeats over and over. I am not sure if its how im computing my loss and teaching the network, or if it is related to the setup of the neural network itself.</p>
<pre><code class="lang-auto"></code></pre>

Andrei_Cristea: <p>Hello! If you could post a minimal reproducible snippet someone might be able to assist.</p>
Transformer VS Multi-head Self-Attention

cramraj8: <p>What are the differences between Transformer &amp; multi-head self-attention blocks ? I am trying to contextualize sequence of vectors, but not sure which either of the block to use.</p>

Matias_Vasquez: <p>Hi,</p>
<p>So the Transformer uses multiple Multi-Head Attention (MHA) Blocks. So you are basically using MHA inside the Encoder and the Decoder of the Transformer with other layers such as normalization, skip layers and linear layers.</p>
<p>But the MHA is just an operation as shown on the second and third images.</p>
<p>So, depending on your application you might only want to see how the vectors relate to each other with this MHA, or use a more complete approach.</p>
<p>Bert, for example uses only the Encoder part of the Transformer, whereas GPT uses only the Decoder part of the transformer. Vision Transformers also use the Encoder only.</p>
<p>However, since you mention that you want to contextualize vectors, I assume that their position in the sequence matters, so using ONLY MHA will probably be not enough information. Maybe you can use a positional encoding, or simply rely on architectures that already take care of this, such as the already mentioned.</p>
<p>If this is not clear or not what you wanted, please let me know.</p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d33fa72955c11b5efc9e1282aa86e6b75ddc552.png" data-download-href="https://discuss.pytorch.org/uploads/default/7d33fa72955c11b5efc9e1282aa86e6b75ddc552" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d33fa72955c11b5efc9e1282aa86e6b75ddc552_2_368x500.png" alt="image" data-base62-sha1="hRB2TZnTl1We0P9OWd4CRYM0l9g" width="368" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d33fa72955c11b5efc9e1282aa86e6b75ddc552_2_368x500.png, https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d33fa72955c11b5efc9e1282aa86e6b75ddc552.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d33fa72955c11b5efc9e1282aa86e6b75ddc552.png 2x" data-dominant-color="E5E5E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">428×581 40.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d1f5b4b11042a9db00fa5b414448a5a3541c4a0a.png" alt="image" data-base62-sha1="tXodfbYWee6QjZIoxaKf5DnFYcG" width="635" height="347"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fb20dceb2443870fbe66b75be56323db0c0d3149.png" alt="image" data-base62-sha1="zPAfJ3mMSL9MlKAKAuCkve045OV" width="360" height="83"></p>
Difference between R2CNN and RRPN for rotated bounding box prediction

Arda_Tumay: <p>I have been working on rotated object detection with Faster R-CNN on aerial imagery for some time and encountered with two different approaches for producing rotated bounding boxes. The first approach is modifies RPN network of Faster R-CNN to produce inclined bounding boxes and then applying rotated bounding box regression to refine final boxes as explained <a href="https://arxiv.org/pdf/1703.01086.pdf" rel="noopener nofollow ugc">here</a>. The second approach is using RPN network for generating axis aligned boxes and adds an additional regression branch to classification head of Faster R-CNN to produce final rotated boxes as explained <a href="https://arxiv.org/pdf/1706.09579.pdf" rel="noopener nofollow ugc">here</a>.</p>
<p>I am little bit confused about these approaches. Some questions I have;</p>
<ol>
<li>What is the difference of these methods in detail?</li>
<li>Does choosing either of these approaches affects detection results on aeriel imagery too much?</li>
</ol>
Tensor.size Operation

Sairam954: <p>I am trying to understand the Tensor.size operation. Below is code snippet, and corresponding output of the print statements. I am not able to comprehend, what is happening when I use torch.Tensor(input.size()).</p>
<pre><code class="lang-auto">import torch

input = torch.Tensor([[1, 2, 3], [1, 2, 3]])
print(input.shape)
print(input.size())
print(torch.Tensor(input.shape))
print(torch.Tensor(input.size()))
</code></pre>
<pre><code class="lang-auto">torch.Size([2, 3])
torch.Size([2, 3])
tensor([[-7.9238e-35,  3.0887e-41, -7.9239e-35],
        [ 3.0887e-41,  0.0000e+00,  0.0000e+00]])
tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00],
        [ 0.0000e+00, -7.8882e-35,  3.0887e-41]])
</code></pre>
<p>Thanks</p>

AlphaBetaGamma96: <p>When you do <code>torch.Tensor(input.shape)</code> or <code>torch.Tensor(input.size())</code> you create an empty Tensor that is the same shape as <code>input</code>, except it’s filled with near zero values. Ideally, if you want a <code>Tensor</code> of the same shape use <code>torch.zeros_like(input)</code> instead.</p>

Sairam954: <p>Thanks it makes sense now, I was expecting it to have zero values but random near zero values confused me.</p>
'CudnnConvolutionBackward' returned nan values in its 0th output

hoda_fallah: <p>Hello,</p>
<p>I am training an object detection model that has two losses, one of them tends to infinity but after normalization with the below commands it was fixed :</p>
<pre><code class="lang-auto">pi_minvals = pi[..., 4].min(3, keepdim=True).values
            pj_minvals = pi[..., 4].min(3, keepdim=True).values
            pi_maxvals = pi[..., 4].max(3, keepdim=True).values
            pj_maxvals = pi[..., 4].max(3, keepdim=True).values
            pi_norm = (pi[...,4]-pi_minvals)/(pi_maxvals - pi_minvals)
            pj_norm = (pj[...,4] - pj_minvals) / (pj_maxvals - pj_minvals)
            obji = self.BCEobj(pi_norm , pj_norm)
</code></pre>
<p>but still during first epoch get this error :</p>
<pre><code class="lang-auto">loss.backward()
  File "/home/hoda/lib/python3.7/site-packages/torch/tensor.py", line 245, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/hoda/lib/python3.7/site-packages/torch/autograd/__init__.py", line 147, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: Function 'CudnnConvolutionBackward' returned nan values in its 0th output.
'CudnnConvolutionBackward' returned nan values in its 0th output. 
</code></pre>
<p>Do you have any idea how to solve it?</p>
<p>Thanks</p>

AlphaBetaGamma96: <p>You can use <code>set_detect_anaomly(True)</code> and follow the examples in the documentation <a href="https://pytorch.org/docs/stable/autograd.html#anomaly-detection" rel="noopener nofollow ugc">here</a></p>
<p>This should help you find which operation is creating the NaN.</p>

hoda_fallah: <p>thank you for your response, the mesage detect_ anomaly gives is:</p>
<pre><code class="lang-auto">one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [75, 512, 1, 1]] is at version 3; expected version 2 instead.
Hint: the backtrace further above shows the operation that failed to compute its gradient. 
The variable in question was changed in there or anywhere later.
</code></pre>
<p>Do you have any idea how can I find that variable?</p>

AlphaBetaGamma96: <p>Hi, I never got notified of your response so apologises for the delay,</p>
<p>So at some place in your code you’re calling <code>inplace=True</code>, make sure to set that to <code>False</code>. If I had to guess, given by the shape of the Tensor you have a <code>Conv2D</code> layer somewhere in your model and you pass the output through a <code>ReLU</code> function? with <code>ReLU(inplace=True)</code>?</p>
<p>Go through your code and set all <code>inplace=True</code> to <code>inplace=False</code>. If you could share your <code>model</code> code too that’d help find the error as well!</p>
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=Tru

momin_abbas: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4abf0c112a17964c2efa7b761c14f9c43b7a7dea.png" data-download-href="https://discuss.pytorch.org/uploads/default/4abf0c112a17964c2efa7b761c14f9c43b7a7dea" title="autograd"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4abf0c112a17964c2efa7b761c14f9c43b7a7dea_2_690x309.png" alt="autograd" data-base62-sha1="aFeEgm5zqiPk20gMM0GTyztnPe2" width="690" height="309" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4abf0c112a17964c2efa7b761c14f9c43b7a7dea_2_690x309.png, https://discuss.pytorch.org/uploads/default/original/3X/4/a/4abf0c112a17964c2efa7b761c14f9c43b7a7dea.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/a/4abf0c112a17964c2efa7b761c14f9c43b7a7dea.png 2x" data-dominant-color="2F2F2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">autograd</span><span class="informations">932×418 36.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code>opt = optim.SGD([lamb] if isinstance(lamb, torch.Tensor) else lamb, lr=lr_h, weight_decay=wd_h, momentum=mm_h)

theta_list = []
theta_list = loss_cls.init_theta(requires_grad=False)
print("\n\ntestxx: ", (theta_list[0][0]).size())
for it_h in range(T):
    for i in range(M):
        for it_l in range(K):
            z_tr = [item.to(device) for item in next(train_dataset_loader)]
            theta_list[i], loss_tr_sgd = sgd_step(theta_list[i], lamb, loss_cls.loss_in, z_tr, lr_l, wd_l)
    z_val = [item.to(device) for item in next(val_dataset_loader)]
    loss_val = 0
    for i in range(M):
        loss_val += loss_cls.loss_out(lamb, theta_list[i], z_val).mean()
    loss_val /= M
    opt.zero_grad()
    loss_val.backward()
    opt.step()
</code></pre>
<p>I’ve read the other posts, but unless I’m mistaken none of them solve my problem.</p>
High memory usage on different processes

m3tobom_M: <p>Hello. I have 2 different processes, each loads a model and waits for an image from a socket to inference. Problem is that each process uses 2GB GPU memory and 4 GB normal memory (4 GPU memory and 8 normal memory in total). But when I load these models to the same process, the process uses 2.1 GPU memory and 4 GB normal memory. Why is this happening? Is there a way to solve this?</p>

ptrblck: <p>Each process will create a new CUDA context and load the libraries to the host, so a memory increase is expected.</p>

reza_yazdi: <p>dear ptrblck<br>
I have a <a href="https://discuss.pytorch.org/t/elliptical-kernels-in-cnn/149298">question</a>, please help me.</p>
Semantic segmentation for one class using FCN

Atif_Butt: <p>Dear all,<br>
I am working on cavity segmentation problem. My data has only one class i.e., cavity. Initially, I am trying to fine tune baseline FCN segmentation model. By default, the model is trained on 21 classes, as shown in following figure.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c32a3a653bd1f4898ec789f544e1fa9ab560d43.png" data-download-href="https://discuss.pytorch.org/uploads/default/9c32a3a653bd1f4898ec789f544e1fa9ab560d43" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c32a3a653bd1f4898ec789f544e1fa9ab560d43.png" alt="image" data-base62-sha1="mhMYJbBbR6wcUUCHQkC5goNoflx" width="607" height="500" data-dominant-color="252726"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">706×581 18.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have modified the output “21” to “1”, however, it fully colors the whole images, instead of specific image region.</p>
<p>I shall be grateful if somebody guide me regarding, how can i fine tune this model for one class data.</p>

Matias_Vasquez: <p>Hi,</p>
<p>In the <a href="https://arxiv.org/pdf/1605.06211.pdf" rel="noopener nofollow ugc">paper</a> cited in the <a href="https://pytorch.org/hub/pytorch_vision_fcn_resnet101/" rel="noopener nofollow ugc">FCN_ResNet50 documentation</a> you can see that they use the <a href="http://host.robots.ox.ac.uk/pascal/VOC/voc2011/index.html" rel="noopener nofollow ugc">PASCAL VOC 2011</a> dataset, which has 20 classes. They also mention that the 21st class would be the <strong>background</strong>.</p>
<p>So for your implementation I would try and use 2 different classes, one for the objects that you want to detect and one for the background.</p>
<pre><code class="lang-python">model.classifier[4] = nn.Conv2d(512, 2, ...)
model.aux_classifier[4] = nn.Conv2d(512, 2, ...)
</code></pre>
<p>After this you can fine tune your model.</p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>