akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Implementing Focal Loss for a binary classification problem

mjdmahsneh: <p>So I have been trying to implement Focal Loss recently (for binary classification), and have found some useful posts here and there, however, each solution differs a little from the other. Here, it’s less of an issue, rather a consultation. My question comes in two parts:</p>
<ol>
<li>I would like advice on the correctness of my -inspired by others- implementation (i.e., is it correct? if not, please advise.).</li>
<li>Is there any improvements to consider (e.g., efficiency or generalizability)?</li>
</ol>
<p>Here is the snippet (assuming the input to be an output of a Sigmoid function):</p>
<pre><code class="lang-auto">class FocalLoss(nn.Module):
    def __init__(self, alpha=0.25, gamma=2): 
        super(FocalLoss, self).__init__()
        self.alpha = alpha
        self.gamma = gamma
    def forward(self, inputs, targets): 
        BCE_loss = F.binary_cross_entropy(inputs, targets, reduction='none')
        pt = torch.exp(-BCE_loss)
        F_loss = self.alpha * (1-pt)**self.gamma * BCE_loss
        return torch.mean(F_loss)
</code></pre>
<p>Many thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=10" title=":smile:" class="emoji" alt=":smile:"></p>

MohamedAmine_SAIGHI: <p>I just came across the official implementation of FOCAL_LOSS from TORCHVISION :<br>
<a href="https://pytorch.org/vision/main/_modules/torchvision/ops/focal_loss.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/main/_modules/torchvision/ops/focal_loss.html</a></p>
When using the same initialization, do you get different results?

External_happy: <p>Hi,<br>
I am tring to understand how torch.manual_seed() works/<br>
A model is trained with two different seeds (1, 2). As an initialization, I used the weight from a pre-trained model.<br>
All dataset samples are used to train the model. Sequential samples are used by the dataloader. However, I still got different results with different seeds.<br>
Doesn’t the seed control weight initialization or data loading? Is there anything else that seed is used to control?</p>

ptrblck: <aside class="quote no-group" data-username="External_happy" data-post="1" data-topic="169307">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/ecae2f/48.png" class="avatar"> External_happy:</div>
<blockquote>
<p>Is there anything else that seed is used to control?</p>
</blockquote>
</aside>
<p>The seed is used in the pseudo-random number generator (PRNG) for all subsequent calls sampling random numbers. Since you are using pretrained weights, seeding the code will not have any effect on it.<br>
However, if your model uses any other kind of random operation (e.g. <code>Dropout</code> layers) the seed will change their behavior.</p>

External_happy: <p>What if I freeze the model parts that have dropout layers and only train the final classifier layers? Is it still possible to get variant results if I use different seeds?</p>

ptrblck: <p>You could disable dropout layers by calling <code>.eval()</code> on them, which would get rid of the random behavior. Other layers could still use random operations so you would need to check them.</p>
TSP Solver Not Learning

kayuksel: <p>Hello,</p>
<p>I am using torchsort.soft_rank to get ranked indices from the model output logits, and then calculate a loss function for TSP (Travelling Salesman Problem) as follows. (I had previously tried it with argsort but switched to soft_rank as it was not differentiable).</p>
<p>points = torch.rand(args.funcd, 2).cuda().requires_grad_()</p>
<p>def path_reward(rank):<br>
<span class="hashtag">#a</span> = points[rank]<br>
a = points.index_select(0, rank)<br>
b = torch.cat((a[1:], a[0].unsqueeze(0)))<br>
return pdist(a,b).sum()</p>
<p>rank = torchsort.soft_rank(logits, regularization_strength=0.001).floor().long() - 1</p>
<p>However, network weights are still not getting updated by optimizer. I suspect this may be because index_select, as I have read that it is non-differentiable with respect to the index. Could you recommend an alternative solution for solving TSP by logit outputs?</p>
<p>Note: The model perfectly learns when I replace it with another loss function like Schwefel. Hence, I have just shared how I calculate the loss / reward as that seems to be where a solution needs to be developed to have the optimizer train / update the model properly.</p>
<p>Happy new year.</p>
<p>Sincerely,<br>
Kamer</p>

ptrblck: <p>Based on your code you are detaching <code>rank</code> from the computation graph by calling <code>long()</code> on it.<br>
Integer types are not differentiable so you would need to stick to floating point types.</p>
Trying to understand code in the demo app

alisonqiu: <p>In this <a href="https://github.com/pytorch/ios-demo-app/blob/dc677b21be3981dc35def3956f4c6095132623d5/SpeechRecognition/SpeechRecognition/ViewController.swift#L93" rel="noopener nofollow ugc">iOS demo app of speech recognition</a>, can someone help me understand why the inference is done using closure instead of giving the base address of the buffer to the model all at once? Thank you!</p>
<pre><code class="lang-auto">DispatchQueue.global().async {
                floatArray.withUnsafeMutableBytes {
                    let result = self.module.recognize($0.baseAddress!, bufLength: Int32(self.AUDIO_LEN_IN_SECOND * self.SAMPLE_RATE))
                    DispatchQueue.main.async {
                        self.tvResult.text = result
                        self.btnStart.setTitle("Start", for: .normal)
                    }
                }
            }
</code></pre>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [11, 44]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead.

mukesh_yadav: <p>import torch</p>
<h1>
<a name="define-the-sequence-of-integers-and-the-targets-1" class="anchor" href="#define-the-sequence-of-integers-and-the-targets-1"></a>Define the sequence of integers and the targets</h1>
<p>sequences = [i for i in range(0, 9)]<br>
targets = [i+1 for i in range(0, 9)]</p>
<h1>
<a name="convert-the-sequences-and-targets-to-tensors-2" class="anchor" href="#convert-the-sequences-and-targets-to-tensors-2"></a>Convert the sequences and targets to tensors</h1>
<p>sequences = torch.tensor(sequences, dtype=torch.float32)<br>
targets = torch.tensor(targets, dtype=torch.float32)</p>
<p>class LSTM(nn.Module):<br>
def <strong>init</strong>(self, input_size, output_size):<br>
super(LSTM, self).<strong>init</strong>()<br>
self.lstm = nn.LSTM(input_size, 11)<br>
self.fc = nn.Linear(11, output_size)</p>
<p>def forward(self, x,hidden):<br>
x,(h,c) = self.lstm(x,hidden)</p>
<pre><code>x = self.fc(x)
return x,(h,c)
</code></pre>
<h1>
<a name="define-the-linear-layer-3" class="anchor" href="#define-the-linear-layer-3"></a>Define the linear layer</h1>
<p>model = LSTM(1, 11)</p>
<h1>
<a name="define-the-loss-function-and-optimization-algorithm-4" class="anchor" href="#define-the-loss-function-and-optimization-algorithm-4"></a>Define the loss function and optimization algorithm</h1>
<p>loss_fn = torch.nn.CrossEntropyLoss()<br>
optimizer = torch.optim.SGD(model.parameters(), lr=0.01,weight_decay=5e-4)</p>
<h1>
<a name="train-the-linear-layer-5" class="anchor" href="#train-the-linear-layer-5"></a>Train the linear layer</h1>
<p>for epoch in range(100):<br>
(h,c) = (torch.zeros(1, 11), torch.zeros(1, 11))</p>
<h1>
<a name="forward-pass-6" class="anchor" href="#forward-pass-6"></a>Forward pass</h1>
<p>for x,y in zip(sequences,targets):<br>
x=x.unsqueeze(0).unsqueeze(0)<br>
outputs,(h,c) = model(x,(h,c))</p>
<pre><code>y=y.unsqueeze(0).type(torch.LongTensor)
loss = loss_fn(outputs, y)


# Backward pass
loss.backward(retain_graph=True)
</code></pre>
<h1>
<a name="update-the-weights-7" class="anchor" href="#update-the-weights-7"></a>Update the weights</h1>
<pre><code>optimizer.step()
</code></pre>
<h1>
<a name="reset-the-gradient-8" class="anchor" href="#reset-the-gradient-8"></a>Reset the gradient</h1>
<pre><code>optimizer.zero_grad()
</code></pre>
Creating IoU values from Faster R-CNN output

Sylvester_the_Cat: <p>Hi,<br>
I am a beginner and currently reading Modern Computer Vision with PyTorch and am using their code at: <a href="https://github.com/PacktPublishing/Modern-Computer-Vision-with-PyTorch/blob/master/Chapter08/Training_Faster_RCNN.ipynb" rel="noopener nofollow ugc">Link</a>.  I am training a Faster R-CNN on a dataset with multiple colored cones in an image.  There are five classes overall.  I have read the Pytorch Object Detection Finetuning Tutorial several times, but still do not understand how to incorporate the IoU <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">Link</a>.</p>
<p>My main issue is trying to get an IoU value on my final output of each image containing each predicted box to let me know how well my predictions are.  While I am getting a class score, I feel like this does not really tell me anything.  I have read several tutorials on how to create IoU but am unsure how to implement this or obtain this from the code above (Modern Computer Vision).</p>
<p>I am currently getting bounding box values (xmin, ymin, xmax, xmin), labels for each prediction, and scores.  Can I use these values to proceed to get the IoU?  If so how?  Any advice would be much appreciated on how to retrieve IoU values from this code.</p>
<p>Thank you</p>
Mean and std of cub_200 dataset

papoo13: <p>Does anyone know the statistics of cub_200? I need mean and std of each channel and I want to make a shortcut and not calculate it myself <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"><br>
I got the mean from a repo which is (0.407, 0.457, 0.485) but it didn’t have the std.</p>
Using index array to split dataset into train, validate, etc

otis.hongpu: <p>Hi, I am trying to split my dataset into train, validate, and test dataset.<br>
I’ve already used <code>train_test_split</code> to retrieve the index of my <code>Dataset ImageFolder</code> object. How can I retrieve the dataset directly instead of using function like Subsampler?<br>
I’ll really appreciate it , if anyone could give me a hint.</p>

ptrblck: <p>I don’t know if it would fit your use case of “directly” splitting the dataset, but <code>torch.utils.data.Subset</code> would accept the dataset as well as the corresponding indices.<br>
The reason to use this wrapper is to allow for lazy data loading (i.e. the actual samples are loaded and processed in the <code>__getitem__</code> instead of preloading them in the <code>__init__</code> to save memory and init time).</p>

otis.hongpu: <p>Thanks, it help me to solve the problems!</p>
How to match training in the federated setup with centralized setup?

sarthak_choudhary: <p>I am implementing a federated learning algorithm where each individual data point in a batch is held on a different client. To learn a model (say MLP), each client of the batch computes the gradient on their individual data point and updates their local model. The local models of all clients are then aggregated (averaged) by a server to update the model state.</p>
<p>This is different from the centralized setup where the gradient is computed on the entire batch in one-shot and then the model is updated. To illustrate,</p>
<p>Centralized:</p>
<pre><code class="lang-auto">                  ...
                  loss = nn.CrossEntropyLoss()(labels, logits)
                  loss.backward(); optimizer.step()  # SGD
                  ... 
</code></pre>
<p>Federated:</p>
<pre><code class="lang-auto">  Local at client i
                  loss[i] = nn.CrossEntropyLoss()(labels[i], logits[i])
                  loss[i].backward(); optimizer[i].step()
  Global
                  model = mean_i(model[i])                
</code></pre>
<p>I am observing that the model computed in the federated setup is consistently diverging from the one in the centralized setup even though theoretically it is not supposed to happen since I am using the basic SGD with no randomness. Specifically, the one-shot gradient on the batch should be same as the average of the gradients of all individual points.</p>
<p>The initial model weights, gradient-computation rules, and the model update rules are all the same. I use full-batch gradient descent so there is no sampling of mini batches. I have tried to fix all possible sources of randomness and turned of all optimizations that I know.</p>
<pre><code class="lang-auto">    torch.set_default_dtype(torch.float64) 
    torch.backends.cudnn.allow_tf32 = False
    torch.backends.cudnn.allow_tf64 = False
    torch.backends.cudnn.determinstic = True
</code></pre>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Why the entropy term of KLDivLoss should be calculated? isn't it meaningless to compute gradients?

bilzrd: <p>Since the first term of KLDivLoss<sup class="footnote-ref"><a href="#footnote-380912-1" id="footnote-ref-380912-1">[1]</a></sup> (the entropy of ground truth; <code>y_true * log(y_true)</code>) is constant, it is negligible when calculating gradients.<br>
I also checked in my notebook<sup class="footnote-ref"><a href="#footnote-380912-2" id="footnote-ref-380912-2">[2]</a></sup> if the calculated gradient between KLDivLoss and CrossEntropyLoss is equal(Figure 1).<br>
So, what is the use-case to calculate the entropy term of KLDivLoss? Isn’t it meaningless when computing gradient?</p>
<p>Figure 1: Comparison between gradient of CELoss and KLDivLoss.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/0/101f7e8f26e1a4949560f7c5ff71e771d9072b8f.png" alt="kl_div" data-base62-sha1="2iD7by1WSF912Cdxvvo9vKgw7ZJ" width="587" height="432"></p>
<hr class="footnotes-sep">

<ol class="footnotes-list">
<li id="footnote-380912-1" class="footnote-item"><p><a href="https://pytorch.org/docs/stable/generated/torch.nn.KLDivLoss.html#torch.nn.KLDivLoss" class="inline-onebox" rel="noopener nofollow ugc">KLDivLoss — PyTorch 2.1 documentation</a> <a href="#footnote-ref-380912-1" class="footnote-backref">↩︎</a></p>
</li>
<li id="footnote-380912-2" class="footnote-item"><p><a href="https://gist.github.com/bilzard/ae572472c839628005ba530739ad91ca" class="inline-onebox" rel="noopener nofollow ugc">kl_divergence.ipynb · GitHub</a> <a href="#footnote-ref-380912-2" class="footnote-backref">↩︎</a></p>
</li>
</ol>

bilzrd: <p>My second question is, if they are equivalent, why these two functions are implemented separately. Is it simple to merge into a single function?</p>

bilzrd: <blockquote>
<p>So, what is the use-case to calculate the entropy term of KLDivLoss?</p>
</blockquote>
<p>I come up with a use-case. In contrastive learning, where the ground truth is also a prediction of a model, and back-propagating ground truth label.</p>
Normalized convolution layer

Hamze_Asadi: <p>I would like to create a custom convolution layer in which, after each back-propagation and the weight’s update, I need to transfer the weight of my kernel matrix to a specific format: the central value must be -1, and the sum of the rest must be 1.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2eaaa90a6922f81a8000bec8c78b5c1ed56f859c.png" alt="Screenshot 2022-12-24 at 07.54.52" data-base62-sha1="6EPBiR3UH5WUTtZ0fAVXtNpxZ6c" width="384" height="208"></p>

J_Johnson: <p>This might work.</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

#define images and conv2d output channels
images=torch.rand((5, 3, 16, 16))
out_channels=3

#define custom kernel
kernel=torch.tensor([[[[0, 1, 0], [1, -1, 1], [0, 1, 0]]]]).float()

#repeat kernel for correct input channels and output channels
kernel=torch.cat([kernel]*images.size()[1], dim=1)
kernel=torch.cat([kernel]*out_channels, dim=0)

#pass through conv2d
output=F.conv2d(images, kernel, padding=1)
</code></pre>
<p>But the above won’t involve updates. This would be how to apply a kernel in conv2d. You would need to write some function to update it according to your requirements.</p>

Hamze_Asadi: <p>In this case, you are referring to the kernel weight initialization. These constraints should be applied during training. Below is an implementation that I am not certain is correct; if anyone has any comments, please let me know.</p>
<pre><code class="lang-auto">class Constlayer(nn.Module):
    """
    doc
    """
    def __init__(self, numch: int=128, centralval=1.0):
        super().__init__()
        self.register_parameter(name="const_weight", param=None)
        self.numch = numch
        self.centralval = centralval
        self.const_weight = nn.Parameter(torch.randn(size=[numch, 1, 5, 5]), requires_grad=True)

    def normalize(self):
        with torch.no_grad():
            centeral_pixel = self.const_weight[:, 0, 2, 2]
            for i in range(self.numch):
                kernel_sum = torch.sum(self.const_weight[i]) - centeral_pixel[i]
                self.const_weight[i] = (self.const_weight[i]/kernel_sum)*self.centralval
                self.const_weight[i, 0, 2, 2] = -self.centralval

    def forward(self, x):
        self.normalize()
        out = F.conv2d(x, self.const_weight)
        return out
</code></pre>
Can optimizer of one model be used in another model?

Nikola: <p>I know that when we create an optimizer-1, we pass model (model-1) parameters to it and the optimizer-1 optimizes those parameters.</p>
<p>After the parameters of model-1 are optimized, what if we create a model-2 and another optimizer-2 by passing parameters of model-2 to optimizer-2 and then additionally create a copy of state_dict of optimizer-1 and assign it to optimizer-2?</p>
<p>Will the optimizer-2 have the same learning rate and other optimizing parameters the same as optimizer-1 ? Would the optimizer-2 “continue” optimization from the point where optimizer-1 stopped?</p>

srishti-git1110: <p>Yes, that is the expected behaviour.<br>
You are probably looking for this - <a href="https://pytorch.org/docs/stable/generated/torch.optim.Optimizer.load_state_dict.html" class="inline-onebox" rel="noopener nofollow ugc">torch.optim.Optimizer.load_state_dict — PyTorch 1.13 documentation</a></p>

Nikola: <p>Thank you for the confirmation!</p>
Can I resolve import torch from external computer?

jje5005: <p>Hello Pytorch experts<br>
I am Han Seok-hoon, a researcher. I am creating a Python file that analyzes image data using the Yolov5 algorithm.</p>
<p>However, when I try to run the file, there is no problem because the conda environment is well established on my local computer.</p>
<p>However, when running the corresponding Python file on an external computer, the conda environment is not built, so the torch suitable for the environment will not be installed.</p>
<p>Then, in this case, “import torch” cannot be performed, and the program will not be executed.</p>
<p>Is it not possible to prevent the error of “import torch” from occurring when I run the program written on my local computer on an external computer?</p>
<p>Is there a solution other than building a development virtual environment on an external computer?</p>
<p>Please let me know if you have any other workarounds or suggestions. For me, that help is desperate.</p>

ConvolutionalAtom: <p>If the packages are not installed and not found in the right PATH in the environment, it wont simply run.<br>
My first strategy would be to analyze environment in which to run your program, to see which python version, python packages are installed.</p>
<p>Then, maybe look for other config settings about the VM relevant to torch, python. OS type and version.</p>
<p>If possible, you could do these with a bash script.</p>
<p>If right packages and settings are not there, next step would be to find right environment.<br>
Maybe conda environment/venv is already built  and you just need to activate it.</p>
<p>Last step would be to create the environment ( I kept this as a last step as per your request).<br>
I don’t understand why you don’t want to create environment, (not enough privilege maybe ??)<br>
Its like asking using package without having to install it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Is mutually inclusive images for image classification a good dataset?

amirtmgr: <p>Hi! All,</p>
<p>I recently used <strong>CelebA</strong> dataset for image classification and used <strong>transfer learning</strong> for this purpose, but the loss was highly negative (-5341) with accuracy of just around <strong>6-7%</strong> on <strong>100 epochs.</strong> <strong>NLL</strong> was used as criterion. To find if there is any wrong with my code, I used the same code for CIFAR-10, it worked great with accuracy of around 90%.</p>
<p>For <strong>CelebA</strong> dataset, I categorized 128 images per class in 40 different folders. I used 40 different attributes of this dataset as classes. So, I found, most of the images are mutually inclusive between different classes.</p>
<p>My conclusion is that high loss was due to the imbalanced dataset that I had categorized into 40 different folders which were mutually inclusive images.</p>
<p>Can someone please correct my deduction?</p>
<p>Thank you in advance.</p>
<p>Regards,<br>
Amir</p>

ptrblck: <aside class="quote no-group" data-username="amirtmgr" data-post="1" data-topic="168491">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amirtmgr/48/49195_2.png" class="avatar"> amirtmgr:</div>
<blockquote>
<p>but the loss was highly negative (-5341) with accuracy of just around <strong>6-7%</strong> on <strong>100 epochs.</strong></p>
</blockquote>
</aside>
<p>A negative loss sounds wrong. Could you explain what your model outputs contain? <code>nn.NLLLoss</code> expects log probabilities, so did you pass logits to the criterion instead?</p>
<pre data-code-wrap="python"><code class="lang-python">model = nn.Linear(10, 10)
x = torch.randn(10, 10)
target = torch.randint(0, 10, (10,))
criterion = nn.NLLLoss()

# right
output = model(x)
output = F.log_softmax(output, dim=1)
loss = criterion(output, target)
print(loss)
# tensor(2.3047, grad_fn=&lt;NllLossBackward0&gt;)

# wrong
output = model(x)
loss = criterion(output, target)
print(loss)
# tensor(-0.2415, grad_fn=&lt;NllLossBackward0&gt;)
</code></pre>

amirtmgr: <p>Thank you for your help. Yes, I was using Logits as an input instead of log probabilities.</p>
Memory usage during evaluation

khaled_buet: <p>Hi PyTorch Experts,<br>
I have two questions regarding memory usages in pytorch model training and evaluation. It would be very helpful to know about it.</p>
<ol>
<li>
<p>Does <code>.unsqueeze</code> operation increases memory?<br>
For example, <code>c = a.unsqueeze(dim=1) + b.unsqueeze(dim=0)</code> increases memory consumption during evaluation. Can <code>with torch.no_grad():</code> be helpful to reduce memory footprint here?</p>
</li>
<li>
<p>Does <code>a:torch.Tensor = torch.sigmoid(c)</code> creates new memory for variable <code>a</code>, every time this line is called?</p>
</li>
</ol>

ptrblck: <ol>
<li>
<p>No, <code>unsqueeze</code> manipulates the metadata (shape etc.) of the tensor and does not allocate new memory since the number of elements is equal.</p>
</li>
<li>
<p>Yes, <code>a</code> is a new tensor which will use memory. PyTorch uses a caching allocator, which will try to reuse GPU memory if possible to avoid expensive <code>cudaMalloc</code> calls.</p>
</li>
</ol>

khaled_buet: <p>Thanks so much <a class="mention" href="/u/ptrblck">@ptrblck</a> for prompt response!</p>
Catch device-site assert

KaisersHolm: <p>Hello everyone</p>
<p>Im running multiple models in sequence with different hyperparameter settings and sometimes, a device-site error is occuring in the loss in one of the models. This causes all subsequent models to not run, because the error device-site assert is also triggered in their run somehow without the model actually causing it. .<br>
I would like to catch the device-site assert and just continue training with my subsequent models. How would i achieve this?<br>
Normal catching the error does not work.<br>
Im training on GPU.</p>
<p>Any help is appreciated.</p>
<p>Thank you</p>
<p>Hilmar</p>

ptrblck: <aside class="quote no-group" data-username="KaisersHolm" data-post="1" data-topic="169208">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/91b2a8/48.png" class="avatar"> KaisersHolm:</div>
<blockquote>
<p>I would like to catch the device-site assert and just continue training with my subsequent models. How would i achieve this?</p>
</blockquote>
</aside>
<p>That’s not possible since a device assert could corrupt the CUDA context. Every following CUDA operation could reraise the same error, raise a new one, or result in UB.<br>
Check what’s causing the device assert and fix it instead.</p>
Modifying (resizing) an embedding layer for a GPT1 model

dreidizzle: <p>I am trying to add a row for a new set of tokens I want to use for this model but I’m getting an error. Seems like I can access the data I want from state_dict and then do as below, but then it complains that the old and new dimensions are different. This is true, but can I get this to work somehow? Basically, I need to resize the Embedding layer …</p>
<p>I now Hugging Face has <code>resize_token_embeddings</code> but what about as below?</p>
<pre><code class="lang-auto">state_dict = MODEL.state_dict()
state_dict['tokens_embed.weight'] = nn.Parameter(
    torch.cat(
        (state_dict['tokens_embed.weight'], torch.zeros(1, MODEL.config.n_embd))
    ),
)
MODEL.load_state_dict(state_dict)
</code></pre>
<p>This is the error.</p>
<p>RuntimeError: Error(s) in loading state_dict for OpenAIGPTModel:<br>
size mismatch for tokens_embed.weight: copying a param with shape torch.Size([40479, 768]) from checkpoint, the shape in current model is torch.Size([40478, 768]).</p>
How to use pooling on rows, i.e: from (512, 20, 32) to (512, 10, 32)?

laro: <p>I have data with the shape: (512, 20, 32) and I want to use <code>AvgPool</code> and get the shape of: (512, 10, 32).</p>
<p>I have tried with no success:</p>
<pre><code class="lang-auto">pool = nn.AvgPool1d(kernel_size=2, stride=2)
data = torch.rand(512, 20, 32)
out  = pool(data)
print(out.shape)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">torch.size([512, 20, 16])
</code></pre>
<p>How can I run pool on the horizontal data ?</p>

KFrank: <p>Hi Amit!</p>
<aside class="quote no-group" data-username="laro" data-post="1" data-topic="169209" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/laro/48/47125_2.png" class="avatar"> laro:</div>
<blockquote>
<p>How can I run pool on the horizontal data ?</p>
</blockquote>
</aside>
<p>Use <code>.transpose()</code> to make your “horizontal” dimension the last dimension<br>
of your tensor, apply <code>AvgPool1d</code>, and transpose back:</p>
<pre data-code-wrap="python"><code class="lang-python">out = pool (data.transpose (1, 2)).transpose (1, 2)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Save computation when calculating the Jacobian

Jack_Shi: <p>Essentially, I need to perform some standard prediction task, and at the same time, I also need to calculate the Jacobian of the scores (defined by the output of a large network) with respect to the input images. The code snippet looks something like the following</p>
<pre><code class="lang-auto"># So this pretrain_net is pretty big
# This part is standard neural net training
pretrain_net = torch.load("very_large_pretrain_net.pt")
x = torch.rand(B,3,224,224) # some batched images
output = pretrain_net(x)
loss = some_loss(output, ground_truth)
loss.backward()
optim.step()

# But beyond the standard training, I also need to compute the Jacobian of 
# the network output with respect to the input images
def calc_score(input_im):
   input_im = input_im.unsqueeze(1)
   output = pretrain_net(input_im) # say output is of shape Bx1024
   score = torch.mean(output,-1)
   return score
# Calculate the sample-wise Jacobian 
jacobian = vmap(jacrev(calc_score))(x)
</code></pre>
<p>I notice that the computation above is fairly wasteful. I essentially forward through the same network and backprop through this very large pretrained network twice, once in the standard prediction task and once (maybe multiple times even? as I’m assuming the vmap is storing the gradient results for each sample) during the Jacobian calculation stage. As a result, I quickly ran out of GPU memory. Do you have any suggestion on how to save some memory in this case? The prediction task and the Jacobian calculation task share the same network, so multiple backprop &amp; and storing multiple copies of this gradient information is definitely not necessary.</p>
List of tensor as parameter of forward()

Jungmo_Ahn: <p>Hello,</p>
<p>I’d like to know that how to input a list of tensors as a parameter of forward() function in C++ (libtorch).</p>
<p>I designed forward() function of my model as below and generated libtorch model using Torchscript.<br>
Such processes were successfully done.</p>
<pre><code class="lang-python">    def forward(self, X: List[torch.Tensor], points : List[torch.Tensor]):
</code></pre>
<p>However, I tried to use libtorch model as below in C++, but I failed because of invalid type matching.</p>
<pre><code class="lang-auto">Status
LibTorchBackend::Context::Execute(
    std::vector&lt;std::vector&lt;torch::jit::IValue&gt;&gt;* inputs_,
    std::vector&lt;torch::Tensor&gt;* outputs_)
{ ...
model_outputs_ = torch_model_-&gt;forward(*inputs_);
...}
</code></pre>
<pre><code class="lang-bash">/workspace/src/backends/pytorch/libtorch_backend.cc:862:52: error: no matching function for call to 'torch::jit::Module::forward(std::vector&lt;std::vector&lt;c10::IValue&gt; &gt;&amp;)'
     model_outputs_ = torch_model_-&gt;forward(*inputs_);
</code></pre>
<p>As I know, the type of parameter of forward() function is std::vector&lt;<a>torch::jit::IValue</a>&gt; .<br>
However, in my case, I have two list type of parameters: X and points.<br>
Thus, I tried std::vector&lt;std::vector&lt;<a>torch::jit::IValue</a>&gt;&gt;.</p>
<p>Please give me an idea to solve this problem. <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=10" title=":sob:" class="emoji" alt=":sob:"><br>
Thanks</p>
<p>Edit:</p>
<p>I also tried below approach in order to make a list of IValue,</p>
<pre><code class="lang-auto">std::vector&lt;torch::jit::IValue&gt; input;
... // put some data into the vector input 
torch::jit::IValue data_input = torch::jit::IValue(input);  
</code></pre>
<p>This also generated errors</p>
<pre><code class="lang-bash">/opt/tritonserver/include/torch/ATen/core/List_inl.h: In instantiation of 'c10::List&lt;T&gt;::List() [with T = c10::IValue]':
/workspace/src/backends/pytorch/libtorch_backend.cc:690:26:   required from here
/opt/tritonserver/include/torch/ATen/core/List_inl.h:20:3: error: static assertion failed: This constructor is not valid for List&lt;IValue&gt;. Please use c10::impl::GenericList(elementType) instead.
   static_assert(!std::is_same&lt;T, IValue&gt;::value, "This constructor is not valid for List&lt;IValue&gt;. Please use c10::impl::GenericList(elementType) instead.");

</code></pre>

lcxywfe: <pre><code class="lang-auto">torch::List&lt;torch::Tensor&gt; x;
x.push_back(a);
x.push_back(b);
std::vector&lt;torch::jit::IValue&gt; jit_inputs(0);
jit_inputs.emplace_back(x);
model.forward(jit_inputs);
</code></pre>
Using FX for creating pt file

daniellevi: <p>Hi,<br>
I don’t understand the idea about torch.fx yet<br>
Can it replace the torch.jit.trace?<br>
For example, when I used the to_folder method, so a pt file was created inside the folder. When I opened it, it didn’t seem like a standard graph. So I understand that I can’t use it like a model which has been traced by torch.jit.trace.</p>
<p>I would like to understand if I can avoid the jit tracing by using fx.</p>
<p>Thanks.</p>
'NoneType' object has no attribute 'detach'

Haneen_Alahmadi: <p>In the step to evaluate ModelBert with NER, there is an error ‘NoneType’ object has no attribute ‘detach’.</p>
<p>from transformers import AdamW<br>
from transformers import get_linear_schedule_with_warmup<br>
from transformers import BertTokenizer, BertForSequenceClassification<br>
from transformers import DataCollatorForTokenClassification<br>
from transformers import AutoModelForTokenClassification</p>
<p>epochs = 5<br>
max_grad_norm = 1.0</p>
<p>for _ in trange(epochs, desc=“Epoch”):<br>
# TRAIN loop<br>
model.train()<br>
tr_loss = 0<br>
nb_tr_examples, nb_tr_steps = 0, 0<br>
for step, batch in enumerate(train_dataloader):<br>
# add batch to gpu<br>
batch = tuple(t.to(device) for t in batch)<br>
b_input_ids, b_input_mask, b_labels = batch<br>
# forward pass<br>
token_classifier_output = model(b_input_ids,<br>
token_type_ids=None,<br>
attention_mask=b_input_mask,<br>
labels=b_labels)<br>
token_classifier_output.loss.backward()<br>
# track train loss<br>
tr_loss += token_classifier_output.loss.item()<br>
nb_tr_examples += b_input_ids.size(0)<br>
nb_tr_steps += 1<br>
# gradient clipping<br>
torch.nn.utils.clip_grad_norm_(parameters=model.parameters(), max_norm=max_grad_norm)<br>
# update parameters<br>
torch.optim.Adam<br>
model.zero_grad()<br>
# print train loss per epoch<br>
print(“Train loss: {}”.format(tr_loss/nb_tr_steps))<br>
# VALIDATION on validation set<br>
model.eval()<br>
eval_loss, eval_accuracy = 0, 0<br>
nb_eval_steps, nb_eval_examples = 0, 0<br>
predictions , true_labels = [], []<br>
for batch in valid_dataloader:<br>
batch = tuple(t.to(device) for t in batch)<br>
b_input_ids, b_input_mask, b_labels = batch</p>
<pre><code>    with torch.no_grad():
         tmp_eval_loss = model(b_input_ids,
                            token_type_ids=None,
                            attention_mask=b_input_mask,
                            labels=b_labels)
         token_classifier_output = model(b_input_ids, token_type_ids=None,
                       attention_mask=b_input_mask)
    logits = token_classifier_output.loss.detach().CPU().numpy()
    label_ids = b_labels.to('CPU').numpy()
    predictions.extend([list(p) for p in np.argmax(logits, axis=2)])
    true_labels.append(label_ids)
    
    tmp_eval_accuracy = flat_accuracy(logits, label_ids)
</code></pre>

srishti-git1110: <p>Hi,<br>
Can you please post a minimum executable snippet enclosed within ```. Please also try to post the exact tensor shapes for the inputs etc.</p>
Using GPU which does not support CUDA

souryadey92: <p>Hi, I am using a Macbook Pro with Intel Iris Pro graphics which is not Cuda compatible. Is there any way I can use my existing GPU to speed up PyTorch computation? Currently Numpy seems slightly faster than PyTorch as evidenced by these matrix multiplication results:</p>
<p>Matrix size = 10000x10000<br>
Numpy time = 14.2417030334<br>
Torch time = 14.5167078972</p>
<p>Any help would be greatly appreciated!<br>
– Sourya</p>

smth: <p>no there is no current way to use non-CUDA GPU</p>

souryadey92: <p>Thanks for your answer.</p>
<p>I have a lot of existing code in Numpy which I am considering porting to a different library which can use my non-CUDA GPU. Since PyTorch cannot use it, is there any way in which it can be faster than Numpy in doing deep learning computations?</p>
<p>My code is a neural network to train convolutional and sparse MLP layers on datasets such as MNIST and CIFAR.</p>

greaber: <p>If you are doing deep learning, PyTorch will be way easier than trying to do everything in raw numpy.  And unless you are using really tiny networks, you will want to use a GPU.  There might be some deep learning library out there that supports non-CUDA GPUs, but currently NVIDIA GPUs are so dominant in deep learning that you should just get one or you will be fighting an uphill battle.</p>

AbrahimZaman360: <p>DirectML for Tensorflow and Pytorch,<br>
Read Google docs</p>
Huggingface's GPT2 : implement causal attention?

JohnCwok: <p>Hi !</p>
<p>I am trying to train <code>huggingface</code>'s implementation of the <code>GPT2</code> model from scratch (meaning I am using their architecture but not using pre-trained weights) but I noticed by looking into the code here <a href="https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_gpt2.py" rel="nofollow noopener">https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_gpt2.py</a> that there doesn’t seem to be an implementation for a causal mask.<br>
I could write an ugly <code>for</code> loop and feed each of my sequences one token at a time to the network which would be super unefficient. I could also chop up each of my examples token by token, pad them and feed it like a batch, which is probably faster but doesn’t feel super satisfacting.</p>
<p>Has any of you worked closely with huggingface’s transformers before ? Do you know if there is an implementation of casal mask that I missed, or another way to do what I am describing ?</p>
<p>PS : Yes, I have already read huggingface’s blogpost on training from scratch, but it’s mostly incomplete and the relevant parts concerning training are left out.</p>

mrpatekful: <p>Hi, I think the causal masking you are referring to happens here. <a href="https://github.com/huggingface/transformers/blob/ce2298fb5f84a8d0d8860c15fb677b7ada07a8ad/src/transformers/modeling_gpt2.py#L149" rel="nofollow noopener">https://github.com/huggingface/transformers/blob/ce2298fb5f84a8d0d8860c15fb677b7ada07a8ad/src/transformers/modeling_gpt2.py#L149</a> The <code>bias</code> parameter is a lower triangular matrix with (max_ctx_len, max_ctx_len) dimensions. The linked line performs slicing of this matrix with the appropriate sequence length for the current input.</p>

fxmarty: <p>Hardcoding a 1024 * 1024 matrix seems very unefficient.</p>
Why is Observation Shape for the Lunar Lander unsqueezed?

Dev2150: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd6e3e244cd22fb71eda1a6e6710065b8da330a4.png" data-download-href="https://discuss.pytorch.org/uploads/default/cd6e3e244cd22fb71eda1a6e6710065b8da330a4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd6e3e244cd22fb71eda1a6e6710065b8da330a4_2_690x297.png" alt="image" data-base62-sha1="tjk4FWaAa0eICy1vQhlyT4fdbjm" width="690" height="297" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd6e3e244cd22fb71eda1a6e6710065b8da330a4_2_690x297.png, https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd6e3e244cd22fb71eda1a6e6710065b8da330a4.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd6e3e244cd22fb71eda1a6e6710065b8da330a4.png 2x" data-dominant-color="1E1F21"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">945×407 18.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am using observation = observation[0] or perhaps I should use the squeeze() method</p>
<p>Anyway, how is this observation shape advantageous?</p>

vmoens: <p>Hey thanks for this<br>
Can you elaborate a bit more your issue? Not sure I fully grasp the question</p>
Tensorboard's distributed view doesn't appear

dake: <p>Hi,</p>
<p>As illustrated in <a href="https://pytorch.org/tutorials/intermediate/tensorboard_profiler_tutorial.html" rel="noopener nofollow ugc">pytorch-profiler tutorial</a>, the distributed view should appear in tensorboard if I applied DDP during training.</p>
<p>However, the distributed option is not available when I run tensorboard.</p>
<p>What’s the reason for this?</p>
<p>My pytorch version is 1.12.0.</p>
Is there a method to parallel the following pytorch code

Bowen: <p>I have the following pytorch code that define a new matrix operation on two matrix points and primitives</p>
<pre><code class="lang-auto">def operations(points, primitives):
    """
    points shape: (batch size, number_of_points, 3)
    primitives shape: (batch_size, number_of_primitives,7)
    """
    gradient = torch.zeros(batch_size,number_of_points,number_of_primitives)
    for i in range(batch_size):
        
        temp_points = points[i,:,:]
        
        temp_primitives= primitives[i,:,:]
        temp = torch.zeros(number_of_points,number_of_primitives)
        for k in range(number_of_points):
            for j in range(number_of_primitives):
                temp[k,j] = torch.norm(temp_points[k,:]*temp_primitives[j,:3]+temp_primitives[j,3:6])
        gradient[i,:,:] = temp
    return gradient
</code></pre>
<p>Is there any method to parallelize such code to speed up?Thanks!</p>
<p>The above serialized code is implemented by myself, which is utilized in a deep-learning work. Is there any method to parallelize it? Everytime I run my code, the data loader of pytorch will throw an error <code>RuntimeError: DataLoader worker (pid 255034) is killed by signal: Killed. </code> even though I set the number of worker is zero. Thanks!</p>
Implementation of the alpha-loss function

Mora: <p>Hi,<br>
I came across this parametric loss function here: <a href="https://sankar.engineering.asu.edu/a-class-of-parameterized-loss-functions-for-classification-optimization-tradeoffs-and-robustness-characteristics/" class="inline-onebox" rel="noopener nofollow ugc">A Class of Parameterized Loss Functions for Classification: Optimization Tradeoffs and Robustness Characteristics | Lalitha Sankar</a></p>
<p>I was wondering if there is a simple way to implement this alpha-loss in pytorch for multi-class classification. I found an implementation on gitHub (link below), but it seems that it’s only for binary classification (?)</p>
<p>Implementation: <a href="https://github.com/SankarLab/AlphaLoss-TransIT-Code-MNIST-CIFAR/blob/main/Noisy_MultiClass/CIFAR10/CNN_2_1/cifar-noise-alpha.py#L114" class="inline-onebox" rel="noopener nofollow ugc">AlphaLoss-TransIT-Code-MNIST-CIFAR/cifar-noise-alpha.py at main · SankarLab/AlphaLoss-TransIT-Code-MNIST-CIFAR · GitHub</a></p>
<p>Thanks,</p>
With or without torch.no_grad()

bertrjul: <p>Will the output be the same with or without torch.no_grad()? torch.no_grad() impacts the autograd engine and deactivates it, in order to speed up the computation. Does it also impact the results? In both cases, model.eval() is on.</p>

ptrblck: <p><code>torch.no_grad()</code> will only disable Autograd and will avoid storing the forward activations. It will not change any behavior of the model. However, if the model has built-in randomness, which cannot be disabled via <code>model.eval()</code>, <code>torch.no_grad()</code> also won’t change it.</p>
Using multiple optimizers

mikayagoda: <p>Hi everyone,<br>
I need to implement the following:<br>
Optimize on a perturbation delta (a tensor) that is added to an input to a nerual network in order to create an adversarial example for the model.<br>
After that optimize on the model’s weights in order to make it robust to this specific input (the adversarial example).<br>
Afterwards, I need to use the model for some downstream task.</p>
<p>I tried to implement that using 3 different optimizers and faced an error:<br>
“RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [768, 512]] is at version 101; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!”</p>
<p>After some investigation I realized that the problem is probably the fact that the second optimizer changed the model’s weights inplace. (In general - optimizer.step() changes the optimization paramter inplace).</p>
<p>I also tried to use one optimizer with some optimization parameters, and changing the learning rates of the parameters that I want to remain static at certain times to 0.<br>
This also resulted in the same error.</p>
<p>I saw that an option to use multiple optimizers is performing the backward step of all optimizers and only after that performing the optimizer step.<br>
But this won’t fit in this case.</p>
<p>Does anyone have an idea how to overcome this issue?</p>
<p>Thank you!</p>

ptrblck: <p>You might be running into a similar issue as described <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">here</a>. After one optimizer updates (some) parameters the corresponding forward activations become stale as they were created by the old parameter set. Trying to calculate the gradients with these stale forward activations and the updated parameters is wrong and will raise the error.<br>
The linked post describes the issue in more detail.</p>
How did DDP benchmark?

JuyiLin: <p>Are there some demo codes for the DDP training benchmark? How to confirm that the all process are killed?</p>

agu: <p>Maybe the DDP part of this repo may be helpful: <a href="https://github.com/mrshenli/ptd_benchmark" class="inline-onebox" rel="noopener nofollow ugc">GitHub - mrshenli/ptd_benchmark</a></p>

JuyiLin: <p>I find pytorch/benchmarks/distributed/ddp/compare/compare_ddp.py in the repo.</p>

JuyiLin: <pre><code class="lang-python">    with profile(
        activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
        # record_shapes=True, # Causes seg fault in export_chrome_trace
        # with_stack=True, # Causes seg fault with EFA
        # with_flops=True, # Causes seg fault in export_chrome_trace
        record_shapes=False,
        with_stack=False,
        with_flops=False,
        on_trace_ready=my_tensorboard_trace_handler(f"tb/{now.strftime('%Y_%m_%d_%H_%M_%S')}", rank, use_gzip=True)
    ) if args.profile else contextlib.nullcontext() as prof:
        for i in range(n_iters):
            before_forward_event.record()           
            out = model(inputs)
            after_forward_event.record()
</code></pre>
<p>I wonder whether it is inaccurate.<br>
<code>profile</code> may bring extra cost to the model, and <code>event.record</code> may take more time compared with removing <code>with profile</code></p>

agu: <p>You are right; I think the profiler does introduce some slight overhead. However, we find that that slight overhead is typically negligible.</p>
<p>When you want to benchmark DDP, what are the metrics you are interested in?</p>

JuyiLin: <p>Thank you for your reply. I am interested in the forward and backward time in each GPU and the average of each GPU.</p>

mrshenli: <blockquote>
<p>Thank you for your reply. I am interested in the forward and backward time in each GPU and the average of each GPU.</p>
</blockquote>
<p>Hey <a class="mention" href="/u/juyilin">@JuyiLin</a> here are some examples of how to get those numbers:</p>
<p>getting fwd/bwd time: <a href="https://gist.github.com/mrshenli/8c39c35612218e0d6d772910bca2f737" class="inline-onebox" rel="noopener nofollow ugc">event_demo.py · GitHub</a><br>
getting DDP comm time: <a href="https://gist.github.com/mrshenli/eff5fdb654b5b4779ff4687a848d638d" class="inline-onebox" rel="noopener nofollow ugc">ddp_comm_time.py · GitHub</a></p>