akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Any help for a facial keypoint detection model?

Mehdi: <p>Hi !</p>
<p>I’m training a model for facial keypoints detection, using the Helen dataset.<br>
I’m using a Imagenet pretrained mobilenetV2 as backbone, retraining only the final layers for 10 epochs on the full dataset.<br>
I’m getting very poor results and I wanted to know whether someone could help me out. Basically, the model sticks all the points in the center and after the first epoch both the training and validation losses stop decreasing.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/4/4a1028910099bfb67b6ef4a7a89bfdca270f1567.png" alt="Figure_1" data-base62-sha1="azbX26id6T09mWUFRLVIn5gq6cn" width="640" height="480"></p>
<p>I’m not doing particularly fancy things and I don’t see what prevents training.<br>
First, for the model, :</p>
<pre><code class="lang-auto">backbone =  models.mobilenet_v2(pretrained = False)
backbone.load_state_dict(torch.load('./mb_net.pth')) 


features = backbone.features 
predictions = nn.Sequential(nn.AdaptiveAvgPool2d(output_size=(1, 1)),
                                    Flatten(),  
                                    nn.Linear(1280, 1000), 
                                    nn.Tanh(), 
                                    nn.Linear(1000, 1000),
                                    nn.Tanh(),
                                    nn.Linear(1000,512), 
                                    nn.ReLU(inplace = True), 
                                    nn.Linear(512, 
for p in features.parameters():
    p.requires_grad = False
self.predictor = nn.Sequential(features, predictions)

</code></pre>
<p>Then, for training, I do:</p>
<pre><code class="lang-auto">predictor = FaceModel()
loss_fn = nn.MSELoss(size_average = False)
adam = optim.Adam([p for p in predictor.parameters() if p.requires_grad], lr = 3e-4)

for epoch in range(epochs):
        predictor = predictor.to(device)
        predictor.train()
        train_loss = 0. 
        for i, sample in enumerate(train_loader): 


            x = sample['image']
            y = sample['points']

            preds = predictor(x.to(device))
            loss = loss_fn(preds, y.to(device))

            adam.zero_grad()
            loss.backward()
            adam.step()
            
           # keep track of metrics, 
           # validation loop... 


</code></pre>
<p>Concerning data augmentation, I’m first rescaling the images to be 224*224, then I ensure they are float values, comprised between 0 and 1 and I also make sure the keypoints values lie between 0 and 1. I apply horizontal flipping with p = 0.5 and that’s it. Should I use Imagenet stats to normalize image channels ?<br>
Other than that, I really don’t see an obvious error. Could maybe someone point me in the right direction ?<br>
Thanks a lot !</p>

tylerzhu: <p>When you apply your horizontal flipping, you most likely aren’t re-assigning keypoints, so the network is learning to get the best of both worlds and output the average. Namely, you need not only only flip the values of the keypoints, but also re-map them to their new keypoints, i.e. the left eye points now go to the right eye points.</p>
Backprop with multiplication on sparse tensor

Nicolas_Beltran: <p>Hi!<br>
I am using sparse tensors and need to perform a matrix multiplication using csr format but have not been able to do so as it appears as if it wasn’t implemented. I attach below my code and the error that I get. Interestingly enough if I use tensors in a coo format I don’t have this issue at all. Does anyone know what could be causing this issue?</p>
<pre><code class="lang-auto">i = [[1, 0], [1, 0]]
v = [1,1]
spm = torch.sparse_coo_tensor(i, v, torch.Size([2,2]), dtype=torch.float).to_sparse_csr()
m = torch.ones(2,2, requires_grad=True, dtype=torch.float)
a = torch.mm(spm, m)
l = torch.sum(a**2)
l.backward()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c42a5e8430c6fb3d4b9438e503e952aa8ea495bf.png" data-download-href="https://discuss.pytorch.org/uploads/default/c42a5e8430c6fb3d4b9438e503e952aa8ea495bf" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c42a5e8430c6fb3d4b9438e503e952aa8ea495bf_2_690x365.png" alt="image" data-base62-sha1="rZmmj6PDyKIh1JkJ3MXSKX98f0j" width="690" height="365" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c42a5e8430c6fb3d4b9438e503e952aa8ea495bf_2_690x365.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c42a5e8430c6fb3d4b9438e503e952aa8ea495bf_2_1035x547.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c42a5e8430c6fb3d4b9438e503e952aa8ea495bf_2_1380x730.png 2x" data-dominant-color="DEDEDE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1562×828 139 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How can I use pytorch to effectively zoom into the image?

Yosin: <p>Previously to zoom into an image I used the following (example) code:</p>
<pre><code class="lang-auto">import numpy as np
import cv2

image = np.zeros([64,64)]
image[10:-10,10:-10] = 1

# zoom into image:
imageZoomIn = cv2.resize(image[1:-1,1:-1] , (64,64))
#zoom out:
imageZoomOut = cv2.resize(image , (65,65))[1:-1, 1:-1)
#multiples Zooms:

imageBigZoom = image
for i in range(zoomScale):
    imageBigZoom = cv2.resize(imageBigZoom [1:-1,1:-1] , (64,64))
</code></pre>
<p>Is ther a way to do this effectively in pytorch with whole batches? I want to train my NN with distorted data, which can be bigger or smaller than it is supposed to be.</p>

ptrblck: <p>I think you could use <code>transforms.Crop</code> followed by <code>transforms.Resize</code> (or vice versa) to achieve the same results as in your current code snippet.</p>
How to realize VAR in pytorch

LiuWJ: <p>I want to realize VAR by pytorch, where for time shot t, Y[t]=W1Y[t-1]+W2Y[t-3]. The general part of my pytorch code is just like the below one：</p>
<pre><code class="lang-auto">f1=torch.nn.Linear(input_dim,output_dim)
f2=torch.nn.Linear(input_dim,output_dim)
for i in range(3,Y.shape[0]):
    Y[i]=Y[i]+f1(Y[i-1])+f2(Y[i-3])
</code></pre>
<p>However, when I backward the loss function of Y, there became an error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 100]], which is output 0 of UnsqueezeBackward0, is at version 5040; expected version 5039 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>Is there an easy way to realize VAR by pytorch? Thank you!</p>
nn.MaxPool2D layer

yuri: <p>Since batchnorm layer gathers statistics during the training step and reuse them later during inference, we have to define a new batchnorm layer every time it is used. The question is if this also applies to maxpooling or is it enough to define it once and use multiple times<br>
if we take the follwing script as example is it enough to only use max_pooling1 layer for both usages<br>
thnks</p>
<pre><code class="lang-auto">self.conv1 = nn.Conv2d(out_channel_4, out_channel_5, kernel_size=(9, 9), bias=True, stride=1, padding='same')
self.batch_Norm1 = nn.BatchNorm2d(out_channel_3, momentum=momentum)
self.conv2 = nn.Conv2d(out_channel_3, out_channel_4, kernel_size=(9, 9), bias=True, stride=1, padding='same')
self.batch_Norm2 = nn.BatchNorm2d(out_channel_4, momentum=momentum)
self.max_pooling1 = nn.MaxPool2d(2, stride=2)
self.max_pooling2 = nn.MaxPool2d(2, stride=2)

    def forward(self, x1):
        # block 1.1
        x1 = self.conv1(x1)
        x1 = self.batch_Norm1(x1)
        x1 = F.relu(x1)
        x1 = self.max_pooling1(x1)
        x1 = self.conv2(x1)
        x1 = self.batch_Norm2(x1)
        x1 = F.relu(x1)
        x1 = self.max_pooling2(x1)
</code></pre>

my3bikaht: <p>Correct mean and variance of the whole dataset needed to do proper normalization, but batchnorm has access only to batch part of data each step. That’s why batchnorm is saving intermediate values and updates them each batch-step.</p>
<p>Maxpool only returns max value in the pool in input data. Thus, you don’t need to define different maxpooling instances if they have same parameters.</p>
What is param.grad the gradient of?

snowball: <p>I’m slightly confused as to what param.grad is calculating mathematically?</p>
<p>Here, it says “Computes and returns the sum of gradients of outputs with respect to the inputs.”<br>
<a href="https://pytorch.org/docs/stable/autograd.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/autograd.html</a></p>
<p><strong>Can someone help me interpret this?</strong><br>
Suppose I’m looking at the weight of one neuron in a fully connected hidden layer denoted as w. Suppose my batch size is 64.<br>
Is param.grad for the w the partial derivative dL/dw calculated for each of the 64 raw inputs, and then summed?</p>

KFrank: <p>Hi Snowball!</p>
<aside class="quote no-group" data-username="snowball" data-post="1" data-topic="136751" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/ce7236/48.png" class="avatar"> snowball:</div>
<blockquote>
<p>Is param.grad for the w the partial derivative dL/dw</p>
</blockquote>
</aside>
<p>Yes.</p>
<aside class="quote no-group">
<blockquote>
<p>calculated for each of the 64 raw inputs, and then summed?</p>
</blockquote>
</aside>
<p>Not exactly.</p>
<p>Let <code>w</code> be a <code>Parameter</code> (or for than matter, just a <code>Tensor</code> that<br>
has <code>requires_grad = True</code>, but is not wrapped in a <code>Parameter</code>),<br>
and let <code>L</code> be a scalar (that is, a tensor with a single element) that<br>
has been calculated from <code>w</code> (and a bunch of other <code>Parameter</code>s<br>
and non-<code>Parameter</code> numbers).</p>
<p>If <code>param.gran</code> starts out as <code>0.0</code> (or <code>None</code>), then calling<br>
<code>L.backward()</code> populates <code>w.grad</code> with <code>dL / dw</code>.  Note that autograd<br>
is calculating the gradient of a single scalar, <code>L</code>, not of a batch of<br>
scalars that then get further processed.</p>
<p>In the <em>typical</em> use case <code>L</code> is the value of a loss function obtained by<br>
passing a batch of inputs through your model to produced a batch<br>
of predictions that are then compared with a batch of ground-truth<br>
targets using a loss function.  “Standard” pytorch loss functions will,<br>
by default, calculate the mean of the losses for each batch sample<br>
of the batch to obtain the single scalar value <code>L</code>.</p>
<p>So, to be clear, autograd computes the gradient of a single scalar<br>
<code>L</code> that as already be averaged over the whole batch, rather than<br>
computing the gradient of a whole batch of <code>L</code>s and then averaging<br>
(or summing) those gradients together.</p>
<p>Consider the case where <code>w</code> is a single scalar, and<br>
<code>L = w**2 + 7*w + 53</code>.  No batch, no sum over batch samples;<br>
autograd simply computes the derivative of the given quadratic<br>
polynomial, as computed to produce the value <code>L</code>, with respect to <code>w</code>.</p>
<p>Best.</p>
<p>K. Frank</p>
GPU utilization is low during training

Convolution: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=10" title=":grinning:" class="emoji" alt=":grinning:"><br>
I’m having a problem on my laptop, where the task manager shows the GPU utilization is fairly low (mostly below 38%) during training, but the power consumption is very high. I’m fairly new to pytorch and programming with GPU and use it mostly for academic assignments. Is it possible to improve the utilization of the GPU to make the training more efficient?</p>
<p>I’ll try to provide as much info as I can:<br>
OS: windows 10<br>
CPU: Intel(R) Core™ i7-7500U<br>
RAM: 8.00 GB<br>
GPU: Nvidia Geforece GTX 950M</p>
<p>I’m working on Pycharm pro and sometimes Jupyter Notebook on chrome.<br>
During training the task manager shows the GPU is &lt; 38%, Memory is ~83% and CPU ~34%.<br>
I’ve checked that my GPU is available, and during training I see my .py file on the Nvidia control panel.</p>

ptrblck: <p>I would recommend to check the <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html">Performance Guide</a> and check if you could enable e.g. <code>cudnn.benchmark</code> or use automatic mixed-precision training.<br>
Also, you could profile the code using the PyTorch Profiler and see where the bottlenecks are (e.g. in the data loading or the model itself).</p>
Pytorch acceleration

brianw0924: <p>I am wondering what is the best way to accelerate Pytorch or what implementation is most commonly used?</p>
<p>write C/C++? pytorch lightning? or others?</p>
<p>what is the ecosystem in this field? people often choose what method to accelerate their pytorch training?</p>

JuanFMontesinos: <p>Well,<br>
Pytorch is “accelerated” and it’s really fast. The major problem is not about pytorch but the python code you write around it.<br>
Check this blogpost <a href="https://paulbridger.com/posts/video-analytics-pipeline-tuning/" class="inline-onebox" rel="noopener nofollow ugc">Object Detection from 9 FPS to 650 FPS in 6 Steps | paulbridger.com</a> to have an idea where the problems come from. The only thing you can optimize is the dataloader. Still depends a lot on the field.</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html">Performance Guide</a> could also be useful for general best practices.</p>

brianw0924: <p>Thanks!! it helps a lot!!</p>
Vulkan concatenation operator

julien-thierry: <p>Hi,</p>
<p>Based on the tutorial:<br>
<a href="https://pytorch.org/tutorials/prototype/vulkan_workflow.html#c-api" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Vulkan Backend User Workflow — PyTorch Tutorials 1.8.1+cu102 documentation</a></p>
<p>The operator <code>_cat</code> is supposed to be supported in the vulkan backend. Yet when I try run an exported model using it on android, I get the following error:</p>
<p><code>RuntimeError: Could not run 'aten::_cat' with arguments from the 'Vulkan' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::_cat' is only available for these backends: [CPU, QuantizedCPU, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, Tracer, Autocast, Batched, VmapMode].</code></p>
<p>Is there any reason this is happening?</p>
<p>Thanks,</p>
<p>Julien</p>

iseeyuan: <p>It’s likely that the kernel of ‘aten::_cat’ is not implemented for Vulkan, or the implementation is not compiled with PyTorch. cc <a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a></p>

dhruvbird: <p>cc <a class="mention" href="/u/ssjia">@ssjia</a> - do you have some thoughts on this?</p>

beback4u: <p>Vulkan <code>cat</code> operator is supported with limitations; a) 4 dimensional inputs only b) works for height and channel dim. PR 66103, 66669 and 67207 are for the Vulkan <code>cat</code> operator implementation.</p>
CNN to recognize hand-face interaction

lb74: <p>Hello,</p>
<p>I’m currently involved in a project that aims at preventing risky hand-face interactions (skin touching, eye rubbing…). But we also want to distinguish these type of interactions from interaction like teeth brushing, eating, glasses adjustment…<br>
Our proposed method is to use wearable devices and to test different machine learning model for this application.<br>
For my part I use a wristband with an IMU to use the data from an accelerometer and a gyroscope. I want to classify the data in 12 classes of different movement type. The data will consist of a set of time series (the feautures could be 6-axis accelerometer and gyroscope, maybe the euler angles and the FFT of these signals) on a population of 12 people with at least 10 examples per class per people.<br>
I need to design a CNN model, train it and evaluate it but I don’t have much experience with pytorch and deep learning. How do we design a model for time-series classification (number of conv1d layer, activation function, dropout…)? How do we chose all the needed parameters (kernel size, batch size, number of epoch, number of filters…)?</p>
<p>Thank you for your help</p>
C++/libtorch equivalent of ".grad=None"?

Hamster: <p>Is there a way to achieve the equivalent of <code>.grad=None</code> in C++? (in order to later replace <code>.grad().zero_()</code> when the gradient is defined)</p>
<p>Thank you!</p>

dfalbel: <p>Maybe <code>t.mutable_grad() = torch::Tensor({})</code>? AFAICT <code>None</code> is equivalento to an undefined tensor in python.</p>
Optimize_for_mobile get a larger model file

Leroy: <p>I followed the mobile tutorial for a NMT model. After optimize_for_mobile and _save_for_lite_interpreter, I get a larger model file which is 127MB include encoder and decoder. But my pytorch model is an fp16 model with only 97MB before.<br>
Is this situation correct? Are there any steps I’m missing?</p>

Linbin: <p>cc <a class="mention" href="/u/kimishpatel">@kimishpatel</a>. I don’t think we support fb16.</p>
Can a GIoU loss (generalized intersection over union) be used after an STN module (spatial transformer network)?

flaviomaia77: <p>I have a model that uses an STN module for number detection, and mean square error loss. But I would like to replace it for GIoU, because MSE doesnt take into account how much of the target area has been detected, only how close individual coordinates are close to the target. But I wonder if this makes sense. Has anyone tried it, or has some insight?</p>
What does the c10::NotImplementedError from the C++ frontend mean?

Timangar: <p>The following C++ pytorch code throws a “c10:NotImplementedError”</p>
<blockquote>
<p><span class="hashtag-raw">#include</span> &lt;torch/torch.h&gt;</p>
<p>struct valnetImpl : torch::nn::Module<br>
{<br>
valnetImpl::valnetImpl() :<br>
fc1(register_module(“fc1”, torch::nn::Linear(384, 64))),<br>
fc2(register_module(“fc2”, torch::nn::Linear(64, 64))),<br>
fc3(register_module(“fc3”, torch::nn::Linear(64, 1)))<br>
{}<br>
torch::Tensor forward(const torch::Tensor&amp; x);</p>
<p>torch::nn::Linear fc1, fc2, fc3;<br>
};<br>
TORCH_MODULE(valnet);</p>
<p>int main(void)<br>
{<br>
torch::Device device = (torch::cuda::is_available()) ? torch::kCUDA : torch::kCPU;<br>
valnet v;<br>
v-&gt;to(device);<br>
//error: c10::NotImplementedError<br>
}</p>
</blockquote>
<p>I would be grateful for your help as to why this happens.</p>

Matej_Kompanek: <p>Your code runs fine in my environment, is there any reason why forward method is not implemented ?</p>
<pre><code class="lang-auto">torch::Tensor forward(torch::Tensor x)
{
	x = fc1-&gt;forward(x);
	x = fc2-&gt;forward(x);
	x = fc3-&gt;forward(x);
	return x;
}
</code></pre>

Timangar: <p>Thank you for your reply. It turned out that something during the installation had gone wrong.</p>
What is general approach to write logistic regression in PyTorch?

GamarChoba: <p>I have been working on a university project where I stuck on one point. I want to know what is the general approach to writing logistic regression in PyTorch? I have written some code here which I guess working fine.</p>
<p>Code:</p>
<pre><code class="lang-auto">class LogisticRegression(torch.nn.Module):
    """
    Logistic regression model
    inherits the torch.nn.Module which is the base class 
    for all neural network modules.
    """
    def __init__(self, input_dim, output_dim):
        """ Initializes internal Module state. """
        super(LogisticRegression, self).__init__()
        # TODO define linear layer for the model
        self.linear = torch.nn.Linear(input_dim,output_dim)
        

    def forward(self, x):
        """ Defines the computation performed at every call. """
        # What are the dimensions of your input layer?
        # TODO flatten the input to a suitable size for the initial layer
        output = self.linear(x)
        # TODO run the data through the layer
        
        return outputs
</code></pre>
<p>There are two additional questions that I do not understand:<br>
What are the dimensions of your input layer? and how to find it?<br>
Run the data through the layer. How to run data through the layer? and<br>
Flatten the input to a suitable size for the initial layer?</p>
<p>Thanks in advance.</p>