akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Inception v3 inference problems

username: Taha_Iftikhar
cooked: <p>Hello ,<br>
I am having a problem using my own re-trained inception_v3 for real-time inference.</p>
<p>First off I have a question. When saving my model after training, is it enough to just save it’s state dictionary with</p>
<blockquote>
<p>torch.save(model.state_dict(), ‘model.pth’)</p>
</blockquote>
<p>or do I also have to save the model with it and then the state dictionary?</p>
<blockquote>
<p>torch.save(model, ‘model.pth’)<br>
torch.save(model.state_dict(), ‘model.pth’)</p>
</blockquote>
<p>Second,<br>
my issue with inference is I am trying to use the re-trained inception_v3, which i only saved and loaed the state_dictionary for, it’s giving issues when i try to use my webcam and Opencv.<br>
This is the code:</p>
<blockquote>
<p>cam=cv2.VideoCapture(0)<br>
while True:<br>
ret,frame = cam.read()<br>
frame = torch.Tensor(frame)<br>
new_frame=frame.unsqueeze(0)<br>
incept.eval()<br>
out=incept(new_frame.permute(0,3,1,2)) <span class="hashtag-raw">#model</span> =model(inputs.permute(0, 3, 1, 2)) for proper channel placement<br>
cv2.imshow(‘frame’,frame)<br>
if cv2.waitKey(1) &amp; 0xFF == ord(‘q’):<br>
break<br>
cam.release()<br>
cv2.destroyAllWindows()</p>
</blockquote>
<p>I am now getting an error stating:<br>
TypeError: Expected Ptr<a>cv::UMat</a> for argument ‘mat’. Is there a better way to run a pytroch model for inference over webcam feed from opencv?</p>

username: ptrblck
cooked: <p>Yes, saving the <code>state_dict</code> is sufficient to restore the model for your deployment use case.</p>
<p>The error message seems to be raised by OpenCV and I don’t know which method is raising it.<br>
Are you able to read frames from your <code>cam</code> object using OpenCV?</p>

username: Taha_Iftikhar
cooked: <p>Hello,<br>
so I was able to read frames yes. I think there was an issue with my model to begin with.<br>
Either way, I managed to fix my inference problems by following the ideas by Htut Lynn Aung on this issue:</p><aside class="quote" data-post="1" data-topic="23928">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/htut_lynn_aung/48/6027_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/pytorch-trained-model-on-webcam/23928">Pytorch trained model on Webcam</a> 
  </div>
  <blockquote>
    Is there any way to do real time image classification with webcam using Pytorch trained model?
  </blockquote>
</aside>
Onnx Export with Dropout Layer = Training, rest eval

username: iSymphony
cooked: <p>Hi there,</p>
<p>I am implementing a bayesian neural network. For this I need the dropout layers to be active during inference while the rest of the layers are in eval mode. This is done with</p>
<pre><code class="lang-auto">model.eval()
model.classifier_layer[2].train(True) # Dropout layer
</code></pre>
<p>This works perfectly fine. However, I want to implement it on real-time on a raspberry pi. I export the model with torch.onnx.export(). For this I found the argument <em>training</em> of the onnx.export function. Here I have 3 choices: TrainingMode.EVAL, TrainingMode.TRAINING, TrainingMode.PRESERVE. I can’t use training or eval because then the whole model would be in training/eval mode. If I apply the TrainingMode.PRESERVE and export the model. The output of the model is always the same which implies that the dropout layer is not in training mode.</p>
<p>Is there any way I can get only the dropout layer to be in the training mode when exporting the model or am I doing anything wrong?</p>
<p>Many thanks, I appreciate the help!</p>

username: ptrblck
cooked: <p>I’m not familiar with ONNX and don’t know why the <code>PRESERVE</code> mode is not working.<br>
However, if for some reason the dropout layers are disabled during the export (have you verified it by e.g. just exporting a single dropout layer?), you could most likely use a custom module and hardcode the <code>training</code> argument:</p>
<pre><code class="lang-python">class MyDropout(nn.Module):
    def __init__(self, p):
        self.p = p

    def forward(self, x):
        x = F.dropout(x, p=p, training=True)
        return x
</code></pre>
<p>and replace all layers with this custom implementation.</p>
Data loader without labels?

username: f3ba
cooked: <p>Is there a way to the <code>DataLoader</code> machinery with unlabeled data?</p>

username: ptrblck
cooked: <p>Yes, <code>DataLoader</code> doesn’t have any conditions on the number of outputs of your <code>Dataset</code> as seen here:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.randn(100, 1)
        
    def __getitem__(self, index):
        x = self.data[index]
        return x
    
    def __len__(self):
        return len(self.data)

dataset = MyDataset()
loader = DataLoader(
    dataset,
    batch_size=5,
    num_workers=2,
    shuffle=True
)

for data in loader:
    print(data.shape)
</code></pre>

username: naty88
cooked: <p>Hello, I am working with DataLoader for the first time and have some problems.<br>
I have my data in json file like {“doc_id_1”: [sentence1, sentence2, …], …} and want to create a DataLoader with these data to get then a sentence embedding. The problem is that, there are no labels (doc_id is not a true label, it is just the id of the document).</p>
<p>I have defined the class straightforward like in the example above:</p>
<pre><code class="lang-auto">class DocDataset(Dataset):
    def __init__(self, json_file):
        self.data = json.load(open(json_file))

    def __getitem__(self, index):
        x = self.data[index]
        return x

    def __len__(self):
        return len(self.data)

json_file = 'dataset.json'
dataset = DocDataset(json_file)
loader = DataLoader(dataset, batch_size=5, num_workers=2, shuffle=True)
</code></pre>
<p>I might have a problem in <strong>getitem</strong> function, because I have a “KeyError”.</p>
<p>Thank you an advance!</p>

username: naty88
cooked: <p>Hello again, I think, I could solve the problem with KeyError:</p>
<pre><code class="lang-auto">    def __getitem__(self, index):
        for i, sent_list in enumerate(self.data.values()):
            x = list(self.data.values())[i]
        return x
</code></pre>
<p>I can’t get data.shape like in your example, since I’m working with list. And according to the next Error I’ve got, this must not be a list. So I defined it still wrong. Could you please tell me, where I get wrong?</p>
<pre><code class="lang-auto">train_dataloader = DataLoader(dataset, batch_size=5, num_workers=2, shuffle=True)
train_loss = losses.CosineSimilarityLoss(model)
#Tune the model
model.fit(train_objectives=[(train_dataloader, train_loss)], epochs=1, warmup_steps=100)
</code></pre>
<pre><code class="lang-auto">AttributeError: 'list' object has no attribute 'texts'
</code></pre>

username: ptrblck
cooked: <p>I’m not sure where the new error is raised, as I cannot see any usage of the <code>text</code> attribute. Could you check, which function is trying to access this attribute and make sure it’s using the right object?</p>
__forward__impl meaning?

username: ronald-xie
cooked: <p>Hi, Could someone explain why shufflenetV2 code in torchvision need to use __forward__impl?</p>
<p>The explanation in the code say that I can check with TorchScript super().</p>
<p>But I have checked with TorchScript tutorial and have not found that why we need to use __forward__impl().</p>
<p>Is that coupled with TorchScript and can speed up the inference when model is being deployed?</p>

username: siahuat0727
cooked: <p>I found more explanation <a href="https://github.com/pytorch/vision/blob/e8dded4c05ee403633529cef2e09bf94b07f6170/torchvision/models/mobilenetv2.py#L190" rel="noopener nofollow ugc">here</a>.</p>
Random State, Manual Seed & Oscillating Loss

username: qdl
cooked: <p>Hello,</p>
<p>I have a simple model with three <a href="https://docs.dgl.ai/en/0.4.x/_modules/dgl/nn/pytorch/conv/graphconv.html" rel="noopener nofollow ugc">GraphConv</a> layers (<a href="https://www.dgl.ai/" rel="noopener nofollow ugc">DGL</a>) and one <code>nn.Linear</code> layer, trying to do predictions over a graph with 700 vertices (560 training vertices in a semi-supervised GCN setting). About 10 days ago, when I first wrote it, I was consistently getting high training/test accuracies of &gt;= 95%. Something changed one of the days and I can’t figure out what (it is such an extremely simple model) that made it behave in a way where the loss would either go down painstakingly slowly or start oscillating. This meant that the accuracy numbers dropped below 40%. But this wasn’t consistent either. Once every 10-15 runs, the accuracy would shoot up to &gt; 85% and then fall back down. All of this was on the exact same training set (same training vertices).</p>
<p>This made me suspect that perhaps something was up with the random state. So I set <code>torch.manual_seed(0)</code> at the start of the program. Now I have accuracy at 95%. Also, as is expected with a fixed RNG seed, the accuracy is consistent across runs. So, surely my hunch about this being related to RNG was right, but my fix feels like a band-aid on the symptom and not a real root cause analysis. As to the question of why I am starting at such a terrible state is simply beyond me. I also <em>highly</em> doubt that my simple use-case has uncovered an underlying PyTorch/DGL bug. So something is up with my model.</p>
<p>Does anyone have an obvious hunch, say without looking at the actual code itself? If you need to see the model itself, then I can post it here, but really it is as simples as:</p>
<pre><code class="lang-auto">nn.ReLU()(dgl.nn.pytorch.GraphConv(10, 10)) x 3
nn.Sigmoid()(nn.Linear(10, 4))
</code></pre>
<p>Also, this is possibly a <code>DGL</code> forum question, but since <code>torch.manual_seed(0)</code> seemed to alleviate the symptoms, I thought I might have better luck starting with a broader audience.</p>
<p>I have also cross-posted it in the DGL forum: <a href="https://discuss.dgl.ai/t/random-state-manual-seed-oscillating-loss/1884" class="inline-onebox" rel="noopener nofollow ugc">Random State, Manual Seed &amp; Oscillating Loss - Questions - Deep Graph Library</a></p>

username: ptrblck
cooked: <p>The dependency on the random seen could point towards e.g. a bad initialization of the model and I agree that it feels rather like a band-aid than a proper solution.</p>
<p>It’s hard to isolate the real root cause, since you don’t have the initial code anymore and thus cannot compare different runs. However, I would check the initialization as well as shuffling of the dataset etc., which are all depending on the seed.</p>
Reduce the GPU memory use during inference

username: KaiHoo
cooked: <p>Hello, are there any methods to reduce the GPU memory use during inference? I have already used torch.no_grad. Thanks!</p>

username: KaiHoo
cooked: <p>I tried to fuse batch norm and convolutions, but it has very limited reduction. In theory, we can forward one layer and free the GPU use, but I do not know how to do this.</p>
<p>EDIT: I made a mistake. This is indeed one possible solution</p>
Multi-GPU torch.load() out of memory

username: tcsn_wty
cooked: <p>I was training a model with 1 GPU device and just now figured out how to train with 2 GPU devices. Therefore I paused the training and resume after adding in lines of code to use 2 GPUs. Firstly, loading the checkpoint would cause torch.load() out of memory no matter I use 1 GPU or 2 GPUs. Then I followed some posts to first load the check point to CPU and delete checkpoint, and the model was able to resume on 1 GPU.</p>
<pre><code class="lang-auto">checkpoint = torch.load('checkpoint/ckpt.t7', map_location=torch.device("cpu"))
model.load_state_dict(checkpoint['state'])
del checkpoint
torch.cuda.empty_cache()
</code></pre>
<p>However, when I tried to use 2 GPUs it caused a cuda out of memory error. I wasn’t able to find any related posts specifically regarding OOM regarding multi-GPUs especially it worked just fine with single GPU. I wonder if there are any tricks playing with multi-GPU when loading checkpoint?</p>
Backwarding only respect with a part of network, at first

username: Simone256
cooked: <p>Hello everyone, I am struggling with the backward() function for this architecture I’m working with:</p>
<pre><code class="lang-auto">from losses import mmd_NTK
from models import FullyConnected, Convolutional

latent_size = 16
net = FullyConnected(latent_size=latent_size)

noise = torch.randn((noise_batch_size, latent_size))
input_images, input_labels = next(iter(train_loader))
generated_images = net(noise)

classifier = Convolutional()
classifier.load_state_dict(torch.load('./trained_classifier.pth'))

loss = mmd_NTK(input_images, generated_images, classifier)
</code></pre>
<p>I’ve skipped many steps of the actual code, but I believe it’s clear what I’m doing. I will like to optimize the parameters of the network <code>net</code> backproping with respect to loss. But I have some problems before… In particular my <code>mmd_NTK</code> function is defined as follows</p>
<pre><code class="lang-auto">def mmd_NTK(batch_images, batch_generated_images, classifier):
    vec1 = get_NTK_avg_feature(batch_images, classifier)
    vec2 = get_NTK_avg_feature(batch_generated_images, classifier)
    return distance(vec1, vec2)
</code></pre>
<p>My loss is set to be equal to the distance of this two vectors, that are the derivatives of some output function with respect to the parameters of another network, <code>classifier</code>, which is already trained, and that I don’t want to train anymore.</p>
<pre><code class="lang-auto">def get_NTK_avg_feature(images, classifier):
    outputs = classifier(images)
    avg_outputs = outputs.mean(dim=0)
    squared_outputs = (avg_outputs ** 2).sum()  # This is to get a number as output
    squared_outputs.backward()
    parameters_grad = []
    for param in iter(classifier.parameters()):  # Now I fill a list with all the derivatives...
        parameters_grad.append(param.grad)
    return parameters_grad
</code></pre>
<p>Unfortunately when I run this a second time pytorch complains, in particular it tells that I’m trying to backward through the graph a second time. The problem arises because when I backward in the <code>get_NTK_avg_feature</code>, the first network <code>net</code> is also taken into account. I wouldn’t want this. But I also don’t want to completely erase the dependency of my loss (the distance between vec1 and vec2) with respect to <code>net</code>, since I will optimize it with respect its parameters. Any clue?</p>
<p>P.s. Also, it turns out that vec1 and vec2 are equal, or better, if I compute vec2 the value inside vec1 gets automatically overwritten with the value of vec2. This is an independent problem though…</p>
Loss function for multi class semantic segmentation

username: benkolber
cooked: <p>Hello!</p>
<ol>
<li>I am training a semantic segmentation model, specifically the deeplabv3 model from torchvision.</li>
<li>I am training this model on the CIHP dataset, a dataset consisting of human images and 20 class labels for different body parts (arm, leg, face etc…)</li>
<li>I am lost as to how to compute the loss for the following tensors:</li>
</ol>
<pre><code class="lang-auto">input.shape = (batch_size, 3, 512, 512)
mask.shape = (batch_size, 1, 512, 512)
</code></pre>
<p>the output of the model:</p>
<pre><code class="lang-auto">model = models.segmentation.deeplabv3_resnet101(pretrained=False)
model.classifier = DeepLabHead(2048, 20)
</code></pre>
<p>is a tensor of shape <code>(batch_size, 20, 512, 512)</code>, where 20 are the different possible classes.</p>
<p>When I try to calculate the loss as follows:<br>
<code>loss = criterion(outputs['out'], masks)</code></p>
<p>I get the following warning:</p>
<p><code>Using a target size (torch.Size([10, 1, 512, 512])) that is different to the input size (torch.Size([10, 20, 512, 512])). This will likely lead to incorrect results due to broadcasting.</code></p>
<p>The warning makes sense to me, but then how would one calculate the loss for this type of model?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>I assume you are working on a multi-class segmentation use case and are thus using <code>nn.CrossEntropyLoss</code> as the criterion. The error message doesn’t seem to fit this loss function, so you might want to switch to it. Once this is done, remove the “channel dimension” in your <code>masks</code> tensors, as they are not expected. <code>nn.CrossEntropyLoss</code> expects a model output tensors in the shape <code>[batch_size, nb_classes, height, width]</code> containing the logits for each pixel and a target tensor in the shape <code>[batch_size, height, width]</code> containing the class indices for each pixel in the range <code>[0, nb_classes-1]</code>.</p>

username: benkolber
cooked: <p>Thank you very much!! Very helpful <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
LSTM error while trying to update the hidden state

username: Vlad_Dumitru
cooked: <p>I am trying to train an LSTM while keeping its hidden state (LSTM stateful) until the moment when I am going to start a new epoch(episode). But here it’s come an interesting situation because I am getting the following error while trying to do so:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has<br>
been modified by an inplace operation: [torch.cuda.FloatTensor [705,<br>
25]] is at version 3; expected version 2 instead. Hint: the backtrace<br>
further above shows the operation that failed to compute its gradient.<br>
The variable in question was changed in there or anywhere later. Good<br>
luck!</p>
</blockquote>
<p>And this is happening just if I am trying to keep my hidden state if, for example, I will use a thin line instead:</p>
<p><code>dist, _ = self.actor(state, h_out)</code> and also removing <code>retain_graph=True</code> everything is going to work just fine.</p>
<p>Could any of you help me to understand what is going on here and how can I fix this, please?</p>
<p>I have my training loop here:</p>
<pre><code>for ep in range(conf.num_episode):
    state = env.reset()
    step = 0

    qnet_agent.hidden = None
    qnet_agent.hidden_2 = None
    while True:
        step += 1
        frames_total += 1

        epsilon = calculate_epsilon(frames_total)

        action, smart_decision = qnet_agent.select_action(state, epsilon)

        new_state, reward, done, info = env.step(action)

        memory.push(state, action, new_state, reward, done)

        qnet_agent.optimize()
        state = new_state

        if done:
            steps_total.append(step)
            break
</code></pre>
<p>And here is my optimize function:</p>
<pre><code> def optimize(self):
    if len(self.memory) &lt; self.config.batch_size:
        return

    state, action, new_state, reward, done = self.memory.sample(batch_size=self.config.batch_size)

    state = torch.Tensor(np.array(state)).to(device)
    new_state = torch.Tensor(np.array(new_state)).to(device)
    reward = torch.Tensor(reward).to(device)
    action = torch.LongTensor(action).to(device)
    done = torch.Tensor(done).to(device)

    h_out = self.hidden
    dist, self.hidden = self.actor(state, h_out)
    dist = torch.distributions.Categorical(dist)

    advantage = reward + (1 - done) * self.config.gamma * self.critic(new_state).squeeze(1) - self.critic(state).squeeze(1)

    critic_loss = advantage.pow(2).mean()
    self.optimizer_critic.zero_grad()
    critic_loss.backward()
    self.optimizer_critic.step()

    actor_loss = -dist.log_prob(action) * advantage.detach()
    self.optimizer_actor.zero_grad()
    actor_loss.mean().backward(retain_graph=True)
    self.optimizer_actor.step()
</code></pre>
Runtime dependence on hidden nodes

username: PhysicsIsFun
cooked: <p>Greetings!</p>
<p>I am training a single-hidden-layer perceptron on the CIFAR10 sets. My network thus has 32*32 input nodes (I turned the images to greyscale), N hidden nodes, and 10 output nodes.<br>
For a fix learning rate and number of training epochs, I found that my runtime barely depends on the number N of hidden nodes. E.g. N=200 gives around 5 minutes runtime, N=10 still almost 5 minutes.<br>
How is this possible? Decreasing the number N by a factor of 20 also drastically reduces the number of learnable parameters. Does it have anything to do with the way PyTorch applies its linear layers?</p>
<p>Best<br>
PiF</p>

username: ptrblck
cooked: <p>The actual model workload is tiny, so that changing the size of this single layer might not be directly visible in the end2end training time.<br>
You would have to properly profile the code parts to determine how much time is actually spent on the training itself.<br>
The current runtime might be determined largely by the data loading and processing time or even the dispatching or kernel launch overheads.</p>

username: PhysicsIsFun
cooked: <p>Interesting! Didn’t consider memory issues as I am usually not focussing on images. Cheers!</p>
Overloading nn.embedding method in nn.model?

username: Jnune02
cooked: <p>Hello, new user here and new to machine learning and pytorch framework. So, I have a sequence of vector embeddings that are generated by running input sequences through a GAN discriminator and capturing the output. I want to take this sequence and feed it back into another discriminator to generate embeddings over the sequence of the original embeddings.</p>
<p>The issue here is that the vector space over the potential combination of vector embeddings in a sequence of, say, even 5 vectors in sequence, is HUGE. Far more than can be held in my working environment, and I plan on doing this a 3rd time as well.</p>
<p>I was thinking that I could potentially override the nn.embedding functionality to implement a system other than a simple lookup table. Essentially, instead of trying to store every possible vector combination as another vector, I instead store mappings between input vectors and vector transformations. e.g. given one vector, the embeddings method looks up a transformation, applies to the given vector, and then spits out a new vector to represent the next vector in the sequence. The stored transformations essentially become the vector lookup table instead of trying to capture all possible vector combination sequences.</p>
<p>I asking here however because I’m very green at PyTorch. This is actually part of my Senior Project project at my university, and me being the overambitious little scrub that I am, chose a project that was… near the upper ends of what I have experience with, lol.</p>
<p>So, I really want to know: Is there a better way of doing this? My project entails using GAN networks to process nested sequences of information. I’m essentially using a first level GAN to processes the innermost sequences, but then I need to feed the data back and processes sequences of level 1 sequences, and then once again, where it is sequences of level 2 sequences of level 1 sequences.</p>
<p>Also, over the general topic of processing nested sequences, does anyone have any advice or information on that research topic in general? e.g. “Processing nested sequences of sequences” using GAN networks? This semester has been super information overload, haha, and I’m going to be needing all the help I can get in order to pull of my Senior Project.</p>
<p>Thank you very much for your time and in advance!</p>
Python vs C++ when it comes to pytorch

username: Rahul_pillai1
cooked: <p>I want to know the difference between python and c++ when it comes to pytorch.<br>
Like whats their purpose in the AI world. Also the position of python and c++ in the industry.<br>
Which one should i go for as a college student, also someone who knows intermediate c++ and why.<br>
Also suggest a good learning resource.</p>
PSNR and SSIM decrease after increasing training image

username: cbd
cooked: <p>First i trained model with 800 images. During the validation i took the image from same dataset and i got the PSNR of 19. I take the test image from cross dataset. When i test it on trained model it is increased to 20.</p>
<p>Then i increase the training image to 1200 instead of 800 and keep other parameter same. In validation PSNR increase to 22 but during testing image of cross data set, it decrease to 19.</p>
<p>Why testing image PSNR decreasing even validation PSNR increase? What will be the solution to increase the PSNR of testing image?</p>
Conv1d for horizontal gradient

username: MauroGentile
cooked: <p>Hello,</p>
<p>I need to extract the horizontal gradient of a binarized image.<br>
As a toy example, consider one single row of the image of length 10<br>
What I do is:</p>
<pre><code class="lang-auto">row = np.array([0,1,1,1,1,0,1,1,1,0]).reshape(1,1,10)
row = torch.Tensor(row)

m = nn.Conv1d(1, 1, 2, stride=1)
dx=np.array([1, -1])
m.weight=nn.Parameter(torch.from_numpy(dx).float().unsqueeze(0).unsqueeze(0))
output = m(row)
</code></pre>
<p>As output, I would expect:</p>
<pre><code class="lang-auto">[ 0,  1,  0,  0,  0, -1,  1,  0,  0, -1,  0]

</code></pre>
<p>But I am actually having random values that change at every run…</p>
<p>Why so?</p>
<p>PROBLEM SOLVED: I must set bias = False</p>
Understanding DARTS coding

username: promach
cooked: <p>Any idea what <code>def __init__</code> is <a href="https://translate.google.com/translate?sl=auto&amp;tl=en&amp;u=http://khanrc.github.io/nas-4-darts-tutorial.html" rel="noopener nofollow ugc">trying to achieve here</a> ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a1e0cab27d42adc9e15e8fcde02b146df8e3271f.png" data-download-href="https://discuss.pytorch.org/uploads/default/a1e0cab27d42adc9e15e8fcde02b146df8e3271f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1e0cab27d42adc9e15e8fcde02b146df8e3271f_2_531x500.png" alt="image" data-base62-sha1="n62ubJjepgmDNeSVU0hEN3Clfyf" width="531" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1e0cab27d42adc9e15e8fcde02b146df8e3271f_2_531x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1e0cab27d42adc9e15e8fcde02b146df8e3271f_2_796x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/1/a1e0cab27d42adc9e15e8fcde02b146df8e3271f.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">905×852 104 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Can't overfit after small change in data

username: Adir_R
cooked: <p>Hi,</p>
<p>I’m using the following repo:<a href="https://github.com/OshriHalimi/shape_completion" class="inline-onebox" rel="noopener nofollow ugc">GitHub - OshriHalimi/shape_completion</a> for partial shape reconstruction. Their architecture is Encoder-Decoder, where the encoder is 3 conv1 layers with relu between them and linear as final layer, the decoder is 5 conv1 layers.<br>
When I make an overfit test, I give it as input always the same partial shape with the desired reconstruction, it works as expected, and after about 30 epochs there are nice results. when I make simple change of deleting about 20% of the mesh vertices(and complete those missing with coping random vertices until I get full number of vertices again), the network does not learn anything, although the completion is still rather pretty simple.<br>
The loss function is MSE between the output mesh and desired one, with ADAM optimizer.</p>
<p>I will appreciate any help to understand the reason for this failure.<br>
Thanks</p>
Training with multiple copies of a model

username: mbehzad
cooked: <p>Hi,</p>
<p>How can I <strong>correctly</strong> create multiple copies of a model and backpropagate the loss correctly? So I have a <code>model</code> and I am simply did the following:</p>
<pre><code class="lang-auto">model1 = model
model2 = model
model3 = model
</code></pre>
<p>Firstly, is this correct or should I use deepcopy (or how is deepcopy different from the above)?</p>
<p>Secondly, <strong>after</strong> I create/copy model1, model2, and model3, when I execute the line:</p>
<pre><code class="lang-auto">model.to(device)
</code></pre>
<p>I notice that all the four models are moved to <code>device</code>. So if I move the <code>model</code> to <code>gpu</code>, the other 3 models also moved to <code>gpu</code> and same for <code>cpu</code>. Why is that?</p>
<p>Thirdly, I try to delete the <code>model</code> (or the copied models) by calling <code>del model</code>, but it is still not deleted and <code>'model' in locals()</code> return <code>True</code> (same <code>True</code> is returned for other models even when I delete them via <code>del</code>). Any explanation for this behavior?</p>
<p>Lastly, and importantly, I want to compute the loss of each model independently. Something like:</p>
<pre><code class="lang-auto">loss1 = criterion(outputs1, labels)
loss2 = criterion(outputs2, labels)
loss3 = criterion(outputs3, labels)
</code></pre>
<p>How can I aggregate the loss (is <code>loss = loss1 + loss2 + loss3</code> okay?) such that this total <code>loss</code> is backpropagated to all the models. Also, how to call <code>_.backward()</code> and <code>_.step()</code> functions in this case.</p>
<p>Any help on this will be much appreciated.</p>

username: god_sp33d
cooked: <p>because model1, model2, model3 and model are referring to the same object. You have to use copy.deepcopy() to create an independent copy of the model. your second, question would be solved with copy.deepcopy()</p>
<p>yes, you can compute</p>
<pre><code class="lang-auto">loss1 = criterion(outputs1, labels)
loss2 = criterion(outputs2, labels)
loss3 = criterion(outputs3, labels)
loss = loss1 + loss2 + loss3
loss.backward()
</code></pre>
<p>regarding optimizer.step(), you first have to add all your models parameters to the optimizer before calling the step.</p>
<pre><code class="lang-auto">torch.optim.SGD(list(model1.parameters()) + list(model2.parameters() + list(model3.parameters())
</code></pre>
Error while train a model on my own dataset

username: Najeh_Nafti
cooked: <p>i cannot train my own dataset on a model defined on github and i got this error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 230, in &lt;module&gt;
    main()
  File "train.py", line 227, in main
    run(config)
  File "train.py", line 137, in run
    'start_itr': state_dict['itr']})
  File "/content/drive/MyDrive/ME/BigGAN-PyTorch/utils.py", line 573, in get_data_loaders
    load_in_mem=load_in_mem, **dataset_kwargs)
TypeError: 'NoneType' object is not callable
</code></pre>

username: ptrblck
cooked: <p>It’s a bit hard to debug this issue and I would suggest to create an issue in the GitHub repository, as the code authors should be more familiar with their code base.<br>
That being said, based on the error message a <code>None</code> object is called, so you could check, which of the methods or arguments is <code>None</code>, while a valid method is expected.</p>

username: Kapil_Rana
cooked: <p>From the error,  I feel that there is an error in the data-loader, Please check the dataset and files. then dataloader</p>
CUDA out of memory error when calling the train script in a loop from another python script

username: theairbend3r
cooked: <p>Hi all,</p>
<p>I’m trying to run the same pytorch training script with different arguments (argparse) from another python script. I’m using <code>os.system()</code> for the same.</p>
<p>Here’s what I’m trying to do -<br>
<code>train.py</code> = &gt; the script which contains the train-loop.<br>
<code>runner.py</code> =&gt; the file which runs the train script in a loop.</p>
<pre><code class="lang-auto"># runner.py
for hp in hyperparams:
    os.system(f"CUDA_VISIBLE_DEVICES=1 python train.py --arg1 hp")
</code></pre>
<p>A few models get trained but I eventually end up getting a CUDA out of memory error. My guess is that the GPU memory is not being cleared after every loop. What can I do to mitigate this?</p>

username: god_sp33d
cooked: <p>Does it train completely for the first set of hyperparams in the for loop?</p>

username: theairbend3r
cooked: <p>Yep. For instance, if there were 10 models, it will successfully train 8 and then give a CUDA error for 9 and 10.</p>
Merging two or more models together

username: TheoCoombes
cooked: <p>Is it possible to merge many models together, whilst maintaining the same overall model structure?</p>
<p>For example, I have three models trained on separate parts of the same dataset. I would like to then merge them into one model that maintains the same structure of the model (so I can use the same scripts etc.).</p>
<p>Is it possible to do so?<br>
Would the model be worse than if I trained one single model?<br>
Would this scale?</p>
<p>Many thanks in advance</p>

username: ptrblck
cooked: <p>I’m not sure, what “merge” means in this case and if you want to reduce the different parameter sets to a single one or if you want to create a model ensemble.<br>
In the former case, you could use <a href="https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008/2">this approach</a>, but note <a href="https://discuss.pytorch.org/t/training-linear-combination-of-pytorch-models/75349/4">this concern</a>. In the latter case you could use <a href="https://discuss.pytorch.org/t/combining-trained-models-in-pytorch/28383/2">this approach</a>.</p>
<aside class="quote no-group" data-username="TheoCoombes" data-post="1" data-topic="117730">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/theocoombes/48/36129_2.png" class="avatar"> TheoCoombes:</div>
<blockquote>
<p>Would the model be worse than if I trained one single model?</p>
</blockquote>
</aside>
<p>This would be my concern, but you should run your experiments and see, if it could work for your use case. E.g. <a href="https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/">Stochastic Weight Averaging</a> also works, but would average “similar” checkpoints, not completely different models, which might have converged to different minima.</p>

username: TheoCoombes
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks a lot for the fast response</p>
<blockquote>
<p>I’m not sure, what “merge” means in this case</p>
</blockquote>
<p>I simply want to add the trained data from the 2nd model into the first model.</p>
<p>Taking a look, it seems the former link could be solution to my question.</p>
New device backend, trying to implement all operators

username: ConnollyLeon
cooked: <p>Hi, we are trying to implement all native operators to make good use of our GPU device (not from NVIDIA).</p>
<p>Could anyone please tell us what operators  we should implement?</p>
<p>I’ve checked the native_functions.yaml file, but still not pretty sure where to begin. Are those operators that are with a CUDA dispatch key the only operators that we should implement?</p>
Add custom attributes to Node in JIT

username: harryjing
cooked: <p>Is there any way to add custom attributes to JIT Node ? I need to use this attribute when I run custom passes on JIT Graph.</p>

username: Bob.Zhu
cooked: <p>Hi, did you find the solution?<br>
I tried register_buffer and register_parameter APIs (they are on module level but not Node, but they fit my need), but neither works. I got below error when calling them for a loaded JIT model.<br>
"<br>
RuntimeError: Can’t add a new parameter after ScriptModule construction.<br>
"<br>
And below error when calling a new ScriptModule.<br>
"<br>
AttributeError: cannot assign buffer before Module.<strong>init</strong>() call<br>
"<br>
any input will be appreciated.</p>
<p>thanks,</p>
Run user defined function on multi-GPU

username: javirk
cooked: <p>Hello,<br>
How can I run a user-defined function on multi-GPU in parallel? As a toy example, given function f, I would like to run f(a) on cuda:0 and f(b) on cuda:1 in parallel:</p>
<pre><code class="lang-auto">def f(x):
    return torch.matmul(x, x)

a = torch.rand((5, 5))
b = torch.rand((5, 5))

f(a)
f(b)
</code></pre>
<p>Thanks!</p>

username: JuanFMontesinos
cooked: <p>you just need to allocate a and b in the gpus that you want and pytorch will do the trick.</p>
NVIDIA K80 for PyTorch

username: nicofish
cooked: <p>I was wondering if anyone of the forum has any experience using a K80 in 2021 with PyTorch.</p>
<p>I am running a GTX 1070 with 8GB of VRAM, and my models are too big. I have purchased a GT710 to free up some of the VRAM occupied by my operating system, but it appears by current project is going to require 16+GB of VRAM.</p>
<p>Originally I was hoping to be able to purchase a pair of RTX 3060’s or get NVIDIA’s researcher discount on  a Titan, but it appears that that is not going to happen in the near future.</p>
<p>Because of this I have been wondering if a K80 could be a viable, cost effective alternative for me.</p>
<p>Has anyone used a K80 recently with PyTorch? If so what is performance like? is it worth it considering I can purchase one for &lt;300 USD?</p>
<p>As always thank you very much for your time and assistance.</p>

username: nicofish
cooked: <p>In case anyone ever stumbles upon this. I have abandoned this idea due to the fact that I would have to mod rather loud fans on to the K80 heat sink. I will simply wait until graphics card prices drop down to normal.</p>

username: abdulrazaq
cooked: <p>I am interested in this question. I do not mind the loud noise or cooling.</p>
Should an MLP learn slower than a single linear layer?

username: agt
cooked: <p>I have a big architecture that uses a few MLPs like this: linear-&gt;ReLU-&gt;linear. After 30 epochs, it gets to a loss of around .5. But if I replace all the MLPs with single linear layers, it gets to a loss of around .06 in the same number of epochs. I am trying to fit a small set of test data.</p>
<p>Is this an issue? And why does it happen?</p>
Check if grad is enabled

username: ZeroMax
cooked: <p>Hi,<br>
I’m looking for a way to check if gradient calculation is enabled or not from within my net. Is there a way to do this from within PyTorch? Something like: torch.check_grad_enabled → bool…</p>
<p>Thanks.</p>

username: sio277
cooked: <p>You can check it by accessing <code>torch.Tensor</code>'s <code>requires_grad</code> attribute, which returns True if gradient should be calculated on that Tensor. Note that it has a contagious behavior, that is, if <code>A.requires_grad=True</code> for some Tensor <code>A</code>, all Tensors computed from <code>A</code> have the <code>requires_grad</code> attribute True.</p>
Spatial Transformer Networks - Why Interpolation/Sampling?

username: Horst_G
cooked: <p>Hey,<br>
After transformation is computed and the sampling grid (transformed input grid) is generated, the sampling process (assigning values to grid) starts. Is the following explanation for the sampling process correct?: The sampling grid is no longer integer, but usually decimal (float). Because the output grid is supposed to be integer again, we interpolate each integer value in the output using the sampling grid (which has a direct association with the input grid and therefore values) and its neighbors in the input grid?</p>
Does "parameter.grad" is the gradient*lr or just gradient

username: Btorch
cooked: <p>Hi, I want to measure the gradient norm, I have a working code, I just want to interpret it :</p>
<pre><code>        for m,p in enumerate(model.parameters()):
            param_norm = p.grad.data.norm(2)
            total_norm += param_norm.item()
</code></pre>
<p>Now I wonder does p.grad is the already multiplied by the learning rate or the pure gradient?</p>
<p>Thanks a lot!</p>

username: sio277
cooked: <aside class="quote no-group" data-username="Btorch" data-post="1" data-topic="117722">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/btorch/48/12833_2.png" class="avatar"> Btorch:</div>
<blockquote>
<p>Now I wonder does p.grad is the already multiplied by the learning rate or the pure gradient?</p>
</blockquote>
</aside>
<p>No, it is just a gradient itself <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20">.</p>
Optimizer.step() vs. optimizer.step

username: joepareti54
cooked: <p>the following code only works if</p>
<pre><code class="lang-auto">optimizer.step()
</code></pre>
<p>is used. But if</p>
<pre><code class="lang-auto">optimizer.step
</code></pre>
<p>is in use, there is no training progress, so I believe the parameters are not updated, yet there is no warning or exception being thrown.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision.transforms as transforms
import torchvision.datasets as dsets
import numpy as np
import torch.utils.data as DL
import torch.nn.functional as F
import time
#
TRF = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
#
T_DSET = dsets.MNIST(root= './data1', transform=TRF, train=True, download=True  )
#
V_DSET  = dsets.MNIST(root= './data1', transform=TRF, train=False, download=True  )
TRAINLOADER = DL.DataLoader(dataset=T_DSET, batch_size=32, shuffle=True)
V_LOADER = DL.DataLoader(dataset=V_DSET, batch_size=32)
#
m = nn.LogSoftmax(dim= 1)
loss = nn.NLLLoss()
device = 'cuda'
#
class Model(nn.Module):
    def __init__(self,H1, H2):
        super(Model, self).__init__()
        self.layer1 = nn.Linear(28*28, H1)
        self.layer2 = nn.Linear(H1, H2)
        self.layer3 = nn.Linear(H2, 10)
    def forward(self, x):
        S = x.shape[0]
        x = x.view(S, -1)
        x = F.relu(self.layer1(x))
        x = F.relu(self.layer2(x))
        x = (self.layer3(x))
        x = m(x)
        return x
#
MODEL = Model(256, 128)
#
optimizer = torch.optim.SGD(MODEL.parameters(), lr=0.01)
MODEL.to(device)
#
for epoch in range(10):
    running_loss = 0
    start = time.time()
    for X,y in TRAINLOADER:
        X, y = X.to(device), y.to(device)
        optimizer.zero_grad()
        yhat = MODEL(X)
        LOSS = loss(yhat, y)
        LOSS.backward()
        optimizer.step
#        optimizer.step()
        running_loss = running_loss + LOSS.item()
    end = time.time()
    val_loss = 0
    with torch.no_grad():
        for XX, yy in V_LOADER:
            XX, yy = XX.to(device), yy.to(device)
            yhatv = MODEL(XX)
            VL = loss(yhatv, yy)
            val_loss = val_loss + VL.item()
        print(running_loss/len(TRAINLOADER), val_loss/len(V_LOADER), (end-start))
#
#calculate the prediction probabilities for a given input image
MODEL.to('cpu')
my_iter = iter(V_DSET)
for i in range(55):
    Kx, Ky = next(my_iter)
#
OUT = torch.exp(MODEL(Kx))
print('ist ',Ky)
print('Soll ',OUT)


</code></pre>

username: sio277
cooked: <p><code>optimizer.step()</code> is a method defined in <code>torch.optim.Optimizer</code>, which is a base class for all optimizers. You should ‘call’ it for your parameters to be updated. Check this <a href="https://pytorch.org/docs/stable/optim.html#algorithms" class="inline-onebox" rel="noopener nofollow ugc">torch.optim — PyTorch 1.8.1 documentation</a>.</p>
Grid_sample docs vs usage

username: dorukkarinca
cooked: <p>In <a href="https://discuss.pytorch.org/t/solved-torch-grid-sample/51662/2" class="inline-onebox">[SOLVED]Torch.grid_sample? - #2 by ptrblck</a> the <code>grid</code> parameter of <code>grid_sample</code> appears to have the <code>y, x</code> order because <code>grid = torch.stack((meshy, meshx), 2)</code>. But in <a href="https://pytorch.org/docs/stable/nn.functional.html#grid-sample" rel="noopener nofollow ugc">the documentation</a>, it says "the size-2 vector <code>grid[n, h, w]</code> specifies <code>input</code> pixel locations <code>x</code> and <code>y</code>". So is the grid order <code>y, x</code> or <code>x, y</code>? In other words, does <code>grid[..., 0]</code> sample along the height or the width of the image?</p>
Values replacements in custom defined loss throw error

username: MauroGentile
cooked: <p>Hi,</p>
<p>I am using a custom defined loss function to implement a model I have read in a paper which requires me change all pixels in the output of the NN which do not meet a condition.</p>
<p>This command (in the Loss function) throws an error:<br>
output[target != 1] = 1</p>
<p>The error I get is “one of the variables needed for gradient computation has been modified by an inplace operation”</p>
<p>Any suggestion to solve this?<br>
Thanks for helping</p>
<p>UPDATE: Giving a second thought, this makes sense: in the Loss I am changing the output. I should rather create a clone and operating on it…I do not get the error anymore.  I am not sure if it works. Will let you know for collective knowledge</p>
Import torch without cuda support

username: micahr123
cooked: <p>When pytorch loads the cuda libraries into ram it use a huge chunk of memory. I am using multiple process. Only some of the processes need the cuda libraries (most can use the cpu). Is there a way to “import torch” without loading the cuda libraries on some processes and load the cuda libraries for others?</p>

username: ptrblck
cooked: <p>Do you mean the host or device memory?<br>
I’m not sure how the host memory behaves, but the device memory shouldn’t be used, if you are not calling any CUDA operations in your script.</p>

username: micahr123
cooked: <p>Host memory (CUDA DLLs loaded into ram or pagefiles). If I have 20 processes and each uses 5GB of ram, I would need 100GB of RAM+pagefiles to be able to run. See picture below.</p>
<p>.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f114644be96e4cdd32c733e9a1c0a5292aa10c8d.png" data-download-href="https://discuss.pytorch.org/uploads/default/f114644be96e4cdd32c733e9a1c0a5292aa10c8d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f114644be96e4cdd32c733e9a1c0a5292aa10c8d.png" alt="image" data-base62-sha1="yoGL5NorZpOP9rY5RmrfmWOfwaV" width="690" height="100" data-dominant-color="E3E3E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1130×165 11.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Pytorch profiler bug

username: weiqqi1028
cooked: <p>I use <code>tensorboard_trace_handler()</code> to store the trace json file, and I found in my json file, I have this invalid entry:</p>
<pre><code class="lang-auto">{
    "ph": "X", "cat": "Memcpy",
    "name": "Memcpy DtoD (Device -&gt; Device)", "pid": 0, "tid": "stream 7",
    "ts": 1618002603703134, "dur": 0,
    "args": {
      "device": 0, "context": 1,
      "stream": 7, "correlation": 549926, "external id": 61893,
      "bytes": 4, "memory bandwidth (GB/s)": inf
    }
  }
</code></pre>
<p>Notice the <strong>inf</strong> in the last line. This is not a valid entry so pytorch-tb-profiler has trouble opening it. Can this be a bug in Pytorch profiler?</p>

username: ptrblck
cooked: <p>This sounds like a bug, so could you create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> so that we can track and check it?</p>