akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How can I multiply all my model weights that saved in checkpoint by -1?

username: sbht_sbht
cooked: <p>I want to try multiplying the checkpoint weights by -1 to try to find the best minimum. How can i do this?</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/sbht_sbht">@sbht_sbht</a>,</p>
<p>If you want to multiply all weights by 1, you can try something like this,</p>
<pre><code class="lang-auto">for param in model.parameters():
  param.mul_(-1)
</code></pre>
<p>Although, I don’t know why multiplying all weights by one will help you find the minimum, given that your model is non-linear this will most likely drastically change your model’s performance.</p>
<p>If you’re trying to maximize a loss value, you multiply the gradients by -1 not the weights.</p>

username: sbht_sbht
cooked: <p>Thanks! Yeah, I know that, but I just want to try and see what happens</p>
Saving you best model during stochastic training

username: Ken_Semanov
cooked: <p>Is there a canonical way of caching the best model weights during training?    I am training using an <code>nn.Module</code>  class, using SGD with momentum and learning rate.  On rare occasions, the model with the smallest loss() occurred somewhere in the middle of the epochs.  I would like to somehow set aside that model for later use, rather than just using the model weights from the final epoch.</p>
Output LSTM Pack Padded Sequence to Linear Layer

username: shubhvachher
cooked: <pre><code class="lang-python">class MyModel(someBaseModel):
    def __init__(self):
        super().__init__()

        self.gru = nn.GRU(20, 50, 1, batch_first=True)  # input, hiddensize, layers
        self.classifier = nn.Linear(50, 4)

    def forward(self, xb):  # xb is a packed padded sequence of size (batch_size*max_length, 20)
        out, hn = self.gru(xb)  # out is packed padded sequence with data of size (bs*ml, 50)
        out = self.classifier(out.data)  # (bs*ml, 4) Now it is not a packed padded sequence but normal tensor
        return out
</code></pre>
<p>And my loss calculation looks like:</p>
<pre><code class="lang-python">def training_step(batch):
    inputs, labels = batch
    inputs = inputs.to("cuda:2")  # Packed Padded Sequence of size (bs*ml, 20)
    # labels = torch.Tensor(labels).to("cuda:2")  # Won't work since all labels are of different lengths

    batch_sizes = inputs.batch_sizes
    out = model(inputs)    # Generate predictions
    out_padded, out_lengths = pad_packed_sequence(torch.nn.utils.rnn.PackedSequence(out, batch_sizes), batch_first=True)
    # out_padded is of shape (bs, ml, 4)

    loss = torch.zeros(out_padded.size(0))
    for i in range(out_padded.size(0)):
        msel = F.mse_loss(out_padded[i][:out_lengths[i]], labels[i].to("cuda:2"))
        loss[i] = msel

    acc = torch.mean(loss)
    return loss, acc

for batch in train_loader:
    loss, acc = model.training_step(batch)
    train_losses.append(torch.mean(loss))
    train_accs.append(acc)
    torch.sum(loss).backward()  # [l.backward() for l in loss] won't work: RuntimeError: Trying to backward through the graph a second time
    optimizer.step()
    optimizer.zero_grad()
</code></pre>

username: shubhvachher
cooked: <p>Does this look right? I’m really unsure what the <code>torch.sum(loss).backward()</code> step would be doing in terms of accumulating gradients in both the GRU as well as the Linear weights… Is there a better way?</p>
<p>I really couldn’t find much info on how to best use the <code>pack_sequence</code> function.</p>

username: Messiah
cooked: <p>Did you end up finding a solution to this question?</p>
How to obtain the high order derivative of a custom function?

username: wangz
cooked: <p>Hi, I have a question w.r.t how to obtain the high order derivative of a custom <code>torch.autograd.Function</code>. If the <code>forward</code> and <code>backward</code> of the function is not implemented with pytorch, but scipy, with the <code>backward</code> in the <code>Function</code>, the first order derivative could be obtained. However, what about the high order derivative?</p>

username: soulitzer
cooked: <p>You won’t be able to get higher-order gradients for free if your backward isn’t written in a differentiable way using torch ops. You’ll need to write another custom function for the backward.</p>
<p>See <a href="https://pytorch.org/tutorials/intermediate/custom_function_double_backward_tutorial.html#when-backward-is-not-tracked" class="inline-onebox" rel="noopener nofollow ugc">Double Backward with Custom Functions — PyTorch Tutorials 1.13.0+cu117 documentation</a></p>
Deprecation of register_backward_hook problematic

username: slobodaapl
cooked: <p>I’ve started recently needing and using the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_backward_hook" rel="noopener nofollow ugc">register_backward_hook</a> and I found the warning that this function is deprecated in favor of the full backwards hook, which I find confusing because using the full backwards hook in various applications would require more redundant code and be more error prone.</p>
<p>Consider that one wants to manipulate the gradients of the output layer of their neural network, and only the output layer. A registered backward hook makes this simple, but a full backward hook requires one to add a redundantly running if statement to verify the correct module is being targeted.</p>
<p>Am I perhaps missing something that makes the use of a full backward hook concise and neat for this application that I’m unaware of?.. To me it seems like the simple backward hook is very useful.</p>

username: soulitzer
cooked: <p>You can also register a full backward hook to a specific module, perhaps you were using global hooks?</p>
<p>One difference between a regular backwards hook and a full backwards hook is that full backwards hook fires when the gradient wrt the inputs the module are computed, so you’d need to register another dummy module and then register hook if you’d like mutate gradients before running backward for that module. (We also have a new type of hook available on the latest nightly that fires when the gradients for the module outputs are computed, but you would be unable to modify the gradients).</p>
<p>Another difference to note for full backwards (pre)hook is that if your module returns multiple outputs, only a single hook would be registered and when that hook is called you’d be able to see gradients wrt all the outputs.</p>
"Missing token" for time series data

username: dbbgia
cooked: <p>Just as there are “Missing tokens” in language modeling problems to direct prediction to only the missing “words”, is there an equivalent “missing token” for numerical values to <em>impute</em> missing values in a time series? This feels trivial but I’m seemingly stumped on how to do this!</p>

username: AbdulsalamBande
cooked: <p>Hello, If I have understood you question correctly. You can treat your “numerical values in a time series” as <em>series of characters</em> or <em>series of words</em> in a language model.</p>

username: dbbgia
cooked: <p><a class="mention" href="/u/abdulsalambande">@AbdulsalamBande</a> I see, but I feel that is cumbersome. Say I want really high precision, a floating point number broken into “characters” would be really long.</p>
<p>Is there a standard workflow where the network’s attention can be directed to only missing numerical values? For instance, if my input is</p>
<pre><code class="lang-auto">input = np.array([[0.5, 1.2, 0.2], [0.6, 1.3, 0.3], [np.nan, np.nan, np.nan], [0.8, 1.4, 0.5], [0.9, 1.5, 0.6]])
</code></pre>
<p>I’d like the model to 1) copy over the input values where it is not missing 2) calculate the loss based on the prediction of only the missing values (np.nan).</p>
Accuracy slowly decreasing

username: dneprdva
cooked: <p>Hi guys, could you help me to explain my results? I m training dataset with 130000 images with 3 categories (very imbalanced 1, 10, 10). And I recieve these results.</p>
<blockquote>
<p>Epoch: 1 | train_loss: 1.0891 | train_acc: 0.5807 | val_loss: 1.0139 | val_acc: 0.6900<br>
Epoch: 2 | train_loss: 1.0873 | train_acc: 0.5587 | val_loss: 0.9711 | val_acc: 0.7617<br>
Epoch: 3 | train_loss: 1.0924 | train_acc: 0.5481 | val_loss: 1.0545 | val_acc: 0.5200<br>
Epoch: 4 | train_loss: 1.0906 | train_acc: 0.5476 | val_loss: 0.9557 | val_acc: 0.7779<br>
Epoch: 5 | train_loss: 1.0899 | train_acc: 0.5465 | val_loss: 0.9806 | val_acc: 0.7359<br>
Epoch: 6 | train_loss: 1.0905 | train_acc: 0.5524 | val_loss: 1.0477 | val_acc: 0.4923<br>
Epoch: 7 | train_loss: 1.0891 | train_acc: 0.5536 | val_loss: 1.0416 | val_acc: 0.5334<br>
Epoch: 8 | train_loss: 1.0891 | train_acc: 0.5424 | val_loss: 1.0141 | val_acc: 0.6006<br>
Epoch: 9 | train_loss: 1.0900 | train_acc: 0.5343 | val_loss: 1.0208 | val_acc: 0.6013<br>
Epoch: 10 | train_loss: 1.0888 | train_acc: 0.5391 | val_loss: 1.0387 | val_acc: 0.5357<br>
Epoch: 11 | train_loss: 1.0860 | train_acc: 0.5336 | val_loss: 0.9196 | val_acc: 0.8061<br>
[INFO] Total training time: 11991.201 seconds</p>
</blockquote>
<p>Training accuracy is slowly decreasing, why? And val acc is jumping.</p>

username: ptrblck
cooked: <p>I don’t know if you are trying to counter the class imbalance, but if you are using e.g. a <code>WeightedRandomSampler</code> or a weighted loss, the accuracy could decrease due to the <a href="https://en.wikipedia.org/wiki/Accuracy_paradox">accuracy paradox</a> and might not be the best metric for your use case.</p>

username: dneprdva
cooked: <p>I use this one</p>
<blockquote>
<p>nn.CrossEntropyLoss(weight=torch.tensor([0.59,6.24,6.39])).to(device)</p>
</blockquote>
<p>But now I have another problem, the train accuracy is increasing but train loss almost stays at one level.</p>
Freelance - Pytorch, GAT

username: aleksandrpikul134
cooked: <ol>
<li>
<p>There is such an article (<a href="https://arxiv.org/abs/2110.01200" class="inline-onebox" rel="noopener nofollow ugc">[2110.01200] AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks</a>), it uses the graph attention network (GAT) to detect voice forgery. Here you need to pay attention to Fig 1. In fact, you need to make such an algorithm, but not for voice, but for faces.</p>
</li>
<li>
<p>This circuit itself (Fig 1) has an Encoder. You can take ResNet18 as it, but without the last layer - average polling + FC.</p>
</li>
<li>
<p>Next, you need to take the GAT itself. For example, here is such an implementation (<a href="https://github.com/Diego999/pyGAT" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Diego999/pyGAT: Pytorch implementation of the Graph Attention Network model by Veličković et. al (2017, https://arxiv.org/abs/1710.10903)</a>).</p>
</li>
<li>
<p>Next, take some database of faces. It is important that the size of the images fit ResNet18, otherwise you will need to resize.</p>
</li>
<li>
<p>Put it all together. That is, a face is given at the input, then ResNet18 comes, then GAT. Then you need to look at the diagram, but if I’m not mistaken, then there is a certain vector, then FC and human marks (I can describe in more detail in the messages).</p>
</li>
<li>
<p>Train</p>
</li>
</ol>
<p>You can offer your price.</p>
<p>Or help find a person who will help with the implementation</p>

username: ptrblck
cooked: <p>I would recommend to change the category to “Jobs” so that more interested people would see it.</p>
Why is my REINFORCE algorithm not learning?

username: desert_ranger
cooked: <p>I am training a REINFORCE algorithm on the CartPole environment. Due to the simple nature of the environment, I expect it to learn quickly. However, that doesn’t happen.</p>
<pre><code class="lang-auto">#I referred to this when writing the code - https://github.com/DeepReinforcementLearning/DeepReinforcementLearningInAction/blob/master/Chapter%204/Ch4_book.ipynb

import numpy as np
import gym
import torch
from torch import nn
env = gym.make('CartPole-v0')
learning_rate = 0.0001
episodes = 10000

def discount_rewards(reward, gamma = 0.99):
    return torch.pow(gamma, torch.arange(len(reward)))*reward
def normalize_rewards(disc_reward):
    return disc_reward/(disc_reward.max())

class NeuralNetwork(nn.Module):
    def __init__(self, state_size, action_size):
        super(NeuralNetwork, self).__init__()
        self.state_size = state_size
        self.action_size = action_size
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(state_size, 300),
            nn.ReLU(),
            nn.Linear(300, 128),
            nn.ReLU(),
            nn.Linear(128, 128),
            nn.ReLU(),
            nn.Linear(128, action_size),
            nn.Softmax()
        )

    def forward(self,x):
        x = self.linear_relu_stack(x)
        return x

model = NeuralNetwork(env.observation_space.shape[0], env.action_space.n)
opt = torch.optim.Adam(params = model.parameters(), lr = learning_rate)
score = []
for i in range(episodes):
    print("i = ", i)
    state = env.reset()
    done = False
    transitions = []

    tot_rewards = 0
    while not done:

        act_proba = model(torch.from_numpy(state))
        action = np.random.choice(np.array([0,1]), p = act_proba.data.numpy())
        next_state, reward, done, info = env.step(action)
        tot_rewards += 1
        transitions.append((state, action, tot_rewards))


    if i%50==0:
        print("i = ", i, ",reward = ", tot_rewards)
    score.append(tot_rewards)
    reward_batch = torch.Tensor([r for (s,a,r) in transitions])
    disc_rewards = discount_rewards(reward_batch)
    nrml_disc_rewards = normalize_rewards(disc_rewards)
    state_batch = torch.Tensor([s for (s,a,r) in transitions])
    action_batch = torch.Tensor([a for (s,a,r) in transitions])
    pred_batch = model(state_batch)
    prob_batch = pred_batch.gather(dim=1, index=action_batch.long().view(-1, 1)).squeeze()
    loss = -(torch.sum(torch.log(prob_batch)*nrml_disc_rewards))
    opt.zero_grad()
    loss.backward()
    opt.step()
</code></pre>

username: vmoens
cooked: <p>Hey!<br>
So it seems that you’re not doing state = next_state at the end of your while loop, is that indended?<br>
Happy to debulk things further if needed</p>

username: desert_ranger
cooked: <p>Yes, I was doing a bunch of things wrong. Here is the complete solution - <a href="https://stackoverflow.com/questions/74214034/why-is-my-reinforce-algorithm-not-learning/74215692#74215692" class="inline-onebox" rel="noopener nofollow ugc">Why is my REINFORCE algorithm not learning? - Stack Overflow</a></p>
<p>Thanks for your help!</p>
Help for custom loss function

username: Eli1
cooked: <p>Hello everyone, I am trying to implement the following loss function:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/6/068ca12690dd2c9198f55c768a34cf66160e7a6c.png" data-download-href="https://discuss.pytorch.org/uploads/default/068ca12690dd2c9198f55c768a34cf66160e7a6c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/068ca12690dd2c9198f55c768a34cf66160e7a6c_2_690x227.png" alt="image" data-base62-sha1="VWa3d3KW2CCJa0OmtrxM1eu6VS" width="690" height="227" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/068ca12690dd2c9198f55c768a34cf66160e7a6c_2_690x227.png, https://discuss.pytorch.org/uploads/default/original/3X/0/6/068ca12690dd2c9198f55c768a34cf66160e7a6c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/6/068ca12690dd2c9198f55c768a34cf66160e7a6c.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">779×257 19.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have to perform a SGD to parameterize qθ(x), and q̃θ(x) is the parametrized part of the function.</p>
<p>So far I’ve come up with something like this:</p>
<pre><code class="lang-auto">def objective(p, output):
  x,y = p
  a = minA
  b = minB
  r = 0.1

  XA = 1/2 -1/2 * torch.tanh(100*((x - a[0])**2 + (y - a[1])**2 - (r + 0.02)**2))
  XB = 1/2 -1/2 * torch.tanh(100*((x - b[0])**2 + (y - b[1])**2 - (r + 0.02)**2))
  q = (1-XA)*((1-XB)* output - (XB))
  print("In obective function, q is", q)
  output_grad, = torch.autograd.grad(q, (x,y), retain_graph=True, create_graph=True)
  q = output_grad**2

  return q
</code></pre>
<p>This problem can be viewed as an unsupervised minimization problem. I am using as training data points that I’ve indepentely sampled from a distribution.<br>
The training is:</p>
<pre><code class="lang-auto">for e in range(epochs) :
 for configuration in total:
   # Training pass
   optimizer.zero_grad()

   #output is q~
   output = model(configuration)

   #loss is the objective function we defined
   loss = objective(configuration, output)
   loss.backward()
  
   optimizer.step()
</code></pre>
<p>In particular, I get the error: “RuntimeError: One of the differentiated Tensors does not require grad”.</p>
<p>Thank you to anyone who will help me.</p>
Validation loss overfitting with higher accuracy

username: Mohamed_Taha
cooked: <p>i have trained a model using ResNet implementation for Radio wave modulation classification and it’s working pretty well until finished training.</p>
<p>the test accuracy on validation dataset is continue to improve however, the validation loss increases,</p>
<p>so the question is "is this overfiting the model or it’s normal to increase in loss but get higher accuracy i don’t even know train it more or what to do right now</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/0/20b5ddede52c2dd562fd36f48729686f1b849c1c.jpeg" alt="photo_2022-11-06_11-30-18" data-base62-sha1="4FmVtPnRY74AYCa4AB0H4DcXAsY" width="640" height="480"></p>
Ctc loss becomes negative after 50k steps of training

username: barakb
cooked: <p>Although theoretically it is not possible, I’m getting negative loss when using ctc loss.<br>
Running in batch gives the save results, so Ill share the shapes of my tensors for B=1:<br>
<strong>Log_prob</strong> size is:[1500,73].<br>
<strong>Targets</strong> size is:[18],  and value:<br>
tensor([16,  9, 10, 11,  2, 10, 45, 17, 72, 10, 17, 72, 13,  5, 72, 12, 35, 11]).<br>
<strong>Input_lengths</strong> equal to tensor(1500).<br>
<strong>Target_lengths</strong> equal to tensor(18).<br>
This produces a loss of -1.3</p>
<p>My CTCloss is defined as:</p>
<pre><code>    self.loss = nn.CTCLoss(
        blank=72,zero_infinity =True
    )
</code></pre>
<p>As mentioned, the loss (and entire training) worked perfectly , only after 50k steps it suddenly became negative.</p>
<p>I’ve seen online that if the target size larger then input size this might create negative loss, but as you can see this is not the case.<br>
I’ll be happy to share anymore information.</p>
<p>Thanks for your help!!!</p>
32 float weight convert 16 float model?

username: henry_Kang
cooked: <p>Hello. I got my trained model with a good segmentation result.<br>
However, this is still little bit slow.<br>
I am wondering, is there any way I can convert this model to another type for speed?<br>
My model is Mobilenet v3 base small version model.<br>
I see that there is half-precision or 16 float convert that can make the model faster than the pure 32 float model.<br>
My GPU is titan V.<br>
Thank you.</p>

username: surya00060
cooked: <p>Hi</p>
<p>If you want to truncate/reduce precision the weights of the trained model, you can do</p>
<pre><code class="lang-auto">net = Model()
net.half()
</code></pre>
<p>which converts all FP32 tensor to FP16 tensor.</p>

username: henry_Kang
cooked: <p>Thank you I will try. Do you think this can reduce the inference time?</p>

username: ptrblck
cooked: <p>This can reduce the inference time, if TensorCores are used for the workload.<br>
However, note that FP16 can easily overflow such that your model might return NaNs, if you are not careful (especially during training).<br>
We thus recommend to use <a href="https://pytorch.org/docs/master/amp.html">automatic mixed precision training</a>, which can be used in the nightly binaries or from a master build.</p>
<p>Nevertheless, you could try to directly call <code>half()</code> and see, if the inference workload suffers from these issues.</p>

username: RAJA_PARIKSHAT
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> would converting the model to half-precision reduce run time inference on CPU as well?</p>

username: ptrblck
cooked: <p>I don’t know what the current support of half-precision on the CPU is so let’s wait for some CPU experts.</p>

username: MD_DIDARUL_ISLAM
cooked: <p>Henry, did that technique work for your model ??</p>
Develop multitask model based on vision transformer

username: nikiguo93
cooked: <p>Hi I used hugging face vision transofrmer to build my multitask model but encountered errors like:</p>
<p>TypeError: linear(): argument ‘input’ (position 1) must be Tensor, not ImageClassifierOutput</p>
<p>The code for the architecture looks like the following:</p>
<p>class multi_output_model(torch.nn.Module):</p>
<pre><code>def __init__(self,categories,regression_out,frozen_layer=False):
    super(multi_output_model,self).__init__()
    
    #vit = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224-in21k')
    self.vit = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224-in21k')
    #self.features = torch.nn.ModuleList(vit.children())[:-1]
    #self.features = torch.nn.Sequential(*self.features)
    #in_features = vit.classifier.in_features
    in_features = self.vit.config.hidden_size
    if frozen_layer:
        self.freeze_feature_layers()
    self.classification = torch.nn.Linear(in_features,categories)
    self.regression = torch.nn.Linear(in_features,regression_out)

        
def forward(self,input_imgs):
    #output = self.features(input_imgs)
    output = self.vit(input_imgs)
    #output = output.logits.view(input_imgs.size(0), -1)
    output_classification = self.classification(output)
    output_regression = self.regression(output)
    output_regression =torch.reshape(output_regression,[output_regression.size()[0],17,100])
    #return [output_regression,output_classification]
    return SequenceClassifierOutput(
        logits=[output_regression,output_classification],
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

def freeze_feature_layers(self):
    
    for n,p in self.features.named_parameters():
        p.requires_grad = False
</code></pre>
<p>Can any body help me with such problem?<br>
Looking forward to hearing from you!</p>

username: clerkie
cooked: <p>It looks like you’re trying to run your code on a CPU when your model is on a GPU. Make sure to put your model on the CPU before trying to run it on the CPU.</p>
MviT how to change the number of classes

username: Xiaohao_Lin
cooked: <p>Dear all,</p>
<p>I am trying to use the MVIT_V2_S function in torchvision 0.14.<br>
I wanted to load the pretrained weights on Kinetics-400, which has 400 output classes. However, my classification task only has 10 classes. I tried to pass in</p>
<p>torchvision.models.video.mvit_v2_s(weights=‘DEFAULT’, num_classes=10)</p>
<p>But received the following error:  ValueError: The parameter ‘num_classes’ expected value 400 but got 10 instead.</p>
<p>May I know how can I fix this issue?</p>
<p>Link: <a href="https://pytorch.org/vision/main/models/generated/torchvision.models.video.mvit_v2_s.html#mvit-v2-s" class="inline-onebox" rel="noopener nofollow ugc">mvit_v2_s — Torchvision main documentation</a></p>

username: ptrblck
cooked: <p>Load the original pretrained model first, then manipulate the last layer(s) as needed to adapt them to your use case. <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html">The finetuning tutorial</a> might be a good start.</p>
New open source automata project

username: Avery_Thomas
cooked: <p>I’m looking to fork a PyTorch project for research but need help from experts. The new open source project would involve taking this existing project and making a few changes, I just don’t have a coding background: <a href="https://github.com/chenmingxiang110/Growing-Neural-Cellular-Automata" class="inline-onebox" rel="noopener nofollow ugc">GitHub - chenmingxiang110/Growing-Neural-Cellular-Automata: A reproduction of growing neural cellular automata using PyTorch.</a></p>
<p>Instead of using a starting image the input data would take the first time step and use that as the input for the next time step and so on. The output image would be a graphical representation of the stacked time changes of all alive cells.</p>
<p>I’m hoping the result generates a 3D hypergraph that has the same qualities as the original GitHub project like self healing.</p>
<p>I want to see what the algorithm does when parts are corrupted or removed. My theory is the neural network will have some interesting characteristics.</p>
Understanding precision and reacall in torchmetric

username: Innat
cooked: <p>Documentation. <a href="https://torchmetrics.readthedocs.io/en/stable/classification/precision.html?highlight=Precision" class="inline-onebox" rel="noopener nofollow ugc">Precision — PyTorch-Metrics 1.3.0.post0 documentation</a></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf82e375d63f856d6d93cb6ef780a1493a73b1bd.png" alt="image" data-base62-sha1="rkbCHsE9DFzlAUZwmh0lnBxQaqV" width="320" height="104"></p>
<pre><code class="lang-auto">from torchmetrics import Precision
from torchmetrics.classification import BinaryPrecision

preds  = torch.tensor([0, 0, 0, 0])
target = torch.tensor([0, 0, 0, 0])

general_precision = Precision(num_classes=2)
bin_precision = BinaryPrecision()

# both should be same?
general_precision(preds, target)
# (tensor(1.)

bin_precision(preds, target) 
# tensor(0.))

</code></pre>
<p>From equation, and from <code>preds</code> and <code>target</code>, as there are now <code>True Positive</code>, the numerator becomes <code>0</code> and so the answer should be <code>0</code>. But in general precision, how does it calcuate this?</p>
<p>[ Same goes to (<code>F1Score - BinaryF1Score</code>, <code>BinaryRecall, Recall</code>). ]</p>

username: Innat
cooked: <p>ok, -_-</p>
<pre><code class="lang-auto">precision = Precision(num_classes=1, multiclass=False)
</code></pre>
How to assign value of a multidimensional tensor by indexing a tensor?

username: alessandro_mondin
cooked: <p>I have a tensor tcls of shape  (n_objects, n_classes) and a 1 dim tensor for classes targets[…, 5][obj]. I would like to set for each row “n” of tcls the “nth” element of  targets[…, 5][obj].</p>
<pre><code class="lang-auto">tcls = torch.zeros_like(preds[..., 5:][obj], device=config.DEVICE)
new_tcls = "how can I do it?"
lcls = self.BCE_cls(preds[..., 5:][obj], new_tcls)  # BCE
</code></pre>
<p>Any suggestion?</p>
<p>I know I could do I with a naive for loop, but I am striving to learn it in a “linear algebra” way to enhance GPU computations</p>

username: alessandro_mondin
cooked: <p>After searching all around stackoverflow and PyTorch forum I found the solution here: <a href="https://discuss.pytorch.org/t/fill-value-to-matrix-based-on-index/34698/5" class="inline-onebox">Fill value to matrix based on index - #5 by ptrblck</a></p>
<p>Here it’s how it should be, I have no clue why…</p>
<pre><code class="lang-auto">tcls = torch.zeros_like(preds[..., 5:][obj], device=config.DEVICE)
        tcls[torch.arange(tcls.size(0)), targets[..., 5][obj].long()] = targets[..., 5][obj].float()
        lcls = self.BCE_cls(preds[..., 5:][obj], tcls)  # BCE```</code></pre>
How to Get Each Model Replica's Gradient

username: cbats
cooked: <p>Hi,</p>
<p>After I call loss.backward(), PyTorch can auto calculate the average gradients of all model replicas(towers) when I use torch.nn.DataParallel(). Is there any way to get each model replica(tower)'s gradient?</p>
<p>Example::<br>
&gt;&gt;&gt; net = torch.nn.DataParallel(model, device_ids=[0, 1, 2])<br>
&gt;&gt;&gt; output = net(input_var)<br>
&gt;&gt;&gt; loss = criterion(output, label_var)<br>
&gt;&gt;&gt; loss.backward()<br>
"""</p>
<p>Thanks</p>

username: cbats
cooked: <p>Thank you all, I got it by specialized the Gather function.</p>
<p>Btw, I wonder why replicate function will be iterative call when DataParallel Module forward rather than call once when init? <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/data_parallel.py#L67" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/data_parallel.py#L67</a></p>

username: cbats
cooked: <p>Sorry, I made a mistake. Actually I solve it by specialized the Broadcast’s backward function(which is a reduce operation).</p>
<p>Thanks.</p>

username: wish
cooked: <p>I also met with this problem. I thought it is the Scatter::backward() gathers the grads until this post remind me it actually is Broadcast::backward().<br>
Thanks very much:)</p>

username: cruzas2
cooked: <p>Hello!</p>
<p>I am interested in something similar. In particular, I am interested in accessing the replicated model parameters. Would you be able to provide a code snippet to show how you were able to access the replicated model gradients?</p>
Seq2seq for time series prediction

username: NoWay2Guess
cooked: <p>Hello guys, I have a project related to the time series prediction, I currently use a many2many model, and I want to replace it using a seq2seq model. Since pretty much all the models I can find are NLPs. I manage to build an simple seq2seq model myself, but I not sure if it is the correct way of doing it.</p>
<p>Here is an sample of what I am doing for my project.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b557266f4d14acf776c802bdaaf415bffafbe0d2.png" alt="image" data-base62-sha1="pSd8HUD4MNzElUXWqz42lsKdN3s" width="394" height="278"></p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1H_msZ5ZCFZnMRBQE5Mm2e6xsnWZi7Vu2?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/ac0e6bae38aa2b5a52a4f6e10b43c43d/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1H_msZ5ZCFZnMRBQE5Mm2e6xsnWZi7Vu2?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/1H_msZ5ZCFZnMRBQE5Mm2e6xsnWZi7Vu2?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Also question about “start” and “end” token added in the NLP model, I used the last element in the input dataset as the input for the first time step in the decoder to replace the “start” token, is this a correct way of doing it, and what about the ''end" token, should I just ignore it?</p>
Finding the indices of duplicate rows in a tensor

username: ZSheikhb
cooked: <p>Is there any way to find the indices of repeated (duplicate) rows in a 2D or 3D tensor using pytorch?</p>
<p>i.e. in the following tensor, is there any pytorch function that I can use in order to identify the indices of the repeated rows</p>
<pre><code class="lang-auto">tensor([[ 1.00, -1.00,  3.00],
        [ 2.00,  1.00,  0.00],
        [ 0.00,  2.00, -4.00],
        [ 2.00,  1.00,  0.00],
        [ 3.00, -2.00,  0.50],
        [ 1.00, -1.00, -0.50],
        [-2.50,  0.00,  2.30],
        [-0.50, -0.70,  0.60],
        [ 2.00,  1.00,  0.00]])
</code></pre>
Converting .pth (PyTorch) to .ckpt (PyTorch Lightning)

username: MateiBejan
cooked: <p>Hey there! I am looking for an elegant solution for converting a .pth (PyTorch) model file into a .ckpt (PyTorch Lightning) model file.</p>
<p>The solution I’m considering right now is parsing all the layers in the .pth’s state_dict and building a model which would be saved in .ckpt, but I’d prefer something less arduous.</p>
<p>Any Ideas?</p>
Simplest way of rotating a 3D tensor volume on the GPU

username: Omroth
cooked: <p>I have a 3D volume in a tensor on the GPU and I’d like to rotate it by a 3x3 (or transform it via a 4x4 affine) matrix.  Is there a built-in for this?</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/omroth">@Omroth</a>,</p>
<p>Can you share a minimal reproducible example?</p>
Replicating torch.permute with other operations

username: spiegelball
cooked: <p>Since torch.permute is buggy on mps device (<a href="https://github.com/pytorch/pytorch/issues/88183" class="inline-onebox" rel="noopener nofollow ugc">Permute followed by torch.nn.functional.interpolate gives wrong results on mps backend · Issue #88183 · pytorch/pytorch · GitHub</a>), I am looking for a generic replacement of permute operations. Obviously, chained invokations of torch.transpose do the job, but they are buggy on mps too. I was thinking of some combination of .view and transpose, but didn’t come up with a generic “formula” yet.</p>

username: KFrank
cooked: <p>Hi Spiegelball!</p>
<aside class="quote no-group" data-username="spiegelball" data-post="1" data-topic="165025" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/spiegelball/48/46248_2.png" class="avatar"> spiegelball:</div>
<blockquote>
<p>I am looking for a generic replacement of permute operations.</p>
</blockquote>
</aside>
<p>You could try <code>einsum()</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.12.0'
&gt;&gt;&gt; t = torch.ones (2, 3, 4, 5)
&gt;&gt;&gt; p = torch.einsum ('ijkl -&gt; jilk', t)
&gt;&gt;&gt; t.shape
torch.Size([2, 3, 4, 5])
&gt;&gt;&gt; p.shape
torch.Size([3, 2, 5, 4])
</code></pre>
<p>Having said that, it would not surprise me if <code>einsum()</code> uses <code>permute()</code><br>
and / or <code>transpose()</code> internally, so you might end up right back in the soup.</p>
<p>Best.</p>
<p>K. Frank</p>

username: spiegelball
cooked: <p>Thanks for the input, sadly using einsum leads to the same problem.</p>
Ninja Error on Android when building project

username: toni
cooked: <p>Hi, I cloned a branch from the repository called <code>ci-all/orig/release/1.7.1</code> and when I try to build the program I am getting the following error:</p>
<pre><code class="lang-auto">Build command failed.
Error while executing process /Users/&lt;username&gt;/Library/Android/sdk/cmake/3.10.2.4988404/bin/ninja with arguments {-C /Users/&lt;username&gt;/Desktop/pytorch/android/pytorch_android/.cxx/cmake/debug/armeabi-v7a fbjni pytorch_jni}
ninja: Entering directory `/Users/&lt;username&gt;/Desktop/pytorch/android/pytorch_android/.cxx/cmake/debug/armeabi-v7a'

ninja: error: '../../../../src/main/jniLibs/armeabi-v7a/libtorch.a', needed by '../../../../build/intermediates/cmake/debug/obj/armeabi-v7a/libpytorch_jni.so', missing and no known rule to make it
</code></pre>
<p>I am very new to PyTorch and I couldn’t figure this error out.<br>
I would really appreciate if anyone can help me with this. Thanks in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: kimishpatel
cooked: <p><a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a> can you take a look?</p>

username: IvanKobzarev
cooked: <p>Hi <a class="mention" href="/u/toni">@toni</a>,<br>
What command do you run?</p>
<p>To build pytorch_android you should use:</p>
<pre><code class="lang-auto">sh ./scripts/build_pytorch_android.sh
</code></pre>
<p>which will build all 4 abis by default. Or to specify abi as argument.</p>
<pre><code class="lang-auto">sh ./scripts/build_pytorch_android.sh arm64-v8a
</code></pre>

username: vjayd
cooked: <p>Still same error with the above command</p>
How to learning to generate a single tensor (2xNxN) from a batch of features?

username: Aldebaran
cooked: <p>What kind of neural network (NN) do you suggest for learning to generate a single tensor with shape <code>(2, n, n)</code> from a batch of features with shape <code>(batch_size, max_length_size, representations_size)</code>?</p>

username: AbdulsalamBande
cooked: <p>The input seems to be of NLP type. If the application is not large and the max_length_size is not very large, you can try an LSTM. Else, you can try a transformer. You can also squeeze the input and try a vanilla  NN.</p>
Creating trainable masks for model parameters

username: Vibhu
cooked: <p>Hello everyone,</p>
<p>I’m working with GPT-2 and I need to create a trainable mask for each weight layer. In other words, for each weight layer W, there is a matrix M of the same shape such that during the forward pass, a thresholding function is applied to M in order to create a mask, which is then applied to W. During the backward pass, the matrix M would need to be updated, in order to better sample the appropriate masks. Also, the backward function for the thresholding function would need to be a straight-through estimator, so that the gradients aren’t 0.</p>
<p>I tried this approach but <code>self.pre_mask</code> (the matrix M) doesn’t get updated during the backward pass:</p>
<pre><code class="lang-auto">class ThresholdFunction(torch.autograd.Function):

    @staticmethod
    def forward(ctx, input):
        # 0.5 is the threshold
        return (input &gt; 0.5).float()

    @staticmethod
    def backward(ctx, grad_output):
        # Treats the threshold function like an identity function
        return grad_output


class StraightThroughEstimator(nn.Module):
    def __init__():
        super(StraightThroughEstimator, self).__init__()

    def forward(self, x):
        x = ThresholdFunction.apply(x)
        return x


class Masked(nn.Module):

    def __init__(self, orig_layer):
        super().__init__()
        # the matrix W 
        self.orig_layer = orig_layer
        # the matrix M
        self.pre_mask = torch.rand(self.orig_layer.weight.size(), requires_grad=True)
        self.masking_layer = StraightThroughEstimator()

    def forward(self, *x):

        mask = nn.Parameter(self.masking_layer.forward(self.pre_mask))
        masked_layer = self.orig_layer
        masked_layer.weight = nn.Parameter(self.orig_layer.weight * mask)
        output = masked_layer(*x)

        return output


class GPT2_with_mask(nn.Module):

    def __init__():
        super().__init__()
        self.model = GPT2LMHeadModel.from_pretrained('gpt2')
        # Example of modifying a GPT-2 layer so that it has a trainable mask
        self.model.transformer.h[0].attn.c_attn = Masked(self.model.transformer.h[i].attn.c_attn)
</code></pre>
<p>I’m also aware that using hooks might be another approach, but I’m not sure how to best proceed.</p>
<p>If it helps, this is the paper that I’m trying to implement: <a href="https://arxiv.org/pdf/2004.12406.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2004.12406.pdf</a></p>
<p>Any inputs or suggestions would be really appreciated, thank you.</p>
Time Series Augmentation in pytorch forecasting

username: Jose_Peeterson
cooked: <p>Hi,</p>
<p>I have count time series of demand data and some covariates like weather information every hour. I have used 168 hours (7 days) for encoder and 24 hours (next day) for decoder in DeepAR  pytorch forecasting. E.G. using MTWTFSS for encoder to predict M (Monday)</p>
<p>After doing much testing I find that the 24 hours in prediction is more correlated with NOT the previous 7 days. It is more correlated with the same days in the past week. So I would need to use</p>
<p>MMMMMMM (mondays of previous weeks) to predict M (next monday).</p>
<p>Is it possible to tell TimeSeriesDataset object to train using this type of inputs?</p>
<p>I cannot manually create this like below</p>
<p>MMMMMMM TTTTTTTT WWWWWWWW…</p>
<p>because it will take any subsequence inside this time series like MMMMTTTT to use for encoder (MMMMTTT) and decoder (T). I do not want this. so is there a way to tell  TimeSeriesDataset object to only sample the time series sequentially from the beginining without any overlaps during training? So that I can just feed the input time series as</p>
<p>MMMMMMM TTTTTTTT WWWWWWWW…</p>
Issue in regression and 3DCNN algorithm

username: Ghazaleh
cooked: <p>Hi, I have developed a 3DCNN for regression of video images. The output measures RMSE and MAE. The problem is the value of RMSE and MAE is close to zero for 10 epoch and 5 fold cross validation. Others applied same database and different models. but most of the previous works results for RMSE and MAE is between 6 to 10. I think something is wrong in my code which measures RMSE and MAE very low. Any help appreciated.  I was wondering if I implemented my algorithm based on 3DCNN and Regression in a correct way or not.</p>

username: ptrblck
cooked: <p>Double post from <a href="https://discuss.pytorch.org/t/rmse-results-close-to-zero/165062">here</a>. Without more information about your use case and your concern about a low loss it’s hard to help, so please add more information.</p>

username: Ghazaleh
cooked: <p>Thanks for your answer. It is an algorithm to inference depression level from AVEC database. I trained and evaluated the system with 10 epoch and applying the 5 fold cross validation. In each epoch the result for training and validation RMSE and MAE is close to zero for example: Train RMSE: 0.0167, Train MAE: 0.01 and Val RMSE: 0.0060, Val MAE: 0.01.  The problem is other researchers in the same database with different deep learning algorithms achieved RMSE and MAE something between 6 to 10. Please let me if anything else I should share. I can share the code if needed. I was wondering if I implemented my algorithm based on 3DCNN and Regression in a correct way or not.</p>
Questions about torch.multiprocessing

username: ArchieGertsman
cooked: <p>I’m trying to reduce the overheads of communicating between processes, and my idea is to utilize the shared nature of torch tensors. All the data that I need to send between processes is numerical and can be stored as tensors. I have a few questions:</p>
<ul>
<li>if I send a collection of tensors through the Queue, will those be shared? Or are tensors only shared when they they are the only object being sent through the Queue? In the latter case, can I specify that I want each tensor in the collection to be shared manually?</li>
<li>if tensors are shared, then does that mean they don’t get pickled when sent though a Queue? What about a collection of tensors? Really, I would like to know if it is fast to send a collection of shared tensors between processes. Note: the shapes of the tensors are subject to change, so I can’t just put everything in one big tensor of fixed shape, unless I pad it.</li>
<li>how large is the overhead of copying data to shared memory and then sending it through a Queue versus not utilizing shared memory at all (i.e. just using python’s multiprocessing module)?<br>
Thanks!</li>
</ul>
Language Models' interpretability

username: sev777
cooked: <p>Is there any efficient way to analyze how a piece of knowledge is encoded in LMs?</p>

username: AbdulsalamBande
cooked: <p>Honestly. Understanding the machanics of Transformers can really help you. <a href="https://www.youtube.com/watch?v=oCWFyt2kWLg&amp;ab_channel=MachineLearningwithPytorch" rel="noopener nofollow ugc">Pytorch Transformer encoder</a></p>
RuntimeError: PyTorch is not linked with support for opencl devices

username: vjayd
cooked: <p>RuntimeError: PyTorch is not linked with support for opencl devices When trying to set tensor.device(“opencl”)</p>

username: artyom-beilis
cooked: <p>There is no native support of OpenCL in pytorch - opencl device is just rudimentary device for something that was planned/existed at some time.</p>
<p>However you can use OpenCL for deep learning using out of tree backend.</p>
<p>There is a <a href="https://github.com/artyom-beilis/pytorch_dlprim/" rel="noopener nofollow ugc">relatively new project</a> based on OpenCL <a href="https://github.com/artyom-beilis/dlprimitives" rel="noopener nofollow ugc">dlprimitives library</a> that works on AMD, Intel and nVidia aritechures.</p>
<p>This project is relatively new and far from being complete but already gives support for training of many different vision network types and gives decent performance.</p>
<p>So follow the build instructions and try it.</p>
<p>See <a href="https://discuss.pytorch.org/t/out-of-tree-opencl-backend-now-easy-to-install/164521">this discussion</a> as well</p>
RuntimeError: Found dtype Long but expected Float for the loss.backward()

username: MichalBaryla
cooked: <p>Hi,<br>
I try to run my googlenet model on CelebA data set (list_landmarks_align_celeba as a label) and i facing some issues with loss.backward() in the code below:</p>
<pre><code class="lang-auto">model = GoogLeNet()
criterion = nn.functional.mse_loss

model.to(device)
optimizer = optim.Adam(model.parameters(), lr=0.001)

inputs, labels = next(iter(trl))
inputs, labels = inputs.to(device), labels.to(device)

optimizer.zero_grad()

# Start the forward pass
prediction0, aux_pred_1, aux_pred_2 = model(inputs)

real_loss = criterion(prediction0, labels)
aux_loss_1 = criterion(aux_pred_1, labels)
aux_loss_2 = criterion(aux_pred_2, labels)

loss = real_loss + 0.3 * aux_loss_1 + 0.3 * aux_loss_2
loss.backward()
# optimizer.step()

</code></pre>
<p>with this out put:</p>
<pre><code class="lang-auto">out shape is   torch.Size([2, 128, 4, 4])
out shape is   torch.Size([2, 128, 4, 4])
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_3736/3018090650.py in &lt;module&gt;
     16 aux_loss_2 = criterion(aux_pred_2, labels)
     17 loss = real_loss + 0.3 * aux_loss_1 + 0.3 * aux_loss_2
---&gt; 18 loss.backward()
     19 # optimizer.step()

~/anaconda3/envs/py35-2/lib/python3.9/site-packages/torch/_tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    485                 inputs=inputs,
    486             )
--&gt; 487         torch.autograd.backward(
    488             self, gradient, retain_graph, create_graph, inputs=inputs
    489         )

~/anaconda3/envs/py35-2/lib/python3.9/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    195     # some Python versions print out the first line of a multi-line function
    196     # calls in the traceback and some print out the last line
--&gt; 197     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    198         tensors, grad_tensors_, retain_graph, create_graph, inputs,
    199         allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass

RuntimeError: Found dtype Long but expected Float

</code></pre>

username: srishti-git1110
cooked: <p>Hi, could you please check if by any chance the data type of the <code>labels</code> tensor long?</p>

username: MichalBaryla
cooked: <p>Yes, variable labels was tensor long and predicted values were Float. I switched type of labeles, prediction0, aux_pred_1,  aux_pred_2,  to torch.double and it worked. Thanks for help.</p>