akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Efficiently computing gradients for each example in a large dataset

username: alirs
cooked: <p>Given a trained model (M), I’m interested in computing the utility of new (unseen) examples in a pool (for an active learning task).  For this, I need to compute the magnitude of the gradient when M is trained on each new example.  In code, it is something like:</p>
<pre><code class="lang-auto">losses, grads = [], []
for i in range(X_pool.shape[0]):
    pred = model(X_pool[i:i+1])
    loss = loss_func(pred, y_pool[i:i+1])

    model.zero_grad()
    loss.backward()

    losses.append(loss)
    grads.append(layer.weight.grad.norm())
</code></pre>
<p>However, this is quite slow when there is a large pool of examples, especially since this will be an inner loop in my scenario.  How can this code be improved for efficiency?</p>
<p>I appreciate any suggestions or pointers.  Thanks!</p>

username: Vania_Todorova
cooked: <p>did you ever figure out nything ? i m doing a linear regression with 1 layer but its diverging … any advice?</p>
How to train PyTorch transfer learning tutorial with more then 1 GPU

username: radeeplearning
cooked: <p>Hi there,</p>
<p>I am currently following the PyTorch transfer learning tutorial in: <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a></p>
<p>I have been able to complete tutorial and train on both a CPU and 1 GPU.</p>
<p><strong>I am utilising Google Cloud Platform Notebook Instances and using 4 NVIDIA Tesla k80 x 4 GPU. It is here that I run into a Server Connection Error (invalid response: 504) error when I train the network on more than 1 GPU</strong></p>
<p>The screen shot of the error is attached<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/ae06790143450204c8575d105d4608f72e608cef.png" data-download-href="https://discuss.pytorch.org/uploads/default/ae06790143450204c8575d105d4608f72e608cef" title="error"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/ae06790143450204c8575d105d4608f72e608cef_2_298x500.png" alt="error" data-base62-sha1="oPuWKc8aUz7bmHG5GJK49m4hTsj" width="298" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/ae06790143450204c8575d105d4608f72e608cef_2_298x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/a/ae06790143450204c8575d105d4608f72e608cef_2_447x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/a/ae06790143450204c8575d105d4608f72e608cef_2_596x1000.png 2x" data-dominant-color="0F0F0F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error</span><span class="informations">1080×1807 90.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I enable Data Parallelism through the code shown below</p>
<pre><code class="lang-auto">model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 2)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

## Using 4 GPUs
if torch.cuda.device_count() &gt; 1:
    model_ft = nn.DataParallel(model_ft)
model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=25)
</code></pre>
<p>Please advice if I am taking some wrong step in enabling multi GPU with pytorch</p>
W.grad becomes None

username: birzhanm
cooked: <p>I am having a trouble with understanding the workings of .backward() and .grad() in PyTorch. Particularly, w.grad seems to become None type in the following code. Could you help to clarify what’s going on here. I am using PyTorch version 1.0.0. Thanks.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f03da5a1787f7b9c45ca1a40cdb88c28353d348d.png" data-download-href="https://discuss.pytorch.org/uploads/default/f03da5a1787f7b9c45ca1a40cdb88c28353d348d" title="17"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f03da5a1787f7b9c45ca1a40cdb88c28353d348d_2_690x377.png" alt="17" data-base62-sha1="yhgFBgF8oXXrDxVXrU8lDGIHSCp" width="690" height="377" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f03da5a1787f7b9c45ca1a40cdb88c28353d348d_2_690x377.png, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f03da5a1787f7b9c45ca1a40cdb88c28353d348d_2_1035x565.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/f03da5a1787f7b9c45ca1a40cdb88c28353d348d.png 2x" data-dominant-color="E9EAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">17</span><span class="informations">1140×623 68.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: rasbt
cooked: <p>Not sure what’s going on there, but your issue is probably fixed if you change <code>x_data</code> to</p>
<pre><code>x_data = torch.tensor([1., 2., 3.])
</code></pre>
<p>and the same for <code>y_data</code>.</p>

username: chenhao1umbc
cooked: <p>l.backward()<br>
with torch.no_grad():<br>
w = w - 0.01*w.grad<br>
w.requires_grad = True</p>
Custom Loss Function RCE

username: goffta
cooked: <p>Hi,</p>
<p>I’m trying to implement a custom loss function that is a regularization of the standard cross entropy loss function. Essentially I’m adding a penalty parameter. I was wondering the best way to implement this loss function - as a nn.Module, or just a def myloss() type. I can provide specifications for the function if necessary.</p>
<p>Thanks in advance!</p>

username: ptrblck
cooked: <p>If your custom loss function uses internal parameters or some other arguments, I would use the <code>nn.Module</code> approach as it will properly encapsulate all attributes.<br>
On the other hand you could just define a method, if you are using a pure functional approach.</p>

username: goffta
cooked: <p>Thanks for the input! I think I’ll go with the nn.Module approach. If I do this, I shouldn’t need to define a backward function, only a forward, correct?</p>

username: ptrblck
cooked: <p>If you are using PyTorch methods only (no numpy etc.) you can just define <code>forward</code>, as Autograd will automatically create the backward.</p>

username: goffta
cooked: <p>Before implementing the RCE (adding penalty parameter), I wanted to make sure my model works with the regular cross entropy function (the RCE required for my model only guarantees smoothness in probability distribution - when penalty parameter is set to 0 in my custom loss, it is the normal CE loss function). However, I don’t believe I have the data in correct format. When calling:</p>
<pre><code class="lang-auto">nn.CrossEntropyLoss()(out, target.squeeze())
</code></pre>
<p>I get a run time error:</p>
<pre><code class="lang-auto">RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15
</code></pre>
<p>A little background:<br>
My data looks like this:</p>
<pre><code class="lang-auto">1.0, 72
0.9741266547569558, 69
0.8379395394315619, 68
0.6586391650305975, 77
0.8268797607858336, 55
0.1315648101238786, 69
0.016174165344821745, 74
0.09840399297894489, 87
0.6690187519456767, 65
0.38906138226255316, 68
</code></pre>
<p>The data is representative of: y,C(dy), where y is the output of a defined mathematical function, and C(dy) is the class to which each point should belong. (With this specific data set, I have 144 classes.)</p>
<p>dy is generated by taking y[i+1] - y[i]. The number of classes for the entire data set is:</p>
<pre><code class="lang-auto">int(np.floor(abs(dy.amax - dy.amin) / 0.04)).
</code></pre>
<p>The 0.04 is just a specified classification ‘bin’ width.</p>
<p>The line before I call the loss function, my out is a tensor of size [20,100], and my target is a tensor of size [20,100] (after a target.squeeze().</p>
<p>Any ideas?</p>
<p>Edit: Fix formatting.</p>

username: goffta
cooked: <p>After further investigation, I did not have the correct class. I ended up removing everything relating to numpy and used pure pytorch methods. I fixed the loss using CEL by calling:</p>
<pre><code class="lang-auto">loss = nn.CrossEntropyLoss()(out, torch.max(target,1)[1])
</code></pre>
<p>However, I’m not quite sure what changing the targets argument actually did.</p>
How to use an LSTM where the input is not needed (e.g. decoder)

username: pinocchio
cooked: <p>I want to implement a decoder but I don’t need its input. Do I just pass a <code>torch.zeros</code> for its input?</p>
<p>e.g.</p>
<pre><code class="lang-auto">def main_test_zero_input_to_lstm():
    ## model params
    n_layers, nb_directions = 1, 1
    hidden_size = 64
    input_size = 3
    decoder = nn.LSTM(input_size=input_size,hidden_size=hidden_size).to(device)
    print(lstm.hidden_size)
    ## initialize the hidden state(s)
    batch_size = 1
    h_n = torch.randn(n_layers*nb_directions, batch_size, hidden_size)
    c_n = torch.randn(n_layers*nb_directions, batch_size, hidden_size)
    hidden = (h_n, c_n)
    ## pass through fake data
    seq_len, embedding_dim = 1, input_size
    zeros_fake_data = torch.zeros(seq_len, batch_size, embedding_dim)
    #zeros_fake_data = torch.zeros(seq_len, embedding_dim) # this throws an error, we need the batch dimension!!!
    ## do a decoder step
    out, hidden = decoder(zeros_fake_data, hidden)
    ##
    print(f'out size equals hidden state size: {out.size() ==  hidden[0].size()}')
    print(f'out.size() = {out.size()}')
    print(out)

if __name__ == '__main__':
    print('start')
    main_test_zero_input_to_lstm()
    print('DONE \a')
</code></pre>
<p>or is this bad practice? Perhaps I do need its input from previous step…?</p>
Plz help how to write this code in pytorch

username: Sunita_Choudhary
cooked: <p>def build_combined_onehot(FLAGS, NUM_FILTERS, FILTER_LENGTH1, FILTER_LENGTH2):<br>
XDinput = Input(shape=(FLAGS.max_smi_len, FLAGS.charsmiset_size))<br>
XTinput = Input(shape=(FLAGS.max_seq_len, FLAGS.charseqset_size))</p>
<pre><code>encode_smiles= Conv1D(filters=NUM_FILTERS, kernel_size=FILTER_LENGTH1,  activation='relu', padding='valid',  strides=1)(XDinput)
encode_smiles = Conv1D(filters=NUM_FILTERS*2, kernel_size=FILTER_LENGTH1,  activation='relu', padding='valid',  strides=1)(encode_smiles)
encode_smiles = Conv1D(filters=NUM_FILTERS*3, kernel_size=FILTER_LENGTH1,  activation='relu', padding='valid',  strides=1)(encode_smiles)
encode_smiles = GlobalMaxPooling1D()(encode_smiles) #pool_size=pool_length[i]


encode_protein = Conv1D(filters=NUM_FILTERS, kernel_size=FILTER_LENGTH2,  activation='relu', padding='valid',  strides=1)(XTinput)
encode_protein = Conv1D(filters=NUM_FILTERS*2, kernel_size=FILTER_LENGTH1,  activation='relu', padding='valid',  strides=1)(encode_protein)
encode_protein = Conv1D(filters=NUM_FILTERS*3, kernel_size=FILTER_LENGTH1,  activation='relu', padding='valid',  strides=1)(encode_protein)
encode_protein = GlobalMaxPooling1D()(encode_protein)



encode_interaction = keras.layers.concatenate([encode_smiles, encode_protein])
#encode_interaction = keras.layers.concatenate([encode_smiles, encode_protein], axis=-1) #merge.Add()([encode_smiles, encode_protein])

# Fully connected 
FC1 = Dense(1024, activation='relu')(encode_interaction)
FC2 = Dropout(0.1)(FC1)
FC2 = Dense(1024, activation='relu')(FC2)
FC2 = Dropout(0.1)(FC2)
FC2 = Dense(512, activation='relu')(FC2)


predictions = Dense(1, kernel_initializer='normal')(FC2) 

interactionModel = Model(inputs=[XDinput, XTinput], outputs=[predictions])
interactionModel.compile(optimizer='adam', loss='mean_squared_error', metrics=[cindex_score]) #, metrics=['cindex_score']


print(interactionModel.summary())
#plot_model(interactionModel, to_file='figures/build_combined_onehot.png')

return interactionModel</code></pre>

username: ptrblck
cooked: <p>Have a look at the general approach of writing a model in <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#define-a-convolutional-neural-network" rel="nofollow noopener">this tutorial</a>.<br>
You could use the PyTorch equivalent methods for the ones you’ve posted, e.g.</p>
<ul>
<li>Conv1D corresponds to <code>nn.Conv1d</code> in PyTorch</li>
<li>
<code>keras.layers.concatenate</code> can be done via <code>torch.cat((a, b), dim=your_dim)</code> in the <code>forward</code> method</li>
<li>Dense corresponds to <code>nn.Linear</code> in PyTorch</li>
</ul>
<p>Let us know, if you get stuck! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to change learning rate with c++?

username: zhijie1996
cooked: <p>how to use lr_schedule with c++ API ? for example: if I want to change the lr every 3 epoch, how to make it.<br>
torch::optim::SGD optim(model-&gt;parameters(), torch::optim::SGDOptions(1e-3).momentum(0.9));</p>

username: zhijie1996
cooked: <p>I find a way that directly changing ‘optim.options.learning_rate_’ by custom.</p>

username: mhubii
cooked: <p>Hi <a class="mention" href="/u/zhijie1996">@zhijie1996</a>, could you post your solution here? Would be awesome <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Sampling sub tensors from a n dimensional tensor

username: Boazrciasn
cooked: <p>Hello, I am trying to implement some kind of importance sampling from a pseudo in the following code,</p>
<p>inp = torch.rand([32, 20, 10, 16])<br>
inp_norms = torch.norm(inp, dim=3, keepdim=True, p=2)<br>
print(inp.size())<br>
sums = torch.sum(inp_norms, dim=1, keepdim=True)<br>
probs = inp_norms/sums<br>
t = torch.split(probs, dim=2, split_size_or_sections=1)<br>
sampls = torch.multinomial(inp_norms, num_samples=15, replacement=True)<br>
samples = torch.stack([torch.multinomial(tt, num_samples=15, replacement=True) for tt in t], dim=2)<br>
print(samples.size())</p>
<p>I would like to get 15 samples from dim=1 of inp for each element in dim=2. samples contain the indices of the samples but I can not get the samples from inp. (I do not want to use 2 for loops one for batch dimension and second for dim=2 of inp.) Is there any other way that I can follow, I would be glad if someone can at least give me keywords to search:D</p>
<p>index_select and gather did not do good to me or at least I could not understand them enough to convert to my problem.</p>
<p>Best and thanks in advance!<br>
Barış</p>
Train with some Image path from csv not present in Image directory: Dataloader

username: Mdhvince
cooked: <p>Hello everyone,</p>
<p>I would like to remove some images from my Image directory, and I was wonder if the PyTorch Dataloader is able to train on custom dataset by reading image path from csv then if the image is not found, juste pass to another image.</p>
<p>Is the PyTorch Dataloader can handle this or I have to remove these images from the csv file ?</p>
<p>Thank you all,</p>
<p>Medhy</p>

username: ptrblck
cooked: <p>If you are fine with some repeated images, you could check in the <code>__getitem__</code> method of your <code>Dataset</code>, if the current file exists, and if not just sample a new random index.<br>
This approach will repeat as many valid images as there are missing ones.</p>
<p>Would that work for you?</p>

username: Mdhvince
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, interesting but unfortunately I prefer not to have repeated images. Is there a way to juste pass to the next iteration if the image is not present ?</p>
How does one check if a tensor/parameter in pytorch is trainable?

username: pinocchio
cooked: <p>How does one check if a tensor/parameter in pytorch is trainable?</p>

username: pinocchio
cooked: <p>check if it has flag <code>requires_grad</code> to true:</p>
<pre><code class="lang-auto">Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; x = torch.randn(3,4)
&gt;&gt;&gt; x
tensor([[ 0.5905,  0.7622,  0.5487,  0.4738],
        [-0.2488,  0.1633,  2.3185, -0.6563],
        [ 1.0734, -0.3375, -0.5587,  0.3377]])
&gt;&gt;&gt; x.grad
&gt;&gt;&gt; x.requires_grad
False
&gt;&gt;&gt; x.requires_grad = True
&gt;&gt;&gt; x
tensor([[ 0.5905,  0.7622,  0.5487,  0.4738],
        [-0.2488,  0.1633,  2.3185, -0.6563],
        [ 1.0734, -0.3375, -0.5587,  0.3377]], requires_grad=True)
&gt;&gt;&gt;
</code></pre>
<p>source: <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html</a></p>

username: albanD
cooked: <p>Yes a Tensor will have it’s <code>.grad</code> field populated if <code>requires_grad=True</code> and <code>is_leaf=True</code>.</p>
Overwrite backward propagation for nn.Parameter for weight quantization

username: Hengck
cooked: <p>i have written a torch.autograd.Function for quantization as:</p>
<pre><code class="lang-auto">class Quantize(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        return input.round()
    @staticmethod
    def backward(ctx, grad_output):
        return grad_output #pass through 

fake_uniform_quantize = Quantize.apply
</code></pre>
<p>how can i apply to a weight parameter? , E.g</p>
<pre><code class="lang-auto">self.kernel = nn.Parameter(torch.Tensor(out_channel, in_channel, kernel_size, kernel_size))
self.kernel = fake_uniform_quantize(self.kernel)
</code></pre>
<p>But, I end up with the error:</p>
<pre><code class="lang-auto">TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'kernel' (torch.nn.Parameter or None expected)
</code></pre>

username: Hengck
cooked: <p>i solved the problem by</p>
<pre><code class="lang-auto">self.kernel = nn.Parameter(torch.Tensor(out_channel, in_channel, kernel_size, kernel_size))
kernel = fake_uniform_quantize(self.kernel)

then ...

x = F.conv2d(x, kernel, None, stride=self.stride, padding=0, dilation=self.dilation, groups=self.group)

</code></pre>
Multiple GPUs with user defined functions

username: rahul
cooked: <p>I am trying to use multiple GPUs for a deep neural network. When I restrict to a neural network with forward and backward functions, torch nn dataparallel works fine. However, I would like to include one more function in the neural network like this</p>
<pre><code>class MyNN(nn.module):
def __init__(self):
    super().__init__()
    self.layer = nn.Linear(10, 20)
def forward(self, x):
    x = self.block(x)
    return x
def evaluate(self,x):
   return some_scalar_value(x,self)
</code></pre>
<p>The function evaluate returns some scalar value. I am wondering how to average over the values obtained by multiple GPUs? If I had to simply do forward and backward pass, I could simply use</p>
<p>model = MyNN()<br>
model = torch.nn.Dataparallel(model,device_ids)</p>

username: Miguel_Rodriguez
cooked: <p>i have the same problema, how solve it?</p>

username: kaiwang13
cooked: <pre><code class="lang-python">eval_model = MyNN()
train_model = torch.nn.Dataparallel(eval_model,device_ids)
</code></pre>
<p>Use <code>eval_model</code> for evaluation and <code>train_model</code> for training. The parameters are shared.</p>

username: kaiwang13
cooked: <p>Please check my solution above for reference.</p>