akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch efficiently multiplying by zeros

whoab: <p>Let’s say we have dropout or something else that zero’s out part of the state space. Would a forward pass and backward pass still take the same time compared to the non-zero’d out input?</p>

whoab: <p>And if so, why. We should at least be able to avoid matrix multiplications and just sum the biases. There should be a shortcut.</p>
<p>Note: I am not talking about ‘mostly zero’ sparse matrices. I am talking about something more fundamental; you do not need to multiply by a matrix if the input is zero.</p>

Mazhar_Shaikh: <p>In my honest opinion, It comes down to a trade off. If you want to conditionally compute some operation, then the cost of the condition should also be considered.<br>
Total Cost of operations  = Cost of conditional check*Number of elements + Cost of multiply-add * Number of non-zero elements<br>
In most cases, The reduction in cost due to multiply add is not enough to compensate the additional cost of conditional checks.<br>
For sparse tensors, where more than 70% of the elements are zero, the trade off favors having checks.<br>
Assumption: Cost of multiply-add of 32-bit number is O(32 squared) while cost of checking whether 32-bit number is non-zero is O(32).</p>
Training Loss not improving so much

Sam_Fst: <p>Epoch: 0 – Training Loss: 0.057411<br>
Epoch: 1 –  Training Loss: 0.056986<br>
Epoch: 2 – Training Loss: 0.057286<br>
Epoch: 3 –  Training Loss: 0.057266</p>
<p>Anyone knows why it is sooo slow ??<br>
My data set have 1248 images for training</p>
<p>CODE:</p>
<p>model = models.densenet121(pretrained = True)</p>
<h1>manter os features parameters</h1>
<p>for param in model.parameters():<br>
param.requires_grad = False<br>
criterion = nn.NLLLoss()</p>
<p>optimizer = optim.Adam(model.parameters(), lr=0.003)</p>
<p>for e in range(epochs):<br>
# keep track of training and validation loss<br>
running_loss = 0.0<br>
running_corrects = 0.0</p>
<pre><code>for inputs, label in (dataloaders['train']):
    model.train()
    # IF GPU is availible
    if train_on_gpu:
        inputs, label = inputs.cuda(), label.cuda()
    
    optimizer.zero_grad()
    with torch.set_grad_enabled(True):
        logps = model(inputs)
        _, preds = torch.max(logps, 1) # tecnica nova de validacao
        loss = criterion(logps, label)
        loss.backward()
        optimizer.step()
    
        running_loss += loss.item()
        running_corrects += torch.sum(preds == label.data)</code></pre>

JuanFMontesinos: <p>I haven’t check but does densenet softmax already incorporated inside the model?</p>

Sam_Fst: <p>no , but this is not the problem here</p>

Mazhar_Shaikh: <p>Could you tell me why you are using this line?</p>
<aside class="quote no-group" data-username="Sam_Fst" data-post="1" data-topic="51651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_fst/48/14347_2.png" class="avatar"> Sam_Fst:</div>
<blockquote>
<p>for param in model.parameters():<br>
param.requires_grad = False</p>
</blockquote>
</aside>
<p>Seems to me that setting the requires grad to False, prevents the optimizer from calculating gradients for the parameters and prevents any optimization from taking place.</p>
Loss momentum behavior when accumulating gradients

Ali2500: <p>There have been a couple of general discussions about accumulating gradients before before (<a href="https://discuss.pytorch.org/t/how-to-implement-accumulated-gradient/3822" class="inline-onebox">How to implement accumulated gradient？</a> and <a href="https://discuss.pytorch.org/t/how-to-implement-accumulated-gradient-in-pytorch-i-e-iter-size-in-caffe-prototxt/2522/4" class="inline-onebox">How to implement accumulated gradient in pytorch (i.e. iter_size in caffe prototxt)</a>).</p>
<p>But I want to know how loss momentum behaves when we’re training a model like this:</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

# suppose we want to accumulate gradients for 'N' steps:
for i, input in enumerate(data_loader):
    output = model(input)
    loss = criterion(output, input) / N
    loss.backward()

    if i % N == 0 and i &gt; 0:
        optimizer.step()
        optimizer.zero_grad()
</code></pre>
<p>Are the loss momentum values overwritten at each call of <code>loss.backward()</code> or (desirably) when <code>optimizer.step()</code> is called?</p>

Mazhar_Shaikh: <p>Hi Ali,<br>
The loss variable is reassigned every iteration due to the line.</p>
<pre><code class="lang-auto">loss = criterion(output, input) / N
</code></pre>
<p>The momentum of the gradient is maintained by the optimizer for the parameters/tensors that it is tracking, in this case those are model.parameters().<br>
Hope this helps.</p>
Which algorithm PyTorch use to optimize L1 penalty say in case of LASSO

Suryabhan_Singh: <p>I really want to know which algorithm Pytorch use to optimize L1 penalty.<br>
Like scikit-learn clearly mentions that - “We use the truncated gradient algorithm proposed by Tsuruoka et al. 2009 for L1 regularization (and the Elastic Net).” (<a href="https://scikit-learn.org/stable/modules/sgd.html#implementation-details" rel="nofollow noopener">https://scikit-learn.org/stable/modules/sgd.html#implementation-details</a>).</p>
<p>I really searched a lot but I couldn’t find the answer.</p>

Mazhar_Shaikh: <p>Pytorch lets you choose which optimization algorithm to use for any penalty that you would like to optimize.Here’s an example that uses SGD to optimize the L1 penalty. <a href="https://stackoverflow.com/a/47196174" rel="nofollow noopener">link</a><br>
You could replace the SGD with <a href="https://pytorch.org/docs/stable/optim.html#algorithms" rel="nofollow noopener">any other optimizer</a> available in PyTorch or make your own custom one.</p>
Design my own resnet using torch.nn.Sequential, possible?

Chieh_Wu: <p>Hello</p>
<p>I have been playing with a basic fully connected neuralNet using the Sequential function.</p>
<blockquote>
<p>dcn.NN = torch.nn.Sequential(<br>
torch.nn.Linear(dcn.d, dcn.hidden_d, bias=True),<br>
torch.nn.ReLU(),<br>
torch.nn.Linear(dcn.hidden_d, dcn.hidden_d, bias=True),<br>
torch.nn.ReLU(),<br>
torch.nn.Linear(dcn.hidden_d, dcn.hidden_d, bias=True),<br>
torch.nn.ReLU(),<br>
torch.nn.Linear(dcn.hidden_d, dcn.hidden_d, bias=True),<br>
torch.nn.ReLU(),<br>
torch.nn.Linear(dcn.hidden_d, dcn.output_d, bias=True),<br>
torch.nn.Softmax(dim=1)<br>
)</p>
</blockquote>
<p>I know that pytorch has a pre-packaged resnet in the library. But to my understanding, it is a CNN. If I want a basic fully connected neuralNet with resnet structure, I assume I would need to build it myself?  If so how do I do it using the Sequential function?   Or do I have to do it without using the Sequential method?</p>
<p>Thank you in advanced.</p>
<p>Chieh</p>

smth: <p>you have to do it without the Sequential container, because you have to do the <code>+=</code> operation (i.e. residual connection)</p>

pinocchio: <p>I also want to know. Duplicate: <a href="https://discuss.pytorch.org/t/how-to-have-residual-network-using-only-sequential-blocks/51541" class="inline-onebox">How to have Residual Network using only Sequential Blocks?</a></p>
Calculating gradient for each layer without using loss.backward()

JakeNocentino: <p>Hi all,</p>
<p>I am in a bit of a predicament here. So right now, I am combining a neural network implemented in PyTorch and a conditional random field (graphical model) which is implemented in C++. I am passing in the output of the 2nd last layer of the neural network to the CRF. The CRF then uses that layer to calculate an AUC for its performance and also calculates the gradient of the second last layer. I then receive the gradient that the CRF calculated and place it in the .grad attribute of the second last layer.</p>
<p>At this point, my problem arises. I want to start back-propagation from this layer to all other parameters in my network, but this is hard to figure out to do without calling <code>loss.backward()</code> since each parameter is its own leaf node in the computational graph. Does anyone know any legit ways or any “hacky” ways to solve this?</p>
<p>Thanks!</p>

JuanFMontesinos: <p>Hi,<br>
Why don’t you define a custom autograd function involving CRF?<br>
This way typical pytorch workflow would keep working without “hacks”.</p>
<p><a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html</a></p>
<p>forgot to mention you can create c++ modules for pytorch</p>

JakeNocentino: <p>Hi,<br>
So you’re saying I could create a custom autograd function and that call backwards on the 2nd last layer in PyTorch?</p>

JuanFMontesinos: <p>You said you are computing auc  and backpropagating from there. As it is c++ you are pasting the gradient.</p>
<p>You can create a python wrapper which takes as input that layer and returns the auc as a tensor relying in that c++ implementation. This way you will be able to call auc.backward</p>