akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Alternating Least Squares as a custom loss

username: Pkpklong
cooked: <p>Hi everyone,</p>
<p>I’m sort of a beginner in Pytorch and I’m trying to implement a implicit matrix factorization model which uses Alternating Least Squares (ALS). I’m trying to see if it’s even possible to implement ALS as an optimizer or loss function to update gradients because I’m not seeing a way to “fix” one embedding’s gradient temporarily in order to calculate the gradient of another.</p>
<p>Any thoughts or inputs is much appreciated.</p>

username: tom
cooked: <p>As you have two separate (sets of) parameters to train: What you’d typically do is to define two optimizers (for users and items) and set the other’s <code>requires_grad</code> to false when computing an update for either.<br>
The next question could be whether you get significantly worse results when computing simultaneous updates instead of alternating ones, I must admit I’d probably try that, too.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Pkpklong
cooked: <p>Thanks a lot Tom for getting me started on this! The reason why I’m even considering ALS is because SGD with a small batch size is very slow using a single GPU, which makes sense because I’m not maximizing the GPU utilization by using a smaller batch size. However, with a larger batch, although SGD becomes blazing fast, my model converges to worse local minima, since a larger batch size would indicate less noise in the data. Noise is what we want when escaping local minimas.</p>
<p>Do you think I should continue pursuing ALS?</p>
How to run network with multiple independent inputs in parallel in Pytorch?

username: Jan_Vainer
cooked: <p>I have a neural network with two separate vectors as inputs, similar to <a href="https://stackoverflow.com/questions/51700729/how-to-construct-a-network-with-two-inputs-in-pytorch" rel="nofollow noopener">this question</a>. Both inputs are encoded and then processed further. But until then, the encoding of those inputs is completely independent. How can I parallelize the encoding phase in pytorch? A minimal example of my code:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(params):
        super(MyModel self).__init__()

        self.encoder1 = Encoder1()
        self.encoder2 = Encoder2()

    def forward(x1, x2):
        # how to calculate both encodings in parallel?
        enc1 = self.encoder1(x1)
        enc2 = self.encoder2(x2)
        return some_func(enc1, enc2)
</code></pre>
<p>I checked pytorch forums and learned a bit about DataParallel, but i am not sure how to fit it to my case and on only 1 GPU.</p>

username: mrshenli
cooked: <p>DataParallel won’t help here if there is only one GPU, but you could use multiple CUDA streams. For example,</p>
<pre><code class="lang-auto">s0 = torch.cuda.Stream()
s1 = torch.cuda.Stream()
with torch.cuda.stream(s0):
    enc1 = self.encoder1(x1)

with torch.cuda.stream(s1):
    enc2 = self.encoder2(x2)
</code></pre>

username: Jan_Vainer
cooked: <p>Thank you very much! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Anchor Generator in Faster R-CNN implementation

username: farhad-bat
cooked: <p>I was reading Faster R-CNN codes that can be find <a href="https://github.com/pytorch/vision/blob/ed5b2dc3a5e7411d8b40cc7e526e151983e99cf9/torchvision/models/detection/faster_rcnn.py#L24" rel="nofollow noopener">here</a>. I don’t understand part of it that I explain it below.</p>
<pre><code class="lang-auto">if rpn_anchor_generator is None:
            anchor_sizes = ((32,), (64,), (128,), (256,), (512,))
            aspect_ratios = ((0.5, 1.0, 2.0),) * len(anchor_sizes)
            rpn_anchor_generator = AnchorGenerator(
                anchor_sizes, aspect_ratios
            )
</code></pre>
<p>This code is for anchor generator when argument in None, It with be default setting for anchor generator. why anchor sizes is ((32,), (64,), (128,), (256,), (512,)) ? what is it mean? is it mean backbone contains 5 set of output? What backbone is assumed?</p>
<pre><code class="lang-auto"> box_roi_pool = MultiScaleRoIAlign(
                featmap_names=[0, 1, 2, 3],
                output_size=7,
                sampling_ratio=2)
</code></pre>
<p>Why roi has featmap [0,1,2,3] ? What backbone is assumed? why it is different from anchor generator?</p>
DCGAN tutorial - generator does not converge

username: albusdemens
cooked: <p>Hi, I am trying to generate trees from an images dataset I prepared. As a tool, I am building on the <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="noopener nofollow ugc">DCGAN tutorial</a>. During the training, the loss function of the generator does not converge (see image below). Do you have any suggestion on how to fix this? I already tried changing the learning rate from 0.0002 to 001, but the behavior of the loss function did not change. Thanks for the help!</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f217f98c0f4536a6638f36543cdb6e60fe03285a.png" alt="index" data-base62-sha1="yxEUDU2iyxyOnM9QA2z8bOIVQ0i" width="606" height="333"></p>

username: ptrblck
cooked: <p>Are you resetting some parameters after one epoch or are you now shuffling the real dataset?<br>
The loss spikes look a bit weird to me.</p>
LSTM with LBFGS optimizer, purpose of detach()

username: mjeat21
cooked: <p>I am building an LSTM net for time-series prediction. I found an example from <a href="https://github.com/pytorch/examples/blob/master/time_sequence_prediction/train.py" rel="nofollow noopener">github</a> and tried to implement it myself.</p>
<p>I have a class to monitor the training process. The training function is as below:</p>
<pre><code>def trainModel(self, epochMax=100):
    for i in range(epochMax):
        self.h = torch.zeros(layerNo, datasetNo, D_H)
        self.c = torch.zeros(layerNo, datasetNo, D_H)
        self.h = self.h.to(device)
        self.c = self.c.to(device)

        def closure(): #this is needed for LBFGS optimizer
            #***Why do I need these four lines?
            self.h = self.h.detach()
            self.c = self.c.detach()
            self.h = self.h.requires_grad_()
            self.c = self.c.requires_grad_()
            
            yPred, h_temp, c_temp = self.model(self.XTrainAll, self.h, self.c)
            self.optimizer.zero_grad()
            self.h, self.c = h_temp, c_temp
            loss = self.lossFn(yPred, self.YTrainAll)
            # print('loss:', loss.item())
            loss.backward()
            return loss
        loss = self.optimizer.step(closure)
</code></pre>
<p>As seen from the example code, they do not implement the <code>detach()</code> function. But for my case, if these lines are missing, the error <code>RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</code> appears.</p>
<p>I have gone through this <a href="https://discuss.pytorch.org/t/solved-training-a-simple-rnn/9055">post</a> but I am still not clear about the difference between my code and the github code.</p>
<p>Also, is there a way to pass <code>h</code> and <code>c</code> into the <code>closure()</code> function without defining them as class variables? Python complains about undefined variables when I define them as just <code>h</code> and <code>c</code> outside of the function. Having <code>global h</code> at the beginning of <code>closure()</code> does not help.</p>
<p>Thanks in advance.</p>

username: Yaroslav_Bulatov
cooked: <p>Regarding h,c variables, if there’s an assignment to <code>h</code> anywhere in the function, Python will treat h as a local symbol that shadows <code>h</code> from the outer context.</p>
<p>Assignment to an element of <code>h</code> does not trigger this behavior, so the the standard workaround is to wrap the variable from outer context into a singleton list.</p>
<pre><code class="lang-auto">myvar = 5
myvar = [myvar]
def closure():
  myvar[0]=6

closure()
print(myvar[0]) # =&gt; 6
</code></pre>
What is an example of using Hessian Vector Product in Learning using Pytorch?

username: pinocchio
cooked: <p>I was looking at the famous paper MAML (Model Agnostic Meta-Learning) and they say they use Hessian-Vector products. How do they use them:</p>
<ol>
<li>
<p>in mathematics? i.e. how do the Hessian vector products get involved in the update rule?</p>
</li>
<li>
<p>in the code. How do they update the parameters being learned?</p>
</li>
</ol>
<p>Or is there a nice example of how Hessian Vector products are used in (meta) training?</p>
<hr>
<p>cross posted:</p>
<ul>
<li><a href="https://www.quora.com/unanswered/What-is-an-example-of-using-the-Hessian-vector-product-in-learning-using-PyTorch" rel="nofollow noopener">https://www.quora.com/unanswered/What-is-an-example-of-using-the-Hessian-vector-product-in-learning-using-PyTorch</a></li>
<li><a href="https://www.reddit.com/r/pytorch/comments/dhcn8c/what_is_an_example_of_using_hessian_vector/" rel="nofollow noopener">https://www.reddit.com/r/pytorch/comments/dhcn8c/what_is_an_example_of_using_hessian_vector/</a></li>
<li><a href="https://discuss.pytorch.org/t/what-is-an-example-of-using-hessian-vector-product-in-learning-using-pytorch/58061" class="inline-onebox">What is an example of using Hessian Vector Product in Learning using Pytorch?</a></li>
</ul>
R: element 0 of tensors does not require grad and does not have a grad_fn

username: Nidutt_Bhuptani
cooked: <p>So i am trying to Run a neural network that tries to detect the YOGA style. I scrapped dataset from google for 13 diiferent YOGA positions. Here is my NN. Kindly help as i am getting this error<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
<pre><code class="lang-auto">
dl = torch.utils.data.DataLoader(YogaPoseDataset('classifier_train_data2.csv', trn_tfms, 
                                                           pose_id_to_name, pose_name_to_id),
                                           bs, 
                                           shuffle=True)
model = resnet34(pretrained=True)
model.fc = nn.Linear(512, 14)   
model_utils.freeze_all_layers(model)
model = model.to(device)

model.train()
n_epochs = 2
lr = 0.00001
loss_fn = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr = lr)

print("Fine till here")

total_steps = 0
for e in range(n_epochs):
    avg_batch_accuracy = []
    for batch, labels in dl:
        optimizer.zero_grad()
        
        batch = batch.to(device)
        labels = labels.to(device)
        print(labels)
        
        preds = model(batch)
        print(preds)
        loss = loss_fn(preds, labels)
        loss.backward()
        optimizer.step()
                
        pred_labels = preds.argmax(dim=1)
        total_steps +=1
        batch_accuracy = (labels == pred_labels).float().mean()   
        avg_batch_accuracy.append(batch_accuracy.item())
        model_utils.print_training_loss_summary(loss.item(), total_steps, e+1, n_epochs, len(dl))
    
    print('avg batch accuracy:{:.2f}'.format(np.array(avg_batch_accuracy).mean()))
</code></pre>

username: tom
cooked: <p>The error message wants to tell you it doesn’t have anything w.r.t. which you would want to compute grads.<br>
One might naïvely think <code>freeze_all_layers</code> could cause training to bite on ice here.</p>
<p>Best regards</p>
<p>Thomas</p>