akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
LSTM data configuration

username: Hyundong
cooked: <p>I want to learn a LSTM neural network with 20 timesteps and 1 feature, but I ask for help when it comes to data configuration difficulties.</p>
<p>Data is time-series data, one data is composed as follows, and the total data is 9000.</p>
<p>t1 - value1<br>
t2 - value2<br>
t3 - value3<br>
t4 - value4<br>
…<br>
t19 - value19<br>
t20 - value20<br>
(20 timesteps)</p>
<p>Here, I want to take 18 timesteps as inputs from the beginning and 2 as outputs (predictions).<br>
And, of the 9000 datasets, we want to use 80% for training and 20% for testing.<br>
In this case, the input dimension I think is (batch=?, seq_len=18, feature=1).<br>
So, here I understand the data composition for 1 dataset, but I don’t know how to construct the total dataset.</p>
<p>In addition, I would like to organize the data as shown in the picture below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/f/efca7781e2132bf25b4d5106a631803812738f9e.png" data-download-href="https://discuss.pytorch.org/uploads/default/efca7781e2132bf25b4d5106a631803812738f9e" title="question"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/efca7781e2132bf25b4d5106a631803812738f9e_2_690x229.png" alt="question" data-base62-sha1="ydhTIg4dFOp7Wi1b7tDxXuIUM9M" width="690" height="229" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/efca7781e2132bf25b4d5106a631803812738f9e_2_690x229.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/f/efca7781e2132bf25b4d5106a631803812738f9e_2_1035x343.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/f/efca7781e2132bf25b4d5106a631803812738f9e.png 2x" data-dominant-color="4D525D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">question</span><span class="informations">1057×351 17 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
(data is independent, not time-series, just number of samples.)</p>
<p>I want your kind help.<br>
Thank you.</p>
Unfold is slower than Convolution

username: rahulmoh
cooked: <p>Hi,</p>
<p>I’m trying to implement a custom Convolution layer in PyTorch and need to use the im2col functionality in order to convert the convolution operation into matrix multiplication. To do so, I am utilizing the Unfold class in PyTorch. I benchmarked the performance and it seems like Unfold by itself is slower than Conv2d. I used torch.cuda.synchronize() to make sure the benchmarking was done properly. This is a problem for me since my goal is to eventually speed up the Convolutional layer – is there any way to do the Unfold operation more efficiently in PyTorch? Appreciate any tips.</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>This is expected, as <code>unfold</code> would explicitly create the matrix so that you are able to perform the <code>matmul</code> afterwards. Some libraries such as <code>cudnn</code> ship with tuned algorithms (for common conv layer setups) in order to speed up the operation.</p>
Loss value decreases slowly

username: Ella
cooked: <p>Hi everyone,</p>
<p>I have an issue with my UNet model, in the upsampling stage, I concatenated convolution layers with some layers that I created, for some reason my loss function decreases very slowly, after 40-50 epochs my image disappeared and I got a plane image with some pixels pattern.  Are there any suggestions on what it might be, and how can I fix it?<br>
I used perceptual loss combined with MSE loss<br>
torch.optim.SGD(prior_weights, lr=0.00001, momentum=0.9, weight_decay=0.0001)<br>
Thank<br>
Epoch:   0   Loss:  0.0021450244821608067<br>
Epoch:   1   Loss:  0.002144632861018181<br>
Epoch:   2   Loss:  0.0021442414727061987<br>
Epoch:   3   Loss:  0.002143850550055504<br>
Epoch:   4   Loss:  0.0021434600930660963<br>
Epoch:   5   Loss:  0.0021430705673992634<br>
Epoch:   6   Loss:  0.0021426803432404995<br>
Epoch:   7   Loss:  0.0021422908175736666<br>
Epoch:   8   Loss:  0.002141902456060052<br>
Epoch:   9   Loss:  0.002141514327377081<br>
Epoch:   10   Loss:  0.00214112619869411<br>
Epoch:   11   Loss:  0.0021407371386885643<br>
Epoch:   12   Loss:  0.0021403473801910877</p>

username: ptrblck
cooked: <p>It’s a bit hard to speculate what might be the root cause of this issue.<br>
I would generally recommend to try to overfit a small dataset (e.g. just 10 samples) by playing around with some hyperparamters and make sure your current model as well as training setup is able to do so.<br>
Once this is done you could scale up the use case again.</p>
How to create ROC Curve for Resnet NN

username: snowflaker
cooked: <p>Hi, trying to take the resnet50 model I have defined in PyTorch and generate an ROC curve-unsure of what to insert code-wise to generate the data for an ROC curve</p>
<pre><code class="lang-auto">for epoch in range(3): 
    running_loss = 0.0
    for i, data in enumerate(trainloader_aug, 0):
        
        inputs, labels = data
        inputs, labels = Variable(inputs.cuda()), Variable(labels.cuda())
        optimizer.zero_grad()
        outputs = (net(inputs)).cuda()
        loss = criterion(outputs, labels)
        loss.backward()        
        optimizer.step()

        running_loss += loss.data[0]
        if i % 10 == 9: # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' % (epoch+1, i+1, running_loss / 1000))
            running_loss = 0.0

    correct = 0
    total = 0    
    for data in testloader:
        images, labels = data
        images = images.cuda()
        labels = labels.cuda()
        outputs = net(Variable(images)) 
        outputs = outputs.cuda()
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum()
      
</code></pre>

username: snowflaker
cooked: <p>I would greatly appreciate some assistance with this. Thank you.</p>

username: ptrblck
cooked: <p>You could use the <a href="https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html"><code>sklearn.metrics.roc_curve</code></a> example and adapt the code to use the targets and predictions for the ROC curves.</p>
ValueError: Expected target size (128, 44), got torch.Size([128, 100]), LSTM Pytorch

username: yf19001
cooked: <p>I want to build a model, that predicts next character based on the previous characters.<br>
I have spliced text into sequences of integers with length = 100(using dataset and dataloader).<br>
I want to predict character at each timestep.</p>
<p>Dimensions of my input and target variables are:</p>
<pre><code>inputs dimension: (batch_size,sequence length). In my case (128,100)
targets dimension: (batch_size,sequence length). In my case (128,100)
</code></pre>
<p>After forward pass I get dimension of my predictions: (batch_size, sequence_length, vocabulary_size) which is in my case (128,100,44)</p>
<p>but when I calculate my loss using <code>nn.CrossEntropyLoss()</code> function:</p>
<pre><code>batch_size = 128
sequence_length   = 100
number_of_classes = 44
# creates random tensor of your output shape
output = torch.rand(batch_size,sequence_length, number_of_classes)
# creates tensor with random targets
target = torch.randint(number_of_classes, (batch_size,sequence_length)).long()

# define loss function and calculate loss
criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
print(loss)
</code></pre>
<p>I get an error:</p>
<pre><code>ValueError: Expected target size (128, 44), got torch.Size([128, 100])
</code></pre>
<p>Question is: how should I handle calculation of the loss function for many-to-many LSTM prediction? I want to predict a character at each timestep. According to <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">nn.CrossEntropyLoss</a> Dimension must be(N,C,d1,d2…dN), where N is batch_size,C - number of classes.</p>

username: ptrblck
cooked: <p>As given in the docs, the class dimension should be in <code>dim1</code>, so this would work:</p>
<pre><code class="lang-python">output = torch.rand(batch_size, number_of_classes, sequence_length)
</code></pre>
<p>In your training script, you could <code>permute</code> the output to match these dimensions.</p>