akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to calculate loss for small values?

username: ROODAY
cooked: <p>I’m trying to create a sequence to sequence lstm that will output a series of human poses in the form of 3D coordinates. The input data I’m training in is kept within a 1x1x1 bounding box, so all the coordinate values are &lt;1. Using traditional MSE for example will give me very small loss values even when the error is pretty high since all the values are &lt;1. I also realized that since the poses I have are from video frames (24 FPS), the frame to frame movement is very small, so my model ends up predicting a single pose many times. I figured the best way to ensure the model outputs a series of distinct poses would be to calculate loss on the diff of the values (e.g. np.diff(values)). However these values would be even smaller, so I tried to solve this with custom loss functions that calculate percentage based error.</p>
<p>However these attempts didn’t really solve my issue. When I tried to do percentage based error on just the coordinates themselves, I had noisy output that somewhat moved around but the poses weren’t anatomically correct (the point representing the center of the hip would be way off for example). Trying to do so on just the diff of the values didn’t give sensible output either (the coordinates were all clustered as a blob). I tried doing MAPE and relative percent difference on the diffs of the values and that ended up with my model returning NaN. Could anyone give me some insight on potential errors with my approach/better approaches?</p>
<p>For context, here are the error functions I created to try and do percentage error:</p>
<pre><code class="lang-auto">def MAPELoss(output, target):
  return torch.mean(torch.abs((target - output) / target))

def RPDLoss(output, target):
  return torch.mean(torch.abs(target - output) / ((torch.abs(target) + torch.abs(output)) / 2))

def MAPELoss_Diff(output, target):
  output = output[1:] - output[:-1]
  target = target[1:] - target[:-1]
  return torch.mean(torch.abs((target - output) / target))

def RPDLoss_Diff(output, target):
  output = output[1:] - output[:-1]
  target = target[1:] - target[:-1]
  return torch.mean(torch.abs(target - output) / ((torch.abs(target) + torch.abs(output)) / 2))
</code></pre>
Oversampling the minor classes(only single sample for two classes)

username: TanmDL
cooked: <p>Hi,<br>
I have a imbalance dataset = [3, 41, 29, 7, 14, 20, 1, 15, 1, 36, 64, 22, 4, 28, 10, 2]<br>
I want to make a dataloader which can generate equal number of samples for for each class (e.g. 4*16=64 batch_size). I have used WeightedRandomSampler but I was not getting the exact results.<br>
The demo code is given below:</p>
<h5>count function</h5>
<p>def count_(x):<br>
x_unique = x.unique(sorted=True)<br>
x_unique_count = torch.stack([(x==x_u).sum() for x_u in x_unique])<br>
return x_unique_count</p>
<p>train_weights = 1./torch.as_tensor(count, dtype=torch.double)   ##### torch.double<br>
train_sampleweights = train_weights[train_dataset.target]<br>
train_sampler = WeightedRandomSampler(weights=train_sampleweights, num_samples = len(train_sampleweights))</p>
<h6>dataloader</h6>
<p>balance_loader = DataLoader(train_dataset,<br>
sampler=train_sampler,<br>
batch_size=64)<br>
for i,(data, target_) in enumerate(balance_loader):<br>
<span class="hashtag">#print</span>(target)<br>
count = count_(target_)<br>
print(torch.sum(count))<br>
print(count)</p>
<p>##<span class="hashtag">#results</span><br>
tensor([3, 2, 4, 4, 5, 5, 4, 4, 3, 5, 5, 2, 3, 4, 9, 2])<br>
tensor([4, 4, 5, 7, 1, 3, 6, 7, 5, 2, 4, 4, 4, 5, 3])<br>
tensor([3, 9, 6, 5, 2, 2, 4, 2, 4, 7, 3, 1, 5, 5, 2, 4])<br>
tensor([3, 3, 6, 3, 2, 4, 5, 2, 8, 3, 2, 7, 4, 6, 4, 2])<br>
tensor([2, 1, 2, 3, 5, 7, 2, 3, 4, 2, 3, 2, 4, 1, 2])</p>
Getting a subset(given their indices) of data from the data loader

username: Siddharth_Soni
cooked: <p>I need to train my neural network on a selected subset of the FashionMNIST training dataset. The indices of these are known. How can I pass the indices to the DataLoader so the samples are generated only from these indices.</p>
<pre><code class="lang-auto">train_fm = torchvision.datasets.FashionMNIST("./data/",
                                             train=True,
                                             transform=
                                             torchvision.transforms.
                                             Compose
                                             ([torchvision.transforms.ToTensor()]
                                              ),
                                             target_transform=None,
                                             download=True)
train_loader = torch.utils.data.DataLoader(
    train_fm, batch_size=100
)
</code></pre>
<p>Can i use the sampler argument to get it done? How?<br>
Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Siddharth_Soni
cooked: <p>The below solution is what I was looking for.<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">train_loader_2 = torch.utils.data.DataLoader(
    train_fm, batch_size=100,
    sampler =  sampler.SubsetRandomSampler(&lt;array containing indices&gt;)
)
</code></pre>

username: Berfu_Buyukoz
cooked: <p>thank you, friend! It worked for me too.</p>
How to make a Gym Environment Iterable Dataset

username: Etienne_Perot
cooked: <p>I don’t really understand the iterable dataset. I would like to stream different videos in different process, and that each video would be present in the batch like this:<br>
<span class="hashtag">#batch1:</span><br>
video0_frame0<br>
video1_frame0<br>
…<br>
videoB_frame0</p>
<p><span class="hashtag">#batch2:</span><br>
video0_frame1<br>
video1_frame1<br>
…<br>
videoB_frmae1</p>
<p>you get the picture.</p>
<p>i am trying like this:</p>
<pre><code class="lang-auto">from __future__ import absolute_import
from __future__ import division
from __future__ import print_function


import torch
from torch.utils.data import Dataset, IterableDataset, DataLoader
import gym
import numpy as np
import cv2


class GymEnv(object):
    def __init__(self, env_name, niter=1000):
        self.env = gym.make(env_name)
        self.niter = niter
    
    def __iter__(self):
        self.env.reset()
        for _ in range(self.niter):
            action = np.random.randint(0, self.env.action_space.n)
            observation, reward, done, info = self.env.step(action)
            if done: 
                self.env.reset()
            else:
                yield observation, reward, done, info
            yield self.env.step(action)


class Gym(IterableDataset):

    def __init__(self, env_name='SpaceInvaders-v0', niter=10000):
        self.env_name = env_name
        self.niter = niter
        self.start, self.end = 0, niter

    def preprocess(self, step):
        #We only have the text in the file for this case
        observation, reward, done, info = step
        return observation

    def __iter__(self):
        env_iter = GymEnv(self.env_name, self.niter)
        return map(self.preprocess, env_iter)

          
def worker_env_init(worker_id):
    worker_info = torch.utils.data.get_worker_info()
    dataset = worker_info.dataset  # the dataset copy in this worker process
    overall_start = dataset.start
    overall_end = dataset.end
    # configure the dataset to only process the split workload
    dataset.env_name = ['SpaceInvaders-v0', 'Pong-v0'][worker_info.id]

    print('dataset: ', dataset.env_name)


if __name__ == '__main__':
    
    ds = Gym()
    dl = DataLoader(ds, batch_size=2, num_workers=2, worker_init_fn=worker_env_init)

    # for data in ds:
    #     cv2.imshow('test', data[..., ::-1])
    #     cv2.waitKey(5)

    for idx, data in enumerate(dl):
        print('batch_idx: ', idx)
        for i in range(len(data)):
            img = data[i].cpu().numpy()
            cv2.imshow('env#'+str(i), img[..., ::-1])
        
        cv2.waitKey(0)
</code></pre>
<p>but batches basically contains several times the same image and the videos are alternatively in one batch or the over, never in one batch at the same time</p>
Resuming from model checkpoints

username: cswangjiawei
cooked: <p>I want to resume from model checkpoints, but the result of the model trained from the checkpoint on the development set is different from that of the model trained from scratch at the same epoch. Such as, if the training starts from scratch, at the third epoch, the accuracy of the model on the development set is 91.41666666666667, however, if the model trained from the checkpoint(checkpoint is second epoch), the accuracy of the model on the development set is 90.675 at the third epoch. I save the model using the following code：</p>
<pre><code class="lang-auto">checkpoint = {
            'epoch': epoch,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'best_acc': best_acc
        }
        torch.save(checkpoint, checkpoint_dir + os.sep + 'last_epoch.pt')
</code></pre>
<p>and I restore the model using the following code:</p>
<pre><code class="lang-auto">checkpoint = torch.load(checkpoint_dir+os.sep+'last_epoch.pt')
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
initepoch = checkpoint['epoch'] + 1
best_acc = checkpoint['best_acc']
</code></pre>
<p>I think the reason of the difference is the  DataLoader RNG, How I solve it?</p>

username: ptrblck
cooked: <p>Did you seed the code properly and is this drop in accuracy reproducible?<br>
I.e. do you always get these numbers if you rerun the entire code and try to just retrain the last epoch?</p>
<p>If you didn’t use any seeds (and are using e.g. <code>torch.backends.cudnn.benchmark = True</code>), I would assume your final accuracy might get some variance.</p>

username: cswangjiawei
cooked: <p>I used seed, the code is as follows:</p>
<pre><code class="lang-auto">seed_num = 123
random.seed(seed_num)
torch.manual_seed(seed_num)
np.random.seed(seed_num)
</code></pre>
<p>If I run the code from scratch twice, the accuracy of the same epoch on the development set is the same.<br>
Training from the breakpoint caused a change in the data shuffle order (the first epoch followed by training from the checkpoint at the end of the second epoch training is different from the data shuffle order of the third epoch training from scratch), I guess this It is the reason that the accuracy of the same epoch model on the development set is different. How to make the data shuffle when training from the checkpoint the same as training from the scratch?</p>
Please Need help with Prediction Neural Network

username: XxaemaethxX
cooked: <p>Hi, for my senior design we have are using the YOLOv3 network for objection detection of the game of casino game of Roulette (specifically identifying the 0 pocket and Roulette ball). We already have annotated those images (~2,000ish) data set at this point and our getting decent object detection results. We also have coded functions that compute speed (pixels/second), radius/distance (pixels), angular velocity (degrees/second), theta (degrees), acceleration (degrees/second) per frame and output that data to a txt file ( can be CSV if needed). I was figuring that I could use a predictor neural net for like predicting Weather, Stock Market or House prices, since the general concepts seem the same.</p>
<p>Here’s a sample : <a href="https://www.youtube.com/watch?v=HaMlKvNqCVs" rel="nofollow noopener">https://www.youtube.com/watch?v=HaMlKvNqCVs</a><br>
<a href="https://www.youtube.com/watch?v=HaMlKvNqCVs" rel="nofollow noopener">Beating Roulette</a></p>
<p>I thought outputting the parameters of each frame, like the initial speed/velocity of the ball, and where it finally lands (in what pocket) in a python data frame and run some Regression on it could produce “decent” results, i.e.</p>
<pre><code class="lang-auto">speed_initial    velocity_initial    acceleration_initial    theta_initial   pocket_final
   500,454              400                  433                  308             17
.....                .........            ......                 ......          .....
</code></pre>
<p>Obviously I would need a lot of data, but I thought the neural network would be able to build a model of what entry speeds and angles result in landing in some specific pocket.</p>
<p>Are there any better approaches? My teammate was planning on using a binary classifier to divide the wheel in 2-halves and the output parameters (speed, accel, theta, radius , etc…) into a binary classifier as a tensor…</p>
<p>If anyone has any suggestions please me know! I know there’s experts on these forums that may have done similar prediction models that maybe i can tweak/twist to fit this roulette prediction!</p>
RuntimeError: The size of tensor a (38) must match the size of tensor b (49) at non-singleton dimension 3

username: Shrutika994
cooked: <p>File “A1.py”, line 121, in <br>
outputs = model(inputs)<br>
File “/home/shru/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/shru/Nucleus/A1_.py”, line 247, in forward<br>
x3 = self.Att4(d4, x3)<br>
File “/home/shru/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/shru/Nucleus/A1_.py”, line 124, in forward<br>
psi = self.relu(g1 + x1)<br>
RuntimeError: The size of tensor a (38) must match the size of tensor b (49) at non-singleton dimension 3</p>

username: ptrblck
cooked: <p>I’m not sure what your use case is of what your model does exactly, but <code>g1</code> and <code>x1</code> cannot be added with different shapes.<br>
If you post the model definition, we could try to help debugging.</p>
Anaconda installation

username: stone19
cooked: <p>When Anaconda has been installed, there appears that “The system could not find the specified path”</p>

username: Eta_C
cooked: <p>Could you give more details?</p>

username: stone19
cooked: <p>I have downloaded the anaconda. But when installation has completed, I tested in w+R(cmd), there is no conda list.</p>

username: stone19
cooked: <p>In addition, anaconda installation could be Windows system?</p>

username: ptrblck
cooked: <p>Do you have problems with the Anaconda installation or with installing PyTorch?<br>
I’m currently not sure, what the error message means.<br>
If you search for “Anaconda” on Windows, do you see the Anaconda terminal?<br>
If so, could you install PyTorch there and execute it in the same environment?</p>
Get features from Mask R-CNN object

username: drapado
cooked: <p>Hi, as in this <a href="https://discuss.pytorch.org/t/extract-features-from-f-rcnn-mask-rcnn/56487/4">post</a> I want to get the feature vector prior to the classification layer of the final detected objects (final output of Mask R-CNN). How could this be done with the pytorch implementation of Mask R-CNN?<br>
Thanks!</p>

username: ptrblck
cooked: <p>Is the suggestion from the linked topic not working?<br>
If so, could you explain your use case a bit and how it differs from theirs? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Is L2 regularization through weight decay reflected in loss function?

username: MrRobot
cooked: <h2><a name="problem-1" class="anchor" href="#problem-1"></a>Problem</h2>
<p>I am following Andrew Ng’s deep learning course on Coursera. He <em>warns</em> that forgetting adding L2 regularization term into loss function might lead to wrong conclusions about convergence.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/dbd0934f48216400502c3b9e1aab7fcfe575b796.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/dbd0934f48216400502c3b9e1aab7fcfe575b796" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbd0934f48216400502c3b9e1aab7fcfe575b796_2_690x386.jpeg" alt="image" data-base62-sha1="vmzr598AYmMuNRUFNd0QtHA7yOa" width="690" height="386" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbd0934f48216400502c3b9e1aab7fcfe575b796_2_690x386.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/d/b/dbd0934f48216400502c3b9e1aab7fcfe575b796.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/b/dbd0934f48216400502c3b9e1aab7fcfe575b796.jpeg 2x" data-dominant-color="F6F7F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">748×419 84 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I know the L2 regularization could be implemented through <code>weight_decay</code> argument in <code>Adam(model.parameters(), lr=1e-4, weight_decay=1.0)</code>. However, I am not sure</p>
<ul>
<li>Whether regularization term is somehow <strong>automatically</strong> added to loss value when <code>weight_decay</code> is not 0?</li>
<li>If the answer to the first point is “no” and I add regularization through something like following, would it be like I add double regularization to my network?</li>
</ul>
<pre data-code-wrap="python"><code class="lang-python">for block in model.children():
  loss += torch.norm(block.weight)
</code></pre>
<ul>
<li>The second point assumes that blocks are all <code>nn.Linear</code>, but what if I would like to get the <strong>true</strong> loss term for CNN and LSTM (where <code>block.weight</code> might be replaced to architecture-specific weight tensor).</li>
</ul>
Is this a correct use of nn.Transformer?

username: vainaijr
cooked: <pre><code class="lang-auto">import torch, torch.nn as nn
</code></pre>
<pre><code class="lang-auto">class A(nn.Module):
  def __init__(self):
    super().__init__()
    self.embed_src = nn.Embedding(2, 10)
    self.embed_target = nn.Embedding(2, 10)
    self.transformer = nn.Transformer(10, 2)
    self.lin = nn.Linear(10, 2)
    self.softmax = nn.Softmax(dim=-1)
  def forward(self, inp, tgt):
    embed_src = self.embed_src(inp)
    embed_target = self.embed_target(tgt)
    # print(embedding.shape)
    output = self.transformer(embed_src.view(len(inp), 1, -1), embed_target.view((len(tgt)), 1, -1))
    output = self.lin(output)
    print('output.shape', output.shape, 'output', output)
    print('embed_target', embed_target.view(2, 1, -1))
    print('softmax probabilities', self.softmax(output))
    return output.permute(0, 2, 1)
</code></pre>
<pre><code class="lang-auto">model = A()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
</code></pre>
<pre><code class="lang-auto">dataset = [ 
            [([0, 1, 0, 1, 0, 1]), ([[0], [1]])],
            [([1, 0, 1, 0, 1, 0]), ([[1], [0]])],
            [([0, 0, 0, 0, 0, 0]), ([[0], [0]])],
            [([1, 1, 1, 1, 1, 1]), ([[1], [1]])],
            [([1, 1, 0, 0, 1, 1]), ([[0], [0]])],
            [([0, 0, 1, 1, 0, 0]), ([[1], [1]])]
           ]
</code></pre>
<pre><code class="lang-auto">tensor_dataset = []
for i in range(6):
  tensor_dataset.append([torch.tensor(dataset[i][0]), torch.tensor(dataset[i][1])])
</code></pre>
<pre><code class="lang-auto">tensor_dataset
</code></pre>
<p>[<br>
[tensor([0, 1, 0, 1, 0, 1]), tensor([[0], [1]])],<br>
[tensor([1, 0, 1, 0, 1, 0]), tensor([[1], [0]])],<br>
[tensor([0, 0, 0, 0, 0, 0]), tensor([[0], [0]])],<br>
[tensor([1, 1, 1, 1, 1, 1]), tensor([[1], [1]])],<br>
[tensor([1, 1, 0, 0, 1, 1]), tensor([[0], [0]])],<br>
[tensor([0, 0, 1, 1, 0, 0]), tensor([[1], [1]])]<br>
]</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
</code></pre>
<pre><code class="lang-auto">for i in range(1000):
  optimizer.zero_grad()
  print(tensor_dataset[i%6][0], tensor_dataset[i%6][1])
  loss = criterion(model(tensor_dataset[i%6][0], tensor_dataset[i%6][1]), tensor_dataset[i%6][1])
  print(loss)
  loss.backward()
  optimizer.step()
</code></pre>
<p>do I need to set self.embed_target requires_grad as False?<br>
is output.permute(0, 2, 1) correct way to use with nn.CrossEntropyLoss?</p>
Using non-system compiler and undefined references to pthread

username: mahmoodn
cooked: <p>On a Centos which has gcc 4.8.5 by default, I have built gcc-6.1.0 from source and want to tell pytorch to use that.</p>
<p>I have specified CC and CXX as below:</p>
<pre><code class="lang-auto">$ export CC=/storage/users/mnaderan/tools/gcc-6.1.0/bin/gcc
$ export CXX=/storage/users/mnaderan/tools/gcc-6.1.0/bin/g++
</code></pre>
<p>However, I get this error:</p>
<pre><code class="lang-auto">Performing C SOURCE FILE Test CMAKE_HAVE_LIBC_PTHREAD failed with the following output:
Change Dir: /storage/users/mnaderan/pt/pytorch/build/CMakeFiles/CMakeTmp

Run Build Command(s):/bin/gmake cmTC_bceed/fast &amp;&amp; gmake: Warning: File `Makefile' has modification time 19 s in the future
/bin/gmake -f CMakeFiles/cmTC_bceed.dir/build.make CMakeFiles/cmTC_bceed.dir/build
gmake[1]: Entering directory `/storage/users/mnaderan/pt/pytorch/build/CMakeFiles/CMakeTmp'
gmake[1]: Warning: File `CMakeFiles/cmTC_bceed.dir/flags.make' has modification time 19 s in the future
Building C object CMakeFiles/cmTC_bceed.dir/src.c.o
/storage/users/mnaderan/tools/gcc-6.1.0/bin/gcc   -DCMAKE_HAVE_LIBC_PTHREAD -fPIE   -o CMakeFiles/cmTC_bceed.dir/src.c.o   -c /storage/users/mnaderan/pt/pytorch/build/CMakeFiles/CMakeTmp/src.c
Linking C executable cmTC_bceed
/storage/users/mnaderan/tools/cmake-3.15.4/bin/cmake -E cmake_link_script CMakeFiles/cmTC_bceed.dir/link.txt --verbose=1
/storage/users/mnaderan/tools/gcc-6.1.0/bin/gcc  -DCMAKE_HAVE_LIBC_PTHREAD   -rdynamic  CMakeFiles/cmTC_bceed.dir/src.c.o  -o cmTC_bceed
CMakeFiles/cmTC_bceed.dir/src.c.o: In function `main':
src.c:(.text+0x2f): undefined reference to `pthread_create'
src.c:(.text+0x3b): undefined reference to `pthread_detach'
src.c:(.text+0x4c): undefined reference to `pthread_join'
src.c:(.text+0x60): undefined reference to `pthread_atfork'
collect2: error: ld returned 1 exit status
</code></pre>
<p>Any idea?</p>
Converting tensor to numpy array

username: Muppet
cooked: <p><code>x.cpu().detach().numpy()</code> seems quite convoluted. Is that the right way to transform a tensor to a numpy array? Is there a shortform?</p>
<p>Currently doing this (tensor is on GPU):</p>
<pre><code>    action = action.cpu().detach().numpy()
    log_prob = log_prob.cpu().detach().numpy()
    state_value = state_value.cpu().detach().numpy()
</code></pre>
<p>Can this maybe be written more efficiently as a with block? Can this be done in place? Is there a lot of overhead doing the above?</p>

username: beaupreda
cooked: <p>Hello,</p>
<p>Yes, this is the right way. You could always wrap the operations inside a function that looks like that:</p>
<pre><code class="lang-auto">def to_numpy(tensor):
    return tensor.cpu().detach().numpy()
</code></pre>
<p>I do not think a <code>with</code> block would work, and as far as I know, you can’t do those operations inplace (except <code>detach_</code>). The main overhead will be in the <code>.cpu()</code> call, since you have to transfer data from the GPU to the CPU.</p>