akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Getting 0 accuracy with resnet model

username: sven_cicca
cooked: <p>Hello,</p>
<p>I am new to pytorch and am following this tutorial: <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html#helper-functions" rel="nofollow noopener">Finetuning Torchvision</a></p>
<p>I am getting 0 accuracy while training and evaluating the model, I am not sure why…</p>
<p>Here is my training method:</p>
<pre><code class="lang-auto">def train_model(model, dataloaders, criterion, optimizer, num_epochs=25):
    since = time.time()

    val_acc_history = []
    
    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['imagenet_12_train', 'imagenet_12_val']:
            if phase == 'imagenet_12_train':
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'imagenet_12_train'):

                    outputs = model(inputs)
                    loss = criterion(outputs, labels)

                    _, preds = torch.max(outputs, 1)

                    # backward + optimize only if in training phase
                    if phase == 'imagenet_12_train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            epoch_loss = running_loss / len(dataloaders[phase].dataset)
            epoch_acc = running_corrects.double() / len(dataloaders[phase].dataset)

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'imagenet_12_val' and epoch_acc &gt; best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())
            if phase == 'imagenet_12_val':
                val_acc_history.append(epoch_acc)

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model, val_acc_history
</code></pre>
<p>Thanks!</p>

username: ptrblck
cooked: <p>The code looks alright.<br>
Do you see the loss decreasing?<br>
A 0% accuracy is weird, as you should at least get the random accuracy (i.e. <code>1/num_classes</code>) for a balanced dataset.</p>
Regression of unbalanced data

username: bigyeet
cooked: <p>Hi, I am doing a CNN LSTM regression of a wildly unbalanced dataset. All 6 labels are unbalanced in different ways. How would I use the weightedrandomsampler to deal with this? Examples I’ve seen were all for classification. Thanks a ton</p>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/dealing-with-imbalanced-datasets-in-pytorch/22596/36">This post</a> might be helpful. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Pytorch dataloader removes negative values

username: Gul_Zain
cooked: <p>Hello everyone!<br>
I am working with pytorch dataloaders and have my own custom dataset initialized. I am normalizing data inside the custom dataset class between -1 and 1. but when i check the output of dataloader, min and max value of the output is 1 and 1. When i print values from inside dataset class, the values are normalized from -1 to 1. I am not using ANY pytorch transformations. I am doing all these operations using opencv and then is returning the image as torch.FloatTensor(input tensor normalized in -1 and 1 range). Below is the initialization provided.</p>
<pre><code class="lang-auto">dataloaders = {
            'train': DataLoader(NYUImagesDataset(), batch_size=batch_size, shuffle=True, num_workers=0),
            'test': DataLoader(NYUImagesDataset(mode="test"), batch_size=batch_size, shuffle=True, num_workers=0)
        }
</code></pre>
<p>NYUImageDataset is my custom dataset class and the internal functioning of this class, i have described above. Looking forward to helpful responses.</p>

username: ptrblck
cooked: <p><code>DataLoader</code> should not change anything as shown in this example:</p>
<pre><code class="lang-python">data = torch.empty(1000, 1).uniform_(-1, 1)
dataset = TensorDataset(data)
loader = DataLoader(dataset, batch_size=5, shuffle=True)

for data in loader:
    print(data[0].min(), data[0].max())
</code></pre>
<p>Could you use a similar check and iterate your <code>Dataset</code> as well as the <code>DataLoader</code>?</p>
Extracting features from googlelenet pool 5 layer

username: KUSHAL_JAIN
cooked: <p>Many papers use Google Lenet pool5 layer’s as features for images/video-frames. I need a feature vector whose dimension is 1024. I figured out the corresponding layer in PyTorch <code>torchvision.models.googlelenet()</code> is the <code>AdaptiveAvgPool2d</code> layer. However, the ouputs from this layer for one image are of shape [1, 1024, 1, 1]. Is this output correct? Can I simply drop last 2 dimensions for the feature vector?</p>
<pre><code class="lang-auto">model = models.googlenet(pretrained=True)
lenet = nn.Sequential(*list(model.children())[:-2])
lenet(torch.randn(1,3,224,224)).shape
</code></pre>
<p>torch.Size([1, 1024, 1, 1])</p>

username: ptrblck
cooked: <p>Yes, you could <code>squeeze</code> these dimensions and process your activation as <code>[batch_size 1024]</code>.</p>
How to access conv2d inside desne blocks?

username: Mohamed_Moustafa
cooked: <p>For a pretrained model such as densenet161, how can I access one of the conv2d layers inside one of the dense blocks?</p>

username: ptrblck
cooked: <p>You could directly access this layer, e.g. via:</p>
<pre><code class="lang-python">model = models.densenet161()
model.features.denseblock4.denselayer21.conv2
</code></pre>
<p>Generally, you could print the model architecture and use the specified module names to get the wanted layer.</p>
CUDA out of memory Problem!

username: triplet02
cooked: <p>Hello!<br>
I am University Student who studying in Speech Recognition in Korea.</p>
<p>Recently I faced a annoying situation, and REALLY need some help.<br>
It’s an out of memory error, and error message is like below:<br>
<em>“CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 12.00 GiB total capacity; 9.42 GiB already allocated; 6.67 MiB free; 9.71 GiB reserved in total by PyTorch)”</em></p>
<p>My question is, although there is total 12.00 GIB of memory capacity, why my model struggles for memory shortage, which is using only 9.42 GIB, or 9.71 GIB at maximum .</p>
<p>Is there a extra reserved memory space that CUDA(or Pytorch) allocates internally to deal with specific situation?</p>
<p>If so, is there any way to stop it?<br>
It’s really frustrating for poor university student who can’t afford to get additional GPU, haha.</p>
<p>It’ll be lovely if someone’s gonna help me.<br>
Thank you very much!</p>

username: ptrblck
cooked: <p>The message points to a small chunk of free memory (~6MB), which is not sufficient for your use case.<br>
This could happen e.g. due to memory fragmentation.<br>
Try to lower your batch size and run the code again. Alternatively you could also have a look at <a href="https://pytorch.org/docs/stable/checkpoint.html">torch.utils.checkpoint</a> to trade compute for memory.</p>
Load pt files as datasets from separate folders with folder names as labels

username: Janine
cooked: <p>Hi all,</p>
<p>I am trying to load a bunch of .pt files from separate folders with the folder names as labels, .the pt files are tensors with size [height, weight, channel].</p>
<p>I assume something could work in a similar way as the ImageFolder or DataFolder. But when I tried the following script, it throws me some error:</p>
<pre><code class="lang-auto">data_dir = './data'  # the folder has multiple subfolders which contain the pt files

data_loader = torch.utils.data.TensorDataset(
         datasets.DataFolder(data_dir, transforms.Compose([
                transforms.Normalize(mean=mean,std=std),
                ])),
		batch_size = 1, orkers = 0,
        )
</code></pre>
<p>Then I got the following error:</p>
<pre><code class="lang-auto">AttributeError: module 'torchvision.datasets' has no attribute 'DataFolder'
</code></pre>
<p>I am just wondering how could this be correctly done in the DataFolder manner. The dataloader I am trying to create here will be used for evaluation only, not for training. I am new to PyTorch and any help is greatly appreciated. Thank you!</p>

username: Janine
cooked: <p>I have somehow managed to achieve my goal using the following code:</p>
<pre><code class="lang-auto">data_dir = './data'  # the folder has multiple subfolders which contain the pt files
imgs_pt = []
labels_pt = []

i = 0
for root, dirs, files in os.walk(data_dir):
    for file in files:
        path = os.path.join(root, file)
        img_pt = torch.load(path)
        img_pt = img_pt.permute(2,0,1).unsqueeze(0)
        img_pt = img_pt.add(-mean_.cpu()).mul(1/std_.cpu())
        img_pt = img_pt.squeeze()
        label_pt = torch.tensor(i-1)
        labels_pt.append(label_pt)
        imgs_pt.append(img_pt)
    i += 1

labels_pt = torch.tensor(labels_pt)
print(labels_pt.shape)
print(labels_pt)
imgs_pt = torch.stack(imgs_pt)
print(imgs_pt.shape)

data = torch.utils.data.TensorDataset(imgs_pt, labels_pt)
data_loader = torch.utils.data.DataLoader(data,
                                          batch_size=1,
                                          shuffle=False,
                                          num_workers=0)
</code></pre>
<p>Just wondering if pytorch has a plan to embed such functionality for pt files as for images and numpy files, like ImageFolder and DataFolder</p>
_pickle.UnpicklingError: invalid load key, '\x0a'

username: Kais_Kotamy
cooked: <p>hi there<br>
I am trying to load a .pb model through this functions below and I get this error<br>
“_pickle.UnpicklingError: invalid load key, ‘\x0a’.”<br>
any help please</p>
<pre><code class="lang-auto">def load_network_pkl(run_id_or_result_subdir_or_network_pkl, snapshot=None):
    return load_pkl(locate_network_pkl(run_id_or_result_subdir_or_network_pkl, snapshot))

def load_pkl(filename):
    with open(filename, 'rb') as file:
        return legacy.LegacyUnpickler(file, encoding='latin1').load()

&gt; #the load() function is from pickle package 

class LegacyUnpickler(pickle.Unpickler):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
</code></pre>
Pytorch model quantization for mobile device

username: Chame_call
cooked: <p>Hello there.<br>
I’m wondering can we quantize any model in pytorch or there are some constraints on it?<br>
If so then what are those constraints?<br>
thanks in advance…</p>

username: hx89
cooked: <p>The constraints I can think of are:</p>
<ul>
<li>Op support, if some ops in the model don’t have quantized version, they need to be implemented or otherwise skipped during quantization.</li>
<li>Accuracy constraint, quantization will introduce some errors to the output, if the quantized model has error larger than tolerance, you may need to skip the quantization of some operators in the model to get acceptable accuracy.</li>
</ul>
Pytorch: slice and stack a matrix along dimension 0

username: kaiseryet
cooked: <p>So I want to slice a matrix of size (n<strong>2, n</strong>2) to n<strong>2 of (n, n) matrices stacked along the dimension 0, resulting in a (n</strong>2, n, n) tensor. e.g.:</p>
<pre><code class="lang-auto">a = torch.arange(1,82).view(9,9) # this is the matrix to work on
b = a.view(3,3,3,3) # note that here n=3
print(b.permute(0,2,1,3))
</code></pre>
<p>The result is:</p>
<pre><code class="lang-auto">tensor([[[[ 1.,  2.,  3.],
          [10., 11., 12.],
          [19., 20., 21.]],

         [[ 4.,  5.,  6.],
          [13., 14., 15.],
          [22., 23., 24.]],

         [[ 7.,  8.,  9.],
          [16., 17., 18.],
          [25., 26., 27.]]],


        [[[28., 29., 30.],
          [37., 38., 39.],
          [46., 47., 48.]],

         [[31., 32., 33.],
          [40., 41., 42.],
          [49., 50., 51.]],

         [[34., 35., 36.],
          [43., 44., 45.],
          [52., 53., 54.]]],


        [[[55., 56., 57.],
          [64., 65., 66.],
          [73., 74., 75.]],

         [[58., 59., 60.],
          [67., 68., 69.],
          [76., 77., 78.]],

         [[61., 62., 63.],
          [70., 71., 72.],
          [79., 80., 81.]]]])
</code></pre>
<p>Almost there, except it’s a (3, 3, 3, 3) tensor, instead I want:</p>
<pre><code class="lang-auto">tensor([[[ 1.,  2.,  3.],
          [10., 11., 12.],
          [19., 20., 21.]],

         [[ 4.,  5.,  6.],
          [13., 14., 15.],
          [22., 23., 24.]],

         [[ 7.,  8.,  9.],
          [16., 17., 18.],
          [25., 26., 27.]],

        [[28., 29., 30.],
          [37., 38., 39.],
          [46., 47., 48.]],

         [[31., 32., 33.],
          [40., 41., 42.],
          [49., 50., 51.]],

         [[34., 35., 36.],
          [43., 44., 45.],
          [52., 53., 54.]],

        [[55., 56., 57.],
          [64., 65., 66.],
          [73., 74., 75.]],

         [[58., 59., 60.],
          [67., 68., 69.],
          [76., 77., 78.]],

         [[61., 62., 63.],
          [70., 71., 72.],
          [79., 80., 81.]]])
</code></pre>
<p>I can’t figure out how to do this… (<code>view(9, 3, 3)</code> wouldn’t work, it messed up the ordering of elements in my (3, 3) submatrices) On the other hand, if there is any other way do operate on those (3, 3) slices of matrix <code>a</code> (inverse, matrix multiplication, etc.), I sure would like to hear about it…</p>

username: kaiseryet
cooked: <p>Problem solved by using <code>.reshape(9, 3, 3)</code>. I guess create a new copy is necessary because what I want to do broke the original ordering…</p>
UNet output always scaled down in intensity

username: Robin_Lobel
cooked: <p>For some reason, even with simple cases, the masks I get back using the original UNet model (with padding) do not have the same intensity as the masks it was trained with. The shape of the masks infered by the UNet are very good, but it seems the intensity is always scaled at 80%-90% of the original masks intensity.</p>
<p>In my case, the intensity is pretty important as I want to extract signals mixed with noise. If the intensity is not scaled correctly, I can’t fully extract the signal or fully remove the noise.</p>
<p>I tried 2 approaches : training mix vs signal, and mix vs mask of the signal. Both approaches result in the same conclusion : great shape detection, and great intensity detection, but the intensity is always scaled by some ratio for a reason I don’t know.</p>
<p>The UNet model I’m using is as follow:</p>
<p>Contracting/Expanding Blocks: Conv2d, ReLU, DropOut, Conv2d, DropOut<br>
Downsampling: MaxPool2d<br>
Upsampling: ConvTranspose2d<br>
For reference, here’s my UNet code: <a href="https://github.com/divideconcept/PyTorch-libtorch-U-Net/blob/master/unet.h" rel="nofollow noopener">https://github.com/divideconcept/PyTorch-libtorch-U-Net/blob/master/unet.h</a></p>
<p>I apply padding (with zeroes) at each 3x3 Conv2d to avoid cropping the image. Could it because of that ?</p>
<p>My mix image values are always between [0 1], and so is the target signal image.</p>

username: jankunze
cooked: <p>Hi,</p>
<p>no elaborate answer but just few thoughts and questions:</p>
<p>To understand your dataset better, how many classes do you have in your mask images?</p>
<p>Did you already check that your masks are loaded correctly by displaying them? Make sure the mask images from your data loader resemble the mask images from your database on your PC. Could normalization be an issue?</p>
<p>What loss function do you use?</p>
<p>Can you post a code snippet for getting the prediction mask  image?</p>

username: Robin_Lobel
cooked: <p>Here are my inputs (left), the inference from the model (middle) and the response I was waiting (right).<br>
As you can see the shape is quite good, but the intensity is slightly off by 10-20% compared to the target image.<br>
Yes I checked the source, inference and targets multiple times.<br>
FYI I don’t do any normalization steps, but the values are always between 0 and 1.<br>
Here’s my UNet code: (pytorch c++): <a href="https://github.com/divideconcept/PyTorch-libtorch-U-Net/blob/master/unet.h" rel="nofollow noopener">https://github.com/divideconcept/PyTorch-libtorch-U-Net/blob/master/unet.h</a></p>
<p>My training loop is as follow (pytorch c++) :</p>
<p><code>optim.zero_grad();</code><br>
<code>result = model-&gt;forward(source[b]);</code><br>
<code>loss = torch::mse_loss(result, target[b]);</code><br>
<code>loss.backward();</code><br>
<code>optim.step();</code></p>

username: Robin_Lobel
cooked: <p>Breaking News : removing the DropOut nodes solved the intensity problem, and changing the loss function from MSE(L2) to MAE(L1) helped a lot with the accuracy of the lower values.</p>
Replacing indexing assigment with scatter for 5D tensor

username: copah
cooked: <p>I want to convert a model that uses in-place index assignment to an onnx model. According to the docs, the conversion doesn’t support in-place index assignment (<a href="https://pytorch.org/docs/stable/onnx.html#limitations" rel="nofollow noopener">https://pytorch.org/docs/stable/onnx.html#limitations</a>), so I’m trying to rewrite the operation to a scatter operation, but I’m having difficulties doing this.</p>
<p>The current implementation looks something like this:</p>
<pre><code class="lang-auto">    for i in range(1, max_dimension):
            combined_tensor[:, num_channels:, i, :, i:] = right_tensor[:, :, :, :-i]
</code></pre>
<p><code>combined_tensor</code> is of shape : <code>(batch_size, num_channels*2, max_dimension, width, height)</code> and <code>right_tensor</code> is of shape <code>(batch_size, num_channels, width, height)</code>.</p>
<p>I want to build the indices to supply to the scatter operation, so I’m utilizing meshgrid in this way:</p>
<pre><code class="lang-auto">    idx = torch.meshrid(
            range(batch_size),
            range(num_channels, num_channels*2),
            range(max_dimension),
            range(width),
            range(height)[::-1]
        )
</code></pre>
<p>But I am unsure if this is the correct approach. Any guidance would be appreciated.</p>