Where is the implementation of torchvision roi_align?

Question:
I can’t find anywhere the implementation of this code:

      github.com

    pytorch/vision/blob/68f511eb3510ab6fffd258c8aaeffdc1a574212c/torchvision/ops/roi_align.py#L61 1

                  Tensor[K, C, output_size[0], output_size[1]]: The pooled RoIs.
              """
              if not torch.jit.is_scripting() and not torch.jit.is_tracing():
                  _log_api_usage_once(roi_align)
              _assert_has_ops()
              check_roi_boxes_shape(boxes)
              rois = boxes
              output_size = _pair(output_size)
              if not isinstance(rois, torch.Tensor):
                  rois = convert_boxes_to_roi_format(rois)
              return torch.ops.torchvision.roi_align(
                  input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned
              )

class RoIAlign(nn.Module):
              """
              See :func:`roi_align`.
              """

    def __init__(

Context:
may be this is what you are looking

      github.com

    pytorch/vision/blob/eac3dc7bab436725b0ba65e556d3a6ffd43c24e1/torchvision/ops/roi_align.py#L13

          import torch
          from torch import nn, Tensor
          from torch.jit.annotations import BroadcastingList2
          from torch.nn.modules.utils import _pair
          from torchvision.extension import _assert_has_ops

from ..utils import _log_api_usage_once
          from ._utils import convert_boxes_to_roi_format, check_roi_boxes_shape

def roi_align(
              input: Tensor,
              boxes: Union[Tensor, List[Tensor]],
              output_size: BroadcastingList2[int],
              spatial_scale: float = 1.0,
              sampling_ratio: int = -1,
              aligned: bool = False,
          ) -> Tensor:
              """
              Performs Region of Interest (RoI) Align operator with average pooling, as described in Mask R-CNN.


Multi-label models trained on single-label data

Question:
I wonder if it is a good idea to train a multi-label model on a single-labelled data set (using e.g. BCELoss). Is this likely to generalize well to images with multiple of the categories present? Even more important, possibly: would such a model be likely to respond correctly if no category is present?
Has anyone tried this?

Context:
Just found this article myself:

      openaccess.thecvf.com

Cole_Multi-Label_Learning_From_Single_Positive_Labels_CVPR_2021_paper.pdf 2

  506.20 KB

It claims that learning from single labels can work even when the single labeled training data actually had multiple classes represented in each image. (My plan was to use training images with just a single class present in each image.)
F


Question:
I’m trying to use the torchvision pascal voc package
from torchvision.datasets import VOCDetection

My problem is that I’m having issues when resizing the images with transforms.
data_transforms = transforms.Compose([
        transforms.Resize(size=(256,300)),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])

dataset = VOCDetection(root="./", download=True, transform=data_transforms)
train_size = int(len(dataset) * 0.9)
val_size = int(len(dataset) - train_size)
train, val = random_split(dataset, [train_size, val_size])

train_loader = DataLoader(train, batch_size=32, num_workers=4)
val_loader = DataLoader(val, batch_size=32, num_workers=4)

I’m passing the size as a tuple, still I’m getting this when I run:
RuntimeError: each element in list of batch should be of equal size

If I pass something like   transforms.Resize(size=(256)) I get a reasonble error
RuntimeError: stack expects each tensor to be equal size, but got [3, 256, 341] at entry 0 and [3, 341, 256] at entry 1

which makes sense based on transforms.resize documentation. I’m confused because even passing it as a tuple, still I get the error.
I never worked with  this detection datasets , so if there’s anything I should know I appreciate the help.

Context: is it right to use batch sizes larger than 1? or is just standard like segmentation to use 1?


Ssdlite320_mobilenet_v3_large failed to run with torchscript

Question:
Hi there,
I am trying to use torchscript to trace the ssdlite320_mobilenet_v3_large provided in torchvision. However, there was unexpected errors when I follow the docs.
import torch
import torch as th
import torch.nn as nn
from torchvision import models
import torch.onnx

model = models.detection.ssdlite320_mobilenet_v3_large(pretrained=True)
model.eval()

x = [torch.rand(3, 320, 320), torch.rand(3, 500, 400)]

predictions = model(x) # successfully run without error
model_trace = torch.jit.trace(model, x) # fail to run

The error log
/home/ligeng/.local/lib/python3.8/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2157.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Traceback (most recent call last):
  File "ssd_torchscript.py", line 15, in <module>
    model_trace = torch.jit.trace(model, x)
  File "/home/ligeng/.local/lib/python3.8/site-packages/torch/jit/_trace.py", line 741, in trace
    return trace_module(
  File "/home/ligeng/.local/lib/python3.8/site-packages/torch/jit/_trace.py", line 958, in trace_module
    module._c._create_method_from_trace(
  File "/home/ligeng/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ligeng/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1090, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/ligeng/.local/lib/python3.8/site-packages/torchvision/models/detection/ssd.py", line 310, in forward
    images, targets = self.transform(images, targets)
  File "/home/ligeng/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ligeng/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1090, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/ligeng/.local/lib/python3.8/site-packages/torchvision/models/detection/transform.py", line 100, in forward
    for k, v in t.items():
AttributeError: 'Tensor' object has no attribute 'items'

Context:
torch:1.10.0+cu102
torchvision:0.11.1+cu102


Speed up Dataloader using the new Torchvision Transforms support for Tensor, batch computation, GPU

Question:
Hello there,
According to the following torchvision release 58 transformations can be applied on tensors and batch tensors directly. It says:

torchvision transforms are now inherited from nn.Module and can be torchscripted and applied on torch Tensor inputs as well as on PIL images. They also support Tensors with batch dimension and work seamlessly on CPU/GPU devices

Here a snippet:
import torch
import torchvision.transforms as T

transforms = torch.nn.Sequential(
    T.RandomCrop(224),
    T.RandomHorizontalFlip(p=0.3),
    T.ConvertImageDtype(torch.float),
    T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
)

tensor_image = torch.randint(0, 256, size=(3, 256, 256), dtype=torch.uint8)

# works directly on Tensors
out_image1 = transforms(tensor_image)

# on the GPU
out_image1_cuda = transforms(tensor_image.cuda())

# with batches
batched_image = torch.randint(0, 256, size=(4, 3, 256, 256), dtype=torch.uint8)
out_image_batched = transforms(batched_image)

How can we utilize this fact to improve our dataloaders performance.
Loading and Transforming ImageNet images is taking on my computer approx 5s to load a 256-images batch. Given the fact that I will move the input tensors to device anyway why not apply transformations after doing this step?
The typical work flow is to define a transformation composition and set it up for a torch dataset. The composition transforms PIL images to tensors in the final step as following:
import torchvision.transforms as T
transform = T.compose([
                     T.Resize(...),
                     T.Crop(....),
                     T.Flip(...),
                      # And finally
                     T.ToTensor(),
])

This doesn’t make use of the fact that transformations can be applied on device tensors directly.
My thought is to add transformation statement inside the data loading loop as following:
for i,(input,target) in enumerate(dataloader):
      # move data to GPU
      input = input.to(device)
      target = target.to(device)
      # APPLY transformations
      input = transform(input)
      # feed to model
      output = model(input)

However this approach is not clean and seamless as simply setting a transformation to a dataset. Though I am afraid it is the only possible approach.
I am not sure if this approach will even speed up loading. Will it?
Any thoughts?

Context: Hi! Have you figured this out? I’m also interested on improving computation times for transformations.


CNN for image classification - mat1 and mat2 cannot be multiplied

Question:
Hi, I’m trying to run a cnn for image classification.  Very new to this, so using a mashup of tutorial code and my own inputs.  I get this error when I try to run:
mat1 and mat2 shapes cannot be multiplied (12x403328 and 1266750x120)
All my image inputs are 750x563 jpgs.
Torch size is:
Shape of X [N, C, H, W]:  torch.Size([12, 3, 563, 750])
Shape of y:  torch.Size([12]) torch.int64

Any help would be appreciated - I’ve seen similar posts, but can’t figure out how to apply those answers to my code
if __name__ == '__main__':
    import torch
    from torchvision import datasets, transforms

    import numpy as np # for transformation

    import torch # PyTorch package
    import torchvision # load datasets
    import torchvision.transforms as transforms # transform data
    import torch.nn as nn # basic building block for neural neteorks
    import torch.nn.functional as F # import convolution functions like Relu
    import torch.optim as optim # optimzer

    train_transform = transforms.Compose([transforms.ToTensor(),transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])])
    train_dataset = datasets.ImageFolder('H:/01 - Tech Refs/data2/train', transform=train_transform)
    train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=12, shuffle=True, num_workers=4)

    test_transform = transforms.Compose([transforms.ToTensor(),transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])])
    test_dataset = datasets.ImageFolder('H:/01 - Tech Refs/data2/test', transform=test_transform)
    test_dataloader = torch.utils.data.DataLoader(test_dataset, batch_size=12, shuffle=False,num_workers=4)

    for X, y in test_dataloader:
        print("Shape of X [N, C, H, W]: ", X.shape)
        print("Shape of y: ", y.shape, y.dtype)
        break

            #self.fc1 = nn.Linear(3*563*750, 120)

    device = "cuda" if torch.cuda.is_available() else "cpu"
    print(f"Using {device} device")

    class NeuralNetwork(nn.Module):
        def __init__(self):
            super().__init__()
            self.conv1 = nn.Conv2d(3, 6, 5)
            self.pool = nn.MaxPool2d(2, 2)
            self.conv2 = nn.Conv2d(6, 16, 5)
            self.fc1 = nn.Linear(3*563*750, 120)
            self.fc2 = nn.Linear(120, 84)
            self.fc3 = nn.Linear(84, 17)

        def forward(self, x):
            x = self.pool(F.relu(self.conv1(x)))
            x = self.pool(F.relu(self.conv2(x)))
            x = torch.flatten(x, 1) # flatten all dimensions except batch
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))
            x = self.fc3(x)
            return x

    model = NeuralNetwork().to(device)

    loss_fn = nn.CrossEntropyLoss()
    optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

    def train(dataloader, model, loss_fn, optimizer):
        size = len(dataloader.dataset)
        model.train()
        for batch, (X, y) in enumerate(dataloader):
            X, y = X.to(device), y.to(device)

            # Compute prediction error
            pred = model(X)
            loss = loss_fn(pred, y)

            # Backpropagation
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            if batch % 100 == 0:
                loss, current = loss.item(), batch * len(X)
                print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")

    def test(dataloader, model, loss_fn):
        size = len(dataloader.dataset)
        num_batches = len(dataloader)
        model.eval()
        test_loss, correct = 0, 0
        with torch.no_grad():
            for X, y in dataloader:
                X, y = X.to(device), y.to(device)
                pred = model(X)
                test_loss += loss_fn(pred, y).item()
                correct += (pred.argmax(1) == y).type(torch.float).sum().item()
        test_loss /= num_batches
        correct /= size
        print(f"Test Error: \n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \n")

    epochs = 5

    for t in range(epochs):
        print(f"Epoch {t+1}\n-------------------------------")
        train(train_dataloader, model, loss_fn, optimizer)
        test(test_dataloader, model, loss_fn)
    print("Done!")

Context:
Hi Luke!

 luke0:

mat1 and mat2 shapes cannot be multiplied (12x403328 and 1266750x120)
All my image inputs are 750x563 jpgs.

By the time you get to fc1 – where the error is occurring – your images
are no longer of the same size as when passed into forward().

    class NeuralNetwork(nn.Module):
        def __init__(self):
            super().__init__()
            self.conv1 = nn.Conv2d(3, 6, 5)

Conv2d changes (reduces) the size of your image.

            self.pool = nn.MaxPool2d(2, 2)

MaxPool2d also reduces the size of your image.

        def forward(self, x):
            x = self.pool(F.relu(self.conv1(x)))
            x = self.pool(F.relu(self.conv2(x)))
            x = torch.flatten(x, 1) # flatten all dimensions except batch
            x = F.relu(self.fc1(x))
            ....

In forward(), print out the shape of x (initially your input image), first
at the beginning,  after the application of conv1(), after the first application
of pool(), after conv2(), after the second pool(), and then again after
flatten().  You will see the height and width of x shrinking.  The shape
of x after flatten will tell you the number in_features you need for fc1
and it will (presumably) be 403328 rather than 1266750.
Best.
K. Frank


Question:
Hello all,
Could you please help me with a doubt? I have a 2D NumPy array that represents an image in black and white (shape: 1x224x224). After converting the NumPy array to a PyTorch tensor, I want to apply transformations to the image (using ToPILImage). Nevertheless, I got a noisy image as a result.
The NumPy array in matplotlib looks like this:

But the ToPILImage() figures looks noisy:

My question is:

How can I convert the PyTorch tensor (in black and white) to the PIL object, so it looks more like the first matplotlib image? I am using transforms.ToPILImage() at the moment.

Thank you so much,

Context:
Could you check the dtype and shape of the numpy array before transforming it to a PIL.Image and compare it to the result of the transformation? Based on the output the transformation might cause overflows e.g. if the dtype is also changed.
If you get stuck, please post an executable code snippet to reproduce this issue so that we could help in debugging.


Question:
Is there an easy way to convert from RGB to Lab color space and vice versa in Pytorch? I want to do it in a differentiable way within the model.
Thanks

Context:
I’m not aware of an easy shortcut, but the conversion is a composition of differentiable operations, so you can just implement the forward computation and backprop through it.
You can copy my implementation in go-colorful 207 which includes this conversion here 280, it does the conversion by passing through XYZ space, follow the code for equations.
The only non-trivial part in the chain is a couple data-dependent switches, you could either use this trick 280 or masked_scatter_ 31 for that, or you might get away with just ignoring them because they apply in the very low-luminosity range only.


SOTA techniques in image segmentation & detection, *medical images

Question:
Looking for inputs on state of the art work in the detection and segmentation work for medical images.
Thanks

Context: I would check if e.g. MONAI 2 provides some desired models, which is part of the PyTorch ecosystem and deals with medical datasets.


Oddly getting 32 dimensions when I resize a 3D tensor?

Question:
I’m trying to resize a tensor through the Resize function of torchvision.transforms module; but - as the tile is self-descriptive - I’m getting a 32D output when I apply the resize function of a 3D tensor. I’ve added the code snippet below. The shape of the input and the output are (32, 32, 3) and (32, 75, 75), respectively. Could you please guide me through the thing I miss?
from torchvision import datasets, models, transforms as T
tensor_x = torch.Tensor(X)  # converting the NumPy array to Tensor
resize_fn = T.Resize((75, 75))
out = resize_fn(tensor_x)

Also, I’d like to ask why the transforms that I’ve provided did not apply to the dataset below. My aim is to resize the samples of CIFAR-10 from (32, 32, 3) to (75, 75, 3). But I still get the original shape for the samples, (32, 32, 3).
from torchvision import datasets, models, transforms as T

m_transform = T.Compose([
    T.Resize((75, 75)),
    T.ToTensor()
])

train_set = datasets.cifar.CIFAR10('/PyTorch-Datasets', train=True, download=False), transform=m_transform)
test_set = datasets.cifar.CIFAR10('/PyTorch-Datasets', train=False, download=False), transform=m_transform)

Context:
Resize expects a channels-first tensor, so it should have the shape [3, 32, 32] in your case as seen here:
tensor_x = torch.randn(3, 32, 32)
resize_fn = T.Resize((75, 75))
out = resize_fn(tensor_x)


Custom resnet50 weights on pytorch faster rcnn backbone

Question:
Hi,
I want to detect heart using stanford chestxray dataset with the help of torchvision.models.detection.fasterrcnn_resnet50_fpn. Now when i set torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True) it will have the pretrained weights which i dont want. For my problem, i have already trained a resnet 50 model using stanford chestxray dataset and i want those weights of the checkpoints as the weights of the backbone for the faster rcnn object detector. Please help me how to use them in the torchvision model. Thanks

Context: You could either try to load your state_dict into the backbone via accessing model.backbone and changing some keys in the state_dict if needed or alternatively create a new model as seen here 2 with your custom backbone.


How to implement oversampling in Cifar-10?

Question:
Hi,
I need to train a convolution network using some oversampling technique in the Cifar-10 database. But I do not know how to do it in Pytorch. First I need to simulate the problem of class imbalance at the dataset, because CIFAR-10 is a balanced dataset. And then apply some oversampling technique. Could someone give me an example?

Context:
Would it work for you if you just oversample a specific class?

train_dataset = torchvision.datasets.CIFAR10(root='YOUR_PATH,
                                             transform=torchvision.transforms.ToTensor())
target = train_dataset.train_labels
class_sample_count = np.unique(target, return_counts=True)[1]
print(class_sample_count)

# oversample class 0
class_sample_count[0] = 50

weight = 1. / class_sample_count
samples_weight = weight[target]
samples_weight = torch.from_numpy(samples_weight)
sampler = WeightedRandomSampler(samples_weight, len(samples_weight))

train_loader = DataLoader(
    train_dataset, batch_size=10, num_workers=1, sampler=sampler)

for i, (data, target) in enumerate(train_loader):
    print "batch index {}, class 0 {}, other classes {}".format(
        i,
        len(np.where(target.numpy() == 0)[0]),
        len(np.where(target.numpy != 0)[0]))


Question:
I am reading video and split them into batched. I cant fit a whole video in my batch. So i split them using collate function. Shape of video is (9*32,3,224,224).
Output shape from data loader after using collate function is
torch.Size([9, 32, 3, 224, 224]) where index 0 is batch. I if pass this bath data , if will get memory error. Is there a way to send a batch of 3 .?
Here is code
dl=DataLoader(DataReader(df,train_aug), batch_size =1,collate_fn=collate_fn)

here is custom dataset
class DataReader(Dataset):
    def __init__(self, dataset, transform=None):
        self.df = dataset
        self.transform = transform

    def __getitem__(self, index):
        x=self.df['path'][index]
        y=self.df['pulse'][index]
        x=readframes(x)
        x1=[]
        for i in x:
            img=cv2.cvtColor(src=i, code=cv2.COLOR_BGR2RGB)
            img=train_aug(image=img)['image']
            x1.append(img)
        return torch.stack(x1),np.float32(y)

    def __len__(self):
        return len(self.df)

readframes is a function to extracrt frames of a video.
def reshaped(i,bs=32):
    i=i[len(i)%bs::].reshape(len(i)//bs,-1,3,224,224)
    return i
def collate_fn(batch):#batch is a tuple. where first value is batch.,2nd is ether array or label
    bs=32
    data=np.concatenate([reshaped(i[0]) for i in batch])
    label=np.concatenate([[i[1]]*(len(i[0])//bs) for i in batch])
    data,label=data[0:4],label[0:4]
    return torch.tensor(data,dtype=torch.float),torch.tensor(label)

Context: The collate_fn will get all loaded samples and will create the batch. Removing already loaded and processed samples that late sounds a bit wasteful. Wouldn’t it work to not load the undesired frames in the first place?


Get batch from collate_fn of Pytorch data loader

Question:
I am reading video and split them into batched. I cant fit a whole video in my batch. So i split them using collate function. Shape of video is (128,3,224,224).
Where 128 is frames, and 3 is channels (RGB).
Output shape from data loader after using collate function is
torch.Size([4, 32, 3, 224, 224]) where index 0 is batch. I if pass this bath data , if will get memory error. Is there a way to send a batch of 2 .?
Here is the code for reproduce-ability purpose.
Generate dummy data
features=[]
labels=[]
#here the data is in ram, but actually its in disk and i cant load all data to ram to manipulate.
for i in range(5):
    features.append(np.random.randint(low=0, high=256, size=(128,3,224,224)))
    labels.append(np.random.randint(low=0, high=2))

dataset
class DataReader(Dataset):
    def __init__(self,features,labels):
        self.features = features
        self.labels = labels

    def __getitem__(self, index):
        x=self.features[index]
        y=self.labels[index]
        return x,y

    def __len__(self):
        return len(self.features)

dataloader
def reshaped(i,bs=32):
    i=i[len(i)%bs::].reshape(len(i)//bs,-1,3,224,224)
    return i
def collate_fn(batch):
    data=reshaped(batch[0])
    print(data.shape)
    label=[batch[1]]*len(data)
    return data,label
dl=DataLoader(DataReader(features,labels), batch_size =None, num_workers=0,pin_memory=True,shuffle=False,collate_fn=collate_fn)
batch=next(iter(dl))

batch[0] shape is (4, 32, 3, 224, 224) even with a single batch the generated data is large, so i want to send (2, 32, 3, 224, 224) then another (2, 32, 3, 224, 224)
Dont get confused with batch=None, i am just disabling automatic batching. Practically batch=None load one batch. I convert that one batch to four batches in collate function.
But the issue is that now all four will be sent further and i want only 2 of them to be sent, then next 2 to the model, instead of all four at once.

Context: Double post from here 1.


Question:
I am trying to implement a paper that uses the activations of an Inception v3 model with the final softmax removed (so just the logits). If I understand correctly, that is exactly what the inception_v3 model from torchvision returns? So should I be calling inception_v3 with pretrained=True, input_transforms=True and aux_logits=False? I don’t know what the last parameter does (I think it means auxiliary logits?) but I doubt I need that for my task since my goal is to get that 1000 length tensor with logit outputs for each of the 1000 ImageNet classes.
Also, with input_transforms=True, what kind of input will that model expect? I know that there is some sort of inherent normalization that goes on inside the model itself. Given this, what range of values should my input be in so that the model functions properly?
I apologize for the wall of questions. This is my first time working with pre-trained models.

Context:
Entropy:

(so just the logits). If I understand correctly, that is exactly what the inception_v3 model from torchvision returns?

Yes, the torchvision implementation returns the logits from the last layer as well as the aux. logits, if specified as seen here.

 Entropy:

I don’t know what the last parameter does (I think it means auxiliary logits?) but I doubt I need that for my task since my goal is to get that 1000 length tensor with logit outputs for each of the 1000 ImageNet classes.

Yes, that’s right and based on your description you could set it to False.

 Entropy:

Also, with input_transforms=True, what kind of input will that model expect?

transform_input=True will apply this transformation, which was introduced to match the Inception/Google implementation.


Question:
Hey, im trying to implement a vae and the model is
class VAE(nn.Module):
    def __init__(self, z_dims=60):
        super(VAE, self).__init__()
        self.encoder = q_zx(z_dims=z_dims)
        self.decoder = p_xz(z_dims=z_dims, batch_size=batch_size)

    def encode(self, x):
        mu_z, logvar_z = self.encoder(x)
        # mu_z = (1, z_dims)
        # logvar_z = (1, z_dims)
        return mu_z, logvar_z

    def decode(self, x):
        mu_x, logvar_x = self.decoder(x)
        # mu_x = (batch, 1, 64, 64)
        # logvar_x = (batch, 1, 64, 64)
        return mu_x, logvar_x

    def sample(self, mu_z, logvar_z):
        std = torch.exp(0.5*logvar_z)
        eps = torch.randn_like(std)
        return mu_z + eps*std

    def loss_function(self, mu_x, logvar_x, x, mu_z, logvar_z):
        mu_x = torch.flatten(mu_x, start_dim = 1)
        logvar_x = torch.flatten(logvar_x, start_dim=1)
        loss_rec = torch.mean(-torch.sum(
        (-0.5 * np.log(2.0 * np.pi))
        + (-0.5 * logvar_x)
        + ((-0.5 / torch.exp(logvar_x)) * (x.view(-1, 64*64) - mu_x) ** 2.0),
        dim=1,
        ))
        KLD = torch.mean(-0.5 * torch.sum(1 + logvar_z - mu_z.pow(2) - logvar_z.exp()))
        return loss_rec + KLD

    def forward(self, x):
        mu_z, logvar_z = self.encode(x)
        z_sampled = self.sample(mu_z, logvar_z)
        mu_x, logvar_x = self.decode(z_sampled)
        return mu_z, logvar_z, mu_x, logvar_x

The dimensions of mu_z, logvar_z, mu_x, logvar_x is written in the encode and decode func
The train function i use is
def train(epoch):
    model.train()
    train_loss = 0
    for batch_idx, (data, _) in enumerate(train_loader):
        data = data.to(device)
        optimizer.zero_grad()
        mu_z, logvar_z, mu_x, logvar_x = model(data)
        loss = model.loss_function(mu_x, logvar_x, data, mu_z, logvar_z)
        loss.backward()
        train_loss += loss.mean().item()
        optimizer.step()
        if batch_idx % 1200 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                       100. * batch_idx / len(train_loader),
                       loss.item() / len(data)))```

However my loss in nan
why is that happening??

Thanks

Context:
In your loss_function(…), like …
var_x = torch.clip(torch.exp(logvar_x), min=1e-5)
loss_rec = torch.mean(-torch.sum(
        (-0.5 * np.log(2.0 * np.pi))
        + (-0.5 * logvar_x)
        + ((-0.5 / var_x) * (x.view(-1, 64*64) - mu_x) ** 2.0),
        dim=1,
        ))

By the way, if a vae model learn to std in decoder, it tend to go 0 and become deterministic (so, usually fixed to some small value, like 0.1). See Taming VAE paper by Rezende and Viola for more discussion about this ([1810.00597] Taming VAEs 1).


How can I solve it ,TypeError: cannot pickle ‘dict_keys’ object?

Question:
class spatial_dataset(Dataset):
    def __init__(self, dic, root_dir, mode, transform=None):

    self.keys = dic.keys()
    self.values=dic.values()
    self.root_dir = root_dir
    self.mode =mode
    self.transform = transform

def __len__(self):
    return len(self.keys)

def load_ucf_image(self,video_name, index):
        path = self.root_dir + '/' + video_name.split('_')[0] + '/' + 'v_' + video_name + '/'

    print(path+'Image'+str(index)+'.jpg')
    img = Image.open(path+'Image'+str(index)+'.jpg')
    transformed_img = self.transform(img)
    img.close()

    return transformed_img

def __getitem__(self, idx):
    print(idx)
    if self.mode == 'train':
        video_name, nb_clips = list(self.keys)[idx].split(' ')
        print(video_name, nb_clips)
        num_clip = int(nb_clips)
        print(num_clip)
        clips = []
        clips.append(random.randint(1, int(num_clip/3)))
        clips.append(random.randint(int(num_clip/3), int(num_clip*2/3)))
        clips.append(random.randint(int(num_clip*2/10), num_clip+1))
        # for i in range(10):
        #     clips.append(random.randrange(1, num_clip, 1))
        print(clips)

    label = list(self.values)[idx]
    label = int(label)-1

    if self.mode=='train':
        data =[]
        for i in range(len(clips)):
            key = 'img'+str(i)
            index = clips[i]
            data.append(self.load_ucf_image(video_name, index))
        data = np.array(data)
        label = np.array(label)
        sample = (data, label)

    return sample

dic.training is dictionary, which have;
{‘videoname_1 framecount_1’: label1,‘videoname_2 framecount_2’ : label2, …}
training_set = spatial_dataset(dic=dic_training, root_dir=data_path, mode='train', transform = transforms.Compose([
                transforms.RandomCrop(224),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])
                ]))
        print('==> Training data :',len(training_set),'frames')
        print("training_set: ",training_set[1][0])

Till Above, it is running but when I use this below code it shows me error (mentioned below)
train_loader = DataLoader(
            dataset=training_set,
            batch_size=BATCH_SIZE,
            shuffle=True,
            num_workers=num_workers)

    for z,i in train_loader:
        print(i)

This is error it is showing me;
 2
It is something because I am using dictionary to store values, I then converted it into the numpy arrays and list but still it is showing me this error I don’t know where it is going wrong.

Context: Maybe your issue is related to this one 134, so you might want to take a look at it.


Question:
This is the code in Tensorflow
class DownShiftedConv2d(tf.keras.layers.Layer):
    def __init__(self, num_filters, filter_size=(2, 3), dilation_rate=(1, 1), kernel_regularizer='L2', padding='VALID',
                 **kwargs):
        super().__init__(**kwargs)
        if isinstance(filter_size, int):
            self.filter_size = (filter_size, filter_size)
        else:
            self.filter_size = filter_size
        self.wnconv = tf.keras.layers.Conv2D(num_filters, filter_size, strides=dilation_rate,
                                             kernel_regularizer=kernel_regularizer, padding=padding)

    def call(self, inputs, *args, **kwargs):
        output = tf.pad(inputs, [[0, 0], [self.filter_size[0] - 1, 0],
                                 [int((self.filter_size[1] - 1) / 2), int((self.filter_size[1] - 1) / 2)],
                                 [0, 0]])
        output = self.wnconv(output)
        return output

I want to translate this code from Tensorflow to Pytorch but don’t know the correct way to add L2 regularizer.
This is my code :
class DownShiftedConv2D(nn.Module):
    def __init__(self, in_channels, num_filters, filter_size=(2, 3), dilation_rate=(1, 1), padding='valid',
                 **kwargs):
        super(DownShiftedConv2D, self).__init__(**kwargs)
        self.in_channels = in_channels
        self.num_filters = num_filters
        self.filter_size = filter_size
        self.dilation_rate = dilation_rate

        if isinstance(filter_size, int):
            self.filter_size = (filter_size, filter_size)
        else:
            self.filter_size = filter_size
        self.wnconv = nn.Conv2d(in_channels=in_channels, out_channels=num_filters, kernel_size=filter_size,
                                stride=dilation_rate, padding=padding)

    def forward(self, inputs, *args, **kwargs):
        output = F.pad(inputs, (0,0,
                                self.filter_size[0] - 1, 0,
                                int((self.filter_size[1] - 1) / 2), int((self.filter_size[1] - 1) / 2),
                                0, 0))

Can anyone help me ? Thank you

Context:
You could add the weight’s L2 norm to the loss before optimizing
l2_lambda = 0.01
l2_reg = torch.tensor(0.)
for name, param in model.named_parameters():
    if 'conv' in name:
        l2_reg += torch.linalg.norm(param)
loss += l2_lambda * l2_reg

AFAIK the weight_decay parameter in your optimizer will apply to all parameters in the network, and not just the conv layers.


How to do weight normalization in last classification layer?

Question:
Many loss functions in face recognition will norm feature or norm weight before calculate softmax loss, such as normface(https://arxiv.org/abs/1704.06369 30), L2-softmax(https://arxiv.org/abs/1703.09507 25) and so on.
I’d like to know how to norm weight in the last classification layer.
self.feature =  torch.nn.Linear(7*7*64, 2) # Feature extract layer
self.pred = torch.nn.Linear(2, 10, bias=False) # Classification layer

I want to replace the weight parameter in self.pred module with a normalized one.
In another word, I want to replace weight in-place, like this:
self.pred.weight = self.pred.weight / torch.norm(self.pred.weight, dim=1, keepdim=True)

When I trying to do this, there is something wrong:
TypeError: cannot assign 'torch.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)

I am new comer to pytorch, I don’t know what is the standard way to handle this. Thanks a lot!
Here is the whole code:
class Model(torch.nn.Module):

    def __init__(self):
        super(Model, self).__init__()

        self.backbone = torch.nn.Sequential(

            torch.nn.Conv2d(1, 8, kernel_size=7, stride=1, padding=3),

            # 28 * 28
            torch.nn.Conv2d(8, 16, kernel_size=5, stride=1, padding=2),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(stride=2, kernel_size=2),

            # 14 * 14
            torch.nn.Conv2d(16, 32, kernel_size=3, stride=1, padding=1),
            torch.nn.ReLU(),
            torch.nn.Conv2d(32, 32, kernel_size=3, stride=1, padding=1),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(stride=2, kernel_size=2),

            # 7 * 7
            torch.nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
            torch.nn.ReLU(),
            torch.nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1),
            torch.nn.ReLU(),

        )

        self.feature =  torch.nn.Linear(7*7*64, 2) # Feature extract layer
        self.pred = torch.nn.Linear(2, 10, bias=False) # Classification layer

        ## something wrong here
        self.pred.weight = self.pred.weight / torch.norm(self.pred.weight, dim=1, keepdim=True)

        for m in self.modules():
            if isinstance(m, torch.nn.Conv2d):
                torch.nn.init.xavier_normal_(m.weight)
            elif isinstance(m, torch.nn.BatchNorm2d):
                torch.nn.init.constant_(m.weight, 1)
                torch.nn.init.constant_(m.bias, 0)

    def forward(self, x):

        x = self.backbone(x)
        x = x.view(-1, 7 * 7 * 64)
        x = self.feature(x)
        x = self.pred(x)

        return x

Besides,

if I don’t want to use view function in forward function, how to deal with it in __init__ function？
how I can extract the weight, which is before normalization and after normalization?
If I want to norm feature which is the output feature map in the last two fc layer  in __init__ function, how to do it?

Thanks a lot!

Context:
Besides, I have read the doc about WeightNorm(https://pytorch.org/docs/stable/_modules/torch/nn/utils/weight_norm.html 522)
I just need one normalized weight, and do not need both of g and v.


Using pined memory causes out-of-memory error even though batch size is set to low values

Question:
I am fine-tuning on a custom dataset using ResNet. I have met with this strange issue. When I try to train the model, I see the following error message:
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCCachingHostAllocator.cpp line=271 error=2 : out of memory
Traceback (most recent call last):
  File "train.py", line 282, in <module>
    main()
  File "train.py", line 141, in main
    epoch)
  File "train.py", line 179, in train
    for batch_idx, (data, target) in enumerate(train_loader):
  File "/home/haojiedong/tools/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 316, in __next__
    batch = pin_memory_batch(batch)
  File "/home/haojiedong/tools/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 200, in pin_memory_batch
    return [pin_memory_batch(sample) for sample in batch]
  File "/home/haojiedong/tools/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 200, in <listcomp>
    return [pin_memory_batch(sample) for sample in batch]
  File "/home/haojiedong/tools/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 194, in pin_memory_batch
    return batch.pin_memory()
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCCachingHostAllocator.cpp:271

I tried to reduce the batch size from 64 all the way to batch size 1. But this error still persists. Then I tried to change the training code and use pin_memory=False in the dataloader parameter. Now I can train the mode with batch size 64 without any error.
I have always used pin_memory=True in the dataloader and  I have never experienced this kind of error before.  Why is this failing now?
The pytorch version I am using is 0.4.1.

Context: Can anyone comment on this issue? Have you ever encountered such issue?


How to use the Inception model for transfer learning?

Question:
I have created a torchvision model for transfer learning, using the pre-built ResNet50 base model, like this:
        # Create base model from torchvision.models
        model = resnet50(pretrained=True)
        num_features = model.fc.in_features

        # Define the network head and attach it to the model
        model_head = nn.Sequential(
            nn.Linear(num_features, 512),
            nn.ReLU(),
            nn.Dropout(0.25),
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(256, num_classes),
        )
        model.fc = model_head

Now I wanted to use the Ineception v3 model instead as base, so I switched from resnet50() above to inception_v3(), the rest stayed as is. However, during training I get the following error:

TypeError: cross_entropy_loss(): argument ‘input’ (position 1) must be Tensor, not InceptionOutputs

So how can one use the Inception v3 model from torchvision.models as base model for transfer learning?

Context:
For the records:
I found the solution here 2.


Pytorch: Different Input within Stacking Ensemble & Apex library!

Question:
I shortly will describe my project:
+2 input for 2 networks: image(3x256x256) & 20 features(1 tensor of 20 number)
+Stacking ensemble. more 2
This is my model:
class MetaMelanoma(nn.Module):
    def __init__(self,out_dim=9,n_meta_features=0,n_meta_dim=[512, 128]):
            super(MetaMelanoma,self).__init__()
            self.enet = timm.create_model('tf_efficientnet_b0_ns',pretrained=True)
            self.n_meta_features = n_meta_features
            self.dropouts = nn.ModuleList([
            nn.Dropout(0.5) for _ in range(5)])
            in_ch = self.enet.classifier.in_features
            self.meta = nn.Sequential(
                nn.Linear(n_meta_features, n_meta_dim[0]),
                nn.BatchNorm1d(n_meta_dim[0]),
                Swish_Module(),
                nn.Dropout(p=0.3),
                nn.Linear(n_meta_dim[0], n_meta_dim[1]),
                nn.BatchNorm1d(n_meta_dim[1]),
                Swish_Module(),
                nn.Linear(n_meta_dim[1],9)
            )
            self.myfc = nn.Linear(in_ch, out_dim)
            self.enet.classifier = nn.Identity()
    def extract(self, x):
        x = self.enet(x)
        return x
    def forward(self, x,x_meta):
        x = self.extract(x).squeeze(-1).squeeze(-1)
        if self.n_meta_features > 0:
            x_meta = self.meta(x_meta)
            x = torch.cat((x, x_meta), dim=1)
        for i, dropout in enumerate(self.dropouts):
            if i == 0:
                out = self.myfc(dropout(x))
            else:
                out += self.myfc(dropout(x))
        out /= len(self.dropouts)

        return out

In order to train the network, I need Apex Nvidia library for optimizer and loss function more
Problem: To install Apex, we need old cuda version satisfying pytorch version. However, I am using a sever from my school, I can’t change cuda version. So anyone have any solution for this, maybe the same library or protocol?

Context: apex.amp is deprecated so use the native mixed-precision training utility via torch.cuda.amp. You can find the examples here 2.


Are there any constraints to the image sizes for torchvision models?

Question:
The torchvision.models documentation mentions that all models require an image input size of “at least 224x224”, with some exceptons, like the Inception model, which requires exactly (?) 299x299. Now I am wondering, for all other models, are really any other image dimensions fine as well?
I understand that technically these “modern” models can all handle other dimensions as well. But are there any constraints to the image dimensions, or better: are there any dimensions that work better than others? Or can I use any “random” sizes, like for example 474x474 or 834x834, or whatever? Or should I follow some “best practices” when choosing the image dimensions?

Context:
If I am not mistaken, they use an average pooling layer after convolutions. So the image data can go through the model without popping up errors. That is not to say that the comprehension of the model will be the same for all image sizes. It depends on the structure of the model.
For instance, a Conv2d layer with a given kernel may be able to capture a particular feature of the image important to classification at one size, but not when that same feature is much larger. (Read up on kernels to understand why.)
Due to the way neural networks learn, it’s not like there is a one size fits all answer to this. It may also be dataset dependent. But generally speaking, if you have enough compute resources, you should trying resizing beforehand and see what size works best for your application. Or if you don’t have those resources, just preprocess your images to the smallest size allowed, as that will also equate to less calculations/compute time during training or inference.


Modifing segmentation masks for only a few specific classes

Question:
Hello,
Right now I have pseudo-labels for an image dataset from DeepLabV3+ that are of shape [H W, 1] where each pixel is an integer from 0 to 18 representing the class of the pixel (using cityscapes pretraied models).
Now, what I want to do is focus on only a few classes (say for example, 10, 12, 13) and map the rest of the classes to some background class. What is the best way to do this in PyTorch?
I guess numpy would also work.
Thanks!

Context:
Hi Dryden!

 drydenwiebe:

Now, what I want to do is focus on only a few classes (say for example, 10, 12, 13) and map the rest of the classes to some background class.

You therefore want four classes, “background”, “class 10”, “class 12”, and
“class 13.”  You should then label them with with the four integer class
labels 0, 1, 2, 3.
Set up a translation table that maps the integers {0, 1, 2, ..., 17, 18}
to your desired smaller set of class labels, {0, 1, 2, 3}:
translate_table = torch.tensor ([0] * 19)
translate_table[10] = 1
translate_table[12] = 2
translate_table[13] = 3

Let’s say that labels is your pseudo-labels tensor (in your example
of shape [H, W, 1]).  Then:
mapped_labels = translate_table[labels]

will be the class labels for your reduced four-class problem (and will
still have shape [H, W, 1]).
Best.
K. Frank


Question:
I am trying to extract features.
extracted_features = []
main_extracted_features = []
for batch in range(100):
print(“Batch Number”,batch)
data_for = new_data[batch*100:(batch+1)*100]
for i in range(len(data_for)):
img = torch.tensor(data_for[i]/255).float()
ft_extract = feature_extractor(img.to(device))
extracted_features.append(ft_extract)
main_extracted_features.append(extracted_features)
print(np.array(extracted_features).shape)
print(np.array(main_extracted_features).shape)
and then crashes
Screenshot 2021-01-09 at 1.16.28 PM996×524 73.6 KB

Context: It would seem like colab is running out of memory. I think it is because of not using .detach() function but not sure, have to look at more of the code.


RuntimeError: Found dtype Double but expected Float - PyTorch

Question:
I am new to pytorch and I am working on DQN for a timeseries using Reinforcement Learning and I needed to have a complex observation of timeseries and some sensor readings, so I merged two neural networks and I am not sure if that’s what is ruining my loss.backward or something else.
I know there is multiple questions with the same title but none worked for me, maybe I am missing something.
First of all, this is my network:
class DQN(nn.Module):
  def __init__(self, list_shape, score_shape, n_actions):
    super(DQN, self).__init__()

    self.FeatureList =  nn.Sequential(
            nn.Conv1d(list_shape[1], 32, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.Conv1d(32, 64, kernel_size=4, stride=2),
            nn.ReLU(),
            nn.Conv1d(64, 64, kernel_size=3, stride=1),
            nn.ReLU(),
            nn.Flatten()
        )

    self.FeatureScore = nn.Sequential(
            nn.Linear(score_shape[1], 512),
            nn.ReLU(),
            nn.Linear(512, 128)
        )

    t_list_test = torch.zeros(list_shape)
    t_score_test = torch.zeros(score_shape)
    merge_shape = self.FeatureList(t_list_test).shape[1] + self.FeatureScore(t_score_test).shape[1]

    self.FinalNN =  nn.Sequential(
            nn.Linear(merge_shape, 512),
            nn.ReLU(),
            nn.Linear(512, 128),
            nn.ReLU(),
            nn.Linear(128, n_actions),
    )

  def forward(self, list, score):
    listOut = self.FeatureList(list)
    scoreOut = self.FeatureScore(score)
    MergedTensor = torch.cat((listOut,scoreOut),1)
    return self.FinalNN(MergedTensor)

I have a function called calc_loss, and at its end it return the MSE loss as below
  print(state_action_values.dtype)
  print(expected_state_action_values.dtype)
  return nn.MSELoss()(state_action_values, expected_state_action_values)

and the print shows float32 and float64 respectively.
I get the error when I run the loss.backward() as below
LEARNING_RATE = 0.01
optimizer = optim.Adam(net.parameters(), lr=LEARNING_RATE)

for i in range(50):
  optimizer.zero_grad()
  loss_v = calc_loss(sample(obs, 500, 200, 64), net, tgt_net)
  print(loss_v.dtype)
  print(loss_v)
  loss_v.backward()
  optimizer.step()

and the print output is as below:
torch.float64
tensor(1887.4831, dtype=torch.float64, grad_fn=)
Update 1:
I tried using a simpler model, yet the same issue, when I tried to cast the inputs to Float, I got an error:
RuntimeError: expected scalar type Double but found Float

What makes the model expects double ?

Context:
Update 2:
I tried to add the below line on top after the torch import but same issue of RuntimeError: Found dtype Double but expected Float
>>> torch.set_default_tensor_type(torch.FloatTensor)

But when I used the DoubleTensor I got:
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.DoubleTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor


No GPU utilization although CUDA seems to be activated

Question:
Hi,
I have an Alienware laptop with GeForce GTX 980M , and I’m trying to run my first code in pytorch - using transfer learning with resnet.
The thing is that I get no GPU utilization although all CUDA signs in python seems to be ok:
print(“torch.cuda.is_available()   =”, torch.cuda.is_available())
print(“torch.cuda.device_count()   =”, torch.cuda.device_count())
print(“torch.cuda.device(‘cuda’)   =”, torch.cuda.device(‘cuda’))
print(“torch.cuda.current_device() =”, torch.cuda.current_device())
print(“torch.cuda.get_device_name(0) =”,torch.cuda.get_device_name(0))
cudnn.benchmark = True
#Additional Info when using cuda
if device.type == ‘cuda’:
print(torch.cuda.get_device_name(0))
print(‘Memory Usage:’)
print(‘Allocated:’, round(torch.cuda.memory_allocated(0)/10243,1), ‘GB’)
print('Cached:   ', round(torch.cuda.memory_cached(0)/10243,1), ‘GB’)
torch.cuda.is_available()   = True
torch.cuda.device_count()   = 1
torch.cuda.device(‘cuda’)   = <torch.cuda.device object at 0x0000021F70F15D68>
torch.cuda.current_device() = 0
torch.cuda.get_device_name(0) = GeForce GTX 980M
GeForce GTX 980M
Memory Usage:
Allocated: 0.0 GB
Cached:    0.0 GB
I tried googling it and also looked here but came up with nothing helpful.
running nvidia-smi also shows that there is no GPU memory usage, and the following message:
    Process ID                  : 19792
        Type                    : C
        Name                    : C:\Users\itaim\AppData\Local\Programs\Python\Python36\python.exe
        Used GPU Memory         : Not available in WDDM driver model

attaching task manager screenshot and nviidia-smi output:

image.png1792×1443 211 KB

nvidia-smi -l
C:\Users\itaim>nvidia-smi -l
±----------------------------------------------------------------------------+
| NVIDIA-SMI 419.35       Driver Version: 419.35       CUDA Version: 10.1     |
|-------------------------------±---------------------±---------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 980M   WDDM  | 00000000:01:00.0 Off |                  N/A |
| N/A   66C    P0    69W /  N/A |    917MiB /  8192MiB |     51%      Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     19792      C   …cal\Programs\Python\Python36\python.exe N/A      |
±----------------------------------------------------------------------------+
nvidia-smi -q
C:\Users\itaim>nvidia-smi -q
==============NVSMI LOG==============
Timestamp                           : Wed Mar 27 01:09:08 2019
Driver Version                      : 419.35
CUDA Version                        : 10.1
Attached GPUs                       : 1
GPU 00000000:01:00.0
Product Name                    : GeForce GTX 980M
Product Brand                   : GeForce
Display Mode                    : Disabled
Display Active                  : Disabled
Persistence Mode                : N/A
Accounting Mode                 : Disabled
Accounting Mode Buffer Size     : 4000
Driver Model
Current                     : WDDM
Pending                     : WDDM
Serial Number                   : N/A
GPU UUID                        : GPU-544e0a63-680d-08ca-c2da-1ef57adb2fff
Minor Number                    : N/A
VBIOS Version                   : 84.04.87.00.08
MultiGPU Board                  : No
Board ID                        : 0x100
GPU Part Number                 : N/A
Inforom Version
Image Version               : N/A
OEM Object                  : N/A
ECC Object                  : N/A
Power Management Object     : N/A
GPU Operation Mode
Current                     : N/A
Pending                     : N/A
GPU Virtualization Mode
Virtualization mode         : None
IBMNPU
Relaxed Ordering Mode       : N/A
PCI
Bus                         : 0x01
Device                      : 0x00
Domain                      : 0x0000
Device Id                   : 0x13D710DE
Bus Id                      : 00000000:01:00.0
Sub System Id               : 0x07081028
GPU Link Info
PCIe Generation
Max                 : 3
Current             : 3
Link Width
Max                 : 16x
Current             : 8x
Bridge Chip
Type                    : N/A
Firmware                : N/A
Replays Since Reset         : 0
Replay Number Rollovers     : 0
Tx Throughput               : 59000 KB/s
Rx Throughput               : 26000 KB/s
Fan Speed                       : N/A
Performance State               : P0
Clocks Throttle Reasons
Idle                        : Not Active
Applications Clocks Setting : Not Active
SW Power Cap                : Not Active
HW Slowdown                 : Not Active
HW Thermal Slowdown     : N/A
HW Power Brake Slowdown : N/A
Sync Boost                  : Not Active
SW Thermal Slowdown         : Not Active
Display Clock Setting       : Not Active
FB Memory Usage
Total                       : 8192 MiB
Used                        : 827 MiB
Free                        : 7365 MiB
BAR1 Memory Usage
Total                       : 256 MiB
Used                        : 229 MiB
Free                        : 27 MiB
Compute Mode                    : Default
Utilization
Gpu                         : 39 %
Memory                      : 25 %
Encoder                     : 0 %
Decoder                     : 0 %
Encoder Stats
Active Sessions             : 0
Average FPS                 : 0
Average Latency             : 0
FBC Stats
Active Sessions             : 0
Average FPS                 : 0
Average Latency             : 0
Ecc Mode
Current                     : N/A
Pending                     : N/A
ECC Errors
Volatile
Single Bit
Device Memory       : N/A
Register File       : N/A
L1 Cache            : N/A
L2 Cache            : N/A
Texture Memory      : N/A
Texture Shared      : N/A
CBU                 : N/A
Total               : N/A
Double Bit
Device Memory       : N/A
Register File       : N/A
L1 Cache            : N/A
L2 Cache            : N/A
Texture Memory      : N/A
Texture Shared      : N/A
CBU                 : N/A
Total               : N/A
Aggregate
Single Bit
Device Memory       : N/A
Register File       : N/A
L1 Cache            : N/A
L2 Cache            : N/A
Texture Memory      : N/A
Texture Shared      : N/A
CBU                 : N/A
Total               : N/A
Double Bit
Device Memory       : N/A
Register File       : N/A
L1 Cache            : N/A
L2 Cache            : N/A
Texture Memory      : N/A
Texture Shared      : N/A
CBU                 : N/A
Total               : N/A
Retired Pages
Single Bit ECC              : N/A
Double Bit ECC              : N/A
Pending                     : N/A
Temperature
GPU Current Temp            : 66 C
GPU Shutdown Temp           : 96 C
GPU Slowdown Temp           : 91 C
GPU Max Operating Temp      : 101 C
Memory Current Temp         : N/A
Memory Max Operating Temp   : N/A
Power Readings
Power Management            : N/A
Power Draw                  : 41.15 W
Power Limit                 : N/A
Default Power Limit         : N/A
Enforced Power Limit        : N/A
Min Power Limit             : N/A
Max Power Limit             : N/A
Clocks
Graphics                    : 1126 MHz
SM                          : 1126 MHz
Memory                      : 2505 MHz
Video                       : 1036 MHz
Applications Clocks
Graphics                    : 1038 MHz
Memory                      : 2505 MHz
Default Applications Clocks
Graphics                    : 1038 MHz
Memory                      : 2505 MHz
Max Clocks
Graphics                    : 1126 MHz
SM                          : 1126 MHz
Memory                      : 2505 MHz
Video                       : 1036 MHz
Max Customer Boost Clocks
Graphics                    : N/A
Clock Policy
Auto Boost                  : N/A
Auto Boost Default          : N/A
Processes
Process ID                  : 20840
Type                    : C
Name                    : C:\Users\itaim\AppData\Local\Programs\Python\Python36\python.exe
Used GPU Memory         : Not available in WDDM driver model
and my code:
import torch
import torchvision
import torch.nn as nn
import torch.nn.functional as F
from torchvision import transforms, datasets, models
from torch.utils.data import Dataset, DataLoader
from torch.autograd import Variable
import torch.optim as optim
from torch.optim import lr_scheduler
from torch.utils.data.sampler import SubsetRandomSampler
import torch.backends.cudnn as cudnn
import matplotlib.pyplot as plt
import numpy as np
import torch.optim as optim
import pandas as pd
import os
from PIL import Image
import pickle
import urllib
from urllib.request import urlopen
import time
import copy
device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)
print(“torch.cuda.is_available()   =”, torch.cuda.is_available())
print(“torch.cuda.device_count()   =”, torch.cuda.device_count())
print(“torch.cuda.device(‘cuda’)   =”, torch.cuda.device(‘cuda’))
print(“torch.cuda.current_device() =”, torch.cuda.current_device())
print(“torch.cuda.get_device_name(0) =”,torch.cuda.get_device_name(0))
cudnn.benchmark = True
#Additional Info when using cuda
if device.type == ‘cuda’:
print(torch.cuda.get_device_name(0))
print(‘Memory Usage:’)
print(‘Allocated:’, round(torch.cuda.memory_allocated(0)/10243,1), ‘GB’)
print('Cached:   ', round(torch.cuda.memory_cached(0)/10243,1), ‘GB’)
class DogsTrainingDataset(Dataset):
def __init__(self,text_file,root_dir):
    """
    Args:
        text_file(string): path to text file
        root_dir(string): directory with all train images
    """
    self.name_frame = pd.read_csv(text_file,sep=",",usecols=range(1))
    self.label_frame = pd.read_csv(text_file,sep=",",usecols=range(1,2))
    self.label_idx_frame = pd.read_csv(text_file,sep=",",usecols=range(2,3))
    self.root_dir = root_dir
    self.transform = transforms.Compose([
         transforms.Resize(224),
         transforms.CenterCrop(224),
         transforms.ToTensor(),
         transforms.Normalize( mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
         ])

def __len__(self):
    return len(self.name_frame)

def __getitem__(self, idx):
    img_name = os.path.join(self.root_dir, self.name_frame.iloc[idx, 0]) + '.jpg'
    label_str = self.label_frame.iloc[idx, 0]
    label = self.label_idx_frame.iloc[idx, 0]
    image = Image.open(img_name)
    image = self.transform(image)
    #labels = labels.reshape(-1, 2)
    return [image, label, label_str]

def imshow(img, title=None):
inp = img.numpy().transpose((1, 2, 0))
mean = np.array([0.485, 0.456, 0.406])
std = np.array([0.229, 0.224, 0.225])
inp = std * inp + mean
inp = np.clip(inp, 0, 1)
plt.imshow(inp)
if title is not None:
plt.title(title)
plt.pause(0.001)  # pause a bit so that plots are updated
def train_model(model, criterion, optimizer, scheduler, validation_split, dataloaders, num_epochs=25):
since = time.time()
best_model_wts = copy.deepcopy(model.state_dict())
best_acc = 0.0
best_epoch_idx = 0
train_loss = []
train_acc = []
val_loss = []
val_acc = []

val_size = np.floor(len(dataloaders['train'].dataset) * validation_split)
train_size = len(dataloaders['train'].dataset) - val_size

for epoch in range(num_epochs):
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 10)

    # Each epoch has a training and validation phase
    for phase in ['train', 'val']:
        if phase == 'train':
            dataset_size = train_size
            scheduler.step()
            model.train()  # Set model to training mode
        else:
            dataset_size = val_size
            model.eval()   # Set model to evaluate mode

        running_loss = 0.0
        running_corrects = 0

        # Iterate over data.
        for inputs, labels, labels_str in dataloaders[phase]:
            inputs = inputs.to(device)
            labels = labels.to(device)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward
            # track history if only in train
            with torch.set_grad_enabled(phase == 'train'):
                #if(phase == 'train'):
                #    outputs,aux = model(inputs)
                #else:
                #    outputs = model(inputs)
                outputs = model(inputs)
                _, preds = torch.max(outputs, 1)
                loss = criterion(outputs, labels)

                # backward + optimize only if in training phase
                if phase == 'train':
                    loss.backward()
                    optimizer.step()

            # statistics
            running_loss += loss.item() * inputs.size(0)
            running_corrects += torch.sum(preds == labels.data)
        epoch_loss = running_loss / dataset_size
        epoch_acc = running_corrects.double() / dataset_size

        print('{} Loss: {:.4f} Acc: {:.4f}'.format(
            phase, epoch_loss, epoch_acc))
        # save loss and accuracy
        if phase == 'train':
            train_loss.append(epoch_loss)
            train_acc.append(epoch_acc)
        else:
            val_loss.append(epoch_loss)
            val_acc.append(epoch_acc)
        # deep copy the model
        if phase == 'val' and epoch_acc > best_acc:
            best_acc = epoch_acc
            best_epoch_idx = epoch
            best_model_wts = copy.deepcopy(model.state_dict())

time_elapsed = time.time() - since
print('Training complete in {:.0f}m {:.0f}s'.format(
    time_elapsed // 60, time_elapsed % 60))
print('Best val Acc: {:4f}'.format(best_acc))

# plot loss and accuracy

plt.figure()
plt.xlabel('# of epochs')
plt.title('Accuracy')
plt.plot(range(num_epochs), train_acc, label='Train Accuracy')
plt.plot(range(num_epochs), val_acc, label='Validation Accuracy')
plt.legend()
plt.figure()
plt.xlabel('# of epochs')
plt.title('Loss')
plt.plot(range(num_epochs), train_loss, label='Train Loss')
plt.plot(range(num_epochs), val_loss, label='Validation Loss')
plt.legend()
print('best epoch index:',best_epoch_idx)
# load best model weights
model.load_state_dict(best_model_wts)
return model

dogsTrainSet = DogsTrainingDataset(text_file =‘C:\Users\itaim\Master\TeamLily\labels.csv’,
root_dir = ‘C:\Users\itaim\Master\TeamLily\train’)
num_epochs = 10
batch_size = 16
validation_split = .2
shuffle_dataset = True
random_seed= 42
Creating data indices for training and validation splits:
dataset_size = len(dogsTrainSet)
indices = list(range(dataset_size))
split = int(np.floor(validation_split * dataset_size))
if shuffle_dataset :
np.random.seed(random_seed)
np.random.shuffle(indices)
train_indices, val_indices = indices[split:], indices[:split]
Creating PT data samplers and loaders:
train_sampler = SubsetRandomSampler(train_indices)
val_sampler = SubsetRandomSampler(val_indices)
handle data loaders
dogsTrainLoader = DataLoader(dogsTrainSet, batch_size=16, sampler=train_sampler, pin_memory=True)
dogsValLoader = DataLoader(dogsTrainSet, batch_size=16, sampler=val_sampler, pin_memory=True)
dataloaders = {‘train’: dogsTrainLoader , ‘val’: dogsValLoader}
init model and its attributes
model = models.resnet50(pretrained=True)
#model.aux_logit=False
option 1 - use the model as feature extractor (train only classifier)
for param in model.parameters():
param.requires_grad = False
num_ftrs = model.fc.in_features
model.fc = nn.Linear(num_ftrs, 120)
model = model.to(device)
criterion = nn.CrossEntropyLoss()
choose SG with momentum
optimizer = optim.SGD(model.fc.parameters(), lr=0.001, momentum=0.9)
Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)
train the model and evaluate on the validation set
print(’#########################################’)
print(‘running with SGD’)
print(’#########################################’)
model_trained1 = train_model(model, criterion, optimizer,
exp_lr_scheduler, validation_split, dataloaders, num_epochs=25)
print(’#########################################’)
print(‘running with Adam’)
print(’#########################################’)
optimizer = optim.Adam(model.fc.parameters(), lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)
exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)
model_trained2 = train_model(model, criterion, optimizer,
exp_lr_scheduler, validation_split, dataloaders, num_epochs=25)
any suggestions?
Thanks in advance

Context:
Hi,
You can print the inputs before feeding them to the model to make sure they are on the gpu.
It is possible that your model is so small that the gpu is barely used as well, you can try increasing the batch size to see if the gpu usage increases.


Create/edit PyTorch tensor using OpenGL?

Question: In order to create polygonal masks I’m currently using Pillow’s ImageDraw to draw them. Then, I can get the corresponding numpy arrays and upload to GPU. But I’m thinking about creating them directly on the GPU using OpenGL, via, say, pyglet or glumpy. I found somewhere else how to pass PyTorch tensors to CuPy using data_ptr() and the current CUDA stream, and I wonder whether something along those lines can be used to “draw” to a PyTorch tensor using OpenGL. Does anyone know how to do that?

Context:
You can, but it’s going to require a bunch of work including C/C++ code.

You need render to a texture using pyglet or glumpy. Get the texture ID (an int)

Use the CUDA OpenGL API to register the texture: https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__OPENGL.html 40

Use the CUDA Graphics API to bind the resource to a pointer. https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__INTEROP.html#group__CUDART__INTEROP 19

Either cudaMemcpy that pointer into a Tensor’s data_ptr() or use torch::from_blob to wrap it as a Tensor.

You’ll also need to call the appropriate unregister functions in destruction and make sure that the texture is in a matching format as the Tensor (i.e. float32 vs. int8)


Model Optimization using Low Rank Transformation in MMDetection

Question:
Hello, I am new to mmdetection and model optimization. I am using mmdetection to use mask RCNN with SWIN transformer as the backbone. As the model is big I was trying to reduce its size.As far as I know , pruning as implemented in pytorch would not change the model size but would simply create a mask.I have also tried L1 unstructured quantization with success and was currently trying low rank transformations. I can see the transformation happening when I compare the print(model) before and after the low-rank transformation. However, the model is not able to infer anything. I see no bounding box that appear. The following is what I have tried :
config='/content/mmdetection/configs/swin/mask_rcnn_swin-t-p4-w7_fpn_1x_coco.py'
# Setup a checkpoint file to load
checkpoint = 'checkpoints/mask_rcnn_swin-t-p4-w7_fpn_1x_coco_20210902_120937-9d6b7cfa.pth'
# initialize the detector
model = init_detector(config,checkpoint)

Once the model is initialized I use the following for low-rank transformation
def low_rank_transform(model):
    svd_classifier_layers=[]
    L=40
    for module_name, module in model.named_children():
      if module_name=='proj':
        continue
      if len(list(module.children())) > 0:
        replace_layers(module)
      if isinstance(module, torch.nn.Linear):
        W=module.weight.data
        # print(W)
        # U,S,V=torch.svd(W)
        U, S, Vh = torch.linalg.svd(W)
        V = Vh.transpose(-2, -1).conj()
        W1 = U[:,:L]
        W2 = torch.diag(S[:L]) @ V[:,:L].t()
        if module.bias is not None:
          # print("W1:",W1)
          layer_1 = nn.Linear(in_features=module.in_features,
                            out_features=L, bias=True)
          layer_1.weight.data = W2
          svd_classifier_layers.append(layer_1)

          layer_2 = nn.Linear(in_features=L,
                            out_features=module.out_features , bias=True)
          layer_2.weight.data = W1
          layer_2.bias.data = module.bias.data
          svd_classifier_layers.append(layer_2)
          # print("Layer2",layer_2.weight.data)
          # break

        else:
          layer_1 = nn.Linear(in_features=module.in_features,
                            out_features=L, bias=False)
          layer_1.weight.data = W2
          svd_classifier_layers.append(layer_1)

          layer_2 = nn.Linear(in_features=L,
                            out_features=module.out_features , bias=False)
          layer_2.weight.data = W1
          # layer_2.bias.data = module.bias.data
          svd_classifier_layers.append(layer_2)

        try:
          # print("we are now trying to replace",module,"with",svd_classifier_layers,"in a sequnetial way")
          # n =int(module_name)
          # model[n]=nn.Sequential(*svd_classifier_layers)
          setattr(model, module_name, nn.Sequential(*svd_classifier_layers))
        except:
          setattr(model, module_name, nn.Sequential(*svd_classifier_layers))

Then I simply call the above function by passing only the backbone of the model.
low_rank_transform(model):(model.backbone)
I can see the expected change in the architecture where a single Linear layer gets replaced by two linear layers. However, nothing happens during inference.
Can someone help me with what is that I might be doing wrong here?

Context: I have observed on using the inference_detector() that the confidence of the model is relatively low and which is obvious. Can someone suggest a better way to do low-rank transformations on a model? or perhaps how to change the model’s score threshold so as to get bounding boxes in the image.


When opening RGB image in grayscale, can 2 colors be collapsed in the same gray level?

Question:
I am working on Audi A2D2 dataset 2D segmentation.
The segmentation mask are provided as 3-channel, RGB images (despite having only 55 classes).
In order to process the labels faster in my data loader, I’d like to open them as grayscale even though they are RGB. Because there are only 55 different RGB pixel values, it should be possible to convert to gray scale and keeping unique pixel value for the 55 classes, since there are 256 possible shades of grey.
from torchvision.io import read_image
from torchvision.io.image import ImageReadMode

read_image(path, mode=ImageReadMode.RGB)

I’m wondering: is there a risk of 2 RGB pixel being assigned the same grayscale value, when opening an RGB picture with ImageReadMode.GRAY?

Context:
Hi Olivier!

 Olivier-CR:

is there a risk of 2 RGB pixel being assigned the same grayscale value

Yes, the risk is real, and such an occurrence is in fact quite likely:  See the
so-called “birthday paradox.”
If the person who prepared the segmentation masks were kind, then he
could have purposely chosen RGB values that map to distinct grayscale
values.  (And if he were diabolical, he could have chosen 55 RGB values
that all map to the same grayscale value.)  But if the RGB values were
chosen somewhat “randomly,” then you will likely have such a collision.
So …   Do you feel lucky?
Best.
K. Frank


Question:
Hi, can someone ping me to an example of partial transfer learning with EfficientNet? For instance, unfreezing the last two blocks of the network?
Thanks in advance!

Context:
You can freeze all parameters of the model first via:
for param in model.parameters():
    param.requires_grad_(False)

and later unfreeze the desired blocks by printing the model (via print(model)) and use the corresponding module names to unfreeze their parameters. E.g. assuming the last two “blocks” are named block1 and block2, this should work:
for param in model.block1.parameters():
    param.requires_grad_(True)
for param in model.block2.parameters():
    param.requires_grad_(True)


Convolution operation with L2 normalized weights

Question:
Hi all,
Is there a way normalize (L2) the weights of a convolution kernel before performing the convolution?
For a fully connected layer, I’d go about doing something like:
# __init__()
weights = nn.Parameter(torch.Tensor(in_size, out_size))

# forward
forward(x):
    w = F.normalize(weights)
    out = torch.mm(x, w)
    return out

But I’m not exactly sure how this would work on convolutions, even if I’m using 1x1 convolution kernels.
Any ideas?

Context:
You could use the same approach, but call F.conv2d instead of torch.mm to perform a convolution.
Also, the input and weight shapes would be different, since F.conv2d expects an input of [batch_size, channels, height, width] and a weight of [out_channels, in_channels, height, width].


Question:
Hi,
In many papers, I will see the heat map, how to get it, what is the API of pytorch?

image.png675×517 58.5 KB

image.jpg508×699 166 KB

thanks

Context:
plt.imshow(arr, map=‘jet’)
arr: is  output of conv2d  ?
thanks


Question:
I want to implement a linear function: y = [w_1x_1+b_1; w_2x_2+b_2;…;w_kx_k+b_k] the size of input x is (b,k*c), where b is the batch size, k is the number of groups, c is the number of channels. As shown below:

linear.jpg1024×832 29.2 KB

And a Grouped convolution layer is also needed. The grouped convolution layer just like using the convolution setting groups=k. The difference is that the weight and bias is not expected to be shared between groups.
Although this operation can be implemented with multiple linear or convolutional layers, I don’t think it is convenient.
Actually, I hope that different data may be processed with different weights. I want to transfer the batch to the channel, like (b, c, h, w)->(1, bxc, h, w), so that a sample corresponds to a group of channels, and each group of channels is processed with different weights. And then reshaping it back.

Context: did you find an elegant way to do that ?


CRNN with CTC loss predicts only blank labels

Question:
Hello,
I am trying to train CRNN with CTC loss. But, after some iterations, the model predicts only blank labels. I have tried solutions provided to similar problems. But, none worked in my case.
I am providing code, Colab notebook, and dataset.
Any help will be really appreciated.
Thanks in advance.

import os
import sys
import cv2
import tqdm
import glob
import torch
import torchvision
from torch import nn
from PIL import Image
from itertools import groupby
import matplotlib.pyplot as plt
from collections import OrderedDict
import torchvision.transforms as transforms
from torch.nn.modules.pooling import MaxPool2d
from torch.nn.modules.batchnorm import BatchNorm2d
from torch.utils.data import Dataset, DataLoader

cwd = os.getcwd()

"""# Model"""

class BiLSTM(torch.nn.Module):
    def __init__(self, n_In, n_hidden, n_Out):
        super(BiLSTM, self).__init__()
        self.rnn = torch.nn.LSTM(n_In, n_hidden, bidirectional=True)
        self.embedding = torch.nn.Linear(n_hidden * 2, n_Out)

    def forward(self, input):
        output, _ = self.rnn(input)
        T, b, h = output.size()
        t_rec = output.view(T * b, h)
        output = self.embedding(output)
#         output = sigmoid(output)
#         m = nn.Sigmoid()
        output = output.view(T, b, -1)
        return output

# n_In = 512
# n_hidden = 30
# n_Out = 62
# rnn = BiLSTM(n_In, n_hidden, n_Out)

class CRNN(torch.nn.Module):
    def __init__(self, n_classes):
        self.n_classes = n_classes
        super(CRNN, self).__init__()
        self.cnn = torch.nn.Sequential(OrderedDict([

            ("conv0" , torch.nn.Conv2d(in_channels = 1, out_channels = 64, kernel_size = 3, stride = 1, padding = 1)),
            ("relu0" , torch.nn.ReLU()),
            ("pooling0" , torch.nn.MaxPool2d(kernel_size = 2, stride = 2)),

            ("conv1" , torch.nn.Conv2d(in_channels = 64, out_channels = 128, kernel_size = 3, stride = 1, padding = 1)),
            ("relu1" , torch.nn.ReLU()),
            ("pooling1" , torch.nn.MaxPool2d(kernel_size = 2, stride = 2)),

            ("conv2" , torch.nn.Conv2d(in_channels = 128, out_channels = 256, kernel_size = 3, stride = 1, padding = 1)),

            ("batchnorm2" , torch.nn.BatchNorm2d(256)),
            ("relu2" , torch.nn.ReLU()),

            ("conv3" , torch.nn.Conv2d(in_channels = 256, out_channels = 256, kernel_size = 3, stride = 1, padding = 1)),
            ("relu3" , torch.nn.ReLU()),

            ("pooling2" , torch.nn.MaxPool2d(kernel_size=(2, 2), stride=(2, 1), padding=(0, 1))),

            ("conv4" , torch.nn.Conv2d(in_channels = 256, out_channels = 512, kernel_size = 3, stride = 1, padding = 1)),

            ("batchnorm4" , torch.nn.BatchNorm2d(512)),
            ("relu4" , torch.nn.ReLU()),

            ("conv5" , torch.nn.Conv2d(in_channels = 512, out_channels = 512, kernel_size = 3, stride = 1, padding = 1)),
            ("relu5" , torch.nn.ReLU()),

            ("pooling3" , torch.nn.MaxPool2d(kernel_size=(2, 2), stride=(2, 1), padding=(0, 1))),

            ("conv6" , torch.nn.Conv2d(in_channels = 512, out_channels = 512, kernel_size = 2, stride = 1, padding=0)),
            ("batchnorm6" , torch.nn.BatchNorm2d(512)),
            ("relu6" , torch.nn.ReLU())

        ]) )

        # self.cnn = self.conv_block
        self.lstm = torch.nn.Sequential( BiLSTM(512, 256, 256),
                                        BiLSTM(256, 256, n_classes)    )

        self.sigmoid = nn.Sigmoid()

    def forward(self, input):
        conv_output = self.cnn(input)
        b, c, h, w = conv_output.size()
        conv_output = conv_output.squeeze(2)
        # print(f"conv_output.shape : {conv_output.shape}")
        conv_output = conv_output.permute(2, 0, 1)  # [w, b, c]
#         print(f"conv_output.shape : {conv_output.shape}")
        lstm_op = self.lstm(conv_output)
        lstm_op = self.sigmoid(lstm_op)
        # output = lstm_op.transpose(1,0)
        return lstm_op

blank_label = 0
alphabets = "~0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'.- "
n_classes = len(alphabets)
model = CRNN(n_classes)
print(model)

# cwd

# files = glob.glob(cwd + "/cropped_data/*.jpg")
# files = glob.glob(cwd + "/Dataset/out/*.jpg")
files = glob.glob(cwd + "/Dataset/dataset1/test/*.jpg")
files.sort()
# files

"""# Utility Functions"""

# alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxys0123456789-"

alphabet_list = [a for a in alphabets]

alphabet_dict = {}
for i,c in list(enumerate(alphabet_list)):
    alphabet_dict[c]=i

def encode(string):
    output_list = []
    for char in string:
        output_list.append(alphabet_dict[char])
    return output_list

def decode(List):
    string = ""
    for element in List:
        string+=alphabets[int(element)]
    return string

def clean(string):
    new_string = ""
    for i in range(len(string)):
        if string[i] != '-' and (not (i > 0 and string[i - 1] == string[i])):
            new_string+=string[i]
    return new_string

def image_resize(image, width = None, height = None, inter = cv2.INTER_AREA):
    dim = None
    (h, w) = image.shape[:2]
    if width is None and height is None:
        return image
    if width is None:
        r = height / float(h)
        dim = (int(w * r), height)
    else:
        r = width / float(w)
        dim = (width, int(h * r))
    resized = cv2.resize(image, dim, interpolation = inter)
    return resized

def run_model(img):
    image = image_resize(img, height = 32)
    print(image.shape)

    # plt.imshow(image)
    # plt.show()
    gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    image_t = torch.Tensor(gray_image)
    image_t = torch.unsqueeze(image_t, 2)
    image_t = image_t.permute([2, 0, 1])
    image_t = torch.unsqueeze(image_t, 0)
    print(f"image_t.shape : {image_t.shape}")
    op = model.forward(image_t)
    return op

def draw_tsmaps(image, output):
    image = image_resize(image, height = 200)
    NOStamps = len(output)
    h,w,_ = image.shape
    distance = w / NOStamps
    y1 = 0
    y2 = h-1
    for stamp in range(NOStamps):
        x1 = int(stamp * distance)
        cv2.line(image,(x1,y1),(x1,y2),(0,255,0),1)

        image = cv2.putText(image, output[stamp], (x1,h//2), cv2.FONT_HERSHEY_SIMPLEX,
                           1, (255,0,0), 1, cv2.LINE_AA)

    return image

def predict(frame, dev=False):
    if not dev:
        output = run_model(frame)
    else:
        output = model.forward(frame)
#     print(output.shape)
#     n_o_t,_,_ = output.shape
    _, output = output.max(2)
    output = output.data
    output = output.reshape(-1)
    output = output.tolist()
    print(output)
    output = decode(output)
    clean_output = clean(output)
    return output,clean_output

for file in files[:10]:
    image = cv2.imread(file)
    output, clean_output = predict(image)

    n_o_t = len(output)
    image = draw_tsmaps(image, output)
    plt.imshow(image)
    plt.show()

    print(f"{n_o_t} results")

    print(f"output : {output}")
    print(f"clean_output : {clean_output}")

    print("_______________________________________________________________________")

"""# Defining Dataloaders"""

def custom_collate(batch):
    label_padding_value = alphabet_dict[' ']
    width = [item['img'].shape[2] for item in batch]
    indexes = [item['idx'] for item in batch]
    imgs = torch.ones([len(batch), batch[0]['img'].shape[0], batch[0]['img'].shape[1],
                       max(width)], dtype=torch.float32)
    for idx, item in enumerate(batch):
        try:
            imgs[idx, :, :, 0:item['img'].shape[2]] = item['img']
        except:
            print(imgs.shape)
    item = {'img': imgs, 'idx':indexes}
    if 'label' in batch[0].keys():
        labels = [item['label'] for item in batch]
        len_labels = [len(label) for label in labels]
        max_size_label = max(len_labels)
        for label in labels:
            label.extend([label_padding_value] * (max_size_label - len(label)))
        item['label'] = labels

        item['label'] = torch.Tensor(item['label'])
    return item

class OCRDataset(Dataset):
    def __init__(self, opt):
        super(Dataset, self).__init__()
        self.path = os.path.join(opt['path'], opt['imgdir'])
        self.images = os.listdir(self.path)
        self.nSamples = len(self.images)
        f = lambda x: os.path.join(self.path, x)
        self.imagepaths = list(map(f, self.images))
       	transform_list =  [#transforms.Resize((128,128)),
                           transforms.Grayscale(1),
                            transforms.ToTensor(),
                            transforms.Normalize((0.5,), (0.5,))]
        self.transform = transforms.Compose(transform_list)
        self.collate_fn = custom_collate

    def __len__(self):
        return self.nSamples

    def __getitem__(self, index):
        assert index <= len(self), 'index range error'
        imagepath = self.imagepaths[index]
        imagefile = os.path.basename(imagepath)
        img = Image.open(imagepath)
        if self.transform is not None:
            img = self.transform(img)
        item = {'img': img, 'idx':index}
        item['label'] = imagefile.split('_')[0]
        item['label'] = encode(item['label'])
        return item

cwd

batch_size = 16
opt_train = {
    "path" : cwd,
    # "imgdir" : "namewise"
    "imgdir" : "Dataset/dataset1/train"
}

opt_valid = {
    "path" : cwd,
    # "imgdir" : "namewise"
    "imgdir" : "Dataset/dataset1/valid"
}

opt_test = {
    "path" : cwd,
    # "imgdir" : "namewise"
    "imgdir" : "Dataset/dataset1/test"
}

train_data = OCRDataset(opt_train)
train_loader = DataLoader(train_data, shuffle=True, drop_last=True, batch_size=batch_size, num_workers=0, collate_fn = custom_collate)

valid_data = OCRDataset(opt_valid)
valid_loader = DataLoader(valid_data, shuffle=False, drop_last=True, batch_size=batch_size, num_workers=0, collate_fn = custom_collate)

test_data = OCRDataset(opt_test)
test_loader = DataLoader(test_data, shuffle=False, drop_last=True, batch_size=batch_size, num_workers=0, collate_fn = custom_collate)

criterion = nn.CTCLoss(blank=0, reduction='mean', zero_infinity=True)
# optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
# optimizer = torch.optim.SGD(model.parameters(), lr=0.001)#, momentum=0, dampening=0, weight_decay=0, nesterov=False)
optimizer = torch.optim.Adadelta(model.parameters())

def average(losses):
    return sum(losses)/len(losses)

def calc_accuracy(Y_train, Y_pred):
    train_correct = 0
    train_total = 0
    _, max_index = torch.max(Y_pred, dim=2)
    for i in range(batch_size):
        raw_prediction = list(max_index[:, i].detach().cpu().numpy())
        prediction = torch.IntTensor([c for c, _ in groupby(raw_prediction) if c != blank_label])
        if len(prediction) == len(Y_train[i]) and torch.all(prediction.eq(Y_train[i])):
            train_correct += 1
        train_total += 1
    accuracy = train_correct / train_total
    return accuracy

def train():
    losses = []
    accuracies = []
    for L in tqdm.tqdm(train_loader):
        X_train = (L['img'])
        Y_train = (L['label'])
        # print(f"Y_train: {Y_train}")

        batch_size = X_train.shape[0]
        optimizer.zero_grad()
        Y_pred = model(X_train).cuda()
        Y_pred_0 = Y_pred.permute(1, 0, 2)
        input_lengths = torch.IntTensor([len(t) for t in Y_pred_0])
        target_lengths = torch.IntTensor([len(t) for t in Y_train])
        Y_pred = Y_pred.log_softmax(dim=2)
        loss = criterion(Y_pred, Y_train, input_lengths, target_lengths)
        losses.append(loss)
        loss.backward()
        optimizer.step()
        accuracy = calc_accuracy(Y_train, Y_pred)
        accuracies.append(accuracy)

    avg_loss = average(losses)
    avg_accuracy = average(accuracies)

    print(f"Training Loss : {avg_loss}")
    print(f"Training Accuracy : {avg_accuracy}")

def valid():

    losses = []
    accuracies = []
    for L in tqdm.tqdm(train_loader):

        X_train = (L['img'])
        Y_train = (L['label'])

        batch_size = X_train.shape[0]
        optimizer.zero_grad()
        Y_pred = model(X_train).cuda()
        Y_pred_0 = Y_pred.permute(1, 0, 2)
        input_lengths = torch.IntTensor([len(t) for t in Y_pred_0])
        target_lengths = torch.IntTensor([len(t) for t in Y_train])
        Y_pred = Y_pred.log_softmax(dim=2)
        loss = criterion(Y_pred, Y_train, input_lengths, target_lengths)
        losses.append(loss)

        accuracy = calc_accuracy(Y_train, Y_pred)
        accuracies.append(accuracy)

    avg_loss = average(losses)
    avg_accuracy = average(accuracies)

    print(f"Validation Loss : {avg_loss}")
    print(f"Validation Accuracy : {avg_accuracy}")

epochs = 200
for epoch in range(epochs):
    train()
    valid()

for file in files[:10]:
    image = cv2.imread(file)
    output, clean_output = predict(image)

    n_o_t = len(output)
    image = draw_tsmaps(image, output)
    plt.imshow(image)
    plt.show()

    print(f"{n_o_t} results")

    print(f"output : {output}")
    print(f"clean_output : {clean_output}")

    print("_______________________________________________________________________")

Link of the Colab notebook:
https://drive.google.com/file/d/10d_RLxEkzl8F2yCvalNKF-zmULyaY6Pg/view?usp=sharing
Link of the Dataset:
https://drive.google.com/drive/folders/1a0U3j-fkxW1nUvYQiqi327xVMBkjRk-K?usp=sharing

Context:
The problem is solved.
I was using a sigmoid after the last layer. Removing it solved the problem.
Here is the architecture of the current working model:
CRNN(
  (cnn): Sequential(
    (conv0): Conv2d(1, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu0): ReLU()
    (pooling0): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu1): ReLU()
    (pooling1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (conv2): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (batchnorm2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu2): ReLU()
    (conv3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu3): ReLU()
    (pooling2): MaxPool2d(kernel_size=(2, 2), stride=(2, 1), padding=(0, 1), dilation=1, ceil_mode=False)
    (conv4): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (batchnorm4): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu4): ReLU()
    (conv5): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu5): ReLU()
    (pooling3): MaxPool2d(kernel_size=(2, 2), stride=(2, 1), padding=(0, 1), dilation=1, ceil_mode=False)
    (conv6): Conv2d(512, 512, kernel_size=(2, 2), stride=(1, 1))
    (batchnorm6): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu6): ReLU()
  )
  (lstm): Sequential(
    (0): BiLSTM(
      (rnn): LSTM(512, 256, bidirectional=True)
      (embedding): Linear(in_features=512, out_features=256, bias=True)
    )
    (1): BiLSTM(
      (rnn): LSTM(256, 256, bidirectional=True)
      (embedding): Linear(in_features=512, out_features=67, bias=True)
    )
  )
)


Question: i have trained a model on gpu but i want to deploy it on cpu… i got that error “AttributeError: ‘_DenseLayer’ object has no attribute ‘memory_efficient’” when i train densenet model but the inference works fine with any other pretrained model

Context: can you show the full traceback here? also are you saying densenet pretrained inference runs fine?


Question:
Hello everyone, I have just started learning Pytorch and i got a problem while trying to create dataloader from my customized datasets  which is contains 20 files “data_x.mat” stored in specific folder, I need to use them with data-loader, can anyone help me how to create this class to get iterable batches :
Mydata is

10 files each is dict {“Training_Patches”: shape(760,120,21,21)  /  “Label”: shape(1,760)}  stored in .mat file
10 files each is dict {“Testing_Patches”: shape(420,120,21,21)  /  “Label”: shape(1,420)}  stored in .mat file
Here is what i thing it should looks like, Any suggestion,ideas, help would be Appreciated

class Datasets(Dataset):
    def __init__(self):
        self.tensors = []
        self.labels = []
       for i  in range(len(data_i["Training_patches"])):
                    self.tensors.append(data_i["Training_patches"])
                    self.labels.append(data_i["labels"])

    def __getitem__(self, index):
        # return one item on the index
        return

    def __len__(self):
        # return the data length
        return

Context:
Would you like to load each file as one sample or is a whole batch saved in each file?
Were you able to load the data using scipy.io.loadmat?
Here is a dummy example for your dataset:

class MyDataset(Dataset):
    def __init__(self, mat_paths):
        self.paths = mat_paths

    def __getitem__(self, index):
        # Load .mat
        data = io.loadmat(self.paths[index])
        x = torch.from_numpy(data['Training_patches'])
        y = torch.from_numpy(data['labels'])

        return x, y

    def __len__(self):
        return len(self.paths)


When and how shall we use the “torch.nn.MultiMarginLoss”?”

Question:
Can I use the torch.nn. MultiMarginLoss instead of using the cross-entropy loss for a classification task?

Should I apply softmax function to the logits before I send the logits into the “MultiMarginLoss” to make the logits as normalized probobility?

Context:
crissallan:

Can I use the torch.nn. MultiMarginLoss instead of using the cross-entropy loss for a classification task?

Yes.

 crissallan:

Should I apply softmax function to the logits before I send the logits into the “MultiMarginLoss” to make the logits as normalized probobility?

I think typically not. Speaking of which: I would not call them “logits” for this reason but “score” or so because the former always has a “unnormalized log probabilities” ring to me, which is not the case here.
Best regards
Thomas


Stack expects each tensor to be equal size, but got [163, 256, 256] at entry 0 and [160, 256, 256] at entry 1

Question:
I am working with the OAI MRI dataset for knee osteoarthritis classification. Each one of 435 MRIs I got has to be classified to a grade. For each MRI in a folder, there are 160 2D images. I created this function to read the dataset:
def dicom2array(path):
  dicom = pydicom.read_file(path)
  data = dicom.pixel_array
  data = (data - np.min(data)) / (np.max(data) - np.min(data))
  data = cv2.resize(data, (256,256))
  return data

def load_3d_dicom_images(scan_id): #returns an object with shape (160,256,256)
  files = sorted(glob.glob(f"/content/drive/MyDrive/subjects/{scan_id}/*/*"))
  img = np.array([dicom2array(a) for a in files])
  return img

class MyCustomDataset(Dataset):
  def __init__(self, path = "/content/drive/MyDrive/subjects"):
    dataframe = pd.read_csv("/content/drive/MyDrive/KL_grade.csv", sep = ';')
    self.labels = {}
    id = list(dataframe["id"])
    grades = list(dataframe["grade"])
    for i,g in zip(id, grades):
      self.labels[str(i).zfill(5)] = g

    self.ids = [a.split("/")[-1] for a in sorted(glob.glob(f"/content/drive/MyDrive/subjects/" + "/*"))]

  def __len__(self):
    return len(self.ids)

  def __getitem__(self, idx):
    imgs = load_3d_dicom_images(self.ids[idx])
    label = self.labels[self.ids[idx]]

    return torch.tensor(imgs, dtype = torch.float32), torch.tensor(label, dtype = torch.long)

I tried to train the model with a resnet18 from the monai library. When I set the batch_size=10 or more cuda runs out of memory (I run it on google colab). It works with batch_size=1 but when I set it equal to 2 I get this error:
stack expects each tensor to be equal size, but got [163, 256, 256] at entry 0 and [160, 256, 256] at entry 1
Can anybody help me ?

Context:
Based on the error message some samples seem to have 160 channels while others have 163.
Due to this, the collate_fn cannot stack the samples to a single batch and raises the error.
Check the shape of each sample and make sure the channel dimension as well as the spatial size are the same.


Question:
Hello,
I have been trying to debug an issue where, when working with a dataset, my RAM is filling up quickly. It turns out this is caused by the transformations I am doing to the images, using transforms.
My code is very simple:
  for dir1 in os.listdir(img_folder):
      for file in os.listdir(os.path.join(img_folder, dir1)):
          image_path = os.path.join(img_folder, dir1,  file)
          with Image.open(image_path) as img_pil:
            normalize = transforms.Normalize(mean=mean,std=std)
            preprocess = transforms.Compose([
              transforms.Resize((img_size,img_size)),
              transforms.ToTensor(),
              normalize
            ])
            img_pil = preprocess(img_pil)

Without running the “preprocess code”, the memory is emptied correctly upon opening and closing images.
I have tried defining the normalize and preprocess function outside the loop, but memory was still accumulating.
Am I missing something? Is there a way to free up the memory that is being occupied by the transformation steps?
NB: same issue arises when using a dataloader. But I didn’t know what was causing it, that’s how I ended up here.
Thanks

Context:
You can use batch transforms outside of loading loop (it is probably much faster too).

 fnak:

img_pil = preprocess(img_pil)

I wonder if redefining img_pil inside with loop is causing this issue for the PIL library.