How can I use sklearn.inverse transform on my predictions

Question:
I am currently training a model in pytorch for regression purposes(the model consists of some dense layers).
I have used sklearn’s standard scaler on the data which consists of an image and the ground truth regression targets.
How can I use the scaler’s inverse transform on the predicted output from the model during evaluation stage.
The training code is shown below
for epoch in range(num_epochs):
  model.train()
  for batch_idx,(img,gt_pose) in enumerate(train_loader):
    img=img.to(device)
    gt_pose=gt_pose.to(device)
    pred_pose=model(img)
    train_loss=loss(pred_pose,gt_pose)
    train_loss=train_loss/accum_iter
    train_loss.backward()
    if((batch_idx+1)%accum_iter==0) or (batch_idx+1==len(train_loader)):
      optimizer.step()
      optimizer.zero_grad()

    print ('Epoch: %03d/%03d | Batch %04d/%04d | Training loss: %.4f'
                   %(epoch, num_epochs, batch_idx,
                     len(train_loader), train_loss))
    train_losses.append(train_loss.item())
  model.eval()
  for batch_idx,(img_val,gt_pose_val) in enumerate(val_loader):
    img_val=img_val.to(device)
    gt_pose_val=gt_pose.to(device)
    pred_pose_val=model(img_val)
    loss_val=loss(pred_pose_val,gt_pose_val)
    print ('Epoch: %03d/%03d | Batch %04d/%04d | Validation loss: %.4f'
                   %(epoch, num_epochs, batch_idx,
                     len(train_loader), loss_val))
    valid_losses.append(loss_val.item())

Model evaluation
sc=StandardScaler()
img=img.to(device)#same transformations applied on image as during training
torch.manual_seed(123)
with torch.no_grad():
  pred=model(img)
  pred=pred.detach().cpu().numpy()
#sc.inverse_transform(pred) gives an error

if i do  sc.inverse_transform(pred),i get an error of the form scalar needs to be fit on the data.

Context: You would need to fit the StandardScaler before using transform or inverse_transform, which is missing in the posted code snippet. Either reuse the original scaler object or store the sc.mean_ and sc.scale_ and undo the normalization manually.


How can I preprocess video data while preserving all frames?

Question:
I want to batch video data for action counting task. say, I want to get (Batch, Channel, Frame, Height, Width)
But video data frames are not same each other and It looks like not to possible to apply frame subsampling… because action counting task needs to fed all video frames!
Is there better way to batch video data preserving video frames??

Context:
To best of my knowledge, it is impossible.
However, if you construct them into a graph, please see PyG Documentation — pytorch_geometric 2.0.4 documentation


Significant difference in performance between Torchvision and Kornia image augmentations

Question:
I have trained two models that use the same sequence of image augmentations but in Torchvision and Kornia and I’m observing a significant difference in the performance of these models. I understand that despite fixing random seeds, these augmentations might still be different which might cause some difference in the test accuracy but on average, I assume that both of these models should end with similar accuracy which is not the case.
# PyTorch transformation
train_orig_transform = transforms.Compose([
    transforms.RandomResizedCrop(32),
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.RandomApply([transforms.ColorJitter(0.4, 0.4, 0.4, 0.1)], p=0.8),
    transforms.RandomGrayscale(p=0.2),
    transforms.ToTensor(),
    transforms.Normalize([0.4914, 0.4822, 0.4465], [0.2023, 0.1994, 0.2010])
    ])

This is the kornia version of the above code

class KorniaAugmentationModule(nn.Module):
    def __init__(self, batch_size=512):
        super().__init__()
        # These are standard values for CIFAR10
        self.mu = torch.Tensor([0.4914, 0.4822, 0.4465])
        self.sigma = torch.Tensor([0.2023, 0.1994, 0.2010])

        self.hor_flip_prob = 0.5
        self.jit_prob = 0.8
        self.gs_prob = 0.2

        self.crop = K.RandomResizedCrop(size=(32, 32), same_on_batch=False)
        self.hor_flip = K.RandomHorizontalFlip(p=self.hor_flip_prob, same_on_batch=False)
        self.jit = K.ColorJitter(brightness=0.4, contrast=0.4, saturation=0.4, hue=0.1, p=self.jit_prob, same_on_batch=False)
        self.rand_grayscale =  K.RandomGrayscale(p=self.gs_prob, same_on_batch=False)

        self.normalize = K.Normalize(self.mu, self.sigma)

    # Note that I should only normalize in test mode; no other type of augmentation should be performed

    def forward(self, x, params=None, mode='train'):
        B = x.shape[0]
        if mode == 'train':
                x = self.crop(x, params['crop_params'])
                x = self.hor_flip(x, params['hor_flip_params'])
                x[params['jit_batch_probs']] = self.jit(x[params['jit_batch_probs']], params['jit_params'])
                x = self.rand_grayscale(x, params['grayscale_params'])

        x = self.normalize(x)
        return x

This is the only difference between the two models. Rest of the code for training and testing is a common shared code.
These are the plots I obtain for training loss and testing accuracy of the models.
Plots for Training Loss curve and Test Accuracy - Kornia (orange) vs Torchvision (green)
Screen Shot 2020-09-27 at 6.56.35 PM1428×562 52.9 KB
The difference in test accuracies between the two models is nearly ~11% which is very significant.
Is this expected? Could you suggest ways to debug this issue?

Context:
Maybe it’s due to the difference of torchvision.transform.ToTensor and kornia.image_to_tensor (or any similar to_tensor)
torchvision’s ToTensor scale the pixels to [0, 1] range, and kornia does not and keeps the original pixel value (in [0,255]), and normal to_tensor methods don’t too. This means the normalization is with incorrect data, and network is fed with wrong data distribution, which may lead to performance problem


Question:
I made a simple example of a cnn layer where convolutional weights are defined as linear combination of predefined filters. The goal is to train the coefficients of linear combination while keeping predefined filters fixed. This can be easily achieved in tensorflow using tf.nn.conv2d. However here model.weight is not changing and model.weight.grad=None?
Can anyone please have a look? Thanks.
from __future__ import print_function
import torch.nn.functional as F
# from torch.autograd import Variable
# import copy
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

class ConvNet(nn.Module):
    def __init__(self, device='cuda'):

        super(ConvNet, self).__init__()
        self.device = device
        self.filter = torch.Tensor(
            [[[0.06,  0,      0],
              [0.1,   0,    0.2],
              [0.06,  0.1,   0]],
             [[0.1,   0,      0],
             [0.2,    0,      0],
             [0.1,    0,      0]]]).to(self.device)
        # print(self.filter.shape)
        self.weight = nn.Parameter(torch.Tensor(1, 1, 2).to(self.device))
        self.bias = nn.Parameter(torch.Tensor(1)).to(self.device)
        nn.init.xavier_uniform_(self.weight)
        self.bias.data.uniform_(-1, 1)
        self.kernel = nn.Parameter(
            torch.einsum("ijk, klm -> ijlm",
            self.weight, self.filter).to(self.device),
            requires_grad=False)
        # print(self.kernel.shape)
        self.bn1 = nn.BatchNorm2d(1)
        self.fc = nn.Linear(1*13*13, 10)

    def forward(self, x):
        out = F.conv2d(input=x, weight=self.kernel, bias=self.bias)
        out = self.bn1(out)
        out = nn.ReLU()(out)
        out = nn.MaxPool2d(kernel_size=2, stride=2)(out)
        # print("out.shape: ", out.shape)
        out = out.reshape(out.size(0), -1)
        # print("out.shape: ", out.shape)
        out = self.fc(out)
        return out

def main(argv=None):
    # Device configuration
    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
    # Hyper parameters
    num_epochs = 1
    batch_size = 1024
    learning_rate = 0.001
    log_interval = 10

    # MNIST dataset
    train_dataset = datasets.MNIST(
    root='../../data/',
    train=True,
    transform=transforms.ToTensor(),
    download=True)
    train_loader = torch.utils.data.DataLoader(
    dataset=train_dataset,
    batch_size=batch_size,
    shuffle=True)

    model = ConvNet(device=device)
    model.to(device)
    for name, param in model.named_parameters():
        print(name, '\t\t', param.shape)
    optimizer = optim.SGD(model.parameters(), lr=learning_rate)
    loss_fn = nn.CrossEntropyLoss()
    for epoch in range(1, num_epochs + 1):
        model.train()
        for batch_idx, (data, target) in enumerate(train_loader):
            data, target = data.to(device), target.to(device)
            optimizer.zero_grad()
            output = model(data)
            loss = loss_fn(output, target)
            # print(model.weight)
            # print(model.kernel)
            # print(model.weight.grad)
            a = list(model.parameters())[0].clone()
            loss.backward()
            optimizer.step()
            b = list(model.parameters())[0].clone()
            print(torch.equal(a.data, b.data))
            if batch_idx % log_interval == 0:
                _, predicted = torch.max(output.data, 1)
                total = target.size(0)
                correct = (predicted == target).sum().item()
                print('batch Accuracy: {} %'.format(100 * correct / total))

if __name__ == '__main__':
    main()

Context:
gist.github.com

    https://gist.github.com/InnovArul/ca0db578769186bbaf094c5f8416a8bc 146
filter_learning.py
from __future__ import print_function
import torch.nn.functional as F
# from torch.autograd import Variable
# import copy
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

This file has been truncated. show original

Remember that pytorch is based on dynamic computation graphs, where as tensorflow is based on static computation graphs. Hence, you need to calculate filter in forward() rather than __init__().

You do not need to move individual parameters to cuda. Rather try to move whole model to cuda.


PyTorch version of `skimage.draw.polygon`? coordinates to mask

Question:
Hello everyone.
I’ve been trying quite a bit to figure out a solution here.
I’d like to go from a set of points to a masked image representation of these points. (Filling the pixels within)
skimage.draw.polygon allows me to do just that.
Is there anyone that can assist with a PyTorch implementation?
Many thanks!

Context: The internal _polygon 1 method used in skimage seems to iterate the passed coordinates and appends the indices which are inside the polygon, so you could try to adapt this code in PyTorch and reuse it.


Cats and dogs PyTorch dataset for Lucidrain vision transformer

Question:
I am trying to run the cats and dogs example for vision transformer by lucidrain but the following Webpage from PyTorch doesn’t exist anymore. What is the link I should use for retrieving the train.zip and test.zip files?
https://pytorch.org/docs/stable/torchvision/datasets.html 1

Screen Shot 2022-01-21 at 5.09.43 PM1274×792 35.7 KB

This dataset is required for running the cell #9 here vit-pytorch/cats_and_dogs.ipynb at main · lucidrains/vit-pytorch · GitHub
The author has linked to the PyTorch website that I linked above in this issue: how to get train.zip. · Issue #13 · lucidrains/vit-pytorch · GitHub

Context:
The datasets docs can be found here: torchvision.datasets — Torchvision 0.11.0 documentation 1
You can generally use the search in the docs to check if pages were moved.


Dataloader with multipile image sizes - using collate_fn correctly

Question:
Hi,
I have a dataset with different image sizes, and I see the way to go is with custom collate_fn.
I’ve seen some good answers here, but it looks like I still need to fix my collate_fn.
I searched for the max height and width in the same batch, and add zeros to (right&top) when the size is smaller.
I get the correct sizes with my collate_fn, but there’s a problem and I get the following error:
inputs, labels = data

ValueError: too many values to unpack (expected 2)
A simple code is attached, so it’s very easy to reproduce (Still looks odd here…)

import os
import numpy as np
import torch
import timm

import torch.nn as nn
import torch.nn.functional as F
f>rom torch.optim import SGD,Adam,lr_scheduler

from torchvision import transforms,models
from torchvision.datasets import ImageFolder

def my_collate(batch):

max_h = 0
max_w = 0
for x, y in batch:
print('y: ',y )
h,w = x.shape[1], x.shape[2]
if h>max_h:
max_h=h
if w>max_w:
max_w = w

   new_batch = []
   for x, y in batch:
    if (x.shape[1] != max_h) or (x.shape[2] != max_w):
        print('Fix (like padding?)')
        h,w = x.shape[1], x.shape[2]
        pad_w = torch.zeros(3, h, max_w-w)
        new_x = torch.cat((x,pad_w),2) # adjust width
        new_h, new_w = new_x.shape[1], new_x.shape[2]
        pad_h = torch.zeros(3, max_h-h, new_w) # adjust height
        new_x = torch.cat((new_x,pad_h),1)
        new_batch.append((new_x,y))
    else:
        new_batch.append((x,y)) # max dimensions -- > just add to the new batch

print('done')
return new_batch

train_transform = transforms.Compose([
transforms.Resize((224)),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

dataset_path = ‘/home/diffrent_img_sizes_dataset’
train_dataset = ImageFolder(root=os.path.join(dataset_path , ‘train’), transform=train_transform)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=8, collate_fn=my_collate, num_workers=2, drop_last=True, shuffle=True)

model = timm.models.mobilenetv3_large_100(pretrained=True)
criterion = nn.CrossEntropyLoss()
optimizer = SGD(model.parameters(), lr=0.005, momentum=0.9)
lr_schedule = lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.9)

for epoch in range(1):
for i, data in enumerate(train_loader, 0):
inputs, labels = data
if torch.cuda.is_available():
inputs, labels = inputs.cuda(),labels.cuda()
optimizer.zero_grad()
outputs = model(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
out = torch.argmax(outputs.detach(),dim=1)
lr_schedule.step()
In the collate_fn, if I create x_new_list and y_list and use

return x_new_list, y_list

Then I still get an error (a different one)
How do I fix my collate_fn?
Is there a common way to tackle the different sizes rather than adding zeros this way?

Context: I succeeded to make it work! Now the collate function returns [ [N,C,H,W], [N]]- first is the images and second the labels. Not sure though it’s the best way to tackle that


Visualizing the outputs of kernels, instead of the outputs of filters

Question:
Hello
As I have seen, in many posts in the forum, peoples have explained about visualizing featuermaps(i.e., the outputs of the filters.)
As the mentioned featuremaps are the summation of all outputs of kernels, my question is how we can access to these featuremaps outputted from kernels, right before summing up to produce final featuremaps?
Thank you

Context:
Each “featuremap” is created by a single kernel. I.e. a conv layer using the setup nn.Conv2d(in_channels=3, out_channels=10, kernel_size=3), the output activation (featuremap) will have the shape [batch_size, out_channels=10, height, width]. The out_channels dimension corresponds to the number of filters/kernels in the layer (in the default setup) and each filter creates one activation map.

 hamedB:

right before summing up to produce final featuremaps?

I guess this refers to the applied operation of each filter on the input activation. If you want to split the channels of the filter and apply them separately, you could use the groups argument during the instantiation of the conv layer.


Question:
Hello. I got my trained model with a good segmentation result.
However, this is still little bit slow.
I am wondering, is there any way I can convert this model to another type for speed?
My model is Mobilenet v3 base small version model.
I see that there is half-precision or 16 float convert that can make the model faster than the pure 32 float model.
My GPU is titan V.
Thank you.

Context:
Hi
If you want to truncate/reduce precision the weights of the trained model, you can do
net = Model()
net.half()

which converts all FP32 tensor to FP16 tensor.


How can l use the pre-trained Resnet to extract feautres from my own dataset?

Question:
Hello,
l want to extract features of my own dataset from the last hidden layer of ResNet (before softmax).
l defined the following :
import torchvision.models as models
resnet152 = models.resnet152(pretrained=True,requires_grad=False)
modules=list(resnet152.children()[:-1])
resnet152=nn.Sequential(*modules)

Is it correct ?
What’s next ?
Thank you

Context:
Have you run the code? There are several mistakes. First, resnet152.children() is a generator, and hence is not subscriptable. Also, resnet152 doesn’t take a requires_grad argument. The following code will work:
import torch
import torch.nn as nn
import torchvision.models as models
from torch.autograd import Variable
resnet152 = models.resnet152(pretrained=True)
modules=list(resnet152.children())[:-1]
resnet152=nn.Sequential(*modules)
for p in resnet152.parameters():
    p.requires_grad = False

Of course, the outcome depends on what you want to achieve. This code returns a model consisting of all layers of resnet152 bar the last one (a fully connected layer), with fixed parameters


2D convolution with different kernel per location

Question:
Math generally is less ambiguous than words so let me just show some equations of what I’m trying to implement. (1) is standard 2D convolution with unit batch size, 3x3 kernel size, and unit stride, and (2) is what I want to implement.

image.png1191×383 16.4 KB

In equation (2), the kernel w is modulated via element-wise product by another tensor d whose values depend on both the spatial index (i, j) as well as the dummy “convolution index” (m, n). Note that the convolutional kernel w is learnable while the modulating tensor d is known. Another way to formulate this is to use d and w to form a new kernel of shape (3, 3, H, W, K_in, K_out) whose values depend not just on the convolution index (m, n) but also on the spatial index (i, j).
Is there a simple way to express this computation without rewriting the CUDA kernels for conv2d?

Context: Don’t know if it’s possible to use conv2d for this, but maybe play with fold/unfold ops to avoid writing cuda kernel: https://pytorch.org/docs/stable/_modules/torch/nn/modules/fold.html 72


Missing keys & unexpected keys in state_dict when loading self trained model

Question:
Hi, there:
I’ve encountered this problem and got stucked for a while. I have a labeled image dataset in a considerable large scale and I chose to train a vgg16 on it just starting from pytorch’s imagenet example 61.
I firstly organize data into three splits, namely train, val, test; under each of them are bunches of subdirectory organized by class labels, like:
train
  label0
  label1
  ...
val
  label0
  label1
  ...
test
  file0
  file1
  ...

and the command:
export CUDA_VISIBLE_DEVICES=device_id
python3 main.py /path/to/my/dataset -a vgg16 -b 32 --lr 0.001

and the training seems to be fine — with nearly 90% of top-5 accuracy. The model file name is model_best.pth.tar
After that I would like to infer some images using my model, it fails with the follow error:
RuntimeError: Error(s) in loading state_dict for VGG:
        Missing key(s) in state_dict: "features.0.weight", "features.0.bias", "features.2.weight", "features.2.bias", "features.5.weight", "features.5.bias", "features.7.weight", "features.7.bias", "features.10.weight", "features.10.bias", "features.12.weight", "features.12.bias", "features.14.weight", "features.14.bias", "features.17.weight", "features.17.bias", "features.19.weight", "features.19.bias", "features.21.weight", "features.21.bias", "features.24.weight", "features.24.bias", "features.26.weight", "features.26.bias", "features.28.weight", "features.28.bias".
        Unexpected key(s) in state_dict: "features.module.0.weight", "features.module.0.bias", "features.module.2.weight", "features.module.2.bias", "features.module.5.weight", "features.module.5.bias", "features.module.7.weight", "features.module.7.bias", "features.module.10.weight", "features.module.10.bias", "features.module.12.weight", "features.module.12.bias", "features.module.14.weight", "features.module.14.bias", "features.module.17.weight", "features.module.17.bias", "features.module.19.weight", "features.module.19.bias", "features.module.21.weight", "features.module.21.bias", "features.module.24.weight", "features.module.24.bias", "features.module.26.weight", "features.module.26.bias", "features.module.28.weight", "features.module.28.bias".

Could anyone give me some advice?
Thanks in advance.
EDIT: the loading snippet:
import torch
from torchvision import models
model = models.__dict__[args.arch]() # arch is fed as 'vgg16'
model.cuda()
checkpoint = torch.load(model_file_name) # ie, model_best.pth.tar
model.load_state_dict(checkpoint['state_dict'])

Context:
It seems you’ve used nn.DataParallel to save the model.
You could wrap your current model again in nn.DataParallel or just remove the .module keys.
Here 7.4k is a similar thread with some suggestions.


Opinion about Batch Preprocessing vs Real Preprocessing for images

Question:
Hello,
I’d like your opinion on some approaches for applying preprocessing to images for deep learning (eg: semantic segmentation).

Important note :

Take for account that this would be performed in a pipeline therefore every new training would re-apply this (consider the disk to be emptied between training for instance). Also take for granted that we have access to cpus and gpus.

Approach 1 : Batch Transforms, Real time data augmentation
This approach would essentially apply the transforms like resize, rescale (/255) and toTensor in a batch and save the transformed images on disk for training.
Then during training, only the data augmentation would be left to be applied (for the training set only).
TLDR: Transforms (images) => onDisk => Training + Data aug on the fly

Approach 2 : Batch Transforms, Batch data augmentation
This approach would apply the transforms like resize, rescale (/255),toTensor and the data augmentation for the training set then save it on disk for training.
Then during training, we’d use a dataset with transforms=None meaning no Transforms are applied, it would simply read the already processed images.
TLDR : Transforms(images) => Data Aug(images) => onDisk => Training

Approach 3 : Batch Data augmentation, Real time Training + Transforms
Same principle but for data aug (for training set)
Data augmentation (images) => onDisk => Training + Transforms on the fly

Approach 4 : Real time transforms, real time data augmentation
Here nothing is saved to disk besides the raw train/valid/test images.
Training + Transforms + Data augmentation on the fly.
Important things to keep in mind :

Keep in mind that we could decide to perform preprocessing on the cpu or gpu if we use any batch approach, this might or might not be advantageous.
We also might want to consider if having them on disk is useful to save it for reproducibility and lineage (in case we want to go back from a model in production for instance and see what it was trained on).
We also must keep in mind that if we perform the data aug on the fly then it is re-applied after each epoch therefore this might or might not improve model accuracy.

So what do you guys think ?

Context:
This sounds like a valid approach, as the data augmentation is applied on each sample during training and could potentially save some preprocessing time. In any case I would profile the image decoding (e.g. JPEG decoding) vs. loading the raw binary data (speed and file size). E.g. loading a 1200x1600 JPEG encoded file with a size of ~310kB results in a tensor of ~23MB, since raw pixels are stored if you are not resizing it.

This approach wouldn’t use any data augmentation and I would then claim it’s invalid. Even if you are applying it once, this could be seen as creating a “new” dataset (with flipped, cropped etc. images) which won’t be randomly transformed during training.

Same as 1. with the difference to store the augmented images on disk, which I believe shouldn’t yield any advantage.

Standard approach, so valid.


How to reduce latency in ConvTranspose2d?

Question:
Question: How to reduce latency(computational cost) in ConvTranspose2d?
I compared latency between nn.Conv2d, Upsample(mode=nearest), ConvTranspose2d, Upsample(mode=bilinear) in different batch size. It looks like deconvolution operation takes a lot of time in large batch (=64). Is there a way to reduce computation time for deconvolution?
Thank you
Comparison Table
[------------------------------------- upsample module comparison ------------------------------------]
                        |  nearest(scale=2)  |  conv(kernel=3)  |  deconv(scale=2)  |  bilinear(scale=2)
4 threads: --------------------------------------------------------------------------------------------
      [8, 2048, 8, 8]   |        30.4        |       94.6      |        134.6      |        4838.2
      [16, 2048, 8, 8]  |        58.4        |       94.3      |        249.4      |        9682.3
      [32, 2048, 8, 8]  |       111.6        |      119.1      |        480.1      |       19434.2
      [64, 2048, 8, 8]  |       219.9        |      206.1      |      12399.6      |       38844.8

Times are in microseconds (us).

Comparison Code
import os

os.environ['CUDA_DEVICE_ORDER'] = 'PCI_BUS_ID'
os.environ['CUDA_VISIBLE_DEVICES'] = '4'

import torch
import torch.nn as nn
import torch.utils.benchmark as benchmark
import torchvision.models as models
from itertools import product

def get_upsample_module(mode, upsample, ch):
    if mode == 'deconv':
        return nn.Sequential(
            torch.nn.ConvTranspose2d(ch, ch, upsample, stride=upsample, dilation=1, groups=ch, bias=False),
            torch.nn.BatchNorm2d(ch)
        ).cuda()
    else:
        return nn.Upsample(scale_factor=upsample, mode=mode).cuda()

def get_downsample_module(kernel, ch):
    return nn.Sequential(
        torch.nn.Conv2d(ch, ch, kernel, stride=1, padding=1, dilation=1, groups=ch, bias=False),
        torch.nn.BatchNorm2d(ch)
    ).cuda()

results = []

batch_size = [8, 16, 32, 64]
channel_size = [2048]
image_size = [8]
mode = ['nearest', 'conv', 'deconv', 'bilinear']
scale_factors = [2]

for b, c, n in product(batch_size, channel_size, image_size):
    label = 'upsample module comparison'
    sub_label = f'[{b}, {c}, {n}, {n}]'
    x = torch.rand((b, c, n, n)).cuda()

    for method, upsample in product(mode, scale_factors):
        if method == 'conv':
            upsample += 1
            model = get_downsample_module(upsample, c)
        else:
            model = get_upsample_module(method, upsample, c)
        with torch.cuda.amp.autocast():
            results.append(benchmark.Timer(
                stmt='model(x)',
                setup='from __main__ import model',
                globals={'x': x},
                label=label,
                sub_label=sub_label,
                description=f"{method}(scale={upsample})",
                num_threads=4
            ).blocked_autorange(min_run_time=1))

compare = benchmark.Compare(results)
compare.colorize()
compare.print()

My Environment:

OS: Linux 18.04
Python: 3.7.11
Pytorch: 1.10.0
CUDA: 11.3

Context: You could add torch.backends.cudnn.benchmark = True to let cuDNN profile different kernels and select the fastest one and use to(memory_format=torch.channels_last) since you are using amp, which could give another speedup if permutations are avoided.


Question:
Dear all,
I’m currently working on a resnet18 backbone + convTranspose2D to upsample the image output. Here is the entire model:
Sequential(
(0): Sequential(
(0): Sequential(
(0): Sequential(
(0): Sequential(
(0): Sequential(
(0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
(3): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
(4): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(1): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(5): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(6): Sequential(
(0): BasicBlock(
(conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(7): Sequential(
(0): BasicBlock(
(conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
)
(1): ConvTranspose2d(512, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
)
(1): ConvTranspose2d(256, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
)
(1): ConvTranspose2d(128, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
)
(1): ConvTranspose2d(64, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
)
(1): ConvTranspose2d(64, 5, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
)
The problem is the lost function is converging very slowly :
tensor(1.5996)
tensor(1.5947)
tensor(1.5921)
tensor(1.5919)
tensor(1.5911)
tensor(1.5900)
tensor(1.5896)
tensor(1.5892)
tensor(1.5889)
tensor(1.5887)
I tried various learning rates (10^-2,10^-3,10^-4)and different optimizer (Adam, SGD) and obtained the same result. I’m using crossentropyloss for multi-classification problem. Changing the learning rate doesn’t really impact anything with make me skeptical about my model. I also tried to use a single image (and therefore a batchsize of 1) as training in order to reduce the complexity but same result.
Thanks

Context: As conv2d does, convtranspose2d usually needs batchnorm and activation.


Question:
I have this code and I spent a lot of time figuring out why the size is still the same [28,28]:
class FashionDataset(Dataset):
    def __init__(self,dataset_dir,train_option,transform=None):
        self.dataset = datasets.FashionMNIST(root=dataset_dir,
                                           train=train_option,
                                           download=True)
        self.transform = transform
        self.labels = self.dataset.targets

    def __len__(self):
        return len(self.dataset)

    def __getitem__(self, index):
        data, target = self.dataset[index]
        data = self.transform(data)
        return index, data, target

transform = transforms.Compose([transforms.ToTensor(),transforms.Resize((224, 224))])

train_dataset = FashionDataset(dataset_dir = dataset_dir, train_option = True, transform = transform)

test_dataset = FashionDataset(dataset_dir = dataset_dir, train_option = False, transform = transform)

train_loader = DataLoader(train_dataset,
                          batch_size=batch_size,
                          sampler=SubsetRandomSampler(range(len(train_dataset))))

test_loader = DataLoader(test_dataset,
                         batch_size=batch_size,
                         shuffle=False)

Thank you for your help in advance.

Context:
Your code works as expected:
idx, x, y = next(iter(train_loader))
print(x.shape)
# > torch.Size([10, 1, 224, 224])

idx, x, y = next(iter(test_loader))
print(x.shape)
# > torch.Size([10, 1, 224, 224])


How to get the scores from torchvision fasterRCNN based on an image and bounding box coordinates

Question:
Hi,
I am using  torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True) for object detection. Now this method gives me the scores as one of the outputs. Now i have the co-ordinates of the bounding box, so i want to get the scores based on that image and the bounding box.
For example, i want to do something like this =>
num_classes = 2
in_features = model.roi_heads.box_predictor.cls_score.in_features

model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
chkpt_path = "/chkpt_path"
model.load_state_dict(torch.load(chkpt_path))
model.to(device)
images, target = iter(val_data_loaders).next()
images = list(image.to(device, dtype=torch.float) for image in images)
[outputs = model(images)] # replace this with the following
outputs = model(images, precomputed_bbox)

Please suggest me how to do this? I want the score from this

Context:
I’m not sure that I clearly understand your problem.
You want to get the scores of your custom bboxes from the image, right?
I think it is hard to get because inside the model, there are several modules that generate anchors and cutout the boxes which has a score under threshold and NMS…


Question:
Hello everybody,
I want to translate and rotate a map matrix using the affine_grid and grid_sample method but want to avoid using a full 2d tensor because of very large RAM demand. My data is quite sparse, therefore I reduce the size by only keeping coordinates where values actually exist and reconstruct the map when needed. However I am struggeling to see whether there is any way to use/configure the grid_sample function to transform sparse matrices of this format:
x       y         value
[[4],   [5]],     [0.12],
[[4],   [10]],   [0.35],
[[24], [12]],   [0.84],
[[14], [595]], [0.55],
…
Does anybody know whether this won’t be possible using sample_grid? The values don’t need to be included at all costs either. Thank you!

Context:
Hi,
I have exactly the same problem. Did you find a solution?
Thanks,
Thibault


Question: ^^What is the best way to perform hyper parameter search in PyTorch? Are there frameworks that can ease this process?

Context:
@kevinzakka has implemented hypersearch 2.3k.
There are still some TODOs, so alternatively you could have a look at Skorch 3.0k which allows you to use the scikit-learn grid search / random search.


Question:
I am receiving this error while trying to run my script.
Can anyone help?
Traceback (most recent call last):
File “train.py”, line 724, in
run_training(model, trainCases, epoch, lp, max_image_shape)
File “train.py”, line 343, in run_training
output = model(mris_batch_tensor)
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 493, in call
result = self.forward(*input, **kwargs)
File “/content/drive/My Drive/MRI to SCT/models.py”, line 65, in forward
upconcat2_feat = self.upconcat2(conv_up1_2_3_feat, conv_down5_6_7_feat)
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 493, in call
result = self.forward(*input, **kwargs)
File “/content/drive/My Drive/MRI to SCT/models.py”, line 211, in forward
x_conv = self.W_x(x)
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 493, in call
result = self.forward(*input, **kwargs)
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/container.py”, line 92, in forward
input = module(input)
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 493, in call
result = self.forward(*input, **kwargs)
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py”, line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size 256 512 1 1, expected input[8, 256, 32, 32] to have 512 channels, but got 256 channels instead

Context: Please let me know your self.W_x and dimension of x


Modules defined in __init__() but not used in forward() still affect the results

Question:
I have defined a channel attention block in the init, but didn’t use it in the forward function.  However, the network performance still improved a lot. I’m confused. Any suggestions? Thanks!
class ImgFeatMSNet2(nn.Module):
    '''
    image and feature space multi-scale deconvolution network: multiscale image do Wiener separately and merge in refine net
    '''

    def __init__(self, n_colors, channel_atten=True):
        super(ImgFeatMSNet2, self).__init__()
        # params
        self.channel_atten = channel_atten
        # self.n_colors = n_colors
        n_resblock = 3
        n_feats1 = 16
        n_feats = 32
        kernel_size = 5
        padding = 2

        self.FeatModule1 = FeatModule(
            n_in=n_colors, n_feats=n_feats1, kernel_size=kernel_size, padding=padding, act=True)

        self.InBlock1 = InBlock(n_in=n_feats1 + n_colors, n_feats=n_feats,
                                kernel_size=kernel_size, padding=padding, act=True)

        if channel_atten:
            self.ChannelAttention1 = ChannelAttention(
                in_planes=n_feats1+n_colors, ratio=2)
            self.ChannelAttention2 = ChannelAttention(
                in_planes=n_feats1 + n_feats+n_colors, ratio=4)

        self.RefineUnet = RefineUnet(
            n_feats=n_feats, n_resblock=n_resblock, kernel_size=kernel_size, padding=padding, act=True)

        self.OutBlock_a = OutBlock_a(
            n_feats=n_feats, n_resblock=n_resblock, kernel_size=kernel_size, padding=padding)
        self.OutBlock_b = OutBlock_b(
            n_feats=n_feats, n_out=n_colors, kernel_size=kernel_size, padding=padding)

    def forward(self, input):
        # scale1: x0.5; scale1: x1.0
        n, c, h, w = input.shape

        feat_2 = self.FeatModule1(input)  # get image feature
        wd_1 = torch.cat((feat_2, input), 1)  # cat image and feature

        # The following codes are not used
        # if self.channel_atten:
        #     wd_1 = self.ChannelAttention1(wd_1)

        in_1 = self.InBlock1(wd_1)
        refine_1 = self.RefineUnet(in_1, int(round(h/2)), int(round(w/2)))
        out_a1 = self.OutBlock_a(refine_1)
        out_b1 = self.OutBlock_b(out_a1)

        return out_b1

Context: Creating an additional layer would initialize all parameters (assuming this layer contains parameters) by sampling random numbers through the pseudorandom number generator (PRNG). This would affect all future calls of the PRNG and would not yield the same results even if the additional layer is not used in the forward. Changing the seed would have a similar effect and if your model is sensitive to the seed it would mean that the overall training is not particularly stable.


Unet training Error: The size of tensor a (16) must match the size of tensor b (6) at non-singleton dimension 1

Question:
I’m trying to train a Unit model on LandCoverNet dataset, which is a satellite imagery dataset that contains input images and corresponding land cover type masks.
I have created a custom dataset to get my images and masks:
# Create custom dataset that accepts 4 channels images
from torch.utils.data import Dataset, DataLoader, sampler
from pathlib import Path
from PIL import Image
import matplotlib.pyplot as plt
import os
import numpy as np
import rasterio as rio
from torchvision import transforms, datasets, models
# We have two dir: inputs(folder for each image) and tatgets

class LandCoverNetDataset(BaseDataset):

  CLASSES = ['otherland', 'cropland', 'pastureland', 'bare soil', 'openwater', 'forestland']

  def __init__(self, inputs_dir, targets_dir,
               classes = None,
               augmentation=None ,
               preprocessing = False,
               pytorch=True):

    super().__init__()
    self.samples = []
    self.pytorch = pytorch
    self.augmentation = augmentation
    self.preprocessing = preprocessing

    # Convert str names to class values on masks
    self.class_value = [self.CLASSES.index(cls.lower()) for cls in classes]

    # Create dictionary for images and targets
    for sub_dir in os.listdir(inputs_dir):
      files = {}
      files = {
         'img_bands' : os.path.join(inputs_dir, sub_dir),
          'target' : os.path.join(targets_dir, sub_dir[:13] + "_LC_10m.png")
        }
      self.samples.append(files)

  def __len__(self):
    return len(self.samples)

  def normalize(self, band):

    '''Notmalize a numpy array to have values between 0 and 1'''
    band_min, band_max = band.min(), band.max()
    np.seterr(divide='ignore', invalid='ignore')
    normalized_band = ((band - band_min)/(band_max - band_min))
    #Remove any nan value and subtitute by zero
    where_are_NaNs = isnan(normalized_band)
    normalized_band[where_are_NaNs] = 0
    return normalized_band

  def open_as_array(self, idx, include_ndvi = False):
    '''
      Merge the 4 bands into one image and normalize the bands
    '''
    # List indivisual bands in each image folder
    # Stack them togather
    list_bands = []
    for img_file in os.listdir(self.samples[idx]['img_bands']):
      # Get the ndvi band
      if 'NDVI' in img_file:
        ndvi_band = os.path.join(self.samples[idx]['img_bands'], img_file)
      else:
        # Get the rgb bands
        band = rio.open(os.path.join(self.samples[idx]['img_bands'], img_file)).read(1)

        if self.preprocessing:
          # preprocess the bands before stacking them (only rgb)
          band = self.normalize(band)
        list_bands.append(band)

    # Stack the bands
    raw_rgb = np.stack(list_bands, axis=2).astype('float32')

    if include_ndvi:
      # Include the NDVI band in the input images
      ndvi = np.expand_dims(rio.open(ndvi_band).read(1).astype('float32'), 2)
      raw_rgb = np.concatenate([raw_rgb, ndvi], axis=2)

    if self.augmentation:
      transformed = self.augmentation(image = raw_rgb)
      raw_rgb  = transformed["image"]

    if self.preprocessing:
      # transpose to tensor shape
      raw_rgb = raw_rgb.transpose((2,0,1)).astype('float32')

    return raw_rgb

  def open_mask(self, idx):
    # Extract certain classes from mask

    mask = cv2.imread(self.samples[idx]['target'], 0)
    masks = [(mask == v) for v in self.class_value]
    mask = np.stack(masks, axis=-1).astype('long')

    if self.augmentation:
      transformed = self.augmentation(image = mask)
      mask  = transformed["image"]

    if self.preprocessing:
      # preprocess the mask
      mask = self.normalize(mask)
      # transpose to tensor shape
      mask = mask.transpose((2, 0, 1)).astype('long')
      mask = mask[0, :, :]
    return mask

  def __getitem__(self, idx):
    x = torch.tensor(self.open_as_array(idx, include_ndvi=True), dtype=torch.float)
    y = torch.tensor(self.open_mask(idx), dtype=torch.long)

    return x, y

  def open_as_pil(self, idx):
    arr = 256*self.open_as_array(idx)
    return Image.fromarray(arr.astype(np.uint8), 'RGB')

  def __repr__(self):
    s = 'Dataset class with {} files'.format(self.__len__())
    return s

The input here is 4 bands.
This is the shape of the first batch for both input/target
torch.Size([16, 4, 224, 224])
torch.Size([16, 224, 224])

I’m using a model from segmentation-models-pytorch library, and here is how I customized it for my case:

ENCODER = 'se_resnext50_32x4d'
ENCODER_WEIGHTS = 'imagenet'
ACTIVATION = 'softmax2d'
DEVICE = 'cuda'

model = smp.FPN(ENCODER, classes=len(CLASSES), activation=ACTIVATION)

# Replace the model.conv1 to accept 4 channels
# first: copy the layer's weights
weight = model.encoder.layer0.conv1.weight.clone()
model.encoder.layer0.conv1 = nn.Conv2d(4, 64,kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
with torch.no_grad():
  model.encoder.layer0.conv1.weight[:, :3] = weight
  model.encoder.layer0.conv1.weight[:, 3] = model.encoder.layer0.conv1.weight[:, 0]

loss = smp.utils.losses.NLLLoss()
metrics = [
    smp.utils.metrics.IoU(threshold=0.5),
]
optimizer = torch.optim.SGD([
    dict(params=model.parameters(), lr=0.001, weight_decay=1e-8, momentum=0.9),
])

# create epoch runners
# it is a simple loop of iterating over dataloader`s samples
train_epoch = smp.utils.train.TrainEpoch(
    model,
    loss=loss,
    metrics=metrics,
    optimizer=optimizer,
    device=DEVICE,
    verbose=True,
)

valid_epoch = smp.utils.train.ValidEpoch(
    model,
    loss=loss,
    metrics=metrics,
    device=DEVICE,
    verbose=True,
)

# train model for 40 epochs

And here is my training loop

# train model for 40 epochs

max_score = 0

for i in range(0, 40):

    print('\nEpoch: {}'.format(i))
    train_logs = train_epoch.run(train_loader)
    valid_logs = valid_epoch.run(valid_loader)

    # do something (save model, change lr, etc.)
    if max_score < valid_logs['iou_score']:
        max_score = valid_logs['iou_score']
        torch.save(model, './best_model.pth')
        print('Model saved!')

    if i == 25:
        optimizer.param_groups[0]['lr'] = 1e-5
        print('Decrease decoder learning rate to 1e-5!')

At first, the target shape was [16, 6, 224, 224] but I had an error and found this thread that it should be [batch_size, height, width]
That’s why I added this line in the Dataset class : mask = mask[0, :, :]
to get ride of the number of classes dim, and here where things get confusing for me, because the output of me model is torch.Size([10, 6, 224, 224]).
This is the entire error message:
Epoch: 0
train:   0%|          | 0/157 [00:00<?, ?it/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-215-2ae39e205dee> in <module>()
      7
      8     print('\nEpoch: {}'.format(i))
----> 9     train_logs = train_epoch.run(train_loader)
     10     valid_logs = valid_epoch.run(valid_loader)
     11

3 frames
/usr/local/lib/python3.6/dist-packages/segmentation_models_pytorch/utils/functional.py in iou(pr, gt, eps, threshold, ignore_channels)
     32     pr, gt = _take_channels(pr, gt, ignore_channels=ignore_channels)
     33
---> 34     intersection = torch.sum(gt * pr)
     35     union = torch.sum(gt) + torch.sum(pr) - intersection + eps
     36     return (intersection + eps) / union

RuntimeError: The size of tensor a (16) must match the size of tensor b (6) at non-singleton dimension 1

I have been working on this for days now and would appreciate any help!
Thanks a lot!

Context:
Rasha_Salim:

That’s why I added this line in the Dataset class : mask = mask[0, :, :]

I understand that it is a 6-class segmentation problem?
I am not sure what the initial target shape [16, 6, 224, 224] means. Is it [batch size, num classes (one hot), H, W]?
If the target was one hot, you should do mask = torch.argmax(mask, dim=1).

 Rasha_Salim:

here where things get confusing for me, because the output of me model is torch.Size([10, 6, 224, 224]).

This is confusing now. If the batch size is 16, the output of the model should have been [16, 6,224,224].


RuntimeError: mat1 dim 1 must match mat2 dim 0 for Sequential of DINO model

Question:
I’m getting the runtime error when trying to Sequentialise my linear layer with a DINO backbone from torch.hub.
import torch
import torch.distributed as dist

class LinearClassifier(torch.nn.Module):
    def __init__(self, dim, num_labels=1000):
        super(LinearClassifier, self).__init__()
        self.num_labels = num_labels
        self.linear = torch.nn.Linear(dim, num_labels)
        self.linear.weight.data.normal_(mean=0.0, std=0.01)
        self.linear.bias.data.zero_()

    def forward(self, x):
        # flatten
        x = x.view(x.size(0), -1)

        # linear layer
        return self.linear(x)

dist.init_process_group('gloo', init_method='file:///tmp/somefile', rank=0, world_size=1)
# load backbone
model = torch.hub.load('facebookresearch/dino:main', 'dino_vits8')

#Setup linear layer
linear_classifier = LinearClassifier(1536, 1000)
linear_classifier = linear_classifier.cuda()
linear_classifier = torch.nn.parallel.DistributedDataParallel(linear_classifier)
state_dict = torch.hub.load_state_dict_from_url(url="https://dl.fbaipublicfiles.com/dino/dino_deitsmall8_pretrain/dino_deitsmall8_linearweights.pth")['state_dict']
linear_classifier.load_state_dict(state_dict, strict=True)

#Sequentialise
model = torch.nn.Sequential(model,
                            linear_classifier)

x = torch.ones((1, 3, 224, 224))
out = model(x)
print("out: " + out)

This is what the print of the final layers looks like:

image1084×1240 44.7 KB

Context:
@Tizian_Th
The output of the Dino model is a [1,384] tensor, so if you want to connect two sequential layers, the output of the previous should have the same dimension as the input of the next. If you substitute
linear_classifier = LinearClassifier(384, 1000)

It should work


Accessing ImageFolder() imgs after using ConcatDataset()

Question:
Hello,
I have a question about how I can access the images from the call to ImageFolder() after feeding it into ConcatDataset(). This does seem to execute fine, but the ConcatDataset object doesn’t have an attribute for imgs that ImageFolder() has.
Here is the code I am using to setup the training data, the commented line is what I originally used. Now I am trying to take two datasets of images.
size = (224, 224)
data_transform = ImageTransform(size)
# train_data = torchvision.datasets.ImageFolder(root=train_dir, transform=data_transform.train_transform)

train_dataA = torchvision.datasets.ImageFolder(root=train_dirA, transform=data_transform.train_transform)
train_dataB = torchvision.datasets.ImageFolder(root=train_dirB, transform=data_transform.train_transform)
train_data = ConcatDataset([train_dataA, train_dataB])

val_data = torchvision.datasets.ImageFolder(root=val_dir, transform=data_transform.val_transform)
train_loader = torch.utils.data.DataLoader(train_data,
                                          batch_size=batch_size,
                                          shuffle=True)
val_loader = torch.utils.data.DataLoader(val_data,
                                          batch_size=batch_size,
                                          shuffle=True)
dataloaders_dict = {'train': train_loader, 'val': val_loader}

In another block I try and access the train_data.imgs, but it gives me an error of
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-30-e1395bae7815> in <module>()
----> 1 weights = make_weights_for_balanced_classes(train_data.imgs, carClasses) #len(train_data.classes)
      2 weights[1] /= 2
      3 weights = torch.FloatTensor(weights)
      4 print(weights)

AttributeError: 'ConcatDataset' object has no attribute 'imgs'

Thanks in advance!

Context:
My example code was accessing a random attribute to show that you would need to use the .datasets[index] approach to use it after wrapping the datasets into the ConcatDataset.
In your current code snippet you are passing the shape of the underlying tensors to the method, which sounds wrong so just manipulate your code from:
train_data.imgs

to
train_data.datasets[index].imgs


ResNet, Bottleneck, Layers, groups, width_per_group

Question:
I was on the web looking for how to implement ResNet in Pytorch, and found this code, but I have couple of days looking for information on the web to explain how it works and I have not found anything, that is why I hopw to find information here of the parameters what are they and how  they are used, or documentations.
Thanks.
import torch.nn as nn
import torchvision.models as models

class MyResNeXt(models.resnet.ResNet):
    def __init__(self, training=True):
        super(MyResNeXt, self).__init__(block=models.resnet.Bottleneck,
                                        layers=[3, 4, 6, 3],
                                        groups=32,
                                        width_per_group=4)
        self.fc = nn.Linear(2048, 1)

why do we init the ResNet with the Bottleneck, any docs?
2.- layers [3, 4, 6, 3] this depends on the type of resnet? 18, 50, 101? etc
3.- groups?
4.- width per group, can we use different sizes? 8, 16, 32, etc?
5.- self.fc = nn.Linear(2048, 1) is is binary, but why not 2 ? in Keras you can use 2 as the final output

Context:
The sublocks of the resnet architecture can be defined as BasicBlock or Bottleneck based on the used resnet depth. E.g. resnet18 and resnet32 31 use BasicBlock, while resnet>=50 24 use Bottleneck.

Yes. Your mentioned configuration would fit resnet34 and resnet50 as seen here 56.

Bottleneck layers support the groups argument to create grouped convolutions. (line of code 100)

Again, a ResNeXt-specific setup for the Bottleneck layer. You could try different values, but would most likely have to look into the paper, how these values would interact with the channels etc.

You could treat your binary classification use case as a 2 class multi-class classification use case, if you set the number of output features to 2.


Trying to understand the input shape convention and notation

Question:
I am confused with the input shape convention that is used in Pytorch in some cases:

The nn.Layer’s input is of shape (N,∗,H_in) where N is the batch size, H_in is the number of features and ∗ means “any number of additional dimensions”. What exactly are these additional dimensions and how the nn.Linear is applied on them?

The nn.Conv1d’s input is of shape (N, C_in, L) where N is the batch size as before, C_in the number of input channels, L is the length of signal sequence.

The nn.Conv2d’s input is of shape (N, C_in, H, W) where N is the batch size as before, C_in the number of input channels, H is the height and W the width of the image.

The nn.BatchNorm1d’s input is of shape (N, C) or (N, C, L) where N is the batch size as before. However what does the C and L denote here? It seems that C = number of features, L = number of channels, based on the description in documentation “2D or 3D input (a mini-batch of 1D inputs with optional additional channel dimension)”. This is inconsistent with the nn.Conv1d notation.

The nn.BatchNorm2d’s input is of shape (N, C, H, W) where N is the batch size as before, H and W are the height and width of the image respectively. What does the C denote here? Is it the number of features as in nn.BatchNorm1d or the number of channels as in nn.Conv2d? It seems to be the number of channels since we are talking about “a 4D input (a mini-batch of 2D inputs with additional channel dimension)”, but then in documentation we have the line “num_features – C from an expected input of size (N, C, H, W)”, so C is both number of channels and number of features which is weird. So perhaps num_features should be renamed to num_channels.

Context:
Any number of dimensions are supported and the linear layer would use them as if you would loop through these dimensions as seen here:

lin = nn.Linear(10, 10)
x = torch.randn(2, 10, 10)
out = lin(x)
out2 = torch.stack([lin(x_.squeeze(1)) for x_ in x.split(1, dim=1)], dim=1)
print((out - out2).abs().max())
# > tensor(0., grad_fn=<MaxBackward1>)

and 3. yes, that’s correct.

nn.BatchNorm1d accepts inputs in [N, C, L], where C is optional. C is the channel dimension and num_features in the batchnorm layers will use either C or L depending on the input shape as described in the docs:

num_features – C from an expected input of size (N,C,L) or L from input of size (N,L)

The part in the Shape section is inconsistent, as it mentions the inputs as (N, C) or (N, C, L). A PR is welcome in case you want to fix it.

C denotes the channel dimension and thus the features in the batchnorm layer.

 denexo:

so C is both number of channels and number of features which is weird.

Why is this weird? This dimensions is used as the channel dimension in the input and named features in the normalization layer.

 denexo:

So perhaps num_features should be renamed to num_channels.

This might be a possibility for batchnorm, but wouldn’t match other norm layers, which do not necessarily use the channel dimension only.


Getting the “TypeError: ‘int’ object is not subscriptable” error while constructing a basic LSTM network?

Question:
Hey everyone,
I’m getting the error below while constructing a basic LSTM network for research purposes (which is based on the source code available in this GitHub repository 1). Could you please help me to find out the issue?
p.s. I’m utilizing the Google Colab platform.
TypeError Traceback (most recent call last)

[<ipython-input-10-8883ef0b1eb5>](https://iyhna5ktvk-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20200708-085602-RC00_320186311#) in <module>() 195 196 if __name__ == '__main__': --> 197 main()

6 frames

[/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py](https://iyhna5ktvk-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20200708-085602-RC00_320186311#) in check_forward_args(self, input, hidden, batch_sizes) 520 expected_hidden_size = self.get_expected_hidden_size(input, batch_sizes) 521 --> 522 self.check_hidden_size(hidden[0], expected_hidden_size, 523 'Expected hidden[0] size {}, got {}') 524 self.check_hidden_size(hidden[1], expected_hidden_size,

TypeError: 'int' object is not subscriptable

import os
import random
import time

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import Dataset
from torchtext import data
from torchtext.datasets import IMDB

def epoch_time(start_time, end_time):
    elapsed_time = end_time - start_time
    elapsed_mins = int(elapsed_time / 60)
    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))
    return elapsed_mins, elapsed_secs

class LSTM_IMDB(nn.Module):
    def __init__(self, input_dim, embedding_dim, hidden_dim, output_dim):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.embedding = nn.Embedding(input_dim, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, text):
        embedded = self.embedding(text)

        lstm_out, hidden=self.lstm(embedded, self.hidden_dim)

        #stack up the lstm output
        lstm_out=lstm_out.contiguous().view(-1, hidden)

        return self.fc(lstm_out)

def binary_accuracy(preds, y):
    """
    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8
    """

    # round predictions to the closest integer
    rounded_preds = torch.round(torch.sigmoid(preds))
    correct = (rounded_preds == y).float()  # convert into float for division
    acc = correct.sum() / len(correct)
    return acc

def train(model, iterator, optimizer, criterion):
    epoch_loss = 0
    epoch_acc = 0

    model.train()

    for batch in iterator:
        optimizer.zero_grad()

        predictions = model(batch.text).squeeze(1)

        loss = criterion(predictions, batch.label)

        acc = binary_accuracy(predictions, batch.label)

        loss.backward()

        optimizer.step()

        epoch_loss += loss.item()
        epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

def evaluate(model, iterator, criterion):
    epoch_loss = 0
    epoch_acc = 0

    model.eval()

    with torch.no_grad():
        for batch in iterator:
            predictions = model(batch.text).squeeze(1)

            loss = criterion(predictions, batch.label)

            acc = binary_accuracy(predictions, batch.label)

            epoch_loss += loss.item()
            epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

def main():
    SEED = 1234
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    torch.manual_seed(SEED)
    torch.backends.cudnn.deterministic = True

    TEXT = data.Field(tokenize='spacy')
    LABEL = data.LabelField(dtype=torch.float)

    train_data, test_data = IMDB.splits(TEXT, LABEL)
    train_data, valid_data = train_data.split(split_ratio=0.5, random_state=random.seed(SEED))

    print(f'Number of training examples: {len(train_data)}')
    print(f'Number of validation examples: {len(valid_data)}')
    print(f'Number of testing examples: {len(test_data)}')

    MAX_VOCAB_SIZE = 25_000
    TEXT.build_vocab(train_data, max_size=MAX_VOCAB_SIZE)
    LABEL.build_vocab(train_data)

    BATCH_SIZE = 64
    train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(
        (train_data, valid_data, test_data),
        batch_size=BATCH_SIZE,
        device=device)

    INPUT_DIM = len(TEXT.vocab)
    EMBEDDING_DIM = 100
    HIDDEN_DIM = 256
    OUTPUT_DIM = 1

    model = LSTM_IMDB(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM)
    model.to(device)

    lr = 1e-2
    criterion = nn.BCELoss()
    criterion = criterion.to(device)
    optimizer = optim.SGD(model.parameters(), lr=lr, momentum=0.9)

    N_EPOCHS = 5

    best_valid_loss = float('inf')

    for epoch in range(N_EPOCHS):

        start_time = time.time()

        train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
        valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)

        end_time = time.time()

        epoch_mins, epoch_secs = epoch_time(start_time, end_time)

        if valid_loss < best_valid_loss:
            best_valid_loss = valid_loss
            torch.save(model.state_dict(), 'tut1-model.pt')

        print(f'Epoch: {epoch + 1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
        print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc * 100:.2f}%')
        print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc * 100:.2f}%')

if __name__ == '__main__':
    main()

Context:
Could you check, if you are accidentally passing an int to the model instead of a tensor?
The error is raised, if you are trying to index an int via:
1[0]
> TypeError: 'int' object is not subscriptable

which seems to be called somewhere inside the forward method.


Freezing Batch norm layers while keeping them in the graph

Question:
Hi
If we set requires_grad to False for batch norm layers of a model, the batch norm layers do not remain in the graph. In this case, I cant fine tune these layers later if I want to. Is there any way we can freeze the layers, yet keep them in the graph so that they can be trained later?

Context: You could set the requires_grad attributes of the batchnorm parameters to True again if you want to train them later.


Image normalization for UNET with resnet pretrained encoder

Question:
Working on healthcare image datasets what should be the normalization stats used on these images, imagenet stats or stats from the new dataset being trained on?
Does it even matter which we choose?

Context: I would expect the custom stats to work better especially if the actual data stats differ between the medical images and ImageNet, but would also recommend to test both approaches.


Torch version affects the network’s training performance

Question:
I am opening this issue because apparently depending on which version of pytorch you are using, the training result will be different. Here are the 3px error evaluation curves of on a minimal example of overfitting the network on a single image for 300 epochs:

The purple line is trained with Pytorch 1.7.0 and the orange line is trained with Pytorch 1.5.1. As you can see, with version 1.7.0 the error rate is flat 100%, while version 1.5.1 the error rate is dropping. Reason for this is that the BatchNorm function has changed between version 1.5.1 and Pytorch 1.7.0. In version 1.5.1, if I disable track_running_stats here, both evaluation and training will use batch stats. However in Pytorch 1.7.0, it is forced to use running_mean and running_var in evaluation mode, while in training the batch stats is used. With track_running_stats disabled, the running_mean is 0 and running_var is 1, which is clearly different from the batch stats.
How can I maintain this feature beyond version 1.5（evaluation using batch stats）？

Context:
DeH40_Sung:

In version 1.5.1, if I disable track_running_stats here, both evaluation and training will use batch stats. However in Pytorch 1.7.0, it is forced to use running_mean and running_var in evaluation mode, while in training the batch stats is used. With track_running_stats disabled, the running_mean is 0 and running_var is 1, which is clearly different from the batch stats.

I don’t think that’s true, as seen in the code for 1.5.0 vs. 1.7.0. As you can see, in both versions the running stats are initialized as None.
Also, this code snippet shows the same behavior:
import torch
import torch.nn as nn

print(torch.__version__)

bn = nn.BatchNorm2d(3, track_running_stats=False)
print(bn.running_mean, bn.running_var)
x = torch.randn(2, 3, 24, 24) * 5 + 2

out = bn(x)
print(out.min(), out.max(), out.mean(), out.std())

bn.eval()
out = bn(x)
print(out.min(), out.max(), out.mean(), out.std())

Output:
1.5.0+cpu
None None
tensor(-3.5741, grad_fn=<MinBackward1>) tensor(3.5072, grad_fn=<MaxBackward1>) tensor(1.6557e-09, grad_fn=<MeanBackward0>) tensor(1.0001, grad_fn=<StdBackward0>)
tensor(-3.5741, grad_fn=<MinBackward1>) tensor(3.5072, grad_fn=<MaxBackward1>) tensor(1.6557e-09, grad_fn=<MeanBackward0>) tensor(1.0001, grad_fn=<StdBackward0>)

1.7.0+cpu
None None
tensor(-3.3744, grad_fn=<MinBackward1>) tensor(3.2699, grad_fn=<MaxBackward1>) tensor(3.8633e-09, grad_fn=<MeanBackward0>) tensor(1.0001, grad_fn=<StdBackward0>)
tensor(-3.3744, grad_fn=<MinBackward1>) tensor(3.2699, grad_fn=<MaxBackward1>) tensor(3.8633e-09, grad_fn=<MeanBackward0>) tensor(1.0001, grad_fn=<StdBackward0>)

so the difference might be coming from another part of the code.


CUDA error: unknown error. CUDA kernel errors might be asynchronously reported at some other API call

Question:
Hi all,
I am a beginner of PyTorch and CV. I encounter a problem when trying to use mmaction2 to extract features from video clips. Following the tutorial from here, I tried to run a single video test and my command is
python3 tools/misc/clip_feature_extraction.py \
configs/recognition/i3d/i3d_r50_video_32x2x1_100e_kinetics400_rgb.py \
pretrained/i3d_r50_video_32x2x1_100e_kinetics400_rgb_20200826-e31c6f52.pth \
--video-list examples/inputs/video_list_single.txt \
--video-root examples/inputs/video \
--out examples/outputs/examples_feature.pkl

However, I got the a RuntimeError: CUDA error: unknown error.
load checkpoint from local path: pretrained/i3d_r50_video_32x2x1_100e_kinetics400_rgb_20200826-e31c6f52.pth
[                                                  ] 0/1, elapsed: 0s, ETA:Traceback (most recent call last):
  File "tools/misc/clip_feature_extraction.py", line 229, in <module>
    main()
  File "tools/misc/clip_feature_extraction.py", line 217, in main
    outputs = inference_pytorch(args, cfg, distributed, data_loader)
  File "tools/misc/clip_feature_extraction.py", line 118, in inference_pytorch
    outputs = single_gpu_test(model, data_loader)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/mmcv/engine/test.py", line 33, in single_gpu_test
    result = model(return_loss=False, **data)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/mmcv/parallel/data_parallel.py", line 50, in forward
    return super().forward(*inputs, **kwargs)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 166, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/mmaction/models/recognizers/base.py", line 264, in forward
    return self.forward_test(imgs, **kwargs)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/mmaction/models/recognizers/recognizer3d.py", line 99, in forward_test
    return self._do_test(imgs).cpu().numpy()
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/mmaction/models/recognizers/recognizer3d.py", line 63, in _do_test
    feat = self.extract_feat(imgs)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 98, in new_func
    return old_func(*args, **kwargs)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/mmaction/models/recognizers/base.py", line 163, in extract_feat
    x = self.backbone(imgs)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/mmaction/models/backbones/resnet3d.py", line 854, in forward
    x = res_layer(x)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/container.py", line 141, in forward
    input = module(input)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/mmaction/models/backbones/resnet3d.py", line 318, in forward
    out = _inner_forward(x)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/mmaction/models/backbones/resnet3d.py", line 305, in _inner_forward
    out = self.conv1(x)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/mmcv/cnn/bricks/conv_module.py", line 201, in forward
    x = self.conv(x)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/mmcv/cnn/bricks/wrappers.py", line 80, in forward
    return super().forward(x)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 590, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/xxx/miniconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 586, in _conv_forward
    input, weight, bias, self.stride, self.padding, self.dilation, self.groups
RuntimeError: CUDA error: unknown error
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

If I set CUDA_LAUNCH_BLOCKING=1, i.e., CUDA_LAUNCH_BLOCKING=1 python3 ..., nothing more is shown.
I am not sure what causes the error, but I guess might be CUDA or PyTorch setup problems, since the codes can work properly on the other machine. FYI, I list the environment of the two machine below.

Device 1 (has error)
Device 2 (no error)

Platform
WSL2, Ubuntu 20.04.3
WSL2, Ubuntu 20.04.3

GPU
GeForce GTX 1080 Ti, Driver=510.06, CUDA=11.6
GeForce RTX 2060, Driver=510.06, CUDA=11.6

PyTorch
pytorch=1.10.1, py=3.7, cuda=11.3.1
pytorch=1.10.1, py=3.7, cuda=11.3.1

My question is what causes the error and how I can fix it? Thanks very much.

Context: Are you able to build and run any CUDA examples in the first setup at all or are all CUDA applications crashing?


Neural network reproducibility and metaheuristics

Question:
hi, i have a problem with the neural network optimization reproducibility with the black hole algorithm (metaheuristics). I am maximizing the accuracy by exploring the learning ratio, but at the time of reproducing the results I am not able to obtain the same. My optimal result is in iteration 20 of the black hole algorithm. I do not understand why I cannot reproduce the results of the network in an isolated way. I am using the seed as follows, I am attaching code. In advance thank you very much for the help.
import random
from numpy.random import seed
torch.manual_seed(1)
torch.cuda.manual_seed_all(1)
np.random.seed(1)
random.seed(1)
seed(1)

class BlackHole:

    in_features = 23520
    n_etiquetas = 4

    tipo = "clasificacion"
    directorio_de_entrenamiento = "parches_2_patologo2_aumentado_10"
    rutas_csv_entrenamiento = "train_a.csv"
    directorio_de_validacion = "parches_2_patologo2_aumentado_10"
    rutas_csv_validacion = "validacion_a.csv"

    train_simple_dataset = SimpleDataLoader(rutas_csv_entrenamiento,directorio_de_entrenamiento, tipo,"entrenamiento")
    dataloader_entrenamiento = DataLoader(train_simple_dataset, batch_size=60)

    valid_simple_dataset = SimpleDataLoader(rutas_csv_validacion,directorio_de_validacion, tipo,"validacion")
    dataloader_validacion = DataLoader(valid_simple_dataset, batch_size=60)

    model = Net(in_features,n_etiquetas)
    criterion = nn.CrossEntropyLoss()

    if torch.cuda.is_available():
        model.cuda()

    params =model.parameters()
    epocas = 30

    Total_de_Estrellas = 5
    Dimencion_de_vector = 1
    Iteraciones = 100
    constelacion = []
    ancho = 224
    alto = 224
    directorio = "./union_imagenes/"
    registos_excel = "./nuevo_conjunto.xlsx"

    def run(self):
        inicio_mh = time.time()

        print("Creando Poblacion")
        blackhole_mov = self.Create_blackhole_mov()
        self.constelacion = self.creacion_constelacion()
        print("Iniciando problacion")
        for iteracion in range(self.Iteraciones):
            print("ITERACION: {}".format(iteracion))
            self.constelacion = self.fit_constelacion()
            print('constelacion 0', self.constelacion)
            blackhole = np.copy(self.constelacion[self.detectar_agujero()])
            if(np.float64(blackhole[1])>np.float64(blackhole_mov[1])):
                blackhole = blackhole_mov
            print("blackhole de poblacion: ",blackhole)
            wandb.log({"blackhole poblacion":  blackhole[1],"epoch": iteracion})

            self.constelacion = np.copy(self.movimiento(blackhole[0]))
            self.constelacion = self.fit_constelacion()
            print("Poblacion con movimiento: {}".format(self.constelacion))
            blackhole_mov = np.copy(self.constelacion[self.detectar_agujero()])
            print("blackhole de poblacion mov: ",blackhole_mov)
            if(np.float64(blackhole[1])<np.float64(blackhole_mov[1])):
                blackhole_mov = blackhole
            print("BlackHole Ganador!:".format(blackhole_mov))
            wandb.log({"blackhole mov":  blackhole_mov[1],"epoch": iteracion})

            blackhole_mov[2]=self.horizonte_evento(blackhole_mov)

            self.constelacion = self.Absorcion(blackhole_mov)

        fin_mh  =time.time()
        total = (fin_mh-inicio_mh)/60
        print("Tiempo utilizado: {}".format(total))
        print('Best Black Hole Found: ',' '.join(map(str,blackhole_mov[0])),'Fit:', blackhole_mov[1],'Radius',blackhole_mov[2],'loss', blackhole_mov[3])
        return blackhole_mov

    """
        Metodos de Creacion
    """

    def creacion_estrellas(self):
        position = np.append(np.float64(np.random.uniform(0.0001,0.0002,size=(1, self.Dimencion_de_vector))),np.random.randint(100, size=1))
        metrica = 0
        radio = 0
        loss = 0
        acc = 0
        area_bajo_acc = 0
        star = [position,metrica,radio,loss,acc,area_bajo_acc]
        return np.array(star)

    def creacion_constelacion(self):
        constelacion = []
        for i in range(self.Total_de_Estrellas):
            constelacion.append(self.creacion_estrellas())
        return np.array(constelacion)

    def Create_blackhole_mov(self):
        position = np.zeros(self.Dimencion_de_vector)
        fit = 10000
        radio = 0
        star = [position,fit,radio]
        return np.array(star)

    """
        Metodos relacionados con Movimiento de Constelacion
    """

    def movimiento(self, blackhole):
        for i in range(self.Total_de_Estrellas):
            self.constelacion[i,0] = self.constelacion[i,0] + np.random.uniform()*(blackhole - self.constelacion[i,0])
        return self.constelacion

    def distancia(self,blackhole,constellation):
        return np.sum((blackhole[0]-constellation[0])<blackhole[2])

    """
        Metodos Relacionados Con Absorcion
    """

    def Absorcion(self,blackhole):
        index=int(np.where(self.constelacion[:,1] == min(self.constelacion[:,1]))[0][0])
        for i in range(self.Total_de_Estrellas):
            if(i == index):
                None
            else:
                if(self.distancia(blackhole,self.constelacion[i])==self.Dimencion_de_vector):
                    self.constelacion[i]= self.creacion_estrellas()
        return self.constelacion

    def horizonte_evento(self,blackhole):
        return np.float64((np.divide(blackhole[1], np.sum(self.constelacion[:,1], dtype=np.float64) ,dtype=np.float64  )))

    """
        Metodos Relacionados con Fit
    """
    def fitX(self,importancia):
        #return VGG16m().modelo(importancia[0],self.x_train,self.y_train,self.x_test,self.y_test)
        print("importancia: ",importancia)
        return VGG16m().run(importancia[0],self.x_test,self.x_train,self.y_test,self.y_train)

    def fit_constelacion(self):
        for i in range(self.Total_de_Estrellas):
            print("Estrella {}:".format(i))
            print("constelacion[i]: {} ".format(self.constelacion[i]))
            self.constelacion[i][1], self.constelacion[i][3],self.constelacion[i][4],self.constelacion[i][5] = self.fit(self.constelacion[i])
        return self.constelacion

    def detectar_agujero(self):
        return int(np.where(self.constelacion[:,1] == min(self.constelacion[:,1]))[0][0])

    def fit(self, lr):
        print(lr)
        dataloader_entrenamiento = self.dataloader_entrenamiento
        model = self.model
        criterion = self.criterion
        epocas = self.epocas

        optimizador = optim.Adagrad(model.parameters(), lr=lr[0][0], lr_decay=0, weight_decay=0, initial_accumulator_value=0, eps=1e-10)
        inicio = time.time()
        for epoch in range(0,epocas):
            #entrenamiento.train(dataloader_entrenamiento, model, criterion, optimizador, epoch)
            model = entrenamiento.train(dataloader_entrenamiento, model, criterion,
                                    optimizador, epoch, losses,accuracy_,recall_,precision_,kappa_,accuracy,precision,recall,cohenkappa,enviar_wand)
            #entrenamiento.validacion(dataloader_validacion, model,criterion, epoch)
            #entrenamiento.validacion(dataloader_validacion, model, criterion, epoch,val_accuracy_,val_recall_,val_precision_,val_kappa_,val_losses,val_cohenkappa,val_accuracy,enviar_wand)
            model,total_accuracy,lossbh = entrenamiento.validacion(dataloader_entrenamiento, model, criterion,
                                    epoch, losses,val_accuracy,val_precision,val_recall,val_cohenkappa,enviar_wand)
            scheduler.step()
        print("Entrenamiento 1 Finalizado")
        print("Resultados Acc:{} Loss:{}".format(total_accuracy.item(),lossbh.item()))
        fin = time.time()
        total = (fin - inicio)/60
        print("Tiempo de Entrenamiento: {}".format(total))
        return -(total_accuracy.item()),lossbh.item(), 0, 0

Context: The reproducibility docs 1 give you more information e.g. on how to enable deterministic algorithms etc., which might be helpful.


Question:
Hi, i try to use a LSTM for regression task (2 target)
My input data is with theese columns(Index, value1, value2, value3, value4, value5 and target1 and target2)
i want to pass at every time step in lstm an array like this:
[0,0,0,0,0] where the number 0  is an array of 5 values (value1, value2, value3, value4, value5) (the first row without target)
at second time steps:
[0,0,0,0,1]  where the number 1 is an array of 5 values (value1, value2, value3, value4, value5) (the second row without target)
and so on.
so this is my class dataset:
from torch.utils.data.dataset import Dataset
import numpy as np
import pandas as pd
from os import path
import torch

# CREAZIONE DELLA CLASSE CHE CI PERMETTE DI CARICARE LE FEATURES

class Dataset(Dataset):

    """
    Carica il dataset per l'eseprimento  che ci permette di caricare le feature
    """

    def __init__(self, base_path, csv_Name):

        """Input:
            dataset CSV
        """
        self.base_path = base_path
        self.csv_Name = csv_Name
        path_finale = path.join(base_path, csv_Name)

        self.file = pd.read_csv(path_finale)

    def __getitem__(self, index):
        #print(index)
        names = []
        index1 = index - 4
        index2 = index - 3
        index3 = index - 2
        index4 = index - 1
        index5 = index

        if index1 < 0:
            index1 = 0
        if index2 < 0:
            index2 = 0
        if index3 < 0:
            index3 = 0
        if index4 < 0:
            index4 = 0

        name1, rss11, rss21, rss31, rss41, rss51, coordinateX1, coordinateY1 = self.file.iloc[index1]
        name2, rss12, rss22, rss32, rss42, rss52, coordinateX2, coordinateY2 = self.file.iloc[index2]
        name3, rss13, rss23, rss33, rss43, rss53, coordinateX3, coordinateY3 = self.file.iloc[index3]
        name4, rss14, rss24, rss34, rss44, rss54, coordinateX4, coordinateY4 = self.file.iloc[index4]
        name5, rss15, rss25, rss35, rss45, rss55, coordinateX5, coordinateY5 = self.file.iloc[index5]
        name1 = int(name1)
        name2 = int(name2)
        name3 = int(name3)
        name4 = int(name4)
        name5 = int(name5)
        names.append(name1)
        names.append(name2)
        names.append(name3)
        names.append(name4)
        names.append(name5)

        array1 = torch.tensor([rss11, rss21, rss31, rss41, rss51])
        array2 = torch.tensor([rss12, rss22, rss32, rss42, rss52])
        array3 = torch.tensor([rss13, rss23, rss33, rss43, rss53])
        array4 = torch.tensor([rss14, rss24, rss34, rss44, rss54])
        array5 = torch.tensor([rss15, rss25, rss35, rss45, rss55])
        array = torch.cat((array1, array2, array3, array4, array5))
        array = array.reshape(1,25)
        coordinateX = coordinateX5
        coordinateY = coordinateY5

        return {"ID": names, "Array": array, 'Movement': np.array([coordinateX, coordinateY], dtype='float')}

    def __len__(self):
        return len(self.file)

My LSTM Model is like this:
import torch
import torch.nn as nn
class LSTM_CLASS(nn.Module):
    def __init__(self, num_classes, input_size, hidden_size, num_layers):
        super(LSTM_CLASS, self).__init__()
        self.num_classes = num_classes #2
        self.num_layers = num_layers
        self.input_size = input_size
        self.hidden_size = hidden_size

        self.lstm = nn.LSTM(input_size= self.input_size,
                            hidden_size=self.hidden_size,
                            num_layers= self.num_layers,
                            batch_first=True)

        self.fc1 = nn.Linear(hidden_size, hidden_size)

        self.fc2 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):

        ula, (h_out,_) = self.lstm(x)
        [_, _, features] = h_out.shape
        out = self.fc1(h_out)
        out = out.view(-1, features)
        relu = nn.ReLU()
        out = relu(out)
        out = self.fc2(out)

        return out

and i call the model, create the dataloaders like this:
    model = LSTM.LSTM_CLASS(num_classes = 2, input_size=25, hidden_size=128, num_layers=1)

    train_dataset = Dataset('../Dataset/','121_train_seq.csv')
    valid_dataset = Dataset('../Dataset/','121_valid_seq.csv')

    train_loader = DataLoader(train_dataset, batch_size=16, num_workers=2)
    valid_loader = DataLoader(valid_dataset, batch_size=16, num_workers=2)

i don’t understand with (with suffle = False on dataloader), in get_item od the class dataset i have this situation:
[0, 0, 0, 0, 0]
[0, 0, 0, 0, 1]
[0, 0, 0, 1, 2]
[0, 0, 1, 2, 3]
[0, 1, 2, 3, 4]
[1, 2, 3, 4, 5]
[2, 3, 4, 5, 6]
[3, 4, 5, 6, 7]
[4, 5, 6, 7, 8]
[5, 6, 7, 8, 9]
[6, 7, 8, 9, 10]
[7, 8, 9, 10, 11]
[8, 9, 10, 11, 12]
[9, 10, 11, 12, 13]
[10, 11, 12, 13, 14]
[11, 12, 13, 14, 15]
why from now a skip of 16 (my BS?)
[28, 29, 30, 31, 32]
[29, 30, 31, 32, 33]
[30, 31, 32, 33, 34]
[31, 32, 33, 34, 35]
[32, 33, 34, 35, 36]
[33, 34, 35, 36, 37]
[34, 35, 36, 37, 38]
[35, 36, 37, 38, 39]
[36, 37, 38, 39, 40]
[37, 38, 39, 40, 41]
[38, 39, 40, 41, 42]
[39, 40, 41, 42, 43]
[40, 41, 42, 43, 44]
[41, 42, 43, 44, 45]
[42, 43, 44, 45, 46]
[43, 44, 45, 46, 47]
[60, 61, 62, 63, 64]
[61, 62, 63, 64, 65]
[62, 63, 64, 65, 66]
[63, 64, 65, 66, 67]
[64, 65, 66, 67, 68]
[65, 66, 67, 68, 69]
[66, 67, 68, 69, 70]
[67, 68, 69, 70, 71]
[68, 69, 70, 71, 72]
where do i wrong?
for completeness this is my training procedure:
def train(model, train_loader, valid_loader, exp_name = "LSTM",  lr=0.0001, epochs=1000, wd = 0.000001):
    criterionX = nn.SmoothL1Loss()
    criterionZ = nn.SmoothL1Loss()

    optimizer = Adam(params=model.parameters(),lr = lr, weight_decay=wd)
    scheduler = StepLR(optimizer, step_size=50, gamma=0.5)#per ogni 100 epochs, lr si divide per due

    # meters
    lossX_meter = AverageValueMeter()
    lossZ_meter = AverageValueMeter()
    lossT_meter = AverageValueMeter()

    # device
    device = "cuda" if torch.cuda.is_available() else "cpu"
    model.to(device)
    loader = {"train": train_loader, "test": valid_loader}

    loss_X_logger = VisdomPlotLogger('line', env=exp_name, opts={'title': 'LossX', 'legend': ['train', 'test']})
    loss_Z_logger = VisdomPlotLogger('line', env=exp_name, opts={'title': 'LossZ', 'legend': ['train', 'test']})
    loss_T_logger = VisdomPlotLogger('line', env=exp_name, opts={'title': 'Total_Loss', 'legend': ['train', 'test']})
    visdom_saver = VisdomSaver(envs=[exp_name])
    for e in range(epochs):
        for mode in ["train", "test"]:
            lossX_meter.reset()
            lossZ_meter.reset()
            lossT_meter.reset()
            model.train() if mode == "train" else model.eval()
            with torch.set_grad_enabled(mode == "train"):  # abilitiamo i gradienti in training
                for i, batch in enumerate(loader[mode]):
                    x = batch["Array"].to(device)
                    dx = batch['Movement'][:, 0].float().to(device)
                    dz = batch['Movement'][:, 1].float().to(device)
                    output= model(x)

                    out1, out2 = output[:, 0], output[:, 1]

                    l1 = criterionX(out1, dx)
                    l2 = criterionZ(out2, dz)
                    loss = l1+l2

                    if mode == "train":
                        optimizer.zero_grad()
                        loss.backward()
                        optimizer.step()

                    n = x.shape[0]  # numero di elementi nel batch

                    lossX_meter.add(l1.item() * n, n)#update meter to ploot
                    lossZ_meter.add(l2.item() * n, n)
                    lossT_meter.add(loss.item()* n, n)
                    if mode == "train":
                        loss_X_logger.log(e + (i + 1) / len(loader[mode]), lossX_meter.value()[0], name=mode)
                        loss_Z_logger.log(e + (i + 1) / len(loader[mode]), lossZ_meter.value()[0], name=mode)
                        loss_T_logger.log(e + (i + 1) / len(loader[mode]), lossT_meter.value()[0], name=mode)
            loss_X_logger.log(e + (i + 1) / len(loader[mode]), lossX_meter.value()[0], name=mode)
            loss_Z_logger.log(e + (i + 1) / len(loader[mode]), lossZ_meter.value()[0], name=mode)
            loss_T_logger.log(e + (i + 1) / len(loader[mode]), lossT_meter.value()[0], name=mode)

        scheduler.step()
        visdom_saver.save()
        torch.save(model.state_dict(), '%s.pth' % exp_name)

    return model

Can you help me?

Context: Have you tried num_workers=1 and does that still give the same issue?


Why doesn’t the accuracy change when I train

Question: e7f09ecfddac61e5f6228c97258c7d31170×1592 95.4 KB

Context: @hunter137 there are multiple reasons why this might happen. Lack of convergence etc. Please provide your code and data link so that people can take a look


Question:
For the below code from https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py 8 I was trying to implement quantization. Since presently it doesn’t has Conv, BatchNorm and Relu as a single block, for classification class what should be the fuse_model.
cfg=[64,"M",128,"M",256,256,"M",512,512,"M",512,512,"M"]
def vgg_make(cfg,batch_norm):
    emp=[]
    inp_c=3
    for i,val in enumerate(cfg):
        if val=="M":
            emp+=[nn.MaxPool2d(kernel_size=(2,2), stride=2)]
        else:
            conv=nn.Conv2d(inp_c, val, kernel_size=(3,3), padding=1)
            if batch_norm:
                emp+=[conv, nn.BatchNorm2d(val), nn.ReLU(inplace=True)]
            else:
                emp+=[conv, nn.ReLU(inplace=True)]
            inp_c=val
    return nn.Sequential(*emp)

backbone=vgg_make(cfg, batch_norm=True)

classifer = nn.Sequential(nn.Linear(512*7*7,4096),
                          nn.ReLU(inplace=True),
                          nn.Dropout(),
                          nn.Linear(4096,4096),
                          nn.ReLU(inplace=True),
                          nn.Dropout(),
                          nn.Linear(4096,10))

class classfication(nn.Module):
    def __init__(self, backbone, classifer):
        super().__init__()
        self.backbone=backbone
        self.classifer=classifer
        self.quant = QuantStub()
        self.dequant = DeQuantStub()
    def forward(self, x):
        x=self.quant(x)
        x=self.backbone(x)
        x=torch.flatten(x, 1)
        x=self.classifer(x)
        x=self.dequant(x)
        return x

    def fuse_model(self):
        for m in self.modules():
            pass

Context:
Unable to fuse convolution, batchnorm and relu.
Each entry in the module list looks something like
  (0): ConvBNReLU(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU6(inplace=True)
  )

At the time of fusion using below code i get the error.
 def fuse_model(self):
     for m in self.modules():
        if type(m) == ConvBNReLU:
           fuse_modules(m, ['0', '1', '2'], inplace=True)

NotImplementedError: Cannot fuse modules: (<class 'torch.nn.modules.conv.Conv2d'>, <class 'torch.nn.modules.batchnorm.BatchNorm2d'>, <class 'torch.nn.modules.activation.ReLU6'>)

Please let me know the way around.


Unable to load pretrained weight into custom model in Pytorch Lightning

Question:
Hello folks,
I want to retrain a custom model with my data. I can load the pretrained weights (.pth file) into the model in Pytorch and it runs but I want more functionality and refactored the code into Pytorch Lightning. I am having trouble loading the pretrained weight into the Pytorch Lightning model. The Pytorch Lightning code works but I have limited data and don’t have enough data to train from scratch.
Pytorch Model:
class BDRAR(nn.Module):
      def __init__(self, stat=1):
          super(BDRAR, self).__init__()
          resnext = ResNeXt101()
          self.layer0 = resnext.layer0
          self.layer1 = resnext.layer1
          self.layer2 = resnext.layer2
          self.layer3 = resnext.layer3
          self.layer4 = resnext.layer4

My Lightning model looks like this:
 class liteBDRAR(pl.LightningModule):
         def __init__(self):
              super(liteBDRAR, self).__init__()
              self.model = BDRAR()
              print('Model Created!')

         def forward(self, x):
              return self.model(x)

My run:
KeyError: 'state_dict
path = './ckpt/BDRAR/3000.pth'
bdrar = liteBDRAR.load_from_checkpoint(path,  strict=False)
trainer = pl.Trainer(fast_dev_run=True, gpus=1)
trainer.fit(bdrar)

I get the following error:
    "keys = model.load_state_dict(checkpoint["state_dict"], strict=strict) **KeyError: 'state_dict**'"

I will appreciate any help. Thank you so much.
@williamFalcon

Context: Would it work, if you load the state_dict into the internal model without using the Lightning method via liteBDRARobject.model.load_state_dict()?


Out of memory error when resume training even though my GPU is empty

Question:
I am training a classification model and I have saved some checkpoints. When I try to resume training, however, I got out of memory errors:

Traceback (most recent call last):
File “train.py”, line 283, in
main()
File “train.py”, line 86, in main
optimizer.load_state_dict(checkpoint[‘optimizer’])
File “/home/haojiedong/tools/anaconda3/lib/python3.6/site-packages/torch/optim/optimizer.py”, line 96, in load_state_dict
state_dict = deepcopy(state_dict)
File “/home/haojiedong/tools/anaconda3/lib/python3.6/copy.py”, line 150, in deepcopy
y = copier(x, memo)
File “/home/haojiedong/tools/anaconda3/lib/python3.6/copy.py”, line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File “/home/haojiedong/tools/anaconda3/lib/python3.6/copy.py”, line 150, in deepcopy
y = copier(x, memo)
File “/home/haojiedong/tools/anaconda3/lib/python3.6/copy.py”, line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File “/home/haojiedong/tools/anaconda3/lib/python3.6/copy.py”, line 150, in deepcopy
y = copier(x, memo)
File “/home/haojiedong/tools/anaconda3/lib/python3.6/copy.py”, line 240, in deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File “/home/haojiedong/tools/anaconda3/lib/python3.6/copy.py”, line 161, in deepcopy
y = copier(memo)
File “/home/haojiedong/tools/anaconda3/lib/python3.6/site-packages/torch/tensor.py”, line 23, in deepcopy
new_storage = self.storage().deepcopy(memo)
File “/home/haojiedong/tools/anaconda3/lib/python3.6/site-packages/torch/storage.py”, line 28, in deepcopy
new_storage = self.clone()
File “/home/haojiedong/tools/anaconda3/lib/python3.6/site-packages/torch/storage.py”, line 42, in clone
return type(self)(self.size()).copy(self)
RuntimeError: CUDA error: out of memory

The error is in optimizer.load_state_dict().  The code to load the checkpoint is like this:
if args.resume:
    if os.path.isfile(args.resume):
        print("=> loading checkpoint '{}'".format(args.resume))
        checkpoint = torch.load(args.resume)
        args.start_epoch = checkpoint['epoch']
        best_acc = checkpoint['best_acc']
        model.load_state_dict(checkpoint['state_dict'])
        optimizer.load_state_dict(checkpoint['optimizer'])
        print("=> loaded checkpoint '{}' (epoch {})".format(
            args.resume, checkpoint['epoch']))
    else:
        print("=> no checkpoint found at '{}'".format(args.resume))

I checked the target GPU, it is actually empty.  I am currently using pytorch version 0.4.1.

Context:
Could you try to load the checkpoints onto the CPU first using the map_location argument in torch.load 138?
After it was successful, try to push your model onto the GPU again.


ResNet152 - Flipping features vs Flipping video then computing features

Question:
I’m currently using ResNet152 to extract features from a video, where the extracted features are of shape (1842, 2048). I would like to also extract features for the same video, but horizontally flipped. I see 2 ways to do so:

Flip the video horizontally, then re-compute features
Flip the ResNet152 features for each frame from the already-computed features on the original video

The 2nd option is attractive when considering computing resources - how would that be achieved? Are ResNet152 features able to be “flipped” like this? If so, I’m guessing it can be done with torch.flip?
Thank you for reading! Hoping someone can guide me in the right direction.

Context:
I’m not sure but don’t think that flipping the intermediate features would work.
If you only flip the input image, e.g. the conv kernels would not be flipped and the intermediate activations (as well as the ouput) would be different. Think about kernels used as edge detectors which work on a flipped image, so I would probably try to use the first approach of flipping the inputs.