Question:
I got this user warning when I tried to implement the augmentation for my data, has anyone met this problem?
UserWarning: Argument fill/fillcolor is not supported for Tensor input. Fill value is zero
warnings.warn(“Argument fill/fillcolor is not supported for Tensor input. Fill value is zero”)

Context: Can you post a minimal reproducible example for this error?


Assert boxes.size(1) == 5, ‘The boxes tensor shape is not correct as Tensor[K, 5]’ AssertionError: The boxes tensor shape is not correct as Tensor[K, 5]

Question:
Hello All,
I am getting the following error while using the from torchvision.ops import roi_align function in Pytorch, while using the function roi_align()
assert boxes.size(1) == 5, 'The boxes tensor shape is not correct as
Tensor[K, 5]'
AssertionError: The boxes tensor shape is not correct as Tensor[K, 5]

I went through the Pytorch doc and see that it required for it to be of dimension 5, where the 1st column represents the batch index.
boxes (Tensor[K, 5] or List[Tensor[L, 4]]): the box coordinates in (x1, y1,
x2, y2) format where the regions will be taken from. If a single Tensor is
passed,then the first column should contain the batch index. If a list of
Tensors is passed, then each Tensor will correspond to the boxes for
an element in a batch

I am not quite sure what does the batch index mean here . In my training case, I have set the Batch Size to be 1 and size of boxes which is currently being passed is torch.Size([66, 4]) (where 66 is something which would vary on each batch.)
Thank you!

Context: Hi, did you find solution to your problem.


What is the difference between 1x1 con and avergaepool1d?

Question: what is the difference between 1x1 conv and avergaepool1d ?

Context: A 1x1 conv layer will use a kernel size of 1x1 (i.e. a single pixel) and would act as a linear layer in the default setup. The nn.AdaptiveAvgPool1d layer works on 3D inputs in the shape [batch_size, channels, seq_len] and returns a desired output shape in the temporal dimension by applying an adaptive windowing approach to calculate the average of the input values.


Normalization while fine tuning a trained model

Question:
Consider a scenario where I want to fine tune a trained model for a downstream task.
The trained model used [0, 1] normalization during its training. Now, is it okay if I do a z-score normalization [ input = (input - mean) / std ] while fine tuning?
I am not sure how to make sense of this scenario. Please help.

Context: I would guess your initial loss might be a bit higher using this new normalization approach, as your model would get “new” samples. Fine tuning the model might work, but again I would guess that you might get better results if you fine tune the entire model (not only the last layer(s)). These are just my guesses, so I would recommend to run some experiments and compare both approaches (and please share the results if you plan on running these tests!  ).


RuntimeError: Unsupported marker type 0x4f on 2nd epoch

Question:
Greetings,
I get this error when I train a resnet18 model from Torchvision on a custom dataset. It only happens on the 2nd epoch after a few dozen iterations.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-63-e338034a4492> in <module>()
----> 1 model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,num_epochs=25)

8 frames
/usr/local/lib/python3.7/dist-packages/torchvision/io/image.py in decode_image(input, mode)
    200         output (Tensor[image_channels, image_height, image_width])
    201     """
--> 202     output = torch.ops.image.decode_image(input, mode.value)
    203     return output
    204

RuntimeError: Unsupported marker type 0x4f

I use these transforms:
transform = nn.Sequential(
            T.Resize([256, ]),
            T.CenterCrop(224),
            T.ConvertImageDtype(torch.float),
            T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        )

Does anyone have an idea of what could be wrong?

Context: The image decoding is failing and torchvision seems to reraise the error. Are you able to load the file with another library and if so, could you save it using a different file name, and try to load it again via torch.ops.image.decode_image?


Set Dropout to be True for all the different layers of a Transformer model!

Question:
I am working on image classification using a Transformer! The problem is that of overfitting, I am getting training accuracy of 1, but validation and test accuracy is around 70%. I wanted to ask how can I set dropout to be true in different layers of the model. Here is the architecture of the model :
VisionTransformer(
(patch_embed): PatchEmbed(
(proj): Conv2d(3, 768, kernel_size=(16, 16), stride=(16, 16))
(norm): Identity()
)
(pos_drop): Dropout(p=0.5, inplace=True)
(blocks): Sequential(
(0): Block(
(norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=768, out_features=2304, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=768, out_features=768, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): Identity()
(norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(act): GELU()
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(1): Block(
(norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=768, out_features=2304, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=768, out_features=768, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): Identity()
(norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(act): GELU()
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(2): Block(
(norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=768, out_features=2304, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=768, out_features=768, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): Identity()
(norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(act): GELU()
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(3): Block(
(norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=768, out_features=2304, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=768, out_features=768, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): Identity()
(norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(act): GELU()
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(4): Block(
(norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=768, out_features=2304, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=768, out_features=768, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): Identity()
(norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(act): GELU()
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(5): Block(
(norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=768, out_features=2304, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=768, out_features=768, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): Identity()
(norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(act): GELU()
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(6): Block(
(norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=768, out_features=2304, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=768, out_features=768, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): Identity()
(norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(act): GELU()
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(7): Block(
(norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=768, out_features=2304, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=768, out_features=768, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): Identity()
(norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(act): GELU()
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(8): Block(
(norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=768, out_features=2304, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=768, out_features=768, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): Identity()
(norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(act): GELU()
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(9): Block(
(norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=768, out_features=2304, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=768, out_features=768, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): Identity()
(norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(act): GELU()
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(10): Block(
(norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=768, out_features=2304, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=768, out_features=768, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): Identity()
(norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(act): GELU()
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(11): Block(
(norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=768, out_features=2304, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=768, out_features=768, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): Identity()
(norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(act): GELU()
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
)
(norm): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
(pre_logits): Identity()
(head): Linear(in_features=768, out_features=2, bias=True)
I want to set dropout to be 0.5 in all the different layers. Starting from the first layer when I do : model_ft._modules["pos_drop"] = nn.Dropout(0.5, inplace=True), it works for the first instance of the dropout but when I want to do the same thing for the second dropout and I try model_ft._modules["blocks"].attn.proj_drop = nn.Dropout(0.5, inplace=True), it throws an error.
The real problem is that I don’t know how to access the dropout layers in the network and set them all to non-zero values.
I would be very thankful to you if you could help me with how to access the different layers of  the model and set dropout to be true in all of them.

Context:
You could use model.apply to iterate all modules, check if the current module is an nn.Dropout, and change the p value as seen here:
class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(10, 10)
        self.drop1 = nn.Dropout(p=0.0)
        self.fc2 = nn.Linear(10, 10)
        self.drop2 = nn.Dropout(p=0.0)
        self.fc3 = nn.Linear(10, 10)
        self.drop3 = nn.Dropout(p=0.0)

model = MyModel()
print(model)

def set_dropout_p(m, p):
    if isinstance(m, nn.Dropout):
        m.p = p

model.apply(lambda m: set_dropout_p(m, p=0.5))
print(model)


Question:
I used “a = torch.inverse(mat_inverse)” line in my code. It works fine on google colab but gives below error when run on GPU.
RuntimeError: CUDA error: invalid argument
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
With CUDA_LAUNCH_BLOCKING=1 it shows below error. Is it related to memory requirement?
RuntimeError: CUDA error: invalid argument

Context: Could you post a minimal code snippet to reproduce the issue as well as the output of python -m torch.utils.collect_env, please?


Question:
I am working on training a CNN with transfer learning on resnet50. My script was working until I made some modifications. I manually split my dataset into train and validation folders. And I setup weights on the training dataset since it was unbalanced. I am receiving an error when I run the training. Here is my code and the error:
train_data_dir = '/home/ec2-user/SageMaker/data/frame_150/train'
val_data_dir = '/home/ec2-user/SageMaker/data/frame_150/val'
batch_size = 100
data_transforms = {
    'train': transforms.Compose([transforms.ToTensor(),
                                 transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])]),
    'val': transforms.Compose([transforms.ToTensor(),
                                 transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])
}
image_datasets = {'train': datasets.ImageFolder(train_data_dir, data_transforms['train']),
                  'val': datasets.ImageFolder(val_data_dir, data_transforms['val'])
               }
def make_weights_for_balanced_classes(images, nclasses):
    count = [0] * nclasses
    for item in images:
        count[item[1]] += 1
    weight_per_class = [0.] * nclasses
    N = float(sum(count))
    for i in range(nclasses):
        weight_per_class[i] = N/float(count[i])
    weight = [0] * len(images)
    for idx, val in enumerate(images):
        weight[idx] = weight_per_class[val[1]]
    return weight

weights = make_weights_for_balanced_classes(image_datasets['train'].imgs, len(image_datasets['train'].classes))
weights = torch.DoubleTensor(weights)
sampler = torch.utils.data.sampler.WeightedRandomSampler(weights, len(weights))

dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=batch_size, sampler=sampler, num_workers=4) for x in ['train', 'val']}

dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
            if phase == 'train':
                scheduler.step()

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

model_ft = models.resnet50(pretrained=True)
num_ftrs = model_ft.fc.in_features
# Here the size of each output sample is set to 2.
# Alternatively, it can be generalized to nn.Linear(num_ftrs, len(class_names)).
model_ft.fc = nn.Linear(num_ftrs, 2)
model_ft = model_ft.to(device)
criterion = nn.CrossEntropyLoss()
# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)
# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=6)

ERROR:
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-29-c677c5f760af> in <module>()
      1 model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
----> 2                        num_epochs=6)

<ipython-input-26-6f4f613da14c> in train_model(model, criterion, optimizer, scheduler, num_epochs)
     20
     21             # Iterate over data.
---> 22             for inputs, labels in dataloaders[phase]:
     23                 inputs = inputs.to(device)
     24                 labels = labels.to(device)

~/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
    343
    344     def __next__(self):
--> 345         data = self._next_data()
    346         self._num_yielded += 1
    347         if self._dataset_kind == _DatasetKind.Iterable and \

~/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    854             else:
    855                 del self._task_info[idx]
--> 856                 return self._process_data(data)
    857
    858     def _try_put_index(self):

~/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _process_data(self, data)
    879         self._try_put_index()
    880         if isinstance(data, ExceptionWrapper):
--> 881             data.reraise()
    882         return data
    883

~/.local/lib/python3.6/site-packages/torch/_utils.py in reraise(self)
    392             # (https://bugs.python.org/issue2651), so we work around it.
    393             msg = KeyErrorMessage(msg)
--> 394         raise self.exc_type(msg)

IndexError: Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/ec2-user/.local/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/ec2-user/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ec2-user/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ec2-user/.local/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 137, in __getitem__
    path, target = self.samples[index]
IndexError: list index out of range

Context:
Can you try that:  drop_last=True
DataLoader(dataset, batch_size=1, shuffle=False, sampler=None,
           batch_sampler=None, num_workers=0, collate_fn=None,
           pin_memory=False, drop_last=False, timeout=0,
           worker_init_fn=None)

https://pytorch.org/docs/stable/data.html 6

drop_last  ( bool ,  optional ) – set to  True  to drop the last incomplete batch, if the dataset size is not divisible by the batch size. If  False  and the size of dataset is not divisible by the batch size, then the last batch will be smaller. (default:  False )


Sporadic “RuntimeError: std::bad_alloc” doing forward pass on CPU during deployment on certain hardware

Question:
The problem
I’ve got a fairly basic multi-task CNN I’ve built in Pytorch (efficientnet_pytorch package used for the body, pytorch-lightning used during training).
I’m now deploying the model to a simple Tornado web app. For the app I have a test suite which involves running multiple forward passes through this. The app only runs on the CPU.
All these tests pass on my local machine (Macbook Pro) and my development machine (a beefy Google Cloud box).
The tests fail, and the app often crashes when I run them on the deployment machine (an AWS t3.medium), with the error below.
Fixes I’ve tried

I noticed this in Docker, but running directly on the box gives the same issue.
I tried trying both an AWS t3 and an m5 box in case it was a memory problem (4GB->8GB), but that didn’t fix it, and watching the memory it doesn’t seem to be approaching the limits.
I’ve also upgraded to pytorch 1.10, which had no effect.
Putting the forward pass in a dumb try/except loop trying it multiple times fix the problem - so far it succeeds every time on the second attempt. That’s not an ideal solution though.

Does anyone have any thoughts on what the issue might be, solutions, or any suggestions for getting a more informative stack trace?
Thanks very much!
The error
self = Conv2dStaticSamePadding(
  3, 40, kernel_size=(3, 3), stride=(2, 2), bias=False
  (static_padding): ZeroPad2d(padding=(0, 1, 0, 1), value=0.0)
)
x = tensor([[[[-2.1008, -2.1008, -2.1008,  ..., -2.1008, -2.1008,  0.0000],
          [-2.1008, -2.1008, -2.1008,  ..., -2..., -1.7870,  ..., -1.7870, -1.7696,  0.0000],
          [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000]]]])

    def forward(self, x):
        x = self.static_padding(x)
>       x = F.conv2d(x, self.weight, self.bias, self.stride, self.padding, self.dilation, self.groups)
E       RuntimeError: std::bad_alloc

Context: Are you seeing the issue by simply executing the forward pass of your model or is it only raised in your deployment setup using the web server etc.?


Question: Okay so I have a pt file that I got from Roboflow. My question is, how do I load it into a script?

Context: Assuming the file was stored via torch.save you could use torch.load to load it.


Why do we multiply loss function with a constant?

Question:
Why do we multiply loss with a constant sometimes? I have seen people use
loss = nn.functional.binary_cross_entropy_with_logits(logits, labels)
loss *= labels.size(1)
or sometimes
loss = nn.functional.binary_cross_entropy_with_logits(logits, labels)
loss *= labels.size(0)
I am not sure the difference between using no constant multiply with loss, multiply with num of classes, multiply with batch size. Any body can give me an explanation ?

Context:
The loss is sometimes scaled with the batch size to calculate the epoch loss without adding a small potential error due to varying batch sizes. In particular, the last batch might often be smaller than the test, if the number of samples isn’t divisible by the batch size without a remainder.
In that case, adding the current batch losses to a running loss via epoch_loss += loss.item() and dividing by the number of batches (len(loader)) would add a small error.
If you scale the loss with the batch size before adding it to the running loss and divide by the number of samples (len(dataset)), you would avoid it.


Question:
Hi,
I am working with a pretrained VGG16 model to classify images. I have added my own layers after the model like this:
model = torchvision.models.vgg16(pretrained=True)
model.classifier.add_module('7', nn.Linear(in_features=1000, out_features=500,bias=True))
model.classifier.add_module('8', nn.ReLU())
model.classifier.add_module('9', nn.Linear(in_features=500, out_features=100,bias=True))
model.classifier.add_module('10', nn.ReLU())
model.classifier.add_module('11', nn.Linear(in_features=100, out_features=67,bias=True))

When I start the training, I print the output of the model and I see it’s nan. I don’t know what going on? I’ve ensured my input doesn’t contain nan and this is my dataloader:
class dataload(Dataset):

    def __init__(self, x, transform=None):
        self.data = x

    def __len__(self):
        return len(self.data)

    def __getitem__(self, i):

        img = mpimg.imread(self.data[i])/255.0
        img = img.transpose((2, 0, 1))
        img = torch.from_numpy(img).float()
        tmp = np.int32(filenames[i].split('/')[-1].split('_')[0][1])
        label = np.zeros(67)
        label[tmp] = 1
        label = torch.from_numpy(label).float()

        # if self.transform:
        #     sample = self.transform(sample)

        return img,label

What could be going on?

Context:
I don’t know what might be causing the issue, as I’m not able to reproduce the NaN outputs using your code and random inputs:
model = models.vgg16(pretrained=True)
model.classifier.add_module('7', nn.Linear(in_features=1000, out_features=500,bias=True))
model.classifier.add_module('8', nn.ReLU())
model.classifier.add_module('9', nn.Linear(in_features=500, out_features=100,bias=True))
model.classifier.add_module('10', nn.ReLU())
model.classifier.add_module('11', nn.Linear(in_features=100, out_features=67,bias=True))

for _ in range(100):
    x = torch.randn(1, 3, 224, 224)
    out = model(x)
    print(torch.isnan(out).any())


Add gaussian noise to parameters while training

Question:
I tried to add gaussian noise to the parameters using the code below but the network won’t converge. Any though why? I used cifar10 dataset with lr=0.001
 import torch.nn as nn
 import torch.nn.functional as F
 import torch
 __all__ = ['simplenet_cifar']

 38 class Simplenet(nn.Module):
 37     def __init__(self):
 36         super(Simplenet, self).__init__()
 35         self.conv1 = nn.Conv2d(3, 6, 5)
 34         self.relu_conv1 = nn.ReLU()
 33         self.pool1 = nn.MaxPool2d(2, 2)
 32         self.conv2 = nn.Conv2d(6, 16, 5)
 31         self.relu_conv2 = nn.ReLU()
 30         self.pool2 = nn.MaxPool2d(2, 2)
 29         self.fc1 = nn.Linear(16 * 5 * 5, 120)
 28         self.relu_fc1 = nn.ReLU()
 27         self.fc2 = nn.Linear(120, 84)
 26         self.relu_fc2 = nn.ReLU()
 25         self.fc3 = nn.Linear(84, 10)
 24
 23         self.noise_conv1 = torch.randn(nn.Parameter(self.conv1.weight).size())*0.6 + 0
 22         self.noise_conv2 = torch.randn(nn.Parameter(self.conv2.weight).size())*0.6 + 0
 21
 20     def forward(self, x):
 19         x = self.conv1(x)
 18         self.conv1.weight =  add_noise(nn.Parameter(self.conv1.weight), self.noise_conv1)
 17         x = self.pool1(self.relu_conv1(x))
 16         x = self.conv2(x)
 15         self.conv2.weight =  add_noise(nn.Parameter(self.conv2.weight), self.noise_conv2)
 14         x = self.pool2(self.relu_conv2(x))
 13         # x = self.pool1(self.relu_conv1(self.conv1(x)))
 12         # x = self.pool2(self.relu_conv2(self.conv2(x)))
 11         x = x.view(-1, 16 * 5 * 5)
 10         x = self.relu_fc1(self.fc1(x))
  9         x = self.relu_fc2(self.fc2(x))
  8         x = self.fc3(x)
  7         return x
  6
  5 def add_noise(weights, noise):
  4     with torch.no_grad():
  3         weight_noise = nn.Parameter(weights + noise.to("cuda"))
  1     return weight_noise

Context: Injecting noise into the model might act as a regularizer, but note that your current noise is static and you would most likely want to resample it in each forward pass. I don’t know how large the stddev should be to work properly.


Question:
I want to use the trained model in C++ with libtorch.
However, it failed when I want to load a gpu model
using torch::jit::script::Module;
Module module = torch::jit::load("resnetGPU.pt", torch::kCUDA);

The gpu model is trained in linux, and the pytorch version is: 1.9.0+cu111, and the libtorch version is libtorch-win-shared-with-deps-debug-1.9.0+cu111.
The c++ project is built in win10, and the vs version is vs2022.
The code to generate the model is:
import torch
import torchvision.models as models
model = models.resnet50(pretrained=True).cuda()
model=model.eval()
resnet = torch.jit.trace(model, torch.rand(1, 3, 224, 224).cuda())
resnet.save('resnetGPU.pt')

Any suggestion is appreciated~~

Context:
It’s a little hard to tell what’s going on without the error logs.  Could you copy/paste the error message here?
My best guess is that the cuda/torch DLL files aren’t getting found when the executable runs.  Make sure they’re on your system path.


infenrence with python and c++ got the different answer

Question:
I trained a mobilenetv3 model with cpu and test it. Although both python and c++ gives the right answer, but their output tensors were different.
python output tensors:
tensor([[-6.8434,  7.5787,  2.4852,  2.0013,  3.2962, -7.6010, 10.4773, -0.4331,
          1.4424,  0.0946, -1.4620, -1.4271, -3.0626, -0.4510, -6.5796, -3.5845,
         -8.1056]], grad_fn=<ViewBackward>)

c++ output tensors:
Columns 1 to 8 -5.9655   1.8631   2.1338   2.5918   1.9550  -7.2201  10.4710   1.2880
Columns 9 to 16 -1.9018   1.3437  -3.1746   2.3237  -0.1883   2.3990  -5.5839  -4.7663
Columns 17 to 17 -8.7208
[ CPUFloatType{1,17} ]

why is this happenning?
Besides, I also tested the inference speed of python and c++. pytorch costs 0.21s while libtorch costs 0.23s. What makes libtorch slower than pytorch? What can I do to solve this problem?
My python training, inference program and my c++ inference program are as follows:
python training:
class MobileNetV3_large(nn.Module):
    # (out_channels,kernel_size,exp_channels,stride,se,nl)
    cfg=[
        (16,3,16,1,False,'RE'),
        (24,3,64,2,False,'RE'),
        (24,3,72,1,False,'RE'),
        (40,5,72,2,True,'RE'),
        (40,5,120,1,True,'RE'),
        (40,5,120,1,True,'RE'),
        (80,3,240,2,False,'HS'),
        (80,3,200,1,False,'HS'),
        (80,3,184,1,False,'HS'),
        (80,3,184,1,False,'HS'),
        (112,3,480,1,True,'HS'),
        (112,3,672,1,True,'HS'),
        (160,5,672,2,True,'HS'),
        (160,5,960,1,True,'HS'),
        (160,5,960,1,True,'HS')
    ]
    def __init__(self,num_classes=17):
        super(MobileNetV3_large,self).__init__()
        self.conv1=nn.Conv2d(3,16,3,2,padding=1,bias=False)
        self.bn1=nn.BatchNorm2d(16)

        self.layers = self._make_layers(in_channels=16)
        self.conv2=nn.Conv2d(160,960,1,stride=1,bias=False)
        self.bn2=nn.BatchNorm2d(960)

        self.conv3=nn.Conv2d(960,1280,1,1,padding=0,bias=True)
        self.conv4=nn.Conv2d(1280,num_classes,1,stride=1,padding=0,bias=True)

    def _make_layers(self,in_channels):
        layers=[]
        for out_channels,kernel_size,exp_channels,stride,se,nl in self.cfg:
            layers.append(
                Bottleneck(in_channels,out_channels,kernel_size,exp_channels,stride,se,nl)
            )
            in_channels=out_channels
        return nn.Sequential(*layers)

    def forward(self,x):
        out=Hswish(self.bn1(self.conv1(x)))
        out=self.layers(out)
        out=Hswish(self.bn2(self.conv2(out)))
        out=F.avg_pool2d(out,7)
        out=Hswish(self.conv3(out))
        out=self.conv4(out)

        a,b=out.size(0),out.size(1)
        out=out.view(a,b)
        return out

MAX_EPOCH=50
BATCH_SIZE=64
LR=0.0001
log_interval=3
val_interval=1

split_dir=os.path.join(".","data","splitData")
train_dir=os.path.join(split_dir,"train")
valid_dir=os.path.join(split_dir,"valid")

train_transform=transforms.Compose([
    transforms.Resize((224,224)),
    transforms.ToTensor(),
])

valid_transform=transforms.Compose([
    transforms.Resize((224,224)),
    transforms.ToTensor(),
])

train_data=flowerDataset(data_dir=train_dir,transform=train_transform)
valid_data=flowerDataset(data_dir=valid_dir,transform=valid_transform)

train_loader=DataLoader(dataset=train_data,batch_size=BATCH_SIZE,shuffle=True)
valid_loader=DataLoader(dataset=valid_data,batch_size=BATCH_SIZE)

net=MobileNetV3_large(num_classes=17)
if torch.cuda.is_available():
    net.cuda()

criterion=nn.CrossEntropyLoss()

optimizer=optim.Adam(net.parameters(),lr=LR, betas=(0.9, 0.99))

train_curve=list()
valid_curve=list()
net.train()
accurancy_global=0.0
for epoch in range(MAX_EPOCH):
    loss_mean=0.
    correct=0.
    total=0.
    running_loss = 0.0

    for i,data in enumerate(train_loader):
        img,label=data
        img = Variable(img)
        label = Variable(label)
        if torch.cuda.is_available():
            img=img.cuda()
            label=label.cuda()

        out=net(img)
        optimizer.zero_grad()
        loss=criterion(out,label)
        print_loss=loss.data.item()

        loss.backward()
        optimizer.step()
        if (i+1)%log_interval==0:
            print('epoch:{},loss:{:.4f}'.format(epoch+1,loss.data.item()))
        _, predicted = torch.max(out.data, 1)
        total += label.size(0)
        correct += (predicted == label).sum()
    print("============================================")
    accurancy=correct / total
    if accurancy>accurancy_global:
        torch.save(net.state_dict(), './weights/best.pkl')
        print("准确率由:", accurancy_global, "上升至:", accurancy, "已更新并保存权值为weights/best.pkl")
        accurancy_global=accurancy
    print('第%d个epoch的识别准确率为:%d%%' % (epoch + 1, 100*accurancy))
torch.save(net.state_dict(), './weights/last.pkl')
print("训练完毕，权重已保存为:weights/last.pkl")

python inference:
class Detector(object):
    def __init__(self,net_kind,num_classes=17):
        super(Detector, self).__init__()
        self.net = torch.jit.load(r"E:\my_files\python_program\python_project\MobileNetV3\jitmodel.pth")

        self.net.eval()
        if torch.cuda.is_available():
            self.net.cuda()

    def load_weights(self,weight_path):
        self.net.load_state_dict(torch.load(weight_path))

    def detect(self,weight_path,pic_path):
        img = cv2.imread(pic_path)
        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

        img = cv2.resize(img, (244,244))
        trans = transforms.ToTensor()
        img_tensor = trans(img)
        print(img_tensor.shape)
        img_tensor = torch.unsqueeze(img_tensor, 0)

        if torch.cuda.is_available():
            img_tensor=img_tensor.cuda()
        print(img_tensor.shape)
        net_output = self.net(img_tensor)
        print(net_output)
        _, predicted = torch.max(net_output.data, 1)
        result = predicted[0].item()
        print(result)

if __name__=='__main__':
    detector=Detector('large',num_classes=17)
    start=time.time()
    detector.detect('./weights/best.pkl','./6.jpg')
    end=time.time()
    print(end-start)

c++ inference:
void Classfier(cv::Mat &image)
{
  torch::Tensor img_tensor = torch::from_blob(
      image.data, {1, image.rows, image.cols, 3}, torch::kByte);
  img_tensor = img_tensor.permute({0, 3, 1, 2});
  img_tensor = img_tensor.toType(torch::kFloat);
  img_tensor = img_tensor.div(255);
  torch::jit::script::Module module =
      torch::jit::load("/home/peter-233/project/torchtest/jitmodel.pth");
  torch::Tensor output = module.forward({img_tensor}).toTensor();
  std::cout << output << std::endl;
  auto max_result = output.max(1, true);
  auto max_index = std::get<1>(max_result).item<float>();
  std::cout << max_index << std::endl;
}

int main()
{
  cv::Mat image = cv::imread("/home/peter-233/project/torchtest/6.jpg");
  cv::resize(image, image, cv::Size(224, 224));
  cv::cvtColor(image, image, cv::COLOR_BGR2RGB);
  high_resolution_clock::time_point beginTime = high_resolution_clock::now();
  Classfier(image);
  high_resolution_clock::time_point endTime = high_resolution_clock::now();
  milliseconds timeInterval = std::chrono::duration_cast<milliseconds>(endTime - beginTime);
  cout << "Running Time：" << timeInterval.count() << "ms" << endl;
  return 0;
}

```jsx

Context:
You could try to use a static input (e.g. torch.ones) and compare the outputs between the Python frontend and libtorch. If these values differ by a larger error it would mean that the model itself is not performing the same operations and you should look into the forward definition to isolate the difference.
On the other hand, if the outputs are equal for a static input, the issue would most likely be in the data loading and preprocessing.


CNN cannot overfit even with large number of epochs

Question:
Hello!
I have the following code for classification on 2x128x128 pictures, which cannot even overfit at all, however loss tends to zero when applying huge number of epochs (~500).
In case of classification between 15 classes: accuracy on training set goes up until only 60%, accuracy on test set is around 58%.
While in case of 5 classes: these values are 84 and 75 respectively.
(Still cannot overfit even with many epochs and using no regularization.)
I am pretty sure I’m making a rookie mistake, but could not figure it out yet.
Architecture:
nn.Sequential(
        nn.Conv2d(2, 17, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
        nn.BatchNorm2d(17, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
        nn.LeakyReLU(negative_slope=0.01, inplace=True),
        nn.Conv2d(17, 32, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1)),
        nn.BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
        nn.LeakyReLU(negative_slope=0.01, inplace=True),
        nn.Conv2d(32, 48, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
        nn.BatchNorm2d(48, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
        nn.LeakyReLU(negative_slope=0.01, inplace=True),
        nn.Conv2d(48, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1)),
        nn.BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
        nn.LeakyReLU(negative_slope=0.01, inplace=True),
        nn.Conv2d(64, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
        nn.BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
        nn.LeakyReLU(negative_slope=0.01, inplace=True),
        nn.Conv2d(96, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1)),
        nn.BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
        nn.LeakyReLU(negative_slope=0.01, inplace=True),
        nn.Conv2d(128, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
        nn.BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
        nn.LeakyReLU(negative_slope=0.01, inplace=True),
        nn.Conv2d(192, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1)),
        nn.BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
        nn.LeakyReLU(negative_slope=0.01, inplace=True),
        nn.Conv2d(256, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
        nn.BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
        nn.LeakyReLU(negative_slope=0.01, inplace=True),
        nn.Conv2d(192, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1)),
        nn.BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
        nn.LeakyReLU(negative_slope=0.01, inplace=True),
        nn.AdaptiveAvgPool2d(output_size=2),
        nn.Flatten(start_dim=1, end_dim=-1),
        nn.Dropout(p=0.2, inplace=False),
        nn.Linear(in_features=512, out_features=256, bias=True),
        nn.LeakyReLU(negative_slope=0.01, inplace=True),
        nn.Dropout(p=0.2, inplace=False),
        nn.Linear(in_features=256, out_features=numClasses, bias=True),
        nn.LogSoftmax(dim=1))

Training:
def TrainModel(model, numEpochs, lossFunction, optimizer, dataSource, saveToFolder, scheduler=None):
epochLossSum = 0.0
counter = 0.0
for epoch in range(numEpochs):
    print("Epoch: ", epoch+1)

    for i, batch in enumerate(dataSource, 0):
        inputs = batch['input']
        labels = batch['label']

        outputs = model(inputs)
        loss = lossFunction(outputs, labels)
        epochLossSum = epochLossSum + loss
        counter = counter + 1
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        if scheduler is not None:
            scheduler.step()

    print("Average loss in epoch ", epoch + 1)
    print('%.10f' % (epochLossSum/counter))
    epochLossSum = 0.0
    counter = 0.0

print('Finished Training')
torch.save(model.state_dict(), saveToFolder)
print('Model saved to ', saveToFolder)

Accuracy check:
def ModelAccuracy(model, trainingData, validationData):
goodTrain = 0.0
badTrain = 0.0
goodVal = 0.0
badVal = 0.0
for i, batch in enumerate(validationData, 0):
    inputs = batch['input']
    labels = batch['label']
    inputs = inputs.unsqueeze(0)
    labels = labels.unsqueeze(0)
    prediction = np.argmax(model(inputs).detach().numpy(), axis=1)
    if(labels.item() == prediction[0]):
        goodVal = goodVal + 1
    else:
        badVal = badVal + 1
print("Accuracy on validation set: ", 100*goodVal/(goodVal+badVal), "%")

for i, batch in enumerate(trainingData, 0):
    inputs = batch['input']
    labels = batch['label']
    inputs = inputs.unsqueeze(0)
    labels = labels.unsqueeze(0)
    prediction = np.argmax(model(inputs).detach().numpy(), axis=1)
    if(labels.item() == prediction[0]):
        goodTrain = goodTrain + 1
    else:
        badTrain = badTrain + 1

print("Accuracy on training set: ", 100*goodTrain/(goodTrain+badTrain), "%")

And putting these together:
if __name__ == "__main__":

classifier = DHMClassifier(n_classes=numClasses)
classifier = classifier.to(torch.device("cuda"))
classifier = classifier.train()

trainingData = DataAssembler(dataSource, train=True, onCuda=True)
trainingLoader = DataLoader(trainingData, batch_size=10, num_workers=0)

optimizer = optim.Adam(classifier.parameters(), lr=0.0001)
lossFunction = nn.NLLLoss(weight=trainingData.GetBalancerCUDA())
TrainModel(classifier, 500, lossFunction, \
    optimizer, trainingLoader, savingDirectory, scheduler=None)

classifier = classifier.eval()
validationData = DataAssembler(dataSource, train=False, onCuda=False)
EvaluatedModel = DHMClassifier(n_classes=numClasses)
EvaluatedModel.load_state_dict(torch.load(savingDirectory))

EvaluatedModel = EvaluatedModel.to(torch.device("cpu"))
trainingData.DictToCPU()
ModelAccuracy(EvaluatedModel, trainingData, validationData)

Any help or hints are really appreciated. (I know that some code parts are not very efficient, but that shouldn’t cause problem I believe.)

Context:
Unrelated to this issue, but you are accumulating the loss tensor and would thus store the entire computation graph with it here:
epochLossSum = epochLossSum + loss

If epochLossSum is only used for printing purposes, call .detach() on loss before accumulating it.
I can’t see any obvious mistake in your code, so would recommend to try to overfit a small dataset (e.g. just 10 samples) and check if your model is able to do so by playing around with some hyperparameters.


Custom loss function even when just handing down MSELoss: expected float, got double. how is that possible

Question:
I am currently trying to get another loss function to work, but I always get the same error:
Found dtype Double but expected Float
But how is that possible?
class MSLELoss(nn.Module):
    def __init__(self):
          super().__init__()
          self.mse = nn.MSELoss()

      def forward(self, pred, actual):
          return self.mse(pred,actual)

criterion = nn.MSELoss() works fine while
criterion = MSLELoss() does not…
What am I missing?
Thanks in advance!

Context:
I cannot reproduce the issue and both approaches work as expected:
criterion = nn.MSELoss()

x = torch.randn(10, 10)
y = torch.randn(10, 10).double()

loss = criterion(x, y)

class MSLELoss(nn.Module):
    def __init__(self):
        super().__init__()
        self.mse = nn.MSELoss()

    def forward(self, pred, actual):
        return self.mse(pred,actual)

my_criterion = MSLELoss()
loss = my_criterion(x, y)


Question:
Hi,
I want to implement single class classification(output should be just single probability number). is there any example? or guide ?
I did multi-class classification, but I couldn’t modify it to single class… (if I use single class, training always shows loss: 0)

Context:
A single-class classification wouldn’t make much sense, since your model wouldn’t learn anything and will just output a high probability for the single class.
A “perfect” classifier would thus be:
def predict():
    return 0

as no other class indices are valid.
Assuming you are dealing with a binary classification, you could treat it as a 2-class multi-class classification (i.e. with nn.CrossEntropyLoss as the criterion) and use 2 output values or alternatively you could use a single output neuron and use nn.BCEWithLogitsLoss. The lower the output value of this neuron the more likely class0, the higher the more likely class1 was detected.


Question:
i need to extract the feature by ignoring the layer of classification … so I tried this code but I think it is wrong
class Identity(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return x

PATH = 'model.pth'
model = Model()
model.load_state_dict(torch.load(PATH),strict=False)
model.head = Identity()

the structure of the model is
       (1): TransformerBlock(
          dim=768, input_resolution=(7, 7), num_heads=24, window_size=7, shift_size=0, mlp_ratio=4.0
          (pool_layers): ModuleList()
          (norm1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
          (attn): WindowAttention(
            dim=768, window_size=(7, 7), num_heads=24
            (qkv): Linear(in_features=768, out_features=2304, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=768, out_features=768, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (softmax): Softmax(dim=-1)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=768, out_features=3072, bias=True)
            (act): GELU()
            (fc2): Linear(in_features=3072, out_features=768, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
      )
    )
  )
  (norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
  (avgpool): AdaptiveAvgPool1d(output_size=1)
  (head): Identity()

Context:
logy:

so I tried this code but I think it is wrong

I couldn’t see any obvious pointer to determine this code as wrong. How did you find out that this code is wrong?


Question:
Hey there,
I intend to build a densenet121-based multi-head CNN. My goal is to use the first three denseblocks as a shared set of layers and then create multiple branches using the architecture of the fourth denseblock.
My code is below:
class Densenet121_v1(nn.Module):
    def __init__(self, num_classes = [1,4,2,8,3]):
        super(Densenet121_v1,self).__init__()
        original_model = torchvision.models.densenet121(pretrained=True)
        self.num_classes=num_classes

        self.trunk=original_model.features[:-2]

        self.branch1=original_model.features[-2:]
        self.branch2=original_model.features[-2:]
        self.branch3=original_model.features[-2:]
        self.branch4=original_model.features[-2:]
        self.branch5=original_model.features[-2:]

        #self.features=nn.Sequential(*self.trunk,*self.branch)
        #self.features = nn.Sequential(*list(original_model.children())[:-1])

        self.classifier1=(nn.Linear(1024, self.num_classes[0]))
        self.classifier2=(nn.Linear(1024, self.num_classes[1]))
        self.classifier3=(nn.Linear(1024, self.num_classes[2]))
        self.classifier4=(nn.Linear(1024, self.num_classes[3]))
        self.classifier5=(nn.Linear(1024, self.num_classes[4]))

    def forward(self, x):

        # shared trunk
        trunk=self.trunk(x)

        #block1:
        f1 = self.branch1(trunk)
        b1 = F.relu(f1, inplace=False)
        b1 = F.adaptive_avg_pool2d(b1, (1, 1)).view(f1.size(0), -1)
        y1 = self.classifier1(b1)

        #block2:
        f2 = self.branch2(trunk)
        b2 = F.relu(f2, inplace=False)
        b2 = F.adaptive_avg_pool2d(b2, (1, 1)).view(f2.size(0), -1)
        y2 = self.classifier2(b2)

        #block3:
        f3 = self.branch3(trunk)
        b3 = F.relu(f3, inplace=False)
        b3 = F.adaptive_avg_pool2d(b3, (1, 1)).view(f3.size(0), -1)
        y3 = self.classifier3(b3)

        #block4:
        f4 = self.branch4(trunk)
        b4 = F.relu(f4, inplace=False)
        b4 = F.adaptive_avg_pool2d(b4, (1, 1)).view(f4.size(0), -1)
        y4 = self.classifier4(b4)

        #block5:
        f5 = self.branch5(trunk)
        b5 = F.relu(f5, inplace=False)
        b5 = F.adaptive_avg_pool2d(b5, (1, 1)).view(f5.size(0), -1)
        y5 = self.classifier5(b5)

        #order predictions
        output= torch.cat((y1,y2[:,0:3],torch.unsqueeze(y4[:,0],1),torch.unsqueeze(y2[:,3],1),
                            y4[:,1:3],torch.unsqueeze(y5[:,0],1),
                            torch.unsqueeze(y3[:,0],1),torch.unsqueeze(y4[:,3],1),
                            torch.unsqueeze(y3[:,1],1),y4[:,4:7],
                            torch.unsqueeze(y5[:,1],1),torch.unsqueeze(y4[:,7],1),
                            torch.unsqueeze(y5[:,2],1)),1)

        return output

model=Densenet121_v1()
out=model(torch.rand(16,3,224,224))

When I inspect the computational graph using tensorboard I see that some tensors resulting from batchnorm layers in branch1 are being fed into the remaining branches. This does not happen with the remaining branches:

final1920×872 62.5 KB

How can I avoid this? The heads should be completely independent of each other.

Context:
SofiaCP:

Either way, would something like this work during backprop? The idea is that during backprop the common trunk is updated by all losses but the subbranches are only updated by the losses of their outputs.

Yes this would work. For example, the following work seems to follow this paradigm.
ensemblenet: end-to-end optimization of multi-headed models 3

 SofiaCP:

I’m not sure how to do that

This might help.

      gist.github.com

    https://gist.github.com/InnovArul/871cf866b737b306eb073f3c37a6a8fd 2

  reset_params.py
  import torchvision, copy
import torch, torch.nn as nn

def reset_all_weights(model: nn.Module) -> None:
    """
    refs:
        - https://discuss.pytorch.org/t/how-to-re-set-alll-parameters-in-a-network/20819/6
        - https://stackoverflow.com/questions/63627997/reset-parameters-of-a-neural-network-in-pytorch
        - https://pytorch.org/docs/stable/generated/torch.nn.Module.html
    """
    This file has been truncated. show original


Apply different Transform (Data Augmentation) to Train and Validation

Question:
My dataset folder is prepared as Train Folder and Test Folder. When I conduct experiments, I further split my Train Folder data into Train and Validation.
However, transform is applied before my split and they are the same for both my Train and Validation. My question is how to apply a different transform in this case?
Transoform Code:
data_transform = transforms.Compose([
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
    ])

Dataset Code
train_data = datasets.ImageFolder(base_path + '/train/',
                                           transform=data_transform)

Train and Validation Split and Loader Code
# obtain training indices that will be used for validation
num_train = len(train_data)
indices = list(range(num_train))
np.random.shuffle(indices)
split = int(np.floor(valid_size * num_train))
train_idx, valid_idx = indices[split:], indices[:split]

# define samplers for obtaining training and validation batches
train_sampler = SubsetRandomSampler(train_idx)
valid_sampler = SubsetRandomSampler(valid_idx)

# prepare data loaders
train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size,
    sampler=train_sampler, num_workers=num_workers)
valid_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size,
    sampler=valid_sampler, num_workers=num_workers)

Context:
I think a good solution can be found here: Changing transforms after creating a dataset - #7 by Brando_Miranda 9
train_dataset = MyDataset(train_transform)
val_dataset = MyDataset(val_transform)
train_indices, val_indices = sklearn.model_selection.train_test_split(indices)
train_dataset = torch.utils.data.Subset(train_dataset, train_indices)
val_dataset = torch.utils.data.Subset(val_dataset, val_indices)


Question:
In Pytorch, I cannot find a straightforward possibility to do a convolution (nn.conv2d) with periodic boundary conditions. For example, take the tensor
[[1,2,3],
[4,5,6],
[7,8,9]]
and any 3x3 filter. A convolution with periodic boundary conditions could in principle be done by doing a periodic padding to 5x5
[[9,7,8,9,7],
[3,1,2,3,1],
[6,4,5,6,4],
[9,7,8,9,7],
[3,1,2,3,1]]
and subsequently a convolution with the filter in “padding=0” mode. Unfortunately, I cannot find a padding method that supports this periodic padding.
Is there a simple walkaround?

Context: You can use nn.ReflectionPad2d 151 for that, or the functional interface F.pad 58


How to adjust SmoothL1Loss for particular need

Question:
Hello,
I have been experimenting with L1 and MSE losses in Pytorch and noticing that L1Loss performs better because it is more robust to outliers. I discovered SmoothL1Loss which seems to be the best of both worlds. I understand that it behaves like MSELoss for error<1 and like L1Loss otherwise. My dataset only contains values between 0 and 1. Therefore the largest possible error is 1. Does this mean the function behaves identical to MSELoss? Is it possible adjust the threshold in anyway to work better for my problem?
Regards,

Context:
Hi,
The boundary between “like MSELoss” and “L1Loss” is defined by the beta factor (to the best of my knowledge).


Shape mismatch while implementing VGG11*: Runtime Error

Question:
img_rows, img_cols = 64, 64
in_channels = 3
batch_size = 1
input_batch = (batch_size, in_channels, img_rows, img_cols)

filter_amt = 64
kernel_size = 3
pool_size = 2
num_output_classes = 200
padding = 1
stride = 1

pool_dim = (pool_size, pool_size)
conv_filter = (kernel_size, kernel_size)

import torch
import torch.nn as nn

class VGG9(nn.Module):
    def __init__(self, in_channels, num_output_classes):
        super(VGG9, self).__init__()
        self.in_channels = in_channels
        self.num_classes = num_output_classes

        self.conv_layers = nn.Sequential(
            nn.Conv2d(in_channels = self.in_channels, out_channels = 64, kernel_size = kernel_size, stride = stride, padding = padding),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size = 2, stride = 2),
            nn.Conv2d(64, 128, kernel_size = kernel_size, stride = stride, padding = padding),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size = 2, stride = 2),
            nn.Conv2d(128, 256, kernel_size = kernel_size, stride = stride, padding = padding),
            nn.ReLU(),
            nn.Conv2d(128, 256, kernel_size = kernel_size, stride = stride, padding = padding),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size = 2, stride = 2),
            nn.Conv2d(256, 512, kernel_size = kernel_size, stride = stride, padding = padding),
            nn.ReLU(),
            nn.Conv2d(512, 512, kernel_size = kernel_size, stride = stride, padding = padding),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size = 2, stride = 2)
        )

        self.linear_layers = nn.Sequential(
            nn.Linear(in_features = 512 * 8 * 8, out_features = 4096), # 64/(2^3) - 4 'blocks', each increasing the output by a power of 2, except the first block
            nn.ReLU(),
            nn.Dropout2d(0.25),
            nn.Linear(in_features = 4096, out_features = 4096),
            nn.ReLU(),
            nn.Dropout2d(0.25),
            nn.Linear(in_features = 4096, out_features = self.num_classes)
        )

    def forward(self, x):
        x = self.conv_layers(x)
        x = x.view(x.size(0), -1)
        x = self.linear_layers(x)
        return x

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
vgg9 = VGG9(in_channels = in_channels, num_output_classes = num_output_classes).to(device)
total_params = sum(p.numel() for p in vgg9.parameters())
print(total_params)

image_tensor = torch.randn(input_batch).to(device)
outputs = vgg9(image_tensor)
print(outputs.shape)

21818952
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-14-21c3dc6880f6> in <module>()
      5
      6 image_tensor = torch.randn(input_batch).to(device)
----> 7 outputs = vgg9(image_tensor)
      8 print(outputs.shape)

6 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    441                             _pair(0), self.dilation, self.groups)
    442         return F.conv2d(input, weight, bias, self.stride,
--> 443                         self.padding, self.dilation, self.groups)
    444
    445     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Given groups=1, weight of size [256, 128, 3, 3], expected input[1, 256, 16, 16] to have 128 channels, but got 256 channels instead

Context:
The shape mismatch is raised in these layers:
            nn.Conv2d(128, 256, kernel_size = kernel_size, stride = stride, padding = padding),
            nn.ReLU(),
            nn.Conv2d(128, 256, kernel_size = kernel_size, stride = stride, padding = padding),

as the former conv layer creates an output activation with 256 channels while the latter expects 128 input channels.


Is there an automatic way to analyze the bottleneck of the model

Question:
Hello!
This is a simple but general problem. Recently I was training a deep learning model using Pytorch, but it seems that no matter how I modified the layer number, channels or layer output size, the training loss stop improving after  hundreds of iterations. Changing the optimizing algorithm did not help much. I wonder is there an automatic way which can help me analyze which layer is the bottleneck that affects the decrease of training loss or model parameter updating?
Thanks for your help!

Context:
Hi.
The answer to your question is very complex. You are assuming that there is a layer which is the bottleneck of your model, but in fact many different factors can affect your training. here are just a few examples:

Data - your dataset may hide a limit on the success of your model. if your dataset contains errors in labeling for example, or if it is very difficult your network will never be able to reach accuracy over the errors in the dataset.
Weight decay - if your optimizer is using weight decay, you may have set it too high. That means that you are limiting you model from learning the data better. Decreasing that wd could of course, cause overfitting, but that is another issue.
Learning rate - when you have reached the plato in loss, you may want do decrease you learning rate in order to allow your model to decent into lower minima. Consider using a learning rate scheduler and decrease the lr when you reach the plato.

many other factors could affect the model training. if you are using a known model and dataset, look for the training recipe for this model, it will give you a good start.


Why all out of a sudden google colab runs out of memory?

Question:
@ ptrblck I apologize in advance . I know I should not mention anyone specific and wait for the comunity to answer but I am  in a In a tight spot. It would mean alot   if you could help out.
So I have been trying to extract the features of a  some pictures for a research case study .  when trying to load  an image from dataset_loader_ and try to use the model.forward() method for 34 iteration on my dataset_loader_object. this happens even using the cpu instead of gpu:(in this case google colab just crashes) I get  a

CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 11.17 GiB total capacity; 10.62 GiB already allocated; 832.00 KiB free; 10.66 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I know this is indicating that I have ran out of cuda memory And here  is the model . and the forward decleration :
class vgg16(torch.nn.Module):
    def __init__(self, pretrained=True):
        super(vgg16, self).__init__()
        vgg_pretrained_features = tv.vgg16(pretrained=pretrained).features
        self.layerss = torch.nn.Sequential()

        for x in range(30):
            self.layerss.add_module(str(x), vgg_pretrained_features[x])
        self.layerss.eval()

    def forward(self, x):
      output=[]

      for i,layer in enumerate( self.layerss):
        # print (i)
        x=layer(x)
        max_=[img_.max() for chanel in x for img_ in chanel]
        min_=[img_.min() for chanel in x for img_ in chanel]
        output.append(max_+min_)
      return output
model=vgg16()
model.to(device)

But the out of memory erore disapears when I Replace:
        max_=[img_.max() for chanel in x for img_ in chanel]
        min_=[img_.min() for chanel in x for img_ in chanel]

whit this
        x=layer(x)
        max_=[img_.max().detach().numpy() for chanel in x for img_ in chanel]
        min_=[img_.min().detach().numpy() for chanel in x for img_ in chanel]

but why is this happening ? **Is a tensor obj really  that large that  34 containers of type tensor can suck  up the space of 10GIB in just matters of seconds ?Am i using the model wrong ?? it’s worth mentioning that the output of the  forward function is a list  of lists  containing  tensors
and I used this function to **monitor the cuda memory :
``
def mem_report():
  print("CPU RAM Free: " + humanize.naturalsize( psutil.virtual_memory().available ))

  GPUs = GPUtil.getGPUs()
  for i, gpu in enumerate(GPUs):
    print('GPU {:d} ... Mem Free: {:.0f}MB / {:.0f}MB | Utilization {:3.0f}%'.format(i, gpu.memoryFree, gpu.memoryTotal, gpu.memoryUtil*100))

and as mentioned everything looks fine until the for loop : here is the loop code
features=[]
dataset_ =tid2008_dataloader(path,image_name_dir,first_time=True)
start=time.time()
for batch_idx,data in enumerate(dataset_):
  data = data.to(device=device)
  features.append (model.forward(data))
  del data

It’s worth mentioning that the images  are the size of 384 * 512*3

Context:
In your code you are appending the output of the forward method to features which will not only append the output tensor but the entire computation graph with it. Since you are iterating the entire dataset_ your memory usage would then grow in each iteration until you could be running out of memory.
I don’t know what you are doing with features, but the common approach is to train the model using a mini-batch, which would free the computation graph with its intermediate tensors, instead of accumulating the outputs of the entire dataset and potentially call backward() once per epoch.


Image. fromarray() throws up error TypeError: Cannot handle this data type

Question:
I am trying to make a custom dataset, But as I convert it to PIL image, I am Getting the error:
TypeError: Cannot handle this data type

> class Gen_Data(Dataset):
> >         def __init__(self, p, transform = None):
> >                 super(Gen_Data,self).__init__()
> >                 self.data = []
> >                 self.transform = transform
> >
> >                 for v in p:
> >                         (self.data).append(v)
> >
> >         def __getitem__(self, index):
> >                 img = np.asarray(self.data[index],dtype=np.uint8)
> >                 print (img)
> >                 print (type(img))
> >                 img = Image.fromarray(img.astype('uint8'))
> >
> >                 if self.transform is not None:
> >                         img  = self.transform(img)
> >
> >                 return (img)
> >
> >         def __len__(self):
> >                 return len(self.data)

Where p is a numpy nd array.
I am not quite understanding, how do I overcome this issue.
Traceback (most recent call last):
**  File “/usr/local/torch3/lib/python3.5/site-packages/PIL/Image.py”, line 2428, in fromarray**
**    mode, rawmode = _fromarray_typemap[typekey]**
KeyError: ((1, 1, 64), ‘|u1’)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
**  File “pytorch_CelebA_DCGAN.py”, line 343, in **
**    print (inception_score.inception_score(our_dataset, cuda=True, batch_size=32, resize=True, splits=10))**
**  File “/home/bansa01/spectral_normalization/DCGAN-Pytorch2/DCGAN/inception_score.py”, line 50, in inception_score**
**    for i, batch in enumerate(dataloader, 0):**
**  File “/usr/local/torch3/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 259, in next**
**    batch = self.collate_fn([self.dataset[i] for i in indices])**
**  File “/usr/local/torch3/lib/python3.5/site-packages/torch/utils/data/dataloader.py”, line 259, in **
**    batch = self.collate_fn([self.dataset[i] for i in indices])**
**  File “pytorch_CelebA_DCGAN.py”, line 37, in getitem**
**    img = Image.fromarray(img.astype(‘uint8’))**
**  File “/usr/local/torch3/lib/python3.5/site-packages/PIL/Image.py”, line 2431, in fromarray**
**    raise TypeError(“Cannot handle this data type”)**
TypeError: Cannot handle this data type

Context: Have you handled this problem?


Question:
In below code “tensor_a” is one dimensional tensor with all value 1. From “tensor_b” i want to find out the index for which all the channel values are greater than 100 and “3rd channel” value should be  greater than value of “1st channel” and “2nd channel”. That index value in “tensor_a” should be replace with 0.
In below code, “tensor_a” first value satisfy that condition(as first value of each channel is 160,185,210 respectively) .so b[0,0,0,0] should be 0.
b=torch.ones(1,1,3,3)
a=torch.FloatTensor([[[[160, 56, 20],
          [54, 6, 97],
          [65, 119, 56]],

         [[185, 13, 90],
          [10, 6, 220],
          [67, 88, 4]],

         [[210, 92, 74],
          [48, 217, 86],
          [42, 12, 24]]]])

Context:
This should work:
a = torch.tensor([[[[160, 56, 20],
                    [54, 6, 97],
                    [65, 119, 56]],

                   [[185, 13, 90],
                    [10, 6, 220],
                    [67, 88, 4]],

                   [[210, 92, 74],
                    [48, 217, 86],
                    [42, 12, 24]]]]).float()

idx = (a[:, 2] > a[:, 0]) & (a[:, 2] > a[:, 1]) & (a[:, 2] > 100.)
result = idx.float()
print(result)
> tensor([[[1., 0., 0.],
           [0., 1., 0.],
           [0., 0., 0.]]])


How to copy a predefined nn.Sequential properly and make it trainable?

Question:
I want to copy a pre-defined nn.Sequential() like the torchvision ResNet layer1 and make a new module that is trainable, without going through the pain of using _make_layer function. Is there an easy and safe way to do this?
Something like this:
class MyModel():
    def __init__(self):
        model = torchvision.models.resnet18(pretrained=True)
        self.layer1 = model.layer1
        self.layer11 = safe_copy_way(self.layer1)

    def forward(self, x):
        x = self.layer1(x)
        x = self.layer11(x)
        return x

Context: Yes, you can reassign modules as seen in your code snippet. You would have to change the attribute names of course to valid ones and you could use copy.deecopy for safe_copy_way.


Question:
As you can see I need to use this kind of data during my training process, so IO has been a huge problem.
I also use DDP, which makes me tougher to adjust my code.
Is there a way in pytorch can handle this problem? Help me, please.

Context: What kind of IO problem are you seeing?


Question:
Hello Everyone,
I am new to pyotorch and learning quickly.
I have a dataset of images with 5 category/classes with 2 label.
For prediction of those 2 labels I plan to do an custom ensemble approach.
I want to train separate but same network(densenet-121) for each category/classes. And then the resulting model would predict out of these 2 label. I could not find a similar example for pytorch, If anyone has anyidea or could give me an concrete example would be very helpful.
For example: let’s say I have X,Y,Z,P,Q classes who have label 0, 1
Now I want train total 5 networks. densenet-121_X with the data of X, densenet-121_Y, with the data of Y and so on.
And finally somehow combine those trained model to output a single prediction of 0 or 1 given any test data from X,Y,Z,P,Q classes.  I cant not figure out how to do it and not sure if it is feasible too.
I saw this example Combining Trained Models in PyTorch 470 which is somewhat similar to my idea. But in the end instead of passing two input, I would like to pass one input to get single prediction either 0 or 1 .
Similar to bagging apporach:

bagging.png719×389 76.1 KB

instead of test sample there would images from each category
Thanks a lot in advance for help.

Context:
You could train the 5 classifiers separately as binary classifiers and try to combine them afterwards as shown in your figure.
For this you could pass all pretrained models to the linked ensemble model (maybe remove the last classification layer of each model), and add a classifier head on top of their concatenated outputs.
Would that work or are you thinking about another approach?


Question:
Hello, I am new to PyTorch and I want to make a classifier for 3D DICOM MRIs. I want to use the pretrained resnet18 from monai library but I am confused with the input dimensions of the tensor. The shape of the images in my dataloader is [2,160,256,256] where 2 is the batch_size, 160 is the number of dicom images for each patient and 256x256 is the dimension of the images. When I try to run the model I get this error: Expected 5-dimensional input for 5-dimensional weight [64, 3, 7, 7, 7], but got 4-dimensional input of size [2, 160, 256, 256] instead
If I unsqueeze the tensor before feeding it to the model I get: Given groups=1, weight of size [64, 3, 7, 7, 7], expected input[1, 2, 160, 256, 256] to have 3 channels, but got 2 channels instead Can anybody help me figure this out ?

Context:
Hi gkrisp!

 gkrisp9:

Do you think it’s better to use a resnet with 2d convolutions and override the first layer to have input of 160 channels

It would almost certainly be worse, if I understand correctly that you
are working with 3d images.
The point is that your images have substantive spatial structure.
That is, just as a pixel with x = 17 is right next to x = 18, but far
away from x  = 148, a slice in your 3d “z-stack” with z = 17 is
right next to the z = 18 z-slice, but far away from z = 148.
Convolutions know about and respect this structure, while a general
fully-connected layer does not (and the in_channels → out_channels
part of a convolutional layer is fully connected).
You could use a Conv2d with in_channels = 160 (and with all of
those extra connections, it would be a superset of the Conv3d so,
in principal, could perform the same inference if properly trained),
but such a network would have to “learn” the spatial structure of your
z dimension, which could take a long time or not happen at all with
realistic amounts of training data and time.
As a general rule, if your problem has some particular structure and
there is a natural way of “telling” your network about that structure
by building that structure into the architecture, you’re much better off
doing so, rather than forcing your network to “learn” that structure.

model.features[0] = nn.Conv2d(160,64,7,7,7)

As an aside, if your model were pretrained, so, in particular,
model.features[0] were pretrained, doing this would throw away
that part of the pretraining, which could be costly.

 KFrank:

Your first error was caused because your input didn’t have the nBatch
dimension. This dimension can have size one if you want, but it has to
be present

The input shape is [2,160,256,256] where theoretically 2 is the nBatch.

Yes, my mistake.  You said in your first post that 2 is your batch size.
The basic point still stands, however.  Pytorch’s convolutional layers
require both a batch and a channels dimension, even if they are
“trivial,” singleton (that is, size = 1) dimensions.
So, if your input image has shape [2, 160, 256, 256], with nBatch = 2,
and no explicit channels dimension, you have to add the required
channel dimension, e.g.:
image = image.unsqueeze (1)

However, now you have nChannels = 1, and your pretrained resnet
requires nChannels = 3, so probably the best solution (if you want to
keep the benefits of your pretrained model) is to expand() your input
to have 3 (redundant) channels:
image = image.unsqueeze (1).expand (-1, 3, -1, -1, 1)

Now image will have shape [2, 3, 160, 256, 256].
(Note that expand() just provides a logical view into your 1-channel
image – the 2 additional channels don’t actually “exist” in the sense
of being stored in memory or being able to have values that differ from
those of the one “real” channel.)
Best.
K. Frank


Slicing mobilenet V3 matrix shape mistmatch

Question:
I have been trying to obtain the output of all but last linear layer of Mobilenet v3. The last module in small mobilnetv3 is a Sequential one:
print(mobilenet_v3_small.classifier)
Sequential(
  (0): Linear(in_features=576, out_features=1024, bias=True)
  (1): Hardswish()
  (2): Dropout(p=0.2, inplace=True)
  (3): Linear(in_features=1024, out_features=1000, bias=True)
)

I would like only the first Linear layer and all the preceding network. Here’s my attempt:
class mobilenetv3(nn.Module):
    def __init__(self):
        super().__init__()
        mobilenet_v3_small = models.mobilenet_v3_small(pretrained=True)
        self.mobilenet_preclassifier = nn.Sequential(mobilenet_v3_small.features,
                                                     mobilenet_v3_small.avgpool)
        self.mobilenet_linear = nn.Sequential(mobilenet_v3_small.classifier[:1])

    def forward(self, x):
        x = self.mobilenet_preclassifier(x)
        x = self.mobilenet_linear(x)
        return x

But I keep getting the following error:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (576x1 and 576x1024).
Even if I simply assemble the complete mobilenetv3 module by module (without slicing the classifier layer), the error remains the same. So, same error is seen  if I do:
class mobilenetv3(nn.Module):
    def __init__(self):
        super().__init__()
        self.m = nn.Sequential(mobilenet_v3_small.features,
                               mobilenet_v3_small.avgpool,
                               mobilenet_v3_small.classifier)

    def forward(self, x):
        x = self.m(x)
        return x

This points to the fact that I’m somehow assembling the modules incorrectly leading to the issue. Would be grateful if someone can point to the appropriate way to do it and maybe explain the problem with my approach. Thanks in advance.

Context:
Wrapping submodules into nn.Sequential containers often fails, as you might be missing functional API calls used in the forward method.
In your case, this torch.flatten operation is missing and will yield the shape mismatch.
You could either add it as an nn.Flatten to your nn.Sequential model or derive a custom class and override the forward method.


Question:
Hi everyone - really would appreciate your help on this. I’ve been trying to speed up my resnet (basically resnet34, I customized it to expand the features by adding an additional block at the end). To this end, I added nn.DataParallel as a wrapper for my model. The code below worked entirely fine prior to the additions of nn.DataParallel (and using model.cuda() and inputs.cuda() vs to(device) which I used prior). The first run of this model, which is done with only the final output layer and batchnorm layers unfrozen, works fine. I then take this model and unfreeze all weights in all layers of the network, and run this model through the training loop once again. This produces the following error. I’m really not sure how to fix this, although I’m guessing it likely has to do with using the previously trained model and putting that through another training loop (inside of which I again call model.cuda()). I’ve pasted as much of my code as I thought would be relevant, including the training loop and calls to partial unfreezing. Would really appreciate your help and thanks again!
Traceback (most recent call last):
File “/wynton/protected/home/ichs/dmandair/BRCA/train_val.py”, line 395, in
trained_model = train_model(trained_model, criterion, optimizer_ft, exp_lr_scheduler, dataloader_dict, device, NUM_EPOCHS)
File “/wynton/protected/home/ichs/dmandair/BRCA/train_val.py”, line 155, in train_model
outputs = model(inputs)
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1102, in _call_impl
return forward_call(*input, **kwargs)
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.
9/site-packages/torch/nn/parallel/data_parallel.py”, line 168, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py”, line 178, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py”, line 86, in parallel_apply
output.reraise()
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torch/_utils.py”, line 434, in reraise
raise exception
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py”, line 61, in _worker
output = module(*input, **kwargs)
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1102, in _call_impl
return forward_call(*input, **kwargs)
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py”, line 168, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.
9/site-packages/torch/nn/parallel/data_parallel.py”, line 178, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py”, line 86, in parallel_apply
output.reraise()
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torch/_utils.py”, line 434, in reraise
raise exception
RuntimeError: Caught RuntimeError in replica 1 on device 1.
Original Traceback (most recent call last):
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py”, line 61, in _worker
output = module(*input, **kwargs)
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1102, in _call_impl
return forward_call(*input, **kwargs)
File “/wynton/protected/home/ichs/dmandair/BRCA/train_val.py”, line 222, in forward
x = self.base(x)
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1102, in _call_impl
return forward_call(*input, **kwargs)
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.
9/site-packages/torch/nn/modules/container.py”, line 141, in forward
input = module(input)
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1102, in _call_impl
return forward_call(*input, **kwargs)
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torch/nn/modules/conv.py”, line 446, in forward
return self._conv_forward(input, self.weight, self.bias)
File “/wynton/protected/home/ichs/dmandair/anaconda3/envs/ONC_EXP/lib/python3.9/site-packages/torch/nn/modules/conv.py”, line 442, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument weight in method wrapper__cudnn_convolution)
def train_model(model, criterion, optimizer, scheduler, dataloaders, device, num_epochs=4):
     since = time.time()
     model = nn.DataParallel(model, device_ids=[0, 1, 2, 3])
     model = model.cuda()
     best_model_wts = copy.deepcopy(model.state_dict())
     best_acc = 0.0
     for epoch in range(num_epochs):
          print('Epoch {}/{}'.format(epoch, num_epochs - 1))
          print('-' * 10)
    # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
             if phase == 'train':
                 model.train()  # Set model to training mode
             else:
                 model.eval()  # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels, _, _ in dataloaders[phase]:
                inputs = inputs.cuda()
                labels = labels.cuda()
                # zero the parameter gradients
                optimizer.zero_grad()
            # forward
            # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        torch.nn.utils.clip_grad_norm_(model.parameters(), MAX_NORM)
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
            if phase == 'train':
                scheduler.step()

                epoch_loss = running_loss / len(train_tile_ds)
                epoch_acc = running_corrects.double() / len(train_tile_ds)
            else:
                epoch_loss = running_loss / len(val_tile_ds)
                epoch_acc = running_corrects.double() / len(val_tile_ds)

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

def get_new_base(orig_model):
    return nn.Sequential(*list(orig_model.children())[:-2])

def create_new_block(in_channel, out_channel):
    conv_layer = nn.Conv2d(in_channel, out_channel, kernel_size=(1, 1), stride=(1, 1), bias=False)
    batch_layer = nn.BatchNorm2d(out_channel, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    relu = nn.ReLU(inplace=True)
    average_pool = nn.AdaptiveAvgPool2d(output_size=(1, 1))
    return [conv_layer, batch_layer, relu, average_pool]

class CustomResNet(nn.Module):
    def __init__(self, out_c, out_feat):
        super().__init__()
        base = models.resnet34(pretrained=True)

        base_channels_op = base.fc.in_features
        self.base = get_new_base(base)
        self.new_layers = nn.Sequential(*create_new_block(base_channels_op, out_c))
        self.final = nn.Linear(out_c, out_feat)

    def forward(self, x):
        x = self.base(x)
        x = self.new_layers(x)
        x = torch.flatten(x,1)
        return self.final(x)

def set_grad(m, b):
    if isinstance(m, (nn.Linear,nn.BatchNorm2d)): return
    if hasattr(m, 'weight'):
        for p in m.parameters(): p.requires_grad_(b)

train_loader = DataLoader(train_tile_ds, batch_size=BS,
                                       shuffle = True, num_workers=4)
validation_loader = DataLoader(val_tile_ds, batch_size=BS, shuffle = False, num_workers=4)

dataloader_dict = {'train': train_loader, 'val': validation_loader}

mod_resnet = CustomResNet(768, 2)

mod_resnet.apply(partial(set_grad, b=False))

criterion = nn.CrossEntropyLoss()

optimizer_ft = optim.SGD([{'params': mod_resnet.base.parameters(), 'lr': 1e-2},
            {'params': mod_resnet.new_layers.parameters(), 'lr': 1e-3},
            {'params': mod_resnet.final.parameters(), 'lr': 1e-3}
        ], momentum=0.9)

exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

trained_model = train_model(mod_resnet, criterion, optimizer_ft, exp_lr_scheduler, dataloader_dict, device, NUM_EPOCHS)

optimizer_ft = optim.SGD([{'params': trained_model.module.base.parameters(), 'lr': 1e-3},
            {'params': trained_model.module.new_layers.parameters(), 'lr': 1e-3},
            {'params': trained_model.module.final.parameters(), 'lr': 1e-3}
        ], momentum=0.9)

trained_model = train_model(trained_model, criterion, optimizer_ft, exp_lr_scheduler, dataloader_dict, device, NUM_EPOCHS)

Context:
Check the .device attribute of the model parameters as well as the input data in the forward method of your model and try to isolate which tensor creates the device mismatch.
Once this is done, check how this tensor/parameter is created and if you are manually pushing this tensor to a specific device instead of letting nn.DataParallel handle it.


Legality of adding new layers to built-in faster rcnn with forward hooks?

Question:
I am looking for a way to add new conv2d and relu layers in between the backbone and RPN of Faster R-CNN. I have already created a custom nn.module for new layers. Moreover, I discovered the existence of forward hooks that is called after each forward pass. So I thought that I can get the output activations of the last layer of the backbone of resnet and feed these activations to my new module and get activations from it so that I can append these new activations to input of RPN.
Is this a valid way to modify such kinds of networks?
If it is a valid way, does my new nn.module get gradient updates as usual?
Thank in advance for any help.

Context: Nock Nock. Is there anyone out there to have an idea about this?


Can neural network outputs discrete values[0:255]?

Question:
Hello,
My network predicts the values of missing pixels of a grayscale image. However, since the image is normalized, the network gives the normalized values between 0 and 1 with float32 datatype. I guess if the network could output discrete values in [0,1,2,…,255], it would be much more simpler and faster. Should I just put the threshold function that discretizes the float32 output? Do you have any ideas?

Context:
Converting from “tensor” images in [0,1] to unsigned bytes “PIL” images is a fairly standard transformation (e.g., just multiply by 255 and cast) as in to_pil_image:
torchvision.transforms.functional — Torchvision main documentation.
Alternatively, if you wanted a “discrete” output, you could frame it as a 255-class classification problem per pixel.


Question:
I am trying to use an inner class as my nn but it seems that the superclass does not recognize the name. Does anybody have any solutions for this?
this is the snippet of code that is getting an error:
class FashionMNISTmodel:
    class FashionNet(nn.Module):
        def __init__(self, activationfcn):
            super(FashionNet,self).__init__()

This is where I initialize the inner class:
        self.net = self.FashionNet(activationfcn = self.actfcn).to(device)

this is the error i’m getting:
NameError: name ‘FashionNet’ is not defined

Context:
This works for me:
class FashionMNISTmodel:
    def __init__(self):
        self.net = self.FashionNet(torch.relu)

    class FashionNet(nn.Module):
        def __init__(self, activationfcn):
            super().__init__()

model = FashionMNISTmodel()
print(model)
print(model.net)


Debugging Broken UNet Implementation - Can’t Overfit

Question:
I read through a related thread 1 and didn’t quite get to an answer so I thought I’d try and raise this issue on my own:
I tried implementing a UNet for image-to-image learning.  To debug it I’m trying to overfit on a single example.  It learns most of the image perfectly well, but the area where the convolutions seem to overlap in the final image are encountering problems.  I’m not sure where my issue is.

chrome_2021-12-12_15-28-27546×836 266 KB

One can see the top (output now) versus the bottom (as it should be).
I have a full implementation and the related training code uploaded here: GitHub - JosephCatrambone/UNet
But here is the relevant model architecture:
class DoubleConv(nn.Module):
	def __init__(self, in_channels: int, out_channels: int):
		super(DoubleConv, self).__init__()
		self.op = nn.Sequential(
			# No bias 'cause we're using BatchNorm.  It will get cancelled out.
			nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=False),
			nn.BatchNorm2d(out_channels),
			nn.LeakyReLU(inplace=True),

			nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=False),
			nn.BatchNorm2d(out_channels),
			#nn.LeakyReLU(inplace=True),
		)

	def forward(self, x):
		return self.op(x)

class UNet(nn.Module):
	def __init__(self, in_channels:int = 3, out_channels: int = 3, feature_counts=None):
		super(UNet, self).__init__()

		# Prevent modification of mutable default.
		if feature_counts is None:
			feature_counts = [64, 128, 256, 512]

		# Need nn.ModuleList instead of List for batch evals.
		self.downsamples = nn.ModuleList()
		self.bottleneck = DoubleConv(feature_counts[-1], feature_counts[-1]*2)
		self.upsamples = nn.ModuleList()
		self.finalconv = nn.Conv2d(feature_counts[0], out_channels, kernel_size=1)  # 1x1 conv -> Change # feats.
		self.pool = nn.MaxPool2d(kernel_size=2, stride=2)

		# Downsample-Reduction step.
		num_channels = in_channels
		for f_count in feature_counts:
			self.downsamples.append(DoubleConv(in_channels=num_channels, out_channels=f_count))
			num_channels = f_count

		# Up-steps.
		for f_count in reversed(feature_counts):
			self.upsamples.append(nn.ConvTranspose2d(f_count*2, f_count, kernel_size=2, stride=2))
			self.upsamples.append(DoubleConv(f_count*2, f_count))

	def forward(self, x):
		skip_connections = list()  # Don't need ModuleList because this is not retained.
		for dwn in self.downsamples:
			x = dwn(x)
			skip_connections.append(x)
			x = self.pool(x)

		x = self.bottleneck(x)
		skip_connections.reverse()

		for idx in range(0, len(self.upsamples), 2):
			x = self.upsamples[idx](x)
			sk = skip_connections[idx//2]

			# It's possible that due to integer division the sizes slightly mismatch.
			if x.shape != sk.shape:
				x = torch.nn.functional.interpolate(x, size=sk.shape[2:])

			assert len(sk.shape) == 4  # So we don't accidentally unpinch another dimension.
			concat_skip = torch.cat((sk, x), dim=1)  # Dim 1 is channel-dimension.  [b, c, h, w]
			x = self.upsamples[idx+1](concat_skip)

		return self.finalconv(x)

My best guesses for the source of the issue are something to do with the upscaling and the concatenation – maybe I’m joining along the wrong axis.  Or, more likely, there’s something wrong with the padded convolution, but I can’t seem to make it work without the same-size conv2d.
Any input or debugging tips would be appreciated.

Context: I guess the issue is caused by the upsampling layers, which are using transposed convs as well as plain nn.Conv2d layers with overlapping kernels. I would probably try to either remove the nn.Conv2d layers or make sure the kernel do not overlap.


Should model be in train mode when saving it?

Question:
When saving a model like the following snip of code, does it then matter if the model is in train() or eval() mode? Does train() and eval() change the model.state_dict()? And if so, should model.train() be called before saving the model?
for epoch in range(5):
    # prepare model for training
    model.train()
    for i, (local_batch, local_labels) in enumerate(training_generator):
        ....

    # prepare model for evaluation
    model.eval()
    for i, (local_batch, local_labels) in enumerate(validation_generator):
        ....

    torch.save(model.state_dict(),  os.path.join('models', f'model_{epoch+1:03}.pt'))

Context:
Kalle:

Does train() and eval() change the model.state_dict()?

No, it doesn’t. Calling model.train() or model.eval() will change the internal self.training flag to True or False, respectively. This flag is then used in some modules (e.g. nn.Dropout and batchnorm layers) so switch the behavior in their forward method.


Yolov5 - Different result from saved augmentation data and use ‘augment’ = True

Question:
Hi there,
When I train yolov5 pytorch on my custom dataset, I tried an experiment.
Ep1: Firstly, I train my dataset (folder Dataset1) with args ‘agment’ = True. Then I save all image augmentation and label files when it create_dataLoader to Dataset2 folder (with train/val split like Dataset1).
Ep2: Next, I trained model with this augmentation dataset (Dataset2) and set up ‘augment’=False.
This problem is, the Ep1 (training with ‘augment’ = True) got the better results than Ep2 which new augment data I saved. I get map@.5 = 0.978 on Ep1 and map@.5=0.685 on Ep2 .I don’t know why results are different, whereas I checked results between two approaches many times.
Below is result I training Dataset2.
Thanks a lot <3

results2400×1200 247 KB

Context: Based on this usage 1 of the augment argument, the data and labels are augmented in each iteration and thus epoch. If you are randomly transforming the data only once and load it afterwards, you wouldn’t use any data augmentation anymore and are feeding static input to the model.


How to split tensors with overlap and then reconstruct the original tensor?

Question: I encountered a problem. My network is trained with tensors of size BxCx128x128, but I need to verify its image reconstruction performance with images of size 1024x1024. To make the reconstruction smooth, I need to split my input of size BxCx1024x1024 into BxCx128x128  tensors with overlap, which are then fed to the network for reconstruction. Then, the reconstructed tensors of size BxCx128x128 should be used for reconstructing the tensor of size BxCx1024x1024 by averaging the overlaping elements. Note that (Size(in)-128)/stride may be not a integer. How to use padding strategy to ensure that every element is cropped at least for one time? How to implement the process of recovering the BxCx1024x1024 tensor from overlapping BxCx128x128 tensors? Could anyone give me some suggestions? Thanks advance for your consideration.

Context:
fold should work in your use case.
Here is a small example creating the expected input shape step by step:
B, C, W, H = 2, 3, 1024, 1024
x = torch.randn(B, C, H, W)

kernel_size = 128
stride = 64
patches = x.unfold(3, kernel_size, stride).unfold(2, kernel_size, stride)
print(patches.shape) # [B, C, nb_patches_h, nb_patches_w, kernel_size, kernel_size]

# perform the operations on each patch
# ...

# reshape output to match F.fold input
patches = patches.contiguous().view(B, C, -1, kernel_size*kernel_size)
print(patches.shape) # [B, C, nb_patches_all, kernel_size*kernel_size]
patches = patches.permute(0, 1, 3, 2)
print(patches.shape) # [B, C, kernel_size*kernel_size, nb_patches_all]
patches = patches.contiguous().view(B, C*kernel_size*kernel_size, -1)
print(patches.shape) # [B, C*prod(kernel_size), L] as expected by Fold
# https://pytorch.org/docs/stable/nn.html#torch.nn.Fold

output = F.fold(
    patches, output_size=(H, W), kernel_size=kernel_size, stride=stride)
print(output.shape) # [B, C, H, W]

Let me know, if this works for you.