Repository: Lightning-Universe/lightning-flash

Issue:
Default InputTransform for video classification provide no normalization
## 🐛 Bug

Normalization does not work.

### To Reproduce

```bash
tensor([[[[[157., 157., 158.,  ..., 168., 171., 173.],
           [157., 157., 158.,  ..., 192., 189., 187.],
           [158., 158., 159.,  ..., 201., 214., 217.],
           ...,
           [ 81.,  81.,  82.,  ...,  90.,  91.,  93.],
           [ 81.,  81.,  82.,  ...,  88.,  87.,  87.],
           [ 81.,  81.,  82.,  ...,  87.,  86.,  86.]],

          [[158., 157., 154.,  ..., 177., 177., 177.],
           [159., 158., 156.,  ..., 177., 177., 178.],
           [159., 158., 157.,  ..., 178., 178., 180.],
           ...,
           [ 83.,  83.,  83.,  ..., 114., 112., 107.],
           [ 83.,  83.,  83.,  ..., 113.,  98.,  93.],
           [ 83.,  83.,  83.,  ..., 232., 211., 207.]],

          [[160., 160., 159.,  ..., 112., 102.,  86.],
           [166., 166., 166.,  ..., 116., 103.,  86.],
           [175., 176., 175.,  ..., 114., 105.,  86.],
           ...,
           [ 81.,  81.,  83.,  ...,  93.,  92.,  93.],
           [ 82.,  83.,  83.,  ..., 108., 104., 103.],
           [ 82.,  82.,  82.,  ...,  99.,  97.,  97.]],

          ...,

```


#### Code sample

```python
import os
from argparse import ArgumentParser

from torch.utils.data.sampler import RandomSampler

import flash
from flash.core.finetuning import NoFreeze
from flash.core.data.utils import download_data
from flash.video import VideoClassificationData, VideoClassifier

if __name__ == '__main__':
    parser = ArgumentParser()
    parser.add_argument('--seed', type=int, default=1234)
    parser.add_argument('--backbone', type=str, default="x3d_xs")
    parser.add_argument('--download', type=bool, default=True)
    parser.add_argument('--train_folder', type=str, default=os.path.join(os.getcwd(),
                        "./data/kinetics/train"))
    parser.add_argument('--val_folder', type=str, default=os.path.join(os.getcwd(),
                        "./data/kinetics/val"))
    parser.add_argument('--predict_folder', type=str, default=os.path.join(os.getcwd(),
                        "./data/kinetics/predict"))
    parser.add_argument('--max_epochs', type=int, default=1)
    parser.add_argument('--learning_rate', type=float, default=1e-3)
    parser.add_argument('--gpus', type=int, default=None)
    parser.add_argument('--fast_dev_run', type=int, default=False)
    args = parser.parse_args()


    if args.download:
        # Dataset Credit:Download a video clip dataset.
        # Find more datasets at https://pytorchvideo.readthedocs.io/en/latest/data.html
        download_data("https://pl-flash-data.s3.amazonaws.com/kinetics.zip",
                      os.path.join(os.getcwd(), "data/"))

    datamodule = VideoClassificationData.from_folders(
        train_folder=args.train_folder,
        val_folder=args.val_folder,
        predict_folder=args.predict_folder,
        batch_size=8,
        clip_sampler="uniform",
        clip_duration=2,
        video_sampler=RandomSampler,
        decode_audio=False,
        num_workers=2,
    )

    for batch in datamodule.train_dataloader():
       print(batch['video'])
       break
 
    model = VideoClassifier(backbone=args.backbone, num_classes=datamodule.num_classes, pretrained=False)

    trainer = flash.Trainer(max_epochs=args.max_epochs, gpus=args.gpus, strategy='ddp', fast_dev_run=args.fast_dev_run)
    trainer.finetune(model, datamodule=datamodule, strategy=NoFreeze())
```

### Expected behavior

The input values to the model should be in range [0-1].

### Environment
 
 - OS (e.g., Linux): Linux
 - Python version: 3.8
 - PyTorch/Lightning/Flash Version (e.g., 1.10/1.5/0.7): 
 - GPU models and configuration: 
 - Any other relevant information:

### Additional context

By default [VideoClassInputTransform](https://github.com/PyTorchLightning/lightning-flash/blob/master/flash/video/classification/input_transform.py#L35) performs necessary preprocessing step. One step that is missing is dividing by 255. i.e. adding `Lambda(lambda x: x/255.)` function.

I can create a PR with a quick fix.



Discussion:
Hey @dudeperf3ct Thanks for reporting this! Definitely happy to have a PR that resolves this :smiley: I think the `per_batch_transform_on_device` is expected to apply a normalization (which won't have happened until after the batch is converted to the device in the `on_after_batch_transfer` hook of the DataModule). Possibly a simple fix that might make things cleaner would be to replace the `per_batch_transform_on_device` for just `per_batch_transform` which will happen inside the dataloder.

We may _also_ need the divide by 255 depending on what the normalization transform expects.
@ethanwharris do you mean the [default input transformations](https://github.com/PyTorchLightning/lightning-flash/blob/master/flash/video/classification/input_transform.py) provide input values in range [0-1] to the model after they are moved to particular device or we have to add some normalization function to take care of it? 


@dudeperf3ct It looks like we need an additional transform to scale the input values to (0, 1), we do then have the normalization after that assumes it's input to be in (0, 1). So if it isn't then that's a bug. Would welcome a PR to fix it :smiley:

Resolving patch:
diff --git a/flash/video/classification/input_transform.py b/flash/video/classification/input_transform.py
--- a/flash/video/classification/input_transform.py
+++ b/flash/video/classification/input_transform.py
@@ -30,6 +30,10 @@
     ClipSampler, LabeledVideoDataset, EncodedVideo, ApplyTransformToKey = None, None, None, None
 
 
+def normalize(x: torch.Tensor) -> torch.Tensor:
+    return x / 255.0
+
+
 @requires("video")
 @dataclass
 class VideoClassificationInputTransform(InputTransform):
@@ -48,7 +52,8 @@ def per_sample_transform(self) -> Callable:
             per_sample_transform = [CenterCrop(self.image_size)]
 
         return ApplyToKeys(
-            "video", Compose([UniformTemporalSubsample(self.temporal_sub_sample)] + per_sample_transform)
+            "video",
+            Compose([UniformTemporalSubsample(self.temporal_sub_sample), normalize] + per_sample_transform),
         )
 
     def per_batch_transform_on_device(self) -> Callable:


Test patch: