Package: video-transformers
Easiest way of fine-tuning HuggingFace video classification models.🚀 Featuresvideo-transformersuses:🤗acceleratefor distributed training,🤗evaluatefor evaluation,pytorchvideofor dataloadingand supports:creating and fine-tunining video models usingtransformersandtimmvision modelsexperiment tracking withneptune,tensorboardand other trackersexporting fine-tuned models inONNXformatpushing fine-tuned models intoHuggingFace Hubloading pretrained models fromHuggingFace HubAutomatedGradio app, andspacecreation🏁 InstallationInstallPytorch:condainstallpytorch=1.11.0torchvision=0.12.0cudatoolkit=11.3-cpytorchInstall pytorchvideo and transformers from main branch:pipinstallgit+https://github.com/facebookresearch/pytorchvideo.git
pipinstallgit+https://github.com/huggingface/transformers.gitInstallvideo-transformers:pipinstallvideo-transformers🔥 UsagePrepare video classification dataset in such folder structure (.avi and .mp4 extensions are supported):train_rootlabel_1video_1video_2...label_2video_1video_2......
val_rootlabel_1video_1video_2...label_2video_1video_2......Fine-tune Timesformer (from HuggingFace) video classifier:fromtorch.optimimportAdamWfromvideo_transformersimportVideoModelfromvideo_transformers.backbones.transformersimportTransformersBackbonefromvideo_transformers.dataimportVideoDataModulefromvideo_transformers.headsimportLinearHeadfromvideo_transformers.trainerimporttrainer_factoryfromvideo_transformers.utils.fileimportdownload_ucf6backbone=TransformersBackbone("facebook/timesformer-base-finetuned-k400",num_unfrozen_stages=1)download_ucf6("./")datamodule=VideoDataModule(train_root="ucf6/train",val_root="ucf6/val",batch_size=4,num_workers=4,num_timesteps=8,preprocess_input_size=224,preprocess_clip_duration=1,preprocess_means=backbone.mean,preprocess_stds=backbone.std,preprocess_min_short_side=256,preprocess_max_short_side=320,preprocess_horizontal_flip_p=0.5,)head=LinearHead(hidden_size=backbone.num_features,num_classes=datamodule.num_classes)model=VideoModel(backbone,head)optimizer=AdamW(model.parameters(),lr=1e-4)Trainer=trainer_factory("single_label_classification")trainer=Trainer(datamodule,model,optimizer=optimizer,max_epochs=8)trainer.fit()Fine-tune ConvNeXT (from HuggingFace) + Transformer based video classifier:fromtorch.optimimportAdamWfromvideo_transformersimportTimeDistributed,VideoModelfromvideo_transformers.backbones.transformersimportTransformersBackbonefromvideo_transformers.dataimportVideoDataModulefromvideo_transformers.headsimportLinearHeadfromvideo_transformers.necksimportTransformerNeckfromvideo_transformers.trainerimporttrainer_factoryfromvideo_transformers.utils.fileimportdownload_ucf6backbone=TimeDistributed(TransformersBackbone("facebook/convnext-small-224",num_unfrozen_stages=1))neck=TransformerNeck(num_features=backbone.num_features,num_timesteps=8,transformer_enc_num_heads=4,transformer_enc_num_layers=2,dropout_p=0.1,)download_ucf6("./")datamodule=VideoDataModule(train_root="ucf6/train",val_root="ucf6/val",batch_size=4,num_workers=4,num_timesteps=8,preprocess_input_size=224,preprocess_clip_duration=1,preprocess_means=backbone.mean,preprocess_stds=backbone.std,preprocess_min_short_side=256,preprocess_max_short_side=320,preprocess_horizontal_flip_p=0.5,)head=LinearHead(hidden_size=neck.num_features,num_classes=datamodule.num_classes)model=VideoModel(backbone,head,neck)optimizer=AdamW(model.parameters(),lr=1e-4)Trainer=trainer_factory("single_label_classification")trainer=Trainer(datamodule,model,optimizer=optimizer,max_epochs=8)trainer.fit()Fine-tune Resnet18 (from HuggingFace) + GRU based video classifier:fromvideo_transformersimportTimeDistributed,VideoModelfromvideo_transformers.backbones.transformersimportTransformersBackbonefromvideo_transformers.dataimportVideoDataModulefromvideo_transformers.headsimportLinearHeadfromvideo_transformers.necksimportGRUNeckfromvideo_transformers.trainerimporttrainer_factoryfromvideo_transformers.utils.fileimportdownload_ucf6backbone=TimeDistributed(TransformersBackbone("microsoft/resnet-18",num_unfrozen_stages=1))neck=GRUNeck(num_features=backbone.num_features,hidden_size=128,num_layers=2,return_last=True)download_ucf6("./")datamodule=VideoDataModule(train_root="ucf6/train",val_root="ucf6/val",batch_size=4,num_workers=4,num_timesteps=8,preprocess_input_size=224,preprocess_clip_duration=1,preprocess_means=backbone.mean,preprocess_stds=backbone.std,preprocess_min_short_side=256,preprocess_max_short_side=320,preprocess_horizontal_flip_p=0.5,)head=LinearHead(hidden_size=neck.hidden_size,num_classes=datamodule.num_classes)model=VideoModel(backbone,head,neck)Trainer=trainer_factory("single_label_classification")trainer=Trainer(datamodule,model,max_epochs=8)trainer.fit()Perform prediction for a single file or folder of videos:fromvideo_transformersimportVideoModelmodel=VideoModel.from_pretrained(model_name_or_path)model.predict(video_or_folder_path="video.mp4")>>[{'filename':"video.mp4",'predictions':{'class1':0.98,'class2':0.02}}]🤗 Full HuggingFace IntegrationPush your fine-tuned model to the hub:fromvideo_transformersimportVideoModelmodel=VideoModel.from_pretrained("runs/exp/checkpoint")model.push_to_hub('model_name')Load any pretrained video-transformer model from the hub:fromvideo_transformersimportVideoModelmodel=VideoModel.from_pretrained("runs/exp/checkpoint")model.from_pretrained('account_name/model_name')Push your model to HuggingFace hub with auto-generated model-cards:fromvideo_transformersimportVideoModelmodel=VideoModel.from_pretrained("runs/exp/checkpoint")model.push_to_hub('account_name/app_name')(Incoming feature) Push your model as a Gradio app to HuggingFace Space:fromvideo_transformersimportVideoModelmodel=VideoModel.from_pretrained("runs/exp/checkpoint")model.push_to_space('account_name/app_name')📈 Multiple tracker supportTensorboard tracker is enabled by default.To add Neptune/Layer ... tracking:fromvideo_transformers.trackingimportNeptuneTrackerfromaccelerate.trackingimportWandBTrackertrackers=[NeptuneTracker(EXPERIMENT_NAME,api_token=NEPTUNE_API_TOKEN,project=NEPTUNE_PROJECT),WandBTracker(project_name=WANDB_PROJECT)]trainer=Trainer(datamodule,model,trackers=trackers)🕸️ ONNX supportConvert your trained models into ONNX format for deployment:fromvideo_transformersimportVideoModelmodel=VideoModel.from_pretrained("runs/exp/checkpoint")model.to_onnx(quantize=False,opset_version=12,export_dir="runs/exports/",export_filename="model.onnx")🤗 Gradio supportConvert your trained models into Gradio App for deployment:fromvideo_transformersimportVideoModelmodel=VideoModel.from_pretrained("runs/exp/checkpoint")model.to_gradio(examples=['video.mp4'],export_dir="runs/exports/",export_filename="app.py")ContributingBefore opening a PR:Install required development packages:pipinstall-e."[dev]"Reformat with black and isort:python-mtests.run_code_styleformat