Dense Video Captioning Using Pytorch | by Raymond Cheng | Towards Data Science

The deep learning task, Video Captioning, has been quite popular in the intersection of Computer Vision and Natural Language Processing for the last few years. In particular, Dense Video Captioning, which is a subfield, has been gaining some traction among researchers. Dense Video Captioning is the task of localizing interesting events from an untrimmed video and producing individual textual description for each event.
Dense Video Captioning is challenging as it requires a strong contextual representation of the video, as well as being able to detect localized events. Most models tackle this problem by decomposing it into two steps: detecting event proposals from the video and then, generate sentences for each event. The current state-of-the-art algorithm Bi-modal Transformer with Proposal Generator (BMT) proposes to combine two channels of inputs in order to conduct dense video captioning: visual and audio information. It achieves state-of-the-art performance on ActivityNet Captions Dataset, which consists of thousands of videos paired with captions associated with certain timeframes.
The BMT architecture consists of three main components: Bi-modal Encoder, Bi-modal Decoder, and finally the Proposal Generator.
First, the audio and visual of a video is encoded using VGG and I3D, respectively. After feature extraction, the VGG and I3D features are passed to the bi-modal encoder layers where audio and visual features are encoded to what the paper calls as, audio-attended visual and video-attended audio. These features are then passed to the proposal generator, which takes in information from both modalities and generates event proposals.
After the event proposal generation, the video is trimmed according to the event proposals. Each short clip is passed into the entire again, starting from feature extraction. Audio and visual features are extracted, passed into the bi-modal encoder, then to the bi-modal decoder layers. Here, the decoder layers take in two inputs: the outputs of the last layer from the bi-modal encoder and also, the GloVe embeddings of the last generated caption sequence. Finally, the decoder decodes the internal representation and generates the next word based on the probability distribution, which is added to the previous caption sequence.
In this article, we will show you how to use pre-trained BMT to perform dense video captioning given a video. No model training is needed.
Download the paper’s official repository via:
git clone --recursive https://github.com/v-iashin/BMT.gitcd BMT/
Download VGG and I3D model and also GloVe embeddings. The script will save them in the ./data and ./.vector_cache folder.
bash ./download_data.sh
Setup a conda environment with all the required libraries and dependencies:
conda env create -f ./conda_env.ymlconda activate bmtpython -m spacy download en
Now, you can get the video that you want. As an example, I will get a short documentary about the recent global coronavirus pandemic from Youtube. I got this one:
You can download Youtube videos using online downloaders, but please use it carefully and at your own risk! After downloading it, you can save it somewhere that you like. I created a test folder under the BMT project folder and copied the downloaded video to the test folder.
mkdir test# copied video to the test directory
After getting the video, now it’s time to extract I3D features by first creating the conda environment and then running the python script:
cd ./submodules/video_featuresconda env create -f conda_env_i3d.ymlconda activate i3dpython main.py \    --feature_type i3d \    --on_extraction save_numpy \    --device_ids 0 \    --extraction_fps 25 \    --video_paths ../../test/pandemic.mp4 \    --output_path ../../test/
Extract VGGish features using a similar procedure:
conda env create -f conda_env_vggish.ymlconda activate vggishwget https://storage.googleapis.com/audioset/vggish_model.ckpt -P ./models/vggish/checkpointspython main.py \    --feature_type vggish \    --on_extraction save_numpy \    --device_ids 0 \    --video_paths ../../test/pandemic.mp4 \    --output_path ../../test/
After running the above scripts, the I3D and VGGish features will be saved in the test directory. The saved features include RGB visual features (pandemic_rgb.npy), optical flow features (pandemic_flow.npy), and audio features (pandemic_vggish.npy).
Navigate back to the main project folder and then activate the bmt environment which was set up previously. Finally, we can run video captioning using the below command:
cd ../../conda activate bmtpython ./sample/single_video_prediction.py \    --prop_generator_model_path ./sample/best_prop_model.pt \    --pretrained_cap_model_path ./sample/best_cap_model.pt \    --vggish_features_path ./test/pandemic_vggish.npy \    --rgb_features_path ./test/pandemic_rgb.npy \    --flow_features_path ./test/pandemic_flow.npy \    --duration_in_secs 99 \    --device_id 0 \    --max_prop_per_vid 100 \    --nms_tiou_thresh 0.4
The prop_generator_model_path and pretrained_cap_model_path specify the proposal generator model path and the caption model path. Because we are using both pre-trained models, we can directly link it to the path where the pre-trained models were saved before. vggish_features_path, rgb_features_path, and flow_features_path are the paths for which the corresponding features are saved, duration_in_secs is the duration of the video in seconds, device_id is the GPU number to use, max_prop_per_vid is the maximum number of proposals to search for (a hyperparameter), and finally nms_tiou_thresh is the non-maximum suppression threshold parameter.
After executing dense video captioning, here is the result that is printed:
[  {'start': 2.9, 'end': 25.2, 'sentence': 'A person is seen sitting on a table and leads into a woman speaking to the camera'},  {'start': 0.0, 'end': 61.9, 'sentence': 'A man is seen speaking to the camera and leads into several shots of people working on a building'},  {'start': 0.0, 'end': 12.7, 'sentence': 'A woman is sitting in a chair talking to the camera'},  {'start': 78.6, 'end': 100.0, 'sentence': 'The man then takes the cube and shows the cube off the table'},  {'start': 0.2, 'end': 5.2, 'sentence': 'A man is sitting on a chair with a woman sitting on a chair and talking'},  {'start': 38.8, 'end': 100.0, 'sentence': 'A man is seen speaking to the camera and leads into several shots of people working out and speaking to the camera'}]
The result gives a list of start and end times, as well as sentences describing the content of the video within that period of time. After watching and comparing it with the video itself, I must say that the model performs pretty well in terms of understanding the video and taking the audio into account!
This can be applied to any video so feel free to try it out~! This article is based on the official repository, so make sure to also check it out.
This article shows you how you can generate multiple captions for different timeframes within a single video using a pre-trained model. The model’s performance is shown to be pretty well and there are more improvements that can possibly be applied to the model and the dense video captioning task in the future.
For more tutorials regarding deep learning, please feel free to check out: