Package: videosum
DescriptionCode for thevideosumPython package. Given a video file, this package produces a single-image storyboard that summarises the video.CitationIf you use this code in your research, please citethe paper:@article{GarciaPerazaHerrera2023,
	author = {Luis C. Garcia-Peraza-Herrera and Sebastien Ourselin and Tom Vercauteren},
	title = {VideoSum: A Python Library for Surgical Video Summarization},
        journal = {arXiv preprint arXiv:2303.10173},
	year = {2023}
}Supported platformsUbuntu >= 20.04Python >= 3.10Use the package from a Docker containerIf you want to quickly try the package, this is the easiest way, provided that you have Docker installed with GPU support. If you do not have it, you can followthistutorial to install Docker with GPU support.
Alternatively, you can install thevideosumpackage frompipor from source, following the next sections of this README.Download repository:$gitclonehttps://github.com/luiscarlosgph/videosum.git
$cdvideosum/dockerBuild Docker image:$dockerbuild--build-argUSER=$(whoami)--build-argUID=$(id-u)--build-argGID=$(id-g)-tluiscarlosgph/videosum:latest.RunvideosumDocker container:$dockerrun--volume$HOME:/mnt/user_home--namevideosum--runtimenvidialuiscarlosgph/videosum:latest&The--volumeargument mounts your home directory (outside the container) into the directory/mnt/user_homeinside the container.
This is so that you can access the videos that need to be summarised from within the container, and also save the output storyboards
in a folder that is accessible from outside the Docker container.Get a terminal on the container:$dockerexec--user$(whoami)--workdir$HOME-itvideosum/bin/zshSummarise the test video:$python-mvideosum.run--input/opt/videosum/test/data/video.mp4--output/mnt/user_home/storyboard.jpg--fps2--nframes16--height1080--width1920--algoinceptionAfter running this command you should have the summary of the test video saved asstoryboard.jpgin you home directory.Install dependenciesIf you are not using the videosum Python package via Docker (as explained in the previous section), but you are planning to install it natively in your system, you must install all the dependencies detailed in this section.Python version3.10: works well withfaissversion1.7.4. You can find a howto guide to install this version of Pythonhere.PyTorchversion1.12.1+cu116, you can install it running:$pipinstalltorch==1.12.1+cu116torchvisiontorchaudio--extra-index-urlhttps://download.pytorch.org/whl/cu116`ffmpeg: in Ubuntu/Debian, run$ sudo apt install ffmpegto install it.swig: in Ubuntu/Debian, run$ sudo apt install swigto install it.faiss-gpuversion1.7.4: you can find a howto guide to install ithere.A good way to see all the commands that need to be executed to install all the dependencies is by checking theDockerfile.Install with pip$ python3 -m pip install videosum --userInstall from source$ git clone https://github.com/luiscarlosgph/videosum.git
$ cd videosum
$ python3 setup.py install --userRun video summarisation on a single video$ python3 -m videosum.run --input video.mp4 --output collage.jpg --nframes 100 --height 1080 --width 1920 --algo timeOptions:--input: path to the input video file.--output: path where the output collage will be saved.--nframes: number of frames that you want to see in the collage image.--height: height of the collage image.--width: width of the collage image.--time-segmentation: set it to either0or1. If 1, the clustering results are displayed in a bar underneath the collage (i.e. the columns of the bar represent the frames of the video, and the colours represent the clustering label).--fps: number of frames you want to read per second of video, used to downsample the input video and have less frames to describe and cluster.--time-smoothing: weight in the range[0.0, 1.0]that regulates the importance of time for clustering frames. A higher weight will result in a segmentation of frames over time closer to that of thetimemethod.--processes: number of processes to use when summarising a folder of videos.--metric: set it to True if you want to compute theFrechet Inception Distancebetween the frames in the summary and the frames in the original video.--algo: algorithm used to select the key frames of the video.time: evenly spaced frames are selected.inception: k-medoids clustering (l2-norm metric) on InceptionV3 latent space vectors.uid: in order to compute the distance between two images their InceptionV3 latent space vectors are computed, a univariate Gaussian is estimated for each of the two latent vectors, and the 2-Wasserstein distance is computed between the two Gaussians and used as clustering metric for k-medoids.uidstands for Univariate Inception Distance.scda: k-medoids clustering (l2-norm metric) on SCDA image descriptors (Wei et al. 2017 Selective Convolutional Descriptor Aggregation for Fine-Grained Image Retrieval). In this package we use InceptionV3 as opposed to VGG-16, which was the model used by Wei et al.Run video summarisation on multiple videosPointing the command line parameter-ior--inputto a folder of videos is enough. In this case, the path indicated by-oor--outputwill be used as output folder, each video summary will have the same filename as the video but a.jpgfile extension.The parameter--processesallows you to select the number of videos to summarise in parallel. This is necessary because some of the summarisation methods use GPU memory, which is typically a limiting factor. If the number of processes is too high you might get a CUDA out of memory error.$python3-mvideosum.run-i<input_folder>-o<output_folder>-n16--width1920--height1080-ainception--fps1--time-segmentation1--processes5Exemplary code snippetimportcv2importvideosum# Choose the number of frames you want in the summarynframes=100# Choose the dimensions of the collagewidtth=1920height=1080# Choose the algotrithm that selects the key framesalgo='inception'# The options are: 'time', 'inception', 'uid', 'scda'# Create video summariser objectvs=videosum.VideoSummariser(algo,nframes,width,height)# Create collage imageim=vs.summarise('video.mp4')# Save collage to filecv2.imwrite('collage.jpg',im)# Retrieve a list of Numpy/OpenCV BGR images corresponding to the key frames of the videokey_frames=vs.get_key_frames('video.mp4')# Print the video frame indices of the key frames, available after calling summarise() or get_key_frames()print(vs.indices_)# Print the video frame cluster labels, available after calling summarise() or get_key_frames()print(vs.labels_)Exemplary resultThe storyboards have a bar underneath that is produced when the--time-segmentation 1option is passed.
This bar shows how frames have been clustered over time, with a colour for each cluster, and black vertical lines representing the key frames.Exemplary video:hereSummary based ontimealgorithm:$ python3 -m videosum.run --input test/data/video.mp4 --output test/data/time.jpg --nframes 16 --height 1080 --width 1920 --algo time --time-segmentation 1Summary based oninceptionalgorithm:$ python3 -m videosum.run --input test/data/video.mp4 --output test/data/inception.jpg --nframes 16 --height 1080 --width 1920 --algo inception --time-segmentation 1Summary based onuidalgorithm:$ python3 -m videosum.run --input test/data/video.mp4 --output test/data/uid.jpg --nframes 16 --height 1080 --width 1920 --algo uid --time-segmentation 1Summary based onscdaalgorithm:$ python3 -m videosum.run --input test/data/video.mp4 --output test/data/scda.jpg --nframes 16 --height 1080 --width 1920 --algo scda --time-segmentation 1Run unit testing$ python3 setup.py testRun timing script$python3-mvideosum.timingMethodTime for a 1h video sampled at 1fpstime13sinception86suid216sscda74sUse this package as an unsupervised spatial feature extractorIf you have 2D RGB images and you want to obtain a feature vector for them, you can do so like this:InstallvideosumPython package:$ pip install videosum --userExtract feature vectors for your images:importcv2importvideosum# Read a BGR image from fileim=cv2.imread('test/data/test_time.png',cv2.IMREAD_UNCHANGED)# Extract latent space spatial feature vector for the imagemodel=videosum.InceptionFeatureExtractor('vector')vec=model.get_latent_feature_vector(im)# Here you can pass an image (H, W, 3)# or a batch of images (B, H, W, 3)# Print vector dimensionsprint(vec)print('Shape:',vec.shape)The output:[0.343185960.117948030.04767929...0.097318720.1.1942172]Shape:(2048,)AuthorLuis Carlos Garcia Peraza Herrera (luiscarlos.gph@gmail.com), 2022-present.LicenseThis code repository is shared under anMIT license.