Summarizing videos in 300 lines of code
By: Laurent Picard
Tags: Programming, Machine Learning, Python, Google App Engine, Technology

🎞️ Visual Summary

Code structure

It’s interesting to split the code into 2 main classes:

StorageHelper for local file and cloud storage object management

for local file and cloud storage object management VideoProcessor for graphical processings

Here is a possible core function:

Note: If exceptions are raised, it’s handy to log them with logging.exception() to get a stack trace in production logs.

Class StorageHelper

The class manages the following:

The retrieval and parsing of video shot annotations

The download of source videos

The upload of generated visual summaries

File names

The source video is handled in the with statement context manager:

Note: Once downloaded, the video uses memory space in the /tmp RAM disk (the only writable space for the serverless function). It's best to delete temporary files when they're not needed anymore, to avoid potential out-of-memory errors on future invocations of the function.

Annotations are retrieved with the methods storage.Blob.download_as_string() and json.loads() :

The parsing is handled with this VideoShot helper class:

Video shot info can be exposed with a getter and a generator:

The naming convention was chosen to keep consistent object paths between the different buckets. This also lets you deduce the video path from the annotation URI:

The video is directly downloaded with storage.Blob.download_to_filename() :

On the opposite, results can be uploaded with storage.Blob.upload_from_string() :

Note: from_string means from_bytes here (Python 2 legacy). Pillow supports working with memory images, which avoids having to manage local files.

And finally, here is a possible naming convention for the summary files:

Class VideoProcessor

The class manages the following:

Video frame extraction

Visual summary generation

Opening and closing the video is handled in the with statement context manager:

The video summary is a grid of cells which can be rendered in a single loop with two generators:

Note: shot_ratio is set to 0.5 by default to extract video shot middle frames.

The first generator yields cell images:

The second generator yields cell positions:

OpenCV easily allows extracting video frames at a given position:

Choosing the summary grid composition is arbitrary. Here is an example to compose a summary preserving the video proportions:

Finally, Pillow gives full control on image serializations:

Note: Working with in-memory images avoids managing local files and uses less memory.

Local development and tests

You can use the main scope to test the function in script mode:

Test the function:

cd ~/$PROJECT_ID

python3 -m venv venv

source venv/bin/activate



pip install -r $PROJECT_SRC/gcf2_generate_summary/requirements.txt



VIDEO_NAME="gbikes_dinosaur.mp4"

ANNOTATION_URI="gs://$ANNOTATION_BUCKET/$VIDEO_BUCKET/$VIDEO_NAME.json"



python $PROJECT_SRC/gcf2_generate_summary/main.py $ANNOTATION_URI Downloading -> /tmp/SUMMARY_BUCKET/VIDEO_BUCKET/VIDEO_NAME

Generating summary...

Uploading -> VIDEO_BUCKET/VIDEO_NAME.summary004.jpeg

Note: The uploaded video summary shows 4 shots.

Clean up:

deactivate

rm -rf venv

Function entry point

Note: This function will be called whenever an annotation file is uploaded to the bucket defined as a trigger.

Function deployment

GCF_NAME="gcf2_generate_summary"

GCF_SOURCE="$PROJECT_SRC/gcf2_generate_summary"

GCF_ENTRY_POINT="gcf_generate_summary"

GCF_TRIGGER_BUCKET="$ANNOTATION_BUCKET"

GCF_ENV_VARS="SUMMARY_BUCKET=$SUMMARY_BUCKET"

GCF_TIMEOUT="540s"

GCF_MEMORY="512MB"



gcloud functions deploy $GCF_NAME \

--runtime python37 \

--source $GCF_SOURCE \

--entry-point $GCF_ENTRY_POINT \

--update-env-vars $GCF_ENV_VARS \

--trigger-bucket $GCF_TRIGGER_BUCKET \

--region $GCF_REGION \

--timeout $GCF_TIMEOUT \

--memory $GCF_MEMORY \

--quiet

Notes:

The default timeout for a Cloud Function is 60 seconds. As you’re deploying a background function with potentially long processings, set it to the maximum value (540 seconds = 9 minutes).

You also need to bump up the memory a little for the video and image processings. Depending on the size of your videos and the maximum resolution of your output summaries, or if you need to generate the summary faster (memory size and vCPU speed are correlated), you might use a higher value (1024MB or 2048MB).

Deploying function (may take a while - up to 2 minutes)...done.

availableMemoryMb: 512

entryPoint: gcf_generate_summary

environmentVariables:

SUMMARY_BUCKET: b3-summaries...

...

status: ACTIVE

timeout: 540s

updateTime: 'YYYY-MM-DDThh:mm:ss.mmmZ'

versionId: '1'

Here is how it looks like in the Cloud Console:

Production tests

Make sure to test the function in production. You can upload an annotation file in the 2nd bucket:

VIDEO_NAME="gbikes_dinosaur.mp4"

ANNOTATION_FILE="$VIDEO_NAME.json"

ANNOTATION_URI="gs://$ANNOTATION_BUCKET/$VIDEO_BUCKET/$ANNOTATION_FILE"

gsutil cp $ANNOTATION_URI .

gsutil cp $ANNOTATION_FILE $ANNOTATION_URI

rm $ANNOTATION_FILE

Note: This reuses the previous local test annotation file and overwrites it. Overwriting a file in a bucket also triggers attached functions.

Wait a few seconds and query the logs to check that the function has been triggered:

gcloud functions logs read --region $GCF_REGION LEVEL NAME EXECUTION_ID TIME_UTC LOG

...

D gcf2_generate_summary ... ... Function execution started

I gcf2_generate_summary ... ... Downloading -> /tmp/SUMMARY_BUCKET/VIDEO_BUCKET/VIDEO_NAME

I gcf2_generate_summary ... ... Generating summary...

I gcf2_generate_summary ... ... Uploading -> VIDEO_BUCKET/VIDEO_NAME.summary004.jpeg

D gcf2_generate_summary ... ... Function execution took 11591 ms, finished with status: 'ok'

The 2nd function is operational and the pipeline is in place! You can now do end-to-end tests by copying new videos in the 1st bucket.

Results

Download the generated summary on your computer:

cd ~/$PROJECT_ID

gsutil cp -r gs://$SUMMARY_BUCKET/**.jpeg .

cloudshell download *.jpeg

Here is the visual summary for gbikes_dinosaur.mp4 (4 detected shots):

You can also directly preview the file from the Cloud Console: