Visual Geometry Group
Department of Engineering Science, University of Oxford

Automated Tagging of Image andVideo Collections using FaceRecognition

Andrew Brown

Andrew Zisserman

Ernesto CotoThe Goal



Provide TAGs or retrieve frames in videos using
face recognition
Two example datasets
The BFI Untagged dataset

Tags:

?

Film: “Slumdog Millionaire”

The BBC News dataset

Tags:

?

Programme: “Newsnight”

Visual Challenges for Face Recognition
Performing this task using Face Recognition can
be challenging
AgeVisual Challenges for Face Recognition
Performing this task using Face Recognition canbe challenging
AgeAppearance

Visual Challenges for Face Recognition
Performing this task using Face Recognition can
be challenging
AgeAppearance
Expression

Visual Challenges for Face Recognition

Performing this task using Face Recognition canbe challenging
AgeAppearance
Expression

Viewpoint

The Approach
1. Detect Faces

The Approach
1. Detect Faces

2. Represent each face by a vector

The Approach
1. Detect Faces

2. Represent each face by a vector

3. Recognize a face from a gallery using closest distance between vectors
vs.
John
GoodmanJean
DujardinCharlieChaplin…

The Approach
1. Detect Faces

2. Represent each face by a vector

3. Recognize a face from a gallery using closest distance between vectors
vs.
John
Goodman

JeanDujardin

Charlie
Chaplin…

For this to work, need vectors to only represent identity, and not be affectedby expression, pose, lighting, age, etc

Basic workhorse - face to vector
Convolutionalarchitecture

NeuralNetwork(CNN)

SE-ResNet-50-256D architecture with 30M parametersdeep256D
featurevector

J. Hu, L. Shen, G. Sun, “Squeeze-and-Excitation Networks”, IEEE Conference on Computer Vision and Pattern
Recognition, 2018

Trained on the VGGFace2 dataset

Face Dataset (VGGFace2)

3M+ face images, 300+ images for each of 9000 identities

Q. Cao, L. Shen, W. Xie, O. M. Parkhi, A. Zisserman, “VGGFace2: A dataset for recognising faces across pose and age”,
International Conference on Automatic Face and Gesture Recognition, 2018

Basic workhorse - face to vector
Convolutionalarchitecture

NeuralNetwork(CNN)

deep

256D
featurevector

Once the network is trained, it can be used to generatefeature vectors for previously unseen people

Automatic Tagging Example
The British Film Institute provided us withNames of 11K people of interest Metadata for movies/tv-shows (title, year) and their cast

Tags:?

Film: “Four Weddings and a Funeral”
GOAL: Provide tags for these 11K people on
dataset with 46K images

Tagging the images

Web Image SearchObtaining identity images
Use web image search engines (e.g. Google)
Works well for “famous” people (~6.5K in the dataset) Can automatically determine if famous or not using clusteringDisplaying the image ranking
Results are ordered starting from the best at thetop-left corner Location of the detected face can be highlighted
Text query:
Rank:

Rank:

1

263

74859

The BFI Browser
Provides functionality to search for people by nameor by movie/tv-show title
 Filter results by “featured” or “non-featured” in thecredits
Anomalies Found
1. Images labelled with the wrong filme.g. all of the images from "War Horse" were labelled as
being from "Anterior and Posterior Plaster Beds"
The Labelled Film Name: “Anterior andPosterior Plaster Beds” – 1936Correct Film Name: “War Horse” - 2011

Steven Spielberg is tagged in thisimage, but is not in the cast/crewAnomalies Found
2. Actors/actresses left out of the cast/crew

Geraldine Somerville left out of thecast of “Cracker” (TV series)

Donald Sutherland left out of the cast of“Commander in Chief”(TV series)

Anomalies Found
3. People appearing on sets when they are not in
the cast/crew

Marilyn Monroe appears in“Ritz” (Film)

Sharon Stone appears on“Richard & Judy” (TV Series)

Tony Blair appears on“Richard & Judy” (TV Series)

1,821 anomalies identified in total across the dataset
The decision of whether an anomaly is a mislabellingor a surprise appearance can be done automaticallyFace Recognition on Video Data

The BBC News Search system performs visual searches
over a large video dataset (~10K hours of video, ~5M
keyframes, ~1.5 TB)
Four search categories are available, including People

Available at: http://zeus.robots.ox.ac.uk/bbc_search/Video Demo

Automatic Tagging in News videosSimilarly to the BFI browser, we could pre-generate lists ofresults where specific people appears

Extensions: Compound Queries
Retrieve frames containing both a target face and a target
scene
Query: “Barack Obama in the Office”

Extensions: Compound Queries
Retrieve frames containing multiple people

Visual Geometry Group
Department of Engineering Science, University of Oxford

More information and demos at
http://www.robots.ox.ac.uk/~vgg/

Andrew Brown

Andrew Zisserman

Ernesto Coto