Abstract

Mahmoud Azab, Mingzhe Wang, Max Smith, Noriyuki Kojima, Jia Deng, Rada Mihalcea
01:02:00 --> 01:02:01

Jack, must you go?

Input

®
e a new model for speaker naming
nthatMovies
01:02:01 --> 01:02:04
leverages visual, textual, andTime for me to go row
with other slaves.
odalities in an unified optimization. inToMovies
evaluate the performance of
01:02:07 --> 01:02:08
Good night, Rose.
, we introduce a new dataset coniyuki
Kojima,Rada Mihalcea
six
episodesof theJia
BigDeng,Bang The,owUniversity
offullMichiganand eighteen
movies coveroriyuki
Kojima,JiaDeng,Rada
Mihalcea
ntgenres. Our experiments show
adeng,mihalcea}@umich.edu
Jack, must
Names:
ng,UniversityofMichigan
ultimodal model significantly outJack
you go?
ProblemDefinition:
jiadeng,mihalcea}@umich.edu
Rose……
everal competitive baselines on the……
Good night
eighted
F-scoremetric.
TodemonGiven a movie video and its subtitles, label each segment of theeffectivenessof with
our framework,
subtitles
the nameweof its corresponding speaker
end-to-end memory network model
Joint Optimization
Input
Outputges our speaker naming
model andate-of-the-art
results on the subtitles
01:02:00 --> 01:02:01
01:02:00-->you01:02:01
Jack,mustgo?
MovieQA
2017
Challenge.
Jack, must you go?

University of Michigan{mazab,mzwang,mxsmith,kojimano,jiadeng,mihalcea}@umich.edu

forforme
row
Time
meto
to gogo row
ctionTimewithotherwithotherslaves.
slaves.

OutputInput

Inference

Introduction

01:02:01
01:02:01-->-->01:02:04
01:02:04

Jack: Hold On!
Jack: All right!

Linitial
Linitial + Lgender
Linitial + Lnegative
Linitial + Ldistribution
Linitial + LM ultipleInstance

Framework & Model

Table 4: Analysis of the effect of adding each com-

Name Reference Classifier

Sentence Embedding
Voice-based GenderClassifier

MFCC + F0 statistics

Rose: I trust you.

I-vector Extraction+Joint Optimization

OutputWeighted Sum

Inference

OutputInference

rdndSoftmax

stConvsegments of subtitles). We use the segments inTo encourage our predictions to be consistentwhich we know the speaker’s name and manuallywith the dialogue-based priors, we penalize theobtain the ground truth gender label from IMDB.
square error between the distributions of predicVGG()
We extract the signal energy, 20 Mel-frequency
tions and name mentions priors in the following
cepstral coefficients (MFCCs) along with theirequation:
XXfirst and second derivatives, in addition to timeL (f ) =( (f (x )) d )(4)
and frequency-based absolute fundamental frequency (f0) statistics as features to represent eachwheredistheratioofnamejmentions
inall
subsegment Datain the subtitles. The f0 statistics has been
titles.
found to improve the automatic gender detection
Final Framework. Combining the loss in Eqn. 1
performance for short speech segments (Levitanand multiple losses with different constraints, weet al., 2016), which fits our case since the medianobtain our unified optimization problem:
duration of the dialogue turns in our dataset is 2.6• 437 different character
names
Matrix representingseconds.
⇤f=arg
min
L(f)
+L(f)
assignmentof1
2MI
initialwith each subtitle segment manually labeled with its corresponding
fThe
MFCC
featuresareextracted
using
step
names ato subtitles
speaker namesize of 16 msec over a 64 msec window using the+ 3 Lneg (f ) + 4 Lgender (f ) + 5 Ldis (f )methodfrom (Mathieu et al., 2010), while the f0• Publicly available
at: http://lit.eecs.umich.edu/downloads.html(5)
Negative constraint:• More 1 and 2 person reference
Gender constraint:statistics are extracted using a step size of 25 msec• 2 and 3 person
ref.of theAlls
arehyper-parameters
tobetuned→ morelikelyto speak more•
Voiceandname• s: I am flying, Jack.
•Maincharacters
speak
more
genders
mustmatch→scannot
beJack.
over a 50 msec window as the default configuraon development set. We also include the constrainttion
in(Eybenetal.,2013).
Wethenusethesefeathat
predictionsfordifferent
character
names
must
This research is based upon work funded by a Samsung research grant

EFigure 3

based CMemN2• Fine-tuning
parameters
using4
moviesandtestingon
therestThe M
In order to analyze the effect of the errors that
PrecisionF-scoredifferenseveral of the modules (e.g.,gender Recalland namerefB1: MFMC
0.09100.2749 0.1351
mationerenceclassifiers)propagate
intothesystem,
we
B2: DRA
0.22560.1819 0.1861
for manalso
our framework
by 0.2876
replacing
each0.2317
one of
B3: test
Gender-based
DRA
0.2349Model (Skip-thoughts)*
0.3468truth
0.28690.2680are not
theOurcomponents
with its groundinformation.
OurMovie
Model (TF-IDF)*
0.35790.29330.2805
Question
Answerstial
syst
AsOurseen
in
Table
5,
theresults
obtainedin
thisModel (iVectors)
0.2151
0.2347 0.1786
A1:
ShewaskilledwethusOur
Model(Visual)*
0.33480.26590.2555setting show significant
improvement
What did Mike’swife, as hewith the reA3: Leukemia
OurFargo
Model (Visual+iVectors)*
0.3371
0.2720 0.2617
which inplacement
ofeachcomponentinourframework,says,
diefrom?
Our Model (TF-IDF+iVectors)*
0.35490.2835 0.2643
A5: Complication
informawhichsuggeststhat additional
work0.2975
on these
comOur Model(TF-IDF+Visual)*
0.33850.2821
A1: Sketch her in h
Our Modelwill(all)*Whatdoes
Rose0.3720
ask
Jack0.3108
to do 0.2920
answer q
ponents
havepositiveimplications
ontheTitanic
A3: Take a picture
inherroom?
To deoverallsystem.
Table
3:Comparison
betweentheaverageofA5:Takeapicture• Replacing different auxiliary classifiers input with ground-truth
namingmacro-weighted average of precision, recall and fTable 6: Example of questions
andRecallanswers
from theonMQA
PrecisionF-scorean escore of the baselines and our model. * means stacorrect
answers
totheircorrespondingquestion.
Our
Model
0.37200.31080.2920tistically significant (t-test p-value < 0.05) whenet al.,
Voice Gender (VG)
0.42180.3449 0.3259
comparedtobaseline
B3.
tionalM
VG + Name Gender (NG)0.44120.3790 0.3645
3 shows
the+architectureof0.4403
our model.
relies on
VG + NG
Name Ref
0.3938 0.3748
Table7
includestheresultsofoursystemonspeakergoal is to assign names to speakers, and not necTable
5:
Comparison
betweenourmodelwhile
theessarily
validation
andtestsets, along
withwetheconbest network
solve
this
coreference
problem,
Extrinsic
Evaluation:
replacingdifferent
components
with
their
groundsystemswork,
showing
ing framsider theintroduced
assignmentinofprevious
the subtitle
segments
totruthinformation.
• We build
afullQA
model
toexplore
theeffectivenessofthethat
SC-MemN2N
achieves
best Thus,
perfor- for eachanyourof thespeaker’s aliases
to be thecorrect.
speakernamingmodel
onmovieQA
task
predicte
mance.
measureduringFurthermore,
the evaluation,towemap allthe
theeffectivenesscharacters’
4
A1:
Jones
Q:
Who
sees
Bruce
andtles.
To
our model’s
outputto the names
inofaliases
addingfrom
the speakernames
andmasking,
wetest
A2: Gordon
Selina together
in Florence?"
7the Speaker
NamingforMovie
A3: Alfred
groundtruth
annotations.
Our mapping
does
sent eac
our model
afterremoving
the namesfrom thenetA4: Fox
Understanding
not (C-MemN2N).
includeRelated
other referent
nouns“Dad,”
A5:
Blakeof a 300work
As W2Vseen
fromsuch
the asresults,
thecharacters“Buddy,”etc.;
ifasegment
gets
assigned
toany
5etal.,
2
gainofSC-MemN2N
isstatisticallysignificant
W2V
speakers is FCa critical task for underSubtitlesIdentifying+such terms,
itisconsideredamisprediction.
sentence
speakercomparednames
toaversion
ofthesystem
thatdoes
notstanding the dialogue and storyline in movies.
FCWe
compareourmodelagainstthreebaselines:
tions
aninclude
thespeakernames(C-MemN2N).
FigureFigure 4
MovieQA is a challenging dataset for movie unFigure
2: F
B1:
Most-frequentlymentioned
characterconquestion
4 derstanding.
shows the performanceboth C-MemN2N
andThe datasetofconsistsof 14,944∑ multion
typ⨷
SNE(Vansists of selecting the most frequently mentioned
anansw
SC-MemN2N
models byaboutquestiontype. The
retiple choice questions
408 movies.
Eachingcharacte
iVectocharacter in the dialogue as the speaker for all thesults
suggestourspeaker
the resent inst
questionhas thatfiveanswers
and naming
only onehelpsof them⨷subtitles. Even though it is a simple baseline, itcharacte
Softmax
model
betterdistinguishbetweencharacters,andinmovisachieves
correct.
The
datasetisdivided
intothreesplits:
an accuracy of 27.1%, since the leading
andmas
train,
validation,andtestaccordingtothemovie
tithatprepending
thespeaker
namestothesubtificult
sePrediction
characters tend to speak the most in the movies.
weighted
Importantly,
theretheare ability
no overlapping
moviestletles.
segments
improvesofthememory
sion
fro
3
B2: Distribution-driven random assignment
formshttp:/
the4
betweenthesplits.
Table
6
shows
examplesofthenetworktocorrectly
identify
thesupporting
facts
plicated
We stconsistsofrandomly
assigning
character
names
tures
comb
Subtitles
Speaker +Weights
questionandanswers
intheMovieQA
dataset.
toinferthfromthestory
that
answers
agivenquestion.
includeEmbeddingMentionMask
according to a distribution that reflects their fractogether in
andofinction of mentions in all the subtitles.
tancete
Subtitles
tic
B3: Gender-based distribution-driven random
MethodThecons
ine
val
test
datasetassignment consists of selecting the speaker
resultof th
SSCB-W2V (Tapaswi et al., 2016)24.8 23.7
names based on the voice-based gender detecposed
maredifficul
SSCB-TF-IDF (Tapaswi et al., 2016) 27.6 26.5
tion classifier. This baseline randomly selects thebaseline
ure2show
SSCB Fusion (Tapaswi et al., 2016)27.7
character name that matches the speaker’s genderborEmbedmodelt
MemN2N (Tapaswi et al., 2016)38.0 36.9
according to the distribution of mentions of thewhich
iscan
ach
Understanding visual regions
37.4
names
inthe
matching
gendercategory.
techniquetask of
RWMN (Na et al., 2017)40.4 38.5
The
results
obtainedwith
ourproposedunifiedsimilarvec
C-MemN2N (w/o SN)
40.6
The
optimization
framework
andthethreebaselines
dissimilar
SC-MemN2N (Ours)
42.7 39.4
introduc
are shown in Table 3. We also report the perforvisualizati
ableandfromance
the optimization
framework
difshowTable
7: ofPerformancecomparison
forusing
the subtiW2V

01:02:07-->01:02:08
01:02:08
Rose:I'm flying. Jack.
01:02:07-->
Open your eyes.
peakers
andtheirnames
inmovies,
Good night, Rose.
Goodnight, Rose.
general, is a primary task for manys problems, including automatic subSpeaker Labeled
Talking
Face
Figure 1: Overview of our approach for speaker
Face
Subtitles
Video
Names:
Jack, content-basedmust
Hu et al.,2015),
video
Embedding
tracksnaming.
Jack
yougo?
Names:
Jack, mustretrieval(Zhangetal.,2009),
video
Rose
……
Jack
you go?
……
nightn (TapaswietGoodal.,2014), and video
Rose
……
……
Good nightspeakers who did not appear in the video frameNames Identification and Reference Classifier:erstanding(Tapaswiet al., 2014). It
were not assigned any names, which is common inenging task, asJoint
theOptimization
visual appearance
Target Labels
Weak Labels
movies and TV shows. Other available sources ofters Motivation:changes over the course of theJoint Optimization
information such as scripts were only used to ex- - [Sheldon, Dr. Cooper, Sheldon Cooper] —> Sheldon
several factors such as scale, cloth• Speaker
namingisimportantforvideo
understanding,indexing
Hey,Sheldon!
2
Rose: I trustandyou.
tract cues about the speakers’ names to associate - [Mrs. Cooper, Mary Cooper, Mary]
Hold On! (Arandjelovic
ion, andJack
so : forth—> Mary Cooperandsummarization- Did you see Leonard? 3
the faces in the videos with their corresponding - [Leonard]005; Everinghametal.,
2006).
The—>Leonard
Jack: All right!
Rose: I trust you. character name (Everingham et al., 2006; Tapaswi
Jack:yourHoldeyes.
On!
- I am Penny.
1Rose:I'mflying.
Jack. to generalize:•Existingmethods
arehardOpen
movie data with speakers’ names- [Howard, Howard Wolowitz]—> Howard
etcannotal., 2015;
Bäumletnew
al., 2013;
Sivic et al., 2009);
l in a numberofapplications,suchas•Jack
Use
supervisedapproaches,
workonmovies: All right!
gureanswering
1: OpenOverviewof our approachforspeakeron
etRoseal.,:I'm2016),
flying.
Jack. however since scripts are not always available, theyour(Tapaswieyes.
• Rely on scripts/cast lists to applicability
get speakerofnames
and labelsthesemethods
is somehow limited.Talking Face Embedding:ming.
ntification of character relationships
Ignoretext,movie
use only
vision andspeech2009),• orautomatic
captionOther
studies focused on the problem of speakere 1: Overview of our approach for speaker2015).who did not appear in the video frame
recognition without naming, using the speech
eakersng.
ere
assignedrelied
any names,
which isousnotstudies
primarily
oncommonvi- inmodality as a single source of information. WhileoviesContributions:andTV shows. Other
sources oftion(Arandjelovic
and available
Zisserman,
some of these studies attempted to incorporate
formationsuchasscriptsand
wereaimed
only used
to exghametal.,2006),
forthe visual
modality,
their goal was to cluster thekers•who
did notappear weakly
in the video
frame
Proposeanovelsupervised
unifiedmultimodal
act cues about the speakers’ names to associateifferent
taskofface
tracklabeling;
speechsegments
ratherthan
namethespeakers
notassigned
anynames,
whichiscommonin
optimization
framework
forspeakernaminge faces in the videos with their correspondingies
andnameTV shows.
Otheravailable
sources ofaracter(Everinghametal.,
2006;
Tapaswi• Construct new speaker naming dataset of 18 movies and 6mation
asetscripts
wereto exal.,
2015;suchBäumlal., 2013;
Siviconly
et al.,used2009);
Kepisodes
ofTV
shows
wever
are not always
available,
thecues since
aboutscripts
the speakers’
names
to associatej2
plicabilityof these
methods
somehow
limited.
Achievestate-of-the-art
performance on movieQA subtitles
aces• inthevideos
withis theircorrespondingijdis
Othernamestudiesfocused on the problemof speaker
challenge
acter(Everinghamet al., 2006;
Tapaswij=1
cognition without naming, using the speech, 2015; Bäuml et al., 2013; Sivic et al., 2009);
Joint
Optimization:odality as a single source of information. While
eversince
not always
available, theme ofthesescripts
studiesare
attempted
to incorporate•
Approachspeakernamingasa
transductive
learning
problemwithconstraints
jof these their
methods
is somehow
ecability
visual modality,
goal wasto clusterlimited.
the• Model the objective function as a linear combination of different losses
eechsegments
rather on
thanthenamethe speakers
her studies
focusedproblemof speakergnition without naming, using the speechMulti-instance constraint:•Construct
anew
datasetconsistingof:•
Assign
names
based
on
first
personality as a single source of information. Whilend person reference is some speaker in the•2reference (“I am Jack”)•24
videos(18movies&
6
episodes
ofaTV
show)
e of these studies attempted to incorporate
conversation• Assign same name to speakers withvisual •modality,
s1: I think we can do it.
31,019 theirturnsgoal was to cluster thesimilar facial, textual, acoustic featuress2: I don’t think so, Jack.
ch segments
ratherthannamethespeakers• 21.99 hours of dialog

ndF-score
0.07750.12100.03610.06080.2189ponentof the loss function to the initial loss.
Intrinsic
Evaluation:
Character Names

Audio

Recall0.15760.18450.07460.15700.2941

Experiments & Results

Framework Overview:

Subtitles

Precision
0.06310.11600.08250.10500.3058

W2V

Speaker Naming in Movies

Linit without the other parts is very low due to thesparsity of first person references and errors thatthe person reference classifier introduces.