Advanced
Multimodal Machine Learning
Lecture 1.1: Introduction
Louis-Philippe Morency* Original version co-developed with Tadas Baltrusaitis
1Your Instructor and TAs This Semester (11-777)
Louis-Philippe Morency
morency@cs.cmu.edu
Office: GHC-5411

Carla De Oliveira Viegas
cviegas@andrew.cmu.edu

Harsh Jhamtani
jharsh@andrew.cmu.edu

Zhun Liuzhunl@andrew.cmu.edu

Varun Lakshminarasimhan
vbl@andrew.cmu.edu

Ying Shenyshen2@andrew.cmu.edu
2

Lecture Objectives▪ Introductions▪ What is Multimodal?
▪ Multimodal communicative behaviors▪ A historical view of multimodal research▪ Core technical challenges▪ Representation, translation, alignment, fusion
and alignment▪ Course syllabus and project assignments▪ Grades and course structureWhat isMultimodal?
4What is Multimodal?

Multimodal distribution➢ Multiple modes, i.e., distinct “peaks”(local maxima) in the probability
density function

What is Multimodal?

Sensory Modalities

Multimodal Communicative Behaviors

Verbal▪

Lexicon▪ Words▪

Syntax▪ Part-of-speech▪ Dependencies▪

Visual▪

Gestures▪ Head gestures▪ Eye gestures▪ Arm gestures▪

Body language▪ Body posture▪ Proxemics▪

Eye contact▪ Head gaze
▪ Eye gaze▪

Facial expressions▪ FACS action units▪ Smile, frowning

Pragmatics▪ Discourse acts

VocalProsody▪ Intonation▪ Voice quality▪ Vocal expressions▪ Laughter, moans▪

7What is Multimodal?
Modality
The way in which something happens or is experienced.
• Modality refers to a certain type of information and/or therepresentation format in which information is stored.
• Sensory modality: one of the primary forms of sensation,
as vision or touch; channel of communication.

Medium (“middle”)
A means or instrumentality for storing or communicating
information; system of communication/transmission.
• Medium is the means whereby this information isdelivered to the senses of the interpreter.
8Multiple Communities and Modalities

Psychology

MedicalSpeech

VisionLanguage

Multimedia

Robotics

Learning

Examples of Modalities❑ Natural language (both spoken or written)❑ Visual (from images or videos)❑ Auditory (including voice, sounds and music)❑ Haptics / touch❑ Smell, taste and self-motion❑ Physiological signals▪ Electrocardiogram (ECG), skin conductance❑ Other modalities▪ Infrared images, depth images, fMRI

A Historical View
11

Prior Research on “Multimodal”
Four eras of multimodal research➢ The “behavioral” era (1970s until late 1980s)➢ The “computational” era (late 1980s until 2000)➢ The “interaction” era (2000 - 2010)➢ The “deep learning” era (2010s until …)❖ Main focus of this course

19701980

1990
1220002010

Language and Gestures
David McNeill
University of Chicago
Center for Gesture and Speech Research“For McNeill, gestures are in effect the speaker’s
thought in action, and integral components of speech,
not merely accompaniments or additions.”❑ TRIVIA: Justine Cassell was a student of David McNeill
1970

1980

1990
1320002010

The McGurk Effect (1976)

Hearing lips and seeing voices – Nature
1970

1980

1990
1420002010

The McGurk Effect (1976)

Hearing lips and seeing voices – Nature
1970

1980

1990
1520002010➢ The “Computational” Era(Late 1980s until 2000)

1) Audio-Visual Speech Recognition (AVSR)

1970

1980

1990
1620002010➢ The “Computational” Era (Late 1980s until 2000)

2) Multimodal/multisensory interfaces
Rosalind Picard
Affective Computing iscomputing that relates to, arises
from, or deliberately influences
emotion or other affectivephenomena.
❑ TRIVIA: Rosalind Picard came from thesame group (MIT, Sandy Pentland)
1970

1980

1990
1720002010➢ The “Computational” Era (Late 1980s until 2000)

3) Multimedia Computing[1994-2010]“The Informedia Digital Video Library Project automatically combines speech,
image and natural language understanding to create a full-content searchable
digital video library.”
1970

1980

1990
1820002010➢ The “Interaction” Era (2000s)

1) Modeling Human Multimodal Interaction
AMI Project [2001-2006, IDIAP]• 100+ hours of meeting recordings• Fully synchronized audio-video• Transcribed and annotated

CHIL Project [Alex Waibel]• Computers in the Human Interaction Loop
• Multi-sensor multimodal processing• Face-to-face interactions❑ TRIVIA: Samy Bengio started at IDIAP working on AMI project
1970

1980

1990
1920002010

➢ The “Interaction” Era (2000s)

1) Modeling Human Multimodal Interaction
CALO Project [2003-2008, SRI]
• Cognitive Assistant that Learns and Organizes• Personalized Assistant that Learns (PAL)• Siri was a spinoff from this project

SSP Project [2008-2011, IDIAP]• Social Signal Processing• First coined by Sandy Pentland in 2007
• Great dataset repository: http://sspnet.eu/❑ TRIVIA: LP’s PhD research was partially funded by CALO ☺
1970

1980

1990
2020002010➢ The “deep learning” era (2010s until …)

Representation learning (a.k.a. deep learning)• Multimodal deep learning [ICML 2011]• Multimodal Learning with Deep Boltzmann Machines [NIPS 2012]• Visual attention: Show, Attend and Tell: Neural Image Caption
Generation with Visual Attention [ICML 2015]

Key enablers for multimodal research:••••

New large-scale multimodal datasets
Faster computer and GPUS
High-level visual features“Dimensional” linguistic features

1970

Our course focuses on this era!
1980

1990
21

2000

2010Core TechnicalChallenges
22Core Challenges in “Deep” Multimodal ML

Tadas Baltrusaitis, Chaitanya Ahuja,
and Louis-Philippe Morency

https://arxiv.org/abs/1705.09406

These challenges are non-exclusive.
23

Core Challenge 1: Representation

Definition: Learning how to represent and summarize multimodal data in awaythat exploits the complementarity and redundancy.

A Joint representations:Representation

Modality 1

Modality 2

24

Joint Multimodal Representation
Joyful tone

“I like it!”
“Wow!”

Joint Representation(Multimodal Space)25

Tensed voice

Audio-visual speech recognition[Ngiam et al., ICML 2011]•Bimodal Deep Belief Network

DepthMultimodal

Joint Multimodal Representations

Multimodal Representation

Image captioning
Multimodal Deep Boltzmann Machine

Audio-visual emotion recognition

DepthVerbal•

DepthVideo[Srivastava and Salahutdinov, NIPS 2012][Kim et al., ICASSP 2013]•Deep Boltzmann Machine

Visual26Verbal

Multimodal Vector Space Arithmetic[Kiros et al., Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models, 2014]27

Core Challenge 1: Representation

Definition: Learning how to represent and summarize multimodal data in awaythat exploits the complementarity and redundancy.

A Joint representations:

B Coordinated representations:Representation

Modality 1

Modality 2

28Repres. 1

Repres 2

Modality 1

Modality 2

Coordinated Representation: Deep CCA

View 𝐻𝑥

Learn linear projections that are maximally correlated:𝒖∗ , 𝒗∗ = argmax 𝑐𝑜𝑟𝑟 𝒖𝑻 𝑿, 𝒗𝑻 𝒀

View 𝐻𝑦𝒖,𝒗

𝑯𝒙𝒖𝒗
𝒀𝑿
Andrew et al., ICML 2013

29···
𝑼···
𝑾𝒙···
Text
𝑿···
𝑽···
𝑾𝒚···
Image𝒀𝑯𝒚

Core Challenge 2: Alignment
Definition: Identify the direct relations between (sub)elements from two or
more different modalities.
Modality 2

Modality 1

A Explicit Alignment

t2

t3

tn

Fancy algorithm

t1
t4

t5

tn

The goal is to directly find correspondences
between elements of different modalities

B Implicit Alignment
Uses internally latent alignment of modalities inorder to better solve a different problem

30Temporal sequence alignment

Applications:- Re-aligning asynchronousdata- Finding similar data acrossmodalities (we can estimate
the aligned cost)- Event reconstruction frommultiple sources

Alignment examples (multimodal)Implicit Alignment

Karpathy et al., Deep Fragment Embeddings for Bidirectional Image Sentence Mapping,https://arxiv.org/pdf/1406.5679.pdf

33

Core Challenge 3: Fusion

Definition: To join information from two or more modalities to perform a
prediction task.
A Model-Agnostic Approaches
2) Late Fusion

1) Early Fusion
Modality 1

Modality 1

Classifier

Modality 2

Classifier

Classifier

Modality 2

34Core Challenge 3: Fusion

Definition: To join information from two or more modalities to perform a
prediction task.
B Model-Based (Intermediate) Approaches
1) Deep neural networksMultiple kernel learning2) Kernel-based methods

yℎ1𝐴3) Graphical models

𝒙𝑨𝟏

ℎ2𝐴
ℎ1𝑉
𝒙𝑽𝟏𝒙𝑨𝟐

ℎ4𝐴

ℎ3𝐴
ℎ2𝑉
𝒙𝑽𝟐

𝒙𝑨𝟑

ℎ3𝑉
𝒙𝑽𝟑

𝒙𝑨𝟒

ℎ5𝐴

ℎ4𝑉

𝒙𝑨𝟓

𝒙𝑽𝟒

Multi-View Hidden CRF

35

ℎ5𝑉
𝒙𝑽𝟓

Core Challenge 4: Translation
Definition: Process of changing data from one modality to another, where thetranslation relationship can often be open-ended or subjective.

B Model-driven

A Example-based

36

Core Challenge 4 – Translation

Visual gestures(both speaker andlistener gestures)Transcriptions+Audio streams

Marsella et al., Virtual character performance from speech, SIGGRAPH/Eurographics
Symposium on Computer Animation, 2013

Core Challenge 5: Co-Learning
Definition: Transfer knowledge between modalities, including theirrepresentations and predictive models.

Prediction

Modality 1

Modality 2
Help during
training
38

Core Challenge 5: Co-Learning

A Parallel

B Non-Parallel

39

C HybridTaxonomy of Multimodal Research [ https://arxiv.org/abs/1705.09406 ]Representation▪ Joint
o Neural networks
o Graphical models
o Sequential▪ Coordinatedo Similarity
o Structured

Translation▪ Example-basedo Retrievalo Combination▪ Model-basedo Grammar-basedo Encoder-decodero Online prediction

Alignment▪ Explicito Unsupervisedo Supervised▪ Implicito Graphical models
o Neural networksFusion▪ Model agnostic
o Early fusiono Late fusiono Hybrid fusion▪ Model-based
oooKernel-based
Graphical models
Neural networks

Co-learning▪ Parallel datao Co-trainingo Transfer learning▪ Non-parallel data
▪ Zero-shot learning▪ Concept grounding▪ Transfer learning▪ Hybrid data
▪ Bridging

Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency, Multimodal Machine Learning: A Survey and Taxonomy

Real world tasks tackled by MMML
▪ Affect recognition▪
▪
▪

Emotion
Persuasion
Personality traits▪ Media description▪
▪
▪

Image captioning
Video captioning
Visual Question Answering▪ Event recognition▪
▪

Action recognition
Segmentation▪ Multimedia information retrieval▪

Content based/Cross-media

Multimodal Applications[ https://arxiv.org/abs/1705.09406 ]Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency, Multimodal Machine Learning: A Survey and Taxonomy

Course Syllabus43

Three Course Learning Paradigms

Research paper reading
and group discussion

Course project assignments(60% of your grade)(40% of your grade)

Course lectures(including guest lectures)
44

Course StructureGroup discussion andstudent presentations

Course lectures

45

Course Recommendations and Requirements
1

Ready to read at least 9 papers this semester !
• 9 research papers as part of the weekly reading assignments• Asked to answer research questions about each paper2

Already taken a machine learning course• Strongly recommended for students to have taken anintroduction machine learning course• 10-401, 10-601, 10-701, 11-663, 11-441, 11-641 or 11-741

3Motivated to produce a high-quality course project• Three course project assignments• Designed to enhance state-of-the-art algorithms

46

Course Project▪ Pre-proposal (in 2 weeks)▪ Define your dataset, research task and teammates▪ First project assignment (in 5 weeks)▪ Experiment with unimodal representations▪ Explore/discuss simple baseline model(s)▪ Midterm project assignment (in 10 weeks)▪ Implement and evaluate state-of-the-art model(s)▪ Discuss new multimodal model(s)▪ Final project assignment (in 14 weeks)▪ Implement and evaluate new multimodal model(s)▪ Discuss results and possible future directions

Course Project Guidelines▪ Dataset should have at least two modalities:▪ Natural language and visual/images▪ Teams of 3 or 4 students▪ The project should explore algorithmic novelty▪ Possible venues for your final report:▪ NAACL 2019, ACL 2019, IJCAI 2019, ICML 2019

▪ We will discuss on Thursday about project ideas▪ GPU resources available:▪ Amazon AWS and Google Cloud PlatformExamples of Previous Course Projects▪ Select-Additive Learning: Improving Generalization in Multimodal
Sentiment Analysis▪

https://arxiv.org/abs/1609.05244▪ Preserving Intermediate Objectives: One Simple Trick to Improve
Learning for Hierarchical Models▪

https://arxiv.org/abs/1706.07867▪ Gated-Attention Architectures for Task-Oriented Language
Grounding▪

https://arxiv.org/abs/1706.07230▪ Efficient Low-rank Multimodal Fusion with Modality-Specific Factors▪

https://arxiv.org/abs/1806.00064▪ Multimodal Sentiment Analysis with Word-Level Fusion andReinforcement Learning▪

https://arxiv.org/abs/1802.00924

Process for Selecting your Course Project▪ Thursday 8/30: Lecture describing available
multimodal datasets and research topics▪ Monday 9/3: Submit a short paragraph listing
your top 3 choices▪ Tuesday 9/4: During the later part of the lecture,
we will have a discussion period to help withteam formation▪ Sunday 9/12: Pre-proposals are due. You shouldhave selected your teammates, dataset and task

Course Grades▪ Discussion and lecture participation 20%▪ Reading assignments 20%▪ First project assignmento Report and presentation 15%▪ Mid-term project assignmento Report and presentation 15%▪ Final project assignmento Report and presentation 30%
51

Equal Contribution by All Teammates!

▪ Each team will be required to create a GitHub
repository which will be accessible by TAs▪ Each report should include a description of thetask from each teammate▪ Please let us know soon if you have concerns
about the participation levels of your teammates

Lecture Schedule
Classes
Week 1
8/28 & 8/30
Week 2
9/4 & 9/6
Week 3
9/11 & 9/13
Week 4
9/18 & 9/20

LecturesCourse introduction
Thursday 8/30 in•
Research and technical challenges
DH A302•
Multimodal applications and datasets
Basic mathematical concepts
Project•
preferences due
Language, image and audio representations•Loss functions and basic neural networks on Monday night
Convolutional neural networks and optimization
Pre-proposal due•
Neural network optimization
on Sunday 9/12•
Convolutional neural networks
Recurrent neural networks•
Backpropagation Through Time•
Gated networks and LSTM

53

Lecture Schedule
Classes
Week 5
9/25 & 9/27
Week 6
10/3 & 10/5
Week 7
10/9 & 10/11
Week 8
10/16 & 10/18

LecturesMultimodal representation learning•
Multimodal auto-encoders•
Multimodal joint representationsFirst project assignment - Presentations

Thursday in DH A302 .
Proposal due: 10/7.

Multivariate statistics and coordinated representations•
Deep canonical correlation analysis•
Non-negative matrix factorization
Multimodal alignment and attention models•
Explicit alignment and dynamic time warping•
Implicit alignment and attention models

54Lecture Schedule
Classes
Week 910/23 – 10/25
Week 1010/30 & 11/1
Week 11
11/6 & 11/8
Week 12
11/13 & 11/15

LecturesMultimodal optimization•
Practical deep model optimization•
Variational approaches
Probabilistic graphical models• Boltzmann distribution and CRFs
• Continuous and fully-connected CRFs
Mid-term project assignment - Presentations

Thursday in DH A302.
Midterm due on 11/11.

Multimodal fusion and new directions•
Multi-kernel learning and fusion•
New directions in multimodal machine learning55

Lecture Schedule
Classes
Week 13
11/20 & 11/22
Week 14
11/27 & 11/29

Week 15TBD* Final *Lectures
Thanksgiving week (+ Project preparation)
Multi-lingual representations and models• Neural machine translation• Guest lecture: Graham Neubig
Final project assignment - Presentations

Thursday in DH A302.
Final project due: 12/9.

56

Piazza https://piazza.com/cmu/fall2018/11777/home• Announcements• Question/Answers• Lecture slides• Reading assignments• Project resources• Room assignments
57

Gradescope – Entry Code: M576ZG• Submit your weekly answers
to reading questions• Submit your project reports• View the comments from yourgraded reports

58