explain this deeply i don't understand : The Linear Method of Learning
When attempting to understand a broad field of study (like the underlying computer science of self-driving cars), it’s often difficult to know where the right entry point is.

As a result, most people assume that the best path forward is to start with the basics, build up a general foundation of knowledge, and then continue towards finer and finer levels of detail.

I call this the Linear Method of Learning.

Using the linear method to learn the computer science of self-driving cars would look something like this:

Learn multivariable calculus
Learn linear algebra
Learn basic computer science fundamentals
Learn about general machine learning concepts
Learn about computer vision concepts
Learn how to code in Python (a coding language commonly used for machine learning)
Learn how to use TensorFlow (a special machine learning library for Python)
Learn how computer vision is applied to create self-driving car software
Learn how to write Python and TensorFlow code to build the relevant programs
Etc…
While this method may eventually work, it’s inefficient and probably not effective.

Firstly, if I start by learning multivariable calculus, how do I know which parts of multivariable calculus are relevant to self-driving cars and which parts aren’t? I don’t. So, I’ll have to learn all of it. Same for linear algebra, and computer science fundamentals, etc.

In other words, if I start with the most general pieces of knowledge, I have no way to prioritize what I’m learning, and so, I’m ultimately forcing myself to learn everything just in case.

Additionally, because I’m first learning the foundational concepts in a general, abstract sense, it’s much harder for me to relate what I’m learning to things I already know. Since effective learning is essentially figuring out how to attach new pieces of information to currently existing knowledge in a meaningful way, the Linear Method of Learning also fails in this regard.

So, while most people approach learning in this linear fashion, it’s a pretty poor method to actually learn anything in a reasonable timeframe.

The V-Method of Learning
Instead, I use a different method, which I call the V-Method of Learning.

Here’s how the V-Method of Learning works:

I start with a specific, well-documented example of my end goal
I try to understand how this example works
For everything I don’t understand about the example, I research the underlying concepts
If I don’t understand the underlying concepts, I research the underlying concepts of the underlying concepts, until I feel I’ve exhausted this path (either by reaching understanding or by reaching a point of diminishing returns)
Eventually, I pigeonhole down enough different paths to start seeing patterns in the important underlying concepts
I study these relevant underlying concepts, slowly working my way up the knowledge chain, until I’m back at the level of detail of the original example
Finally, I reproduce the example based on my new hierarchical knowledge
I call this the “V-Method” because I start at the finest level of detail, dive deep towards the directly-applicable foundational concepts, and then work my way back up towards the finest level of detail — a conceptual V.

The V-Method is much more effective than the Linear Method because I’m able to 1. Learn in the order of relevance to my ultimate goal, 2. Learn the foundational concepts in the context of something tangible, and 3. Build and organize my knowledge in a hierarchical, interrelated way.

As a result, this method is much more efficient, effective, and engaging.

So, here’s how I plan to apply the V-Method to this month’s challenge:

Look for sample, open-source self-driving car code on Github (Github is a popular repository for code, which basically means I can find a lot of other people’s software projects there)
Work my way line-by-line through the code
For every line of code I don’t understand at an intuitive level (which will be most of them), begin my descent through the layers of underlying concepts
Identify patterns in what I’m constantly looking up / researching and determine the most important foundational concepts
Study these foundational concepts
Work my way back up the layers of underlying concepts until I can effectively explain to myself each line of code from the sample Github project
If this still sounds a bit confusing, hopefully it will start making more sense once I actually start.

My first step is to search Github for a good sample project…

Certainly! Let's dive deep into the concepts of the Linear Method of Learning and the V-Method of Learning as described in the passage. We'll explore what each method entails, their advantages and disadvantages, and why the V-Method might be more effective for certain learning goals.

The Linear Method of Learning
Overview:
The Linear Method of Learning is a traditional approach where learning progresses in a sequential, step-by-step manner from the most basic foundational concepts to more advanced and specific topics. It assumes that a strong general foundation is necessary before tackling complex or specialized subjects.

How It Works:

When faced with a broad field of study, learners following the Linear Method start at the beginning of the knowledge hierarchy. They aim to build a comprehensive understanding by covering all the foundational material before moving on to applications or specialized areas.

Example – Learning the Computer Science Behind Self-Driving Cars:

Learn Multivariable Calculus
Learn Linear Algebra
Learn Basic Computer Science Fundamentals
Learn General Machine Learning Concepts
Learn Computer Vision Concepts
Learn to Code in Python
Learn to Use TensorFlow
Learn How Computer Vision Is Applied in Self-Driving Cars
Learn to Write Code for Relevant Programs
Challenges with the Linear Method:

Inefficiency Due to Lack of Focus:
Breadth Over Depth: By starting with broad topics, you might spend time on aspects that are not directly relevant to your goal.
Time-Consuming: Covering all foundational areas can take a significant amount of time, delaying progress toward your specific objective.
Difficulty in Prioritization:
Unclear Relevance: Without a clear connection to the end goal, it's challenging to determine which topics are most important.
Overlearning: You may end up learning unnecessary material "just in case" it's needed later.
Abstract Learning Without Context:
Limited Engagement: Learning abstract concepts without seeing how they apply can reduce motivation and interest.
Poor Retention: Without practical application, it's harder to remember and understand new information.
Why It May Not Be Effective:

The Linear Method doesn't account for the specific needs of the learner's goal. It treats all foundational knowledge as equally important, which can lead to inefficiencies. Moreover, abstract learning without immediate application can make it difficult to see the relevance and maintain engagement.

The V-Method of Learning
Overview:
The V-Method of Learning is an approach that starts with a specific, tangible example related to the learner's end goal. It involves diving deep into this example, identifying gaps in understanding, and exploring underlying concepts as needed. The learner then builds back up to a comprehensive understanding of the example through focused study.

Visual Representation:

Imagine a "V" shape:

Left Side (Descent): Start at the surface with a specific example and dive deep into the underlying concepts.
Bottom Point: Reach the foundational knowledge required.
Right Side (Ascent): Climb back up by integrating these concepts to fully understand and recreate the example.
How It Works:

Start with a Specific Example:
Choose a well-documented case or project that represents your end goal. For self-driving cars, this could be an open-source code repository.
Attempt to Understand the Example:
Go through the example in detail, trying to grasp how it works.
Identify Knowledge Gaps:
Note areas or concepts you don't understand.
Research Underlying Concepts:
Dive into resources to learn about these specific areas.
Delve Deeper as Needed:
If initial research doesn't suffice, explore more fundamental concepts that underpin what you're studying.
Identify Patterns in Key Concepts:
Recognize recurring themes or core principles that are essential to understanding the example.
Study Relevant Foundational Concepts:
Focus on these key areas to build a solid understanding.
Integrate Knowledge and Reconstruct the Example:
Apply what you've learned to understand and potentially recreate the original example.
Advantages of the V-Method:

Relevance and Efficiency:
Targeted Learning: Only study concepts directly related to your goal.
Time-Saving: Avoid spending time on unrelated topics.
Contextual Learning:
Immediate Application: Concepts are learned in the context of their application.
Better Retention: Understanding how concepts work in practice aids memory.
Hierarchical Knowledge Building:
Structured Understanding: Build knowledge in layers, connecting practical examples to underlying theory.
Flexibility: Adjust depth of learning based on necessity.
Increased Engagement:
Motivation Through Progress: Seeing immediate relevance keeps you motivated.
Active Problem-Solving: Engaging directly with real-world examples enhances critical thinking.
Potential Challenges:

Overlooking Broader Context:
Focusing too narrowly might cause you to miss out on complementary knowledge.
Dependency on Quality Examples:
The effectiveness relies on the availability of good, well-documented examples.
Requires Self-Direction:
You need to be proactive in identifying and filling knowledge gaps.
3. Detailed Comparison
Starting Point:

Linear Method: Begins with general foundational knowledge without immediate connection to the end goal.
V-Method: Starts with a specific example directly related to the end goal.
Learning Path:

Linear Method: Sequential progression from basics to specifics.
V-Method: Non-linear exploration based on immediate needs; deep dives into relevant topics.
Efficiency:

Linear Method: Can be inefficient due to unnecessary breadth.
V-Method: More efficient by focusing on relevant areas.
Relevance:

Linear Method: Not always clear how early learning connects to the goal.
V-Method: Immediate relevance as learning is driven by the specific example.
Engagement:

Linear Method: May feel abstract and disengaging.
V-Method: Practical application keeps learning engaging.
Retention:

Linear Method: Abstract knowledge may be harder to retain.
V-Method: Contextual learning aids memory and understanding.
4. Why the V-Method Can Be More Effective
Focused Learning:

By starting with a specific example, you focus your efforts on what's necessary to achieve your goal.
Contextual Understanding:

Learning within the context of a real-world application helps make abstract concepts concrete.
Time Efficiency:

You avoid spending time on areas that are not directly relevant, making your learning process faster.
Active Learning:

Engaging with material actively by problem-solving enhances comprehension and retention.
Building a Knowledge Network:

By connecting specific examples to underlying principles, you create a web of understanding that's easier to navigate and expand upon.
Intrinsic Motivation:

Seeing progress toward your goal maintains motivation and reduces the likelihood of burnout.
5. Practical Application of the V-Method
Step-by-Step Guide Using the V-Method:

Let's walk through how someone might use the V-Method to learn about self-driving car software.

Step 1: Find a Specific Example

Locate an open-source self-driving car project on GitHub. Ensure it's well-documented and manageable in scope.
Step 2: Attempt to Understand the Project

Begin by reading the project's README file and documentation.
Try running the code if possible to see it in action.
Step 3: Identify What You Don't Understand

As you review the code, list any functions, libraries, or concepts that are unfamiliar.
Step 4: Research Underlying Concepts

For each unfamiliar item, seek out resources to learn about them.
Example: If you encounter a Python function you don't know, look up its documentation.
If the project uses TensorFlow and you haven't used it before, find tutorials on TensorFlow basics.
Step 5: Dive Deeper as Needed

If you realize you need to understand how neural networks work, study the basics of neural networks.
If neural networks rely on linear algebra concepts you're unfamiliar with, learn those specific aspects of linear algebra.
Step 6: Identify Key Foundational Areas

Notice if certain topics keep coming up (e.g., convolutional neural networks, image processing).
Prioritize learning these areas thoroughly.
Step 7: Build Back Up

Return to the project with your new knowledge.
See how the concepts you learned apply to the code.
Step 8: Recreate or Modify the Project

Using your understanding, try to recreate parts of the project or make modifications.
This hands-on practice solidifies your learning.
6. Illustrative Example
Scenario: Learning to Develop a Self-Driving Car Simulation

Starting Point:

You find a GitHub repository for a simple self-driving car simulation that uses machine learning to navigate a virtual environment.
Procedure:

Review the Simulation:
Run the simulation to see how the car behaves.
Observe how the code is structured.
Identify Knowledge Gaps:
You notice the use of reinforcement learning algorithms you aren't familiar with.
The code uses Python libraries like NumPy and OpenAI Gym.
Research Reinforcement Learning:
Learn the basics of reinforcement learning and how it's applied in the simulation.
Understand the Libraries:
Study how NumPy is used for mathematical operations.
Explore how OpenAI Gym provides environments for training algorithms.
Delve into Mathematical Concepts:
If the algorithms use concepts like Markov Decision Processes, learn the relevant probability and statistics.
Revisit the Simulation Code:
With your new knowledge, go back to the code and see how these concepts are implemented.
Modify the Simulation:
Try changing parameters or adding new features.
Perhaps implement a different reinforcement learning algorithm.
Outcome:

You gain a practical understanding of how self-driving car simulations are built.
You learn specific machine learning concepts relevant to your goal.
You develop programming skills in the context of a real project.
7. Addressing Potential Challenges
Ensuring Foundational Knowledge Isn't Overlooked:

While focusing on immediate needs, periodically assess if you're missing broader foundational concepts that could enhance your understanding.
Finding Quality Examples:

Look for well-documented projects with active communities.
Use educational platforms or tutorials that combine practical projects with theory.
Staying Organized:

Keep a learning journal to track what you've learned and what you need to explore further.
Set milestones to measure progress.
8. When to Use Each Method
Linear Method Suitable For:

Formal Education:
When following a curriculum designed to provide comprehensive knowledge.
Certifications or Exams:
When you need to cover all topics that might be tested.
V-Method Suitable For:

Goal-Oriented Learning:
When you have a specific project or skill you want to acquire.
Rapid Skill Acquisition:
When you need to learn efficiently due to time constraints.
9. Conclusion
The choice between the Linear Method and the V-Method depends on your learning objectives, time constraints, and personal preferences.

Why the V-Method Is Effective:

Efficiency: Focuses on learning what's necessary, saving time.
Engagement: Keeps you motivated through practical application.
Contextual Learning: Enhances understanding by connecting theory to practice.
Adaptive: Allows you to adjust the depth and breadth of learning based on your needs.
Key Takeaways:

Start with a clear goal and a specific example.
Be proactive in identifying and filling knowledge gaps.
Use practical application to reinforce learning.
Build a network of interconnected knowledge for better retention.
The V-Method transforms learning from a linear journey into a dynamic exploration tailored to your goals. It leverages curiosity and practical engagement to deepen understanding, making it a powerful approach for complex and broad fields where traditional methods may fall short.

Next Steps:

To apply the V-Method to your learning:

Identify Your Goal:
Be specific about what you want to achieve.
Find a Relevant Example:
Look for projects, code samples, or case studies that align with your goal.
Engage Actively:
Dive into the example, and don't be afraid to get your hands dirty with code or experiments.
Stay Curious:
Let your questions guide your learning journey.
Reflect and Iterate:
Regularly assess your understanding and adjust your learning path as needed.
Feel free to ask any questions or request further examples if there's anything still unclear!

how to determine which method to use for different task or goals

Determining Which Method to Use for Different Tasks or Goals

Choosing between the Linear Method of Learning and the V-Method of Learning depends on several factors related to your specific tasks or goals, your learning style, existing knowledge, time constraints, and the nature of the subject matter. Below is a detailed guide to help you determine which method may be more suitable for your situation.

Assess the Nature of Your Goal
Linear Method:
Long-Term Mastery: If your goal is to achieve a deep, comprehensive understanding of a subject area.
Structured Learning Path: When following a curriculum or preparing for standardized exams and certifications.
Complex Fields with Prerequisites: Fields like medicine, law, or theoretical physics that require a sequential build-up of knowledge.
V-Method:

Specific, Practical Objectives: When you have a particular project or skill you want to acquire quickly.
Immediate Application: If you need to apply knowledge to a task right away, such as developing a prototype.
Project-Based Learning: Ideal for goals that are best approached through hands-on experience.
Example:

Linear Method: Studying to become a licensed engineer, where you must understand fundamental principles before practicing.
V-Method: Learning to build a mobile app for your business within a short timeframe.
2. Evaluate Your Existing Knowledge and Experience
Linear Method:

Beginner Level: If you're entirely new to a subject and need to build up from the basics.
Need for Context: When understanding the foundational theories is crucial to make sense of advanced topics.
V-Method:

Some Background Knowledge: If you already have a basic understanding and can navigate new concepts with guided problem-solving.
Comfort with Learning on the Go: You're adept at filling knowledge gaps as they arise during practical application.
Example:

Linear Method: Someone new to programming starts with learning fundamental concepts before writing code.
V-Method: An experienced programmer learning a new language by modifying existing code.
3. Consider Your Learning Style and Preferences
Linear Method:

Sequential Learners: Prefer a step-by-step approach, building knowledge systematically.
Theoretical Understanding: Enjoy delving into abstract concepts before seeing their practical applications.
Comfort in Structure: Thrive in environments with clear guidelines and progression.
V-Method:

Hands-On Learners: Learn best by doing and experimenting with real examples.
Contextual Understanding: Grasp concepts better when they see how they are applied.
Flexible Approach: Comfortable with a less structured learning path and adaptable to changes.
Example:

Linear Method: A student who likes to read textbooks cover-to-cover before attempting exercises.
V-Method: A learner who jumps straight into building a DIY project to understand electronics.
4. Analyze Time Constraints
Linear Method:

Ample Time Available: When you have the luxury of time to explore subjects in-depth.
No Immediate Deadlines: Suitable for long-term educational goals without pressing timelines.
V-Method:

Limited Time: Need to achieve proficiency quickly due to deadlines or urgent needs.
Efficient Learning Required: Must prioritize learning relevant aspects to accomplish a task.
Example:

Linear Method: Preparing over several years for a career that requires extensive education.
V-Method: Learning essential presentation skills for an upcoming conference next month.
5. Examine the Complexity and Prerequisites of the Subject
Linear Method:

High Complexity, Strong Foundation Needed: Subjects that are intricate and require a solid base to understand advanced concepts (e.g., quantum mechanics).
Safety and Regulations: Fields where lack of foundational knowledge can lead to errors with serious consequences (e.g., civil engineering).
V-Method:

Modular Learning Possible: Areas where you can understand parts without needing to grasp the whole initially.
Low Risk of Errors: Fields where mistakes are easily correctable and don't have severe implications (e.g., hobbyist photography).
Example:

Linear Method: Learning to design aircraft, where physics and engineering principles are critical.
V-Method: Picking up cooking skills by trying out recipes and learning techniques as you go.
6. Determine the Learning Context
Linear Method:

Formal Education Settings: Schools, universities, or structured courses that follow a curriculum.
Mentorship Programs: Apprenticeships or programs where learning is guided progressively.
V-Method:

Self-Directed Learning: Personal projects, hobbies, or interests pursued independently.
Professional Development: Acquiring new skills for work-related tasks without formal training.
Example:

Linear Method: Enrolling in a degree program with mandatory courses.
V-Method: Learning new software to increase productivity at work through online tutorials.
7. Use a Decision-Making Guide
Ask Yourself the Following Questions:

What is my ultimate goal?
Is it broad and long-term (Linear) or specific and immediate (V-Method)?
How much do I already know about the subject?
Am I starting fresh (Linear) or building on existing knowledge (V-Method)?
What is my preferred learning style?
Do I like structured learning (Linear) or exploring through practice (V-Method)?
What time constraints am I under?
Do I have ample time (Linear) or need to learn quickly (V-Method)?
How complex is the subject, and what are the risks involved?
Does it require deep foundational knowledge (Linear) or can I learn incrementally (V-Method)?
What resources are available to me?
Do I have access to structured courses and mentors (Linear), or do I have real-world examples to study (V-Method)?
8. Recognize That a Hybrid Approach May Be Beneficial
Sometimes, combining both methods can provide the best results:

Start with the V-Method: Dive into a specific project to gain practical exposure and identify key areas where you lack knowledge.
Integrate the Linear Method: For those identified areas, return to foundational learning to build a deeper understanding.
Example:

While developing a web application (V-Method), you realize you need a better grasp of data structures. You then take a structured course on data structures (Linear Method) to enhance your skills.
9. Examples of Tasks and Suitable Methods
A. Learning to Play a Musical Instrument

V-Method: Start by learning to play a simple song, picking up techniques and theory as needed.
B. Studying for a Professional Certification

Linear Method: Follow the exam syllabus systematically to cover all required knowledge areas.
C. Building a Personal Website

V-Method: Use a website builder or templates to get started, learning HTML and CSS as you customize.
D. Understanding Advanced Mathematics

Linear Method: Begin with fundamental concepts and progress through increasingly complex topics to ensure comprehension.
E. Learning a New Language for Travel

V-Method: Focus on conversational phrases and common vocabulary for immediate use, learning grammar rules as needed.
10. Final Recommendations
Choose the Linear Method If:
Your Goal Requires Comprehensive Knowledge: Essential for academic degrees, professional qualifications, or careers needing in-depth expertise.
You Prefer Structure: Enjoy learning in a systematic way with clear progression.
Choose the V-Method If:
You Have a Specific Project: Ideal for achieving concrete goals efficiently.
You Thrive on Practical Engagement: Learn best by doing and applying knowledge directly.
Consider a Hybrid Approach:
Adaptability: Start with one method and switch or blend as your needs evolve.
Continuous Learning: Use the V-Method for immediate needs but plan for Linear Method learning to solidify and expand your knowledge over time.
Remember, the most effective learning method aligns with your personal goals, circumstances, and preferences. Be flexible and willing to adjust your approach as you gain more insight into what works best for you.

Tips for Success:

Set Clear Objectives: Define what you want to achieve and by when.
Monitor Your Progress: Regularly assess your understanding and adjust your learning plan accordingly.
Seek Feedback: Engage with others, join communities, or find a mentor to enhance your learning experience.
Stay Motivated: Choose topics and projects that genuinely interest you to maintain enthusiasm.
By thoughtfully considering these factors, you can select the learning method that optimizes your educational journey and leads to successful outcomes in your tasks or goals.

let us say i'm trying to use v learning as end goal to make this .what steps to take to make this using v learning method :

Iterative Text-based Editing of Talking-heads Using Neural Retargeting
XINWEI YAO, Stanford University
OHAD FRIED, The Interdisciplinary Center Herzliya
KAYVON FATAHALIAN, Stanford University
MANEESH AGRAWALA, Stanford UniversityTarget Video
Why yell or worry over silly items? [rest] Marvel movies are not cinema [rest]
Iteration 1
[rest] Marvel movies are not really
cinema [rest]
Iteration 2 Iteration N
Lip motion
from repository
40 sec
......
[rest] Marvel movies really are not
cinema [rest] [smile:1.0](wording edit) (wording edit) (mouth gesture)
Fig. 1. Our iterative text-based tool for editing talking-head video takes 2-3 minutes of a target video as input and is designed to support an iterative editing
workflow. On each iteration the user might edit the wording of the speech (itr. 1 and 2), refine mouth motions if necessary to reduce artifacts, manipulate the
performance by inserting mouth gestures (itr. N) or change the overall speaking style. Unlike previous techniques that require hours our tool takes about 40
seconds to generate each iteration, making it practical for users to explore a variety of different edits as they iterate. Our approach is to retarget lip motion
from a repository of source actor video to the target actor. The frame shown for each iteration corresponds to the red edit text/gesture below the frame.
We present a text-based tool for editing talking-head video that enables an
iterative editing workflow. On each iteration users can edit the wording of
the speech, further refine mouth motions if necessary to reduce artifacts and
manipulate non-verbal aspects of the performance by inserting mouth ges-
tures (e.g. a smile) or changing the overall performance style (e.g. energetic,
mumble). Our tool requires only 2-3 minutes of the target actor video and it
synthesizes the video for each iteration in about 40 seconds, allowing users
to quickly explore many editing possibilities as they iterate. Our approach
is based on two key ideas. (1) We develop a fast phoneme search algorithm
that can quickly identify phoneme-level subsequences of the source repos-
itory video that best match a desired edit. This enables our fast iteration
loop. (2) We leverage a large repository of video of a source actor and de-
velop a new self-supervised neural retargeting technique for transferring the
mouth motions of the source actor to the target actor. This allows us to work
with relatively short target actor videos, making our approach applicable in
many real-world editing scenarios. Finally, our refinement and performance
controls give users the ability to further fine-tune the synthesized results.
CCS Concepts: • Computing methodologies → Motion processing; Com-
putational photography; Reconstruction; Graphics systems and interfaces.
Authors’ addresses: Xinwei Yao, xinwei.yao@cs.stanford.edu, Stanford University,
Department of Computer Science; Ohad Fried, ohad.fried@post.idc.ac.il, The Inter-
disciplinary Center Herzliya, Department of Computer Science; Kayvon Fatahalian,
kayvonf@cs.stanford.edu, Stanford University, Department of Computer Science; Ma-
neesh Agrawala, maneesh@cs.stanford.edu, Stanford University, Department of Com-
puter Science.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@acm.org.
© 2020 Association for Computing Machinery.
XXXX-XXXX/2020/11-ART $15.00
https://doi.org/10.1145/nnnnnnn.nnnnnnn
Additional Key Words and Phrases: text-based video editing, talking-heads,
phonemes, retargeting
ACM Reference Format:
Xinwei Yao, Ohad Fried, Kayvon Fatahalian, and Maneesh Agrawala. 2020.
Iterative Text-based Editing of Talking-heads Using Neural Retargeting. 1,
1 (November 2020), 14 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn
1 INTRODUCTION
Tools for editing talking-head video using transcripts have made it
possible to easily remove filler words, emphasize phrases, correct
mistakes, and try different wordings of the speech [Berthouzoz et al .
2012; Fried et al . 2019; Suwajanakorn et al . 2017; Thies et al . 2020].
Many of these tools can synthesize high-quality results that closely
match the appearance of the unedited video. Such tools have the
potential to enable a variety of post-capture editing applications
including re-phrasing dialogue in a film scene, dubbing commercials
to a new language, developing dialogue for a conversational video
assistant, and fixing wording errors in an online lecture.
Yet, current transcript-based video editing tools are impractical
for use in many real-world editing scenarios for four main reasons.
(1) Slow feedback loop hinders iterative editing. Synthesizing the
edited result at high-quality is often extremely slow. For example,
while viewers report that Fried et al.’s [2019] results appear very
realistic, their approach takes hours to generate a few seconds of
edited video. The slow feedback loop — time between specifying an
edit and seeing the result — significantly hinders iterative editing
(e.g. trying different phrasings of dialogue).
(2) Require hours of target talking-head video. To produce real-
istic results, many of these tools [Fried et al . 2019; Suwajanakorn
et al. 2017] require hours of video of the target talking-head actor.
, Vol. 1, No. 1, Article . Publication date: November 2020.
arXiv:2011.10688v1 [cs.CV] 21 Nov 2020
2 • Yao, et al.
Some tools further require the actor to speak a set of specialized
phrases (e.g. TIMIT corpus [Garofolo et al. 1993]). In practice how-
ever, many video editing projects lack access to such large amounts
of target actor video.
(3) Missing controls for refining results. None of these editing
tools provide controls for manually refining the lip motions of the
synthesized results, making it impossible to fix objectionable arti-
facts these editing tools sometimes generate (e.g. mouth doesn’t
fully close on \m, \b, \p phonemes).
(4) Missing controls for adjusting non-verbal performance. None
of these editing tools include controls for adjusting the target actor’s
non-verbal performance by inserting mouth gestures (e.g. a smile)
or changing the overall speaking style (e.g. mumbling, energetic).
In this work we present an iterative talking-head video edit-
ing tool that explicitly addresses all four of these issues. While
our approach builds on the high-quality synthesis technique of
Fried et al. [2019], we make several new contributions. We signifi-
cantly reduce the time required to synthesize video (from hours to
about 40 seconds for a 6 word edit) by developing a fast algorithm
for searching the source repository for the desired lip motions. We
lower the data requirement on the target actor video (2-3 minutes
are usually enough) by leveraging a large repository of video from a
source actor and use a new self-supervised neural retargeting tech-
nique to transfer their lip motions to the target actor. We provide
controls to refine results by allowing users to smooth over jumpy
transitions and force mouth closure on the results of the automated
synthesis pipeline. Finally, we enable insertion of non-verbal mouth
gestures with the same text interface, as well as controls to switch
between different speaking styles by using a version of the source
repository with the desired style.
As shown in Figure 1 our tool enables an iterative editing work-
flow. Given a short video of the target actor, the user can edit the
transcript and our tool synthesizes the corresponding video in un-
der a minute. The user can inspect the feedback and further adjust
wording, refine the lip motions and/or insert mouth gestures and
quickly see how the adjustment affects the synthesized video. Note
that our work focuses on generating video from text; to obtain the
corresponding speech audio, we rely on either having access to the
actor speaking the new content (e.g. from a prerecorded library
of the actor’s speech or recorded by the actor in real-time during
editing), text-to-speech voice synthesis [van den Oord et al . 2016]
or voice cloning [Jia et al. 2018; Kumar et al. 2019].
We demonstrate a variety of iterative editing sessions facilitated
by our tool and we conduct user studies which show that our syn-
thesized results are rated as “real” for 56.2% of the sentence-long
edits and for 64.9% of the phrase-long edits – slightly better than the
previous state-of-the-art approach of Fried et al. [2019]. Together
these results suggest that our algorithm provides the speed, data
efficiency and controls necessary for a practical iterative editing
workflow while maintaining high-quality synthesis results.
2 RELATED WORK
Video-driven talking-head synthesis. A common approach to syn-
thesizing a talking-head video is to use a “driving” video from a dif-
ferent actor that has the desired motion, expression and speech, and
transfer those elements to the primary talking-head. Early attempts
used facial landmarks from a video to retrieve frames of a different
person and play them back directly [Kemelmacher-Shlizerman et al .
2010] or after warping [Garrido et al. 2014]. Opting for a lower data
requirement, several approaches synthesize video given only one
or a few photos of the target person, either by morphing and blend-
ing [Averbuch-Elor et al . 2017] or using neural networks [Geng et al .
2018; Pumarola et al. 2019; Wiles et al. 2018; Zakharov et al. 2019].
These methods are successful in producing short expression videos,
but are less convincing for full sentences. Several approaches use
a tracked head model, to decouple properties (e.g. pose, identity,
expressions) to produce convincing results [Garrido et al. 2015; Kim
et al . 2019, 2018; Thies et al . 2016; Vlasic et al. 2005]. We similarly
use a tracked head model to decouple such properties. All of these
previous methods require a driving video to specify the desired
output head motion and expression. In this work we specify those
properties via text, which is often a simpler, lower-cost interaction.
Voice-driven talking-head synthesis. Another approach for talking-
head synthesis is to drive it with voice. The pioneering work of
Bregler et al. [1997] creates talking-heads through a combination
of alignment and blending, and was improved upon in various fol-
lowups [Chang and Ezzat 2005; Ezzat et al. 2002; Liu and Ostermann
2011]. Others have used human voice-driven synthesis to dub non-
humans [Fried and Agrawala 2019]. Several methods synthesize
a talking-head given only one or a few video frames in addition
to the voice track [Chen et al. 2019; Chung et al . 2017; Song et al .
2019; Vougioukas et al. 2018, 2019; Zhou et al . 2019]. However, the
result is a fixed frame with a moving inner-face region, or a tightly
cropped head, and is easily distinguishable from realistic video.
Suwajanakorn et al. [2017] demonstrate that using a large reposi-
tory of video (17 hours) can produce convincing synthesis results.
In work concurrent to ours, Thies et al. [2020] produce video from
speech. We compare to their results in Section 5.3. None of these
voice-drive methods provide refinement and performance controls
that are essential for iterative editing.
Text-driven talking-head synthesis. Most related to our work are
methods that perform text-based video editing and synthesis. Wang
et al. [2011] synthesize a talking-head, and allow control over facial
expressions, but the head is floating in space and is not part of a pho-
torealistic video. Mattheyses et al. [2010] synthesize audio-visual
speech from text, but the resulting videos have no head motion
making them unrealistic. Berthouzoz et al. [2012] can edit talking-
head video by cutting, copying and pasting transcript text. However,
they do not allow synthesis of new words to change phrasing or
fix flubbed lines. ObamaNet [Kumar et al . 2017] synthesizes both
audio and video from text, using a large dataset of 17 hours of the
president’s speeches. While predominantly an audio-based method,
Thies et al. [2020] also show text-based results by incorporating a
text-to-speech system. The work of Fried et al. [2019] most closely
resembles ours, but it requires over 1 hour of target video and takes
, Vol. 1, No. 1, Article . Publication date: November 2020.
Iterative Text-based Editing of Talking-heads Using Neural Retargeting • 3repository
target video
Phoneme
Alignment
M UW1 V IY0 Z
3D Head Model
Registration
B R AW1 N F AA1 ...
S IH1 N AH0 M AH0 ...
repository
target video
Fig. 2. Our preprocessing pipeline annotates both the source repository
and the target video with phonemes and registers a 254-parameter 3D head
model to each frame of each video.
hours to produce a result, while our tool requires 2–3 minutes of
target video and produces results in about 40 seconds. We compare
our results to both Thies et al. and Fried et al. in Sections 5.3 and 5.4
and find that the quality of our results is similar to both of these tech-
niques. Moreover we provide refinement and performance controls
that are missing in all previous text-driven talking-head synthesis
tools, but are critical for a practical video editing tool.
3 METHOD
Given a short talking-head video of a target actor (often 2-3 minutes
in length), and an edit of the video transcript, our system synthe-
sizes new video of the target actor matching the edit. An edit is
specified as a replacement of one continuous sequence of words in
the original transcript with a new sequence of words. Since a short
target actor video is unlikely to contain all the lip motions necessary
to convincingly synthesize the sequence of phonemes in the edit,
we leverage a large repository of video from a different, source ac-
tor. Specifically, we pre-capture an hour of a source actor speaking
the TIMIT corpus [Garofolo et al. 1993] which includes the most
common phoneme combinations (coarticulations) in English and
we retarget their lip motions to the target actor during synthesis.
Our approach for quickly synthesizing the edited result is based
on the approach of Fried et al. [2019] but involves several critical
modifications. As in Fried et al., our preprocessing pipeline (Figure 2)
annotates both the repository and target videos with phonemes and
registers a parametric 3D head model to the face in each frame of
each video. Our synthesis pipeline (Figure 3) provides a new, fast
phoneme search algorithm that finds subsequences of phonemes in
the source video that match the desired edit. It then stitches together
the corresponding parameters of the 3D head model for the source
actor across subsequence boundaries to smooth the lip motions. We
introduce a new self-supervised neural retargeting step that adapts
the parameters representing the lip motion of the source actor to
those of the target actor and blend the resulting parameters into
the target video. Finally we render photorealistic frames from the
parameters using neural rendering [Tewari et al. 2020].
We briefly summarize how we adapt each step in Fried et al.’s
pipelines to our problem in Sections 3.1 and 3.2. We then present
the details of our new algorithms; fast phoneme search and stitch-
ing in Section 3.3 and neural retargeting algorithm in Section 3.4.
In Section 3.5, we describe the iterative refinement and performance
controls enabled by our approach.
3.1 Preprocessing Pipeline
Our preprocessing pipeline annotates each frame of the repository
and the target videos in two main steps, (1) phoneme alignment
and (2) registration of a parametric 3D head model. These resulting
phoneme and face parameter annotations are used by our synthesis
pipeline to establish correspondences between the target video and
source repository. Note that the repository is only annotated once
and the resulting annotated repository is then bundled as part of the
system. In contrast, the target video must be annotated each time a
new target video is given as input.
Phoneme Alignment. The phoneme alignment step takes as input
a video (repository or target) paired with its text transcript, and
computes the identity and timing of the phonemes in the video.
Specifically, we use P2FA [Rubin et al . 2013; Yuan and Liberman
2008] to convert the transcript into phonemes and align them to the
audio speech track of the video. This produces an ordered sequence
𝑉 = (𝑣1, . . . , 𝑣𝑛 ) of phonemes, where each phoneme 𝑣𝑖 contains its
name, start time and end time. If the transcript is not available, we
can obtain one using a transcription service such as Google Cloud
Speech-to-Text [2020a], or rev.com [2020].
3D Head Model Registration. We fit a parametric head model [Blanz
and Vetter 1999; Thies et al . 2016] to each frame of video using a
monocular head tracker [Garrido et al. 2016]. At every frame, the
fitted model includes 80 parameters for 3D facial geometry, 80 for
facial reflectance, 3 for head pose, 27 for scene illumination and 64
for face and lip expressions. In the fitting procedure we hold the
facial geometry and reflectance parameters constant across all the
frames of the same actor, but we allow the pose, illumination and
expression to vary across time.
3.2 Synthesis Pipeline
Our synthesis technique is based on matching phonemes in the
edit to phonemes in the repository. Therefore, we first convert the
input text of the edit from words into a sequence of phonemes
𝑊 = (𝑤1, . . . , 𝑤𝑚 ), where each phoneme contains its name, start
time and end time. Specifically, we convert the edit into audio using
either text-to-speech voice synthesis [goo 2020b; van den Oord et al .
2016] or voice cloning techniques [lyr 2020; Kumar et al. 2019], and
then apply P2FA [Rubin et al. 2013; Yuan and Liberman 2008] to
time-align the resulting speech to the phonemes of the edit. Note
that our synthesis algorithm only uses the timing of the phonemes
and does not use any other aspect of the synthesized speech audio
signal. If the user has access to the audio of the target actor saying
the new content (either from a prerecorded library of the actor’s
speech or recorded by the actor in real-time during editing) they
can run our fast synthesis pipeline with the timings obtained from
the real-voice recordings.
Fast Phoneme Search and Stitching. The fast phoneme search and
stitching step is designed to quickly find the best subsequences
of phonemes in the repository and then stitch together the cor-
responding expression parameters of the source actor 3D head
model, in order to produce the the edit 𝑊 we wish to synthesize.
Our new algorithm operates three orders of magnitude faster than
Fried et al. [2019] and finds the best repository subsequences in
, Vol. 1, No. 1, Article . Publication date: November 2020.
4 • Yao, et al.Neural Rendering
composite
Fast Phoneme Search
and Stitching
... Expansion to
Full Parameters
...
other face
parameters
background
pixels
target actor
expression
parameters
source actor
expression
parameters
... and curious volume of ...
... about the oblong box just ...
fox result videofox
GAN
... quick brown fox jumped ...
Aligned phonemes and
expression parameters
network
Neural
Retargeting
... quick brown spider fox
jumped ...
edit
repository
... N EH1 V ER0 M AO1 R ...
target video
... K W IH1 K B R AW1 N ...
Fig. 3. Our synthesis pipeline adapts the pipeline of Fried et al. [2019] by introducing a fast phoneme search and stitching step (yellow), and a self-supervised
neural retargeting step (yellow). The input to our pipeline is a target video (green) and an edit (green) – here changing the word “spider” to “fox”. Our fast
phoneme search finds phonemes in the repository that visually match the desired edit – here the “v” in volume and the “ox” in box. We then stitch together
the corresponding facial expression parameters of the 3D head model for the source repository actor, and use a novel neural retargeting model to translate
those parameters into those for the target actor. Next, we expand the retargeted expression parameters to the full face parameters (e.g. pose, illumination) for
the target actor. Finally we render photorealistic frames from the parameters using a neural rendering approach that first composites the lower part of the
face rendered from the 3D head model, with background pixels from the original target video and then uses a generative adversarial network (GAN) to map
the composites to photorealistic frames.
seconds rather than hours. We present this fast algorithm in detail
in Section 3.3.
Neural Retargeting. The retargeting step converts a sequence
of expression parameters for the source actor into those for the
target actor. We introduce a learned retargeting model, trained in a
self-supervised manner from corresponding pairs of repository and
target video sequences and transforms the expression parameters
as detailed in Section 3.4. The result is a sequence of target actor
face expression parameters corresponding to the edit.
Expansion to Full Parameters. Next, we combine the synthesized
target actor expression parameters with geometry, reflectance, pose
and illumination parameters from the input target video to produce
a sequence of full face parameters for the target actor corresponding
to the edit. Specifically, we take an interval of frames around the
edit location in the target video, retime it to account for the duration
of the edit, and use the geometry, reflectance, pose and illumination
parameters from the retimed interval.
Neural Rendering. The neural rendering step takes the sequence
of full face parameters for the target actor and first generates a
composite image in which the lower face region is a rendering
of the 3D head model, while the upper part of the head and the
surrounding background are from the original target video, but
retimed to match the length of the edit. It then uses a GAN trained
on the target video to complete the image-to-image translation from
composite image to photorealistic frame.
3.3 Fast Phoneme Search and Stitching
Our synthesis pipeline takes an edit 𝑊 specified as a sequence of
phonemes with timings (𝑤1, . . . , 𝑤𝑚 ) and starts by finding match-
ing subsequences of video in the source repository 𝑉 , that can be
combined to produce𝑊 . More precisely, we partition the edit𝑊 into
phoneme subsequences (𝑊1,𝑊2, . . . ,𝑊𝑘 ) and for each subsequence
𝑊𝑖 find its best match 𝑉𝑖 in the repository 𝑉 . Fried et al. [2019] use
a brute-force method that considers all possible partitions split(𝑊 )
of the edit 𝑊 , and all possible matches with subsequences of 𝑉 to
find (𝑉1, 𝑉2, . . . , 𝑉𝑘 ) that minimizes the objective:
𝐶 (𝑊 , 𝑉 ) = min
(𝑊1,𝑊2,...,𝑊𝑘 ) ∈split(𝑊 )
(𝑉1,𝑉2,...,𝑉𝑘 )
𝑘∑︁
𝑖=1
𝐶match (𝑊𝑖 , 𝑉𝑖 ) + 𝐶len (𝑊𝑖 )
where 𝐶match is a custom Levenshtein edit distance [1966] between
two phoneme subsequences that takes into account the phoneme
label, the viseme label and the timing difference, and 𝐶len penalizes
short subsequences. In order to find subsequences that transition
well at their endpoints, during the search, we expand each subse-
quence 𝑊𝑖 by a single phoneme on either end. Thus, adjacent sub-
sequences overlap by two context phonemes. We find that this new
context expansion approach better captures co-articulation effects
between the subsequences, than the algorithm of Fried et al. which
does not use context expansion (see user studies in Section 5.4).
We further modify Fried et al.’s search algorithm in three key
ways to obtain a speedup of over 3 orders of magnitude: (1) we
propose a fast alternative to to the Levenshtein distance, (2) we
reduce the size of the search space on 𝑊𝑖 and, (3) given 𝑊𝑖 , we
use a viseme-based indexing scheme to quickly find the optimal
𝑉𝑖 . Finally, we stitch together source actor expression parameters
corresponding to the 𝑉𝑖 s to produce a single coherent sequence of
expression parameters.
(1) Fast alternative to edit distance. The full Levenshtein edit dis-
tance allows substitution, insertion and deletion of phonemes when
computing 𝐶match . However, we have observed that when the
matching subsequences between the edit𝑊 and the repository video
𝑉 contain phoneme insertions or deletions, the final synthesized
video appears out-of-sync with the audio; it either contains extrane-
ous mouth motions due to phoneme insertion, or it misses mouth
motions due to deletion. In practice we find it is beneficial to disal-
low insertions and deletions and only allow phoneme substitutions.
Given a subsequence 𝑊𝑖 of the edit 𝑊 , this approach forces 𝐶match
to only consider subsequences 𝑉𝑗 of the repository 𝑉 that contain
the same number of phonemes as 𝑊𝑖 . We can therefore replace the
Levenshtein distance with the sum of element-wise substitution
, Vol. 1, No. 1, Article . Publication date: November 2020.
Iterative Text-based Editing of Talking-heads Using Neural Retargeting • 5
cost which requires linear time in the number of phonemes rather
than the quadratic time required for computing the full Levenshtein
distance [Wagner and Fischer 1974].
(2) Reduce search space for partitioning. The brute force search
considers all possible partitions of 𝑊 into (𝑊1, . . . ,𝑊𝑘 ). But, an
extremely long edit subsequence 𝑊𝑖 is unlikely to have a good
match with a repository subsequence 𝑉𝑖 . Thus, we can reduce the
search space of possible partitions by capping the maximum length
of the 𝑊𝑖 ’s to 𝐿. In our experience, 99% of the matches found by
the brute force search are of length 6 or less, and we therefore set
𝐿 = 6. This approach reduces the number of partitions to search.
More importantly, it typically reduces the number of distinct 𝑊𝑖 ’s
we need to consider by over an order of magnitude, especially when
𝑊 the full edit sequence is itself very long.
(3) Viseme-based index to search repository. For each edit subse-
quence 𝑊𝑖 we consider in our search space, we must find the optimal
𝑉𝑖 in the repository with respect to 𝐶match . Instead of checking all
possible subsequences in the repository, we impose an additional
constraint on 𝑉𝑖 that restricts the set of 𝑉𝑖 we consider to only
the most likely match candidates and allows us to build an index
structure on the set of 𝑉𝑖 to retrieve the likely candidates quickly.
As in Fried et al., our 𝐶match cost function considers phonemes
to match when they appear visually similar – that is, their corre-
sponding visemes match. By imposing the restriction that 𝑉𝑖 start
with the same viseme n-gram as 𝑊𝑖 we can pre-compute an index
for the repository using viseme n-grams as the key and the location
of the n-gram in the source repository video as the value. At search
time, we look up all possible candidate 𝑉𝑖 ’s using this index and only
compute the 𝐶match for them. While this indexing approach speeds
up the search significantly it also reduces the space of subsequence
matches the search considers. In general, the longer the n-gram key,
the stronger the reduction and the more likely it is that no good
match will be found. In practice, we find that using a bi-gram index
best balances this trade-off between search speed and result quality.
Stitching. After we find the best matching phoneme subsequences
𝑉1, . . . , 𝑉𝑘 from the repository, we look up the expression parameters
of the source actor’s 3D head model corresponding to each phoneme,
and linearly re-time them to match the phoneme durations specified
in the edit. We then stitch together adjacent subsequences by first
linearly blending the the expression parameters across the overlap-
ping context phonemes and then applying a Gaussian filter over a
window of 4 frames around the transition boundaries between the
subsequences to further smooth the transition.
We have found in practice, that errors in tracking expression pa-
rameters, timing misalignments and our linear blending can some-
times fail to fully close the mouth of the 3D head model at the begin-
ning of \m, \b, and \p phonemes. However, proper mouth closures
for these phonemes is crucial for producing perceptually realistic
results [Agarwal et al. 2020]. We therefore force the desired mouth
closure by linearly blending in a closed-mouth expression from the
repository at the beginning of all \m, \b, and \p phonemes with a
default length of 2 frames. Note that the closed-mouth expressions
are manually annotated once during repository preprocessing and...... ......
...... ......
repository
target video
S
TA TB
Fig. 4. To automatically build training data for our neural retargeting model,
we consider a sequence of phonemes 𝑆 (red) in the repository and find up to
two best matches 𝑇 𝐴 and 𝑇 𝐵 (red) in the target video. We then find matches
for the next repository sequence (green) starting at the last phoneme of the
previous sequence. The overlap allows us to capture the phoneme transition
out of the final phoneme in the red subsequence.
we automatically use the one closest to the parameter values at the
point of insertion.
Overall, the fast search strategy reduce the runtime of the phoneme
search process by three orders of magnitude compared to the brute-
force approach. It takes around 5 seconds to find and stitch snippets
𝑉1, . . . , 𝑉𝑘 for an edit 𝑊 of 20 phonemes in an hour-long repository.
3.4 Neural Retargeting
Given a stitched-together sequence of face expression parameters
for the source actor in the repository, the goal of retargeting is to gen-
erate a matching sequence of expression parameters for the target
actor. We have developed a self-supervised neural network model for
retargeting and in Section 5.1 we show that using a neural network
for retargeting produces higher-quality results than baseline meth-
ods such as directly copying source actor expression parameters to
the target actor, or applying a linear retargeting model.
Self-supervised training data. To train our retargeting network we
require sequences of expression parameters for the source and target
actor that correspond to one another with respect to their mouth
motions. Assuming that uttering the same sequence of visemes
will produce similar mouth motions, we automatically construct
corresponding pairs of training data by finding the longest matching
sequences of phonemes between the source repository video and
the target video, as follows.
Since we apply our retargeting model to a stitched-together sub-
sequences of phonemes that can come from anywhere in the source
repository, we would like the training sequences to cover as much of
the repository as possible. Therefore, we start from the first phoneme
𝑠1 in the repository, and find the longest sequence 𝑆 = (𝑠1, . . . , 𝑠𝑘 ) for
which there is at least one corresponding sequence 𝑇 = (𝑡1, . . . , 𝑡𝑘 )
in the target video where 𝑡𝑖 and 𝑠𝑖 belongs to the same viseme group
(i.e. phonemes that require the same lip expressions are in the same
viseme group). We take up to two best matches in the target video
with respect to 𝐶match score defined in Section 3.3, and call them
𝑇 𝐴 = (𝑡𝐴
1 , . . . , 𝑡𝐴
𝑘 ) and 𝑇 𝐵 = (𝑡𝐵
1 , . . . , 𝑡𝐵
𝑘 ). We add the pairs (𝑆,𝑇 𝐴)
and (𝑆,𝑇 𝐵 ) to the set of phoneme sequences in correspondence,
and continue the scan through the source repository at 𝑣𝑘 , until
we finish scanning through the entire repository for subsequence
matches (Figure 4). To quickly find the best 𝑇 𝐴 and 𝑇 𝐵 sequences
in the target video, we apply fast search techniques discussed in
Section 3.3. Finally, to convert each resulting phoneme sequence
, Vol. 1, No. 1, Article . Publication date: November 2020.
6 • Yao, et al.F
s i
t i
unroll F0
0
00 s 1
t 1
F0
t 2
F
t3
...... F
tT-2
tT-1
tT
s 2
s 1
0 s 1 s 2 s 3 s T-2 s T-1 s T
Fig. 5. The retargeting network with the unrolled recurrent unit 𝐹 , which
looks two frames back at each time step and uses the previous two outputs
as its state. The diagram shows how the RNN processes the input of𝑇 frames
of source actor parameters (𝑠1, 𝑠2, . . . , 𝑠𝑇 ) and produces the prediction for
𝑇 frames of target actor parameters (𝑡1, 𝑡2, . . . , 𝑡𝑇 ).
pair into a parameter sequence pair, we linearly retime the target
video phoneme interval to the corresponding repository interval
and similarly interpolate the expression parameters. The retargeting
model is trained once for each new target video.
Neural Network Architecture. We employ a recurrent neural net-
work (RNN) manually unrolled for 𝑇 time-steps to encode the tem-
poral dynamics of the facial expressions and regress from source
parameters to target parameters (Figure 5). The resulting network
takes as input 𝑇 frames of 64 expression parameters for the source
actor, and outputs 𝑇 frames of 64 parameters for the target actor.
At the core of the network is the recurrent unit that is made up
of 3 fully connected layers of 1024 nodes with relu activation. The
inputs to the recurrent unit are 64 expression parameters of the
current time-step, as well as the parameters and outputs of the re-
current unit for the previous 𝐻 time-steps. To facilitate learning of
deviations from the identity transformation, the output layer of the
recurrent unit with 64 nodes and no activation produces the residual
values that are added to the input source parameters element-wise
to obtain the prediction for the target parameters for the current
time-step. We zero-pad the unavailable inputs at the first 𝐻 time-
steps. We empirically found that setting 𝐻 = 2 and 𝑇 = 7 produced
high-quality retargeting results.
Loss function. Our loss function is a linear combination of a data
term and a temporal regularizer with regularization weight 𝜆:
L = 1
𝑇
𝑇∑︁
𝑖=1
∥𝐹 (𝑠𝑖 ) − 𝑡𝑖 ∥1 + 𝜆∥𝐹 (2) (𝑠𝑖 )∥2
where 𝑠𝑖 , 𝑡𝑖 are 64-dimensional vectors representing the 𝑖th time-
step of the source and target parameters respectively, 𝐹 (𝑠𝑖 ) is a
64-dimensional vector of the predicted target actor parameters and
𝐹 (2) (𝑠𝑖 ) is a 64-dimensional vector that is the second temporal dif-
ference vector, or acceleration, of the predicted values. Empirically
we found that using an 𝐿1 norm for the data term significantly
outperforms 𝐿2 by generating more expressive motions and better
preserving mouth closures. The temporal regularization term is
needed to make the network predict temporally stable parameters.
Hyperparameters and training. Since the network takes a fixed-
size input of 𝑇 frames, we run a sliding window on each matching
parameter sequences to obtain the training examples. Experimen-
tally we set the temporal window 𝑇 = 7 frames. For training we set
𝜆 = 10, and dropout rate at 25%, 50% and 25% for the three layers
in the recurrent unit, respectively. To train the network we use
stochastic gradient descent with the Adam solver [Kingma and Ba
2015] and set an initial learning rate of 0.0002 with an exponential
decay rate of 0.5. We employ gradient clipping [Pascanu et al. 2013]
to avoid exploding gradients. We train the network with minibatch
size 100 and training typically converges within 100 epochs.
Inference. At inference time, we convert a sequence of source
actor expression parameters into target actor parameters. Since
our retargeting model accepts fixed-size 𝑇 frames of input and
produces 𝑇 frames of output, we run a sliding window of length
𝑇 over the new sequence of source actor expression parameters
at inference time. Each frame is covered by exactly 𝑇 such sliding
windows. In order to obtain a more temporally stable output, at
each frame we average the 𝑇 outputs produced by those 𝑇 sliding
windows as the final output of the frame. The result is a synthesized
sequence of target actor expression parameters that animate the
face to speak the new content of the edit with the desired timing. We
then proceed to expand these expression parameters into parameters
for the whole head and use neural rendering to generate the video
frames (Section 3.2).
Training our retargeting model typically requires 2–3 minutes
of target actor video speaking arbitrary speech to produce high-
quality synthesis results. Retargeting allows our tool to leverage the
large repository of controlled source actor video (speech consists
of TIMIT sentences) to generate the target actor lip motions and
opens our tool to many practical applications where large amounts
of controlled target actor video are not available.
3.5 User Controls
This speed of our synthesis pipeline opens the door to interactive
user controls for iteratively refining the edit and further manipulat-
ing the the facial performance.
Refinement Control: Smoothing jumpy transitions. Our synthesis
pipeline stitches together different subsequences of expression pa-
rameters from the source repository by smoothing over a window of
4 frames around the transition boundary (Section 3.3 Fast Phoneme
Search and Stitching). At times however, some transitions may still
appears jumpy even after this smoothing. We allow the user to
inspect the result and further refine it by manually specifying the in-
terpolation radius (in number of frames) at user-specified transition
boundaries to better smooth out visibly jumpy transitions.
Refinement Control: Adjusting mouth closure. As noted in Sec-
tion 3.3 mouth closure on \m, \b and \p phonemes is crucial for the
mouth motions to appear realistic. Thus our stitching procedure
automatically inserts 2 closed mouth frames at the beginnings of
these three phonemes to ensure the mouth closes correctly for them.
We further allow users refine any synthesized result by extending
(or reducing) the length of the inserted closed mouth frames.
Performance Control: Inserting mouth gestures. Users can also in-
sert non-verbal mouth gestures (e.g. a smile) into an edit. To enable
such performance control we manually annotate mouth gestures
including rest, closed-mouth smile, regular teeth-showing smile, big
, Vol. 1, No. 1, Article . Publication date: November 2020.
Iterative Text-based Editing of Talking-heads Using Neural Retargeting • 7
open-mouth smile, sad, scream, mouth gesturing left and mouth ges-
turing right, in the repository video. These segments can then be
retrieved by our fast phoneme search just like any other phoneme
annotation.
Since the annotations are on the repository, this manual annota-
tion only needs to be done once during repository preprocessing.
Note however, that users do not label the target video with these
mouth gestures and our retargeting network is never explicitly
trained with corresponding pairs of mouth gesture frames between
the repository and target videos. Nevertheless, we have found that
our retargeting network is able to generalize to unseen expressions
and produce good quality expression parameters for the target actor.
With these annotations, the user can add special mouth gesture
directives like [smile] anywhere in their edit of the transcript, and
our tool constructs a “generalized phoneme” edit sequence 𝑊 that
contains phonemes and such directives. Any mouth gesture that
appears in 𝑊 is given a default duration of 0.5 seconds that the user
can override with an explicit duration e.g. [smile:1.5s]. We employ
a special substitution cost in 𝐶match described in Section 3.3 for
“gesture phonemes” that is set to infinity for a non-match to ensure
that we retrieve the correct “phoneme” match for the gesture. When
there are multiple candidates, 𝐶match takes duration into account
and picks the gesture with duration closest to the query. The rest of
the editing pipeline (Section 3.2) is otherwise unmodified.
Performance Control: Adjusting speaking style. Our tool allows the
user to select a different speaking style for the synthesized result by
using a version of the repository with the desired style. In addition
to the default repository which captures a “neutral” speaking style,
we have recorded an “energetic” repository of our source actor with
more pronounced mouth movements, and a “mumble” repository
with significantly less mouth movements. Figure 6 (third row) shows
frames from these alternative repositories.
Importantly, we do not have to retrain our neural retargeting
model (Section 3.4) for each additional style repository. We train
this retargeting model once using only the default neutral repository.
We have found that our default retargeting model can extrapolate
to retarget subsequences of source actor face parameters retrieved
from other speaking style repositories of the same actor. Moreover,
the other repository videos can be captured at different times, with
different background and the source actor can even be wearing dif-
ferent clothes or have a different hairstyle. Thus, our tool generates
videos with different speaking styles by using one of the alternative
style repositories in the fast phoneme search step, but leaves the
remainder of the synthesis pipeline unchanged.
3.6 Implementation Details
We implemented the fast phoneme search in Python and both
our neural retargeting model and the GAN renderer in Tensor-
Flow [Abadi et al . 2015]. The monocular head tracker and renderer
are written in C++ with shader language extensions.
In preprocessing the repository and target videos, phoneme align-
ment takes one third of the video time, and face registration takes
110 ms per frame. It takes 30 minutes to generate training data for
our neural retargeting model and another 30 minutes to train it on
#words #phonemes #frames search (s) render (s) total (s)
1 4 24 1.51 9.96 12.39
3 15 49 2.67 12.94 20.30
6 25 72 5.32 17.74 28.95
8 39 105 7.57 24.33 37.97
10 49 134 11.60 31.19 50.60
Table 1. Runtime of our tool on a variety of edit lengths. Search time scales
roughly linearly with the number of phonemes, and render time scales
linearly with the number of frames. Even for long edits of 10 words, our
system can generate video in approximately a minute.
one NVIDIA GTX 1080Ti. Training the GAN for neural rendering
takes 17 hours on one NVIDIA Tesla V100.
In our synthesis pipeline, our fast phoneme search requires 5
seconds for a typical edit of 5 words containing 20 phonemes. Retar-
geting inference speed is 10K fps. Composite images are rendered
at 12 fps and final GAN rendering takes 7 fps on two NVIDIA GTX
1080Ti. All together, a typical 5 word edit takes around 30 seconds
for the full video generation (Table 1).
It should be possible to further reduce the feedback time by paral-
lelizing our synthesis pipeline. Phoneme search could be distributed
where each worker job is responsible for searching a fraction of the
repository. Both parts of the neural rendering step – forming the
composite images from target actor head parameters and applying
the GAN to generate photorealistic frames – are parallelizable by
distributing the frames. We have performed initial experiments on
parallelizing the GAN rendering which is the main bottleneck in
our pipeline. Distributing the GAN rendering across a cluster of
8 NVIDIA Tesla V100s achieved a rendering rate of 24fps, a 3.4x
speedup from the original 7fps for this step. Note that this speed
up rate includes image compression overhead. Overall this experi-
ment cuts the end-to-end synthesis time from 40 to 20 seconds for
a typical 8-word sentence. Similarly parallelizing the other parts
of the pipeline and using sufficient hardware we believe that the
end-to-end video generation feedback time could be reduced signif-
icantly. Streaming the frames as they are ready could also further
reduce the latency from issuing an edit to seeing the first frames of
the result, enabling real-time interactive editing sessions.
4 RESULTS
Figures 1 and 6 show examples of iterative text-based editing ses-
sions for a variety of target videos including recordings of graduate
students, YouTube video and a take from filming a dialog scene.
We encourage readers to watch the videos in our supplementary
materials to see how our text-based interface facilitate the iterative
editing workflow used in each of these sessions.
Session 1: Talking-head with glasses. Our first session works on a
2.5 minute target video (Figure 1). The editor explores ways for the
actor to parody Martin Scorsese and express that Marvel movies
are not to be considered real cinema. They first synthesize “Marvel
movies are not cinema” from a resting pose. Feeling it is too blunt,
they slightly change the wording to “Marvel movies are not really
cinema”, and eventually settle on the firmer statement “Marvel
movies really are not cinema”. They then insert a smile at the end
to soften the overall tone. Our tool is able to produce results with
synchronized mouth motions at each step.
, Vol. 1, No. 1, Article . Publication date: November 2020.
8 • Yao, et al.Target Video Iteration 1 Iteration 2 Iteration N
......
Jane may earn more money
by working hard
[rest] Just a jump to the left
and a step to the right
[rest] Just a jump to the left
[mouth_left] and a step to the right
[rest] Just a jump to the left
[mouth_left:1] and a step to the right
[mouth_right:1](wording edit) (mouth gesture)
(mouth gesture)
Lip motion
from repository
40 sec
......
He tried to blame it on terrorists I will make him suffer I will have him pay for what he didI will make him suffer [smile_closed:1]
(wording edit) (mouth gesture) (wording edit)
Lip motion
from repository
40 sec
......
There’s this very clear path May the force be with you
(speaking style: mumble) (speaking style: energetic)
May the force be with you May the force be with you
(wording edit)
Lip motion
from repository
40 sec
Mumble
repository Energetic
repository
Fig. 6. Editing sessions facilitated by our tool on target videos of 1 to 5 minutes in length. In these sessions our tool lets the editor change wording, insert
mouth gestures, refine mouth motions and manipulate speaking style. Here we highlight a few of these edits in each session. Our tool finds the source actor
mouth motions that conform to the specified edit operation and then retargets them to the target actor. Please see our supplemental materials and video for
full video results as well as screen recordings of the edit sessions in our interface.
Session 2: Talking-head with stubble. Our second session works
on a 3.5 minute target video (Figure 6 first row). The editor explores
ways for the actor to give the instruction to start the time warp by
jumping to the left then stepping to the right. They first synthe-
size the instruction phrase, then add gestures “[mouth_left]” and
“[mouth_right]” to the corresponding location in the dialog. Next,
feeling the gestures go too quickly, they lengthen the gestures to one
second each. Our tool produces realistic video with mouth motions
synchronized to the audio and the gesture directives.
Session 3: Movie scene. Our third session works on a target video
of a single take from a dialogue scene (Figure 6 second row). The
target video is a challenging one because it is only 1 minute long,
and the actress speaks for only 30 seconds in the take. Our tool
nevertheless is able to synthesize compelling results for this session.
In the session, the editor prototypes ways for the actress to express
her hatred towards the murderer of her sister’s dog. They first try
“I will make him suffer”. Then for added creepiness, add a tight-
lipped smile of 1 second duration at the end. Finally, they settle
on a less hostile line instead. While the neural renderer struggles
with the lack of data to produce images as sharp as those from the
previous two sessions, our tool is still able to produce realistic and
synchronized mouth motions that give the user a good sense of how
the scene would look with the alternative line and gesture.
Session 4: Interview. Our fourth session works on a 5 minute
excerpt of a YouTube video (Figure 6 third row). The editor explores
different delivery styles for the phrase “may the force be with you”.
They first synthesize the phrase with the default repository, then
switch to a mumble style and finally to an energetic style. While
the mouth movements match the audio, there is visibly less motion
with the mumble style and more motion with the energetic style.
5 EVALUATION
To evaluate our tool we analyze the quality of the synthesized video
as we vary the algorithmic methods (e.g. fast phoneme search, neu-
ral retargeting) used in our synthesis pipeline, and as we vary the
data (e.g. length of target video, repository or edit) provided to the
algorithm. We then compare the quality of our results to those of
previous work. Finally we report on user studies that quantitatively
evaluate the quality of our synthesized results. Unless otherwise
, Vol. 1, No. 1, Article . Publication date: November 2020.
Iterative Text-based Editing of Talking-heads Using Neural Retargeting • 9Fried et al. [2019]
Our Fast Phoneme
Search and Stitching
that’s all, people
P IY1 P AH0
Only Our Phoneme
Context Expansion
Fig. 7. Comparison of phoneme search and stitching method. While the
three methods usually yield visually indistinguishable frames, both the
baseline method in Fried et al. [2019] and that with phoneme context
expansion fail to close the mouth at the first “P” in people, whereas our fast
phoneme search and stitching algorithm fully closes it.
noted, the evaluations in this section use results from our auto-
matic synthesis pipeline with no additional user refinement. Readers
should refer to supplemental materials to evaluate the video results
presented in this section.
5.1 Varying the Algorithmic Methods
Comparison of phoneme search and stitching methods. We compare
the impact of using ablated versions of our fast phoneme search and
stitching algorithm (Section 3.3) with the phoneme search method of
Fried et al. [2019]. More specifically, because their synthesis pipeline
does not include neural retargeting we treat Fried et al.’s approach
as a baseline method and build two comparison pipelines. The first
one adds only phoneme context expansion (Section 3.3) to the base-
line stitching method. The second pipeline replaces their phoneme
search and stitching method with the full version of our fast method
(we call this version of the pipeline “Modified Fried et al. [2019]” in
later comparisons). All three pipelines assume access to an hour of
target video which serves as the repository. Figure 7 and videos in
the supplemental materials show that results generated using our
fast phoneme search and stitching method are often indistinguish-
able from the those generated by the baseline. The main differences
that do appear are often subtle as our forced mouth closure on \m, \b,
and \p phonemes reduces open mouth artifacts and our new context
aware stitching (Section 3.3) across subsequence transitions yields
smoother, less jittery lip motions. More importantly our method
takes only seconds to run, which is three orders of magnitude faster
than the hours required by baseline Fried et al. [2019], making it
possible for the user to iterate on edits.May the odds be ever in your favor
Source Actor
Composite ImageGround Truth
Composite Image Neural Render
LinearCopy Ours
Fig. 8. Comparison of retargeting methods. Copying expression parameters
yields an unnatural, rounded mouth shape on the target actor, while linear
regression fails to close his mouth for the “v” sound in the word “favor”. In
contrast, our retargeting method achieves a good match in lip shape to the
source actor and properly matches the shape required for the “v” sound.I think my mask of sanity is about to slip
30-second target 1-minute target 2.5-minute target 5-minute target
Fig. 9. With 2.5 minutes or more target video, our approach produces rela-
tively sharp, realistic frames. With only 30 seconds or 1 minute of target
video, the frames become noticeably blurry especially around the mouth
and teeth (Zoom in at the front teeth to see the difference).
Comparison of retargeting methods. Our neural retargeting method
(Section 3.4) transforms a sequence of source actor expression pa-
rameters to those of the target actor. We compare our method with
two simpler baseline retargeting methods. The copy baseline directly
copies the source actor parameters to the target actor. The linear
baseline replaces our retargeting network with a linear model. More
specifically, we manually chose 2.5 minutes of target video contain-
ing phrases that exactly matched phrases in the source repository
and established a frame-to-frame correspondence by re-timing the
target phonemes to match the lengths of the source phonemes. We
then applied linear regression on the source and target parameter
pairs to obtain the linear baseline model. Figure 8 and the supple-
mental materials show that our neural retargeting model produces
the best results, while direct copying produces uncanny mouth
shapes and the linear model often fail to close the mouth, causing
out-of-sync lip motions.
5.2 Varying the Amount of Data
Varying the length of the target video. We examine the effect of
different amounts of target video by applying our tool on 10 minute,
5 minute, 2.5 minute, 1 minute and 30 second subsets of a target
video. More target video generally results in sharper images, higher
, Vol. 1, No. 1, Article . Publication date: November 2020.
10 • Yao, et al.5 min repository1 hour repository
I have to return some video tapes
IY0 IY0 OW0 T
Fig. 10. Effect of small repository size. As shown here, the result from the
5-minute repository look less temporally stable as the transition from IY0 to
OW0 is more drastic than the result from the full-hour repository. With 5-
minute repository, the actor also has an incorrect closed-mouth at T because
we do not have as good a selection of phoneme coarticulations with a small
repository as we do with a full-hour one.Suwajanakorn et al. [2017]
(14 hours target video)
Suwajanakorn et al. [2017]
(3 min target video)
... saved many lives ...
M EH1 N IY0
Ours
(3 min target video)
Fig. 11. Comparison to Synthesizing Obama [Suwajanakorn et al . 2017].
With 3 minutes of Obama video, Suwajanakorn et al. [2017] cannot give
realistic mouth motions (top row). It produces similar frames throughout
the word “many”, and in particular fails to close the mouth at \m. Our
method produces well-synchronized mouth shapes and closures with only 3
minute of target video (middle row), while Suwajanakorn et al. [2017] needs
14 hours of target video to produce a good result (bottom row).
quality mouth interiors and smoother mouth motions. But the dif-
ference is subtle when target video exceeds 5 minutes, and at 2.5
minutes the results remain plausible. With a 30 second target video
however, although the results still have well-synchronized mouth
motions, our neural renderer struggles and produces noticeably
blurrier images, as shown in Figure 9 and videos in supplemental
materials. Please zoom in on the front teeth to see the difference.
Varying the amount of repository video. We examine the effect
of different amounts of repository video by applying our tool onFried et al. [2019]
(3.5 min target video)
Ours
(3.5 min target video)
Jane may earn more money by working hard
JH JH EY1 N
Fried et al. [2019]
(1 hour target video)
Fig. 12. Comparison to Fried et al. [2019]. Mouth motions from Fried et
al. [2019] are less temporally coherent as shown from phoneme ’JH’ to
’EY1’, both with 3.5 minutes (top row) and with 1 hour (middle row) of
target video. With 3.5 minutes, Fried et al. [2019] also produced an incorrect
mouth closure during the second ’JH’ frame. Our results (bottom row) have
smoother mouth motions, and we use only 3.5 minutes of target video.
a 3.5 minute target video with 60 minute, 30 minute, 10 minute
and 5 minute subsets of repository data. Generally a larger reposi-
tory leads to better results. Figure 10 and videos in supplemental
materials show that as the repository shrinks, mouth motions be-
come choppier as it becomes harder to find long matching phoneme
subsequences in the repository and our tool has to introduce more
transitions. However, the quality degrades gracefully.
Varying the length of the edit. We examine the effect of synthe-
sizing edits of different lengths by synthesizing increasingly longer
portions of the sentence “only the most accomplished artists ob-
tain popularity”; starting by only synthesizing the first word and
sequentially adding words until the full sentence is synthesized by
our pipeline. Videos in supplemental materials show that, while
the full sentence synthesis still has good mouth motions and looks
plausible, it generally produce results that contain more artifacts
than shorter edits, since with more phonemes to synthesize there
are more opportunities for artifacts to emerge. Our user studies
(Section 5.4) also found that results from short edits are rated more
real than full-sentence syntheses.
5.3 Comparisons to Other Methods
Comparisons with Synthesizing Obama [Suwajanakorn et al . 2017].
Suwajanakorn et al. [2017] have presented a technique for taking
audio speech of Obama as input and synthesizing a video of Obama
saying the speech. We compare our results to theirs using only 3
minutes of video of Obama. As shown in Figure 11, our approach
gives more plausible and synchronized mouth motions compared to
their method which can fail to close the mouth. While our approach
, Vol. 1, No. 1, Article . Publication date: November 2020.
Iterative Text-based Editing of Talking-heads Using Neural Retargeting • 11NVP [Thies et al. 2019]Ours
so for example military use and ...
Z AE1 M P
Fig. 13. Comparison to Neural Voice Puppetry (NVP) [Thies et al . 2020].
The actor fails to close his mouth with Neural Voice Puppetry at \m and \p,
whereas our approach with no user refinement fully closes it. We refer to
the supplemental materials for video comparison.Ours
There’s this very clear path...
Vougioukas et al. [2019] Chen et al. [2019]
IH1 P AE1 AE1
Fig. 14. Comparison to Chen et al. [2019] and Vougioukas et al. [2019]. All
three methods produce good lip-sync, but our result has better quality due
to sharper images and natural head motion.
produces good results with only 3 minutes of Obama video, their
method needs 14 hours.
Comparisons with Fried et al. [2019]. Our work builds on Fried
et al.’s [2019] synthesis pipeline. However, they require over an
hour of target video to produce high-quality results. We compare
our method against theirs using 3 minutes of target video, as well
as theirs using an hour of target video. Figure 12 and videos in
supplemental materials show that our result has more temporally
coherent mouth motions than Fried et al. [2019], both when they
use the same 3 minutes of target video, and when they use 1 hour
of target video. We further compare our results to those for Fried et
al. [2019] in the user studies in Section 5.4.
Comparisons with Neural Voice Puppetry [Thies et al. 2020]. Con-
current to our work, Neural Voice Puppetry (NVP) can synthesize
talking head videos from audio signal input, or from text by using a
synthetic voice to obtain the audio signal. Given an audio speech
track, we compare our result to theirs by applying our method on the
phoneme timings extracted from the audio. Figure 13 and videos in
supplemental materials show that while both approaches generate
mouth motions that synchronize with the audio, our fully automatic
result (without user refinement) generates closed-mouth frames at
the desired phonemes (\m, \b, \p), whereas Neural Voice Puppetry
leaves the mouth open for many of these phonemes. In addition, un-
like NVP, our tool allows the user to iteratively refine the automatic
results and adjust the performance. We further compare our results
to NVP in the user studies in Section 5.4.
Comparisons with Chen et al. [2019] and Vougioukas et al. [2019].
Both Chen et al. [2019] and Vougioukas et al. [2019] generate talk-
ing head videos from audio input and a single frame of the target
actor. Given an audio speech track, we compare our result to theirs
by applying our method on the phoneme timings extracted from
the audio. Figure 14 and videos in supplemental materials show
that while all three approaches produce good lip-sync with proper
mouth closures, both Chen et al. [2019] and Vougioukas et al. [2019]
produce videos of less resolution than our result. In addition, their
results do not have the natural head motion in our result, and by al-
ways centering the video around the cropped head, their results can
contain warping artifacts in the background, making them ill-suited
for incorporation into a video-editing workflow.
5.4 User Studies and Automatic Metrics
We use both user studies and automatic metrics to quantitatively
evaluate the quality of the video generated by our editing tool. In
the user studies, we investigate both short and long edits, while
ablating the target video length and the neural retargeting step. We
compare to previous work [Fried et al . 2019; Thies et al. 2020] and
to ground-truth video. We follow the study design used in previous
talking-head synthesis research [Fried et al. 2019; Kim et al. 2018].
Specifically, participants see one video at a time in randomized order
and are asked to rate the statement “This video clip looks real to
me” on a 5-point Likert scale ranging from strongly disagree (1) to
strongly agree (5). All videos used in our studies are available in the
supplemental material.
User study 1: Short Phrases (1 – 4 words). Short phrases are the
main type of result shown in Fried et al. [2019]. Such edits are useful
for minor fix-ups on existing sentences. In this study we compare
our automatic synthesis results (“Ours”) to 3 versions of Fried et
al. [2019]. (1) Their method with the same amount of target video as
used by our tool, which is less than 5 minutes in all cases. (2) Their
method with 1 hour of target video, which is their recommended
amount, and more than 12 times the amount we use. (3) A version of
their method with our fast phoneme search and stitching algorithm
(Section 3.3), but with 1 hour of target video (“Modified Fried et
al. [2019]” in Section 5.1), to evaluate the effect of ablating our neural
retargeting step. We also compare to ground truth video recordings.
We recruited 110 participants to view 25 videos each (5 conditions
for each of 5 edits). We report Likert scale responses in Table 2
(“Short Phrase”). The differences between all pairs, except “Ours”
vs. “Modified Fried”, are statistically significant. All p-values have
, Vol. 1, No. 1, Article . Publication date: November 2020.
12 • Yao, et al.
been adjusted for multiple testing and are reported in supplemental
materials.
Our tool outperforms Fried et al. [2019] both when using 5 min-
utes of data and 1 hour of data. We believe this is due to our results
having more accurate mouth closures and better temporal coherency
in mouth motions. Results are similar for our tool and Modified Fried,
indicating that our neural retargeting step does not have much neg-
ative effect on result quality. Together these results also suggest that
our fast phoneme search with stitching that forces closed mouths
for \m, \b, and \p phonemes leads to higher-quality synthesis than
the slow phoneme search and stitching approach used originally
by Fried et al. [2019]. Although a gap still remains between our
synthesized results and ground-truth videos, our results for short
edits are rated as real almost two thirds of the time.
User study 2: Full Sentences (6 – 9 words). Full sentence synthesis is
more challenging, since longer synthesis equates to a larger chance
of inaccurate matches and synthesis artifacts. However, synthesiz-
ing full sentences as opposed to short phrases opens up more use
cases (Section 4). Investigating full-sentence synthesis also empha-
sizes the quality differences between methods. The conditions in
this user study are the same as for user study 1. We recruited 153
participants to view 25 videos each (5 conditions for each of 5 sen-
tences). We report Likert scale responses in Table 2 (“Full Sentence”).
The differences between all pairs, except “Fried < 5 min” vs. “Fried

1 hr”, are statistically significant. All p-values have been adjusted
for multiple testing and are reported in supplemental materials.
Our tool produces the highest-quality results, followed by Modi-
fied Fried with over 1 hour of data, then by Fried et al. [2019]. Similar
to user study 1, here our results have better mouth closures and
smoother mouth motions than Fried et al. [2019]. It is worth noting
that our results are even better than Modified Fried. We believe this
is because our tool has a higher-quality source repository which be-
comes more salient when the edits are long. It shows the advantage
of our approach to decouple source repository from target video,
as data quality improvements to the repository can benefit many
different target videos. The one-time cost of building a high-quality
repository amortizes across all the edits that use it.
User study 3: our tool vs Neural Voice Puppetry [Thies et al. 2020].
The third user study compares our results to those of Neural Voice
Puppetry [Thies et al. 2020], where we show viewers videos gen-
erated by the two methods from the same audio speech track. We
recruited 90 participants to view 8 videos each (4 from each of the
two methods, Table 3). The audio tracks used in the videos are not
the actor’s real voice, and we believe this is the predominant reason
for overall lower scores (for both methods). The difference between
conditions is not statistically significant, and our results have sim-
ilar mean scores to those of NVP. Nevertheless, as mentioned in
Section 5.3, closely examining the videos generated by the two ap-
proaches, we find that our method often does a better job of closing
the mouth on \m, \b, and \p phonemes. We also note that while our
user studies evaluate our automatic results, unlike NVP, our tool
also provides refinement and performance controls that can be used
to improve results over the course of an interactive editing session.
Likert response (%)
Condition Length 5 4 3 2 1 Mean ‘Real’
Short Phrase Fried et al. [2019] < 5 min 19.5 28.0 11.3 22.1 19.0 3.1 47.6%
Fried et al. [2019] > 1 hr 24.1 31.5 13.7 20.0 10.7 3.4 55.6%
Modified Fried > 1 hr 27.5 39.9 13.3 13.9 5.4 3.7 67.4%
Ours < 5 min 30.37 34.2 14.1 15.2 5.7 3.7 64.9%
Ground truth n/a 40.1 37.8 11.5 9.8 0.9 4.1 77.8%
Full Sentence Fried et al. [2019] < 5 min 14.7 22.9 11.1 25.0 26.3 2.7 37.6%
Fried et al. [2019] > 1 hr 14.6 22.5 12.2 26.3 24.5 2.8 37.1%
Modified Fried > 1 hr 16.5 31.5 14.1 20.7 17.2 3.1 48.0%
Ours < 5 min 17.9 38.2 16.0 20.1 7.7 3.4 56.2%
Ground truth n/a 39.0 42.1 7.3 8.8 2.9 4.1 81.1%
Table 2. Results from user studies on short phrases and full sentences.
The “Length” column shows the length of the input target video for each
method. We compare to previous work [Fried et al . 2019] and to ground-
truth recordings. We report percentage of each answer on a 5-Point Likert
scale, the mean score, and percent of videos that received a score of 4 or
5 (‘real’). The difference between conditions is significant in both studies
(Kruskal-Wallis test, 𝑝 < 10−20 each). A followup Tukey’s range test shows
that all pairwise comparisons are statistically significant (𝑝 < 0.008 each)
except for “Ours” vs. “Modified Fried” for short edits and “Fried < 5 min”
vs. “Fried > 1 hr” for full sentences. Note that Tukey’s procedure adjusts
the p-values for multiple comparisons. We report all adjusted p-values in
the supplemental materials. Using our fast phoneme search and stitching
algorithm improves results from Fried et al. [2019]. Our tool outperforms
the method of Fried et al. [2019] when they use the same amount of target
video data, and when they use 12x the amount of data. For full-sentence
synthesis, our tool also outperforms Modified Fried even while they use 12x
the amount of data.
Likert response (%)
Condition Length 5 4 3 2 1 Mean ‘Real’
Ours < 5 min 6.0 9.6 10.6 26.2 47.5 2.0 15.6%
NVP < 5 min 6.0 10.6 11.3 24.8 47.4 2.0 16.6%
Table 3. Results from user study on our tool and Neural Voice Puppetry
(NVP) [Thies et al . 2020]. We compare to NVP and report percentage of each
answer on a 5-Point Likert scale, as well as mean score and percent of videos
that received a score of 4 or 5 (‘real’). Our tool and NVP received similar
mean scores and the difference is not statistically significant (Kruskal-Wallis
test, 𝑝 = 0.84).
Automatic Metrics. For videos in user study 1 and 2 where we
have ground-truth recordings, we further evaluate the results using
automatic metrics against grouth truth videos. To measure recon-
struction quality, we compute structural similarity index (SSIM
[Zhou Wang et al . 2004]) and peak signal-to-noise ratio (PSNR). To
measure lip motion quality, we compute the Landmarks Distances
(LMD) [Chen et al . 2018]. We report the results in Table 4. Although
our methods achieve favorable scores on many of these metrics, the
score differences are quite small and visual differences can be subtle.
We believe the user studies provide a better and more trustworthy
measure of video quality.
, Vol. 1, No. 1, Article . Publication date: November 2020.
Iterative Text-based Editing of Talking-heads Using Neural Retargeting • 13
Condition Length SSIM PSNR LMD
Short Phrase Fried et al. [2019] < 5 min 0.89929 25.08146 4.57492
Fried et al. [2019] > 1 hr 0.89925 25.09057 4.47108
Modified Fried > 1 hr 0.89934 25.09670 4.38139
Ours < 5 min 0.89921 25.09349 4.57060
Full Sentence Fried et al. [2019] < 5 min 0.96339 32.74630 3.75896
Fried et al. [2019] > 1 hr 0.96552 32.94852 3.67367
Modified Fried > 1 hr 0.97578 35.00073 2.90959
Ours < 5 min 0.97630 35.12082 3.18670
Table 4. Results of automatic metrics. We compare our results to 3 versions
of Fried et al. [2019] for both short phrase edits and full sentence syntheses.
Best score is bolded. Our tool tops SSIM and PSNR for full sentence syn-
theses and ranks second after Modified Fried for LMD on full sentence and
PSNR on short edits.
6 LIMITATIONS AND FUTURE WORK
We have demonstrated an iterative text-based tool for editing talking-
head dialogue and performance that can be applied to many real-
world editing scenarios in which only a few minutes of target actor
video is available. However, our approach does have several limita-
tions that could be addressed in future work.
Further reduce feedback loop time. Our tool currently requires
about 30 seconds to synthesize a typical 5 word edit. While this feed-
back loop time allows users to try a variety of edits and refinements,
seeing a synthesized result immediately (in real-time) would allow
even more iteration and exploration of design space. As noted in
Section 3.6, parallelization of our fast phoneme search and neural
rendering steps as well as streaming playback of the synthesized
video could reduce the feedback loop time significantly.
Improve quality of synthesis results. Although our method com-
pares favorably in quality with previous talking-head synthesis
techniques (Section 5.4), there is still a gap in realism between our
results and ground-truth videos. As our method relies on a rich
repository of source video to provide mouth motions for phoneme
coarticulations, it may be possible to improve synthesis by develop-
ing higher-quality repositories. One approach may be to leverage
existing work in text-driven 3D human mouth animation [Edwards
et al . 2016] to render unlimited amounts of mouth motions to serve
as the repository. Another direction is to build multiple reposito-
ries of many different source actors and then given a target video,
develop techniques to pick the best source actor for the target.
Performance controls over full face. Our current approach focuses
on synthesizing lip motions that match the target edit. While our
tool offers controls for inserting mouth gestures and changing the
speaking style, the effects of these controls are limited to the lower
part of the face. Others have demonstrated techniques for controlling
more of the head, including the ability to change head pose, gaze
direction and whole face expressions [Kim et al . 2018] However,
these techniques often introduce artifacts in the hair and with the
clothes at the neckline. Adding such full face controls in an artifact-
free manner remains an open research direction.
Previsualizing dialogue using existing film scenes. When writing
dialogue, scriptwriters have to imagine the sound and appearance
of the scene. Using our video editing tool with a catalog of video
from existing film scenes might allow such scriptwriters to quickly
visualize the dialogue in different settings and with different actors.
Users might search for scenes based on their settings and actors
using a tool like SceneSkim [Pavel et al. 2015] and our tool could be
used to insert the new dialogue.
7 ETHICAL CONSIDERATIONS
Our editing tool is designed to enable an iterative workflow for
removing filler words, adjusting phrasing, or correcting mistakes in
a talking-head video. While such tools can facilitate content creation
and storytelling, tools like ours, that let users manipulate what a
target actor is saying, can also be misused. We follow the guidelines
suggested by Fried et al. [2019] for ethically using such tools. (1)
Video generated by our tool should be transparent about the fact
that it has been manipulated. (2) Actors must give consent to any
manipulation before a resulting video is shared widely.
We also recognize that these guidelines alone will not stop bad
actors from using tools like ours to create false statements and slan-
der others. Therefore, it is also critical for researchers to continue
developing tools for detecting, fingerprinting, and verifying such
video manipulation. Openly publishing the technical details of our
tool can increase public awareness and help detection efforts. Ul-
timately these issues may also require regulations and laws that
penalize misuse while allowing creative and consensual use cases.
8 CONCLUSION
Iterative editing is central to many content-creation tasks and is
especially common in video editing. We have shown how to enable
such iterative editing in the context of editing talking-head video
using a text-based interface that allows changes to wording and
facial performance while providing refinement controls. Whether
an editor trying different ways to phrase the dialogue in a film,
developing dialogue for a conversational agent, or correcting a
mistake in a lecture, such iteration is often essential for finding the
most appropriate result. We believe such tools that facilitate video
editing can democratize content-creation and enable many more
people to tell their stories.
REFERENCES
2020a. Google Cloud Speech to Text API. https://cloud.google.com/speech-to-text
2020b. Google Cloud Text to Speech API. https://cloud.google.com/text-to-speech
2020. Lyrebird AI. https://www.descript.com/lyrebird-ai
2020. Rev. https://rev.com
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig
Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghe-
mawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing
Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dande-
lion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster,
Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent
Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin
Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-
Scale Machine Learning on Heterogeneous Systems. https://www.tensorflow.org/
Software available from tensorflow.org.
Shruti Agarwal, Hany Farid, Ohad Fried, and Maneesh Agrawala. 2020. Detecting Deep-
Fake Videos from Phoneme-Viseme Mismatches. In Workshop on Media Forensics at
CVPR. Seattle, WA, USA.
, Vol. 1, No. 1, Article . Publication date: November 2020.
14 • Yao, et al.
Hadar Averbuch-Elor, Daniel Cohen-Or, Johannes Kopf, and Michael F. Cohen. 2017.
Bringing Portraits to Life. 36, 6 (Nov. 2017), 196:1–13. https://doi.org/10.1145/
3130800.3130818
Floraine Berthouzoz, Wilmot Li, and Maneesh Agrawala. 2012. Tools for Placing Cuts
and Transitions in Interview Video. ACM Trans. Graph. 31, 4, Article 67 (July 2012),
8 pages. https://doi.org/10.1145/2185520.2185563
Volker Blanz and Thomas Vetter. 1999. A Morphable Model for the Synthesis of
3D Faces. In Proceedings of the 26th Annual Conference on Computer Graphics and
Interactive Techniques (SIGGRAPH ’99). ACM Press/Addison-Wesley Publishing Co.,
USA, 187–194. https://doi.org/10.1145/311535.311556
Christoph Bregler, Michele Covell, and Malcolm Slaney. 1997. Video Rewrite: Driving
Visual Speech with Audio. In Proceedings of the 24th Annual Conference on Computer
Graphics and Interactive Techniques (SIGGRAPH ’97). ACM Press/Addison-Wesley
Publishing Co., USA, 353–360. https://doi.org/10.1145/258734.258880
Yao-Jen Chang and Tony Ezzat. 2005. Transferable Videorealistic Speech Animation.
143–151. https://doi.org/10.1145/1073368.1073388
Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. 2018. Lip
movements generation at a glance. In Proceedings of the European Conference on
Computer Vision (ECCV). 520–535.
Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. 2019. Hierarchical cross-
modal talking face generation with dynamic pixel-wise loss. In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition. 7832–7841.
Joon Son Chung, Amir Jamaludin, and Andrew Zisserman. 2017. You said that?. In
British Machine Vision Conference.
Pif Edwards, Chris Landreth, Eugene Fiume, and Karan Singh. 2016. JALI: an animator-
centric viseme model for expressive lip synchronization. ACM Transactions on
Graphics (TOG) 35, 4 (2016), 1–11.
Tony Ezzat, Gadi Geiger, and Tomaso Poggio. 2002. Trainable Videorealistic Speech
Animation. 21, 3 (July 2002), 388–398. https://doi.org/10.1145/566654.566594
Ohad Fried and Maneesh Agrawala. 2019. Puppet Dubbing. In Eurographics Symposium
on Rendering - DL-only and Industry Track, Tamy Boubekeur and Pradeep Sen (Eds.).
The Eurographics Association. https://doi.org/10.2312/sr.20191220
Ohad Fried, Ayush Tewari, Michael Zollhöfer, Adam Finkelstein, Eli Shechtman, Dan B
Goldman, Kyle Genova, Zeyu Jin, Christian Theobalt, and Maneesh Agrawala. 2019.
Text-Based Editing of Talking-Head Video. ACM Trans. Graph. 38, 4, Article 68 (July
2019), 14 pages. https://doi.org/10.1145/3306346.3323028
John S Garofolo, Lori F Lamel, William M Fisher, Jonathan G Fiscus, and David S Pallett.
1993. DARPA TIMIT acoustic-phonetic continous speech corpus CD-ROM. NIST
speech disc 1-1.1. NASA STI/Recon technical report n 93 (1993).
Pablo Garrido, Levi Valgaerts, Ole Rehmsen, Thorsten Thormaehlen, Patrick Pérez,
and Christian Theobalt. 2014. Automatic Face Reenactment. 4217–4224. https:
//doi.org/10.1109/CVPR.2014.537
Pablo Garrido, Levi Valgaerts, Hamid Sarmadi, Ingmar Steiner, Kiran Varanasi, Patrick
Pérez, and Christian Theobalt. 2015. VDub: Modifying Face Video of Actors for
Plausible Visual Alignment to a Dubbed Audio Track. 34, 2 (May 2015), 193–204.
https://doi.org/10.1111/cgf.12552
Pablo Garrido, Michael Zollhöfer, Dan Casas, Levi Valgaerts, Kiran Varanasi, Patrick
Pérez, and Christian Theobalt. 2016. Reconstruction of Personalized 3D Face Rigs
from Monocular Video. ACM Trans. Graph. 35, 3, Article 28 (May 2016), 15 pages.
https://doi.org/10.1145/2890493
Jiahao Geng, Tianjia Shao, Youyi Zheng, Yanlin Weng, and Kun Zhou. 2018. Warp-
guided GANs for Single-photo Facial Animation. In SIGGRAPH Asia 2018 Technical
Papers (Tokyo, Japan) (SIGGRAPH Asia ’18). ACM, New York, NY, USA, Article 231,
231:1–231:12 pages. http://doi.acm.org/10.1145/3272127.3275043
Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Patrick Nguyen,
Ruoming Pang, Ignacio Lopez Moreno, Yonghui Wu, et al . 2018. Transfer learning
from speaker verification to multispeaker text-to-speech synthesis. In Advances in
neural information processing systems. 4480–4490.
Ira Kemelmacher-Shlizerman, Aditya Sankar, Eli Shechtman, and Steven M. Seitz. 2010.
Being John Malkovich. 341–353. https://doi.org/10.1007/978-3-642-15549-9_25
Hyeongwoo Kim, Mohamed Elgharib, Michael Zollhöfer, Hans-Peter Seidel, Thabo
Beeler, Christian Richardt, and Christian Theobalt. 2019. Neural Style-Preserving
Visual Dubbing. ACM Trans. Graph. 38, 6, Article 178 (Nov. 2019), 13 pages. https:
//doi.org/10.1145/3355089.3356500
H. Kim, P. Garrido, A. Tewari, W. Xu, J. Thies, N. Nießner, P. Pérez, C. Richardt, M.
Zollhöfer, and C. Theobalt. 2018. Deep Video Portraits. ACM Transactions on
Graphics 2018 (TOG) (2018).
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization.
In 3rd International Conference on Learning Representations, ICLR 2015, San Diego,
CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann
LeCun (Eds.). http://arxiv.org/abs/1412.6980
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh,
Jose Sotelo, Alexandre de Brebisson, Yoshua Bengio, and Aaron Courville. 2019.
MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis.
arXiv:1910.06711 [eess.AS]
Rithesh Kumar, Jose Sotelo, Kundan Kumar, Alexandre de Brebisson, and Yoshua Bengio.
2017. ObamaNet: Photo-realistic lip-sync from text. arXiv:1801.01442 [cs.CV]
VI Levenshtein. 1966. Binary Codes Capable of Correcting Deletions, Insertions and
Reversals. In Soviet Physics Doklady, Vol. 10. 707.
Kang Liu and Joern Ostermann. 2011. Realistic facial expression synthesis for an
image-based talking head. https://doi.org/10.1109/ICME.2011.6011835
Wesley Mattheyses, Lukas Latacz, and Werner Verhelst. 2010. Optimized photorealistic
audiovisual speech synthesis using active appearance modeling. In Auditory-Visual
Speech Processing. 8–1.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training
recurrent neural networks. In International conference on machine learning. 1310–
1318.
Amy Pavel, Dan B Goldman, Björn Hartmann, and Maneesh Agrawala. 2015. Sce-
neskim: Searching and browsing movies using synchronized captions, scripts and
plot summaries. In Proceedings of the 28th Annual ACM Symposium on User Interface
Software & Technology. ACM, 181–190.
A. Pumarola, A. Agudo, A.M. Martinez, A. Sanfeliu, and F. Moreno-Noguer. 2019.
GANimation: One-Shot Anatomically Consistent Facial Animation. (2019).
Steve Rubin, Floraine Berthouzoz, Gautham J Mysore, Wilmot Li, and Maneesh
Agrawala. 2013. Content-based tools for editing audio stories. In Proceedings of
the 26th annual ACM symposium on User interface software and technology. ACM,
113–122.
Yang Song, Jingwen Zhu, Dawei Li, Andy Wang, and Hairong Qi. 2019. Talking
Face Generation by Conditional Recurrent Adversarial Network. In Proceedings of
the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19.
International Joint Conferences on Artificial Intelligence Organization, 919–925.
https://doi.org/10.24963/ijcai.2019/129
Supasorn Suwajanakorn, Steven M. Seitz, and Ira Kemelmacher-Shlizerman. 2017.
Synthesizing Obama: Learning Lip Sync from Audio. ACM Trans. Graph. 36, 4,
Article 95 (July 2017), 13 pages. https://doi.org/10.1145/3072959.3073640
Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan
Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner,
Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt,
Maneesh Agrawala, Eli Shechtman, Dan B. Goldman, and Michael Zollhöfer. 2020.
State of the Art on Neural Rendering. Computer Graphics Forum (2020). https:
//doi.org/10.1111/cgf.14022
Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias
Nießner. 2020. Neural Voice Puppetry: Audio-driven Facial Reenactment. In European
Conference on Computer Vision. Springer, 716–731.
Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, and Matthias
Nießner. 2016. Face2Face: Real-Time Face Capture and Reenactment of RGB Videos.
2387–2395. https://doi.org/10.1109/CVPR.2016.262
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals,
Alexander Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016.
WaveNet: A Generative Model for Raw Audio. In Arxiv. https://arxiv.org/abs/1609.
03499
Daniel Vlasic, Matthew Brand, Hanspeter Pfister, and Jovan Popović. 2005. Face Transfer
with Multilinear Models. 24, 3 (July 2005), 426–433. https://doi.org/10.1145/1073204.
1073209
Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. 2018. End-to-End Speech-
Driven Facial Animation with Temporal GANs. arXiv:1805.09313 [eess.AS]
Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. 2019. Realistic Speech-
Driven Facial Animation with GANs. International Journal of Computer Vision (13
Oct 2019). https://doi.org/10.1007/s11263-019-01251-8
Robert A Wagner and Michael J Fischer. 1974. The string-to-string correction problem.
Journal of the ACM (JACM) 21, 1 (1974), 168–173.
Lijuan Wang, Wei Han, Frank Soong, and Qiang Huo. 2011. Text-driven 3D Photo-
Realistic Talking Head. In INTERSPEECH 2011 (interspeech 2011 ed.). International
Speech Communication Association. https://www.microsoft.com/en-us/research/
publication/text-driven-3d-photo-realistic-talking-head/
O. Wiles, A.S. Koepke, and A. Zisserman. 2018. X2Face: A network for controlling
face generation by using images, audio, and pose codes. In European Conference on
Computer Vision.
Jiahong Yuan and Mark Liberman. 2008. Speaker identification on the SCOTUS corpus.
In In Proceedings of Acoustics 2008. Citeseer.
Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. 2019.
Few-Shot Adversarial Learning of Realistic Neural Talking Head Models.
arXiv:1905.08233 [cs.CV]
Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. 2019. Talking Face
Generation by Adversarially Disentangled Audio-Visual Representation. In AAAI
Conference on Artificial Intelligence (AAAI).
Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. Image quality
assessment: from error visibility to structural similarity. IEEE Transactions on Image
Processing 13, 4 (2004), 600–612.
, Vol. 1, No. 1, Article . Publication date: November 2020.

Certainly! Let's apply the V-Method of Learning to help you achieve your goal of creating or understanding the project described in the paper titled "Iterative Text-based Editing of Talking-heads Using Neural Retargeting." This is a complex and cutting-edge topic in the field of computer vision and deep learning, and using the V-Method will allow you to focus your efforts efficiently.

Overview of Your Goal
Objective: To understand and possibly develop a system for iterative text-based editing of talking-head videos using neural retargeting, as described in the research paper.

Step-by-Step Application of the V-Method

Start with a Specific, Well-Documented Example
Action: Locate an open-source implementation or code repository related to the paper.
Search for Existing Code:
Check if the authors have released the code:
Visit the paper's website or academic page.
Check repositories like GitHub for any related projects.
If the exact code isn't available, look for similar projects:
First Order Motion Model for Image Animation: GitHub Repository
Neural Voice Puppetry: GitHub Repository
Face2Face: Real-Time Face Capture and Reenactment: GitHub Repository
Goal: Obtain a working codebase that performs similar tasks—editing talking-head videos using neural networks.

Attempt to Understand How This Example Works
Action:
Running the Code:
Install necessary dependencies and try to run the provided examples.
Observe the input and output to get a sense of the system's capabilities.
Reading the Documentation:
Go through the README files and any available documentation.
Understand the system's high-level architecture.
Goal: Get hands-on experience with the system's functionality and its overall workflow.

Identify What You Don't Understand
Action:
As you explore the code and documentation, make a list of concepts, technologies, and terminologies that are unfamiliar.
Examples:

Neural Retargeting: What is it, and how does it work?
Phoneme Alignment: How are phonemes aligned with video frames?
3D Head Model Registration: What methods are used for this?
GANs (Generative Adversarial Networks): How are they applied in this context?
Facial Expression Parameters: What are they, and how are they manipulated?
Monocular Head Tracking: How does it function?
Goal: Create a roadmap of topics you need to delve deeper into.

Research the Underlying Concepts
For each unfamiliar concept, dive into learning resources to build your understanding.
Action:

Neural Retargeting:
Definition: Transferring motions or expressions from one subject to another using neural networks.
Resources:
Research papers on neural retargeting.
Tutorials on neural style transfer and its applications.
Phoneme Alignment:
Definition: Synchronizing phonemes (speech sounds) with video frames.
Resources:
Basics of phonetics and phonology.
Tools like P2FA (Penn Phonetics Lab Forced Aligner).
Articles on audio-visual synchronization.
3D Head Model Registration:
Definition: Fitting a 3D model to 2D images of faces to capture head geometry and expressions.
Resources:
Tutorials on 3D morphable models.
Open-source tools for face alignment and tracking (e.g., OpenFace).
Generative Adversarial Networks (GANs):
Definition: A class of machine learning frameworks where two neural networks contest with each other.
Resources:
Introductory articles on GANs.
Courses on deep learning focusing on GANs.
Implementation tutorials (e.g., GANs in TensorFlow or PyTorch).
Facial Expression Parameters:
Definition: Numerical representations of facial expressions used in animations and models.
Resources:
Studies on facial action coding systems.
Papers on expression transfer between models.
Monocular Head Tracking:
Definition: Estimating head movements and facial expressions from a single video stream.
Resources:
Computer vision courses covering object tracking.
Publications on face tracking techniques.
Goal: Build foundational knowledge in each area directly relevant to the project.

Dive Deeper as Needed
If initial resources aren't sufficient, go deeper into the foundational concepts.
Action:

Mathematics and Algorithms:
Linear algebra (matrix operations, transformations).
Optimization techniques used in model fitting.
Machine Learning Fundamentals:
Neural network architectures.
Training and validation processes.
Loss functions and backpropagation.
Computer Vision Basics:
Image processing techniques.
Feature extraction and matching.
Goal: Ensure you have the necessary background to understand the more advanced topics.

Identify Patterns in Key Foundational Areas
As you learn, you'll notice recurring themes and essential concepts.
Action:

Common Techniques:
Use of convolutional neural networks (CNNs) in image analysis.
Application of recurrent neural networks (RNNs) for sequential data.
Importance of data preprocessing and augmentation.
Frequently Mentioned Tools and Libraries:
TensorFlow and PyTorch for building neural networks.
OpenCV for computer vision tasks.
DLib and other face detection libraries.
Goal: Prioritize learning these areas in more depth, as they are critical to the project.

Study These Relevant Foundational Concepts
Dedicate time to deeply understand these key areas.
Action:

Enroll in Online Courses:
Deep Learning Specialization by Andrew Ng (Coursera)
CS231n: Convolutional Neural Networks for Visual Recognition (Stanford Online)
Introduction to Computer Vision (Udacity)
Read Books and Tutorials:
"Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow" by Aurélien Géron.
"Deep Learning" by Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
Work on Smaller Projects:
Implement a simple GAN on a dataset like MNIST to understand how they work.
Create a face detection script using OpenCV.
Goal: Solidify your understanding through structured learning and practical application.

Integrate Knowledge and Reconstruct the Example
Return to the initial example with your new knowledge.
Action:

Re-read the Paper:
Go through the paper again, now focusing on sections that were previously challenging.
Take notes, and summarize each section in your own words.
Examine the Code in Detail:
Analyze how the main components are implemented.
Debug the code by stepping through key functions.
Map the code to the concepts you've learned.
Modify the Code:
Make small changes to see how they affect the output.
Experiment with different parameters or inputs.
Goal: Achieve a comprehensive understanding of how the system works end-to-end.

Reproduce the Example and Build Upon It
Apply what you've learned to create your version or extend the project.
Action:

Set Up a Development Environment:
Install necessary tools: Python, necessary libraries (TensorFlow, PyTorch), CUDA for GPU acceleration.
Data Acquisition:
Gather datasets required for training, such as facial video data.
Understand ethical considerations and privacy concerns when using datasets.
Implement Key Components:
Phoneme Alignment:
Use tools like Gentle Forced Aligner or Montreal Forced Aligner for phoneme segmentation.
3D Head Model Registration:
Implement or use existing models like 3D Morphable Models (3DMM).
Neural Retargeting:
Build and train neural networks for retargeting expressions.
Neural Rendering:
Implement GANs for rendering photorealistic images.
Consider models like Pix2Pix or CycleGAN as starting points.
Testing and Evaluation:
Run the system on sample inputs.
Evaluate the quality of the output.
Compare your results with those presented in the paper.
Iterate and Improve:
Fine-tune model parameters.
Try different network architectures.
Address any gaps or issues that arise.
Goal: Develop a working prototype and deepen your practical expertise.

Putting It All Together: A Suggested Plan
Week 1: Project Familiarization and Planning
Day 1-2: Locate and set up an example codebase.
Day 3-4: Run the code and note areas of confusion.
Day 5-7: Begin researching unfamiliar concepts.
Week 2: Learning Core Concepts
Day 8-14: Deep dive into core topics (Neural Networks, GANs, Computer Vision).
Week 3: Expanding Knowledge and Starting Implementation
Day 15-17: Study specific topics (Phoneme Alignment, 3D Head Models).
Day 18-21: Begin modifying the code; implement small features.
Week 4: Reconstructing and Building
Day 22-25: Reproduce key components of the system.
Day 26-28: Test the system, evaluate results, and iterate.
Additional Tips
Stay Organized:
Keep a learning journal documenting what you learn each day.
Use tools like Anki or Quizlet for memorizing key concepts.
Join Communities:
Participate in forums like Reddit's r/MachineLearning or Stack Overflow.
Engage with the academic community by attending webinars or virtual conferences.
Seek Collaborations:
Connect with others interested in similar projects.
Consider finding a mentor or joining a study group.
Be Patient and Persistent:
Accept that some concepts may take time to fully understand.
Celebrate small victories along the way.
Conclusion
By applying the V-Method of Learning to your goal, you're focusing your efforts on what truly matters for achieving it. Starting with a specific example allows you to contextualize your learning, making it more efficient and engaging. As you dive deeper into the necessary concepts, you'll build a strong, relevant knowledge base that directly supports your objective.

Remember, the key steps are:

Start with a specific example.
Identify and research unfamiliar concepts.
Dive deeper where necessary.
Recognize patterns and key foundational areas.
Study these areas thoroughly.
Integrate your knowledge back into the example.
Reproduce and build upon the example.
Good luck on your learning journey! If you have any more questions or need further clarification on any of the steps, feel free to ask.

can you execute the plan and give me detailed step by step on what to do . also can we use github projects components and libraries and pieces things up together .if so how


Certainly! Let's execute the plan step by step, providing detailed instructions on what you need to do to understand and potentially recreate the project described in the paper "Iterative Text-based Editing of Talking-heads Using Neural Retargeting." We'll also discuss how to leverage existing GitHub projects, components, and libraries, and how to piece them together.

Overview of the Goal
Objective:
To understand and possibly develop a system for iterative text-based editing of talking-head videos using neural retargeting, as described in the research paper.

Approach:
We'll apply the V-Method of Learning to achieve this goal, focusing on practical steps, leveraging existing resources, and integrating components from GitHub projects.

Step-by-Step Plan
Step 1: Start with a Specific, Well-Documented Example
Action: Locate an open-source implementation or related projects.

Search for Existing Code:
Check if the authors have released code:
Visit the paper's page or the authors' profiles.
Look for supplementary materials or code repositories.
Search GitHub and other repositories:
Use keywords like "talking-head synthesis," "neural retargeting," "text-based video editing."
Explore projects with similar goals.
Identify Relevant Projects: Since the exact code from the paper isn't available, we'll focus on projects that are similar or address parts of the system. Recommended GitHub Projects:
Neural Voice Puppetry
Repo: JustusThies/NeuralVoicePuppetry
Description: Audio-driven facial reenactment.
First Order Motion Model for Image Animation
Repo: AliaksandrSiarohin/first-order-model
Description: Image animation by a driving video.
Face2Face: Real-Time Face Capture and Reenactment
Repo: TadasBaltrusaitis/OpenFace (face alignment and tracking)
Text-based Editing of Talking-head Video
Website: https://www.ohadf.com/projects/text-based-editing
Note: While code isn't publicly available, the project page provides insights and may have updates.
Choose a Project to Start With: We'll begin with Neural Voice Puppetry as it closely aligns with the paper's objectives.
Step 1 Actions:

Clone the Repository:

git clone https://github.com/JustusThies/NeuralVoicePuppetry.git
Review Documentation:
Read the README.md file thoroughly.
Note installation instructions, dependencies, and requirements.
Set Up the Environment:
Ensure you have a compatible GPU (NVIDIA with CUDA support).
Install necessary libraries and frameworks (e.g., PyTorch, TensorFlow).
Step 2: Attempt to Understand How This Example Works
Action:

Run the Provided Examples:
Follow the instructions to run a demo.
Observe the inputs (audio, source video) and outputs (reenacted video).
Analyze the Codebase:
Main Components:
Data Processing: Input handling, preprocessing steps.
Model Architecture: Neural networks used, layers, and functions.
Inference Pipeline: How the model generates outputs.
Key Files to Examine:
models/: Contains model definitions.
inference.py: Script used for running inference.
utils/: Helper functions and utilities.
Documentation and Research Papers:
Read the paper associated with Neural Voice Puppetry:
Thies, J., et al. "Neural Voice Puppetry: Audio-driven Facial Reenactment." ECCV 2020.
Take notes on methodologies and techniques used.
Step 3: Identify What You Don't Understand
Action:

Create a list of concepts and components that require further understanding:

Neural Retargeting: How are expressions mapped between source and target?
Phoneme Alignment and Forced Alignment Tools: Techniques for aligning audio and text.
3D Head Model Registration: Methods for fitting a 3D model to a face.
Generative Adversarial Networks (GANs): Their role in image synthesis.
Facial Expression Parameters: How expressions are represented numerically.
Monocular Head Tracking: Capturing head movements from a single camera.
Step 4: Research the Underlying Concepts
Action:

For each item on your list, dive into resources to build foundational knowledge.

1. Neural Retargeting
Definition:
Transferring motion or expressions from a source to a target using neural networks.
Resources:
Papers:
Wiles, O., Koepke, A. S., & Zisserman, A. "X2Face: A network for controlling face generation using images and pose codes." ECCV 2018.
Tutorials:
Deep learning lectures on style transfer and domain adaptation.
2. Phoneme Alignment
Definition:
Aligning phonetic units (phonemes) with spoken audio.
Tools:
Montreal Forced Aligner: Link
Gentle Forced Aligner: Link
Learning Material:
Basics of phonetics and phonology.
Understanding speech recognition systems.
3. 3D Head Model Registration
Definition:
Fitting a 3D model to 2D facial images to capture geometry and expressions.
Resources:
Face Alignment and Tracking Libraries:
OpenFace: Link
Dlib: Link
Papers:
Blanz, V., & Vetter, T. "A morphable model for the synthesis of 3D faces." SIGGRAPH 1999.
Tutorials:
Implementing face alignment and 3D reconstruction from images.
4. Generative Adversarial Networks (GANs)
Definition:
Deep learning models where two networks (generator and discriminator) compete to produce realistic data.
Resources:
Books:
"Generative Deep Learning" by David Foster.
Online Courses:
DeepLearning.AI's GAN specialization on Coursera.
5. Facial Expression Parameters
Definition:
Numerical representations (coefficients) that define facial expressions in models.
Resources:
Facial Action Coding System (FACS).
Papers on expression transfer and facial animation.
6. Monocular Head Tracking
Definition:
Estimating head movement and facial expressions using a single camera.
Resources:
Computer vision courses focusing on object tracking.
Implementation of facial landmark detection.
Step 5: Dive Deeper as Needed
Action:

For complex topics, immerse yourself further.

Mathematics:
Linear algebra (vectors, matrices).
Calculus (optimization techniques).
Machine Learning Fundamentals:
Neural networks (architecture, backpropagation).
Overfitting and regularization.
Computer Vision Basics:
Image processing.
Feature detection and matching.
Recommended Courses:
CS231n: Convolutional Neural Networks for Visual Recognition (Stanford): Link
Deep Learning Specialization by Andrew Ng on Coursera: Link
Step 6: Identify Patterns in Key Foundational Areas
Action:

Recognize commonly used techniques and tools:

Convolutional Neural Networks (CNNs): For image analysis.
Recurrent Neural Networks (RNNs) / Temporal Models: For sequence data.
Autoencoders: For encoding and decoding representations.
Common Libraries:
PyTorch and TensorFlow for deep learning.
OpenCV for computer vision tasks.
Step 7: Study These Relevant Foundational Concepts
Action:

Focus on mastering:

GANs:
Understand the training dynamics.
Implement simple GAN models.
Facial Recognition and Tracking:
Implement face detection using OpenCV.
Explore facial landmark detection with Dlib.
Audio Processing:
Use LibROSA for audio feature extraction.
Study spectrograms and MFCCs.
Implement Mini-Projects:
Build a simple face swapper.
Create a voice-to-lip movement synchronizer.
Step 8: Integrate Knowledge and Reconstruct the Example
Action:

Re-express the Codebase:
Comment and Document:
Go through the Neural Voice Puppetry code and add comments.
Map Functions to Concepts:
Understand how each part of the code relates to the concepts you've learned.
Data Flow Visualization:
Create flowcharts illustrating:
Input processing.
Model inference steps.
Output generation.
Explore Alternative Implementations:
Look into other repositories for different approaches.
Compare methodologies and results.
Step 9: Reproduce the Example and Build Upon It
Action:

1. Set Up Your Development Environment
Hardware Requirements:
A machine with an NVIDIA GPU supporting CUDA.
Software Setup:
Operating System: Ubuntu 18.04 or higher is recommended.
Python Environment:
Install Python 3.7 or compatible version.
Use conda or virtualenv for environment management.
CUDA and cuDNN:
Install CUDA Toolkit: Link
Install cuDNN: Link
2. Install Necessary Libraries
Deep Learning Frameworks:
PyTorch: Installation Guide
TensorFlow (if required): Installation Guide
Additional Libraries:
OpenCV: For image processing.

pip install opencv-python
Dlib: For face detection.

pip install dlib
Audio Processing:
LibROSA: For audio analysis.

pip install librosa
3. Data Acquisition
Datasets:
VoxCeleb2: Link
Large-scale dataset with audio-visual data.
LRW (Lip Reading in the Wild): Link
Focused on mouth region videos aligned with speech.
Collecting Custom Data:
Record videos of yourself or others speaking.
Ensure proper lighting and frontal face positioning.
Ethical Considerations:
Obtain consent from participants.
Respect data privacy and intellectual property rights.
4. Implement Key Components
A. Phoneme Alignment

Install Forced Alignment Tool:
Montreal Forced Aligner (MFA): Link
Process Audio Data:
Transcribe audio files manually or using ASR tools.
Use MFA to align phonemes with the audio.
Example Command:

mfa_align /path/to/data /path/to/dictionary /path/to/acoustic_model /path/to/output
B. 3D Head Model Registration

Face Alignment:
Use OpenFace or Dlib to detect facial landmarks.
3D Model Fitting:
Implement a 3D Morphable Model (3DMM).
Tutorial: 3DMM_fitting
Understand Face Tracking Algorithms:
Read about techniques like Active Appearance Models (AAMs).
C. Neural Retargeting

Model Architecture:
Design a neural network that maps source expression parameters to target parameters.
Considerations:
Use RNNs or Temporal Convolutional Networks to capture temporal dependencies.
Input: Source facial expression parameters over time.
Output: Target facial expression parameters.
Training Data:
Need paired data of source and target expressions.
Self-Supervised Learning:
As in the paper, find corresponding sequences between source and target to train the model.
Implementing the Model:
Use PyTorch for building and training the network.

import torch
import torch.nn as nn

class NeuralRetargetingModel(nn.Module):
    def __init__(self):
        super(NeuralRetargetingModel, self).__init__()
        # Define layers

    def forward(self, x):
        # Define forward pass
        return x
D. Neural Rendering

GANs for Image Synthesis:
Implement a Pix2Pix GAN model for translating composites to photorealistic images.
Resources:
Pix2Pix Paper: Link
PyTorch Implementation: Link
Training the GAN:
Prepare training data with input-output image pairs.
Preprocessing:
Generate composite images from the target video.
Pair them with the original frames for supervised learning.
Implementing the Training Loop:
Define generator and discriminator architectures.
Set up loss functions (adversarial loss, L1 loss).
Use batching and data loaders for efficient training.
5. Integration of Components
Data Flow:
Audio Input → Phoneme Alignment → Expression Parameters
Video Input → 3D Model Fitting → Expression Parameters
Processing Pipeline:
Map phonemes to facial expressions using the trained neural retargeting model.
Use the GAN to render the final frames.
Synchronizing Modules:
Ensure that timings between audio, phonemes, and video frames are consistent.
6. Testing and Evaluation
Run Inference:
Use test audio and video inputs to generate outputs.
Evaluate Results:
Visual inspection for realism.
Quantitative metrics:
Lip-Sync Error: Compare generated lip movements with ground truth.
Structural Similarity Index (SSIM): Measure image quality.
Iterative Refinement:
Identify issues (e.g., artifacts, misalignments) and adjust models.
Fine-tune hyperparameters.
Using GitHub Projects, Components, and Libraries
Yes, you can use GitHub projects, components, and libraries. Here's how:

Leverage Existing Code:
Use ready-made implementations for complex tasks (e.g., face tracking, GANs).
Examples:
Face Alignment: Use OpenFace.
GAN Implementations: Use the Pix2Pix or CycleGAN repositories.
Ensure Compatibility:
Check the software requirements and dependencies of each component.
Use virtual environments to manage different library versions.
Understand Licensing:
Verify the licenses (e.g., MIT, GPL) to ensure you're compliant.
Integrate Components:
Modular Design:
Treat each component as a module with clear input and output interfaces.
Data Formats:
Use standard formats (e.g., JSON, NumPy arrays) for data interchange.
Handling Dependencies:
Resolve any conflicting dependencies between libraries.
Customize and Extend:
Modify components as needed for your specific use case.
Add or alter functionality while respecting the original license agreements.
Contribute Back:
If you improve upon or fix issues in a library, consider contributing back via pull requests.
Final Recommendations
Stay Organized:
Use a project management tool (e.g., Trello, Asana) to track tasks.
Keep your code in a version control system (Git).
Document Everything:
Maintain thorough documentation for your code and processes.
Community Engagement:
Join forums like Stack Overflow, Reddit's r/MachineLearning, and specialized communities.
Attend virtual meetups or webinars on related topics.
Be Mindful of Ethical Considerations:
Understand the implications of generating synthetic media.
Ensure respectful and lawful use of data and technology.
Iterative Learning:
As you build, continually test and validate your understanding.
Don't hesitate to revisit foundational concepts if needed.
Seek Feedback:
Share your progress with peers or mentors.
Engage in code reviews to improve code quality.
By following this detailed plan, you'll progressively build a solid understanding of the project and the underlying concepts. Utilizing GitHub repositories and libraries allows you to focus on integrating and customizing existing solutions rather than reinventing the wheel.

Remember: The key to success in such a complex project is iterative learning and development. Break down the problem, tackle each component methodically, and integrate them step by step. Along the way, you'll deepen your expertise in machine learning, computer vision, and software integration.

If you have any questions or need further clarification on any of the steps, feel free to ask!


is is possible to piece up components from similar implementations and only code the algorithms and make it work  ? if so how