Programming Environments

CHI 2015, Crossings, Seoul, Korea

T
TextAliv
ve: Integ
grated Design
D
EnvironmEment forr Kinetic
c Typography
Jun KatoTomooyasu Nakanno
Masaataka Goto
National Institute of Advanceed Industrial Science andd Technologyy (AIST), Tsukuba, Japaan{junn.kato, t.nakaano. m.goto}}@aist.go.jpp
A
ABSTRACTTThis paper pressents TextAliv
ve, a graphical tool that allow
ws
innteractive editting of kineticc typography videosv
in whichlyyrics or transcripts are animmated in syncchrony with thheccorresponding musicmor speecch. While existiing systems havveaallowed the deesigner and cassual user to crreate animationns,mmost of them dod not take intoo account syncchronization wiith
aaudio signals. They
T
allow preddefined motions to be applied tooobjects and paarameters to beb tweaked, butb it is usualllyimmpossible to ex
xtend the predeefined set of motionmalgorithm
msw
within these syystems. We th
herefore propose an integrated
ddesign environm
ment featuring (1) GUIs that designersdcan useu
too create and edit animatio
ons synchronized with aud
diossignals, (2) inttegrated tools that programm
mers can use toimmplement anim
mation algorithhms, and (3) a framework forfbbridging the in
nterfaces for designersdand programmers.
pAppreliminary useer study with designers, prrogrammers, anndccasual users demonstrated its capability in authoring
a
vario
ous
kkinetic typograp
phy videos.
A
Author Keywords

K
Kinetic typograaphy; animationn; creativity sup
pport tool;
inntegrated desig
gn environmentt; live programmming.
A
ACM Classifica
ation KeyworddsH
H.5.2. Informattion interfaces anda presentation (e.g., HCI):U
User interfaces – GUI; D.2.6. Software Engin
neering:
P
Programing Env
vironments – Inntegrated envirronments.
IN
NTRODUCTIO
ON

K
Kinetic typograaphy, a techniq
que for movinng and reshapinng
teext in a videoo, is widely usedu
in musicc, narrative, anndccommercial vid
deos to deliver text informatiion in a dynammic
w
way. It can delliver the emotiional feeling associated
a
with
ha
teext in additionn to the text’s literal meaning [17]. When it isuused in music and narrativee videos, text is animated inssynchrony withh vocal music or speech. Th
he animated teext
aaugments the listening experieence with visuaal effects, helpinngliisteners better comprehendcwh
hat is being voccalized.
K
Kinetic typograaphy videos hav
ve been producced with generaalppurpose tools forf creating aniimations (e.g., Adobe Flash or
A
After Effects [1]). While such tools provide a fine granulariityof conntrol over the animations, prroducing the ffinal result
takes long time eveen for the exxperienced useer because
al synchronizatiion of the textt animation annd audio is
manualtediouss. In additionn, general toools do not prrovide any
guideliines to help noovice users prooduce convinciing effects.
Tools specifically deesigned for maaking kinetic ttypography
videos (e.g., ActiveeText [18] oor Kinedit [7]) provide
predefiined templates of effective annimations, fromm which theuser caan choose. Succh domain-speecific tools, thoough, have
had a certain limitaation in their flexibility. Thhat is, themming neededd to create new
w animation tem
mplates has
programto be ddone in separatee development environments.
With tthe goal of puushing forwardd the state off the art in
kineticc typography deesign, we preseent TextAlive, a graphicaltool thhat allows inteeractive editinng of kinetic ttypography
videos (Figure 1). While toools for desiggners andprogrammmers have usually beeen designed separately,
TextAllive is designedd to benefit botth. We thereforre call it an
integraated design ennvironment (ann integrated deevelopment
environnment is one designed onlly for the proogrammer).
Thankss to its dualityy for the desiggner and progrrammer, its
researcch contributionn is three-fold. First, from ann input pairconsistting of audio and text infoormation, TextAlive can
automaatically compoose a playable kinetic typograaphy video
that serrves as a concrrete starting poiint for the desiggner. It also
providees interactive uuser interfacess for editing thhe video insynchrorony with the aaudio signals. S
Second, TextA
Alive allows
live prrogramming off animation teemplates with its built-in
developpment environm
nment that provvides continuouus graphicalfeedbacck to the proggrammer. Thirrd, its unique framework
allows the programm
mer to easily exxtend its GUI so that the
programmmer or desiggner can intuitiively debug orr customize
the parrameters of anim
mation templattes.
The rem
mainder of thiss paper is orgaanized as follow
ws. Related
work iis introduced tto highlight oour research coontribution.
Next, the design and implemeentation of T
TextAlive’s
interacction for the deesigner and proogrammer are iintroduced.

P
Permission to make digital or hard copiies of all or part of thist work for person
nal
oor classroom use is granted without feee provided that coopies are not made or
ddistributed for profit or commercial addvantage and that coopies bear this notiice
aand the full citation on the first page. CopyrightsCfor com
mponents of this woorkoowned by others than ACM must beb honored. Abstraacting with credit isppermitted. To copy otherwise, or repub
blish, to post on serv
vers or to redistribu
ute
too lists, requires priior specific permisssion and/or a fee. Request permissio
ons
frfrom Permissions@aacm.org.
C
CHI 2015, April 18 - 23 2015, Seoul, RepublicRof KoreaC
Copyright is held by
y the owner/author(ss). Publication rightts licensed to ACM.
A
ACM 978-1-4503-3145-6/15/04…$15.00
hhttp://dx.doi.org/10.1145/2702123.2702
2140

Figure 1. Overview off the TextAlive ssystem.

3403

Programming Environments

CHI 2015, Crossings, Seoul, Korea

Then, the results of a preliminary user study are discussed.
Finally, the conclusion and future work are presented. For
better understanding, readers are encouraged to see thedemonstration video and/or visit http://textalive.jp to try out
the system before further reading.

tool. It provides more precise and flexible control over the
text animation and an abstraction of an animation templatethat is reusable not only by the programmer but also by
designers and casual users.

RELATED WORK
Tools for Creating Kinetic Typography

While this work focuses on text animation, it is closely
related to prior work on tools for general animation. Adobe
Flash and After Effects [1] are professional tools used tocreate various kinds of animation. The user specifies key
frames for objects, and the system smoothly interpolates the
motion between the frames. These tools have good support
for direct manipulation, such as visualizing object trajectories
and using drag-and-drop to change the motion or seek time
periods. Compared with these professional tools, TextAlive
has a limited, or rather focused, GUI for editing text
animations synchronized with audio signals, including atimeline interface. The user does not specify key frames, butrather timings of each text unit being vocalized. The timing
information is then passed to animation templates that output
fluid motions. TypeMonkey [2] is a plug-in for After Effects
that has additional user interfaces for choosing text
animations from templates and editing their parameters forthe purpose of making kinetic typography videos. While it
reduces the pain of key-framing, it still requires manualsynchronization of the animation and audio. In addition, it
has the same issue as other tools for kinetic typography: thelimited number of templates restricts the user’s expressivity.

Various tools for creating kinetic typography videos havebeen proposed. However, they focus on graphical aspects ofcreating videos. To our knowledge, none of them deals withaudio signals synchronized with the videos. This is because
their primary use of kinetic typography is for augmenting
text-based communication. TextAlive, in contrast, aims toaugment audio listening experience with dynamic graphicalrepresentation of text information. Prior work on editing
time-series multimedia includes synchronized editing oftranscription and narration [21]. This is a complementary
technique and can be integrated into TextAlive.
The existing tools for kinetic typography videos target
programmers, designers, or casual users, not all three.
TextAlive is built on top of these. For instance, a framework
to help programmers create kinetic typography videos by
writing Java source code has been proposed [16]. TextAlive
shares some concepts with this framework, such as thehierarchical structures of phrases, words, and characters forrepresenting text information. The ActiveText [18] andKinedit [7] systems allow designers and casual users to input
text, choose animations from several templates, and edit theirparameters by using sliders and other standard GUIs.
TextAlive also provides similar features. A combination ofnatural language processing (NLP) and heuristics enables theextraction of emotional information from text and automatic
generation of the associated videos [20]. While the existing
techniques benefit from NLP, they do not allow userinteraction, which is essential if the user is to express his/hercreativity.
The existing tools have limitations because of their targetingcertain users. While tools for designers and casual users
provide interactive GUIs, they typically force the user to use
a predefined set of motion algorithms and do not allow any
extensions to be implemented. The tools for programmers, on
the other hand, have extendibility based on their APIs, but
their development environments are usually equipped with
text-based user interfaces and do not provide interactive
feedback on the source code changes.
There is a visual programming environment that overlays
code elements on top of the animation and allows binding a
set of elements to text units [3]. Each element represents one
typography property (font size, color, style, or scaling), andthe set of elements affects the properties of nearby text in a
way that depends on the distance between a set element and atext unit. It does not allow precise control of properties, nordoes it provide any abstraction, preventing reuse of thecreated animations. Instead of using visual programming,
TextAlive integrates text-based programming into a graphicalTools for Creating Animations

Tools that let casual users make animations, such as
Microsoft PowerPoint and Movie Maker, do not require keyframing. However, the resulting motions are usually limited
to those following simple trajectories. In particular, various
sketching interfaces allowing the casual user to create
animations have been proposed. K-Sketch [4] allows users tospecify the motion of objects by drawing trajectories, which
is implemented as our “Trace Path” animation. Draco [15] isa sketch-based tool to animate multiple objects. We also
target casual users who make animations of multiple objects(text units) but with more finely grained control for eachobject. In addition, TextAlive integrates a programming
environment that allows authoring of reusable animation
templates.
Tools for Live Programming

Live programming [19] is a technique to provide theprogrammer with continuous feedback about the program
being developed. It aims to get rid of the gulf of execution by
providing a seamless programming experience with regard towriting code, executing the program, and debugging it.
Live programming in TextAlive is inspired by Victor’s demo
[23] that allows the programmer to dynamically rewrite
parameters of graphical applications during their execution.
In our system, the programmer can simply click the “update”
button, and there is no need to re-launch the entire program
and reconfigure the parameters. It allows the programmer torewrite the logic of animation templates without losing theirparameter information. VisionSketch [13] uses a similar3404

Programming Environments

CHI 2015, Crossings, Seoul, Korea

teechnique, wheere the logic of image proceessing algorithmmsccan be dynamiically replacedd, but it does notn preserve thhesstate informatio
on when the logic is updatted. Subtext [6][ccontinuously revalidatesrth
he program withoutw
expliccit
ccompilation. It updatesuthe proogram output up
pon the typing of
eeach character in the source code editor. We
W intentionallly
aavoided such implicitirevalidation and keept the “updatte”
bbutton because the implicit revalidationr
flu
ushes the screen
vvery often and is
i distracting.
P
Professional too
ols for animatiion are typicallly equipped wiith
sscripting enginees. Adobe Flassh and Adobe After
A
Effects, forfeexample, have a built-in sccripting enginee to control thhepproperties of ob
bjects. Some too
ols meant for casualcusers, suchaas Microsoft PowerPoint,P
alsso allow such scripting. Whiilemmost scripting engines allow
w conversationaal programminng,thhese tools do not have deccent support forf bridging th
heaactivities of wrriting code andd running the program.
p
On th
he
oother hand, Jux
xtapose [12] is capable of geenerating a slid
derbbound to a variiable value, andd Unity allows the programm
mer
too extend its GU
UI for runtime parameter tuniing. Within theese
eenvironments, GUI widgets bridge that gap
g between thhestatic ddefinition and the runtime bbehavior of thee program.
TextAllive takes the ssame approachh, balancing thee variety ofwidgetts (more than Juxtapose) andd the lines of code (lessthan U
Unity). Moreeover, TextAllive is specialized foranimattion authoring. For instance, a set of APIs iis provided
for eassily animatingg text units. T
The Stage intterface canvisualizze object trajeectories calculaated from the user code.
Whereaas the prior woork intended foor game develoopment hassuch viisualizations [223, 9], TextAlivve uses the trajeectories not
only foor visualizationn but also for seeeking time periiods.
TEXTA
ALIVE: AN INTTEGRATED DE
ESIGN ENVIROONMENT

This seection introducces the interacttion design of TextAlive.
The firrst subsection describes interractive editingg of kinetic
typograaphy videos byy designers or casual users. T
This sort ofeditingg does not rrequire prior knowledge of kinetic
typograaphy or proogramming. T
The second subsection
describbes live progrramming of animation tem
mplates by
novice programmers. These days, itt is not uncomm
mon for thedesigneer to have programmingg skills. Forr instance,
programmming skills in scripting aare required too precisely
controll many objectts when usingg existing tools such as
Adobe After Effects [[1]. In TextAlivve, programmiing abilities
enable designers and casual users aas well as progrrammers tocontroll not only the currently preseent objects on the screen
but alsoo their later reuuse and custom
mization.
Interacctive Editing oof Kinetic Typoography Video
os

TextAllive can autom
matically creaate a kinetic ttypography
video wwhen given an audio file and its transcriptionn. Whereas
the exiisting tools reequire the userr to create a vvideo fromscratchh and to spend much time onn manual synchhronizationof audiio and text, TeextAlive signifficantly reducess the effort
requireed. The composed video can be interactively edited sothat it mmatches the useer’s style.
Automaatic Video Com
mposition and T
Timing Correctio
onF
Figure 2. Timeliine interface.

F
Figure 3. Editor interface andd integrated souurce code editoor;the user can swiitch between them by clicking the buttons (f).

The usser first inputs a pair consistinng of a song annd its lyrics
or a narration andd its transcripption. Then, tthe systemestimattes the timinggs of each chaaracter in the text being
vocalizzed [8]. Througgh this processs, not only chaaracters but
also woords that consiist of characterss and phrases tthat consistof wordrds and correspond to one linee in the original text cometo havee timing inform
mation about thhe starting andd ending of
the voccalization. The system then asssigns a defaultt animation
templaate to each textt unit (a characcter, word, or pphrase) andsynthessizes a playablee video. If a soong is providedd, its chorus
parts aare detected annd assigned a mmore gorgeouss animation
templaate (explained in the implem
mentation secttion). Beat
informmation is also aanalyzed, and th
the result is vissualized on
the Timmeline (Figuree 2). After thiss automatic synnthesis, thevideo iis shown on thhe Stage interfaace (Figure 1), and it will
alwayss be playable by clicking tthe “play” buttton in theTimelinne interface. A
Any operation ddescribed hereaafter can be
carriedd out interactivvely so that thhe video can bbe instantly
updatedd.
TextAllive cannot alw
ways estimate coorrect parameteers. If there
is an eerror in the esttimation, the tiiming informattion can be3405

Programming Environments

CHI 2015, Crossings, Seoul, Korea

interactively corrected in many ways before being saved for
use the next time the same pair of audio and text is loaded.
For instance, text units in the Timeline can be drag-anddropped to instantly fix their timing information. Drawing a
rectangle in the Timeline selects multiple text units thatoverlap the rectangle. Chorus parts are highlighted in thebottom row of the Timeline, and clicking them selects the
text units vocalized during the chorus. Once the text units are
selected, the “align left,” “center,” “right,” and “justify”
buttons in the Editor interface (Figure 3b) can be used to fix
their timings collectively. Timing information can also be
copied and pasted (Figure 3c), and this is quite useful forgroups of text units with the same structure, such as repeating
chorus parts.
The seek bar in the Timeline visualizes the sequentialchanges in vocal volume information extracted from theoriginal audio signals, thereby helping the user to estimate
correct timings visually. The seek bar allows one to select a
specific time region. While a region is selected, the “play”
button plays the corresponding part repeatedly, allowing one
to concentrate on correcting the timing and editing theanimation of the specific part. Automatic synchronization
estimating the timings of the text units contained in theselected region can be applied again by clicking “auto-sync”
in the Editor interface (Figure 3a).
Font and Animation Template Assignment

An automatically created video does not at all reflect theuser’s intention. To make it his/her own, the user selects text
units in the Timeline and changes their properties, including
fonts, and animation templates with the Editor interface(Figure 3d, e). As in timing correction, property changes canbe collectively made on multiple text units. For instance,
when a phrase is selected, changes in the font style and size
are applied to all of the characters in the phrase.
Predefined animation templates allow novice users to create
aesthetic kinetic typography. To avoid implausible visual
effects, they are not always applicable to every type of text
unit: phrases, words, and characters have different sets ofassignable animation templates. In addition, multiple
templates can be assigned to a single text unit at the sametime. For example, “Sliding Animation” can be assigned to aphrase, and “Hopping Animation” can be assigned to thecharacters in that phrase. “Karaoke Animation” that changes
the text color before/after vocalization can also be assigned tothe characters (Figure 4). Combinations of different

Figure 4. A simple example of kinetic typography effects
generated from different sets of preset animation templates.

templates that can be assigned to suitable types of text units
allow flexible yet effective kinetic typography design.
Animation Customization

Each animation template has parameters for customizing its
appearance, and these parameters can be interactively
manipulated with the GUI widgets shown on the Editor(Figure 3g) or the Stage. These widgets are dynamicallygenerated by the template implementation, which is
described in detail in the next subsection. They appear when
the corresponding template is selected in the Editor interface.
While the Stage can usually show only the current frame ofinterest in the video, it would be desirable that the user can
instantly see how an object’s motion is affected by parametertuning. To make this possible, the trajectories of the selected
text units can be visualized on the Stage (Figure 1). The
trajectories can be used to seek time periods in the way
shown in prior work [5].
Live Programming of Animation Templates

While each animation template can be customized to meetthe user’s needs, it still limits the resulting animation to acertain extent. General tools for creating animations are often
equipped with scripting engines with which the animation oflots of objects can be controlled in a mathematically precise
way. Such scripting, though, typically does not allow laterreuse because it lacks an abstraction mechanism. There is notmuch graphical feedback while the user is writing code either.
TextAlive aims to address this issue by enabling “liveprogramming” of animation templates: the user canseamlessly go back and forth between editing animation andimplementing templates (Figure 3f). The implementation canbe updated by just one click and there is no notion ofcompilation or execution. The program that creates theanimations continues running virtually. The update process
updates not only the behavior of the text units but also the
GUI widgets specific to the template (Figure 3g). Thesewidgets help the programmer debug the template and help
the user interactively customize it.
Implementation of Animation Templates

To read the implementation of an existing animation template,the user selects the template and clicks the “edit” button in
the Editor interface. TextAlive then shows the code editor,
with which the source code written in Java can be edited(Figure 3h). Each template is defined as a Java class that
implements a specific Java interface. The template
implementation can be updated by clicking the “update”
button, which instantly updates animations of the text units
that use the template. This update can be done at any moment,such as during the playing of a song or when the user isobserving the motion of text units. To create a new template,the user can either click the “new” button or change the classname in the source code of any existing template and click
the “update” button.
Updating running software is generally called a hot swap. It
replaces existing objects using old code with dynamically3406

Programming Environments

CHI 2015, Crossings, Seoul, Korea

generated new objects using new code. Since such a processsimply discards the state information of the old objects, thestate information (in our case, customized parameters of ananimation template) usually needs special care, such as by
the programmer writing some code to store and restore it.
TextAlive handles this transparently, which frees the
programmer from having to write boilerplate code and allows
him/her to concentrate on the algorithm.
Implementation of GUI Widgets for Each Template

As discussed in Animation Customization, each template has
dedicated GUI widgets to customize its appearance. “Sliding
Animation,” for instance, has a box of radio buttons to selectits horizontal alignment and a slider to control its sliding
angle. In order to implement a GUI widget, the programmer
should declare a Java public field and write a comment blockright before the field declaration in the source code. The
comment should be written in a simple format: “@uiWidgetName([options]).” The generated GUI widget isbound to the public field; GUI operations on the widgets
change the field values.
The widgets are not only meant to help designers and casual
users manipulate the parameters intuitively. They can also
help the programmer debug and test the template during thedevelopment process. For instance, the programmer cantemporarily create a box of radio buttons for switching
between multiple implementations having a similar effectand check which one works best. Once he/she knows which
one is best, he/she removes the radio buttons and publishes it
for other users.
Below is the list of preset GUI widgets available in thecurrent implementation of TextAlive (Figure 5). Other
widgets can be easily added by writing Java source code.
@ui Slider(min, max) shows a slider to change the value ofthe corresponding integer field between min to max.
@ui Check() shows a check box to switch between true andfalse values of the corresponding boolean field.

Figure 5. Comments and GUI widgets generated from them.

@ui Radio(“label1”, v1, “label2”, v2, …) shows a box ofradio buttons used to select a value.
@ui Color() shows a button with a colored rectangle so thatthe user can choose a color with the color picker dialog.
@ui File() shows a button used to choose a file whose path isstored in the corresponding String field.
@ui Track() shows a check box used to enable/disabledrawing a path by dragging the mouse on the Stage. When apath is drawn, its data is represented as a list of (x, y)
coordinates and stored in the corresponding List field. This isused to implement the “Follow Path” animation template thatmakes text move along the specified path.
IMPLEMENTATIONThis section introduces the current implementation of the
TextAlive system. It first gives an overview of the systemalong with the techniques used in the implementation, andthen it explains the system in more detail. The first
subsection explains how the creation of plausible kinetic
typography videos is supported, and the second explains how
a live programming experience is provided.
Overview

The architecture of TextAlive is shown in Figure 6. It iscurrently implemented with Java 7 (64-bit) and tested on a
personal computer with a Windows 8.1 (64-bit) OS. Input
sound files should use the MPEG-2 Audio Layer III format
for playback in the system and are converted to the monaural
MS WAV format using FFmpeg for further audio signal
processing. Input text is written in a natural language (either
Japanese or English) and is converted into a series ofphoneme information by using MeCab, open-source software
for morphological analysis and part-of-speech tagging.
The audio signal is synchronized with the corresponding
phoneme information [8] by attaching timing information to
the phrases, words, and characters of the original text. If the
input sound file is a song, the beat information and the chorus

Figure 6. Software architecture of the TextAlive system.

3407

Programming Environments

CHI 2015, Crossings, Seoul, Korea

part are detected [10]. This information is referenced forautomatic video composition. The Java source code ofanimation templates is dynamically compiled with the
Eclipse Compiler for Java and loaded into the Java VM with
the Java Reflection API. In order to store and restore the state
information of the animation templates, a text-based JSON
format is used as a temporary backup. The animation data isalso saved as a JSON file (Figure 9).
Implementation for Kinetic Typography Design
Automatic Video Composition

Given the automatically estimated parameters, TextAlive
automatically assigns animation templates to text units andgenerates a playable kinetic typography video. By default, it

Figure 7. Example implementation of an animation template.

simply assigns the “Sliding Animation” to all phrases and the“Karaoke Animation” to all characters. If the input sound is asong, phrases in the chorus part are assigned the “Sliding andRotating Animation” and characters in the chorus part are
additionally assigned “Hopping Animation” so that thechorus part looks more prominent than the other parts.
The strategy that defines how to assign animation templates
is currently written as Java source code and can be easily
customized. We plan to implement a GUI for choosing andimplementing new strategies.
Text Animation

In a lot of kinetic typography videos, phrases, words, andcharacters have their own motion algorithms independent
from the others. In order to animate text information in ahuman-readable way, characters often keep their positions
relative to each other and are aligned horizontally or
vertically. Multiple words are often organized into a phrasethat is prevented from overlapping with other phrases. Such
motion algorithms can be described in a simple and clean
way by using the coordinate system relative to the parent text
units (e.g., characters to words and words to phrases).
Given the characteristics of text animation, each text unit
holds its timing information (when it starts and ends being
vocalized), a list of assigned animation templates, a font
name, and one or more characters contained in the unit. Eachanimation template is required to implement a method named
animate(time) whose argument is the current time. An
example implementation is shown in Figure 7. The template
has access to its assigned text unit, and the animate method
modifies the rendering parameters of the text unit. The
parameters include font color, an alpha compositing rule
(how to blend the text graphic with the existing pixel values),
and an affine transform matrix (a three-by-three matrix thatdefines how to transform the text graphic and includes
translation, rotation, and scaling) relative to the parent text
unit. Since the animate method can be called with any timevalue at any time, the method is usually expected to be a purefunction – in other words, it should return a value without
modifying the state of the animation template object.
While changing the rendering parameters can reshape theappearance of the text unit as a vector graphic, there are
kinetic typography effects that cannot be supported by such
reshaping and instead demand pixel-based operations. They
include adding blur or shadows to the text graphics. To
support such effects, an animation template can implement
background(graphic context, time) and foreground(graphic
context, time) methods that render graphics behind and in
front of the text units. Since these methods are called after
rendering all text units in the off-screen buffer by using theanimate(time) method, they have access to the pixel-based
representation of the text units.

Figure 8. Pseudocode for rendering each frame.

TextAlive plays and renders the video in a 30-frames-persecond format. The pseudocode for rendering each frame isshown in Figure 8. Graphic units are optional units that hold3408

Programming Environments

CHI 2015, Crossings, Seoul, Korea

the start and end times as well as a list of animation templates
mainly used for rendering the background or foreground tocomplement or emphasize text information.
Trajectory Rendering

Trajectory rendering allows the user to see the past and future
motion at a glance. The same procedure as used for rendering
frames is repeatedly called without actually using renderinggraphics, and the absolute time-series positions of theselected text units are visualized as their trajectories.
The trajectory can also be used as an interface to seek time
periods. Since the system knows the correspondence between
a specific point in the trajectory and timing, the current
mouse position can easily be converted into timing
information by finding the closest point in the trajectory.
When the trajectory is not straight-forward, though,following the closest point sometimes results in a big jump in
time, which is not desired. To avoid this problem, thedistances between the mouse position and the next orprevious point in the trajectory are calculated.
Implementation for Live Programming ExperienceHot-swap for Updating Animation Templates

Animation templates are represented by Java classes that arecompiled and dynamically loaded into the Java VM when
TextAlive launches. After the user edits their source code, thecorresponding classes are recompiled and the resulting Java
bytecode is reloaded into the VM.
Such a process, i.e., replacing classes without restarting theVM, is called a hot-swap. A general hot-swap simply
replaces an old class definition with a new one and is not
sufficient for updating animation templates. If the updatingprocess only updated the class definition, existing objects
with old definitions would not be updated. To see theupdated motion, the user would need to remove the objects,
re-assign the animation template, and reconfigure parameters.
To avoid such hassle, TextAlive automatically extracts state
information from old objects, creates new objects with theupdated implementation, and makes its best effort to restore
the state information. Before a hot-swap, a backup of thestate information of the existing objects is created by calling
the Java Reflection API to serialize all public fields of eachobject. It is used after the hot-swap to construct objects withthe updated class definition, in effect replacing the old
objects. The process of restoring state information is identicalto that of loading videos, which is described later.

the widget class name and its options. The option values areevaluated using a subset of the Java programming language,
thereby providing dynamic parameters to widgets. The
language supports simple mathematical calculations andread-only access to useful parameters such as the resolutionof the video (the size of the Stage). For instance, a slider
with the maximum value of the height of the Stage can begenerated by writing “@ui Slider(0, stageHeight)” as shownon line 17 of Figure 7.
Saving and Loading Videos

TextAlive saves a kinetic typography video in plain text
JSON format. It consists of a hierarchical architecture ofphrases, words, and characters. Each text unit has the
vocalization start and end times and a list of information
about the assigned animation templates. Unlike phrases andwords, characters are also saved with their font name and size.
Each animation template is represented by its name and thecustomized parameters. Figure 9 shows an example of a
JSON object of a video.
Loading videos is exactly the reverse of saving them, except
for the chance of animation templates being updated. If an
animation template has an updated definition, one or more
public fields might be added or removed. To construct ananimation template object with an updated definition,
TextAlive makes its best effort to maintain the state
information; newly added fields have their default values,
and old values of removed fields are simply discarded.
PRELIMINARY USER STUDY

We conducted a preliminary study to gain user feedback on
the TextAlive system and to investigate the capabilities,
limitations, and potential of our interaction design. Seven
users with varying levels of expertise in authoring videos andprogramming graphical applications participated in the study.
They were asked to use the system and to create a kinetic
typography video with one favorite song or narration. They
were asked to compare their use of the system against theirprior experience on tools for creating videos and graphicalapplications.
Participants

Seven participants aged 23 to 30 years old (mean 27) took
part in the study. Each was paid $30 for their participation.
P1-4 are designers and casual users (two females and two
males). P1 is an amateur singer who has used Adobe After
Effects [1] to create a simple slide show for her music video;
P2 is an amateur illustrator who has no prior experience inStatic Code Analysis for Generating GUI Widgets

The GUI widgets for customizing animation templates are
generated by static code analysis upon compilation of theclass definition. The widgets can collectively change thevalues of multiple variables (i.e., the same parameters ofmultiple animation template objects) when multiple text units
are selected in the Timeline.
Each comment that begins with @ui and is located rightbefore a public field declaration is parsed and divided into

Figure 9. JSON-formatted text for saving the animation.

3409

Programming Environments

CHI 2015, Crossings, Seoul, Korea

creating videos; P3 is an amateur disk jockey who has
extensive experience in mixing music and recording
narrations for Rakugo (a traditional form of Japanesestorytelling) but no video authoring experience; P4 is aninteraction designer who has used Motion, Final Cut Pro,
openFrameworks, and Unity [22] for creating videos.
P5-7 are programmers and are all male; P5 has a small
amount of experience creating demonstration videos for his
software; P6 is an experienced user of Adobe After Effects (>2 years) and has created music videos; P7 has a hobby ofwriting songs and has created music videos with AviUtl,
which is freeware for casual video authoring.
Experimental setup

All participants used a standard set of laptop computers witha full HD (1920 x 1080 pixels) display and a mouse (Figure10). Some of the participants brought their headphones or
earphones; others used earphones we provided. The
TextAlive system occupied the entire screen during the study.
The experiment followed the steps described below and took2-3 hours for each participant.
Pre-experiment questionnaire: Each participant was asked tofill out a form asking his/her experience in authoring songs,
narrations, videos, and programming.
Introduction and demonstration: Each participant was given a

5-15 minute introduction to the TextAlive system by the
instructor. The instructor then demonstrated the timing
correction, font and animation template assignment, andanimation customization using the GUI widgets. For the
programmers (P5-7), the instructor also demonstrated the useof the integrated source code editor to modify and create an
animation template.
Kinetic typography design: Each participant was given 2hours maximum to create a kinetic typography video of
his/her favorite subject with the system. The participant wasallowed to bring data of a song, narration, or image to be
used in the video or could choose from a list of the songs we
prepared. Since it is usually impossible to create a video foran entire song or narration in 2 hours, the participant was
asked to work on his/her favorite part of the song or narration.
During this open-ended task, the participant was free to ask
any questions about the usage of the system, including how
to use the preset animation templates and APIs for
implementing animation templates. The participant wasallowed to end the study when he/she was satisfied playing
with the system.
Post-experiment questionnaire: Each participant was asked tofill out a form asking five questions about the usability of thesystem (selected answers on a 7-point Likert scale), and threequestions about the good and bad points of the user interface.
Results and Lessons Learned

All participants could successfully create kinetic typography
videos for their favorite part of a song or narration, some ofwhose screenshots are shown in Figure 11. Two participants
used a song we prepared, while the others used their favorite
song or narration. The results of the post-experimentquestionnaire consisting of the mean, standard deviation, andpercentage of positive responses (>4 on a 7-point Likert
scale) are shown in Table 1.
Everyone who participated in the study was positive about
how suitable TextAlive is for authoring kinetic typography
videos (Q5). There were no answers representing negative
impressions of the TextAlive system on Q1-3, but there wereseveral neutral answers (= 4 points). The neutral answers
were found to come from non-critical reasons such as theparticipant not being enthusiastic enough in creating kinetic
typography videos (P1) and practical reasons that can beaddressed by making minor revisions to the system such asadding keyboard shortcuts to frequently used commands (P7).
Those who used the live programming feature (P5 and P7)
indicated they felt the need for technical support in order to
use the system. Their negative answers stem from theprogramming experience described later.
Automatic video composition was useful for both novice andexperienced users: All participants liked the automatic video

composition feature a lot. P7 commented “automaticallyestimating timing information and synthesizing a playable
video is extremely useful, reducing the effort always neededfor existing tools I have used.” P3 commented “it is nice that
we no longer need to create but just edit the video.”
The abstraction of phrases, words, and characters was
welcomed: Participants with prior experience of creating

videos with text information such as subtitles (P1, P4, P6)
especially favored the hierarchical abstraction, enabling
assignment of different animation templates to each phrase,
word, and character. P1 noted that “the capability ofmodifying the properties of each character is very useful.”This feature is often missing from existing video authoring
tools.
The abstraction of animation templates suggested potential
applications: While animating text in a specific time period

Figure 10. Setup of the user study. (a) A participant using oursystem. (b) A screenshot taken during the study.

requires manual parameter tuning in existing authoring tools,
animation templates in TextAlive adapt the animation to fit
in the period. This allows the user to almost instantly animate
text in synchrony with sound. P3 and P4 expressed their
desire to play “text jockey” during their live disc jockey
performances. P3 also foresees using it for Rakugo
storytelling.

3410

Programming Environments

CHI 2015, Crossings, Seoul, Korea

No need to stop the video, enabling a fluid experience: Many

users appreciated that almost every operation on the systeminteractively updates the resulting video. P5 commented that“there was no need to pause or stop the video. With
TextAlive, I could select the time region of interest and playthe video continuously while editing it. The only exception
was when I got too tired to listen to the same part of the song.”
P6 appreciated that “animation templates can be edited inplace (without launching external editors.)”
Having a variety of animation templates counts: P6
appreciated that the system allows novice users to create
good looking videos simply by choosing animations from alist of preset templates. P1 and P7 requested more variety of
templates. P4 wrote that “more variety of animation
templates would make TextAlive more appealing, but too
many templates might overwhelm the user.” Actually, P3 andP7 complained that the current combo box for choosing an
animation template is not informative enough. They
commented that more detailed explanations or graphicalpreviews would be desirable. In addition, P6 pointed out thedifficulty of predicting the result of combining multiple
animation templates, suggesting that we provide preset
combinations of animation templates.
The Timeline interface is at the heart of synchronizing audio
and video: All users appreciated the usability of the Timeline

interface dedicated to timing correction. P1, P2 and P5
especially appreciated the wave-form visualization of vocal
sound and the visualization of beat information. In addition,
P1 and P5 requested snapping to the beat information whilemoving or resizing text units in the Timeline.
The Stage interface should allow (more) direct manipulation:

P3 appreciated that most of the interfaces for customizing
appearance of the selected units are gathered in the Editor
interface. On the other hand, P2 commented that most of the# Question
1 I would like to use it frequently.
2 I found it unnecessarily complex.
3 I thought it was easy to use.
4 I needed technical support to use it.
5 I thought it is suitable for authoring kinetic
typography videos.

The TextAlive system
Mean
SD
%
5.57
1.20
5/7
2.71
1.02
0/7
4.86
0.75
4/7
3.29
1.74
2/7
5.57

0.75

7/7

Table 1. Results of the post-experiment questionnaire.

preset animation templates only use the widgets in the Editor
interface and the animation templates should make more useof the Stage interface for their customization. Currently, the“Follow Path” animation is the only preset template thatallows the user to interact with the Stage to interactively
update the tracing path. P1, P3 and P4 requested that simple
animations should be customized by direct manipulation on
the Stage, e.g., allowing drag-and-drop of text units to fix
their positions at a certain timing. However, it is not straightforward to support direct manipulation of objects whose
motion is defined by a combination of multiple algorithms.
Since each animation template transforms the coordinatesand does not know how the other templates are implemented,
the parameters for the template to move the assigned unit to
the mouse cursor position cannot be easily retrieved. The
process is identical to finding a set of parameters for an
unknown function that returns a specific value. Similar
problems in physical simulations have been addressed byintroducing novel interaction techniques that might beapplicable to our case [11]. More improvements to thesketching interaction might also be helpful, as has been
indicated in prior work [15].
Programming ensures extendibility but needs training: Users

who tried the programming interface (P5-7) all appreciatedthe extendibility of TextAlive, but at the same time, found it
difficult to learn in 2-3 hours. P5 hesitated to use the feature.
P7 commented that “the live programming feature allows
fine adjustments of animation but requires prior knowledge,
making me procrastinate in its use for some time.” However,
he was the most active person to create animation templates
by the end of the experiment and also commented that “thechanges in code could be instantly applied to the video,
which is nice.” P6 appreciated how easy it is to provide GUI
widgets for tweaking parameters. Given that the live
programming feature itself is not considered harmful, more
software engineering effort to support more “learnable
programming” [24] is desired.
CONCLUSION

We have described TextAlive, a tool for creating kinetic
typography videos. It can automatically estimate the timingof each character being vocalized. Instead of providing a
blank canvas and forcing the user to create videos fromscratch, TextAlive analyzes a given pair of audio and text
information and creates a playable and editable video. The
video serves as a foundation on which the user can create
his/her own video. The user interface for interactive editingof the video synchronized with audio signals was highly
appreciated by the participants of the user study who hadvarious professional backgrounds. The capability of editing
animation templates by writing programs ensures theextendibility of the TextAlive system as a tool that cancontinue to evolve through use. While the user study
confirmed its usefulness, its learning curve should be gentler,
and more software engineering help provided.

Figure 11. Screenshots of kinetic typography videos in various
styles created by the participants (selected and trimmed.)3411

Programming Environments

CHI 2015, Crossings, Seoul, Korea

Although the number of websites for sharing videos andaudio files is increasing and more people are getting involvedin creating new media content, it is still not easy to create
videos synchronized with audio. Music videos on suchwebsites typically feature the artists and illustrations. The
lyrics are often placed in the safe area near the edges of thevideo or are not rendered on the video at all. Kinetic
typography is effective at expressing the message of a songor narration visually. The tool described in this paper
enhances the expressiveness of casual users by enabling themto create kinetic typography videos. While its current
implementation is limited to placing objects on a twodimensional plane, we are working on an extension to threedimensional space.
TextAlive is available online at http://textalive.jp. Currently,
many songs and narrations and their lyrics and transcriptions
are available on the Internet. Preprocessing such information
allows the user to create kinetic typography videos without
first wading through the synchronization process. Corrected
timing information can be shared with others, enabling
crowdsourcing of such timing information. Animation
templates can also be easily shared with others, meaning thatthe integrated design environment can be extended
cooperatively. While this study focused on the experience ofusers with different backgrounds, the proposed interaction
design enables various forms of collaboration (e.g.
programmer with designer, designer with end user, etc.).
Investigating such interactions would make an interesting
follow up.
ACKNOWLEDGMENTS

This work was supported in part by JST, CREST.
REFERENCES

1. Adobe After Effects.
http://www.adobe.com/products/aftereffects.html.
2. aescripts + aeplugins. TypeMonkey.
http://aescripts.com/typemonkey/.
3. Chao, C. M. and Maeda, J. Concrete programming
paradigm for kinetic typography. In Proc. IEEE VL 1997,
446-447.
4. Davis, R. C., Colwell, B. and Landay, J. A. K-Sketch: a“kinetic” sketch pad for novice animators. In Proc. CHI
2008, 413-422.
5. Dragicevic, P., Ramos, G., Bibliowitcz, J.,
Nowrouzezahrai, D., Balakrishnan, R. and Singh, K.
Video browsing by direct manipulation. In Proc. CHI
2008, p.237-246.
6. Edwards, J. Subtext: uncovering the simplicity of
programming. In Proc. OOPSLA 2006, p.505-518.
7. Forlizzi, J., Lee, J. and Hudson, S. The Kinedit system:
affective messages using dynamic texts. In Proc. CHI
2003, 377-384.

8. Fujihara, H., Goto, M., Ogata, J., and Okuno, H. Lyric
Synchronizer: automatic synchronization system between
musical audio signals and lyrics. IEEE Journal of
Selected Topics in Signal Processing, 5(6), 2011.
9. Fukahori, K., Sakamoto, D., Kato, J. and Igarashi, T.
CapStudio: an interactive screencast for visual application
development. Ext. Abstracts CHI 2014, 1453-1458.
10. Goto, M, Yoshii, K., Fujihara, H., Mauch, M. andNakano, T. Songle: A web service for active music
listening improved by user contributions, In Proc. ISMIR
2011, 311-316.
11. Ha, S., McCann, J., Liu, C. K. and Popovic, J. Physics
storyboards. Computer Graphics Forum 32, 2 (Proc. EG
2013), 133-142.
12. Hartmann, B., Yu, L., Allison, A., Yang, Y. andKlemmer, S. Design as exploration: creating interface
alternatives through parallel authoring and runtime
tuning. In Proc. UIST 2008, 91-100.
13. Kato, J., Sakamoto, D. and Igarashi, T. VisionSketch:
Integrated support for example-centric programming ofimage processing applications. In Proc. GI 2014, 115122.
14. Kazi, R., Chevalier, F., Grossman, T. and Fitzmaurice, G.
Kitty: Sketching dynamic and interactive illustrations. In
Proc. UIST 2014, 395-405.
15. Kazi, R., Chevalier, F., Grossman, T., Zhao, S. and
Fitzmaurice, G. Draco: bringing life to illustrations withkinetic textures. In Proc. CHI 2014, 351-360.
16. Lee, J., Forlizzi, J. and Hudson, S. The kinetic typography
engine: an extensible system for animating expressive
text. In Proc. UIST 2002, 81-90.
17. Lee, J., Jun, S., Forlizzi, J. and Hudson, S. Using kinetic
typography to convey emotion in text-basesd
interpersonal communication. In Proc. DIS 2006, 41-49.
18. Lewis, J. and Weyers, A. ActiveText: a method for
creating dynamic and interactive texts. In Proc. UIST
1999, 131-140.
19. McDirmid, S. Living it up with a live programming
language. In Proc. OOPSLA 2007, 623-638.
20. Minakuchi, M. and Tanaka, K. Automatic kinetic
typography composer. In Proc. ACE 2005, 221-224.
21. Rubin, S., Berthouzoz, F., Mysore, G. J., Li, W. andAgrawala, M. Content-based tools for editing audio
stories. In Proc UIST 2013, 113-122.
22. Unity. http://unity3d.com/.
23. Victor, B. Inventing on Principle (2012).
http://vimeo.com/36579366.
24. Victor, B. Learnable Programming (2012).
http://worrydream.com/LearnablePrgramming/.

3412