MusicCommentator: Generating Comments
Synchronized with Musical Audio Signals
by a Joint Probabilistic Modelof Acoustic and Textual Features
Kazuyoshi Yoshii and Masataka Goto
National Institute of Advanced Industrial Science and Technology (AIST)
Central 2, 1-1-1 Umezono, Tsukuba, 305-8568 Ibaraki, Japan{k.yoshii,m.goto}@aist.go.jp
Abstract. This paper presents a system called MusicCommentator thatsuggests possible comments on appropriate temporal positions in a musical audio clip. In an online video sharing service, many users can provide
free-form text comments for temporal events occurring in clips not for
entire clips. To emulate the commenting behavior of users, we proposea joint probabilistic model of audio signals and comments. The systemtrains the model by using existing clips and users’ comments given to
those clips. Given a new clip and some of its comments, the model is used
to estimate what temporal positions could be commented on and whatcomments could be added to those positions. It then concatenates possible words by taking language constraints into account. Our experimental
results showed that using existing comments in a new clip resulted in
improved accuracy for generating suitable comments to it.
Keywords: Audio and language processing, user communication modeling, probabilistic music-comment association, comment generation.

1

Introduction

Commenting plays important roles in the entertainment culture of the consumer
generated media (CGM). We can access many online content-sharing services
such as YouTube (video), MySpace (music), and Flickr (photo) that enable users
not only to present their original works but also to comment on works createdby others. Users who view the same work can communicate with each other bycommenting. For example, users provide tags [1], describe positive or negative
reviews [2], and show their agreement or disagreement with speciﬁc comments.
This kind of user communication has recently been facilitated.
Commenting can be viewed as collaborative creation [3] in an advanced form ofuser communication: pseudo-synchronized communication. We can see evidenceof this in a video sharing service named Nico Nico Douga (whose name means“video making people smile” in Japanese),1 where users can provide comments
1

Nico Nico Douga reserves over 17 hundred million comments by ten million users.
http://www.nicovideo.jp/, http://en.wikipedia.org/wiki/Nico Nico Douga.

S. Natkin and J. Dupire (Eds.): ICEC 5709, pp. 85–97, 2009.
c IFIP International Federation for Information Processing 2009


86

K. Yoshii and M. Goto

Numerous simultaneous comments
covering most of the original video
Overlaid comments are
flowing from right to left

Playback position

SynchronizedTemporal Comment
position

ASCII art (text art) by skilled users

Fig. 1. Screenshots captured from a video sharing service named Nico Nico Douga

at arbitrary temporal positions in the video. A unique feature of this service isthat recent comments of many users are overlaid directly onto the video andsynchronized to a speciﬁc playback time as shown in Fig. 1. This gives users asense of sharing the viewing experiences. That is, users feel as if they enjoyed
the same video together in real time although their comments were provided at
diﬀerent dates and times in the real world. Some kinds of comments therefore
can add remarkable and interesting eﬀects to the original video. For instance, we
often see barrages, where so many identical or similar comments collaboratively
made by many users are piled up to a degree that the original video is almost
completely hidden, and skilled users create cool drawings2 by combining many
comments (characters) provided at temporally and spatially diﬀerent positions.
Because commenting is much easier than creating new video clips from scratchor reediting existing ones. it can be an important popularized way of creation.
Novice users, however, sometimes feel anxiety when commenting, wondering
Is my comment suitable to the occasion? Implicit rules seem to be shared amongusers who collaborate to provide comments, so novice users had better experience
what kinds of comments are given by other users and what kinds of temporal
events are annotated. Another issue is that video creators can hardly predict
what comments will be given to their works, and the fear of being insulted often
makes them hesitant about presenting their work to the public, especially theﬁrst time they try. Besides these practical issues, from an academic viewpoint, weare interested in the relationships between music and comments and investigate
whether it is really possible to generate comments in a human-like fashion.
We therefore developed a system, called MusicCommentator, that can generate comments that are most likely to be provided at speciﬁc temporal positions
in a music video clip. It can help novice users by suggesting comments suitable
to the occasion and can help encourage video creators to present their work byletting them virtually experience having comments made about their work. In
this study we deal with music, one of the audio parts within video clips, as theﬁrst step toward handling all the information in the clips.
2

ASCII art or text art: http://en.wikipedia.org/wiki/ASCII art.

Generating Comments Synchronized with Musical Audio Signals

87

The rest of this paper is organized as follows. First, Section 2 introduces
related work. Then, Section 3 speciﬁes the commenting problem and Section 4
explains how to build our system. Section 5 reports on our experiments. Finally,
Section 6 summarizes the key ﬁndings of this paper.

2

Related Work

Several studies have been conducted to predict suitable words to a given musical piece by using the audio signal as input. These studies typically estimate
how strongly each word is associated with a given piece. For example, Whitman
and Rifkin [4] used a kernel method to predict words that will appear in music
reviews. Turnbull et al. [5] attempted to associate audio content with semantically meaningful words by using a Gaussian mixture model (GMM) of acoustic
features for each word. The output is obtained as sentences by ﬁlling slots withpredicted words in sentence templates manually prepared beforehand. BertinMahieux et al. [6] used an ensemble learning method called AdaBoost to predict
social tags, which are free-form text labels at a song or artist level.
Our study diﬀers from the previous studies in two ways. First, we deal withcomments that are not given to an entire piece but provided at arbitrary temporal positions in it. It is thus necessary to determine what temporal positions
can be annotated in a given piece. Second, we try to generate comments as
natural-languages sentences. These goals make our attempt very challenging.

3

Problem Speciﬁcation

The input data for the MusicCommentator task contains N audio clips (audio
tracks of video clips) and their comments provided by users. Note that we focus
on audio tracks in this paper even if we deal with video clips. Let n (1 ≤ n ≤ N )be the index of an audio clip. This data is used to train a computational modelof commenting. When the system is given a new audio clip (and some existing
comments on it), the objective is to add reasonable comments at appropriate
temporal positions by using the model. Audio clips are represented as acoustic
features and comments are represented as textual features.
1. Acoustic Features: We use mel-frequency cepstrum coeﬃcients (MFCCs)
and their delta components because these features have been eﬀectively used
for characterization of detection of musical genres and moods [7]. Calculating
MFCCs at each frame,3 we can obtain a temporal sequence of feature vectors.
(n)
Let at be a feature vector of frame t in clip n.
2. Textual Features: We deﬁne three kinds of textual features of comments.
(a) Bag-of-Words Features: These features represent the content of provided comments. We split all Japanese free-form comments into words4
3
4

A “frame” here is a short duration (256 ms) to be analyzed in an audio clip.
Some words have single morphemes while others have two or more morphemes withinthem. The inﬂectional word “loved,” for example, consists of the base morpheme“love” and the inﬂectional morpheme “ed” (past tense).

88

K. Yoshii and M. Goto

with a Japanese morphological analyzer called Mecab [8]. Because diﬀerent words including the same base morpheme are semantically identical,
we do not distinguish them. Removing auxiliary words and extracting
signiﬁcant words whose numbers of occurrences are higher than a threshold, we get a vocabulary consisting of V words. Then, we count how
many times each word occurs. For example, if a frame contains threecomments, “I love it,” “It is loved,” and “Love song,” the average num(n)(n)(n)
ber of occurrences of the verb “love” is 0.66. Let w t = {wt,1 , · · · , wt,V }(n)be a bag-of-words vector, where wt,v (1 ≤ v ≤ V ) represents the numberof occurrences of word v per comment at frame t in clip n.
(b) Comment Density: This indicates the number of comments in eachframe. Note that feature values in each clip are normalized with respect
to its length and the number of comments. This feature is used to learn(n)what temporal positions should be annotated in a target clip. Let dt
be a feature value of frame t in clip n.
(c) Average Length of Comments: This indicates the average number ofwords in a single comment and is used to learn how long comments could
be generated. In the above example with “love”, the value of this feature(n)
is 2.66 ((3 + 3 + 2) / 3). Let lt be an average length of comments offrame t in clip n.
(n)

(n)

(n)

(n)

(n)

A set of these features is given by ot = {at , wt , dt , lt }. When clip n
contains Tn frames, the observable features O (n) and O are given by O (n) =(n)(n){o1 , · · · , oTn } and O = {O(1) , · · · , O (N ) }.

4

MusicCommentator

MusicCommentator takes a constructive approach that tries to clarify the cognitive mechanism of humans by building and examining a computational model
emulating it. As shown in Fig. 2, the system comprises a learning phase in which
the system tries to acquire a sense of commenting (i.e., build a computationalmodel of what comments are suitable to speciﬁc acoustic features) by experiencing many comments provided by users, and a commenting phase in which themodel is used to generate comments suitable to the occasion. We will discuss
how to design the model and then explain the two phases.
4.1

Model Formulation

Considering the characteristics of target data, we think that a reasonable model
should meet the following three requirements:
1. Joint Modeling of Acoustic and Textual Features: When users wantto produce new comments, they seem to simultaneously take into accountthe content of musical audio signals and the content of comments provided
by other users. This suggests that well-balanced integration of them will
enable the model to yield reasonable comments.

Generating Comments Synchronized with Musical Audio Signals

Audio clips for training

New audio clip

Musical audio signal

Musical audio signal“Great performance” “She has a beautiful voice”

Existing users’ comments

Clip 1 Users’ comments given to temporal positions
Clip 2
Clip N
Joint probabilistic model ofaudio signals and comments

Comments③

Outlining stage Determine contents and positions“beautiful” and “this” arelikely to jointly occur①

Audio and comments

Assembling stage

Adapted uni-gram
Tri-gram“cool” is likely
to occur②④
Uni-gram

89

Generate sentences“This is a beautiful performance” “Cool”

Bi-gramGeneral language models

Learning phase

Commenting phaseFig. 2. Overview of MusicCommentator

2. Temporal Modeling of Acoustic and Textual Features: Because music is one of temporal medium, its temporal characteristics such as moodtransitions should be captured by using a temporal model. Similarly, we
focus on topic transitions in comments that are synchronized with music.
3. Cross-modal Feature Binding through Temporal Contexts: We canassume that a single latent state, which can be conceptually interpreted as amood or topic, is shared behind the observable features of audio signals andcomments at each frame.
To meet these requirements, we propose a joint probabilistic model of multimodal features by extending a standard hidden Markov model (HMM), as shown
(n)
(n)
(n)
in Fig. 3. Let K be the number of latent states and let z t = {zt,1 , · · · , zt,K } bea state representation at frame t in clip n, where zt,k = 1 and {zt,k = 0|k = k  }if the model stays at state k  (1 ≤ k  ≤ K). We deﬁne latent state sequences(n)(n)
Z (n) and Z as Z (n) = {z 1 , · · · , z Tn } and Z = {Z (1) , · · · , Z (N ) }.
Our HMM can, like standard HMMs, be characterized by a set θ of threekinds of parameters {π, A, φ}. π is a set of initial probabilities {π1 , · · · , πK },(·)
where πk ≡ p(z1,k = 1). A is a transition matrix {Ajk |1 ≤ j, k ≤ K}, where
(n)

(·)

(n)

(·)

Ajk ≡ p(zt,k = 1|zt−1,j = 1). φ is a set of parameters of output distributions
that calculate the likelihoods of observable features.
Acoustic and textual features at a frame are associated with the same state.
Let bk be a joint output distribution of state k. This calculates the likelihood(n)(n)
of ot , which is given by bk (ot ). This indicates how likely the four kinds(n)(n) (n) (n)of features {at , w t , dt , lt } jointly occur from state k. We assume that(n)bk (ot ) can be decomposed into the following four likelihoods:
(n)

(n)

(n)

(n)

(n)

bk (ot ) = ba,k (at ) bw,k (w t ) bd,k (dt ) bl,k (lt ),

(1)

90

K. Yoshii and M. Goto

Shared state sequence in audio clip n

zt−(n1)zt(n)Gaussian mixture model(GMM)

zt+(n1)Gaussian
GaussianMel-frequency cepstrum(n)coefficients (MFCCs) a

Multinomialt

Acoustic features(n)Comment density dt

Average length of comments l (n)Bag-of-words features wt(n)t

Textual features

Fig. 3. Overview of our ergodic hidden Markov model

where ba,k , bw,k , kd,k , and bl,k are designed as follows:
1. Output Distributions of Acoustic Features: ba,k is a Gaussian mixture
model (GMM) of state k as in typical HMMs for speech recognition. Let M
be the number of mixtures and let ga,k,m , μa,k,m , and Σ a,k,m be the weight,
mean, and covariance of the m-th Gaussian in the GMM of state k.
2. Output Distributions of Textual Features: bw,k is a multinomial distribution for bag-of-words features. Its parameters are given by pk = {pk,1 , · · · ,pk,V }. bd,k is a standard Gaussian representing the distribution of comment
densities. Let μd,k and Σd,k be the mean and variance of the Gaussian associated with state k. Similarly, bl,k is also a Gaussian for comment lengths,
and its mean and variance are given by μl,k and Σl,k .
Let φk be the set of parameters of output distributions of state k, given by
φk = {{ga,k,m , μa,k,m , Σ a,k,m |1 ≤ m ≤ M }, pk , μd,k , Σd,k , μl,k , Σl,k }. In total, aset of all parameters of output distributions φ is given by φ = {φ1 , · · · , φK }.
Our model is an ergodic HMM, which allows any state transition at any time,
because we cannot identify correct sequences of states in training data. In speech
recognition, manual transcriptions of speech signals (i.e., phoneme sequences)can be directly transformed into state sequences because each phoneme is deﬁned
as a combination of several states. Left-to-Right HMMs, where state transitions
are limited to match the transcriptions, are therefore commonly used in speech
recognition. In contrast, we use the HMM in an unsupervised fashion.
4.2

Learning Phase

This section explains how to estimate the unknown parameters θ = {π, A, φ}.
Let p(O|θ) be the likelihood of observable variables O. Instead of directly maximizing the incomplete likelihood p(O|θ), we try to maximize the expected complete likelihood of observable variables O and latent variables Z by using the
Expectation-Maximization (EM) algorithm [9]. The complete likelihood is
T
T
N
n
n



(n)
(n) (n)(n) (n)p(O, Z|θ) =p(z 1 |π)p(z t |z t−1 )p(ot |z t ),
(2)
n=1

t=2

t=1

Generating Comments Synchronized with Musical Audio Signals

91

(n)

z1,k
(n)
where p(z 1 |π) is given by Kπ
, which is an initial probability that clip
k=1 k
(n)
n starts at a state speciﬁed by z 1 . We then deﬁne the Q function as follows:p(Z|O, θold ) log p(O, Z|θ),Q(θ|θ old ) =(3)
Z
where θold is a set of the current parameters and p(Z|O, θold ) is a posterior
probability of latent variables Z. Q(θ|θold ) indicates the expected complete loglikelihood of all variables O and Z when we regard θ as a variable of the function.
Thus, the optimized parameters are obtained by maximizing Q(θ|θold ) and arethen set to θold next time. This is iterated until Q(θ|θ old ) converges.
In the E-step of the EM algorithm, the objective is to calculate the posterior
distribution p(Z|O, θold ). For convenience, we deﬁne some new symbols as:

(n)

(n)

(n)

(n)

(n)

(n)

γ(z t ) ≡ p(z t |O, θold ), ξ(z t−1 , z t ) ≡ p(z t−1 , z t |O, θ old ),

(4)

(n)
(n)
(n) (n)(n)γ(y t,k ) ≡ p(y t,k |O, θold ) = p(y t,k |z t )γ(z t ),

(5)

(n)

(n)

(n)

(n)

where γ(z t ) is a posterior distribution of latent variable z t . ξ(z t−1 , z t )(n)(n)
is a joint posterior distribution of adjacent latent variables z t−1 and z t . For
(n)
(n)
(n)
each t, γ(z t ) consists of K probabilities that sum up to unity. ξ(z t−1 , z t )is expressed as a K × K probability matrix whose elements sum up to unity.
(n)
(n)
(n)
(n)
Let γ(zt,k ) be the conditional probability of zt,k = 1 and let ξ(zt−1,j , zt,k )
(n)

(n)

be that of zt−1,j = zt,k = 1, given O and θold . These probabilities can be(n)eﬃciently calculated by using the forward-backward algorithm [11]. y t,k is a
(n)

(n)

vectorial variable, {yt,k,1 , · · · , yt,k,M }. This shows which Gaussian is responsible(n)for generating at(n)among M Gaussians in GMM ba,k , where yt,k,m = 1 and{yt,k,m = 0|m = m } when the m -th Gaussian is responsible. p(y t,k |z t ) isexpressed as a K × M probability matrix that reserves the responsibilities of(n)
KM Gaussians of GMMs {ba,1 , · · · , ba,K } for observation at .
In the M-step, we try to maximize the Q function Q(θ|θ old ). Substituting
Eqn. (2) for Eqn. (3), we get(n)Q(θ|θ old ) =

(n)

N 
K

n=1 k=1(n)γ(z1,k ) log πk +

Tn 
N 
K 
K


(n)

(n)

(n)

ξ(zt−1,j , zt,k ) log Ajk

n=1 t=2 j=1 k=1

+

Tn 
N 
K


(n)

(n)

γ(zt,k ) log p(ot |φk ), (6)n=1 t=1 k=1(n)where the last term can be decomposed into four terms as log p(ot |φk ) =
(n)
(n)
(n)
(n)
log ba,k (at )+log bw,k (w t )+log bd,k (dt )+log bl,k (lt ) We can thus independently update the parameters of four kinds of distributions (GMM ba,k , multinomial distribution bw,k , and two Gaussians bd,k and bl,k ) by using the Lagrange
multiplier method. We get the updating formula as follows:

92

K. Yoshii and M. GotoN(n)n=1 γ(z1,k )πk = N K
,
(n)
n=1
k=1 γ(z1,k )

N

N

Tn
(n)
(n)
t=2 ξ(zt−1,j , zt,k )n=1
Ajk = N K T
,
(n)
(n)
n
t=2 ξ(zt−1,j , zt,l )
n=1
l=1

Tn
(n)
t=1 γ(yt,k,m )
n=1
ga,k,m = N T M
,
(n)
n
t=1
n=1
m=1 γ(yt,k,m )NΣ a,k,m =n=1

Nμd,k =

N

μa,k,m =

Tn

4.3

Tn
(n) (n)t=1 γ(zt,k )ltn=1,
N Tn(n)t=1 γ(zt,k )
n=1

pk =

N

Σd,k =

N
μl,k =

N(n)(n)− μa,k,m )2
t=1 γ(yt,k,m )(at,N Tn
(n)
γ(y
)
t=1
n=1
t,k,m

Tn
(n) (n)t=1 γ(zt,k )dtn=1,
N Tn(n)t=1 γ(zt,k )
n=1

Σl,k =

Tn

n=1

N
n=1

Tn
(n)
(n)
t=1 γ(yt,k,m )atn=1,
N Tn(n)t=1 γ(yt,k,m )
n=1

Tn
(n)
(n)
t=1 γ(zt,k )w tn=1,
N Tn
(n)
γ(z
)
t=1
n=1
t,k

(n)
(n)
− μd,k )2
t=1 γ(zt,k )(dt,N Tn(n)t=1 γ(zt,k )
n=1

Tn

(n)
(n)
− μl,k )2
t=1 γ(zt,k )(lt.
N Tn(n)t=1 γ(zt,k )
n=1

(7)

Commenting Phase

This section explains how to provide comments suitable to a target audio clip.
Like the training data, the audio signal and provided comments are characterized
by a sequence of acoustic features, a = {a1 , · · · , aT  }, three sequences of textual
features, w  = {w1 , · · · , w T  }, d = {d1 , · · · , dT  }, and l = {l1 , · · · , lT  }, whereT  is the number of frames. This phase consists of an outlining stage and an
assembling stage. The latter estimates how many comments and what content
should be provided at each frame. The former concatenates a suitable numberof words in an appropriate order by taking language constraints into account.
Outlining Stage. We ﬁrst determine a most likely sequence of latent statesin the target, z  = {z 1 , · · · , z T  }, with the Viterbi algorithm [10]. When zt,kisˆ
1 at frame t (1 ≤ t ≤ T ), the most likely density there, dt , is given by themode (most likely observation) of the Gaussian bd,k , i.e., mean μd,k . From thedensity distribution over the entire clip, we can determine how many comments
is 1, we can get mostshould be provided in each frame. Similarly, when zt,klikely bag-of-words features (occurrence probabilities of signiﬁcant words) ŵt to
be pk .
We here cannot generate sentences that are appropriate as natural language,i.e., reasonable sequences of words, because bag-of-words features only outlines
the content of comments. Therefore, we should solve the following problems:
1. We do not have occurrence probabilities of non-signiﬁcant words such as conjunctions and auxiliary verbs, which are indispensable for natural language.
2. We do not have individual occurrence probabilities of inﬂectional words thathave the same base morpheme within them (see 2a in Section 3).

Generating Comments Synchronized with Musical Audio Signals

93

Significant words andtheir inflectional words Other words
Generaluni-gram
Duplicate multinomial probabilities
Do nothing
to those of corresponding words
Normalize updated probabilities
Adapteduni-gram

Normalize original probabilities

α1−αFig. 4. Adaptation of general uni-gram to multinomial distribution

3. We cannot determine an appropriate order of words because the current
model does not take into account sequential relations between words.
For example, suppose that two words “it” and “love” are highly likely to occur
and the comment length is likely to be three. We cannot synthesize a comment
like “It is loved” or “I love it” because the probabilities of “is” and “loved” arenot given and we therefore do not know which sentence is more appropriate.
Assembling Stage. To solve the three problems described above, we propose
a comment generation method based on adaptation of general language models(uni-, bi-, and tri-grams) that are learned from numerous comments of all clips
in the training data. Unlike what we do in the learning phase, we distinguishbetween diﬀerent words that have the same base morpheme (e.g., we distinguish“took” from “taken”’). The uni-gram can be used for solving the ﬁrst and second
problems, and the bi- and tri- grams contribute to solving the third one.
Fig. 4 shows a sketch of how the probabilities ŵt of signiﬁcant words at framet are incorporated into the general uni-gram, which includes all the words appearing in the training data. We duplicate the probability of each signiﬁcantword to those of its inﬂectional words that have diﬀerent surface expressions.
For example, the probabilities of words “took” and “taken” are set to be thesame as that of word “take.” Then, because the sum the probabilities of significant words and their inﬂectional ones is greater than 1, the probabilities are
scaled so that their sum is α, which is a control parameter given in advance. On
the other hand, the probabilities of other words containing non-signiﬁcant words
and their inﬂectional ones are scaled so that their sum is 1 − α. As a result, we
now get the adapted uni-gram (a set of occurrence probabilities of all words).
is 1 at frame t, we assume that a most likely comment (wordWhen zt,ksequence) ĉt and a most likely comment length (number of words) l̂t should bedetermined according to the following generative model:{ĉt , l̂t } = argmax p(c, l; θk ) = argmax p(c|l; θk )p(l; θk ),
c,l
c,l

(8)

94

K. Yoshii and M. Goto

Uni-gram
probability

Bi-gramprobability

SilE

Tri-gram
probability

1

…

…

All words

i −2

i −1

…

SilB

…

Endingsymbol…

Beginning
symbolilFig. 5. Probability calculation on word trellis

where p(l; θk ) is a likelihood that a comment generated from state k consistsof l words. Its value is calculated according to Gaussian bl,k and p(c|l; θk ) is aconditional probability that comment c is generated when its length is given byl. Note that for readability we hereafter omit the estimated parameter θk . To
get ĉ, we have only to calculate argmaxc p(c|l) for each length.
To estimate argmaxc p(c|l), we propose a method that can ﬁnd a most likely
path of words on a trellis including all words by using the Viterbi algorithm [10].
As shown in Fig. 5, each node corresponds to a speciﬁc word and the observation probabilities of words in each column are given by the adapted uni-gram.
Transition probabilities between nodes are determined as bi- and tri-grams. We
let SilB and SilE be special symbols (silent words) that indicate the beginningand ending of comments. The likelihood of comment c is given by l
p(c|l) = p(w1 |SilB)p(wi |wi−2 , wi−1 ) p(SilE|wl−1 , wl ),(9)i=2

where wi is the i-th word in comment c and w0 is SilB. p(wi |wi−2 , wi−1 ) is
an adapted trigram probability, which is calculated with linear interpolation of
the general tri- and bi- grams pt , pb and the adapted uni-grams pu as follows:p(wi |wi−2 , wi−1 ) ← βt pt (wi |wi−2 , wi−1 ) + βb pb (wi |wi−1 ) + βu pu (wi ), where βt ,βb , and βu are weighting factors of the tri-, bi-, and uni-grams.

5

Evaluation

We experimentally evaluated how accurately the system predicted commentsthat are freely provided on arbitrary temporal positions by users.
5.1

Conditions

The audio clips (tracks) we used were included in the video clips taken from themusic category of the video sharing service Nico Nico Douga. Speciﬁcally, we focus on music performances whose titles included “Ensoushitemita” (“We/I played

Generating Comments Synchronized with Musical Audio Signals(a) Content Accuracy(b) Positioning Accuracy
Hiitemita

10
8
6

Ensoushitemita

4
2
0%

(%)

F-value

F-value

(%)

95

80
75
70

Hiitemita
Ensoushitemita

65
25% 50% 75%
0% 25% 50%
Amount of users’ comments for generating system’s comments

75%

Fig. 6. Results of experimental comment generation

something, not limited to musical instruments, e.g., music box and wooden gong”),
or “Hiitemita” (“We/I played piano or stringed instruments, e.g., violin and guitar”). Some of them were performed by multiple people, e.g., sessions, bands, or
ensembles. There are many popular clips that follow these conventional naming
rules in the title. We collected the most popular 100 “Ensoushitemita”-category
clips that are shorter than 10 minutes according the the number of comments thatroughly reﬂects its popularity. Then, the ﬁrst 1100 comments, which were available
as many as possible in all the clips, were extracted. Note that the ﬁrst 1100 comments do not mean the 1100 comments taken from the beginning within each clip,
but mean the 1100 comments taken from the beginning of its submission to thevideo sharing service. As for the “Hiitemita” category having more comments, wewere able to extract 2400 comments from each of 100 clips. The control parameters
were set as V = 2082, 2278, K = 200, M = 8, α = 0.9, and βt = βb = βu = 1.0
by trial and error.
The experiments were conducted in the way of 4-fold cross fold validation.
First, all audio clips with provided comments were randomly divided into fourgroups. Three groups were used as a training set in the learning phase and theother group was used as a test set in the commenting phase. Switching the choiceof test set, we conducted four trials. 4-fold cross fold validation was furthermore
performed in each trial by dividing the provided comments of each test clip into
four groups. To estimate a most likely sequence of states in a given clip, thesystem used either no comments (i.e., only acoustic features) or one, two, or
three groups (i.e., 0%, 25%, 50%, or 75%) of comments on the clip. That is, wetested four settings. The remaining comments was used as ground truth.
To evaluate the results, we calculated the word-based F-value, which is given
R
by F = P2P+R, where P and R are the precision and recall rates. We focused oneach word of the system-generated comments. A word in a system’s comment
was considered reasonable if it appeared in users’ comments annotated in theneighborhood of the system’s comment. The error tolerance was set to 5 seconds.
P =#appropriate words#appropriate words, R=, (10)#words of system’s comments#words of users’ comments

96

5.2

K. Yoshii and M. Goto

Results

As shown in Fig. 6(a), the F-values could be improved even if only 25% of users’comments of a target clip was available for adding new comments. Although theF-values reached at most 10%, we think these results were promisingly reasonablebecause it is impossible to completely predict what comments are provided byusers at a word level even for humans. Note that when we evaluated only the
temporal positions and lengths of generated comments (i.e., allowed errors inword selection), the F-values were around 70%, as shown in Fig. 6(b). One may
say that it is enough to list most likely words as a rough suggestion. However, we
believe that sentences of natural language are much better in terms of readability
although they are often grammatically or semantically strange because n-grams
cannot all inter-word dependencies contained in a sentence.
The F-values were not furthermore improved when we increased the amountof users’ comments over 25% for adding new comments. This indicates that thecurrent system cannot deal with widely diverse comments. That is, the modelcannot create various comments that are essentially diﬀerent from each other intheir meanings once a speciﬁc state is determined for given acoustic and textual
features. Comments freely provided by humans without constraints are widely
diverse. A major reason that the F-values for the “Ensoushitemita” category
were lower than those for the “Hiitemita” category could be the wider diversityof the comments on former clips. The title “Hiitemita” means limited kinds of
instruments such as piano and guitar were used in the video.
We also found that the current system is not always useful because generalcomments like “it is very cool” and “great” tend to be generated. The F-valuesof frequently used positive words such as “cool” or “great” were around 40%.
This was closely related to the limitation of the statistical approach. If we canuse a huge amount of users’ comments for training the HMM (we actually usedover 100,000 comments), the probabilistic model tries to capture universal characteristics of the data. However, it is not appropriate to spoil the diversity ofhumans’ comments for our task. We should tackle this problem in the future.

6

Conclusion

We presented MusicCommentator that generates comments (short sentences ofnatural language) and provide them at appropriate temporal positions. The system is based on a multi-modal HMM that associates acoustic features with
textual ones through latent sequences of states. These sequences correspond to
temporal transitions of both musical moods and comment topics. To estimate
the parameters of the HMM, we used a likelihood maximization method so thatmany examples of how users have provided comments can be well explained withthe model. Given a new audio clip, the system concatenates suitable words inan appropriate order by using general language models.
The experimental results were promising but revealed that we are still far
from the ultimate goal of building a computer that can express the impressions ofvideo clips as natural language as humans do. Because commenting is one of theGenerating Comments Synchronized with Musical Audio Signals

97

most sophisticated cognitive functions of humans, it would be hard to preciselyemulate even if we use the state-of-the-art techniques of machine learning. We
think, however, that our study is an important ﬁrst challenge. We plan to improve
MusicCommentator by incorporating advanced methods of recognizing musicalcontent such as rhythm and melody. This kind of multi-aspect modeling could
help the system generate comments that are more appropriate and diverse.
Acknowledgement. This study was partially supported by CREST, JST.

References
1. Ames, M., Naaman, M.: Why We Tag: Motivations for Annotation in Mobile andOnline Media. In: ACM CHI, pp. 971–980 (2007)
2. Nakamura, S., Shimizu, M., Tanaka, K.: Can Social Annotation Support Users in
Evaluating the Trustworthiness of Video Clips? In: ACM WICOW (2008)
3. Hamasaki, M., Takeda, H., Nishimura, T.: Network Analysis of Massively Collaborative Creation of Multimedia Contents –Case Study of Hatsune Miku videos on
Nico Nico Douga–. In: uxTV, pp. 165–168 (2008)
4. Whitman, B., Rifkin, R.: Musical Query-by-Description as a Multiclass Learning
Problem. In: IEEE MMSP 2002, pp. 153–156 (2002)
5. Turnbull, D., Barrington, L., Torres, D., Lanckriet, G.: Semantic Annotation and
Retrieval of Music and Sound Eﬀects. IEEE Trans. on ASLP 16(2), 467–476 (2008)
6. Bertin-Mahieux, T., Eck, D., Maillet, F., Lamere, P.: Autotagger: A Model for
Predicting Social Tags from Acoustic Features on Large Music Databases. J. of
New Music Research (JNMR) 37(2), 115–135 (2008)
7. Tzanetakis, G., Cook, P.: Musical Genre Classiﬁcation of Audio Signals. IEEE
Trans. on SAP 10(5), 293–302 (2002)
8. Kudo, T., Yamamoto, T., Matsumoto, Y.: Applying Conditional Random Fields
to Japanese Morphological Analysis. In: EMNLP (2004)
9. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum Likelihood from Incomplete
Data via the EM Algorithm. J. R. Stat. Soc (B) 39(1), 1–38 (1977)
10. Forney, G.D.: The Viterbi Algorithm. IEEE 61(3), 268–278 (1973)
11. Baum, L.E., Petrie, T., Soules, G., Weiss, N.: A Maximization Technique Occurringin the Statistical Analysis of Probabilistic Functions of Markov Chains. Annals of
Mathematical Statistics 41(1), 164–171 (1970)