Computational Modeling of Emotional Content in Music

Computational Modeling of Emotional Content in Music Kristine Monteith (kristinemonteith@gmail.com) Tony Martinez (martinez@cs.byu.edu) Dan Ventura (ventura@cs.byu.edu) Department of Computer Science Brigham Young University Provo, UT 84602 Abstract associate emotions with various facial expressions (Tolbert, 2001). Scherer presents a framework for formally describ- ing the emotional effects of music and then outlines factors that contribute to these emotions, including structural, per- formance, listener, and contextual features (Scherer, 2001). In this paper, we focus on some of the structural aspects of music and the manner in which they contribute to emotions in music. We present a cognitive model of characteristics of mu- sic responsible for human perception of emotional content. Our model is both discriminative and generative; it is capable of detecting a variety of emotions in musical selections, and also of producing music targeted to a specific emotion. We present a system designed to model characteristics which contribute to the emotional content of music. It creates n-gram models, Hidden Markov Models, and entropy-based models from corpora of musical selections representing various emotions. These models can be used both to identify emo- tional content and generate pieces representative of a target emotion. According to survey results, generated selections were able to communicate a desired emotion as effectively as human-generated compositions. Keywords: Music cognition; learning; music composition. computational modeling; Introduction Music and emotion are intrinsically linked; music is able to express emotions that cannot adequately be expressed by words alone. Often, there is strong consensus among listen- ers as to what type of emotion is being expressed in a par- ticular piece (Gabrielsson & Lindstrom, 2001; Juslin, 2001). There is even some evidence to suggest that some perceptions of emotion in music may be innate. For example, selections sharing some acoustical properties of fear vocalizations, such as sudden onset, high pitch, and strong energy in the high fre- quency range, often provoke physiological defense responses (Ohman, 1988). Researchers have demonstrated similar low- level detection mechanisms for both pleasantness and novelty. (Scherer, 1984, 1988). There also appears to be some inborn preference for consonance over dissonance. In studies with infants, researchers found that their subjects looked signifi- cantly longer at the source of sound and were less likely to squirm and fret when presented with consonant as opposed to dissonant versions of a melody (Zentner & Kagan, 1996). There are a variety of theories as to what aspects of mu- sic are most responsible for eliciting emotional responses. Meyer theorizes that meaning in music comes from following or deviating from an expected structure (Meyer, 1956). Slo- boda emphasizes the importance of associations in the per- ception of emotion in music and gives particular emphasis to association with lyrics as a source for emotional meaning (Sloboda, 1985). Kivy argues for the importance of cultural factors in understanding emotion and music, proposing that the “emotive life” of a culture plays a major role in the emo- tions that members of that culture will detect in their music (Kivy, 1980). Tolbert proposes that children learn to associate emotion with music in much the same way that they learn to Related Work A number of researchers have addressed the task of modeling musical structure for the purposes of building a generative musical system. Conklin summarizes a number of statisti- cal models which can be used for music generation, includ- ing random walk, Hidden Markov Models, stochastic sam- pling, and pattern-based sampling (Conklin, 2003). These approaches can be seen in a number of different studies. For example, Hidden Markov Models have been used to harmo- nize melodies, considering melodic notes as observed events and a chord progression as a series of hidden states (Allan & Williams, 2005). Similarly, Markov chains have been used to harmonize given melody lines, focusing on harmonization in a given style in addition to finding highly probable chords (Chuan & Chew, 2007). Wiggins, Pearce, and Mullensiefen present a system de- signed to model factors such as pitch expectancy and melodic segmentation. They also demonstrate that their system can successfully generate music in a given style (Wiggins, Pearce, & Mullensiefen, 2009). Systems have also been developed to produce compositions with targeted emotional content. Delgado, Fajardo, and Molina-Solana use a rule-based sys- tem to generate compositions according to a specified mood (Delgado, Fajardo, & Molina-Solana, 2009). Rutherford and Wiggins analyze the features that contribute to the emotion of fear in a musical selection and present a system that allows for an input parameter that determines the level of “scariness” in the piece (Rutherford & Wiggins, 2003). Oliveira and Car- doso describe a wide array of features that contribute to emo- tional content in music and present a system that uses this