Automatic music mood detection

A system and methods use music features extracted from music to detect a music mood within a hierarchical mood detection framework. A two-dimensional mood model divides music into four moods which include contentment, depression, exuberance, and anxious/frantic. A mood detection algorithm uses a hierarchical mood detection framework to determine which of the four moods is associated with a music clip based on the extracted features. In a first tier of the hierarchical detection process, the algorithm determines one of two mood groups to which the music clip belongs. In a second tier of the hierarchical detection process, the algorithm then determines which mood from within the selected mood group is the appropriate, exact mood for the music clip. Benefits of the mood detection system include automatic detection of music mood which can be used as music metadata to manage music through music representation and classification.

TECHNICAL FIELD

The present disclosure relates to music classification, and more particularly, to detecting the mood of music from acoustic music data.

BACKGROUND

The recent significant increase in the amount of music data being stored on both personal computers and Internet computers has created a need for ways to represent and classify music. Music classification is an important tool that enables music consumers to manage an increasing amount of music in a variety of ways, such as locating and retrieving music, indexing music, recommending music to others, archiving music, and so on. Various types of metadata are often associated with music as a way to represent music. Although traditional information such as the name of the artist or the title of the work remains important, these metadata tags have limited applicability in many music-related queries. More recently, music management has been aided by the use of more semantic metadata, such as music similarity, style and mood. Thus, the use of metadata as a means of managing music has become increasingly focused on the content of the music itself.

Music similarity is one important metadata that is useful for representing and classifying music. Music genres, such as classical, pop, or jazz, are examples of music similarities that are often used to classify music. However, such genre metadata is rarely provided by the music creator, and music classification based on this type of information generally requires the manual entry of the information or the detection of the information from the waveform of the music.

Music mood information is another important metadata that can be useful in representing and classifying music. Music mood describes the inherent emotional meaning of a piece of music. Like music similarity metadata, music mood metadata is rarely provided by the music creator, and classification of music based on the music mood requires that the mood metadata be manually entered, or that it be detected from the waveform of the music. Music mood detection, however, remains a challenging task which has not yet been addressed with significant effort in the past.

Accordingly, there is a need for improvements in the art of music classification, which includes a need for improving the detectability of certain music metadata from music, such as music mood.

SUMMARY

A system and methods detect the mood of acoustic musical data based on a hierarchical framework. Music features are extracted from music and used to determine a music mood based on a two-dimensional mood model. The two-dimensional mood model suggests that mood comprises a stress factor which ranges from happy to anxious and an energy factor which ranges from calm to energetic. The mood model further divides music into four moods which include contentment, depression, exuberance, and anxious/frantic. A mood detection algorithm determines which of the four moods is associated with a music clip based on features extracted from the music clip and processed through a hierarchical detection framework/process. In a first tier of the hierarchical detection process, the algorithm determines one of two mood groups to which the music clip belongs. In a second tier of the hierarchical detection process, the algorithm determines which mood from within the selected mood group is the appropriate, exact mood for the music clip.

DETAILED DESCRIPTION

Overview

The following discussion is directed to a system and methods that use music features extracted from music to detect music mood within a hierarchical mood detection framework. Benefits of the mood detection system include automatic detection of music mood which can be used as music metadata to manage music through music representation and classification. The automatic mood detection reduces the need for manual determination and entry of music mood metadata that may otherwise be needed to represent and/or classify music based on its mood.

Exemplary Environment

FIG. 1illustrates an exemplary computing environment100suitable for detecting music mood. Although one specific computing configuration is shown inFIG. 1, various computers may be implemented in other computing configurations that are suitable for performing music mood detection.

The computing environment100includes a general-purpose computing system in the form of a computer102. The components of computer102may include, but are not limited to, one or more processors or processing units104, a system memory106, and a system bus108that couples various system components including the processor104to the system memory106.

The system bus108represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. An example of a system bus108would be a Peripheral Component Interconnects (PCI) bus, also known as a Mezzanine bus.

Computer102includes a variety of computer-readable media. Such media can be any available media that is accessible by computer102and includes both volatile and non-volatile media, removable and non-removable media. The system memory106includes computer readable media in the form of volatile memory, such as random access memory (RAM)110, and/or non-volatile memory, such as read only memory (ROM)112. A basic input/output system (BIOS)114, containing the basic routines that help to transfer information between elements within computer102, such as during start-up, is stored in ROM112. RAM110contains data and/or program modules that are immediately accessible to and/or presently operated on by the processing unit104.

Computer102may also include other removable/non-removable, volatile/non-volatile computer storage media. By way of example,FIG. 1illustrates a hard disk drive116for reading from and writing to a non-removable, non-volatile magnetic media (not shown), a magnetic disk drive118for reading from and writing to a removable, non-volatile magnetic disk120(e.g., a “floppy disk”), and an optical disk drive122for reading from and/or writing to a removable, non-volatile optical disk124such as a CD-ROM, DVD-ROM, or other optical media. The hard disk drive116, magnetic disk drive118, and optical disk drive122are each connected to the system bus108by one or more data media interfaces126. Alternatively, the hard disk drive116, magnetic disk drive118, and optical disk drive122may be connected to the system bus108by a SCSI interface (not shown).

Any number of program modules can be stored on the hard disk116, magnetic disk120, optical disk124, ROM112, and/or RAM110, including by way of example, an operating system126, one or more application programs128, other program modules130, and program data132. Each of such operating system126, one or more application programs128, other program modules130, and program data132(or some combination thereof) may include an embodiment of a caching scheme for user network access information.

A user can enter commands and information into computer system102via input devices such as a keyboard134and a pointing device136(e.g., a “mouse”). Other input devices138(not shown specifically) may include a microphone, joystick, game pad, satellite dish, serial port, scanner, and/or the like. These and other input devices are connected to the processing unit104via input/output interfaces140that are coupled to the system bus108, but may be connected by other interface and bus structures, such as a parallel port, game port, or a universal serial bus (USB).

A monitor142or other type of display device may also be connected to the system bus108via an interface, such as a video adapter144. In addition to the monitor142, other output peripheral devices may include components such as speakers (not shown) and a printer146which can be connected to computer102via the input/output interfaces140.

Computer102may operate in a networked environment using logical connections to one or more remote computers, such as a remote computing device148. By way of example, the remote computing device148can be a personal computer, portable computer, a server, a router, a network computer, a peer device or other common network node, and the like. The remote computing device148is illustrated as a portable computer that may include many or all of the elements and features described herein relative to computer system102.

Logical connections between computer102and the remote computer148are depicted as a local area network (LAN)150and a general wide area network (WAN)152. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet. When implemented in a LAN networking environment, the computer102is connected to a local network150via a network interface or adapter154. When implemented in a WAN networking environment, the computer102includes a modem156or other means for establishing communications over the wide network152. The modem156, which can be internal or external to computer102, can be connected to the system bus108via the input/output interfaces140or other appropriate mechanisms. It is to be appreciated that the illustrated network connections are exemplary and that other means of establishing communication link(s) between the computers102and148can be employed.

In a networked environment, such as that illustrated with computing environment100, program modules depicted relative to the computer102, or portions thereof, may be stored in a remote memory storage device. By way of example, remote application programs158reside on a memory device of remote computer148. For purposes of illustration, application programs and other executable program components, such as the operating system, are illustrated herein as discrete blocks, although it is recognized that such programs and components reside at various times in different storage components of the computer system102, and are executed by the data processor(s) of the computer.

Exemplary Embodiments

FIG. 2is a block diagram representation of an exemplary computer102illustrating exemplary components suitable for facilitating music mood detection. Computer102includes one or more music clips200formatted as any of variously formatted music files including, for example, MP3 (MPEG-1 Audio Layer3) files or WMA (Windows Media Audio) files. Computer102also includes a music mood detection algorithm202configured to extract music features204from a music clip200, and to classify the music clip according to a hierarchical mood detection framework/process given the extracted music features204. Accordingly, the music mood detection algorithm202generally includes a music feature extraction tool206and a hierarchical music mood detection process208. It is noted that these components (i.e., algorithm202, extraction tool206, hierarchical mood detection process208) are shown inFIG. 2by way of example only, and not by way of limitation. Their illustration in the manner shown inFIG. 2is intended to facilitate discussion of music mood detection on a computer102. Thus, it is to be understood that various configurations are possible regarding the functions performed by these components. For example, such components might be separate stand alone components or they might be combined as a single component on computer102.

In general, the music mood detection algorithm202extracts certain music features204from a music clip200using music feature extraction tool206. Mood Detection algorithm202then determines a music mood (e.g., Contentment, Depression, Exuberance, Anxious/Frantic,FIGS. 3 and 4) for the music clip200by processing the extracted music features204through the hierarchical mood detection process208. The algorithm202employs a two-dimensional mood model proposed by Thayer, R. E. (1989),The biopsychology of mood and arousal, Oxford University Press (hereinafter, “Thayer”). The two-dimensional model adopts the theory that mood is comprised of two factors: Stress (happy/anxious) and Energy (calm/energetic), and divides music mood into four clusters: Contentment, Depression, Exuberance and Anxious/Frantic as shown inFIG. 3.

InFIG. 3, Contentment refers to happy and calm music, such as Bach's “Jesus, Joy of Man's Desiring”; Depression refers to calm and anxious music, such as the opening of Stravinsky's “Firebird”; Exuberance refers to happy and energetic music such as Rossini's “William Tell Overture”; and Anxious/Frantic refers to anxious and energetic music, such as Berg's “Lulu”. Such definitions of the four mood clusters are explicit and discriminatable. In addition, the two-dimensional structure provides important cues for computational modeling. Therefore, the two-dimensional model is applied in the music mood detection algorithm202.

As mentioned above, the music feature extraction tool206extracts music features from a music clip200. Music mode, intensity, timbre and rhythm are important features associated with arousing different music moods. For example, major keys are consistently associated with positive emotions, whereas minor ones are associated with negative emotions. However, the music mode feature is very difficult to obtain from acoustic data. Therefore, only the remaining three features, intensity feature204(1), timbre feature204(2), and rhythm feature204(3) are extracted and used in the music mood detection algorithm202. In Thayer's two-dimensional mood model shown inFIG. 3, the intensity feature204(1) corresponds to “energy”, while both the timbre feature204(2) and the rhythm feature204(3) correspond to “stress”.

To begin the music mood detection process, a music clip200is first down-sampled into a uniform format, such as a 16 KHz, 16 bit, mono-channel sample. It is noted that this is only one example of a uniform format that is suitable, and that various other uniform formats may also be used. The music clip200is also divided into non-overlapping temporal frames, such as 32 microsecond-long frames. The 32 microsecond frame length is also only an example, and various other non-overlapping frame lengths may also be suitable. In each frame, an octave-scale filter bank is used to divide the frequency domain into several frequency sub-bands:

[0,ω02n),[ω02n,ω02n-1),…⁢[ω022,ω021](1)
where w0refers to the sampling rate and n is the number of sub-band filters. In a preferred implementation, 7 sub-bands are used.

In general, timbre features and intensity features are then extracted from each frame. The means and variances of the timbre features and intensity features of all the frames are calculated across the whole music clip200. This results in a timbre feature set and an intensity feature set. Rhythm features are also extracted directly from the music clip. In order to remove the relativity among these raw features, a Karhunen-Loeve transform is performed on each feature set. The Karhunen-Loeve transform is well-known to those skilled in the art and will therefore not be further described. After the Karhunen-Loeve transform, each of the resulting three feature vectors is mapped into an orthogonal space, and each resulting covariance matrix also becomes diagonal within the new feature space. This procedure helps to achieve a better classification performance with the Gaussian Mixture Model (GMM) classifier discussed below. Additional details regarding the extraction of the three features (intensity feature204(1), timbre feature204(2), and rhythm feature204(3)) are provided as follows.

As mentioned above, intensity features are extracted from each frame of a music clip200. In general, intensity is approximated by the root mean-square (RMS) of the signal's amplitude. The intensity of each sub-band in a frame is first determined. An intensity for each frame is then determined by summing the intensities of the sub-bands within each frame. Then all the frame intensities are averaged for the whole music clip200to determine the overall intensity feature204(1) of the music clip. Intensity is important for mood detection because its contrast among the music moods is usually significant, which helps to distinguish between moods. For example, intensity for the music moods of Contentment and Depression is usually small, but for the music moods of Exuberance and Anxious, it is usually big.

Timbre features are also extracted from each frame of a music clip200. Both spectral shape features and spectral contrast features are used to represent the timbre feature. The spectral shape features and spectral contrast features that represent the timbre feature are listed and defined in Table 1. Spectral shape features, which include centroid, bandwidth, roll off and spectral flux, are widely used to represent the characteristics of music signals. They are also important for mood detection. For example, the centroid for the music mood of Exuberance is usually higher than for the music mood of Depression because Exuberance is generally associated with a high pitch whereas Depression is associated with a low pitch. In addition, octave-based spectral contrast features are also used to represent relative spectral distributions due to their good properties in music genre recognition.

TABLE 1Definition of Timbre FeaturesThe Feature NameDefinitionSpectralCentroidMean of the short-time Fourier amplitudeShapespectrum.FeaturesBandwidthAmplitude weighted average of the differencesbetween the spectral components and the centroid.Roll off95thpercentile of the spectral distribution.Spectral2-Norm distance of the frame-to-frame spectralFluxamplitude difference.SpectralSub-bandAverage value in a small neighborhood aroundContrastPeakmaximum amplitude values of spectralFeaturescomponents in each sub-band.Sub-bandAverage value in a small neighborhood aroundValleyminimum amplitude values of spectralcomponents in each sub-band.Sub-bandAverage amplitude of all the spectralAveragecomponents in each sub-band.

As mentioned above, rhythm features are also extracted directly from the music clip. Rhythm is a global feature and is determined from the whole music clip200rather than from a combination of individual frames. Three aspects of rhythm are closely related with people's mood response. These are, rhythm strength, rhythm regularity, and rhythm tempo. For example, in the Exuberance mood cluster shown inFIG. 3, the rhythm is usually strong and steady with a fast tempo, while in the Depression mood cluster, music usually has a slow tempo and no distinct rhythm pattern. Therefore, these three features (i.e., rhythm strength, regularity, and tempo) are extracted accordingly. Because rhythm features are usually apparent through instruments whose sounds are prominent in the lower and higher sub-bands (e.g., bass instruments and snare drums, respectively), only the lowest sub-band and highest sub-band are used to extract rhythm features.

After an amplitude envelope is extracted from these sub-bands by using a half hamming (raise cosine) window, a Canny estimator is used to estimate a difference curve, which is used to represent the rhythm information. Use of a half hamming window and a Canny estimator are both well-known processes to those skilled in the art, and they will therefore not be further described. The peaks above a given threshold in the difference curve (rhythm curve) are detected as instrumental onsets. Then, three features are extracted as follows:Average Strength: the average strength of the instrumental onsets.Average Correlation Peak: the average of the maximum three peaks in the auto-correlation curve. The more regular the rhythm is, the higher the value is.Average Tempo: the maximum common divisor of the peaks of the auto-correlation curve.

As illustrated inFIG. 4, the music mood detection algorithm202performs mood detection through a hierarchical mood detection framework/process208based on the three extracted feature sets (i.e., intensity feature204(1), timbre feature204(2), and rhythm feature204(3)) and Thayer's two-dimensional mood model. The different extracted features (e.g., intensity feature204(1), timbre feature204(2), and rhythm feature204(3)) perform differently in discriminating between different music moods (e.g., Contentment, Depression, Exuberance, Anxious). Accordingly, as shown below, the hierarchical mood detection process208has the advantage of making it possible to use the most suitable features in different tasks. Moreover, like other hierarchical methods, it can make better use of sparse training data than its non-hierarchical counterparts.

In the hierarchical mood detection process208illustrated inFIG. 4, a Gaussian Mixture Model (GMM) is utilized to model each feature set. In constructing each GMM, the Expectation Maximization (EM) algorithm is used to estimate the parameters of the Gaussian component and mixture weights. The initialization is performed using the K-means algorithm. The EM and K-means algorithms are well-known to those skilled in the art and they will therefore not be further described.

The basic flow of the hierarchical mood detection process208is illustrated inFIG. 4, and can be generally described as follows. It is noted first, however, that the ensuing discussion presumes that the music features204have already been extracted from the music clip200by the music feature extraction tool206of the music mood detection algorithm202.

As shown inFIG. 4, for a given music clip200, the music clip200is first classified into Group1(Contentment and Depression) or Group2(Exuberance and Anxious) based on its intensity feature204(1) information. This is done because the energy of the Contentment and Depression moods is usually much less than the energy of the Exuberance and Anxious moods. Thus, discrimination between these 2 mood groups is very accurate on the basis of the intensity feature204(1) alone. To classify the music clip into different groups, simple Bayesian criteria are employed, as

P⁡(G1❘I)P⁡(G2❘I)⁢{≥1,Select⁢⁢G1⁢<1,Select⁢⁢G2(2)
where Girepresents different mood group, I represents the intensity feature set. Given the intensity feature, I, the probabilities of Group1and Group2are determined. Group1is selected if the probability of Group1is greater than or equal to the probability of Group2. Otherwise, Group2is selected.

Then classification is performed in each group (i.e., for whichever group is selected according to equation (2) above) based on timbre and rhythm features. In each group, the probability of being an exact mood given timber feature204(2) and rhythm feature204(3) can be calculated as
P(Mj|G1,T,R)=λ1×P(Mj|T)+(1−λ1)×P(Mj|R)j=1,2
P(Mj|G2,T,R)=λ2×P(Mj|T)+(1−λ2)×P(Mj|R)j=3,4  (3)
where Mjis the mood cluster, T and R represent timbre and rhythm features respectively, and λ1and λ2are two weighting factors to emphasize different features for the mood detection in different mood groups. After each probability is obtained, Bayesian criteria, similar to Equation 2, are again employed to classify the music clip200into an exact music mood cluster.

In Group1, the tempo of both mood clusters (i.e., Contentment and Depression moods) is usually slow and the rhythm pattern is generally not steady, while the timbre of Contentment is usually much brighter and more harmonic than that of Depression. Therefore, the timbre features are more important than the rhythm features in the classification in Group1. On the contrary, in Group2(i.e., Exuberance and Anxious moods), rhythm features are more important. Exuberance usually has a more distinguished and steady rhythm than Anxious, while their timbre features are similar, since the instruments of both mood clusters are mainly brass. On this basis, weighting factor λ1is usually set larger than 0.5, while weighting factor λ2is set at less than 0.5. Experiments indicate that the optimal average accuracy is archived when λ1=0.8, λ2=0.4. This confirms that the hierarchical mood detection process208provides the advantage of stressing different music features in different classification tasks to achieve improved results.

Exemplary Methods

Example methods for detecting the mood of acoustic musical data based on a hierarchical framework will now be described with primary reference to the flow diagram ofFIG. 5. The methods apply to the exemplary embodiments discussed above with respect toFIGS. 1–4. While one or more methods are disclosed by means of flow diagrams and text associated with the blocks of the flow diagrams, it is to be understood that the elements of the described methods do not necessarily have to be performed in the order in which they are presented, and that alternative orders may result in similar advantages. Furthermore, the methods are not exclusive and can be performed alone or in combination with one another. The elements of the described methods may be performed by any appropriate means including, for example, by hardware logic blocks on an ASIC or by the execution of processor-readable instructions defined on a processor-readable medium.

A “processor-readable medium,” as used herein, can be any means that can contain, store, communicate, propagate, or transport instructions for use or execution by a processor. A processor-readable medium can be, without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. More specific examples of a processor-readable medium include, among others, an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM) (magnetic), a read-only memory (ROM) (magnetic), an erasable programmable-read-only memory (EPROM or Flash memory), an optical fiber (optical), a rewritable compact disc (CD-RW) (optical), and a portable compact disc read-only memory (CDROM) (optical).

At block502of method500, three music features204are extracted from a music clip200. The extraction may be performed, for example, by a music feature extraction tool206of music mood detection algorithm202. The extracted features are an intensity feature204(1), a timbre feature204(2), and a rhythm feature204(3). The feature extraction includes converting (down-sampling) the music clip into a uniform format, such as a 16 KHz, 16 bit, mono-channel sample. The music clip200is also divided into non-overlapping temporal frames, such as 32 microsecond-long frames. The frequency domain of each frame is divided into several frequency sub-bands (e.g., 7 sub-bands) according to equation (1) shown above.

Extraction of the intensity feature includes calculating the RMS signal amplitude for each sub-band from each frame. The RMS signal amplitudes are summed across the sub-bands of each frame to determine a frame intensity for each frame. The intensity feature of the music clip200is then found by averaging the frame intensities.

Extraction of the timbre feature includes determining spectral shape features and spectral contrast features of each sub-band of each frame and then determining these features for each frame. The spectral shape features and spectral contrast features that represent the timbre feature are listed and defined above in Table 1. Calculations of the spectral shape and spectral contrast features are based on the definitions provided in Table 1. Such calculations are well-known to those skilled in the art and will therefore not be further described. Spectral shape features include a frequency centroid, bandwidth, roll off and spectral flux. Spectral contrast features include the sub-band peak, the sub-band valley, and the sub-band average of the spectral components of each sub-band.

Extraction of the rhythm feature is based on the whole music clip200rather than a combination of individual sub-bands and frames. Only the lowest sub-band and highest sub-band of the frames are used to extract rhythm features. An amplitude envelope is extracted from these sub-bands using a half hamming (raise cosine) window. A Canny estimator is then used to estimate a difference curve, which is used to represent the rhythm information. The half hamming window and Canny estimator are both well-known processes to those skilled in the art, and they will therefore not be further described. The peaks above a given threshold in the difference curve (rhythm curve) are detected as instrumental onsets. Then, an average rhythm strength feature is determined as the average strength of the instrument onsets, an average correlation peak (representing rhythm regularity) is determined as the average of the maximum three peaks in the auto-correlation curve (obtained from difference curve), and the average rhythm tempo is determined based on the maximum common divisor of the peaks of the auto-correlation curve (obtained from difference curve).

At block504of method500, the music clip200is classified into a mood group based on the extracted intensity feature204(1). The classification is an initial classification performed as a first stage of a hierarchical music mood detection process208. The initial classification is done in accordance with equation (2) shown above. The mood group into which the music clip200is initially classified, is one of two mood groups. Of the two mood groups, one is a contentment-depression mood group, and the other is an exuberance-anxious mood group. The initial classification into the mood group includes determining the probability of a first mood group based on the intensity feature. The probability of a second mood group is also determined based on the intensity feature. If the probability of the first mood group is greater than or equal to the probability of the second mood group, then the first mood group is selected as the mood group into which the music clip200is classified. Otherwise, the second mood group is selected. Thus, the initial classification classifies the music clip200into either the contentment-depression mood group or the exuberance-anxious mood group.

At block506of method500, the music clip is classified into an exact music mood from within the selected mood group from the initial classification. Therefore, if the music clip has been classified into the contentment-depression mood group, it will now be further classified into an exact mood of either contentment or depression. If the music clip has been classified into the exuberance-anxious mood group, it will now be further classified into an exact mood of either exuberance or anxious. Classifying the music clip into an exact mood is done in accordance with equation (3) above. Classifying the music clip therefore includes determining the probability of a first mood based on the timbre and rhythm features in accordance with equation (3) shown above. The probability of a second mood is also determined based on the timbre and rhythm features. The first mood and the second mood are each a particular mood within the mood group into which the music clip was initially classified (e.g., contentment or depression from the contentment-depression mood group, or exuberance or anxious from the exuberance-anxious mood group). If the probability of the first mood is greater than or equal to the probability of the second mood, then the first mood is selected as the exact mood into which the music clip200is classified. Otherwise, the second mood is selected as the exact mood.

CONCLUSION