Method and apparatus for attaching metadata

A method for attaching metadata includes steps of specifying at least two positions in head portions of bars of music data of music, and segmenting into segments a duration extending between the specified two positions and attaching time-series metadata to the segments.

CROSS REFERENCES TO RELATED APPLICATIONS

The present invention contains subject matter related to Japanese Patent Application JP 2006-073185 filed in the Japanese Patent Office on Mar. 16, 2006, the entire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a method and apparatus for attaching time series music metadata.

2. Description of the Related Art

Audio reproducing apparatuses employing a large-capacity storage medium such as a semiconductor memory or a hard disk are currently in widespread use. More and more users are now storing and enjoying a vast amount of music data stored on not only an audio reproducing apparatus but also on a storage unit in an information device such as a personal computer or a cellular phone. Albums may be managed on removable recording media such as a compact disk (CD) or tape on a per medium basis, while a vast amount of music data for several hundreds to tens of thousands of songs can be handled on a single device.

Random accessing to a large amount of songs is possible without replacing a medium, and users can produce a playlist of songs across albums. Metadata attaching service system currently draws attention to efficiently search for a song among numerous songs and to group songs. The metadata attaching service system acquires music metadata, such as song title, artist name, album name, etc. from a database and then attaches the music metadata to music data. The metadata attaching service system acquires not only song titles, and artist names but also genre of each song and to some degree, tempo information.

The metadata attaching service system is basically developed to search for songs and to group songs, and actually used for searching and grouping purposes. The manner of listening to a song reproduced as a result of search remains unchanged from the known manner.

Tools are available for professional or semi-professional music creators who may compose and arrange songs. Such tools manage the tempo and beat position of a song in time axis, and edit music data with reference to the position. Similarly, some of available disk jockey software programs store position information of beats to superimpose a plurality of songs in synchronization with the beat.

Information regarding such beats is based on the premise that tempo remains unchanged throughout one song. In commercially available recorded content, tempo typically fluctuates in several localized portions of each song, and it is usually difficult to keep track of accurate beat positions.

If an apparatus automatically recognizes timing of bars and beats of music from live music waveforms, metadata of the beat position is acquired from a vast amount of music content recorded on an existing CD, and possibility is opened to a new entertainment.

Attempts have been made to automatically extract temp or beat.

For example, Japanese Unexamined Patent Application Publication No. 2002-116754 discloses one extraction method. In accordance with the disclosed method, autocorrelation of a music waveform signal is calculated as a time-series signal, beat structure of music is analyzed based on the calculated autocorrelation, and the tempo of the music is then extracted from the resulting beat structure.

In accordance with Japanese Patent No. 3066528, sound pressure data of each of a plurality of frequency bands is produced from music data, a frequency band that predominantly provides rhythm is identified from the plurality of frequency bands, and a rhythm component is then estimated from a variation period in the pressure data of the identified frequency band.

SUMMARY OF THE INVENTION

Techniques for automatically extracting rhythm, beat, and tempo are divided into two types. In one type, a music signal is analyzed in time domain as disclosed in Japanese Unexamined Patent Application Publication No. 2002-116754, and in the other type, a music signal is analyzed in frequency domain as disclosed in Japanese Patent No. 3066528.

If the music signal is analyzed in the time domain, the beat and the time-series waveform do not match each other accurately, and high accuracy is not achieved in extraction process. If the music signal is analyzed in the frequency domain, extraction accuracy is relatively easily improved. But data resulting from the frequency analysis contains many beats other than beats of a particular musical note. It is extremely difficult to identify the beat of each musical note from all beats. Since musical tempo fluctuates greatly, extracting the beat of a particular musical note keeping track of the musical tempo fluctuation is extremely difficult.

It is thus desirable to provide a metadata attaching method and a metadata attaching apparatus for easily and accurately attaching time-series metadata, such as beat position and bar, to music data.

In accordance with one embodiment of the present invention, a method for attaching metadata, includes steps of specifying at least two positions in head portions of bars of music data of music, and segmenting into segments a duration extending between the specified two positions and attaching time-series metadata to the segments.

At least one of the two positions in the head portion of the bar may be manually specified by a listener who is listening to the music.

The time-series metadata may be calculated by dividing the duration extending between the two specified positions by one of the length of the bar and the number of bars.

In accordance with embodiments of the present invention, the beat position in music rhythm or the head position of each bar are accurately extracted from music data.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

In accordance with embodiments of the present invention, beat positions are easily and accurately extracted from an audio signal (an acoustic signal). At least two delimiters (heads of bars) are specified to music data of the audio signal, and the beat positions (including the head positions of the bars) are extracted by segmenting a duration between the two specified delimiters into equal intervals. At least one delimiter specified is manually input by a listener who is actually listening to the music.

FIG. 1illustrates a system in accordance with one embodiment of the present invention. As shown inFIG. 1, an audio signal supplier1supplies an audio signal including music data to a beat extracting apparatus2. An automatic beat extractor3in the beat extracting apparatus2extracts beats. The beat automatically extracted by the automatic beat extractor3is supplied to a manual authoring unit4where a human operator manually adds time-series metadata5to the beat. The manual authoring unit4may directly perform a beat extraction process on the music data from the audio signal supplier1and then add time-series metadata5.

The beat position is accurately extracted from the music data to attach accurate time-series metadata5onto the music data. To this end, frequency analysis or sound pressure analysis is performed by signal processing the audio signal of the music data in the known arts. If the beats of the music are clear and the beat intervals are constant, an accurate beat position can be extracted in the automatic extraction process with a high probability, and music data of the beat position is thus generated.

However, if a song played by humans is recorded, the tempo of the song fluctuates as shown inFIG. 2. The beat intervals are not constant.FIG. 2is a plot of an instantaneous beat per minute (BPM) of a song played live and recorded with respect to time (beat). Depending on songs, mean BPM gradually increases or decreases over a range of several bars rather than varying in a localized fluctuation.

In such songs, the mean BPM is not so significant with regard to representing the tempo of the song. Even if the beat positions are equally calculated using BPM, the calculated beat positions can be largely different from the actual beat positions.

If signal processing extracts accurately the beat position and fluctuations are overcome to some extent, it is still difficult to accurately extract the head portion of each bar, and the type and start position of melody.

A human can manually accurately attach time-series metadata to the music data. However, several hundred to thousand beat positions need to be attached on a per song basis. If several hundreds or thousands of songs are handled, such manual attaching operations become a vast amount of work and are not practicable.

In accordance with the system of one embodiment of the present invention, with the manual authoring unit4, the user manually attaches data having a high degree of abstraction recognizable by humans only, based on the beat automatically extracted through the signal processing of the automatic beat extractor3ofFIG. 1. The automatic beat extraction process performed by the automatic beat extractor3cannot theoretically achieve 100% accuracy level. Even if the automatic beat extractor3erratically detects a beat, the manual authoring unit4can easily correct the erratic detection. The automatic beat extractor3ofFIG. 1is not essential. The user can perform the beat attaching process to the music data from the audio signal supplier1using the manual authoring unit4.

FIG. 3illustrates a graphical operation input screen, namely a graphical user interface (GUI) to perform the process of the manual authoring unit4in the beat extraction system. As shown inFIG. 3, a waveform of the music data of the audio signal to be processed is displayed on a waveform display section301. Also displayed on the waveform display section301is beat information to be discussed later. Two waveform display sections301illustrated inFIG. 3are employed for stereo left and right channels, respectively. Also employed is a display section302for displaying the music data of the audio signal in a wide range. The display section302displays instantaneous BPM with respect to time axis (beat).

FIG. 4illustrates a beat display mode displayed on the waveform display section301.

The waveform display section301displays audio waveform data of a song. For example, the waveform display section301includes on an upper half thereof a beat editing area304for use in finalizing or editing beat information based on the pre-extracted beat. The waveform display section301includes on a lower half thereof a beat extracting area303where lines (vertical bars) indicating beat positions appears overlapping on the audio waveform, as beat information automatically extracted by the automatic beat extractor3. The type of beat information corresponds to the function of the automatic beat extractor3.

FIG. 5illustrates only the beat information displayed on the beat extracting area303and the beat editing area304. As a result of the automatic beat extraction performed through the signal processing, the beat information displayed on the beat extracting area303ofFIG. 5includes first type beats Ba that are determined as being synchronized with each original beat, second type beats Bb that are not automatically extracted but automatically interpolated in synchronization with the original beat, and third type beats Bc that are automatically extracted but not the original beat (so-called up beat). In the discussion that follows, the first type beat Ba and the second type beat Bb are referred to as an in beat, and the third type beat Bc is referred to an out beat. These types of beats Ba-Bc are preferably shown in different colors for easy recognition. For example, the first type color Ba is shown in a light blue line, the second type color Bb is shown in a pink line, and the third type color Bc is shown in an orange line. The automatic beat extractor3will be described in detail later.

Based on the beats extracted and displayed on the beat extracting area303, more accurate beat information is displayed on the beat editing area304. The beat information displayed on the beat editing area304ofFIG. 5includes a head portion Bd of each bar and a normal beat Be. To determine more accurately the beat position and the position of each bar, the following problems need to be solved:

(1) The results of the automatic beat extraction process are not always 100% accurate;

(2) It is extremely difficult for the head portion of each bar and the beat to be recognized through the automatic beat extraction process, and a 100% correct answer cannot be reached.

(3) It is difficult for an apparatus to recognize an erratic extraction as an error, and a human operator needs to actually listen to the song to manually correct the error.

(4) If all process is manually performed, several hundred to several thousand beats need to be manually attached on a per song basis. Such operations are a great deal of work.

(5) The position of a manually input beat is likely to be less accurate than a position extracted through the signal processing.

(6) The beat and tempo of the song itself may greatly fluctuate.

In view of the above problems, embodiments of the present invention are developed with the following guidelines adopted:

(a) The position of the head portion of the bar can be recognized by humans only and is thus not left to automatic extraction (The music perception ability of humans is excellent).

(b) The beat position extracted through the automatic beat extraction process is used if the beat position is not erratic.

(c) If a beat is erratically extracted, the position of the erratically extracted beat is set to be easily identifiable.

(d) If a beat is erratically extracted, manual correction of the erratic beat is set to be easily performed.

The solution in accordance with the guidelines (a)-(d) are specifically described below.

The guideline (a): The position of the head portion of the bar can be recognized by humans only and is thus not left to automatic extraction is followed as below.

Each song is composed of bars, and each bar is composed of beats. In the duple time, each bar is composed of two beats, and in the quadruple time, each bar is composed of four beats.

When actually listening to the music data to determine the head portion of the bar, a human operator specifies two points (Z1and Z2) that are obviously identified as the head portion of each of the bars as shown inFIG. 6, and inputs the number of bars, n, within a duration between the two points. The beats within the duration and the bars are automatically determined (interpolated). By listening to the music data being reproduced, the operator taps a tap button311on the GUI screen ofFIG. 3(or clicks on a mouse) at timing the operator recognizes the head portion of each bar. The head portion of the bars is thus specified. At least two points need to be specified. Since the start position of a song is typically the head portion of a bar, the start position of the song is thus substituted for one of the two points. In such a case, only one point needs to be manually specified while listening to the song.

When the two points (Z1and Z2) as the head portions of the bars are specified, and the number of bars n is input, the length of bar LBar is represented by LBar=(Z2−Z1)/n, and the length of each beat (beat interval or beat duration) LB is presented by Lb=(LBar/number of beats). As shown inFIG. 7, for example, by inputting the number of bars n between the two specified points Z1and Z2, n=142, the bar length LBar and beat interval Lb are determined.

The bar length is determined from the two specified points Z1and Z2with the number of bars known. However, counting the number of bars is troublesome. Conversely, if the bar length LBar is known, the number of bars may be determined. A method of determining each bar and each beat position from the bar length LBar and the specified points Z1and Z2is described below. It should be noted that the specified points Z1and Z2represent the head portions of the bars and that the number of bars falling within the duration between the specified points Z1and Z2is an integer value. When the number of bars is determined from the bar length, an approximate length of one bar is specified, and a correct bar length closest to the number of bars n being an integer is thus determined. As shown inFIG. 8, the specified points Z1and Z2are considered to be fixed ends of a vibration.

The points Z1and Z2are specified as the head portions of the bars, and the number of bars n falling between the two points Z1and Z2is an integer. An approximate bar length is specified so that the number of bars automatically becomes an integer. Let La represent an approximate bar length and LBar represent a bar length that allows n bars just fit between the two points Z1and Z2. The bar length LBar is determined from the following equation:
nf=((Z2−Z1)/La)+0.5
n=INT(nf)
LBar=(Z2−Z1)/n
where 0.5 is added on the left side of the equation determining nf to round off the equation determining n. INT(x) in the equation determining n is a function to round off a fractional part of x (to determine an integer).

Bars and beats are timed between the specified points Z1and Z2if the number of bars or the approximate length of the bar is known. Neither the number of bars nor the approximate bar length is known, a method discussed below is effective.

In the preceding method, the automatically extracted beats include in beats and out beats. The in beats are song's original beats extracted in accordance with an automatic extraction algorithm. Whether the in beat is the head portion of the bar is not known and the number of beats is not known, but the number of in beats is half or twice the actual number of beats. Taking advantage of this feature, the head portion of the bar is set based on the in beat, and the head portion of each tentative bar is regularly set for each time (such as four-four time, three-four time, etc.) specified by the user. The head portions of the tentative bars are thus arranged within the duration. The bar length LBar is easily determined by setting the number of tentative bars between the specified points Z1and Z2as being n or by calculating a difference between the head portions of the bars.

The automatically extracted beats may be half or twice the original time. In that case, the automatically extracted beats may be doubled or halved. By listening to the reproduced music data, the human operator can immediately recognize whether the extracted beats are half or twice the original time. In accordance with the embodiment of the present invention, an interface is provided to perform immediately recalculation by halving or doubling the number of bars n between the specified points Z1and Z2. Buttons312and313ofFIG. 3are used to double and halve the number of bars n, respectively. When the number of bars n is determined, the recalculation of the beat positions is easily performed. An accurate number of bars and an accurate bar length are thus easily determined with a few operation steps.

In accordance with one embodiment of the present invention, an interface to allow the user to manually increment or decrement the number of bars n by +1 or −1 (with buttons314and315as shown inFIG. 3) respectively is provided. If the time length between the specified points Z1and Z2is long, an error involved increases. The number of bars n determined from the length La can be larger or smaller than the actual number by 1 or so. If the beat rises when the user listens to the reproduced music data, the number of bars n may be incremented by 1 or decremented by 1, and the bar length LBar will be automatically recalculated to reposition the beat. This operation is performed based on the premise that the number of bars n between the specified points Z1and Z2is an integer value.

The guideline (b): The beat position extracted through the automatic beat extraction process is used if the beat position is not erratic. This guideline is followed as described below.

The head positions, the number, and the lengths of the bars are determined in order to interpolate the bars and beats between the specified points Z1and Z2. If the bars and the beats are interpolated equally, the actual beat position and the calculated beat position become shifted from each other with respect to a song that may suffer from beat interval fluctuation shown inFIG. 2.

In accordance with one embodiment of the present invention, that inconvenience is overcome by accounting for the beat information automatically extracted through the signal processing. More specifically, the beat position information obtained through the automatic beat extraction process of the automatic beat extractor3ofFIG. 1accounts for the beat fluctuation, and is thus used intact as much as is possible. If there is a possibility that the automatically extracted beat is in error, the beat position information at that position is not used but the calculated position information is used. With reference toFIGS. 9-1and9-2, that process is described below.

FIGS. 9-1and9-2are flowcharts illustrating a process of attaching the position information of bar and beat successively from the point Z1specified at the head portion of the bar by the human operator toward the specified point Z2.

In step S401ofFIG. 9-1, the bar length LBar is determined. As previously described, the bar length LBar is determined from the following equation using the positions of the specified points Z1and Z2and the number of bars n:
LBar=(Z2−Z1)/n
where the number of bars n is determined from the above-described approximate length La.

In step S402, the tentative beat length Lb is determined from the following equation, i.e., by dividing the bar length LBar determined in step S401by the time:
Lb=LBar/time

In step S403, a variable Pa indicating the position of the beat timed last during this process (the position of the beat recently attached) is initialized by substituting the specified point Z1as an initial value for the variable Pa (Pa=Z1). In step S404, a variable PBar indicating the position of the head portion of the bar timed last (the position of the head portion of the bar recently attached) is initialized by substituting the position of the specified point Z1for the variable PBar (PBar=Z1).

In step S405, the value of Pa+Lb is substituted for a variable Pc indicating the position of a next beat candidate (Pc=Pa+Pa). A position advancing from the last (latest) beat position Pa by the tentative beat length Lb is set to be a position Pc as a next beat candidate. Whether an automatically extracted beat is present within a range of a threshold value Ti (within a window width) centered on the candidate position Pc. It is then determined in step S406whether the candidate position Pc is smaller than the specified point Z2(Pc<Z2). If the determination in step S406is yes, in other words, if the candidate position Pc is ahead of the specified point Z2in time, processing proceeds to step S407. If the determination in step S406is no, processing ends.

It is determined in step S407whether an automatically extracted in beat is present within the range of the threshold value Ti centered on the candidate position Pc. If the determination in step S407is yes, processing proceeds to step S408. In step S408, the beat information is attached at the position Pi of the in beat. In step S409, Pa indicating the latest position where the beat has been timed is updated for later process by substituting Pi for the variable Pa. Processing proceeds to step S415.

If it is determined in step S407that no automatically extracted in beat is found, processing proceeds to step S410. If it is determined in step S410whether an automatically extracted out beat is present within a range of a threshold value To centered on the candidate position Pc (namely, within a window width, Ti>To in this embodiment). If the determination in step S410is yes (with an out beat present) the beat information is attached at the position Po of the output beat in step S411. In step S412, the variable Pa indicating the latest position where the beat has been timed is updated for later process by substituting Po (Pa=Po). Processing proceeds to step S415.

If the determination in step S410is no (with no automatically extracted out beat), processing proceeds to step S413. The beat information is attached at the position of the position candidate Pc. In step S414, the variable Pa indicating the latest position where the beat has been timed is updated for later process by substituting Pc for Pa (Pa=Pc). Processing proceeds to step S415.

It is determined in step S415whether the finally timed beat (the beat at the position of the variable Pa indicating the latest position) is the head portion of a bar. If the determination in step S415is yes, processing proceeds to step S416. If the determination in step S415is no, processing returns to step S405.

In step S416, the length of the nearest bar is determined by calculating Pa−PBar based on the beat position Pa timed last and the position PBar of the head portion of the bar timed last. The beat length candidate Lb is then determined from the determined bar length and the time. More specifically,
Lb=(Pa−PBar)/time

In step S417, the variable PBar indicating the latest position of the head portion of the bar is updated by substituting Pa for PBar (PBar=Pa). Processing returns to step S405.

Steps S405through S417are repeated until the determination in step S406becomes no, namely, until the candidate position Pc reaches the specified point Z2.

FIG. 10illustrates how the head portions of each beat and each bar are timed, thereby illustrating the beat extracting area303displaying the automatically extracted beats, and the beat editing area304where the head portions of the beats and the bars are timed in response to the extracted beats. The waveform of the music data is not shown inFIGS. 5 and 10. The first type color Ba discussed with reference toFIG. 5corresponds to the beat at the position Pi ofFIG. 10as the automatically extracted in beat. The second type color Bb corresponds to the beat at the position Po ofFIG. 10as the automatically extracted out beat.

With reference toFIG. 10, the specified point Z1is the initial value of the last (latest) beat position Pa and the bar head portion position PBar as described in steps S403and S404ofFIG. 9-1. The sum of the last (latest) beat position Pa (=Z1) and the tentative beat length Lb are determined as the position Pc of a next beat candidate341. It is then determined whether an automatically extracted beat is present within the range of the threshold value Ti (within the window width) centered on the candidate position Pc.

With reference toFIG. 10, an in beat342automatically extracted from within the range of the threshold range Ti centered on the position Pc of the beat candidate341is present in the beat extracting area303. The determination in step S407ofFIG. 9-2is yes, and processing proceeds to step S408. A beat343is thus timed in the beat editing area304at the position Pi of the extracted beat342. In step S409, the variable Pa indicating the last (latest) beat position is updated by substituting the position Pi of the beat343for the variable Pa (Pa=Pi).

A position advancing by the tentative beat length Lb from the updated Pa is set to be a position Pc of a next beat candidate344. When an automatically extracted beat is examined from within the rate of the threshold value Ti centered on the position Pc, an output beat345and an in beat346are present in the beat extracting area303ofFIG. 10. As understood from the flowchart ofFIG. 9-1, if the determination in step S407as to whether an in beat is present is yes, processing proceeds to step S408. A beat347is timed in the beat editing area304at the position Pi of the in beat346within the range of the threshold value Ti. The variable Pa is updated with the position Pi (Pa=Pi).

An automatically extracted beat is examined from within the range of the threshold Ti centered on the beat candidate348at a position advancing by the tentative beat length Lb from the position Pa of the last (latest) timed beat347. An out beat349is present but no in beat is present in the beat extracting area303ofFIG. 10. For example, a next in beat250is out of the range of the threshold value Ti. The determination in step S407ofFIG. 9-2is no, and processing proceeds to step S410. It is determined in step S410whether an automatically extracted output beat is present within the range of the threshold value To centered on the position Pc of the beat candidate348(Ti>To in this embodiment). Since the output beat349in the beat extracting area303ofFIG. 10falls within the range of the threshold value To, the determination in step S410is yes. Processing proceeds to step S411. A beat351is timed in the beat editing area304at the position Po of the output beat349. In step S412, the variable P1is updated by substituting the position Po for the variable Pa (Pa=Po).

No in beat is present within the range of the threshold value Ti centered on the position Pc of a beat candidate352at a position advancing by the tentative beat length Lb from the position Pa of the beat351. No out beat is present either within the range of the threshold value To centered on the position Pc. The determination in step S407of FIG.9-2is no, the determination in step S410is no, and processing proceeds to step S413. A beat353for interpolation is timed at the position Pc of the beat candidate352.

The automatically extracted beat present within the range of a predetermined threshold value with respect to the candidate position Pc of a next beat is used as is (drawn to the position of the automatically extracted beat). If there is no automatically extracted beat within the range of the predetermined threshold value, an interpolating beat is timed at the candidate position Pc of the next beat.

The guideline (c): If a beat is erratically extracted, the position of the erratically extracted beat is set to be easily identifiable is followed as described below.

In the beat editing area304ofFIG. 10, a beat may be timed between the specified points Z1and Z2. The beats343,347,351,353, etc. may be displayed in different colors in the beat editing area304depending on whether the automatically extracted beat displayed in the beat extracting area303is found within the window width, namely within the range of the threshold value Ti or To centered on the beat candidate position Pc.

In accordance with one embodiment of the present invention, the beats343,347, etc drawn to the in beat position Pi of the automatically extracted beats are displayed in light blue (or blue), the beat351, etc. drawn to the output beat position Po of the automatically extracted beat is displayed in yellow (or orange), and the beat353, etc. interpolated through calculation because of no automatically extracted beat found within the predetermined threshold range is displayed in red (or pink). By analogy with traffic lights, the beats are displayed in light blue (blue), orange, and red in the high to low reliability order.

In this way, if the automatically extracted beat position and the bar position and the beat attaching position within the duration between the specified points Z1and Z2fall within the range of the threshold value Ti in the vicinity of the candidate position, the beat is displayed in light blue (blue).

If an erratic extraction takes place during the automatic extraction process, the extracted beat position and the beat attaching position within the duration between the specified points Z1and Z2deviates from each other. In most of the cases where the beat is not correctly extracted, the tempo may greatly change over several to tens of bars in a song, or an accurate in beat cannot be picked up in a song that is not distinct in beating. In such a case, the number of beats falling within the duration between the specified points Z1and Z2cannot be divided by the number of the bars. Since the specified points Z1and Z2are the head portions of the bars, the number of beats falling within the duration must be an integer multiple of the number bars. However, if an erratic extraction takes place, the number of beats falling within the duration fails to match the actual number of beats. Any beat fails to fall within the range of the threshold value Ti or To ofFIG. 10. This state continues until a beat shifted by one beat falls within the threshold. As a result, the red beats are consecutively displayed in the beat editing area304. Beat shifting can thus be easily found.

An error, an accumulated error and a phase shift of each beat in the instantaneous BPM are visualized in graph. The reliability of each beat automatically extracted through the signal processing is thus predicted.FIG. 11Aillustrates in the display section302a sound performed live or recorded where the beats fluctuate greatly.FIG. 11Billustrates in the display section302a sound reproduced on a computer. As shown inFIGS. 11A and 11B, a curve Qa represents the error in the instantaneous BPM of beats, a curve Qb represents an accumulated error in the instantaneous BPM of beats, and a curve Qc represents a phase shift.

If the automatic extraction works with the BPM of the song stabilized as shown inFIG. 11B, the three curves Qa, Qb, and Qc generally overlap. If the beats greatly fluctuate as shown inFIG. 11A, the curves Qa, Qb, and Qc shift from each other. In this way, the reliability of the automatically extracted beats is recognized at a glance.

The guideline (d): If a beat is erratically extracted, manual correction of the erratic beat is set to be easily performed, is followed as described below.

As previously described, the cause for an erratic extraction is that the tempo may greatly change over several to tens of bars in a song, or that an accurate in beat cannot be picked up in a song that is not distinct in beating. In addition, a composer or a player may intentionally partially change the time.

If the beat cannot be extracted from the song, an erratic extraction is easily detected in the methods discussed with reference toFIGS. 10 and 11Aand11B. Preferably, at least two durations are set up in such a song, and accurate beat positions and accurate head portion positions of the bars are determined. With reference toFIG. 12, at least three points Z1, Z2, and Z3are specified, and the processes discussed with reference toFIGS. 9-1,9-2and10are performed for each duration between the specified points Z1, Z2, and Z3.

The specified points Z1, Z2, . . . are at the head portions of the bars, the length of the bar, the number of bars, and the beats from each of the specified points Z1, Z2, . . . to a next specified point are individually managed. The durations are divided into ones where the automatic extraction has been successful and ones where the automatic extraction has been unsuccessful, and the above-described calculation is performed for each duration with bar length different from duration to duration.

Separately from duration setting, the manual authoring unit4ofFIG. 1can input, as the time-series music data, music structure information regarding a melody structure of a song, including an intro, melody A, melody B, climax, and an ending of the song. More specifically, as shown inFIG. 13, marks451indicating position information of change for identifying each portion of the melody structure are attached and displayed in response to a user operation input, together with the beat position and the bar head portion position of the music data in the waveform display section301. The change position responsive to the melody structure is different from the specified points Z1, Z2, . . . , but is at the head portion positions of the bars in many cases. The tempo and the time may be changed in the melody structure from duration to duration. For example, the intro may be played at a tempo slower than a tempo in the other portion.

In accordance with embodiments of the present invention, the time-series metadata for an application that requires accurate beat position of a song can be easily and accurately attached. Information such as an accurate melody, accurate code progression, lyric, etc. may be attached in accordance with the beat position.

A music remixing application requiring accurate metadata, and a device and a software program requiring accurate timing control in synchronization with the beat can be manufactured.

The music metadata extraction system is described below. The music metadata extraction system includes the beat extracting apparatus2having the manual authoring unit4ofFIG. 1. The manual authoring unit4manually attaches the beat position and the bar head portion position. The music metadata extraction system also includes the automatic beat extractor3.

FIG. 14is a block diagram illustrating a hardware structure including the beat extracting apparatus2ofFIG. 1in accordance with one embodiment of the present invention. An apparatus10ofFIG. 14, including a personal computer, performs not only the functions of the automatic beat extractor3and the manual authoring unit4ofFIG. 1but also performs a music content delivery function and a music reproducing function.

The automatic beat extractor3obtains an automatically extracted beat serving as raw data (input data) to the manual authoring unit4ofFIG. 1. The automatic beat extractor3is not an essential element in the present invention. The higher the beat extraction accuracy in the automatic beat extractor3, the more the time for the manual authoring is saved. The beat extraction and the bar head portion extraction are performed at a high quality level. The use of a high-quality automatic beat extractor3is still useful. From this point of view, a preferable example of the automatic beat extractor3ofFIG. 1is described in detail.

In the apparatus10ofFIG. 14, a central processing unit (CPU)101, a read-only memory (ROM)102and a random-access memory (RAM)103are connected to a system bus100. The ROM102stores a variety of programs, and the CPU101performs processes on the RAM103as a working area in accordance with the programs.

Also connected to the system bus100are an audio data decoder104, a media drive105, a communication network interface (I/F)107, an operation input interface109, a display interface111, an input-output (I/O) port113, an input-output (I/O) port114, an input interface115, and a hard disk drive (HDD)121. A series of data units to be processed by elements is supplied to respective elements via the system bus100.

The media drive105reads music data of music content recorded on a disk106, such as a compact disk (CD) or a digital versatile disk (DVD), into the system bus100.

An operation input unit110such as a keyboard and a mouse is connected to the operation input interface109.

As previously discussed with reference toFIG. 3, the display112displays a GUI screen during a manual authoring operation. During the automatic beat extraction, the display112can display data in synchronization with the extracted beat or a doll or a robot dancing to the extracted beat.

The I/O port113connects to an audio reproducing section117and a beat extractor11. The I/O port114connects to the beat extractor11. The beat extractor11corresponds to the automatic beat extractor3ofFIG. 1.

The input interface115connects to an input unit116including an analog-to-digital (A/D) converter116A, a microphone terminal116B, and a microphone116C. An audio signal and a music signal picked up by the microphone116C are converted into a digital audio signal by the A/D converter116A. The digital audio signal is then supplied to the input interface115. The input interface115acquires the digital audio signal into the system bus100. The digital audio signal (time-series waveform signal) captured by the system bus100is recorded in a format .wav on the HDD121. The digital audio signal captured via the input interface115is not directly supplied to the audio reproducing section117.

Upon receiving music data from one of the HDD121and the media drive105via the system bus100, an audio data decoder104decodes the music data to restore the digital audio signal. The audio data decoder104then supplies the restored digital audio signal to the I/O port113via the system bus100. The I/O port113supplies the signal audio signal transferred via the system bus100to each of the beat extractor11and the audio reproducing section117.

The system bus100acquired data from a medium106such as a known CD via the media drive105. Uncompressed audio content downloaded by a listener and stored on the HDD121is directly captured by the system bus100. On the other hand, compressed audio content is returned to the system bus100via the audio data decoder104. The digital audio data is captured by the system bus100via the input unit116and the input interface115(the digital audio signal is not limited to a music signal and may include a voice signal, and other audio band signal) and then stored on the HDD121. The digital audio data is then returned to the system bus100.

In the apparatus10in accordance with one embodiment of the present invention, the digital audio signal (corresponding to the time-series waveform signal), captured by the system bus100, is transferred to the I/O port113and then to the beat extractor11.

The beat extractor11in accordance with one embodiment of the present invention includes a beat extraction processor12and a beat alignment processor13. The beat extraction processor12extracts beat position information of a rhythm of a song. The beat alignment processor13generates beat period information using the beat position information extracted by the beat extraction processor12, and aligns the beats of the beat position information extracted by the beat extraction processor12in accordance with the beat period information.

Upon receiving the digital audio signal recorded in a .wav file as shown inFIG. 15, the beat extraction processor12extracts coarse beat information from the digital audio signal and outputs the resulting coarse beat information as music data recorded in a .mty file. Based on all music data recorded in the .mty file and music data corresponding to a music portion assumed to have the same tempo, the beat alignment processor13aligns the beat position information extracted by the beat extraction processor12, and outputs the resulting beat position information as music data recorded in a .may file. Extracted beat position information having stepwise increased high accuracy is thus provided. The beat extractor11will be described in detail later.

The audio reproducing section117includes a digital-to-analog (D/A) converter117A, an output amplifier117B, and a loudspeaker117C. The I/O port113supplies the digital audio signal transferred via the system bus100to the D/A converter117A in the audio reproducing section117. The D/A converter117A converts the digital audio signal supplied via the I/O port113into an analog audio signal, and the analog audio signal is supplied to the loudspeaker117C via the output amplifier117B. The loudspeaker117C converts into a sound the analog audio signal supplied from the D/A converter117A via the output amplifier117B.

The display interface111connects to a display112, such as a liquid-crystal display (LCD). The display112displays a beat component and a tempo value extracted from the music data of the music content. The display112displays an animation screen or lyric to the music.

The communication network interface107connects to the Internet108. The music reproducing apparatus10accesses a server storing attribute information of the music content via the Internet108, thereby transmitting identification information of music content with acquisition request for attribute information as a key word. The music reproducing apparatus10causes a hard disk on the HDD121to store the attribute information transmitted from the server in response to the acquisition request.

The beat extractor11in the music reproducing apparatus10in accordance with one embodiment of the present invention extracts the beat position information of a rhythm of a song based on the feature of the digital audio signal to be discussed below.

FIG. 16Aillustrates a time-series waveform of the digital audio signal. The time-series waveform ofFIG. 16Aincludes portions indicating an instantaneous large peak. The portion indicating a large peak may correspond to a beat of a drum.

Actually listening to the song having the time-series waveform of the digital audio signal shown inFIG. 16Areveals that more beats are timed at generally regular intervals, although such beats are invisibly hidden in the time-series waveform of the digital audio data. The actual beat components of the rhythm of the song cannot be extracted from the large peak values of the time-series waveform alone shown inFIG. 16A.

FIG. 16Billustrates a spectrogram of the digital audio signal having the time-series waveform ofFIG. 16A. In the spectrogram of the digital audio signal ofFIG. 16B, beat components hidden in the time-series waveform ofFIG. 16Aemerge as portions instantaneously greatly changing in power spectrum. Actual listening to the song reveals that the portions instantaneously greatly changing in power spectrum correspond to the beat components. The beat extractor11regards the portions instantaneously greatly changing in power spectrum as the beat components of the rhythm.

The rhythm period and beats per minute (BPM) of the song can be known by extracting the beat component and measuring the beat period.

As shown inFIG. 17, the beat extraction processor12includes a power spectrum calculator12A, a rate of change calculator12B, an envelope follower12C, a comparator12D, and a binarizer12E.

The power spectrum calculator12A receives the digital audio signal made of a time-series waveform of a song shown inFIG. 18A.

The digital audio signal supplied from the audio data decoder104is supplied to the power spectrum calculator12A in the beat extraction processor12.

The power spectrum calculator12A cannot extract the beat component from the time-series waveform at a high accuracy level. The power spectrum calculator12A performs a fast Fourier transfer (FFT) process on the time-series waveform, thereby calculating a spectrogram ofFIG. 18B.

The definition in the FFT process is preferably set to be 5 to 30 ms in real time with a sample count being 512 or 1024 if a sampling frequency of the digital audio signal input to the beat extraction processor12is 48 kHz. Parameter values set in the FFT process is not limited to the ones set herein. The FFT process is preferably performed while window function such as hanning window or hamming window is applied with the window overlapped.

The power spectrum calculator12A supplies the calculated spectrum to the rate of change calculator12B.

The rate of change calculator12B calculates a rate of change of the power spectrum supplied from the power spectrum calculator12A. More specifically, the rate of change calculator12B differentiates the power spectrum supplied from the power spectrum calculator12A, thereby calculating a rate of change. By repeatedly differentiating the constantly varying power spectrum, the rate of change calculator12B outputs a detected signal having a extracted beat waveform ofFIG. 18C. The positively transitioned peaks of the extracted beat waveforms ofFIG. 18Care regarded as the beat component.

Upon receiving the detected signal from the rate of change calculator12B, the envelope follower12C imparts a hysteresis characteristic having an appropriate time constant to the detected signal. The envelope follower12C thus removes chattering from the detected signal, and then supplies the chattering free detected signal to the comparator12D.

The comparator12D sets an appropriate threshold level, and eliminates a low-level noise component from the detected signal supplied from the envelope follower12C, and then supplies the low-level noise free detected signal to the binarizer12E.

The binarizer12E binarizes the detected signal supplied from the comparator12D to leave the detected signal having a level equal to or higher than the threshold level. The binarizer12E outputs beat position information, indicating position of beat components of P1, P2, and P3in time and recorded in a .mty file, as the music data.

The beat extraction processor12extracts the beat position information from the time-series waveform of the digital audio signal, and then outputs the beat position information in the .mty file as the music data. The elements in the beat extraction processor12includes internal parameters set. By changing the internal parameters, the effect of operations of each element is modified. The internal parameters are automatically optimized. But the user can manually override the parameter setting using the operation input unit110.

The beat intervals of the beat position information of the song, extracted and recorded in the .mty file by the beat extraction processor12as the music data, are typically regular as shown inFIG. 19A.

The beat alignment processor13aligns beat position information in a music portion providing the same song or the same tempo, out of the beat position information extracted by the beat extraction processor12.

The beat alignment processor13extracts regularly timed beats, such as A1through A11ofFIG. 19A, timed at regular time intervals, from the metadata of the beat position information extracted and recorded in the .mty file by the beat extraction processor12. The beat alignment processor13does not extract irregularly timed beats, such as B1through B4. In accordance with one embodiment of the present invention, the regularly timed beats are timed at regular intervals, such as those of quarter notes.

The beat alignment processor13calculates accurate mean period T from the metadata of the beat position information extracted and recorded in the .mty file by the beat extraction processor12, and extracts, as regular interval beats, beats having a time interval equal to the mean period T.

The extracted regularly timed beats alone causes a blank period as shown inFIG. 19A. As shown inFIG. 19B, the beat alignment processor13adds interpolation beats C1through C3at positions where otherwise the regular beats would be timed. The beat position information relating to all beats timed at regular intervals is thus obtained.

The beat alignment processor13defines in beats as beats that are in equal phase with the regular interval beats, and extracts the in beats. The in beats are those synchronized with actual music beats, and include the regular interval beats. The beat alignment processor13defines out beats as beats that are out of phase with the regular interval beats, and excludes the out beats. The out beats are those that are not synchronized with the actual music beats (such as quarter note beats). The beat alignment processor13needs to discriminate between the in beats and the out beats.

To discriminate between the in beat and the out beat, the beat alignment processor13defines a constant window width W centered on the regular interval beat as shown inFIG. 20. The beat alignment processor13determines that a beat contained within the window width W is an in beat, and that a beat not contained within the window width W is an out beat.

When no regular interval beat is not contained in the window width W, the beat alignment processor13adds an interpolation beat to interpolate between the regular interval beats.

As shown inFIG. 21, the beat alignment processor13extracts regular interval beats A11through A20, and an in beat D11substantially in phase with the regular interval beat A11. The beat alignment processor13also extracts interpolation beats C11through C13. The beat alignment processor13does not extract out beats B11through B13as quarter note beats.

Since music beats fluctuate in time, the number of in beats extracted from a song suffering a large fluctuation is decreased. An extraction error, called beat slip, may be caused.

The value of the window width W is set to be large in a song suffering from a large fluctuation. The number of in beats extracted is thus increased, and the extraction error is reduced. The window width W may be normally constant. In a song suffering from a large fluctuation, the window width W as a parameter may be adjusted to be larger.

The beat alignment processor13imparts, to the metadata, beat attribute indicating the in beat contained in the window width W and the out beat not contained in the window width W. If no extracted beat is present within the window width W, the beat alignment processor13automatically adds an interpolation beat, and gives beat attribute as the interpolation beat as the metadata. The metadata forming the beat information includes the above-described beat position information and the beat attribute, and records the metadata in a metadata file (.may). The elements contained in the beat alignment processor13have internal parameters such as the window width W, and the effect of the operation of the beat alignment processor13is modified by modifying the internal parameters.

The beat extractor11automatically extracts highly accurate beat information from the digital audio signal by performing two-step data processing with the beat extraction processor12and the beat alignment processor13. The beat extractor11performs not only the in beat/out beat determination process but also the beat interpolation process, thereby resulting in the beat information of quarter note intervals over an entire song.

The beat extractor11discussed as a specific example of the manual authoring unit4ofFIG. 1can musically synchronize a pulse-code modulated (PCM) audio waveform (sampling source) having no timestamp information such as the beat position information with another medium. The data size of the time stamp information such as the beat position information ranges from several Kbytes to tens of Kbytes, and is extremely small, as small as one-thousandth the data size of the audio waveform. Memory capacity and operation steps may be reduced, thereby permitting the user to easily handle the beat position information. Even if a song suffers from tempo change or rhythm fluctuation, beats can be accurately extracted from the entire song. By synchronizing the music with another medium, a new entertainment may be created.

The beat information extracted by the beat extractor11is transferred to the manual authoring unit4ofFIG. 1to obtain the metadata of highly accurate beat position and bar head.

With the hardware structure ofFIG. 14, the manual authoring unit4ofFIG. 1performs the previously described manual authoring functions using software.

FIG. 22is a functional block diagram illustrating software modules of the manual authoring unit4in the beat extraction system ofFIG. 1.

As shown inFIG. 22, an operating system200inputs data to and outputs data from a keyboard110A, a mouse110B, a display112, a loudspeaker117C, and a storage device201, each used as the operation input unit110ofFIG. 14. The storage device201stores acoustic data202containing the music data and metadata203, and includes the HDD121ofFIG. 14, for example.

Software modules implemented on the operating system200includes a user input processor205, a metadata manager206, a display manager207, an automatic beat detector208, a file input-output unit209, an acoustic data manager210, and an acoustic reproducing unit211.

The user input processor205determines a user operation input entered via the input device such as the keyboard110A and the mouse110B, and issues process commands to modules responsive to the user operation input. When the user selects a duration mark input operation using the mouse110B, the user input processor205sets itself to a duration mark input state, and notifies the display manager207that the user input processor205is in the duration mark input state.

When the user then clicks on the mouse110B within an input enabled area, the user input processor205converts clicked coordinates into a position of the acoustic data in time axis and instructs the metadata manager206to place a duration mark at the position in time axis.

The metadata manager206includes a duration manager221and metadata body222. The metadata body222includes lists of data trains as shown inFIG. 23B. Each data train has a structure shown inFIG. 23A. The data train ofFIG. 23Aincludes bar number, metadata type including beat, melody, and accent, and sample position. The metadata is chronologically managed. The metadata body222is managed in a list structure. When new data to be inserted in editing operation is created, a reference destination pointer and a reference source pointer are simply changed.

The duration manager221manages durations. The duration manager221stores a structure ofFIG. 24as a unit. As shown inFIG. 24, the one unit includes a start sample, an end sample, a duration BPM, a beat in duration, the number of bars in duration, and a duration bar length.

The metadata manager206generates a duration based on duration information informed of by the user input processor205, and determines the beat position, based the bar position informed of by the user input processor205, in accordance with the process discussed with reference toFIGS. 9-1and9-2. The beat position calculation is performed on a per duration basis.

The metadata manager206deletes, moves, inserts, and searches for the metadata body222and the duration data.

The display manager207, including an acoustic data display225and a metadata display226, calculates a range of time axis currently displayed on the screen of the display112, and searches for the metadata contained in the range using the metadata manager206. The display manager207displays the metadata in a color or an icon responsive to the type of acquired metadata on the screen of the display112.

The display manager207further displays the waveform of the acoustic data, statistical data, the bar number, and a numerical value of a sample position of a sample currently being edited.

The automatic beat detector208corresponds to one of the automatic beat extractor3ofFIG. 1and the beat extractor11ofFIG. 14. The automatic beat detector208automatically extracts the music beat from the music data, and notifies the metadata manager206of the position of each automatically extracted beat. An automatic beat extraction algorithm may be used as long as the algorithm generates information satisfying the requirement of information to be supplied to the metadata manager206.

The file input-output unit209opens specified acoustic data, reads data of samples of the number specified by a specified position, and returns the data to a request source. The file input-output unit209outputs the metadata body to a file on a secondary storage such as the storage device201, and reads the data from the metadata file on the secondary storage into the metadata body222.

The acoustic data manager210stores the acoustic data read via the file input-output unit209, and transfers the acoustic data to each of the acoustic reproducing unit211and the display manager207. The storage device201searches for and stores a variety of information incident to the acoustic data (such as a quantization bit number, a sampling rate, the number of channels, etc.).

The acoustic reproducing unit211reproduces the acoustic data from a specified point, and outputs the acoustic data to an actual acoustic output device such as the loudspeaker117C.