Patent Description:
A mashup is a fusion or mixture of disparate elements, and, in media, can include, in one example, a recording created by digitally synchronizing and combining background tracks with vocal tracks from two or more different songs (although other types of tracks can be "mashed-up" as well). A mashing up of musical recordings may involve removing vocals from one first musical track and replacing those vocals with vocals from at least one of second musically-compatible track, and/or adding vocals from the second track to the first track.

Listeners are more likely to enjoy mash-ups created from songs the users already know and like. Some commercially available websites enable users to listen to playlists suited to the users' tastes, based on state-of-the-art machine learning techniques. However, the art of personalizing musical tracks themselves to users' tastes has not been perfected.

Also, a mashup typically does not work to combine two entire songs, because most songs are much too different from each other for that to work well. Instead, a mashup typically starts with the instrumentals of one song as the foundation, and then the vocals are inserted into the instrumentals one short segment at a time. Any number of the vocal segments can be inserted into the instrumentals, and in any order that may be desired.

However, if two vocal and instrumental segments are not properly aligned, then they will not sound good together.

It is with respect to these and other general considerations that embodiments have been described. Also, although relatively specific problems have been discussed, it should be understood that the embodiments should not be limited to solving the specific problems identified in the background. <CIT> discloses audio signal separation and recombination known in the art of audio mixing. Thereby analysis methods are applied for separation of audio and automatic separation and/or recombination of the audio.

It is an object of the invention to overcome the shortcomings in the prior art. This object of the invention is solved by the independent claims. Specific embodiments are defined in the dependent claims.

In the following detailed description, references are made to the accompanying drawings that form a part hereof, and in which are shown by way of illustrations specific embodiments or examples. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the present disclosure. Embodiments may be practiced as methods, systems or devices. Accordingly, embodiments may take the form of a hardware implementation, an entirely software implementation, or an implementation combining software and hardware aspects. The following detailed description is therefore not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims and their equivalents.

Example aspects described herein can create new musical tracks that are a mashup of different, pre-existing audio tracks, such as, e.g., musical tracks. By example and without limitation, at least one component of a musical track, such as a vocal component, can be combined with at least part of another musical track, such as an instrumental or background track (also referred to as an "accompaniment track"), to form a mashup of those tracks. According to an example aspect herein, such a musical mashup can involve various procedures, including determining musical tracks that are musically compatible with one another, determining, from those tracks, segments that are compatible with one another, performing beat and downbeat alignment for the compatible segments, performing refinement of transitions between the segments, and mixing the segments of the tracks.

Before describing the foregoing procedures in more detail, examples of at least some types of information that can be used in the procedures will first be described. Example aspects of the present application can employ various different types of information. For example, the example aspects can employ various types of audio signals or tracks, such as mixed original signals, i.e., signals that include both an accompaniment (e.g., background instrumental) component and a vocal component, wherein the accompaniment component includes instrumental content such as one or more types of musical instrument content (although it may include vocal content as well), and the vocal component includes vocal content. Each of the tracks may be in the form of, by example and without limitation, audio files for each of the tracks (e.g. mp3, wav, or the like). Other types of tracks that can be employed include solely instrumental tracks (e.g., tracks that include only instrumental content, or only an instrumental component of a mixed original signal), and vocal tracks (e.g., tracks that include only vocal content, or only a vocal component of a mixed original signal). In one example embodiment herein, a 'track' may include an audio signal or recording of the applicable content, a file that includes an audio recording/signal of applicable content, a section of a medium (e.g., tape, wax, vinyl) on which a physical (or magnetic) track has been created due to a recording being made or pressed there, or the like. Also, for purposes of this description, the terms "background" and "accompaniment" are used interchangeably.

In one example embodiment herein, vocal and accompaniment/background (e.g., instrumental) tracks (or components) can be obtained from mixed, original tracks, although in other examples they may pre-exist and can be obtained from a database.

Example aspects of the present application also can employ song or track segmentation information for creating mashups. For example, song segmentation information can include the temporal positions of boundaries between sections of each track.

An additional type of information that can be employed to create mashups can include segment labelling information. Segment labelling information identifies (using, e.g., particular IDs) different types of track segments, and track segments may be labeled according to their similarity. By example and without limitation, segments that are included in a verse (which tends to be repeated) of a song may have a same label, segments that are included in a chorus of a song may have a same label, and the like. In one example, segments that are considered to be similar to one another (and which thus have a same label) are deemed to be within a same cluster.

Of course, the above examples given for how to obtain vocal and accompaniment tracks, song segmentation information, and segment labelling information, are intended to be representative in nature, and, in other examples, vocal and/or accompaniment tracks, song segmentation information, and/or segment labelling information may be obtained from any applicable source, or in any suitable manner known in the art.

Additional information that can be employed to create mashups also can include tempo(s) of each track, a representation of tonality of each track (e.g., a twelve-dimensional chroma vector), beat/downbeat positions in each track (e.g., temporal positions of beats and downbeats in each track), information about the presence of vocals (if any) in time in each track, energy of each of the segments in the vocal and accompaniment tracks, or the like. The foregoing types of information can be obtained from any applicable source, or in any suitable manner known in the art. In one example, at least some of the foregoing information is obtained for each track (including, e.g., separated tracks) using a commercially available audio analysis tool, such as the Echo Nest analyzer. In other examples, the aforementioned types of information may pre-exist and can be obtained from a database.

According to one example, determining information about the presence of vocals involves mining original-instrumental pairs from a catalogue of music content, extracting strong vocal activity signals between corresponding tracks, exploiting the signal(s) to train deep neural networks to detect singing voice, and recognizing the effects of this data source on resulting models. In other example embodiments herein, information (vx) about the presence of vocals can be obtained from loudness of a vocal track obtained from a mixed, original signal, such as, e.g., a vocal track obtained according the Jansson application identified above.

Additional information that can be employed to create mashups can include acoustic feature vector information, and loudness information (e.g., amplitude). An acoustic feature vector describes the acoustic and musical properties of a given recording. An acoustic feature vector can be created manually, by manually quantifying the amount of given properties, e.g. vibrato, distortion, presence of vocoder, energy, valence, etc. The vector can also be created automatically, such as by using the amplitude of the signal, the time-frequency progression, or more complex features.

Each of the above types of information associated with particular tracks and/or with particular segments of tracks, can be stored in a database in association with the corresponding tracks and/or segments. The database may be, by example and without limitation, one or more of main memory <NUM>, portable storage medium <NUM>, and mass storage device <NUM> of the system <NUM> of <FIG> to be described below, or the database can be external to that system <NUM>, in which case it can be accessed by the system <NUM> by way of, for example, network <NUM> and peripheral device(s) <NUM>. For purposes of this description, the various types of information are shown as information <NUM> stored in mass storage device <NUM> of <FIG>, although of course the information <NUM> can be stored in other storage devices as well, or in lieu of mass storage device <NUM>, as described above.

<FIG> shows an example flowchart representation of how an automashup can be performed based on a candidate track that includes vocal content, and a background or query track, according to an example embodiment herein. In this example, the algorithm to perform the automashup creates a music mashup by sequentially adding vocal segments of one or more track(s) (of one song) on top of one or more segments of a background track, (of, e.g., another song), and/or by replacing vocal content of one or more segments of a background track (of one song) that includes the vocal content, with vocal content of the one or more track(s) (of, e.g., another song). Inputs to the algorithm can include, by example, a background track (e.g., including instrumental or vocal/instrumental content) (also referred to herein as a "query track" or "base track"), such as track <NUM> of <FIG>, and a (potentially large) set of vocal candidate tracks, including track <NUM> having vocal content, each of which may be obtained from the database and/or in accordance with the method(s) described in the Jansson application, for example.

In one example embodiment herein, with respect to tracks <NUM>, <NUM>, the content of track <NUM> is from a different song than the content from track(s) <NUM>, although in other examples the content of at least some tracks <NUM>, <NUM> may be from the same song(s). For purposes of this description, the track <NUM> also is referred to herein as a "target" or "candidate" track <NUM>. Also, each track <NUM>, <NUM> includes respective segments, wherein segments of the candidate or target track <NUM> are also referred to herein as "candidate segments" or "target segments", and segments of the query track <NUM> also are referred to herein as "query segments". <FIG> shows a representation of a query track <NUM> having query segments <NUM>, and candidate tracks <NUM> having candidate segments <NUM>, based on which a mashability score <NUM> can be determined, according to an example aspect herein. The query segments <NUM> may include, by example and without limitation, instrumental or vocal/instrumental content, (e.g., of one song), and the candidate segments <NUM> may include, by example and without limitation, at least vocal content (of, e.g., at least one other song). Of course, the scope of the invention is not limited to these examples only, and the segments <NUM>, <NUM> may include other types of content arrangements than those described above.

As represented in <FIG>, the candidate track includes vocals <NUM> and the query track <NUM> includes separated vocal component/track <NUM> and separated instrumental component/track <NUM>. In addition, additional track features 112a of the query track and additional track features 110a of the candidate track <NUM> are also identified from the query track <NUM> and candidate track <NUM>. Track features 110a and 112a can include, for example, acoustic features (such as tempo, beat, musical key, likelihood of including vocals, and other features as described herein). Information regarding loudness 114b and tonality (e.g., tonal representation) 114a are obtained based on the vocal component <NUM> of the candidate track <NUM>. Information regarding loudness 118b and tonality (e.g., tonal representation) 118a based on the separated instrumental component/track <NUM> and information regarding at least loudness 116a based on the separated vocal component/track <NUM> of the query track <NUM> are obtained.

The information represented by reference numerals 110a, 112a, <NUM>, 114a, 114b, <NUM>, 116a, <NUM>, 118a and 118b is employed in an algorithm to perform an automashup that results in a mashup track <NUM>, according to an example aspect herein. It should be note that, although candidate track <NUM> is shown and described above for convenience as including instrumental content, in some cases it also may include at least some vocal content as well, depending on the application of interest.

A procedure <NUM> according to an example aspect herein, for determining whether individual segments of a query track (e.g., an accompaniment track) <NUM> under consideration are to be kept, or have content (e.g., vocal content) replaced or added thereto from one or more candidate (e.g., vocal) tracks <NUM>, during an automashup of the tracks <NUM>, <NUM>, will now be described, with reference to <FIG> and <FIG>. In one example embodiment herein, and as described above, the content of query track <NUM> used in the procedure <NUM> is from a different song than the content from the one or more candidate track(s) <NUM> used in the procedure <NUM>, although in other examples the content of at least some tracks <NUM>, <NUM> used in the procedure <NUM> may be from the same song(s). Also, although at least some parts of the below description may be described in the context of procedure <NUM> being performed for one query track <NUM> and one candidate track <NUM>, the scope of the invention is not so limited, and the procedure can involve more than two tracks, such as, by example, a query track <NUM> and a plurality of candidate tracks <NUM>, wherein each track <NUM>, <NUM> may include content from different songs (or, in other examples, at least some of the same songs.

In one example embodiment herein, the procedure <NUM> employs at least some of the various types of information <NUM> as described above, including, without limitation, information about the likelihood of a segment containing vocals (vx) (e.g., at beats of segments), downbeat positions, song segmentation information (including start and end positions of segments), and segment labelling information (e.g., IDs), and the like. As described above, each type of information may be stored in a database in association with corresponding tracks <NUM>, <NUM> and/or segments <NUM>, <NUM> associated with the information <NUM>.

Referring to <FIG>, query segments <NUM> of the query track <NUM> that have less than a predetermined number of bars (e.g., eight bars) are filtered out and discarded (step <NUM>), while others are maintained. In steps <NUM> and <NUM> scores (e.g., two scores) are determined for a first one of the maintained query segments <NUM>. More particularly, in step <NUM>, a first score (K_keep_vx) is calculated by determining, for all beats of the currently considered query segment <NUM>, a mean value of the probability of the segment <NUM> containing vocals at each beat, based on the information about the likelihood of the segment <NUM> containing vocals (vx) at those beats, wherein in one example embodiment, that information may be obtained from the database. In step <NUM>, which includes substeps 206a to 206d, a second score (K_keep_rep) is determined. More particularly, in sub-step 206a, given a predetermined ideal number of repetitions (e.g., two) (i.e., an amount of segments of query track <NUM> (or, in another example embodiment, of a candidate track <NUM>) having the same segment ID) represented by the term "ideal_num_reps", an intermediate value ("score_rep") is determined according to the following formula (F1): <MAT> where Nrepet represents a number of segments <NUM> of the query track <NUM> that have the same segment labelling information (e.g., the same segment ID) as the currently considered query segment <NUM>, score_rep represents the intermediate score, and ideal_num_reps represents the predetermined ideal number of repetitions.

If the value of score_rep is greater than value '<NUM>' ("Yes" in sub-step 206b), then in sub-step 206c, the value of score_rep is set as follows, according to formula (F2): <MAT>.

On the other hand, if the value of score_rep is less than or equal to value '<NUM>' ("No" in sub-step 206b), then the value of score_rep that was determined in step 206a is maintained.

In either case, after sub_step 206b, control passes to sub-step 206d, where a value for the second score (K_keep_rep) is determined according to the following formula (F3): <MAT>.

Then, control passes to step <NUM> where a value of a "keep score" K_keep is determined according to the following formula (F3'), for the segment <NUM> under consideration: <MAT>.

Next, control passes via connector A to step <NUM> of <FIG>, where a determination is made as to whether the query track <NUM> includes additional query segments <NUM> that have not yet been considered. If "Yes" in step <NUM>, then control passes back to step <NUM> where the procedure <NUM> continues in the above described manner, but for a next segment <NUM> in a sequence of segments <NUM> of the query track <NUM>. If "No" in step <NUM>, then control passes to step <NUM> where any segments <NUM> that were processed as described above (in steps <NUM> to <NUM>) are clustered according to their IDs. In particular, according to one example embodiment herein, step <NUM> includes determining labels (e.g., IDs) (e.g., based on segment labelling information among information <NUM>) of those segments <NUM>, and then clustering together segments <NUM> having the same labels. As a result of step <NUM>, there may be as many clusters determined as there are unique segment labels (IDs).

In a next step <NUM>, a mean K_keep score for each of the clusters (i.e., a mean of the K_keep score values for segments <NUM> from each respective cluster) is determined, and then control passes to step <NUM>, where a set of segments <NUM> from the cluster with the greatest determined mean K_keep score is selected. Then, in step <NUM>, it is determined which segments <NUM> have a length of less than a predetermined number of bars (e.g., <NUM> bars), and those segments are added to the selected set of segments, according to one example embodiment herein, to provide a combined set of segments <NUM>. The combined set of segments <NUM> resulting from step <NUM> is deemed to be assigned to "S-keep", and thus each segment <NUM> of the combined set will be maintained (kept) with its original content, whether the content includes vocal content, instrumental content, or both.

To determine segments "(S_subs)" for which the original vocal content included therein will be replaced, and to determine segments (S_add) to which vocals from other songs will be added (versus replaced), the remaining set of segments <NUM> that had not been previously assigned to S_keep are employed. More specifically, to determine segments S_add, those ones of the remaining segments <NUM> (i.e., those not resulting from step <NUM>) that are deemed to not contain vocal content are identified. In one example embodiment herein, identification of such segments <NUM> is performed as described in the Humphrey application (and/or the identification may be based on information <NUM> stored in the database), and can include determining a mean probability that respective ones of the segments <NUM> contain vocal content (at each of the beats) (step <NUM>). Then, for each such segment <NUM>, a determination is made as to whether the mean determined therefor is lower than a predetermined threshold (e.g., <NUM>) in step <NUM>. If the mean for respective ones of those segments <NUM> is not lower than the predetermined threshold (i.e., if the mean equals or exceeds the predetermined threshold) ("No" in step <NUM>), then those respective segments <NUM> are deemed to be segments (S_subs) for which the original vocals thereof will be replaced (i.e., each such segment is assigned to "S_subs") (step <NUM>). If the mean calculated for respective ones of the segments <NUM> identified in step <NUM> is lower than the predetermined threshold ("Yes" in step <NUM>), then those segments <NUM> are deemed to be segments (S_add) to which vocals from other, candidate tracks <NUM> will be added (i.e., each such segment is assigned to "S_add") (step <NUM>).

A procedure <NUM> to perform automashups using the segments (S_subs) and (S_add), according to an example aspect herein, will now be described, with reference to <FIG>. The procedure <NUM> is performed for each respective segment <NUM> assigned to S_subs and S_add. In step <NUM>, a search is performed to find/identify one or more compatible candidate (e.g., vocal) segments <NUM> for a first segment <NUM> from among the segments <NUM> that were assigned to S_subs and S_add. In one example embodiment herein, step <NUM> involves performing a song suggester procedure and a segment suggestion procedure, and computing one or more mashability scores for the segment <NUM> (of the query track <NUM> under consideration) and segments <NUM> from candidate tracks <NUM>. In one example embodiment herein, the song suggester procedure is performed in accordance with procedure <NUM> of <FIG> to be described below, and the segment suggestion procedure is performed in accordance with procedure <NUM> of <FIG> to be described below. Also, in one example embodiment herein, the mashability score is performed as will be described below.

Then, in step <NUM>, beat and downbeat alignment is performed for the segment <NUM> under consideration and the candidate (e.g., vocal) segment(s) <NUM> determined to be compatible in step <NUM>. In step <NUM>, transition refinement is performed for the segment <NUM> under consideration and/or the candidate segment(s) <NUM> aligned in step <NUM>, based on, for example, segmentation information, beat and downbeat information, and voicing information, such as that stored among information <NUM> in association with the tracks <NUM>, <NUM> and/or segments <NUM>, <NUM> in the database. Then, in step <NUM>, those segments <NUM>, <NUM> are mixed. In one example, mixing includes a procedure involving time-stretching and pitch shifting using, for example, pysox or a library such as elastique. By example, in a case where that segment <NUM> was previously assigned to S_subs, mixing can include replacing vocal content of that segment <NUM>, with vocal content of the aligned segment <NUM>. Also by example, in a case where the segment <NUM> was previously assigned to S_add, mixing can include adding vocal content of the segment <NUM> to the segment <NUM>.

In a next step <NUM>, a determination is made as to whether a next segment <NUM> among segments (S_subs) and (S_add) exists in the query track <NUM>, for being processed in the procedure <NUM>. If "Yes' in step <NUM>, then control passes back to step <NUM> where the procedure <NUM> is performed again for the next segment <NUM> of the track <NUM>. If "No" in step <NUM>, then the procedure ends in step <NUM>. As such, the procedure <NUM> is performed (in one example embodiment) in sequential order, from a first segment <NUM> of the query track <NUM> until the last segment <NUM> of the query track <NUM>. The procedure also can be performed multiple times, based on the query track <NUM> and multiple candidate tracks <NUM>, such that a mashup is created based on multiple ones of the tracks <NUM>. Also in a preferred embodiment herein, to reduce processing load and the amount of time required to perform procedure <NUM>, the number of candidate tracks <NUM> that are employed can be reduced prior to the procedure <NUM>, by selecting best options from among the candidate tracks <NUM>. This is performed by determining a "song mashability score" (e.g., score <NUM> of <FIG>), which will be described in detail below.

As a result of the procedure <NUM>, a mashup track <NUM> (<FIG>) is provided based on the query track <NUM> and at least one candidate track <NUM> under consideration. The mashup track <NUM> includes, by example, one or more segments <NUM> that were assigned to S_keep, one or more other segments <NUM> having vocal content (from one or more candidate tracks <NUM>) that was used to replace vocal content of an original version of those other segments <NUM> in step <NUM>, and one or more further segments <NUM> having vocal content (from one or more candidate tracks <NUM>) that was added to those further segments <NUM> in step <NUM>. In the mashup track <NUM>, beat positions in the query track <NUM> are mapped with corresponding beat positions of the candidate track(s) <NUM>.

Before describing how a song mashability score is determined, the song suggester procedure <NUM> according to an example aspect herein will first be described. In one example embodiment herein, the song suggester procedure <NUM> involves calculating a song mashability score defining song mashability. To do so, a number of different types of scores are determined or considered to determine song mashability, including, by example and without limitation, an acoustic feature vector distance, a likelihood of including vocals, closeness in tempo, and closeness in key.

An acoustic vector distance score is represented by "Ksong (acoustic)". In one example embodiment herein, an ideal normalized distance between tracks can be predetermined such that segments under evaluation are not too distant from one another in terms of acoustic distance. The smaller the distance between the query and candidate (e.g., vocal) tracks, the higher is the score. Of course, in other example embodiments herein, the ideal normalized distance need not be predetermined in that manner. Also, it is within the scope of the invention for the ideal normalized distance to be specified by a user, and/or the ideal normalized distance may be such that the segments under evaluation are not close in space (i.e., and therefore the segments may be from songs of different genres) to achieve a desired musical effect, for example.

In one example embodiment herein, an acoustic feature vector distance score Ksong(acoustic) is determined according to the procedure <NUM> of <FIG>. In step <NUM>, the acoustic vector of the original query track <NUM> under consideration (e.g., in procedure <NUM>) is determined, without separation (query-mix_ac). In step <NUM>, a cosine distance between query-mix_ac and all vectors of the candidate tracks <NUM> is determined. In one example embodiment herein, step <NUM> determines a respective vector of acoustic feature vector distance between the query track <NUM> and each candidate track <NUM>, using a predetermined algorithm. The predetermined algorithm involves using random projections and building up a tree. At every intermediate node in the tree, a random hyperplane is selected, that divides the space into two subspaces. The hyperplane is chosen by sampling a plurality (e.g., two) of points from the subset and taking the hyperplane equidistant from them. The foregoing is performed k times to provide a forest of trees, wherein k is tuned as deemed needed to satisfy predetermined operating criteria, considering tradeoffs between precision and performance. In one example, a Hamming distance packs the data into <NUM>-bit integers under the hood and uses built-in bit count primitives. All splits preferably are axis-aligned. A Dot Product distance reduces the provided vectors from dot (or "inner-product") space to a more query friendly cosine space.

In another example embodiment herein, the predetermined algorithm is the Annoy (Approximate Nearest Neighbors Oh Yeah) algorithm, which can be used to find nearest neighbors. An Annoy tree is a library with bindings for searching for points in space close to a particular query point. The Annoy tree can form file-based data structures that can be mapped into memory so that various processes may share the same data. In one example, and as described above, an Annoy algorithm builds up binary trees, wherein for each tree, all points are split recursively by random hyperplanes. A root of each tree is inserted into a priority queue. All trees are searched using the priority queue, until there are search_k candidates. Duplicate candidates are removed, a distance to candidates is computed, candidates are sorted by distance, and then top ones are returned.

In general, a nearest neighbor algorithm involves steps such as: (a) start on an arbitrary vertex as a current vertex, (b) find out a shortest edge connecting the current vertex with an unvisited vertex V, (c) set the current vertex to V, (d) mark V as visited, and (e) if all the vertices in domain are visited, then terminate. The sequence of the visited vertices is the output of the algorithm.

Referring again to <FIG>, a next step <NUM> includes normalizing the vector of acoustic feature vector distance(s) determined in step <NUM> by its maximum value, to obtain normalized distance vector(s) (step <NUM>), or, in other words, a resulting final vector of acoustic feature vector distances (Vdist), wherein, in one example embodiment, Vdist is within the interval [<NUM>,<NUM>].

Then, for a given candidate track <NUM> with index j, formula (F4) is performed in step <NUM> to determine a distance ("difference") between the final vector of acoustic feature vector distances (Vdist) and an ideal normalized distance: <MAT> where "Vdist[j]" is the final vector of acoustic feature vector distances for candidate track <NUM> with index j, and "ideal_norm_distance" is the ideal normalized distance. In one example embodiment herein, the ideal normalized distance ideal_norm_distance can be predetermined, and, in one example, is zero ('<NUM>'), to provide a higher score to acoustically similar songs.

A value of "Ksong(acoustic)" (the acoustic feature vector distance score) is then determined in step <NUM> according to the following formula (F5): <MAT> where "difference" is defined as in formula (F4).

In the foregoing manner, the acoustic feature vector score Ksong(acoustic) is determined.

As described above, another type of information that is used to determine a mashability score is information about the presence of vocals (if any) in time, or, in other words, information representing the likelihood that a segment in question contains vocals. As described above, information about the presence of vocals (if any) in time, for a candidate track <NUM>, can be obtained according to the method described in the Humphrey application, although this example is not exclusive, and the information can be obtained from among the information <NUM> stored in a database. For convenience, information representing the likelihood that a segment in question contains vocals is referred to herein as a "vocalness likelihood score".

In one example embodiment herein, a greater likelihood of a track segment including vocals means a greater score. Such a relationship can be useful in situations where, for example, users would like to search for tracks <NUM> which contain vocals. In another example scenario (e.g., a DJ wanting to mix together songs) the vocalness likelihood score may be ignored.

In one example embodiment herein, a vocalness likelihood score can be determined according to procedure <NUM> of <FIG>. In step <NUM> a likelihood of each beat of a candidate track <NUM> under consideration containing vocals, is determined. In one example embodiment, step <NUM> is performed in accordance with the procedure(s) described in the Humphrey application, or, in another example, step <NUM> can be performed based on a likelihood information obtained from among information <NUM> in the database. Next, in step <NUM> an average of the likelihood determined in step <NUM> for each musical measure of the track <NUM>, is determined. Next, in step <NUM> a maximum value among averages determined in step <NUM> for all measures is determined (and is represented by "Ksong(vocalness)"). Procedure <NUM> is performed for each candidate track <NUM>.

Another type of information that is used to determine a mashability score is closeness in tempo. For determining a score for closeness in tempo, according to an example embodiment herein, that score, which is represented by "Ksong(tempo)", is determined according to the following formula (F6): <MAT> where tempo_cand and tempo_query are the tempi of the candidate and query tracks <NUM>, <NUM>, respectively (e.g., such tempi can be retrieved from the database), and K_tempo is a factor to control the penalty of the difference between tempi. Tempo can be determined in many ways. One example includes: tempo = <NUM>/median (durations), where durations are the durations of the beats in a song. In one example embodiment herein, the closer the candidate and query tracks <NUM>, <NUM>, are in beats-per-minute (bpm), the higher is the score Ksong(tempo) (in a logarithmic scale).

Another type of information that is used to determine a mashability score is closeness in key, which is defined by a "closeness in key score" Ksong(key). The manner in which a closeness in key score Ksong(key) is determined according to an example embodiment herein, will now be described. The closeness in key score Ksong(key) measures how close together tracks <NUM>, <NUM> are in terms of musical key. In one example embodiment herein, "closeness" in key is measured by way of a difference in semitones of keys of tracks <NUM>, <NUM>, although this example is non-limiting. Also in one example embodiment herein, the smaller the difference (in semitones) between the semitones of tracks <NUM>, <NUM>, then the greater is the score Ksong(key). <FIG> shows a representation of a known cycle of fifths, representing how major and minor keys and semitones relate to one another in Western musical theory.

<FIG> shows a procedure <NUM> for determining closeness in key, according to an example embodiment herein. In step <NUM>, a determination of the key and of each track <NUM>, <NUM> (and the pitch at each beat of segments of the tracks <NUM>, <NUM>) under consideration is made. The key and the pitch of a segment is determined using methods described in the Jehan reference discussed above. According to an example embodiment herein, if the tracks <NUM>, <NUM> under consideration are determined to be in the same type of key (e.g., both are in a major key, or both are in a minor key) ("Yes" in step <NUM>), then the keys determined in step <NUM> are passed to step <NUM> to calculate the score Ksong(key), in a manner as will be described below.

Referring again to step <NUM>, if two tracks <NUM>, <NUM> under consideration are not both in a major key, or are not both in a minor key ("No" in step <NUM>), then, prior to determining the score Ksong(key), the relative key or pitch corresponding to the key or pitch, respectively, of one of those tracks <NUM>, <NUM> is determined (step <NUM>). For example, each pitch in the major key in Western music is known to have an associated relative minor, and each pitch in the minor key is known to have a relative major. Such relationships between relative majors and minors may be stored in a lookup table stored in a database (such as the database described above). <FIG> represents one example of the lookup table (LUT) <NUM>. To determine a relative major or minor of a key of a particular track <NUM>, <NUM> in step <NUM>, the key of the track <NUM>, <NUM> can be correlated to in the lookup table <NUM>, and the relative major or minor key associated with the correlated-to key can be accessed/retrieved from the table <NUM>, wherein the relative key is in the same key type (e.g., major or minor) as the other track <NUM>, <NUM> under consideration. By example and without limitation, where a candidate track <NUM> is determined to be in a key of A major in step <NUM>, and the query track <NUM> is determined to be in a key of D minor in step <NUM>, then it is determined in step <NUM> that those tracks <NUM>, <NUM> have different key types ("No" in step <NUM>). Control then passes to step <NUM> where, in one example embodiment herein, D minor is correlated to a key in the lookup table <NUM>, to access the relative major (e.g., F major) stored in association therewith in the lookup table <NUM>. The accessed key (e.g., F major) is then passed with the A major key to step <NUM> to calculate the score Ksong(key) based thereon, in a manner to be described below.

Step <NUM> will now be described. In step <NUM>, a determination is made of the difference in semitones between the root notes of the keys received as a result of the performance of step <NUM> or <NUM>, wherein the difference is represented by variable "n_semitones". In one example herein, the difference n_semitones can be in a range between a minimum of zero "<NUM>" and a maximum of six "<NUM>", although this example is not limiting.

By example, if a candidate track <NUM> under consideration is in a major key and has a root pitch class of A major, and the query track <NUM> under consideration also is in a major key and has a root pitch class of B major ("Yes" in step <NUM>), then in step <NUM> a determination is made of the difference (in semitones) between those root pitch classes, which in the present example results in a determination of two ('<NUM>') semitones (i.e., n_semitones = <NUM>). In another example, in a case in which the candidate track <NUM> under consideration is in a major key and has a root pitch class of C major, and the query track <NUM> under consideration is in a minor key and has a root pitch class of G minor ("No" in step <NUM>), then the relative minor of C major (e.g., A minor) is correlated to and accessed from the lookup table <NUM> in step <NUM>, and is provided to step <NUM> along with G minor. In step <NUM>, a determination is made of the difference (in semitones) between those root pitch classes, which in the present example results in a determination of two ('<NUM>') semitones (i.e., n_semitones = <NUM>).

Step <NUM> will now be described. According to an example embodiment herein, step <NUM> is performed to determine the closeness in key score, using the following formula (F6): <MAT> where the variable Ksong(key) represents the closeness in key score, variable n_semitones represents the difference determined in step <NUM>, and mode_change_score_penalty is pre-set equal to '<NUM>' if both songs are in a same key type (in the case of "Yes" in step <NUM>), or is equal to a value of a constant K_mode_change_score, which represents a penalty for requiring a change in key type (in the case of "No" in step <NUM>). In one example embodiment herein, constant K_mode_change_score is equal to a predetermined value, such as, by example and without limitation, <NUM>. Also in formula (F6), and according to one example embodiment herein, K_semitone_change is equal to a predetermined value, such as, by example and without limitation, <NUM>. Which particular value is employed for the variable K_semitone_change depends on how much it is desired to penalize any transpositions that may be required to match both key types (i.e., in the case of "No" in step <NUM>), and can depend on, for example, the quality of a pitch shifting algorithm used, the type (e.g., genre) of music used, the desired musical effect, etc..

According to an example aspect herein, a song mashability score (represented by variable (Ksong[j])) between the query track <NUM>, and each of the candidate tracks <NUM>, can be determined. Reference is now made to <FIG> which shows a procedure <NUM> for determining a song mashability score, with respect to a given jth candidate track <NUM> under consideration. In step <NUM>, an acoustic feature vector distance Ksong(acoustic)[j] is determined, wherein in one example embodiment herein, the acoustic feature vector distance is determined in the manner described above and shown in <FIG> with respect to the jth candidate track <NUM>. In step <NUM>, a determination is made of the likelihood that a segment under consideration includes vocals (in other words, a vocalness likelihood score Ksong(vocalness)[j] is determined), with respect to the jth candidate track <NUM>. In one example embodiment herein, the determination is made in the manner described above and shown in <FIG>. In step <NUM>, a closeness in tempo score (Ksong(tempo)[j]) is determined for tracks under consideration (e.g., the query track <NUM> and the jth candidate track <NUM> under consideration). In one example embodiment herein, that score is determined as described above and represented by formula F6, with respect to the jth candidate track <NUM>. In step <NUM>, a determination is made of a closeness in key score Ksong(key)[j], to measure the closeness of the keys of those tracks <NUM>, <NUM> under consideration. According to one example embodiment herein, step <NUM> is performed as described above and shown in <FIG>, with respect to the jth candidate track <NUM> although this example is not limiting. In step <NUM>, a song mashability score Ksong is determined as the product of the scores determined in steps <NUM> to <NUM>. In particular, the song mashability score Ksong[j], for the query track <NUM> and given candidate track (j), is represented by formula (F7): <MAT>.

In one example embodiment herein, the resulting vector Ksong [j] has Nc components, where Nc corresponds to the number of candidate tracks. Steps <NUM> to <NUM> of procedure <NUM> can be performed with respect to each of the j candidate tracks <NUM> to yield respective scores Ksong [j] for each such track <NUM>. Also in one example embodiment herein, song mashability score Ksong [j] determined for the j candidate tracks <NUM> can be ordered in descending order (in step <NUM>) from greatest score to least score (although in another example, they may be ordered in ascending order, from least score to greatest score).

In one example embodiment herein, to limit the number of tracks that may be employed for mashing up, certain ones of the j candidate tracks <NUM> can be eliminated based on predetermined criteria. As an example, respective mashability scores Ksong [j] determined for respective ones of the j candidate tracks <NUM> can be compared individually to a predetermined threshold value (step <NUM>). If a score is less than the predetermined threshold value ("No" in step <NUM>), then the respective candidate track <NUM> is discarded (step <NUM>). If a score is equal to or greater than the predetermined threshold value ("Yes" in step <NUM>), then the respective candidate track <NUM> under consideration is maintained (selected) in step <NUM> (for eventually being mashed up in step <NUM> of <FIG>). In one example embodiment herein, step <NUM> additionally can include selecting only a predetermined number of the candidate tracks <NUM> for which the predetermined threshold was equaled or exceeded in step <NUM>. By example only, step <NUM> can include selecting the candidate tracks <NUM> having the twenty greatest Ksong [j] scores, for being maintained, and the other tracks <NUM> can be discarded.

Having described the manner in which song mashability is determined according to an example embodiment herein, a procedure for finding a segment, such as, e.g., a candidate (e.g., vocal) segment <NUM>, with high mashability relative to a query track (e.g., an accompaniment track) <NUM> according to another example aspect herein, will now be described, with reference to <FIG>. The procedure, which also is referred to herein as a "segment suggestion procedure <NUM>" and which will be described below in the context of <FIG>, is performed such that, for each of the query segments <NUM> (of the query track <NUM>) assigned to S_subs and S_add (in steps <NUM> and <NUM>, respectively), compatible vocals from candidate tracks <NUM> under consideration are searched for and identified, wherein in one example embodiment herein, the candidate tracks <NUM> are those maintained (selected) in step <NUM> of <FIG> described above. As will be described in detail below, the procedure <NUM> involves determining a segment-wise compatibility score. That is, for each of the segments (S_subs and S_add) <NUM> in the query track <NUM>, respective compatibility scores between the query track segment <NUM> and respective segments <NUM> from corresponding ones of the maintained candidate tracks <NUM> is determined. In one example, the compatibility score ("segment mashability score") is based on "vertical mashability (V)" and a "horizontal mashability (H)". Before describing the segment suggestion procedure <NUM> of <FIG> in detail, vertical mashability and horizontal mashability will first be described.

<FIG> and <FIG> show a procedure <NUM> for determining vertical mashability, according to an example aspect herein. In some examples, steps <NUM>-<NUM> of the procedure <NUM>, described herein, can be performed in an order other than the one shown in <FIG> and <FIG>. In other examples, more or less number of steps may be performed than the ones show in <FIG> and <FIG>.

In one example embodiment herein, to enable a vertical mashability score to be calculated, a minimum length of segments (in terms of the number of beats thereof) is first determined in step <NUM>, using the following formula (F8): <MAT> where variable Nbeats represents a minimum length of segments (in terms of number of beats), Nvoc represents the number of beats of the candidate (e.g., vocal) segment <NUM> under consideration, and variable Nacc represents the number of beats of the query segment <NUM> under consideration from the query track <NUM>. In the initial performance of step <NUM>, the segments under consideration include a first query segment <NUM> of the query track <NUM> and a first candidate segment <NUM> of the candidate track <NUM> under consideration.

In a next step <NUM>, a tempo compatibility between the candidate segment <NUM> and the query segment <NUM> is determined (in one example, the closer the tempo, the higher is a tempo compatibility score K_seg_tempo, to be described below). In one example embodiment herein, step <NUM> can be performed according to procedure <NUM> shown in <FIG>. In step <NUM>, inter-beat distances (in seconds) in each respective segment <NUM>, <NUM> are determined. Inter-beat distances can be derived as the difference between consecutive beat positions. In step <NUM>, the respective determined inter-beat distances are multiplied by a predetermined value (e.g., <NUM>/<NUM>, such as to convert from inter-beat distances in seconds to tempi in beats-per-minute), to produce resulting vectors of values representing time-varying tempi of the respective segment <NUM>, <NUM> (i.e., a time-varying tempo of segment <NUM>, and a time-varying tempo <NUM> of segment <NUM>). Then, in step <NUM> the median value of the vector (from step <NUM>) is determined for each respective segment <NUM>, <NUM>, to obtain a single tempo value for the respective segment <NUM>. Then, a tempo compatibility score K_seg_tempo is determined in step <NUM> according to the following formula (F9): <MAT> where K_seg_tempo represents the tempo compatibility score, min_score represents a predetermined minimum value for that score (e.g., <NUM>), tempo_candidate represents the tempo value obtained for the candidate segment <NUM> in step <NUM>, tempo_query represents the tempo value obtained for the query segment <NUM> in step <NUM>, and K is a value to control a penalty due to tempo differences. K is a predetermined constant, (e.g. <NUM>). The higher the value of K, the lower the score. In other words, it is more important that the query and candidate have similar tempi. It is noted that, the closer the tempi of the segments <NUM>, <NUM> are, the greater is the score.

Referring again to <FIG>, after tempo compatibility (e.g., score K_seg_tempo) is determined in step <NUM>, harmonic progression compatibility (also referred to herein as "harmonic compatibility") is determined in step <NUM>. In one example embodiment herein, the closer the harmonic compatibility of segments <NUM>, <NUM> under consideration, the higher is the score. Also, in one example embodiment herein, step <NUM> can be performed according to procedure <NUM>' shown in <FIG>. In step <NUM>' beat synchronized chroma feature vectors are determined for each of the query segment <NUM> and candidate segment <NUM> under consideration, by determining, for each respective segment <NUM>, <NUM>, an average of chroma values within each beat of the respective segment <NUM>, <NUM>. In one example embodiment herein, the chroma values are obtained from among the information <NUM> in the database using methods described in the Jehan reference discussed above. In step <NUM>' a Pearson correlation between the beat synchronized chroma feature vectors determined in step <NUM>', is determined for each of the beats of the segments under consideration. For example, the segments may include a segment of the query track (chroma values taken only from the accompaniment), and one segment of the candidate track underanalysis (only computing chroma values of the vocal part). In step <NUM>' a median value (med_corr) of vectors of beat-wise correlations determined in step <NUM>' is calculated. Then, in step <NUM>' a harmonic (progression) compatibility score (K_seg_harm_prog) is determined using formula (F10) below, according to an example embodiment herein: <MAT> wherein K_seg_harm_prog represents the harmonic compatibility score, and med_corr represents the median value determined in step <NUM>'.

Another factor involved in vertical mashability is normalized loudness compatibility. Referring again to <FIG>, before, or after or in parallel with when harmonic progression compatibility is determined in step <NUM>, normalized loudness compatibility is determined in step <NUM>. In one example embodiment herein, the closer the normalized loudness of query and candidate segments <NUM>, <NUM>, the higher is a loudness compatibility score. In one example embodiment herein, the loudness compatibility score is determined in step <NUM> according to procedure <NUM> of <FIG>. In steps <NUM> to <NUM>, a determination is made of the relative loudness of the query and target segments <NUM>, <NUM> within the complete tracks. More particularly, for each of the query segment <NUM> and the candidate segment <NUM> under consideration, a loudness of each of the beats of the respective segment is determined (step <NUM>), wherein the loudness, in one example embodiment, may be obtained from among the information <NUM> stored in the database. The determined loudness of each segment <NUM>, <NUM> is divided by a maximum loudness of any beat in the corresponding track (i.e., the query track <NUM> or candidate track <NUM>, respectively), to obtain a vector of size Nbeats for the segment, where Nbeats corresponds to the number of beats in the segment (step <NUM>). Then, for each vector determined in step <NUM>, a median value of the vector is determined in step <NUM> (as a "median normalized loudness"). The median value determined for the query segment <NUM> in step <NUM> is referred to as "query_loudness", and the median value determined for the candidate segment <NUM> in step <NUM> is referred to as a "target_loudness". In step <NUM> a normalized loudness compatibility score, represented by K_seg_norm_loudness, is determined according to the following formula (F11): <MAT> where K_seg_norm_loudness represents the normalized loudness compatibility score, target_loudness represents a loudness of the candidate (target) segment <NUM> (as determined in step <NUM>), and query_loudness represents a loudness of the query segment <NUM> (as also determined in step <NUM>).

Another factor involved in vertical mashability is vocal activity detection on the segment of the candidate (e.g., vocal) track <NUM> under consideration. Referring again to <FIG>, after the normalized loudness compatibility score is determined in step <NUM>, vocal activity detection is performed in step <NUM> for the candidate track <NUM>. In one example embodiment herein, a higher vocal activity in a segment results in a higher vocal activity score. In the present example embodiment, K_seg_vad represents a mean normalized loudness of beats of the candidate track <NUM>. The relationship between K_seg_vad and vertical mashability is described in further detail in formula F17 below. In another example embodiment herein, a voice activity detector can be employed to address possible errors in vocal source separation.

Beat-stability can be another factor involved in vertical mashability. Beat-stability, for a candidate segment <NUM>, is the stability of beat duration in a candidate segment <NUM> under consideration, wherein, in one example embodiment herein, a greater beat stability results in a higher score. Beat stability is determined in step <NUM> of <FIG>. Step <NUM> is preferably performed according to procedure <NUM> of <FIG>. In step <NUM>, a relative change between durations of consecutive beats in the candidate segment <NUM> is determined, according to the following formula (F12): <MAT> where i corresponds to the index of a beat, and delta_rel[i] is a vector representing a relative change between durations of consecutive beats in the candidate segment <NUM> under consideration. In one example embodiment herein, "dur" represents a duration, the vector (delta_rel[i]) has a size represented by (Nbeats - <NUM>), and formula (F12) provides a maximum value.

In step <NUM>, a beat stability score, K_seg_beat_stab, is determined according to the following formula (F13): <MAT>.

Another factor involved in vertical mashability is harmonic change balance, which measures if there is a balance in a rate of change in time of harmonic content (chroma vectors) of both query and candidate (target) segments <NUM>, <NUM>. Briefly, if musical notes change often in one of the tracks (either query or candidate), the score is higher when the other track is more stable, and vice versa.

Harmonic change balance is determined in step <NUM> of <FIG>, which is connected to <FIG> via connector B. Details of how harmonic change balance is determined, according to one example embodiment herein, are shown in procedure <NUM>' of <FIG>. In step <NUM>' a length of the segments <NUM>, <NUM> under consideration is restricted to that of one of the segments <NUM>, <NUM> with a minimal amount of beats (Nbeats) (i.e., either the query segment <NUM> or the candidate segment <NUM>). Next, a harmonic change rate between consecutive beats is determined, for each of the query track <NUM> and candidate track <NUM> under consideration, as follows. A Pearson correlation between consecutive beat-synchronised chroma vectors is determined, for all beats of each track <NUM>, <NUM> (step <NUM>'), to provide a vector (Nbeats - <NUM>) of correlation values. In step <NUM>', the correlation is mapped to change rate values according to formula (F13): <MAT>.

As a result, a vector is obtained with (Nbeats - <NUM>) change rate values for both candidate and query tracks, <NUM>, <NUM>, wherein the change rate value for the candidate (e.g., vocal) track <NUM> is represented by "CRvoc", and the change rate value for the query (accompaniment) track <NUM> is represented by "CRacc".

A Harmonic Change Balance (HCB) vector is then determined in step <NUM>' according to the following formula (F15): <MAT> where HCB[i] represents a Harmonic Change balance, value [i] corresponds to each element of the change rate vectors, CRvoc is the change rate value for the candidate (e.g., vocal) track <NUM>, and CRacc is the change rate value for the query track <NUM>.

A Harmonic change balance score (K_harm_change_bal) is then determined in step <NUM>' according to the following formula (F16): <MAT>.

Another factor involved in vertical mashability is segment length. In one example embodiment herein, the closer the lengths of the query and candidate segments <NUM>, <NUM> (measured in beats) are to each other, then the greater is a segment length score K_len. Segment length is measured in step <NUM> of <FIG> by a segment length score (K_len), which is determined according to the following formula (F17): <MAT> wherein K_len represents the segment length score, Nvoc represents a length of a candidate segment <NUM> under consideration, and Nacc represents a length of a query segment <NUM> under consideration.

According to an example embodiment herein, vertical mashability is measured by a vertical mashability score (V), which is determined as the product of all the foregoing types of scores involved with determining vertical mashability. According to one example embodiment herein, the vertical mashability score (V) is determined according to the following formula (F18), in step <NUM>: <MAT> where the symbol ^ represents a power operator, the term W_seg_harm_prog represents a weight for the score K_seg_harm_prog, the term W_seg_tempo represents a weight for the score K_seg_tempo, the term W_seg_vad represents a weight for the term K_seg_vad, the term W_seg_beat_stab represents a weight for the term K_seg beat_stab, the term W_harm_change_bal represents a weight for the term K_harm_change_bal, and the term W_len represents a weight for the term K_len.

The weights enable control of the impact or importance of each of the mentioned scores in the calculation of the overall vertical mashability score (V). In one example embodiment herein, one or more of the weights have a predetermined value, such as, e.g., '<NUM>'. Weights of lower value result in the applicable related score having a lesser impact or importance on the overall vertical mashability score, relative to weights having higher scores, and vice versa.

Horizontal mashability will now be described in detail. A horizontal mashability score (H) considers a closeness between consecutive tracks. In one example embodiment, to determine horizontal mashability, tracks from which vocals may be employed (i.e., candidate tracks <NUM>) for a mashup are considered.

To determine horizontal mashability, a distance is computed between the acoustic feature vectors of the candidate track <NUM> whose segment <NUM> is a current candidate and a segment <NUM> (if any) that was previously selected as a best candidate for a mashup. The smaller the distance, the higher is the horizontal mashability score. Determining horizontal mashability also involves considering a repetition of the selected segment <NUM>. <FIG> represents acoustic feature vector determination and repetitions, used to determine horizontal mashability.

In one example embodiment herein, an acoustic feature vector distance is determined according to procedure <NUM> of <FIG>. In step <NUM>, the acoustic feature vector of the candidate track <NUM> from which a current segment i under consideration (a selected segment) is determined, without separation (selected-mix_ac). The acoustic feature vector of the candidate track is computed from the acoustic vector of the selected song for vocal segment i. In step <NUM>, a cosine distance between selected-mix_ac and all acoustic feature vectors of candidate tracks <NUM> for segment i+<NUM> is determined. In one example embodiment herein, step <NUM> determines a respective vector of acoustic feature vector distances between the query track <NUM> and each candidate track <NUM>, using a predetermined algorithm.

A next step <NUM> includes normalizing the distance vector (from step <NUM>) by its maximum value, to obtain a normalized distance vector (step <NUM>). A final vector of acoustic feature vector distances (Vsegdist) is within the interval [<NUM>,<NUM>].

For a given candidate track <NUM> with index j, formula (F19) is performed in step <NUM> to determine a distance ("difference") between the final vector of acoustic feature vector distances (Vsegdist) and an ideal normalized distance: <MAT> where Vsegdist[j] is the final vector of acoustic feature vector distances (determined in step <NUM>), and "ideal_norm_distance" is the ideal normalized distance. In one example embodiment herein, the ideal normalized distance ideal_norm_distance can be predetermined, and, in one example, is zero ('<NUM>'), to provide a higher score for acoustically similar tracks (to allow smooth transitions between vocals in terms of style/genre).

A value of K_horiz_ac is then determined in step <NUM> according to the following formula (F20): <MAT> where K-horiz_ac represents a horizontal acoustic distance score of the candidate track <NUM> with index j.

The manner in which the number of repetitions of a given segment <NUM> is determined (e.g., to favor changing between vocals of different tracks/segments), will now be described with reference to the procedure <NUM> of <FIG>. For a given candidate segment <NUM> under consideration, in step <NUM> a determination is made of the number of times the specific segment <NUM> of a candidate track <NUM> has already been previously selected as the best candidate in searches of candidate segments <NUM> (e.g., vocal segments) for being mixed with previously considered query segments <NUM>, wherein the number is represented by "num_repet". Then, in step <NUM> a value for a number of repetitions (K_repet) of the candidate segment <NUM> under consideration is determined according to the following formula (F21): <MAT> where, as described above, num_repet is equal to the number of times the specific segment <NUM> has already been previously selected as the best candidate in searches of candidate segments <NUM> (e.g., vocal segments) for being mixed with previously considered query segments <NUM>.

A procedure <NUM> for determining a horizontal mashability score according to an example aspect herein will now be described, with reference to <FIG>. Since a search for compatible vocals is performed sequentially(i.e., segment-wise) in one example embodiment herein, a first segment <NUM> under consideration is assigned a horizontal mashability score H equal to '<NUM>' (step <NUM>). For each of additional following segment searches, a horizontal mashability score is determined between the given candidate segment <NUM> (under consideration) of a candidate track <NUM>, and a previously selected candidate segment <NUM> (a segment <NUM> previously determined as a best candidate for being mixed with previous query segments <NUM>), as will now be described. For example, in step <NUM>, for the given segment <NUM> under consideration, a determination is made of a horizontal acoustic feature vector distance score K_horiz_ac for the segment <NUM>. In one example embodiment herein, step <NUM> is performed according to procedure <NUM> of <FIG> described above. In a next step <NUM>, a determination is made of a repetition score K_repet for the segment. In one example embodiment herein, step <NUM> is performed according to procedure <NUM> of <FIG> described above. Then, in step <NUM>, a horizontal mashability score H is determined according to the following formula (F22): <MAT> where H represents the horizontal mashability score, and W_horiz_ac and W_repet are weights that allow control of an importance or impact of respective scores K_horiz_ac and K_repet in the determination of value H. In one example embodiment herein, W_horiz_ac = W_repet = <NUM> by default.

Referring now to <FIG>, a procedure <NUM> for determining a mashability score (M) for each candidate segment <NUM> will now be described. In step <NUM> a key distance score (Ksong(key)) is determined, wherein in one example embodiment herein, step <NUM> is performed according to procedure <NUM> of <FIG>. In step <NUM> a normalized distance in tracks' acoustic feature vector (Ksong(acoustic)) is determined, wherein in one example embodiment herein, step <NUM> is performed according to procedure <NUM> of <FIG>. In step <NUM>, a vertical mashability score V for the segment <NUM> is determined, wherein in one example embodiment herein, step <NUM> is performed according to procedure <NUM> of <FIG> and <FIG>. In step <NUM>, a horizontal mashability score H for the segment <NUM> is determined, wherein in one example embodiment herein, step <NUM> is performed according to procedure <NUM> of <FIG>. In step <NUM>, a total mashability score M[j] is determined according to the following formula (F23): <MAT> where M[j] represents the total mashability score for a jth segment <NUM> under consideration, Ksong(key)[j] represents the key distance score for the segment <NUM>, Ksong(acoustic)[j] represents the acoustic feature vector calculated for the segment <NUM>, V[j] represents the vertical mashability score for the segment <NUM>, and H[j] represents the horizontal mashability score H for the segment <NUM>. Steps <NUM> to <NUM> can be performed for each segment <NUM> of candidate track(s) <NUM> under consideration.

After computing the score (M) for all segments <NUM> of all candidate tracks <NUM> under consideration, the segment <NUM> with the highest total mashability score (M) is selected (step <NUM>), although in other example embodiments, a sampling between all possible candidate segments can be done with a probability which is proportional to their total mashability score. The above procedure can be performed with respect to all segments <NUM> that were assigned to S-subs and S_add of the query track <NUM> under consideration, starting from the start of the track <NUM> and finishing at the end of the track <NUM>, to determine mashability between those segments <NUM> and individual ones of the candidate segments <NUM> of candidate tracks <NUM> that were selected as being compatible with the query track <NUM>.

As described above with respect to the procedure <NUM> of <FIG>, in step <NUM>, beat and downbeat alignment is performed for a segment <NUM> under consideration (a segment <NUM> assigned to S_subs or S_add) and a candidate (e.g., vocal) segment(s) <NUM> determined to be compatible with the segment <NUM> in step <NUM>. Also, in step <NUM>, transition refinement is performed for the segment <NUM> under consideration and/or the candidate segment(s) <NUM> aligned in step <NUM>, wherein each step <NUM> and <NUM> may be performed based on, for example, segmentation information, beat and downbeat information, and/or voicing information, such as that stored among information <NUM> in association with the corresponding tracks <NUM>, <NUM> and/or segments <NUM>, <NUM> in the database. Then, in step <NUM>, those segments <NUM>, <NUM> are mixed. The manner in which those steps <NUM>, <NUM>, and <NUM> are performed according to one example embodiment herein, will now be described in greater detail.

Alignment in step <NUM> of procedure <NUM> involves properly aligning the candidate (e.g., vocal) segment <NUM> with the segment <NUM> under consideration from the query track <NUM> to ensure that, once mixing occurs, the mixed segments sound good together. As an example, if a beat of the candidate segment <NUM> is not aligned properly with a corresponding beat of the segment <NUM>, then a mashup of those segments would not sound good together and would not be in an acceptable musical time. Proper alignment according to an example aspect herein avoids or substantially minimizes that possibility.

Also by example, another factor taken into consideration is musical phrasing. If the candidate segment <NUM> starts or ends in the middle of a musical phrase, then a mashup would sound incomplete. Take for example a song like "I Will Always Love You," by Céline Dion. If a mashup were to select a candidate (e.g., vocal) segment that starts in the middle of the vocal phrase "I will always love you," (e.g., at ". ays love you" and cut off "I will alw. "), then the result would sound incomplete. Thus, in one example embodiment herein it is desired to analyze vocal content of the candidate segment <NUM> to determine whether the vocal content is present at the starting or ending boundary of the segment <NUM>, and, if so, to attempt to shift the starting and/or ending boundaries to the start or end of the musical phrase so as to not cut the musical phrase off in the middle of the musical phrase.

In one example embodiment herein, segment refinement in step <NUM> is performed according to procedure <NUM> of <FIG>. First, preliminary segment boundaries (including a starting and ending boundary) are identified for a segment <NUM> of a candidate track <NUM> (step <NUM>). The start and ending boundaries are then analyzed to determine a closest downbeat temporal location thereto (step <NUM>). In one example embodiment herein, steps <NUM> and <NUM> are performed based on segmentation information, beat and downbeat information, and/or voicing information (such as that stored among information <NUM>) for the query track <NUM> under consideration. Next, in step <NUM>, a preliminary segment boundary (e.g., one of the starting and ending boundaries) that varies from the downbeat temporal location is corrected temporally to match the downbeat location temporally (step <NUM>). <FIG> represents start and ending boundaries <NUM>, <NUM> identified in step <NUM>, a closest downbeat location <NUM> identified in step <NUM>, and variation of boundary <NUM> to a corrected position <NUM> matching the downbeat location <NUM> in step <NUM>.

Vocal activity in the candidate track <NUM> is then analyzed over a predetermined number of downbeats around the downbeat location (e.g., <NUM> beats, either before or after the location in time) (step <NUM>), based on the beat and downbeat information, and voicing information. For a preliminary starting boundary of the candidate (e.g., vocal) segment <NUM>, a search is performed (step <NUM>) for the first beat in the candidate track before that segment boundary in which the likelihood of containing vocals is lower than a predetermined threshold (e.g., <NUM>, on a scale from <NUM> to <NUM>, where <NUM> represents full confidence that there are not vocals at that downbeat and <NUM> represents full confidence that there are vocals at that downbeat). The first downbeat before the starting boundary that meets that criteria is selected as the final starting boundary for the candidate segment <NUM> (step <NUM>). This is helpful to avoid cutting a melodic phrase at the start of the candidate segment <NUM>, and alignment between candidate and query segments <NUM>, <NUM> is maintained based on the refined downbeat location. Similarly, for the ending boundary of the candidate segment <NUM>, a search is performed (step <NUM>) for the first beat in the candidate track after the segment boundary in which the likelihood of containing vocals is lower than the threshold (e.g., <NUM>), and that downbeat is selected as the final ending boundary of the candidate segment <NUM> (step <NUM>). This also is helpful to avoid cutting a melodic phrase at the end of the segment <NUM>.

As such, by virtue of procedure <NUM>, the boundaries of the candidate segment <NUM> are adjusted so that the starting and ending boundaries of a segment are aligned with a corresponding downbeat, and the starting and ending boundaries can be positioned before or after a musical phrase of vocal content (e.g., at a point in which there are not vocals). The procedure <NUM> can be performed for more than one candidate track <NUM> with respect to the query track <NUM> under consideration, including for all segments selected (even segments from different songs) as being compatible.

It is helpful to align the starting and ending boundaries with the downbeats. For example, if the corresponding insertion point of the instrumentals is also selected at a downbeat (of the instrumentals), then, when the two are put together by aligning the starting boundary of the vocals with the insertion point of the instrumentals, the beats will automatically also be aligned.

As described above, in procedure <NUM> of <FIG>, segments <NUM>, <NUM> are mixed. According to an example embodiment herein, mixing is performed based on various types of parameters, such as, by example and without limitation, (<NUM>) a time-stretching ratio: determined for each beat as a ratio between lengths of each of the beats in both tracks <NUM>, <NUM>; (<NUM>) a pitchshifting ratio: an optimal ratio, relating to an optimal transposition to match keys of the tracks; (<NUM>) a gain (in dB) to be applied to vocal content; and (<NUM>) transitions.

<FIG> shows a procedure <NUM> for mixing segments <NUM>, <NUM>, and can be performed as part of step <NUM> described above. The procedure <NUM> includes cutting the candidate (e.g., vocal) segments <NUM> from each of the candidate tracks <NUM>, based on the refined/aligned boundaries determined in procedure <NUM> (step <NUM>). A next step includes applying one or more gains to corresponding candidate (e.g., vocal) segments <NUM> (step <NUM>).

The particular gain (in dB) that is applied to a segment in step <NUM> can depend on the type of the segment, according to an example embodiment herein. Preferably, for query segments <NUM> that have been assigned to S_keep, the original loudness thereof is maintained (i.e., the gain = <NUM>). For segments <NUM> assigned to S_subs and S_add, on the other hand, a loudness of beats of the tracks <NUM>, <NUM> is employed and a heuristically determined value is used for a gain (in dB). <FIG> shows a procedure <NUM> for determining a gain for segments <NUM> to be used in place of or to be added to query segments <NUM> assigned to S_subs and S_add, respectively. In step <NUM> a loudness of each beat of tracks <NUM>, <NUM> is determined, based on, for example, information <NUM>, wherein the loudness of each beat is determined as the mean loudness over the duration of the beat, in one example embodiment herein. Then, in step <NUM>, a determination is made of a median loudness (in dB) of each of the beats of the candidate segment <NUM> of the candidate track <NUM>, wherein the median is represented by variable Lvocal. In step <NUM> a determination is made of a maximum loudness (in dB) of each of the beats of the candidate segment <NUM> of the track <NUM>, wherein the maximum loudness is represented by variable MaxLvocal. Then, in step <NUM> a determination is made of a median loudness (in dB) of each beat of the segment <NUM>, based on the query track <NUM>, wherein that median loudness is represented by variable Laccomp. The determination is based on the separation of the vocals from the accompaniment track as seen <NUM> from <FIG>. In step <NUM> a determination is made of the gain to be applied to the particular segment <NUM>, based on the following formula (F24): <MAT>.

As a result of the "Gain" being determined for a particular candidate segment <NUM> (to be used in place of or to be added to a query segment <NUM> assigned to S_subs or S_add, respectively, in step <NUM>), that Gain is applied to the segment <NUM> in step <NUM>.

After step <NUM>, time-stretching is performed in step <NUM>. Preferably, time-stretching is performed to each beat of respective candidate (e.g., vocal) tracks <NUM> so that they conform to beats of the query track <NUM> under consideration, based on a time-stretching ratio (step <NUM>). In one example embodiment herein, the time-stretching ratio is determined according to procedure <NUM> of <FIG>. In step <NUM> of procedure <NUM>, lengths of beats of the tracks <NUM>, <NUM> under consideration are determined, based on, for example, information <NUM>. Then, in step <NUM>, for each beat of track <NUM>, a time-stretching ratio is determined as a ratio of the length of that beat to the length of the corresponding beat of candidate track <NUM>. Thus, in step <NUM> of procedure <NUM>, for each beat of the candidate track <NUM>, the length of the beat is varied based on the corresponding ratio determined for that beat in step <NUM>.

Step <NUM> includes performing pitch shifting to each candidate (e.g., vocal) segment <NUM>, as needed, based on a pitch-shifting ratio. In some embodiments, the pitch-shifting ratio is computed while computing the mashability scores discussed above. For example, the vocals are pitch-shifted by n_semitones, where n_semitones is the number of semitones. In some embodiments, the number of semitones is determined during example step <NUM> discussed in reference to <FIG>.

Then, the procedure <NUM> can include applying fade-in and fade-out, and/or high pass filtering or equalizations around transition points, using determined transitions (step <NUM>). In one example embodiment herein, the parts of each segment <NUM> (of a candidate track <NUM> under consideration) which are located temporally before initial and after the final points of the refined boundaries (i.e., transitions), can be rendered with a volume fade in, and a fade out, respectively, so as to perform a smooth intro and outro, and reduce clashes between vocals of different tracks. Fade in and Fade out can be performed in a manner known in the art. In another example embodiment herein, instead of performing a fade in step <NUM>, low pass filtering can be performed with a filter cutoff point that descends from, by example, <NUM>, at a transition position until <NUM> at the section initial boundary, in a logarithmic scale (i.e., where no filtering is performed at the boundary). Similarly, instead of performing a fade out in step <NUM>, a low pass filtering can be performed, with an increasing cutoff frequency, from, by example, <NUM> to <NUM>, in logarithmic scale. Depending on the length of the transition (which depends on the refinement to avoid cutting vocal phrases), a faster or slower fade in or fade out can be provided (i.e., the longer the transition the slower the fade in or fade out). In some embodiments, the transition zone is the zone between the refined boundary using vocal activity detection and the boundary refined only with downbeat positions.

Referring again to <FIG>, after step <NUM> is performed, the segment(s) <NUM> to which steps <NUM> to <NUM> were performed are mixed (i.e., summed) with the corresponding segment(s) <NUM> of the query track <NUM> under consideration. By example, in a case where a segment <NUM> was previously assigned to S_subs, mixing can include replacing vocal content of that segment <NUM>, with vocal content of the corresponding candidate segment <NUM> to which steps <NUM> to <NUM> were performed. Also by example, in a case where the segment <NUM> was previously assigned to S_add, mixing can include adding vocal content of the segment <NUM> to which steps <NUM> to <NUM> were performed, to the segment <NUM>.

Another example aspect herein will now be described. In accordance with this example aspect, an automashup can be personalized based on a user's personal taste profile. For example, users are more likely to enjoy mashups created from songs the users know and like. Accordingly, the present example aspect enables auto-mashups to be personalized to individual users' taste profiles. Also in accordance with this example aspect, depending on the application of interest, there may not be enough servers available to be able to adequately examine how every track might mash up with every other track, particularly in situations where a catalog many (e.g., millions) of tracks is involved. The present example aspect reduces the number of tracks that are searched for and considered/examined for possible mash-ups, thereby alleviating the number of servers and processing power required to perform mash-ups.

A procedure <NUM> according to the present example aspect will now be described, with reference to the flow diagram shown in <FIG>. In step <NUM>, a determination is made of a predetermined number P1 (e.g., <NUM>) of most liked mixed, original tracks of at least some users of a mashup system herein, such as computation system <NUM> to be described below. For example, the determination may be made with respect to all users of the system, with respect to only a certain set of users, with respect to only specific, predetermined users, and/or with respect to only users who prescribe to a specific service provided by the system. In one example embodiment herein, the determination in step <NUM> is performed for each such user (i.e., for each such user, the predetermined number P1 of the user's most liked mixed, original tracks is determined). Also, in one example embodiment herein, the determination can be made by analyzing the listening histories of the users or user musical taste profiles.

Next, in step <NUM>, tracks that were determined in step <NUM> are added to a set S1. In some example embodiments herein, there may be one set S1 for each user, or, in other example embodiments, there may be a single set S1 that includes all user tracks that were determined in step <NUM>. In the latter case, where there is overlap of tracks, only a single version of the track is included in the set S1, thereby reducing the number of tracks.

Then, in step <NUM>, audio analysis algorithms are performed to the tracks from set S1, and the resulting output(s) are stored as information <NUM> in the database. In one example embodiment herein, the audio analysis performed in step <NUM> includes determining the various types of information <NUM> in the manner described above. By example only and without limitation, step <NUM> may include separating components (e.g., vocal, instrumental, and the like) from the tracks, determining segmentation information based on the tracks, determining segment labelling information, performing track segmentation, determining the tempo(s) of the tracks, determining beat/downbeat positions in the tracks, determining the tonality of the tracks, determining information about the presence of vocals (if any) in time in each track, determining energy of each of the segments in the vocal and accompaniment tracks, determining acoustic feature vector information and loudness information (e.g., amplitude) associated with the tracks, and/or the like. In at least some cases, algorithms performed to determine at least some of the foregoing types of information can be expensive to run and may require a high level of processing power and resources. However, according to an example aspect herein, by reducing the total available number of tracks to only those included in the set S1, a reduction of costs, processing power, and resources can be achieved.

For each user for which the determination in step <NUM> originally was made, a further determination is made in step <NUM>, of a predetermined number P2 (e.g., the top <NUM>) of the respective user's most liked mixed, original tracks. In one example embodiment herein, the determination in step <NUM> can be made by making affinity determinations for the respective users, in the above-described manner. Next, in step <NUM>, tracks that were determined in step <NUM> are added to a set S2, wherein, in one example embodiment herein, there is set S2 for each user (although in other example embodiments, there may be a single set S2 that includes all user tracks that were determined in step <NUM>).

Then, in step <NUM> an intersection of the tracks from the sets S1 and S2 is determined. In one example embodiment herein, step <NUM> is performed to identify which tracks appear in both sets S1 and S2. According to an example embodiment herein, in a case where set S1 includes tracks determined in step <NUM> for all users, and where each set S2 includes tracks determined in step <NUM> for a respective one of the users, then step <NUM> determines the intersection between tracks that are in the set S1 and the set S2, and is performed for each set S2 vis-a-vis the set S1. In an illustrative, non-limiting example in which the predetermined numbers P1 and P2 are <NUM> and <NUM>, respectively, the performance of step <NUM> results in there being between <NUM> and <NUM> tracks being identified per user in step <NUM>. The identified tracks for each respective user are then assigned to a corresponding set SU (step <NUM>).

In another example embodiment herein, step <NUM> is performed based on multiple users. By example and without limitation, referring to <FIG>, it is assumed that a top predetermined number (e.g., two) of tracks <NUM> are identified among mixed, original tracks <NUM>-<NUM>, from a set <NUM> associated with a user A (i.e., where the tracks <NUM>-<NUM> were identified as those for which User A has an affinity), and that a top predetermined number (e.g., two) of tracks <NUM> are identified among mixed, original tracks <NUM>-<NUM> from a set <NUM> associated with a user B (i.e., where the tracks <NUM>-<NUM> were identified as those for which User B has an affinity). In such an example case, step <NUM> is performed to identify <NUM> those tracks from the sets <NUM> and <NUM> that intersect or overlap (e.g., track <NUM>) with one another, and to include the intersecting track in a set <NUM>. In one example embodiment herein, step <NUM> also comprises including the tracks <NUM> (e.g., tracks <NUM>-<NUM>) from set <NUM> (e.g., tracks <NUM>-<NUM>) and a non-overlapping one (e.g., track <NUM>) among the tracks <NUM> from set <NUM>, in set <NUM>, wherein as represented in <FIG>, track <NUM>, track <NUM>, and track <NUM> are shown in set <NUM> in association with user A and track <NUM> and track <NUM> are shown in association with user B. Also in one example embodiment herein, the set <NUM> may represent set SU.

Referring again to <FIG>, in one example aspect herein, a next step <NUM> is performed by providing each track in the set SU (or per-user set SU) to a waveform generation algorithm that generates a waveform based on at least one of the tracks, and/or to the song suggester algorithm described above. By example, a particular track from the set SU can be employed as the query track <NUM> in procedure <NUM> (<FIG>) described above, and at least some other ones of the tracks from the set SU can be employed as the candidate tracks <NUM>. In some example embodiments herein, each track of the set SU can be employed as a query track <NUM> in separate, respective iterations of the procedure <NUM>, and other ones of the tracks from the set SU can be employed as corresponding candidate tracks <NUM> in such iterations. In another example embodiment herein, only those tracks of set SU that are not provided to the waveform generation algorithm, and which are associated with a particular user, are employed for use in the song suggester algorithm of procedure <NUM>, resulting in a set of mashups of size |SU| mashing up various combinations of a particular user's most popular tracks.

In some example embodiments herein, the results of more than one user's affinity determinations (in procedure <NUM>) can be employed as mashup candidates, and musical compatibility determinations and possible resulting mashups can be performed for those tracks as well in the above-described manner, whether some tracks overlap across users or not. In still another example, only tracks for which a predetermined number of users are determined to have an affinity are employed in the musical compatibility determinations and possible mashups. In still another example where more than one user's affinity determinations are employed as mashup candidates, the intersection between those results and each user's full collection of tracks is determined and employed and the intersecting tracks are employed in musical compatibility determinations and possible mashups. At least some of the results of the intersection also can be employed to generate a waveform.

By virtue of the above procedure <NUM>, the number of tracks that are searched for and considered/examined for possible mash-ups can be reduced based on user profile(s), thereby alleviating the number of servers and processing power required to perform mash-ups.

In accordance with another example embodiment herein, a collage can be created of images (e.g., album cover art) associated with musical tracks that are employed in a "mashup" of songs. In one example embodiment herein, each pixel of the collage is an album cover image associated with a corresponding musical track employed in a mashup, and the overall collage forms a profile photo of the user. A process according to this example aspect can include downloading a user's profile picture, and album art associated with various audio tracks, such as those used in mashups personalized for the user. Next, a resize is performed of every album art image to a single pixel. A next step includes obtaining the color (e.g., average color) of that pixel and placing it in a map of colors to the images they are associated with. This gives the dominant color of each piece of album art. Next steps include cropping the profile picture into a series of 20x20 pixels, and then performing a resize to one pixel on each of these cropped pictures, and then finding a nearest color in the map of album art colors. A next step includes replacing the cropped part of the picture with the album art resized to, by example only, 20x20 pixels. As a result, a collage of the album art images is provided, and, in one example embodiment herein, the collage forms a profile image of the user.

According to still another example embodiment herein, titles are formulated based on titles of songs that are mashed up. That is, titles of mashed up tracks are combined in order to create a new title that includes at least some words from the titles of the mashed up tracks. Prior to being combined, the words from each track title are categorized into different parts of speech using Natural Language Processing, such as by, for example, the Natural Language Toolkit (NLTK), which is a known collection of libraries and tools for natural language processing in Python. A custom derivation tree determines word order so that the combined track names are syntactically correct. Various possible combinations of words forming respective titles can be provided. In one example embodiment herein, out of all the possible combinations, the top <NUM>% are selected based on length. The final track name is then randomly chosen from the <NUM>%. The track names can then be uploaded to a data storage system (e.g., such as BigTable), along with other metadata for each track. From the data storage system, the track names can be retrieved and served in real-time along with the corresponding song mashups. In an illustrative example, the following (four) track titles T are employed as inputs: T = {Shine on Me, I Feel Fantastic, Rolling Down the Hill, Wish You Were Here}. An algorithm according to an example embodiment herein selects the following words W from those titles T: W = {shine, feel, fantastic, rolling, down, hill, wish, you, were, here}. Based on those words, the following possible combined titles are generated: "Wish the Hill," "The Shine was Rolling," and "The Fantastic Shine".

As can be appreciated in view of the above description, at least some example aspects herein employ source separation to generate candidate (e.g., vocal) tracks and query (e.g., accompaniment) tracks, although in other example embodiments, stems can be used instead, or a multitrack can be employed where separation is therefore not needed). In other example embodiments herein, full tracks can be employed (without separation of vocals and accompaniment components).

Also, at least some example aspects herein can determine which segments to keep of an original, mixed track, which ones to replace with content (e.g., vocal content) from other tracks, and which ones to have content from other tracks added thereto. For those segments in which vocals from other songs/tracks are added, it can be determined whether source (e.g., vocal) separation is needed to be performed or not on a query track (e.g., accompaniment track) by using vocal activity detection information, among information <NUM>.

At least some example embodiments herein also employ a song mashability score, using global song features, including, by example only, acoustic features derived from collaborative filtering knowledge. At least some example embodiments herein also employ a segment mashability score, including various types of musical features as described above.

At least some example embodiments herein also at least implicitly use collaborative filtering information (i.e., using acoustic feature vectors for improving recommendations of content (e.g., vocals) to be mixed with query (e.g., instrumental) tracks, and selection of content in contiguous segments. Presumably, the more similar they are, then the more likely it is for them to work well together in a mashup. However, this is a configurable parameter, and, in other examples, users may elect to foster mixes of more different songs, instead of more similar ones.

At least some example aspects herein also employ refinement of transitions between lead (vocal) parts, by using section, downbeat, and vocal activity detection for finding ideal transition points, in order to avoid detrimentally cutting melodic phrases.

<FIG> is a block diagram showing an example computation system <NUM> constructed to realize the functionality of the example embodiments described herein.

The computation system <NUM> may include without limitation a processor device <NUM>, a main memory <NUM>, and an interconnect bus <NUM>. The processor device <NUM> (<NUM>) may include without limitation a single microprocessor, or may include a plurality of microprocessors for configuring the system <NUM> as a multi-processor acoustic attribute computation system. The main memory <NUM> stores, among other things, instructions and/or data for execution by the processor device <NUM>. The main memory <NUM> may include banks of dynamic random access memory (DRAM), as well as cache memory.

The system <NUM> may further include a mass storage device <NUM> (which, in the illustrated embodiment, has LUT <NUM> and stored information <NUM>), peripheral device(s) <NUM>, portable non-transitory storage medium device(s) <NUM>, input control device(s) <NUM>, a graphics subsystem <NUM>, and/or an output display interface <NUM>. A digital signal processor (DSP) <NUM> may also be included to perform audio signal processing. For explanatory purposes, all components in the system <NUM> are shown in <FIG> as being coupled via the bus <NUM>. However, the system <NUM> is not so limited. Elements of the system <NUM> may be coupled via one or more data transport means. For example, the processor device <NUM>, the digital signal processor <NUM> and/or the main memory <NUM> may be coupled via a local microprocessor bus. The mass storage device <NUM>, peripheral device(s) <NUM>, portable storage medium device(s) <NUM>, and/or graphics subsystem <NUM> may be coupled via one or more input/output (I/O) buses. The mass storage device <NUM> may be a nonvolatile storage device for storing data and/or instructions for use by the processor device <NUM>. The mass storage device <NUM> may be implemented, for example, with a magnetic disk drive or an optical disk drive. In a software embodiment, the mass storage device <NUM> is configured for loading contents of the mass storage device <NUM> into the main memory <NUM>.

Mass storage device <NUM> additionally stores a song suggester engine <NUM> that can determine musical compatibility between different musical tracks, a segment suggestion engine <NUM> that can determine musical compatibility between segments of the musical tracks, a combiner engine <NUM> that mixes or mashes up musically compatible tracks and segments, an alignment engine <NUM> that aligns segments to be mixed/mashed up, and a boundary connecting engine <NUM> that refines boundaries of such segments.

The portable storage medium device <NUM> operates in conjunction with a nonvolatile portable storage medium, such as, for example, a solid state drive (SSD), to input and output data and code to and from the system <NUM>. In some embodiments, the software for storing information may be stored on a portable storage medium, and may be inputted into the system <NUM> via the portable storage medium device <NUM>. The peripheral device(s) <NUM> may include any type of computer support device, such as, for example, an input/output (I/O) interface configured to add additional functionality to the system <NUM>. For example, the peripheral device(s) <NUM> may include a network interface card for interfacing the system <NUM> with a network <NUM>.

The input control device(s) <NUM> provide a portion of the user interface for a user of the computer <NUM>. The input control device(s) <NUM> may include a keypad and/or a cursor control device. The keypad may be configured for inputting alphanumeric characters and/or other key information. The cursor control device may include, for example, a handheld controller or mouse, a trackball, a stylus, and/or cursor direction keys. In order to display textual and graphical information, the system <NUM> may include the graphics subsystem <NUM> and the output display <NUM>. The output display <NUM> may include a display such as a CSTN (Color Super Twisted Nematic), TFT (Thin Film Transistor), TFD (Thin Film Diode), OLED (Organic Light-Emitting Diode), AMOLED display (Activematrix Organic Light-emitting Diode), and/or liquid crystal display (LCD)-type displays. The displays can also be touchscreen displays, such as capacitive and resistive-type touchscreen displays. The graphics subsystem <NUM> receives textual and graphical information, and processes the information for output to the output display <NUM>.

<FIG> shows an example of a user interface <NUM>, which can be provided by way of the output display <NUM> of <FIG>, according to a further example aspect herein. The user interface <NUM> includes a play button <NUM> selectable for playing tracks, such as tracks stored in mass storage device <NUM>, for example. Tracks stored in the mass storage device <NUM> may include, by example, tracks having both vocal and non-vocal (instrumental) components (i.e., mixed signals), tracks including only instrumental or vocal components (i.e., instrumental or vocal tracks, respectively), query tracks, candidate tracks, etc..

The user interface <NUM> also includes forward control <NUM> and reverse control <NUM> for scrolling through a track in either respective direction, temporally. According to an example aspect herein, the user interface <NUM> further includes a volume control bar <NUM> having a volume control <NUM> (also referred to herein as a "karaoke slider") that is operable by a user for attenuating the volume of at least one track. By example, assume that the play button <NUM> is selected to playback a song called "Night". According to one non-limiting example aspect herein, when the play button <NUM> is selected, the "mixed" original track of the song, and the corresponding instrumental track of the same song (i.e., wherein the tracks may be identified as being a pair according to procedures described above), are retrieved from the mass storage device <NUM>. As a result, both tracks are simultaneously played back to the user, in synchrony. In a case where the volume control <NUM> is centered at position <NUM> in the volume control bar <NUM>, then, according to one example embodiment herein, the "mixed" original track and instrumental track both play at <NUM>% of a predetermined maximum volume. Adjustment of the volume control <NUM> in either direction along the volume control bar <NUM> enables the volumes of the simultaneously played back tracks to be adjusted in inverse proportion, wherein, according to one example embodiment herein, the more the volume control <NUM> is moved in a leftward direction along the bar <NUM>, the lesser is the volume of the instrumental track and the greater is the volume of the "mixed" original track. For example, when the volume control <NUM> is positioned precisely in the middle between a leftmost end <NUM> and the center <NUM> of the volume control bar <NUM>, then the volume of the "mixed" original track is played back at <NUM>% of the predetermined maximum volume, and the instrumental track is played back at <NUM>% of the predetermined maximum volume. When the volume control <NUM> is positioned all the way to the left end <NUM> of the bar <NUM>, then the volume of the "mixed" original track is played back at <NUM>% of the predetermined maximum volume, and the instrumental track is played back at <NUM>% of the predetermined maximum volume.

Also according to one example embodiment herein, the more the volume control <NUM> is moved in a rightward direction along the bar <NUM>, the greater is the volume of the instrumental track and the lesser is the volume of the "mixed" original track. By example, when the volume control <NUM> is positioned precisely in the middle between the center positon <NUM> and rightmost end <NUM> of the bar <NUM>, then the volume of the "mixed" original track is played back at <NUM>% of the predetermined maximum volume, and the instrumental track is played back at <NUM>% of the predetermined maximum volume. When the volume control <NUM> is positioned all the way to the right along the bar <NUM>, at the rightmost end <NUM>, then the volume of the "mixed" original track is played back at <NUM>% of the predetermined maximum volume, and the instrumental track is played back at <NUM>% of the predetermined maximum volume.

In the above manner, a user can control the proportion of the volume levels between the "mixed" original track and the corresponding instrumental track.

Of course, the above example is non-limiting. By example, according to another example embodiment herein, when the play button <NUM> is selected, the "mixed" original track of the song, as well as the vocal track of the same song (i.e., wherein the tracks may be identified as being a pair according to procedures described above), can be retrieved from the mass storage device <NUM>, wherein, in one example, the vocal track is obtained according to one or more procedures described above, such as that shown in <FIG>, or is otherwise available. As a result, both tracks are simultaneously played back to the user, in synchrony. Adjustment of the volume control <NUM> in either direction along the volume control bar <NUM> enables the volume of the simultaneously played tracks to be adjusted in inverse proportion, wherein, according to one example embodiment herein, the more the volume control <NUM> is moved in a leftward direction along the bar <NUM>, the lesser is the volume of the vocal track and the greater is the volume of the "mixed" original track, and, conversely, the more the volume control <NUM> is moved in a rightward direction along the bar <NUM>, the greater is the volume of the vocal track and the lesser is the volume of the "mixed" original track.

In still another example embodiment herein, when the play button <NUM> is selected to play back a song, the instrumental track of the song, as well as the vocal track of the same song (wherein the tracks are recognized to be a pair) are retrieved from the mass storage device <NUM>. As a result, both tracks are simultaneously played back to the user, in synchrony. Adjustment of the volume control <NUM> in either direction along the volume control bar <NUM> enables the volume of the simultaneously played tracks to be adjusted in inverse proportion, wherein, according to one example embodiment herein, the more the volume control <NUM> is moved in a leftward direction along the bar <NUM>, the lesser is the volume of the vocal track and the greater is the volume of the instrumental track, and, conversely, the more the volume control <NUM> is moved in a rightward direction along the bar <NUM>, the greater is the volume of the vocal track and the lesser is the volume of the instrumental track.

Of course, the above-described directionalities of the volume control <NUM> are merely representative in nature, and, in other example embodiments herein, movement of the volume control <NUM> in a particular direction can control the volumes of the above-described tracks in an opposite manner than those described above, and/or the percentages described above may be different that those described above, in other example embodiments. Also, in one example embodiment herein, which particular type of combination of tracks (i.e., a mixed original signal paired with either a vocal or instrumental track, or paired vocal and instrumental tracks) is employed in the volume control technique described above can be predetermined according to pre-programming in the system <NUM>, or can be specified by the user by operating the user interface <NUM>.

Referring again to <FIG>, the input control devices <NUM> will now be described.

Input control devices <NUM> can control the operation and various functions of system <NUM>.

Input control devices <NUM> can include any components, circuitry, or logic operative to drive the functionality of system <NUM>. For example, input control device(s) <NUM> can include one or more processors acting under the control of an application.

Each component of system <NUM> may represent a broad category of a computer component of a general and/or special purpose computer. Components of the system <NUM> (<NUM>) are not limited to the specific implementations provided herein.

Software embodiments of the examples presented herein may be provided as a computer program product, or software, that may include an article of manufacture on a machine-accessible or machine-readable medium having instructions. The instructions on the non-transitory machine-accessible machine-readable or computer-readable medium may be used to program a computer system or other electronic device. The machine- or computer-readable medium may include, but is not limited to, floppy diskettes, optical disks, and magneto-optical disks or other types of media/machine-readable medium suitable for storing or transmitting electronic instructions. The techniques described herein are not limited to any particular software configuration. They may find applicability in any computing or processing environment. The terms "computer-readable", "machine-accessible medium" or "machine-readable medium" used herein shall include any medium that is capable of storing, encoding, or transmitting a sequence of instructions for execution by the machine and that causes the machine to perform any one of the methods described herein. Furthermore, it is common in the art to speak of software, in one form or another (e.g., program, procedure, process, application, module, unit, logic, and so on), as taking an action or causing a result. Such expressions are merely a shorthand way of stating that the execution of the software by a processing system causes the processor to perform an action to produce a result.

Some embodiments may also be implemented by the preparation of application-specific integrated circuits, field-programmable gate arrays, or by interconnecting an appropriate network of conventional component circuits.

Some embodiments include a computer program product. The computer program product may be a storage medium or media having instructions stored thereon or therein which can be used to control, or cause, a computer to perform any of the procedures of the example embodiments of the invention. The storage medium may include without limitation an optical disc, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory, a flash card, a magnetic card, an optical card, nanosystems, a molecular memory integrated circuit, a RAID, remote data storage/archive/warehousing, and/or any other type of device suitable for storing instructions and/or data.

Stored on any one of the computer-readable medium or media, some implementations include software for controlling both the hardware of the system and for enabling the system or microprocessor to interact with a human user or other mechanism utilizing the results of the example embodiments of the invention. Such software may include without limitation device drivers, operating systems, and user applications. Ultimately, such computer-readable media further include software for performing example aspects of the invention, as described above.

Included in the programming and/or software of the system are software modules for implementing the procedures described herein.

While various example embodiments of the present invention have been described above, it should be understood that they have been presented by way of example, and not limitation. It will be apparent to persons skilled in the relevant art(s) that various changes in form and detail can be made therein. Thus, the present invention should not be limited by any of the above described example embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claim 1:
A method for combining audio tracks, comprising:
determining at least one music track (<NUM>) that is musically compatible with a base music track (<NUM>);
aligning the at least one music track (<NUM>) and the base music track (<NUM>) in time;
separating the at least one music track (<NUM>) into an accompaniment component and a vocal component (<NUM>); and
adding the vocal component (<NUM>) of the at least one music track (<NUM>) to the base music track (<NUM>);
wherein the determining includes determining a vertical musical compatibility between segments (<NUM>, <NUM>) of the base track (<NUM>) and the at least one music track (<NUM>) based on a tempo compatibility, a harmonic compatibility, a loudness compatibility, vocal activity, and/or beat stability;
characterized in that the tempo compatibility between the segment (<NUM>) of the music track (<NUM>) and the segment (<NUM>) of the base track (<NUM>) is determined by determining a tempo compatibility score which is higher the closer the tempo between the segments (<NUM>, <NUM>) is;
wherein the harmonic compatibility between the segment (<NUM>) of the music track (<NUM>) and the segment (<NUM>) of the base track (<NUM>) is determined by determining a harmonic compatibility score which is higher the closer the harmonic compatibility of the segments (<NUM>, <NUM>) is;
wherein the loudness compatibility between the segment (<NUM>) of the music track (<NUM>) and the segment (<NUM>) of the base track (<NUM>) is determined by determining a loudness compatibility score which is higher the closer the normalized loudness of the segments (<NUM>, <NUM>) is;
wherein the vocal activity is detected on the segment (<NUM>) of the music track (<NUM>) where a higher vocal activity in the segment (<NUM>) results in a higher vocal activity score;
wherein the beat stability is determined on the segment (<NUM>) of the music track (<NUM>) where a greater beat stability in the segment (<NUM>) results in a higher beat stability score;
wherein a vertical mashability score, on which a compatibility score of the vertical musical compatibility is based, is determined as the product of the scores of tempo compatibility, harmonic compatibility, loudness compatibility, vocal activity, and/or beat stability on which the vertical musical compatibility is based.