Patent Description:
Various technologies allow end users to separate audio tracks or objects after formal productions of music or other audio has been mixed. Such technologies include DOLBY ATMOS, blind signal separation (BSS), and MPEG Spatial Audio Object Coding (SAOC). Music producers generally want to limit an end user's ability to apply these technologies in efforts to maintain the integrity of the originally-produced music.

<NPL>, describes a standard for integrating multiple audio tracks with appropriate additional information, enabling users to experience various preset mixes and to make their own mixes complying with interactivity rules imposed by the music composers.

al "On the Informed Source Separation Approach for Interactive Remixing in Stereo" presents a comparison between two informed source separation (ISS) systems and relates the ISS approach in various configurations with conventional coding of separate tracks for interactive remixing in stereo. The compared systems are Undetermined Source Signal Recovery (USSR) and Enhanced Audio Object Separation (EAOS).

C "Frequency-Domain Source Identification and Manipulation in Stereo Mixes for Enhancement, Suppression and Re-Panning Applications" describes a frequency-domain framework for source identification, separation and manipulation in stereo music recordings. Based on a simplified model of the stereo mix, this paper described how a similarity measure between the Short-Time Fourier Transforms (STFT) of the input signals is used to identify time-frequency regions occupied by each source based on the panning coefficient assigned to it during the mix.

The following summary is merely intended to be exemplary. The summary is not intended to limit the scope of the claims.

In one exemplary aspect, a method comprises: providing an audio file comprising two or more discrete tracks; separating the two or more discrete tracks; setting a limit on an amount at least one of the two or more discrete tracks may be altered; and outputting the separated and limited discrete tracks as a file for use by an end user.

In accordance with another exemplary aspect, a method comprises: receiving an audio file comprising two or more discrete tracks; analyzing the two or more discrete tracks to determine a set limit on possible remixing for at least one of the two or more discrete tracks; editing at least one of the two or more discrete tracks; and remixing the two or more discrete tracks based on the determined set limit on possible remixing for the at least one of the two or more discrete tracks.

In accordance with another exemplary aspect, an apparatus comprises: at least one processor; and at least one non-transitory memory including computer program code configured to, with the at least one processor, cause the apparatus at least to: receive an audio file comprising two or more discrete tracks; separate the two or more discrete tracks from each other; analyze the two or more discrete tracks to determine a set limit on possible remixing for at least one of the two or more discrete tracks; and allow a user to remix the two or more discrete tracks based on the determined set limit on possible remixing for the at least one of the two or more discrete tracks.

Referring to the Figures, examples of methods of allowing an end user to alter, within defined limits, a mix of instrument and voice tracks are disclosed herein. Also disclosed herein are examples of apparatuses for carrying out such methods. Although some technologies (e.g., CD, MP3, and the like) do not allow the end user to alter a mix of tracks, more recent technologies do make such allowances. Such altering may be via the use of technology formats such as DOLBY ATMOS, blind signal separation (BSS), and MPEG Spatial Audio Object Coding (SAOC). For example, DOLBY ATMOS allows a completely free editing of instrument and voice tracks. BSS, however, provides a slightly less free editing ability due to little or no user-available information about the source signals or mixing processes used therefor. Editing using BSS may also result in products that suffer from separation artefacts. Similarly, in using MPEG SAOC, free editing is allowed, but audio quality may suffer due to changes in the object separation that are too drastic relative to the source track.

In any editing technology, music recordings may be recognized using any suitable method for the automatic identification of sound recordings. Some music files may be separated into instrument and voice tracks using any suitable algorithm (e.g., DUET BSS algorithm). Additionally, the instrument in each track may be recognized by any suitable automatic musical instrument recognition algorithm. A dominant instrument in a song can also be recognized by analyzing the music for a dominant tag.

In using any of the foregoing technologies to separate audio tracks, the end user may have a tendency to excessively modify music or other audio to their liking. For example, a guitar player may want to better hear the guitar track in a song in order to learn the notes or chords, a vocalist may want to understand lyrics, a person viewing a movie may want to hear the explosions louder or to filter out background noise in order to hear what a character is saying, or a person may want to simply hear only a voice track without hearing music, a laugh track, or other background sound in an effort to keep the overall volume at a minimum.

However, audio- or music producers generally do not want to allow end users complete control for one or more reasons. In particular, for reasons related to copyright, artists and producers want to limit the ability of an end user to completely single out instruments into separate tracks because such separated tracks may be used for unauthorized projects. Drastic changes to a song can also make that song sound bad, and when such versions of that song are widely disseminated (e.g., over the Internet), the original artist may incur negative reviews and/or bad publicity. Furthermore, it is well-noted that removing a laugh track from a comedy show often eliminates or reduces any comic effect.

Accordingly, both artists and producers generally dislike having their work products excessively modified by end users. To address issues of excessive modification of produced audio, music artists and music producers may desire to curtail the ability of an end user to modify a song (or other work product) by setting limits (either automatically or manually) regarding the extent to which the end user can remix audio files (for example, combine, pan, add reverberation, add autotuning ability, alter levels (e.g., levels of amplification relative to other levels), and the like). Such limits may include separating audio sources based on their location (e.g., direction of sound propagation) and playback of the separated audio sources. Other limits may be imposed by manipulation of playback options based on metadata associated with the audio. The examples disclosed herein are directed to imposing limits on the separation of instrument tracks (which is different from separation based on direction of sound propagation) as well as limiting the separation quality in playback.

The examples disclosed herein are directed to methods and tools for music producers to limit an end user's manipulation (using a user interface (UI)) of an audio mix of a song after instrument tracks have been separated. The manipulation may have either automatic limits or limits that can be set by the music producers. The examples described herein take into account the quality of the separation system when deciding the limits. In the alternative, a music producer can take into account the quality of the separated instrument tracks. Furthermore, the end user may be able to exceed the limits and listen to the result, but they may be prevented from saving or publishing (where license is given) remixes that exceed the limits. In addition, the examples disclosed herein may be used to enable inexperienced end users to remix their own recorded content with the help of BSS while limiting the risk of compromising the quality of the recorded content due to over remixing.

Referring to <FIG>, one example of a method of setting limits on the ability to alter a final mix, not encompassed by the wording of the independent claims, is designated generally by the reference number <NUM> and is hereinafter referred to as "method <NUM>. " Method <NUM> is capable of working with traditional audio files, e.g., with a final mix by the producers only. In method <NUM>, a producer manually sets the limits to which the end user may alter the final producer mix.

Method <NUM> is initiated with a music production step <NUM>, a first separation step <NUM>, and a limit setting step <NUM>. In the music production step <NUM>, a music producer uses discrete tracks of instruments and vocals (e.g., one instrument per track, one vocal per track, etc.) to make a preferred mix of a song. The music producer may then output and/or save the preferred mix in a particular format (e.g., mono, stereo, <NUM>, or the like). The limits are stored as metadata alongside the actual song audio. In some examples, the song audio is stored in MPEG SAOC format, and the metadata is added to the free metadata field in the SAOC format. In some other examples, the song audio is stored in MPEG SAOC format and the metadata is added to an MP4 container. In some other examples, the song audio is stored as AAC (advanced audio coding), and the metadata is added to the free metadata field in the AAC format. With MPEG SAOC the tracks are stored within the format, and modifying the stored tracks is easy.

With AAC the tracks of the preferred mix may need to be separated into discrete tracks before modification, as indicated in the first separation step <NUM>. An audio object separation system is used both in the music production end and at the user end. The separation system may need to be the same so that the metadata that the music producers decide on is applicable also at the user end. With some formats, the preferred mix may be separated into instrument tracks using BSS. If SAOC is used, however, the separation is built in. In any case, however, although the tracks are considered to be discrete, the separation thereof may not be perfect due to limitations in the production (e.g., particularly with regard to AAC and SAOC).

In the limit setting step <NUM>, the music producer listens to the separated tracks with different remixes and determines allowed limits of user remixing. In this way, the music producer ensures that the determined limits are suitable for the used separation quality. Naturally, the producers can also consider aspects other than separation quality when deciding the limits. Such aspects may include artistic views and the like. The producers may also set limits such as, for example, "Vocal track cannot be attenuated" (so the limit may be zero, i.e., no alteration of the track is allowed), "Background singing track must always be at least <NUM> dB less than lead vocal track," and the like. In some examples, the limits may be determined using a computer program.

After the music production step <NUM>, the first separation step <NUM>, and/or the limit setting step <NUM>, the separated/limited audio and metadata are outputted as an output file <NUM>. In various examples, the output file <NUM> may be a finished production of an audio file by the music producer for the end user and may comprise instrument and/or vocal tracks.

If desired, the output file <NUM> may be encrypted so that only an approved version of a player program may decrypt it. In this way it can be made so that the end user cannot use other player programs to remix the songs excessively. In other words, a digital rights management (DRM) scheme may estimate the degree of change (as defined by the allowed limits for editing or remixing) before allowing saving, copying, or sharing of the content. The output file <NUM> is then stored, streamed, and/or sent to the player program in a sending step <NUM>.

When the player program receives the output file <NUM> from the sending step <NUM>, the player program checks for the existence of metadata and allows the end user to remix the song accordingly. As shown in a second separation step <NUM>, the audio file of the preferred mix may be separated into instrument and/or voice tracks using BSS. If, on the other hand, the separation in the second separation step <NUM> is via SAOC, the separation is built in. In examples in which DOLBY ATMOS is used in the second separation step <NUM>, at least some of the tracks may already be discrete because DOLBY ATMOS generally maintains the separation of tracks through the sending step <NUM>. As shown in a remix step <NUM>, the metadata is received, an analysis of the metadata is made to ascertain limits as to how much the end user may remix the tracks, and the end user listens to the audio and may remix it to the extent allowable using a UI. In doing so, the end user may not exceed the limits set in the metadata by the producers in the limit setting step <NUM>.

In some examples, however, the end user may be permitted to exceed the set limits in a limit exceeding step <NUM>. In the limit exceeding step <NUM>, the end user may exceed the set limits temporarily, but the result may not be stored. In some other examples, the end user may be allowed to exceed the set limits only when the end user is constantly manipulating a control (e.g., holding a finger on a control button).

Referring to <FIG>, another example of a method of setting limits on the ability to alter a final mix is designated generally by the reference number <NUM> and is hereinafter referred to as "method <NUM>. " In method <NUM>, the original tracks are supplied in addition to final preferred mix by the producers. In the system of method <NUM>, limits are automatically set with the assistance of the original track information.

Method <NUM> is initiated with a music production step <NUM>, a first separation step <NUM>, a quality estimation step <NUM>, and a limit setting step <NUM>. In the music production step <NUM>, original instrument tracks are available from the music production. A music producer makes a preferred mix of a song and outputs and/or saves it in a particular format (e.g., mono, stereo, <NUM>, or the like).

As shown in the first separation step <NUM>, the preferred mix is separated into instrument tracks. The separated instrument tracks may be slightly different from the original instrument tracks because the separation may not be perfect. The separation can be done using BSS for traditional music files and SAOC for SAOC files.

In the quality estimation step <NUM>, the quality of the separation is estimated using different criteria. In some examples only the file format is considered in the quality evaluation. SAOC files are considered to have high quality, whereas BSS files are considered to have lower quality. Different metrics of the files may also be calculated, for example, the system may calculate cross correlation between the original instrument tracks and the separated tracks. Higher correlation generally corresponds to higher quality.

In the limit setting step <NUM>, limits may be set for how the end user can remix the files. Typically, higher quality files can be edited more, and lower quality files less. The limits may be written into metadata.

In addition to separation quality related limits, there may be other limits. The player program may analyze different characteristics of the tracks of the song. The analysis may include recognizing instruments using automatic instrument recognition algorithms, recognizing the dominant instrument of the song, recognizing the tempo of the song, recognizing frequency- or time-domain characteristics such as impulses, zero-crossing rate, spectral tilt, overall spectrum, transient or sinusoidal nature of the song, and the like. The analysis can be performed for the whole song or only for parts of the song. The end user may remix the song using a UI. The player program software sets limits for end user remixing based on the analysis. The limits may be, for example, "The dominant instrument cannot be attenuated," "Vocal tracks cannot be attenuated," "Songs with tempo > <NUM> bpm cannot be sped up," and the like. Also, limits can be used in combination with music analysis (e.g., "The dominant instrument cannot be attenuated more than <NUM> dB," "Vocal tracks cannot be attenuated," and the like). These limits may also be written into the metadata.

The separated/estimated/limited audio and metadata are then outputted as an output file <NUM> as instrument tracks. The output file <NUM> is then stored, streamed, and/or sent to the end user in a sending step <NUM>.

When the player program receives the output file <NUM> from the sending step <NUM>, the player program receives a music song file with metadata that describes the limits for editing. The player program separates the song into audio object tracks in a second separation step <NUM>. In the second separation step <NUM>, the audio file may be separated using BSS or SAOC (with the separation built in).

As shown in a remix step <NUM>, the end user may remix the song using a UI. The end user may not remix the song so that the remix would exceed the limits set in the metadata. The limits may be, for example, "An instrument may be set louder or quieter but not more than described in the metadata. " For each track the metadata may contain limits, e.g., in Decibels. Such limits may be, for example, "The level of Track <NUM> can by modified by the user from - <NUM> dB to +<NUM> dB" if the separation quality is low, and "The level of Track <NUM> can by modified by the user from -<NUM> dB to +<NUM> dB" if the separation quality is high.

In some examples of method <NUM> (as with method <NUM>), the end user may exceed the set limits in a limit exceeding step <NUM>. In the limit exceeding step <NUM>, the end user may exceed the set limits temporarily, but the result may not be stored. In some other examples, the end user may be allowed to exceed the set limits only when the end user is constantly manipulating a control (e.g., holding a finger on a control button).

Referring to <FIG>, another example of a method of automatically setting limits without information pertaining to the original instrument tracks is designated generally by the reference number <NUM> and is hereinafter referred to as "method <NUM>. " Method <NUM> operates with traditional audio files, e.g., with the final mix by the producers only.

Method <NUM> comprises a music production step <NUM> in which one or more original instrument tracks are available from the music production. A music producer makes a preferred mix of a song and outputs and/or saves it in a particular format (e.g., mono, stereo, <NUM>, or the like). After the music production step <NUM>, the preferred mix is outputted as an output file <NUM> of instrument tracks and stored, streamed, and/or sent to the end user in a sending step <NUM>.

In a separation step <NUM>, a player program receives a music song file with no metadata. The player program separates the preferred mix into instrument tracks. The separation can be done using BSS for traditional music files and SAOC for SAOC files.

Quality of the separation may then be estimated in an end user quality estimation step <NUM>. The end user quality estimation step <NUM> may employ different criteria. In some examples only the file format is considered in the quality evaluation. SAOC files are considered to have high quality, whereas BSS files are considered to have lower quality. Different metrics of the files may also be calculated, for example, the system may calculate cross correlation between the separated tracks. If the tracks have low correlation, then the separation may be considered to have a high success rating. However, if the tracks have high correlation, then the separation may be considered to have a lower success rating.

As shown in a limit setting step <NUM>, limits are set with regard to how the end user may remix the files. In such a step, higher quality files can be edited more, and lower quality files can be edited less. Such limits may be, for example, "The level of Track <NUM> can by modified by the user from -<NUM> dB to +<NUM> dB" if the separation quality is low, and "The level of Track <NUM> can by modified by the user from -<NUM> dB to +<NUM> dB" if the separation quality is high.

In addition to separation quality related limits there may be other limits. For example, the player program may analyze different characteristics of the tracks of the song. The analysis may include recognizing instruments using automatic instrument recognition algorithms, recognizing the dominant instrument of the song, recognizing the tempo of the song, recognizing frequency- or time-domain characteristics such as impulses, zero-crossing rate, spectral tilt, overall spectrum, transient or sinusoidal nature of the song, and the like. The analysis can be performed for the whole song or only for parts of the song. The user may remix the song using a UI. The player program software sets limits for user remixing based on the analysis. The limits may be, for example, "The dominant instrument cannot be attenuated," "Vocal tracks cannot be attenuated," "Songs with tempo > <NUM> bpm cannot be sped up," and the like. Also, limits can be used in combination with music analysis (e.g., "The dominant instrument cannot be attenuated more than <NUM> dB," "Vocal tracks cannot be attenuated," and the like).

As shown in a remix step <NUM>, the end user may remix the song using a UI. The end user may not remix the song so that the remix would exceed the limits set.

The end user may exceed the set limits in a limit exceeding step <NUM>. In the limit exceeding step <NUM>, the end user may exceed the set limits temporarily, but the result may not be stored. In some other examples, the end user may be allowed to exceed the set limits only when the end user is constantly manipulating a control (e.g., holding a finger on a control button).

Referring to <FIG>, the instrument tracks from any of the foregoing examples may be received into an apparatus or electronic device for use by the end user. The apparatus or electronic device may be a user interface (UI) designated by the reference number <NUM> and hereinafter referred to as "UI <NUM>. " UI <NUM> comprises a controller <NUM> and a display <NUM>. The controller <NUM> may include a processor <NUM> and a memory <NUM>, with software <NUM> being located in the memory <NUM>. The instrument tracks may be music files <NUM>.

Referring to <FIG>, the UI <NUM> may comprise a simple slider-based control on the display <NUM> where each slider <NUM> represents a track in a song. The end user may change each track by manipulating its corresponding slider <NUM>.

Referring to <FIG>, the UI <NUM> may be a graphical user interface (GUI) that displays images <NUM>, <NUM>, <NUM> of the artists of a band and allows the end user to manipulate the images <NUM>, <NUM>, <NUM> so as to simultaneously manipulate the audio mix of the artists. The GUI may render use of the UI <NUM> more intuitive as compared to a UI incorporating a slider. The images <NUM>, <NUM>, <NUM> may be, for example, album cover art. Each image <NUM>, <NUM>, <NUM> may have different colors or visual effect to indicate whether the instrument associated with the image can or cannot be edited.

Particularly in examples in which the UI <NUM> is graphical, the music files <NUM> may include visual files where visual objects of the visual files have been mapped to instruments and/or voices, i.e., music objects in the music files <NUM>. The end user may then manipulate the visual objects and at the same time automatically manipulate the music objects. Manipulation of the music objects includes, but is not limited to, selecting a band member visual object to increase their track volume, and the like. As shown in <FIG>, an image <NUM> of a band member, as a visual object, may be pinched bigger to increase their track volume. Manipulation may also include moving a band member visual object to move the auditory spatial position of their track, e.g., moving the band member's associated track from a left speaker to a right speaker and vice versa. As shown in <FIG>, audio effects like autotune <NUM>, bass/treble <NUM>, balance between speakers <NUM>, reverb, equalization, etc. may appear as visual objects. The end user may then control the amount of these effects by manipulating the visual object. Each track may have its own effects.

Referring to <FIG>, the UI <NUM> may use any suitable method for recognizing a song in a music file. For example, the UI <NUM> may access an online (or other) database <NUM>, which the UI <NUM> may use to search for the artists recorded in the music file. The UI <NUM> may further search online images of the artists and/or their instruments. The UI <NUM> may further divide the music file into instrument/voice tracks (using, for example, the DUET BSS algorithm) and may also recognize the instrument in each track (using, for example, any suitable automatic musical instrument recognition algorithm). The UI <NUM> may map each track to an artist based on the knowledge of the artist's instrument. Images of the artists or their instruments may be displayed, and the end user may manipulate individual tracks based on such. Furthermore, if there are several players with the same instrument, the UI <NUM> may sum those tracks together instead of an artist image showing an instrument image or a group or composite image of the artists.

Other aspects of the UI <NUM> may include displaying different pictures on the display <NUM> at different parts of songs when players change or are silent. Additionally, a record label associated with the song may provide several example mixes with associated pictures where band member visual objects match the mix. Such a system may automatically choose an end user-preferred mix based on either user settings or the end user's previous habits. For example, if the end user always mixes the bass player louder, a mix may be selected where the bass player is loudest of all the players. The end user may be supplied with a visual confirmation of the mix because the system shows him or her an image where the bass player visual object is larger than others.

Remixing by the end user on the UI <NUM> may also be channel-based so that the end user amplifies/attenuates an instrument differently in different channels in multichannel audio. In this way, the end user may control the perceived spatial location of the instrument. The differing amount of amplification may also be limited in metadata or by automatic analysis, e.g., higher quality files associated with the spatial location can be changed more than lower quality files.

In any example of the UI <NUM>, the end user may exceed the limits set by the music producers. In the UI <NUM>, this may be visible, for example, on a touch display device so that the control moves slower after the limit has been exceeded and the end user has to swipe many times to move the control. The control may be a slider as shown in <FIG> or an icon. The end user may listen to the song with limits that exceed the remixing, but the remixed version may be refused saving and/or transmitting by the UI <NUM>.

In examples of the UI <NUM> in which the end user may exceed the limits set by the producers, the exceeded limits may be noted on the display <NUM> by being made visible, for example, on a touch display device so that the remix exceeds limits only if the user holds his finger on the slider and keeps it above the slider maximum value. When the user releases his finger the slider returns to the maximum value.

In addition to using the examples described herein for music files, the examples described herein are applicable for audio broadcasts, teleconferencing, movie audio, audio books, and the like. The examples described herein may also be used with an end user's own recorded content because allowing end users to remix their own content is a beneficial feature (but traditional remix with BSS may require some expertise, which may cause the end-result to sound less than desirable, thereby possibly causing end users to stop using it). However, with the examples described herein, BSS can be used by inexperienced end users with less of a risk of them detracting from the overall quality of their own content.

With regard to the examples described herein: limits for remixing may be set individually for each instrument; artists can set instrument-wise limits even if audio is sent using backwards-compatible audio formats such as AAC or MP3 using audio object separation and metadata; in some examples the limits for remixing may be set automatically; in some examples automatic instrument-wise limits may be used even without metadata; and an intuitive way for end users to create personal music mixes is realized.

Referring now to all of the Figures described herein, any of the foregoing examples may be implemented in software, hardware, application logic, or a combination of software, hardware, and application logic. The software, application logic, and/or hardware may reside in the UI <NUM> to remix audio. If desired, all or part of the software, application logic, and/or hardware may reside at any other suitable location. In an example, the application logic, software, or an instruction set is maintained on any one of various computer-readable media. A "computer-readable medium" may be any media or means that can contain, store, communicate, propagate, or transport instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer. A computer-readable medium may comprise a computer-readable storage medium that may be any media or means that can contain or store the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer.

Claim 1:
A computer implemented method (<NUM>), comprising:
providing (<NUM>) an audio file based on two or more discrete audio tracks;
separating (<NUM>) the audio file into two or more separated audio tracks;
estimating a quality of the separation based on a cross-correlation between the discrete audio tracks and the separated audio tracks and/or determining a file format of the audio file;
setting (<NUM>) a limit on an amount of alteration of a track volume and/or a spatial position and/or an amount of an effect, for at least one of the two or more separated audio tracks, wherein the at least one set limit is based on at least the estimated quality of the separation and/or the file format of the audio file; and
outputting (<NUM>) the separated audio tracks as a file and the at least one set limit as metadata of the file for use by a user to alter the track volume and/or the spatial position and/or the amount of the effect.