Transcribing voiced musical notes for creating, practicing and sharing of musical harmonies

Method for transcription of voiced musical note includes segmenting an electronic audio signal into a plurality of musical note segments, and sampling each note segment to obtain a plurality of audio samples. For each audio sample, an autocorrelation is computed to determine a probability value associated with certain audio frequencies contained within the audio sample. Local maxima are identified in the energy associated with one or more audio frequencies comprising each audio sample and a corrective function applied to reduce the occurrence of octave errors. A true pitch of each note segment is then determined by converting the pitch identification problem to one involving a shortest path through a graph or network of node. Edge weights are computed for a plurality of adjacent nodes i, j comprising the graph, where each node represents one of the musical note segments.

BACKGROUND

Statement of the Technical Field

This disclosure concerns automated methods and devices which facilitate the creating, practicing and sharing of music, and more particularly concerns methods for transcribing voiced musical notes.

DESCRIPTION OF THE RELATED ART

In the music field, harmony involves a combination of concurrently sounded musical notes which produce a pleasant listening effect. In this regard, harmony is generally understood to require at least two separate tones or voices to be sounded simultaneously. For example, a simple form of harmony would involve a second note sounded at a pitch that is double the pitch of a basic melody. A Third harmony is one in which the harmony part is a musical Third above the original pitches comprising the melody. Most commonly, a musical harmony will comprise between three and six voice parts or tones.

In theory, various harmony parts (e.g. vocal harmony parts) could be created separately and then combined to create a song. But in order to create vocal harmonies that have a pleasing sound, vocalists will often work together with producers. For example, this process can take place in a recording studio with the various vocalists present so that they can practice with one another and hear how the combined vocal parts will sound when combined together.

SUMMARY

This document concerns a method for accurate transcription of voiced musical notes. A microphone is used to convert sound waves comprised of monophonic audio produced by a voiced rendition of the musical composition, into an electronic audio signal. The electronic audio signal is then stored in a data storage device. An electronic processing device receives certain manual user inputs which specify timing information for the musical composition to be transcribed. For example, the user specified timing information can be selected from the group consisting of a music time signature and a tempo associated with the musical composition. The electronic audio signal is then processed in the electronic processing device to determine a true pitch of each musical note of the voiced rendition.

The true pitch is determined in the electronic processing device through a series of steps which can begin by segmenting the electronic audio signal into a plurality of musical note segments. For example, in some scenarios each of the musical note segments can be selected to comprise an eighth note. Each note segment is sampled to obtain a plurality of audio samples. According to one aspect, the audio samples are preferentially selected from portions of the note segment which exclude leading and trailing edges of the note segment

For each audio sample, an autocorrelation is computed to determine a probability value associated with certain audio frequencies contained within the audio sample. The resulting plot of probability values will vary in magnitude over a frequency range associated with audio produced by a human voice. Within such frequency range there will be one or more instances involving the occurrence of local maxima or peaks in the energy associated with one or more audio frequencies comprising each audio sample. A corrective function can be applied to help eliminate one or more of the local maxima which are determined to be associated with octave errors. After octave errors are accounted for in this way for each audio sample, the audio frequency associated with each of the one or more local maxima can be quantized to a nearest musical note on the western musical scale.

A true pitch of each note segment is then determined by converting the pitch identification problem to one involving a shortest path through a graph or network of nodes, where each node corresponds to a possible pitch of the corresponding note segment. For purposes of such a network, a virtual note or pitch can be included to represent periods of silence. In a shortest path solution, it is necessary to compute edge weights between nodes. According to one aspect of the solution presented herein, edge weights are computed for a plurality of adjacent nodes i, j comprising the graph, where each node represents one of the musical note segments.

The edge weights between any two adjacent nodes i, j are computed by averaging the negative log likelihood of the probability values which are determined for the audio frequency associated with each musical note, a determined over the range of samples which are associated with each particular audio segment. It will be appreciated that these probability values which are averaged are based on the strength of the local maxima at such frequencies as determined in connection with each audio sample. Each resulting edge weight assigned between adjacent nodes i and j is then a negative log likelihood of the probability that a next musical note segment has a pitch associated with node j, assuming that node i was selected. Based on this shortest path analysis, a true pitch to be assigned each of the musical note segments is determined.

According to one aspect, the corrective function reduces the occurrence of octave errors by giving preference to a first audio frequency associated with a first local maxima having the lowest audio frequency of the one or more local maxima. This preference being conditional on the others of the one or more local maxima associated with the same audio sample having a probability value magnitude within a predetermined threshold range of the first local maxima. In some scenarios, the corrective function applied for this purpose is the integral of a Kumaraswamy distribution.

The process of determining the true pitch of each note can further involve applying a regularization function when assigning the edge weights to minimize a possibility that an oscillating pitch will (e.g., an oscillating pitch associated with vibrato singing) will significantly affect assigned edge weights. In some scenarios, the regularization function can be comprised of a Laplace distribution.

After the pitch of the musical note segments are determined, the process can continue by selectively concatenating contiguous groups of the musical note segments which are determined to have the same pitch, into notes of longer duration. The resulting transcription data which specifies a true pitch of each of the musical note in the voiced rendition can then be stored in a data storage device.

In some scenarios, the voiced rendition can comprise a first musical harmony part of the musical composition, and the electronic processing device displays on a note grid the musical notes which correspond to the true pitch which has been determined. Further, the musical composition can be comprised of at least a second musical harmony part. In such a scenario, the electronic processing device can further display on the note grid, concurrent with the musical notes comprising the first musical harmony part, those musical notes which are associated with at least the second musical harmony part.

The process can also include determining in real-time a pitch of a second voiced rendition of the first musical harmony part. An audio analysis process applied for the real-time determining of the pitch of the second voiced rendition is advantageously selected so that it is different from the audio analysis process applied for determining the true pitch of the voiced rendition. A pitch indicator can be displayed in conjunction with the note grid to indicate whether musical notes of the second voiced rendition match a specified pitch and timing of the musical notes of the first musical harmony part. Further the second musical harmony part can be caused by the electronic processing device to be audibly played through a loudspeaker concurrent with determining the pitch of the second voiced rendition of the first harmony part.

DETAILED DESCRIPTION

It will be readily understood that the solution described herein and illustrated in the appended figures could involve a wide variety of different configurations. Thus, the following more detailed description, as represented in the figures, is not intended to limit the scope of the present disclosure, but is merely representative of certain implementations in various different scenarios. While the various aspects are presented in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

A solution disclosed herein concerns a Music Harmony Tool (MHT) and method of using same. For a performer who seeks to learn, practice, share and/or collaboratively create harmonies there are a number of challenges which must be overcome. A harmony can involve a plurality of different parts in which performers are singing different notes at the same time. A singer seeking to create, learn or practice a harmony part must focus on the specific notes required for their part of the harmony without being distracted by other parts of the harmony. Further, different performers necessary for creating a harmony may be unavailable at the same time, may be physically separated by large distances and/or may have different skill levels requiring different amounts of practice time. Not all participants may want to participate in all practice sessions. But for singers with less experience, it can be difficult to master a particular harmony without the presence of others, and an inexperienced singer may not be able to tell when the harmony part they are singing is being performed properly. Accordingly, an MHT disclosed herein provides certain advantages for learning creating, practicing and collaborating in regards to musical harmonies.

Referring now toFIG. 1it can be observed that an MHT system can in some scenarios include an application server102which has access to user account data store103, and a main data store105. One or more clients comprising user equipment (UE) computer systems1061-106ncan communicate with the application server using a computer data network108. The UE computer systems1061-106ncan comprise any suitable type of computing device that is capable of carrying out the methods and functions described herein. In some scenarios, the user equipment can comprise a desktop computer with suitable network interface connections to carry out certain data communication operations as described herein. In other scenarios, the user equipment can comprise a portable data communication device such as a smart phone, a tablet computer, or a laptop computer. Other types of computing systems which can be used for this purpose include dedicated computing devices which are designed to exclusively carry out the methodologies and functions described herein. Although a network-based arrangement is presented herein, it should be understood that several aspects of the solution can also be implemented in a non-networked computer system. These various aspects and features are described below in greater detail

The application server102can comprise a computer program and associated computer hardware that provides MHT services to the UE computer systems1061-106nto assist in carrying out one or more of the methods and functions described herein. The user account data store103can contain certain user account data114pertaining to individual users who have established user accounts to facilitate access and use of the MHT. In some embodiments, the user account data store103can comprise user account data such as passwords, email addresses, practice session scores reflecting user proficiency, and so on. The user account data114can also include other types of user authentication data, digital certificates, and/or a transaction log. The main data store105can comprise music data files1101,1102, . . .110nassociated with one or more songs. As described below in greater detail, each of the music data files1101,1102, . . .110ncan include digital data representative of one or more harmony parts created by one or more users.

The computer data network108is a comprised of a data communication network suitable to facilitate communication of data files, user data and other types of information necessary to implement the MHT and MHT services described herein. Computer network108can also facilitate sharing with UE computer systems1061-106ncertain computing resources that are available at application server102. Exemplary networks which can be used for this purpose can include packet data networks operating in accordance with any communication protocol now known, or known in the future. The organizational scope of such network can include but is not limited to one or more of an intranet, an extranet, and the Internet.

FIGS. 2A-2Fcomprise a set of flow charts that are useful for understanding a method for creating, sharing and practicing musical harmonies.FIG. 2Ashows that an MHT process200can begin at202and continue at204where a graphical user interface (GUI) comprising a home screen is displayed on a display screen of a UE (e.g. any one of UE1061-106n). An example of a home screen350facilitated by a UE1061-106nis shown inFIG. 3. The home screen350includes exemplary user selectable control elements352,354,356,358which allow users to select various MHT functions such as “Rehearsal”, “Lessons”, “Build Harmonies” and “Create New Song”. At least one other user selectable control element (e.g., control element360) can facilitate user access to certain user data such as user account data114. In some scenarios, the UE home screen can be presented on a display screen of a UE1061-106n. According to one aspect, the display screen can be a touch screen display though which a user can select available functions by touching one or more of the selectable control elements. In other scenarios, a computer pointing device, keyboard or other type of user interface can be provided for this purpose.

From the home screen a UE1061-106nreceive one or more user input selections206. The process continues in steps208,212,216and220where a determination is made as to whether the user has selected a particular option. For example, selections can include a “Lessons” session option208, a Rehearsal session option212, or a Build Harmony session option216. A user can also select a terminate session option220. Depending on the user selection, the UE will transition to display the Lessons GUI at210, a Rehearsal GUI at214, or a Build Harmony GUI at218to begin the selected session as shown.

If a lesson session option is selected at208, then the process continues at224inFIG. 2B. At224, the UE receives a user input selection to begin a practice session with respect to a particular segment of a musical selection or song. The user's choice of a particular song can be entered in a song selection screen400aas shown inFIG. 4A. In response to a user selection of a particular song (e.g., Song 3), the UE1061. . .106ncan retrieve certain data comprising a particular segment of a musical selection or song. For example, in some scenarios the data comprising a segment of a particular musical selection or song1101-110ncan be retrieved from the main data store105in response to a request from a UE directed to the application server102. In some scenarios, the segment can comprise the entire musical selection or song. In other scenarios the segment can comprise a portion of the musical selection.

Once the particular song has been selected, the process continues on to225where the user can select a particular harmony part that he wishes to sing. For example, in a scenario shown inFIG. 4Bthe user can select the part of Soprano, Alto, Tenor or Bass from a part selection screen400b. In some scenarios, the text associated with each identified part can be visually coded in accordance with a pattern or color. Once the system receives the user selection of a particular part that the user wishes to practice, the process continues on to226where selected song is loaded in preparation for the practice session. This condition is illustrated inFIG. 4Cwhich involves a graphic display400cof a musical note scale in which particular musical note legends or annotations410are aligned along a vertical note axis411of the display.

InFIG. 4C, each musical note block404,405,406,407is visually coded by means of a selected visual indicator such as a cross-hatching pattern or color. In some scenarios, the pattern or color of the musical note blocks associated with a particular harmony part can be chosen to correspond with a visual coding (e.g., color or pattern) of one of the various harmony parts listed inFIG. 4B. For example, if text specifying the “Soprano” part is presented in blue colored text inFIG. 4B, then the musical note blocks404which correspond to the Soprano part inFIG. 4Ccan be presented in the same color blue. If text specifying the “Alto” part is presented in red colored text inFIG. 4B, then the musical note blocks405which correspond to the Alto part inFIG. 4Ccan be presented in the same color red. Consequently, the user can more easily identify through such visual coding which notes are associated with each part.

It can be observed inFIG. 4Cthat the musical note blocks407which correspond to the particular harmony part selected by the user can be highlighted or otherwise marked to facilitate identification. These marked musical note blocks are sometimes referred to herein as target musical note blocks. Once the practice session begins, the user is intended to sing these highlighted or visually marked target musical note blocks.

It can be observed inFIG. 4Cthat each musical note block404,405,406,407is vertically aligned with the appropriate annotated note text in note legend410. The musical note blocks are also aligned along the time axis412with a particular time during the particular musical segment of the selected song. For example, inFIG. 4Ctarget musical note block407aligned with a particular time within a song to indicate the note that is required at a particular time for that particular harmony part. The duration of each note in the musical harmony is indicated by the width W of each note block along a direction aligned with time axis412. Time scale is marked by vertical lines408,409which extend transverse to the time axis. For example, lines408can indicate musical measures or bars, and lines409indicate beats within each measure. A current time axis403can specify the point in time where the user is intended to begin singing.

When the practice session actually begins, the measures and musical note blocks404,405,406,407will automatically begin to move or scroll (e.g., from left to right) on the screen in accordance with the music tempo and the passage of time. In this regard, the musical note blocks can be said to scroll in a direction aligned with the time axis. More generally, the user's view into the note grid can be understood as being allowed to slide as the song progresses so that a “window” appears to the user to scroll along the time axis412. The resulting note grid which is displayed can be advantageously centered in the display on the current time axis403within a particular song the user is practicing. This process is described below in greater detail. Accordingly,

The practice session actually begins at227and228with the UE audibly reproducing the music segment, and also scrolling the music note blocks on the display screen. The scrolled notes will include the target musical note blocks407which the user is expected to sing with respect to the reproduced segment of music.FIG. 4Dshows a display screen400dcorresponding to steps227and228. Optionally, the process can further comprise displaying230to the user certain information or textual guidance (not shown inFIG. 4D) explaining how the particular segment is to be performed or sung.

At232a user will sing notes having a pitch in accordance with the target musical note blocks408, while observing the target musical note blocks408which are scrolled on screen on screen with the passage of time. Using the contextual information provided in steps227,228and230, the user attempts while singing to match the pitch of their voice to the note that is specified by the target musical note blocks. As the user sings, the UE captures the resulting sound associated with such singing with a microphone so as to generate an audio signal. The UE further performs certain processing operations at234to analyze the pitch of the musical notes that are sung. The results of such analysis are displayed on screen at236in the form of visual cues414a,414bso that the user can understand how well the singing performance pitch is matching the specified target notes. These visual cues are presented or displayed to the user. In some scenarios, the visual cues can be presented in the form of micro-notes, having a duration which is substantially less than a duration of an entire measure. For example, in some scenarios each micro-note can be one musical beat or less. Micro-notes are selected to have a relatively short duration relative to each measure so as to facilitate continually updating the information presented to the user regarding the pitch that they are singing. It can be observed inFIG. 4Dthat the visual cues414a,414bare displayed in real time at a location along the time axis412to indicate actual timing of sung notes. Of course other types of visual cues are also possible and it should be appreciated that any such alternative arrangement of visual cue can also be acceptable for purposes of the solution presented herein.

FIG. 4Dshows how the time-aligned visual cues are generated and displayed in real time as the user sings. In the example shown, visual cues414ahave the correct pitch corresponding to target musical note blocks407. As such, visual cues414aare displayed directly overlaid on the target musical note blocks407. Conversely, visual cues414bhave the incorrect pitch as compared to the target musical note blocks407and therefore appear above or below the target musical note blocks407to indicate that the sung pitch is too high or too low.

In the scenario shown inFIG. 4D, notes407that are about to be sung enter from the right side of the screen and are intended to be voiced by the user as they scroll past the current time axis403. The notes then depart the screen to the left. Visual cues regarding a timing of the user's singing can be provided by observing how well the leading and trailing edges of a particular visual cue414a,414balign with the position of the target musical note blocks407along the time axis412. For example, inFIG. 4Da timing misalignment “t” between the leading edge of a target musical note block407and the leading edge of a visual cue414aindicates that the user was late in beginning to sing the particular note. The foregoing visual cues can be visually observed and compared by a user in real-time to target musical note blocks407

The visual cues shown inFIG. 4Dare useful for conveying information concerning the timing and pitch of musical notes to be voiced by the user. In some scenarios, it can be desirable to modify the visual cues to also convey information concerning the relative loudness of the harmony part that is to be sung. This additional information can be visually conveyed in any suitable manner. For example, in some scenarios the information can be conveyed by adjusting the “thickness/height” of the target musical note blocks407in the vertical direction (i.e., in directions aligned with the vertical note axis411). Thicker/taller notes would indicate notes to be sung more loudly (forte), and narrower/skinnier notes would indicate quieter (piano) sections. Of course other variations are also possible. For example, the saturation, brightness or intensity of the displayed note blocks could be varied to indicated differences in how loud the musical note blocks are to be voiced. In other scenarios, the color of the musical note blocks can be varied.

The visual feedback provided to a user by the UE during practice gives real-time input to the singer. This feature greatly helps with training and improving vocals. For example, it eliminates the need for a user to record an entire track then go back and listen to it afterward. Accordingly, a user can avoid the slow and tedious feedback process common to conventional methods.

At238a determination is made as to whether the lesson corresponding to the particular segment is completed. If not (238: No) then the process returns to step232where the user continues to sing and receive feedback from the UE. If the lesson is completed with respect to the particular segment (238: Yes) then the process continues on to240where the UE can calculate and display a score with respect to the particular practice session. In some scenarios, this score can be used to help guide the user toward appropriate exercises or further practice sessions that are automatically selected to help the user improve his performance. For example, in some scenarios, the score provided to the user can be segmented to reveal performance associated with different portions of a harmony. In such a scenario, the user can be directed to those areas where the users performance is less than satisfactory. In other scenarios, where the user's score reveals a more generalized weakness in their singing ability, then the system can select musical singing exercises which are particularly tailored to improve certain singing skills.

At242, the user can select whether to proceed with the next lessor or music segment. If so (242: Yes) then the process returns to224for practicing the next segment. Otherwise the process can terminate or return to the home screen at244.

Referring once again toFIG. 2A, If a rehearsal session option is selected at212, then the process continues at246inFIG. 2C. At246the UE receives a user input selection to load a Song Library. For example, this song library can be requested from an application server102. Information concerning the selected Song Library can be displayed to the user in a screen similar to screen400aas shown inFIG. 4A. Once the Song Library is loaded, the user can interact with the GUI to subsequently select at248a particular song for rehearsal.

The user can optionally begin the rehearsal by selecting a particular harmony part (first harmony part) of the song which has been loaded so that it can be played out loud using a UE loudspeaker. Accordingly, the process can continue at250with the UE receiving a user input specifying a particular harmony part which is to be played out loud. Shown inFIG. 2Eis a display screen illustrating how the user can interact with the UE to facilitate such playback. Here, the user has previously selected Song 3 from the music library and has selected to play out loud the Soprano part. The UE can receive a user input through the UE by the user selecting the “Play” command424which causes the UE to play the user selected harmony parts. Consequently, the first harmony part which has been selected can be heard by the user during the rehearsal session. A volume or amplitude of the playback can be controlled such as by a slider or other type of control element420.

The user can further indicate a particular harmony part (second harmony part) of the song which the user wishes to actually practice. Accordingly, the process can continue at252with the UE receiving a user input specifying a particular harmony part which is to be practiced. To facilitate the rehearsal or practice session, the UE can receive from the user through the GUI one or more selections to control a playback volume254of certain harmony parts of the song (e.g. to control the playback volume of a first harmony part while the user rehearses practices singing the second harmony part). In such a scenario, it is advantageous for the first harmony part to be played back to the user by means of headphones while the user performs the second harmony part. Such an arrangement allows the UE to separately capture the second harmony part performed by the user without audio interference caused by playback of the first harmony part. In some scenarios, the user can select a plurality of different harmony parts which are to be played back while the user rehearses or sings a selected harmony part. A volume or amplitude of the playback can be controlled by a slider or other type of control element420.

The UE can be caused to play a certain harmony part as described herein by selecting the “Play” command424. The UE will respond to such input by playing the user selected harmony parts (e.g. while the user sings a different selected harmony part for practice/rehearsal purposes). Alternatively, the UE can be caused to concurrently play the user selected harmony parts and record a different harmony part, which is concurrently sung by the user. This functionality can be initiated when the UE receives a user input selecting the “Record” control426. In either scenario, the UE will display visual cues (e.g. highlighted target musical note blocks428) to indicate the harmony part which is to be sung by the user. Such display can be arranged in a manner similar to that described herein with respect to the practice screen shown inFIG. 4D. More particularly, at258the UE visually displays highlighted target musical note blocks428for the user part while the user sings his/her part. The UE can concurrently also visually display harmony notes422for the harmony parts other than user part as those other parts are being audibly played back by the UE

As the rehearsal process progresses, the UE can advantageously display at260real-time visual feedback to the user on a display screen of the UE. Such visual feedback can be similar to the feedback described herein with respect toFIG. 4Dto indicate how well the user's singing performance matches a desired harmony part. Visual cues can be provided to indicate how well the user performance matches a desired performance with respect to one or more of pitch, volume and/or timing with respect to user's selected part.

At262a decision can be made as to whether the selected song or segment is done playing. If not (262: No) then the process continues at258. At any point during the session while the song is playing, the UE can receive through its user interface at264a user input to rewind or fast forward through portions of the selected song. Further, at266the UE can receive through its user interface a user input to optionally select to save the harmony part that has just been recorded during the rehearsal session. At268, the user can select to have the rehearsal process terminate at270(270: Yes), whereby the UE can terminate or return to the home screen. If the process is not terminated (268: No), then the process can continue or return to246so the process continues.

An MHT disclosed herein can also facilitate creation of entirely new songs by the user. The user can record a harmony to begin the process and define that harmony as being part of a new song. Further, the user can create one or more harmony parts of the newly created song. This process is best understood with reference toFIG. 2D. More particularly, the UE can receive at216a user selection to “Create New Song” at216. This user selection can be indicated by a user activation of a “Create New Song” GUI control356in home screen350. At218, the process displays the “Create New Song” GUI shown inFIG. 5A. Here, the user can insert a song title, set the beats per minute (BPM) that the new song is intended to have, and select a time signature for the song (e.g., 3/4 time or 4/4 time). As shown inFIG. 5B, the user can then proceed to set a title (e.g., “Soprano”) for a particular part that they will be creating.

The process then proceeds to step272inFIG. 2D. The system can wait at272for a user input selection indicating that the user is ready to begin recording a new harmony. The system determines at274whether a user has indicated that they are ready to begin recording a new harmony. If so (274: Yes) then the process continues on to step276where the UE uses a microphone and associated audio circuitry to record a harmony part. If creating a new song, then the harmony part can be any harmony part the user wishes to create.

As the UE records the harmony part, it can display a preliminary indication of the pitch of the user's voice as shown inFIG. 5C. The user's voice pitch can be presented in real time as the user is singing in the form of micro-notes502. As shown inFIG. 5C, the micro-notes are of relatively short duration. For example, in some scenarios each micro-note502can be 1 beat or less in duration. The UE will continue recording and wait for a user input to stop recording at278. When such an input is received (278: Yes) it will serve as an indication that the user is done creating the harmony part. At this point, the micro-notes are combined by the UE as shown inFIG. 5Dto form musical note blocks504. For example, this step can be performed in some scenarios through the use of post processing operations performed by the UE after the recording operation has been stopped.

The UE can then offer the user the opportunity to play the part at280, in which case the newly recorded harmony part will be played at282. The user may be provided with the opportunity at284to decide whether the newly recorded harmony part should be saved. If so (284: Yes) then the process continues on to step296inFIG. 2E.

At286the user can be prompted as to whether they would like to add another harmony part. If so, the process can return to272. Otherwise the user can elect to terminate the Build Harmony section at288. If so (288: Yes) then the session terminates at290or returns to the MHT home screen.

FIG. 5Eillustrates an example of a UE screen display in a scenario where the user has elected at286to add a further harmony part to the song. InFIG. 5Ethe part named “Soprano” has been created as “track1” and a second part (e.g. “Alto”) can be added by the user as “track2” by using the keypad510. If the user chooses to proceed with the creation of the second part (297: Yes) inFIG. 5E, they can activate a user interface control508to cause the UE to create the second harmony part. The process can then return to step272inFIG. 2D.

In some scenarios, the process can continue at298where the user can be prompted to indicate whether they wish to add a musical collaborator to participate in the creation of one or more harmony parts. If no collaborator is to be added (298: Yes) the process can terminate at299or can return to a home screen. If a collaborator is to be invited to participate, then the process can continue with step302shown inFIG. 2F.

At302the UE can receive user input from the user specifying an email address, user name, and/or role (e.g. a harmony part) of a potential collaborator. Thereafter, at304the UE can request and receive confirmation that the identified person is to be invited to participate as a collaborator with respect to a specified harmony or song. At308, the UE can communicate with the application server102to determine whether the identified person is an existing user of the MHT system described herein for which user account data is available. If so, then the application server102can cause the person to be notified (e.g. via email or text message) at309. For example, the application server102can send a message to the identified person that they have been invited to participate as a collaborator in connection with creation of a particular song.

In some scenarios, the person invited to collaborate is not an existing user (308: No), in which case the UE can cause the person to be notified (e.g. via email or text message) that they have been invited to participate in creating a song using the MHT tool. For example, such invitation can be generated by an application server102. A response may be subsequently received from an invitee at312(e.g. received at the application server). If the response indicates rejection of the invitation (314: No) then the application server can send a notification to the UE which initiated the invitation that the collaboration request has been declined. However, if the invitation is accepted (314: Yes) then the process continues to318where the invited user is requested at318to download an MHT software component (an application) to the invitee's UE. Thereafter, the invitee can download the software component to their UE and can create an MHT account at320by communicating account data to the application server102. Having downloaded the required software component and established themselves as a new user of the MHT system, the invited user or collaborator can then accept a particular song on which they have been invited to collaborate. The process terminates at324and/or can return to a system home screen.

During a process of creating a new song or modifying an existing song it can sometimes be desirable to edit an existing harmony track. This process is illustrated inFIGS. 6A and 6B. The process can involve displaying a note grid for a particular track that has been recorded. The note grid can be similar to the note grids described herein with respect toFIGS. 4C-4Dand/orFIG. 5D. The user can manipulate a highlighted vertical note region bar605. According to one aspect, the UE can be configured using a note size selector control610to vary a width p of the bar, and thereby choose from 16th, 8th, quarter, half, or whole notes.

In the scenario shown inFIG. 6A, the user has used next and back scroll controls602a,602bto selectively control which portion of a note grid is displayed. The user has set the note region bar to correspond to a quarter note. In the displayed note grid, the user can control operate user interface so that the note region bar605aligns with a particular note to be edited.

Once a particular note has been selected, the user can make use of a plurality of available editing controls608to perform a user initiated operation on the selected note. For example, the available editing control608can include a delete control612to permit the user to delete the selected note, an add control614to allow the user to add a new note, a merge control616to allow the merging of two existing notes in the note grid, and a divide control618which allows the user to divide a note into two notes of shorter duration.

In the example shown inFIGS. 6A and 6B, the note which the user has marked for editing is the third quarter note604in a measure606. The user marks this note604with note region bar605to indicate that note604has been selected for editing. The user then activates the divide control618to split the quarter note604from note622. The user further makes use of the user interface to lower the pitch of the note from G3 to F3. For example, in a UE equipped with a touch-screen, this adjustment in pitch to the highlighted or marked note604can be accomplished by using a finger to slide or drag the note604to correspond to a different note on the scale. The edited harmony track can be saved by the UE when the user has completed all necessary edits.

Real-time pitch detection (RTPD) processing in the solution presented herein is implemented by a novel technique involving evaluation of a monophonic audio signal as produced by an individual human singer. Specifically, it is designed to give accurate feedback to a singer about their actual pitch and amplitude relative to the known “correct pitch and amplitude”. The RTPD processing involves a frequency-domain based algorithm using a modified Fast Fourier Transform (FFT). The FFT is optimized for audio frequencies which are singable by the human voice (approx. 60 Hz-1400 Hz). The process is described below in further detail with reference toFIGS. 7 and 8.

The process begins at702and continues at704where a constant-Q transform is applied to detect pitch information of from an audio signal sample or chunk. As is known, a standard FFT transform can be used to faithfully convert chunks of time-domain data (audio signal) into frequency-domain data (spectrogram). In a conventional FFT, this frequency-domain spectrogram is linear. However, the human ear distinguishes pitch on a pseudo-logarithmic scale. Therefore, instead of a traditional FFT, the RTPD processing algorithm in the present solution uses a “constant-Q” transform where “Q” is the ratio of the center frequency to the bandwidth of a corresponding logarithmic filter.

The constant-Q transform is well-known in the field of signal processing and therefore will not be described herein in detail. However, it will be understood that the constant-Q transform is computed by applying a FFT after first applying a logarithmic filter to the underlying data. As is known, this process can be equivalently implemented as applying the FFT with a logarithmic kernel. See, e.g., Brown, J. C., & Puckette, M. S. (1992). An efficient algorithm for the calculation of a constant Q transform.The Journal of the Acoustical Society of America,92(5), 2698. http://doi.org/10.1121/1.404385).

In practice, a large number of parameters must be chosen for this constant Q transform, and these parameters can be optimized for the special case of giving real-time feedback to human singers. For example, an RTPD processing algorithm can advantageously involve the application of the following specific parameters, which were identified empirically as a result of experiments with audio recordings of professional singers:FFT bins per octave is set to 48, in order to create sufficient frequency resolution for feedback purposes while maintaining real-time performance on modern mobile devices.Frequency to bandwidth ratio of the logarithmic filter is set to a value between 250 and 300 Hz, which provides a good tradeoff between filtering spurious microphone/background noise and precision. For example, in some scenarios, the value can be chosen to be 275 Hz.Length of the audio signal used is set to 2048 samples (42.7 ms sample length for 48 kHz sampled audio, or 50 ms sample length for 41.1 kHz sampled audio), which provides sufficient frequency resolution for this application without introducing a significant processing delay before feedback can be provided to the singer.

As is known, the window size of an FFT that is selected for pitch detection will affect the frequency resolution. A bin is a spectrum sample of the audio signal, and can be understood as defining the frequency resolution of the FFT window. Frequency resolution for pitch detection is improved by increasing the FFT size, i.e., by increasing the number of bins per octave. However, an excessive number of bins can result in processing delays that render real-time pitch detection impractical. The parameters identified herein have been determined to provide acceptable resolution of singing pitch while still facilitating real-time processing for the benefit of the singer.

With the foregoing parameters applied, an RTPD processing algorithm detects the current pitch based on the constant Q transform of a short audio signal with N samples QA=Q(a0 . . . N) by detecting the peak frequencies and generating at706a histogram of potential “pitch candidates”. An example histogram800showing the results of this process are illustrated inFIG. 8A. The plot inFIG. 8Ashows estimated pitch frequency versus time. The particular plot shown inFIG. 8Awas obtained by applying the constant-Q transform described herein to audio of a tenor singing eight notes of a major scale. These eight notes start at a low frequency of approximately 220 Hz and end at a high frequency of approximately 440 Hz.

In histogram ofFIG. 8A, each bin804is represented as a shaded rectangle. Each column802of bins804is created from one sample of the audio signal. For example, in a scenario where the duration of the audio signal sample is set to 2048 as described above, the audio signal sample associated with each bin would have a duration of 42.7 ms (in the case of 48 kHz sampled audio). Further, it can be observed inFIG. 8A, that each rectangular bin804is shaded to an extent. In the example shown, the darker shaded rectangles represent higher estimated power levels in that bin. The bins which correspond to the eight major notes which are sung in this example are identified by reference number806.

The process continues by qualifying the pitch candidates obtained in706. To qualify pitch candidates, the RTPD processing algorithm can search those bins which correspond to each particular audio sample. For example, this search can proceed from lowest to highest frequency within a predetermined frequency search range to identify locally maximal bins (i.e., bins having a locally maximal power level) which are associated with a particular audio sample. Shown inFIG. 8Bis a plot which is useful for understanding this process.FIG. 8Bshows how the power level which is associated with bins covering various frequency ranges will vary with respect to frequency for given audio sample or chunk. This plot can be understood conceptually with reference toFIG. 8Awhich shows that bin power variations along an axis line812can be evaluated among a set of bins in a particular column that are associated with one particular audio sample or chunk. It can be observed inFIG. 8Bthat a resulting plot of bin power versus frequency will include one or more local maxima or local peaks814,816,818where the power level of particular frequency ranges is noticeably of greater magnitude as compared to other nearby frequencies. Each local maxima or peak will have a contour or shape which can be mathematically characterized.

Starting from the lowest frequency is useful to ensure that the correct peak is identified. Partial onsets with lower frequency (e.g., local peak814) do not typically have higher energy than the correct pitch (e.g., local peak816in this example). In contrast, for the case of human singing, partial onsets at higher frequencies (e.g., local peak818) will sometimes have slightly more energy in them as compared to the correct pitch. Accordingly, by scanning the peaks from the lowest to highest frequency and only accepting pitch candidates which exceed a certain threshold proportional to the current best (i.e., greatest magnitude) peak (for example at least 110% larger than the current best peak), partial onsets with a higher energy can be avoided in most cases. Further, the RTPD processing algorithm is advantageously optimized for human voice by searching a range 62 Hz-1284 Hz, and frequencies outside this range are not considered as potential matches. This optimization speeds up computation significantly.

In the flowchart shown inFIG. 7, the above-described qualifying process is referenced at708where, locally maximal bins in each column802are identified, after which a Gaussian distribution is fit to the data in the nearest five bins804relative to the locally maximal bin (two lower bins, the center bin which is the local maxima, and two higher bins). This fitting process allows the contours of the local maxima (e.g., local maxima814,816,818) to be characterized. According to one aspect, the Gaussian distribution can be fit to the data by using the maximum likelihood method. This fitting process is applied in this way as a basis to estimate and/or characterize the shape and amplitude of the peak. The power level pi, standard deviation σi, and mean frequency of that Gaussian distribution is recorded in a peak candidate list.

At710the process continues by correcting octave errors. As used herein, the term “octave error” refers to errors in pitch identification which are off by approximately one octave (known as a partial onset). When attempting to identify a pitch, conventional algorithms will often simply choose the pitch candidate having the greatest magnitude power level. But as may be appreciated from the plot inFIG. 8, a pitch candidate represented by a bin with the greatest magnitude of signal power does not always correspond to the true pitch in associated with an audio signal sample. Early-onset (lower than true pitch) and harmonic (higher than true pitch) partial onsets often accompany the voice of a singer due to the complexity of the human vocal system. These types of octave errors manifest themselves as the groups of bins (e.g. bin groups808,810) which appear below and above bins806inFIG. 8A. Conventional algorithms that do not account for these effects are prone to making octave errors. In this regard it may be noted that the term “octave error” is in some respects a misnomer, since early-onset and harmonic pitches are not in general off by precisely an octave. However, it remains a useful shorthand way of referring to pitch detection errors which are generally off by approximately one octave.

Octave errors in the solution presented herein are advantageously corrected using the following corrective function to re-score each bin associated with an FFT of a particular audio sample:

pi*=pi+∑ω∈Ω⁢∫x∈Gi⁢QA⁡(ω-x)·Gi⁡(x)⁢dx
where
pi, is the power of the i-th peak candidate,
Ωi∈N=69.203 ln(i) is a discrete set of partial onsets unique to human vocal system,
ω is a partial onset frequency in Ω
Giis the Gaussian distribution fit to the i-th peak candidate,
x is a frequency offset in the domain of Ω, and
QAis the constant-Q transform of the audio signal.
Conceptually, this equation adds additional energy to each peak candidate when energy distribution or shape of the peak within the partial onsets of that pitch look similar to the energy distribution of the peak candidate under consideration. An example of such a scenario is illustrated within column802ofFIG. 8, wherein a partial onset peak present in bin group808and/or810might look similar to a peak candidate bin in bin group806. Note that this corrective function presented herein only makes sense in the “constant Q” domain and not after a traditional FFT. This correction factor is effective at removing spurious pitch candidates without partial onset support (i.e. unlikely to be due to human singing) and reinforces the true pitch. At712the RTPD will determine the true pitch of each audio sample by selecting the pitch with largest p*ivalue from among the bins associated with that particular audio sample.

At714and716the RTPD processing algorithm removes common pitch tracking errors that occur during the beginning of sung notes and around harsh consonants where no true pitch exists. It achieves this result by using a bilateral filter to remove outlier detected pitches and by automatically filling holes or gaps where no pitch could be detected. As is known, a bilateral filter uses two kernels simultaneously. For purposes of the present solution, the spatial kernel is a Gaussian operating on the pitch frequency, and the range kernel is a Gaussian operating on the audio signal amplitude. This can also be thought of as a weighted average at each audio sample, where the weights are determined based on similarity in both pitch and amplitude. After outlier pitch values are removed, the result can be displayed at718. Thereafter, the process can terminate at720or continue with other processing.

Note Transcription Algorithm

The solution disclosed herein also concerns a process involving note transcription. This note transcription process is illustrated inFIGS. 9A and 9B. The process utilizes a note transcription (NT) algorithm. The NT algorithm is an audio analysis procedure for converting an audio recording of an individual singer (monophonic audio) to a sequence of non-overlapping notes with pitch and duration information (suitable for creating sheet music or a MIDI file, for example). While RTPD is optimized for real-time feedback and provides information on the pitch hundreds of times per second, NT is optimized for detecting individual notes aligned to the time signature and tempo of a song. In practice, NT must often make difficult decisions when singers are slightly between two keys (C vs C# for example), which necessitates deeper contextual awareness than RTPD. Utilization of two entirely different algorithms in this way is noteworthy in this context. By selectively applying a different audio analysis process or method to what might otherwise be understood to be basically the same problem, a non-obvious result is obtained for achieving improvements in both real-time pitch detection and in note transcription. These performance enhancements would not be possible one algorithm or the other was used exclusively for both types of audio analysis.

To facilitate creation of a new harmony, it is advantageous to have (1) the audio of the singer for each harmony track (independently) and (2) some user-specified timing information applicable of the “sheet music” (which can include one or more of the time signature, tempo, and notes) over the duration of the song. Accordingly, the process900can begin at902and continue at904where the system receives a user input manually specifying a time signature and tempo which is applicable to a transcribed sheet music.

Conventional note transcription solutions do not accept the song tempo as input, since this information is not available in a typical unstructured setting. In contrast, the NT algorithm is intended to be embedded into an application where this information can be provided by the user before a particular song is sung. Since the time signature and tempo are set in advance, the search space for possible note configurations is greatly reduced. Consequently, the NT algorithm is able to improve the accuracy and fidelity of transcription results over conventional note transcription solutions.

After the user has manually entered the time signature and tempo data, the user can sing and record each part or track of the harmony at906. Thereafter, the process can continue whereby the one or more audio tracks associated with the user singing is automatically analyzed by the NT algorithm. As explained below in greater detail, the result is an automatic production or transcription of the corresponding “sheet music”, which can later be edited.

According to one aspect, the NT algorithm can solve the transcription problem by converting the problem of audio transcription into a shortest-path problem. This process is best understood with reference toFIG. 10, which is a simplified diagram showing a shortest path through the network that defines the optimal pitch assignment for each note. In graph theory, the shortest path problem involves finding a path between two vertices (or nodes) in a graph such that the sum of the weights of its constituent edges is minimized. In the present solution, each column1004of nodes in a graph1000represents one eighth note duration in the song, and individual nodes1002will represent a possible pitch for this eighth note.

The weight assigned to each edge1006between adjacent nodes i and j is the negative log likelihood of the probability that the next note has the pitch associated with node j, assuming that node i was selected. This edge weight balances the probability that node j is in the song independent of any other information, along with a regularization factor that takes into account the previous note. The regularization is important in cases, for example, where a singer is slightly off pitch or oscillating between notes; in this case, even though a note may appear to be the most likely in isolation, making a global decision based on the entire path through the notes via the regularization factor allows us to choose one continuous pitch for several notes in a row rather than oscillate incorrectly between notes because each is slightly more likely in isolation. The shortest path from source s to sink t will include one node per column1004, identifying which pitch should be assigned to the eighth note.

There are several steps involved with the assigning weights to each edge1006of the graph. The process first segments at908the entire audio signal A into fractional notes or note segments based on the current known tempo: {A0, A1, A2, . . . }. In some scenarios described herein, the fractional notes or note segments can comprise eighth notes. The specific choice of eighth notes is arbitrary, and not considered critical to the solution. Accordingly, other note durations are also possible. Still, for the purposes of the presenting the solution herein, it is convenient to describe the process with respect to eighth notes. Each eighth note of audio is then subsampled at910into multiple standard length (potentially overlapping) “chunks” {c0, c1, . . . , cM}∈Ai. In a solution presented herein, these chunks can comprise 4096 subsamples (each 93 ms in duration for 44.1 kHz audio, and 85 ms in duration for 48 kHz audio) and eight chunks per note are selected. However, the solution is not limited in this regard and subsamples of different durations and alternative note segmentations are also possible.

The chunks of audio signal obtained at910can either be sampled regularly from the eighth note of audio, or sampled at randomized times within the eighth note. According to one scenario, the chunks are randomly sampled from among a distribution that samples more frequently near the middle of the eighth note. Such a scenario can help avoid data obtained near the leading and trailing edges of notes that might contain samples of previous/next notes when the singer is not perfectly on tempo.

After the chunks have been obtained at910, the process continues by computing at912the autocorrelation R (T) of each chunk with itself, normalized by the size of the overlapping domain, to detect the probability of each frequency per chunk. This process can be understood with reference to the following expression:

R⁡(τ)=∑i=04096⁢(c⁡(i+τ)-c⁡(i))24096-τ
whereτ is the lag time in samples,c(i) is the ithsample from audio in the chunk, andc(i+τ) is a sample from the same chunk, offset by the time lag.R(τ)=0 for an audio signal perfectly periodic every i samples. The frequency probability is defined as S(f)=R(4096/f)−1.

An example result of the foregoing autocorrelation process is presented inFIG. 11.FIG. 11is a graph showing sample index plus frequency probability value (y axis), plotted against musical note values (x axis) for a small selection of chunks. InFIG. 11, each line1102corresponds to one chunk and the peaks (e.g., peaks1104a,1104b) in each line indicate a local probability maxima corresponding to certain frequencies listed along the frequency axis. For example, it can be observed inFIG. 10that local maxima1104acorresponds approximately to note E2, and local maxima1104bcorresponds approximately to note B2. In this regard,FIG. 11is useful for illustrating one difficulty with autocorrelation-based methods for evaluating frequency probability. In particular, it can be observed that for any given chunk or line, a local probability maxima is often detected at multiples of the true note frequency.

The occurrence of multiple local probability maxima as shown inFIG. 11can be understood as true octave errors, and are due to the periodic nature of the audio signal. To correct for this problem and in order to assign the appropriate probabilities, process900continues at914by further processing each monotonically increasing local maxima in order from lowest to highest frequency {0, f0, f1. . . , fN}, and assigning them probabilities. Probabilities are advantageously assigned by making use of a mathematical process that gives preference to the original lowest frequency when the probabilities are similar, without over-penalizing frequencies that may have a peak undertone as long as the probability is significantly lower. An example of such a process is the integral of a Kumaraswamy distribution over the probability range, {0, f0, f1, . . . , fN}, as follows:

These probabilities are then quantized at916to their nearest notes on the western musical scale. For example, in some scenarios this quantization can be accomplished by linearly distributing the probability to the nearest note centers.

Another difficulty when detecting pitch in human singing is vibrato (notes sung with significantly oscillating pitch). Such vibrato type singing diffuses the note probability between adjacent notes, and can cause a single long note to appear as many oscillating nearby notes. The NT algorithm in process900can correct for this potential error at918. The vibrato correction involves introducing a regularization factor to the edge weights906, equal to a Laplace distribution

At920, to compute the final edge weights906between any two adjacent nodes i and j902, we use every chunk from the j-th eighth note, and average the negative log likelihood of that node's pitch being the correct pitch for this note:

E⁡(i,j)=1M⁢∑M⁢-log⁡(l⁡(fi-fj)*P⁡(fj))
where
i, j indicate two nodes in the graph,
M is the number of chunks,
l(fi-fj) is the Laplace distribution applied to the difference in frequency of the nodes,
P(fj) is the note probability indicated above.
Edge weights connected to the source or sink node are not important in this case (all set to 1).
Intuitively, the edge weight equation balances the probability of each note in isolation based on just the eighth note's audio signal, with a regularization factor that prefers long uninterrupted notes over many quick back and forth notes, which is important in cases where a singer may be off-pitch and halfway between two options, or in cases of vibrato.

The process continues at922by inserting one or more “virtual note” nodes in the graph to represent periods of silence or the lack of any sung note. In other words, each eighth note can be assigned f=∅ to indicate periods of silence or a lack of any actual note. The edge weight leading to each silent node is set to:

E⁡(i,s)=1M⁢∑M⁢(1-c_10⁢⁢A_)
where
i is any node and s is the silent “virtual note” node,
M is the number of chunks,
cis the average intensity of the audio signal in one chunk, and
Ā is the average intensity in the audio signal from the entire song.
This sets the edge weight based on how loud a chunk is compared to the average song loudness (quiet chunks are more likely to indicate a silent note). Note also that no vibrato correction is applied for silent notes.

At924the process continues by constructing the graph using for each node the musical notes identified in steps908-922. The shortest path through this graph can be solved efficiently using conventional methods at926. This solution will represent the optimal (highest probability) note selection, since the minimum sum of edge weights (i.e. shortest path) equals the highest product of probabilities:
min(Σ−log(P))=−Σmax(log(P))=−log(max(ΠP))
The path determined in926can be converted at928into a sequence of eighth notes. According to one aspect, adjacent notes with the same pitch can be merged at930into larger notes (quarter, half notes, and so on). However, other solutions are also possible and it may be useful to consider changes at the edges of notes to determine whether to merge notes or leave them separated. The note transcription process can then terminate at932or can continue with other processing. The transcribed note data resulting from the process can be imported into the application along with the recorded singing track.

The systems described herein can comprise one or more components such as a processor, an application specific circuit, a programmable logic device, a digital signal processor, or other circuit programmed to perform the functions described herein. Embodiments can be realized in one computer system or several interconnected computer systems. Any kind of computer system or other apparatus adapted for carrying out the methods described herein is suited. A typical combination of hardware and software can be a general-purpose computer system. The general-purpose computer system can have a computer program that can control the computer system such that it carries out the methods described herein.

Embodiments of the inventive arrangements disclosed herein can be realized in one computer system. Alternative embodiments can be realized in several interconnected computer systems. Any kind of computer system or other apparatus adapted for carrying out the methods described herein is suited. A typical combination of hardware and software can be a general-purpose computer system. The general-purpose computer system can have a computer program that can control the computer system such that it carries out the methods described herein. A computer system as referenced herein can comprise various types of computing systems and devices, including a server computer, a personal computer (PC), a laptop computer, a desktop computer, a network router, switch or bridge, or any other device capable of executing a set of instructions (sequential or otherwise) that specifies actions to be taken by that device. In some scenarios, the user equipment can comprise a portable data communication device such as a smart phone, a tablet computer, or a laptop computer.

Referring now toFIG. 12, there is shown a hardware block diagram comprising a computer system1200. The machine can include a set of instructions which are used to cause the computer system to perform any one or more of the methodologies discussed herein. In a networked deployment, the machine can function as a server, such as application server102. In some scenarios, the exemplary computer system1200can correspond to each of the user equipment computer systems1061-106n. In some embodiments, the computer1200can operate independently as a standalone device. However, embodiments are not limited in this regard and in other scenarios the computer system can be operatively connected (networked) to other machines in a distributed environment to facilitate certain operations described herein. Accordingly, while only a single machine is illustrated it should be understood that embodiments can be taken to involve any collection of machines that individually or jointly execute one or more sets of instructions as described herein.

The computer system1200is comprised of a processor1202(e.g. a central processing unit or CPU), a main memory1204, a static memory1206, a drive unit1208for mass data storage and comprised of machine readable media1220, input/output devices1210, a display unit1212(e.g. a liquid crystal display (LCD), a solid state display, or a cathode ray tube (CRT)), and a network interface device1214. Communications among these various components can be facilitated by means of a data bus1218. One or more sets of instructions1224can be stored completely or partially in one or more of the main memory1204, static memory1206, and drive unit1208. The instructions can also reside within the processor1202during execution thereof by the computer system. The input/output devices1210can include a keyboard, a mouse, a multi-touch surface (e.g. a touchscreen). The input/output devices can also include audio components such as microphones, loudspeakers, audio output jacks, and so on. The network interface device1214can be comprised of hardware components and software or firmware to facilitate wired or wireless network data communications in accordance with a network communication protocol utilized by a data network100,200.

The drive unit1208can comprise a machine readable medium1220on which is stored one or more sets of instructions1224(e.g. software) which are used to facilitate one or more of the methodologies and functions described herein. The term “machine-readable medium” shall be understood to include any tangible medium that is capable of storing instructions or data structures which facilitate any one or more of the methodologies of the present disclosure. Exemplary machine-readable media can include magnetic media, solid-state memories, optical-media and so on. More particularly, tangible media as described herein can include; magnetic disks; magneto-optical disks; CD-ROM disks and DVD-ROM disks, semiconductor memory devices, electrically erasable programmable read-only memory (EEPROM)) and flash memory devices. A tangible medium as described herein is one that is non-transitory insofar as it does not involve a propagating signal.

Embodiments disclosed herein can advantageously make use of well-known library functions such as OpenAL or AudioKit to facilitate reading and writing of MP3 files, and for handling audio input/output functions. These audio input/output functions can include for example microphone and speaker connectivity, volume adjustments, wireless networking functionality and so on).

Further, it should be understood that embodiments can take the form of a computer program product on a tangible computer-usable storage medium (for example, a hard disk or a CD-ROM). The computer-usable storage medium can have computer-usable program code embodied in the medium. The term computer program product, as used herein, refers to a device comprised of all the features enabling the implementation of the methods described herein. Computer program, software application, computer software routine, and/or other variants of these terms, in the present context, mean any expression, in any language, code, or notation, of a set of instructions intended to cause a system having an information processing capability to perform a particular function either directly or after either or both of the following: a) conversion to another language, code, or notation; or b) reproduction in a different material form.

The MHT described herein advantageously combines qualitative self-assessment through hearing singer's voice with objective assessment through visualization of pitch, dynamics, and rhythm. It shows target notes and actual dynamics in an innovative way. It further allows for collaborative, asynchronous, virtual harmonizing via recording and track sharing. Multiple tracks of different formats can be synced with recorded tracks to allow seamless practicing in several modes and instant feedback. A pitch of each note that has been sung and the amplitude of each note can be updated at a rate which is selected to provide a smooth and accurate user experience with respect to note tracking and feedback. For example, in some scenarios, note updates can occur at a rate of every 18 ms (60 Hz) or 33 ms (30 Hz). Of course these values are merely provided as examples and are not intended to limit the range of note update frequency. The display format facilitates visualization of many aspects of the voice, with a particularly innovative treatment of the voice's amplitude (volume), a design challenge met with a volume meter (a perpendicular bar that runs horizontally along a note, with the left being lowest volume and the right being loudest).

The described features, advantages and characteristics of the various solutions disclosed herein can be combined in any suitable manner. One skilled in the relevant art will recognize, in light of the description herein, that the disclosed systems, devices and/or methods can be practiced without one or more of the specific features. In other instances, additional features and advantages may be recognized in certain scenarios that may not be present in all instances.