Automatic composer

Methods and apparatuses are disclosed for automatically composing a song are disclosed. In an embodiment, a method includes receiving music performance data by a processor. The processor may then segment the music performance data based on at least one structural attribute into at least a first musical segment, where the first musical segment is associated with at least one musical attribute. The processor may then determine an affinity value for the first musical segment based on the at least one musical attribute. The affinity value represents a degree of similarity between the first musical segment and a second musical segment having the at least one musical attribute. The processor may then generate a musical composition based on the affinity values associated with the first musical segment and the second musical segment.

CROSS-REFERENCES TO RELATED APPLICATIONS

The following regular U.S. patent applications are being filed concurrently, and the entire disclosure of the other applications are incorporated by reference into this application for all purposes:Application Ser. No. 14/871,978, filed Sep. 30, 2015, entitled “Automatic Music Recording and Authoring Tool”;Application Ser. No. 14/871,982, filed Sep. 30, 2015, entitled “Automatic Music Recording and Authoring Tool”;Application Ser. No. 14/871,902, filed Sep. 30, 2015, entitled “MUSIC ANALYSIS PLATFORM”; andApplication Ser. No. 14/871,897, filed Sep. 30, 2015, entitled “MUSIC ANALYSIS PLATFORM”.

BACKGROUND

Musical compositions are pieces of musical work that contain an arrangement of melody, harmony, and rhythm. Creators of such musical compositions are known as composers, who decide how the melody, harmony, and rhythm are arranged. Modern technology has advanced to assist composers in developing musical compositions. For instance, software applications have been developed to provide composers an interface with which musical pieces may be constructed and sampled (e.g., heard by the composer) in real time. These types of software applications perform calculations on a digital representation of a musical piece which may be referred to as “music performance data.” The music performance data may then be manipulated by such software applications. Often, composers utilize modern technology to develop music compositions from beginning to end in one progression. Despite these technological advances, however, modern technology limits composers' abilities to experience different variations of their work, thereby stifling their creative potential. Accordingly, improvements to such modern technology are desired.

SUMMARY

Embodiments provide methods and systems for automatically generating a musical composition from music performance data to provide an interactive way of inspiring a composer to create musical pieces.

In some embodiments, a method includes receiving, by a processor, music performance data, and segmenting, by the processor, the music performance data based on at least one structural attribute into at least a first musical segment. The first musical segment may be associated with at least one musical attribute. Also, the first musical segment may have at least one of a corresponding prologue, epilogue, and verse. The method may include determining, by the processor, an affinity value for the first musical segment based on the at least one musical attribute. The affinity value may represent a degree of similarity between the first musical segment and a second musical segment having the at least one musical attribute. The method may further include generating, by the processor, a musical composition based on the affinity values associated with the first musical segment and the second musical segment.

In certain embodiments, a non-statutory computer-readable medium having a computer-readable program code configured to cause a processor to perform operations including receiving music performance data and analysis data, and segmenting the music performance data based on at least one structural attribute into at least a first musical segment. The first musical segment may be associated with at least one musical attribute. Additionally, the first musical segment may have at least one of a corresponding prologue, epilogue, and verse. The operations may include determining an affinity value for the first musical segment based on the at least one musical attribute. The affinity value may represent a degree of similarity between the first musical segment and a second musical segment having the at least one musical attribute. The operations may further include generating a musical composition based on the affinity values associated with the first musical segment and the second musical segment.

In some embodiments, a system may include a user interface, one or more data processors coupled to the user interface, and one or more non-transitory computer-readable storage media containing instructions configured to cause the one or more data processors to perform operations including receiving music performance data and analysis data, and segmenting the music performance data based on at least one structural attribute into at least a first musical segment. The first musical segment may be associated with at least one musical attribute. Additionally, the first musical segment may have at least one of a corresponding prologue, epilogue, and verse. The operations may include determining an affinity value for the first musical segment based on the at least one musical attribute. The affinity value may represent a degree of similarity between the first musical segment and a second musical segment having the at least one musical attribute. The operations may further include generating a musical composition based on the affinity values associated with the first musical segment and the second musical segment, and presenting the musical composition to the user interface.

DETAILED DESCRIPTION

Embodiments describe a method and system for an automatic composer, which can be configured to automatically generate a musical composition from music performance data, or assist the user in re-composing that music performance data. Music performance data may be one or more representations of sound. For instance, music performance data may be a piece of music in the form of a digital recording. The automatic composer may utilize the music performance data to generate a musical composition. The musical composition may include musical segments that are arranged differently than when the musical segments were originally arranged in the music performance data. In embodiments, the generated musical composition may be presented to a user, e.g, played and/or modified and/or displayed to the user.

To generate a musical composition, an automatic composer may segment music performance data into one or more musical segments according to embodiments of the present invention. Each musical segment may then be assigned information pertaining to its musical profile. For instance, an affinity value representing a degree of similarity between two musical segments may be calculated and assigned to each of the two musical segments. Depending on the affinity value, the musical segments may then paired with one another to form a part of, or an entire, musical composition. The musical composition may be generated without extensive interaction from a user.

Embodiments allow a user to automatically create musical compositions. The user does not need to manually segment music performance data into musical segments by hand, nor does the user need to manually recompose the musical segments together into a musical composition. Additionally, the recomposed musical segments may have similar musical sound such that the musical composition is a cohesive musical piece. As a result, embodiments may save the user a substantial amount of time and effort, while also allowing the user to modify music performance data in various ways that were not originally imagined.

I. Audio Processing System

The automatic composer, according to embodiments, may be part of a post-processing system for an audio processing system. That is, the automatic composer may receive data from the audio processing system, and may utilize that data to automatically generate musical compositions, according to embodiments of the present invention. To better understand how the automatic composer plays a role in a larger system, the audio processing system will be discussed herein.

FIG. 1is a schematic diagram depicting an audio processing system100according to certain aspects of the present disclosure. The audio processing system100can be embodied in one or more pieces of hardware, such as a single device (e.g., smartphone or computer), multiple devices directly coupled together (e.g., a rack of equipment), multiple devices remotely coupled together (e.g., multiple computers communicatively coupled together via a network), or any combination thereof. The audio processing system100can include an audio processor108capable of accessing audio data. Audio data can include any data received by the audio processor108that is representative of a sound. Audio data can be provided as an audio signal120or an audio file122.

An audio signal120can be any analog or digital signal being performed or created in real time. In some cases, audio signals120can be created by a live instrument102and provided to the audio processor108through an audio input104. In some cases, audio signals120can be sound waves originating from a live instrument102(e.g., an acoustic guitar, a piano, a violin, a flute, or other traditional or non-traditional instrument capable of producing sound waves) that are picked up by an audio input104that is a microphone (e.g., a dynamic microphone, condenser microphone, ribbon microphone, fiber optic microphone, condenser microphone, hydrophone, or any other device capable of generating an electrical signal representative of a sound wave). In some cases, audio signals120can originate from voice (e.g., a singer or chorus), speakers (e.g., a pre-recorded sound or a live-played sound), nature-based sounds (e.g., wind noises or water noises), or other sources besides traditional instruments which can be received by an audio input104that is a microphone.

In some cases, audio signals120can be analog electrical signals originating from a live instrument102(e.g., electric guitar, electric piano, electric violin, Theremin, or other traditional or non-traditional instrument capable of producing an electrical signal corresponding to a sound wave) and received by an audio input104that is a line input.

In some cases, audio signals120can be digital signals originating from a live instrument102(e.g., a Musical Instrument Digital Interface (MIDI) controller, a computer-based digital instrument, or other traditional or non-traditional instrument capable of producing a digital signal representative of a sound wave) and received by an audio input104that is a digital signal processor. In some cases, audio signals120that are digital signals can be provided directly to the audio processor108.

In some cases, other equipment, such as preamplifiers, digital signal processors, compressors, analog-to-digital converters, and the like, can be included as part of the audio input104or coupled between the audio input104and the audio processor108.

In addition to or instead of receiving an audio signal120, the audio processor108can receive audio data in the form or an audio file122. Audio file122can be any audio data stored in a file that is representative of an audio signal120, such as a waveform audio file, Moving Picture Experts Group (MPEG)-1 or MPEG 2 Audio Layer III (MP3) file, Apple Lossless Audio Codec (ALAC), or any other file containing audio data. In some cases, an audio file122can be included in a file containing more than just audio data, such as a video file or other file. The audio file122can be stored on a data store106. Data store106can be any storage medium accessible to the audio processor108, such as built-in memory (e.g., flash storage in a smartphone), external memory (e.g., an external hard drive of a computer), or remotely accessible memory (e.g., a hard drive of a computer accessible to the audio processor108via a network, such as the internet). In some cases, an audio file122can be generated in real time (e.g., by a computer-based instrument) and need not be previously stored in a data store prior to being provided to the audio processor108.

In some cases, the audio file122is a streaming file that is provided to the audio processor108through a communication link, such as a wireless or wired network connection. The streaming file can originate from a remote source, such as a recording device placed a distance from the audio processor108or a server accessible through a network (e.g., the Internet). In an example, a smartphone can act as a recording device and can be coupled to a computer via a communication link (e.g., WiFi or Bluetooth connection), where the computer acts as the audio processor108. In that example, the smartphone can receive audio signals120at a microphone and store the audio signals as an audio file122which can be transmitted to the computer for further processing.

The audio processor108can process any incoming audio data. The audio processor108can include one or more of an automatic start/stop engine110, an audio recording engine112, an audio analyzing engine114, and an audio buffer116. The audio processor108can include more or fewer components. The audio processor108can be embodied in one or more data processors, such as central processing units (CPUs), application-specific integrated circuits (ASICs), microprocessors, or other devices or components capable of performing the functions associated with the audio processor108.

The audio buffer116can include memory capable of storing incoming audio data. The audio buffer116can be stored on volatile or non-volatile memory. The audio buffer116can store a predetermined amount of audio data, such as a predetermined size (e.g., in bytes) or a predetermined length (e.g., in seconds) of audio data. In some cases, the audio buffer116can store the last n seconds of incoming audio data. The audio buffer116can overwrite itself in real time so that the last n seconds or last n bytes of audio data are always available. In an example, the audio buffer116can store approximately five seconds worth of audio data, although shorter or longer audio buffers116can be used. In some cases, the size or length of the audio buffer116can be manually set, such as by a setting of a program or application utilizing the audio buffer116. In some cases, the size or length of the audio buffer116can be automatically set, such as automatically increasing the size of the audio buffer116if a determination is made that current size of the audio buffer116is insufficient for its current purposes, or automatically decreasing the size of the audio buffer116if a determination is made that the current size of the audio buffer116exceeds is current purposes. In some cases, the size of the audio buffer116can be automatically scaled based on certain settings or parameters, such as a recording mode (e.g., more or less sensitive), input choice (e.g., line input versus microphone input), environmental parameters (e.g., noisy environment versus a quiet environment or steady noise environment versus an environment with occasional disruptive noises).

The automatic start/stop engine110can include one or more of an automatic start detector and an automatic stop detector. The automatic start/stop engine110can process incoming audio data (e.g., from an audio input104, from a data store106, or from the audio buffer116). In some cases, the automatic start/stop engine110can dynamically analyze the contents of the audio buffer116to determine if a start event has occurred. In some cases, the automatic start/stop engine110can dynamically analyze and compare the first half of the audio buffer116with the second half of the audio buffer116to determine if a start event has occurred in the middle of the audio buffer116.

The automatic start/stop engine110can look for characteristics (e.g., mathematical, calculated, musical, or other characteristics) of the audio data that are indicative of a start event. The start event can correspond to a time at which a desired action is to take place. For example, upon detecting a start event, the automatic start/stop engine110can initiate recording of the incoming audio data, such as by copying some or all of the audio buffer116(e.g., that portion of the audio buffer116that occurs at or after the start event) into an audio file124of a data store118and begin appending audio file124with real time audio data using the audio recording engine112. Upon detecting a start event, the automatic start/stop engine110can also initiate analysis of the incoming audio data using the audio analyzing engine. The automatic start/stop engine110can trigger other tasks upon detection of a start event.

In some cases, the automatic start/stop engine110can look for a pre-determined start event, such as the presence of musical content in the audio data. In some cases, the automatic start/stop engine110can look for other start events, such as detection of a count-off (e.g., speech recognition of “one, two, three, four”) or detection of a particular characteristics such as a note, chord, or sequence of notes or chords (e.g., if a user wishes to record a second take of an existing recording, the automatic start/stop engine110can detect when the incoming audio data has characteristics similar to the beginning characteristics of the existing recording). In some cases, the automatic start/stop engine110can be used to trigger an action upon detection of musical content, versus noise or non-musical speech.

The automatic start/stop engine110can also analyze incoming audio data to determine a stop event (e.g., similarly to how a start event is determined). The stop event can be similar to and opposite from the start event, or can be otherwise defined. Upon detection of the stop event, the automatic start/stop engine110can trigger an action to stop (e.g., recording of incoming audio data) or trigger another action to be performed (e.g., transmitting the audio file124or beginning of post-processing the audio file124). In an example use case, an automatic start/stop engine110can be used to automatically remove non-musical content from a radio station being recorded; the automatic start/stop engine110can automatically start recording (e.g., to create a new audio file124or append an existing audio file124) upon detection of musical content and can automatically stop or pause recording upon detection of non-musical content.

According to embodiments of the present invention, audio file124may include music performance data, which may be data that represents the detected musical performance containing musical content. The music performance data may be further processed by an automatic composer to allow a user to automatically compose a new song by rearranging segments of the music performance data into a new musical composition.

The audio recording engine112can store incoming audio data as an audio file124stored on a data store118. The data store118can be the same data store as data store106, or can be a different data store118. Data store118can be any suitable storage medium accessible to the audio processor108, such as internal memory, external memory, or remote memory. In some cases, audio recording engine112can access audio buffer116to prepend any incoming audio data with some or all of the audio data stored in the audio buffer116. In some cases, the audio recording engine112can append an existing audio file124, such as if an audio file124was created using some or all of the audio data stored in the audio buffer116.

The audio analyzing engine114can process incoming audio data (e.g., from live audio signals120or existing audio files122) to generate metadata126related to audio data124. The metadata126can correspond to musical properties of the audio data, such as a melody transcription, a chord transcription, one or more key signatures, or other such musical properties of the audio data. The metadata126can be stored as an independent file on the data store118and be related to the audio file124. In some cases, the metadata126and the audio file124can be stored as parts in the same data file. In some cases, metadata126can be encoded directly into the audio file124(e.g., as signals that are demodulatable from the audio signal in the audio file124).

The audio analyzing engine114can perform one or more of real time (e.g., approximately real time or dynamic) and non-real time (e.g., post-processing of an entire audio file124) analysis of audio data. In some cases, the audio analyzing engine114can perform an initial real time analysis of incoming audio data (e.g., as being played from a live instrument102) to determine some musical properties or estimates of musical properties, and then perform an additional non-real time analysis of the audio file124to determine some musical properties or validate estimated musical properties.

In some cases, an audio analyzing engine of another device (e.g., a remote server) can perform additional processing to determine or validate one or more musical properties of the audio data (e.g., of audio file124). In some cases, the audio processor108can transmit the audio file124, the metadata126, or both to the other device for further processing. For example, the further composing may include automatically composing a song utilizing an automatic composer, according to embodiments of the present invention. Upon processing the received data, the other device can transmit new or updated data to the audio processor108(e.g., a new audio file124, new metadata126, or both). Continuing along the aforementioned example, the new or updated data may be a musical composition containing musical segments that are rearranged from music performance data as contained in audio file124.

In some cases, the audio processor108can be coupled to an output device, such as a display130or an audio output132, although other output devices can be used. The audio processor108can produce outputs through the output device(s) related to any processes occurring in the audio processor108, such as an audio analyzing process. In an example, the audio analyzing engine114can output musical properties to a display130(e.g., computer monitor or smartphone screen) in real time while the audio data is being received by the audio processor108. In another example, the audio analyzing engine114can use the detected musical properties to generate an accompaniment (e.g., a bass line generated based on detected chord progressions) which can be played through an audio output132(e.g., a speaker or line out).

As described herein, the audio processor108can output data (e.g., audio files124and metadata126) to a data store118. In some cases, outputting data can involve transmitting (e.g., streaming over a network connection) the data to a another device. For example, an audio processor108of a smartphone can receive an audio signal120from a live instrument102, record incoming audio data as an audio file124, analyze the audio data using the audio analyzing engine114to generate metadata126, and transmit the audio file124and metadata126(e.g., through real time streaming) to a computer located remote from the smartphone.

A. Recording Environment

FIG. 2is a schematic diagram depicting a recording environment200according to certain aspects of the present disclosure. An input phase222and an output phase224are shown. During the input phase222, the an audio processing device202can receive audio data from one or more sources. During the output phase224, the audio processing device226, which can be audio processing device202at a later point in time or another audio processing device, can process or display metadata228related to the audio data received during the input phase222. An audio processing device202,226can be any suitable device for receiving and processing audio data, such as a smartphone having a line input208(e.g., ⅛″ headset jack) and a microphone210. An audio processing device202,226can be the audio processing system100ofFIG. 1. The elements ofFIG. 2are not necessarily shown to scale.

The audio processing device202can receive audio data through a cable206coupled to the line input208. The line input208can receive line level, microphone level, or other level input. Any suitable instrument or audio device can be coupled to the cable206, such as an guitar204having an electric pickup. Examples of other suitable audio devices include electric pianos, microphone preamplifiers, a media player (e.g., MP3 player or compact disc player), a media receiver (e.g., radio receiver or internet streaming audio receiver), or other device capable of generating an audio signal. In some cases, the line input208can be coupled to multiple instruments or audio devices through the use of splitters, mixers, or other such audio equipment.

The audio processing device202can receive audio data through a microphone210. The audio data can be sound waves218from an instrument216or sound waves214from another audio source. An instrument216can be any traditional or non-traditional instrument capable of generating acoustic sound waves detectable by microphone210. Examples of other audio sources include a speaker212(e.g., home stereo speakers or loudspeakers at a public venue), nature-based sounds (e.g., wind noises or water noises), or any other source of sound waves214.

The audio processing device202can receive audio data from one or more audio sources at a time. For example, the audio processing device202can receive audio data from multiple instruments216through the microphone210, multiple instruments214through the line input208, or multiple instruments204,216through the line input208and microphone210, respectively.

The audio processing device202can perform operations on the incoming audio data, such as those described herein and with reference to audio processor108ofFIG. 1. The operations may result in generation of metadata that may be used for post-processing.

FIG. 3is a schematic representation of a metadata usage environment300according to certain aspects of the present disclosure. Metadata usage environment300can be any environment for making use of metadata304associated with audio data302. Metadata304and audio data302can be stored (e.g., in a file on a data store, such as data store118ofFIG. 1) or can be provided in real time (e.g., approximately real time) from an audio analyzing engine (e.g., audio analyzing engine114ofFIG. 1). In embodiments, metadata usage environment300may post-process metadata to perform useful functions, such as functioning as an automatic accompaniment engine, a segmenting engine, an automatic composing engine, and a song metrics analyzing engine, as will be discussed herein.

The metadata usage environment300can operate on a suitable device, such as an audio processor (e.g., audio processor108ofFIG. 1), an audio processing device (e.g., audio processing device202,226ofFIG. 2), or any other device suitable for making use of the metadata304, such as a computer or smartphone. Several examples for using the metadata304are described with reference to the metadata usage environment300, however the metadata304can be used in additional ways as well.

The metadata usage environment300can include an automatic accompaniment engine306. The automatic accompaniment engine can use received metadata304, and optionally received audio data302, to generate an accompaniment. The accompaniment can be a collection of musical notes, chords, drum beats, or other musical sounds determined to musically fit with the audio data302. The automatic accompaniment engine306can use musical properties identified in the metadata304associated with the audio data302to determine an accompaniment that satisfies a harmonic or musical fit with the audio data302.

For example, audio data302may include a melody316played by a guitar314. The metadata304may include a melody transcription for the melody316played by the guitar314, as well as an identified key signature for the audio data302. The automatic accompaniment engine306can use the key signature and melody transcription from the metadata304to identify other notes to play that would fill possible chords at various points in the piece (e.g., at the downbeat of every two measures). A device318(e.g., a smartphone or computer) implementing the automatic accompaniment engine306can play an accompaniment320based on the notes identified to fill possible chords. In some cases, the accompaniment320can be saved as another audio file or added to the audio data302. In other cases, the accompaniment320can be performed by the device318(e.g., through a speaker, a line output, or a MIDI output to a MIDI instrument) as the audio data302is being played. In some cases, where the audio data302and metadata304are being provided in real time, the device318may generate an accompaniment320to play along with a live performer.

The automatic accompaniment engine306can use any metadata304to generate the accompaniment. In some cases, certain metadata304can have a stronger weighting than other metadata (e.g., an identified key can have a stronger weight towards identifying what notes to play in an accompaniment than a melody transcription). The automatic accompaniment engine306can assign a confidence score for each attribute of the accompaniment (e.g., when to play a sound, for what duration to play the sound, what notes or chords to include in the sound, and the like) based on how well that attribute fits with the metadata304.

Metadata usage environment300can include an automatic musical segmenting engine308. The automatic musical segmenting engine308can use metadata304to split audio data302into a collection322of musical segments324,326. Any number of musical segments can be included in a collection322. The automatic musical segmenting engine308can segment the audio data302based on musical attributes, such as chords, tempos, key signatures, measures, meters, musical figures, musical motifs, musical phrases, musical periods, musical sections, and other such attributes that are discernable from the audio data302, metadata304, or both.

In an example, audio data302for a song may have associated metadata304that includes rhythmic data and melody transcriptions. The automatic musical segmenting engine308can identify any combination of rhythmic patterns and melody patterns and segment the audio data302where the patterns repeat to create audio segments324,326. In another example, the automatic musical segmenting engine308can simply use rhythmic data (e.g., from metadata304) to determine the downbeat of measures and segment the audio data302according to a manually set number of measures.

The metadata usage environment300can include an automatic composing engine310.

Automatic composing engine310may include lines of code and/or hardware and accompanying firmware configured to operate as an automatic composer, according to embodiments of the present invention. The automatic composing engine310can create a song328by piecing together any number of individual audio segments330,332,334,336. The song328can include only unique audio segments330,332,334,336(e.g., no audio segment repeats), or can include one or more repeating audio segments (e.g., audio segment330in the example shown inFIG. 3). Each audio segment330,332,334,336can be a segment324,326(e.g., from the automatic musical segmenting engine308). In some cases, each audio segment330,332,334,336is a distinct audio file that has not been processed by an automatic musical segmenting engine308.

The automatic composing engine310can use metadata304associated with the segments330,332,334,336to determine a desirable order in which to arrange the audio segments330,332,334,336. The automatic composing engine310can determine a correlation score between the beginning and ending of each audio segment330,332,334,336and arrange the audio segments330,332,334,336based on the correlation scores. The correlation scores can take into account musical properties, such as key, melodic transcription, chord transcription, rhythmic data, tempo, and other such properties. Other evaluation methods can be used to determine a musical affinity between adjacent segments.

In some cases, the automatic composing engine310can specifically select an order of audio segments330,332,334,336that is designed to produce an interesting song328(e.g., having varied musical properties between adjacent segments). For example, an automatic composing engine310may create a song328that includes a segment330identified as having a first chord progression, followed by a segment332identified as having a second chord progression in the same key as segment330, followed by segment330again, followed by a segment334identified as having only melody transcription and no chord transcriptions, followed by a segment336identified as having a resolution (e.g., a held consonance note after a dissonant chord).

In some cases, one or more segments can be identified as an intro or outro segment, in which case the automatic composing engine310can use those segments exclusively at the beginning or end of the song328, respectively. Intro and outro segments can be identified manually or automatically. Automatically identified intro and outro segments can be identified based on presence in an original piece (e.g., the first and last segments corresponding to the beginning and end of an audio file processed by an automatic musical segmenting engine308may be automatically labeled as intro and outro, respectively). Automatically identified intro and outro segments can also be identified based on musical properties of the segment itself.

In some cases, the automatic composing engine310can select a subset of audio segments from a larger set of audio segments for use in a song328. For example, an automatic composing engine310may have access to a set of 80 audio segments (e.g., from multiple collections322of audio segments created using an automatic musical segmenting engine308on a plurality of audio files). The automatic composing engine310may select which out of the set of 80 audio segments to use in the final song328. This selection process can be based on any combination of manual settings (e.g., a user desiring a two minute song) and musical properties (e.g., selecting all segments that match a particular key signature).

In some cases, the automatic composing engine310can allow a user to manipulate the order of the segments. The automatic composing engine310can store historical information related to the past manual placement of audio segments in relation to other audio segments and in relation to an overall song328. The automatic composing engine310can learn from this historical information and use the historical information to improve its audio segment ordering and selection processes. In some cases, the historical information can be used to adjust the weighting of certain musical properties and can recognize patterns in audio segment placement.

AlthoughFIG. 3illustrates automatic musical segmenting engine308as a separate engine from automatic composing engine310, embodiments are not so limited. For instance, automatic segmenting engine308may be a part of automatic composing engine310. Accordingly, automatic segmenting engine308may be a subfunction of automatic composing engine310, as will be discussed in more detail herein.

The metadata usage environment300can include a song metrics analyzing engine312. The song metrics analyzing engine312can analyze any attributes of the metadata304associated with audio data302. The song metrics analyzing engine312can be used to determine patterns, relationships, averages, or other metrics associated with musical properties of the audio data302. For example, the song metrics analyzing engine312can determine the most common chord used in a piece, the number of times each note was used in a piece, the average tempo or tempo changes throughout a piece, and other metrics. The song metrics analyzing engine312can provide metrics data338to other engines or devices for further use. Metrics data338from multiple songs can be compared and further analyzed, such as to determine correlations between multiple songs.

In an example, a song metrics analyzing engine312can be used on a set of songs to generate metrics data338regarding the key signatures, chords, notes, tempos, and other musical properties of each song in the set. Comparison of the metrics data338can be used to order the songs (e.g., for a playlist or an album) in a meaningful way. For example, metrics data338can be used to order similar songs adjacent one another. In another example, metrics data338can be used to order songs so that similar songs (e.g., with similar chord or note distributions, similar tempos, similar keys, or other similar characteristics) are not directly adjacent one another (e.g., to improve variety in a playlist or album).

The ability to obtain audio data302and associated metadata304, as well as to use the audio data302, metadata304, or both brings substantial benefit to music enthusiasts, including performers, technicians, and listeners alike. For example, the use of an audio processor108having an automatic start/stop engine110as described inFIG. 1can simplify the recording process for a musician. As another example, the ability to analyze incoming audio data to generate metadata (e.g., metadata126generated by the audio analyzing engine114ofFIG. 1) can enable many different uses of the recordings or live performances (e.g., as seen inFIG. 3). Furthermore, the aspects described herein will enable musicians to record, analyze, and manipulate their music in new and unique ways.

It can be appreciated that may functions can be performed from utilizing metadata of audio files. These functions may be complex functions that require several processing steps, as will be discussed herein for an automatic composer.

FIG. 4is a simplified block diagram400for an automatic composer402, according to embodiments of the present invention. Automatic composer402may be program code stored on a memory device (e.g., another server) configured to be executed by a processor to perform a function, such as generating a musical composition as will be discussed herein. Alternatively, automatic composer402may be a combination of hardware and software specially configured to perform the function. For example, automatic composer402may be a data processing system containing software configured to perform the function.

In embodiments, automatic composer402may receive an input and generate a meaningful output. For example, music performance data404may be received by automatic composer402. In embodiments, music performance data404may include audio data302and associated metadata304as discussed inFIG. 3. For instance, music performance data404may be a single digital recording or a collection of digital recordings and their corresponding data related to melody, harmony, and rhythm. Automatic composer402may use music performance data404(which includes corresponding music analysis data as will be discussed further herein) to generate a musical composition406. In embodiments, automatic composer402may generate musical composition406by initially segmenting music performance data404into one or more musical segments. The musical segments may then be arranged into a cohesive piece of musical work, thereby resulting in the generation of musical composition406.

In some embodiments, automatic composer402may generate musical composition406from music performance data404based upon sets of rules. For instance, automatic composer402may generate musical composition406based upon two sets of rules: segment creation rules408and affinity rules410. Segment creation rules408may be a list of structural attributes of musical pieces that are desired to be present in each musical segment. For instance, segment creation rules408may be a list that includes a number of beats and bars regardless of tempo, chord sequences, rhythmic structure, and the like. Affinity rules410may be a list of musical attributes of musical pieces that are desired to be shared amongst each musical segment in musical composition406. As an example, affinity rules410may be a list that includes chord progression, beats, rhythm, and the like. The details and purposes of segment creation rules408and affinity rules410will be discussed further herein.

In embodiments, automatic composer402may include functional engines that are each configured to perform a different function for generating musical composition406. For instance, automatic composer402may include a segment creator engine412, affinity calculating engine414, and composer engine416. Each engine412,414, and416may be lines of program code stored on a memory device configured to be executed by a processor. In some embodiments, engines412,414, and416may include hardware and firmware. The interaction between the three engines may enable automatic composer402to generate musical composition406from music performance data404based upon segment creation rules408and affinity rules410. Details of these engines are discussed further herein.

III. Segment Creator Engine

As mentioned herein, automatic composer402may segment music performance data404into a plurality of musical segments. To perform this function, automatic composer402may include segment creator engine412as shown inFIG. 5.

FIG. 5is a block diagram illustrating the operation of a segment creator engine, such as segment creator engine412, according to embodiments of the present invention. Segment creator engine412may be a subfunction of automatic composer402that is configured to perform a small part of a greater function. For instance, segment creator engine412may be configured to segment music performance data into one or more musical segments such that automatic composer402may use the musical segments to generate a musical composition.

In some embodiments, segment creator engine412receives music performance data404. Music performance data404may be generated by a preprocessing engine (not shown). The preprocessing engine may be any suitable body of computer code that can analyze audio files to extract data, such as data pertaining to melody, harmony, and rhythm from an audio file. As an example, the preprocessing engine may be audio processor108discussed inFIG. 1. The analysis of each audio file may be appended to the audio file as metadata, which may be utilized by subsequent processing. In embodiments, music performance data404may include one or more audio files and analyses data pertaining to melody, harmony, and rhythm. For instance, music performance data404may include one or more audio files, i.e., audio data302, and associated analysis data, i.e., metadata304, discussed inFIG. 3. It is to be appreciated that any number of audio files and analysis data may be included as music performance data404. For instance, a single audio file and analysis data may be included as music performance data404. Alternatively, a number N of audio files and analysis data ranging from1to N may be included as music performance data404. That is, music performance data404may include 1staudio file and analysis data502-1through Nthaudio file and analysis data502-N.

Segment creator engine412may receive music performance data404and subsequently segment music performance data404into one or more musical segments. As shown inFIG. 5, segment creator engine412may segment musical performance data404into a plurality of musical segments506. For example, segment creator engine412may segment musical performance data404into a number M of musical segments506, i.e., 1stmusical segment506-1to Mthmusical segment506-M. Musical segments506may be stored in a musical segments library504, which may be an allocation of memory in a memory bank configured to store musical segments506-1through506-M. Alternatively, musical segments library504may consist of a list of addresses linking to specific locations in memory where data for musical segments506-1through506-M are located.

In certain embodiments, music performance data404may be segmented based upon segment creation rules408. Segment creation rules408may determine how audio files and analysis data502in music performance data404will be segmented by segment creator engine412. Segment creation rules408may be a list of structural attributes of musical pieces that are desired to be present in each musical segment. Structural attributes may be related to an underlying musical framework of a musical piece. The musical framework of a musical piece may include properties of a musical segment that are unrelated to how a musical segment sounds, such as, but not limited to, number of beats and bars regardless of tempo, chord sequences, rhythmic structure, spectral similarity over time, baseline similarity, role of the musical segment in the original music performance data (e.g., whether the musical segment is an intro, chorus, bridge, and the like), presence of vocal content, specific instruments detection (e.g., whether the musical segment is a guitar or a piano piece), and the like. As an example, if segment creation rules408contain a structural attribute specifying four musical bars, segment creator engine412may segment each audio file502into a plurality of musical segments506where each musical segment506-1through506-M contains only four musical bars.

It is to be appreciated that musical segments library504may include musical segments506-1through506-M that have been stored at different periods of time. For instance, 1stmusical segment506-1may have been stored several days or months prior to the time at which 2nd musical segment506-2was stored. Furthermore, musical segments506-1through506-M may be segments of different audio files502-1through502-N. As an example, 1stmusical segment506-1may be a segment of 1staudio file502-1and 2ndmusical segment506-2may be a segment of 2ndaudio file (not shown). On the other hand, musical segments506-1through506-M may be segments of the same audio file. For instance, 3rdmusical segment (not shown) and 4thmusical segment (not shown) may be segments of 2ndaudio file (not shown).

Additionally, it is to be appreciated that each musical segment506-1through506-M may still contain analysis data, e.g., metadata, from the preprocessing engine (not shown). Thus, although musical segments506are each a portion of audio files502, each musical segment506-1through506-M may include data pertaining to its melody, harmony, and rhythm. This analysis information may be utilized to determine a degree of similarity between musical segments, as will be discussed further herein.

A. Musical Segment

Each musical segment created by segment creator engine412may include distinct parts. In certain embodiments, each musical segment may include a prologue, an epilogue, and/or a verse.

A prologue may be a portion of an audio file that is devoid of musical data. For instance, a prologue may not have melody, harmony, or rhythm. Additionally, a prologue may be a portion of an audio file that immediately precedes a portion of an audio file that has melody, harmony, or rhythm. As an example, a prologue may be a portion of an audio file where a musician takes a breath before playing an instrument. Thus, the prologue may represent a beginning of a musical piece.

Similar to a prologue, an epilogue may also be a portion of an audio file that is devoid of musical data. However, in contrast to a prologue, an epilogue may be a portion of an audio file that immediately follows a portion of an audio file that has melody, harmony, or rhythm. For instance, an epilogue may be a portion of an audio that includes audio of a person talking or audio containing non-harmonic background noise. It may represent an ending of a musical piece.

In contrast to both a prologue and an epilogue, a verse is a portion of an audio file that has musical data. As an example, a verse may be a portion of an audio file that has melody, harmony, and/or rhythm. In embodiments, a verse may be a riff, a chorus, a solo piece, and the like.

Each musical segment may contain one or a combination of a prologue, an epilogue, and a verse.FIGS. 6A-6Dillustrate different types of musical segments that can be created by segment creator engine412. As shown inFIG. 6A, an exemplary musical segment600may include all three parts: a prologue602, an epilogue604, and a verse606. Prologue602immediately precedes verse606, and epilogue604immediately follows verse606.

It is to be appreciated that musical segments do not have to include all three parts.FIG. 6Billustrates an exemplary musical segment608that includes prologue602and verse606but not epilogue604.FIG. 6Cillustrates an exemplary musical segment610having epilogue604and verse606but no prologue602.FIG. 6Dillustrates an exemplary musical segment612having only verse606and no prologue602or epilogue604. AlthoughFIGS. 6A-6Ddo not illustrate a musical segment having only a prologue and/or an epilogue, one skilled in the art understands that musical segments may also be created to have a prologue and/or an epilogue without a verse.

In embodiments, a musical segment may also include transitions614and616at the beginning and/or end of a verse.FIG. 6Dillustrates verse606having transitions614and616at both a beginning and an end of verse606. Transitions614and616may be modifications of verse606to enhance seamless transition between musical segments. For example, transition614may gradually increase an audio level of verse606to provide a gradual beginning of verse606. Transition616may gradually decrease an audio level of verse606to provide a gradual ending of verse606

B. Exemplary Segmentation of an Audio File

To better describe segmentation of an audio file,FIGS. 7A and 7Billustrate an exemplary segmentation of an audio file into a plurality of musical segments, according to embodiments of the present invention. InFIG. 7A, an audio file700is shown as having a prologue702, an epilogue704, and a body706between prologue702and epilogue704. Audio file700may be a musical piece where prologue702is an introductory portion that is devoid of musical data (i.e., having no melody, harmony, and rhythm). Following prologue702is body706, which may include musical data such as melody, harmony, and rhythm. In some embodiments, body706may include various rifts, choruses, and the like. Following body706may be epilogue704, which is an ending portion that may be devoid of musical data.

In embodiments, audio file700may be segmented into a plurality of musical segments as shown inFIG. 7B. For instance, audio file700may be segmented into musical segments720,722,724, and726. Each musical segment may be a part of audio file700. As an example, musical segment720may include prologue702and a verse710. Verse710may be a portion of body706that includes musical data such as melody, harmony, and rhythm. Other musical segments, such as segments722,724, and726may contain other parts of audio file700. For example, musical segment722may only include a verse712, and musical segment724may only include a verse714. Verses712and714may be parts of body706that contain musical data. In embodiments, musical segments720,722,724, and726contain similar structural attributes as determined by segment creation rules708discussed herein with respect toFIG. 5. For instance, musical segment720,722,724, and726may each have four bars, four chords, and the like.

Segmenting audio file700into musical segments720,722,724, and726allows automatic composer402to manipulate the order of musical segments720,722,724, and726to generate a musical composition that is different than audio file700. However, in order for automatic composer402to perform such manipulation, automatic composer402may determine which musical segments are compatible with one another.

IV. Affinity Calculating Engine

Determining compatibility may be performed by calculating an affinity value. The affinity value may be a numerical value that represents a degree of similarity between two musical segments. In embodiments, the affinity value may be associated with one or more musical attributes shared by the two musical segments. According to embodiments of the present invention, this affinity value may be calculated by an affinity calculating engine, such as affinity calculating engine414shown inFIG. 8.

Calculating an affinity value may allow automatic composer402to utilize the affinity value to identify musical segments that are similar in musical sound. The identified musical segments may be combined to form a musical composition. Combining musical segments having a degree of similarity provides for a smooth transition between them, thereby resulting in a musical composition that is musically coherent.

FIG. 8is a block diagram illustrating the operation of affinity calculating engine414, according to embodiments of the present invention. Affinity calculating engine414may be a subset of automatic composer402that is configured to perform a small part of a greater function. For instance, affinity calculating engine414may be configured to calculate an affinity value for pairings of musical segments such that automatic composer402may utilize the affinity value to generate a musical composition, e.g., musical composition406inFIG. 4.

In certain embodiments, affinity calculating engine414receives a plurality of musical segments from a musical segments library. For instance, affinity calculating engine414may receive musical segments506-1through506-M in musical segments library204that were created by segment creation engine412.

Once musical segments506are received by affinity calculating engine414, affinity calculating engine414may perform calculations and output affinity values802. In embodiments, each affinity value802may represent a degree of similarity between two musical segments. In certain embodiments, affinity calculating engine414may determine an affinity value802for each possible pairing of musical segments. In other embodiments, affinity calculating engine414may determine more than one affinity value802for each possible pairing of musical segments. Such affinity values802may then be linked or appended to corresponding musical segments to form a new segments library804. Accordingly, new segments library804may include a plurality of musical segments and affinity values806, where each musical segment and affinity values802includes data pertaining to a musical segment and its corresponding affinity values. In embodiments, new segments library804may be an updated version of musical segments library504that replaces musical segments library504.

According to embodiments, affinity values802may be calculated based upon a set of affinity rules410. Affinity rules410may include a selection of one or more musical attributes. Musical attributes may include properties of a musical segment that relate to how the musical segment sounds. For instance, musical attributes may include characteristics such as, but not limited to, chord progression, spectral content, beats, rhythm, and harmonic scale. There may be several different types of spectral content. As an example, spectral content may be defined by a spectral distribution of audio data (FFT) localized at the beginning and at the end of verses, at the ending of prologues, or at the beginning of epilogues. Spectral content may also be defined by peaks at each frequency of the overall spectral distribution of a verse. Furthermore, spectral content may be defined by the shape and characteristics (e.g., the width, phase, characteristics, modulation, harmonics distribution) of relevant spectral peaks. It is to be appreciated that musical attributes are different than structural attributes in that musical attributes relate to the arrangement of tones, melodies, chords, harmonies, and the like of a musical piece, while structural attributes are more related to the underlying musical framework of a musical piece. Affinity rules410may determine what musical attributes will be shared between musical segments in a musical composition, as will be discussed further herein with respect toFIGS. 10A-10B and 12.

In embodiments, affinity rules410determine how affinity values802are to be calculated. For example, if affinity rules410are selected to include musical attributes such as chord progression and harmonic scale, then affinity values802may be a calculated numerical value representing a degree of similarity between two musical segments based upon chord progression and harmonic scale. Affinity values802may be a single number that represents a degree of similarity between two musical segments based upon any combination and number of musical attributes. One skilled in the art understands that embodiments are not limited to just two musical attributes.

To provide flexibility and user friendliness, affinity rules410may be selected by a user. For instance, a user who desires to arrange segments506-1through506-M according to chord progression and harmonic scale, may select chord progression and harmonic musical attributes to be affinity rules410. If the user would like to change the established affinity rules410, the user may deselect certain musical attributes and select new musical attributes. In addition to having a user select musical attributes of affinity rules410, a default set of musical attributes may be encoded within affinity calculating engine414such that a user does not have to select the musical attributes. The selected musical attributes for the default configuration may be determined by a programmer according to a design goal.

Determining an affinity value between two segments may include calculating an affinity subvalue between two musical segments. The affinity subvalue may be a number that represents a degree of similarity between a shared musical attribute between two musical segments. An affinity subvalue may be distinguished from an affinity value because an affinity subvalue pertains to only one musical attribute shared between two musical segments, while an affinity value pertains to one or more musical attributes shared between two musical segments. Thus, an affinity subvalue may be a more basic determination of a degree of similarity between musical segments, while an affinity value may be a more complex determine of a degree of similarity between musical segments.

Determining an affinity value may further include combining affinity subvalues. The combined affinity subvalues may correspond to the selected musical attributes established by the set of affinity rules. As an example, if the set of affinity rules includes chord progression and harmonic scale, then the affinity subvalues associated with chord progression and harmonic scale may be added together to determine the affinity value. Details of how an affinity value is calculated may be shown inFIG. 9.

FIG. 9is a simplified block diagram illustrating an exemplary affinity calculating engine, such as affinity calculating engine414, according to embodiments of the present invention. Affinity calculating engine414may include a plurality of affinity functions902. For instance, affinity calculating engine414may include a number Y of affinity functions902ranging from902-1to902-Y. Each affinity function902-1through902-Y may be a section of program code that is specifically configured to calculate an affinity subvalue904based upon a specific musical attribute. In embodiments, an affinity subvalue904is determined for every musical attribute, regardless of what is selected in affinity rules410. As an example, 1staffinity function902-1may be configured to calculate a degree of similarity based upon chord progression. 2ndaffinity function902-2may be configured to calculate a degree of similarity based upon harmonic scale. It is to be appreciated that any other affinity function may be configured to determine an affinity subvalue for any other musical attribute.

Once affinity subvalues904are calculated for every musical attribute, certain affinity subvalues904that are associated with the selected musical attributes in affinity rules410may be factored together by function906. Function906may receive data from affinity rules410pertaining to which musical attributes are selected. Only those musical attribute selected by affinity rules410may be multiplied together to determine an affinity value908. Accordingly, affinity value908may be a degree of similarity between 1stand 2ndmusical segments506-1and506-2based upon the musical attributes selected in affinity rules410. For instance, affinity value908may be a degree of similarity between 1stand 2ndmusical segments506-1and506-2with regards to chord progression and harmonic scale.

In embodiments, affinity value908may be a normalized value. For example, function906may not only multiply/combine affinity subvalues together, but function906may also normalize the resulting calculation such that the normalized affinity value908of a musical segment ranges between 0-1. Any other standardization format may be used to calculate affinity value908. It is to be appreciated that the following discussion calculates affinity value908by merely multiplying together corresponding affinity subvalues for ease of discussion, but is not limited to such calculation methods.

In embodiments, the calculated affinity value908may then be linked or appended to corresponding 1stand 2ndmusical segments506-1and506-2to form 1stand 2ndmusical segment and affinity values806-1and806-2in new segments library804, as discussed herein with respect toFIG. 8. Accordingly, 1stmusical segment and affinity values806-1may include affinity value908, which may represent its similarity to 2ndmusical segment and affinity values806-2. Likewise, 2ndmusical segment and affinity values806-2may include affinity value908, which may represent its similarity to 1stmusical segment and affinity values806-1.

Although the discussion herein relates to only 1stand 2ndmusical segments, one skilled in the art understands that similar operations apply to any two musical segments without departing from the spirit and scope of the present invention.

B. Exemplary Calculation of Affinity

FIGS. 10A and 10Bare block diagrams for illustrating how the affinity values are calculated, according to embodiments of the present invention. Specifically,FIG. 10Ais a block diagram illustrating an exemplary calculation of affinity subvalues for a 1stmusic segment1002.FIG. 10Bis a block diagram illustrating an exemplary calculation of affinity values for the 1stmusic segment1002. One skilled in the art understands that even thoughFIGS. 10A and 10Bshow calculations for only 1stmusical segment1002, the same discussion applies to any other musical segment.

Affinity subvalues for each of the three music segments are calculated for four different musical attributes: 1stmusical attribute1008-1, 2ndmusical attribute1008-2, 3rdmusical attribute1008-3, and 4thmusical attribute1008-4.

Affinity calculating engine414may determine an affinity subvalue1010for each possible segment pairing and for each musical attribute1008. For instance, affinity subvalues1010may be determined for every possible pairing between 1stmusical segment1002and all other musical segments, e.g., 2ndand 3rdmusical segments1004and1006. This may be repeated for each musical attribute1008. Accordingly, 1stmusical segment1002may have eight affinity subvalues1010A-1010H associated with 1stmusical segment1002. In embodiments, the eight affinity subvalues1010A-1010H may be linked or appended to 1stmusical segment1002to form 1stmusical segment and affinity values806-1and stored in new segments library804as discussed inFIG. 8.

AlthoughFIG. 10Aillustrates calculating affinity subvalues1010for only 1stmusical segment1002, it is to be appreciated that this calculation may be performed for all other musical segments, such as 2ndmusical segment1004and 3rdmusical segment1006. Corresponding affinity subvalues may also be linked or appended to 2ndand 3rdmusical segments1004and1006in musical segments library504inFIG. 8, and then stored in new segments library804.

As shown inFIG. 10B, exemplary affinity values, such as affinity value608inFIG. 9, are calculated according to certain affinity rules, such as affinity rules1012,1014, and1016. Affinity rules1012,1014, and1016may each have different selected musical attributes. For instance, affinity rule1012may have 1stmusical attribute1008-1selected, affinity rule1014may have 1stand 2ndmusical attributes1008-1and1008-2selected, and affinity rule1016may have 1st, 2nd, and 3rdmusical attributes1008-1,1008-2, and1008-3selected. As aforementioned herein, the musical attributes may be selected by a user or be programmed to be selected by default.

According to affinity rule1012, only one musical attribute is selected: 1stmusical attribute1008-1. Thus, the affinity value for affinity rule1012is calculated to be the corresponding affinity subvalue since there is no other affinity subvalue with which to add. Accordingly, the affinity value for 1stand 2ndmusical segments1002and1004is 0.8, as shown by affinity subvalue1010A inFIG. 10A. The affinity value for 1stand 3rdmusical segments1002and1006is 0.2.

Based upon affinity rule1014, two musical attributes are selected: 1stand 2ndmusical attributes1008-1and1008-2. Thus, the affinity value for affinity rule1014is calculated as the multiplication of corresponding affinity subvalues for each attribute for the segment pair. For instance, the affinity value for 1stand 2ndmusical segments1002and1004is 0.08, which is the multiplication of affinity subvalue1010A (0.8) and1010C (0.1). The affinity value for 1stand 3rdmusical segments1002and1006is 0.18, which is the multiplication of affinity subvalues1010B (0.2) and1010D (0.9).

Furthermore, according to affinity rule1016, three musical attributes are selected: 1st,2nd, and 3rdmusical attributes1008-1,1008-2, and1008-3. As a result, the affinity value for 1stand 2ndmusical segments1002and1004is 0.056, which is the multiplication of affinity subvalues1013A (0.8),1010C (0.1), and1010E (0.7). The affinity value for 1stand 3rdmusical segments1002and1006is 0.072, which is the multiplication of affinity subvalues1010B (0.2),1010D (0.9), and1010F (0.4).

Each of affinity rules1012,1014, and1016are examples of how different affinity rules410may result in different affinity subvalues1010. Depending on what the user selects, or what is selected by default, affinity subvalues1010may vary. Accordingly, a user may change the set of affinity rules to achieve different musical compositions. According to embodiments, musical compositions may be generated by a composer engine, as will be discussed further herein. It is to be appreciated that the scale shown inFIGS. 10A and 10Bare merely exemplary, and that other embodiments are not limited to such scoring schemes.

V. Composer Engine

The musical segments and affinity subvalues may be received by a composer engine. The composer engine may be lines of program code stored on a memory device configured to be executed by a processor to perform a specific function. In embodiments, the composer engine may be configured to generate a musical composition. The musical composition may be generated by arranging a plurality of musical segments together into a musical piece. The musical segments may be arranged according to affinity values determined by a set of affinity rules, such as affinity rules410.

FIG. 11is a simplified block diagram illustrating an exemplary composer engine, such as composer engine416, according to embodiments of the present invention. Composer engine416may receive musical segments and affinity values806from new segments library804and subsequently arrange them into a musical composition406. In embodiments, musical composition406includes a plurality of rearranged musical segments1102. Each rearranged musical segment1102may be a musical segment806from new segments library804arranged differently than when musical segment806was arranged as a portion of its music performance data.

According to embodiments, musical composition406may be generated by rearranging musical segments1106from new segments library1104based upon affinity values, e.g., affinity values1102inFIG. 11, calculated according to a set of affinity rules, e.g., affinity rules410inFIG. 11. Composer engine416may analyze the affinity values for each musical segment, i.e., musical segment and affinity values1106. Composer engine416may then pair together musical segments having a predetermined affinity value. For instance, composer engine416may combine musical segments1106having an affinity value greater than a certain threshold affinity value, or composer engine416may combine musical segments1106having a highest affinity value. Combining those musical segments having predetermined affinity values results in a music composition whose rearranged musical segments1102may be similar to one another in musical sound such that the resulting composition is a cohesive musical piece.

As mentioned herein, an affinity value may be a number that reflects a similarity of two musical segments based upon selected affinity rules. Thus, depending on the selection of affinity rules, musical composition406may be arranged such that its rearranged musical segments1102-1through1102-X have a strong similarity between those musical attributes selected in the affinity rules. In other words, the selected affinity rules may dictate how the musical compositions will sound. For example, if the set of affinity rules select chord progression and harmonic scale as the selected musical attributes, then musical segments1102arranged in musical composition406will be have similar chord progression and harmonic scale.

To ensure that the rearrangement musical segments1102are similar to one another in musical sound, composer engine416may generate musical composition406may pairing musical segments having a highest affinity value, as discussed inFIG. 12herein.

Composer engine416may generate musical compositions by rearranging a plurality of musical segments. Rearranging musical segments may be performed by generating a series of pairs of musical segments. To determine which two musical segments pair well together, composer engine416may analyze affinity values for each possible pair and pair together those musical segments having the highest affinity value.

FIG. 12is a block diagram illustrating an example pairing of musical segments by composer engine416. The example illustrated inFIG. 12may be a continuation of the example discussed inFIG. 10B. In this example, there may be only two possible pairs for 1stmusical segment1002: a pairing with 2ndmusical segment1004or a pairing with 3rdmusical segment1006. This may be because there are only three musical segments in this example. Thus, 1stmusical segment1002can only be paired with either 2ndmusical segment1004or 3rdmusical segment1006. It is to be appreciated that embodiments having more musical segments may result in a greater number of possible pairs.

As shown inFIG. 12, several different possible pairings are illustrated according to sets of affinity rules. Based upon affinity rule1012, the affinity value between 1stmusical segment1002and 2ndmusical segment1004is 0.8, as discussed herein with respect toFIG. 10B, and the affinity value between 1stmusical segment1002and 3rdmusical segment1006is 0.2. Because 1stmusical segment1002has a higher affinity value with 2ndmusical segment1004, composer engine416may pair 1stmusical segment1002with 2ndmusical segment1004. Indication of this selection may be illustrated by its solid lines, as opposed to the dotted lines for the pairing of 1stmusical segment1002with 3rd musical segment1006.

FIGS. 6-12illustrate composer engine416as determining a pairing of musical compositions based upon a multiplication of affinity subvalues. One skilled in the art understands that this is merely one embodiment, and that other embodiments are not limited to such calculations. As already discussed herein, the affinity value may be a normalized value. Additionally, in other embodiments, the affinity value may be an average of affinity subvalues, a mean of affinity subvalues, or any other way of using mathematics to distinguish one value from a plurality of values.

B. Exemplary Musical Composition

According to embodiments, the series of matched pairs may then be arranged into a musical composition. The musical composition may be formed by utilizing the same techniques as discussed herein with regard to pairing musical segments. That is, one musical segment of a pair of musical segments may pair with another musical segment of another pair of musical segments. Thus, a musical composition may be seen as a partially overlapping arrangement of pairs of musical segments, as will be shown herein with respect toFIGS. 13A and 13B.

FIG. 13Aillustrates an exemplary musical composition1300as generated by automatic composer402. Exemplary musical composition1300may be one embodiment of musical composition406discussed inFIG. 4. As shown, musical segments1316may include different arrangements of prologues, epilogues, and verses as mentioned herein. For instance, musical segment1316includes a prologue1302and a verse1306, musical segment1324includes a verse1314and an epilogue1304, and musical segments1320,1318, and1322include only verses1308,1310, and1312, respectively.

Musical segment1316is paired with musical segment1318. Composer engine416may have paired them together based upon an affinity value calculated based upon a set of affinity rules, as discussed herein. To form an entire musical piece, composer engine416may build upon that pair by forming another pair between musical segment1318and1320. Accordingly, musical segment1318may be shared between two separate pairs of musical segments to form a portion of musical composition1300. Thus, there may be a partially overlapping arrangement between pairs of musical segments throughout musical composition1300where each musical segment has a high affinity value with adjacent musical segments. Arranging the musical segments to have a high affinity value with adjacent musical segments may result in similar sounds across musical segments throughout musical composition1300, thereby appearing as a single well composed and cohesive musical piece.

It is to be appreciated that musical segments1316,1318,1320,1322, and1324may each be different from one another, or some may be the same. For example, musical segment1318may be different than every other musical segment such that each musical segment has its own distinctive arrangement of musical notes. However, in other examples, musical segment1318may be repeated. That is, musical segment1322may be a copy of musical segment1320such that verse1312is the same as verse1310. The same applies to a series of musical segments where two or more sequential musical segments are repeated. This repeating may be referred to as “looping”.

FIG. 13Aillustrates musical composition1300against a musical bar backdrop1326to show how the musical framework of each musical segment may be substantially similar. This similarity may be established by a set of segment creation rules, such as segment creation rules408inFIGS. 1 and 2, that determines how music performance data is to be segmented by a segment creator engine, e.g., segment creator engine412. In embodiments, musical segments1316,1318,1320,1322, and1324are shown vertically offset from one another to make it easier to perceive the distinctive musical segments. The musical segments, however, can be arranged in other ways. For instance, the musical segments can be arranged to be directly adjacent to one another as shown inFIG. 13B.

FIG. 13Billustrates musical composition1300in a linear format where each musical segment is arranged directly adjacent to one another. In embodiments, transitions1328may be positioned between musical segments such that each transition1328is between each verse. Transitions1328may minimize any audible disjointedness between musical segments created by joining two musical segments with one another that were not originally created as such. In certain embodiments, transitions1328may be a cross-fade. As a cross-fade, transitions1328may fade out of one verse while simultaneously turning up another verse at the interface of both verses. For instance, verse1306may fade out while verse1308turns up at the first transition1328. In embodiments, transitions1328is an overlapping/combination of transitions614and616discussed inFIG. 6D.

C. Sources for Generating a Musical Composition

According to embodiments, an automatic composer can generate a musical composition from music performance data, and analysis data. The automatic composer generates the musical composition by segmenting the music performance data into musical segments and stores them in a segment library. The automatic composer then takes musical segments from the segment library and combines them into the musical composition. The musical composition may be a musical piece that is arranged differently than the music performance data.

FIG. 14Aillustrates an exemplary music performance data1400and an exemplary musical composition1401generated by an automatic composer, such as automatic composer402, according to embodiments of the present invention. In this example, music performance data1400includes one musical piece having a prologue, an epilogue, and a plurality of verses 1-5 in sequential order. The prologue, epilogue, and verses 1-5 may be parts of musical segments as discussed herein with respect toFIG. 4B. Thus, one skilled in the art understands that althoughFIG. 14Ashows a prologue, an epilogue, and verses 1-5, the illustration applies to musical segments as well.

Music performance data1400may be segmented and rearranged by the automatic composer to generate musical composition1401. In embodiments, musical composition1401may include verses 1-5 but rearranged to be in a different order than how they were arranged as music performance data1400. Additionally, verses, such as verse1and verse3, may be repeated in other parts of musical composition1401. As a result, musical composition1401may be a musical piece that has an arrangement of verses 1-5 in a particular order that may be entirely new and unique.

In embodiments where music performance data includes more than one musical piece, the resulting musical composition may include segments from more than one musical piece. For instance, music performance data1402may include two musical pieces: first musical piece1402A and second musical piece1402B, each having a prologue, an epilogue, and a plurality of verses 1-5 in sequential order. Second musical piece1402B is shaded to indicate which prologue, epilogue, and verse belongs to second musical piece1402B. Music performance data1402may be segmented and rearranged by the automatic composer to generate musical composition1403. In embodiments, musical composition1403may include verses 1-5 from both music performance data1402A and1402B but rearranged to be in a different order than how they were originally arranged before being recomposed by the automatic composer. As a result, musical composition1401may be a musical piece that has an arrangement of one or more verses 1-5 from both music performance data1402A and1402B in a particular order that may be entirely new and unique.

AlthoughFIG. 14Billustrates music performance data1402has including two separate music performance data1402A and1402B as sources for generating a musical composition1403, embodiments are not limited to such sources. For example, a segments library may contain musical segments created from other music performance data that have been segmented at a different period of time. These segments may be used by the automatic composer to generate a musical composition, according to embodiments of the present invention.

VI. Method of Automatically Composing a Song

FIG. 15is a flow chart illustrating a method for generating a musical composition from music performance data, according to embodiments of the present invention. At block1502, music performance data and analysis data may be received by a processor. The processor may contain code for an automatic composer, such as automatic composer402discussed herein. In embodiments, music performance data may be received by a segment creator engine of the automatic composer engine. As an example, segment creator engine415may receive music performance data404as discussed herein with respect toFIG. 4. In embodiments, music performance data404includes analysis data pertaining to melody, harmony, and rhythm of music performance data404.

At block1504, the music performance data may be segmented based on at least one structural attribute into at least a first musical segment. For instance, the music performance data may be segmented by the segment creator engine, such as segment creator engine415discussed herein. The structural attribute may be a property of the music performance data relating to the underlying musical framework of a musical piece, such as number of bars, chord sequences, rhythmic structure, spectral similarity over time, baseline similarity, and the like.

In embodiments, the first musical segment may be associated with at least one musical attribute. A musical attribute may include properties of a musical segment that relate to how the musical segment sounds. For instance, musical attributes may be characteristics such as, but not limited to, chord progression, spectral content, beats, rhythm, and harmonic scale. Musical attributes may differ from structural attributes in that musical attributes may relate to the arrangement of tones, melodies, chords, harmonies, and the like of a musical piece, while structural attributes may relate to the underlying musical framework of a musical piece.

In embodiments, the first musical segment may have at least one of a corresponding prologue, epilogue, and a verse. A prologue may be a portion of an audio file that is devoid of musical data. Additionally, a prologue may be a portion of an audio file that immediately precedes a portion of an audio file that has melody, harmony, or rhythm. An epilogue may also be a portion of an audio file that is devoid of musical data. However, in contrast to a prologue, an epilogue may be a portion of an audio file that immediately follows a portion of an audio file that has melody, harmony, or rhythm. In contrast to both a prologue and an epilogue, a verse is a portion of an audio file that has musical data. A verse may be a rift, a chorus, a solo piece, and the like.

At block1506, an affinity value for the first musical segment may be determined based on the at least one musical attribute. The affinity value may represent a degree of similarity between the first musical segment and a second musical segment having the at least one musical attribute. In embodiments, the affinity value is calculated by an affinity calculating engine, such as affinity calculating engine414inFIG. 4. The affinity calculating engine may receive the musical segments and calculate affinity values for each possible musical segment pairing. The affinity calculating engine may include a plurality of affinity functions where each affinity function is configured to calculate an affinity subvalue for a particular musical attribute. The affinity subvalue may be number reflecting a degree of similarity between the particular musical attribute of two musical segments.

An affinity value may be calculated by referencing the affinity subvalues for musical attributes selected in a set of affinity rules. The set of affinity rules may contain a selection of musical attributes that is desired to be shared between the first and second musical segments in a resulting musical composition. In embodiments, the affinity value may be calculated by adding together the affinity subvalues for the musical attributes selected in the set of affinity rules.

At block1508, a musical composition may be generated based upon the affinity values associated with the first musical segment and the second musical segment. In embodiments, a composer engine, such as composer engine416, generates the musical composition. The composer engine may pair segments with one another having a highest affinity value. Combining those musical segments having predetermined affinity values results in a music composition whose rearranged musical segments may be similar to one another in musical sound such that the resulting musical composition is a cohesive musical piece

In embodiments, the musical composition may be presented to a user. For example, the musical composition may be outputted to a user interface from which the user may see and hear the musical composition. Additionally, the user interface may allow the user to interact with the automatic composer to generate inputs for establishing segment creation rules408and selecting affinity rules, as discussed herein. Examples of such a user interface is shown herein with respect toFIGS. 16A-16C.

VII. User Interface

FIG. 16Ais a screenshot of an exemplary user interface1600for an automatic composer, i.e., automatic composer402, according to embodiments of the present invention. User interface1600may be a program window displayed on a display screen of a computing device, such as a computer, tablet, laptop, smartphone, and the like. The automatic composer may be a program executed by a processor. The processor may be coupled to the display screen such that the processor may present user interface1600to the user.

User interface1600may provide information to a user via visual output showing music performance data as well as outputted musical compositions. For example, user interface1600shows a music performance data1602. Music performance data1602may be an audio file for a musical piece. The musical piece may be a live recording of a musical performance, or a stored audio file of a musical piece. Music performance data1602may be presented to the user such that the user may reference music performance data1602when comparing it to music compositions generated by the automatic composer.

As shown inFIG. 16A, two musical compositions are shown: a first musical composition1604, and a second musical composition1606. In embodiments, first and second musical compositions1604and1606may be generated by the automatic composer and subsequently presented to the user via user interface1600. In the example shown inFIG. 16A, first musical composition1604may be a first order of musical composition that occurred before second musical composition1604. First musical composition1604may be a segmented version of music performance data1602in its original order. First musical composition1604may be in its original order to illustrate how music performance data1602is segmented.

Second musical composition1606may be a second order of musical composition that occurred after the generation of first musical composition1604. Second musical composition1606may be a rearranged version of music performance data1602including a plurality of musical segments1607. Each musical segment1607of music performance data1606may be a portion of music performance data1602that is arranged in a different location than when it originally was presented as music performance data1602. Each rearranged musical segment1607in second musical composition1606may have a high affinity with one another such that second musical composition1606is a cohesive musical piece.

The user may control how musical composition1606is arranged and structured by interacting with interactive windows, such as segmentation window1608and composer window1610. Segmentation window1608and composer window1610may allow a user to input information for determining segment creation rules, such as segment creation rules408, and affinity rules, such as affinity rules410. Details of segmentation window1608and composer window1610will be discussed further herein with respect toFIGS. 16B and 16C, respectively.

FIG. 16Bshows an enlarged view of segmentation window1608. In embodiments, segmentation window1608may allow a user to initiate creation of musical segments of music performance data1602, and subsequently display pertinent information relating to each musical segment to the user. Segmentation window1608may include a segment creator region1632within which a plurality of options may be presented to a user. Each option may specify one or more segment creation rules, such as segment creation rules408, upon which segmenting music performance data1602may be based. As shown in the example illustrated inFIG. 16B, segment creator region1632may be a plurality of radio buttons selectable by a user. The user may select one or more radio buttons associated with the desired segment creation rule and initiate creation of the musical segments by pressing a clickable button, such as a clickable button labeled “Create Segments”1634. Once the button is clicked, segmentation window1608may display pertinent information relating to the created musical segments.

In embodiments, segmentation window1608may display the pertinent information in a plurality of rows and columns, where each row conveys information pertaining to a specific musical segment and each column conveys information pertaining to various properties of the musical segment. As shown inFIG. 16B, segmentation window1608has a plurality of rows1611, each row relating to a musical segment created from music performance data1602.

As further shown inFIG. 16B, segmentation window1608may have a plurality of columns1612,1614,1616,1618,1620,1622,1624,1626,1628, and1630. Each column may convey information pertaining to various properties of the musical segment In embodiments. InFIG. 16B, column1612may contain names of each musical segment. Each musical segment may be named according to its specific range of bars from music performance data1602. For example, a first musical segment may be named “Bar1-9 (8 Bars)”, as shown inFIG. 16B. However, it is to be appreciated that any other names may be used for naming musical segments.

Columns1614and1616may contain information pertaining to the start and end bar of each musical segment. For instance, column1614may contain a bar number from which a corresponding musical segment starts, and column1616may contain a bar number at which the corresponding musical segment ends. Column1618may contain information pertaining to a bar length of a musical segment. As shown inFIG. 16B, column1618may include the number “8” showing that the musical segments each contain eight bars, which correlates with the segment creation rules from which they were originally created already discussed herein.

Columns1620and1622may contain radio buttons showing which musical segments contain a prologue and an epilogue. Radio buttons that are checked in column1620may indicate that a prologue exists in the musical segment. Additionally, radio buttons that are checked in column1622may indicate that an epilogue exists in the musical segment. Columns1624and1626may contain information pertaining to tempo. Specifically, column1624may contain information relating to a start tempo of a musical segment, and column1626may contain information relating to an end tempo of a musical segment.

Column1628may contain information pertaining to a chord sequence of a musical segment. For instance, column1628may contain a series of letters in a specific order, representing chords arranged in a specific sequence. Displaying the chord sequence of a musical segment may allow a user to visually perceive the chord sequence. Thus, the user may visually rearrange musical segments without having to hear the chord sequence.

Column1630may contain information pertaining to a section for a musical segment. The section may refer to a specific part of a musical piece. For example, the section may refer to an introduction, a chorus, or any other part of a musical piece. Each section may be generically labeled, such as “section A,” “section B,” section C,” and the like.

In addition to using segmentation window1608to create musical segments, a user may create musical segments by interacting with music performance data1602displayed in the user interface. For instance, the user may click and drag a region of music performance data1602to create a musical segment containing the selected region. Additionally, the user may create musical segments by editing musical segments created through segmentation window1608.

AlthoughFIG. 16Billustrates columns1612,1614,1616,1618,1620,1622,1624,1626,1628, and1630, embodiments are not limited to such columns, nor are they limited to the information presented by the columns. As an example, more or less columns may be implemented in segmentation window1608. Additionally, more or less information may be presented by the columns. Furthermore, more or less options may be provided in the segment creator region1632.

B. Composer Window

FIG. 16Cshows an enlarged view of composer window1610. In embodiments, composer window1608may allow a user to initiate creation of a musical composition, such as musical composition406, and subsequently present the musical composition to the user. Like segmentation window1608, composer window1610may display pertinent information for musical segments in a plurality of rows and columns, where each row conveys information pertaining to a specific musical segment and each column conveys information pertaining to various properties of the musical segment.

As shown inFIG. 16C, composer window1610has a plurality of rows1638. As already mentioned herein, each row may represent a musical segment. Rows1638may represent an arrangement of a musical composition. That is, rows1638may be arranged in a sequential order from top to bottom where a top of the order represents the beginning of the musical composition and the bottom represents an ending of the musical composition. In some embodiments, each row may be placed in composer window1610by clicking-and-dragging the desired rows (i.e., musical segments) from segmentation window1608. In other embodiments, each row may be placed in composer window1610by uploading a file containing rows1638.

As further shown inFIG. 16B, segmentation window1608may have a plurality of columns1640,1642,1644,1616,1648,1650,1652,1654,1656,1658,1660, and1662, many of which are similar to those discussed herein with respect toFIG. 16Bshowing segmentation window1608. For instance, columns1640,1644,1616,1648,1650,1652,1654, and1656are similar to columns1612,1614,1616,1628,1624,1626,1620, and1622inFIG. 16B, respectively. Column1642may contain information regarding whether a musical segment is to be enabled or disable. Enabling/disabling segments provide a quick way to omit one or more segments, where enabling the segment includes the segment and disabling the segment excludes the segment from the musical composition.

Column1658may contain information regarding whether the chord progression of the segment should be shown. The chord progression may be the same information shown in column1628. If shown, the user may be able to reference the chord progression in the composer window for ease of reference.

Column1660may contain pulldown menus regarding whether a crossfade is implemented for a musical segment. A crossfade may be a transition, such as transition1028discussed herein with respect toFIG. 10B, for smoothing a transition between two musical segments to enhance cohesiveness of the musical composition. In embodiments, a user may interact with each pulldown menu to effectuate implementation of a crossfade.

Column1662may contain pulldown menus regarding whether a loop is implemented for a musical segment. When the pulldown menu indicates that the musical segment is a loop, it may convey to a user that the musical segment is going to be repeated multiple times in a row in the musical composition. In embodiments, a user may interact with each pulldown menu to indicate whether a musical segment is a duplicate of another musical segment should be repeated in a row or not (and eventually how many times it should be repeated).

In some embodiments, composer window1610also includes a composer region1636. Composer region1636may include various input components, e.g., pulldown menus, radio buttons, and clickable buttons, that allow the user to modify a composition of the musical composition. Each input component may be configured to specify a specific property of the musical compositions. For example, a pulldown menu may determine how much crossfade should be implemented between rearranged musical segments in the musical composition. In another example, a clickable button may allow a user to randomly rearrange the musical segments based upon predetermined musical attributes, such as chord affinity and mixed affinity. Once the user has configured the input components of composer region1636, composer window1610may allow the user to export the musical composition by clicking an “Export Composition” button1634. The musical composition may be exported as an order of an outputted musical compositions as illustrated inFIG. 16A.

FIG. 16Cillustrates columns1640,1642,1644,1616,1648,1650,1652,1654,1656,1658,1660, and1662; however, embodiments are not limited to such columns, nor are they limited to the information presented by the columns. As an example, more or less columns may be implemented in composer window1610. Additionally, more or less information may be presented by the columns. Furthermore, more or less options may be provided in the composer region1636.

VIII. Computer System

FIG. 17is a simplified block diagram depicting a computer system1700that may incorporate components of various systems and devices described herein according to certain aspects of the present disclosure. In some cases, a computing device can incorporate some or all of the components of computer system1700. Computer system1700may include one or more processors1702that communicate with a number of peripheral subsystems via a bus subsystem1704. These peripheral subsystems may include a storage subsystem1706, including a memory subsystem1708and a file storage subsystem1710, user interface input devices1712, user interface output devices1714, and a network interface subsystem1716.

Bus subsystem1704can provide a mechanism for allowing the various components and subsystems of computer system1700communicate with each other as intended. Although bus subsystem1704is shown schematically as a single bus, in some cases, the bus subsystem may utilize multiple busses.

Processor1702, which can be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of computer system1700. One or more processors1702may be provided. These processors may include single core or multicore processors. In some cases, processor1702can execute a variety of programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can be resident in processor(s)1702and/or in storage subsystem1706. Through suitable programming, processor(s)1702can provide various functionalities described above.

Network interface subsystem1716provides an interface to other computer systems and networks. Network interface subsystem1716serves as an interface for receiving data from and transmitting data to other systems from computer system1700. For example, network interface subsystem1716may enable computer system1700to connect to one or more devices via the Internet. In some cases, network interface1716can include radio frequency (RF) transceiver components for accessing wireless voice and/or data networks (e.g., using cellular telephone technology, advanced data network technology such as 3G, 4G or EDGE, WiFi (IEEE 802.11 family standards, or other mobile communication technologies, or any combination thereof), GPS receiver components, and/or other components. In some cases, network interface1716can provide wired network connectivity (e.g., Ethernet) in addition to or instead of a wireless interface.

User interface input devices1712may include a keyboard, pointing devices such as a mouse or trackball, a touchpad or touch screen incorporated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, audio input devices such as voice recognition systems, microphones, eye gaze systems, and other types of input devices. In general, use of the term “input device” is intended to include all possible types of devices and mechanisms for inputting information to computer system1700. For example, in an iPhone®, user input devices1712may include one or more buttons provided by the iPhone® and a touchscreen which may display a software keyboard, and the like.

User interface output devices1714may include a display subsystem, indicator lights, or non-visual displays such as audio output devices, etc. The display subsystem may be a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, a touch screen, and the like. In general, use of the term “output device” is intended to include all possible types of devices and mechanisms for outputting information from computer system1700. For example, a software keyboard may be displayed using a flat-panel screen.

Storage subsystem1706provides a computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of various aspects disclosed herein. Storage subsystem1706can be implemented, e.g., using disk, flash memory, or any other storage media in any combination, and can include volatile and/or non-volatile storage as desired. Software (programs, code modules, instructions) that when executed by a processor provide the functionality described above may be stored in storage subsystem1706. These software modules or instructions may be executed by processor(s)1702. Storage subsystem1706may also provide a repository for storing data used in accordance with the present invention. Storage subsystem1706may include memory subsystem1708and file/disk storage subsystem1710.

Memory subsystem1708may include a number of memories including a main random access memory (RAM)1718for storage of instructions and data during program execution and a read only memory (ROM)1720in which fixed instructions are stored. File storage subsystem1710may provide persistent (non-volatile) memory storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a Compact Disk Read Only Memory (CD-ROM) drive, an optical drive, removable media cartridges, and other like memory storage media.

Computer system1700can be of various types including a personal computer, a portable device (e.g., an iPhone®, an iPad®, and the like), a workstation, a network computer, a mainframe, a kiosk, a server or any other data processing system. Due to the ever-changing nature of computers and networks, the description of computer system1700depicted inFIG. 17is intended only as a specific example. Many other configurations having more or fewer components than the system depicted inFIG. 17are possible.

The above description illustrates various embodiments of the present invention along with examples of how aspects of the present invention may be implemented. The above examples and embodiments should not be deemed to be the only embodiments, and are presented to illustrate the flexibility and advantages of the present invention as defined by the following claims. For example, although certain embodiments have been described with respect to particular process flows and steps, it should be apparent to those skilled in the art that the scope of the present invention is not strictly limited to the described flows and steps. Steps described as sequential may be executed in parallel, order of steps may be varied, and steps may be modified, combined, added, or omitted. As another example, although certain embodiments have been described using a particular combination of hardware and software, it should be recognized that other combinations of hardware and software are possible, and that specific operations described as being implemented in software can also be implemented in hardware and vice versa.

The specification and drawings are, accordingly, to be regarded in an illustrative rather than restrictive sense. Other arrangements, embodiments, implementations and equivalents will be evident to those skilled in the art and may be employed without departing from the spirit and scope of the invention as set forth in the following claims.