Patent Description:
In a conventional solution, the user selects a song and then operates the user interface to so that the solution, such as a mobile application, provides the user with the musical notation of the song or start playing the song or backing track of the song. The user can then play along with the backing track. <CIT> is known, which discloses a search and match system in which a melody played on a piano is matched inside a database, the system selecting a musical exercise or musical accompaniment accordingly. The system further records historical data of previously played exercises and proposes a mode in which only previously played vocal exercises or musical accompaniments are used for the search/match algorithm.

Playing a song is however often a personal, emotional and expressive experience. Ideally, the player knows the song by heart and is able to just play from the heart and sing. The next best thing is playing with the aid of lyrics and chord charts or tablature. Things that distract from this experience are e.g. scrolling through lyrics while playing causing the user to remove their hand from the instrument, or browsing through lists of songs necessitating the user to switch switching from play mindset to analyze mindset.

However, automatic detection of the song from a freely performed user performance, such as a user playing an instrument at their own tempo and choice of harmonies without adhering to the details of a predefined arrangement/notation of the song is difficult. When playing freely, the performer is usually reading chord-based music notation that describes the harmonic progression of the song with chord names, chord symbols, or chord diagrams without detailing the note-by-note arrangement of the song or the rhythmic patterns. Playing freely is very common when playing the part of an accompanying instrument in a song. The challenge in detecting a specific song from such free playing is that the amount of information contained in a few chords is typically not sufficient to disambiguate the song from among a large collection of songs.

One goal of the invention is that a user utilizing the system and method according to the present disclosure does not need to click through an application user interface or menus in order to choose a song or songs that the user wants to perform. Instead, the user can simply start playing, and the system and method recognize the song from the user play history and current user play performance, which may be used to display written music and/or lyrics for the user automatically. Further, the system and method may provide backing track audio pertaining to the recognized song at the tempo of the user, and at the position where the user is in the song.

According to a first example aspect there is provided computer-implemented method for identifying a song comprising:.

According to a second example aspect there is provided a system or apparatus comprising:.

The apparatus may be or comprise a mobile phone.

The apparatus may be or comprise a smart watch.

The apparatus may be or comprise a tablet computer.

The apparatus may be or comprise a laptop computer.

The apparatus may comprise a smart instrument amplifier, such as a smart guitar amplifier.

The apparatus may comprise a smart speaker, such as a virtual assistant provided speaker.

The apparatus may be or comprise a desktop computer.

The apparatus may be or comprise a computer.

According to a third example aspect there is provided a computer program comprising computer executable program code which when executed by at least one processor causes an apparatus at least to perform the method of the first example aspect.

According to a fourth example aspect there is provided a computer program product comprising a non-transitory computer readable medium having the computer program of the third example aspect stored thereon.

According to a fifth example aspect there is provided an apparatus comprising means for performing the method of the first example aspect.

Any foregoing memory medium may comprise a digital data storage such as a data disc or diskette; optical storage; magnetic storage; holographic storage; opto-magnetic storage; phase-change memory; resistive random-access memory; magnetic random-access memory; solid-electrolyte memory; ferroelectric random-access memory; organic memory; or polymer memory. The memory medium may be formed into a device without other substantial functions than storing memory or it may be formed as part of a device with other functions, including but not limited to a memory of a computer; a chip set; and a sub assembly of an electronic device.

The expression "a number of" refers herein to any positive integer starting from one (<NUM>), e.g. to one, two, or three.

The expression "a plurality of" refers herein to any positive integer starting from two (<NUM>), e.g. to two, three, or four.

<FIG> schematically shows a system <NUM> according to an example embodiment. The system comprises a musical instrument <NUM> and an apparatus <NUM>, such as a mobile phone, a tablet computer, smart instrument amplifier, smart speaker, or a laptop computer. The setting may be for example a user playing an instrument <NUM> and using a user apparatus <NUM> at their home.

<FIG> shows a block diagram of an apparatus <NUM> according to an example embodiment. The apparatus <NUM> comprises a communication interface <NUM>; a processor <NUM>; a user interface <NUM>; and a memory <NUM>.

The communication interface <NUM> comprises in an embodiment a wired and/or wireless communication circuitry, such as Ethernet; Wireless LAN; Bluetooth; GSM; CDMA; WCDMA; LTE; and/or <NUM> circuitry. The communication interface can be integrated in the apparatus <NUM> or provided as a part of an adapter, card, or the like, that is attachable to the apparatus <NUM>. The communication interface <NUM> may support one or more different communication technologies. The apparatus <NUM> may also or alternatively comprise more than one of the communication interfaces <NUM>.

In this document, a processor may refer to a central processing unit (CPU); a microprocessor; a digital signal processor (DSP); a graphics processing unit; an application specific integrated circuit (ASIC); a field programmable gate array; a microcontroller; or a combination of such elements.

The user interface <NUM> may comprise a circuitry for receiving input from a user of the apparatus <NUM>, e.g., via a keyboard; graphical user interface shown on the display of the apparatus <NUM>; speech recognition circuitry; or an accessory device; such as a microphone, headset, or a line-in audio <NUM> connection for receiving the performance audio signal; and for providing output to the user via, e.g., a graphical user interface or a loudspeaker.

The memory <NUM> comprises a work memory and a persistent memory configured to store computer program code and data. The memory <NUM> may comprise any one or more of: a read-only memory (ROM); a programmable read-only memory (PROM); an erasable programmable read-only memory (EPROM); a random-access memory (RAM); a flash memory; a data disk; an optical storage; a magnetic storage; a smart card; a solid-state drive (SSD); or the like. The apparatus <NUM> may comprise a plurality of the memories <NUM>. The memory <NUM> may be constructed as a part of the apparatus <NUM> or as an attachment to be inserted into a slot; port; or the like of the apparatus <NUM> by a user or by another person or by a robot. The memory <NUM> may serve the sole purpose of storing data or be constructed as a part of an apparatus <NUM> serving other purposes, such as processing data.

A skilled person appreciates that in addition to the elements shown in <FIG>, the apparatus <NUM> may comprise other elements, such as microphones; displays; as well as additional circuitry such as input/output (I/O) circuitry; memory chips; application-specific integrated circuits (ASIC); processing circuitry for specific purposes such as source coding/decoding circuitry; channel coding/decoding circuitry; ciphering/deciphering circuitry; and the like. Additionally, the apparatus <NUM> may comprise a disposable or rechargeable battery (not shown) for powering the apparatus <NUM> if external power supply is not available.

<FIG> show a flow chart according to an example embodiment. <FIG> illustrate a process comprising various possible steps including some optional steps while also further steps can be included and/or some of the steps can be performed more than once:.

The method may further comprise any one or more of:.

<FIG> illustrates a general view of an example of an embodiment. The user is shown to play an instrument, namely a guitar in this case, using a mobile apparatus with microphone or line-in to track user's performance, i.e. playing of the instrument, and to detect which notes and chords the user plays. The user may start to perform freely with their instrument and the system and method thereof recognize the song after which the user may be provided with written music and lyrics pertaining to the song via the display of the mobile apparatus. Further, the system and method may after recognition of the song start playing a backing track pertaining to the recognized song via the mobile apparatus, optionally at the place and tempo at which the user is performing the song.

The user performance may comprise, for example, playing of a plurality of opening chords of a song or some other position or part of a song. Additionally, user play history is combined with the detected user performance so that a song corresponding to the user performance may be more effectively detected from a large set of songs. The mobile apparatus is provided with user play history, song information and backing track audio data from an external server or cloud arrangement. As mentioned, such song information may comprise notation of the song but also further information such as practice information or alternative versions of the song and/or various types of visualization of the song or user performance thereof. Alternatively, the mobile apparatus may store at least part of the user play history, song information and backing track data. This way the user may utilize the system and method by very little user intervention or operation of the user interface and focus on free expression on their instrument, which is automatically detected and matched with a song to which song information may then be provided automatically for example to accompany the user's playing or help them play or practice the song.

Many tempo estimation techniques are known and may be used since it is a widely discussed topic in prior art. Examples of estimating user activity, playing position and tempo are discussed hereinbelow, which are all obtained by analyzing the performance audio signal in real time:.

By combining information from the above-mentioned three features, and backing track information, we can tackle the various challenges in tracking the position x(t) and playing tempo of an amateur performer and set a backing track corresponding to the position and playing tempo wherein:.

Any of the above-described methods, method steps, or combinations thereof, may be controlled or performed using hardware; software; firmware; or any combination thereof. The software and/or hardware may be local; distributed; centralized; virtualized; or any combination thereof. Moreover, any form of computing, including computational intelligence, may be used for controlling or performing any of the afore described methods, method steps, or combinations thereof. Computational intelligence may refer to, for example, any of artificial intelligence; neural networks; fuzzy logics; machine learning; genetic algorithms; evolutionary computation; or any combination thereof.

Various embodiments have been presented. It should be appreciated that in this document, words comprise; include; and contain are each used as open-ended expressions with no intended exclusivity.

It is however clear to a person skilled in the art that the invention is not restricted to details of the embodiments presented in the foregoing, but that it can be implemented in other embodiments using different combinations of embodiments without deviating from the characteristics of the invention.

Claim 1:
A computer-implemented method for identifying a song comprising:
- providing a set of musical notation information or harmonic progression information for songs,
- receiving a real-time audio signal of a user performing on an instrument,
- detecting playing activity in successive segments of the user in successive segments from the real-time audio signal,
- detecting notes and/or chords from the real-time audio signal,
- storing user play history information comprising of information of songs a user has played before, and how often the user has previously played said songs, optionally additionally in relation to the timing when the user has played said songs,
- based on the play history information calculating a first probability for at least one song,
the method being characterised by:
- based on a number of first probabilities for a number of songs and based on the detected playing activity and the detected notes and/or chords, estimating the song that the user is performing, wherein
o said estimation comprises calculating a second probability for a number of different songs wherein the second probabilities are defined by the real-time audio signal corresponding with a particular song of the play history combined with first probability associated with said song, and
- providing the song the user is performing or information pertaining to the song.