Real-time audio generation for electronic games based on personalized music preferences

Systems and methods are provided for real-time audio generation for electronic games based on personalized music preferences. An example method includes requesting listening history information from one or more music streaming platforms, the listening history information indicating, at least, music playlists to which a user created or is subscribed. A style preference associated with the user is determined based on the listening history information. A musical cue associated with an electronic game is accessed, with the musical being associated with music to be output by the electronic game based on a game state of the electronic game. Personalized music is generated utilizing one or more machine learning models based on the musical cue and the style preference, wherein the system is configured to provide the personalized music for inclusion in the electronic game.

TECHNICAL FIELD

The present disclosure relates to systems and techniques for generating audio. More specifically, this disclosure relates to generating music for utilization in electronic games.

BACKGROUND

Electronic games generally include pre-recorded audio, and the pre-recorded audio may be output to users during gameplay (e.g., via speakers). While an electronic game is being designed, one or more music composers may create music for the electronic game. This music may be created utilizing musical instrument digital interface (MIDI) software, for example based on musical scores. The music may also be recorded utilizing real-life performers. For example, real-life symphonic audio may be generated using a large-scale symphonic orchestra. The orchestra may be recorded during the performance, and the resulting recording utilized in the electronic game. The music created for the electronic game may be output during gameplay.

For example, certain music may be output based on a current game context. As an example, a start screen may be associated with a particular portion of the created music. As another example, a loading screen may be associated with another portion of the created music. As another example, particular music may be played by the electronic game while a user is performing a particular goal or action within the electronic game. In these examples, the electronic game may access specific music via monitoring the current game context.

Due to technical constraints associated with a size of the resulting electronic game, an amount of time required to create music, and so on, the created music may be of a limited length. It may therefore be appreciated that any created music may be repeatedly played to a user of an electronic game. With respect to the above, an electronic game may output a same portion of music each time the electronic game is started. Additionally, the electronic game may output a same portion of music each time the user reaches a particular portion of the electronic game. Since the user may return to the particular portion, for example to repeat objectives or for each of multiple attempts to complete the particular portion, the electronic game may thus repeatedly play the same portion. With respect to certain types of electronic games, such as online-based games, background music may continuously be played during the electronic game. For example, in a first-person shooter online game, the online game may cause continuous background music to be played.

At present, users of electronic games may thus grow tired or bored of the music being output to them. They may therefore prefer to reduce a volume associated with the in-game music or may play their own music with the in-game music turned off. Thus, there is a need for improved techniques to enable the generation of personalized music while avoiding the technical constraints described above.

SUMMARY OF CERTAIN EMBODIMENTS

Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. Utilizing the techniques described herein, electronic games may present varied, and personalized, music to users of the electronic games. In contrast to current schemes in which music is pre-recorded, the techniques described herein enable real-time (e.g., substantially real-time) generation of music personalized to specific users. Thus, and as will be described, a user experience associated with playing electronic games may be improved. For example, users may avoid having to consistently hear the same music as they go about enjoying an electronic game.

In current electronic games, and with particular emphasis on higher-tier games, music may be utilized to great emotional effect within the games. The music may be used similarly to that of higher-tier movies, for which musical cues and patterns may be designed to evoke specific emotional responses. For example, symphonic music may be employed in movies to evoke feelings of sadness, happiness, excitement, and so on. In an electronic game, symphonic music may also be employed to elicit specific feelings depending on a user's current progression in the electronic game. An example electronic game may be a first-person game (e.g., a first-person shooter, role-playing game, and soon). For this example game, a user may traverse different in-game regions and perform disparate actions. As the user performs actions within the electronic game, an overall tone or feeling may be adjusted.

As an example, if the user encounters a plethora of non-playable characters (e.g., enemies), or other users with respect to an online game, the electronic game may utilize certain cues to evoke excitement. It may be appreciated that musical cues may form a basis, at least in part, for such evocation of excitement. For example, the music may increase in tempo or may be adjusted in style. As another example, the electronic game may output music with different themes, motifs, musical keys, different harmony or dissonance, and so on. With respect to a motif, the electronic game may optionally consistently utilize the motif to evoke excitement.

In this way, as the user plays the electronic game, the electronic game may adjust the music being output to the user. As will be described below, this adjustment may optionally be based on a current game state associated with the electronic game. A game state may represent game information (e.g., information for which the electronic game monitors), such as information associated with a progress of the user, current actions being performed by the user, upcoming events to be triggered, and so on. For example, the electronic game may trigger the output of certain music based on the user collecting a particular class of item. In this example, the music may include a particular theme which the user may associate with the action of collecting the particular class of items.

However, music within electronic games may begin to feel stale as users play the electronic games. For example, if an electronic game continuously utilizes a same, or similar, portion of music to evoke excitement, then users of the electronic game may grow tired of this portion. Thus, one of the uses of the portion (e.g., to evoke excitement) may be diminished. The users may then lower, or eliminate, a volume associated with this in-game music. Additionally, music within electronic games may not comport with certain users' style preferences. Thus, if the portion of music is associated with a first style (e.g., a particular genre of music), certain users may similarly lower, or eliminate, a volume associated with the in-game music. Instead, if this portion of music were to be adjusted in style to a second style, these users may have a much greater affinity, or emotional response, to in-game events, actions, and so on.

As will be described, a system described herein may access one or more music streaming services of a user. The system may obtain information indicative of musical preferences of the user, such as a listening history, playlists created or subscribed to by the user, and so on. The system may then determine a style or genre preference of the user. Based on this genre preference, the system may then utilize machine learning techniques to generate music for a particular electronic game which comports with this genre preference.

To generate the music, the system may access musical cues for the particular electronic game. For example, the musical cues may be created for use in the electronic game and may be indicative of a particular in-game context. An example musical cue may indicate a particular theme, motif, melody, and so on. In some embodiments, a musical cue may indicate a particular emotion or feeling to be achieved from the generated music. These musical cues may be created by a game composer or by a machine learning model. The system may generate music based on musical cues. For example, if a musical cue indicates a particular melody, then the system may generate music based on the melody and a genre preference for a user. As another example, if a musical cue indicates a particular emotion (e.g., happiness), then the system may generate music based on the emotion and genre preference. Therefore, the generated music may be based on specific musical cues, but adjusted for each user based on the user's empirically determined genre preferences.

Since the genre preferences have been determined based on actual listening habits of users, it may be appreciated that the resulting music may have a greater likelihood of keeping a user's interest. In this way, a user experience associated with an electronic game may be improved. As described above, this music may be generated utilizing machine learning models, for example in substantially real-time as the user plays an electronic game. Therefore, the music may optionally be generated such that it varies between uses. For example, a user may play a first-person game and reach a certain point within the game. Each time the user reaches this point, and absent the techniques described herein, the electronic game may output a same, or similar, portion of music (e.g., pre-recorded music). In contrast, the system may generate distinct music each time the user reaches the point. Advantageously, the distinct music may be in conformance with the user's genre preference and the musical cue associated with reaching the point.

The techniques described herein additionally improve upon the functioning of electronic games and prior schemes to generate audio. As described above, prior schemes lacked the technical ability to personalize music for end-users. Thus, electronic games were required to output a same set of music regardless of user preference. In contrast, the techniques described herein enable an electronic game to output consistent music regardless of user, which has been generated to be distinct according to user preference. Additionally, prior schemes required musical composers to utilize complex software, and spend large quantities of time, to generate different, and lengthy, musical audio clips for utilization in an electronic game. These different musical audio clips may then consume storage space of game systems utilized by end-users. In contrast, the techniques described herein allow musical composers to focus on crafting musical cues. The system described herein may then generate complex (e.g., polyphonic) music utilizing these musical cues. Since genre preferences of the end-users may be utilized to generate music, the techniques described herein may substantially expand upon the musical language which is achievable by an electronic game.

The systems, methods, and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for the all of the desirable attributes disclosed herein.

One embodiment discloses a method comprising: by a system of one or more computers, requesting listening history information from one or more music streaming platforms, the listening history information indicating, at least, music playlists to which a user created or is subscribed; determining, based on the listening history information, a style preference associated with the user; accessing a musical cue associated with an electronic game, the musical cue being associated with music to be output by the electronic game based on a game state of the electronic game; and generating, utilizing one or more machine learning models, personalized music based on the musical cue and the style preference, wherein the system is configured to provide the personalized music for inclusion in the electronic game.

Various embodiments of the method may include one, all, or any combination of the following features. Requesting listening history information comprises: receiving, from a user device executing the electronic game, information indicating initiation of the electronic game; obtaining authentication information associated with the user; and requesting the listening history information utilizing the authentication information. The system is configured to subscribe to updates, from the one or more music streaming platforms, of the user's listening history information. Determining a style preference comprises: grouping a threshold number of songs identified in the listening history information; generating the style preference based on the threshold number of songs. Generating the style preference comprises assigning a weight to each of a plurality of musical styles based on the threshold number of songs. Accessing a musical cue comprises: receiving, from a user device executing the electronic game, a unique identifier associated with a musical cue; and accessing one or more databases, and obtaining the musical cue based on the unique identifier. The method further comprises accessing, based on information identifying the electronic game, musical cues associated with the electronic game; and generating personalized music based on a plurality of the musical cues and the style preference. The plurality of the musical cues are selected based on the game state of the electronic game. A musical cue comprises a portion of musical audio or a melody. A machine learning model comprises a recurrent neural network or a convolutional neural network.

Some aspects feature a system comprising one or more computers and non-transitory computer storage media storing instructions that when executed by the one or more computers, cause the one or more computers to perform operations comprising: requesting listening history information from one or more music streaming platforms, the listening history information indicating, at least, music playlists to which a user created or is subscribed; determining, based on the listening history information, a style preference associated with the user; accessing a musical cue associated with an electronic game, the musical cue being associated with music to be output by the electronic game based on a game state of the electronic game; and generating, utilizing one or more machine learning models, personalized music based on the musical cue and the style preference, wherein the system is configured to provide the personalized music for inclusion in the electronic game.

Various embodiments of the system may include one, all, or any combination of the following features. Requesting listening history information comprises: receiving, from a user device executing the electronic game, information indicating initiation of the electronic game; obtaining authentication information associated with the user; and requesting the listening history information utilizing the authentication information. Determining a style preference comprises: grouping a threshold number of songs identified in the listening history information; generating the style preference based on the threshold number of songs. Generating the style preference comprises assigning a weight to each of a plurality of musical styles based on the threshold number of songs. Accessing a musical cue comprises: receiving, from a user device executing the electronic game, a unique identifier associated with a musical cue; and accessing one or more databases, and obtaining the musical cue based on the unique identifier. A machine learning model comprises a recurrent neural network or a convolutional neural network.

Some aspects feature non-transitory computer storage media storing instructions that when executed by a system of one or more computers, cause the computers to perform operations comprising: requesting listening history information from one or more music streaming platforms, the listening history information indicating, at least, music playlists to which a user created or is subscribed; determining, based on the listening history information, a style preference associated with the user; accessing a musical cue associated with an electronic game, the musical cue being associated with music to be output by the electronic game based on a game state of the electronic game; and generating, utilizing one or more machine learning models, personalized music based on the musical cue and the style preference, wherein the system is configured to provide the personalized music for inclusion in the electronic game.

Various embodiments of the computer storage media may include one, all, or any combination of the following features. Requesting listening history information comprises: receiving, from a user device executing the electronic game, information indicating initiation of the electronic game; obtaining authentication information associated with the user; and requesting the listening history information utilizing the authentication information. Determining a style preference comprises: grouping a threshold number of songs identified in the listening history information; generating the style preference based on the threshold number of songs, the style preference comprises an assigned weight to each of a plurality of musical styles, the assignment being based on the threshold number of songs. A machine learning model comprises a recurrent neural network or a convolutional neural network.

Although certain embodiments and examples are disclosed herein, inventive subject matter extends beyond the examples in the specifically disclosed embodiments to other alternative embodiments and/or uses, and to modifications and equivalents thereof.

The details, including optional details, of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other optional features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

DETAILED DESCRIPTION

Overview

This specification describes techniques to generate personalized music based on determined musical style preferences of users (referred to herein as a ‘style preference’). A style preference of a user may indicate information indicative of preferred musical genres to which the user listens. As is known, music may be classified according to disparate features or qualities. These classifications may form categorizations of music, such as genres. Example genres may include symphonic music (e.g., classical, impressionist, modern), jazz, hip-hop, rock including subgenres thereof, electronic music, and so on. As will be described, a system described herein (e.g., the music personalization and generation system100) may determine a style preference of a user and then generate music of a same, or similar, musical style. Advantageously, the generated music may be provided to the user during gameplay of an electronic game. In this way, the music being output to the user may be customized according to the user's preferences.

It may be appreciated that music in electronic games is typically pre-recorded music, for example utilizing MIDI or real-life performers. As described above, this pre-recorded music may grow tiresome to users of the electronic games such that they lower, or eliminate, a volume of the music. Utilizing machine learning techniques, the system may instead generate music in real-time (e.g., substantially real-time) as a user plays an electronic game. Example machine learning techniques may include utilization of artificial neural networks, such as recurrent neural networks and/or convolutional neural networks. These example machine learning techniques will be described in more detail below, with respect to at leastFIG. 2. It may be appreciated that additional machine learning techniques may instead be utilized.

The system described herein may determine a style preference of a user via accessing the user's accounts with one or more music streaming platforms. Example music streaming platforms may include SPOTIFY, APPLE MUSIC, and so on. The system may receive authentication information from the user, such as a user name/password, authentication token, and so on. This authentication information may be utilized to access the user's music streaming platforms. In all situations in which personal information, or authentication information, of a user is utilized, it may be appreciated that these uses may be ‘opt-in’. Thus, users may be required to affirmatively indicate that they wish to receive personalized music.

To provide authentication information, the user may utilize a user interface to specify the authentication information for use by the system. For example, the user interface may represent a log-in screen associated with a music streaming platform. Certain music streaming platforms may additionally enable the sharing of music listening information with outside systems, users, social media platforms, and so on. For example, as a user utilizes a music streaming platform, the listening history may be broadcast by the platform. This broadcast listening history may, for example, be consumed by a social media platform. The social media platform may thus present the user's listening history in substantially real-time, for example in a social network feed. In this way, friends of the user on the social media platform may view the music being listened to by the user. Similarly, the user may indicate that the system described herein is to receive such broadcast listening history. The system may then aggregate the listening history to determine a style preference of the user.

To determine the style preference, the system may optionally access metadata information associated with music being streamed by the user. For example, the metadata information may indicate a genre of each song or a genre of an artist. The system may then determine the genres which are being listened to by the user. As will be described, the system may generate music based on these determined genres. In some embodiments, the system may indicate that the style preference represents a threshold number of songs listened to by the user. For example, the system may access the user's listening history and select a threshold number of the most recently played songs. As will be described, the system may generate music based on these threshold number of songs. For example, the resulting music may represent a combination of features, or styles, extracted from the threshold number of songs by the system.

The determined style preference for the user may be utilized as the user plays an electronic game. As will be described, the user may initiate an electronic game on a user device, such as a video game console, laptop, handheld device, and so on. The system may receive information indicating the initiation, and may access the determined style preference. The style preference may then be utilized by the system to generate music, for example based on contextual information occurring in the electronic game. The generated music may, in some embodiments, be provided for output by the user device via a network (e.g., the internet). In some embodiments, the user device may generate personalized music according to the techniques described herein.

The electronic game may include certain musical cues which may be selected for output by the electronic game based on a current game state. A game state in this specification may be construed broadly, and may indicate contextual information associated with gameplay (e.g., information being monitored by the electronic game). Thus, the musical cues may be selected as the user plays the electronic game. As an example, a certain musical cue may be selected based on the user reaching a particular progression point within the electronic game. An example of a musical cue may include a musical theme, an indication of emotion, and so on. A musical composer for the electronic game may thus simply create a motif, theme, indicate an emotion, and the system may generate corresponding music. Additional examples of musical cues will be described below.

Utilizing a selected musical cue, the system may then generate music to be output during gameplay. As will be described, the system may utilize machine learning techniques to expound upon the selected musical cue. Advantageously, the generated music will be generated according to the determined style preference of the user. Thus, if the user prefers hip-hop then the system may generate music which is in a same style. In contrast, if a different user is playing the same electronic game, the system may generate music in a different style (e.g., jazz).

While the description below focuses on electronic games (e.g., video games), it may be understood that the techniques described herein may be applied to different use cases. For example, music may be generated for utilization in movies. In this example, a user of, for example, a video streaming platform may view a movie or television show via the platform. During the movie, music may be generated which comports with the user's style preferences. Similar to the description herein, the movie may comprise certain musical cues which may be adjusted by the system.

Furthermore, advantageously the techniques described herein may optionally be applied to any game, for example without modifying the electronic games. As an example, an electronic game may include pre-recorded music. The system may analyze the pre-recorded music, and generate personalized music based on the pre-recorded music. Without being constrained by theory, it may be appreciated that a convolutional neural network may utilize raw audio (e.g., audio waveforms). For example, the neural network may utilize WaveNets. This neural network may then generate personalized music according to a user's style preference. As another example, an electronic game may include representations of musical notes for output (e.g., a MIDI representation). The system may similarly utilize these representations to generate personalized music. In these examples, the system may receive audio being generated by the electronic game and replace it with its own generated audio.

Example Block Diagram

FIG. 1is a block diagram illustrating an example music personalization and generation system100generating personalized music102. As described above, the music personalization and generation system100may access a streaming music system120to determine a style preference of a user of a game system110. The music personalization and generation system100may then generate personalized music102for output by the game system110. The personalized music102may be based on current game information114associated with an electronic game being played on the game system110. For example, the current game information114may represent a current game state of the electronic game. The current game information114may be utilized by the system100to select a musical cue142. The music personalization and generation system100may then generate personalized music102using the musical cue142and the style preference of the user.

The music personalization and generation system100may be a system of one or more computers, one or more virtual machines executing on a system of one or more computers, and so on. Optionally, the music personalization and generation system100may represent a cloud-based system which receives information from multitudes of game systems. The game systems may optionally be executing different electronic games. In this way, the music personalization and generation system100may generate personalized music for each user. As will be described, the music personalization and generation system100may be in communication with a game music database140. The game music database140may include musical cues142associated with the different electronic games, such that the music personalization and generation system100may generate personalized music for different electronic games.

The game system110may represent a user device of the user, such as a video game console, laptop or personal computer, handheld device, and so on. The game system110may include a game engine112associated with an electronic game. For example, a user of the game system110may select a particular electronic game via a user interface presented via the game system110. The game system110may then initiate the electronic game for playing by the user.

In some embodiments, the game engine112may perform game processing and render visual elements of the electronic game for presentation via a display. For example, the game engine112may comprise executable code associated with the electronic game. The game system110may utilize graphics processing units, central processing units, and so on, to render the visual elements of the electronic game. Optionally, the electronic game may be played via a streaming platform. For example, the game system110may represent a thin client. The user may utilize an application (e.g., an ‘app’), or a browser, to access the streaming platform. The visual elements, and game processing, may be performed by an outside system (e.g., a cloud-based system). These visual elements may then be provided to the game system110for presentation via a network (e.g., the internet). In these embodiments, the game engine112may thus represent an application associated with the streaming platform.

Upon initiation of the electronic game, the game system110may provide game information114to the music personalization and generation system100. In some embedment's, the system110may provide game information114upon selection of an option to receive personalized music (e.g., an in-game option). The game information114, as described above, may represent information indicative of a current game state. Absent the techniques described herein, it may be appreciated that an electronic game may adjust music being presented to a user based on a current game state. For example, if the user encounters a particular boss then the electronic game may output certain boss, or battle, music. The electronic game may thus utilize the game state as a trigger to access certain music for output to the user.

Similarly, the music personalization and generation system100may utilize received game information114to generate personalized music102appropriate for the current game state. For example, if the electronic game is presenting a start menu, the game information114may thus indicate the start menu. As another example, the user may be controlling a character within a certain region of the game-world. In this example, the game information114may indicate the certain region. As the user manipulates the character within the region, the game information114may be updated to indicate triggers or contextual information associated with the manipulation. In this way, if the user encounters a particular boss, or enters a particular room, the game information114may be updated to reflect these encounters.

The music personalization and generation system100may thus utilize the game information114to select one or more musical cues. These musical cues may be created, for example by a composer, for utilization in certain game contexts. Thus, there may be one or more musical cues associated with battling enemies, certain enemies, and so on. An example musical cue may include a motif, theme, snippet of audio, and so on, which the composer indicates is to be utilized for one or more game contexts. Another example musical cue may include an indication of emotion, or feeling, to be achieved by music generated for one or more game contexts. As an example, a musical cue may indicate that while battling an enemy, the music is to be perceived as causing excitement. Optionally, the musical cue may comprise a data structure (e.g., a vector) in which different emotions may be assigned weights or values. Thus, in the example of battling an enemy, the composer may assign a higher value for excitement than for other emotions. In this way, the composer may enable fine-grain control over the emotion to be achieved by the music.

As described above, the game information114may include information indicative of these game contexts. In some embodiments, the game information114may indicate a unique identifier associated with a musical cue. For example, as the user plays the electronic game, the electronic game may obtain a unique identifier associated with music to be output to the user. The electronic game may then provide the unique identifier to the music personalization and generation system100. The music personalization and generation system100may access a musical cue associated with the received identifier.

Optionally, the electronic game may obtain multiple unique identifiers. For example, if the user is controlling a character within a game world, the electronic game may determine that a multitude of musical choices may be output depending on subsequent actions of the user. Thus, the electronic game may select a subset of the musical choices. As an example, the subset may be based on a likelihood of the user following certain choices. As another example, the subset may be based on a proximity of the user's character to triggers associated with the musical choices. The music personalization and generation system100may then access musical cues for each unique identifier, and generate personalized music102. The game system110may receive the generated personalized music102, and cache the music102. Depending on the actions taken by the user, the game system110may output corresponding personalized music.

To generate personalized music102, for example based on an obtained musical cue, the music personalization and generation system100may utilize listening information122obtained from one or more music streaming platforms. Examples of listening information may include playlists created by the user, playlists subscribed to by the user, specific songs or artists selected by the user, and so on. This listening information122may be utilized by the music personalization and generation system100to determine a style preference of the user.

As illustrated, the music personalization and generation system100may receive authentication information112from the game system110. The authentication information112may include a user name/password associated with the user, an authentication token (e.g., an OAuth token), and so on. Optionally, the music personalization and generation system100may obtain information from a user profile associated with the user. For example, the user may create a user profile accessible to the system100. The user may then include authentication information associated with a music streaming platform utilized by the user. In some embodiments, the user may indicate that the system100is authorized to receive listening information122from a music streaming platform. Thus, the music personalization and generation system100may store authentication information received from a music streaming platform indicating authorization to obtain (e.g., read) the user's listening information122.

FIG. 1illustrates an example playlist124being stored by a music streaming system120associated with an example music streaming platform. The example playlist124may represent a playlist created, or subscribed to, by the user utilizing an application associated with the platform (e.g., a web application, a mobile application, and so on). Information included in the playlist124may be provided to the music personalization and generation system100. Example information may include an indication of the songs included in the playlist124(e.g., song title, artist name, and so on). Example information may further include an indication of metadata associated with the songs. For example, a genre of the song and/or artist may be included in the listening information122. Example information may further include an indication of a different user who created the playlist. The music personalization and generation system100may optionally utilize style preferences of the different user to inform style preferences of the user of the game system110. Optionally, the different user may be an artist on the streaming music platform. The user may have subscribed to this artist's playlist, and the music personalization and generation system100may utilize genre or style information associated with the artist.

To determine a style preference of the user, the music personalization and generation system100may utilize the received listening information122described above. For example, the music personalization and generation system100may aggregate songs the user has listened to within a threshold amount of time. The aggregated songs may be utilized by machine learning techniques to determine a style preference. For example, the style preference may represent a weighted, or learned, combination of the genres represented in the aggregated songs. Thus, if the user mainly listens to folk music, but occasionally listens to heavy metal, the music personalization and generation system100may determine the style preference as an aggregated version of folk music and heavy metal.

The music personalization and generation system100may also select the style preference based on a most listened to genre of music in the listening information122. For example, if a measure of central tendency of a certain genre exceeds a threshold, the music personalization and generation system100may set the genre as the style preference. Optionally, the music personalization and generation system100may utilize a threshold number of prior songs to inform the determination. Optionally, the user may indicate that only certain playlists, songs, or artists, are to be utilized by the music personalization and generation system100when determining the user's style preference. For example, the user may occasionally listen to classical music, but may not prefer that symphonic music be output to the user during gameplay.

In some embodiments, the music personalization and generation system100may determine multiple style preferences for the user. For example, a musical cue may be designated by a composer as being appropriate for certain genres of music (e.g., orchestral, rock n roll, funk, and so on). Thus, when generating personalized music based on this musical cue, the music personalization and generation system100may select a style preference of the user which adheres closest to the genre indicated by the composer. Optionally, the music personalization and generation system100may analyze a musical cue and select a particular style preference based on the analyses. For example, a musical cue may be determined to be atmospheric. In this example, the music personalization and generation system100may select a style preference corresponding to slower, atmospheric, music (e.g., Brian Eno may be included in the user's listening information122). As another example, a musical cue may be determined to have a high tempo, complicated beat patterns, and so on. Thus, the music personalization and generation system100may select a different style preference.

As will be described in more detail below, with respect toFIG. 2, the music personalization and generation system100may then generate personalized music102for the user. In embodiments in which the musical cue comprises a melody, theme, snippet of audio, the system100may generate music which expands upon the musical cue and is based on the style preference of the user. The musical cue may optionally be monophonic, such as melody line. The system100may utilize this melody line to generate complex polyphonic music. For example, the melody line may be provided to a neural network. The neural network may optionally be a sequence model, such as a recurrent neural network utilizing long short-term memory (LSTM) units. The neural network may thus generate music based on probabilistically determining music which best fits the melody line.

The game system110may then receive, and output, the personalized music102. In some embodiments, the music personalization and generation system100may generate multitudes of personalized music102. For example, the system100may access musical cues (e.g., from database140) for the entirety of the electronic game. The system100may then generate personalized music based on the entirety of the musical cues. This generated music may then be provided to the game system110(e.g., to be cached by the system110). Optionally, and as described above, the music personalization and generation system100may generate personalized music102for a subset of musical cues.

WhileFIG. 1described the game system110as providing game information114to the music personalization and generation system100, it may be appreciated that the game system110may generate personalized music. For example, the game engine112may include machine learning models which may be utilized to generate the music102. The game engine112may store the weights, biases, hyperparameters, and so on, which are associated with the machine learning models. Thus, the game engine112may perform a forward pass of a machine learning model utilizing e a musical cue and style preference as an input.

FIG. 2is a block diagram illustrating details of the example music personalization and generation system100. As described above, the music personalization and generation system100may generate personalized music102for inclusion in an electronic game. A user of the electronic game may thus hear music which has been customized according to the user's musical style preference. The music personalization and generation system100may utilize a musical cue142to generate the personalized music142, which a composer or game designer has specified. For example, the musical cue142may be associated with a particular context, or game state, within the electronic game. The music personalization and generation system100may identify a particular musical cue based on game information (e.g., information114). The music personalization and generation system100may then generate personalized music102utilizing the musical cue (e.g., as a seed).

To generate the personalized music102, the music personalization and generation system100includes a style determination engine210. The style determination engine210may access, or receive information from, music streaming platforms utilized by a user. The style determination engine210may also obtain, or receive, playlists associated with music stored by the user. For example, the user may have local storage, or networked storage, which stores albums, songs, and so on, to which the user listens. In some embodiments, the user may specify styles or genres of music of interest to the user. For example, the user may utilize a user interface to select, or specify, styles. The user interface may optionally be presented via a web application associated with the music personalization and generation system100. The user interface may also be presented during an initial set up, or in a settings menu, of the electronic game or game system110.

As described inFIG. 1, the style determination engine210may determine a style preference212for the user. As an example, the style determination engine210may group a threshold number of songs or artists. For example the threshold number may represent a threshold number of recently listened to songs or artists. As another example, the threshold number may represent a threshold number of the most listened to songs or artists. The style determination engine210may then designate this group as the style preference212of the user.

With respect to the above grouping, the style determination engine210may determine features associated with the songs or artists included in the group. As an example, the style determination engine210may assign a weight to each of multiple genres represented in the group. This weight may, in some embodiments, represent a multi-hot encoding of the genres. As another example, the style determination engine210may determine features associated with the songs. For example, the features may indicate style information, such as tempo, key or chord changes, sequences of notes which are utilized, scales utilized, instruments which are represented, and so on. In some embodiments, the group of songs or artists may be provided to a machine learning model which may generate a learned representation or encoding of the style preference212.

The music personalization and generation system100further includes a music generation engine220. The music generation engine220may receive the determined style preference212from the style determination engine210, and utilize the style preference212to generate personalized music. As described above, the personalized music102may be generated utilizing a musical cue142. In some embodiments, the musical cue142may represent an indication of a particular emotion or feeling to be achieved from the personalized music102. In some embodiments, the musical cue142may comprise a musical encoding144A or an audio waveform144B.

In the example of a musical encoding144A, the musical cue142may be discretized into one or more example forms. The musical encoding144A may thus represent an abstraction of a musical portion. As an example, the musical encoding144A may include information identifying musical notes. These musical notes may be obtained from a score generated by a composer or game designer. The musical notes may also be obtained from a MIDI representation of the musical cue (e.g., a MIDI representation of a melody). The encoding144A may include an indication of the musical notes in a multi-hot or single-hot encoding scheme. Additionally, the musical encoding144A may include identifying information associated

The musical encoding144A may further include information indicative of timing. For example, the musical encoding144A may be sampled at a particular rate. For musical notes which are sustained between samplings, the encoding144A may include a label indicating the sustain (e.g., a continued note). For musical notes which are not sustained, the encoding144A may include a label indicating when that a new note has started. Similarly, the musical encoding144A may indicate information such as tempo, time signature, and so on.

While the above includes examples of storing a representation of a musical cue142in a data structure, it may be appreciated that the musical encoding144A may be represented differently. For example, the musical encoding144A may indicate quality information associated with musical notes. In this example, the quality information may indicate a measure of distortion, presence of overtone frequencies, whether it sounds percussive, has reverb, and so on. Thus, a composer or game designer may generate the musical cue142utilizing disparate instruments and the nuances of the sound may be captured in the musical encoding144A.

In the example of an audio waveform144B, a composer or game designer may record a portion of audio. For example, the audio waveform144B may represent a recording of a composer utilizing an instrument. As another example, the audio waveform144B may be generated by a MIDI instrument. In some embodiments, the audio waveform144B may include a portion of complex (e.g., polyphonic) audio. In these embodiments, the audio waveform144B may be a recording a symphony, one or more professional musicians, a complex electronic recording, and so on.

The audio waveform144B may be provided along with label information. The label information may indicate features of the audio waveform, such as instruments represented in the audio waveform144B. Additionally, specific dynamics may be labeled. For example, it may not be apparent from the audio waveform144B how loud or soft the musical cue is to appear. Thus, the dynamics may be encoded (e.g., in a data structure) and utilized to generate the personalized music.

As illustrated inFIG. 2, the musical encoding144A may be utilized as an input to an artificial neural network222. For example, a recurrent neural network, such as a network utilizing long short-term memory units or gated recurrent units, may receive the musical encoding144A. In addition the style preference212may be received by the neural network222. This neural network222may then perform a forward pass through the units. As the received information is processed by the neural network222, the network222may generate musical output. For example, the neural network222may be trained utilizing different input music. To ensure that long-term structure associated with generated output is maintained, the neural network222may leverage one or more techniques. Example techniques may include skip connections, utilizing attention, and so on. In this way, cohesive music may be generated. Optionally, one or more hand tuned models may be utilized to ensure that notes do not substantially vary in pitch, or are not repeated too often.

The style preference212may inform a particular style of the personalized music102generated by the neural network222. For example, during training the neural network222may learn a meaning associated with input styles. As described above, style or genre may be encoded. In some embodiments, the neural network222may learn a style embedding (e.g., as a contextual input, which may be connected with each layer). Based on this style encoding or embedding, the neural network222may be able to translate between style domains.

As illustrated inFIG. 2, the audio waveform144B may be utilized as an input to a different artificial neural network224. For example, a convolutional autoencoder may be utilized. The convolutional autoencoder224may comprise a multitude of convolutional layers. The received audio waveform144B may be transformed from a time series through the convolutional layers. In some embodiments, the convolutional layers may represent WaveNet layers. The convolutional autoencoder may be trained utilizing multiple autoencoder pathways, for example one per style or genre. Subsequent to training, the audio waveform144B may be analyzed by the network224in view of the style preference212. The network224may then generate personalized music102in accordance with the style preference.

Example Flowcharts

FIG. 3is a flowchart of an example process300for associating determined a musical style preference with a user. For convenience, the process300will be described as being performed by a system of one or more computers (e.g., the music personalization and generation system100). In some embodiments, a user device of one or more processors (e.g., the game system110) may perform one or more of the blocks described below.

At block302, the system receives authentication information associated with one or more music streaming platforms. As described above, the user may provide a user name/password associated with the one or more music streaming platforms to the system. Optionally, upon initiation of an electronic game, the electronic game may request authorization to access the user's music streaming platform. Optionally, the electronic game may include a setting to enable the system's access.

At block304, the system accesses listening information from the music steaming platforms. The system may receive playlist information, such as playlists the user has created utilizing the music streaming platforms. The system may also receive streaming history information, such as a recent, or the entirety of, listening history by the user.

At block306, the system determines a style preference of the user. The system may utilize the listening information to determine a style or genre preference. This determined style preference may represent a model associated with styles of music to which the user listens. As described inFIGS. 1-2, the system may group recently listened to songs or artists together. The system may then generate the style preference based on this grouping. For example, the system may assign a weight or value for each of a multitude of musical styles. This assigned weight or value may then be utilized to generate personalized music for the user. In some embodiments, the system may identify a threshold number of styles to which the user listens. For example, these may represent a most commonly listened to number of styles. When generating personalized music, the system may then select from among the styles.

At block308, the system associates the determined style preference with the user. Optionally, the system may store a user profile associated with the user. The style preference of the user may include in this user profile, such that the system may access the style preference when generating personalized music. Optionally, the system may periodically access listening information associated with the user and re-determine style preference information. The system may then associated the determined style preference with the user.

FIG. 4is a flowchart of an example process for generating personalized music based on a style preference. For convenience, the process400will be described as being performed by a system of one or more computers (e.g., the music personalization and generation system100). In some embodiments, a user device of one or more processors (e.g., the game system110) may perform one or more of the blocks described below.

At block402, the system receives an indication of initiation of an electronic game. As described above, the user may utilize a user device, such as a game console, laptop, and so on, to initiate an electronic game. In implementations in which the system provides personalized music via a network (e.g., the internet) to the user device, the system may receive the initiation via the network. In some implementations, the system may represent an engine, or module, included in the user device. In these implementations, the engine or module may receive the initiation. The engine or module may then generate personalized music for inclusion in the electronic game.

At block404, the system obtains a style preference of the user. As described inFIG. 3, the system may store a style preference which has been determined from listening information of the user. In some embodiments, the system may receive information from the user specifying the style preference in lieu of determining the style preference. For example, the system may prompt the user for an indication of preferred style. The user may then select from among the indicated styles, for example in a user interface, or may provide verbal input identifying a style.

At block406, the system receives game information. As the user plays the electronic game, the game state may be adjusted based on gameplay. The system may monitor this game information to ascertain which music is to be played via the electronic game. As described above, the electronic game may provide one or more unique identifiers associated with musical cues to the system. That is, instead of the system utilizing a game state to identify a musical cue, the system may receive information identifying a specific musical cue.

At block408, the system obtains a musical cue associated with the game information. The system may maintain musical cues for the electronic game, and optionally for a multitude of electronic games. Thus, the system may utilize the game information, such as a unique identifier, to obtain the corresponding musical cue. In some embodiments, the system may obtain a multitude of musical cues. For example, upon initiation of the electronic game the system may obtain multiple musical cues. The system may then generate personalized music410based on these musical cues. In this way, the system may rapidly provide music to the electronic game as the user traverses the electronic game.

At block410, the system generates personalized music. As described inFIGS. 1-2, the system may generate personalized music based on one or more obtained musical cues and the style preference. As an example, a musical cue may include a theme, motif, melody, snippet of audio, and so on. In this example, the system may generate music in the user's style preference. The system may then provide (e.g., as a stream, or as a download) the personalized music to the user's user device. In implementations in which the user device generates the personalized music, the engine or module executing on the user device may generate the personalized music. The electronic game may listen for updates from the engine or module, and may include the personalized music in the electronic game.

The system may generate a threshold length of personalized music. The threshold may be based on the musical cue, or may be set by a game designer or composer. For example, certain music may represent background music. For the background music, the system may generate lengthy background music. As another example, certain music may represent short musical cues based on a user triggering some action or event within the electronic game. In this example, the system may generate shorter music. Optionally, the system may receive updates from the user device. For example, with respect to background music the user device may indicate that the user has, or is about to, exceed the generated background music length. In this example, the system may generate additional personalized music to be included in the background.

In some implementations, the system may utilize personalized music which has been generated for one or more other users. For example, instead of generating personalized music for each user, the system may identify one or more other users with similar style preferences to the user. In this example, the system may determine respective measure of similarity between users. With respect to the style preference being a weight or value for a multitude of styles, the system may compute a measure of difference between these weights or values (e.g., an L1 or L2 norm). Thus, the system may obtain personalized music for a similar user. The system may then provide the obtained personalized music to the user. In this way, if two users listen to substantially similar music the system may conserve processing resources associated with generating music.

FIG. 5is a flowchart of an example process for an electronic game to output personalized music. For convenience, the process500will be described as being performed by a user device of one or more processors (e.g., the game system110).

At block502, the user device executes a game. As described herein, the user device may represent a game console, laptop, handheld device, and so on. Thus, the user device may execute the electronic game as stored, or made accessible to, the user device. The user device may also represent a laptop presenting visual information which is rendered by an outside system (e.g., a cloud-based system). In this example, the user device may indicate that the electronic game is to be played. The outside system may then initiate the game, and provide visual information for presentation via the user device.

At block504, the user device provides indication of game initiation. As described in block504, the user device may provide information identifying that the game has started to a particular system (e.g., the system100).

At block506, the user device provides monitored game state information. As described above, the user device may provide information identifying musical cues which are to be output during game play.

AT block508, the user device receives personalized music. The particular system (e.g., the system100) may generate personalized music, and provide it for output by the user device. In some embodiments, the user device may generate the personalized music. In some embodiments, the user device may store (e.g., cache) previously received personalized music. In these embodiments, the user device may identify whether a user's listening history has changed greater than a threshold since the personalized music was received. For example, the user device may access one or more music streaming platforms. As another example, the user device may cause the particular system to access the music streaming platforms. Upon access, the current listening history may be compared to prior listening history. If the listening history is different according to one or more measures, the system may generate new personalized music. If the listening history is not different, the user device may utilize the stored personalized music.

Overview of Computing Device

FIG. 6illustrates an embodiment of computing device610according to the present disclosure. Other variations of the computing device610may be substituted for the examples explicitly presented herein, such as removing or adding components to the computing device610. The computing device610may include a game device, a smart phone, a tablet, a personal computer, a laptop, a smart television, a car console display, a server, and the like. The computing device610may also be distributed across multiple geographical locations. For example, the computing device610may be a cluster of cloud-based servers.

As shown, the computing device610includes a processing unit620that interacts with other components of the computing device610and also external components to computing device610. A game media reader622is included that communicates with game media612. The game media reader622may be an optical disc reader capable of reading optical discs, such as CD-ROMs or DVDs, or any other type of reader that can receive and read data from game media612. One or more of the computing devices may be used to implement one or more of the systems disclosed herein.

Computing device610may include a separate graphics processor624. In some cases, the graphics processor624may be built into the processing unit620. In some such cases, the graphics processor624may share Random Access Memory (RAM) with the processing unit620. Alternatively or additionally, the computing device610may include a discrete graphics processor624that is separate from the processing unit620. In some such cases, the graphics processor624may have separate RAM from the processing unit620. Computing device610might be a handheld game application device, a dedicated game console computing system, a general-purpose laptop or desktop computer, a smart phone, a tablet, a car console, or other suitable system.

Computing device610also includes various components for enabling input/output, such as an I/O632, a user I/O634, a display I/O636, and a network I/O638. I/O632interacts with storage element640and, through a device642, removable storage media644in order to provide storage for computing device610. Processing unit620can communicate through I/O632to store data, such as game state data and any shared data files. In addition to storage640and removable storage media644, computing device610is also shown including ROM (Read-Only Memory)646and RAM648. RAM648may be used for data that is accessed frequently, such as when a video game is being played.

User I/O634is used to send and receive commands between processing unit620and user devices, such as game controllers. In some embodiments, the user I/O634can include a touchscreen input. The touchscreen can be capacitive touchscreen, a resistive touchscreen, or other type of touchscreen technology that is configured to receive user input through tactile inputs from the player. Display I/O636provides input/output functions that are used to display images from the game being played. Network I/O638is used for input/output functions for a network. Network I/O638may be used during execution of a game, such as when a game is being played online or being accessed online, application of fraud detection, and/or generation of a fraud detection model.

Display output signals produced by display I/O636comprise signals for displaying visual content produced by computing device610on a display device, such as graphics, user interfaces, video, and/or other visual content. Computing device610may comprise one or more integrated displays configured to receive display output signals produced by display I/O636. According to some embodiments, display output signals produced by display I/O636may also be output to one or more display devices external to computing device610.

The computing device610can also include other features that may be used with a video game, such as a clock650, flash memory652, and other components. An audio/video player656might also be used to play a video sequence, such as a movie. It should be understood that other components may be provided in computing device610and that a person skilled in the art will appreciate other variations of computing device610.

Program code can be stored in ROM646, RAM648or storage640(which might comprise a hard disk, other magnetic storage, optical storage, other non-volatile storage or a combination or variation of these). Part of the program code can be stored in ROM that is programmable (ROM, PROM, EPROM, EEPROM, and so forth), and part of the program code can be stored in storage640, and/or on removable media such as game media612(which can be a CD-ROM, cartridge, memory chip or the like, or obtained over a network or other electronic channel as needed). In general, program code can be found embodied in a tangible non-transitory signal-bearing medium.

Random access memory (RAM)648(and possibly other storage) is usable to store variables and other game and processor data as needed. RAM648is used and holds data that is generated during the execution of an application and portions thereof might also be reserved for frame buffers, application state information, and/or other data needed or usable for interpreting user input and generating display outputs. Generally, RAM648is volatile storage and data stored within RAM648may be lost when the computing device610is turned off or loses power.

As computing device610reads game media612and provides an application, information may be read from game media612and stored in a memory device, such as RAM648. Additionally, data from storage640, ROM646, servers accessed via a network (not shown), or removable storage media644may be read and loaded into RAM648. Although data is described as being found in RAM648, it will be understood that data does not have to be stored in RAM648and may be stored in other memory accessible to processing unit620or distributed among several media, such as game media612and storage640.