Association via voice

A server has a pool data store that stores ambient sound recordings for matching. A match engine finds matches between ambient sound recordings from devices in the pool data store. The matching ambient sound recordings and their respective devices are then analyzed to determine which device is a source device that provides credentials and which device is a target device that receives credentials. The server then obtains or generates credentials associated with the source device and provides the credentials to the target device. The target device accesses content or services of an account using the credentials.

TECHNICAL FIELD

The present disclosure relates to technical solutions for associating a target device with credentials or an account of a source device.

BACKGROUND

The process of associating a software or hardware product with an account of a user often includes the user providing a username and password using a tactile or virtual keyboard. However, there are technical drawbacks and challenges with this process: not all devices have a keyboard (e.g., voice-first devices), not all keyboards are available for use (e.g., for attention or accessibility reasons), and username-password combinations are not always ideal for authentication, among others. Login difficulties are compounded by frequent switching of accounts, such as among family members sharing a device or when providing guest access.

While advancements have been made in transferring login credentials from one device to another, technical challenges still remain. Some techniques include pairing an already logged-in device with a non-logged-in device and transferring the credentials from the logged-in device to the non-logged-in device via an auxiliary channel, such as using HTTP on a local network (e.g., using zero configuration networking). Other techniques include device pairing over BLUETOOTH and transferring a login token over the connection. Typically, these techniques require an extra communication channel and, depending on the communication technique, consume significant energy resources for both devices. Such techniques further raise concerns when used for guest mode access. For example, where a connection is made via a same WI-FI network, sharing access credentials for the network can be undesirable for security or privacy purposes.

Still other techniques use audio to transmit data, such as using broadband phase difference encoding. Such techniques can synthesize audio directly or by masking with existing audio. For example, Multimedia Information Hiding Technologies for Controlling Data (Kazuhiro Kondo, 2013) describes the use of acoustic orthogonal frequency-division multiplexing at page 94. Some techniques include transferring small amount of data encoded in an audio signal, decode the data, and then use the decoded data to look up large pieces of data available at a backend system. Fast Fourier transform or similar transforms are usable to extract the sequence of tones on the client side and transmit that to the backend to retrieve the larger information. Using audio to transmit data often results in robotic-sounding audio lacking aural aesthetics.

US 2017/0019394 describes playing media items from a guest's media account on a host's media player. A guest device discovers a host media player over a wireless local area network. A token can is provided from the host media player to the guest device. The token is encoded into an audio file, which is played by the host media player. The audio is captured by the guest device and decoded into the token, which is transmitted to a server for verification.

SUMMARY

The present disclosure provides methods, apparatuses, and computer readable products for using audio to associate a target device with credentials of a source device.

In an example, there is a method, including: receiving, from a source device associated with a source device account a source device ambient sound recording; receiving, from a target device, a target device ambient sound recording; matching the source device ambient sound recording and the target device ambient sound recording together based on one or more similarities between the source device ambient sound recording and the target device ambient sound recording; and associating the target device with the source device account responsive to the matching.

In an example, wherein associating the target device with the source device account includes providing credentials to the target device. In an example, the method further includes providing the source device ambient sound recording for matching responsive to determining that the source device ambient sound recording includes a log-in utterance; and providing the target device ambient sound recording for matching responsive to determining that the target device ambient sound recording includes a log-in utterance. In an example, the method further includes obtaining source device signals from the source device; and wherein matching the source device ambient sound recording and the target device ambient sound recording together is further based on the target device signals. In an example, the method further includes obtaining target device signals from the target device, wherein matching the source device ambient sound recording and the target device ambient sound recording together is further based on the target device signals. In an example, the method further includes after the matching and prior to the associating, determining to associate the target device with the source device account based on the source device signals and the target device signals. In an example, determining to associate the target device with the source device account is further based on a hierarchy of device types, the device type of the source device, and the device type of the target device. In an example, the method further includes prior to associating the target device with the source device account, operating the target device in a primary mode associated with a target device account. In an example, associating the target device with the source device account includes operating the target device in a guest mode associated with the source device account. In an example, the target device automatically reverts to the primary mode associated with the target device account after an occurrence of an event. In an example, the event is a device power event. In an example, the method further includes storing the source device ambient sound recording in a pool data store; storing the target device ambient sound recording in the pool data store; and determining matches between recordings in the pool data store. In an example, determining matches between recordings in the pool data store includes the matching of the source device ambient sound recording and the target device ambient sound recording together. In an example, the method further includes removing the source device ambient sound recording and the target device ambient sound recording from the pool data store after the matching of the source device ambient sound recording and the target device ambient sound recording together.

In an example, there is a system that includes one or more processing devices; and a memory device coupled to the one or more processing devices and comprising instructions thereon that, when executed by the one or more processing devices, cause the one or more processing devices to perform any of the previously described methods.

DETAILED DESCRIPTION

The example embodiments presented herein are directed to systems, methods, and non-transitory computer-readable medium products for associating a target device with credentials of a source device based on matching ambient sound recordings from both devices. This is for convenience only, and is not intended to limit the application of the present invention. After reading the following description, it will be apparent to one skilled in the relevant art how to implement the following disclosure in alternative embodiments.

As used herein, “credentials” are data usable for authentication or to gain access to particular content or services. In many examples herein, credentials are associated with a particular account (e.g., credentials usable to log into or otherwise gain access to services associated with an account). Credentials are usable to perform an authentication or association process with an application or server, such as using OAUTH 2.0, OPENID CONNECT (maintained by the OPENID FOUNDATION), SAML (maintained by OASIS of Burlington, Mass.), or other standards, protocols, or techniques. Other uses for and examples of credentials will be apparent to one of skill in the art. In some examples, the credentials are representative of a username and password for an account of a service. In some instances, the credentials are use-limited or time-limited, such as one-session-use credentials or credentials valid for limited amount of time.

As used herein, “target device” refers to a computing device with which credentials (e.g., credentials of an account) are to be associated and “source device” refers to a computing device with which the credentials are currently associated. This distinction is mutable. A device can be a target device in a first process and the same device can be a source device in a second process.

In an example situation, the source device is a smartphone having an audio streaming application. The audio streaming application is associated with a source device account. The audio streaming application uses the source device account to access audio streaming services associated with the source device account (e.g., accessing an audio library associated with the source device account). The user of the source device wants to access the audio streaming service using the source device account on a nearby target device. In the example, the target device is a smart speaker system. Both the source device and the target device receive an utterance, “ahoy computer, log me in with voice”, spoken by the user. Both devices independently wake up from a sleep mode in response to detecting the activation trigger “ahoy computer”. Both devices independently begin recording ambient sound. By recording ambient sound, both devices independently create an ambient sound recording, which includes the “log me in with voice” portion of the utterance. Both devices independently send the ambient sound recording to the remote server. The remote server receives the ambient sound recordings. The remote server uses a speech analysis engine to determine an action to perform based on the ambient sound recordings. The speech analysis engine determines that ambient sound recordings include a phrase such as “log me in with voice” and begin a log-in-with-voice process, which causes the ambient sound recordings to be sent to a match engine. The match engine stores the ambient sound recordings in a pool data store. The match engine determines matching recordings within the pool data store. The match engine matches the source device ambient sound recording (e.g., the recording from the smartphone) with the target device ambient sound recording (e.g., the recording from the smart speaker system) based on audio characteristics and additional signals associated with the respective ambient sound recordings. At this point, the match engine identified the devices that match but not which of the devices is the source device and which device is the target device. The match engine then determines which of the matched devices is the source device and which is the target device. Based on this determination, the match engine identifies from which device to obtain credentials (e.g., the source device) and to which device the credentials will be provided (e.g., the target device). The match engine determines that the source device (e.g., the smartphone) is the source of the account credentials and that the target device (e.g., the smart speaker system) is the device that is the target recipient of the credentials. This determination is based on a hierarchy of device types. The match engine determines the device type of the source device is a smartphone and the device type of the target device is a smart speaker system. The match engine determines source device is the source device and the target device is the target device because the smart speaker systems are higher than a smartphone in the hierarchy of device types. With the source device and target device identified, the server provides credentials of the source device account to the target device. The target device then uses the source device account to access the streaming service and obtain content based on the source device account.

Among other advantages, embodiments disclosed herein allow for the association of an account with a nearby device with the user's voice. In contrast to using audio steganography to transmit login tokens, this association-with-voice process uses the user's own voice, which provides several advantages. For example, the audio content used in the process need not necessarily be specially formatted (e.g., formatted to encode data) for the purpose of logging in to the account. This increases security by obviating the need to audibly broadcast or receive login credentials. It also improves the user interface provided by the device because the device can initiate the process by merely receiving an utterance from the user, which reduces an amount of user input that the device needs to receive (and thus that the user needs to provide). Further, the association-with-voice process provides an intuitive pattern for a voice-based user interface: the device does what the user tells it to do. This not only improves the ease of use of the system but also reduces computing resources that would be used in the process. For instance, the device need not necessarily be specially programmed or include specialized hardware for encoding, decoding, or receiving audio steganography data. In this manner, computing resources are conserved. Other advantages will be apparent to one skilled in the art on consideration of the examples described herein.

System for Using Ambient Sound Recordings to Associate a Target Device with an Account of a Source Device

FIG. 1illustrates an example system100including a source device110, a target device150, and a speech analysis server750. The source device110and the target device150provide an utterance-based user interface. The source device110is a computing device and uses the credentials111to access services or content associated with a source device account114. The target device150is a computing device with which the user wants to associate the source device account114. In many examples, the target device150already includes or is associated with a target device account154via its own set of credentials. The user of the source device110is, for example, a guest in the home of the person that owns the target device150. In such instances, the target device150is likely already associated an account (the target device account154) of its owner. Nonetheless, the guest may want to control playback of the target device150by associating the target device150with his or her account (e.g., the source device account114). In the illustrated example, both the source device110and the target device150receive a same utterance202which causes the target device150to associate with the source device account114of the source device110.

In an example, the speech analysis server750is associated with a streaming audio service and the credentials111are associated with the source device account114. In an example, the source device110uses the credentials111to access one or more streaming audio services associated with the source device account114. The services can include, for example, audio playback services, library management services (e.g., a song library, a playlist library, or an album library, among others), media content purchasing services, taste profile services (e.g. services relating to the likes and dislikes of the user), content sharing services, and account management services, among others.

The utterance202is voice input to the utterance-based user interfaces provided by the source device110and the target device150. In the example system100, the utterance202includes the example phrase “ahoy computer, log me in with voice”. Both devices110,150separately listen to ambient sounds using a microphone or another input source. In turn, the devices110,150process the ambient sounds, to detect and further process utterances spoken in the vicinity of the devices110,150. In the illustrated example, both the source device110and the target device150receive and process the utterance202.

As illustrated the source device110includes a source device activation trigger engine116, and the target device150includes a target device activation trigger engine156. The source device activation trigger engine116detects an activation trigger spoken in the vicinity of the source device110. Responsive to detecting the activation trigger, the source device110creates a source device ambient sound recording118, which is sent from the source device110to the speech analysis engine190for processing. Separately, the target device activation trigger engine156detects an activation trigger spoken in the vicinity of the target device150. Responsive to detecting the activation trigger, the target device150creates and sends a target device ambient sound recording158to the speech analysis engine190for processing. In some examples, creating the source device ambient sound recording118and the target device ambient sound recording158includes creating an audio clip of the ambient sound or establishing a streaming connection with the speech analysis server750. For instance, the target device150records the ambient sounds to a buffer at the target device150and the contents of the buffer are streamed to the speech analysis server750, thereby providing the target device ambient sound recording158to the speech analysis server750.

In the illustrated example, both of the activation trigger engines116,156are trained to detect the activation trigger “ahoy computer”, though other activation triggers can be used. Because the utterance202is detected by both activation trigger engines116,156, both devices110,150separately provide respective ambient sound recordings118,158to the speech analysis engine190for further processing.

The speech analysis engine190processes recordings to understand their contents and select an action to perform based thereon. In an example, the processing involves speech-to-text processing on recordings and performing natural language processing on the resulting text. The speech analysis engine190outputs an intent based on the analysis of the natural language processing. The intent is usable by the speech analysis engine190or another component of the speech analysis server750to select and perform an action associated with the recording.

In the illustrated example, the speech analysis engine190performs analysis on the source device ambient sound recording118and the target device ambient sound recording158. The speech analysis engine190determines from the utterance202contained in the ambient sound recordings118,158to cause a log-in-with-voice command to be executed. In executing the command, the speech analysis engine190provides the ambient sound recordings118,158to the match engine191.

The match engine191determines matches between two or more recordings. The match engine191determines that there is a match if a similarity of the recordings passes a predetermined threshold. In many examples, the similarity is based on whether the recordings are of a same place at a same time. Determining whether the recordings are of a same place at a same time can be based on a variety of factors, including whether the recordings capture a same event (e.g., a user speaking an utterance, same music playing, same car engine noise, etc.). The match engine191uses a variety of data to make its determinations, such as the sonic qualities of the recordings, as well as other signals about the recordings (e.g., locations where the recordings are taken, the time at which the recordings were received, etc.).

The match engine191uses a pool data store196to store recordings and associated data for matching. The match engine191curates the pool data store to remove old entries. When a new recording is provided to the match engine191, the match engine191compares the new recording to those recordings in the pool data store196to determine if there is a match. If there is not a match, then the new recording is added to the pool data store196. If there is a match, then the one or more matching recordings in the pool are removed, and the match engine191provides an output indicating which recordings match. For instance removing the source device ambient sound recording118and the target device ambient sound recording158from the pool data store196after the matching of the source device ambient sound recording118and the target device ambient sound recording158together. In an example, the output includes identifiers (e.g., device identifiers) of the devices that matched. In an example, the match engine191calls an application programming interface (API) of the access management engine192with the matching devices as parameters.

The data in the pool data store196can be handled according to a defined user privacy policy. In some embodiments, data elsewhere in the system100can be handled according to a defined user privacy policy. Generally, data can be used to the extent allowed by the user. In examples, the system100provides customizable privacy parameters. For instance, the system100maintains one or more privacy parameters directing components of the system100as to what data may be stored and how the data is used. One or more parameters are user-customizable, such as through a voice command (e.g., “ahoy computer, don't store what I say” or “ahoy computer, I like my privacy!”). In an example, the system100disallows authentication based on ambient sound recordings from devices unless allowed by the privacy parameters. In some examples, the storage of data is based on a location of the source device110or the target device150. For instance, the system100receives the location of the devices110,150and determines privacy settings based thereon. In addition, the data contained in the pool data store196is stored in accordance with applicable regulations. Data can also be stored and used in association with a defined security policy. For example, data can be encrypted at rest and in motion. For instance, ambient sound recordings are encrypted during transmission and encrypted while stored. In an example, the ambient sound recordings are authenticated in a manner other than using user credentials. In an example, transmission channels (e.g., for the audio input data) are authenticated in a manner other than using user credentials, such as using data regarding the device100itself.

In the illustrated example, the match engine191receives the source device ambient sound recording118. The match engine191also receives the target device ambient sound recording158. In an example, the match engine191receives the source device ambient sound recording118first.

The match engine191compares the source device ambient sound recording118against any recordings in the pool data store196. The match engine191does not find a match. In response to not identifying a match between the source device ambient sound recording118and any recordings in the pool data store196, the match engine191adds the source device ambient sound recording118to the pool data store196. Next, the match engine191receives the target device ambient sound recording158. The match engine191compares the target device ambient sound recording158to the one or more recordings in the pool data store196. The match engine191identifies a match between the source device ambient sound recording118and the target device ambient sound recording158. Responsive to finding the match, the match engine191provides an output to the access management engine192identifying a match between the source device ambient sound recording118and the target device ambient sound recording158. The output indicates that the source device110and the target device150match as part of a log-in-with-voice function.

The access management engine192controls the credentials and access using the accounts. The access management engine192uses the matched devices output from the match engine191and provides credentials122to one of the devices.

AlthoughFIG. 1labels a particular device as a “source device” and a particular device as a “target device”, in many examples the access management engine192does not yet have an identification of which of the devices is the source device and which is the target device. So the access management engine192or the match engine191determines which of the devices in the output is the source device110having the source device account114to be associated and which device is the target device150to receive the source device account114. Example techniques for making the determination involve the use of hierarchies, relative volume, activity logs, and asking the user (which are each described further in relation to operation450inFIG. 4), among others.

In the illustrated example, the access management engine192determines that the source device is the source device because its device type is a smartphone, which is lower in the hierarchy than the target device150, which is a smart speaker system. Based on the identification of the source device110as the source device and the target device150as the target device, the access management engine192provides credentials122to the target device150. In some examples, the access management engine192determines an account associated with the device identifier associated with the source device110. The access management engine192then obtains or generates the credentials122based thereon.

In some examples, the credentials122are identical to the credentials111. In other examples, the credentials122are similar to the credentials111but have one or more different characteristics. Both credentials111,122are associated with a same account, but the credentials122are flagged to distinguish the credentials122from the credentials111, for instance. Flagging the credentials122provides benefits to revoking credentials122at a later time (e.g., manually or automatically when the user leaves the vicinity of the target device150) by allowing the credentials to identified. Further, the user may not want the target device150(or other users thereof) to have full access to his or her account for privacy, security, or account management purposes.

The target device150uses the credentials122to access services or content provided by the speech analysis server750. In an example, the source device110was playing a media content item and playback of the media content item is transferred from the source device110to the target device150automatically in response to the target device150accessing the speech analysis server750using the credentials122. The transfer of playback provides an improved user experience by serving as a confirmation that the association completed successfully.

The source device110and target device150are illustrated as being a smartphone and a media streaming appliance, respectively, but the devices110,150can take a variety of forms, including but not limited to laptop computers, desktop computers, media streaming appliances, smart speakers, Internet-of-things devices, and vehicle head units, among others.

In the illustrated example, the source device ambient sound recording118and the target device ambient sound recording158are based on the utterance202and provided to the speech analysis server750as part of a voice-based user interface, which is described in more detail inFIGS. 2 and 3.

Voice-Based User Interface

Software or hardware products that provide a voice-based user interface are configured to take action in response to utterances.FIGS. 2 and 3describe an example voice-based user interface. For concision, the voice-based user interface is described in relation to the source device110and the source device activation trigger engine116. The target device150and the target device activation trigger engine156has at least similar functionality.

FIG. 2illustrates the example utterance202stored in a data store200(e.g., temporary memory). The utterance202includes an activation trigger portion210, a command portion220, and a parameter portion230. In the illustrated example, the activation trigger portion210corresponds the phrase “ahoy computer”, the command portion220corresponds to the phrase “log me in”, and the parameter portion230corresponds to the phrase “with voice”.

The source device110receives the utterance202. Responsive to the source device activation trigger engine116detecting the activation trigger portion210, the source device activation trigger engine116provides at least a portion of the utterance202to the speech analysis engine190. The speech analysis engine process some or all of the utterance202and identifies one or more of the various portions210,220,230.

Identifying the portions of the utterance202can be performed using a variety of techniques. In one example, segmenting the utterance202into constituent portions is based on the text content of the utterance (e.g., splitting on words or natural-language processing of the text content of the utterance). In another example, segmenting is based on how the utterance202was uttered (e.g., by splitting the utterance based on pauses in the utterance), among other techniques.

Where the speech analysis engine190determines that the utterance202includes the command portion220, the speech analysis engine190determines a command associated with the command portion220(e.g., using natural language processing, look-up tables, or other techniques). Where the utterance202includes a parameter portion230, the speech analysis engine190determines the parameters described by the parameter portion230. The determined command is then executed using the determined parameters. Based on the output of the executed command, the device generates a response or confirmation for the user or takes another action.

Voice-based user interfaces use one or more activation triggers to activate one or more capabilities of an associated device. In many instances, an activation trigger is a predetermined word or phrase spoken by the user to activate a voice-based user interface. In some instances, the activation trigger is formed or detected as particular sound data above or below the threshold of human hearing. In other instances, the activation trigger is an electronic signal received by a device in response to an event, such as actuation of a button or other user input mechanism. In still other instances, the event includes a signal received from another device.

The activation trigger portion210serves several purposes, including overcoming challenges in voice-based user interfaces. Voice-based user interfaces generally operate by obtaining audio input from an ambient environment and appropriately responding to detected speech. As such, voice-based user interfaces receive all utterance in the ambient environment, including utterances not directed to the user interface. The detection of utterances not directed to the interface can present drawbacks. The use of an activation trigger provides several technical advantages, including conserving resources and protecting user privacy.

FIG. 3illustrates an example flow diagram showing a process300for providing a voice-based user interface using the source device activation trigger engine116and the speech analysis engine190. The process300begins at operation310, which involves the source device activation trigger engine116receiving audio input. For instance, the source device activation trigger engine116receives audio input from a microphone of the source device110or a connection to an external audio source (e.g., a BLUETOOTH connection to a device that has a microphone). In some instances, the audio input is streaming audio input. In some instances, the audio input includes discrete slices of audio samples or chunked input. As will be understood, various portions of process300can occur simultaneously, and various buffers or caches can be used to facilitate the comparison of data over time.

In operation312, the source device activation trigger engine116compares the audio input with one or more patterns to determine if the audio input includes an activation trigger. The obtained audio input can include utterances that are directed to the source device110, utterances directed to another target (e.g., another person in the room), ambient room sound, or other sounds. A variety of different techniques can be used to determine if the audio input includes an activation trigger.

In one example, features are extracted from the audio input and provided to a machine learning framework configured to determine the likelihood that the extracted features correspond to an activation trigger. Various techniques can be used for feature extraction. In some examples, mel-frequency cepstrum representation of audio data is used to generate cepstral coefficients features for processing. In other examples, mel-frequency cepstral coefficients, log filter banks, cepstral analysis, or other techniques are used. In other examples, feature extraction is performed using invertible feature extraction (e.g., using Fourier transforms). For instance, instead of relying on features or parameters only from a time domain of a signal, the signal is transformed into a frequency domain using Fourier transformation. Parameters are then extracted from the frequency domain.

Once extracted, the extracted features are analyzed to determine a similarity with a pattern associated with an activation trigger. This analysis can be performed through a variety of techniques including, but not limited to, hidden Markov models, neural networks, and other techniques. In many instances, the source device activation trigger engine116includes a pre-trained or pre-generated pattern against which to compare the features.

At operation313, if the received audio input matches the pattern, then the flow of the process moves to operation314. If the received audio input does not match the pattern, the flow of the process300moves back to operation310. In an example, the audio input matches the pattern if a difference between the features of the audio input and the pattern satisfies a threshold. The process of operations310,312, and313can be referred to as monitoring the output of an audio input device for an activation trigger pattern. Once a match is detected, the flow simultaneously moves to operation314and the monitoring continues.

At operation314, the source device activation trigger engine116provides causes the speech analysis engine190to be activated. The source device activation trigger engine116can activate the speech analysis engine190in a variety of ways, such as sending a signal to the speech analysis engine190, executing a command, accessing an application programming interface associated with the speech analysis engine190, populating a queue with data, or piping output of a recording device or process to the speech analysis engine190, among other techniques.

In operation324, the speech analysis engine190obtains the audio input data. In some examples, this includes some or all of the audio input data received in operation310. For instance, the source device activation trigger engine116buffers the audio input data obtained and, when the audio input data is determined to match an activation trigger pattern, the contents of the buffer are provided to the speech analysis engine190for analysis. In such instances, the audio input obtained in operation324includes the activation trigger portion (e.g., activation trigger portion210as in utterance202ofFIG. 2) in addition to other portions that the speech analysis engine processes (e.g., the command portion220and the parameter portion230as in utterance202ofFIG. 2). Once the audio input data is provided to the speech analysis engine190or a threshold amount of time passes, the buffer is cleared. In other instances, the source device activation trigger engine116provides or redirects audio obtained after detection of the activation trigger to the speech analysis engine190. In such instances, the speech analysis engine190obtains audio input data for a portion of an utterance following the activation trigger portion but not the activation trigger portion itself. In some instances, the audio input is “raw” audio data. In other instances, the audio input includes features extracted from the audio data.

After receiving at least a portion of the audio input, the speech analysis engine190performs operation326, which involves processing the audio input data. Processing audio input takes a variety of different forms. In many examples, processing audio input involves performing speech-to-text transcription of the audio input. In other examples, processing audio input involves determining an intent associated with the utterance. For instance, if the speech analysis engine190were processing the utterance202as audio input, the speech analysis engine190performs speech-to-text transcription on the utterance202to determine that the audio input data corresponds to the text “ahoy computer, log me in with voice”.

At operation328, the speech analysis engine190provides output based on the processing in operation326. As will be understood, the output is based on the type of processing performed. In some instances, the output or audio data is provided to another analysis or processing engine for further processing, such as text analysis, natural language processing, emotion detection, or other processing. In other instances, that additional processing is performed by the speech analysis engine190in operation326, and the results of that additional processing can be provided.

In some examples, the speech analysis engine190determines that the user's intent is to perform a log-in-with-voice process. For example, the speech analysis engine190receives a log-in utterance indicating that a log-in-with-voice process should be executed. For instance, responsive to a log-in utterance corresponding to the text “ahoy computer, log me in with voice”, the process300provides an output indicating that a login with voice command should be executed. In an example, the output involves providing the utterance202(or a representation thereof) to the match engine191for further processing. For instance, providing the source device ambient sound recording118for matching responsive to determining that the source device ambient sound recording118includes a log-in utterance. In another instance, providing the target device ambient sound recording158for matching responsive to determining that the target device ambient sound recording158includes a log-in utterance.

An example process for processing with the match engine191is shown and described inFIG. 4.

Process for Credential or Account Management Using the Match Engine

FIG. 4illustrates a process400for performing credential or account management based on two or more recordings.

At operation410, an ambient sound recording412is received. For instance, receiving, from the source device110associated with the source device account114, the source device ambient sound recording118. In another instance, receiving, from the target device150, the target device ambient sound recording158. The ambient sound recording412is received in any of a variety of ways. In some instances, an application programming interface of the match engine191is called that specifies the ambient sound recording412as a parameter. In some examples, the parameter is an audio file of the ambient sound recording. In other examples, the parameter is a data structure that represents the ambient sound recording (e.g., a data structure specifying extracted features of the ambient sound recording412, such as an audio fingerprint). In other examples, the parameter is a pointer or other link to a location at which the audio file or representation is stored. In some examples, the match engine191includes a queue into which the ambient sound recording412is placed prior to processing.

At operation420, additional signals422associated with the ambient sound recording412are received. The signals422include, for example, signals obtained from a device that produced the ambient sound recording412. In an example, the signals422are source device signals424of the source device110. For instance, the operation420includes obtaining source device signals424from the source device110. In an example, the signals422are target device signals426of the target device150. For instance, the operation420includes obtaining target device signals426from the target device150. The signals422are additional data that can be used to determine whether a match exists for the ambient sound recording412. Examples of signals422include timing information regarding the recording412, such as a time at which a device began or ended recording the ambient sound recording412, a time at which the ambient sound recording412was sent, or a time at which the ambient sound recording412was received, among other times.

Examples of signals422also include a location of a device that made the ambient sound recording412. Location takes various forms and levels of granularity. In some instances, the location includes GPS coordinates, an address, a street name, a neighborhood name, a city name, a county name, a state name, a province name, a region name (e.g., U.S. East), a country name, a continent name, or others. In some instances, the location includes an indication of an accuracy or an estimate of the location (e.g., accurate within 500 feet). In some instances, the location can further describe an altitude of the device that made the ambient sound recording412, or one or more WI-FI SSIDs near the device that made the ambient sound recording412. Example signals422further include a device type of the device that made the ambient sound recording412and a device identifier of the device that made the ambient sound recording412, among other data. In some instances, the information is a last-known value, such as a last-known location.

In some examples, the signals422are provided with or inferred from a message that provided the ambient sound recording412to the speech analysis server750(e.g., by analyzing metadata of the message or a traceroute of the message). In some examples, the signals422are provided as a parameter to the API call that specified the ambient sound recording412. In examples, a component of the server750requests additional data from the device that provided the ambient sound recording412. For instance, responsive to the match engine191receiving the ambient sound recording412, the match engine191requests additional data (e.g., one or more of the signals422discussed above) from the device that recorded the ambient sound recording412.

In some instances, the one or more signals422include additional signals contained within the ambient sound recording412(e.g., a watermark, identifications of other sounds within the recording, and sonic qualities of the ambient sound recording412such as echoes, among others), an indication whether the device that produced the ambient sound recording412is moving, or an indication of a network path the target device used to send the ambient sound recording412, among others.

In an example, the signals422are stored in a data structure in a data store. The data structure can define a variety of fields for storing data, such as a Rec_Time field (e.g., a recording time field for storing a time associated with the ambient sound recording412, such as a time that the ambient sound recording412was made), a location field (e.g., for storing a location where the ambient sound recording412was made), and an altitude field (e.g., for storing the altitude at which the ambient sound recording412was made), among other fields.

At operation430, the ambient sound recording412and the signals422are stored in the pool data store196. For instance, storing the source device ambient sound recording118in a pool data store196. For instance, storing the target device ambient sound recording158in the pool data store196. In some examples, the ambient sound recording412is stored as an audio file. In other examples, the ambient sound recording412is stored as audio features extracted from the audio of the ambient sound recording412, such as an audio fingerprint. In an example, the match engine191(or another component) performs audio fingerprinting on the ambient sound recording412to obtain an audio fingerprint. The match engine191then stores the audio fingerprint in the pool data store196for later comparison.

Various techniques can be used for feature extraction and fingerprinting the ambient sound recording412. In some examples, mel-frequency cepstrum representation of audio data is used to generate cepstral coefficients features for processing. In other examples, mel-frequency cepstral coefficients, log filter banks, cepstral analysis, or other techniques are used. In other examples, feature extraction is performed using invertible feature extraction (e.g., using Fourier transforms). For instance, instead of relying on features or parameters only from a time domain of a signal, the signal is transformed into a frequency domain using Fourier transformation. Parameters are then extracted from the frequency domain.

In an example, the pool data store196includes a table that describes information about the one or more ambient sound recordings in the pool by storing one or more records. In an example, each record in the table includes data associated with a respective ambient sound recording412and signals422associated therewith. For instance, the table can include a first record describing the source device ambient sound recording118(e.g., by storing an audio fingerprint thereof and the source device signals424), and a second record describing the target device ambient sound recording158(e.g., by storing an audio fingerprint thereof and the target device signals426). Each record of the table includes data associated with one or more fields of the table, such as an R_ID field (e.g., a recording identifier field for storing an identifier of the ambient sound recording412associated with the record), an R_Time field (e.g., a recording time field for storing data associated with a time at which the ambient sound recording412associated with the record was created), a D_ID field (e.g., a device identifier field for storing an identifier of the device that created the ambient sound recording412associated with the record), a D_Type field (e.g., a device type field for storing a type of device that created the ambient sound recording412associated with the record), a location field (e.g., for storing a location of the device that created the ambient sound recording412associated with the record), and an A_ID field (e.g., an account identifier field for storing an identifier of the account associated with the ambient sound recording412associated with the record), among other fields (e.g., an audio fingerprint field sorting an audio fingerprint of the ambient sound recording412associated with the record).

At operation440, matches between ambient sound recordings in the pool data store196are determined. For example, matching the source device ambient sound recording118and the target device ambient sound recording158together. For example, matching based on one or more similarities between the source device ambient sound recording110and the target device ambient sound recording158. In many examples, the matches are determined using the audio characteristics of the ambient sound recordings in the pool data store196. The matches are also determined by the match engine191using the signals of the ambient sound recordings as criteria for determining a match. For example, as part of the match determination process, the match engine191further estimates or determines a start time of the ambient sound recording412. In such instances, the match engine191compares the start time of the ambient sound recording412with other ambient sound recordings in the pool data store196to identify ambient sound recordings with sufficiently similar start times. If the start times are sufficiently close (e.g., within a small enough amount of time still able to account for variations in activation trigger processing time, system clock inaccuracy, and differences in when the time was recorded), then such a determination weighs in favor of the ambient sound recordings matching. In some examples, audio synchronization tools (e.g., PLURALEYES by RED GIANT LLC) can be used to match ambient sound recordings in the pool data store196. For instance, two or more ambient sound recordings are provided as input to the audio synchronization tool. If the audio synchronization tool is able to synchronize a pair of the ambient sound recordings, then the ambient sound recordings of the synchronized pair are matched.

In still further examples, the match engine191uses location as a determining factor. For instance, the match engine191receives the location of the device that created the ambient sound recording412. The location can have various levels of specificity including but not limited to region, continent, country, state, province, city, county, ZIP Code, GPS location, among other location information. The match engine191compares the locations of the ambient sound recordings in the pool data store196and uses a similarity in a location as an indication that the ambient sound recordings match. The match engine191compares the altitude or elevation information of the ambient sound recordings in the pool data store196and uses a similarity as an indication that the ambient sound recordings match.

The pool data store196can be queried based on the data contained therein, such as an audio fingerprint or various signals. In instances where the pool data store196includes a database of the relevant information, the database is queried using the audio fingerprint and one or more of the signals422. In other instances, the match engine191iterates through one or more of the recordings and calculates a likelihood that the recording matches another recording in the pool data store196.

The operation can include determining a likelihood that two ambient sound recordings match and if the likelihood satisfies a threshold, then the match between the ambient sound recordings is determined. This likelihood is determinable in a variety of different ways including using the ambient sound recording412and the signals422. The various data stored in the pool data store196can have various weights usable in calculating a likelihood of a match. For instance, similarities in audio fingerprints between ambient sound recordings is heavily weighted in the calculation such that recordings that do not have similar audio fingerprints have a low likelihood of matching.

Aspects such as ambient sound recording start time may be weighted heavily as well, but there may be some variance allowed to account for various factors in discrepancies in time (e.g., due to variations in activation trigger processing time, due to system clocks being off, or due to differences in when the time was recorded). In further examples, the lack of data may be dispositive in some instances and may be highly relevant in others. For instance, a device that produced a recording may not include a GPS device or be otherwise unable to obtain an accurate location. The lack of location for a recording in a pool data store196need not disqualify the ambient sound recording as a potential candidate for a match. In many instances, both filtering and analysis are used. For example the ambient sound recording in the pool data store196are filtered according to relative similarity of audio fingerprints, and then the likelihood is determined further based on additional signals. In an example, a machine learning framework is used to determine the likelihood (e.g., a neural network trained to determine likelihood that recordings match, a decision tree, a heuristic-based framework, among others). The machine learning framework receives the signals422and audio fingerprint for two ambient sound recording as input. As output, the machine learning framework provides a confidence level (e.g., a likelihood) that the two ambient sound recording match.

If no matches are found between ambient sound recordings, then the ambient sound recordings are kept in the pool data store196for later matching. If a recording remains in the pool data store196for sufficiently long, then the match engine191executes a remedial process. For instance, if no matches are found for an ambient sound recording within a threshold amount of time, then the match engine191sends a message to the device that provided that ambient sound recording indicating an error and prompting the device to try again.

If too many matches are found, the match engine191sends a message to devices associated with the matching ambient sound recordings asking the devices for more information (e.g., location information or more precise location information). In an example, the match engine191causes a device to display a particular code (e.g., a string of letters or numbers) and requests that users of devices associated with the accounts enter in the code displayed on the target device (e.g., via an SMS message, an in-app message, or in other manners). If the match engine191receives a response with the particular code, then the match engine191matches the device that sent the message with the device that displayed the code.

The result of operation440is a match between two devices. In an example, the result is a data structure identifying both devices by the device identifiers of the devices.

At operation450, the match engine191determines which of the matched devices is a source device and which of the matched devices is a target device for the purpose of transferring credentials or account access. For instance, after matching and prior to associating, determining to associate the target device150with the source device account114based on the source device signals424and the target device signals426. Various techniques may be used to make this determination.

In some instances, the ambient sound recordings or messages from the device that made the ambient sound recording or sent the message further include an indication of source and target. For instance, the utterance may specify a target (e.g., “ahoy computer, log me in to [name of device] with voice”). The speech analysis engine190determines the source and target using the utterance and tags the recordings based thereon. The match engine191uses the tag to determine whether a device is a source device or a target device.

In many examples, though, there is no explicit identification. In such instances, a hierarchy of device types can be relied on. For example, the order may be: streaming devices, smart speakers, desktops, laptops, tablets, and smartphones. In the example, the device types earlier in the hierarchy have preference to device types later in the hierarchy. For instance, the determining to associate the target device150with the source device account114based on a hierarchy of device types, the device type of the source device110, and the device type of the target device150. In an example, in a match between a smartphone and a smart speaker, the smartphone is designated as the source device110and the smart speaker is designated as the target device150, based on their location in the hierarchy of device types. This hierarchy is just an example and other device types are usable. Generally, device types associated with high-quality sound output will rate higher than device types associated with low quality sound output. So, for instance, a smartphone may rank relatively low, but a smartphone that is paired via BLUETOOTH to a BLUETOOTH-enabled speaker system may rank relatively higher than a smartphone that is not associated with a speaker system because of its perceived increase in sound quality.

In another example, the access management engine192determines a source and target based on relative volume in sound recordings. For instance, a device that provided a lower volume recording will be determined to be the target device and a device that provide a higher quality recording will be a source device. This is because, for example, the source device is likely to be closer to a user than a target device and therefor provide a higher-quality (or at least a higher in volume) recording.

In another example, the match engine191determines the source and the target using activity logs of the devices. For instance, the match engine obtains activity logs describing recent activity of the matched devices. The activity logs can indicate whether the device was recently in motion (e.g., recently used by a user while walking, running, or driving based on motion sensors of the device) or was relatively stable. The activity logs can also indicate whether the device was recently used to play media content items. In an example, the matched devices include a smartphone and a vehicle-based Personal Media Streaming Appliance (PMSA). The match engine191obtains the activity logs of the smartphone, which indicate that the smart phone was recently used to play media content items and was in motion toward the location of the PMSA. The match engine191obtains the activity logs of the PMSA, which indicate that the PMSA was recently activated (e.g., powered on) and is stationary. The match engine uses this data to determine that the smartphone is the source device and the PMSA is the target device responsive to determining that the smartphone was brought to the PMSA, which was recently activated.

In a further example, the determination of source and target by the match engine191is based on multiple factors, including: activity logs, relative volume, and device type. For instance, these factors are provided as input into a machine learning model configured to output an indication of source or target device. The match engine191uses the output of the machine learning model to determine the source and the target.

In yet another example, the match engine191determines the source and the target by prompting a user. For example, the match engine191selects one of the devices and causes that device to ask the user whether it is the device into which the user would like to log in. The speech analysis server750then receives a response from the user (an utterance responding to the question), analyzes the response via the speech analysis engine190, and data associated with the response is provided to the match engine191. If the response indicates that the device is the device that the user would like to log into, then that device is identified as the target device150. If not, then that device is not a candidate for the target device150.

As a result of operation450, a device is identified as a source device and another device is identified as a target device.

At operation460, the access management engine192performs credential or account management based on the identified source and target devices. For example, associating the target device150with the source device account114responsive to the matching. For instance, by the access management engine192providing credentials111associated with the source device110to the target device150. The access management engine192obtains the credentials111of the source device account114. For instance, the access management engine192selects the credentials111from a data store that stores account credentials. In other instances, the access management engine192generates new credentials122associated with the source device account114for providing to the target device150. The access management engine192provides the credentials111to the target device150for use in, for example, causing media playback. In other examples, a device identifier of the target device150is added to an authorized list of devices for the source device account114instead of providing the credentials111to the target device150.

Once the target device150receives the credentials or is otherwise associated with an account, the received credentials are stored locally at the target device150for use in accessing services (e.g., services provided by the media server application684). In instances where the credentials are for use in a guest mode, the credentials are stored in a temporary area or are stored in association with a time limit. In some instances, the access management engine192does not provide the credentials themselves. Instead the access management engine192provides an identifier associated with credentials. For instance, this may be used where the target device150includes already multiple different sets of credentials and the target device150uses the provided identifier in order to select credentials stored locally in the target device to use.

The target device150can then use the received credentials to access services associated with the media server application684. For instance, the target device150sends a playback request to a media server application that includes the credentials (or a token based thereon). In some instances, the access is performed automatically (e.g., to confirm that the credentials are usable) in other instances, the access is performed in response to a user request.

Process for Guest and Primary Modes

FIG. 5illustrates a process500for operating the target device150in a primary mode512and a guest mode522. The process500begins with operation510.

At operation510, the target device150operates in a primary mode512associated with the target device account154. In an example, the primary mode512is a mode in which the target device150remain indefinitely until a log out or switch account command is executed. In many examples, the primary mode512is a mode associated with an account of owner of the target device150. Operating the target device150in the primary mode512associated with a target device account154includes the target device150using the target device account154to access services or content associated with the target device account154. For example, the target device150is a smart speaker and target device account154is an account of the owner of the target device150. The target device150then operates in a primary mode512associated with the target device account154(e.g., the owner's account) and obtains media content items using the target device account154. This includes, for example, accessing a library of media content items associated with the target device account154, and obtaining media content items according to a taste profile associated with the target device account154, among other activities.

While operating in the primary mode512, the target device150receives a guest mode command over a user interface. The guest mode command is a command that causes the target device150to begin the process of entering a guest mode (e.g., causes the process500to transition to operation520). The target device150receives the guest mode command in any of a variety of ways. In an example, the target device150receives the guest mode command over a voice-based user interface. For instance, the target device150receives the utterance “enter guest mode”, which the target device150analyzes (e.g., using natural language processing) and executes a guest mode process in response thereto. In another example, the target device150has a tactile or virtual button that, when activated, causes the target device150to execute a guest mode process in response thereto.

At operation520, the target device150operates in the guest mode522associated with the source device account114. For instance, associating the target device150with the source device account114includes operating the target device in a guest mode522associated with the source device account. In an example, this operation520includes, the target device150operating according to credentials associated with the source device account114. The credentials can be obtained using any of a variety of techniques described herein, including but not limited to those described in relation toFIG. 1. Operating in the guest mode522takes various forms. In some instances, operating in the guest mode522includes the target device150operating with a limited set of permissions compared to the primary mode512. For instance, while operating in the guest mode522, the target device150may be unable to perform one or more of the following actions: downloading tracks to the target device150, modifying an equalizer of the target device150, changing wireless settings of the target device150, changing security settings of the target device150, changing an ownership of the target device150, changing account management settings of the target device150, playing explicit tracks on the target device150, locking the target device150, and obtaining primary mode status on the target device150, among other actions. For example, the target device150operating in the guest mode522, the target device receives user input associated with changing the ownership of the target device150and provides an error message indicating that the action cannot be taken due to lack of permission.

In addition, the guest mode522may be a mode associated with a limited amount of time. For example, the target device150operates in the guest mode522for a limited amount of time (e.g., one hour, one day, one week, one month), until an a certain number of media content items are played (e.g., playback of one, two, three, or more media content items), until the occurrence of a particular event (e.g., receiving a revert command, detecting the presence of a device associated with the target device account) or another event, and then the target device150reverts to the primary mode512and the account associated therewith (e.g., the target device account154). In another example, the device operates in the guest mode522until a reversion command is received, such as receiving user input associated with reverting to the primary mode512over a user interface (e.g., receiving an utterance over a voice based user interface of the target device150or receiving an indication that a virtual or physical user interface element associated with reversion has been actuated). An account operating in a primary mode512can customize the permissions of the guest mode522.

In another example, the target device150operates in the guest mode522until an event occurs (e.g., until a device power event occurs). For instance the target device150automatically reverts to the primary mode512associated with the target device account154after an occurrence of an event. For instance, the target device150operates in the guest mode until the target device150powers on, powers off, enters a sleep mode, enters an inactive state, enters an active state, wakes up, restarts, loses power, or gains power, among others. The target device150then reverts to the primary mode512with the target device account154.

In some examples, operating in a guest mode522includes the target device150storing credentials of the account associated with the primary mode512. In an example, the target device150stores credentials associated with the primary mode in memory for later use. Storing the credentials in memory facilitates the target device150reverting from, for example, operating in the guest mode522associated with the source device account114to the primary mode512associated with the target device account154without re-receiving the credentials associated with the target device account154, which saves the user time and reduces resource consumption (e.g., by not requiring the use associated with the target device account154to re-log into the target device150).

At operation530, the target device150reverts to the primary mode512associated with the target device account154from the guest mode522associated with the source device account114. The reversion is triggered by one or more of the conditions or criteria described in operation520. Reverting can include accessing credentials associated with the target device account154that are stored locally at the target device150. In some examples, reverting includes obtaining the credentials from a server (e.g., the target device150accessing an application programming interface associated with the access management engine192at the media-delivery system604).

Other Embodiments

In some examples, one or both of the target device150and the source device110identify that the user is requesting a log-in process before or after providing a recording to a server. For instance, the target device150may determine that a log-in process is being requested based on detecting over a user interface that a log-in user interface element is activated. In another example, the speech analysis server750sends a message to the target device150, prompting the target device150to provide additional information to facilitate the matching process. In such instances, the source device110can provide additional information with the request usable for identifying the device during the process. For instance, the request can include additional metadata regarding the device including its location, altitude, and nearby WI-FI SSIDs, among other data. Where the identification is performed in response to activation of a particular user interface element, an indicator whether the device is a source or target can be provided. In other examples, the target device150or the source device110provides an audible signal to facilitate matching the recordings (e.g., an encoded message that facilitates matching).

In some examples, the target device150enters an association mode to prepare for becoming associated with a particular account or credentials. The target device150enters the association mode in response to a variety of causes. In some examples, the target device150automatically enters the association mode responsive to determining that the target device150is attempting to operate without credentials. For instance, the target device150may be powering up for a first time by an end user and the target device150was not preconfigured with an account. In other examples, the target device150finished a log-out mode in which the target device150disassociates itself from particular credentials. In other examples, the target device150receives a signal from another device (an electronic signal, an acoustic signal, among others) that causes the target device150to enter the association mode. In still other examples, the target device150enters the association mode responsive to the target device150receiving a user input. In an example, the target device150receives an utterance instructing the target device150to enter an association mode (e.g., the utterance “enter association mode”) as input via an utterance-based user interface. In entering the association mode the target device150prepares to become associated with credentials or an account. In the illustrated example, responsive to entering the association mode, the target device150prepares to record ambient sound. The target device150, activates one or more sound input devices, for instance. Where the target device150is already associated with credentials, entering the association mode further includes disassociating the target device150from existing credentials. In still further examples, entering the association mode takes different forms depending on a kind association mode selected. The target device150is able to operate in a variety of modes, each mode having its own characteristics and permissions. While in the association mode, the target device150generates additional signals (e.g., obtaining a more precise location) and provides the additional signals with messages to the server (e.g., messages with ambient sound recordings) to increase the likelihood that the device will be properly identified as a match.

While many examples herein are described in relation to media content items and streaming services, this disclosure need not be limited to those uses. In an example scenario, the source device110is a smartphone and the target device150is a voice-enabled corporate computer system into which a user wants to log in. One or more of the techniques described herein can be used to facilitate such a log-in process.

Device Environment

FIG. 6, which is made up ofFIG. 6AandFIG. 6B, illustrates an example system600for association via audio. The example system600is a media content playback system for association via audio and includes the source device110including the source device activation trigger engine116and the target device150including the target device activation trigger engine156. The source device110and the target device150each further including a media-playback engine117. The source device110and the target device150are communicatively coupled across a network606to a media-delivery system604.

Source Device

The source device110is a computing device. In some examples, the source device110is a computing device for playing media content items to produce media output. In some examples, the media content items are provided by the media-delivery system604and transmitted to the source device110using the network606. A media content item is an item of media content, including audio, video, or other types of media content, which may be stored in any format suitable for storing media content. Non-limiting examples of media content items include songs, music, albums, audiobooks, music videos, movies, television episodes, podcasts, other types of audio or video content, and portions or combinations thereof.

The source device110plays the media content item for the user. The media content item is selectable for playback with user input. The media content is also selectable for playback without user input, such as by the source device110or the media-delivery system604. In an example, media content is selected for playback by the media-delivery system604based on a user taste profile stored in association with an account.

The source device110selects and plays media content and generates interfaces for controlling playback of media content items. In some examples, the media-playback engine117receives user input over a user interface, such as a touch screen user interface, an utterance-based user interface, tactile user interfaces, virtual user interfaces, or other user interfaces and plays a media content item based thereon.

The source device110can include other input mechanisms including but not limited to a keypad and/or a cursor control device. The keypad receives alphanumeric characters and/or other key information. The cursor control device includes, for example, a handheld controller or mouse, a rotary input mechanism, a trackball, a stylus, and/or cursor direction keys.

As noted above, the source device110plays media content items. In some examples, the source device110plays media content items that are provided (e.g., streamed, transmitted, etc.) by a system external to the media-playback device such as the media-delivery system604, another system, or a peer device. Alternatively, in some examples, the source device110plays media content items stored locally on the source device110. Further, in at least some examples, the source device110plays media content items that are stored locally as well as media content items provided by other systems.

In some examples, the source device110is a computing device, handheld entertainment device, smartphone, tablet, watch, wearable device, or any other type of device capable of playing media content. In yet other examples, the source device110is a media playback appliance, such as an in-dash vehicle head unit, an aftermarket vehicle media playback appliance, a smart assistant device, a smart home device, a television, a gaming console, a set-top box, a network appliance, a BLU-RAY disc player, a DVD player, a media player, a stereo system, smart speaker, an Internet-of-things device, or a radio, among other devices or systems.

In at least some examples, the source device110includes a location-determining device650, a user interface652, one or more processing devices654, a memory device656, a content output device658, a movement-detecting device, a network access device662, and a sound-sensing device664. Other examples may include additional, different, or fewer components.

The location-determining device650is a device that determines the location of the source device110. In some examples, the location-determining device650uses one or more of the following technologies: Global Positioning System (GPS) technology that receives GPS signals from satellites, cellular triangulation technology, network-based location identification technology, WI-FI positioning systems technology, ultrasonic positioning systems technology, and combinations thereof. Examples of the location-determining device650further include altitude- or elevation-determining devices, such as barometers.

The user interface652operates to interact with the user, including providing output and receiving input. The user interface652can be a physical device that interfaces with the user (e.g., touch screen display) or a combination of devices that interact with the user (e.g., speaker and microphone for providing an utterance-based user interface).

In some examples, the user interface includes a touch-screen based user interface. A touch screen operates to receive an input from a selector (e.g., a finger, stylus, etc.) controlled by the user. In some examples, the touch screen operates as both a display device and a user input device. In some examples, the user interface652detects inputs based on one or both of touches and near-touches. In some examples, the touch screen displays a user interface for interacting with the source device110. Some examples of the source device110do not include a touch screen.

Examples of the user interface652include input control devices that control the operation and various functions of the source device110. Input control devices include any components, circuitry, or logic operative to drive the functionality of the source device110. For example, input control device(s) include one or more processors acting under the control of an application.

While some examples of the source device110do not include a display device, where a source device110does include a display device, the source device110will often include a graphics subsystem and coupled to an output display. The output display uses various technologies, such as TFT (Thin Film Transistor), TFD (Thin Film Diode), OLED (Organic Light-Emitting Diode), AMOLED (active-matrix organic light-emitting diode) display, and/or liquid crystal display (LCD)-type displays. The displays can also be touchscreen displays, such as capacitive and resistive-type touchscreen displays.

The one or more processing devices654include one or more processing units, such as central processing units (CPU), digital signal processors, and field-programmable gate arrays, among others.

The memory device656operates to store data and instructions. In some examples, the memory device656stores instructions for the media-playback engine117. Some examples of the memory device656also include a media content cache672. The media content cache672stores media-content items, such as media content items that have been previously received from the media-delivery system604. The media content items stored in the media content cache672are storable in an encrypted or unencrypted format, and decryption keys for some or all of the media content items are also stored. The media content cache672can also store metadata about media-content items such as title, artist name, album name, length, genre, mood, or era. The media content cache672can also store playback information about the media content items, such as the number of times the user has requested to playback the media content item or the current location of playback.

The memory device656typically includes at least some form of computer-readable media. Computer readable media includes any available media that can be accessed by the source device110. By way of example, computer-readable media include computer readable storage media and computer readable communication media.

The content output device658operates to output media content. In many examples, the content output device658provides media output for a user. In some examples, the content output device658provides media output to a target device150. Examples of the content output device658include a speaker assembly having one or more speakers, an audio output jack, a BLUETOOTH transmitter, a display panel, and a video output jack. Other examples are possible as well, such as transmitting a signal through the audio output jack or BLUETOOTH transmitter to reproduce an audio signal by a connected or paired device such as headphones, speaker system, or vehicle head unit.

The network access device662operates to communicate with other computing devices over one or more networks, such as the network606. Examples of the network access device include one or more wired network interfaces and wireless network interfaces. Examples of wireless network interfaces include infrared, BLUETOOTH wireless technology, 802.11a/b/g/n/ac, and cellular or other radio frequency interfaces.

In some examples, the source device110includes a movement-detecting device that senses movement of the source device110, acceleration of the source device110, determines an orientation of the source device110, or includes other detecting devices. In at least some examples, the detecting devices includes one or more accelerometers or other motion-detecting technologies or orientation-detecting technologies.

The network606is an electronic communication network that facilitates communication between the source device110, the media-delivery system604, and in some instances, the target device150. An electronic communication network includes a set of computing devices and links between the computing devices. The computing devices in the network use the links to enable communication among the computing devices in the network. The network606can include routers, switches, mobile access points, bridges, hubs, intrusion detection devices, storage devices, standalone server devices, blade server devices, sensors, desktop computers, firewall devices, laptop computers, handheld computers, mobile telephones, vehicular computing devices, and other types of computing devices.

In various examples, the network606includes various types of links. For example, the network606includes wired and/or wireless links, including BLUETOOTH, ultra-wideband (UWB), 802.11, ZIGBEE, cellular, and other types of wireless links. Furthermore, in various examples, the network606is implemented at various scales. For example, the network606is implemented as one or more vehicle area networks, local area networks (LANs), metropolitan area networks, subnets, wide area networks (such as the Internet), or can be implemented at another scale. Further, in some examples, the network606includes multiple networks, which may be of the same type or of multiple different types.

Target Device

The target device150can include one or more of the components of the source device110. The aspects described herein are relevant to using the source device account114of the source device110on the target device150. In an example, the association is performed using a sound-sensing device664of the target device150. In some examples, it is otherwise difficult for a user to provide account information to the target device150, such as by the target device150lacking a keyboard, touch screen, or other components that facilitate arbitrary input. In some examples, the target device150lacks a direct connection to the target device150over BLUETOOTH, WI-FI, or other electronic communication schemes.

The sound-sensing device664senses sounds proximate the target device150(e.g., sounds within a vehicle in which the target device150is located). In some examples, the sound-sensing device664comprises one or more microphones. In some examples, the sound-sensing device664includes multiple microphones in a sound-canceling arrangement to facilitate operation in a noisy environment (e.g., configured for use in a vehicle). The sound-sensing device664is able to capture sounds from proximate the target device150and create a representation thereof. These representations are analyzed by the target device150or the media-delivery system604.

In some examples, the representations are used to provide an utterance-based user interface. In such examples, speech-recognition technology is used to identify words spoken by the user. The words are recognized as commands affect the behavior of the target device150(e.g., affecting playback of media content by the target device150). Natural language processing and/or intent-recognition technology are usable to determine appropriate actions to take based on the spoken words.

Additionally or alternatively, the sound-sensing device664determines various sound properties about the sounds proximate the user such as volume, dominant frequency or frequencies, among other properties. These sound properties are usable to make inferences about the environment proximate to the target device150, such as whether the sensed sounds correspond to playback of a media content item. In some examples, the sound sensed by the sound-sensing device664are transmitted to media-delivery system604(or another external system) for analysis, such as using speech-recognition, intent-recognition, and media identification technologies, among others.

The media-delivery system604includes one or more computing devices and provides media content items to the source device110, target device150, and, in some examples, other media-playback devices as well. The media-delivery system604includes a media content server680. AlthoughFIG. 6shows single instances of the media content server680and the speech analysis server750some examples include multiple servers. In these examples, each of the multiple servers may be identical or similar and may provide similar functionality (e.g., to provide greater capacity and redundancy, or to provide services from multiple geographic locations). Alternatively, in these examples, some of the multiple servers may perform specialized functions to provide specialized services (e.g., services to enhance media content playback during travel or other activities, etc.). Various combinations thereof are possible as well.

The media content server680transmits stream media to media-playback devices, such as the source device110or target device150. In some examples, the media content server680includes a media server application684, one or more processing devices654, a memory device656, and a network access device662.

In some examples, the media server application684streams music or other audio, video, or other forms of media content. The media server application684includes a media stream service694, a media data store696, and a media application interface698. The media stream service694operates to buffer media content such as media content items706,708, and710, for streaming to one or more streams700,702, and704.

The media application interface698can receive requests or other communication from media-playback devices or other systems, to retrieve media content items from the media content server680. For example, the media application interface698receives a communication from the media-playback engine117.

In some examples, the media data store696stores media content items712, media content metadata714, and playlists716. The media data store696may store one or more databases and file systems, such as the set of data structures800described in relation toFIG. 7. As noted above, the media content items712may be audio, video, or any other type of media content, which may be stored in any format for storing media content.

The account data store718are used to identify users of a media streaming service provided by the media-delivery system604. In some examples, the media-delivery system604authenticates a user via data contained in the account data store718and provides access to resources (e.g., media content items712, playlists716, etc.) to a device operated by a user. In some examples, different devices log into a single account and access data associated with the account in the media-delivery system604. User authentication information, such as a username, an email account information, a password, and other credentials, can be used for the user to log into his or her user account. A device can use stored credentials to log a user into the account on a device.

The media data store696includes user tastes data720. The user tastes data720includes but is not limited to user preferences regarding media content items, such as media content items that the user likes/dislikes, media content item qualities that the user likes/dislikes, historical information about the user's consumption of media content, libraries of media content items, and playlists of media content items, among other user data.

The media content metadata714operates to provide various information associated with the media content items712. In some examples, the media content metadata714includes one or more of title, artist name, album name, length, genre, mood, era, acoustic fingerprints, and other information. The playlists716operate to identify one or more of the media content items712. In some examples, the playlists716identify a group of the media content items712in a particular order. In other examples, the playlists716merely identify a group of the media content items712without specifying a particular order. Some, but not necessarily all, of the media content items712included in a particular one of the playlists716are associated with a common characteristic such as a common genre, mood, or era. The playlists716may include user-created playlists, which may be available to a particular user, a group of users, or to the public.

In some examples, the media server application684or a dedicated access management server provides access management services. In examples, the media server application684exposes application programming interface endpoints usable by calling devices or functions to use access management services, such as services for logging in to an account, obtaining credentials associated with an account, generating credentials associated with an account, and other services.

Although inFIGS. 6A and 6Bonly a single source device110, target device150, and media-delivery system604are shown, in accordance with some examples, the media-delivery system604supports the simultaneous use of devices, and the source device110and the target device150simultaneously access media content from multiple media-delivery systems604. Additionally, althoughFIGS. 6A and 6Billustrates a streaming media-based system for media-playback, other examples are possible as well. For example, in some examples, the source device110includes a media data store and the source device110selects and plays back media content items without accessing the media-delivery system604. Further in some examples, the source device110operates to store previously-streamed media content items in a local media data store (e.g., in the media content cache672).

In at least some examples, the media-delivery system604streams, progressively downloads, or otherwise communicates music, other audio, video, or other forms of media content items to the source device110or target device150for later playback. In accordance with an example, the user interface652receives a user request to, for example, select media content for playback on the source device110.

Software examples of the examples presented herein may be provided as a computer program product, or software, that may include an article of manufacture on a machine-accessible or machine-readable medium having instructions. The instructions on the non-transitory machine-accessible machine-readable or computer-readable medium may be used to program a computer system or other electronic device. The machine- or computer-readable medium may include, but is not limited to, magnetic disks, optical disks, magneto-optical disks, or other types of media/machine-readable medium suitable for storing or transmitting electronic instructions. The techniques described herein are not limited to any particular software configuration. They may find applicability in any computing or processing environment. In some examples, there are one or more processors that operate as a particular program product or engine. In some examples, one or more processors are coupled to a memory storing instructions which when executed cause the one or more processors to operate in a particular manner. In some examples, the one or more processors include two or more sets of processors operating on different devices.

The terms “computer-readable”, “machine-accessible medium” or “machine-readable medium” used herein shall include any medium that is capable of storing, encoding, or transmitting a sequence of instructions for execution by the machine and that causes the machine to perform any one of the methods described herein. Further, it is common in the art to speak of software, in one form or another (e.g., program, procedure, process, application, module, unit, logic, and so on), as taking an action or causing a result. Such expressions are merely a shorthand way of stating that the execution of the software by a processing system causes the processor to perform an action to produce a result.

Some examples include a computer program product. The computer program product may be a storage medium or media having instructions stored thereon or therein which can be used to control, or cause, a computer to perform any of the procedures of the examples of the invention. The storage medium may include without limitation an optical disc, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory, a flash card, a magnetic card, an optical card, nanosystems, a molecular memory integrated circuit, a RAID, remote data storage/archive/warehousing, and/or any other type of device suitable for storing instructions and/or data.

Stored on any one of the computer-readable medium or media, some implementations include software for controlling both the hardware of the system and for enabling the system or microprocessor to interact with a human user or other mechanism utilizing the results of the examples of the invention. Such software may include without limitation device drivers, operating systems, and user applications. Ultimately, such computer-readable media further include software for performing example aspects of the invention, as described above.

Included in the programming and/or software of the system are software modules for implementing the procedures described above.

Various operations and processes described herein can be performed by the cooperation of two or more devices, systems, processes, or combinations thereof.

Set of Data Structures

FIG. 7illustrates an example set of data structures800storable in the media data store696. As illustrated, the set of data structures800includes an account table802, a media content item context data structure804, and a media content item data structure806. For each account record in the account table802, the set of data structures800includes a device data table810, a playback history table812, a favorite table814, and a playback state data structure816. It is noted that, where user data is used, it can be handled according to a defined user privacy policy and can be used to the extent allowed by the user. Where the data of other users is used, it can be handled in an anonymized matter so the user does not learn of the details of other users generally or specifically. In addition, the data contained in the set of data structures800is stored according to a defined security policy and in accordance with applicable regulations.

As illustrated, each account record in the account table802has a relationship with a device data table810, a playback history table812, and a favorite table814. Each device record in the device data table810has a relationship with a playback state data structure816. Each device record in a favorite table814has a relationship with a media content item context data structure804. Each context data structure has a relationship with the media content item data structure806.

The account table802stores one or more account records usable to identify accounts of the media-delivery system604. In some examples, some or all of the data from the pool data store196is obtained from the set of data structures800. For instance, at least some of the data in pool data store196is obtained by querying the set of data structures800for particular data. For instance, the match engine191obtains a device identifier from a message that contained the ambient sound recording, and the match engine191determines the device type of the device by querying the device data table to determine a type of the device. In an example, where a particular account is referred to in this disclosure (e.g., the target device account154), the account is associated with an entry stored in the set of data structures800. In an example, when the target device150accesses the media-delivery system604under the target device account154, the media-delivery system604locates an account record corresponding to the target device account154in the account table802. The media-delivery system604then uses the data associated with the account record in the account table802to provide services associated with that account. For instance, the media-delivery system604provides a media content item described in the favorite table814to the target device150and updates the playback history table812accordingly.

The account table802references one or more other tables, and is referenced by one or more other tables. In an example, each account record of the account table802corresponds to an account. For instance, the target device account154corresponds to a target device account record in the account table802, and the source device account114corresponds to a source device account record in the account table802. Each account record of the account table802includes data associated with one or more fields of the account table802, such as an account ID field, a user ID field, a password field, and a type field. The account ID field stores an identifier of the account record, such as using a number. The user ID field stores an identifier of a user, such as the user's name. The password field stores data associated with a password of the user, such as a hashed and salted password. The type field identifies subscription types associated with the account record.

Each account record identified in the account table802is associated with, and identifies, data for providing various services from the media-delivery system604. In some examples, the data includes the device data table810, the playback history table812, the favorite table814, and the playback state data structure816, among others. In the illustrated example, the tables810,812, and814are primarily described in association with a single record (e.g., the record having the Account ID: 81828). However, it is understood that, in other examples, the tables810,812, and814are structured to be associated with a plurality of accounts

The device data table810identifies one or more devices associated with a particular account record of the account table802. The device data table810is referenced by the account table802or other tables. The device data table810can reference one or more other tables.

In an example, each device record of the device data table810includes data associated with a device. For instance, a first device record of the device data table810corresponds to the source device110, and a second device record of the device data table810corresponds to the target device150once both devices110,150have been associated with the same account. Each device record of the device data table810includes data associated with one or more fields of the device data table810, such as a device ID field (e.g., storing device identifier data, such as an alphanumeric identifier), a name field (e.g., for storing a device name), a status field (e.g., for storing a status of the device, such as whether the device is currently active or inactive), a location field (e.g., for storing a last-known location of the device), and type field (e.g., for storing a type of the device, such as a phone device, a speaker device, or a vehicle head unit).

The playback history table812describes the media content items played by the account by storing one or more playback records. The playback history table812can reference and be referenced by one or more other tables. In an example, each playback record of the playback history table812includes data associated with a media content item played by a respective account or device. Each playback record of the playback history table812includes data associated with one or more fields of the playback history table812, such as a device ID field (e.g., for storing an identifier of the device that caused playback of the playback record), a MCI (Media Content Item) ID field (e.g., for storing an identifier of the media content item that was played back), a start time field (e.g., for identifying the start time at which the media content item was played back), and a location field (e.g., for identifying the location of the device associated with the device ID when playback was initiated).

The favorite table814describes information about favorite media content item contexts associated with the account by storing one or more favorite records. The favorite table814includes information about favorites associated with an account. The favorite table814can reference and be referenced by one or more other tables. In an example, each favorite record of the favorite table814includes data associated with a favorite media content item context (e.g., album or playlist). Each favorite record of the favorite table814includes data associated with one or more fields of the favorite table814, such as an ID field (e.g., for identifying the favorite record) and a context field (e.g., for identifying a media content item context associated with the favorite record).

The context data structure804is a data structure (e.g., record of a table or other data structure) that contains data associated with a media content item context (e.g., album or playlist). The context data structure804can reference and be referenced by one or more tables or other data structures. The context data structure804stores data regarding a particular media content item context in one or more fields, such as an ID field (e.g., for identifying the context data structure804), a title field (e.g., a string naming the context data structure804), a type field (e.g., for describing the type of the media content item context, such as a playlist, album, or television season), and media content item field (e.g., for identifying one or more media content items of the context data structure804)

The media content item data structure806is a data structure (e.g., record of a table or other data structure) that contains data associated with a media content item. The media content item data structure806can reference and be referenced by one or more tables or other data structures. The media content item data structure806stores data regarding a particular media content item in one or more fields, such as an ID field (e.g., storing an identifier of the media content item data structure806), a title field (e.g., storing a title of the media content item data structure806, such as a song title), a content field (e.g., storing the content of the media content item or a link to the content of the media content item data structure806, such as the audio content of a song), and an audio fingerprint field. In an example, the audio fingerprint field stores an audio fingerprint of the content of the media content item data structure806.

The playback state data structure816is a data structure (e.g., a record of a table or other data structure) that contains data associated with a state of a device (e.g., a state associated with a device record of the device data table810). The playback state data structure816can reference and be referenced by one or more tables or other data structures. The playback state data structure816stores data regarding a particular playback state in one or more fields, such as a current context field (e.g., describing a current context from which a device is playing, such as by containing an identifier of the context), a current MCI (Media Content Item) (e.g., describing a current media content item that is playing, such as by containing an identifier of the media content item), a playback mode field (e.g., describing a playback mode of the device, such as shuffle or repeat), a playback speed field (e.g., describing a current playback speed), and a next MCI field (e.g., describing the next media content item to be played).

Various operations and processes described herein can be performed by the cooperation of two or more devices, systems, processes, or combinations thereof.

While various examples of the present invention have been described above, it should be understood that they have been presented by way of example, and not limitation. It will be apparent to persons skilled in the relevant art(s) that various changes in form and detail can be made therein. Thus, the present invention should not be limited by any of the above described example embodiments, but should be defined only in accordance with the following claims and their equivalents. Further, the Abstract is not intended to be limiting as to the scope of the example embodiments presented herein in any way. It is also to be understood that the procedures recited in the claims need not be performed in the order presented.