Patent ID: 12249345

DETAILED DESCRIPTION

Turning now toFIGS.1A and1B, example process flows that demonstrate various aspects of the present disclosure that may be implemented by a client device120and a remote system160are depicted. The client device120illustrated inFIGS.1A and1Bmay include at least the components that are encompassed within the boxes ofFIGS.1A and1Bthat represent the client device120. Similarly, the remote system150illustrated inFIGS.1A and1Bmay include at least the components that are encompassed within the boxes ofFIGS.1A and1Bthat represent the remote system150. As described herein, the process flows ofFIGS.1A and1Bmay be utilized to implement ephemeral learning and/or federated learning of audio-based machine learning (ML) model(s) based on streams of audio data generated via radio stations across the world.

Referring specifically toFIG.1A, assume that a stream of audio data124is received at the client device120. The stream of audio data124may be received via an internal transceiver122A of the client device120(e.g., that is integral to the client device120) or an external transceiver122B that is coupled to the client device (e.g., via an audio jack of the client device120or by other means). Further, the stream of audio data124may be generated various sources, such as a radio station that emits the stream of audio data via a radio tower110A and at different frequencies (e.g., an amplitude modulation (AM) frequency, a frequency modulation (FM) frequency, etc.), an internet radio source110B, and/or other sources of radio. In some implementations, the client device120may cause the stream of audio data124to be buffered (e.g., in one or more buffers132) while the stream of audio data124is initially processed by the client device120to determine whether to implement an ephemeral learning technique and/or federated learning technique on the stream of audio data124, and/or where to implement the ephemeral learning technique and/or the federated learning technique (e.g., locally at the client device120as shown in the process flow ofFIG.1Aor at the remote system160as shown inFIG.1B).

As shown inFIG.1A, the client device120may include various engines to perform various operations. For example, the client device120may include an on-device ML model engine134, a learning engine136, a gradient engine138, a connection status engine140, a language identification engine142, and a deduplication engine144. Further, the remote system may include a remote update engine180and an update distribution engine182. Although the client device120and the remote system160are depicted inFIG.1Aas including particular engines that perform the process flow in a particular manner, it should be understood that is to illustrate various techniques described herein and is not meant to be limiting. For instance, various engines depicted inFIG.1Amay be combined and other engines may be added or omitted, and the process flow described with respect toFIG.1Amay be reordered (e.g., as described with respect toFIG.1B).

Generally, the on-device ML model engine134may process, using an on-device audio-based ML model that is stored in on-device memory or storage of the client device120(e.g., stored in on-device ML model(s) database120A) and that is an on-device counterpart of a global audio-based ML model to be updated (e.g., stored in global ML model(s) database160A) the stream of audio data124to generate one or more predicted outputs134A. The one or more predicted outputs134A generated by the on-device audio-based ML model engine134may be based on a type of the on-device audio-based ML model that is utilized in processing the stream of audio data124(and the type of the global audio-based ML model that is to be updated based on the processing of the stream of audio data124). For instance, assume that the on-device audio-based ML model and the global audio-based ML model are corresponding language representation ML models. In this example, the one or more predicted outputs134A may include, for instance, a rich feature representation of the stream of audio data124, such as a description of one or more sounds captured in the stream of audio data124, a relationship between sounds captured in the stream of audio data124, and/or other features of the stream of audio data124. In contrast, assume that the on-device audio-based ML model and the global audio-based ML model are corresponding multilingual automatic speech recognition (ASR) models. In this example, the one or more predicted outputs134A may include, for instance, recognized text in the given language, such as one or more terms corresponding to the stream of spoken utterances captured in the stream of audio data124. Although the above examples are described with respect to particular audio-based ML models, it should be understood that is for the sake of example and is not meant to be limiting, and that other non-limiting examples of audio-based ML models are described with respect toFIG.2.

Further, the gradient engine136may generate a gradient136A based on the processing of the stream of audio data124by the on-device ML model engine134and/or the one or more predicted output134A generated by the on-device ML model engine134. The gradient136A may be transmitted to the remote system160for utilization in updating the global audio-based ML model that is a global counterpart of the on-device audio-based ML model utilized in processing the stream of audio data124. In generating the gradient136A, the gradient engine136may utilize the learning engine138. In some implementations, the learning engine138may employ one or more supervised learning techniques, whereas in other implementations the learning engine138may employ one or more unsupervised or semi-supervised learning techniques. However, in various implementations, and by virtue of the stream of audio data124being generated by a given radio station, an explicit supervision signal to employ one or more of the supervised learning techniques may not be available. Accordingly, techniques described herein are generally described with respect to one or more of the unsupervised or semi-supervised learning techniques although one or more of the supervised learning techniques may additionally, or alternatively, be utilized by the learning engine138.

In some versions of those implementations, one or more of the unsupervised or semi-supervised learning techniques may correspond to a teacher-student technique. In implementing the teacher-student technique, the learning engine138may process, using one or more corresponding benchmark audio-based ML models, the stream of audio data124to generate one or more benchmark outputs. In these implementations, the one or more corresponding benchmark audio-based ML models may be the same ML model as the on-device audio-based ML model and/or the global audio-based ML model, or another audio-based ML model (e.g., stored in the on-device ML model(s) database120A and/or the global ML model(s) database160A) that is distinct from, but the same type as, the on-device audio-based ML model and the global audio-based ML model. Further, the one or more benchmark outputs may be utilized as a quasi-supervision signal to be utilized in generating the gradient136A. For instance, the gradient engine136may compare the one or more predicted outputs134A to the one or more benchmark output in generating the gradient136A.

In some further versions of those implementations, the one or more benchmark outputs may only be utilized as the quasi-supervision signal in response to determining one or more conditions are satisfied. The one or more conditions can include, for example, whether one or more of the predicted outputs134A satisfy a predicted output threshold, one or more of the benchmark outputs satisfy a benchmark output threshold, and/or other conditions. Put another way, the one or more benchmark outputs may only be utilized as the quasi-supervision signal in response to determining that the one or more benchmark outputs provide a viable supervision signal for the one or more predicted outputs134A.

For example, again assume that the on-device audio-based ML model and the global audio-based ML model are corresponding multilingual automatic speech recognition (ASR) models. In this example, the on-device ML engine134can process, using the on-device multilingual ASR model, the stream of audio data124to generate recognized text in the given language as the one or more predicted outputs134A. Further, the learning engine138can process, using a benchmark multilingual ASR model, the stream of audio data124to generate benchmark recognized text as the one or more benchmark outputs. In this example, and assuming the one or more conditions are satisfied, the gradient engine136can compare the recognized text and the benchmark recognized text in generating the gradient136A.

In additional or alternative versions of those implementations, one or more of the unsupervised or semi-supervised learning techniques may correspond to a masking technique. In implementing the masking technique, the learning engine138may identify a target portion of the stream of audio data124that is less than all of the audio data included in the stream of audio data124. Accordingly, the target portion of the stream of audio data124may be proximate to a prepended portion of the stream of audio data124that precedes the target portion of the stream of audio data124and/or an appended portion of the stream of audio data124that follows the target portion of the stream of audio data124. The target portion of the stream of audio data124may be selected arbitrarily, or selected based on one or more criteria such as a particular segment between n and m seconds of the stream of audio data124being identified as the target portion and/or any other criteria for selecting the target portion of the stream of audio data124. Further, the learning engine138may mask the target portion of the stream of audio data124. Accordingly, the stream one or more predicted outputs134A may include one or more predictions with respect to the masked target portion of the stream of audio data124.

For example, again assume that the on-device audio-based ML model and the global audio-based ML model are corresponding language representation ML models. In this example, the target portion of the stream of audio data124may correspond to a target audio waveform portion of the stream of audio data124, the prepended portion of the stream of audio data124may correspond to a prepended audio waveform portion that precedes the target audio waveform portion, and the appended portion of the stream of audio data124may correspond to an appended audio waveform portion that is subsequent to the target audio waveform portion. Further, the on-device ML engine134can process, using the on-device language representation ML model, the prepended audio waveform portion that precedes the target audio waveform portion and/or the appended audio waveform portion that is subsequent to the target audio waveform portion to generate the one or more predicted outputs134A, such as a prediction of the target audio waveform portion. Notably, the prediction of the target audio waveform portion may include, for instance, a predicted audio waveform for the target audio waveform, one or more predicted features of the target audio waveform (e.g., a predicted amplitude, a predicted wavelength, a predicted phase, a predicted period, and/or other features), one or more predicted features of the stream of audio data124(e.g., predicted MFCCs, predicted melbank features, and/or other features), and/or other predicted representations of the target portion of the stream of audio data124. Put another way, the on-device ML model engine134may attempt to reconstruct the target audio waveform portion based on processing the prepended audio waveform portion and/or the appended audio waveform portion. In this example, the gradient engine136can compare the predictions made with respect to the target portion of the stream of audio data124to actual features of the stream of audio data124in generating the gradient136A.

Although particular unsupervised or semi-supervised learning techniques are described herein, it should be understood that those techniques are provided for the sake of example and are not meant to be limiting. Rather, it should be understood that any unsupervised or semi-supervised learning technique that may be utilized in generating gradients based on processing the stream of audio data124may be utilized and are contemplated herein.

In some implementations, the gradient136A (and optionally one or more gradients190A generated in the same or similar manner by one or more corresponding additional client devices190that each include the same or similar components and/or engines described with respect to the client device120) may be generated in an ephemeral manner such that the stream of audio data124is not stored in transient memory or storage of the client device120. In some versions of these implementations, the stream of audio data124may be discarded subsequent to generating the gradient136A. Further, the gradient136A may be transmitted to the remote system160in a synchronous manner (e.g., in response to the gradient136A being generated). Put another way, the client device120may cause the stream of audio data124to be processed to generate the gradient136A while it is temporarily available at the client device120(e.g., via the one or more buffers132), and synchronously transmit the gradient136A to the remote system160to reduce memory consumption at the client device120(hence the term “ephemeral learning” due to the ephemeral nature of this learning technique). In additional or alternative implementations, the client device120may update the on-device audio-based ML model that was utilized in generating the gradient136A, and transmit one or more updated on-device weights of the updated on-device audio-based ML model to the remote system160in lieu of the gradient136A.

In additional or alternative implementations, the gradient136A (and optionally one or more gradients190A generated in the same or similar manner by one or more corresponding additional client devices190) may be generated in a federated manner such that the stream of audio data124may not be stored in transient memory or storage of the client device120. In some versions of these implementations, the stream of audio data124may be discarded subsequent to generating the gradient136A. However, the gradient136A may be stored in memory or storage of the client device120and transmitted to the remote system160in an asynchronous manner (e.g., at a temporally distinct time that is subsequent to the gradient136A being generated). Put another way, the client device120may cause the stream of audio data124to be processed to generate the gradient136A, but wait to transmit the gradient136A to the remote system160.

In these implementations, the client device120may determine to implement ephemeral learning or federated learning based on, for example, a connection status of a connection (and optionally based on a connection strength of the connection) between the client device120and the remote system160. For example, the connection status engine140may determine whether the client device120is connected to the remote system160(e.g., over one or more networks, such as one or more local area networks, one or more wide area networks, and/or one or more other networks) and/or a strength of the connection between the client device120and the remote system160(e.g., over one or more of the networks). The connection status engine140may make this determination while the stream of audio data124is transiently stored at the client device120(e.g., in one or more of the buffers132). For instance, in response to the connection status engine140determining that the client device120and the remote system160have a strong and/or stable connection over one or more of networks, the client device120may implement ephemeral learning in generating and transmitting the gradient136A to the remote system160since the connection status enables the gradient136A to be synchronously transmitted to the remote system160. In contrast, in response to the connection status engine140determining that the client device120and the remote system160have a weak and/or unstable connection over one or more of networks, the client device120may implement federated learning in generating and transmitting the gradient136A to the remote system160. These techniques are described in more detail herein (e.g., with respect toFIG.7).

In some implementations, the client device120may only process the stream of audio data124to generate the gradient136A in response to determining that the given language of the stream of spoken utterances captured in the stream of audio data124corresponds to a target language. For example, the language identification engine142may initially process, using one or more language identification models (e.g., stored in the on-device ML model(s) database120A), the stream of audio data124to identify the given language. Further, the language identification engine142may determine whether the given language corresponds to a target language (e.g., stored in target language(s) database142A). Notably, the target language may be one of a plurality of target languages, and may be defined, for example, by a developer to enable the global audio-based ML model to be updated with respect to target language. Put another way, certain languages (e.g., English, Spanish, French, German) may not be of interest since there is a plethora of data available for updating the global audio-based ML model with respect to these languages. Accordingly, these techniques enable the developer to target the specific target languages that are of interest (e.g., the Swazi language of South Africa) since there is little to no data for updating the global audio-based ML model with respect to these languages. These techniques are described in more detail herein (e.g., with respect toFIG.5).

In some implementations, the client device120may only process the stream of audio data124to generate the gradient136A in response to determining that the stream of audio data124has not been previously utilized in generating a gradient for updating the global audio-based ML model or has not has not been previously utilized more than a threshold quantity of instances in generating a gradient for updating the global audio-based ML model. For example, the deduplication engine144may initially process the stream of audio data124to generate an audio-fingerprint for the stream of audio data124. Further, the deduplication engine144may determine whether the audio-fingerprint matches a previously generated audio-fingerprint (e.g., stored in audio-fingerprint(s) database144A). Notably, the audio-fingerprint may correspond to an embedding, an audio hash, and/or any other representation of the stream of audio data124that enables the stream of audio data124to be compared to other streams of audio data. Put another way, the deduplication engine144may be utilized to ensure that the global audio-based ML model is not continuously updated based on the same stream of audio data (e.g., a commercial on the given radio station) to prevent overfitting of the global audio-based ML model to the same stream of audio data, and to ensure diversity of the underlying streams of audio data that are utilized in generating gradients. These techniques are described in more detail herein (e.g., with respect toFIG.5).

In various implementations, and assuming that the gradient136A is transmitted to the remote system160(e.g., in a synchronous or asynchronous manner) and assuming that the one or more gradients190A generated in the same or similar manner by the one or more corresponding additional client devices190, the remote system160may update the global audio-based ML model using a remote update engine180and subsequently distribute updated audio-based ML model(s)182A to the client device120(and optionally one or more of the additional client devices190) using an update distribution engine182. In some versions of those implementations, the remote system160may update the global audio-based ML model in a streaming manner (e.g., as the gradient136A and the one or more gradients190A are received from the respective client devices). In additional or alternative versions of those implementations, the remote system160may store the gradient136A and the one or more gradients190A in one or more databases (e.g., gradient(s) database1768), and may update the global audio-based ML model in response to determining that one or more conditions for updating the global audio-based ML model are satisfied. The one or more conditions for updating the global audio-based ML model may include, for example, a particular time of day, a particular day of week, whether a threshold quantity of gradients are available to update the global audio-based ML model, and/or other conditions.

In updating the global audio-based ML model, the remote system160may update one or more weights of the global audio-based ML model based on the gradient136A and the one or more gradients190A are received from the respective client devices. In some implementations, the remote update engine180can utilize a gradient descent algorithm to update one or more of the global weights. In some versions of those implementations, the remote update engine180may average the gradient136A and the one or more gradients190A that are received from the respective client devices prior to utilizing the gradient descent algorithm prior to updating one or more of the global weights. In additional or alternative versions of those implementations, the remote update engine180may utilize each of the gradient136A and the one or more gradients190A that are received from the respective client devices, or a subset thereof, to update one or more of the global weights using the gradient descent algorithm.

In distributing the updated audio-based ML model(s)182A to the client device120(and optionally one or more of the additional client devices190), the update distribution engine182may determine whether one or more conditions for the updated audio-based ML model(s)182A (or one or more of the global weights thereof) are satisfied. The one or more conditions can be based on whether the client device120is ready to receive the client device120, such as whether the client device120is charging, whether the client device120has at least a threshold state of charge, whether a temperature of the client device120(based on one or more corresponding on-device temperature sensors) is less than a threshold, whether the client device120is being held by a user, temporal condition(s) associated with the client device120(e.g., between a particular time period, every N hours, where N is a positive integer, and/or other temporal conditions), and/or other conditions. Further, the one or more conditions can additionally, or alternatively, be based on other conditions that are specific to the remote system160, such as whether performance of the updated audio-based ML model(s)182A satisfying a performance threshold, whether the updated audio-based ML model(s)182A has updated based on a threshold quantity of gradients, and/or other conditions.

Accordingly, when the client device120(and optionally one or more of the additional client devices190) receives the updated audio-based ML model(s)182A (or one or more of the global weights thereof), the on-device audio-based ML model (or one or more of the on-device weights thereof) may be replaced with the updated audio-based ML model(s)182A (or one or more of the global weights thereof). This process may be repeated to continue updating the global audio-based ML model. In some implementations, this process may be repeated to continue updating the global audio-based ML model until it is determined that the global audio-based ML model has converged.

Referring specifically toFIG.1B, again assume that the stream of audio data124is received at the client device120. However, inFIGS.1B and1ncontrast withFIG.1B, the ephemeral learning is performed by the remote system160and not locally at the client device120and/or one or more of the additional client devices190. Put another way, inFIGS.1B and1ncontrast withFIG.1B, the client device120and/or one or more of the additional client devices190are utilized as a proxy for the remote system to obtain the stream of audio data124from the client device120and one or more streams of audio data190B from a corresponding one of the additional client devices190.

Notably, inFIG.1B, the remote system160includes remote-based counterparts of many of the components and engines described with respect toFIG.1A. For example, one or more buffers172of the remote systemFIG.1Bmay be utilized in the same or similar manner described above with respect to the one or more buffers132of the client device120; global ML model engine174of the remote system160may be utilized to generate one or more predicted output174A in the same or similar manner described with respect to the on-device ML model engine134, but using the global audio-based ML model stored in the global ML model(s) database160A; gradient engine176of the remote system160may be utilized to generate a gradient176A at the remote system160in the same or similar manner described with respect to the gradient engine136of the client device; learning engine178of the remote system160may cause one or more supervised, unsupervised, or semi-supervised learning techniques to be utilized in generating the gradient176A in the same or similar manner described with respect to the learning engine138of the client device120; language identification engine184of the remote system160may be utilized in the same or similar manner described with respect to the language identification engine142of the client device120, but utilize remote memory or storage (e.g., target language(s) database184A) to determine whether the given language of the stream of spoken utterances captured in the stream of audio data124corresponds to a target language; and deduplication engine186of the remote system160may be utilized in the same or similar manner described with respect to the deduplication engine144of the client device, but utilize remote memory or storage (e.g., audio-fingerprint(s) database186A) to determine whether the stream of audio data124has been previously utilized in updating the global audio-based ML model. However, it should be noted that the remote system160does not include a remote counterpart to the connection status engine140since this determination may be made exclusively at the client device120. Accordingly, it should be understood that the ephemeral learning described herein may be performed locally at the client device120, remotely from the client device120(e.g., at the remote system160), or both.

Turning now toFIG.2, a client device250is illustrated in an implementation where various on-device ML engines are included as part of (or in communication with) an automated assistant client240is depicted. The respective ML models are also illustrated interfacing with the various on-device ML engines. Other components of the client device250are not illustrated inFIG.2for simplicity.FIG.2illustrates one example of how the various on-device ML engines of and their respective ML models can be utilized by the automated assistant client240in performing various actions.

The client device250inFIG.2is illustrated with one or more microphones211, one or more speakers212, one or more vision components213, and display(s)214(e.g., a touch-sensitive display). The client device250may further include pressure sensor(s), proximity sensor(s), accelerometer(s), magnetometer(s), and/or other sensor(s) that are used to generate other sensor data that is in addition to audio data captured by the one or more microphones211. The client device250at least selectively executes the automated assistant client240. The automated assistant client240includes, in the example ofFIG.2, language representation engine222, voice activity detection engine224, hotword detection engine226, ASR engine228, multilingual ASR engine230, continued conversation engine232, language identification engine234, and voice identification engine236. The automated assistant client240further includes speech capture engine216and visual capture engine218. It should be understood that the ML engines and ML models depicted inFIG.2are provided for the sake of example, and are not meant to be limiting. For example, the automated assistant client240can further include additional and/or alternative engines, such as a text-to-speech (TTS) engine and a respective TTS model, an endpoint detector engine and a respective endpoint detector model, and/or other engine(s) along with associated ML model(s). Moreover, it should be understood that one or more of the engines and/or models described herein can be combined, such that a single engine and/or model can perform the functions of multiple engines and/or models described herein.

One or more cloud-based automated assistant components270can optionally be implemented on one or more computing systems (collectively referred to as a “cloud” computing system) that are communicatively coupled to client device250via one or more networks as indicated generally by299. The cloud-based automated assistant components270can be implemented, for example, via a cluster of high-performance servers. In various implementations, an instance of the automated assistant client240, by way of its interactions with one or more of the cloud-based automated assistant components270, may form what appears to be, from a user's perspective, a logical instance of an automated assistant as indicated generally by295with which the user may engage in a human-to-computer interactions (e.g., spoken interactions, gesture-based interactions, and/or touch-based interactions).

The client device250can be, for example: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a vehicle of the user (e.g., an in-vehicle communications system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker, a smart appliance such as a smart television (or a standard television equipped with a networked dongle with automated assistant capabilities), and/or a wearable apparatus of the user that includes a computing device (e.g., a watch of the user having a computing device, glasses of the user having a computing device, a virtual or augmented reality computing device). Additional and/or alternative client devices may be provided.

The one or more vision components213can take various forms, such as monographic cameras, stereographic cameras, a LIDAR component (or other laser-based component(s)), a radar component, etc. The one or more vision components213may be used, e.g., by the visual capture engine218, to capture image data corresponding to vision frames (e.g., image frames, laser-based vision frames) of an environment in which the client device250is deployed. In some implementations, such vision frame(s) can be utilized to determine whether a user is present near the client device250and/or a distance of a given user of the client device250(e.g., the user's face) relative to the client device250. Such determination(s) can be utilized, for example, in determining whether to activate the various on-device ML engines depicted inFIG.2, and/or other engine(s). Further, the speech capture engine218can be configured to capture a user's spoken utterance(s) and/or other audio data captured via the one or more of the microphones211.

As described herein, streams of audio data can be processed by the various engines depicted inFIG.2to make predictions at the client device250using corresponding ML models and/or at one or more of the cloud-based automated assistant components270using corresponding ML models updated in the manner described herein (e.g., with respect toFIGS.1A,1B, and3-7).

As some non-limiting examples, the respective language representation engines222,272can utilize respective language representation models222A,272A to generate a rich feature representation of a stream of audio data and/or a stream of spoken utterances captured in the stream of audio data; the respective voice activity detection engines224,274can utilize respective voice activity detection models224A,274A to predict whether a stream of audio data includes voice activity of a user of the client device250and/or other users; the respective hotword detection engines226,276can utilize respective language representation models226A,276A to predict whether a stream of audio data includes one or more particular words or phrases to invoke the automated assistant295(e.g., “Ok Assistant”, “Hey Assistant”, “What is the weather Assistant?”, etc.) or certain functions of the automated assistant295); the respective ASR engines228,278can utilize a respective ASR model228A,278A to generate recognized text for a given language based on processing a stream of audio data, or predict phoneme(s) and/or token(s) that correspond to the stream of audio data detected at the client device250and generate the recognized text for the given language based on the phoneme(s) and/or token(s); the respective multilingual ASR engines230,280can utilize a respective ASR model230A,280A to generate recognized text for a plurality of languages based on processing a stream of audio data, or predict phoneme(s) and/or token(s) that correspond to the stream of audio data detected at the client device250and generate the recognized text for the plurality of languages based on the phoneme(s) and/or token(s); the respective continued conversation engines232,282can utilize a respective continued conversation model232A,282A to predict whether further streams of audio data is directed to the automated assistant295(e.g., or directed to an additional user in the environment of the client device250); the respective language identification engines234,284can utilize a respective language identification model234A,284A to predict a given language of a stream of spoken utterances captured in a stream of audio data; and the respective voice identification engines236,286can utilize a respective voice identification model236A,286A to predict whether a stream of audio data captures a stream of spoken utterances of one or more users of the client device250(e.g., by generating a speaker embedding, or other representation, that can be compared to a corresponding actual embeddings for one or more of the user of the client device250).

In some implementations, the client device250and one or more of the cloud-based automated assistant components270may further include natural language understanding (NLU) engines238,294and fulfillment engine240,296, respectively. The NLU engines238,294may perform natural language understanding, utilizing respective NLU models238A,294A, on recognized text, predicted phoneme(s), and/or predicted token(s) generated by, for instance, the ASR engines228,278and/or the multilingual ASR engines230,280to generate NLU data. The NLU data can include, for example, intent(s) that correspond to the spoken utterance and optionally slot value(s) for parameter(s) for the intent(s). Further, the client device250and one or more of the cloud-based automated assistant components270may further include fulfillment engines240,296, respectively. The fulfillment engines240,296can generate fulfillment data utilizing respective fulfillment models or rules240A,296A, and based on processing the NLU data. This fulfillment data can define certain fulfillment that is responsive to user input (e.g., spoken utterances, typed input, touch input, gesture input, and/or any other type of user input) provided by a user of the client device250. The certain fulfillment can include interaction(s) to perform with locally installed application(s) based on the user input, command(s) to transmit to Internet-of-things (loT) device(s) (directly or via corresponding remote system(s)) based on the user input, and/or other resolution action(s) to perform based on the user input. The fulfillment data is then provided for local and/or remote performance/execution of the determined action(s) to cause the certain fulfillment of the user input to be performed. Execution can include, for example, rendering local and/or remote responses (e.g., visually and/or audibly rendering (optionally utilizing an on-device TTS module)), interacting with locally installed applications, transmitting command(s) to loT device(s), and/or other action(s). In other implementations, the NLU engines238,294and the fulfillment engines240,296may be omitted, and the ASR engines228,278and/or the multilingual ASR engines230,280can generate the fulfillment data directly based on the user input. For example, assume the ASR engines228,278and/or the multilingual ASR engines230,280processes, using the respective models, a stream of audio data that captures a spoken utterance of “turn on the lights.” In this example, the ASR engines228,278and/or the multilingual ASR engines230,280can generate a semantic output that is then transmitted to a software application associated with the lights and/or directly to the lights that indicates that they should be turned on.

Notably, the cloud-based automated assistant component(s)270include cloud-based counterparts to the engines and models described herein with respect toFIG.2. However, in some implementations, these engines and models may not be utilized since the engines and models may be transmitted directly to the client device250and executed locally at the client device250, whereas in other implementations, these engines and models may be utilized exclusively when the client device250detects any user input and transmits the user input to the cloud-based automated assistant component(s)270. In various implementations, these engines and models executed at the client device250the cloud-based automated assistant component(s)270may be utilized in conjunction with one another in a distributed manner. Nonetheless, a remote execution module can optionally be included that performs remote execution based on local or remotely generated NLU data and/or fulfillment data. Additional and/or alternative remote engines can be included. As described herein, in various implementations on-device speech processing, on-device image processing, on-device NLU, on-device fulfillment, and/or on-device execution can be prioritized at least due to the latency and/or network usage reductions they provide when resolving a spoken utterance (due to no client-server roundtrip(s) being needed to resolve the spoken utterance). However, one or more cloud-based automated assistant component(s)270can be utilized at least selectively. For example, such component(s) can be utilized in parallel with on-device component(s) and output from such component(s) utilized when local component(s) fail. For example, if any of the on-device engines and/or models fail (e.g., due to relatively limited resources of client device250), then the more robust resources of the cloud may be utilized.

Turning now toFIG.3, a flowchart illustrating an example method300of ephemeral learning implemented by a client device and based on a stream of audio generated by a given radio station is depicted. For convenience, the operations of the method300are described with reference to a system that performs the operations. The system of method300includes one or more processors and/or other component(s) of a client device (e.g., client device120ofFIGS.1A and1B, client device250ofFIG.2, computing device810ofFIG.8, and/or other client devices). Moreover, while operations of the method300are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

At block352, the system receives, from a given radio station being actively consumed by a user of a client device, a stream of audio data that captures a stream of spoken utterances in a given language. The stream of spoken utterances may capture, for instance, spoken utterances that include commercial or advertisement content from the given radio station, spoken utterances that include disc jockey content from the given radio station, spoken utterances that include podcast content from the given radio station, spoken utterance that include music content from the given radio station, and/or other sources of content from the given radio station. However, in some implementations, the system may discard any streams of audio data that capture spoken utterances that include music content from the given radio station. In some implementations, the stream of audio data may be of a fixed length (e.g., 5 seconds, 10 seconds, 15 seconds, etc.). In additional or alternative implementations, the stream of audio data may be of a dynamic length (e.g., a length of commercial or advertisement content, a length of disc jockey, a length of podcast content, etc.).

At block354, the system causes a gradient, for updating a global machine learning (ML) model with respect to the given language, to be generated. The system may return to block352to block352to continue receiving additional streams of audio data that capture additional streams of spoken utterances from the given radio station that is being actively consumed by the user of the client device. This enables the system to perform multiple iterations of the method300ofFIG.3in parallel as streams of audio data are received from the given radio station that is being actively consumed by the user of the client device. Notably, at block354, the gradient is not yet generated. Rather, at block354, the system may perform various operations to determine whether to generate the gradient based on the stream of audio data (e.g., audio-fingerprint operations as described with respect toFIG.5, language identification operation as described with respect toFIG.5, and/or other operations described herein).

At block356, the system may determine whether to generate the gradient locally at the client device or remotely from the client device. In some implementations, the system may determine to generate the gradient locally at the client device by default. In other implementations, the system may determine to generate the gradient remotely from the client device by default. In yet other implementations, the system may determine to generate the gradient locally at the client device in some instances, but generate the gradient remotely from the client device in other instances. For example, the system may prioritize generating the gradient locally at the client device to reduce a quantity of network resources consumed in transmitting the stream of audio data to a remote system, but begin transmitting the stream of audio data to the remote system in response to determining that a threshold quantity of computational resources are being consumed at the client device. Also, for example, the system may periodically switch between causing the gradient to be generated locally at the client device and remotely from the client device to more evenly distribute the computational resources being consumed by the client device and the remote system. It should be understood that these examples are not meant to be limiting, and that any other criteria for determining whether to generate the gradient locally at the client device or remotely from the client device are contemplated herein.

If, at an iteration of block356, the system determines to generate the gradient locally at the client device, then the system may proceed to block358. At block358, the system processes, using an on-device ML model that is an on-device counterpart of the global ML model and that is stored in on-device memory or storage of the client device, the stream of audio data. At block360, the system generates, using an unsupervised or self-supervised learning technique, and based on processing the stream of audio data using the on-device ML model, the gradient. At block362, the system discards the stream of audio data. At block364, the system transmits the gradient to the remote system to cause the remote system to update the global ML model based on the gradient. Put another way, the system may cause the operations of the process flow described with respect toFIG.1Ato be performed locally at the client device.

If, at an iteration of block356, the system determines to generate the gradient remotely from the client device, then the system may proceed to block366. At block366, the system transmits the stream of audio data to the remote system to cause the remote system to update the global ML model based on processing the stream of audio data. Put another way, rather than the system causing the gradient to be generated locally at the client device based on the stream of audio data being processed locally at the client device, the system may cause the stream of audio data to be transmitted to the remote system to enable the remote system to process the stream of audio data and generate the gradient based on the processing of the stream of audio data (e.g., as described with respect toFIG.1B). In doing so, the system may cause the remote system to perform operations. For example, at block366A, the system causes the remote system to process, using the global ML model, the stream of audio data. Further, at block3668, the system causes the remote system to generate, using an unsupervised or self-supervised learning technique, and based on processing the stream of audio data, the gradient. Moreover, at block366C, the system causes the remote system to discard the stream of audio data. Additionally, at block366D, the system causes the remote system to update, based on the gradient, the global ML model.

Accordingly, the system may initially receive the stream of audio data at the client device and may determine whether to generate the gradient to be utilized in updating the global ML model locally at the client device or remotely from the client device. In making this determination, the system may consider computational resources that would be consumed at the client device in generating the gradient and/or network resources that would be consumed in transmitting the stream of audio data to the remote system to reduce consumption of the computational and/or network resources while still enabling the system to cause the global ML model to be updated based on diverse streams of audio data generated by radio stations across the world. As a result, the global ML models that are updated in this manner are more robust to processing and/or understanding more languages of users across the world.

Turning now toFIG.4, a flowchart illustrating an example method400of ephemeral learning implemented by a remote system and based on a stream of audio generated by a given radio station is depicted. For convenience, the operations of the method400are described with reference to a system that performs the operations. The system of method400includes one or more processors and/or other component(s) of a remote system (e.g., remote system160ofFIGS.1A and1B, cloud-based automated assistant component(s)270ofFIG.2, computing device810ofFIG.8, and/or other remote systems). Moreover, while operations of the method400are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

At block452, the system receives, from a given client device, a stream of audio data that captures a stream of spoken utterances in a given language, the stream of audio data being initially received at the given client device and from a given radio station that is being actively consumed by a user of the given client device. At block454, the system processes, using a global ML model, the stream of audio data. The system may return to block452to continue receiving additional streams of audio data. At block456, the system generates, using an unsupervised or self-supervised learning technique, and based on processing the stream of audio data, a gradient. At block458, the system discards the stream of audio data. At block460, the system updates, based on the gradient, the global ML model with respect to the given language.

Put another way, in implementations according to the method400ofFIG.4, the system may utilize the given client device as a proxy for obtaining the stream of audio data. Further, the system may process the stream of audio data to generate a gradient for utilization in updating the global ML model (e.g., as described with respect toFIG.1B). This enables ephemeral learning to not only be performed locally at client devices, but also at remote systems. In additional or alternative implementations, the system may obtain the stream of audio data directly from the given radio station to further reduce consumption of network resources.

Although the method400ofFIG.4is described in a particular manner, it should be understood that is for the sake of example, and is not meant to be limiting. For example, the system may perform multiple iterations the method400ofFIG.4in parallel and with respect to different streams of audio data that are received from the given client device and/or additional client devices. As another example, the system may initially process the stream of audio data to determine whether to generate the gradient for utilization in updating the global ML model (e.g., as described with respect toFIG.6).

Turning now toFIG.5, a flowchart illustrating an example method500of deduplication techniques utilized in ephemeral learning implemented by a client device is depicted. For convenience, the operations of the method500are described with reference to a system that performs the operations. The system of method500includes one or more processors and/or other component(s) of a client device (e.g., client device120ofFIGS.1A and1B, client device250ofFIG.2, computing device810ofFIG.8, and/or other client devices). Moreover, while operations of the method500are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

At block552, the system receives, from a given radio station, a stream of audio data that captures a stream of spoken utterances in a given language. The stream of spoken utterances may capture, for instance, spoken utterances that include commercial or advertisement content from the given radio station, spoken utterances that include disc jockey content from the given radio station, spoken utterances that include podcast content from the given radio station, spoken utterance that include music content from the given radio station, and/or other sources of content from the given radio station. However, in some implementations, the system may discard any streams of audio data that capture spoken utterances that include music content from the given radio station. In some implementations, the stream of audio data may be of a fixed length (e.g., 5 seconds, 10 seconds, 15 seconds, etc.). In additional or alternative implementations, the stream of audio data may be of a dynamic length (e.g., a length of commercial or advertisement content, a length of disc jockey, a length of podcast content, etc.). In some implementations, the stream of audio data may correspond to a stream of audio data that is being actively consumed by the user of the client device while the user is actively listening to the given radio station. In additional or alternative implementations, the stream of audio data may correspond to a stream of audio data that is accessible at the client device via the given radio station, but is not being actively consumed by the user of the client device.

At block554, the system generates, based on processing the stream of audio data, an audio-fingerprint for the stream of audio data. In some implementations, the audio-fingerprint for the stream of audio data may correspond to an embedding. In some versions of these implementations, the system may process, using an encoder-decoder ML model or another ML model (e.g., stored in the on-device ML model(s) database120A ofFIGS.1A and1B), the stream of audio data to generate the embedding (or other lower dimensional representation of the audio data). Further, the embedding may be mapped to an embedding space (or another lower dimensional latent space) that enables the embedding to be compared to a plurality of previously generated embeddings (e.g., stored in the audio-fingerprint(s) database144A ofFIG.1A) for previously encountered streams of audio data (e.g., as described with respect to block556). In additional or alternative implementations, the audio-fingerprint for the stream of audio data may correspond to an audio hash. In some versions of these implementations, the system may process, using a local sensitivity hash (e.g., stored in the on-device ML model(s) database120A ofFIGS.1A and1B), to generate the audio hash (e.g., a vector representation of the stream of audio data). Further, this enables the audio hash to be compared to a plurality of previously generated audio hashes (e.g., stored in the audio-fingerprint(s) database144A ofFIG.1A) for previously encountered streams of audio data (e.g., as described with respect to block556).

At block556, the system determines whether the stream of audio data was previously utilized in generating a gradient for updating a global ML model. For example, and in implementations where the audio-fingerprint corresponds to the embedding, if the embedding generated based on the stream of audio data matches a given previously generated embedding for a given previously encountered stream of audio data, then the system may determine that the stream of audio data was previously utilized in generating a gradient for updating the global ML model. For instance, the system may determine that the embedding generated based on the stream of audio data matches the given previously generated embedding if the embedding and the given previously generated embedding are within a threshold distance in the embedding space (e.g., using Euclidean distance, cosine similarity, and/or another distance measure). Otherwise, the system may determine that the embedding generated based on the stream of audio data does not match any given previously generated embedding.

As another example, and in implementations where the audio-fingerprint corresponds to the audio hash, if the audio hash generated based on the stream of audio data matches a given previously generated audio hash for a given previously encountered stream of audio data, then the system may determine that the stream of audio data was previously utilized in generating a gradient for updating the global ML model. For instance, the system may determine that the audio hash generated based on the stream of audio data matches the given previously generated audio hash if the corresponding vectors representing the streams of audio data satisfy a similarity threshold. Otherwise, the system may determine that the embedding generated based on the stream of audio data does not match any given previously generated embedding.

Notably, in these implementations, a remote system that is communicatively coupled to a population of client devices (e.g., that includes the client device) may maintain and periodically distribute the database of audio-fingerprints (e.g., the audio-fingerprints database186A ofFIG.1B). For instance, each of the client device of the population may generate and transmit the audio-fingerprints generated locally at the client devices to the remote system, and the remote system may store the audio-fingerprints in the database of audio-fingerprints, and along with any audio-fingerprints generated at the remote system. Further, the remote system may distribute the database of audio-fingerprints (or updates thereto) to each of the client devices of the population to ensure that respective databases of audio-fingerprints (e.g., the audio-fingerprints database144A ofFIG.1A) are up-to-date. This ensures that the population of client devices does not continually generate gradients based on the same streams of audio data and ensures sufficient diversity of streams of audio data in generating gradients that are utilized in updating the global ML model.

If, at an iteration of block556, the system determines that the stream of audio data was previously utilized in generating a gradient for updating a global ML model, then the system may proceed to block558. At block558, the system refrains from any further processing of the stream of audio data. At block560, the system discards the stream of audio data. The system may return to block552to perform an additional iteration of the method500.

If, at an iteration of block556, the system determines that the stream of audio data was previously utilized in generating a gradient for updating a global ML model, then the system may proceed to block562. At block562, the system determines whether the given language corresponds to a target language. For example, the system may process, using a language identification model (e.g., stored in the on-device ML model(s) database120A), the stream of audio data (or recognized text generated based on processing the stream of audio data (e.g., generated using a multilingual ASR model)) to identify the given language associated with the stream of spoken utterances captured in the stream of audio data. Further, the system may compare the given language to one or more target languages (e.g., stored in the target language(s) database142A ofFIG.1) to determine whether to generate a gradient to be utilized in updating the global ML with respect to the given language. Put another way, the system may discard streams of audio data that capture spoken utterances in some languages, while continuing to process streams of audio data that capture spoken utterances in other languages.

If, at an iteration of block562, the system determines that the given language does not correspond to the target language, then the system may proceed to block558. At block558, the system refrains from any further processing of the stream of audio data. At block560, the system discards the stream of audio data. The system may return to block552to perform an additional iteration of the method500. If, at an iteration of block562, the system determines that the given language corresponds to the target language, then the system may proceed to block356of the method300ofFIG.3to determine whether to perform ephemeral learning locally at the client device or remotely at the client device. Put another way, the system may cause the client device to perform these deduplication techniques to reduce a quantity of computational and/or network resources that would otherwise be consumed in processing streams of audio data that have been previously processed. Further, and assuming that the system determines that the gradient has not been previously utilized in generating a gradient for utilization in updating the global ML model, the system may implement ephemeral learning locally at the client device and remotely from the client device.

Although the method500ofFIG.5is described in a particular manner, it should be understood that is for the sake of example, and is not meant to be limiting. For example, the deduplication operations of block556and block562may be switched such that the language identification deduplication technique is performed prior to the audio-fingerprinting deduplication technique. Also, for example, one of the deduplication operations of block556and block562may be omitted such that only the language identification deduplication technique is performed or only the audio-fingerprinting deduplication technique is performed.

Turning now toFIG.6, a flowchart illustrating an example method600of deduplication techniques utilized in ephemeral learning implemented by a remote system is depicted. For convenience, the operations of the method600are described with reference to a system that performs the operations. The system of method600includes one or more processors and/or other component(s) of a remote system (e.g., remote system160ofFIGS.1A and1B, cloud-based automated assistant component(s)270ofFIG.2, computing device810ofFIG.8, and/or other remote systems). Moreover, while operations of the method600are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

At block652, the system receives, from a given client device, a stream of audio data that captures a stream of spoken utterances in a given language, the stream of audio data being initially received at the given client device and from a given radio station. At block654, the system generates, based on processing the stream of audio data, an audio-fingerprint for the stream of audio data

At block656, the system determines whether the stream of audio data was previously utilized in generating a gradient for updating a global ML model. If, at an iteration of block656, the system determines that the stream of audio data was previously utilized in generating a gradient for updating a global ML model, then the system may proceed to block658. At block658, the system refrains from any further processing of the stream of audio data. At block660, the system discards the stream of audio data. The system may return to block652to perform an additional iteration of the method600. If, at an iteration of block656, the system determines that the stream of audio data was previously utilized in generating a gradient for updating a global ML model, then the system may proceed to block662.

At block662, the system determines whether the given language corresponds to a target language. If, at an iteration of block662, the system determines that the given language does not correspond to the target language, then the system may proceed to block658. At block658, the system refrains from any further processing of the stream of audio data. At block660, the system discards the stream of audio data. The system may return to block652to perform an additional iteration of the method600. If, at an iteration of block662, the system determines that the given language corresponds to the target language, then the system may proceed to block456of the method400ofFIG.4and utilize ephemeral learning at the remote system to generate the gradient for updating the global ML model.

Put another way, the method600ofFIG.6is the same or similar as the method500ofFIG.5, but the system is implemented by the remote system according to the method600ofFIG.6, rather than the client device according to the method500ofFIG.5. Notably, in these implementations, the stream of audio data may be initially received by the given client device, but the operations of the method600ofFIG.6may be performed by components that are specific to the remote system. Moreover, although the method600ofFIG.6is described with respect to a single client device and a single stream of audio data, it should be understood that is for the sake of example and is not meant to be limiting. Rather, it should be understood that the system may cause multiple iterations of the method600ofFIG.6to be performed in parallel based on different streams of audio data received from different client devices.

Turning now toFIG.7, a flowchart illustrating an example method700of determining whether a client device should implement an ephemeral learning technique or a federated learning technique is depicted. For convenience, the operations of the method700are described with reference to a system that performs the operations. The system of method700includes one or more processors and/or other component(s) of a client device (e.g., client device120ofFIGS.1A and1B, client device250ofFIG.2, computing device810ofFIG.8, and/or other client devices). Moreover, while operations of the method700are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

At block752, the system receives, from a given radio station, a stream of audio data that captures a stream of spoken utterances in a given language (e.g., in the same or similar manner described with respect to block552of the method500ofFIG.5).

At block754, the system determines whether to implement federated learning or ephemeral learning in generating a gradient for updating a global ML model. For example, the system may determine whether to implement federated learning or ephemeral learning based on a connection status between a client device that received the stream of audio data and a remote system and/or a connection strength between the client device and the remote system (e.g., as described with respect to the connection status engine140ofFIG.1A). Additionally, or alternatively, the system may determine whether to implement federated learning or ephemeral learning based on a location of the client device. For example, if the location of the client device is located within a particular geographical region, then the system may determine to implement federated learning rather than ephemeral learning.

If, at an iteration of block754, the system determines to implement federated learning, then the system may proceed to block756. At block756, the system processes, using an on-device ML model that is an on-device counterpart of the global ML model, the stream of audio data. At block758, the system generates, using an unsupervised or self-supervised learning technique, and based on processing the stream of audio data using the on-device ML model, the gradient. At block760, the system asynchronously transmits the gradient to the remote system to cause the remote system to update the global ML model based on the gradient. The operations of block756and block758may be performed in the same or similar manner described with respect to the operations of block358and block360of the method300ofFIG.3, respectively.

However, and in contrast with the operations of block364of the method300ofFIG.3, the transmitting of the gradient to the remote system at block760is asynchronous. In these implementations, the system may have determined to implement federated learning to generate the gradient based on a weak and/or unstable connection between the client device and the remote system. Accordingly, rather than attempting to transmit the gradient to the remote system in response to the system causing the gradient to be generated locally at the client device, the system may store the gradient in transient memory or storage of the client device due the weak and/or unstable connection between the client device and the remote system. Nonetheless, the system may cause the gradient to be transmitted to the remote system to cause the remote system to update the global ML model based on the gradient when a strong and/or stable connection is subsequently established between the client device and the remote system (e.g., over a Wi-Fi network, a cellular network, etc.). In some versions of these implementations, the stream of audio data may be discarded in response to the gradient being generated, whereas in other versions of these implementations, the system may store the stream of audio data in the transient memory or storage of the client device until the gradient is transmitted to the remote system in the asynchronous manner and then discard the stream of audio data.

If, at an iteration of block754, the system determines to implement ephemeral learning, then the system may proceed to block356of the method300ofFIG.3. Put another way, the system may cause the client device to determine whether to implement ephemeral learning or federated learning in generating the gradient. Further, and assuming that the system determines to implement ephemeral learning based on a strong and/or stable connection between the client device and the remote system, the system may proceed to block356of the method300ofFIG.3to determine whether to generate the gradient locally at the client device or remotely from the client device. In implementations where the system determines to implement ephemeral learning locally at the client device at an iteration of block356, the system may transmit the gradient to the remote system in a synchronous manner at an iteration of block364(e.g., in response to the gradient being generated) and without storing the gradient in the transient memory or storage of the client device. In implementations where the system determines to implement ephemeral learning remotely from the client device at an iteration of block356, the system may transmit the stream of audio data to the remote system in a synchronous manner at an iteration of block366.

Although the method700ofFIG.7and the method500ofFIG.5are described as separate methods, it should be understood that is for the sake of illustrating techniques described herein and is not meant to be limiting. For example, techniques described with respect to the method700ofFIG.7and the method500ofFIG.5(or some aspects thereof) may be combined into a single method.

Turning now toFIG.8, a block diagram of an example computing device810that may optionally be utilized to perform one or more aspects of techniques described herein is depicted. In some implementations, one or more of a client device, cloud-based automated assistant component(s), and/or other component(s) may comprise one or more components of the example computing device810.

Computing device810typically includes at least one processor814which communicates with a number of peripheral devices via bus subsystem812. These peripheral devices may include a storage subsystem824, including, for example, a memory subsystem825and a file storage subsystem826, user interface output devices820, user interface input devices822, and a network interface subsystem816. The input and output devices allow user interaction with computing device810. Network interface subsystem816provides an interface to outside networks and is coupled to corresponding interface devices in other computing devices.

User interface input devices822may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as voice recognition systems, microphones, and/or other types of input devices. In general, use of the term “input device” is intended to include all possible types of devices and ways to input information into computing device810or onto a communication network.

User interface output devices820may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of the term “output device” is intended to include all possible types of devices and ways to output information from computing device810to the user or to another machine or computing device.

Storage subsystem824stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem824may include the logic to perform selected aspects of the methods disclosed herein, as well as to implement various components depicted inFIGS.1A,1B, and2.

These software modules are generally executed by processor814alone or in combination with other processors. Memory825used in the storage subsystem824can include a number of memories including a main random access memory (RAM)830for storage of instructions and data during program execution and a read only memory (ROM)832in which fixed instructions are stored. A file storage subsystem826can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations may be stored by file storage subsystem826in the storage subsystem824, or in other machines accessible by the processor(s)814.

Bus subsystem812provides a mechanism for letting the various components and subsystems of computing device810communicate with each other as intended. Although bus subsystem812is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

Computing device810can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device810depicted inFIG.8is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing device810are possible having more or fewer components than the computing device depicted inFIG.8.

In situations in which the systems described herein collect or otherwise monitor personal information about users, or may make use of personal and/or monitored information), the users may be provided with an opportunity to control whether programs or features collect user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current geographic location), or to control whether and/or how to receive content from the content server that may be more relevant to the user. Also, certain data may be treated in one or more ways before it is stored or used, so that personal identifiable information is removed. For example, a user's identity may be treated so that no personal identifiable information can be determined for the user, or a user's geographic location may be generalized where geographic location information is obtained (such as to a city, ZIP code, or state level), so that a particular geographic location of a user cannot be determined. Thus, the user may have control over how information is collected about the user and/or used.

In some implementations, a method performed by one or more processors of a client device is provided and includes receiving, from a given radio station, a stream of audio data that captures a stream of spoken utterances in a given language; generating, based on processing the stream of audio data, an audio-fingerprint for the stream of audio data; determining, based on comparing the audio-fingerprint for the stream of audio data to a database of audio-fingerprints, whether the stream of audio data has been previously utilized in generating a gradient for updating a global machine learning (ML) model with respect to the given language; and in response to determining that the stream of audio data has not been previously utilized in generating a gradient for updating the global ML model with respect to the given language: processing, using an on-device ML model that is stored in on-device storage of the client device and that is an on-device counterpart of the global ML model, the stream of audio data; generating, using an unsupervised or self-supervised learning technique, and based on processing the stream of audio data using the on-device ML model, the gradient; and transmitting the gradient to the remote system to be utilized in updating the global ML model with respect to the given language.

These and other implementations of the technology can include one or more of the following features.

In some implementations, the method may further include, in response to determining that the stream of audio data has been previously utilized in generating a gradient for updating the global ML model with respect to the given language: discarding the stream of audio data.

In some implementations, the method may further include receiving, from the remote system, the database of audio-fingerprints; and storing, in the on-device storage of the client device, the database of audio-fingerprints. In some versions of those implementations, the remote system may have been previously generated the database of audio-fingerprints based on a plurality of corresponding streams of audio data that that are received from the client device and a plurality of additional client devices, and that capture corresponding streams of spoken utterances in a plurality of different languages, including the given language, and from a plurality of different radio stations.

In some implementations, generating the audio-fingerprint for the stream of audio data based on processing the stream of audio data may include processing, using a local sensitivity hash, the stream of audio data to generate an audio hash as the audio-fingerprint. In some versions of those implementations, determining whether the stream of audio data has been previously utilized in generating a gradient for updating a global ML model with respect to the given language based on comparing the audio-fingerprint for the stream of audio data to a database of audio-fingerprints may include comparing the audio hash generated based on processing the stream of audio data to a plurality of previously generated audio hashes, and determining, based on the comparing, whether the stream of audio data has been previously utilized in generating a gradient for updating a global ML model with respect to the given language. The plurality of previously generated audio hashes may have been previously generated based on processing corresponding streams of audio data, and the plurality of previously generated audio hashes may be stored in the database of audio-fingerprints.

In some implementations, generating the audio-fingerprint for the stream of audio data based on processing the stream of audio data may include processing, using an encoder portion of an encoder-decoder ML model, the stream of audio data to generate an embedding as the audio-fingerprint. In some versions of those implementations, determining whether the stream of audio data has been previously utilized in generating a gradient for updating a global ML model with respect to the given language based on comparing the audio-fingerprint for the stream of audio data to a database of audio-fingerprints may include comparing the embedding generated based on processing the stream of audio data to a plurality of previously generated embeddings, and determining, based on the comparing, whether the stream of audio data has been previously utilized in generating a gradient for updating a global ML model with respect to the given language. The plurality of previously generated embeddings may have been previously generated based on processing corresponding streams of audio data, and the plurality of previously generated embeddings may be stored in the database of audio-fingerprints.

In some implementations, the method may further include processing, using an on-device language identification model that is stored in the on-device storage of the client device, the stream of audio data to identify the given language, and determining whether the given language is one of a plurality of target languages. In some versions of those implementations, generating the audio-fingerprint for the stream of audio data may be in response to determining that the given language is one of the plurality of target languages. In some versions of those implementations, the method may further include, in response to determining that the given language is not one of the plurality of target languages: refraining from any further processing of the stream of audio data; and discarding the stream of audio data. In some versions of those implementations, a developer associated with the global ML model may provide an indication of the plurality of target languages.

In some implementations, the remote system may utilize the gradient to update one or more global weights of the global ML model to generate an updated global ML model. In some versions of those implementations, the method may further include receiving, from the remote system, the one or more global weights of the updated global ML model, or receiving, from the remote system, the updated global ML model. In some further versions of those implementations, the method may further include replacing, in the on-device storage of the client device, one or more on-device weights of the on-device ML model with the one or more global weights of the updated global ML model, or replacing, in the on-device storage of the client device, the on-device ML model with the updated global ML model.

In some implementations, the unsupervised or self-supervised learning technique may include one or more of: a teacher-student technique, or a masking technique.

In some implementations, the global ML model may be a global feature extractor model that is updated to extract features from the stream of audio data with respect to the given language.

In some implementations, the global ML model may be a multilingual automatic speech recognition (ASR) model that is updated to recognize text from the stream of audio data with respect to the given language.

In some implementations, a method performed by one or more processors of a client device is provided and includes receiving, from a given radio station, a stream of audio data that captures a stream of spoken utterances in a given language; generating, based on processing the stream of audio data, an audio-fingerprint for the stream of audio data; determining, based on comparing the audio-fingerprint for the stream of audio data to a database of audio-fingerprints, whether the stream of audio data has been previously utilized in generating a gradient for updating a global machine learning (ML) model with respect to the given language; and in response to determining that the stream of audio data has not been previously utilized in generating a gradient for updating the global ML model with respect to the given language: transmitting the stream of audio data to the remote system. Transmitting the stream of audio data to the remote system causes the remote system to: process, using the global ML model, the stream of audio data; generate, using the unsupervised or self-supervised learning technique, and based on processing the stream of audio data using the global ML model, the gradient; and update, based on the gradient, the global ML model.

These and other implementations of the technology can include one or more of the following features.

In some implementations, transmitting the stream of audio data to the remote system may further cause the remote system to, subsequent to generating the gradient, discard the stream of audio data.

In some implementations, a method performed by one or more processors of a remote system is provided and includes receiving, from a given client device, a stream of audio data that captures a stream of spoken utterances in a given language, the stream of audio data being initially received at the given client device from a given radio station; generating, based on processing the stream of audio data, an audio-fingerprint for the stream of audio data; determining, based on comparing the audio-fingerprint for the stream of audio data to a database of audio-fingerprints, whether the stream of audio data has been previously utilized in generating a gradient for updating a global machine learning (ML) model with respect to the given language; and in response to determining that the stream of audio data has not been previously utilized in generating a gradient for updating the global ML model with respect to the given language: processing, using the global ML model, the stream of audio data; generating, using an unsupervised or self-supervised learning technique, and based on processing the stream of audio data using the global ML model, the gradient; and updating, based on the gradient, the global ML model with respect to the given language.

In some implementations, a method performed by one or more processors of a client device is provided and includes receiving, from a given radio station, a stream of audio data that captures a stream of spoken utterances in a given language; determining, based on a connection status between the client device and a remote system, whether to implement federated learning or ephemeral learning to generate a gradient for updating a global machine learning (ML) model with respect to the given language; in response to determining to implement federated learning to generate the gradient for utilization in updating the global ML model with respect to the given language: processing, using an on-device ML model that is stored in on-device storage of the client device and that is an on-device counterpart of the global ML model, the stream of audio data; generating, using an unsupervised or self-supervised learning technique, and based on processing the stream of audio data using the on-device ML model, the gradient; and asynchronously transmitting the gradient to the remote system to be utilized in updating the global ML model with respect to the given language; and in response to determining to implement ephemeral learning to generate the gradient for utilization in updating the global ML model with respect to the given language: processing, using the on-device ML model, the stream of audio data; generating, using the unsupervised or self-supervised learning technique, and based on processing the stream of audio data using the on-device ML model, the gradient; and synchronously transmitting the gradient to the remote system to be utilized in updating the global ML model with respect to the given language.

These and other implementations of the technology can include one or more of the following features.

In some implementations, the method may further include determining to implement federated learning to generate the gradient for utilization in updating the global ML model with respect to the given language based on the connection status between the client device and the remote system indicating that the client device cannot connect to the remote system. In some versions of those implementations, asynchronously transmitting the gradient to the remote system to be utilized in updating the global ML model with respect to the given language may include, subsequent to generating the gradient: determining that a connection has been established between the client device and the remote system; and in response to determining that the connection has been established between the client device and the remote system: transmitting the gradient to the remote system to be utilized in updating the global ML model with respect to the given language.

In some implementations, the method may further include determining to implement ephemeral learning to generate the gradient for utilization in updating the global ML model with respect to the given language based on the connection status between the client device and the remote system indicating that the client device is connected to the remote system. In some versions of those implementations, synchronously transmitting the gradient to the remote system to be utilized in updating the global ML model with respect to the given language may include transmitting the gradient to the remote system to be utilized in updating the global ML model with respect to the given language without any connection having to be subsequently established between the client device and the remote system.

In some implementations, determining whether to implement federated learning or ephemeral learning to generate a gradient for updating a global ML model with respect to the given language may be further based on a location of the client device.

In some implementations, a method performed by one or more processors of a client device is provided and includes receiving, from a given radio station, a stream of audio data that captures a stream of spoken utterances in a given language; determining, based on a connection status between the client device and a remote system, whether to implement federated learning or ephemeral learning to generate a gradient for updating a global machine learning (ML) model with respect to the given language; in response to determining to implement federated learning to generate the gradient for utilization in updating the global ML model with respect to the given language: processing, using an on-device ML model that is stored in on-device storage of the client device and that is an on-device counterpart of the global ML model, the stream of audio data; generating, using an unsupervised or self-supervised learning technique, and based on processing the stream of audio data using the on-device ML model, the gradient; and asynchronously transmitting the gradient to the remote system to be utilized in updating the global ML model with respect to the given language; and in response to determining to implement ephemeral learning to generate the gradient for utilization in updating the global ML model with respect to the given language: synchronously transmitting the stream of audio data to the remote system. Synchronously transmitting the stream of audio data to the remote system causes the remote system to: process, using the global ML model, the stream of audio data; generate, using the unsupervised or self-supervised learning technique, and based on processing the stream of audio data using the global ML model, the gradient; and update, based on the gradient, the global ML model.

These and other implementations of the technology can include one or more of the following features.

In some implementations, synchronously transmitting the stream of audio data to the remote system may further cause the remote system to: discard the stream of audio data subsequent to generating the gradient.

In some implementations, a method performed by one or more processors of a client device is provided and includes receiving, from a given radio station being actively consumed by a user of the client device, a stream of audio data that captures a stream of spoken utterances in a given language; and causing a gradient, for updating a global machine learning (ML) model with respect to the given language and at a remote system, to be generated. In some versions of those implementations, causing the gradient to be generated includes processing, using an on-device ML model that is stored in on-device storage of the client device and that is an on-device counterpart of the global ML model, the stream of audio data; generating, using an unsupervised or self-supervised learning technique, and based on processing the stream of audio data using the on-device ML model, the gradient; discarding the stream of audio data; and transmitting the gradient to the remote system to be utilized in updating the global ML model with respect to the given language. In other implementations, causing the gradient to be generated includes transmitting the stream of audio data to the remote system. Transmitting the stream of audio data to the remote system causes the remote system to: process, using the global ML model, the stream of audio data; generate, using the unsupervised or self-supervised learning technique, and based on processing the stream of audio data using the global ML model, the gradient; discard the stream of audio data; and update, based on the gradient, the global ML model.

In some implementations, a method performed by one or more processors of a remote system is provided and includes receiving, from a given client device, a stream of audio data that captures a stream of spoken utterances in a given language, the stream of audio data being initially received at the given client device from a given radio station that is being actively consumed by a user of the given client device; processing, using a global machine learning (ML) model, the stream of audio data; generating, using an unsupervised or self-supervised learning technique, and based on processing the stream of audio data using the global ML model, a gradient; discarding the stream of audio data; and updating, based on the gradient, the global ML model with respect to the given language.

Various implementations can include a non-transitory computer readable storage medium storing instructions executable by one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), digital signal processor(s) (DSP(s)), and/or tensor processing unit(s) (TPU(s)) to perform a method such as one or more of the methods described herein. Other implementations can include an automated assistant client device (e.g., a client device including at least an automated assistant interface for interfacing with cloud-based automated assistant component(s)) that includes processor(s) operable to execute stored instructions to perform a method, such as one or more of the methods described herein. Yet other implementations can include a system of one or more servers that include one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described herein.