Patent Description:
Speech recognition involves technologies for recognizing and translating spoken words into text. Speech recognition is used in many contexts, including speech-to-text processing and voice interfaces in which people interact with electronic devices using voice commands.

<CIT> discloses an apparatus including an audio receiver to receive audio comprising a keyword to be detected. The apparatus includes a spike transducer to convert the audio into a plurality of spikes. The apparatus includes a spiking neural network to receive one or more of the spikes and generate a spike corresponding to a detected keyword.

<CIT> discloses deep-learning networks for generating mel frequency cepstral coefficients in audio processing pipelines. A first neural network is trained to output auditory features based on input audio samples. A second neural network is trained to output a classification based on input auditory features. An input audio sample is provided to the first neural network. Auditory features are received from the first neural network. The auditory features are provided to the second neural network. A classification of the input audio sample is received from the second neural network.

Edge computing is a distributed computing paradigm that brings computation and data storage closer to the sources of data. Edge computing can improve response time and save bandwidth. Edge devices are the computing devices that process data on a local level. However, edge devices may have limited computing capability compared to a traditional server computer.

Brain-inspired neuromorphic hardware and Spiking Neural Network models (SNNs) which run on the neuromorphic hardware can provide desirable benefits that make them well-suited for speech recognition for an edge device. Neuromorphic processors have the benefits of low power, low latency, and continuous learning. For example, neuromorphic processors have the potential to reduce power consumption during speech recognition inference by several orders of magnitude compared to CPU or GPU accelerators. The neuromorphic processors can enable a wide range of power-constrained devices, such as edge devices, to support speech recognition using a larger vocabulary than other speech recognition techniques such as digital signal processing. Neuromorphic systems process sparse, event-based data which can enable low latency inference across a variety of applications including speech recognition. Neuromorphic systems can support continuous learning algorithms for SNNs.

The techniques described in this document provide an efficient audio to spikes conversion pipeline that converts speech into spikes for input to SNNs and is suitable for real-time operation with low-power neuromorphic processors such that the conversions and speech recognition can be performed quickly and efficiently by edge devices. For example, this document describes a pipeline and corresponding techniques for converting raw audio, including streamed audio, to spikes fast enough for real-time speech recognition at edge devices.

The audio to spikes conversion pipeline can include hyperparameters that control how the pipeline processes audio data to convert the audio data into spike input for an SNN. Hyperparameter sweeps can be performed on the audio to spikes pipeline to optimize, or at least improve, the hyperparameters of the pipeline which results in more efficient (e.g., requiring fewer computations and/or fewer processor cycles) and faster conversions of audio data into spikes. For example, the hyperparameter sweep process can determine the relative importance of hyperparameters and the processing associated with some hyperparameters may no longer be performed if the hyperparameter is not important. This reduces the amount of processing of the audio data (or intermediate forms of the audio data) without negatively impacting the accuracy of the conversion.

A method performed by one or more data processing apparatus includes obtaining audio data; generating frequency domain audio signals that represent the audio data by converting the audio data into a frequency domain; mapping the frequency domain audio signals into a set of Mel-frequency bands to obtain Mel-scale frequency audio signals; performing a log transformation on the Mel-scale frequency audio signals to obtain log-Mel signals; generating spike input including a series of spikes for input to a spiking neural network (SNN) model by converting the log-Mel signals to the series of spikes, converting the log-Mel signals to spike input includes determining a step-wise difference between the log-Mel signals of two adjacent time steps, determining that the step-wise difference satisfies a threshold in either a positive or negative direction, and generating a spike of a corresponding polarity; and providing the spike input as an input to the SNN model. Other embodiments include corresponding computer systems, apparatus, computer program products, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method. A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

Implementations may include one or more of the following features. In some aspects, the method includes executing, the SNN model using the spike input. The spiking neural network processes the series of spikes to generate speech recognition results including text corresponding to the audio data and outputting the speech recognition results.

Some aspects include performing an action based on the speech recognition results. In some aspects, obtaining the audio data comprises obtaining the audio data from a memory or obtaining the audio data from an active microphone.

In some aspects, converting the audio data into the frequency domain includes centering and scaling the audio signals to obtain centered and scaled audio data and converting the centered and scaled audio data into a frequency domain using a sliding window of a predetermined length. In some aspects, converting the audio data into the frequency domain includes converting the audio data using a sliding discrete Fourier transform (SDFT) algorithm.

In some aspects, converting the log-Mel signals into spikes includes determining cumulative sum spikes by calculating a quantized total of each log-Mel signal.

In some aspects, converting the log-Mel features into spikes includes stacking the log-Mel features for a plurality of times to obtain stacked features in a plurality of blocks; and converting the stacked features in the plurality of blocks into spikes.

In some aspects, the one or more components of the audio to spikes conversion pipeline comprises hyperparameters that are adjusted based on a classification accuracy. Some aspects include, for each hyperparameter configuration of a plurality of sets of hyperparameter configurations: applying the hyperparameter configurations to the one or more components of the audio to spikes conversion pipeline; training multiple test SNN models using training audio data that is converted to spike inputs using the one or more components of the audio to spikes conversion pipeline; and testing each test SNN model using testing data, the testing comprising generating result data by determining an accuracy of speech recognition results generated by each test SNN model; and updating the hyperparameters based on the result data for each set of hyperparameters.

Some aspects include determining an importance of each hyperparameter based on the result data for each set of hyperparameters, wherein updating the hyperparameters comprises updating at least one hyperparameter based on the importance of the at least one hyperparameter.

Some aspects include preventing use of a particular hyperparameter in generating the spike input based on the importance of the given hyperparameter.

This specification is generally related to an audio to spikes conversion pipeline that can encode audio data into spikes for input to an SNN that is trained to generate speech recognition results that include predicted speech sounds in the audio data. The audio to spikes conversion pipeline includes a set of computational operations, each of which is configured, e.g., optimized, for efficient real-time execution and to generate sparse information-rich representation of the raw audio signal. The pipeline can include a combination of stages that provide low latency and require minimal computational overhead to convert audio data, which can be streaming audio input received by the pipeline, into spikes. The spikes can be used for real-time speech recognition by power-constrained devices, such as edge devices with low-power neuromorphic processors.

<FIG> is a block diagram of an example environment <NUM> in which a computing device <NUM> processes audio data to recognize speech sounds in the audio data. The example environment <NUM> includes the computing device <NUM>, a remote server <NUM>, a cloud server <NUM>, and a data communication network <NUM>, such as a local area network (LAN), a wide area network (WAN), the Internet, a mobile network, or a combination thereof. However, as described below, all of the audio processing can occur at the computing device <NUM> without transmitting or receiving data from either server <NUM>, <NUM>, which improves the speed at which speech is recognized at the computing device <NUM>, e.g., by precluding roundtrip communications over the network <NUM>, and reduces bandwidth consumption on the network <NUM>.

The computing device <NUM> can obtain audio data from a microphone <NUM> or a storage device <NUM>. In some examples, the microphone <NUM> can be a component (e.g., sensor) included in the computing device <NUM>. In other examples, the microphone <NUM> can be an Internet of Things (IoT) device, networked device, or a wired device, connected with the computing device <NUM>. In some examples, the computing device <NUM> can obtain the audio data from a memory, e.g., memory of the computing device <NUM> or the data storage device <NUM>. In either case, the audio data can be in the form of raw audio (e.g., that has not been processed) and can contain speech in the form of spoken words.

In some implementations, the computing device <NUM> can be an edge device that processes data on a local level. For example, Internet of things (IoT) devices can generate large volumes of data during their operations. Instead of transmitting the large volumes of IoT data to a remote server <NUM> or a cloud server <NUM> over network <NUM>, the edge device can process the IoT data locally reducing the bandwidth needs of the large volumes of data. In some implementations, the edge device can be a power-constrained device compared with the remote server <NUM> and the cloud server <NUM>. Edge devices can be IoT computing devices, smart cameras, smart speakers, embedded computers, mobile devices, and even smart TVs or other connected devices. In a particular example, the computing device <NUM> can be a smart speaker or smartphone that includes and/or is communicably coupled to (e.g., via the network <NUM>) an artificial intelligence (AI) assistant that performs tasks based on recognized speech of a user.

The computing device <NUM> includes an audio to spikes conversion pipeline <NUM> and a neuromorphic processor <NUM>. The audio to spikes conversion pipeline <NUM> can be implemented using hardware (e.g., a processor, such as a central processing unit (CPU)) and/or software.

The audio to spikes conversion pipeline <NUM> is configured to convert audio data into spikes for processing by the neuromorphic processor <NUM>. A series of spikes output by the audio to spikes conversion pipeline <NUM> for audio data can be referred to as speech input that is provided as input to a SNN <NUM> executed by the neuromorphic processor <NUM>. The audio to spikes conversion pipeline <NUM> can include multiple processing stages to convert the audio data into the spikes. These stages are described in more detail with reference to <FIG>. The audio to spikes conversion pipeline <NUM> is efficient and suitable for real-time operation. For example, the stages of the audio to spikes conversion pipeline <NUM> are configured to convert streamed raw audio to spikes fast enough for real-time speech recognition by the neuromorphic processor <NUM>.

The neuromorphic processor <NUM> is configured to perform speech recognition tasks using a SNN <NUM>. For example, the neuromorphic processor <NUM> can execute the SNN <NUM> to predict speech sounds represented by the spikes of the spike input output by the audio to spikes conversion pipeline <NUM>. For example, the SNN <NUM> can process the spikes to generate speech recognition results that include predicted speech sounds, e.g., in the form of textual phonemes, triphones, full words, or other units of sound, corresponding to speech of the audio data.

In some implementations, e.g., if the SNN <NUM> is not configured to output words as the predicted speech sounds, the computing device <NUM> can include a speech recognizer <NUM> that is configured to determine words and/or sentences corresponding to the phonemes or other non-word predicted speech sounds output by the neuromorphic processor <NUM>. For example, the speech recognizer <NUM> can include lexicons and/or language models that are configured to convert phonemes into words and/or sentences. In this way, the computing device <NUM> can convert audio data into words, phrases, and/or sentences without requiring network communications.

In some implementations, the computing device <NUM> can send predicted speech sounds to the server(s) <NUM>, <NUM> for converting the speech sounds into words and/or sentences. For example, one or both servers <NUM>, <NUM> can include the speech recognizer <NUM> or multiple speech recognizer components, e.g., respective language models for various regions and their different languages and dialects. The server(s) <NUM>, <NUM> can provide the words and/or sentences back to the computing device <NUM> or perform an action based on the words and/or sentences. For example, the server(s) <NUM>, <NUM> can include an AI assistant that performs tasks based on spoken commands received from users' computing devices <NUM>.

One or both servers <NUM>, <NUM> can include a hyperparameter adjuster <NUM>, which can be implemented in hardware and/or software. In some implementations, the hyperparameter adjuster <NUM> can be part of the computing device <NUM>. The hyperparameter adjuster <NUM> can evaluate SNNs and adjust hyperparameters of the audio to spikes conversion pipeline <NUM> based on the evaluation. Example hyperparameters of the audio to spikes conversion pipeline <NUM> are described with reference to <FIG>. For example, the hyperparameter adjuster <NUM> can perform hyperparameter sweeps using multiple sets of hyperparameter configurations and select a hyperparameter configuration based on the results of the sweeps. Each sweep can include applying a particular hyperparameter configuration to the audio to spikes conversion pipeline <NUM>, training multiple SNNs using training data and spikes generated by the audio to spikes conversion pipeline <NUM> based on the training data, testing each of the multiple SNNs using testing data, measuring the accuracy of each SNN based on the test. The training data and/or the testing data can include labeled audio data with labels indicating predicted speech sounds, e.g., words or phonemes, corresponding to the audio data.

The hyperparameter adjuster <NUM> can adjust the hyperparameters of the audio to spikes conversion pipeline <NUM> based on the evaluation. For example, the hyperparameter adjuster <NUM> can identify the hyperparameter configuration for which the most accurate SNN was trained during the sweeps and apply that hyperparameter configuration to the audio to spikes conversion pipeline <NUM>.

The hyperparameter adjuster <NUM> can also determine the relative importance of each hyperparameter based on the results of each sweep. For example, if there is a significant change in the accuracy of the SNNs trained during the sweeps when the configuration of a particular hyperparameter is adjusted, then the particular hyperparameter may have a large impact on the accuracy of the SNNs and is therefore important. The hyperparameter adjuster <NUM> can evaluate various factors in assessing the importance of each hyperparameter, such as the variance in accuracy of SNNs with changes to the hyperparameter, whether the hyperparameter contributed to the highest accuracy SNN or at least a threshold number of the highest accuracy SNNs, the computation cost of the hyperparameter, the power required by the hyperparameter, loss function minimization, and/or other appropriate factors.

The hyperparameter adjuster <NUM> can modify the audio to spikes conversion pipeline <NUM> based on the relative importance of the hyperparameters. For example, the hyperparameter adjuster <NUM> can determine updated configurations (e.g., updated code for software components) of the audio to spikes conversion pipeline <NUM> and send the updated configurations to the computing device <NUM>.

In some cases, the hyperparameter adjuster <NUM> can modify the configuration of a hyperparameter corresponding to a software component of the audio to spikes conversion pipeline <NUM> such that the software component is no longer active if the hyperparameter is determined to have no or low importance. The processing of audio data or intermediate forms of audio data based on each hyperparameter has an associated computation cost (e.g., CPU cycles) and latency. By removing those that are less important, the conversion of audio data to spikes can be performed faster and more efficiently with little or no negative impact on accuracy of the speech recognition results of the SNN <NUM>. However, some hyperparameters may be more important for some languages and/or dialects. Thus, having the ability to use many hyperparameters and filter some based on importance enables highly accurate speech recognition with highly efficient speech to spikes conversions.

In some implementations, the hyperparameter adjuster <NUM> selects hyperparameters to use or not use based on the relative importance and/or the computational cost of each hyperparameter. For example, if a hyperparameter is moderately important but has the highest computational cost, the hyperparameter adjuster <NUM> may turn off that hyperparameter to increase the efficiency of the conversion to spikes. In some implementations, a user, e.g., a software developer, can make these selections and configuration of the audio to spikes conversion pipeline <NUM> accordingly. An example process for adjusting the hyperparameters of the audio to spikes conversion pipeline is shown in <FIG> and described below.

The computing device <NUM> can deploy one of the SNNs from the sweeps on the neuromorphic processor <NUM>. For example, the computing device <NUM> can select the highest performing SNN in terms of accuracy or other evaluation criteria for speech recognition using the hyperparameter configuration selected for the audio to spikes conversion pipeline <NUM>.

<FIG> shows four stages <NUM> - <NUM> of the example audio to spikes conversion pipeline <NUM>. The four stages can include, for example, a sampling stage <NUM>, a frequency domain stage <NUM>, a spectrogram stage <NUM>, and a spiking representation stage <NUM>. The processing of each stage can be performed by a respective component of the audio to spikes conversion pipeline <NUM>, such as a respective engine, software module, or processor of the audio to spikes conversion pipeline <NUM>. For example, the sampling stage <NUM> can be performed by a sampling engine, the frequency domain stage <NUM> can be performed by a frequency domain engine, the spectrogram stage <NUM> can performed by a spectrogram engine, and the spiking representation stage <NUM> can be performed by a spiking representation engine. The processing of the stages <NUM>-<NUM> can be performed in software and/or on hardware accelerators, either sequentially or in parallel.

In the sampling stage <NUM>, the audio to spikes conversion pipeline <NUM> starts with obtaining audio data <NUM>, which can be in the form of raw audio or audio that has been preprocessed. Examples of pre-processing can include reducing noise in the audio data, e.g., using a filtering mechanism or through sound localization. The audio data <NUM> can be obtained from a file or memory or from an active microphone. In some examples, the computing device <NUM> can obtain the audio data <NUM> with a predetermined sampling rate. For example, the raw audio can be <NUM>, <NUM>-bit raw audio. The audio data <NUM> can be obtained with other sampling rates and/or bit precision recordings.

In the sampling stage <NUM>, the audio data <NUM> can be centered and/or scaled. For example, the audio to spikes conversion pipeline <NUM> can center the audio data <NUM> by subtracting an estimate of the bias and scaling to achieve a peak-to-peak amplitude threshold, such as a peak-to-peak amplitude of <NUM> or another appropriate value. The audio to spikes conversion pipeline <NUM> can center and scale the audio data <NUM> sample-by-sample. In some cases, the bias is often small enough to be ignored.

In the frequency domain stage <NUM>, the audio to spikes conversion pipeline <NUM> converts the audio data into audio signals in a frequency domain. The audio to spikes conversion pipeline <NUM> can perform this conversion using a Fast Fourier Transform (FFT), a Sliding Discrete Fourier Transform (SDFT), or another appropriate mechanism for obtaining a frequency domain representation of audio data <NUM>. For example, the audio to spikes conversion pipeline <NUM> can apply the SDFT to the audio data <NUM> to encode the audio data to the frequency domain to mimic how the cochlea acts as a filter bank, performing a Fourier decomposition of the incoming audio. The SDFT is an incremental method for computing the power spectrum of the audio data <NUM>. The incremental computation can be faster than a full FFT in some cases, e.g., depending on the hop size of the FFT. Further, SDFT can operate in constant time complexity on a sample-by-sample basis through parallelization and hardware acceleration.

The output of the frequency domain stage <NUM> can include a frequency domain representation of the audio data <NUM> and can include an instantaneous representation of the spectral content of the audio signal, which represents the power of the frequencies that make up the audio signal. These instantaneous representations can be appended in succession to generate a spectrogram <NUM>.

In the spectrogram stage <NUM>, the audio to spikes conversion pipeline <NUM> maps the frequency domain representation of the audio data <NUM> onto a set of frequency bands and can produce a spectrogram <NUM>. For example, the audio spikes conversion pipeline <NUM> can map the frequency domain representation of the audio data <NUM> onto a set of Mel-scale frequency bands to produce a log-Mel spectrogram <NUM>. The audio to spikes conversion pipeline <NUM> can also log transform, center, and scale the frequency domain audio signals again. For example, the generate a log-Mel spectrogram <NUM>, the audio to spikes conversion pipeline <NUM> can map the frequency domain output by the frequency domain stage <NUM> on a Mel-scale that represents how humans perceive speech pitch, apply another transformation that uses overlapping windows of a particular shape (e.g., triangular, cosine, etc.) and width to non-linearily weight the frequencies in accordance with a human level of perception. After weighting the frequencies in this manner, the audio to spikes conversion pipeline <NUM> can take the log of the output to obtain the log-Mel spectrum. Mel-scale frequency bands, or Mel features, are frequency features in the range of human speech. By mapping the frequency domain (e.g., SDFT features) to the Mel-scale frequency bands, the audio to spikes conversion pipeline <NUM> can generate Mel features that represent the human speech portion of the audio data <NUM>, e.g., by highlighting the frequencies that humans are good at differentiating.

In some implementations, the spectrogram stage <NUM> can be optional. In some examples, after computing the frequency domain representation, the power spectra of the frequency domain audio signals are converted to a number of Mel-scale frequency bands (e.g., <NUM>, <NUM>, <NUM>, or another appropriate number of Mel-scale frequency bands), then a log transform is applied to the converted audio signals. At this point in the audio to spikes conversion pipeline <NUM>, the data are log-Mel spectra with temporal resolution equal to the audio sampling rate. In some implementations, the audio to spikes conversion pipeline <NUM> can continue with this resolution. When training SNNs using Back-Propagation-Through-Time (BPTT), the number of time steps in each processed spike raster can severely impact overall training time. To avoid such impact, the audio to spikes conversion pipeline <NUM> can down-sample the spectra to time steps, e.g., time steps of <NUM>, <NUM>, <NUM>, or another appropriate number of milliseconds (ms) time steps.

In some implementations, the audio to spikes conversion pipeline <NUM> stacks the log-Mel signals multiple times. For example, after down-sampling, the <NUM> log-Mel signals can be stacked four times resulting in <NUM> total log-Mel signals. Here, stacking refers to concatenating separate features with the log-Mel features, e.g., over time. For example, these features can include features derives from the log-Mel representation, such as the cumulative sum of a feature over time. In another example, separate arrays can be concatenated into one array, e.g., repeating the same signal represented as an array multiple times.

In the spiking representation stage <NUM>, the audio to spikes conversion pipeline <NUM> converts the log-Mel signals, e.g., stacked log-Mel signals, to a spiking representation <NUM>. The audio to spikes conversion pipeline <NUM> can convert the stacked log-Mel signals to spikes using a step-forward algorithm. <FIG> show an example process of generating spikes using a step-forward algorithm. The step-forward algorithm converts a continuous-valued signal into spikes.

The step-forward algorithm can include one or more of multiple types of delta modulation, which may also be referred to as differential pulse-code modulation. In one type, which can be referred to as discrete derivative, consecutive analog signals are quantized and the difference between the consecutive signals is determined. In another type, which can be referred to as an integral value approach, the difference is taken with respect to an integrator which is basically a value that tracks the signal over time. In other words, the difference between an instantaneous value and the integrated value of the signal over time is used instead of a difference between two consecutive signals. Both types of delta modulation can work in the audio to spikes pipeline <NUM>, but the second type using the integrated signal may provide performance improvements relative to the discrete derivative approach.

In a differential derivative approach, the step-wise difference between the continuous signals can be calculated. If the step-wise difference satisfies a threshold in either the positive or negative direction, a spike of the corresponding polarity is generated. By calculating the step-wise differences, a set of spikes (positive spikes and/or negative spikes) can be generated. The spikes can represent the change of the audio information (e.g., log-Mel signals). The spikes can represent the quantized, non-negative discrete derivative of the log-Mel signals.

In an integrated value approach, the integral of the signal can be computed over time. For each new signal, the difference between a new instantaneous value of the signal and the integrated value is calculated. The set of spikes can be generated based on this difference, similar to the differential derivative approach.

In addition to the set of spikes based on these differences, the audio to spikes conversion pipeline <NUM> can generate another set of spikes based on cumulative sums of the log-Mel signals, e.g., cumulative sum spikes. The cumulative sum spikes can be calculated by calculating a quantized total of each log-Mel signal. The cumulative sum spikes can represent the spike encoding of the sustained value of audio information (e.g., log-Mel signals). As shown in <FIG>, the generated spikes for the audio <NUM> can include the step-wise difference spikes <NUM> and the cumulative sum spikes <NUM>. The generated spikes can be processed by SNNs for speech recognition.

In some implementations, the hyperparameters used in the different stages of the audio to spikes conversion pipeline <NUM> can be determined by the hyperparameter adjuster <NUM>. For example, the hyperparameters can include spike encoding method, threshold, hop length (ms), time step (ms), log spectral features, spectral features, number of channels, and other appropriate hyperparameters. The hyperparameter adjuster can perform hyperparameter sweeps using multiple sets of hyperparameter configurations and select a hyperparameter configuration that results in the best SNN accuracy. For example, the hyperparameter adjuster can perform sweeps for three different spike encoding methods including the Delta method, Rate method, and step-forward method. Based on the results of the sweeps, the hyperparameter adjuster can determine that the best spike encoding method is the step-forward method.

<FIG> is a flow diagram of an example process <NUM> for converting audio data to spikes and processing the spikes using an SNN to recognize speech sounds. In some implementations, at least a portion of the process <NUM> can be executed at the computing device <NUM>.

At step <NUM>, the computing device obtains audio data. For example, the computing device can obtain the audio data in real time from a microphone. The microphone can be located inside the computing device or outside the computing device. In some examples, the computing device can obtain the audio data from a storage device or a memory of the computing device.

In some examples, the computing device can obtain the audio data with a predetermined sampling data. For example, the obtained audio data can be <NUM>, <NUM>-bit raw audio. Other frequencies and bit precision recording can also be used.

In some implementations, the raw audio data can be centered and scaled. For example, the computing device can center the raw audio data by subtracting an estimate of the bias (in practice, the bias is often small enough to be ignored), and scale the raw audio to achieve a predetermined peak-to-peak amplitude, such as <NUM>, <NUM>, <NUM>, or another appropriate value.

At step <NUM>, the computing device converts the audio data (e.g., the centered and scaled audio data) into a frequency domain to obtain frequency domain audio signals. A Fourier transform can be applied to the audio data to encode the audio data to the frequency domain. For example, an SDFT can be applied to incrementally compute the Fourier transform of the audio data as follows: <MAT> where <MAT> is the DFT's frequency at the kth bin at time step t computed from samples ft-M+<NUM>, ft-M+<NUM>,. , ft, where <NUM> ≤ k ≤ M - <NUM>, t is the time step in the sliding window of length T, and G = ej<NUM>π/M is a scaling factor.

This incremental computation can be much faster than the typical method for performing continuous speech recognition with a FFT, where the audio signal is sliced into overlapping windows and a DFT is computed for each window. Further, SDFT is able to operate in constant time complexity on a sample-by-sample basis through parallelization and hardware acceleration.

At step <NUM>, the computing device maps the frequency domain audio signals to a set of Mel-scale frequency bands and obtains log-Mel signals. Specifically, the frequency domain audio signals can be mapped to a set of Mel-scale frequency bands to obtain Mel-scale frequency audio signals. Mel-scale frequency bands, or Mel features, are frequency features in the range of human speech. The Mel-scale frequency audio signals can then be log transformed, to obtain the log-Mel signals.

In some implementations, the computing device can down-sample the log-Mel signals. The down-sampling can preserve performance while allowing for the audio to spikes conversion pipeline <NUM> to be run in real-time. For example, the log-Mel signals can be down-sampled to <NUM> time steps or other appropriate time steps.

The computing device can stack, e.g., concatenating, the log-Mel signals for predetermined times. For example, after computing the SDFT, the frequency domain audio signals are converted to a number (e.g., <NUM>, <NUM>, <NUM>, <NUM>, etc.) of Mel-scale frequency bands. The log-Mel signals can be stacked four times resulting in four blocks of <NUM> total units' log-Mel signals, with each block including <NUM> units' log-Mel signals. For example, the four blocks can represent the log-Mel spikes, the cumulative sum of the log-Mel spikes and two channels for each of them. If binary data is being sent, there is no way to represent a negative. So, negative and positive changes are broken out into two separate channels. Other appropriate numbers of blocks can be used. In some implementations, each log-Mel signal in the stack can be centered and scaled using static estimates of the mean and standard deviation derived from trail runs of data driven feature extraction.

At step <NUM>, the computing device converts the log-Mel signals to spikes. The converting of log-Mel signals to spikes can be using a step-forward algorithm. This algorithm can encode a continuous-values signal into multiple channels of spikes by taking the step-wise difference between the continuous signals. When the step-wise difference satisfies (e.g., exceeds) a threshold in either the positive or negative direction, a spike of the corresponding polarity is generated. Step-forward is useful for neuromorphic speech recognition systems because it can be implemented efficiently on general-purpose processors for a reasonable number of channels.

<FIG> show an example process of generating spikes for audio signals. <FIG> shows the step-forward algorithm using the integral value approach being applied to two streaming signals: artificial input <NUM> and the raw audio recording <NUM>. For each streaming signal, there is a residual signal that represents the integrated value generated by an integral based delta modulation. For example, for raw input <NUM> of the raw audio recording, there is a residual signal <NUM> that represents the integrated value. Using the residual signals, the two streaming signals <NUM> and <NUM> are converted to spikes. For example, the artificial input <NUM> is converted into positive spikes <NUM> and negative spikes <NUM>. The raw audio recording <NUM> is converted into positive spikes <NUM> and negative spikes <NUM>. The step-forward algorithm converts a continuous-valued signal into a binary signal, interpretable as a spike, by quantizing the signal, taking the difference between the raw input <NUM> and the residual signal <NUM>, and applying a threshold.

For example, the instantaneous value of the raw input <NUM> can be compared to the residual signal <NUM>. The difference between these two signals are then used to generate the spikes. Then, the residual signal <NUM> is modified by the quantized result. For example, if the instantaneous value of the raw signal <NUM> is greater than the residual signal <NUM>, the residual signal <NUM> is increased. Similarly, if the instantaneous value of the raw signal <NUM> is less than the residual signal <NUM>, the residual signal <NUM> is decreased.

<FIG> shows the process of generating spikes using the step-forward algorithm. In this example, the spikes are generated for the raw audio recording streaming signal <NUM> in <FIG>. For the raw input <NUM> of the raw audio recording, there is a residual signal <NUM> that represents the integrated value of the signal over time.

The difference between the continuous raw input signals <NUM> and the residual signal <NUM> can be calculated. A difference between the log-Mel signals of two signals is determined. If the difference satisfies a threshold in either the positive or negative direction, a spike of the corresponding polarity is generated. In other words, when the absolute value of the difference satisfies a threshold, a positive or negative spike can be generated. For example, a residual signal X<NUM> <NUM> and the actual signal Y<NUM> <NUM> are shown. The residual signal X<NUM> is compared to the actual signal Y<NUM> and the difference is quantized if it is greater than a threshold. For example, when the difference X<NUM>-Y<NUM> satisfies (e.g., less than) a negative threshold -θ <NUM>, a negative spike <NUM> can be generated. Similarly, when the difference between residual X<NUM> and actual signal Y<NUM> satisfies (e.g., larger than) a positive threshold θ <NUM>, a positive spike <NUM> can be generated.

By calculating the differences, a set of spikes (positive spikes and/or negative spikes) can be generated. The spikes can represent the change of the audio information (e.g., log-Mel signals). That is, only if there are changes or new information available, the changes or new information can be captured and processed. In this way, the processed audio information is reduced, thus allowing for speech recognition on power-constrained devices.

In addition to the set of spikes based on the differences, the computing device can generate spikes based on cumulative sums of the log-Mel signals, e.g., cumulative sum spikes. The cumulative sum spikes can be determined by calculating a quantized total of each log-Mel signal. For example, a cumulative sum can be applied over time to each log-Mel signal to derive cumulative sum spikes. The cumulative sum spikes can represent the spike encoding of the sustained value of audio information (e.g., log-Mel signals). In some implementations, the cumulative sum spikes can be concatenated to the difference spikes. For example, the spikes for the first and second blocks (e.g., units <NUM> through <NUM>) can represent difference spikes, e.g., the quantized, discrete derivative of the continuous signal values. The spikes in the third and fourth blocks (e.g., units <NUM> to <NUM>) can represent the cumulative sum spikes, e.g., sustained value of the log-Mel signals rather than only changes. The multiple sets of spikes including the step-wise difference spikes (e.g., positive signal spikes and negative signal spikes) and the cumulative sum spikes (e.g., positive accumulative spikes and negative accumulative spikes) can be input to SNNs for speech recognition.

In some implementations, the log-Mel features and the cumulative sum can be provided as input to the spiking representation stage <NUM> together. Each channel can then be converted into spikes separately.

At step <NUM>, the computing device can process the spikes using SNNs for speech recognition. For example, the computing device can execute an SNN using the spikes as input. The SNN can process the spikes to generate speech recognition results that include texts, e.g., words or phonemes, corresponding to speech in the audio data. For example, the SNN can predict speech sounds in the form of phonemes based on the spikes received from the audio to spikes conversion pipeline <NUM>. The phonemes can then be processed by another component, e.g., a lexicon and/or language model to generate words or sentences corresponding to the phonemes.

The computing device or another device can perform an action based on the speech recognition result. For example, an AI assistant can perform an action (e.g., play a song, order an item, etc.) based on the words or sentence. The speech recognition results can be displayed in a user interface of a user device. In some examples, the computing device can be the user device. In some other examples, the computing device can be a different device from the user device.

The order of steps in the process <NUM> described above is illustrative only, and the process <NUM> can be performed in different orders. In some implementations, the process <NUM> can include additional steps, fewer steps, or some of the steps can be divided into multiple steps.

The techniques described in this document can be used to train SNNs with the described audio to spikes conversion pipeline <NUM>. For example, a feedforward SNN architecture can include an input layer of <NUM> (or another appropriate number of) neurons that take as input one second audio sample encoded to spikes by the audio to spikes conversion pipeline <NUM> over <NUM> time steps, three hidden layers of <NUM> Leaky Integrate-and-Fire (LIF) neurons each, and an output layer of <NUM> LIP neurons or <NUM> LIF neurons. For example, the input layer can include a number of layers equal to the number of stacked Mel-signals/features.

In some implementations, SNNs can be trained using snnTorch that enables efficient surrogate gradient learning for SNNs. Models can be trained using spike rate output: the output unit corresponding to the target class can be assigned a high target spike rate (e.g., <NUM> spikes/sec), while non-target output units can be assigned a low target spike rate (e.g., <NUM> spikes/sec). The training loss can be calculated by snnTorch using the mean-squared error of the actual spike rates and the target spike rates. Loss can be back-propagated using the fast sigmoid surrogate gradient and parameters can be updated using the Adam optimizer. The output class of the model can be determined by selecting the index of the output unit with the highest total number of spikes for a recording.

In some implementations, the system can train an SNN model using hardware-aware training methods. For example, the system can train an SNN model with the feedforward <NUM>-layer architecture using SLAYER and a functional neuromorphic hardware simulator, e.g., a simulator of Intel's Loihi chip. SLAYER is a surrogate gradient training framework built on pyTorch that supports the Current-Based Leaky Integrate-and-Fire (CUBA LIF) neuron model and provides a direct, hardware-aware training process to simplify the deployment of trained models. SLAYER trains fill-precision shadow weights which get quantized during the forward inference phase to the same precision supported by the target hardware.

The techniques described in this document can determine the hyperparameters for the audio to spikes conversion pipeline <NUM>. In some implementations, the system can interact with a platform that enables distributed model optimization while keeping a shared record of all parameters and performance metrics. The system can conduct a hyperparameter sweep to identify the best combination of hyperparameters for the audio to spikes conversion pipeline <NUM>. For the sweeps, in offline training instead of running real-time conversion and inference, the system can use standard FFTs in the audio to spikes conversion pipeline <NUM> which is more efficient for batch processing than the SDFT that the pipeline normally uses to enable low single sample latency for streamed audio. The sweeps can generate an estimate of the relative importance for each hyperparameter. The relative importance is the degree to which each hyperparameter was useful in predicting a chosen metric. The importance can be calculated from the feature importance values of a random forest trained with the hyperparameters as inputs and the metric as the target output. The classification accuracy can be chosen as the metric to optimize for the sweeps. In some implementations, the hyperparameters can include spike conversion, threshold, hop length (ms), time step (ms), log spectral features, spectral features, number of channels, and other appropriate hyperparameters.

The audio to spikes conversion pipeline <NUM> is an efficient audio processing pipeline for real-time, neuromorphic speech recognition systems. SNNs trained on data output by the audio to spikes conversion pipeline can achieve high accuracy in keyword classification. The pipeline can offer a more suitable spiking benchmark for speech recognition. The efficiency of the pipeline can enable a greater number of spiking speech recognition benchmarks in neuromorphic speech recognition.

The pipeline is more suitable for real-time operation and is suitable for use in a real-time speech recognition system. The techniques described in this document can lead to wider adoption of neuromorphic accelerators in edge devices to provide voice control, which can have cascading impacts on the intelligence that can be feasibly integrated into these devices. Neuromorphic accelerators can offer multiple application-level benefits, including continuous learning and multimodal processing.

<FIG> is a flow diagram of an example process <NUM> for adjusting hyperparameters of an audio to spikes conversion pipeline. In some implementations, at least a portion of the process <NUM> can be executed by a computing device configured to train and assess the performance of machine learning models, such as server <NUM> or server <NUM>. For brevity, the process <NUM> is described as being performed by a server.

At step <NUM>, the server obtains multiple sets of hyperparameter configurations. In some implementations, the hyperparameter configurations include configuring a set of hyperparameters with different values. For example, one hyperparameter can be time step. The configuration of the time step can include <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>. The set of hyperparameters can include spike encoding method, threshold, hop length (ms), time step (ms), log spectral features, spectral features, number of channels, and other appropriate hyperparameters.

At step <NUM>, the server performs sweeps for each hyperparameter configuration. Each sweep can include applying the particular hyperparameter configuration to the audio to spikes conversion pipeline (step <NUM>), training, on the neuromorphic processor, multiple SNNs using training data and spikes generated by the audio to spikes conversion pipeline based on the training data, and testing each of the multiple SNNs using testing data (step <NUM>), and determining the accuracy of each SNN based on the test (step <NUM>). The training data and/or the testing data can include labeled audio data with labels indicating speech sounds, e.g., words or phonemes, corresponding to the audio data.

The server can perform steps <NUM>-<NUM> multiple times to identify hyperparameters and a trained machine learning model to deploy on the audio to spikes conversion pipeline <NUM>. For example, the server can perform steps <NUM>-<NUM> until finding a trained machine learning model that achieves at least a minimum level of accuracy.

At step <NUM>, the server determines the relative importance of the hyperparameters. The server can determine the relative importance of each hyperparameter based on the results of each sweep. For example, if there is a significant change in the accuracy of the SNNs trained during the sweeps when the configuration of a particular hyperparameter is adjusted, then the particular hyperparameter may have a large impact on the accuracy of the SNNs and is therefore important. The server can evaluate various factors in assessing the importance of each hyperparameter, such as the variance in accuracy of SNNs with changes to the hyperparameter, whether the hyperparameter contributed to the highest accuracy SNN or at least a threshold number of the highest accuracy SNNs, and/or other appropriate factors.

In some implementations, the server can use the importance of each hyperparameter in step <NUM> to determine hyperparameters to test using steps <NUM>-<NUM>. For example, if a hyperparameter is found to be unimportant for a large (e.g., at least a threshold number of) tests, the server can remove the process corresponding to the hyperparameter from the pipeline. In another example, the server can adjust one or more of the hyperparameters based on their importance.

At step <NUM>, the server finalizes the hyperparameters of the audio to spikes conversion pipeline <NUM> based on the accuracy of SNNs and/or importance of the hyperparameters. The computing device can turn off the use of one or more hyperparameters based on the relative importance of the hyperparameters. The server can also adjust the configuration of one or more hyperparameters based on the accuracy and/or importance. The processing of audio data or intermediate forms of audio data based on each hyperparameter has an associated computation cost (e.g., CPU cycles) and latency. By removing those that are less important, the conversion of audio data to spikes can be performed faster and more efficiently with little or no negative impact on accuracy of the speech recognition results of the SNN. However, some hyperparameters may be more important for some languages and/or dialects. Thus, having the ability to use many hyperparameters and filter some based on importance enables highly accurate speech recognition with highly efficient speech to spikes conversions.

In some implementations, the computing device selects hyperparameters to user or not use based on the relative importance and/or the computational cost of each hyperparameter. For example, if a hyperparameter is moderately important but has the highest computational cost, the hyperparameter adjuster <NUM> may turn off that hyperparameter to increase the efficiency of the conversion to spikes.

Certain novel aspects of the subject matter of this specification are set forth in the claims below.

Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine readable storage device, a machine readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not, however, a propagated signal.

Each engine can be implemented on any appropriate type of computing device, e.g., servers, mobile phones, tablet computers, notebook computers, music players, e book readers, laptop or desktop computers, PDAs, smart phones, or other stationary or portable devices, that includes one or more processors and computer readable media.

The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit) a neuromorphic research chip, such as Intel's Loihi chip, or a neural network processor, such as BrainChip's Akida™ chip.

Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD ROM disks.

To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) monitor, an LCD (liquid crystal display) monitor, or an OLED display, for displaying information to the user, as well as input devices for providing input to the computer, e.g., a keyboard, a mouse, or a presence sensitive display or other surface. In addition, a computer can interact with a user by sending resources to and receiving resources from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client server relationship to each other.

Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims.

Claim 1:
A method performed by one or more data processing apparatus using one or more components of an audio to spikes conversion pipeline, the method comprising:
obtaining audio data (<NUM>);
generating frequency domain audio signals that represent the audio data by converting the audio data into a frequency domain;
mapping the frequency domain audio signals into a set of Mel-frequency bands to obtain Mel-scale frequency audio signals;
performing a log transformation on the Mel-scale frequency audio signals to obtain log-Mel signals;
generating spike input comprising a series of spikes for input to a spiking neural network, SNN, model (<NUM>) by converting the log-Mel signals to the series of spikes, wherein converting the log-Mel signals to spike input comprises:
determining a step-wise difference between the log-Mel signals of two adjacent time steps;
determining that the step-wise difference satisfies a threshold (<NUM>, <NUM>) in either a positive or negative direction; and
the method further comprising generating a spike (<NUM>, <NUM>, <NUM>, <NUM>) of a corresponding polarity; and
providing the spike input as an input to the SNN model.