A method comprising: receiving, at data processing hardware of a user device, a sequence of input frames that each include respective audio features characterizing streaming audio captured by the user device; generating, by the data processing hardware, a probability score indicating a presence of a hotword in the streaming audio using a memorized neural network, the memorized neural network comprising sequentially-stacked single value decomposition filter (SVDF) layers, wherein each SVDF layer comprises at least one neuron, and each neuron comprises: a respective memory component, the respective memory component associated with a respective memory capacity of the corresponding neuron and comprising a plurality of memory locations; a first stage configured to perform filtering on the respective audio features of each input frame individually and output the filtered audio features to the respective memory component; and a second stage configured to perform filtering on all the filtered audio features residing in the respective memory component; wherein each neuron is configured such that: the first stage performs filtering on the respective audio features of each input frame individually and outputs the filtered audio features to a next available memory location of the respective memory component, wherein once all of the memory locations of the respective memory component are filled, the first stage overwrites the memory location storing the oldest filtered audio features in the respective memory component, and then the second stage performs filtering on all of the filtered audio features residing in the respective memory component; determining, by the data processing hardware, whether the probability score satisfies a hotword detection threshold; and when the probability score satisfies the hotword detection threshold, initiating, by the data processing hardware, a wake-up process on the user device for processing the hotword and/or one or more other terms following the hotword in the audio stream. receiving, at data processing hardware of a user device, a sequence of input frames that each include respective audio features characterizing streaming audio captured by the user device; generating, by the data processing hardware, a probability score indicating a presence of a hotword in the streaming audio using a memorized neural network, the memorized neural network comprising sequentially-stacked single value decomposition filter (SVDF) layers, wherein each SVDF layer comprises at least one neuron, and each neuron comprises: a respective memory component, the respective memory component associated with a respective memory capacity of the corresponding neuron and comprising a plurality of memory locations; a first stage configured to perform filtering on the respective audio features of each input frame individually and output the filtered audio features to the respective memory component; and a second stage configured to perform filtering on all the filtered audio features residing in the respective memory component; wherein each neuron is configured such that: the first stage performs filtering on the respective audio features of each input frame individually and outputs the filtered audio features to a next available memory location of the respective memory component, wherein once all of the memory locations of the respective memory component are filled, the first stage overwrites the memory location storing the oldest filtered audio features in the respective memory component, and then the second stage performs filtering on all of the filtered audio features residing in the respective memory component; a respective memory component, the respective memory component associated with a respective memory capacity of the corresponding neuron and comprising a plurality of memory locations; a first stage configured to perform filtering on the respective audio features of each input frame individually and output the filtered audio features to the respective memory component; and a second stage configured to perform filtering on all the filtered audio features residing in the respective memory component; wherein each neuron is configured such that: the first stage performs filtering on the respective audio features of each input frame individually and outputs the filtered audio features to a next available memory location of the respective memory component, wherein once all of the memory locations of the respective memory component are filled, the first stage overwrites the memory location storing the oldest filtered audio features in the respective memory component, and then the second stage performs filtering on all of the filtered audio features residing in the respective memory component; the first stage performs filtering on the respective audio features of each input frame individually and outputs the filtered audio features to a next available memory location of the respective memory component, wherein once all of the memory locations of the respective memory component are filled, the first stage overwrites the memory location storing the oldest filtered audio features in the respective memory component, and then the second stage performs filtering on all of the filtered audio features residing in the respective memory component; determining, by the data processing hardware, whether the probability score satisfies a hotword detection threshold; and when the probability score satisfies the hotword detection threshold, initiating, by the data processing hardware, a wake-up process on the user device for processing the hotword and/or one or more other terms following the hotword in the audio stream.