A method comprising: receiving, at data processing hardware of a user device, a sequence of input frames characterizing streaming multi-channel audio captured by an array of microphones in communication with the data processing hardware, each channel of the streaming multi-channel audio comprising respective audio features captured by a separate dedicated microphone in the array of microphones; for each input frame: processing, by the data processing hardware, using a three-dimensional, 3D, singular value decomposition filter, SVDF, input layer of a memorized neural network, the respective audio features of each channel of the streaming multi-channel audio in parallel; and generating, by the data processing hardware, using an intermediate layer of the memorized neural network, a corresponding multi-channel audio feature representation based on a concatenation of the respective audio features of each channel of the streaming multi-channel audio; generating, by the data processing hardware, using sequentially-stacked SVDF layers of the memorized neural network, a probability score indicating a presence of a hotword in the streaming multi-channel audio based on the corresponding multi-channel audio feature representation of each input frame; determining, by the data processing hardware, whether the probability score satisfies a hotword detection threshold; and when the probability score satisfies the hotword detection threshold, initiating, by the data processing hardware, a wake-up process on the user device for processing the hotword and/or one or more other terms following the hotword in the streaming multi-channel audio. receiving, at data processing hardware of a user device, a sequence of input frames characterizing streaming multi-channel audio captured by an array of microphones in communication with the data processing hardware, each channel of the streaming multi-channel audio comprising respective audio features captured by a separate dedicated microphone in the array of microphones; for each input frame: processing, by the data processing hardware, using a three-dimensional, 3D, singular value decomposition filter, SVDF, input layer of a memorized neural network, the respective audio features of each channel of the streaming multi-channel audio in parallel; and generating, by the data processing hardware, using an intermediate layer of the memorized neural network, a corresponding multi-channel audio feature representation based on a concatenation of the respective audio features of each channel of the streaming multi-channel audio; processing, by the data processing hardware, using a three-dimensional, 3D, singular value decomposition filter, SVDF, input layer of a memorized neural network, the respective audio features of each channel of the streaming multi-channel audio in parallel; and generating, by the data processing hardware, using an intermediate layer of the memorized neural network, a corresponding multi-channel audio feature representation based on a concatenation of the respective audio features of each channel of the streaming multi-channel audio; generating, by the data processing hardware, using sequentially-stacked SVDF layers of the memorized neural network, a probability score indicating a presence of a hotword in the streaming multi-channel audio based on the corresponding multi-channel audio feature representation of each input frame; determining, by the data processing hardware, whether the probability score satisfies a hotword detection threshold; and when the probability score satisfies the hotword detection threshold, initiating, by the data processing hardware, a wake-up process on the user device for processing the hotword and/or one or more other terms following the hotword in the streaming multi-channel audio.