A method comprising: receiving, at data processing hardware of a user device , a sequence of input frames characterizing streaming multi-channel audio captured by an array of microphones in communication with the data processing hardware , each channel of the streaming multi-channel audio comprising respective audio features captured by a separate dedicated microphone in the array of microphones ; for each input frame : processing, by the data processing hardware , using a three-dimensional, 3D, singular value decomposition filter, SVDF, input layer of a memorized neural network , the respective audio features of each channel of the streaming multi-channel audio in parallel; and generating, by the data processing hardware , using an intermediate layer of the memorized neural network , a corresponding multi-channel audio feature representation based on a concatenation of the respective audio features of each channel of the streaming multi-channel audio ; generating, by the data processing hardware , using sequentially-stacked SVDF layers of the memorized neural network , a probability score indicating a presence of a hotword in the streaming multi-channel audio based on the corresponding multi-channel audio feature representation of each input frame ; determining, by the data processing hardware , whether the probability score satisfies a hotword detection threshold; and when the probability score satisfies the hotword detection threshold, initiating, by the data processing hardware , a wake-up process on the user device for processing the hotword and/or one or more other terms following the hotword in the streaming multi-channel audio . receiving, at data processing hardware of a user device , a sequence of input frames characterizing streaming multi-channel audio captured by an array of microphones in communication with the data processing hardware , each channel of the streaming multi-channel audio comprising respective audio features captured by a separate dedicated microphone in the array of microphones ; for each input frame : processing, by the data processing hardware , using a three-dimensional, 3D, singular value decomposition filter, SVDF, input layer of a memorized neural network , the respective audio features of each channel of the streaming multi-channel audio in parallel; and generating, by the data processing hardware , using an intermediate layer of the memorized neural network , a corresponding multi-channel audio feature representation based on a concatenation of the respective audio features of each channel of the streaming multi-channel audio ; processing, by the data processing hardware , using a three-dimensional, 3D, singular value decomposition filter, SVDF, input layer of a memorized neural network , the respective audio features of each channel of the streaming multi-channel audio in parallel; and generating, by the data processing hardware , using an intermediate layer of the memorized neural network , a corresponding multi-channel audio feature representation based on a concatenation of the respective audio features of each channel of the streaming multi-channel audio ; generating, by the data processing hardware , using sequentially-stacked SVDF layers of the memorized neural network , a probability score indicating a presence of a hotword in the streaming multi-channel audio based on the corresponding multi-channel audio feature representation of each input frame ; determining, by the data processing hardware , whether the probability score satisfies a hotword detection threshold; and when the probability score satisfies the hotword detection threshold, initiating, by the data processing hardware , a wake-up process on the user device for processing the hotword and/or one or more other terms following the hotword in the streaming multi-channel audio . <EOS>