# Lipreading Using Liquid State Machine with STDP-Tuning
###### Abstract  
Lipreading refers to the task of decoding the text content of a speaker based on visual information about the movement of the speaker's lips. With the development of deep learning in recent years, lipreading has attracted extensive research. However, the deep learning method requires a lot of computing resources, which is not conducive to the migration of the system to edge devices. Inspired by the work of Spiking Neural Networks (SNNs) in recognizing human actions and gestures, we propose a lipreading system based on SNNs. Specifically, we construct the front-end feature extractor of the system using Liquid State Machine (LSM). On the other hand, a heuristic algorithm is used to select appropriate parameters for the classifier in the backend. On small-scale lipreading datasets, our recognition accuracy achieves good results. We claim that our network performs better in terms of accuracy and ratio of learned parameters compared to other networks, and has superior advantages in terms of network complexity and training cost. On the AVLetters dataset, our model achieves a 5% improvement in accuracy over traditional methods and a 90% reduction in parameters over the state-of-the-art. lipreading: liquid state machine; STDP +
Footnote †: journal: Applied  
In addition, traditional CNNs require a lot of computing resources, so it is necessary to explore alternative lightweight and energy-efficient lipreading methods. [PERSON] et al. (2018) used dynamic visual sensors and dynamic auditory sensors to transcribe an existing lipreading dataset into an event data stream and used a CNN network to fuse the data streams of the two sensors. This work is the first time to introduce the lip language problem into event domain processing, which is different from the traditional image and video processing. However, this work adopts the method of the artificial neural network and does not explore the method of spiking neural networks that is more suitable for neuromorphic sensors. Spiking Neural Network (SNN) (2018) is a third-generation artificial neural network inspired by brain science, which is composed of biological neuron models. Because of its event-driven characteristics and low power consumption, it has received more and more attention. LSM (2018) is a recurrent neural network based on spiking neurons. It can convert the input spike train into a liquid state. LSM has strong advantages in spatiotemporal data processing, such as gesture recognition and speech signals. These studies (2018; 2019) show the superior performance of LSM in processing spatiotemporal data. However, the method based on a spiking neural network has not been deeply studied in the field of lipreading tasks. Communication between people is not only through voice communication but also involves the interaction of various information such as lipreading, movements, and expressions, and the human brain can handle such tasks well. The spiking neural network as the third-generation neural network inspired by the brain should be more suitable for such tasks. So this paper introduces the spiking neural network into the field of lipreading. In particular, this work considers the spiking neural network's ability to model spatiotemporal sequences and the characteristics of low computational consumption. Based on Liquid State Machines (LSM), we propose a spiking lipreading network model for small-scale lipreading tasks. To improve model performance, we apply STDP rules to our model. Furthermore, we explore the best classifier structures using a heuristic algorithm. In summary, the main contributions of this paper are as follows:  
* We design an LSM structure model with STDP-tuning, which consists of 512 spiking neurons, for the lipreading task. * Under the condition that the LSM structure is fixed, we employ a heuristic search algorithm to search for the best classifier structure parameters. * On the AVLetters dataset, our model achieves a 5% improvement in accuracy over traditional methods.