## 2 Related Work  
Recently, many researchers have proposed deep learning-based AVSR and VSR schemes [1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21]. As mentioned, most conventional VSR research has focused on frontal face images, assuming that VSR systems are in front of speakers, since there are only a few data sets available with multi-angle faces. Here, we introduce several lipreading works focusing not only on frontal but also diagonal and profile images. To develop these schemes, we need a research corpus. One of the public multi-angle VSR data sets is OuluVS2 [22]. An early work of multi-angle lipreading is [1], where a system was trained using either frontal (\(0^{\circ}\)) or profile (\(90^{\circ}\)) faces. According to the experimental results, the frontal view showed a lower word error rate (WER) than the profile view. In [2], the authors built a multi-angle system investigating a frontal (\(0^{\circ}\)) view, a left profile (\(90^{\circ}\)) view and a right profile (\(-90^{\circ}\)) view. They reported significantly better performance when using the frontal view than the others. [PERSON] et al. proposed a novel sequence image representation method called concatenated frame image (CFI) [3]. Two types of data augmentation methods for CFI, and a framework of a CFI-based CNN, were tested. [PERSON] et al. indicated that human lipreaders tend to have higher performance when slightly angled faces are available, presumably because of the visibility of lip protrusion and rounding [4]. In [5], the active appearance model (AAM) was utilized for feature extraction at five angles, and lipreading was examined on a view-dependent system, as well as on a view-independent system using a regression method in a feature space. As a result, the view-dependent system performed the best performance at 30\({}^{\circ}\) in all tests. [PERSON] et al. used principal component analysis (PCA)-based convolutional networks together with Long short-term memories (LSTMs), one of the DL models, in addition to a conventional speech recognition model, hidden Markov models (HMMs) with Gaussian mixture models (GMMs) [6]. They aimed at combining multiple views by employing these techniques. They finally confirmed that the highest performance was obtained at 30\({}^{\circ}\). [PERSON] et al. stated that the highest accuracy was achieved at 60\({}^{\circ}\) in their experiments [22]. [PERSON] et al. showed that profile-view lipreading provides significantly lower WERs than frontal-view lipreading [7]. There is another strategy to conduct transformation to images or incorporate several views with DL technology. There is one work [8] that involved converting faces viewed from various directions to frontal faces using AAMs. The experimental results showed that recognition accuracy was improved even when the face direction changed about 30\({}^{\circ}\) relative to a frontal view. In [9], the authors proposed a scheme called \"View2 View\" using an encoder-decoder model based on CNNs. The method transformed non-frontal mouth region images into frontal ones. Their results showed that the view-mapping system worked well for VSR and AVSR. [PERSON] et al. introduced a pose normalization technique and performed speech recognition from multiple views by generating virtual frontal views from non-frontal images [10]. In [11], [PERSON] et al. proposed an end-to-end multi-view lipreading system based on bidirectional LSTM networks. This model simultaneously extracted features directly from the pixels and performed visual speech classification from multi-angle views. The experimental results demonstrated that the combination of frontal and profile views improved accuracy over the frontal view. [PERSON] et al. also proposed another decision fusion-based lipreading model [12]; they extracted features through a PCA-based convolutional neural network, LSTM network and GMM-HMM scheme. The decision fusion succeeded by combining Viterbi paths. In [13], [PERSON] et al.