# Lip2 Speech: Lightweight Multi-Speaker Speech Reconstruction with Gabor Features
## 7 Discussion  
One key finding shows that while many results in the literature use CNN-based performance, using Gabor features can not only generate improved results, but can generate better results with a much lighter model. We were able to remove a number of layers entirely from the model without negatively affecting performance. This, along with the reduced number of input features, had a significant improvement in training time and a reduction in memory usage. We also found that using fewer features makes for a simpler and more intuitive process, as the features can be visualised. The results also showed that by removing other information, the system was able to maintain performance with multiple speakers with less performance loss than the CNN-based model. We experimented with using the filtered Gabor image as input, but ultimately the best performance came with our proposed GaborFea2 Speech approach. Multi-speaker performance is less analysed in the literature than single-speaker models, and in this paper, the strengths of our proposed approach have been demonstrated. One interesting finding was that our Gabor-based features were consistently better than using a pure CNN-based method. It was hypothesised that this approach would deliver similar results but with a much smaller model, but the improvement with using Gabor features, particularly with multi-speaker models, was larger than initially expected. CNN-based approaches learn their own features, and previous research by the authors into deep learning with speech recognition [71] used the Lucid Visualisation Python Library [72] to investigate CNN-based features, finding that distinct features could be seen, focusing on areas such as the outline and thickness of the lips and the mouth and teeth opening (see [PERSON] et al. for more details [71]. Arguably, by applying a Gabor transform, and then extracting features, there is less speaker-specific irrelevant information being learned, resulting in a reduction of the variability between different speakers. In terms of features, although we do extract the standard width and height of the mouth region, we also extract additional features, including orientation, area, and mass, which allows for a more three-dimensional representation to be constructed, which means that we are arguably still retrieving detailed information. As a result of our preprocessing experiments, it was also discovered that effective speech reconstruction can be achieved by matching the Gabor sequence length with the output length of the feature and appropriately expanding the dimensionality of the feature. This shows that there are still limitations with the existing approach, with possible further improvements from better feature design rather than just using derivatives. The findings emphasise the superior performance of GaborFea2 Speech in multi-speaker speech reconstruction, capitalizing on the independence of Gabor Features across speakers. However, the GaborCNN2 Speech model, a GaborCNN-based approach, exhibits limitations in multi-speaker scenarios, likely due to the CNN model's difficulty in automatically distinguishing between speakers. Additionally, the performance of the Lip2 AudSpec model provides further evidence of the constraints of CNN-based speech reconstruction models in multi-speaker scenarios. In terms of the difference between approaches, there were differences between individual speakers in the single-speaker models, but these performance differences tended to be mirrored with both CNN and Gabor approaches, meaning that if the CNN approach worked well with an individual speaker, then the Gabor approaches would also report better scores. This reflects differences in articulation and speaker style and suggests that the Gabor approach is not learning anything fundamentally different from the CNN approach, just arguably a more optimised version.