###### Abstract  
Speech recognition is widely applied in fields like security, education, and healthcare. While its development drives global information infrastructure and AI strategies, current models still face challenges such as overfitting, local optima, and inefficiencies in decoding accuracy and computational cost. These issues cause instability and long response times, hindering AI's competitiveness. Therefore, addressing these technical bottlenecks is critical for advancing national scientific progress and global information infrastructure. In this paper, we propose improvements to the model structure fusion and decoding algorithms. First, based on the Conformer network and its variants, we introduce a weighted fusion method using training loss as an indicator, adjusting the weights, thresholds, and other related parameters of the fused models to balance the contributions of different model structures, thereby creating a more robust and generalized model that alleviates overfitting and local optima. Second, for the decoding phase, we design a dynamic adaptive decoding method that combines traditional decoding algorithms such as connectionist temporal classification and attention-based models. This ensemble approach enables the system to adapt to different acoustic environments, improving its robustness and overall performance. Additionally, to further optimize the decoding process, we introduce a penalty function mechanism as a regularization technique to reduce the model's dependence on a single decoding approach. The penalty function limits the weights of decoding strategies to prevent over-reliance on any single decoder, thus enhancing the model's generalization. Finally, we validate our model on the Librispeech dataset, a large-scale English speech corpus containing approximately 1000 h of audio data. Experimental results demonstrate that the proposed method achieves word error rates (WERs) of 3.92% and 4.07% on the development and test sets, respectively, significantly improving over single-model and traditional decoding methods. Notably, the method reduces WER by approximately 0.4% on complex datasets compared to several advanced mainstream models, underscoring its superior robustness and adaptability in challenging acoustic environments. The effectiveness of the proposed method in addressing overfitting and improving accuracy and efficiency during the decoding phase was validated, highlighting its significance in advancing speech recognition technology. speech recognition; convolutional neural network; end-to-end model; Conformer; model fusion; dynamic adaptive decoding +
Footnote †: journal: applied sciences  
by 2032 [1]. The advancement of speech recognition not only drives the development of intelligent devices but also has a substantial impact on global information technology infrastructure and artificial intelligence strategies. Speech recognition technology has undergone three major stages of development. Initially, methods based on Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs) were used [2], relying on manual feature extraction and statistical modeling for speech recognition. While these methods performed reasonably well in environments with low noise, they struggled with complex speech inputs. The introduction of deep learning later brought about the application of convolutional neural networks (CNNs) and deep neural networks (DNNs), significantly improving the accuracy of speech recognition, particularly in noisy environments [3]. In recent years, end-to-end architectures have become mainstream, with models such as Transformer. The self-attention mechanism in Transformer models addresses the limitations of traditional models in parallel processing and long-range dependencies, further enhancing recognition accuracy and training efficiency [4].