Plan for 2025:

Collect a dataset from azure and google asr transcriptions and recorded audios to build a fulfield, meaningful dataset using LLM to fix transcriptions which is going to be learned as a ground truth in accordance to recordings.
ASR Fine-tuning (Substitution with proprietary models trained using WER comparison): Fine-tune Nvidia NeMo ASR models using a dataset generated by Azure and Google ASR transcriptions alongside the associated consultation recordings. These recordings come in varying quality and from different microphones. The primary focus will be on Dutch, with additional attention to local accents and regional variations to enhance transcription precision in challenging audio environments. English may also be included.
Dynamic Model Switching: Implement a system that dynamically switches between different ASR or speech enhancement models based on audio quality. By assessing factors like noise level, clarity, and echo presence, the system will select the most appropriate model, reducing costs while ensuring transcription accuracy.
Breaking down into small microservices: Break down the pipeline parts (Diarization + Labeling, Audio processing - Enhancing, Lang Detection, Recognition, Summ and Report generation) into separate microservices, allowing individual components to scale and improve independently based on demand.
Continuous Learning: Implement a continuous learning process by comparing the performance of NeMo's ASR to Google and Azure ASR models (Compared by WER to decide). Identify and store cases where NeMo's transcriptions are less accurate than Google/Azure and build a dataset of these weak spots (both audio and transcription). This dataset will be used for future fine-tuning of NeMo, focusing on its problematic areas, thus improving its overall accuracy over time.
Speech Enhancement Model: Finetune or implement a proprietary speech enhancement model, such as DeepFilterNet or resemble-enhance to handle specific issue like noise in recordings. These models will be optimized for cases that frequently occur in consultations.
Automated Alerts: Set up monitoring tools (e.g., Prometheus, Grafana) to track pipeline performance metrics and trigger alerts for anomalies or failures.
Implementation of word boosting using Nemo framework which is more efficient and impactful than google word boosting
Overview and improvement of summarization, building a fulfield reports based on summaries
Replace gpt4 api with some local light-weight Bert Based Transformer or open-source LLM to summarize raw transcriptions into particular format/ report or summary (or even finetune LLM with early created summaries by gpt and use this model to have the same result without GPT api cost)
Finetune lang detection model for recognize bad quality recordings(in case of if whisper language detection is no capable to be robust in different quality envs)
LLM recommendation for vet. Expanding the responsibilities of LLM by its involvement in generating recommendations for vet like hints based on conversation with Owner about its pet. Medical recommendations emphasizing it in bullet points and next steps during the treatment process. 


P.S just notes
Implementation  of Streaming regime (intent recognition)
Compute the cost of gpu usage for using proprietary approaches with open source models.
Idea of creating a server on site in a clinic to avoid renting gpu.


Can you please improve my thoughts, structure and add more clearity where it's missing