help me revise the speech for ppt slides.
I will be presenting our call-based model for assessing cognitive decline. Based on CD age distribution, we decide to only focus on elder clients, with age above 70.  
For this data, we do have labeled positive case, which is from the cases reported by crew and IP team decisions. The cases are reported based on call level, so we prepared data on the call level as well. We sampled around 10k non-CD clients, and randomly selected one phone call for each client as negative cases. Time range we focus on between Oct. 2022 to May 2024. The overall data size is around 12K. 
Major effort is focus on feature generation. Since crew notice the issue when they talked to client. So initial model only use transcript and age. Later, after talking to IP team and refer to literature, we generate more features from three main channels. Besides commonly used features such as age, gender, race, marital status, we include more to describe clients, like number of adults or children, how long live in the current house, social economic index and features surrounding area, like population density. 
Features how they interact with vanguard crew such as IVR time, talk time, number of phone calls made on the same day, in past week or month. 
Audio features are very import. Like what IP teams make the decision, it is not just what they say, and how they say it really matters. So we generated similar auto feature from transcripts, including spear rate, sentence length, no of short sentences, pause words and number of negative sentiment sentences. These features cannot be directly read from transcript itself, but provide better indication of how they say it. 
We performed a Chi-square test for categorical features and a t-test for numeric features, with all tests indicating statistical significance. The analysis showed that clients experiencing cognitive decline tend to live in households with fewer adults or children. They also spent more time either in the IVR system or talking to crew members and made substantially more phone calls each day, week, or month. Additionally, they tended to speak more slowly, use shorter sentences, take more pauses, and express a higher frequency of negative sentiment.
Let’s turn our attention to the modeling approach we chose. With the data prepared, we chose to implement a model known as the Multimodal Transformer, which can process both tabular features, features listed in the previous slides and text features, referred as call transcript here. This model integrates text embeddings and constructs a neural network alongside other input features. To evaluate model performance, we selected the Precision-Recall AUC (Area Under the Curve) as our metric since our data is imbalanced. Our goal is to accurately identify positive cases rather than merely finding all cases, so precision remains a key focus.

With the 12k data prepared. 70% of them are used to train the model, while 30%, around 3300 are the test data we used to evaluate model here. The baseline model—where we assume all cases are positive—yielded a PR-AUC of 0.1, while our model achieved a PR-AUC of 0.43. The confusion matrix displayed alongside normalizes data along the y-axis, directly reflecting precision. Our model recorded a precision of 0.51 and a recall of 0.31.

Given that we only refer cases with the highest probability to the IP team, we focused on the left side of this graph, where precision reaches around 80%. However, it is essential to remember that our test data’s class ratio is 1:9. In practice, we need to screen approximately 30,000 calls each week, where the positive cases ratio stands at about 1:500 leading to an extremely imbalanced dataset. Consequently, as we refer cases to the business, the false positive rate increases, resulting in a lower precision of 37%. 

Our initial model based on age and transcript only has pr-auc of 0.37. With more features adding to the model, it seems that improving model significantly proved challenging. A BERT model within our multimodal approach for transcript analysis. BERT is relatively small in terms of parameters, and transcripts should be primary drivers since agents identify issues through client interactions. This led us to explore the potential of using a Large Language Model (LLM).

We successfully improved referral precision to 61%. The IP team set five criteria for agents to determine potential cognitive decline in clients. We crafted a prompt for the LLM to assess whether a client exhibits signs of cognitive decline based on the transcript and, if so, to provide a reason code.
Now, let’s take a deeper dive into the analysis of these reason codes. Our analysis of these reason codes revealed that the model lacked confidence in one specific code related to Vanguard services, highlighting the need for a deeper understanding of our offerings.
Combining codes demonstrated that an increased number of identified reasons correlates with higher precision. Referring back to the confusion matrix, we found that the multimodal model misclassified many cases as positive, resulting in a precision of only 37%. However, when we applied the LLM to these referrals, it successfully eliminated 20 negative cases. While it did miss five positive cases, this adjustment improved our precision to 61%.


In summary, our overall approach moving forward will be effective in leveraging both models. Our overall approach going forward will involve using the multimodal model to screen weekly calls for older clients, selecting the top cases as initial referrals. Then, we will use the LLM model to further evaluate these cases and compile a final list for referral. Although we could use the LLM model initially to generate additional features, this would be costly given the need to review 30,000 call transcripts each week.