## 6 Automated Accuracy Evaluation
### Metric Selection  
As mentioned, the gold-standard reference text is based on systematic reviews, making it likely to incorporate technical language. DietNerd, on the other hand, generates text that is meant to be consumer-friendly. Because of this difference in target audiences, the focus in our evaluation is on whether the final conclusions matched semantically rather than syntactically. Therefore, to assess the alignment between DietNerd's generated text and the gold-standard reference text, two similarity metrics were employed: BERTScore [32] and Semantic Textual Similarity.  
BERTScore analyzes textual similarity based on token-level contextual embeddings, focusing on finer lexical details while still taking into consideration semantic similarity. It outputs an F1 score that is rescaled so that values near or below 0 indicate poor similarity and values closer to 1 indicate higher similarity.  
Semantic Textual Similarity evaluates the overall similarity in the conclusions reached by both texts by calculating the cosine similarity on a scale of \(-1\) to 1, with 1 denoting absolute semantic equivalence. To calculate Semantic Textual Similarity, the model all-mpnet-base-v2-negation was used. Built upon the Sentence Transformer base model all-mpnet-base-v2 [33], this model is fine-tuned on the Compilation of ANnotated, Negation-Oriented Text-pairs (CANNOT) dataset, making it much more sensitive to negations [34]. The Universal Sentence Encoder (USE) Q&A model [35] was considered but, since all-mpnet-base-v2's underlying MPNet architecture incorporates permutation-based training, it potentially offers more versatility and flexibility when dealing with the varied sentence structures and complex language commonly found in medical texts.  
While other evaluation metrics are widely used in LLM research, we determined that they were less suitable for our specific objectives. For instance, the Bilingual Evaluation Understudy (BLEU) [36], commonly used in text generation tasks, focuses on n-gram precision, which does not align well with our goal of assessing semantic understanding. BLEU can penalize semantically correct outputs that use different phrasing. Similarly, Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [37] emphasizes lexical overlap, which does not capture the full range of acceptable variations in LLM outputs. Even ROUGE-WE (ROUGE with Word Embeddings) [38], an extension of ROUGE designed to better capture semantic similarities, was not a good fit because its use of static word embeddings limits its ability to account for the different meanings a word might have in different contexts. Our chosen metrics allow for greater flexibility in assessing semantically equivalent but lexically diverse responses, reflecting our research objectives.  
Table 4 demonstrates the evaluation focus of BERTScore versus Semantic Textual Similarity. In this example, BERTScore's F1 score between the reference and generated texts was low at 0.043 reflecting a token-level difference. By contrast, Semantic Textual Similarity's score rated the texts significantly similar at a 0.788 level. The reason for that high score is that the two texts reach the same essential conclusion: PUFAs were shown to have benefits for some health markers but have no proven impact on others.