# A Large Language Model and Qualitative Comparative Analysis-Based Study of Trust in E-Commerce
## 4 Results
### Evaluating the Performance of LLMs  
We applied five widely used evaluation metrics to quantify the performance of each model: accuracy, precision, recall, F1 and specificity scores. These metrics are essential for evaluating the performance of machine learning models, particularly in classification tasks [46]. The evaluation metrics are calculated based on the elements of the confusion matrix: (i) true positive (TP), correctly predicting a positive outcome, (ii) true negative (TN), correctly predicting a negative outcome, (iii) false positive (FP), incorrectly predicting a positive outcome, and (iv) false negative (FN), incorrectly predicting a negative outcome. Precision, defined as the ratio of TP predictions to the sum of TP and FN predictions, measures the accuracy of positive predictions and is crucial when the cost of FPs is high. Recall is the ratio of TP predictions to the sum of TP and FN predictions. It assesses the model's ability to identify all relevant instances, and it is vital when missing positive cases is costly. The F1 score, the harmonic mean of precision and recall, provides a balanced measure of a model's performance, especially when the dataset is imbalanced. Accuracy is the proportion of correct predictions (both TPs and TNs) out of all predictions. Specificity measures the proportion of TNs out of all actual negatives. Specificity is useful for understanding how well the model identifies the absence of a certain aspect. Together, these metrics provide a comprehensive evaluation of model performance, addressing different aspects of classification accuracy and reliability. In this study, we fine-tuned and trained the two LLMs on our manually annotated dataset. After training, we evaluated the performance of the models using a separate validation set, which was also used for error analysis. The validation set consists of manually annotated true responses (aspects present in the review) and the corresponding responses generated by the models (aspects predicted to be in the review). The elements of the confusion matrix are calculated in this case by comparing the set of actual aspects (True_response) and predicted aspects (Predicted_response) for each sample. Specifically, the True_response and Predicted_response columns are first split into sets of aspects for each sample. The union of all possible aspects across the dataset is then obtained to ensure that all relevant aspects are considered in the evaluation. For each aspect, TPs are counted as the instances where the aspect appears in both the actual and predicted sets. FPs are cases where the aspect appears in the predicted set but not in the actual set, indicating an incorrect prediction. FNs occur when the aspect is present in the actual set but missing from the predicted set, representing an aspect the model failed to identify. Finally, TNs are calculated as the instances where the aspect is absent from both the actual and predicted sets, indicating correct identification of an irrelevant aspect. This comparison across all samples allows for calculating precision, recall, F1 score, accuracy, and specificity for each aspect by aggregating these counts. The results for the two models are presented in Tables 4 and 5. The Gamma model exhibited high performance in several aspects, particularly in predicting positive sentiments for shipping (precision: 0.89, recall: 0.97, F1 score: 0.93) and trust (precision: 0.98, recall: 0.97, F1 score: 0.98). It also performed well in predicting negative sentiments for Customer_service (precision: 0.77, recall: 0.94, F1 score: 0.85). The Llama model, on the other hand, showed consistently high performance across various aspects. It excelled in predicting positive sentiments for Refund_process (precision: 1.00, recall: 1.00, F1 score: 1.00) and trust (precision: 0.98, recall: 1.00, F1 score: 0.99). Additionally, it performed strongly in negative sentiments for Refund_process (precision: 1.00, recall: 0.92, F1 score: 0.96) and Customer_service (precision: 0.84, recall: 0.89, F1 score: 0.86).