# Beyond Manual Media Coding: Evaluating Large Language Models and Agents for News Content Analysis
## 4 Results  
### Model-Based Direct LLM Prompting Evaluation  
After executing the finalized codebook across all selected LLMs, alongside the corresponding news content, we collected each model's predicted answers for all questions, applied to every curated article in the dataset. This process yielded a set of model-generated annotations, reflecting how each LLM interpreted and tagged the content based on the provided schema, global prompt, and codebook instructions. Each model's predictions were systematically compared against the ground-truth annotations produced by our expert human annotator. This comparative analysis allowed us to assess the performance of the models across a wide range of question types and tagging tasks, forming the empirical basis for our quantitative assessment. In addition, it enabled us to examine and compare the relative performance of each model within the context of its respective tier, i.e., low, middle, or high.  
The evaluation of F1-scores across 26 questions, categorized into binary, categorical, and multi-label types, reveals clear and consistent performance disparities among the seven models assessed. Table 4 provides an aggregated overview of average F1-scores for each question group (binary, categorical and multi-label) and a global score that represents the model's overall tagging effectiveness across the full codebook on a per-model basis. Overall, the commercial model Claude-3-7-Sonnet (M6) achieved the highest average F1-score at 0.822, demonstrating superior performance across all question types. It was especially strong in multi-label tasks, which proved to be the most challenging category for all models. The model's performance in all question categories suggests a high degree of generalization, making it particularly suitable for applications that demand nuanced judgment across diverse types of analysis tasks.  
DeepSeek-R1 (M4) and its distilled variant DeepSeek-R1-Distill-Qwen (M5) also perform robustly, achieving global F1-scores of 0.800 and 0.787, respectively. These models show particular strength in binary tasks, and while not as dominant as Claude-3-7-Sonnet in multi-label performance, they remain reliable choices for diverse classification needs. Meta-Llama-3-3 (M3) ranks slightly behind these models overall but performs competitively in binary and categorical tasks, making it a solid middle-tier option. On the other end of the spectrum, Meta-Llama-3-1 (M7) underperforms. With a global F1 score of just 0.636, it struggles most in multi-label classification, often showing the lowest scores across those questions. Even in binary and categorical tasks where other models perform well, its results remain well below average, showcasing the difficulties low-tier models have in complex coding-based analysis.  
Meta-Llama-4-Maverick (M1) delivered solid results, outperforming mid-tier models like Qwen3 (M2) and Llama-3-1 (M7), yet it consistently lagged behind DeepSeek-R1 (M4) and Claude-3-7-Sonnet (M6), particularly on categorical and multi-label tasks. This suggests that a large scale alone does not ensure top-tier performance in complex structured coding and reasoning. These findings motivated our development of an agentive system to determine whether Meta-Llama-4-Maverick (M1), using coordinated, role-specialized agents that reason according to their expert roles, could surpass the limitations observed in direct single-model runs.