# Beyond Manual Media Coding: Evaluating Large Language Models and Agents for News Content Analysis
## 3 Materials and Methods
### Evaluation Metrics  
The codebook questions are clearly differentiated by both their type and the nature of their expected responses. At a high level, this differentiation is based on the number of labels permitted per question--specifically, whether the model is expected to select a single value or multiple values. While this classification is useful for pre-processing and formatting during implementation, its primary function is to support a deeper semantic categorization that plays a crucial role in the final evaluation process, as we organize the questions into three distinct groups: binary, categorical, and multi-label. Binary questions require a single boolean response, either true or false, and typically represent the presence or absence of a specific feature or characteristic in the news content. Categorical questions also allow only one response, but from a predefined list of mutually exclusive values, and are used when a piece of content must be classified into a single category within a defined taxonomy. Multi-label \begin{table}
\begin{tabular}{p{85.4 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Agent** & **Profile Instructions** & **Planning and Actions** \\ \hline \multirow{3}{*}{Coordinator} & Orchestrates workflow execution across agents; manages sequence and agent hand-offs & \multirow{3}{*}{Agent coordination} \\ \cline{2-2} \cline{4-4} \multirow{3}{*}{Setup Agent} & Parses article input, extracts metadata, and prepares structured content for downstream agents & \multirow{3}{*}{Performs preprocessing: parses content, extracts identifiers, and stores structured input} \\ \hline \multirow{3}{*}{Media Analyst} & Expert in journalism and media framing; analyzes content classification, authorship, and editorial stance & \multirow{3}{*}{Examines how information is presented; applies framing and authorship heuristics; and addresses media-specific question subset} \\ \cline{2-2} \multirow{3}{*}{Political Analyst} & Specialist in political leadership, governance, and international relations; detects sentiment and ideological framing & \multirow{3}{*}{Analyzes political actors; detects implicit and explicit sentiment; and addresses political and governance questions} \\ \hline \multirow{3}{*}{Trade Economist} & Expert in international trade and economic impact; analyzes tariffs, market responses, and industry implications & \multirow{3}{*}{Identifies affected industries; assesses trade dynamics and economic reasoning} \\ \cline{1-1} \cline{2-2} \multirow{3}{*}{Reviewer Agent} & Synthesis expert; integrates multiple expert perspectives into a single structured schema & \multirow{3}{*}{Maps expert insights to the codebook schema; checks consistency and resolves ambiguity} \\ \hline \multirow{3}{*}{Format Converter} & Schema compliance agent; finalizes JSON structure and ensures output validity & \multirow{3}{*}{Converts structured output into schema-compliant JSON; ensures formatting} \\ \cline{1-1} \cline{2-2} \cline{4-4} \multirow{3}{*}{Reviewer Agent} & Synthesis expert; integrates multiple expert perspectives into a single structured schema & \multirow{3}{*}{Maps expert insights to the codebook schema; checks consistency and resolves ambiguity} \\ \cline{1-1} \cline{2-2} \cline{4-4} \multirow{3}{*}{Format Converter} & Schema compliance agent; finalizes JSON structure and ensures output validity & \multirow{3}{*}{Converts structured output into schema-compliant JSON; ensures formatting} \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Agents’ profiles and planning characteristics. questions permit multiple responses from an also fixed set of predefined values and are appropriate when the content may relate to several applicable categories simultaneously. This three-tiered classification system not only improves prompt design and model alignment but also informs our evaluation metrics, as each group requires distinct performance criteria and error analysis techniques. It ensures that model outputs are both interpretable and consistent with the logical structure of our coding framework.