ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities
Authors: Zhaochen Hong, Haofei Yu, Jiaxuan You
Accepted at: acl25

ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities

Zhaochen Hong1* Haofei Yu1* Jiaxuan You1
1University of Illinois Urbana-Champaign
{zhong42,haofeiy2,jiaxuan}@illinois.edu

Abstract

Evaluating consistency in Large Language Models (LLMs) is crucial for ensuring reliability, particularly in complex, multi-step interactions between humans and LLMs. Traditional self-consistency methods often miss subtle semantic changes in natural language and functional shifts in code or equations, which can accumulate over multiple transformations. To address this, we propose ConsistencyChecker, a tree-based evaluation framework designed to measure consistency through sequences of reversible transformations, including machine translation tasks and AI-assisted programming tasks. In our proposed framework, nodes represent distinct text states, while edges correspond to pairs of inverse operations. Dynamic and LLM-generated benchmarks ensure a fair assessment of the model’s generalization ability and eliminate benchmark leakage. Consistency is quantified based on similarity across different depths of the transformation tree. Experiments on eight models from various families and sizes show that ConsistencyChecker can distinguish the performance of different models. Notably, our consistency scores, computed entirely without using WMT paired data, correlate strongly (\( r > 0.7 \)) with WMT 2024 auto-ranking, demonstrating the validity of our benchmark-free approach. Our implementation is available at https://github.com/ulab-uiuc/consistencychecker.

1 Introduction

Large Language Models (LLMs) have emerged as transformative tools in artificial intelligence, demonstrating remarkable capabilities in natural language understanding (Minaee et al., 2024), generation (Minaee et al., 2024), and complex reasoning (Brown et al., 2020; Chowdhery et al., 2024). These models are increasingly deployed in applications such as multilingual translation pipelines (Xu et al., 2023) and automated code generation (Jiang et al., 2024b), all of which require maintaining semantic and functional consistency across multiple interactions. A fundamental research question thus arises: Can we effectively evaluate the consistency of LLMs over multi-step transformations?

Challenges for evaluating consistency. Evaluating LLM consistency across multi-step transformations is inherently difficult. Existing methods, such as self-consistency sampling (Min et al., 2023; Wang et al., 2022), which typically assess agreement between outputs for a single prompt, could fail to capture how meaning may drift over a sequence of transformations. This limitation is critical in tasks like machine translation and code generation, where minor changes at each iteration,

![Overview of the ConsistencyChecker. It shows a self-consistency tree for evaluating LLMs on machine translations. The root node (r) is the initial English sentence generated by the evaluator. Subsequent nodes (\( v_i \)) are produced by the evaluator using transformation pairs, such as English→Spanish→English (\( f_1, f_1' \)) and English→German→English (\( f_2, f_2' \)). The evaluation framework supports multilingual translations (e.g., French, Czech, Japanese) and can be extended to programming tasks.](page_1012_682_393_355.png)

Figure 1: Overview of the ConsistencyChecker. It shows a self-consistency tree for evaluating LLMs on machine translations. The root node (r) is the initial English sentence generated by the evaluator. Subsequent nodes (\( v_i \)) are produced by the evaluator using transformation pairs, such as English→Spanish→English (\( f_1, f_1' \)) and English→German→English (\( f_2, f_2' \)). The evaluation framework supports multilingual translations (e.g., French, Czech, Japanese) and can be extended to programming tasks.
though locally coherent, can accumulate into major inconsistencies after multiple steps. For example, a translated sentence may gradually shift semantically, and a code snippet may become non-executable after multiple transformations. Effective evaluation must therefore assess semantic and functional consistency across the multi-step trajectory. For simplicity, we consider semantic consistency to be a subset of functional consistency, which mostly applies to natural language, whose functional consistency is equivalent to its semantic information-keeping consistency.

Challenges in evaluating generalization abilities. Most benchmarks aim to evaluate how LLMs perform on unseen data in specific domains, which is to assess their generalization capabilities. Existing evaluation approaches primarily rely on dataset-centric benchmarks (e.g., HumanEval (Chen et al., 2021) for code generation). The dataset construction process includes human annotation or crawling of real-world data from the Internet. However, due to the vast and opaque nature of LLM training data, it has become increasingly difficult to determine whether benchmark examples were truly unseen during training. Several studies have found evidence of benchmark leakage (Xu et al., 2024; Zhou et al., 2025), where evaluation data overlaps with training corpora, undermining the reliability of performance metrics. Therefore, there is a growing need for novel benchmarking methodologies that dynamically generate evaluation data to ensure a more faithful assessment of model generalization.

Novel tree-based benchmark-free evaluation. To address these limitations, we introduce ConsistencyChecker, a tree-based and benchmark-free evaluation framework that operates without reference datasets, enabling cheap and domain-agnostic assessments of LLM generalization capabilities. At its core, ConsistencyChecker constructs self-consistency trees, where each node represents an LLM-generated state in transformations (e.g., output of one translation step), each edge denotes a pair of inverse operations, and each path in the self-consistency tree represents a multi-step transformation sequence operated by the LLM. For instance, starting with an English sentence, we perform a round-trip translation (English-French-English) executed by the LLM, yielding a slightly varied English sentence as a subsequent node. By analyzing consistency across nodes at different depths, ConsistencyChecker quantifies an LLM’s reliability in retaining consistency (e.g. a sentence still conveys the same information, or code still gives the same output given the same inputs) through iterative transformations. ConsistencyChecker highlights two key innovations: (1) utilizing a tree structure to comprehensively evaluate consistency across multi-step transformation processes; (2) utilizing dynamically sampled data from LLMs to prevent benchmark leakage during evaluation.

Key findings. Based on our experimental results on both machine translation and AI-assisted programming tasks, we find that:(1) ConsistencyChecker can effectively distinguish the performance of different models and assign them varying scores, demonstrating its utility as a distinguishable metric for both tasks; (2) GPT-4o-mini ranks highest in machine translation tasks, while Qwen-2.5-32B performs best in code generation tasks, indicating that evaluation results align with these models’ performance on other benchmarks; (3) ConsistencyChecker can operate without relying on predefined benchmarks and still produces rankings highly correlated with expensive evaluations relying on paired data on the WMT 2024 benchmark.

2 Related Works

Round-trip consistency in translations. Machine translation evaluation has evolved from early rule-based systems (Weaver, 1952) to modern transformer-based approaches (Vaswani et al., 2017). Traditional metrics like BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and TER (Snover et al., 2006) rely on reference translations, while benchmarks like WMT 2024 (Barrault et al., 2019) depend on parallel corpora crawled from internet resources, often unavailable in low-resource languages or specialized domains. Round-trip translation, where text is translated between languages and back (van Zaanen and Zwarts, 2006), provides a benchmark-free quality checker analogous to our transformation chains. ConsistencyChecker generalizes this concept, extending it beyond translation to arbitrary functional-preserving operations (e.g., AI-assisted programming), thereby enabling evaluation in scenarios where parallel data or reference outputs are absent.

Consistency-based evaluation. Recent LLM evaluation frameworks, such as Divide-Conquer-Reasoning (Cui et al., 2024) and MT-Eval (Kwan et al., 2024), improve consistency assessment by breaking down tasks and using dynamic bench-
marks. However, they often miss tracking semantic or functional drift across complex, multi-step transformations (Wang et al., 2024). Our work builds on these ideas by introducing an explicit tree-structured evaluation, which systematically measures consistency at every step and path. Uniquely, we also use LLMs to generate dynamic benchmarks, enabling comprehensive and scalable consistency evaluation beyond existing methods.

Formal verification method. The development of reliable systems has traditionally depended on formal verification techniques such as model checking (Clarke et al., 1999), theorem proving (Nipkow et al., 2002), and SMT solvers (Barrett and Tinelli, 2018), all of which require significant computational resources and domain-specific specifications. The recent rise of LLMs for code generation has led to benchmarks like CodeXGLUE (Lu et al., 2021) and ProblemSolving (Hendrycks et al., 2021), but these approaches rely on curated datasets, which limits their general applicability. Inspired by both formal methods and the input/output-based evaluation used by online programming platforms like LeetCode, we instead use a set of test inputs to verify the functionality of LLM-edited Python code. Unlike traditional benchmarks, our approach does not require predefined correct outputs; instead, we concatenate and compare the outputs to assess functional consistency between different nodes.

3 Preliminary

In this section, we first provide detailed information on the definition of the self-consistency tree, which is the key concept for our proposed Consistency-Checker framework. Furthermore, we introduce two derived concepts of the self-consistency tree: paths and forests. These concepts are necessary for the calculation of our proposed consistency score.

3.1 Self-consistency Tree

To evaluate how well LLMs preserve consistency across reversible transformations, we begin by introducing our definition of operation, which refers to real-world tasks like translation or code editing defined by prompts and will be executed by the evaluatee LLM (the LLM to be evaluated). These operations are paired with their inverses to enable round-trip consistency checks. We then define nodes as intermediate model outputs and edges as paired operations. Together, they form the self-consistency tree, a structure that models the multi-step transformation process and allows consistency to be evaluated at different path lengths.

Operation (basic block). An operation is defined as a prompt-driven transformation performed by the evaluated LLM. For each prompt \( p \), we define an operation \( f_p \) and its inverse \( f_p' \), such that applying \( f_p \) followed by \( f_p' \) ideally recovers the original input. Formally,

\[
f_p : c \to c',
\]

where \( c \) is the input (e.g., code or text) and \( c' \) is the output after the transformation. The inverse operation \( f_p' \) is constructed so that \( f_p'(f_p(c)) \approx c \) if the evaluated LLM is ideal.

Node (LLM-generated state). Each node \( v = (c, \mathcal{I}) \) is a tuple representing a single LLM generation and its associated test inputs: (1) \( c \in \Sigma^* \) is the generated content (e.g., code, function, text); (2) \( \mathcal{I} = [i_1, \ldots, i_n] \) is a list of test inputs used to verify functional behavior. Here, \( \Sigma^* \) denotes the set of all possible strings. Regardless of output format (natural language, code, etc.), we treat each generated content \( c \) as an executable function for the evaluation of unified consistency. For semantic similarity evaluation in machine translation tasks, we consider it as a special case where \( I \) is an empty set and the executed outcomes remain \( c \).

![Key concepts in the self-consistency tree (operation, node, and edge). It provides a concrete example of the self-consistency tree in the AI-assisted programming task. The node v contains a function that returns the product of two positive integers, and I is its set of inputs. The prompt p asks the evaluatee LLM to rewrite the code in node v to use a looped sum instead of just multiplying, while p' asks the same LLM to alter it back to simply multiplying.](page_765_180_579_495.png)

Figure 2: Key concepts in the self-consistency tree (operation, node, and edge). It provides a concrete example of the self-consistency tree in the AI-assisted programming task. The node \( v \) contains a function that returns the product of two positive integers, and \( \mathcal{I} \) is its set of inputs. The prompt \( p \) asks the evaluatee LLM to rewrite the code in node \( v \) to use a looped sum instead of just multiplying, while \( p' \) asks the same LLM to alter it back to simply multiplying.
Edge (paired operations). Each edge represents an operation and its inverse, \((f_p, f_{p'})\). It connects two nodes that are evaluated on the same set of test inputs, ensuring that functional consistency can be assessed under identical conditions. Formally, an edge \(e_{ij} = (v_i, v_j)\) connects nodes \(v_i = (c_i, \mathcal{I}_i)\) and \(v_j = (c_j, \mathcal{I}_j)\) if and only if

\[
\left\{
\begin{array}{l}
\mathcal{I}_i = \mathcal{I}_j, \\
c_j = f_{p'}(f_p(c_i)).
\end{array}
\right.
\]

Self-consistency tree. Combining these concepts, a self-consistency tree \(\mathcal{T} = (\mathcal{V}, \mathcal{E})\) models the multi-step process by which an LLM transforms a root input through a sequence of operations. This hierarchical structure enables comprehensive measurement of how well the LLM preserves semantics or functionality over multiple transformation steps, with verification performed using the shared input set \(\mathcal{I}\). Figure 1 illustrates the construction of nodes, edges, and operations. The tree generation process, detailed in Algorithm 1, uses operation pairs and root nodes. Notably, the branching factor at each layer of the tree is determined by the number of available operation pairs.

3.2 Derived Concepts: Path and Forest

Based on the definition of the self-consistency tree, we can extend it and define more useful concepts on top of it. These derived concepts help us calculate our proposed consistency score.

Path. A path in the self-consistency tree is defined as a sequence of nodes \(P = (v_1, ..., v_n)\), where each node is sequentially connected with the next through a valid edge in the self-consistency tree. The path represents a series of inverse operations applied by the evaluatee LLM, and its length corresponds to the number of operation pairs (the depth difference between the starting and ending nodes), allowing us to measure how consistency degrades across a certain number of transformation steps.

Forest. A forest \(\mathcal{F} = \{\mathcal{T}_1, ..., \mathcal{T}_M\}\) consists of multiple self-consistent trees, each rooted at a different initial node. These initial nodes are generated using distinct meta-prompts - e.g., “Write a 400-word paragraph in English about recent advances in computer science”. This design allows us to analyze LLM consistency across diverse tasks and domains by comparing patterns across trees in the forest.

Algorithm 1 Tree Generation Algorithm

Require: Root node \(r\), max depth \(D\), operation pairs \(((p_1, p'_1), \cdots, (p_k, p'_k))\)
Ensure: Tree \(\mathcal{T} = (\mathcal{V}, \mathcal{E})\)
1: \(\mathcal{V} \leftarrow \{ r \}\)
2: \(\mathcal{E} \leftarrow \emptyset\)
3: for \(d \leftarrow 1\) to \(D\) do
4: \quad \(U \leftarrow \{ v \in \mathcal{V} \mid D(v) = d-1 \}\)
5: \quad for each \(v \in U\) do
6: \quad \quad for \(i \leftarrow 1\) to \(k\) do
7: \quad \quad \quad \(c' \leftarrow f_{p'_i}(f_{p_i}(v.c))\)
8: \quad \quad \quad \(v' \leftarrow (c', v.\mathcal{I})\)
9: \quad \quad \quad \(\mathcal{V} \leftarrow \mathcal{V} \cup \{ v' \}\)
10: \quad \quad \quad \(\mathcal{E} \leftarrow \mathcal{E} \cup \{ (v, v') \}\)
11: \quad end for
12: end for
13: end for
14: return \((\mathcal{V}, \mathcal{E})\)

4 ConsistencyChecker: Tree-based LLM evaluation for Generalization Ability

Based on the definitions in Section §3, we now detail the construction of consistency scores in ConsistencyChecker. We begin by defining the similarity score for node pairs, then extend this to path-level, tree-level, and finally forest-level scores, with each step building upon the previous one.

Node-pair similarity score. To quantify the functional and semantic similarity between two connected nodes \(v_i = (c_i, \mathcal{I})\) and \(v_j = (c_j, \mathcal{I})\) that share the same test inputs \(\mathcal{I}\), we first obtain their execution outputs: \(o_i = \mathrm{exec}(c_i, \mathcal{I})\) and \(o_j = \mathrm{exec}(c_j, \mathcal{I})\). The similarity score \(\mathrm{sim}(v_i, v_j)\) is then computed between \(o_i\) and \(o_j\), using measures such as the cosine similarity of semantic embeddings or the BLEU score.

Path-level consistency score. To evaluate whether a node still retains core functionalities after iterative transformations, we measure the end-to-end consistency between the initial and final nodes in a path. For a path on the tree \(P = (v_1, \cdots, v_n)\), the consistency score defined for that is:

\[
C(P) := \mathrm{sim}(v_1, v_n)
\]

This metric evaluates whether the final output \(v_n\) remains functionally equivalent to the original \(v_1\) after a sequence of transformations. For example, for code generation, applying operations of "add logging functionality" followed by "remove log-
ging functionality” should leave the core sorting functionality unchanged, as reflected by this score.

Tree-level consistency score. To assess how well an LLM preserves functional consistency over fixed-length transformation sequences, we define a tree-level consistency score with the help of the path-level consistency score. For a given depth \( n \), we consider all valid \( n \)-step paths in the self-consistency tree as a set: \( \mathcal{P}_n(\mathcal{T}) = \{(v_1, \ldots, v_n) \mid (v_i, v_{i+1}) \in \mathcal{E} \text{ for } i = 1, \ldots, n\} \). The tree-level consistency score is the average of path-level consistency scores over all such paths:

\[
C_n(\mathcal{T}) := \frac{1}{|\mathcal{P}_n(\mathcal{T})|} \sum_{P \in \mathcal{P}_n(\mathcal{T})} C(P)
\]

where \( C(P) \) refers to the consistency score at the path level. This metric captures the model’s ability to maintain core functionality across all \( n \)-step transformation sequences, reflecting both cumulative error and path-specific divergence.

Forest-level consistency score. To assess robustness and allow the measurement of generalization ability at specific domains, we define a consistency score over a forest of self-consistency trees \( \mathcal{F} = \{\mathcal{T}_1, \ldots, \mathcal{T}_M\} \). The score is computed as:

\[
C_n(\mathcal{F}) := \frac{1}{M} \sum_{m=1}^M C_n(\mathcal{T}_m),
\]

where \( C_n(\mathcal{T}) \) denotes the consistency at depth \( n \) of tree \( \mathcal{T} \). Each tree is rooted at a node generated by the evaluator model. A forest with a high \( C_1(\mathcal{F}) \) score but a low \( C_3(\mathcal{F}) \) score indicates consistent performance for short transformation chains but degradation in longer transformation sequences.

ConsistencyChecker. In practice, setting a large value for \( n \) increases computational cost, while a small \( n \) leads to inaccurate estimates that overlook accumulated errors in multi-step transformations. To balance these trade-offs, we select \( n = 3 \) and use the forest-level consistency score \( C_3(\mathcal{F}) \) as the final metric in our ConsistencyChecker framework.

5 Evaluation Tasks and Meta Prompts

We evaluate ConsistencyChecker on two tasks: machine translation and AI-assisted programming. Each task is specified by a meta-prompt that generates a set of root nodes and paired operations for constructing self-consistency trees. In these cases, nodes correspond to either natural language sentences or executable code snippets. These tasks are representative, as they cover the full spectrum of node types—from static text with no external dependencies to code that serves as a solution to a LeetCode-style problem. The machine translation task focuses on assessing semantic consistency, while the AI-assisted programming task focuses on evaluating functional consistency. We set a time limit of 2 seconds per case in the test inputs to ensure that the verification process can end in a predictable time.

Machine translation. The root node of a translation task is a paragraph randomly sampled. Operations involve translating this paragraph into and from another language. We sample three languages from an evaluator, which are typically French, Spanish, and German. This setup ensures that the task captures the nuances of multilingual translation across diverse linguistic contexts.

AI-assisted programming. The root node for each programming task consists of a LeetCode-style coding problem and 20 associated test inputs. Operations are defined as transformations that prompt the model to implement the solution in a different, but functionally equivalent, way. This setup enables us to evaluate the model’s ability to generate code that remains functionally and semantically consistent across varying levels of abstraction and complexity.

Connections between two tasks.. Although the translation and coding tasks differ in complexity and domain, both are implemented within the same framework. For translation, the function takes no inputs and simply returns the target natural language text. For the coding task, the function accepts inputs as required by the problem. This uni-

<table>
  <tr>
    <th colspan="2">Table 1: Meta prompts for evaluation tasks. Instructions about YAML and formatting are abbreviated.</th>
  </tr>
  <tr>
    <th>Machine translation</th>
    <td>Write a 400 word, complicated English paragraph that might appear on a news website.<br>Please do this in a function way, e.g. provide a function called "main" that returns the content as a string.</td>
  </tr>
  <tr>
    <th>AI-assisted programming</th>
    <td>Write a LeetCode-Hard style problem. The problem must be super hard, even a graduate student in computer science will struggle to solve it.<br>Do not attempt to generate long, nested dicts. But it will require a very long and complicated solution.<br>The execution time should be very short. However, it does not need to be super long. It can be shorter, but it must be really hard.<br>Please do this in a functional way, e.g., provide a function called "main" that returns the intended answer.</td>
  </tr>
</table>
fied design makes our framework easily extensible to other tasks, such as step-by-step arithmetic, which can also be represented as Python code.

6 Experimental Settings

Evaluator models. Evaluator models determine the quality of dynamic benchmark generation. All experiments, except those in Table 4, use Qwen-2.5-72B as the evaluator model to generate dynamic benchmarks. For both tasks, we use the same evaluator-generated benchmarks across all models to ensure fair comparison and minimize the variability that fully on-the-fly benchmarks could introduce. Specifically, the evaluator model generates 10 root nodes for each task, which are then used consistently to evaluate all LLMs.

Evaluatee models. Using LLM-generated benchmarks, we evaluate a diverse set of language models across different families and sizes, including GPT-4o-mini (OpenAI et al., 2024), Qwen-2.5 (1.5B, 7B, 14B, 32B, 72B), and LLaMA-3.1 (8B, 70B) (Grattafiori et al., 2024), on both machine translation and AI-assisted programming tasks. Additionally, we assess the correlation between ConsistencyChecker and WMT 2024 metrics for machine translation benchmarks on specific language pairs. The models evaluated in this comparison include Claude-3.5-Sonnet (Anthropic, 2024), GPT-4, Gemini-1.5-Pro (Team et al., 2024), Mistral-Large (Jiang et al., 2023, 2024a), LLaMA-3-70B, and Phi-3-Medium (Abdin et al., 2024).

ConsistencyChecker settings. For the calculation of ConsistencyChecker, we fix \( n = 3 \), \( M = 10 \), and use embedding-based metrics such as NV-Embed-v2\(^1\) to compute similarity scores between node pairs. We then calculate the forest-level consistency score as our main result. All reported scores are presented as means with standard deviations to reflect variability across evaluation runs.

7 Experimental Results

We show experimental results of 8 evaluated models on both machine translation and AI-assisted programming tasks, and analyze the trends and insights for the two tasks separately.

ConsistencyChecker is reliable for evaluating machine translation tasks. As shown in Table 2, larger models consistently achieve higher consistency scores than smaller models within the same family for machine translation tasks. For example, Qwen-2.5-72B achieves a score of 97.2, which is 21.1% higher than Qwen-2.5-1.5B. Similarly, LLaMA-3.1-70B outperforms LLaMA-3.1-8B by 6.5%. Overall, models from the Qwen family generally achieve higher consistency scores than those from the LLaMA family, indicating that Qwen models produce more consistent results. Specially, we find GPT-4o-mini achieves the highest consistency score of 98.0, indicating the highest quality of it on machine translation tasks.

ConsistencyChecker is reliable for evaluating AI-assisted programming tasks. In AI-assisted programming tasks, larger models generally achieve higher functional consistency than smaller ones within the same model family, with Qwen-2.5-72B as an exception. Concretely, Qwen-2.5-32B scores 34.2% higher than Qwen-2.5-1.5B,

Table 2: Evaluation results with ConsistencyChecker. Each score represents LLMs’ ability to maintain semantic similarity in machine translation tasks and preserve functional consistency in programming tasks. Higher score indicates larger ratios for preserving the original information after transformations.

<table>
  <tr>
    <th>Model</th>
    <th>Translation↑</th>
    <th>Programming↑</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td><b>98.0±0.0</b></td>
    <td>76.5±2.7</td>
  </tr>
  <tr>
    <td>Qwen-2.5-1.5B</td>
    <td>80.3±0.5</td>
    <td>63.4±1.4</td>
  </tr>
  <tr>
    <td>Qwen-2.5-7B</td>
    <td>90.0±0.8</td>
    <td>71.7±0.4</td>
  </tr>
  <tr>
    <td>Qwen-2.5-14B</td>
    <td>94.7±0.1</td>
    <td>79.9±1.0</td>
  </tr>
  <tr>
    <td>Qwen-2.5-32B</td>
    <td>96.4±0.0</td>
    <td><b>85.1±1.1</b></td>
  </tr>
  <tr>
    <td>Qwen-2.5-72B</td>
    <td>97.2±0.0</td>
    <td>77.0±1.9</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-8B</td>
    <td>67.5±3.0</td>
    <td>60.4±1.0</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70B</td>
    <td>71.9±3.2</td>
    <td>83.5±1.0</td>
  </tr>
</table>

Table 3: Comparison between ConsistencyChecker and other metrics in WMT 2024 benchmarks. ConsistencyChecker has a high correlation with reliable translation metrics like CometKiwi and AutoRank without utilizing paired data in WMT 2024.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Czech–Ukrainian</th>
  </tr>
  <tr>
    <th>Ours ↑</th>
    <th>CometKiwi ↑</th>
    <th>AutoRank ↓</th>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>98.1</td>
    <td>68.3</td>
    <td>1.7</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>96.4</td>
    <td>67.7</td>
    <td>2.0</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>97.5</td>
    <td>66.8</td>
    <td>2.0</td>
  </tr>
  <tr>
    <td>Mistral-Large</td>
    <td>96.5</td>
    <td>66.6</td>
    <td>2.3</td>
  </tr>
  <tr>
    <td>LLaMA-3-70B</td>
    <td>95.9</td>
    <td>66.1</td>
    <td>2.6</td>
  </tr>
  <tr>
    <td>Phi-3-Medium</td>
    <td>44.9</td>
    <td>42.5</td>
    <td>9.1</td>
  </tr>
  <tr>
    <th colspan="4">English–Chinese</th>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>99.3</td>
    <td>70.3</td>
    <td>1.7</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>99.3</td>
    <td>69.8</td>
    <td>1.8</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>98.1</td>
    <td>69.3</td>
    <td>2.0</td>
  </tr>
  <tr>
    <td>Mistral-Large</td>
    <td>97.5</td>
    <td>66.5</td>
    <td>2.8</td>
  </tr>
  <tr>
    <td>LLaMA-3-70B</td>
    <td>97.9</td>
    <td>66.2</td>
    <td>2.8</td>
  </tr>
  <tr>
    <td>Phi-3-Medium</td>
    <td>96.6</td>
    <td>64.8</td>
    <td>3.1</td>
  </tr>
</table>

1https://huggingface.co/nvidia/NV-Embed-v2
while LLaMA-3.1-70B outperforms LLaMA-3.1-8B by 38.2%. These results underscore the importance of model scale for robust performance on complex transformation tasks.

ConsistencyChecker reaches high correlation with metrics in WMT 2024 without using its paired data. We compare ConsistencyChecker with the WMT 2024 (Kocmi et al., 2024) automatic rankings, focusing on two widely used metrics: CometKiwi (we refer to CometKiwi-DA-XL (Rei et al., 2023)) and AutoRank (the average of normalized CometKiwi and MetricX (Juraska et al., 2024) scores). Both metrics are computationally expensive, requiring large-scale pairwise human data or extensive model inference. In contrast, our approach is highly cost-efficient, as it does not require parallel data or data crawling—benchmarks are dynamically generated by LLMs. As shown in Table 3, we evaluate six representative LLMs—Claude-3.5-Sonnet, Gemini-1.5-Pro, GPT-4, LLaMA-3-70B, Mistral-Large, and Phi-3-Medium. For both CometKiwi and AutoRank, at least four out of six models are ranked similarly under ConsistencyChecker and WMT 2024. Pearson’s correlation coefficients, which are above 0.8 for both language pairs, provide strong evidence of the close agreement between ConsistencyChecker and established evaluation metrics. Detailed results are provided in Appendix §I.

8 Ablation Study

The path length \( n \) that we utilize for the calculation of ConsistencyChecker and the evaluator model that we rely on for dynamic benchmark generation are two key design elements that affect the performance of ConsistencyChecker. Therefore, we conduct ablation studies on them.

Ablation on path length. Figure 3 shows that consistency scores generally decline as path length increases, reflecting greater transformation complexity. For example, in the machine translation task, the LLaMA-3.1-8B model shows a 24.6% drop in consistency from path length 1 to 3, while the Qwen-2.5-72B model shows a smaller 1.9% decrease. This trend holds across most models, regardless of family or size, though the rate of decline varies. In the AI-assisted programming task, the Qwen-2.5-7B model exhibits a 16.0% reduction, again confirming that errors tend to accumulate over longer transformation paths. Smaller drops in the consistency score indicate stronger self-consistency, as the model introduces fewer errors per operation. An exception occurs with the LLaMA-3.1-70B model in the programming task, where the consistency score is higher at path length 3 than at 2. This may be due to the model detecting and correcting syntax errors or bugs that appeared at intermediate steps.

Ablation on evaluator model. As shown in Table 4, different evaluator models (Qwen-2.5-7B and Qwen-2.5-72B) produce similar rankings across evaluated LLMs. For instance, in the translation task at path length 3, Qwen-2.5-7B assigns a score of 76.8, while Qwen-2.5-72B gives a slightly higher score of 78.2. In the programming task, for example, Qwen-2.5-7B evaluated by itself scores 72.2, while evaluation by Qwen-2.5-72B yields 71.7 (0.7% difference). We find that such an alignment becomes more pronounced at longer path lengths, where consistency patterns are clearer. Although absolute consistency scores may vary between evaluators, relative trends remain stable.

![Ablation study on path lengths. Two line graphs showing consistency scores (%) for machine translation and AI-assisted programming tasks as a function of path length.](page_120_120_1208_482.png)

Figure 3: Ablation study on path lengths. For most models and for both translation and programming tasks, the consistency score becomes smaller when the path becomes longer.
9 Discussion

In our main experiments, we use embedding-based models (NV-embed-v2) to compute node-pair similarity \( \operatorname{sim}(\cdot,\cdot) \) and calculate forest-level consistency score \( C_3(\mathcal{F}) \) based on that. However, BLEU (Papineni et al., 2002) remains a classical and widely adopted measure of translation quality. Therefore, we discuss their differences here.

BLEU is highly correlated but slightly worse than embedding-based models for machine translations. When evaluating translation across the five language pairs defined in WMT 2024, the BLEU-based consistency score is well correlated with WMT benchmark metrics, achieving Pearson correlations above 0.7 with three reference metrics in four out of five pairs. Furthermore, the BLEU-based \( C_3(\mathcal{F}) \) consistency score is also strongly correlated with the embedding-based \( C_3(\mathcal{F}) \), with average Pearson and Spearman correlations of 0.89 and 0.87, respectively, across all five pairs. However, embedding-based consistency scores are slightly more correlated with WMT benchmark metrics than BLEU-based scores when averaged across the five language pairs. Full evaluation results are provided in Appendix §I.

BLEU is surprisingly useful for evaluating programming tasks. For the AI-assisted programming task, the correlations were notably strong: Pearson coefficients reached 0.98 (\( C_1(\mathcal{F}) \)), 0.97 (\( C_2(\mathcal{F}) \)), and 0.99 (\( C_3(\mathcal{F}) \)). This high correlation is unexpected, as BLEU is not designed to measure output functionality but is designed to measure n-gram overlaps. However, these results indicate that in this context, BLEU captures meaningful distinctions between outputs, likely because nodes are either fully functional (producing identical outputs for the same inputs), entirely different, or unexecutable due to code errors. As a result, even when outputs are not natural language, BLEU can effectively differentiate between them.

10 Case Study

In this section, we present two case studies involving different trees with the same root node but with different models for tree generation.

Leaf node from different models. The root node contains a function that returns an analysis of AI integration in healthcare systems. We construct self-consistency trees by applying a series of translation operations to the return value of the function: translating it into French, German, and Spanish, then back to English. These trees are evaluated with a path depth of 3, and we compare the outputs of LLaMA-3.1-8B and LLaMA-3.1-70B. The corresponding code for these leaf nodes is shown in Appendix §J. Based on the generated content of the leaf nodes, LLaMA-3.1-70B introduces fewer changes (Levenshtein distance: 66) compared to LLaMA-3.1-8B (Levenshtein distance: 215), indicating stronger consistency on larger models. This aligns with their consistency scores provided by ConsistencyChecker, highlighting the robustness

<table>
  <tr>
    <th rowspan="2">Evaluator</th>
    <th colspan="2">Qwen-2.5-7B</th>
    <th colspan="2">Qwen-2.5-72B</th>
  </tr>
  <tr>
    <th colspan="2">Machine translation</th>
    <th colspan="2">AI-assisted programming</th>
  </tr>
  <tr>
    <td>Qwen-2.5-7B</td>
    <td>76.8±5.1</td>
    <td>90.0±0.8</td>
    <td>72.2±1.9</td>
    <td>71.7±0.4</td>
  </tr>
  <tr>
    <td>Qwen-2.5-72B</td>
    <td><b>78.2±7.3</b></td>
    <td><b>97.2±0.0</b></td>
    <td>79.4±2.3</td>
    <td>77.0±1.9</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-8B</td>
    <td>61.7±4.8</td>
    <td>67.5±3.0</td>
    <td>61.0±1.1</td>
    <td>60.4±1.1</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70B</td>
    <td>30.8±9.2</td>
    <td>71.9±3.2</td>
    <td><b>81.5±2.1</b></td>
    <td><b>83.5±1.0</b></td>
  </tr>
</table>

Table 4: Ablation study on evaluator models. (Top) machine translation consistency scores (%) at complexity level \( C_3(\mathcal{F}) \) for model scales (7B–72B). (Bottom) AI-assisted programming consistency scores (%) at complexity level \( C_3(\mathcal{F}) \) for code transformations.

<table>
  <tr>
    <th rowspan="2">Evaluatee</th>
    <th colspan="3">C_1(\mathcal{F})</th>
    <th colspan="3">C_2(\mathcal{F})</th>
    <th colspan="3">C_3(\mathcal{F})</th>
  </tr>
  <tr>
    <th colspan="3">Machine translation</th>
    <th colspan="3">AI-assisted programming</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>86.0±0.0</td>
    <td>78.6±0.1</td>
    <td>68.0±0.2</td>
    <td>78.4±0.1</td>
    <td>68.2±0.2</td>
    <td>57.3±0.3</td>
  </tr>
  <tr>
    <td>Qwen-2.5-1.5B</td>
    <td>53.4±0.3</td>
    <td>37.6±0.3</td>
    <td>25.8±0.3</td>
    <td>50.6±0.7</td>
    <td>31.4±2.4</td>
    <td>19.7±0.7</td>
  </tr>
  <tr>
    <td>Qwen-2.5-7B</td>
    <td>70.5±0.3</td>
    <td>56.0±0.7</td>
    <td>41.9±0.7</td>
    <td>65.1±0.2</td>
    <td>53.8±0.2</td>
    <td>41.9±0.7</td>
  </tr>
  <tr>
    <td>Qwen-2.5-14B</td>
    <td>76.3±0.1</td>
    <td>65.1±0.2</td>
    <td>53.8±0.2</td>
    <td>71.2±0.1</td>
    <td>57.4±0.2</td>
    <td>45.4±0.2</td>
  </tr>
  <tr>
    <td>Qwen-2.5-32B</td>
    <td>78.4±0.1</td>
    <td>68.2±0.2</td>
    <td>57.3±0.3</td>
    <td>71.2±0.1</td>
    <td>57.4±0.2</td>
    <td>45.4±0.2</td>
  </tr>
  <tr>
    <td>Qwen-2.5-72B</td>
    <td>81.7±0.1</td>
    <td>71.2±0.1</td>
    <td>57.4±0.2</td>
    <td>81.7±0.1</td>
    <td>71.2±0.1</td>
    <td>57.4±0.2</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-8B</td>
    <td>59.5±1.2</td>
    <td>47.6±1.1</td>
    <td>36.6±0.9</td>
    <td>63.1±2.0</td>
    <td>57.3±1.8</td>
    <td>46.9±2.3</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70B</td>
    <td>63.1±2.0</td>
    <td>57.3±1.8</td>
    <td>46.9±2.3</td>
    <td>63.1±2.0</td>
    <td>57.3±1.8</td>
    <td>46.9±2.3</td>
  </tr>
</table>

Table 5: Consistency score results when using BLEU to build ConsistencyChecker. We select Qwen-2.5-72B as the evaluator and select different models as evaluatees. We utilize BLEU to build consistency scores.
Figure 4: Comparison of two self-consistency trees with different performances. This diagram illustrates how Qwen-2.5-7B (left) and Qwen-2.5-72B (right) performed under the same root node and inverse operation pairs, in machine translation tasks. The diagram shows a simplified case where there are only 2 pairs of inverse operations. We use a lighter shade of red to indicate a smaller embedding similarity with the root node.

of larger models in maintaining meaning through multi-step transformations.

Features of under-performing self-consistency trees. We compare self-consistency trees generated by Qwen-2.5-7B and Qwen-2.5-72B from the same root node to analyze patterns of under-performance from the tree structure. The root consists of an around-290-token English paragraph discussing the impact of European Central Bank policies on global markets. Each model is prompted to translate the text to and from French, German, and Spanish. In the case of Qwen-2.5-7B, the French branch yields a significantly degraded output after round-trip translation, with only a 0.33 embedding similarity score to the root. The resulting English is just around 40 tokens and retains only partial information (see Figure 4). Subsequent nodes in this branch show equally low or worse similarity to the root node. By contrast, the German and Spanish branches maintain embedding similarities above 0.86. For Qwen-2.5-72B, all branches - including all child nodes - consistently achieve embedding similarities above 0.90. This illustrates a failure mode of less self-consistent models like Qwen-2.5-7B, and demonstrates how ConsistencyChecker effectively identifies and quantifies such inconsistencies with the help of the tree structure.

11 Conclusion

In this work, we propose ConsistencyChecker, an evaluation framework for assessing the generalization ability of LLMs effectively and efficiently. For our methods, we first introduce the concept of self-consistency trees, which model multi-step transformations as a tree structure. Based on this, we define a consistency score using a self-consistency forest to enable robust evaluation across multiple tasks. Experimental results show that our consistency metric reliably estimates model performance for both translation and programming tasks, and closely matches WMT benchmark metrics such as AutoRank and CometKiwi for machine translation.

Limitations

While this study offers valuable insights into LLM consistency via the ConsistencyChecker framework, it has several limitations. The evaluation is limited to machine translation and code generation, excluding tasks like arithmetic. It primarily covers only 8 models in mainstream model families. Additionally, the study relies solely on automated metrics such as embedding similarity and BLEU, which may not fully reflect human judgment, especially for tasks requiring creativity or nuanced interpretation. Expanding task coverage, model diversity, and incorporating human evaluation in future work would strengthen the generalizability and robustness of the framework.

References

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkrati Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Jun-heng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iler, Sam Ade Jacobs, Mojan Javanheri, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambuddha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuhang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu,
Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. 2024. Ph-3 technical report: A highly capable language model locally on your phone. Preprint, arXiv:2404.14219.

Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. In Anthropic Model Card.

Loïc Barrault, Ondfjej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1–61, Florence, Italy. Association for Computational Linguistics.

Clark Barrett and Cesare Tinelli. 2018. Satisfiability modulo theories. Handbook of model checking, pages 305–343.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR, abs/2005.14165.

Mark Chen, Jerry Twork, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaeck Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2024. Palm: scaling language modeling with pathways. J. Mach. Learn. Res., 24(1).

Edmund M Clarke, Orna Grumberg, and Doron Peled. 1999. Model checking. MIT press.

Wendi Cui, Zhuohang Li, Damien Lopez, Kamalika Das, Bradley A. Malin, Sricharan Kumar, and Jiaxin Zhang. 2024. Divide-conquer-reasoning for consistency evaluation and automatic improvement of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 334–361, Miami, Florida, US. Association for Computational Linguistics.

Aaron Grattafiore, Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Scheflen, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Dan Hendrycks, Steven Basart, Saurav Kadavath, Manas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring coding challenge competence with APPS. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024a. Mixtral of experts. Preprint, arXiv:2401.04088.

Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024b. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515.

Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. Metricx-24: The google submission to the wmt 2024 metrics shared task. arXiv preprint arXiv:2410.03983.

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda,
Roman Grundkiewicz, et al. 2024. Preliminary wmt24 ranking of general mt systems and llms. arXiv preprint arXiv:2407.19884.

Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. 2024. Mt-eval: A multi-turn capabilities evaluation benchmark for large language models. Preprint, arXiv:2401.16745.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Liding Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie LIU. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).

Marcus J Min, Yangruibo Ding, Luca Buratti, Saurabh Pujar, Gail Kaiser, Suman Jana, and Baishakhi Ray. 2023. Beyond accuracy: Evaluating self-consistency of code large language models with identitychain. arXiv preprint arXiv:2310.14053.

Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meyksam Chenaghlou, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. Preprint, arXiv:2402.06196.

Tobias Nipkow, Markus Wenzel, and Lawrence C. Paulson. 2002. Isabelle/HOL: a proof assistant for higher-order logic. Springer-Verlag, Berlin, Heidelberg.

OpenAI, ; Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoochian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonio Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendon Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevido, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Haining Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O’Connell, Ian O’Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikan Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varava, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quionero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justin Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kevin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Konradiuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Afak, Maddie Simens, Madeleine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Na-
talie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizier, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethershim, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi, Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunningham, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyan Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. 2024. Gpt-4o system card. Preprint, arXiv:2410.21276.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, and André F. T. Martins. 2023. Scaling up cometkiwi: Unbabel-ist 2023 submission for the quality estimation shared task. Preprint, arXiv:2309.11925.

Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223–231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas.

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Sorooosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcobber, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti, Colin Gaffney, Samira Daruki, Olcan Sercinoglu, Zach Gleicher, Juliette Love, Paul Voigt-laender, Rohan Jain, Gabriela Surita, Kareem Mohamed, Rory Blevins, Junwhan Ahn, Tao Zhu, Kornraphop Kawintiranon, Orhan Firat, Yiming Gu, Yujing Zhang, Matthew Rahtz, Maanal Faruqui, Natalie Clay, Justin Gilmer, JD Co-Reyes, Ivo Penchev, Rui Zhu, Nobuyuki Morioka, Kevin Hu, Krishna Haridasan, Victor Campos, Mahdis Mahdieh, Mandy Guo, Samer Hassan, Kevin Kilgour, Arpi Vezer, Heng-Tze Cheng, Raoul de Liedekerke, Siddharth Goyal, Paul Barham, DJ Strouse, Seb Noury, Jonas Adler, Mukund Sundararajan, Sharad Vikram, Dmitry Lepikhin, Michela Paganini, Xavier Garcia, Fan Yang, Dasha Valter, Maja Trebacz, Kiran Vodrahalli, Chulayuth Asawaroengchai, Roman Ring, Norbert Kalb, Livio Baldini Soares, Siddhartha Brahma, David Steiner, Tianhe Yu, Fabian Mentzer, Antoine He, Lucas Gonzalez, Bibo Xu, Raphael Lopez Kaufman, Laurent El Shafey, Junhyuk Oh, Tom Henning, George van den Driessche, Seth Odoom, Mario Lucic, Becca Roelofs, Sid Lall, Amit Marathe, Betty Chan, Santiago Ontanon, Luheng He, Denis Teplyashin, Jonathan Lai, Phil Crone, Bogdan Damoc, Lewis Ho, Sebastian Riedel, Karel Lenc, Chih-Kuan Yeh, Aakashka Chowdhery, Yang Xu, Mehran Kazemi, Ehsan Amid, Anastasia Petrushkina, Kevin Swersky, Ali Khodaei, Gowoon Chen, Chris Larkin, Mario Pinto, Geng Yan, Adria Puigdomenech Badia, Piyush Patil, Steven Hansen, Dave Orr, Sebastien M. R. Arnold, Jordan Grimstad, Andrew Dai, Sholto Douglas, Rishika Sinha, Vikas Yadav, Xi Chen, Elena Grivovskaya, Jacob Austin, Jeffrey Zhao, Kaushal Patel, Paul Komarek, Sophia Austin, Sebastian Borgeaud, Linda Friso, Abhimanyu Goyal, Ben Caine, Kris Cao, Da-Woon Chung, Matthew Lamm, Gabe Barth-Maron, Thais Kagohara, Kate Olszewska, Mia Chen, Kaushik Shivakumar, Rishabh Agarwal, Harshal Godhia, Ravi Rajwar, Javier Snaider, Xerxes Dotiwalla, Yuan Liu, Aditya Barua, Victor Ungureanu, Yuan Zhang, Bat-Orgil Batsaikhan, Mateo Wirth, James Qin, Ivo Danihelka, Tulsee Doshi, Martin Chadwick, Jilin Chen, Sanil Jain, Quoc Le, Arjun Kar, Madhu Gurumurthy, Cheng Li, Ruoxin Sang, Fangyu Liu, Lampros Lamprou, Rich Munoz, Nathan Lintz, Harsh Mehta, Heidi Howard, Malcolm Reynolds, Lora Aroyo, Quan Wang, Lorenzo Blanco, Albin Cassirer, Jordan Griffith, Dipanjan Das, Stephan Lee, Jakub Sygnowski, Zach Fisher, James Besley, Richard Powell, Zafarali Ahmed, Dominik Paulus, David Reitter, Zalan Borsos, Rishabh Joshi, Aedan Pope, Steven Hand, Vittorio Selo, Vishan Jain, Nikhil Sethi, Megha Goel, Takaki Makino, Rhys May, Zhen Yang, Johan Schalkwyk, Christina Butterfield, Anja Hauth, Alex Goldin, Will Hawkins, Evan Senter, Sergey Brin, Oliver Woodman, Marvin Ritter, Eric Noland, Minh Giang, Vijay Bolina, Lisa Lee, Tim Blyth, Ian Mackinnon, Machel Reid,
Obaid Sarvana, David Silver, Alexander Chen, Lily Wang, Loren Maggiore, Oscar Chang, Nithya Ataluri, Gregory Thornton, Chung-Cheng Chiu, Oskar Bunyan, Nir Levine, Timothy Chung, Evgeni Eltyshov, Xiance Si, Timothy Lillicrap, Demetra Brady, Vaibhav Aggarwal, Boxi Wu, Yuanzhong Xu, Ross McIlroy, Kartikeya Badola, Paramjit Sandhu, Erica Moreira, Wojciech Stokowiec, Ross Hemsley, Dong Li, Alex Tudor, Pranav Shyam, Elahe Rahimtoroghli, Salem Haykal, Pablo Sprechmann, Xiang Zhou, Diana Mincu, Yujia Li, Ravi Addanki, Kalpesh Krishna, Xiao Wu, Alexandre Frechette, Matan Eyal, Allan Dafoe, Dave Lacey, Jay Whang, Thi Avrahami, Ye Zhang, Emanuel Taropá, Hanzhao Lin, Daniel Toyama, Eliza Rutherford, Motoki Sano, HyunJeong Choe, Alex Tomala, Chalence Safranek-Shrader, Nora Kassner, Mantas Pajarskas, Matt Harvey, Sean Sechrist, Meire Fortunato, Christina Lyu, Gamaleldin Elsayed, Chenkai Kuang, James Lottes, Eric Chu, Chao Jia, Chih-Wei Chen, Peter Humphreys, Kate Baumli, Connie Tao, Rajkumar Samuel, Cicero Nogueira dos Santos, Anders Andreassen, Nemanja Rakicicvic, Dominik Grewe, Aviral Kumar, Stephanie Winkler, Jonathan Caton, Andrew Brock, Sid Dalmia, Hannah Sheahan, Iain Barr, Yingjie Miao, Paul Natesv, Jacob Devlin, Ferrell Bebbeahani, Flavian Prost, Yanhua Sun, Artiom Myaskovsky, Thanumalayan Sankaranarayana Pillai, Dan Hurt, Angeliki Lazaridou, Xi Xiong, Ce Zheng, Fabio Pardo, Xiaowei Li, Dan Horgan, Joe Stanton, Moran Ambar, Fei Xia, Alejandro Lince, Mingqiu Wang, Basil Mustafa, Albert Webson, Hyo Lee, Rohan Anil, Martin Wicke, Timothy Dozat, Abhishek Sinha, Enrique Piqueras, Elahe Dabir, Shyam Upadhyay, Anudhya Boral, Lisa Anne Hendricks, Corey Fry, Josip Djolonga, Yi Su, Jake Walker, Jane Labanowski, Ronny Huang, Vedant Misra, Jeremy Chen, RJ Skerry-Ryan, Avi Singh, Shruti Rijhwani, Dian Yu, Alex Castro-Ros, Beer Changpinyo, Romina Datta, Sumit Bagri, Arnar Mar Hrafnkelsson, Marcello Maggioni, Daniel Zheng, Yury Sulsky, Shaobo Hou, Tom Le Paine, Antoine Yang, Jason Riesa, Dominika Rogozinska, Dror Marcus, Dalia El Badawy, Qiao Zhang, Luyu Wang, Helen Miller, Jeremy Greer, Lars Lowe Sjos, Azade Nova, Heiga Zen, Rahma Chaabouni, Mihaela Rosca, Jiepu Jiang, Charlie Chen, Ruibo Liu, Tara Sainath, Maxim Krikun, Alex Polozov, Jean-Baptiste Lespiau, Josh Newlan, Zeynep Cankara, Soo Kwak, Yunhan Xu, Phil Chen, Andy Coenen, Clemens Meyer, Katerina Tsihlas, Ada Ma, Juraj Gottweis, Jinwei Xing, Chenjie Gu, Jin Miao, Christian Frank, Zeynep Cankara, Sanjay Ganapathy, Ishita Dasgupta, Steph Hughes-Fitt, Heng Chen, David Reid, Keran Rong, Hongmin Fan, Joost van Amersfoort, Vincent Zhuang, Aaron Cohen, Shixiang Shane Gu, Anhad Mohananean, Anastasija Ilic, Taylor Tobin, John Wieting, Anna Bortsova, Phoebe Thacker, Emma Wang, Emily Caveness, Justin Chiu, Eren Sezener, Alex Kaskasoli, Steven Baker, Katie Millican, Mohamed Elhawaty, Kostas Aisopos, Carl Lebsack, Nathan Byrd, Hanjun Dai, Wenhao Jia, Matthew Wiethoff, Elnaz Davoodi, Albert Weston, Lakshman Yagati, Arun Ahuja, Isabel Gao, Golan Pundak, Susan Zhang, Michael Azzam, Khe Chai Sim, Sergi Caelles, James Keeling, Abhanshu Sharma, Andy Swing, YaGuang Li, Chenxi Liu, Carrie Grimes Bostock, Yamini Bansal, Zachary Nado, Ankeesh Anand, Josh Lipschutz, Abhijit Karmarkar, Lev Proleev, Abe Ittycheriah, Soheil Hassas Yeganeh, George Polovets, Aleksandra Faust, Jiao Sun, Alban Rustemi, Pen Li, Rakesh Shivanna, Jeremiah Liu, Chris Welty, Federico Lebron, Anirudh Baddepudi, Sebastian Krause, Emilio Parisotto, Radu Soricut, Zheng Xu, Dawn Bloxwich, Melvin Johnson, Behnam Neyshabur, Justin Mao-Jones, Ren-shen Wang, Vinay Ramasesh, Zaheer Abbas, Arthur Guez, Constant Segal, Duc Dung Nguyen, James Svensson, Le Hou, Sarah York, Kieran Milan, Sophie Bridgers, Wiktor Gworek, Marco Tagliasacchi, James Lee-Thorp, Michael Chang, Alexey Guseynov, Ale Jakse Hartman, Michael Kwong, Ruizhe Zhao, Sheleem Kashem, Elizabeth Cole, Antoine Miech, Richard Tanburn, Mary Phuong, Filip Pavetic, Sebastien Cevye, Ramona Comanescu, Richard Ives, Sherry Yang, Cosmo Du, Bo Li, Zizhao Zhang, Mariko Inuma, Clara Huiyi Hu, Aurko Roy, Shaan Bijwadia, Zhenkai Zhu, Danilo Martins, Rachel Saputro, Anita Gergely, Steven Zheng, Dawei Jia, Ioannis Antonoglou, Adam Sadovsky, Shane Gu, Yingying Bi, Alek Andreev, Sina Samangooei, Mina Khan, Tomas Kocisky, Angelos Filos, Chintu Kumar, Colton Bishop, Adams Yu, Sarah Hodgkinson, Sid Mittal, Premal Shah, Alexandre Moufarek, Yong Cheng, Adam Bloniarz, Jaehoon Lee, Pedram Pejman, Paul Michel, Stephen Spencer, Vladimir Feinberg, Xuehan Xiong, Nikolay Savinov, Charlotte Smith, Siamak Shakeri, Dustin Tran, Mary Chesus, Bernd Bohnet, George Tucker, Tamara von Glehn, Carrie Muir, Yiran Mao, Hideto Kazawa, Ambrose Stone, Kedar Soparkar, Disha Shrivastava, James Cobon-Kerr, Michael Sharman, Jay Pavagadhi, Carlos Araya, Karolis Misinas, Nimesh Ghelani, Michael Laskin, David Barker, Qiujia Li, Anton Briukhov, Neil Housby, Mia Glasee, Balaji Lakshminarayanan, Nathan Schucher, Yunhao Tang, Eli Collins, Hyeontaeck Lim, Fangxiaoyu Feng, Adria Recasens, Guangda Lai, Alberto Magni, Nicola De Cao, Aditya Siddhant, Zoe Ashwood, Jordi Orbay, Mostafa Dehghani, Jenny Brennan, Yifan He, Kelvin Xu, Yang Gao, Carl Saroufim, James Molloy, Xinyi Wu, Seb Arnold, Solomon Chang, Julian Schrittwieser, Elena Buchatskaya, Soroush Radpour, Martin Polacek, Skye Giordano, Ankur Bapna, Simon Tokumine, Vincent Hellendoorn, Thibault Sottiaux, Sarah Cogan, Aliaksai Severyn, Mohammad Saleh, Shantanu Thakoor, Laurent Shefey, Siyuan Qiao, Meenu Gaba, Shuo yin Chang, Craig Swanson, Biao Zhang, Benjamin Lee, Paul Kishan Rubenstein, Gan Song, Tom Kwiatkowski, Anna Koop, Ajay Kanann, David Kao, Parker Schuh, Axel Stjerngren, Golnaz Ghiasi, Gena Gibson, Luke Vilnis, Ye Yuan, Felipe Tiengo Ferreira, Aishwarya Kamath, Ted Klimenko, Ken Franko, Kefan Xiao, Indro Bhattacharya, Miteyan Patel, Rui Wang, Alex Morris, Robin Strudel, Vivek Sharma, Peter Choy, Sayed Hadi Hashemi, Jessica Landon, Mara Finkelstein, Priya Jhakra, Justin Frye, Megan Barnes, Matthew Mauger, Dennis Daun, Khuslen Baatarsukh, Matthew Tung,
Wael Farhan, Henryk Michalewski, Fabio Viola, Felix de Chaumont Quitry, Charline Le Lan, Tom Hudson, Qingze Wang, Felix Fischer, Ivy Zheng, Elspeth White, Anca Dragan, Jean baptiste Alayrac, Eric Ni, Alexander Pritzel, Adam Iwanciuk, Michael Isard, Anna Bulanova, Lukas Zilka, Ethan Dyer, Devendra Sachan, Srivatsan Srinivasan, Hannah Muckenhirn, Honglong Cai, Amol Mandhane, Mukarram Tariq, Jack W. Rae, Gary Wang, Kareem Ayoub, Nicholas FitzGerald, Yao Zhao, Woohyun Han, Chris Alberti, Dan Garrette, Kashyap Krishnakumar, Mai Gimenez, Anselm Levskaya, Daniel Sohn, Josip Matak, Inaki Iturrate, Michael B. Chang, Jackie Xiang, Yuan Cao, Nishant Ranka, Geoff Brown, Adrian Hutter, Vahab Mirrokni, Nanxin Chen, Kaisheng Yao, Zoltan Eyged, Francois Galiee, Tyler Liechty, Praveen Kallakuri, Evan Palmer, Sanjay Ghemawat, Jasmine Liu, David Tao, Chloe Thornton, Tim Green, Mimi Jasarevic, Sharon Lin, Victor Cotruba, Yi-Xuan Tan, Noah Fiedel, Hongkun Yu, Ed Chi, Alexander Neitz, Jens Heitkemper, Anu Sinha, Denny Zhou, Yi Sun, Charbel Kaed, Brice Hulse, Swaroop Mishra, Maria Georgaki, Sneha Kudugunta, Clement Farabet, Izhak Shafran, Daniel Vlasic, Anton Tsitsulin, Rajagopal Ananthanarayanan, Alen Carin, Guolong Su, Pei Sun, Shashank V. Gabriel Carvajal, Jose Broder, Julia Comsa, Alena Repina, William Wong, Warren Weilun Chen, Peter Hawkins, Egor Filonov, Lucia Loher, Christoph Hirschmann, Weiyi Wang, Jingchen Ye, Andrea Burns, Hardie Cate, Diana Gage Wright, Federico Piccinini, Lei Zhang, Chu-Cheng Lin, Ioneg Gog, Yana Kulizhinskaya, Ashwin Sreevatsa, Shuang Song, Luis C. Cobo, Anand Iyer, Chetan Tekur, Guillermo Garrido, Zhuyun Xiao, Rupert Kemp, Huaxiu Steven Zheng, Hui Li, Ananth Agarwal, Christel Ngani, Kati Goshvadi, Rebeca Santamaría-Fernandez, Wojciech Fica, Xinyun Chen, Chris Gorgolewski, Sean Sun, Roopal Garg, Xinyu Ye, S. M. Ali Esfandi, Nan Hua, Jon Simon, Pratik Joshi, Yelin Kim, Ian Tenney, Sahitya Potluri, Lam Nguyen Thiet, Quan Yuan, Florian Luisier, Alexandra Chronopoulou, Salvatore Scellato, Praveen Srinivasan, Minmin Chen, Vinod Koverkathu, Valentin Dalibard, Yaming Xu, Brennan Saeta, Keith Anderson, Thibault Sellam, Nick Fernando, Fantine Huot, Junyueh Jung, Mani Varadarajan, Michael Quinn, Amit Raul, Maigo Le, Ruslan Habalov, Jon Clark, Komal Jalan, Kalesha Bullard, Achintya Singhal, Thang Luong, Boyu Wang, Sujeevan Rajayogam, Julian Eisenschlos, Johnson Jia, Daniel Finchelstein, Alex Yakubovich, Daniel Balle, Michael Fink, Sameer Agarwal, Jing Li, Dj Dvijotham, Shalini Pal, Kai Kang, Jaclyn Konzelmann, Jennifer Beattie, Olivier Dousse, Diane Wu, Remi Crocker, Chen Elkind, Siddhartha Reddy Jonnalagadda, Jong Lee, Dan Holtmann-Rice, Krystal Kallarakkal, Rosanne Liu, Denis Vnuok, Neera Vats, Luca Invernizzi, Mohsen Jafari, Huanjie Zhou, Lilly Taylor, Jennifer Prendki, Marcus Wu, Tom Eccles, Tianqi Liu, Kavya Kopparapu, Francoise Beaufays, Christof Angermueller, Andreea Marzoaca, Shourya Sarcar, Hilal Dib, Jeff Stanway, Frank Perbet, Nejc Trdin, Rachel Sterneck, Andrey Khorlin, Dinghua Li, Xihui Wu, Sonam Goenka, David Madras, Sasha Goldstein, Willi Gierke, Tong Zhou, Yaxin Liu, Yannie Liang, Anais White, Yunjie Li, Shreya Singh, Sanaz Bahargam, Mark Epstein, Sujoy Basu, Li Lao, Adnan Ozturel, Carl Crous, Alex Zhai, Han Lu, Zora Tung, Neeraj Gaur, Alanna Walton, Lucas Dixon, Ming Zhang, Amir Globerson, Grant Uy, Andrew Bolt, Olivia Wiles, Milad Nasr, Ilia Shumailov, Marco Selvi, Francesco Piccinno, Ricardo Aguilar, Sara McCarthy, Misha Khaman, Mrinal Shukla, Vlado Galic, John Carpenter, Kevin Villela, Haibin Zhang, Harry Richardson, James Martens, Matko Bosnjak, Shreyas Rammohan Belle, Jeff Seibert, Mahmoud Alnahlawi, Brian Williams, Sankalp Singh, Annie Louis, Wen Ding, Dan Popovic, Lenin Simicich, Laura Knight, Pulkit Mehta, Nishesh Gupta, Chongyang Shi, Saaber Fatehi, Jovana Mitrovic, Alex Grills, Joseph Pagadora, Tsenduren Munkhdalai, Dessie Petrova, Danielle Eisenbud, Zhishuai Zhang, Damian Yates, Bhavishya Mittal, Nilesh Tripuraneni, Yanis Assael, Thomas Brovelli, Prateek Jain, Mihaeljlo Velimirovic, Canfer Akbulut, Jiaqi Mu, Wolfgang Macherey, Ravin Kumar, Jun Xu, Haroon Qureshi, Gheorghe Comanici, Jeremy Wiesner, Zhitao Gong, Anton Ruddock, Matthias Bauer, Nick Felt, Anirudh GP, Anurag Arnab, Dustin Zelle, Jonas Rothfuss, Bill Rosgen, Ashish Shenoy, Bryan Seybold, Xinjian Li, Jayaram Mudigonda, Goker Erdogan, Jiawei Xia, Jiri Simsa, Andrea Michi, Yi Yao, Christopher Yew, Steven Kan, Isaac Caswell, Carey Radabaugh, Andre Elisseff, Pedro Valenzuela, Kay McKinney, Kim Paterson, Albert Cui, Eri Latorre-Chimoto, Solomon Kim, William Zeng, Ken Durden, Priya Ponnappali, Tiberiu Sosea, Christopher A. Choquette-Choo, James Manyika, Brona Robeneck, Harsha Vashisth, Sebastien Pereira, Hoi Lam, Marko Velic, Denese Owusu-Afriyie, Katherine Lee, Tolga Bolukbasi, Alicia Parrish, Shawn Lu, Jane Park, Balaji Venkatraman, Alice Talbert, Lambert Rosique, Yuchung Cheng, Andrei Sozanschi, Adam Paszke, Praveen Kumar, Jessica Austin, Lu Li, Khalid Salama, Bartek Perz, Wooeyol Kim, Nandita Dukkipati, Anthony Baryshnikov, Christos Kaplantis, XiangHai Sheng, Yuri Cherovnyi, Caglar Unlu, Diego de Las Casas, Harry Askham, Kathryn Tunyasuvunakool, Felix Gimeno, Siim Poder, Chester Kwak, Matt Miecnikowski, Vahab Mirrokni, Alek Dimitriev, Aaron Parisi, Dangyi Liu, Tomy Tsai, Toby Shevlane, Christina Kouridi, Drew Garnon, Adrian Goedekeemeyer, Adam R. Brown, Anita Vijayakumar, Ali Elqursh, Sadegh Jazayeri, Jin Huang, Sara Mc Carthy, Jay Hoover, Lucy Kim, Sandeep Kumar, Wei Chen, Courtney Biles, Garrett Bingham, Evan Rosen, Lisa Wang, Qijun Tan, David Engel, Francesco Pongetti, Dario de Cesare, Dongseong Hwang, Lily Yu, Jennifer Pullman, Srini Narayanan, Kyle Levin, Siddharth Gopal, Megan Li, Asaf Aharoni, Trieu Trinh, Jessica Lo, Norman Casagrande, Roopali Vij, Loic Mattey, Bramandia Ramadhana, Austin Mathews, CJ Carey, Matthew Johnson, Kremena Goranova, Rohin Shah, Shereen Ashraf, Kingshuk Dasgupta, Rasmus Larsen, Yicheng Wang, Manish Reddy Vuyyuru, Chong Jiang, Joana Ijazi, Kazuki Osawa, Celine Smith, Ramya Sree Boppana, Tay-
Ian Bilal, Yuma Koizumi, Ying Xu, Yasemin Altun, Nir Shabat, Ben Bariach, Alex Korchemnyi, Kiam Choo, Olaf Ronneberger, Chimezie Iwuanyanwu, Shubin Zhao, David Soergel, Cho-Jui Hsieh, Irene Cai, Sharig Iqbal, Martin Sundermeyer, Zhe Chen, Elie Burshtein, Chaitanya Malaviya, Fadi Biady, Prakash Shroff, Inderjit Dhillon, Tejas Latkar, Chris Dyer, Hannah Forbes, Massimo Nicosia, Vitaly Nikolaev, Somer Greene, Marin Georgiev, Pidong Wang, Nina Martin, Hanie Sedghi, John Zhang, Praseem Banzal, Doug Fritz, Vikram Rao, Xuezhi Wang, Ji-ageng Zhang, Viorica Patraucean, Dayou Du, Igor Mordatch, Ivan Jurin, Lewis Liu, Ayush Dubey, Abhi Mohan, Janek Nowakowski, Vlad-Dorit Ion, Nan Wei, Reiko Tojo, Maria Abi Raad, Drew A. Hudson, Vaishakh Keshava, Shubham Agrawal, Kevin Ramirez, Zhichun Wu, Hoang Nguyen, Ji Liu, Madhavi Sewak, Bryce Petrin, DongHyun Choi, Ivan Philips, Ziyue Wang, Ioana Bica, Ankush Garg, Jarek Wilkiewicz, Priyanka Agrawal, Xiaowei Li, Danhao Guo, Emily Xue, Naseer Shaik, Andrew Leach, Sadh MNM Khan, Julia Wiesinger, Sammy Jerome, Abhishek Chakladar, Alek Wenjiao Wang, Tina Ornduff, Folake Abu, Alireza Ghaffarkhah, Marcus Wainwright, Mario Cortes, Frederick Liu, Joshua Maynez, Andreas Terzis, Pouya Samangouei, Rihan Mansour, Tomasz Kępa, François-Xavier Aubet, Anton Algymr, Dan Banica, Agoston Weisz, Andras Orban, Alexandre Senges, Ewa Andrejczuk, Mark Geller, Niccolo Dal Santo, Valentin Ankinl, Majd Al Merey, Martin Baumeil, Trevor Strohman, Junwen Bai, Slav Petrov, Yonghui Wu, Demis Hassabis, Koray Kavukcuoglu, Jeff Dean, and Oriol Vinyals. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint, arXiv:2403.05530.

Menno van Zaanen and Simon Worts. 2006. Unsupervised measurement of translation quality using multi-engine, bi-directional translation. In Proceedings of the 19th Australian Joint Conference on Artificial Intelligence: Advances in Artificial Intelligence, AI’06, page 1208–1214, Berlin, Heidelberg. Springer-Verlag.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.

Yiming Wang, Pei Zhang, Baosong Yang, Derek F. Wong, and Rui Wang. 2024. Latent space chain-of-embedding enables output-free llm self-evaluation. Preprint, arXiv:2410.13640.

Warren Weaver. 1952. Translation. In Proceedings of the Conference on Mechanical Translation.

Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2023. A paradigm shift in machine translation: Boosting translation performance of large language models. arXiv preprint arXiv:2309.11674.

Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024. Benchmarking benchmark leakage in large language models. Preprint, arXiv:2404.18824.

Xin Zhou, Martin Weyssow, Ratnadira Widyasari, Ting Zhang, Junda He, Yunbo Lyu, Jianming Chang, Beiqi Zhang, Dan Huang, and David Lo. 2025. Lessleak-bench: A first investigation of data leakage in llms across 83 software engineering benchmarks. Preprint, arXiv:2502.06215.
A  Potential Risks

The potential risks associated with this work include the misuse of the generated artifacts outside of research contexts. To mitigate this, we have included a license in the appendix that specifies the intended use of the artifacts and restricts their use to research purposes only. We release them under the MIT license: https://github.com/ulab-uiuc/consistencychecker.

B  Artifact Use

License compliance  We confirm that our use of existing artifacts complies with their respective licenses. For artifacts created in this work, we explicitly specify intended use in the documentation, ensuring compatibility with original access conditions. Derivatives of data accessed for research purposes are restricted to research contexts only.

Model licenses  Claude-3.5-Sonnet, GPT-4, GPT-4o-mini, and Gemini-1.5-Pro are all proprietary LLMs. In the Qwen family, Qwen-2.5-1.5B, Qwen-2.5-7B, Qwen-2.5-14B, and Qwen-2.5-32B are licensed under Apache-2.0, while Qwen-2.5-72B uses the Qwen license. Phi-3-Medium is under the MIT license. Mistral-Large is under the Mistral Research License. LLaMA-3-70B is licensed under the llama3 license, while LLaMA-3.1-8B and LLaMA-3.1-79B are under the llama3.1 license. nvidia/NV-Embed-v2, the embedding model we used, is under the license CC-BY-NC 4.0.

API compliance  Open-source LLMs greater than 8B are accessed through Fireworks AI API. The Claude-3.5-Sonnet is accessed via official Anthropic endpoints. Gemini-1.5-Pro is visited from Gemini official API endpoints, and so is Mistral-Large, which is from Mistral official API. Phi-3-Medium is accessed through the NVIDIA Nim API. We accept and are compliant to each of their terms, respectively.

Benchmark use  We do not use any existing benchmarks directly; we only reference their published results, and also the language pairs used in WMT 2024 automatic benchmarks.

C  Documentation of Artifacts

We provide comprehensive documentation for the artifacts used and created in this work. This includes details on the domains, languages, and linguistic phenomena covered, as well as the demographic groups represented. This study involves the generation of benchmarks and evaluation log files involving English, German, French, Spanish, Czech, Ukrainian, Japanese, and Chinese.

D  Statistics for Data

We report relevant statistics for the data used and created in this work. The four evaluator models involved in this study each generated 2 benchmark files, with one for the translation task and one for the AI-assisted programming task. Each of these benchmark files contains 10 self-consistent root nodes. For each LLM evaluated on each of these two tasks, it generated a self-consistency tree for each root node, and the degree and height of which are described in the paper.

E  Model Size and Budget

Model size and configuration  We report the number of parameters for each model used in this study:

Claude-3.5-Sonnet: unknown size;
GPT-4: unknown size;
GPT-4o-mini: unknown size;
Gemini-1.5-Pro: unknown size;
Mistral-Large: 128 billion parameters;
LLaMA-3-70B: 70 billion parameters;
Phi-3-Medium: 14 billion parameters;
Qwen-2.5-1.5B: 1.5 billion parameters;
Qwen-2.5-7B: 7 billion parameters;
Qwen-2.5-14B: 14 billion parameters;
Qwen-2.5-32B: 32.5 billion parameters;
Qwen-2.5-72B: 72 billion parameters;
LLaMA-3.1-8B: 8 billion parameters;
LLaMA-3.1-70B: 70 billion parameters.

Computational resources  This project was conducted on 2 NVIDIA RTX A6000 GPUs. All LLMs with fewer than 8 billion parameters were hosted locally using vLLM on these GPUs, with random seed set to 42. Larger models and proprietary APIs (such as GPT-4o-mini, Qwen-2.5-14B, Qwen-2.5-70B, and LLaMA-3.1-72B) were accessed through external APIs.

Model variants  All LLMs in the LLaMA and Qwen families used in this work are instruction-tuned versions. No additional quantization was applied to any model.
F Experimental Settings

We discuss the experimental setup, including hyperparameter search and the best-found hyperparameter values. The temperature for text generation was set to 0.6. The tree structure used for evaluation, on the two tasks, aside from the experiment for comparison with WMT 2024, had built 10 self-consistency tree with an out-degree of 3 (operation pairs) and a height of 3. These parameters were chosen based on preliminary experiments to balance diversity and consistency in the generated outputs. In the experiments to compare our framework with WMT 2024, for language pairs with English as one of the languages, the out-degree is 1 and the height of the trees is 12. For all those pairs without English as one of the two languages, the out-degree is 3 and the height is 3.

G Parameters for Packages

We used the following packages and reported their implementation, model, and parameter settings: nltk.translate.bleu_score: For calculating BLEU scores, we used the sentence_bleu function with SmoothingFunction().method1.

H Information about Use of AI Assistants

We acknowledge the use of AI assistants in the preparation of this work. The AI assistants were used for generating and refining text, as well as for providing suggestions on experimental design and hyperparameter tuning. All outputs generated by AI assistants were reviewed and validated by the authors to ensure accuracy and relevance.

I Additional Experimental Results

Additional results on path length  We also release our results on these 2 defined tasks in these tables: Table 6, 7, and 8, where we report the numbers for \( C_1(\mathcal{F}) \), \( C_2(\mathcal{F}) \), and \( C_3(\mathcal{F}) \). They correspond to and provide more information than the tables Table 4 and 5. They also manifest the trend that under the same circumstances \( C_2(\mathcal{F}) \) is lower than \( C_1(\mathcal{F}) \), and \( C_3(\mathcal{F}) \) is lower than \( C_2(\mathcal{F}) \).

Additional results on correlation scores in WMT 2024  Table 9 shows the comparison between our ConsistencyChecker rankings and WMT 2024 rankings. In most cases, at least 4 out of 6 LLMs share the same order across both evaluations. To further quantify the relationship, we calculate Pearson’s correlation between our consistency scores and WMT’s three automated metrics. Since AutoRank and MetricX are inversely related to performance, we use their absolute values for the correlation analysis. As illustrated in Figure 5, there is a strong correlation between ConsistencyChecker scores and WMT 2024 evaluations. This relationship is evident using both NV-Embed-v2-based cosine embedding similarity and BLEU as the similarity metric.

J Additional Case Study

J.1 Full Examples of Leaf Nodes

We include the full example of one root node and two leaf nodes generated by LLaMA-3.1-8B and LLaMA-3.1-70B for consistency evaluation in Figure 6.

J.2 Full Examples of Self-consistency Tree

In this section, we present two complete self-consistency trees with a depth of 3 and a branching factor of 3 for the machine translation task. These trees are generated by two different models: Qwen-2.5-7B and Qwen-2.5-72B.

This is the full content of the self-consistency tree generated by Qwen-2.5-7B on machine translation tasks.

Root (Level 0, NV-Embed-v2 similarity to root 1.0000). In the ever-evolving landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) stand out as a significant marker
of the shifting tides in international finance. The ECB’s decision to recalibrate its quantitative easing (QE) program, alongside a series of nuanced interest rate adjustments, has sent shockwaves through the financial markets, particularly affecting emerging economies that are intricately linked to the
Eurozone’s economic health. These changes, while designed to stabilize the Euro
and stimulate economic growth within the European Union, have had far-reaching consequences, including capital flight, currency volatility, and heightened risk aversion among investors. The ECB’s forward guidance, which aims to provide clarity and predictability, has been met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to address the diverse and dynamic economic conditions across different regions. In response, smaller financial institutions and central banks in emerging markets have been compelled to adopt a range of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, in an effort to insulate their economies from external shocks and capitalize on the emerging opportunities. The interplay between these global and local economic strategies underscores the complex and
Table 6: Machine translation consistency evaluation with different path lengths. Scores(%) evaluate LLMs’ ability to attain semantics after translating across different languages.

<table>
  <tr>
    <th>Evaluatee</th>
    <th>\( C_1(\mathcal{F}) \)</th>
    <th>\( C_2(\mathcal{F}) \)</th>
    <th>\( C_3(\mathcal{F}) \)</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>99.2<sub>±0.0</sub></td>
    <td>98.7<sub>±0.0</sub></td>
    <td>98.0<sub>±0.0</sub></td>
  </tr>
  <tr>
    <td>Qwen-2.5-1.5B</td>
    <td>91.9<sub>±0.1</sub></td>
    <td>86.5<sub>±0.1</sub></td>
    <td>80.3<sub>±0.5</sub></td>
  </tr>
  <tr>
    <td>Qwen-2.5-7B</td>
    <td>96.4<sub>±0.1</sub></td>
    <td>93.5<sub>±0.3</sub></td>
    <td>90.0<sub>±0.8</sub></td>
  </tr>
  <tr>
    <td>Qwen-2.5-14B</td>
    <td>97.4<sub>±0.0</sub></td>
    <td>96.1<sub>±0.1</sub></td>
    <td>94.7<sub>±0.1</sub></td>
  </tr>
  <tr>
    <td>Qwen-2.5-32B</td>
    <td>97.6<sub>±0.1</sub></td>
    <td>97.4<sub>±0.0</sub></td>
    <td>96.4<sub>±0.0</sub></td>
  </tr>
  <tr>
    <td>Qwen-2.5-72B</td>
    <td>99.1<sub>±0.0</sub></td>
    <td>98.3<sub>±0.0</sub></td>
    <td>97.2<sub>±0.0</sub></td>
  </tr>
  <tr>
    <td>LLaMA-3.1-8B</td>
    <td>89.5<sub>±0.1</sub></td>
    <td>75.7<sub>±1.6</sub></td>
    <td>67.5<sub>±3.0</sub></td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70B</td>
    <td>87.4<sub>±0.7</sub></td>
    <td>80.7<sub>±1.1</sub></td>
    <td>71.9<sub>±3.2</sub></td>
  </tr>
</table>

Table 7: AI-assisted programming consistency evaluation with different path lengths. Scores(%) evaluate LLMs’ ability to preserve functionality after multi-step algorithm transformation.

<table>
  <tr>
    <th>Evaluatee</th>
    <th>\( C_1(\mathcal{F}) \)</th>
    <th>\( C_2(\mathcal{F}) \)</th>
    <th>\( C_3(\mathcal{F}) \)</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>90.6<sub>±0.2</sub></td>
    <td>84.7<sub>±0.6</sub></td>
    <td>76.5<sub>±2.7</sub></td>
  </tr>
  <tr>
    <td>Qwen-2.5-1.5B</td>
    <td>90.2<sub>±0.2</sub></td>
    <td>80.0<sub>±0.6</sub></td>
    <td>63.4<sub>±1.4</sub></td>
  </tr>
  <tr>
    <td>Qwen-2.5-7B</td>
    <td>85.4<sub>±0.2</sub></td>
    <td>80.2<sub>±0.2</sub></td>
    <td>71.7<sub>±0.4</sub></td>
  </tr>
  <tr>
    <td>Qwen-2.5-14B</td>
    <td>90.3<sub>±0.2</sub></td>
    <td>83.1<sub>±0.6</sub></td>
    <td>79.9<sub>±1.0</sub></td>
  </tr>
  <tr>
    <td>Qwen-2.5-32B</td>
    <td>91.1<sub>±0.4</sub></td>
    <td>88.3<sub>±0.7</sub></td>
    <td>85.1<sub>±1.1</sub></td>
  </tr>
  <tr>
    <td>Qwen-2.5-72B</td>
    <td>90.8<sub>±0.2</sub></td>
    <td>85.3<sub>±0.5</sub></td>
    <td>77.0<sub>±1.9</sub></td>
  </tr>
  <tr>
    <td>LLaMA-3.1-8B</td>
    <td>88.1<sub>±0.2</sub></td>
    <td>76.0<sub>±0.5</sub></td>
    <td>60.4<sub>±1.0</sub></td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70B</td>
    <td>87.9<sub>±0.4</sub></td>
    <td>82.7<sub>±0.6</sub></td>
    <td>83.5<sub>±1.0</sub></td>
  </tr>
</table>

Table 8: Consistency results under different evaluator models and different path lengths. (Top) Machine translation consistency scores (%) across model scales (7B–72B) and transform complexity (\( C_1(\mathcal{F})–C_3(\mathcal{F}) \)). (Bottom) Code equivalence preservation scores (%) for algorithm transformations at varying abstraction levels (\( C_1(\mathcal{F})–C_3(\mathcal{F}) \)).

<table>
  <tr>
    <th rowspan="2">Evaluatee</th>
    <th colspan="3">Evaluator</th>
    <th colspan="3">Qwen-2.5-7B</th>
    <th colspan="3">Qwen-2.5-72B</th>
  </tr>
  <tr>
    <th>\( C_1(\mathcal{F}) \)</th>
    <th>\( C_2(\mathcal{F}) \)</th>
    <th>\( C_3(\mathcal{F}) \)</th>
    <th>\( C_1(\mathcal{F}) \)</th>
    <th>\( C_2(\mathcal{F}) \)</th>
    <th>\( C_3(\mathcal{F}) \)</th>
  </tr>
  <tr>
    <td colspan="8"><b>Machine translation</b></td>
  </tr>
  <tr>
    <td>Qwen-2.5-7B</td>
    <td>90.1<sub>±0.9</sub></td>
    <td>82.8<sub>±3.0</sub></td>
    <td>76.8<sub>±5.1</sub></td>
    <td>96.4<sub>±0.1</sub></td>
    <td>93.5<sub>±0.3</sub></td>
    <td>90.0<sub>±0.8</sub></td>
  </tr>
  <tr>
    <td>Qwen-2.5-72B</td>
    <td>90.0<sub>±1.8</sub></td>
    <td>84.4<sub>±4.2</sub></td>
    <td>78.2<sub>±7.3</sub></td>
    <td>99.1<sub>±0.0</sub></td>
    <td>98.3<sub>±0.0</sub></td>
    <td>97.2<sub>±0.9</sub></td>
  </tr>
  <tr>
    <td>LLaMA-3.1-8B</td>
    <td>81.6<sub>±0.5</sub></td>
    <td>73.0<sub>±1.3</sub></td>
    <td>61.7<sub>±4.8</sub></td>
    <td>89.5<sub>±0.1</sub></td>
    <td>75.7<sub>±1.6</sub></td>
    <td>67.5<sub>±3.0</sub></td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70B</td>
    <td>82.3<sub>±1.8</sub></td>
    <td>68.8<sub>±2.3</sub></td>
    <td>30.8<sub>±9.2</sub></td>
    <td>87.4<sub>±0.7</sub></td>
    <td>80.7<sub>±1.1</sub></td>
    <td>71.9<sub>±3.2</sub></td>
  </tr>
  <tr>
    <td colspan="8"><b>AI-assisted programming</b></td>
  </tr>
  <tr>
    <td>Qwen-2.5-7B</td>
    <td>86.2<sub>±0.4</sub></td>
    <td>78.2<sub>±0.8</sub></td>
    <td>72.2<sub>±1.9</sub></td>
    <td>85.4<sub>±0.2</sub></td>
    <td>80.2<sub>±0.2</sub></td>
    <td>71.7<sub>±0.4</sub></td>
  </tr>
  <tr>
    <td>Qwen-2.5-72B</td>
    <td>87.2<sub>±0.2</sub></td>
    <td>80.1<sub>±0.6</sub></td>
    <td>79.4<sub>±2.3</sub></td>
    <td>90.8<sub>±0.2</sub></td>
    <td>85.3<sub>±0.5</sub></td>
    <td>77.0<sub>±1.9</sub></td>
  </tr>
  <tr>
    <td>LLaMA-3.1-8B</td>
    <td>84.4<sub>±0.2</sub></td>
    <td>72.1<sub>±0.4</sub></td>
    <td>61.0<sub>±1.1</sub></td>
    <td>88.1<sub>±0.2</sub></td>
    <td>76.0<sub>±0.5</sub></td>
    <td>60.4<sub>±1.0</sub></td>
  </tr>
  <tr>
    <td>LLaMA-3.1-70B</td>
    <td>91.0<sub>±0.5</sub></td>
    <td>85.6<sub>±1.0</sub></td>
    <td>81.5<sub>±2.1</sub></td>
    <td>87.9<sub>±0.4</sub></td>
    <td>82.7<sub>±0.6</sub></td>
    <td>83.5<sub>±1.0</sub></td>
  </tr>
</table>

interconnected nature of the modern financial system, where the decisions of one major player can have profound implications for the rest of the world.

Node 0 (Level 1, NV-Embed-v2 similarity to root 0.3336). The communication and forward strategies between global and local economic strategies highlight the complexity and interconnectedness of modern financial systems, where one player’s decisions can have far-reaching impacts on the rest of the world.

Node 0-0 (Level 2, NV-Embed-v2 similarity to root 0.2802). The communication and promotion strategy between global and local economic strategies highlights the complexity and interconnection of modern financial systems, where the decisions of a single actor can have reverberating impacts on the rest of the world.

Node 0-0-0 (Level 3, NV-Embed-v2 similarity to root 0.2820). The communication and promotion strategies between global and local economic strategies highlight the complexity and interconnectedness of modern financial systems, where the decisions of a single actor can have repercussions on the rest of the world.

Node 0-0-1 (Level 3, NV-Embed-v2 similarity to root 0.2400). The communications and propaganda strategy between global and local economic strategies illustrates the complexity and the internal connection of modern financial systems, where the decisions of a single actor can affect the rest of the world.

Node 0-0-2 (Level 3, NV-Embed-v2 similarity to root 0.2693). The communication and promotion strategy between global and local economic strategies highlights the complexity and interconnection of modern financial systems, where the decisions of a single actor can have resonant effects on the rest of the world.

Node 0-1 (Level 2, NV-Embed-v2 similarity to root 0.2774). The communication and foresight strategies between global and local business strategies illustrate the complexity and interdependence of modern financial systems, where the decisions of one player can have far-reaching effects on the rest of the world.

Node 0-1-0 (Level 3, NV-Embed-v2 similarity to root 0.2717). The communication and foresight strategy between global and local business strategies illustrates the complexity and interdependence of modern financial systems, where the decisions of one player can have repercussions on the rest of the world.

Node 0-1-1 (Level 3, NV-Embed-v2 similarity to root 0.2615). The communication and prediction strategies between global and regional
Table 9: Comprehensive results on machine translation tasks. Consistency scores (%) across model and transform complexity (\( C_1(\mathcal{F}) \)-\( C_3(\mathcal{F}) \)) using embedding similarity and BLEU scores, versus metrics in WMT 2024 benchmarks. The rows are arranged using WMT rankings.

<table>
  <tr>
    <th rowspan="2">Czech-Ukrainian</th>
    <th colspan="3">Embedding</th>
    <th colspan="3">BLEU</th>
    <th colspan="3">AutoRank↓ MetricX↓ CometKiwi↑</th>
  </tr>
  <tr>
    <th>\( C_1(\mathcal{F}) \)</th>
    <th>\( C_2(\mathcal{F}) \)</th>
    <th>\( C_3(\mathcal{F}) \)</th>
    <th>\( C_1(\mathcal{F}) \)</th>
    <th>\( C_2(\mathcal{F}) \)</th>
    <th>\( C_3(\mathcal{F}) \)</th>
    <th>AutoRank↓</th>
    <th>MetricX↓</th>
    <th>CometKiwi↑</th>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>99.4</td>
    <td>98.9</td>
    <td>98.1</td>
    <td>87.1</td>
    <td>80.4</td>
    <td>67.3</td>
    <td>1.7</td>
    <td>1.0</td>
    <td>0.683</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>98.7</td>
    <td>97.8</td>
    <td>96.4</td>
    <td>78.1</td>
    <td>69.2</td>
    <td>55.5</td>
    <td>2.0</td>
    <td>1.4</td>
    <td>0.677</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>99.2</td>
    <td>98.5</td>
    <td>97.5</td>
    <td>86.0</td>
    <td>78.3</td>
    <td>57.6</td>
    <td>2.0</td>
    <td>1.2</td>
    <td>0.668</td>
  </tr>
  <tr>
    <td>Mistral-Large</td>
    <td>98.6</td>
    <td>97.6</td>
    <td>96.5</td>
    <td>78.3</td>
    <td>68.8</td>
    <td>56.7</td>
    <td>2.3</td>
    <td>1.6</td>
    <td>0.666</td>
  </tr>
  <tr>
    <td>LLaMA-3-70B</td>
    <td>96.4</td>
    <td>96.7</td>
    <td>95.9</td>
    <td>79.2</td>
    <td>70.4</td>
    <td>55.8</td>
    <td>2.6</td>
    <td>2.0</td>
    <td>0.661</td>
  </tr>
  <tr>
    <td>Phi-3-Medium</td>
    <td>71.1</td>
    <td>58.3</td>
    <td>44.9</td>
    <td>28.9</td>
    <td>14.6</td>
    <td>5.6</td>
    <td>9.1</td>
    <td>6.5</td>
    <td>0.425</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">English-German</th>
    <th colspan="3">Embedding</th>
    <th colspan="3">BLEU</th>
    <th colspan="3">AutoRank↓ MetricX↓ CometKiwi↑</th>
  </tr>
  <tr>
    <th>\( C_1(\mathcal{F}) \)</th>
    <th>\( C_2(\mathcal{F}) \)</th>
    <th>\( C_3(\mathcal{F}) \)</th>
    <th>\( C_1(\mathcal{F}) \)</th>
    <th>\( C_2(\mathcal{F}) \)</th>
    <th>\( C_3(\mathcal{F}) \)</th>
    <th>AutoRank↓</th>
    <th>MetricX↓</th>
    <th>CometKiwi↑</th>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>99.4</td>
    <td>99.2</td>
    <td>99.1</td>
    <td>85.8</td>
    <td>83.5</td>
    <td>82.4</td>
    <td>1.8</td>
    <td>1.4</td>
    <td>0.700</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>99.8</td>
    <td>99.8</td>
    <td>99.8</td>
    <td>95.6</td>
    <td>94.8</td>
    <td>94.4</td>
    <td>1.9</td>
    <td>1.4</td>
    <td>0.695</td>
  </tr>
  <tr>
    <td>Mistral-Large</td>
    <td>99.4</td>
    <td>99.3</td>
    <td>99.2</td>
    <td>88.5</td>
    <td>86.9</td>
    <td>85.5</td>
    <td>2.0</td>
    <td>1.5</td>
    <td>0.694</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>99.8</td>
    <td>99.7</td>
    <td>99.6</td>
    <td>93.3</td>
    <td>92.8</td>
    <td>91.4</td>
    <td>2.2</td>
    <td>1.5</td>
    <td>0.688</td>
  </tr>
  <tr>
    <td>LLaMA-3-70B</td>
    <td>99.3</td>
    <td>99.1</td>
    <td>98.8</td>
    <td>86.4</td>
    <td>84.3</td>
    <td>82.5</td>
    <td>2.5</td>
    <td>1.7</td>
    <td>0.686</td>
  </tr>
  <tr>
    <td>Phi-3-Medium</td>
    <td>98.5</td>
    <td>97.9</td>
    <td>97.2</td>
    <td>81.2</td>
    <td>76.2</td>
    <td>72.7</td>
    <td>3.4</td>
    <td>2.0</td>
    <td>0.657</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">English-Japanese</th>
    <th colspan="3">Embedding</th>
    <th colspan="3">BLEU</th>
    <th colspan="3">AutoRank↓ MetricX↓ CometKiwi↑</th>
  </tr>
  <tr>
    <th>\( C_1(\mathcal{F}) \)</th>
    <th>\( C_2(\mathcal{F}) \)</th>
    <th>\( C_3(\mathcal{F}) \)</th>
    <th>\( C_1(\mathcal{F}) \)</th>
    <th>\( C_2(\mathcal{F}) \)</th>
    <th>\( C_3(\mathcal{F}) \)</th>
    <th>AutoRank↓</th>
    <th>MetricX↓</th>
    <th>CometKiwi↑</th>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>99.4</td>
    <td>99.2</td>
    <td>99.0</td>
    <td>89.2</td>
    <td>87.3</td>
    <td>85.8</td>
    <td>1.5</td>
    <td>2.3</td>
    <td>0.744</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>99.3</td>
    <td>99.2</td>
    <td>99.1</td>
    <td>88.1</td>
    <td>86.3</td>
    <td>84.9</td>
    <td>1.7</td>
    <td>2.5</td>
    <td>0.734</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>98.4</td>
    <td>98.1</td>
    <td>97.8</td>
    <td>76.6</td>
    <td>72.2</td>
    <td>69.4</td>
    <td>1.7</td>
    <td>2.7</td>
    <td>0.740</td>
  </tr>
  <tr>
    <td>LLaMA-3-70B</td>
    <td>93.5</td>
    <td>90.5</td>
    <td>87.9</td>
    <td>71.4</td>
    <td>65.4</td>
    <td>60.0</td>
    <td>2.6</td>
    <td>3.5</td>
    <td>0.714</td>
  </tr>
  <tr>
    <td>Phi-3-Medium</td>
    <td>97.4</td>
    <td>96.1</td>
    <td>95.3</td>
    <td>65.5</td>
    <td>57.1</td>
    <td>51.8</td>
    <td>2.8</td>
    <td>3.6</td>
    <td>0.709</td>
  </tr>
  <tr>
    <td>Mistral-Large</td>
    <td>98.2</td>
    <td>97.4</td>
    <td>96.9</td>
    <td>79.2</td>
    <td>73.9</td>
    <td>70.7</td>
    <td>2.9</td>
    <td>3.8</td>
    <td>0.707</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">English-Chinese</th>
    <th colspan="3">Embedding</th>
    <th colspan="3">BLEU</th>
    <th colspan="3">AutoRank↓ MetricX↓ CometKiwi↑</th>
  </tr>
  <tr>
    <th>\( C_1(\mathcal{F}) \)</th>
    <th>\( C_2(\mathcal{F}) \)</th>
    <th>\( C_3(\mathcal{F}) \)</th>
    <th>\( C_1(\mathcal{F}) \)</th>
    <th>\( C_2(\mathcal{F}) \)</th>
    <th>\( C_3(\mathcal{F}) \)</th>
    <th>AutoRank↓</th>
    <th>MetricX↓</th>
    <th>CometKiwi↑</th>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>99.5</td>
    <td>99.4</td>
    <td>99.3</td>
    <td>88.7</td>
    <td>86.9</td>
    <td>1.7</td>
    <td>3.0</td>
    <td>0.703</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>99.4</td>
    <td>99.3</td>
    <td>99.3</td>
    <td>89.4</td>
    <td>88.5</td>
    <td>1.8</td>
    <td>3.1</td>
    <td>0.698</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>98.6</td>
    <td>98.3</td>
    <td>98.1</td>
    <td>76.4</td>
    <td>73.3</td>
    <td>71.4</td>
    <td>2.0</td>
    <td>3.3</td>
    <td>0.693</td>
  </tr>
  <tr>
    <td>Mistral-Large</td>
    <td>97.5</td>
    <td>97.7</td>
    <td>97.5</td>
    <td>82.5</td>
    <td>79.5</td>
    <td>77.6</td>
    <td>2.8</td>
    <td>4.0</td>
    <td>0.665</td>
  </tr>
  <tr>
    <td>LLaMA-3-70B</td>
    <td>98.6</td>
    <td>98.3</td>
    <td>97.9</td>
    <td>82.4</td>
    <td>78.8</td>
    <td>75.9</td>
    <td>2.8</td>
    <td>3.9</td>
    <td>0.662</td>
  </tr>
  <tr>
    <td>Phi-3-Medium</td>
    <td>98.1</td>
    <td>97.3</td>
    <td>96.6</td>
    <td>72.2</td>
    <td>65.9</td>
    <td>61.8</td>
    <td>3.1</td>
    <td>4.2</td>
    <td>0.648</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Japanese-Chinese</th>
    <th colspan="3">Embedding</th>
    <th colspan="3">BLEU</th>
    <th colspan="3">AutoRank↓ MetricX↓ CometKiwi↑</th>
  </tr>
  <tr>
    <th>\( C_1(\mathcal{F}) \)</th>
    <th>\( C_2(\mathcal{F}) \)</th>
    <th>\( C_3(\mathcal{F}) \)</th>
    <th>\( C_1(\mathcal{F}) \)</th>
    <th>\( C_2(\mathcal{F}) \)</th>
    <th>\( C_3(\mathcal{F}) \)</th>
    <th>AutoRank↓</th>
    <th>MetricX↓</th>
    <th>CometKiwi↑</th>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>98.7</td>
    <td>97.7</td>
    <td>96.0</td>
    <td>77.4</td>
    <td>65.4</td>
    <td>49.6</td>
    <td>1.7</td>
    <td>3.5</td>
    <td>0.603</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>98.8</td>
    <td>98.1</td>
    <td>96.8</td>
    <td>80.1</td>
    <td>70.0</td>
    <td>54.0</td>
    <td>1.9</td>
    <td>3.5</td>
    <td>0.595</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>97.6</td>
    <td>96.0</td>
    <td>93.7</td>
    <td>65.3</td>
    <td>51.6</td>
    <td>37.3</td>
    <td>2.1</td>
    <td>3.8</td>
    <td>0.597</td>
  </tr>
  <tr>
    <td>LLaMA-3-70B</td>
    <td>96.2</td>
    <td>93.4</td>
    <td>88.9</td>
    <td>64.6</td>
    <td>48.0</td>
    <td>28.5</td>
    <td>3.1</td>
    <td>4.7</td>
    <td>0.578</td>
  </tr>
  <tr>
    <td>Mistral-Large</td>
    <td>98.1</td>
    <td>96.7</td>
    <td>94.4</td>
    <td>69.4</td>
    <td>55.6</td>
    <td>39.9</td>
    <td>3.5</td>
    <td>4.9</td>
    <td>0.568</td>
  </tr>
  <tr>
    <td>Phi-3-Medium</td>
    <td>93.9</td>
    <td>89.3</td>
    <td>84.9</td>
    <td>52.1</td>
    <td>38.6</td>
    <td>26.0</td>
    <td>4.0</td>
    <td>5.1</td>
    <td>0.552</td>
  </tr>
</table>

business concepts illustrate the complexity and interconnectedness of modern financial systems, where the decisions of one player can have far-reaching effects on the rest of the world.

Node 0-1-2 (Level 3, NV-Embed-v2 similarity to root 0.2721). The communication and forecast strategy between global and local business strategies illustrate the complexity and interdependence of modern financial systems, where the decisions of one player can have long-lasting effects on the rest of the world.

Node 0-2 (Level 2, NV-Embed-v2 similarity to root 0.2783). The strategy of communication and advancement between global and local strategies highlights the complexity and interconnection of modern financial systems, where the decisions of a single player can have long-term repercussions for the rest of the world.

Node 0-2-0 (Level 3, NV-Embed-v2 similarity to root 0.2595). The communication and advancement strategy between global and local strategies highlights the complexity and interconnection of modern financial systems, where the decisions of a single player can have long-term repercussions for the rest of the world.

Node 0-2-1 (Level 3, NV-Embed-v2 similarity to root 0.2852). The strategy of communication and progress balancing between global and local strategies illustrates the complexity and connection of modern financial systems, where the decisions of a single player can have long-term repercussions for the rest of the world.

Node 0-2-2 (Level 3, NV-Embed-v2 similarity to root 0.2687). The strategy of communication and advancement between global and local strategies highlights the complexity and interconnection of modern financial systems, in which the decisions of a single player can have long-term repercussions for the rest of the world.

Node 1 (Level 1, NV-Embed-v2 similarity to root 0.9537). In the constantly changing landscape of global economic policy, the recent measures of the European Central Bank (ECB) stand out as a significant sign of evolving
Figure 5: Correlation between ConsistencyChecker evaluation and metrics in WMT 2024 benchmarks. The three figures on the left column from top to bottom correspond to the Czech-Ukrainian, English-German, and English-Japanese language pairs. The two figures on the right column from top to bottom correspond to the English-Chinese and Japanese-Chinese language pairs. Each of these figures shows the correlation between the ConsistencyChecker scores by embedding and by BLEU at \( C_1(\mathcal{F}), C_2(\mathcal{F}), \) and \( C_3(\mathcal{F}) \), with the three metrics used in WMT 2024, AutoRank, MetricX, and CometKiwi. The results show that both embedding-based and BLEU-based ConsistencyChecker scores share high correlation with the three metrics in WMT 2024.

directions in the international financial world. The ECB's decision to adjust its Quantitative Easing (QE) programs and a series of nuanced interest rate adjustments have sparked shockwaves in financial markets, particularly in emerging economies closely linked to the economic health of the eurozone. These changes, aimed at stabilizing the euro and fostering growth in the European Union, have far-reaching consequences, including capital flight, currency volatility, and heightened risk aversion among investors.

The forward guidance of the ECB, which seeks to provide clarity and predictability, is met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to accommodate the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks in emerging economies are forced to take a range of adaptive measures, from tightening their interest conditions to diversifying their investment portfolios, to shield their economies
Root Node: def main(): return "In the rapidly evolving landscape of artificial intelligence (AI), the integration of these advanced technologies into healthcare systems represents a significant frontier with profound implications for medical diagnostics and patient care. Recent advancements have demonstrated the potential of AI to enhance diagnostic accuracy, streamline clinical workflows, and personalize treatment plans, thereby improving patient outcomes and reducing healthcare costs. However, the deployment of AI in this sensitive domain is not without its challenges. Ethical considerations, such as data privacy, algorithmic bias, and the potential for job displacement, are paramount. Moreover, the regulatory landscape must adapt to ensure that AI applications meet stringent safety and efficacy standards. Despite these hurdles, the collaboration between tech developers, healthcare providers, and regulatory bodies is fostering a cautious yet optimistic approach to AI integration. The future of healthcare may well be defined by the seamless fusion of human expertise and machine intelligence, creating a hybrid model that leverages the strengths of both to deliver more effective and efficient care."

LLaMA-3.1-8B Leaf Node: def main(): return "In the constantly evolving scenario of artificial intelligence (AI), the integration of these advanced technologies in medical systems represents a significant frontier with important implications for medical diagnosis and patient care. Recent advances have demonstrated the potential of AI to improve diagnostic accuracy, simplify clinical workflows, and personalize treatment plans, leading to better patient outcomes and a reduction in medical costs. However, implementing AI in this sensitive domain does not come without its challenges. Ethical considerations, such as data privacy, algorithmic bias, and job displacement, are fundamental. Furthermore, the regulatory landscape must adapt to ensure that AI applications meet strict standards of safety and efficacy. Despite these difficulties, collaboration between technology developers, healthcare providers, and regulatory bodies is promoting a cautious yet optimistic approach to the integration of AI. The future of healthcare may be defined by the seamless fusion of human experience and artificial intelligence, creating a hybrid model that leverages the strengths of both to offer more effective and efficient care."

LLaMA-3.1-70B Leaf Node: def main(): return "In the rapidly evolving landscape of artificial intelligence (AI), the integration of these advanced technologies into healthcare systems represents a significant frontier with profound implications for medical diagnosis and patient care. Recent advances have demonstrated AI's potential to improve diagnostic accuracy, streamline clinical workflows, and personalize treatment plans, thereby enhancing patient outcomes and reducing healthcare costs. However, the deployment of AI in this sensitive domain is not without challenges. Ethical considerations, such as data privacy, algorithmic bias, and potential job displacement, are paramount. Moreover, the regulatory landscape must adapt to ensure that AI applications meet stringent safety and efficacy standards. Despite these obstacles, collaboration between technology developers, healthcare providers, and regulatory agencies is fostering a cautious yet optimistic approach to AI integration. The future of healthcare may be defined by the seamless fusion of human expertise and machine intelligence, creating a hybrid model that leverages the strengths of both to deliver more effective and efficient care."

Figure 6: Example of root nodes and leaf nodes in a self-consistency tree.
international financial world. The ECB's decision to modulate its Quantitative Easing (QE) programs and a series of subtle adjustments to interest rates have had repercussions in financial markets, particularly in emerging economies closely linked to the economic health of the eurozone. These changes, aimed at stabilizing the euro and promoting growth in the European Union, have consequences far beyond, including capital flight, currency volatility, and an increased risk aversion among investors. The ECB's forward guidance, which aims to provide clarity and predictability, is met with mixed reactions, as some market analysts suggest that it lacks the flexibility needed to adapt to diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks of emerging economies are forced to adopt a range of adaptive measures, from reducing interest rates to diversifying investment portfolios, in order to protect their economies from external shocks and seize emerging opportunities. The interaction between these global and local economic strategies highlights the complex and interconnected nature of modern financial systems, where the decisions of a major player can have profound impacts on the rest of the world.

Node 1-0-0 (Level 3, NV-Embed-v2 similarity to root 0.0520). None

Node 1-0-1 (Level 3, NV-Embed-v2 similarity to root 0.8577). In the constantly changing global economic landscape, the recent measures of the European Central Bank (ECB) represent an important signal for the emerging directions on the international financial market. The decision of the ECB to modulate its Quantitative Liquidity Provision (QLD) and make a series of subtle adjustments to interest rates has elicited reactions from the financial market, particularly in emerging economies that rely on the economic condition of the Eurozone. These changes, aimed at stabilizing the Euro and promoting economic growth in the European Union, have consequences far beyond the Eurozone region, including capital flight, currency fluctuations, and increased risk aversion among investors. The ECB's foresight in providing clarity and predictability is met with mixed reactions, as some market analysts believe it lacks the flexibility to adapt to diverse and dynamic economic conditions in various regions. In response to this, smaller financial institutions and central banks in emerging economies must take a series of adequate measures, ranging from reducing interest rates to diversifying investment portfolios, to protect their economies from external shocks and to capitalize on emerging opportunities. The interaction between these global and local economic strategies highlights the complex and interconnected nature of modern financial systems, in which the decisions of a major player can profoundly influence the world economy.

Node 1-0-2 (Level 3, NV-Embed-v2 similarity to root 0.8955). In the constant scenario of global economic change, the recent measures of the European Central Bank (ECB) stand out as a significant sign of emerging directions in international finance. The ECB's decision to modulate its Quantitative Easing (QE) programs and a series of subtle adjustments to interest rates have had repercussions on financial markets, especially in emerging countries closely linked to the economic health of the eurozone. These changes, aimed at stabilizing the euro and promoting growth in the European Union, have consequences beyond, including capital flight, currency volatility, and increased risk aversion among investors. The guidance provided by the ECB for the future, with the aim of providing clarity and predictability, is received with mixed reactions, as some market analysts suggest that it lacks the flexibility needed to adapt to dynamic and diverse economic conditions in different regions. As a response, smaller financial institutions and central banks of emerging countries are obliged to adopt a variety of adaptive measures, from reducing interest rates to diversifying investment portfolios, in order to protect their economies from external shocks and seize emerging opportunities. The interaction between these global and local economic strategies highlights the complex and interconnected nature of modern financial systems, where the decisions of a major player can have profound impacts on the rest of the world.

Node 1-1 (Level 2, NV-Embed-v2 similarity to root 0.9165). In the constantly changing landscape of global economic and political conditions, the recent measures of the European Central Bank (ECB) stand out as a significant indicator of developing trends in the international financial world. The ECB's decision to adjust its Quantitative Easing (QE) programs and implement a series of fine-tuned interest rate adjustments triggers shockwaves in financial markets, especially in emerging economies closely linked to the economic health of the Eurozone. These changes, aimed at stabilizing the euro and promoting growth in the European Union, have far-reaching consequences, including capital flight, currency optimality, and increased risk aversion among investors. The forward-looking but cautious approach of the ECB, which provides clarity and predictability, meets with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to accommodate the diverse and dynamic economic conditions in various regions. In response, smaller financial institutions and central banks in emerging economies are forced to take a range of adaptive measures, from interest rate reticence to diversifying their investment portfolios, in order to shield their economies from external shocks and seize new opportunities. The interaction between these global and local economic strategies highlights the complex and intertwined nature of modern financial systems, where the decisions of one major player can have profound impacts on the rest of the world.

Node 1-1-0 (Level 3, NV-Embed-v2 similarity to root 0.9070). In the constantly evolving global economic and political landscape, the recent measures of the European Central Bank (ECB) stand out as a significant indicator of trends developing in the international financial world. The decision of the ECB to modify its Quantitative Easing (QE) programs and apply a series of delicate interest rate adjustments has repercussions in financial markets, particularly in emerging economies closely linked to the
economic health of the eurozone. These changes, aimed at stabilizing the euro and fostering growth in the European Union, have far-reaching consequences, including capital flight, currency optimization, and increased investor aversion. The ECB’s forward-looking and prudent approach, which brings clarity and predictability, meets with mixed reactions, as some market analysts claim it lacks sufficient flexibility to adapt to the diverse and dynamic economic conditions of various regions. In response, smaller financial institutions and central banks of emerging economies are compelled to adopt a range of adaptive measures, from interest rate restraint to portfolio diversification, to protect their economies from external shocks and seize new opportunities. The interaction between these global and local economic strategies highlights the complex and interwoven nature of modern financial systems, where the decisions of a major player can have profound impacts on the rest of the world.

Node 1-1-1 (Level 3, NV-Embed-v2 similarity to root 0.8573). In the constantly changing landscape of global economic and political conditions, the recent measures of the European Central Bank (ECB) serve as an important indicator of developing trends in the international financial world. The decision of the ECB to adjust its Quantitative Easing programs and implement a series of nuanced interest rate adjustments has waves in the financial markets, especially in development-assisted economies closely linked to the economic health of the Eurozone. These changes, aimed at stabilizing the Euro and promoting growth in the European Union, have far-reaching consequences, including capital flight, currency optimality, and increased risk aversion among investors. The ECB’s advancing but cautious approach, which offers clarity and predictability, garners mixed reactions, as some market analysts argue that it lacks the necessary flexibility to take into account the diverse and dynamic economic conditions in various regions. As a result, smaller financial institutions and central banks in development-assisted economies must take a series of adaptive measures, ranging from interest rate restraint to diversification of their investment portfolios, to protect their economies from external shocks and seize new opportunities. The interaction between these global and local economic strategies highlights the complex and interwoven nature of modern financial systems, where the decisions of a key player can have strong impacts on the rest of the world.

Node 1-1-2 (Level 3, NV-Embed-v2 similarity to root 0.9194). In the ever-changing landscape of the global economic and political scenario, the recent measures of the European Central Bank (ECB) stand out as a significant indicator of emerging trends in the international financial world. The decision of the ECB to adjust its Quantitative Easing (QE) programs and implement a series of refined interest rate adjustments triggers shockwaves in financial markets, particularly in emerging economies closely linked to the economic health of the Eurozone. These changes, aimed at stabilizing the euro and fostering growth in the European Union, have far-reaching consequences, including capital flight, currency optimization, and increased risk aversion among investors. The forward-looking but cautious approach of the ECB, which provides clarity and predictability, receives mixed reactions, as some market analysts argue that it lacks the necessary flexibility to adapt to dynamically and diversely changing economic conditions in various regions. As a response, smaller financial institutions and central banks of emerging economies are compelled to adopt a series of adaptive measures, ranging from reluctance to adjust interest rates to diversifying their investment portfolios, in order to protect their economies from external shocks and seize new opportunities. The interaction between these global and local economic strategies highlights the complex and intertwined nature of modern financial systems, where the decisions of one key player can have a profound impact on the rest of the world.

Node 1-2 (Level 2, NV-Embed-v2 similarity to root 0.9204). In the constantly evolving global landscape of economic policy, the recent measures of the European Central Bank (ECB) stand out as a significant sign of evolving directions in the international financial world. The decision of the ECB to adjust its quantitative easing (QE) programs and a series of subtle interest rate adjustments have sparked shocks in financial markets, especially in emerging economies closely tied to the economic health of the eurozone. These changes, aimed at stabilizing the euro and promoting growth in the European Union, have far-reaching consequences, including capital flight, currency volatility, and increased risk aversion among investors. The future orientation of the ECB, which seeks to provide clarity and predictability, is met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to address the diverse and dynamic economic conditions in different regions. As a response, small financial institutions and central banks in emerging economies are compelled to take a range of adaptive measures, from tightening interest rate conditions to diversifying their investment portfolios, to protect their economies from external shocks and seize emerging opportunities. The interaction between these global and local economic strategies highlights the complex and interconnected nature of modern financial systems, where decisions by a major player can have profound impacts on the rest of the world.

Node 1-2-0 (Level 3, NV-Embed-v2 similarity to root 0.8980). In the constantly evolving global economic landscape, the recent measures of the European Central Bank (ECB) stand out as a significant sign of emerging directions in the international financial world. The ECB’s decision to modify its quantitative easing (QE) programs and a series of subtle adjustments to interest rates have caused ripples in financial markets, particularly in emerging economies closely linked to the economic health of the eurozone. These changes, aimed at stabilizing the euro and promoting development within the European Union, have far-reaching consequences, including capital flight, currency volatility, and increased risk aversion among investors. The future orientation of the ECB, which seeks to bring clarity and
predictability, is met with mixed reactions, as some market analysts believe it lacks the necessary flexibility to respond to diverse and dynamic economic conditions in different regions. In response, financial institutions and central banks in emerging economies are forced to take a range of adaptive measures, from tightening interest rate conditions to diversifying their investment portfolios, to protect their economies against external shocks and seize emerging opportunities. The interaction between these global and local economic strategies highlights the complex and interconnected nature of modern financial systems, where the decisions of a major actor can have deep impacts on the rest of the world.

Node 1-2-1 (Level 3, NV-Embed-v2 similarity to root 0.9313). In the constantly changing global landscape of international finance and economic policy, the recent measures of the European Central Bank (ECB) serve as an important indicator of evolving developments. The ECB’s decision to adjust its Quantitative Easing (QE) programs and implement a series of subtle interest rate adjustments has sparked shocks in financial markets, particularly in emerging economies that have close ties to the economic health of the Eurozone. These changes, aimed at stabilizing the Euro and fostering economic growth in the European Union, have far-reaching consequences, including capital flight, currency fluctuations, and increased risk aversion among investors. The forward-looking orientation of the ECB, which seeks to provide clear and predictable measures, is met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to address the diverse and dynamic economic conditions in different regions. In response, small financial institutions and central banks in emerging economies are obligated to take a range of adaptive measures, from revealing interest rates to diversifying their investment portfolios, to protect their economies from external shocks and seize new opportunities. The interaction between these global and local economic strategies highlights the complex and intertwined nature of modern financial systems, where the decisions of a major player can have profound impacts on the rest of the world.

Node 1-2-2 (Level 3, NV-Embed-v2 similarity to root 0.9081). In the constantly evolving global economic landscape, recent measures by the European Central Bank (ECB) stand out as a significant sign of changing directions in the international financial world. The ECB’s decision to adjust its quantitative easing (QE) programs and a series of subtle adjustments in interest rates have caused shocks in financial markets, especially in emerging countries closely linked to the economic health of the eurozone. These changes, aimed at stabilizing the euro and promoting growth in the European Union, have far-reaching consequences, including capital flight, currency volatility, and increased risk aversion among investors. The ECB’s future orientation, seeking to provide clarity and predictability, is received with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to address diverse and dynamic economic conditions in different regions. As a response, small financial institutions and central banks in emerging countries are compelled to take a series of adaptive measures, from tightening interest rate conditions to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies highlights the complex and interconnected nature of modern financial systems, where the decisions of one major player can have a profound impact on the rest of the world.

Node 2 (Level 1, NV-Embed-v2 similarity to root 0.9469). In the constantly evolving landscape of global economic governance, the recent political adjustments by the European Central Bank (ECB) stand out as a significant marker of changes in international financial waters. The ECB’s decision to recalibrate its quantitative easing (QE) program, along with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly affecting emerging countries closely linked to the economic health of the Eurozone. These changes, designed to stabilize the euro and stimulate economic growth within the European Union, have had broader implications, including capital flight, currency volatility, and an increase in risk aversion among investors. The ECB’s forward guidance, intended to provide clarity and predictability, has received mixed reactions, as some market analysts argue that it lacks sufficient flexibility to address the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks in emerging countries have been forced to adopt a range of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, in order to insulate themselves from external shocks and capitalize on emerging opportunities. The interplay between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where the decisions of one major player can have profound implications for the rest of the world.

Node 2-0 (Level 2, NV-Embed-v2 similarity to root 0.9227). In the constantly evolving global economic landscape, the recent policy adjustments of the European Central Bank (ECB) stand out as a significant marker of changes in international financial waters. The ECB’s decision to recalibrate its quantitative easing (QE) program, along with a series of subtle adjustments to interest rates, has caused ripples in financial markets, particularly affecting emerging countries closely linked to the economic health of the eurozone. These changes, designed to stabilize the euro and stimulate economic growth within the European Union, have broader implications, including capital flight, currency volatility, and increased risk aversion among investors. The forward guidance, aimed at providing clarity and predictability, has received mixed reactions, as some market analysts believe it lacks the necessary flexibility to address the complex and changing economic conditions in different regions. In response, smaller financial institutions and central banks in emerging countries have had to take a series of adaptive measures, from tightening monetary
policy to diversifying portfolios, to protect themselves from external shocks and seize emerging opportunities. The interaction between global and regional economic strategies highlights the complexity and interconnectedness of modern financial systems, where the decisions of one major player can have far-reaching effects on the rest of the world.

Node 2-0-0 (Level 3, NV-Embed-v2 similarity to root 0.9002). In the constantly evolving global economic landscape, the recent policy adjustments by the European Central Bank (ECB) stand out as a significant sign of changes on the horizon for international financial waters. The ECB’s decision to recalibrate its quantitative easing (QE) program, along with a series of subtle interest rate adjustments, has caused ripples in financial markets, particularly affecting emerging economies closely linked to the economic health of the eurozone. These changes, designed to stabilize the euro and stimulate economic growth within the European Union, have broader implications, including capital flight, currency volatility, and increased investor aversion. The future direction, aimed at bringing clarity and predictability, has received mixed reactions, as some market analysts believe it lacks the necessary flexibility to respond to complex and changing economic conditions in different regions. In response, small financial institutions and central banks of emerging economies have had to take a series of adaptive measures, ranging from tightening monetary policy to diversifying portfolios, to protect themselves from external shocks and seize emerging opportunities. The interaction between global and regional economic strategies highlights the complexity and interconnectedness of modern financial systems, where the decisions of a major actor can have reverberating effects on the rest of the world.

Node 2-0-1 (Level 3, NV-Embed-v2 similarity to root 0.9184). In the constantly changing global economic landscape, the recent policy changes of the European Central Bank (ECB) are a significant marker for changes in international financial waters. The ECB’s decision to recalibrate its Quantitative Easing (QE) program, along with a series of subtle adjustments to interest rates, has sent waves through the financial markets, particularly in Emerging Markets that are closely linked to the economic well-being of the Eurozone. These changes, aimed at stabilizing the Euro and promoting economic development within the European Union, have broader implications, including capital flight, currency volatility, and increased risk aversion among investors. The forward guidance strategy, which seeks to provide clarity and predictability, is met with mixed reactions as some market analysts believe it lacks the necessary flexibility to be fair to complex and changing economic conditions in various regions. In response, smaller financial institutions and central banks in Emerging Markets have taken a range of adaptive measures, from abstaining from monetary policy to diversifying portfolios, to protect themselves from external shocks and seize new opportunities. The interaction between global and regional economic strategies highlights the complexity and interdependence of modern financial systems, where the decisions of a major player can have far-reaching effects on the rest of the world.

Node 2-0-2 (Level 3, NV-Embed-v2 similarity to root 0.8992). In the constant scenario of the evolving global economic landscape, the recent political adjustments by the European Central Bank (ECB) stand out as a significant marker of changes in international financial waters. The decision of the ECB to recalibrate its quantitative easing (QE) program, along with a series of subtle adjustments in interest rates, has caused ripples in financial markets, particularly affecting emerging countries that are closely linked to the economic health of the eurozone. These changes, designed to stabilize the euro and stimulate economic growth within the European Union, have broader implications, including capital flight, exchange rate volatility, and increased risk aversion among investors. The forward-looking orientation, aimed at providing clarity and predictability, has received mixed reactions, as some market analysts believe it lacks the necessary flexibility to address changing and complex economic conditions in different regions. As a response, smaller financial institutions and central banks of emerging countries have had to take a series of adaptive measures, ranging from tightening monetary policy to diversifying portfolios, to protect themselves from external shocks and seize emerging opportunities. The interaction between global and regional economic strategies highlights the complexity and interconnectedness of modern financial systems, where the decisions of a major player can have far-reaching effects on the rest of the world.

Node 2-1 (Level 2, NV-Embed-v2 similarity to root 0.9394). In the constantly evolving landscape of global economic governance, the most recent political adjustment of the European Central Bank (ECB) stands as an important landmark for changes in the international financial waters. The decision of the ECB to recalibrate its Quantitative Easing (QE) program, as well as a series of slight interest rate adjustments, have unleashed shockwaves in the financial markets, particularly in emerging countries with close ties to the economic health of the Eurozone. These changes, aimed at stabilizing the Euro and strengthening economic growth within the European Union, have far-reaching implications, including capital flight, currency volatility, and an increase in risk aversion among investors. The ECB’s forecasts, which are intended to provide clarity and predictability, have received mixed reactions, as some market analysts argue that they do not offer enough flexibility to account for the diverse and dynamic economic conditions in various regions. In response, smaller financial institutions and central banks in emerging countries have taken a range of adaptive measures, from lowering monetary policies to diversifying their investment portfolios, to insulate themselves from external shocks and take advantage of new opportunities. The interaction between these global and local economic strategies underscores the complex and intertwined nature of the modern financial system, where the decisions of one major player can have significant implications for the rest of the world.
Node 2-1-0 (Level 3, NV-Embed-v2 similarity to root 0.9184). In the constantly evolving global economic landscape, the recent adjustment of the monetary policy of the European Central Bank (ECB) serves as an important landmark for changes in international financial waters. The decision of the ECB to recalibrate its Quantitative Easing (QE) program, along with a series of minor adjustments to interest rates, has caused ripples in financial markets, particularly among emerging economies closely linked to the economic health of the eurozone. These changes, aimed at stabilizing the euro and strengthening economic growth in the European Union, have wide-ranging implications, including capital flight, currency volatility, and an increased risk appetite among investors. The ECB’s forecasts, which aim to bring clarity and predictability, have received mixed reactions, as some market analysts believe they do not provide enough flexibility to account for diverse and dynamic economic conditions in different regions. In response, financial institutions and central banks of emerging economies have adopted a range of adaptive measures, from easing monetary policies to diversifying investment portfolios, to protect against external shocks and take advantage of new opportunities. The interaction between these global and local economic strategies highlights the complex and interconnected nature of the modern financial system, where the decisions of a major player can have significant implications for the rest of the world.

Node 2-1-1 (Level 3, NV-Embed-v2 similarity to root 0.8603). In the constantly changing landscape of global economic and political governance, the latest political change of the European Central Bank (ECB) represents an important milestone for changes in international financial waters. The decision of the ECB to recalibrate its Quantitative Easing (QE) program, as well as a series of minor interest rate adjustments, has triggered waves of shockwaves in the financial market, particularly in development countries that rely on the economic health of the Eurozone. These changes, aimed at stabilizing the Euro and strengthening economic growth in the European Union, have far-reaching implications, including capital flight, currency volatility, and an increase in risk aversion among investors. The ECB’s forecasts, which aim to increase clarity and predictability, have received mixed reactions, as some market analysts argue that they are not sufficiently flexible to accommodate the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks in developing countries have taken a series of adaptable measures, ranging from reducing monetary policy to diversifying their investment portfolios, to insulate themselves from external shocks and seize new opportunities. The interaction between these global and local economic strategies underscores the complex and interwoven nature of the modern financial system, where the decisions of one major player can have significant impacts on the rest of the world.

Node 2-1-2 (Level 3, NV-Embed-v2 similarity to root 0.9302). In the constant scenario of the evolution of global economic governance, the latest political adjustment by the European Central Bank (ECB) has risen to become an important milestone for changes in international financial waters. The ECB’s decision to recalibrate its Quantitative Easing (QE) program, along with a series of minor adjustments to interest rates, has triggered shockwaves in financial markets, especially in emerging economies with close ties to the economic health of the Eurozone. These changes, aimed at stabilizing the Euro and strengthening economic growth within the European Union, have long-term implications, including capital flight, currency volatility, and an increase in risk aversion among investors. The ECB’s projections, which seek to provide clarity and predictability, have received mixed reactions, as some market analysts argue that they do not offer sufficient flexibility to address the diverse and dynamic economic conditions in various regions. In response, financial institutions and central banks of emerging economies have taken a series of adaptive measures, from lowering their monetary policies to diversifying their investment portfolios, to protect themselves from external shocks and seize new opportunities. The interaction between these global and local economic strategies highlights the complex and interconnected nature of the modern financial system, where decisions by a single dominant player can have significant implications for the rest of the world.

Node 2-2 (Level 2, NV-Embed-v2 similarity to root 0.9398). In the evolving landscape of global economic governance, the recent political adjustments by the European Central Bank (ECB) stand out as a significant marker of changes in international financial waters. The ECB’s decision to recalibrate its quantitative easing (QE) program, along with a series of subtle adjustments in interest rates, has sent shockwaves through financial markets, particularly affecting emerging countries closely linked to the economic health of the European Economic Area. These changes, designed to stabilize the euro and stimulate economic growth within the European Union, have broader implications, including capital flight, currency volatility, and an increase in risk aversion among investors. The future guidance provided by the ECB, intended to offer clarity and predictability, has received mixed reactions, as some market analysts argue that it lacks sufficient flexibility to address the dynamic and diverse economic conditions in different regions. In response, smaller financial institutions and central banks of emerging countries have been forced to adopt a range of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, with the aim of insulating themselves from external shocks and capitalizing on emerging opportunities. The interaction between these global and local economic strategies highlights the complex and interconnected nature of the modern financial system, where the decisions of a single major player can have profound implications for the rest of the world.

Node 2-2-0 (Level 3, NV-Embed-v2 similarity to root 0.9372). In the constantly evolving landscape of global economic governance, the recent political adjustments by the European Central Bank (ECB) stand out as a significant marker of changes in international financial
waters. The ECB’s decision to recalibrate its quantitative easing (QE) program, along with a series of subtle adjustments to interest rates, sent shockwaves through financial markets, particularly affecting emerging economies closely tied to the economic health of the European Economic Area. These changes, designed to stabilize the euro and stimulate economic growth in the European Union, have broader implications, including currency volatility and an increase in risk aversion among investors. The future guidance provided by the ECB, aimed at offering clarity and predictability, has received mixed reactions, as some financial analysts believe it lacks flexibility to respond to dynamic and diverse economic conditions in different regions. In response, smaller financial institutions and central banks of emerging economies have been forced to adopt a range of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, with the goal of insulating these countries from external shocks and capitalizing on emerging opportunities. The interaction between these global and local economic strategies highlights the complex and interconnected nature of the modern financial system, where the decisions of a single major actor can have profound implications for the rest of the world.

Node 2-2-1 (Level 3, NV-Embed-v2 similarity to root 0.9168). In the constantly evolving scenario of global economic and political governance, the recent political adjustment by the European Central Bank (ECB) represents a significant feature of changes in the international financial waters. The ECB’s decision to recalibrate its Quantitative Easing (QE) program, as well as a series of subtle adjustments to interest rates, have triggered shockwaves in the financial markets, particularly in emerging countries closely linked to the economic health of the European Economic and Monetary Union (EMU). These changes, aimed at stabilizing the euro and promoting economic development in the European Union, have broader impacts, including capital flight, currency instability, and increased risk aversion among investors. The future guidance provided to the ECB to offer clarity and predictability has elicited mixed reactions, as some market analysts argue that it does not offer sufficient flexibility to account for the dynamic and diverse economic conditions in various regions. In response, smaller financial institutions and central banks in emerging countries have taken a range of adaptive measures, from the abstinence of monetary policy to the diversification of their investor portfolios, with the goal of insulating themselves from external shocks and capitalizing on emerging opportunities. The interaction between these global and local economic strategies highlights the complex and interconnected nature of the modern financial system, where decisions of a single major player can have far-reaching impacts on the rest of the world.

Node 2-2-2 (Level 3, NV-Embed-v2 similarity to root 0.9388). In the evolutionary landscape of global economic governance, the recent political adjustments made by the European Central Bank (ECB) stand out as a significant marker of changes in international financial waters. The ECB's decision to recalibrate its quantitative easing (QE) program, along with a series of subtle adjustments in interest rates, has sent shockwaves through financial markets, particularly affecting emerging countries that are closely linked to the economic health of the Euro Area. These changes, designed to stabilize the euro and stimulate economic growth within the European Union, have broader implications, including capital flight, currency volatility, and increased risk aversion among investors. The guidance provided by the ECB, intended to offer clarity and predictability, has received mixed reactions, as some market analysts argue that it lacks sufficient flexibility to address the dynamic and diverse economic conditions in different regions. In response, smaller financial institutions and central banks of emerging countries have been forced to adopt a series of adaptive measures, ranging from tightening monetary policies to diversifying their investment portfolios, with the goal of insulating themselves from external shocks and capitalizing on emerging opportunities. The interaction between these global and local economic strategies illustrates the complex and interconnected nature of the modern financial system, where the decisions of a single major player can have profound implications for the rest of the world.

This is the full content of the self-consistency tree generated by Qwen-2.5-72B on machine translation tasks.

Root (Level 0, NV-Embed-v2 similarity to root 1.0000). In the ever-evolving landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) stand out as a significant marker of the shifting tides in international finance. The ECB’s decision to recalibrate its quantitative easing (QE) program, alongside a series of nuanced interest rate adjustments, has sent shockwaves through the financial markets, particularly affecting emerging economies that are intricately linked to the Eurozone’s economic health. These changes, while designed to stabilize the Euro and stimulate economic growth within the European Union, have had far-reaching consequences, including capital flight, currency volatility, and heightened risk aversion among investors. The ECB’s forward guidance, which aims to provide clarity and predictability, has been met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to address the diverse and dynamic economic conditions across different regions. In response, smaller financial institutions and central banks in emerging markets
have been compelled to adopt a range of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, in an effort to insulate their economies from external shocks and capitalize on the emerging opportunities. The interplay between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where the decisions of one major player can have profound implications for the rest of the world.

Node 0 (Level 1, NV-Embed-v2 similarity to root 0.9745). In the constantly evolving landscape of global economic governance, recent policy adjustments by the European Central Bank (ECB) stand out as a significant marker of trends in international finance. The ECB’s decision to recalibrate its quantitative easing (QE) program, coupled with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly affecting emerging economies that are closely tied to the economic health of the eurozone. These changes, although designed to stabilize the euro and stimulate economic growth within the European Union, have had long-term consequences, including capital flight, currency volatility, and increased risk aversion among investors. The ECB’s forward guidance, aimed at providing clarity and predictability, has been met with mixed reactions, some market analysts arguing that it lacks the necessary flexibility to respond to the diverse and dynamic economic conditions across different regions. In response, smaller financial institutions and central banks in emerging markets have been forced to adopt a range of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, in an effort to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have profound implications for the rest of the world.

Node 0-0 (Level 2, NV-Embed-v2 similarity to root 0.9672). In the constantly evolving landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) stand out as a significant marker of trends in international finance. The ECB’s decision to recalibrate its quantitative easing (QE) program, coupled with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly impacting emerging economies closely tied to the economic health of the eurozone. Although these changes are designed to stabilize the euro and boost economic growth within the European Union, they have had long-term consequences, including capital flight, currency volatility, and increased risk aversion among investors. Future guidance from the ECB, aimed at providing clarity and predictability, has been met with mixed reactions, some market analysts arguing that it lacks the necessary flexibility to respond to the diverse and dynamic economic conditions across different regions. In response, smaller financial institutions and central banks in emerging markets have been forced to adopt a range of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, in an effort to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have profound implications for the rest of the world.

Node 0-0-0 (Level 3, NV-Embed-v2 similarity to root 0.9672). In the constantly evolving landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) stand out as a significant marker of trends in international finance. The ECB’s decision to recalibrate its quantitative easing (QE) program, coupled with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly affecting emerging economies closely tied to the economic health of the eurozone. Although these changes are designed to stabilize the euro and stimulate economic growth within the European Union, they have had long-term consequences, including capital flight, currency volatility, and increased risk aversion among investors. The ECB’s future guidance, aimed at providing clarity and predictability, has been met with mixed reactions, some market analysts arguing that it lacks the necessary flexibility to respond to the diverse and dynamic economic conditions across different regions. In response, smaller financial institutions and central banks in emerging markets have been forced to adopt a range of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, in an effort to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have profound implications for the rest of the world.

Node 0-0-1 (Level 3, NV-Embed-v2 similarity to root 0.9424). In the constantly changing landscape of global economic governance, the recent political adjustments by the European Central Bank (ECB) are a significant indicator of trends in the international financial world. The ECB’s decision to recalibrate its Quantitative Easing (QE) program, coupled with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly to emerging economies that are closely tied to the economic health of the Eurozone. While these changes are designed to stabilize the Euro and promote economic growth within the European Union, they have had long-term consequences, including capital flight, currency fluctuations, and increased risk aversion among investors. The future guideline of the ECB, which aims to create clarity and predictability, has been met with mixed reactions, with some market analysts arguing that it lacks the necessary flexibility to respond to the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks in emerging markets have been forced to adopt a range of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to
protect their economies from external shocks and capitalize on new opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a key player can have far-reaching impacts on the rest of the world.

Node 0-0-2 (Level 3, NV-Embed-v2 similarity to root 0.9677). In the constantly evolving landscape of global economic governance, recent policy adjustments by the European Central Bank (ECB) stand out as a significant marker of trends in international finance. The ECB's decision to recalibrate its quantitative easing (QE) program, along with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly impacting emerging economies closely tied to the economic health of the eurozone. While these changes are designed to stabilize the euro and stimulate economic growth within the European Union, they have had long-term consequences, including capital flight, exchange rate volatility, and an increase in risk aversion among investors. The ECB's future guidance, aimed at providing clarity and predictability, has been met with mixed reactions, some market analysts arguing that it lacks the necessary flexibility to respond to the diverse and dynamic economic conditions in different regions. In response, financial institutions and smaller central banks in emerging markets have been forced to adopt a series of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, in an effort to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have profound implications for the rest of the world.

Node 0-1 (Level 2, NV-Embed-v2 similarity to root 0.9274). In the constantly changing landscape of global economic governance, the recent political adjustments by the European Central Bank (ECB) are a significant indicator of trends in the international financial world. The ECB's decision to recalibrate its Quantitative Easing (QE) program, coupled with a series of subtle interest rate changes, has sent shockwaves through financial markets, particularly to emerging countries closely tied to the economic health of the Eurozone. While these changes aim to stabilize the Euro and promote economic growth within the European Union, they have long-term consequences, including capital flight, currency volatility, and increased risk aversion among investors. The future leadership of the ECB, which is intended to provide clarity and predictability, has been met with mixed reactions, with some market analysts arguing that it lacks the necessary flexibility to respond to the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks in emerging markets have been forced to adopt a range of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have far-reaching effects on the rest of the world.

Node 0-1-0 (Level 3, NV-Embed-v2 similarity to root 0.9488). In the constantly evolving landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) are a significant indicator of trends in the international financial world. The ECB's decision to adjust its Quantitative Easing (QE) program, along with a series of small interest rate changes, has sent shockwaves through financial markets, particularly to emerging countries closely tied to the economic health of the eurozone. While these changes aim to stabilize the euro and promote economic growth within the European Union, they have long-term consequences, including capital outflows, exchange rate fluctuations, and increased risk aversion among investors. The future direction of the ECB, which is supposed to provide clarity and predictability, has been met with mixed reactions, some market analysts arguing that it lacks the necessary flexibility to respond to the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks in emerging markets have been forced to adopt a range of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have long-term effects on the rest of the world.

Node 0-1-1 (Level 3, NV-Embed-v2 similarity to root 0.9263). In the constantly changing landscape of global economic governance, the recent political adjustments by the European Central Bank (ECB) are a significant indicator of trends in the international financial world. The ECB's decision to recalibrate its Quantitative Easing (QE) program, coupled with a series of subtle interest rate changes, has sent shockwaves through financial markets, particularly in emerging countries that are closely tied to the economic health of the Eurozone. While these changes aim to stabilize the euro and promote economic growth within the European Union, they have long-term consequences, including capital flight, currency volatility, and increased risk aversion among investors. The future leadership of the ECB, which is expected to provide clarity and predictability, has been met with mixed reactions, with some market analysts arguing that it lacks the necessary flexibility to respond to the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks in emerging markets have been forced to implement a range of adjustment measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player
can have far-reaching impacts on the rest of the world.

Node 0-1-2 (Level 3, NV-Embed-v2 similarity to root 0.9382). In the constantly changing landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) are a significant indicator of trends in international financial markets. The ECB’s decision to recalibrate its Quantitative Easing (QE) program, along with a series of subtle changes in interest rates, has sent shockwaves through financial markets, particularly to emerging countries closely tied to the economic health of the Eurozone. While these changes aim to stabilize the Euro and promote economic growth within the European Union, they have long-term consequences, including capital flight, currency volatility, and an increase in risk aversion among investors. The future leadership of the ECB, which is intended to provide clarity and predictability, has been met with mixed reactions, with some market analysts arguing that it lacks the necessary flexibility to respond to diverse and dynamic economic conditions in different regions. In response, financial institutions and smaller central banks in emerging markets have been forced to adopt a range of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have far-reaching effects on the rest of the world.

Node 0-2 (Level 2, NV-Embed-v2 similarity to root 0.9748). In the constantly evolving landscape of global economic governance, recent policy adjustments by the European Central Bank (ECB) stand out as a significant marker of trends in international finance. The ECB’s decision to recalibrate its quantitative easing (QE) program, along with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly affecting emerging economies that are closely tied to the economic health of the eurozone. These changes, although designed to stabilize the euro and stimulate economic growth within the European Union, have had long-term consequences, including capital flight, currency volatility, and an increase in risk aversion among investors. The ECB’s forward guidance, intended to provide clarity and predictability, has been met with mixed reactions, some market analysts arguing that it lacks the necessary flexibility to respond to the diverse and dynamic economic conditions in different regions. In response, financial institutions and smaller central banks in emerging markets have been forced to adopt a variety of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, in an effort to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have profound implications for the rest of the world.

Node 0-2-0 (Level 3, NV-Embed-v2 similarity to root 0.9653). In the constantly evolving landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) stand out as a significant marker of trends in international finance. The ECB’s decision to recalibrate its quantitative easing (QE) program, coupled with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly affecting emerging economies closely tied to the economic health of the eurozone. These changes, although designed to stabilize the euro and boost economic growth within the European Union, have had long-term consequences, including capital outflows, exchange rate volatility, and increased risk aversion among investors. The ECB’s future guidance, aimed at providing clarity and predictability, has been met with mixed reactions, some market analysts arguing that it lacks the necessary flexibility to respond to the diverse and dynamic economic conditions in different regions. In response, financial institutions and central banks of emerging markets have been forced to adopt a variety of adaptive measures, ranging from tightening monetary policies to diversifying their investment portfolios, in an effort to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have profound implications for the rest of the world.

Node 0-2-1 (Level 3, NV-Embed-v2 similarity to root 0.9697). In the constantly changing landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) stand out as a significant indicator of trends in international finance. The ECB’s decision to recalibrate its quantitative easing (QE) program, along with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly in emerging countries that are closely tied to the economic health of the Eurozone. These changes, although designed to stabilize the Euro and promote economic growth within the European Union, had long-term consequences, including capital flight, currency fluctuations, and an increase in investor risk aversion. The ECB’s forward guidance, intended to provide clarity and predictability, has been met with mixed reactions. Some market analysts argue that it lacks the necessary flexibility to respond to the diverse and dynamic economic conditions in different regions. In response, financial institutions and smaller central banks in emerging markets have been forced to adopt a range of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on new opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have far-reaching effects on the rest of the world.

Node 0-2-2 (Level 3, NV-Embed-v2 similarity
to root 0.9670). In the constantly evolving landscape of global economic governance, recent policy adjustments by the European Central Bank (ECB) stand out as a significant marker of trends in international finance. The ECB's decision to recalibrate its quantitative easing (QE) program, along with a series of subtle interest rate adjustments, has sent ripples through financial markets, particularly affecting emerging economies that are closely tied to the economic health of the eurozone. These changes, although designed to stabilize the euro and stimulate economic growth within the European Union, have had long-term consequences, including capital flight, currency volatility, and an increase in risk aversion among investors. The future guidance of the ECB, aimed at providing clarity and predictability, has been met with mixed reactions, some market analysts arguing that it lacks the necessary flexibility to respond to diverse and dynamic economic conditions in different regions. In response, financial institutions and smaller central banks in emerging markets have been forced to adopt a variety of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, in an effort to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have profound implications for the rest of the world.

Node 1 (Level 1, NV-Embed-v2 similarity to root 0.9453). In the constantly changing landscape of global economic governance, the recent political adjustments by the European Central Bank (ECB) are a significant sign of the shifting currents in international finance. The ECB's decision to adjust its Quantitative Easing (QE) program, along with a series of differentiated interest rate adjustments, has sent shockwaves through the financial markets, particularly affecting emerging economies that are closely tied to the economic health of the Eurozone. These changes, aimed at stabilizing the Euro and boosting economic growth in the European Union, have had far-reaching consequences, including capital flight, currency fluctuations, and increased risk aversion among investors. The forward-looking communication of the ECB, intended to provide clarity and predictability, has been met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to accommodate the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks in developing countries have been forced to implement a range of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to shield their economies from external shocks and capitalize on available opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have far-reaching impacts on the rest of the world.

Node 1-0 (Level 2, NV-Embed-v2 similarity to root 0.9745). In the constantly evolving landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) are a significant sign of changing tides in international finance. The ECB's decision to adjust its Quantitative Easing (QE) program, along with a series of differentiated interest rate adjustments, has sent shockwaves through financial markets, particularly affecting emerging economies that are closely tied to the economic health of the eurozone. These changes, aimed at stabilizing the euro and stimulating economic growth within the European Union, have had significant consequences, including capital outflows, exchange rate fluctuations, and increased risk aversion among investors. The forward guidance from the ECB, intended to provide clarity and predictability, has been met with mixed reactions, some market analysts arguing that it lacks the necessary flexibility to adapt to diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks of developing countries have been forced to implement a range of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on available opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have significant impacts on the rest of the world.

Node 1-0-0 (Level 3, NV-Embed-v2 similarity to root 0.9586). In the constantly evolving landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) are a significant sign of changes in the currents of international finance. The ECB's decision to adjust its Quantitative Easing (QE) program, along with a series of differentiated interest rate adjustments, has sent shockwaves through financial markets, particularly affecting emerging economies closely tied to the economic health of the eurozone. These changes, aimed at stabilizing the euro and stimulating economic growth within the European Union, have had significant consequences, including capital outflows, exchange rate fluctuations, and increased risk aversion among investors. The future guidance of the ECB, intended to bring clarity and predictability, has been met with mixed reactions, some market analysts arguing that they lack the necessary flexibility to adapt to the diverse and dynamic economic conditions in different regions. In response, small financial institutions and central banks of developing countries have been forced to implement a range of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on available opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have significant impacts on the rest of the world.

Node 1-0-1 (Level 3, NV-Embed-v2 similarity to root 0.9468). In the constantly changing
environment of global economic governance, the recent political adjustments by the European Central Bank (ECB) are a significant sign of changes in the international financial world. The ECB’s decision to adjust its Quantitative Easing (QE) program, along with a series of differentiated interest rate adjustments, has sent shockwaves through the financial markets, especially the emerging economies that are closely tied to the economic health of the Eurozone. These changes, aimed at stabilizing the euro and boosting economic growth within the European Union, have had substantial consequences, including capital outflows, exchange rate fluctuations, and increased risk aversion among investors. The forward-looking guidance from the ECB, intended to provide clarity and predictability, has been met with mixed reactions, with some market analysts arguing that it lacks the necessary flexibility to adapt to the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks of developing countries have been forced to implement a range of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on existing opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a key player can have significant impacts on the rest of the world.

Node 1-0-2 (Level 3, NV-Embed-v2 similarity to root 0.9678). In the constantly evolving landscape of global economic governance, recent policy adjustments by the European Central Bank (ECB) are a significant sign of changes in the tides of international finance. The ECB’s decision to adjust its Quantitative Easing (QE) program, along with a series of differentiated interest rate adjustments, has sent shock waves through financial markets, particularly affecting emerging economies that are closely tied to the economic health of the eurozone. These changes, aimed at stabilizing the euro and stimulating economic growth within the European Union, have had significant consequences, including capital flows, exchange rate fluctuations, and an increase in risk aversion among investors. The ECB’s forward guidance, intended to provide clarity and predictability, has been met with mixed reactions, with some market analysts arguing that it lacks the necessary flexibility to adapt to diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks of developing countries have been forced to implement a range of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and take advantage of available opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have significant impacts on the rest of the world.

Node 1-1 (Level 2, NV-Embed-v2 similarity to root 0.9418). In the constantly changing landscape of global economic governance, the recent political adjustments by the European Central Bank (ECB) are a significant sign of the shifting tides in the international financial world. The ECB’s decision to adjust its Quantitative Easing (QE) program, along with a series of different interest rate adjustments, has sent shockwaves through the financial markets, particularly the emerging economies that are closely tied to the economic stability of the Eurozone. These changes, aimed at stabilizing the euro and promoting economic growth in the European Union, have had far-reaching consequences, including capital flight, currency fluctuations, and increased risk aversion among investors. The forward-looking communication of the ECB, which aims to provide clarity and predictability, has been met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to account for the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks in developing countries have been forced to implement a range of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on available opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a key player can have far-reaching impacts on the rest of the world.

Node 1-1-0 (Level 3, NV-Embed-v2 similarity to root 0.9603). In the constantly evolving landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) are a significant sign of trends changing in the international financial world. The ECB’s decision to adjust its Quantitative Easing (QE) program, along with a series of different interest rate adjustments, has sent shockwaves through financial markets, particularly emerging economies that are closely tied to the economic stability of the eurozone. These changes, aimed at stabilizing the euro and promoting economic growth within the European Union, have had significant consequences, including capital outflows, exchange rate fluctuations, and increased risk aversion among investors. The forward guidance from the ECB, which aims to provide clarity and predictability, has received mixed reactions, with some market analysts arguing that it lacks the necessary flexibility to account for the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks of developing countries have been forced to implement a range of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and take advantage of available opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a key player can have significant impacts on the rest of the world.

Node 1-1-1 (Level 3, NV-Embed-v2 similarity to root 0.9420). In the constantly changing landscape of global economic governance, the recent political adjustments by the European Central Bank
(ECB) are a significant sign of the shifting currents in the international financial world. The ECB’s decision to adjust its Quantitative Easing (QE) program, along with a series of interest rate changes, has sent shockwaves through financial markets, particularly through emerging markets that are closely tied to the economic stability of the Eurozone. These changes, aimed at stabilizing the euro and promoting economic growth in the European Union, have had far-reaching consequences, including capital flight, currency fluctuations, and increased risk aversion among investors. The forward-looking communication of the ECB, which aims to provide clarity and predictability, has been met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to accommodate the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks in developing countries have been forced to implement a range of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on available opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a key player can have far-reaching impacts on the rest of the world.

Node 1-1-2 (Level 3, NV-Embed-v2 similarity to root 0.9449). In the constantly changing landscape of global economic governance, recent political adjustments by the European Central Bank (ECB) are a significant sign of changes in the international financial world. The ECB’s decision to adjust its Quantitative Easing (QE) program, along with a series of different interest rate adjustments, has sent shockwaves through financial markets, particularly to emerging economies that are closely tied to the economic stability of the Eurozone. These changes, aimed at stabilizing the euro and promoting economic growth in the European Union, have had far-reaching consequences, including capital flight, currency fluctuations, and an increase in risk aversion among investors. The forward guidance of the ECB, which aims to provide clarity and predictability, has been met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to account for the diverse and dynamic economic conditions in different regions. In response, financial institutions and smaller central banks in developing countries have been forced to implement a range of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and take advantage of available opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a key player can have widespread impacts on the rest of the world.

Node 1-2 (Level 2, NV-Embed-v2 similarity to root 0.9690). In the constantly changing landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) are a significant sign of the shifting tides in international finance. The ECB’s decision to adjust its Quantitative Easing (QE) program, along with a series of differentiated interest rate adjustments, has sent shockwaves through financial markets, particularly affecting emerging economies that are closely tied to the economic health of the Eurozone. These changes, aimed at stabilizing the euro and promoting economic growth in the European Union, have had far-reaching consequences, including capital flight, currency fluctuations, and an increase in risk aversion among investors. The forward-looking communication of the ECB, intended to provide clarity and predictability, has been met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to adapt to the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks of developing countries have been forced to implement a variety of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on available opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have far-reaching impacts on the rest of the world.

Node 1-2-0 (Level 3, NV-Embed-v2 similarity to root 0.9738). In the constantly evolving landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) are a significant sign of changing tides in international finance. The ECB’s decision to alter its Quantitative Easing (QE) program, coupled with a series of differentiated interest rate adjustments, has sent shockwaves through financial markets, particularly affecting emerging economies closely tied to the economic health of the eurozone. These changes, aimed at stabilizing the euro and promoting economic growth in the European Union, have had significant consequences, including capital outflows, currency fluctuations, and increased risk aversion among investors. The ECB’s forward guidance, intended to bring clarity and predictability, has been met with mixed reactions, some market analysts arguing that it lacks the necessary flexibility to adapt to the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks in developing countries have been forced to implement a variety of adaptive measures, ranging from monetary policy tightening to diversification of their investment portfolios, to protect their economies from external shocks and capitalize on available opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have significant impacts on the rest of the world.

Node 1-2-1 (Level 3, NV-Embed-v2 similarity to root 0.9372). In the constantly changing structure of global economic governance, the recent political adjustments by the European Central Bank (ECB) are a clear sign of the shifting
tides in the international financial world. The ECB’s decision to adjust its Quantitative Easing (QE) program, along with a series of differentiated interest rate adjustments, has sent shockwaves through financial markets, particularly to emerging countries that are closely linked to the economic health of the Eurozone. These changes, aimed at stabilizing the Euro and promoting economic growth in the European Union, have had far-reaching consequences, including capital flight, currency fluctuations, and increased risk aversion among investors. The forward-looking communication of the ECB, intended to provide clarity and predictability, has been met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to adapt to the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and the central banks of developing countries have been forced to implement a variety of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and to benefit from available opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have far-reaching impacts on the rest of the world.

Node 1-2-2 (Level 3, NV-Embed-v2 similarity to root 0.9695). In the constantly changing landscape of global economic governance, recent policy adjustments by the European Central Bank (ECB) are a significant sign of the shifting tides in international finance. The ECB’s decision to adjust its Quantitative Easing (QE) program, along with a series of differential interest rate adjustments, has sent shockwaves through financial markets, particularly affecting emerging economies that are closely tied to the economic health of the Eurozone. These changes, aimed at stabilizing the Euro and promoting economic growth in the European Union, have had far-reaching consequences, including capital flight, monetary fluctuations, and an increase in risk aversion among investors. The forward-looking communication from the ECB, intended to provide clarity and predictability, has been met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to adapt to diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks of developing countries have been forced to implement a variety of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on available opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have far-reaching impacts on the rest of the world.

Node 2-0 (Level 2, NV-Embed-v2 similarity to root 0.9709). In the constantly evolving landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) stand out as a significant marker of changes in the currents of international finance. The ECB’s decision to recalibrate its quantitative easing (QE) program, along with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly affecting emerging economies that are intricately linked to the economic health of the Eurozone. These changes, though designed to stabilize the Euro and stimulate economic growth within the European Union, have had far-reaching consequences, including capital flight, currency volatility, and increased risk aversion among investors. The future orientation of the ECB, which seeks to provide clarity and predictability, has been met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to address the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks of emerging markets have been forced to adopt a series of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, in order to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have profound implications for the rest of the world.

Node 2-0-0 (Level 3, NV-Embed-v2 similarity to root 0.9686). In the constantly evolving landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) stand out as a significant marker of changes in the currents of international finance. The ECB’s
decision to recalibrate its quantitative easing (QE) program, coupled with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly affecting emerging economies that are closely tied to the economic health of the eurozone. These changes, although designed to stabilize the euro and boost economic growth within the European Union, have had broader consequences, including capital flight, exchange rate volatility, and increased risk aversion among investors. The future direction of the ECB, aimed at providing clarity and predictability, has been met with mixed reactions, some market analysts arguing that it lacks the necessary flexibility to respond to the diverse and dynamic economic conditions in different regions. In response, small financial institutions and central banks in emerging markets have been forced to adopt a series of adaptation measures, ranging from easing monetary policies to diversifying their investment portfolios, in order to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have profound implications for the rest of the world.

Node 2-0-1 (Level 3, NV-Embed-v2 similarity to root 0.9361). In the constantly changing landscape of global economic governance, the recent political adjustments by the European Central Bank (ECB) are a significant sign of changes in the currents of international financial markets. The ECB’s decision to realign its Quantitative Easing (QE) program, coupled with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly in emerging countries that are closely tied to the economic health of the Eurozone. These changes, although designed to stabilize the Euro and promote economic growth within the European Union, have had far-reaching consequences, including capital flight, exchange rate volatility, and increased risk aversion among investors. The future direction of the ECB, which aims for clarity and predictability, has been met with mixed reactions. Some market analysts argue that it lacks the necessary flexibility to respond to the diverse and dynamic economic conditions in different regions. In response, small financial institutions and central banks in emerging markets have been forced to adopt a range of adaptive measures, from loosening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on new opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have far-reaching implications for the rest of the world.

Node 2-0-2 (Level 3, NV-Embed-v2 similarity to root 0.9701). In the constantly evolving landscape of global economic governance, recent policy adjustments by the European Central Bank (ECB) stand out as a significant marker of changes in the currents of international finance. The ECB’s decision to recalibrate its quantitative easing (QE) program, along with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly affecting emerging economies that are closely tied to the economic health of the eurozone. These changes, although designed to stabilize the euro and stimulate economic growth within the European Union, have had far-reaching consequences, including capital flight, exchange rate volatility, and an increase in risk aversion among investors. The future direction of the ECB, aimed at providing clarity and predictability, has been met with mixed reactions, some market analysts arguing that it lacks the necessary flexibility to respond to the various and dynamic economic conditions in different regions. In response, small financial institutions and central banks in emerging markets have been forced to adopt a series of adaptive measures, ranging from easing monetary policies to diversifying their investment portfolios, in order to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have profound implications for the rest of the world.

Node 2-1 (Level 2, NV-Embed-v2 similarity to root 0.9395). In the constantly changing landscape of global economic governance, the recent political adjustments by the European Central Bank (ECB) stand out as a significant sign of changes in the currents of international finance. The ECB’s decision to adjust its Quantitative Easing (QE) program, along with a series of subtle interest rate adjustments, has sent shockwaves through the financial markets, particularly the emerging economies that are closely tied to the economic health of the Eurozone. These changes, although designed to stabilize the Euro and promote economic growth within the European Union, have had far-reaching consequences, including capital flight, currency volatility, and increased risk aversion among investors. The future orientation of the ECB, which aims to provide clarity and predictability, has been met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to account for the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks of emerging countries have been forced to adopt a series of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have far-reaching implications for the rest of the world.

Node 2-1-0 (Level 3, NV-Embed-v2 similarity to root 0.9659). In the constantly evolving landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) stand out as a significant sign of changes in the currents of international finance. The ECB’s decision to adjust its Quantitative Easing (QE) program, along with a series of subtle interest rate adjustments, has sent shockwaves
through financial markets, particularly emerging economies closely tied to the economic health of the eurozone. These changes, although designed to stabilize the euro and promote economic growth within the European Union, have had far-reaching consequences, including capital flight, currency volatility, and increased risk aversion among investors. The future direction of the ECB, which aims to provide clarity and predictability, has been met with mixed reactions, some market analysts arguing that it lacks the necessary flexibility to account for diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks of emerging countries have been forced to adopt a series of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where the decisions of a major player can have far-reaching implications for the rest of the world.

Node 2-1-1 (Level 3, NV-Embed-v2 similarity to root 0.9380). In the constantly changing landscape of global economic governance, the recent political adjustments by the European Central Bank (ECB) are a significant sign of changes in the currents of international finance. The ECB’s decision to adjust its quantitative easing (QE) program, along with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly through the emerging economies that are closely tied to the economic health of the Eurozone. These changes, although aimed at stabilizing the euro and promoting economic growth within the European Union, have had far-reaching consequences, including capital flight, currency fluctuations, and increased risk aversion among investors. The future orientation of the ECB, which is intended to provide clarity and predictability, has been met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to consider the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks of emerging countries have been forced to adopt a range of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and seize new opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have far-reaching implications for the rest of the world.

Node 2-1-2 (Level 3, NV-Embed-v2 similarity to root 0.9696). In the constantly changing landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) stand out as a significant sign of changes in the currents of international finance. The ECB’s decision to adjust its Quantitative Easing (QE) program, along with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly to emerging economies that are closely linked to the economic health of the Eurozone. These changes, although designed to stabilize the euro and promote economic growth within the European Union, have had far-reaching consequences, including capital flight, currency volatility, and an increase in risk aversion among investors. The future guidance of the ECB, which aims to provide clarity and predictability, has been met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to address the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks of emerging markets have been forced to adopt a series of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, in order to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have profound implications for the rest of the world.

Node 2-2-0 (Level 3, NV-Embed-v2 similarity to root 0.9769). In the constantly evolving landscape of global economic governance, the recent policy adjustments by the European Central Bank (ECB) stand out as a significant marker of changes in the currents of international finance. The ECB’s decision to recalibrate its quantitative easing (QE) program, combined with a series of subtle interest rate adjustments, has sent shock waves through financial markets,
particularly affecting emerging economies that are closely tied to the economic health of the eurozone. These changes, although designed to stabilize the euro and boost economic growth within the European Union, have had significant consequences, including capital flight, currency volatility, and increased risk aversion among investors. The ECB’s future guidance, aimed at providing clarity and predictability, has been met with mixed reactions, some market analysts arguing that it lacks the necessary flexibility to respond to the diverse and dynamic economic conditions in different regions. In response, small financial institutions and central banks in emerging markets have been forced to adopt a series of adaptive measures, ranging from tightening monetary policies to diversifying their investment portfolios, in order to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have profound implications for the rest of the world.

Node 2-2-1 (Level 3, NV-Embed-v2 similarity to root 0.9269). In the constantly changing landscape of global economic governance, the recent political adjustments by the European Central Bank (ECB) stand out as a significant sign of changes in the flows of international finance. The ECB’s decision to recalibrate its Quantitative Easing (QE) program, along with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly affecting emerging economies that are closely tied to the economic health of the Eurozone. These changes, although designed to stabilize the euro and boost economic growth within the European Union, have had far-reaching consequences, including capital flight, currency fluctuations, and increased risk aversion among investors. The future leadership of the ECB, which is supposed to provide clarity and predictability, has been met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to handle the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks of emerging markets have been forced to adopt a range of adaptive measures, from tightening monetary policy to diversifying their investment portfolios, to protect their economies from external shocks and capitalize on new opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a key player can have far-reaching implications for the rest of the world.

Node 2-2-2 (Level 3, NV-Embed-v2 similarity to root 0.9806). In the constantly evolving landscape of global economic governance, recent policy adjustments by the European Central Bank (ECB) stand out as a significant marker of changes in the currents of international finance. The ECB’s decision to recalibrate its quantitative easing (QE) program, along with a series of subtle interest rate adjustments, has sent shockwaves through financial markets, particularly affecting emerging economies that are intricately linked to the economic health of the Eurozone. These changes, although designed to stabilize the Euro and stimulate economic growth within the European Union, have had far-reaching consequences, including capital flight, currency volatility, and an increase in risk aversion among investors. The future guidance of the ECB, which aims to provide clarity and predictability, has been met with mixed reactions, as some market analysts argue that it lacks the necessary flexibility to address the diverse and dynamic economic conditions in different regions. In response, smaller financial institutions and central banks in emerging markets have been forced to adopt a series of adaptive measures, from tightening monetary policies to diversifying their investment portfolios, in order to protect their economies from external shocks and capitalize on emerging opportunities. The interaction between these global and local economic strategies underscores the complex and interconnected nature of the modern financial system, where decisions by a major player can have profound implications for the rest of the world.

Reviewer paper summaries: - A dataset-free method for evaluating LLM's semantic consistency, specifically during multi-step processes in the general domain
- Proposes a tree-based self-consistency checker where each level represents a user interaction
- Evaluation of 4 LLMs, on two tasks (translation and code generation), based on pre-trained embeddings and N-grams metrics
Summary of strengths: - Introduces and formalizes a framework for multistep analysis of the LLM performance based on consistency score that does not require a dataset on the domain
- Uses both BLEU and embedding-based evaluation on the output of the LLMs
- The framework can easily be used for any LLM.
- It is a well-written paper and it is easy to follow both the text and formal definitions.
Summary of weaknesses: - Root nodes and paired operations: Are they validated manually after they are generated by the evaluator LLM?
- Qualitative analysis is very important and it is mentioned in the limitations, especially since AI-assisted code generation does not always produce efficient programs. In addition to that, another way of evaluating the codes is analyzing the complexity to ensure the quality of the code is also consistent. 
- Tables 1 and 2: are they showing the performance of the evaluator models, i.e., are they showing the average of all evaluatee models under the ten roots generated by the evaluator? 
- Is the case study just a task that covers both the code generation and translation in the paper? Is it inspired by a real situation? Based on the design of the study, I didn't gain any additional insight into the performance of the LLMs.
- The evaluations are mainly based on L1 to L3 for a tree. Why higher tree depths are not evaluated? The consistency score for Forests is introduced by never used or analyzed.
- It would be interesting to add a comparison to a benchmark to show if the results are aligned or in contrast with previous findings. Although the authors might argue that it is beyond the scope of the paper.
Comments, suggestions and typos: A suggestion: Figure 1 is not color-blind friendly. You can use software to check how it looks for different colorblindness.
Overall assessment scores: 4