Repository: zetaalphavector/RAGElo (language: python)

Issue:
Solve for over-reliance on CSV files
Currently, RAGElo is over reliant on CSVs (c.f., the examples in the Readme). While good for portability and easy to handle, CSVs should not be used to store the current state of the Evaluators and Queries. 
It's cumbersome, brittle and can be tricky to read and keep track of it. 

Instead, the main objects (Evaluators and Queries) should be able to load user data from some file format (i.e., CSVs to keep backward compatibility) but they should not overwrite the users's input files and keep its outputs and current state in a single, serializable file (JSON for readability and portability).

The current CSV outputs should be used as an export feature that should be manually triggered.


Discussion / hints:

Fix patch:
diff --git a/.gitignore b/.gitignore
index fcf7aa6..6dc91ca 100644
--- a/.gitignore
+++ b/.gitignore
@@ -162,4 +162,6 @@ cython_debug/
 data/*
 .vscode/*
 .python-version
-.idea/*
\ No newline at end of file
+.idea/*
+ragelo/.vscode/settings.json
+uv.lock
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
new file mode 100644
index 0000000..af4d1a2
--- /dev/null
+++ b/.pre-commit-config.yaml
@@ -0,0 +1,10 @@
+repos:
+- repo: https://github.com/astral-sh/ruff-pre-commit
+  # Ruff version.
+  rev: v0.8.1
+  hooks:
+    # Run the linter.
+    - id: ruff
+      args: [ --fix ]
+    # Run the formatter.
+    - id: ruff-format
\ No newline at end of file
diff --git a/README.md b/README.md
index 707f0bf..c7476e2 100644
--- a/README.md
+++ b/README.md
@@ -30,11 +30,13 @@ To use RAGElo as a library, all you need to do is import RAGElo, initialize an `
 from ragelo import get_retrieval_evaluator
 
 evaluator = get_retrieval_evaluator("RDNAM", llm_provider="openai")
-raw_answer, processed_answer = evaluator.evaluate(query="What is the capital of France?", document='Lyon is the second largest city in France.')
-print(processed_answer)
+result = evaluator.evaluate(query="What is the capital of France?", document='Lyon is the second largest city in France.')
+print(result.answer)
+# Output: {'overall': 1}
+print(result.answer["overall"])
 # Output: 1
-print(raw_answer)
-# Output: '"O": 1\n}'
+print(result.raw_answer)
+# Output: '{"overall": 1"}'
 ```
 
 For a more complete example, we can evaluate with a custom prompt, and inject metadata into our evaluation prompt:
@@ -53,13 +55,14 @@ Retrieved document: {d}
 
 The document has a date of {document_date}.
 Today is {today_date}.
-
-WRITE YOUR ANSWER ON A SINGLE LINE AS A JSON OBJECT WITH THE FOLLOWING KEYS:
-- "relevance": 0 if the document is irrelevant, 1 if it is relevant.
-- "recency": 0 if the document is outdated, 1 if it is recent.
-- "truthfulness": 0 if the document is false, 1 if it is true.
-- "reasoning": A short explanation of why you think the document is relevant or irrelevant.
 """
+response_schema = {
+    "relevance": "An integer, either 0 or 1. 0 if the document is irrelevant, 1 if it is relevant.",
+    "recency": "An integer, either 0 or 1. 0 if the document is outdated, 1 if it is recent.",
+    "truthfulness": "An integer, either 0 or 1. 0 if the document is false, 1 if it is true.",
+    "reasoning": "A short explanation of why you think the document is relevant or irrelevant.",
+}
+response_format = "json"
 
 evaluator = get_retrieval_evaluator(
     "custom_prompt", # name of the retrieval evaluator
@@ -67,11 +70,11 @@ evaluator = get_retrieval_evaluator(
     prompt=prompt, # your custom prompt
     query_placeholder="q", # the placeholder for the query in the prompt
     document_placeholder="d", # the placeholder for the document in the prompt
-    scoring_keys_retrieval_evaluator=["relevance", "recency", "truthfulness", "reasoning"], # Which keys to extract from the answer
-    answer_format_retrieval_evaluator="multi_field_json", # The format of the answer. In this case, a JSON object with multiple fields
+    llm_answer_format=response_format, # The format of the answer. Can be either `text`, if you expect plain text to be returned, `JSON` if the answer should be in JSON format, or `structured`, if you provide a Pydantic BaseModel as the response_schema.
+    llm_response_schema=response_schema, # The response schema for the LLM. Required if the llm_answer_format is structured and recommended for JSON.
 )
 
-raw_answer, answer = evaluator.evaluate(
+result = evaluator.evaluate(
     query="What is the capital of Brazil?", # The user query
     document="Rio de Janeiro is the capital of Brazil.", # The retrieved document
     query_metadata={"today_date": "08-04-2024"}, # Some metadata for the query
@@ -91,15 +94,52 @@ Other examples are available as notebooks in the [docs/examples/notebooks folder
 ## 🚀 CLI Quickstart 
 After installing RAGElo as a CLI app, you can run it with the following command:
 ```bash
-ragelo run-all queries.csv documents.csv answers.csv --verbose --data-dir tests/data/
+ragelo run-all queries.csv documents.csv answers.csv --data-dir tests/data/ --experiment-name experiment --output-file experiment.json
+
+🔎 Query ID: 0
+📜 Document ID: 0
+Parsed Answer: Very relevant: The document directly answers the user question by stating that Brasília is the capital of Brazil.
+
+🔎 Query ID: 0
+📜 Document ID: 1
+Parsed Answer: Somewhat relevant: The document mentions a former capital of Brazil but does not provide the current capital.
+
+🔎 Query ID: 1
+📜 Document ID: 2
+Parsed Answer: Very relevant: The document clearly states that Paris is the capital of France, directly answering the user question.
+
+🔎 Query ID: 1
+📜 Document ID: 3
+Parsed Answer: Not relevant: The document does not provide information about the capital of France.
+
+Evaluating Retrieved documents 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4/4  [ 0:00:02 < 0:00:00 , 2 it/s ]
+✅ Done!
+Total evaluations: 4
+🔎 Query ID: 0
+ agent1              🆚   agent2
+Parsed Answer: A
+
+🔎 Query ID: 1
+ agent1              🆚   agent2
+Parsed Answer: A
+
+Evaluating Agent Answers 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/2  [ 0:00:09 < 0:00:00 , 0 it/s ]
+✅ Done!
+Total evaluations: 2
+------- Agents Elo Ratings -------
+agent1         : 1033.0(±0.0)
+agent2         : 966.0(±0.0)
+```
+
+In this example, the output file is a JSON file with all the annotations performed by the evaluators and the final Elo ratings. These can also be loaded directly as a new `Experiment` object:
 
----------- Agent Scores by Elo ranking ----------
- agent1        : 1026.7
- agent2        : 973.3
+```python
+experiment = Experiment.load("experiment", "experiment.json")
 ```
 
-When running as a CLI, RAGElo expects the input files as CSV files. Specifically, it needs a csv file with the user queries, one with the documents retrieved by the retrieval system and one of the answers each agent produced. Here are some examples of the expected format:
+When running as a CLI, RAGElo expects the input files as CSV files. Specifically, it expects a csv file with the user queries, one with the documents retrieved by the retrieval system and one of the answers each agent produced. These files can be passed with the parameters `--queries_csv_file`, `--documents_csv_file` and `--answers_csv_file`, respectively, or directly as positional arguments.
 
+Here are some examples of their expected formats:
 `queries.csv`: 
 ```csv
 qid,query
@@ -110,7 +150,7 @@ qid,query
 
 `documents.csv`:
 ```csv
-qid,did,document_text
+qid,did,document
 0,0, Brasília is the capital of Brazil.
 0,1, Rio de Janeiro used to be the capital of Brazil.
 1,2, Paris is the capital of France.
@@ -134,29 +174,17 @@ While **RAGElo** can be used as either an end-to-end tool or by calling individu
 The `retrieval-evaluator` tool annotates retrieved documents based on their relevance to the user query. This is done regardless of the answers provided by any Agent. As an example, for calling the `Reasoner` retrieval evaluator (reasoner only outputs the reasoning why a document is relevant or not) we can use:
 
 ```bash
-ragelo retrieval-evaluator reasoner queries.csv documents.csv output.csv --verbose --data-dir tests/data/
+ragelo retrieval-evaluator reasoner queries.csv documents.csv --data-dir tests/data/ --experiment-name experiment
 ```
 The output file changes according to the evaluator used. In general it will have one row per document evaluator, with the query_id, document_id, the raw LLM answer and the parsed answer. An example of the output for the reasoner is found here: [tests/data/reasonings.csv](https://github.com/zetaalphavector/RAGElo/blob/master/tests/data/reasonings.csv).
 
 ### 💬 `answers-annotator`
 
-The `answers-annotator` tool annotates the answers generated by the Agents, taking the quality of the documents retrieved by the retrieval pipeline. By default, it uses the `Pairwise` annotator, which generates `k` random pairs of answers for each query and chooses the best answer based on the relevant documents cited in the answer. It relies on the reasonings generated by the `Reasoner` `retrieval-evaluator`.
-
-```bash
-ragelo answer-evaluator pairwise-reasoning queries.csv documents.csv answers.csv --games-evaluations-file  pairwise_answers_evaluations.csv --verbose --data-dir tests/data/
-```
-
-The `pairwise_answers_evaluations.csv` file is a CSV file with both the raw answer and the parsed result for each pair of "games" between two agents. An output file example is provided at [tests/data/pairwise_answers_evaluations.csv](https://github.com/zetaalphavector/RAGElo/blob/master/tests/data/pairwise_answers_evaluations.csv)
- 
-### 🏆 `agents-ranker`
-
-Finally, the `agents-ranker` tool ranks the agents by simulating an Elo tournament where the output of each game is given by the answers from the `answers-annotator`:
+The `answers-annotator` tool annotates the answers generated by the Agents, taking the quality of the documents retrieved by the retrieval pipeline. By default, it uses the `Pairwise` annotator, which generates `k` random pairs of answers for each query and chooses the best answer based on the relevant documents cited in the answer. If the experiment already exists with annotations for the documents, it will try to load these and inject into the prompts for a better context for the LLM. Otherwise, you can pass the `--add-reasoning` flag to run the `Reasoner` retrieval evaluator first.
 
 ```bash
-ragelo agents-ranker elo pairwise_answers_evaluations.csv --agents-evaluations-file agents_ranking.csv --verbose --data-dir tests/data/
+ragelo answer-evaluator pairwise queries.csv documents.csv answers.csv --data-dir tests/data/ --experiment-name experiment --add-reasoning
 ```
-The output of this step is written to the output file `agents_ranking.csv` with columns agent and score: [tests/data/agents_ranking.csv](https://github.com/zetaalphavector/RAGElo/blob/master/tests/data/agents_ranking.csv).
-
 
 ## 🙋 Contributing
 
@@ -180,7 +208,7 @@ python -m build
 - [x] Add option to few-shot examples (Undocumented, yet)
 - [x] Testing!
 - [x] Publish on PyPi
-- [x] Add more document evaluators (Microsoft)
+- [x] Add more document evaluators
 - [x] Split Elo evaluator
 - [x] Install as standalone CLI
 
diff --git a/pyproject.toml b/pyproject.toml
index 509c182..b98cf8b 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -10,7 +10,7 @@ authors = [
 ]
 description = "RAGElo: A Tool for Evaluating Retrieval-Augmented Generation Models"
 readme = "README.md"
-requires-python = ">=3.8"
+requires-python = ">=3.8.1"
 license = { text = "Apache-2.0" }
 dynamic = ["version"]
 classifiers = [
@@ -19,28 +19,33 @@ classifiers = [
     "Intended Audience :: Developers",
     "Intended Audience :: Science/Research",
     "License :: OSI Approved :: Apache Software License",
-    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.12",
     "Topic :: Scientific/Engineering :: Artificial Intelligence",
     "Topic :: Software Development :: Libraries :: Python Modules",
     "Topic :: System :: Benchmark",
 ]
-dependencies = ["openai>=1.10", "tenacity==9.0.0", "typer==0.12.4", "numpy>=1.24", "pydantic>=1.9"]
+dependencies = [
+    "openai>=1.10",
+    "tenacity==9.0.0",
+    "typer==0.12.5",
+    "numpy>=1.24",
+    "pydantic>=1.9",
+    "eval-type-backport>=0.2.0",
+]
 
 [project.optional-dependencies]
-cli = ["typer[all]"]
+eval = ["ir-measures>=0.3.4"]
+
+[dependency-groups]
 dev = [
-    "bandit==1.7.5",
-    "black==23.10.0",
-    "isort==5.12.0",
-    "flake8==6.1.0",
-    "flake8-black==0.3.6",
-    "flake8-isort==6.1.0",
-    "mypy==1.9.0",
-    "Flake8-pyproject==1.2.3",
+    "mypy>=1.9.0",
+    "pydantic>=2.0.0",
+    "pytest>=8.3.3",
+    "ruff>=0.7.4",
     "types-tqdm==4.66.0",
-    "pydantic>=1.9.0",
     "pytest-asyncio==0.23.6",
     "pytest-mock==3.12.0",
+
 ]
 
 [project.scripts]
@@ -53,26 +58,34 @@ Repository = "https://github.com/zetaalphavector/RAGElo"
 [tool.setuptools.packages.find]
 exclude = ["data*"]
 
-[tool.isort]
-profile = "black"
 
 [tool.mypy]
 plugins = ["pydantic.mypy"]
-python_version = "3.8"
+python_version = "3.12"
 show_column_numbers = true
 namespace_packages = true
 follow_imports = "silent"
 warn_redundant_casts = true
 check_untyped_defs = true
 implicit_reexport = true
+ignore_missing_imports = true
 exclude = ["build/", "dist/", "venv/"]
 
 
-[tool.flake8]
-ignore = ['E501', "W503", "E203"]
-per-file-ignores = ['__init__.py:F401,F403']
-exclude = ["build/", "dist/", "venv/"]
+[tool.ruff]
+exclude = ["docs/examples/notebooks"]
+line-length = 119
+fix = true
 
+[tool.ruff.lint.isort]
+required-imports = ["from __future__ import annotations"]
 
 [tool.setuptools-git-versioning]
 enabled = true
+
+[tool.pytest.ini_options]
+markers = [
+    "requires_openai: marks tests as requiring a valid OpenAI api key (deselect with '-m \"not requires_openai\"')",
+    "serial",
+]
+
diff --git a/ragelo/__init__.py b/ragelo/__init__.py
index 64bdd06..f34fc49 100644
--- a/ragelo/__init__.py
+++ b/ragelo/__init__.py
@@ -1,5 +1,7 @@
 """CLI for automatically evaluating Retrieval Augmented Generation (RAG) models."""
 
+from __future__ import annotations
+
 from ragelo.agent_rankers.base_agent_ranker import get_agent_ranker
 from ragelo.evaluators.answer_evaluators import (
     AnswerEvaluatorFactory,
@@ -10,12 +12,23 @@
     get_retrieval_evaluator,
 )
 from ragelo.llm_providers.base_llm_provider import get_llm_provider
-from ragelo.types.types import AgentAnswer, Document, Query
+from ragelo.types import AgentAnswer, Document, Experiment, Query
+
+__all__ = [
+    "get_agent_ranker",
+    "get_answer_evaluator",
+    "get_retrieval_evaluator",
+    "get_llm_provider",
+    "AnswerEvaluatorFactory",
+    "RetrievalEvaluatorFactory",
+    "AgentAnswer",
+    "Document",
+    "Query",
+    "Experiment",
+]
 
 __version__ = "0.1.0"
 __app_name__ = "ragelo"
 __author__ = "Zeta Alpha"
 __email__ = "camara@zeta-alpha.com"
-__description__ = (
-    "CLI for automatically evaluating Retrieval Augmented Generation (RAG) models."
-)
+__description__ = "CLI for automatically evaluating Retrieval Augmented Generation (RAG) models."
diff --git a/ragelo/agent_rankers/__init__.py b/ragelo/agent_rankers/__init__.py
index 3ea25df..885ee72 100644
--- a/ragelo/agent_rankers/__init__.py
+++ b/ragelo/agent_rankers/__init__.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 from ragelo.agent_rankers.base_agent_ranker import AgentRankerFactory
 from ragelo.agent_rankers.elo_ranker import EloRanker
 
diff --git a/ragelo/agent_rankers/base_agent_ranker.py b/ragelo/agent_rankers/base_agent_ranker.py
index e600777..5359504 100644
--- a/ragelo/agent_rankers/base_agent_ranker.py
+++ b/ragelo/agent_rankers/base_agent_ranker.py
@@ -1,12 +1,11 @@
-"""Base models for ranking a set of answers generated by LLMs"""
+from __future__ import annotations
 
-import csv
-from typing import Dict, List, Optional, Tuple, Type, Union, get_type_hints
+from typing import Any, Type, get_type_hints
 
 from ragelo.logger import logger
 from ragelo.types.configurations.agent_ranker_configs import AgentRankerConfig
-from ragelo.types.types import AgentRankerTypes, Query
-from ragelo.utils import load_answer_evaluations_from_csv
+from ragelo.types.experiment import Experiment
+from ragelo.types.types import AgentRankerTypes
 
 
 class AgentRanker:
@@ -20,90 +19,39 @@ def __init__(
         self.name = self.config.ranker_name
         self.agents_evaluations_file = self.config.agents_evaluations_file
 
-    def run(
-        self,
-        queries: Optional[List[Query]] = None,
-        evaluations_file: Optional[str] = None,
-    ) -> Dict[str, int]:
-        """Compute score for each agent"""
+    def run(self, experiment: Experiment) -> Any:
+        """Computes a score for each agent in the experiment"""
         raise NotImplementedError
 
-    def _prepare_queries(
-        self,
-        queries: Optional[List[Query]] = None,
-        evaluations_file: Optional[str] = None,
-    ) -> List[Query]:
-        if queries is None:
-            if evaluations_file is None:
-                raise ValueError(
-                    "Either queries or evaluations_file should be provided"
-                )
-            queries = load_answer_evaluations_from_csv(evaluations_file)
-        return queries
-
     @classmethod
     def from_config(cls, config: AgentRankerConfig):
         return cls(config)
 
-    def get_agents_ratings(self) -> Dict[str, float]:
+    def get_agents_ratings(self) -> dict[str, float]:
         """Returns the score of all players"""
         raise NotImplementedError
 
-    def dump_ranking(self):
-        if not self.config.write_output:
-            return
-        with open(self.agents_evaluations_file, "w") as f:
-            writer = csv.writer(f)
-            writer.writerow(["agent", "score"])
-            for agent, rating in sorted(
-                self.get_agents_ratings().items(), key=lambda x: x[1], reverse=True
-            ):
-                writer.writerow([agent, rating])
-
-    def print_ranking(self):
-        if not self.config.verbose:
-            return
-        scores = sorted(
-            self.get_agents_ratings().items(), key=lambda x: x[1], reverse=True
-        )
-        if self.config.rich_print:
-            try:
-                import rich
-
-                rich.print(
-                    f"-------[bold white] Agent Scores by {self.name} [/bold white]-------"
-                )
-
-                for agent, rating in scores:
-                    rich.print(f"[bold white]{agent:<15}[/bold white]: {rating:.1f}")
-            except ImportError:
-                logger.warning("Rich not installed. Using plain print")
-                self.config.rich_print = False
-        if not self.config.rich_print:
-            print(f"------- Agent Scores by {self.name} -------")
-            for agent, rating in scores:
-                print(f"{agent:<15}: {rating:.1f}")
-
     @classmethod
     def get_config_class(cls) -> Type[AgentRankerConfig]:
         return get_type_hints(cls)["config"]
 
-    def _flatten_evaluations(self, queries) -> List[Tuple[str, str, str]]:
+    def _flatten_evaluations(self, experiment: Experiment) -> list[tuple[str, str, str]]:
         evaluations = []
-        for q in queries:
-            for game in q.pairwise_games:
-                evaluations.append(
-                    (
-                        game.agent_a_answer.agent,
-                        game.agent_b_answer.agent,
-                        game.evaluation.answer,
+        for query in experiment:
+            for game in query.pairwise_games:
+                if game.evaluation is not None:
+                    evaluations.append(
+                        (
+                            game.agent_a_answer.agent,
+                            game.agent_b_answer.agent,
+                            game.evaluation.answer,
+                        )
                     )
-                )
         return evaluations
 
 
 class AgentRankerFactory:
-    registry: Dict[AgentRankerTypes, Type[AgentRanker]] = {}
+    registry: dict[AgentRankerTypes, Type[AgentRanker]] = {}
 
     @classmethod
     def register(cls, name: AgentRankerTypes):
@@ -119,7 +67,7 @@ def inner_wrapper(wrapped_class: Type[AgentRanker]):
     def create(
         cls,
         ranker_name: AgentRankerTypes,
-        config: Optional[AgentRankerConfig] = None,
+        config: AgentRankerConfig | None = None,
         **kwargs,
     ) -> AgentRanker:
         if ranker_name not in cls.registry:
@@ -134,8 +82,8 @@ def create(
 
 
 def get_agent_ranker(
-    ranker_name: Optional[Union[AgentRankerTypes, str]] = None,
-    config: Optional[AgentRankerConfig] = None,
+    ranker_name: AgentRankerTypes | str | None = None,
+    config: AgentRankerConfig | None = None,
     **kwargs,
 ) -> AgentRanker:
     if ranker_name is None:
diff --git a/ragelo/agent_rankers/elo_ranker.py b/ragelo/agent_rankers/elo_ranker.py
index 8239a73..2a58283 100644
--- a/ragelo/agent_rankers/elo_ranker.py
+++ b/ragelo/agent_rankers/elo_ranker.py
@@ -1,10 +1,15 @@
+from __future__ import annotations
+
 import random
-from typing import Dict, List, Optional, Tuple
+
+import numpy as np
 
 from ragelo.agent_rankers.base_agent_ranker import AgentRanker, AgentRankerFactory
 from ragelo.logger import logger
-from ragelo.types import EloAgentRankerConfig
-from ragelo.types.types import AgentRankerTypes, Query
+from ragelo.types.configurations import EloAgentRankerConfig
+from ragelo.types.experiment import Experiment
+from ragelo.types.results import EloTournamentResult
+from ragelo.types.types import AgentRankerTypes
 
 
 @AgentRankerFactory.register(AgentRankerTypes.ELO)
@@ -17,73 +22,79 @@ def __init__(
         config: EloAgentRankerConfig,
     ):
         super().__init__(config)
-        self.score_map = {"A": 1, "B": 0, "C": 0.5}
+        self.score_map = config.score_mapping
 
-        self.agents: Dict[str, int] = {}
-        self.games: List[Tuple[str, str, float]] = []
-        self.computed = False
-        self.initial_score = self.config.initial_score
-        self.k = self.config.elo_k
+        self.agents_scores: dict[str, float] = {}
+        self.wins: dict[str, int] = {}
+        self.losses: dict[str, int] = {}
+        self.ties: dict[str, int] = {}
+        self.total_games: int = 0
+        self.games_played: dict[str, int] = {}
+        self.computed: bool = False
+        self.initial_score: int = self.config.initial_score
+        self.k: int = self.config.elo_k
+        self.std_dev: dict[str, float] = {}
 
-    def run(
-        self,
-        queries: Optional[List[Query]] = None,
-        evaluations_file: Optional[str] = None,
-    ) -> Dict[str, int]:
+    def run(self, experiment: Experiment) -> EloTournamentResult:
         """Compute score for each agent"""
-        queries = self._prepare_queries(queries, evaluations_file)
-        self.evaluations = self._flatten_evaluations(queries)
-        agent_scores: Dict[str, List[int]] = {}
-        for _ in range(self.config.rounds):
-            self.games = self.__get_elo_scores()
-            while self.games:
-                player1, player2, result = self.games.pop()
-                self.__play_one_game(player1, player2, result)
-            for a in self.agents:
-                if a not in agent_scores:
-                    agent_scores[a] = []
-                agent_scores[a].append(self.agents[a])
-                self.agents[a] = self.initial_score
-        # Average the scores over all rounds
+        self.evaluations = self._flatten_evaluations(experiment)
+        agent_scores: dict[str, list[int]] = {}
+        for _ in range(self.config.tournaments):
+            results = self.run_tournament()
+            for agent, score in results.items():
+                agent_scores[agent] = agent_scores.get(agent, []) + [score]
         for a in agent_scores:
-            self.agents[a] = sum(agent_scores[a]) // len(agent_scores[a])
-        self.dump_ranking()
-        self.print_ranking()
-        return self.agents
+            self.std_dev[a] = float(np.std(agent_scores[a]))
+            self.agents_scores[a] = float(np.mean(agent_scores[a]))
+
+        result = EloTournamentResult(
+            agents=list(self.agents_scores.keys()),
+            scores=self.agents_scores,
+            games_played=self.games_played,
+            wins=self.wins,
+            loses=self.losses,
+            ties=self.ties,
+            std_dev=self.std_dev,
+            total_games=self.total_games,
+            total_tournaments=self.config.tournaments,
+        )
+        experiment.add_evaluation(result)
+        return result
 
     def get_agents_ratings(self):
-        if not self.computed:
-            raise ValueError("Ranking not computed yet, Run run() first")
-        return self.agents
+        return self.agents_scores
+
+    def get_ranked_agents(self) -> list[tuple[str, float]]:
+        ranking = sorted(self.get_agents_ratings().items(), key=lambda x: x[1], reverse=True)
+        return [(agent, rating) for agent, rating in ranking]
 
-    def __get_elo_scores(self) -> List[Tuple[str, str, float]]:
-        games: List[Tuple[str, str, float]] = []
+    def run_tournament(self) -> dict[str, int]:
+        agents_scores: dict[str, int] = {}
+        games: list[tuple[str, str, float]] = []
         for agent_a, agent_b, score in self.evaluations:
             score_val = self.score_map[score]
             games.append((agent_a, agent_b, score_val))
-            logger.info(f"Game: {agent_a} vs {agent_b} -> {score_val}")
-            if agent_a not in self.agents:
-                self.agents[agent_a] = self.initial_score
-            if agent_b not in self.agents:
-                self.agents[agent_b] = self.initial_score
         random.shuffle(games)
-        self.computed = True
-        return games
-
-    def __play_one_game(self, player1, player2, result):
-        player1_rating = self.agents[player1]
-        player2_rating = self.agents[player2]
-        expected_score = self.__expected_score(player1_rating, player2_rating)
-
-        self.agents[player1] = self.__update_rating(
-            player1_rating, expected_score, result
-        )
-        self.agents[player2] = self.__update_rating(
-            player2_rating, 1 - expected_score, 1 - result
-        )
+        for agent_a, agent_b, score_val in games:
+            logger.info(f"Game: {agent_a} vs {agent_b} -> {score_val}")
+            if score_val == 1:
+                self.wins[agent_a] = self.wins.get(agent_a, 0) + 1
+                self.losses[agent_b] = self.losses.get(agent_b, 0) + 1
+            elif score_val == 0:
+                self.wins[agent_b] = self.wins.get(agent_b, 0) + 1
+                self.losses[agent_a] = self.losses.get(agent_a, 0) + 1
+            else:
+                self.ties[agent_a] = self.ties.get(agent_a, 0) + 1
+                self.ties[agent_b] = self.ties.get(agent_b, 0) + 1
+            agent_a_rating = agents_scores.get(agent_a, self.initial_score)
+            agent_b_rating = agents_scores.get(agent_b, self.initial_score)
 
-    def __expected_score(self, rating1, rating2):
-        return 1 / (1 + 10 ** ((rating2 - rating1) / 400))
+            expected_score = 1 / (1 + 10 ** ((agent_a_rating - agent_b_rating) / 400))
+            agents_scores[agent_a] = int(agent_a_rating + self.k * (score_val - expected_score))
+            agents_scores[agent_b] = int(agent_b_rating + self.k * ((1 - score_val) - (1 - expected_score)))
+            self.total_games += 1
+            self.games_played[agent_a] = self.games_played.get(agent_a, 0) + 1
+            self.games_played[agent_b] = self.games_played.get(agent_b, 0) + 1
 
-    def __update_rating(self, rating, expected_score, actual_score) -> int:
-        return int(rating + self.k * (actual_score - expected_score))
+        self.computed = True
+        return agents_scores
diff --git a/ragelo/cli/__init__.py b/ragelo/cli/__init__.py
index d6b2440..de914d5 100644
--- a/ragelo/cli/__init__.py
+++ b/ragelo/cli/__init__.py
@@ -1,1 +1,5 @@
+from __future__ import annotations
+
 from ragelo.cli.cli import app
+
+__all__ = ["app"]
diff --git a/ragelo/cli/agent_rankers_cmd.py b/ragelo/cli/agent_rankers_cmd.py
deleted file mode 100644
index 865fdb3..0000000
--- a/ragelo/cli/agent_rankers_cmd.py
+++ /dev/null
@@ -1,24 +0,0 @@
-import typer
-
-from ragelo import get_agent_ranker
-from ragelo.cli.args import get_params_from_function
-from ragelo.cli.utils import get_path
-from ragelo.types import AgentRankerTypes, EloAgentRankerConfig
-from ragelo.utils import load_answer_evaluations_from_csv
-
-typer.main.get_params_from_function = get_params_from_function  # type: ignore
-app = typer.Typer()
-
-
-@app.command()
-def elo(config: EloAgentRankerConfig = EloAgentRankerConfig(), **kwargs):
-    """Evaluate the performance of multiple agents using an Elo ranking system."""
-    config = EloAgentRankerConfig(**kwargs)
-    config.evaluations_file = get_path(config.data_dir, config.evaluations_file)
-    config.agents_evaluations_file = get_path(
-        config.data_dir, config.agents_evaluations_file, check_exists=False
-    )
-    ranker = get_agent_ranker(AgentRankerTypes.ELO, config=config)
-    queries = load_answer_evaluations_from_csv(config.evaluations_file)
-
-    ranker.run(queries=queries)
diff --git a/ragelo/cli/answer_evaluators_cli.py b/ragelo/cli/answer_evaluators_cli.py
new file mode 100644
index 0000000..69eeef7
--- /dev/null
+++ b/ragelo/cli/answer_evaluators_cli.py
@@ -0,0 +1,119 @@
+from __future__ import annotations
+
+import logging
+
+import typer
+
+from ragelo import (
+    Experiment,
+    get_answer_evaluator,
+    get_llm_provider,
+    get_retrieval_evaluator,
+)
+from ragelo.cli.args import get_params_from_function
+from ragelo.cli.utils import get_path
+from ragelo.types import AnswerEvaluatorTypes
+from ragelo.types.configurations.cli_configs import (
+    CLIPairwiseDomainExpertEvaluatorConfig,
+    CLIPairwiseEvaluatorConfig,
+)
+from ragelo.types.types import RetrievalEvaluatorTypes
+
+typer.main.get_params_from_function = get_params_from_function  # type: ignore
+app = typer.Typer()
+
+
+@app.command()
+def pairwise(config: CLIPairwiseEvaluatorConfig = CLIPairwiseEvaluatorConfig(), **kwargs):
+    """An evaluator that evaluates RAG-based answers by comparing the answers of two agents to the same queries.
+
+    example:
+    >> ragelo answer-evaluator pairwise queries.csv answers.csv
+
+    """
+    logging.getLogger("ragelo").setLevel(logging.INFO)
+    kwargs.pop("llm_answer_format", None)
+    kwargs.pop("llm_response_schema", None)
+
+    config = CLIPairwiseEvaluatorConfig(**kwargs)
+    llm_provider = get_llm_provider(config.llm_provider_name, **kwargs)
+
+    queries_csv_file = get_path(config.data_dir, config.queries_csv_file)
+    documents_file = get_path(config.data_dir, config.documents_csv_file)
+    answers_file = get_path(config.data_dir, config.answers_csv_file)
+    output_file = get_path(config.data_dir, config.output_file, check_exists=False) if config.output_file else None
+
+    experiment = Experiment(
+        experiment_name=config.experiment_name,
+        save_path=output_file,
+        queries_csv_path=queries_csv_file,
+        documents_csv_path=documents_file,
+        answers_csv_path=answers_file,
+        verbose=config.verbose,
+        clear_evaluations=config.force,
+        rich_print=config.rich_print,
+        cache_evaluations=config.save_results,
+    )
+
+    kwargs = config.model_dump()
+
+    if config.add_reasoning:
+        reasoner_evaluator = get_retrieval_evaluator(
+            RetrievalEvaluatorTypes.REASONER,
+            llm_provider=llm_provider,
+            rich_print=config.rich_print,
+            verbose=config.verbose,
+        )
+        reasoner_evaluator.evaluate_experiment(experiment)
+        config.include_annotations = True
+        config.include_raw_documents = False
+    else:
+        config.include_annotations = False
+        config.include_raw_documents = True
+
+    evaluator = get_answer_evaluator(AnswerEvaluatorTypes.PAIRWISE, config=config, llm_provider=llm_provider)
+    evaluator.evaluate_experiment(experiment)
+    experiment.save(output_file)
+
+
+@app.command()
+def expert_pairwise(
+    config: CLIPairwiseDomainExpertEvaluatorConfig = CLIPairwiseDomainExpertEvaluatorConfig(),
+    **kwargs,
+):
+    """
+    An evaluator that evaluates RAG-based answers by comparing answers of two agents and impersonating a domain expert.
+    """
+    logging.getLogger("ragelo").setLevel(logging.INFO)
+    kwargs.pop("llm_answer_format", None)
+    kwargs.pop("llm_response_schema", None)
+
+    config = CLIPairwiseDomainExpertEvaluatorConfig(**kwargs)
+
+    llm_provider = get_llm_provider(config.llm_provider_name, **kwargs)
+
+    queries_csv_file = get_path(config.data_dir, config.queries_csv_file)
+    documents_file = get_path(config.data_dir, config.documents_csv_file)
+    answers_file = get_path(config.data_dir, config.answers_csv_file)
+    output_file = get_path(config.data_dir, config.output_file, check_exists=False) if config.output_file else None
+
+    experiment = Experiment(
+        experiment_name=config.experiment_name,
+        save_path=output_file,
+        queries_csv_path=queries_csv_file,
+        documents_csv_path=documents_file,
+        answers_csv_path=answers_file,
+        verbose=config.verbose,
+        clear_evaluations=config.force,
+        rich_print=config.rich_print,
+        cache_evaluations=config.save_results,
+    )
+
+    if config.add_reasoning:
+        reasoner_evaluator = get_retrieval_evaluator(RetrievalEvaluatorTypes.REASONER, llm_provider=llm_provider)
+        reasoner_evaluator.evaluate_experiment(experiment)
+        config.include_annotations = True
+
+    evaluator = get_answer_evaluator(AnswerEvaluatorTypes.PAIRWISE, config=config, llm_provider=llm_provider)
+    evaluator.evaluate_experiment(experiment)
+    experiment.save(output_file)
diff --git a/ragelo/cli/answer_evaluators_cmd.py b/ragelo/cli/answer_evaluators_cmd.py
deleted file mode 100644
index f3091d5..0000000
--- a/ragelo/cli/answer_evaluators_cmd.py
+++ /dev/null
@@ -1,46 +0,0 @@
-import typer
-
-from ragelo import get_answer_evaluator, get_llm_provider
-from ragelo.cli.args import get_params_from_function
-from ragelo.cli.utils import get_path
-from ragelo.types import AnswerEvaluatorTypes
-from ragelo.types.configurations import PairwiseEvaluatorConfig
-from ragelo.utils import (
-    add_answers_from_csv,
-    add_documents_from_csv,
-    load_queries_from_csv,
-)
-
-typer.main.get_params_from_function = get_params_from_function  # type: ignore
-app = typer.Typer()
-
-
-@app.command()
-def pairwise_reasoning(
-    config: PairwiseEvaluatorConfig = PairwiseEvaluatorConfig(), **kwargs
-):
-    """A evaluator that evaluates RAG-based answers pairwise, with document reasoning"""
-    config = PairwiseEvaluatorConfig(**kwargs)
-    config.queries_file = get_path(config.data_dir, config.queries_file)
-    config.answers_file = get_path(config.data_dir, config.answers_file)
-    config.documents_file = get_path(config.data_dir, config.documents_file)
-    config.document_evaluations_file = get_path(
-        config.data_dir, config.document_evaluations_file
-    )
-
-    config.games_evaluations_file = get_path(
-        config.data_dir,
-        config.games_evaluations_file,
-        check_exists=False,
-    )
-
-    llm_provider = get_llm_provider(config.llm_provider_name, **kwargs)
-    evaluator = get_answer_evaluator(
-        AnswerEvaluatorTypes.PAIRWISE,
-        config=config,
-        llm_provider=llm_provider,
-    )
-    queries = load_queries_from_csv(config.queries_file)
-    queries = add_documents_from_csv(config.documents_file, queries=queries)
-    queries = add_answers_from_csv(config.answers_file, queries=queries)
-    evaluator.batch_evaluate(queries)
diff --git a/ragelo/cli/args.py b/ragelo/cli/args.py
index 4e18be8..73ef508 100644
--- a/ragelo/cli/args.py
+++ b/ragelo/cli/args.py
@@ -1,25 +1,26 @@
 """Parse arguments for the cli app"""
+
+from __future__ import annotations
+
 import collections.abc
 import inspect
 import sys
-from typing import Any, Callable, Dict, Union, get_args, get_origin, get_type_hints
+from typing import Any, Callable, Dict, get_args, get_origin, get_type_hints
 
 from typer.models import ArgumentInfo, OptionInfo, ParameterInfo, ParamMeta
 
 from ragelo.types import BaseConfig
-from ragelo.types.configurations.base_configs import _PYDANTIC_MAJOR_VERSION
+from ragelo.types.pydantic_models import _PYDANTIC_MAJOR_VERSION
 
 arguments = {
-    "queries_file",
-    "documents_file",
+    "queries_csv_file",
+    "documents_csv_file",
+    "answers_csv_file",
     "domain_long",
-    "answers_file",
-    "reasoning_file",
-    "evaluations_file",
-    "retrieval_evaluator_name",
-    "answer_evaluator_name",
 }
 
+ignore_args = {"llm_response_schema"}
+
 
 def get_params_from_function(func: Callable[..., Any]) -> Dict[str, ParamMeta]:
     if sys.version_info >= (3, 10):
@@ -40,6 +41,8 @@ def get_params_from_function(func: Callable[..., Any]) -> Dict[str, ParamMeta]:
         if inspect.isclass(annotation) and issubclass(annotation, BaseConfig):
             fields = annotation.get_model_fields()
             for k, v in fields.items():
+                if k in ignore_args:
+                    continue
                 if _PYDANTIC_MAJOR_VERSION == 2:
                     description = v.description  # type: ignore
                     _type = v.annotation  # type: ignore
@@ -50,30 +53,34 @@ def get_params_from_function(func: Callable[..., Any]) -> Dict[str, ParamMeta]:
                     _type = v.type_  # type: ignore
                     _outer_type = v.outer_type_  # type: ignore
                     t_args = get_args(_outer_type)
+                if get_origin(_outer_type) is list:
+                    _type = _outer_type
+                if get_origin(_outer_type) is type(None) or _type is type(None):
+                    continue
+                if get_origin(_outer_type) is dict:
+                    continue
+
                 if not isinstance(v, ParameterInfo):
-                    if get_origin(_outer_type) == list:
-                        _type = _outer_type
-                    # elif _outer_type == Union:
-                    # _type = t_args[0]
-                    # parse the Union type, defaulting to the str type if possible
                     if len(t_args) > 1:
-                        if get_origin(_outer_type) == Union:
-                            _type = t_args[0]
+                        # To resolve the True argument type, first remove any NoneType from the list of types"
+                        _t_args = [t for t in t_args if t is not type(None)]
+                        if len(_t_args) == 1:
+                            _type = _t_args[0]
                         if (
                             get_origin(_outer_type) == collections.abc.Callable
                             or get_origin(_type) == collections.abc.Callable
                         ):
                             # ignore the callable type and move on.
                             continue
+                        elif len(_t_args) > 1:
+                            _type = _t_args[0]
                     if k in arguments:
                         argument = ArgumentInfo(default=v.default, help=description)
-                        params[k] = ParamMeta(
-                            name=k, default=argument, annotation=_type
-                        )
+                        params[k] = ParamMeta(name=k, default=argument, annotation=_type)
                     else:
                         option = OptionInfo(
                             default=v.default,
-                            default_factory=v.default_factory,
+                            default_factory=v.default_factory,  # type: ignore
                             help=description,
                         )
                         params[k] = ParamMeta(name=k, default=option, annotation=_type)
@@ -81,7 +88,5 @@ def get_params_from_function(func: Callable[..., Any]) -> Dict[str, ParamMeta]:
                     params[k] = ParamMeta(name=k, default=v, annotation=_type)
 
         else:
-            params[param.name] = ParamMeta(
-                name=param.name, default=param.default, annotation=annotation
-            )
+            params[param.name] = ParamMeta(name=param.name, default=param.default, annotation=annotation)
     return params
diff --git a/ragelo/cli/cli.py b/ragelo/cli/cli.py
index fcf804f..f1356a4 100644
--- a/ragelo/cli/cli.py
+++ b/ragelo/cli/cli.py
@@ -1,22 +1,21 @@
+from __future__ import annotations
+
+import logging
+
 import typer
 
 from ragelo import (
+    Experiment,
     get_agent_ranker,
     get_answer_evaluator,
     get_llm_provider,
     get_retrieval_evaluator,
 )
-from ragelo.cli.agent_rankers_cmd import app as ranker_app
-from ragelo.cli.answer_evaluators_cmd import app as answer_evaluator_app
+from ragelo.cli.answer_evaluators_cli import app as answer_evaluator_app
 from ragelo.cli.args import get_params_from_function
-from ragelo.cli.retrieval_evaluator_cmd import app as retrieval_evaluator_app
+from ragelo.cli.retrieval_evaluator_cli import app as retrieval_evaluator_app
 from ragelo.cli.utils import get_path
-from ragelo.types import AllConfig
-from ragelo.utils import (
-    add_answers_from_csv,
-    add_documents_from_csv,
-    load_queries_from_csv,
-)
+from ragelo.types import CLIConfig
 
 typer.main.get_params_from_function = get_params_from_function  # type: ignore
 
@@ -26,67 +25,48 @@
 
 app.add_typer(retrieval_evaluator_app, name="retrieval-evaluator")
 app.add_typer(answer_evaluator_app, name="answer-evaluator")
-app.add_typer(ranker_app, name="agents-ranker")
 
 
 @app.command()
-def run_all(config: AllConfig = AllConfig(), **kwargs):
+def run_all(config: CLIConfig = CLIConfig(), **kwargs):
     """Run all the commands."""
-
-    config = AllConfig(**kwargs)
-    if config.retrieval_evaluator_name != "reasoner":
-        raise ValueError("The retrieval evaluator must be a reasoner")
-    if config.answer_evaluator_name != "pairwise":
-        raise ValueError("The answer evaluator must be pairwise")
-    if config.answer_ranker_name != "elo":
-        raise ValueError("The answer ranker must be an elo")
-    config.verbose = True
+    # set ragelo's logger to INFO
+    logging.getLogger("ragelo").setLevel(logging.INFO)
+    config = CLIConfig(**kwargs)
 
     # Parse the LLM provider and remove it from the kwargs
     llm_provider = get_llm_provider(config.llm_provider_name, **kwargs)
 
     # Get the absolute paths for the input and output files, and ensure that they exist.
-    # TODO: Make this more robust.
-    config.queries_file = get_path(config.data_dir, config.queries_file)
-    config.documents_file = get_path(config.data_dir, config.documents_file)
-    config.answers_file = get_path(config.data_dir, config.answers_file)
-    config.reasoning_file = get_path(
-        config.data_dir, config.reasoning_file, check_exists=False
-    )
-    config.answers_evaluations_file = get_path(
-        config.data_dir, config.answers_evaluations_file, check_exists=False
-    )
-    config.agents_evaluations_file = get_path(
-        config.data_dir, config.agents_evaluations_file, check_exists=False
-    )
-    config.games_evaluations_file = get_path(
-        config.data_dir, config.games_evaluations_file, check_exists=False
-    )
-    config.document_evaluations_file = get_path(
-        config.data_dir, config.document_evaluations_file, check_exists=False
+    queries_csv_file = get_path(config.data_dir, config.queries_csv_file)
+    documents_file = get_path(config.data_dir, config.documents_csv_file)
+    answers_file = get_path(config.data_dir, config.answers_csv_file)
+    output_file = get_path(config.data_dir, config.output_file, check_exists=False) if config.output_file else None
+
+    experiment = Experiment(
+        experiment_name=config.experiment_name,
+        save_path=output_file,
+        queries_csv_path=queries_csv_file,
+        documents_csv_path=documents_file,
+        answers_csv_path=answers_file,
+        verbose=config.verbose,
+        clear_evaluations=config.force,
+        rich_print=config.rich_print,
+        cache_evaluations=config.save_results,
     )
 
     kwargs = config.model_dump()
-    retrieval_evaluator = get_retrieval_evaluator(
-        config.retrieval_evaluator_name, llm_provider=llm_provider, **kwargs
-    )
-
-    answers_evaluator = get_answer_evaluator(
-        config.answer_evaluator_name, llm_provider=llm_provider, **kwargs
-    )
-
-    # TODO: Instead of managing a list of queries, we should have a Dataset type that can load the queries and documents directly.
-    queries = load_queries_from_csv(config.queries_file)
-    queries = add_documents_from_csv(
-        documents_file=config.documents_file, queries=queries
-    )
-    queries = add_answers_from_csv(answers_file=config.answers_file, queries=queries)
-
-    retrieval_evaluator.batch_evaluate(queries)
-    answers_evaluator.batch_evaluate(queries)
+    kwargs.pop("llm_answer_format", None)
+    kwargs.pop("llm_response_schema", None)
 
+    retrieval_evaluator = get_retrieval_evaluator("reasoner", llm_provider=llm_provider, **kwargs)
+    answers_evaluator = get_answer_evaluator("pairwise", llm_provider=llm_provider, **kwargs)
     ranker = get_agent_ranker("elo", **kwargs)
-    ranker.run(queries=queries)
+
+    retrieval_evaluator.evaluate_experiment(experiment)
+    answers_evaluator.evaluate_experiment(experiment)
+    ranker.run(experiment=experiment)
+    experiment.save(output_file)
 
 
 if __name__ == "__main__":
diff --git a/ragelo/cli/retrieval_evaluator_cli.py b/ragelo/cli/retrieval_evaluator_cli.py
new file mode 100644
index 0000000..ed9928c
--- /dev/null
+++ b/ragelo/cli/retrieval_evaluator_cli.py
@@ -0,0 +1,145 @@
+from __future__ import annotations
+
+import logging
+
+import typer
+
+from ragelo import Experiment, get_llm_provider, get_retrieval_evaluator
+from ragelo.cli.args import get_params_from_function
+from ragelo.cli.utils import get_path
+from ragelo.types.configurations.cli_configs import (
+    CLIDomainExpertEvaluatorConfig,
+    CLIRDNAMEvaluatorConfig,
+    CLIReasonerEvaluatorConfig,
+)
+from ragelo.types.formats import AnswerFormat
+from ragelo.types.types import RetrievalEvaluatorTypes
+
+typer.main.get_params_from_function = get_params_from_function  # type: ignore
+
+
+app = typer.Typer()
+
+
+@app.command()
+def domain_expert(config: CLIDomainExpertEvaluatorConfig = CLIDomainExpertEvaluatorConfig(), **kwargs):
+    """Evaluator with a domain expert persona.
+
+    This Retrieval Evaluator evaluates the relevance of documents submitted by
+    a user that is an expert in a specific domain. For instance,
+    to evaluate the documents retrieved to queries submitted by a Chemical
+    Engineer that works at ChemCorp Inc:
+
+    ragelo retrieval_evaluator domain_expert queries.csv documents.csv "Chemical Engineering" --company "ChemCorp Inc."
+
+    """
+    logging.getLogger("ragelo").setLevel(logging.INFO)
+    kwargs.pop("llm_answer_format", None)
+    kwargs.pop("llm_response_schema", None)
+
+    config = CLIDomainExpertEvaluatorConfig(**kwargs)
+    llm_provider = get_llm_provider(config.llm_provider_name, **kwargs)
+
+    queries_csv_file = get_path(config.data_dir, config.queries_csv_file)
+    documents_file = get_path(config.data_dir, config.documents_csv_file)
+    output_file = get_path(config.data_dir, config.output_file, check_exists=False) if config.output_file else None
+
+    experiment = Experiment(
+        experiment_name=config.experiment_name,
+        save_path=output_file,
+        queries_csv_path=queries_csv_file,
+        documents_csv_path=documents_file,
+        verbose=config.verbose,
+        clear_evaluations=config.force,
+        rich_print=config.rich_print,
+        cache_evaluations=config.save_results,
+    )
+
+    kwargs = config.model_dump()
+
+    evaluator = get_retrieval_evaluator(
+        RetrievalEvaluatorTypes.DOMAIN_EXPERT, config=config, llm_provider=llm_provider
+    )
+    evaluator.evaluate_experiment(experiment)
+    experiment.save(output_file)
+
+
+@app.command()
+def reasoner(
+    config: CLIReasonerEvaluatorConfig = CLIReasonerEvaluatorConfig(llm_answer_format=AnswerFormat.TEXT),
+    **kwargs,
+):
+    """
+    A document Evaluator that only outputs the reasoning for why a document is relevant.
+    """
+    logging.getLogger("ragelo").setLevel(logging.INFO)
+
+    kwargs.pop("llm_response_schema", None)
+    kwargs["llm_answer_format"] = AnswerFormat.TEXT
+
+    config = CLIReasonerEvaluatorConfig(**kwargs)
+
+    queries_csv_file = get_path(config.data_dir, config.queries_csv_file)
+    documents_file = get_path(config.data_dir, config.documents_csv_file)
+    output_file = get_path(config.data_dir, config.output_file, check_exists=False) if config.output_file else None
+
+    experiment = Experiment(
+        experiment_name=config.experiment_name,
+        save_path=output_file,
+        queries_csv_path=queries_csv_file,
+        documents_csv_path=documents_file,
+        verbose=config.verbose,
+        clear_evaluations=config.force,
+        rich_print=config.rich_print,
+        cache_evaluations=config.save_results,
+    )
+
+    kwargs = config.model_dump()
+
+    llm_provider = get_llm_provider(config.llm_provider_name, **kwargs)
+
+    evaluator = get_retrieval_evaluator(RetrievalEvaluatorTypes.REASONER, config=config, llm_provider=llm_provider)
+    evaluator.evaluate_experiment(experiment)
+    experiment.save(output_file)
+
+
+@app.command()
+def rdnam(config: CLIRDNAMEvaluatorConfig = CLIRDNAMEvaluatorConfig(), **kwargs):
+    """
+    Evaluator based on the paper by Thomas, Spielman, Craswell and Mitra:
+    Large language models can accurately predict searcher preferences.
+    """
+    logging.getLogger("ragelo").setLevel(logging.INFO)
+
+    kwargs.pop("llm_answer_format", None)
+    kwargs.pop("llm_response_schema", None)
+
+    config = CLIRDNAMEvaluatorConfig(**kwargs)
+    queries_csv_file = get_path(config.data_dir, config.queries_csv_file)
+    documents_file = get_path(config.data_dir, config.documents_csv_file)
+    output_file = get_path(config.data_dir, config.output_file, check_exists=False) if config.output_file else None
+
+    experiment = Experiment(
+        experiment_name=config.experiment_name,
+        save_path=output_file,
+        queries_csv_path=queries_csv_file,
+        documents_csv_path=documents_file,
+        verbose=config.verbose,
+        clear_evaluations=config.force,
+        rich_print=config.rich_print,
+        cache_evaluations=config.save_results,
+    )
+
+    kwargs = config.model_dump()
+    kwargs.pop("llm_answer_format")
+    kwargs.pop("llm_response_schema")
+
+    llm_provider = get_llm_provider(config.llm_provider_name, **kwargs)
+
+    evaluator = get_retrieval_evaluator(RetrievalEvaluatorTypes.RDNAM, config=config, llm_provider=llm_provider)
+    evaluator.evaluate_experiment(experiment)
+    experiment.save(output_file)
+
+
+if __name__ == "__main__":
+    app()
diff --git a/ragelo/cli/retrieval_evaluator_cmd.py b/ragelo/cli/retrieval_evaluator_cmd.py
deleted file mode 100644
index e37526e..0000000
--- a/ragelo/cli/retrieval_evaluator_cmd.py
+++ /dev/null
@@ -1,96 +0,0 @@
-import typer
-
-from ragelo import get_llm_provider, get_retrieval_evaluator
-from ragelo.cli.args import get_params_from_function
-from ragelo.cli.utils import get_path
-from ragelo.types.configurations import (
-    DomainExpertEvaluatorConfig,
-    RDNAMEvaluatorConfig,
-    ReasonerEvaluatorConfig,
-)
-from ragelo.types.types import RetrievalEvaluatorTypes
-from ragelo.utils import add_documents_from_csv, load_queries_from_csv
-
-typer.main.get_params_from_function = get_params_from_function  # type: ignore
-
-
-app = typer.Typer()
-
-
-@app.command()
-def domain_expert(
-    config: DomainExpertEvaluatorConfig = DomainExpertEvaluatorConfig(), **kwargs
-):
-    """Evaluator with a domain expert persona.
-
-    This Retrieval Evaluator evaluates the relevance of documents submitted by
-    a user that is an expert in a specific domain. For instance,
-    to evaluate the documents retrieved to queries submitted by a Chemical
-    Engineer that works at ChemCorp Inc:
-
-    ragelo retrieval_evaluator domain_expert queries.csv documents.csv "Chemical Engineering" --company "ChemCorp Inc."
-
-    """
-
-    config = DomainExpertEvaluatorConfig(**kwargs)
-    config.queries_file = get_path(config.data_dir, config.queries_file)
-    config.documents_file = get_path(config.data_dir, config.documents_file)
-    config.document_evaluations_file = get_path(
-        config.data_dir, config.document_evaluations_file, check_exists=False
-    )
-
-    config.verbose = True
-
-    llm_provider = get_llm_provider(config.llm_provider_name, **kwargs)
-    evaluator = get_retrieval_evaluator(
-        RetrievalEvaluatorTypes.DOMAIN_EXPERT, config=config, llm_provider=llm_provider
-    )
-    queries = load_queries_from_csv(config.queries_file)
-    queries = add_documents_from_csv(config.documents_file, queries=queries)
-    evaluator.batch_evaluate(queries)
-
-
-@app.command()
-def reasoner(config: ReasonerEvaluatorConfig = ReasonerEvaluatorConfig(), **kwargs):
-    """
-    A document Evaluator that only outputs the reasoning for why a document is relevant.
-    """
-    config = ReasonerEvaluatorConfig(**kwargs)
-    config.queries_file = get_path(config.data_dir, config.queries_file)
-    config.documents_file = get_path(config.data_dir, config.documents_file)
-    config.document_evaluations_file = get_path(
-        config.data_dir, config.document_evaluations_file, check_exists=False
-    )
-
-    config.verbose = True
-
-    llm_provider = get_llm_provider(config.llm_provider_name, **kwargs)
-    evaluator = get_retrieval_evaluator(
-        RetrievalEvaluatorTypes.REASONER, config=config, llm_provider=llm_provider
-    )
-    queries = load_queries_from_csv(config.queries_file)
-    queries = add_documents_from_csv(config.documents_file, queries=queries)
-    evaluator.batch_evaluate(queries)
-
-
-@app.command()
-def rdnam(config: RDNAMEvaluatorConfig = RDNAMEvaluatorConfig(), **kwargs):
-    """Evaluator based on the paper by Thomas, Spielman, Craswell and Mitra, Large language models can accurately predict searcher preferences."""
-    config = RDNAMEvaluatorConfig(**kwargs)
-    config.queries_file = get_path(config.data_dir, config.queries_file)
-    config.documents_file = get_path(config.data_dir, config.documents_file)
-    config.document_evaluations_file = get_path(
-        config.data_dir, config.document_evaluations_file, check_exists=False
-    )
-    config.verbose = True
-    llm_provider = get_llm_provider(config.llm_provider_name, **kwargs)
-    evaluator = get_retrieval_evaluator(
-        RetrievalEvaluatorTypes.RDNAM, config=config, llm_provider=llm_provider
-    )
-    queries = load_queries_from_csv(config.queries_file)
-    queries = add_documents_from_csv(config.documents_file, queries=queries)
-    evaluator.batch_evaluate(queries)
-
-
-if __name__ == "__main__":
-    app()
diff --git a/ragelo/cli/utils.py b/ragelo/cli/utils.py
index 157405f..83b921f 100644
--- a/ragelo/cli/utils.py
+++ b/ragelo/cli/utils.py
@@ -1,10 +1,9 @@
+from __future__ import annotations
+
 import os
-from typing import Optional
 
 
-def get_path(
-    data_path: Optional[str], file_path: str, check_exists: bool = True
-) -> str:
+def get_path(data_path: str | None, file_path: str, check_exists: bool = True) -> str:
     if data_path is None:
         data_path = "."
     if file_path.startswith("/"):
diff --git a/ragelo/evaluators/__init__.py b/ragelo/evaluators/__init__.py
index eb2fe96..6c1d2cf 100644
--- a/ragelo/evaluators/__init__.py
+++ b/ragelo/evaluators/__init__.py
@@ -5,3 +5,11 @@
     ReasonerEvaluator,
     RetrievalEvaluatorFactory,
 )
+
+__all__ = [
+    "PairwiseAnswerEvaluator",
+    "DomainExpertEvaluator",
+    "RDNAMEvaluator",
+    "ReasonerEvaluator",
+    "RetrievalEvaluatorFactory",
+]
diff --git a/ragelo/evaluators/answer_evaluators/__init__.py b/ragelo/evaluators/answer_evaluators/__init__.py
index 3671972..d441730 100644
--- a/ragelo/evaluators/answer_evaluators/__init__.py
+++ b/ragelo/evaluators/answer_evaluators/__init__.py
@@ -6,6 +6,9 @@
 from ragelo.evaluators.answer_evaluators.chat_pairwise_evaluator import (
     ChatPairwiseEvaluator,
 )
+from ragelo.evaluators.answer_evaluators.custom_pairwise_evaluator import (
+    CustomPairwiseEvaluator,
+)
 from ragelo.evaluators.answer_evaluators.custom_prompt_evaluator import (
     CustomPromptEvaluator,
 )
@@ -20,6 +23,7 @@
     "AnswerEvaluatorFactory",
     "BaseAnswerEvaluator",
     "CustomPromptEvaluator",
+    "CustomPairwiseEvaluator",
     "PairwiseDomainExpertEvaluator",
     "PairwiseAnswerEvaluator",
     "get_answer_evaluator",
diff --git a/ragelo/evaluators/answer_evaluators/base_answer_evaluator.py b/ragelo/evaluators/answer_evaluators/base_answer_evaluator.py
index d911b53..8870c62 100644
--- a/ragelo/evaluators/answer_evaluators/base_answer_evaluator.py
+++ b/ragelo/evaluators/answer_evaluators/base_answer_evaluator.py
@@ -1,42 +1,32 @@
 """Base model for dealing with answer evaluators"""
 
-import asyncio
+from __future__ import annotations
+
 import itertools
 import random
-from concurrent.futures import ThreadPoolExecutor
-from typing import (
-    Any,
-    Callable,
-    Dict,
-    List,
-    Optional,
-    Set,
-    Tuple,
-    Type,
-    Union,
-    get_type_hints,
-)
+import warnings
+from typing import Any, Callable, Sequence, Type, get_type_hints
 
-from tqdm import tqdm
+from tenacity import RetryError
 
 from ragelo.evaluators.base_evaluator import BaseEvaluator
 from ragelo.llm_providers.base_llm_provider import BaseLLMProvider, get_llm_provider
 from ragelo.logger import logger
-from ragelo.types import (
-    AgentAnswer,
-    AnswerEvaluatorResult,
-    AnswerEvaluatorTypes,
-    AnswerFormat,
+from ragelo.types.configurations import (
     BaseAnswerEvaluatorConfig,
-    Document,
     PairwiseEvaluatorConfig,
-    PairwiseGame,
-    Query,
 )
+from ragelo.types.evaluables import AgentAnswer, Document, Evaluable, PairwiseGame
+from ragelo.types.experiment import Experiment
+from ragelo.types.query import Query
+from ragelo.types.results import AnswerEvaluatorResult
+from ragelo.types.types import AnswerEvaluatorTypes
+from ragelo.utils import call_async_fn
 
 
 class BaseAnswerEvaluator(BaseEvaluator):
     config: BaseAnswerEvaluatorConfig
+    evaluable_name: str = "Agent Answer"
 
     def __init__(
         self,
@@ -45,242 +35,250 @@ def __init__(
     ):
         self.config = config
         self.llm_provider = llm_provider
-        self.output_columns = (
-            config.output_columns_answer_evaluator
-            if not self.config.pairwise
-            else config.output_columns_pairwise_evaluator
-        )
-        self.scoring_keys = config.scoring_keys_answer_evaluator
-        self.scoring_key = config.scoring_key_answer_evaluator
-        if isinstance(self.config.answer_format_answer_evaluator, str):
-            self.answer_format = AnswerFormat(
-                self.config.answer_format_answer_evaluator
-            )
-        else:
-            self.answer_format = self.config.answer_format_answer_evaluator
-
-        if self.answer_format == AnswerFormat.MULTI_FIELD_JSON:
-            missing_keys = [
-                key for key in self.scoring_keys if key not in self.output_columns
-            ]
-            self.output_columns.extend(missing_keys)
-        else:
-            if self.scoring_key not in self.output_columns:
-                logger.info(f"Adding scoring key {self.scoring_key} to output columns")
-                self.output_columns.append(self.scoring_key)
-
-        self.pairwise = self.config.pairwise
-
-    async def _async_batch_evaluate(self, queries: List[Query]) -> List[Query]:
-        use_progress_bar = self.config.use_progress_bar
-        failed_queries = 0
-        queries = self.__prepare_queries(queries)
-        tuples_to_eval = self.__get_tuples_to_evaluate(queries)
-
-        if len(tuples_to_eval) == 0:
-            return queries
-        if self.config.rich_print:
-            import warnings
-
-            from tqdm import TqdmExperimentalWarning
-
-            warnings.filterwarnings("ignore", category=TqdmExperimentalWarning)
-            from tqdm.rich import tqdm as rich_tqdm
-
-            pbar_fn = rich_tqdm  # type: ignore
-        else:
-            pbar_fn = tqdm  # type: ignore
-        pbar = pbar_fn(
-            total=len(tuples_to_eval),
-            ncols=100,
-            desc="Evaluating answers",
-            disable=not use_progress_bar,
-            leave=False,
-            position=0,
-        )
-
-        awaitables_ended = False
-        pending: Set[asyncio.Future] = set()
-        aws = map(self._async_evaluate, tuples_to_eval)
-        aws = iter(aws)
-        evaluations = []
-        # while there are pending tasks or not all tasks are done
-        while pending or not awaitables_ended:
-            # while there are less than n_processes pending tasks
-            while len(pending) < self.config.n_processes and not awaitables_ended:
-                try:
-                    aw = next(aws)
-                except StopIteration:
-                    awaitables_ended = True  # all tasks have been scheduled
-                else:
-                    pending.add(asyncio.ensure_future(aw))
-            if not pending:
-                break
-
-            done, pending = await asyncio.wait(
-                pending, return_when=asyncio.FIRST_COMPLETED
-            )
-            while done:
-                evaluation = await done.pop()
-                pbar.update()
-                if evaluation.exception:
-                    failed_queries += 1
-                    continue
-                evaluations.append(evaluation)
-        pbar.close()
-        self._add_answers_evaluations(queries, evaluations)
-        if self.config.verbose:
-            self._print_failed_evaluations(len(evaluations), failed_queries)
-        return queries
 
     def evaluate(
         self,
-        query: Union[Query, str],
-        answer: Optional[Union[AgentAnswer, str]] = None,
-        answer_a: Optional[Union[AgentAnswer, str]] = None,
-        answer_b: Optional[Union[AgentAnswer, str]] = None,
-        retrieved_documents: Optional[Union[List[str], List[Document]]] = None,
-        document_metadata: Optional[List[Dict[str, Any]]] = None,
-        query_metadata: Optional[Dict[str, Any]] = None,
-        answer_metadata=None,
-        answer_a_metadata: Optional[Dict[str, Any]] = None,
-        answer_b_metadata: Optional[Dict[str, Any]] = None,
-    ) -> Tuple[str, Any]:
-        query = self._assemble_query(query, query_metadata)
+        query: Query | str,
+        answer: AgentAnswer | str | None = None,
+        answer_a: AgentAnswer | str | None = None,
+        answer_b: AgentAnswer | str | None = None,
+        retrieved_documents: list[Document | str] | str | None = None,
+        document_metadata: list[dict[str, Any]] | None = None,
+        query_metadata: dict[str, Any] | None = None,
+        answer_metadata: dict[str, Any] | None = None,
+        answer_a_metadata: dict[str, Any] | None = None,
+        answer_b_metadata: dict[str, Any] | None = None,
+    ) -> AnswerEvaluatorResult:
+        """
+        Evaluates a single sample using an answer evaluator.
+        Args:
+            query(Query|str): The query to evaluate.
+                If a string is provided, a Query object will be created with the provided query_metadata.
+            answer(Optional[AgentAnswer|str]): The answer generated by an agent to evaluate.
+                If a string is provided, the AgentAnswer object will be created with the provided answer_metadata
+            answer_a(Optional[AgentAnswer|str]): The first answer generated by an agent to evaluate.
+                Only used if the evaluator is pairwise.
+            answer_b(Optional[AgentAnswer|str]): The second answer generated by an agent to evaluate.
+                Only used if the evaluator is pairwise.
+            retrieved_documents(Optional[list[str]|list[Document]]): The documents retrieved by the agent.
+            document_metadata(Optional[list[dict[str, Any]]): The metadata of the documents retrieved by the agent.
+            query_metadata(Optional[dict[str, Any]]): The metadata of the query.
+            answer_metadata(Optional[dict[str, Any]]): The metadata of the answer.
+            answer_a_metadata(Optional[dict[str, Any]]): The metadata of the first answer.
+                Only used if the evaluator is pairwise.
+            answer_b_metadata(Optional[dict[str, Any]]): The metadata of the second answer.
+                Only used if the evaluator is pairwise.
+        Returns:
+            AnswerEvaluatorResult: The result of the evaluation.
+        """
+        query = Query.assemble_query(query, query_metadata)
+
         if isinstance(retrieved_documents, str):
             retrieved_documents = [retrieved_documents]
+
         if retrieved_documents:
-            retrieved_and_assembled_docs = self._assemble_documents(
-                retrieved_documents, document_metadata
+            retrieved_and_assembled_docs = Document.assemble_documents(
+                retrieved_documents, query.qid, document_metadata
             )
             query.retrieved_docs = retrieved_and_assembled_docs
-        agent: Union[str, Tuple[str, str]]
-        if self.pairwise:
+
+        agent: str | tuple[str, str]
+        evaluable: AgentAnswer | PairwiseGame
+
+        if self.config.pairwise:
             if not answer_a or not answer_b:
                 raise ValueError("Pairwise evaluations require two answers")
-            answer_a = self._assemble_answer(answer_a, answer_a_metadata)
-            answer_b = self._assemble_answer(answer_b, answer_b_metadata)
+            answer_a = AgentAnswer.assemble_answer(answer_a, query.qid, metadata=answer_a_metadata)
+            answer_b = AgentAnswer.assemble_answer(answer_b, query.qid, metadata=answer_b_metadata)
             agent = (answer_a.agent, answer_b.agent)
-            game = PairwiseGame(
+            evaluable = PairwiseGame(
+                qid=query.qid,
                 agent_a_answer=answer_a,
                 agent_b_answer=answer_b,
             )
-
-            def run(coroutine):
-                return asyncio.run(coroutine)
-
-            try:
-                asyncio.get_running_loop()
-                with ThreadPoolExecutor(max_workers=1) as executor:
-                    future = executor.submit(run, self._async_evaluate((query, game)))
-                    result = future.result()
-            except RuntimeError:
-                result = asyncio.run(self._async_evaluate((query, game)))
         else:
             if not answer:
                 raise ValueError("Pointwise evaluations require an answer")
-            answer = self._assemble_answer(answer, answer_metadata)
-            agent = answer.agent
-
-            def run(coroutine):
-                return asyncio.run(coroutine)
-
-            try:
-                asyncio.get_running_loop()
-                with ThreadPoolExecutor(max_workers=1) as executor:
-                    future = executor.submit(run, self._async_evaluate((query, answer)))
-                    result = future.result()
-            except RuntimeError:
-                result = asyncio.run(self._async_evaluate((query, answer)))
+            evaluable = AgentAnswer.assemble_answer(answer, query.qid, metadata=answer_metadata)
+            agent = evaluable.agent
+        result = call_async_fn(self.evaluate_async, (query, evaluable))
+
         if result.exception or result.raw_answer is None or result.answer is None:
             raise ValueError(
                 f"Failed to evaluate qid: {query.qid} agent(s): {agent}",
                 f"Exception: {result.exception}",
             )
-        return result.raw_answer, result.answer
+        return result
 
-    async def _async_evaluate(
-        self, eval_sample: Tuple[Query, Union[PairwiseGame, AgentAnswer]]
-    ) -> AnswerEvaluatorResult:
+    def evaluate_experiment(self, experiment: Experiment, n_threads: int | None = None):
+        """
+        Trigger the evaluation of all the queries in the experiment.
+        The evaluation is done in asynchronously with the number of threads defined in config.n_processes parameter.
+        This can be overwritten by the n_threads parameter.
+
+        Args:
+            experiment(Experiment): The experiment to evaluate.
+            n_threads(int): The number of threads to use for the evaluation.
+                If None, the number of threads defined in the config will be used.
+        """
+        if self.config.pairwise:
+            self.__add_pairwise_games(experiment)
+        super().evaluate_experiment(experiment, n_threads)
+
+    async def evaluate_async(self, eval_sample: tuple[Query, Evaluable]) -> AnswerEvaluatorResult:
+        """
+        Evaluates a single sample asynchronously
+        """
+        agent: str | tuple[str, str]
         query, evaluable = eval_sample
-        agent: Union[str, Tuple[str, str]]
+
+        if not isinstance(evaluable, AgentAnswer) and not isinstance(evaluable, PairwiseGame):
+            type_name = type(evaluable).__name__
+            raise ValueError(f"can't evaluate a {type_name} in an Answer Evaluator")
+
         if evaluable.evaluation is not None and not self.config.force:
-            return evaluable.evaluation  # type: ignore
+            if isinstance(evaluable, AgentAnswer):
+                return AnswerEvaluatorResult(
+                    qid=query.qid,
+                    agent=evaluable.agent,
+                    raw_answer=evaluable.evaluation.raw_answer,
+                    answer=evaluable.evaluation.answer,
+                    pairwise=False,
+                )
+            return AnswerEvaluatorResult(
+                qid=query.qid,
+                agent_a=evaluable.agent_a_answer.agent,
+                agent_b=evaluable.agent_b_answer.agent,
+                raw_answer=evaluable.evaluation.raw_answer,
+                answer=evaluable.evaluation.answer,
+                pairwise=True,
+            )
+
         if isinstance(evaluable, AgentAnswer):
             agent = evaluable.agent
+            prompt = self._build_message(query, evaluable)
         elif isinstance(evaluable, PairwiseGame):
             agent = (evaluable.agent_a_answer.agent, evaluable.agent_b_answer.agent)
+            prompt = self._build_message_pairwise(query, evaluable)
         else:
             raise ValueError(f"Unknown evaluable type {type(evaluable)}")
 
         exc = None
-        if isinstance(evaluable, AgentAnswer):
-            prompt = self._build_message(query, evaluable)
-        else:
-            prompt = self._build_message_pairwise(query, evaluable)
-        try:
-            raw_answer = await self.llm_provider.call_async(prompt)
-        except Exception as e:
-            logger.warning(f"Failed to FETCH answers for qid: {query.qid}")
-            logger.warning(f"agent(s): {agent}")
-            exc = str(e)
-            raw_answer = None
         try:
-            answer = self._process_answer(raw_answer) if raw_answer else None
+            llm_response = await self.llm_provider.call_async(
+                prompt,
+                answer_format=self.config.llm_answer_format,
+                response_schema=self.config.llm_response_schema,
+            )
+            llm_response = self._process_answer(llm_response)
         except ValueError as e:
             logger.warning(
                 f"Failed to PARSE answer for qid: {query.qid} agent(s): {agent}\n"
-                f"Raw answer: {raw_answer}"
+                f"Raw answer: {llm_response.raw_answer}"
             )
             exc = str(e)
-            answer = None
+        except Exception as e:
+            logger.warning(f"Failed to FETCH answers for qid: {query.qid}")
+            logger.warning(f"agent(s): {agent}")
+            if isinstance(e, RetryError):
+                exc = str(e.last_attempt.exception())
+            else:
+                exc = str(e)
         if isinstance(evaluable, AgentAnswer):
-            output_file = self.config.answers_evaluations_file
-            ans = AnswerEvaluatorResult(
+            return AnswerEvaluatorResult(
                 qid=query.qid,
                 agent=evaluable.agent,
-                raw_answer=raw_answer,
-                answer=answer,
+                raw_answer=llm_response.raw_answer,
+                answer=llm_response.parsed_answer,
                 pairwise=False,
                 exception=exc,
             )
-        else:
-            output_file = self.config.games_evaluations_file
-            ans = AnswerEvaluatorResult(
-                qid=query.qid,
-                agent_a=evaluable.agent_a_answer.agent,
-                agent_b=evaluable.agent_b_answer.agent,
-                raw_answer=raw_answer,
-                answer=answer,
-                pairwise=True,
-                exception=exc,
-            )
-        if ans.exception is None:
-            self._dump_response(ans, self.output_columns, output_file)
-        return ans
+        assert isinstance(evaluable, PairwiseGame)
+        return AnswerEvaluatorResult(
+            qid=query.qid,
+            agent_a=evaluable.agent_a_answer.agent,
+            agent_b=evaluable.agent_b_answer.agent,
+            raw_answer=llm_response.raw_answer,
+            answer=llm_response.parsed_answer,
+            pairwise=True,
+            exception=exc,
+        )
+
+    def _get_tuples_to_evaluate(self, experiment: Experiment) -> Sequence[tuple[Query, Evaluable]]:
+        """
+        Creates the list of pairs (query, evaluable) to evaluate
+        """
+        tuples_to_eval: list[tuple[Query, PairwiseGame | AgentAnswer]] = []
+        all_tuples = 0
+        missing_evaluations = 0
+        for q in experiment:
+            if self.config.pairwise:
+                for g in q.pairwise_games:
+                    all_tuples += 1
+                    tuples_to_eval.append((q, g))
+                    if g.evaluation is None:
+                        missing_evaluations += 1
+
+            else:
+                for a in q.answers.values():
+                    all_tuples += 1
+                    tuples_to_eval.append((q, a))
+                    if a.evaluation is None:
+                        missing_evaluations += 1
+
+        if missing_evaluations == 0:
+            logger.info("All answers have been evaluated")
+            if self.config.verbose and not self.config.force:
+                logger.warning(
+                    f"All {all_tuples} answers are already evaluated.\n"
+                    "If you want to re-evaluate them, use the --force flag"
+                )
+
+        return tuples_to_eval
 
-    def _build_message(
-        self, query: Query, answer: AgentAnswer
-    ) -> Union[str, List[Dict[str, str]]]:
+    def _build_message(self, query: Query, answer: AgentAnswer) -> str | list[dict[str, str]]:
         """Builds the message to send to the LLM evaluator"""
         raise NotImplementedError
 
-    def _build_message_pairwise(
-        self, query: Query, game: PairwiseGame
-    ) -> Union[str, List[Dict[str, str]]]:
+    def _build_message_pairwise(self, query: Query, game: PairwiseGame) -> str | list[dict[str, str]]:
         """Builds the message to send to the LLM evaluator"""
         raise NotImplementedError
 
     @classmethod
-    def from_config(
-        cls, config: BaseAnswerEvaluatorConfig, llm_provider: BaseLLMProvider
-    ):
+    def from_config(cls, config: BaseAnswerEvaluatorConfig, llm_provider: BaseLLMProvider):
         return cls(config, llm_provider)
 
     @classmethod
     def get_config_class(cls) -> Type[BaseAnswerEvaluatorConfig]:
         return get_type_hints(cls)["config"]
 
+    def __add_pairwise_games(self, experiment: Experiment):
+        if not self.config.pairwise:
+            return
+        if not isinstance(self.config, PairwiseEvaluatorConfig):
+            raise ValueError("Trying to add pairwise games to a non-pairwise evaluator")
+        for query in experiment:
+            query_agents = list(query.answers.keys())
+            pairs = list(itertools.combinations(query_agents, 2))
+            if self.config.bidirectional:
+                pairs += [(b, a) for a, b in pairs]
+            random.shuffle(pairs)
+
+            # Filter out games that already exist
+            existing_games = {(a.agent_a_answer.agent, a.agent_b_answer.agent) for a in query.pairwise_games}
+            games = [g for g in pairs if g not in existing_games]
+
+            games_to_add = min(self.config.n_games_per_query - len(existing_games), len(games))
+            games = games[:games_to_add]
+            for agent_a, agent_b in games:
+                query.pairwise_games.append(
+                    PairwiseGame(
+                        qid=query.qid,
+                        agent_a_answer=query.answers[agent_a],
+                        agent_b_answer=query.answers[agent_b],
+                    )
+                )
+
     def _prepare_documents(self, query: Query) -> str:
         documents = []
         for did, d in query.retrieved_docs.items():
@@ -301,6 +299,11 @@ def _prepare_documents(self, query: Query) -> str:
                     continue
                 if not self.config.document_filter(str(d.evaluation.answer)):
                     continue
+            if self.config.include_annotations and d.evaluation is None:
+                continue
+            if self.config.include_raw_documents and d.text is None:
+                continue
+
             formatters = {
                 "did": did,
                 "doc": d.text,
@@ -308,100 +311,13 @@ def _prepare_documents(self, query: Query) -> str:
             }
             documents.append(self.config.document_template.format(**formatters))
         if len(documents) == 0:
+            warnings.warn("No documents were retrieved for the answer evaluator")
             return "NO DOCUMENTS WERE RETRIEVED"
         return "\n".join(documents)
 
-    def __prepare_queries(self, queries: List[Query]) -> List[Query]:
-        queries = self._load_retrieved_documents(queries)
-        queries = self._load_document_evaluations(queries, force=False)
-        queries = self._load_agent_answers(queries)
-        queries = self.__add_pairwise_games(queries)
-        queries = self._load_answers_evaluations(queries, force=self.config.force)
-        return queries
-
-    def __add_pairwise_games(self, queries: List[Query]) -> List[Query]:
-        if not self.pairwise:
-            return queries
-        for query in queries:
-            query_agents = list(query.answers.keys())
-            pairs = list(itertools.combinations(query_agents, 2))
-            if not isinstance(self.config, PairwiseEvaluatorConfig):
-                raise ValueError(
-                    "You are trying to generate pairwise games for a pointwise evaluator"
-                )
-            if self.config.bidirectional:
-                pairs += [(b, a) for a, b in pairs]
-            random.shuffle(pairs)
-
-            # Filter out games that already exist
-            existing_games = {
-                (a.agent_a_answer.agent, a.agent_b_answer.agent)
-                for a in query.pairwise_games
-            }
-            games = [g for g in pairs if g not in existing_games]
-
-            games_to_add = self.config.n_games_per_query - len(existing_games)
-            games = games[:games_to_add]
-            for agent_a, agent_b in games:
-                query.pairwise_games.append(
-                    PairwiseGame(
-                        agent_a_answer=query.answers[agent_a],
-                        agent_b_answer=query.answers[agent_b],
-                    )
-                )
-        return queries
-
-    def __get_tuples_to_evaluate(
-        self, queries: List[Query]
-    ) -> List[Tuple[Query, Union[PairwiseGame, AgentAnswer]]]:
-        tuples_to_eval: List[Tuple[Query, Union[PairwiseGame, AgentAnswer]]] = []
-        all_tuples = 0
-        missing_evaluations = 0
-        for q in queries:
-            if self.pairwise:
-                for g in q.pairwise_games:
-                    all_tuples += 1
-                    tuples_to_eval.append((q, g))
-                    if g.evaluation is None:
-                        missing_evaluations += 1
-
-            else:
-                for a in q.answers.values():
-                    all_tuples += 1
-                    tuples_to_eval.append((q, a))
-                    if a.evaluation is None:
-                        missing_evaluations += 1
-
-        if missing_evaluations == all_tuples:
-            logger.info("All answers have been evaluated")
-            if self.config.verbose and not self.config.force:
-                print(
-                    f"All {all_tuples} answers are already evaluated.\n"
-                    "If you want to re-evaluate them, use the force flag"
-                )
-
-        return tuples_to_eval
-
-    def batch_evaluate(self, queries: List[Query]) -> List[Query]:
-        def run(coroutine):
-            return asyncio.run(coroutine)
-
-        try:
-            # Raises RuntimeError if there is no current event loop.
-            asyncio.get_running_loop()
-            # If there is a current event loop, we need to run the async code
-            # in a separate loop, in a separate thread.
-            with ThreadPoolExecutor(max_workers=1) as executor:
-                future = executor.submit(run, self._async_batch_evaluate(queries))
-                result = future.result()
-        except RuntimeError:
-            result = asyncio.run(self._async_batch_evaluate(queries))
-
-        return result
-
 
 class AnswerEvaluatorFactory:
-    registry: Dict[AnswerEvaluatorTypes, Type[BaseAnswerEvaluator]] = {}
+    registry: dict[AnswerEvaluatorTypes, Type[BaseAnswerEvaluator]] = {}
 
     @classmethod
     def register(cls, name: AnswerEvaluatorTypes) -> Callable:
@@ -417,12 +333,14 @@ def inner_wrapper(wrapped_class: Type[BaseAnswerEvaluator]):
     def create(
         cls,
         evaluator_name: AnswerEvaluatorTypes,
-        llm_provider: Union[BaseLLMProvider, str],
-        config: Optional[BaseAnswerEvaluatorConfig] = None,
+        llm_provider: BaseLLMProvider | str,
+        config: BaseAnswerEvaluatorConfig | None = None,
         **kwargs,
     ) -> BaseAnswerEvaluator:
         if evaluator_name not in cls.registry:
-            raise ValueError(f"Unknown evaluator {evaluator_name}")
+            raise ValueError(
+                f"Unknown answer evaluator {evaluator_name}\n" f"Valid options are {list(cls.registry.keys())}"
+            )
         if isinstance(llm_provider, str):
             llm_provider_instance = get_llm_provider(llm_provider, **kwargs)
         else:
@@ -432,25 +350,32 @@ def create(
             type_config = class_.get_config_class()
             valid_keys = [field for field in type_config.get_model_fields()]
             valid_args = {k: v for k, v in kwargs.items() if k in valid_keys}
+            required_fields = [arg for arg, info in type_config.get_model_fields().items() if info.is_required()]
+            for field in required_fields:
+                if field not in valid_args:
+                    raise ValueError(f"Required argument {field} for evaluator {evaluator_name} not provided")
             config = type_config(**valid_args)
         return cls.registry[evaluator_name].from_config(config, llm_provider_instance)
 
 
 def get_answer_evaluator(
-    evaluator_name: Optional[Union[AnswerEvaluatorTypes, str]] = None,
-    llm_provider: Union[BaseLLMProvider, str] = "openai",
-    config: Optional[BaseAnswerEvaluatorConfig] = None,
+    evaluator_name: AnswerEvaluatorTypes | str | None = None,
+    llm_provider: BaseLLMProvider | str = "openai",
+    config: BaseAnswerEvaluatorConfig | None = None,
     **kwargs,
 ) -> BaseAnswerEvaluator:
     if evaluator_name is None:
         # get the name from the config
         if config is None:
-            raise ValueError(
-                "Either the evaluator_name or a config object must be provided"
-            )
+            raise ValueError("Either the evaluator_name or a config object must be provided")
         evaluator_name = config.evaluator_name
     if isinstance(evaluator_name, str):
-        evaluator_name = AnswerEvaluatorTypes(evaluator_name)
+        try:
+            evaluator_name = AnswerEvaluatorTypes(evaluator_name)
+        except ValueError:
+            raise ValueError(f"Unknown answer evaluator {evaluator_name}")
+    if evaluator_name is None:
+        raise ValueError("The evaluator_name must be provided")
     return AnswerEvaluatorFactory.create(
         evaluator_name,
         llm_provider=llm_provider,
diff --git a/ragelo/evaluators/answer_evaluators/chat_pairwise_evaluator.py b/ragelo/evaluators/answer_evaluators/chat_pairwise_evaluator.py
index 74f44b3..e72a50e 100644
--- a/ragelo/evaluators/answer_evaluators/chat_pairwise_evaluator.py
+++ b/ragelo/evaluators/answer_evaluators/chat_pairwise_evaluator.py
@@ -1,7 +1,6 @@
 """Answer Evaluator for conversations between two agents."""
 
-
-from typing import Dict, List, Union
+from __future__ import annotations
 
 from ragelo.evaluators.answer_evaluators.base_answer_evaluator import (
     AnswerEvaluatorFactory,
@@ -9,8 +8,10 @@
 from ragelo.evaluators.answer_evaluators.pairwise_evaluator import (
     PairwiseAnswerEvaluator,
 )
-from ragelo.types import AnswerEvaluatorTypes, PairwiseGame, Query
 from ragelo.types.configurations import PairwiseEvaluatorConfig
+from ragelo.types.evaluables import PairwiseGame
+from ragelo.types.query import Query
+from ragelo.types.types import AnswerEvaluatorTypes
 
 
 @AnswerEvaluatorFactory.register(AnswerEvaluatorTypes.CHAT_PAIRWISE)
@@ -34,8 +35,8 @@ class ChatPairwiseEvaluator(PairwiseAnswerEvaluator):
 the conversations were presented does not influence your decision. Do not allow the \
 length of the responses to influence your evaluation. Be as objective as possible.
 After providing your explanation, output your final verdict by strictly following \
-this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better, \
-and "[[C]]" for a tie.
+this format: 'A' if assistant A is better, 'B' if assistant B is better, \
+and 'C' for a tie.
 
 [User Intent]
 {query}
@@ -55,9 +56,7 @@ class ChatPairwiseEvaluator(PairwiseAnswerEvaluator):
 [The End of Conversation with Assistant B]
 """.strip()
 
-    def _build_message_pairwise(
-        self, query: Query, game: PairwiseGame
-    ) -> Union[str, List[Dict[str, str]]]:
+    def _build_message_pairwise(self, query: Query, game: PairwiseGame) -> str | list[dict[str, str]]:
         documents = self._prepare_documents(query)
         query_metadata = self._get_usable_fields_from_metadata(
             self.prompt, query.metadata, skip_fields=[self.config.query_placeholder]
@@ -78,15 +77,14 @@ def _build_message_pairwise(
             citations = ""
 
         if not game.agent_a_answer.conversation or not game.agent_b_answer.conversation:
-            raise ValueError(
-                "The conversation of the agents cannot "
-                "be empty for the chat_pairwise evaluator"
-            )
+            raise ValueError("The conversation of the agents cannot be empty for the chat_pairwise evaluator")
+        conversation_a = "\n".join([str(msg) for msg in game.agent_a_answer.conversation])
+        conversation_b = "\n".join([str(msg) for msg in game.agent_b_answer.conversation])
         formatters = {
             self.config.query_placeholder: query.query,
             self.config.documents_placeholder: documents,
-            "answer_a": game.agent_a_answer.conversation,
-            "answer_b": game.agent_b_answer.conversation,
+            "answer_a": conversation_a,
+            "answer_b": conversation_b,
             "citations": citations,
             "factors": self.factors,
             "document_rel": self.documents_prompt,
diff --git a/ragelo/evaluators/answer_evaluators/custom_pairwise_evaluator.py b/ragelo/evaluators/answer_evaluators/custom_pairwise_evaluator.py
new file mode 100644
index 0000000..44300e5
--- /dev/null
+++ b/ragelo/evaluators/answer_evaluators/custom_pairwise_evaluator.py
@@ -0,0 +1,61 @@
+from __future__ import annotations
+
+from ragelo.evaluators.answer_evaluators.base_answer_evaluator import (
+    AnswerEvaluatorFactory,
+    BaseAnswerEvaluator,
+)
+from ragelo.llm_providers.base_llm_provider import BaseLLMProvider
+from ragelo.types.configurations import CustomPairwiseEvaluatorConfig
+from ragelo.types.evaluables import PairwiseGame
+from ragelo.types.query import Query
+from ragelo.types.types import AnswerEvaluatorTypes
+
+
+@AnswerEvaluatorFactory.register(AnswerEvaluatorTypes.CUSTOM_PAIRWISE)
+class CustomPairwiseEvaluator(BaseAnswerEvaluator):
+    """A custom pairwise evaluator that allows for additional customization."""
+
+    config: CustomPairwiseEvaluatorConfig
+    system_prompt: str
+    user_prompt: str
+
+    def __init__(
+        self,
+        config: CustomPairwiseEvaluatorConfig,
+        llm_provider: BaseLLMProvider,
+    ):
+        super().__init__(config, llm_provider)
+        self.system_prompt = config.system_prompt
+        self.user_prompt = config.user_prompt
+
+    def _build_message_pairwise(self, query: Query, game: PairwiseGame) -> str | list[dict[str, str]]:
+        system_prompt_msg = {"role": "system", "content": self.system_prompt}
+        messages = [system_prompt_msg]
+        documents = self._prepare_documents(query)
+        query_metadata = self._get_usable_fields_from_metadata(
+            self.user_prompt,
+            query.metadata,
+            skip_fields=[self.config.query_placeholder],
+        )
+        answer_a_metadata = self._get_usable_fields_from_metadata(
+            self.user_prompt,
+            game.agent_a_answer.metadata,
+            skip_fields=[self.config.answer_placeholder],
+        )
+        answer_b_metadata = self._get_usable_fields_from_metadata(
+            self.user_prompt,
+            game.agent_b_answer.metadata,
+            skip_fields=[self.config.answer_placeholder],
+        )
+        formatters = {
+            self.config.query_placeholder: query.query,
+            self.config.documents_placeholder: documents,
+            "answer_a": game.agent_a_answer.text,
+            "answer_b": game.agent_b_answer.text,
+            **query_metadata,
+            **answer_a_metadata,
+            **answer_b_metadata,
+        }
+        user_message = self.user_prompt.format(**formatters)
+        messages.append({"role": "user", "content": user_message})
+        return messages
diff --git a/ragelo/evaluators/answer_evaluators/custom_prompt_evaluator.py b/ragelo/evaluators/answer_evaluators/custom_prompt_evaluator.py
index 965875c..28eb727 100644
--- a/ragelo/evaluators/answer_evaluators/custom_prompt_evaluator.py
+++ b/ragelo/evaluators/answer_evaluators/custom_prompt_evaluator.py
@@ -1,16 +1,19 @@
+from __future__ import annotations
+
 from ragelo.evaluators.answer_evaluators.base_answer_evaluator import (
     AnswerEvaluatorFactory,
     BaseAnswerEvaluator,
 )
 from ragelo.llm_providers.base_llm_provider import BaseLLMProvider
-from ragelo.types import AgentAnswer, AnswerEvaluatorTypes, Query
 from ragelo.types.configurations import CustomPromptAnswerEvaluatorConfig
+from ragelo.types.evaluables import AgentAnswer
+from ragelo.types.query import Query
+from ragelo.types.types import AnswerEvaluatorTypes
 
 
 @AnswerEvaluatorFactory.register(AnswerEvaluatorTypes.CUSTOM_PROMPT)
 class CustomPromptEvaluator(BaseAnswerEvaluator):
     config: CustomPromptAnswerEvaluatorConfig
-    output_file: str = "custom_prompt_evaluations.csv"
 
     def __init__(
         self,
diff --git a/ragelo/evaluators/answer_evaluators/domain_expert_evaluator.py b/ragelo/evaluators/answer_evaluators/domain_expert_evaluator.py
index d021740..4a72c7a 100644
--- a/ragelo/evaluators/answer_evaluators/domain_expert_evaluator.py
+++ b/ragelo/evaluators/answer_evaluators/domain_expert_evaluator.py
@@ -1,6 +1,6 @@
 """Answer Evaluator with a domain expert persona"""
 
-from typing import Dict, List, Union
+from __future__ import annotations
 
 from ragelo.evaluators.answer_evaluators.base_answer_evaluator import (
     AnswerEvaluatorFactory,
@@ -9,15 +9,17 @@
     PairwiseAnswerEvaluator,
 )
 from ragelo.llm_providers.base_llm_provider import BaseLLMProvider
-from ragelo.types import AnswerEvaluatorTypes, PairwiseGame, Query
 from ragelo.types.configurations import PairwiseDomainExpertEvaluatorConfig
+from ragelo.types.evaluables import PairwiseGame
+from ragelo.types.query import Query
+from ragelo.types.types import AnswerEvaluatorTypes
 
 
 @AnswerEvaluatorFactory.register(AnswerEvaluatorTypes.DOMAIN_EXPERT)
 class PairwiseDomainExpertEvaluator(PairwiseAnswerEvaluator):
     config: PairwiseDomainExpertEvaluatorConfig
     prompt = """
-{company_prompt}You are a domain expert in {expert_in}. Your task is to \
+You are a domain expert in {expert_in}.{company_prompt} Your task is to \
 evaluate the quality of the responses provided by two AI assistants \
 tasked to answer the question shown below, based on a set \
 of documents retrieved by a search engine.
@@ -34,8 +36,8 @@ class PairwiseDomainExpertEvaluator(PairwiseAnswerEvaluator):
 the responses were presented does not influence your decision. Do not allow the \
 length of the responses to influence your evaluation. Be as objective as possible.
 After providing your explanation, output your final verdict by strictly following \
-this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better, \
-and "[[C]]" for a tie.
+this format: 'A' if assistant A is better, 'B' if assistant B is better, \
+and 'C' for a tie.
 
 [User Question]
 {query}
@@ -59,18 +61,10 @@ def __init__(
         llm_provider: BaseLLMProvider,
     ):
         super().__init__(config, llm_provider)
-        if not self.config.expert_in:
-            raise ValueError(
-                "You are trying to use the Domain Expert Answer Evaluator. "
-                "For this evaluator, you need to provide the domain the evaluator "
-                "is an expert in the expert_in field."
-            )
         self.expert_in = self.config.expert_in
         self.company = self.config.company
 
-    def _build_message_pairwise(
-        self, query: Query, game: PairwiseGame
-    ) -> Union[str, List[Dict[str, str]]]:
+    def _build_message_pairwise(self, query: Query, game: PairwiseGame) -> str | list[dict[str, str]]:
         documents = self._prepare_documents(query)
         query_metadata = self._get_usable_fields_from_metadata(
             self.prompt, query.metadata, skip_fields=[self.config.query_placeholder]
@@ -103,7 +97,7 @@ def _build_message_pairwise(
             **answer_b_metadata,
         }
         if self.company:
-            formatters["company_prompt"] = self.COMPANY_PROMPT.format(
-                company=self.company
-            )
+            formatters["company_prompt"] = self.COMPANY_PROMPT.format(company=self.company)
+        else:
+            formatters["company_prompt"] = ""
         return self.prompt.format(**formatters)
diff --git a/ragelo/evaluators/answer_evaluators/pairwise_evaluator.py b/ragelo/evaluators/answer_evaluators/pairwise_evaluator.py
index 051c18c..3081c8d 100644
--- a/ragelo/evaluators/answer_evaluators/pairwise_evaluator.py
+++ b/ragelo/evaluators/answer_evaluators/pairwise_evaluator.py
@@ -1,26 +1,36 @@
-import re
-from typing import Dict, List, Union
+from __future__ import annotations
+
+from typing import Literal
+
+from pydantic import BaseModel as PydanticBaseModel
+from pydantic import Field
 
 from ragelo.evaluators.answer_evaluators.base_answer_evaluator import (
     AnswerEvaluatorFactory,
     BaseAnswerEvaluator,
 )
 from ragelo.llm_providers.base_llm_provider import BaseLLMProvider
-from ragelo.types import AnswerEvaluatorTypes, PairwiseGame, Query
+from ragelo.logger import logger
 from ragelo.types.configurations import PairwiseEvaluatorConfig
+from ragelo.types.evaluables import PairwiseGame
+from ragelo.types.formats import AnswerFormat, LLMResponseType
+from ragelo.types.query import Query
+from ragelo.types.types import AnswerEvaluatorTypes
+
+
+class PairWiseAnswerAnswerFormat(PydanticBaseModel):
+    winner: Literal["A", "B", "C"] = Field(..., description="The winner of the pairwise comparison.")
 
 
 @AnswerEvaluatorFactory.register(AnswerEvaluatorTypes.PAIRWISE)
 class PairwiseAnswerEvaluator(BaseAnswerEvaluator):
-    """A evaluator that evaluates RAG-based answers pairwise, with document reasoning and citations."""
+    """An evaluator that evaluates RAG-based answers pairwise, with document reasoning and citations."""
 
     config: PairwiseEvaluatorConfig
     citations_prompt = " Answers cite documents using square brackets."
     document_template_raw_only = "[{did}] {doc}"
     document_template_annotation_only = "[{did}] {annotation}"
-    document_template_raw_and_annotation = (
-        "[RETRIEVED DOCUMENT]\n{doc}\n[DOCUMENT RELEVANCE]\n{annotation}\n"
-    )
+    document_template_raw_and_annotation = "[RETRIEVED DOCUMENT]\n{doc}\n[DOCUMENT RELEVANCE]\n{annotation}\n"
     documents_prompt_relevance_only = (
         "For each reference document, you will be provided with a reasoning "
         "explaining why the document is or is not relevant."
@@ -30,9 +40,7 @@ class PairwiseAnswerEvaluator(BaseAnswerEvaluator):
         "of the document as well as a reasoning  why the document "
         "is or is not relevant."
     )
-    documents_prompt_raw_only = (
-        "You will be provided with the text of each reference document."
-    )
+    documents_prompt_raw_only = "You will be provided with the text of each reference document."
 
     prompt = """
 Please act as an impartial judge and evaluate the quality of the responses provided \
@@ -45,14 +53,14 @@ class PairwiseAnswerEvaluator(BaseAnswerEvaluator):
 Details are only useful if they answer the user question. If an answer \
 contains non-relevant details, it should not be preferred over one that only \
 use relevant information.
-Begin your evaluation by explaining why each answer correctly answers the user \
+Begin your evaluation by explaining whether or nor each answer correctly answers the user \
 question. Then, you should compare the two responses and provide a short explanation \
 on their differences. Avoid any position biases and ensure that the order in which \
 the responses were presented does not influence your decision. Do not allow the \
 length of the responses to influence your evaluation. Be as objective as possible.
 After providing your explanation, output your final verdict by strictly following \
-this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better, \
-and "[[C]]" for a tie.
+this format: "A" if assistant A is better, "B" if assistant B is better, \
+and "C" for a tie.
 
 [User Question]
 {query}
@@ -75,7 +83,6 @@ def __init__(
         llm_provider: BaseLLMProvider,
     ):
         super().__init__(config, llm_provider)
-        self.pattern = re.compile(r"\[\[([^]]+)]].*$(?:(?!\[\[).)*", re.DOTALL)
         self.factors = config.factors
         if config.include_annotations and config.include_raw_documents:
             config.document_template = self.document_template_raw_and_annotation
@@ -86,17 +93,23 @@ def __init__(
         elif config.include_raw_documents:
             config.document_template = self.document_template_raw_only
             self.documents_prompt = self.documents_prompt_raw_only
-        else:
-            raise ValueError(
-                "At least one of include_annotations or include_raw_documents must be True"
-            )
         if config.prompt:
             self.prompt = config.prompt
+        if config.llm_answer_format == AnswerFormat.STRUCTURED:
+            self.config.llm_response_schema = PairWiseAnswerAnswerFormat
+        elif self.config.llm_answer_format != AnswerFormat.JSON:
+            logger.warning("We are using a PairwiseAnswerEvaluator config. Forcing the LLM answer format to JSON.")
+            self.config.llm_answer_format = AnswerFormat.JSON
+            self.config.llm_response_schema = {
+                "answer_a_reasoning": "Your reasoning wether the answer provided by agent A is correct or not.",
+                "answer_b_reasoning": "Your reasoning wether the answer provided by agent B is correct or not.",
+                "comparison_reasoning": "A short explanation of the differences between the two answers.",
+                "winner": "The winner of the pairwise comparison. Either 'A', 'B', or 'C' for a tie.",
+            }
 
-    def _build_message_pairwise(
-        self, query: Query, game: PairwiseGame
-    ) -> Union[str, List[Dict[str, str]]]:
+    def _build_message_pairwise(self, query: Query, game: PairwiseGame) -> str | list[dict[str, str]]:
         documents = self._prepare_documents(query)
+
         query_metadata = self._get_usable_fields_from_metadata(
             self.prompt, query.metadata, skip_fields=[self.config.query_placeholder]
         )
@@ -128,12 +141,16 @@ def _build_message_pairwise(
         }
         return self.prompt.format(**formatters)
 
-    def _process_answer(self, answer: str) -> str:
+    def _process_answer(self, llm_response: LLMResponseType) -> LLMResponseType:
         """Extracts the relevant part of an answer."""
-        match_ans = self.pattern.search(answer)
-        if not match_ans:
-            raise ValueError(f"Could not find answer in {answer}")
-        better_agent = match_ans.group(1)
-        if better_agent not in ["A", "B", "C"]:
-            raise ValueError(f"Unknown answer: {better_agent}")
-        return better_agent
+        if isinstance(llm_response.parsed_answer, PairWiseAnswerAnswerFormat):
+            return LLMResponseType(
+                raw_answer=llm_response.raw_answer,
+                parsed_answer=llm_response.parsed_answer.winner,
+            )
+        if isinstance(llm_response.parsed_answer, dict):
+            return LLMResponseType(
+                raw_answer=llm_response.raw_answer,
+                parsed_answer=llm_response.parsed_answer["winner"],
+            )
+        return llm_response
diff --git a/ragelo/evaluators/base_evaluator.py b/ragelo/evaluators/base_evaluator.py
index bf27878..e1040fe 100644
--- a/ragelo/evaluators/base_evaluator.py
+++ b/ragelo/evaluators/base_evaluator.py
@@ -1,30 +1,30 @@
-import csv
-import json
-import os
+from __future__ import annotations
+
+import asyncio
 import string
 from abc import ABC, abstractmethod
-from typing import Any, Dict, List, Optional, Set, Union
+from collections.abc import Sequence
 
-from tqdm.auto import tqdm
+import rich
 
 from ragelo.llm_providers.base_llm_provider import BaseLLMProvider
-from ragelo.logger import logger
-from ragelo.types import (
-    AgentAnswer,
-    AnswerEvaluatorResult,
-    Document,
-    EvaluatorResult,
-    Query,
-    RetrievalEvaluatorResult,
-)
-from ragelo.types.configurations import AnswerFormat, BaseEvaluatorConfig
+from ragelo.types.configurations import BaseEvaluatorConfig
+from ragelo.types.evaluables import Evaluable
+from ragelo.types.experiment import Experiment
+from ragelo.types.formats import AnswerFormat, LLMResponseType
+from ragelo.types.query import Query
+from ragelo.types.results import EvaluatorResult
+from ragelo.utils import call_async_fn, get_pbar
 
 
 class BaseEvaluator(ABC):
+    """
+    An abstract class for all evaluators. An evaluator is responsible for evaluating a query and an evaluable
+    """
+
     config: BaseEvaluatorConfig
-    scoring_keys: List[str]
-    scoring_key: str
     answer_format: AnswerFormat
+    evaluable_name: str = "Evaluable"
 
     @abstractmethod
     def __init__(
@@ -34,71 +34,86 @@ def __init__(
     ):
         raise NotImplementedError
 
-    @staticmethod
-    def __parse_json(answer: str, keys: Union[str, List[str]]) -> Dict[str, Any]:
-        # Checks if there is block of json in the answer like ```json\n{...}\n```
-        json_block = answer.split("```json\n")
-        if len(json_block) > 1:
-            json_data = json_block[1].split("\n```")[0]
-            return json.loads(json_data)
-        # Otherwise, go line by line
-        if isinstance(keys, str):
-            keys = [keys]
-        json_dict = {}
-        for line in answer.strip().split("\n"):
-            try:
-                json_object = json.loads(line)
-                for k in keys:
-                    if k in json_object:
-                        json_dict[k] = json_object[k]
-            except json.JSONDecodeError:
-                pass
-        return json_dict
+    def evaluate_experiment(self, experiment: Experiment, n_threads: int | None = None):
+        """
+        Trigger the evaluator for all the supported evaluables in the experiment.
+        The evaluation is done in asynchronously with the number of threads defined in config.n_processes parameter.
+        This can be overwritten by the n_threads parameter.
+
+        Args:
+            experiment(Experiment): The experiment to evaluate.
+            n_threads(int): The number of threads to use for the evaluation.
+                If None, the number of threads defined in the config will be used.
+        """
+        n_threads = n_threads or self.config.n_processes
+        call_async_fn(self._evaluate_experiment_async, experiment, n_threads)
 
-    def json_answer_parser(self, answer: str, key: str) -> str:
-        """Parses a Json answer from the LLM and returns a specific key"""
+    @abstractmethod
+    async def evaluate_async(
+        self,
+        eval_sample: tuple[Query, Evaluable],
+    ) -> EvaluatorResult:
+        """Evaluate a single query and evaluable asynchronously."""
+        raise NotImplementedError
 
-        # Finds all valid JSON objects in the answer that contain the key
-        json_dict = self.__parse_json(answer, key)
-        if key not in json_dict:
-            raise ValueError(
-                "Answer does not contain the necessary key\n"
-                f"Expected {key}, found {json_dict.keys()}\n{answer}"
-            )
-        return json_dict[key]
+    async def _evaluate_experiment_async(self, experiment: Experiment, n_threads: int = 1):
+        tuples_to_eval = self._get_tuples_to_evaluate(experiment)
+        pbar = get_pbar(
+            len(tuples_to_eval),
+            self.config.rich_print,
+            desc=f"Evaluating {self.evaluable_name}s",
+        )
+
+        awaitables_ended = False
+        pending: set[asyncio.Future] = set()
+        aws = map(self.evaluate_async, tuples_to_eval)
+        aws = iter(aws)
+        failed = 0
+        evaluations = 0
+        while pending or not awaitables_ended:
+            while len(pending) < n_threads and not awaitables_ended:
+                try:
+                    aw = next(aws)
+                except StopIteration:
+                    awaitables_ended = True
+                else:
+                    pending.add(asyncio.ensure_future(aw))
+            if not pending:
+                break
+            done, pending = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED)
+            while done:
+                evaluation = await done.pop()
+                evaluations += 1
+                pbar.update()
+                if evaluation.exception:
+                    failed += 1
+                    continue
+                experiment.add_evaluation(evaluation, exist_ok=True)
+        pbar.close()
+        if self.config.verbose:
+            self._print_failed_evaluations(evaluations, failed)
 
-    def json_answer_parser_multifields(
-        self, answer: str, keys: List[str]
-    ) -> Dict[str, str]:
-        """Parses a Json answer from the LLM and returns the values from multiple fields"""
-        # Finds all valid JSON objects in the answer that contain the key
-        try:
-            json_dict = json.loads(answer)
-        except json.JSONDecodeError:
-            json_dict = self.__parse_json(answer, keys)
+    @abstractmethod
+    def _get_tuples_to_evaluate(self, queries: Experiment) -> Sequence[tuple[Query, Evaluable]]:
+        raise NotImplementedError
 
-        valid_keys = set(json_dict.keys()).intersection(keys)
-        if len(valid_keys) != len(keys):
-            raise ValueError(
-                "Answer does not contain all necessary keys\n"
-                f"Expected {keys}, found {valid_keys}.\n"
-                f"Full Answer:\n{answer}"
-            )
-        return json_dict
+    def _process_answer(self, llm_response: LLMResponseType) -> LLMResponseType:
+        """Processes the raw answer returned by the LLM. Should be implemented by the subclass if needed."""
+        return llm_response
 
     @staticmethod
-    def _get_fields_from_string(s: str) -> List[str]:
+    def _get_fields_from_string(s: str) -> list[str]:
         """Parse a formatted string and return all the fields in it"""
         field_names = [v[1] for v in string.Formatter().parse(s) if v[1] is not None]
         return field_names
 
     @staticmethod
     def _get_usable_fields_from_metadata(
-        prompt: str, metadata: Optional[Dict[str, str]], skip_fields: List[str] = []
-    ) -> Dict[str, str]:
+        prompt: str, metadata: dict[str, str] | None, skip_fields: list[str] = []
+    ) -> dict[str, str]:
         """Get the fields from the prompt that are in the metadata"""
         expected_fields = BaseEvaluator._get_fields_from_string(prompt)
-        valid_fields: Dict[str, str] = {}
+        valid_fields: dict[str, str] = {}
         if metadata is None:
             return valid_fields
         for field in expected_fields:
@@ -106,456 +121,14 @@ def _get_usable_fields_from_metadata(
                 valid_fields[field] = metadata[field]
         return valid_fields
 
-    @staticmethod
-    def __get_existing_evaluations_from_json(output_file: str) -> List[Dict[str, str]]:
-        return json.load(open(output_file, "r"))
-
-    @staticmethod
-    def __get_existing_evaluations_from_jsonl(output_file: str) -> List[Dict[str, str]]:
-        with open(output_file, "r") as f:
-            return [json.loads(line) for line in f]
-
-    @staticmethod
-    def __get_existing_evaluations_from_csv(output_file: str) -> List[Dict[str, Any]]:
-        existing_lines: List[Dict[str, str]] = []
-        base_columns = ["qid", "did", "agent", "raw_answer", "agent_a", "agent_b"]
-        with open(output_file, "r") as f:
-            reader = csv.DictReader(f)
-            line: Dict[str, Any]
-            for line in reader:
-                line_dict = line
-                if "answer" in line and line["answer"]:
-                    line_dict["answer"] = line["answer"]
-                else:
-                    remaining_keys = {k for k in line.keys() if k not in base_columns}
-                    line_dict["answer"] = {k: line[k] for k in remaining_keys}
-                existing_lines.append(line_dict)
-        return existing_lines
-
-    def _get_existing_evaluations(
-        self, evaluation_file: str, force: bool = False
-    ) -> List[Dict[str, Any]]:
-        existing_lines: List[Dict[str, str]] = []
-        if force and os.path.exists(evaluation_file):
-            logger.warning(f"Removing existing {evaluation_file}!")
-            os.remove(evaluation_file)
-        if not os.path.isfile(evaluation_file):
-            return existing_lines
-        if evaluation_file.endswith(".json"):
-            return self.__get_existing_evaluations_from_json(evaluation_file)
-        if evaluation_file.endswith(".jsonl"):
-            return self.__get_existing_evaluations_from_jsonl(evaluation_file)
-        else:
-            return self.__get_existing_evaluations_from_csv(evaluation_file)
-
-    @staticmethod
-    def _print_response(
-        response: EvaluatorResult,
-        rich_print: bool = False,
-    ):
-        answer: Optional[Union[str, Dict[str, str], int]]
-        if isinstance(response.answer, dict):
-            # Print the answer in a more readable format
-            answer = json.dumps(response.answer, indent=4)
-        else:
-            answer = response.answer
-        response_dict = response.model_dump()
-        agent_a = response_dict.get("agent_a")
-        agent_b = response_dict.get("agent_b")
-        agent = response_dict.get("agent")
-        qid = response_dict.get("qid")
-        did = response_dict.get("did")
-        raw_answer = response_dict.get("raw_answer")
-
-        if rich_print:
-            try:
-                import rich
-
-                rich.print(f"[bold blue]🔎 Query ID[/bold blue]: {qid}")
-                did = response_dict.get("did")
-                if did:
-                    rich.print(f"[bold blue]📜 Document ID[/bold blue]: {did}")
-                if agent_a and agent_b:
-                    rich.print(
-                        f"[bold bright_cyan] {agent_a:<18} [/bold bright_cyan] 🆚  "
-                        f"[bold red] {agent_b}[/bold red]"
-                    )
-                elif agent:
-                    rich.print(
-                        f"[bold bright_cyan]🕵️ Agent[/bold bright_cyan]: {agent}"
-                    )
-                if raw_answer != answer:
-                    rich.print(f"[bold blue]Raw Answer[/bold blue]: {raw_answer}")
-                rich.print(f"[bold blue]Parsed Answer[/bold blue]: {answer}")
-                rich.print("")
-                return
-            except ImportError:
-                logger.warning("Rich not installed. Using plain print")
-        tqdm.write(f"Query ID: {qid}")
-        if did:
-            tqdm.write(f"Document ID: {did}")
-        if agent_a and agent_b:
-            tqdm.write(f"{agent_a} vs {agent_b}")
-        elif agent:
-            tqdm.write(f"Agent: {agent}")
-        tqdm.write(f"Raw Answer: {raw_answer}")
-        tqdm.write(f"Parsed Answer: {answer}")
-        tqdm.write("")
-
-    @staticmethod
-    def __dump_response_csv(
-        answer_dict: Dict[str, Any], output_columns: List[str], output_file: str
-    ):
-        if not any(k in output_columns for k in answer_dict.keys()):
-            raise ValueError(
-                "No parsed answer fields are in the output columns. \n"
-                f"Expected output columns: {output_columns}. \n"
-                f"Answer fields: {answer_dict.keys()}"
-            )
-        answer_dict = {k: answer_dict.get(k, None) for k in output_columns}
-        if not os.path.isfile(output_file):
-            logger.debug(f"Creating new file {output_file}")
-            with open(output_file, "w") as f:
-                writer = csv.DictWriter(f, fieldnames=output_columns)
-                writer.writeheader()
-        with open(output_file, "a") as f:
-            writer = csv.DictWriter(f, fieldnames=output_columns)
-            writer.writerow(answer_dict)
-
-    @staticmethod
-    def __dump_response_jsonl(answer_dict: Dict[str, str], output_file: str):
-        with open(output_file, "a") as f:
-            f.write(json.dumps(answer_dict) + "\n")
-
-    @staticmethod
-    def __dump_response_json(answer_dict: Dict[str, str], output_file: str):
-        """The file is a json-formatted list of dictionaries. Each dictionary is a response.
-        If the file already exists, erase the final closing square bracket and add a comma before adding the new response.
-        """
-        if not os.path.isfile(output_file):
-            logger.debug(f"Creating new file {output_file}")
-            with open(output_file, "w") as f:
-                f.write("[")
-        with open(output_file, "r+") as f:
-            f.seek(0, os.SEEK_END)
-            if f.tell() > 1:
-                f.seek(f.tell() - 1)
-                f.truncate()
-                f.write(",")
-            f.write(json.dumps(answer_dict) + "]")
-
-    def _dump_response(
-        self,
-        response: EvaluatorResult,
-        output_columns: List[str],
-        output_file: str,
-    ):
-        if self.config.verbose:
-            self._print_response(response, self.config.rich_print)
-        if not self.config.write_output:
-            return
-        answer_dict = response.model_dump()
-        if isinstance(answer_dict["answer"], dict):
-            # flatten the dict into the main dict
-            for k, v in answer_dict["answer"].items():
-                answer_dict[k] = v
-            del answer_dict["answer"]
-
-        null_keys = {k for k in answer_dict.keys() if answer_dict[k] is None}
-        unused_keys = (set(answer_dict.keys()) - set(output_columns)) & set(null_keys)
-        for k in unused_keys:
-            del answer_dict[k]
-
-        if output_file.endswith(".csv"):
-            self.__dump_response_csv(answer_dict, output_columns, output_file)
-        elif output_file.endswith(".jsonl"):
-            self.__dump_response_jsonl(answer_dict, output_file)
-        elif output_file.endswith(".json"):
-            self.__dump_response_json(answer_dict, output_file)
-        else:
-            logger.info(
-                "Output file format not recognized. Dumping raw response in csv format."
-            )
-            self.__dump_response_csv(answer_dict, output_columns, output_file)
-
-    def _process_answer(self, answer: str) -> Any:
-        """Processes the LLM evaluator output into some serializable format"""
-        if self.answer_format == AnswerFormat.JSON:
-            return self.json_answer_parser(answer, self.scoring_key)
-        if self.answer_format == AnswerFormat.MULTI_FIELD_JSON:
-            return self.json_answer_parser_multifields(answer, self.scoring_keys)
-        if self.answer_format == AnswerFormat.TEXT:
-            return answer
-
-    @staticmethod
-    def _assemble_query(
-        query: Union[Query, str], query_metadata: Optional[Dict[str, Any]] = None
-    ) -> Query:
-        if isinstance(query, str):
-            query = Query(qid="<no_qid>", query=query)
-        query.add_metadata(query_metadata)
-        return query
-
-    @staticmethod
-    def _assemble_document(
-        document: Union[Document, str], doc_metadata: Optional[Dict[str, Any]] = None
-    ) -> Document:
-        if isinstance(document, str):
-            did = "<no_did>"
-            if doc_metadata:
-                valid_id_fields = ["did", "doc_id", "document_id", "id", "_id"]
-                valid_id_fields = [f for f in valid_id_fields if f in doc_metadata]
-                if valid_id_fields:
-                    did = doc_metadata[valid_id_fields[0]]
-            document = Document(did=did, text=document)
-        document.add_metadata(doc_metadata)
-        return document
-
-    def _assemble_documents(
-        self,
-        documents: Union[List[str], List[Document]],
-        doc_metadata: Optional[List] = None,
-    ) -> Dict[str, Document]:
-        assembled_docs: Dict[str, Document] = {}
-        if doc_metadata and len(documents) != len(doc_metadata):
-            raise ValueError(
-                "The number of documents and document metadata do not match"
-            )
-        if not doc_metadata:
-            doc_metadata = [None] * len(documents)
-
-        for idx, (doc, m) in enumerate(zip(documents, doc_metadata)):
-            if isinstance(doc, str):
-                doc = Document(did=f"doc_{idx}", text=doc, metadata=m)
-            else:
-                doc.add_metadata(m)  # type: ignore
-            assembled_docs[doc.did] = doc  # type: ignore
-        return assembled_docs
-
-    @staticmethod
-    def _assemble_answer(
-        answer: Union[AgentAnswer, str],
-        answer_metadata: Optional[Dict[str, Any]] = None,
-    ) -> AgentAnswer:
-        if isinstance(answer, str):
-            answer = AgentAnswer(agent="<no_agent>", text=answer)
-        answer.add_metadata(answer_metadata)
-        return answer
-
-    def _load_retrieved_documents(
-        self,
-        queries: List[Query],
-        query_id_col: str = "qid",
-        document_id_col: str = "did",
-        document_text_col: str = "document_text",
-    ) -> List[Query]:
-        # Check if we actually need to do something
-        queries_with_documents = len([q for q in queries if len(q.retrieved_docs) > 0])
-        if queries_with_documents == len(queries):
-            logger.debug("All Queries already have retrieved documents. Skipping")
-            return queries
-        documents_read = 0
-        if not os.path.isfile(self.config.documents_file):
-            logger.warning(
-                f"Documents file {self.config.documents_file} not found"
-                "Will not add retrieved documents to queries"
-                f"{queries_with_documents} queries already have documents"
-            )
-            return queries
-
-        queries_idx = {q.qid: idx for idx, q in enumerate(queries)}
-        docs_per_query: Dict[str, Set[str]] = {}
-        for line in csv.DictReader(open(self.config.documents_file)):
-            qid = line[query_id_col].strip()
-            did = line[document_id_col].strip()
-            text = line[document_text_col].strip()
-            extra_metadata = {
-                k: v
-                for k, v in line.items()
-                if k not in [query_id_col, document_id_col, document_text_col]
-            }
-            if qid not in queries_idx:
-                logger.info(f"Query {qid} not in the provided queries. Skipping")
-                continue
-            if qid not in docs_per_query:
-                docs_per_query[qid] = set()
-            if did in docs_per_query[qid]:
-                logger.info(
-                    f"Document {did} already in the retrieved documents for query {qid}. Skipping"
-                )
-                continue
-            docs_per_query[qid].add(did)
-            queries[queries_idx[qid]].add_retrieved_doc(
-                Document(did=did, text=text, metadata=extra_metadata or None)
-            )
-            documents_read += 1
-        logger.info(f"Loaded {documents_read} documents")
-        return queries
-
-    def _load_agent_answers(
-        self,
-        queries: List[Query],
-        query_id_col: str = "qid",
-        agent_col: str = "agent",
-        answer_col: str = "answer",
-    ):
-        queries_with_answers = len([q for q in queries if len(q.answers) > 0])
-        if queries_with_answers == len(queries):
-            logger.debug("All Queries already have answers. Skipping")
-            return queries
-        if not os.path.isfile(self.config.answers_file):
-            logger.warning(
-                f"Answers file {self.config.answers_file} not found. Will not add answers to queries"
-                f"Queries with answers: {queries_with_answers}"
-            )
-            return queries
-        queries_idx = {q.qid: idx for idx, q in enumerate(queries)}
-        answers_read = 0
-        answers_per_query: Dict[str, Set[str]] = {}
-        for line in csv.DictReader(open(self.config.answers_file)):
-            qid = line[query_id_col].strip()
-            agent = line[agent_col].strip()
-            answer = line[answer_col].strip()
-            extra_metadata = {
-                k: v
-                for k, v in line.items()
-                if k not in [query_id_col, agent_col, answer_col]
-            }
-            if qid not in queries_idx:
-                logger.info(f"Query {qid} not in the provided queries. Skipping")
-                continue
-            if qid not in answers_per_query:
-                answers_per_query[qid] = set()
-            if agent in answers_per_query[qid]:
-                logger.info(
-                    f"Answer for agent {agent} already in the answers for query {qid}. Skipping"
-                )
-                continue
-            answers_per_query[qid].add(agent)
-            ans = AgentAnswer(agent=agent, text=answer, metadata=extra_metadata or None)
-            queries[queries_idx[qid]].add_agent_answer(ans)
-            answers_read += 1
-        logger.info(f"Loaded {answers_read} answers")
-        return queries
-
-    def _load_document_evaluations(
-        self, queries: List[Query], force: bool = False
-    ) -> List[Query]:
-        if force:
-            logger.info("Clearing existing document evaluations")
-            for q in queries:
-                for doc in q.retrieved_docs.values():
-                    doc.evaluation = None
-        document_evaluations = [
-            RetrievalEvaluatorResult(**x)
-            for x in self._get_existing_evaluations(
-                self.config.document_evaluations_file,
-                force,
-            )
-        ]
-        return self._add_document_evaluations(queries, document_evaluations)
-
-    def _add_document_evaluations(
-        self,
-        queries: List[Query],
-        evaluations: List[RetrievalEvaluatorResult],
-    ) -> List[Query]:
-        queries_idx = {q.qid: idx for idx, q in enumerate(queries)}
-
-        for evaluation in evaluations:
-            qid = evaluation.qid
-            did = evaluation.did
-            if qid not in queries_idx:
-                logger.info(f"Query {qid} not in the provided queries. Skipping")
-                continue
-            if did not in queries[queries_idx[qid]].retrieved_docs:
-                logger.info(
-                    f"Document {did} not in the retrieved documents for query {qid}. Skipping"
-                )
-                continue
-            queries[queries_idx[qid]].retrieved_docs[did].evaluation = evaluation
-        return queries
-
-    def _load_answers_evaluations(
-        self, queries: List[Query], force: bool = False
-    ) -> List[Query]:
-        if force:
-            logger.info("Clearing existing answers evaluations")
-            for q in queries:
-                for ans in q.answers.values():
-                    ans.evaluation = None
-                for game in q.pairwise_games:
-                    game.evaluation = None
-        evaluations = [
-            AnswerEvaluatorResult(**x)
-            for x in self._get_existing_evaluations(
-                self.config.answers_evaluations_file,
-                force,
-            )
-        ]
-        evaluations += [
-            AnswerEvaluatorResult(**x)
-            for x in self._get_existing_evaluations(
-                self.config.games_evaluations_file, force
-            )
-        ]
-        return self._add_answers_evaluations(queries, evaluations)
-
-    def _add_answers_evaluations(
-        self, queries: List[Query], evaluations: List[AnswerEvaluatorResult]
-    ) -> List[Query]:
-        queries_idx = {q.qid: idx for idx, q in enumerate(queries)}
-        games_idxs = {}
-        for q in queries:
-            games_idxs[q.qid] = {
-                (g.agent_a_answer.agent, g.agent_b_answer.agent): idx
-                for idx, g in enumerate(q.pairwise_games)
-            }
-
-        for evaluation in evaluations:
-            query_idx = queries_idx[evaluation.qid]
-            if evaluation.pairwise:
-                if not evaluation.agent_a or not evaluation.agent_b:
-                    # Should never happen, as the pydantic model enforces this
-                    raise ValueError("Pairwise evaluations require two agents")
-                agents = (evaluation.agent_a, evaluation.agent_b)
-                if agents not in games_idxs[evaluation.qid]:
-                    agents = (evaluation.agent_b, evaluation.agent_a)
-                    if agents not in games_idxs[evaluation.qid]:
-                        logger.info(
-                            f"Pairwise evaluation between {evaluation.agent_a} and {evaluation.agent_b} "
-                            f"not found in query {evaluation.qid}"
-                        )
-                        continue
-
-                game_idx = games_idxs[evaluation.qid][agents]
-                if queries[query_idx].pairwise_games[game_idx].evaluation is not None:
-                    continue
-
-                queries[query_idx].pairwise_games[game_idx].evaluation = evaluation
-            else:
-                if evaluation.agent is None:
-                    # Should never happen.
-                    raise ValueError("Evaluation must have an agent")
-                if queries[query_idx].answers[evaluation.agent].evaluation is not None:
-                    # Prefer object evaluations over file evaluations
-                    continue
-                queries[query_idx].answers[evaluation.agent].evaluation = evaluation
-        return queries
-
-    def _print_failed_evaluations(
-        self, total_evaluations: int, failed_evaluations: int
-    ):
+    def _print_failed_evaluations(self, total_evaluations: int, failed_evaluations: int):
         if self.config.rich_print:
-            try:
-                import rich
-
-                rich.print("✅ Done!")
-                rich.print(f"Failed evaluations: {failed_evaluations}")
-                rich.print(f"Total evaluations: {total_evaluations}")
-                return
-            except ImportError:
-                logger.warning("Rich not installed. Using plain print")
+            rich.print("✅ Done!")
+            if failed_evaluations > 0:
+                rich.print(f"[bold red]Failed evaluations: {failed_evaluations}[/bold red]")
+            rich.print(f"[bold green]Total evaluations: {total_evaluations}[/bold green]")
+            return
         print("✅ Done!")
-        print(f"Failed evaluations: {failed_evaluations}")
+        if failed_evaluations > 0:
+            print(f"Failed evaluations: {failed_evaluations}")
         print(f"Total evaluations: {total_evaluations}")
diff --git a/ragelo/evaluators/retrieval_evaluators/__init__.py b/ragelo/evaluators/retrieval_evaluators/__init__.py
index 4f5e65f..082f56e 100644
--- a/ragelo/evaluators/retrieval_evaluators/__init__.py
+++ b/ragelo/evaluators/retrieval_evaluators/__init__.py
@@ -1,4 +1,3 @@
-from ragelo.agent_rankers.base_agent_ranker import get_agent_ranker
 from ragelo.evaluators.retrieval_evaluators.base_retrieval_evaluator import (
     BaseRetrievalEvaluator,
     RetrievalEvaluatorFactory,
diff --git a/ragelo/evaluators/retrieval_evaluators/base_retrieval_evaluator.py b/ragelo/evaluators/retrieval_evaluators/base_retrieval_evaluator.py
index 051e249..66aace0 100644
--- a/ragelo/evaluators/retrieval_evaluators/base_retrieval_evaluator.py
+++ b/ragelo/evaluators/retrieval_evaluators/base_retrieval_evaluator.py
@@ -2,38 +2,28 @@
 It receives a set of queries used to retrieve a document and their respective retrieved documents,
 and returns a score or a label for each document."""
 
-import asyncio
-from concurrent.futures import ThreadPoolExecutor
-from typing import (
-    Any,
-    Callable,
-    Dict,
-    List,
-    Optional,
-    Set,
-    Tuple,
-    Type,
-    Union,
-    get_type_hints,
-)
+from __future__ import annotations
 
-from tqdm.auto import tqdm
+from typing import Any, Callable, Type, get_type_hints
+
+from tenacity import RetryError
 
 from ragelo.evaluators.base_evaluator import BaseEvaluator
 from ragelo.llm_providers.base_llm_provider import BaseLLMProvider, get_llm_provider
 from ragelo.logger import logger
-from ragelo.types import (
-    AnswerFormat,
-    Document,
-    Query,
-    RetrievalEvaluatorResult,
-    RetrievalEvaluatorTypes,
-)
 from ragelo.types.configurations import BaseRetrievalEvaluatorConfig
+from ragelo.types.evaluables import Document, Evaluable
+from ragelo.types.experiment import Experiment
+from ragelo.types.formats import LLMResponseType
+from ragelo.types.query import Query
+from ragelo.types.results import RetrievalEvaluatorResult
+from ragelo.types.types import RetrievalEvaluatorTypes
+from ragelo.utils import call_async_fn
 
 
 class BaseRetrievalEvaluator(BaseEvaluator):
     config: BaseRetrievalEvaluatorConfig
+    evaluable_name: str = "Retrieved document"
 
     def __init__(
         self,
@@ -42,223 +32,109 @@ def __init__(
     ):
         self.config = config
         self.llm_provider = llm_provider
-        self.document_evaluations_file = config.document_evaluations_file
-        self.output_columns = config.output_columns_retrieval_evaluator
-        self.scoring_key = config.scoring_key_retrieval_evaluator
-        self.scoring_keys = config.scoring_keys_retrieval_evaluator
-        if isinstance(self.config.answer_format_retrieval_evaluator, str):
-            self.answer_format = AnswerFormat(
-                self.config.answer_format_retrieval_evaluator
-            )
-        else:
-            self.answer_format = self.config.answer_format_retrieval_evaluator
-
-        if self.answer_format == AnswerFormat.MULTI_FIELD_JSON:
-            missing_keys = [
-                key for key in self.scoring_keys if key not in self.output_columns
-            ]
-            self.output_columns.extend(missing_keys)
-        else:
-            if self.scoring_key not in self.output_columns:
-                logger.info(f"Adding scoring key {self.scoring_key} to output columns")
-                self.output_columns.append(self.scoring_key)
-
-    async def _async_batch_evaluate(self, queries: List[Query]) -> List[Query]:
-        use_progress_bar = self.config.use_progress_bar
-        queries = self.__prepare_queries(queries)
-        tuples_to_eval = self.__get_tuples_to_evaluate(queries)
-        if self.config.rich_print:
-            import warnings
-
-            from tqdm import TqdmExperimentalWarning
 
-            warnings.filterwarnings("ignore", category=TqdmExperimentalWarning)
-            from tqdm.rich import tqdm as rich_tqdm
-
-            pbar_fn = rich_tqdm  # type: ignore
-        else:
-            pbar_fn = tqdm  # type: ignore
-        pbar = pbar_fn(
-            total=len(tuples_to_eval),
-            ncols=100,
-            desc="Evaluating retrieved documents",
-            disable=not use_progress_bar,
-            leave=False,
-            position=0,
-        )
-        awaitables_ended = False
-        pending: Set[asyncio.Future] = set()
-        aws = map(self._async_evaluate, tuples_to_eval)
-        aws = iter(aws)
-        evaluations = []
-        failed = 0
-        while pending or not awaitables_ended:
-            while len(pending) < self.config.n_processes and not awaitables_ended:
-                try:
-                    aw = next(aws)
-                except StopIteration:
-                    awaitables_ended = True  # all tasks have been scheduled
-                else:
-                    pending.add(asyncio.ensure_future(aw))
-            if not pending:
-                break
+    def evaluate(
+        self,
+        query: Query | str,
+        document: Document | str,
+        query_metadata: dict[str, Any] | None = None,
+        doc_metadata: dict[str, Any] | None = None,
+    ) -> RetrievalEvaluatorResult:
+        """Evaluates a single query-document pair. Returns the raw answer and the processed answer."""
+        query = Query.assemble_query(query, query_metadata)
+        document = Document.assemble_document(document, query.qid, doc_metadata)
+        result = call_async_fn(self.evaluate_async, (query, document))
 
-            done, pending = await asyncio.wait(
-                pending, return_when=asyncio.FIRST_COMPLETED
+        if result.exception or result.raw_answer is None or result.answer is None:
+            raise ValueError(
+                f"Failed to evaluate qid: {query.qid} did: {document.did}",
+                f"Exception: {result.exception}",
             )
-            while done:
-                evaluation = await done.pop()
-                pbar.update()
-                if evaluation.exception:
-                    failed += 1
-                    continue
-                evaluations.append(evaluation)
-        pbar.close()
-        self._add_document_evaluations(queries, evaluations)
-        if self.config.verbose:
-            self._print_failed_evaluations(len(evaluations), failed)
-        return queries
+        return result
 
-    async def _async_evaluate(
+    async def evaluate_async(
         self,
-        eval_sample: Tuple[Query, Document],
+        eval_sample: tuple[Query, Evaluable],
     ) -> RetrievalEvaluatorResult:
         query, document = eval_sample
+        if not isinstance(document, Document):
+            type_name = type(document).__name__
+            raise ValueError(f"can't evaluate a {type_name} in a Retrieval Evaluator")
         exc = None
         if document.evaluation is not None and not self.config.force:
             return document.evaluation  # type: ignore
         prompt = self._build_message(query, document)
         try:
-            raw_answer = await self.llm_provider.call_async(prompt)
+            llm_response = await self.llm_provider.call_async(
+                prompt,
+                answer_format=self.config.llm_answer_format,
+                response_schema=self.config.llm_response_schema,
+            )
+            llm_response = self._process_answer(llm_response)
+        except ValueError as e:
+            logger.warning(f"Failed to PARSE answer for qid: {query.qid} document id: {document.did}")
+            try:
+                llm_response = LLMResponseType(raw_answer=llm_response.raw_answer, parsed_answer=None)
+            except Exception:
+                llm_response = LLMResponseType(raw_answer="", parsed_answer=None)
+            exc = str(e)
         except Exception as e:
             logger.warning(f"Failed to FETCH answers for qid: {query.qid}")
             logger.warning(f"document id: {document.did}")
-            exc = str(e)
-            raw_answer = None
-        try:
-            answer = self._process_answer(raw_answer) if raw_answer else None
-        except ValueError as e:
-            logger.warning(
-                f"Failed to PARSE answer for qid: {query.qid} "
-                f"document id: {document.did}\n"
-                f"Raw answer: {raw_answer}"
-            )
-            exc = str(e)
-            answer = None
-        ans = RetrievalEvaluatorResult(
+            if isinstance(e, RetryError):
+                exc = str(e.last_attempt.exception())
+            else:
+                exc = str(e)
+            try:
+                llm_response = LLMResponseType(raw_answer=llm_response.raw_answer, parsed_answer=None)
+            except Exception:
+                llm_response = LLMResponseType(raw_answer="", parsed_answer=None)
+        return RetrievalEvaluatorResult(
             qid=query.qid,
             did=document.did,
-            raw_answer=raw_answer,
-            answer=answer,
+            raw_answer=llm_response.raw_answer,
+            answer=llm_response.parsed_answer,
             exception=exc,
         )
-        if ans.exception is None:
-            self._dump_response(
-                ans, self.output_columns, self.document_evaluations_file  # type: ignore
-            )
-        return ans
 
-    def __prepare_queries(self, queries: List[Query]) -> List[Query]:
-        queries = self._load_retrieved_documents(queries)
-        queries = self._load_document_evaluations(queries, force=self.config.force)
-        return queries
-
-    def __get_tuples_to_evaluate(
-        self, queries: List[Query]
-    ) -> List[Tuple[Query, Document]]:
+    def _get_tuples_to_evaluate(self, experiment: Experiment) -> list[tuple[Query, Evaluable]]:
         tuples_to_eval = []
         all_tuples = 0
         missing_evaluations = 0
-        for q in queries:
+        for q in experiment:
             for d in q.retrieved_docs.values():
                 if d.evaluation is None:
                     missing_evaluations += 1
                 tuples_to_eval.append((q, d))
                 all_tuples += 1
-        if missing_evaluations == all_tuples:
+        if missing_evaluations == 0:
             logger.info("All documents have already been evaluated")
             if self.config.verbose and not self.config.force:
-                print(
+                logger.warning(
                     f"All {all_tuples} documents are already evaluated.\n"
                     "If you want to re-evaluate documents, use the --force flag."
                 )
         return tuples_to_eval
 
-    def evaluate(
-        self,
-        query: Union[Query, str],
-        document: Union[Document, str],
-        query_metadata: Optional[Dict[str, Any]] = None,
-        doc_metadata: Optional[Dict[str, Any]] = None,
-    ) -> Tuple[str, Any]:
-        """Evaluates a single query-document pair. Returns the raw answer and the processed answer."""
-        query = self._assemble_query(query, query_metadata)
-        document = self._assemble_document(document, doc_metadata)
-
-        def run(coroutine):
-            return asyncio.run(coroutine)
-
-        try:
-            # Raises RuntimeError if there is no current event loop.
-            asyncio.get_running_loop()
-            # If there is a current event loop, we need to run the async code
-            # in a separate loop, in a separate thread.
-            with ThreadPoolExecutor(max_workers=1) as executor:
-                future = executor.submit(run, self._async_evaluate((query, document)))
-                result = future.result()
-        except RuntimeError:
-            result = asyncio.run(self._async_evaluate((query, document)))
-        if result.exception or result.raw_answer is None or result.answer is None:
-            raise ValueError(
-                f"Failed to evaluate qid: {query.qid} did: {document.did}",
-                f"Exception: {result.exception}",
-            )
-        return result.raw_answer, result.answer
-
-    def batch_evaluate(self, queries: List[Query]) -> List[Query]:
-        def run(coroutine):
-            return asyncio.run(coroutine)
-
-        try:
-            # Raises RuntimeError if there is no current event loop.
-            asyncio.get_running_loop()
-            # If there is a current event loop, we need to run the async code
-            # in a separate loop, in a separate thread.
-            with ThreadPoolExecutor(max_workers=1) as executor:
-                future = executor.submit(run, self._async_batch_evaluate(queries))
-                result = future.result()
-        except RuntimeError:
-            result = asyncio.run(self._async_batch_evaluate(queries))
-
-        return result
-
     def _build_message(
         self,
         query: Query,
         document: Document,
-    ) -> Union[str, List[Dict[str, str]]]:
+    ) -> str | list[dict[str, str]]:
         """Builds the prompt to send to the LLM."""
         raise NotImplementedError
 
     @classmethod
-    def from_config(
-        cls, config: BaseRetrievalEvaluatorConfig, llm_provider: BaseLLMProvider
-    ):
+    def from_config(cls, config: BaseRetrievalEvaluatorConfig, llm_provider: BaseLLMProvider):
         return cls(config, llm_provider)
 
     @classmethod
     def get_config_class(cls) -> Type[BaseRetrievalEvaluatorConfig]:
         return get_type_hints(cls)["config"]
 
-    @staticmethod
-    def _construct_list_of_answers(
-        answers: List[Dict[str, str]]
-    ) -> List[RetrievalEvaluatorResult]:
-        return [RetrievalEvaluatorResult(**x) for x in answers]
-
 
 class RetrievalEvaluatorFactory:
-    registry: Dict[RetrievalEvaluatorTypes, Type[BaseRetrievalEvaluator]] = {}
+    registry: dict[RetrievalEvaluatorTypes, Type[BaseRetrievalEvaluator]] = {}
 
     @classmethod
     def register(cls, evaluator_name: RetrievalEvaluatorTypes) -> Callable:
@@ -276,40 +152,41 @@ def inner_wrapper(
     def create(
         cls,
         evaluator_name: RetrievalEvaluatorTypes,
-        llm_provider: Union[BaseLLMProvider, str],
-        config: Optional[BaseRetrievalEvaluatorConfig] = None,
+        llm_provider: BaseLLMProvider | str,
+        config: BaseRetrievalEvaluatorConfig | None = None,
         **kwargs,
     ) -> BaseRetrievalEvaluator:
+        if evaluator_name not in cls.registry:
+            raise ValueError(
+                f"Unknown retrieval evaluator {evaluator_name}\n" f"Valid options are {list(cls.registry.keys())}"
+            )
         if isinstance(llm_provider, str):
             llm_provider_instance = get_llm_provider(llm_provider, **kwargs)
         else:
             llm_provider_instance = llm_provider
-        if evaluator_name not in cls.registry:
-            raise ValueError(
-                f"Unknown retrieval evaluator {evaluator_name}\n"
-                f"Valid options are {list(cls.registry.keys())}"
-            )
         if config is None:
             class_ = cls.registry[evaluator_name]
             type_config = class_.get_config_class()
             valid_keys = [field for field in type_config.get_model_fields()]
             valid_args = {k: v for k, v in kwargs.items() if k in valid_keys}
+            required_fields = [arg for arg, info in type_config.get_model_fields().items() if info.is_required()]
+            for field in required_fields:
+                if field not in valid_args:
+                    raise ValueError(f"Required argument {field} for evaluator {evaluator_name} not provided")
             config = type_config(**valid_args)
         return cls.registry[evaluator_name].from_config(config, llm_provider_instance)
 
 
 def get_retrieval_evaluator(
-    evaluator_name: Optional[Union[RetrievalEvaluatorTypes, str]] = None,
-    llm_provider: Union[BaseLLMProvider, str] = "openai",
-    config: Optional[BaseRetrievalEvaluatorConfig] = None,
+    evaluator_name: RetrievalEvaluatorTypes | str | None = None,
+    llm_provider: BaseLLMProvider | str = "openai",
+    config: BaseRetrievalEvaluatorConfig | None = None,
     **kwargs,
 ) -> BaseRetrievalEvaluator:
     if evaluator_name is None:
         # get the name from the config
         if config is None:
-            raise ValueError(
-                "Either the evaluator_name or a config object must be provided"
-            )
+            raise ValueError("Either the evaluator_name or a config object must be provided")
         evaluator_name = config.evaluator_name
     if isinstance(evaluator_name, str):
         evaluator_name = RetrievalEvaluatorTypes(evaluator_name)
diff --git a/ragelo/evaluators/retrieval_evaluators/custom_prompt_evaluator.py b/ragelo/evaluators/retrieval_evaluators/custom_prompt_evaluator.py
index e045bb6..0f89cb0 100644
--- a/ragelo/evaluators/retrieval_evaluators/custom_prompt_evaluator.py
+++ b/ragelo/evaluators/retrieval_evaluators/custom_prompt_evaluator.py
@@ -1,10 +1,14 @@
+from __future__ import annotations
+
 from ragelo.evaluators.retrieval_evaluators.base_retrieval_evaluator import (
     BaseRetrievalEvaluator,
     RetrievalEvaluatorFactory,
 )
 from ragelo.llm_providers.base_llm_provider import BaseLLMProvider
-from ragelo.types import Document, Query, RetrievalEvaluatorTypes
 from ragelo.types.configurations import CustomPromptEvaluatorConfig
+from ragelo.types.evaluables import Document
+from ragelo.types.query import Query
+from ragelo.types.types import RetrievalEvaluatorTypes
 
 
 @RetrievalEvaluatorFactory.register(RetrievalEvaluatorTypes.CUSTOM_PROMPT)
diff --git a/ragelo/evaluators/retrieval_evaluators/domain_expert_evaluator.py b/ragelo/evaluators/retrieval_evaluators/domain_expert_evaluator.py
index 55dc438..3db4881 100644
--- a/ragelo/evaluators/retrieval_evaluators/domain_expert_evaluator.py
+++ b/ragelo/evaluators/retrieval_evaluators/domain_expert_evaluator.py
@@ -1,6 +1,8 @@
 """Evaluator with a domain expert persona"""
 
-from typing import Tuple
+from __future__ import annotations
+
+from tenacity import RetryError
 
 from ragelo.evaluators.retrieval_evaluators import (
     BaseRetrievalEvaluator,
@@ -8,18 +10,17 @@
 )
 from ragelo.llm_providers.base_llm_provider import BaseLLMProvider
 from ragelo.logger import logger
-from ragelo.types import (
-    Document,
-    Query,
-    RetrievalEvaluatorResult,
-    RetrievalEvaluatorTypes,
-)
 from ragelo.types.configurations import DomainExpertEvaluatorConfig
+from ragelo.types.evaluables import Document, Evaluable
+from ragelo.types.formats import AnswerFormat
+from ragelo.types.query import Query
+from ragelo.types.results import RetrievalEvaluatorResult
+from ragelo.types.types import RetrievalEvaluatorTypes
 
 
 @RetrievalEvaluatorFactory.register(RetrievalEvaluatorTypes.DOMAIN_EXPERT)
 class DomainExpertEvaluator(BaseRetrievalEvaluator):
-    sys_prompt = """
+    system_prompt = """
 You are a domain expert in {expert_in}.{company_prompt_1} You are tasked \
 with evaluating the performance of a retrieval system for question \
 answering in this domain. The question answering system will be used \
@@ -79,7 +80,6 @@ class DomainExpertEvaluator(BaseRetrievalEvaluator):
 - 0: indicates that the retrieved document is not relevant to the query
 - 1: the document is somewhat relevant to the query
 - 2: the document is highly relevant to the query
-Please only answer with a single number.
 """.strip()
 
     COMPANY_PROMPT_1 = " You work for {company}."
@@ -93,44 +93,27 @@ def __init__(
         llm_provider: BaseLLMProvider,
     ):
         super().__init__(config, llm_provider)
-        if not self.config.expert_in:
-            raise ValueError(
-                "You are trying to use the Domain Expert Retrieval Evaluator. "
-                "For this evaluator, you need to provide the domain the evaluator "
-                "is an expert in the expert_in field."
-            )
-
+        if self.config.llm_answer_format != AnswerFormat.JSON:
+            logger.warning("We are using the Domain Expert Evaluator config. Forcing the LLM answer format to JSON.")
+            self.config.llm_answer_format = AnswerFormat.JSON
         self.expert_in = self.config.expert_in
-        self.domain_short = (
-            f" {self.config.domain_short}" if self.config.domain_short else ""
-        )
-        self.sys_prompt = self.sys_prompt.format(
+        self.domain_short = f" {self.config.domain_short}" if self.config.domain_short else ""
+        self.system_prompt = self.system_prompt.format(
             expert_in=self.expert_in,
             company_prompt_1=(
-                self.COMPANY_PROMPT_1.format(company=self.config.company)
-                if self.config.company
-                else ""
+                self.COMPANY_PROMPT_1.format(company=self.config.company) if self.config.company else ""
             ),
             company_prompt_2=(
-                self.COMPANY_PROMPT_2.format(company=self.config.company)
-                if self.config.company
-                else ""
+                self.COMPANY_PROMPT_2.format(company=self.config.company) if self.config.company else ""
             ),
             domain_short=(
-                self.DOMAIN_SHORT.format(domain_short=self.config.domain_short)
-                if self.config.domain_short
-                else ""
+                self.DOMAIN_SHORT.format(domain_short=self.config.domain_short) if self.config.domain_short else ""
             ),
         )
-        self.extra_guidelines = (
-            self.config.extra_guidelines if self.config.extra_guidelines else []
-        )
-        # self.reasoner_eva
+        self.extra_guidelines = self.config.extra_guidelines if self.config.extra_guidelines else []
 
     def __build_reason_message(self, query: Query, document: Document) -> str:
-        guidelines = "\n".join(
-            [f"- {guideline}" for guideline in self.extra_guidelines]
-        )
+        guidelines = "\n".join([f"- {guideline}" for guideline in self.extra_guidelines])
         reason_prompt = self.reason_prompt.format(
             query=query.query,
             doc_content=document.text,
@@ -139,22 +122,28 @@ def __build_reason_message(self, query: Query, document: Document) -> str:
         )
         return reason_prompt
 
-    async def _async_evaluate(
-        self, eval_sample: Tuple[Query, Document]
-    ) -> RetrievalEvaluatorResult:
+    async def evaluate_async(self, eval_sample: tuple[Query, Evaluable]) -> RetrievalEvaluatorResult:
         query, document = eval_sample
+        assert isinstance(document, Document)
         reason_message = self.__build_reason_message(query, document)
-        messages_reasoning = [
-            {"role": "system", "content": self.sys_prompt},
+        exc = None
+        if document.evaluation is not None and not self.config.force:
+            return document.evaluation  # type: ignore
+        messages = [
+            {"role": "system", "content": self.system_prompt},
             {"role": "user", "content": reason_message},
         ]
-        exc = None
         try:
-            reasoning_answer = await self.llm_provider.call_async(messages_reasoning)
+            return_answer = await self.llm_provider.call_async(messages, answer_format=AnswerFormat.TEXT)
+            reasoning_answer = return_answer.parsed_answer
+            assert isinstance(reasoning_answer, str)
         except Exception as e:
             logger.warning(f"Failed to FETCH reasonings for qid: {query.qid}")
             logger.warning(f"document id: {document.did}")
-            exc = str(e)
+            if isinstance(e, RetryError):
+                exc = str(e.last_attempt.exception())
+            else:
+                exc = str(e)
             return RetrievalEvaluatorResult(
                 qid=query.qid,
                 did=document.did,
@@ -162,26 +151,23 @@ async def _async_evaluate(
                 answer=None,
                 exception=exc,
             )
-        messages_score = messages_reasoning.copy()
+        messages_score = messages.copy()
         messages_score.append({"role": "assistant", "content": reasoning_answer})
         messages_score.append({"role": "user", "content": self.score_prompt})
         try:
-            score_answer = await self.llm_provider.call_async(messages_score)
-            score_answer = self._process_answer(score_answer)
-        except ValueError as e:
-            logger.warning(f"Failed to parse scores for qid: {query.qid}")
-            logger.warning(f"document id: {document.did}")
-            score_answer = None
-            exc = str(e)
+            score_answer = await self.llm_provider.call_async(
+                messages_score,
+                answer_format=AnswerFormat.JSON,
+                response_schema=self.config.llm_response_schema,
+            )
         except Exception as e:
             logger.warning(f"Failed to FETCH scores for qid: {query.qid}")
             logger.warning(f"document id: {document.did}")
-            score_answer = None
             exc = str(e)
         return RetrievalEvaluatorResult(
             qid=query.qid,
             did=document.did,
             raw_answer=reasoning_answer,
-            answer=score_answer,
+            answer=score_answer.parsed_answer,
             exception=exc,
         )
diff --git a/ragelo/evaluators/retrieval_evaluators/few_shot_evaluator.py b/ragelo/evaluators/retrieval_evaluators/few_shot_evaluator.py
index 56ef9bd..e2ec059 100644
--- a/ragelo/evaluators/retrieval_evaluators/few_shot_evaluator.py
+++ b/ragelo/evaluators/retrieval_evaluators/few_shot_evaluator.py
@@ -1,12 +1,14 @@
-from typing import Dict, List
+from __future__ import annotations
 
 from ragelo.evaluators.retrieval_evaluators import (
     BaseRetrievalEvaluator,
     RetrievalEvaluatorFactory,
 )
 from ragelo.llm_providers.base_llm_provider import BaseLLMProvider
-from ragelo.types import Document, Query, RetrievalEvaluatorTypes
 from ragelo.types.configurations import FewShotEvaluatorConfig
+from ragelo.types.evaluables import Document
+from ragelo.types.query import Query
+from ragelo.types.types import RetrievalEvaluatorTypes
 
 
 @RetrievalEvaluatorFactory.register(RetrievalEvaluatorTypes.FEW_SHOT)
@@ -20,24 +22,24 @@ def __init__(
     ):
         super().__init__(config, llm_provider)
 
-        self.prompt = config.few_shot_user_prompt
-        self.sys_prompt = config.system_prompt
+        self.user_prompt = config.few_shot_user_prompt
+        self.system_prompt = config.system_prompt
         self.assistant_prompt = config.few_shot_assistant_answer
         self.few_shots = config.few_shots
 
-    def _build_message(self, query: Query, document: Document) -> List[Dict[str, str]]:
-        system_prompt_msg = {"role": "system", "content": self.sys_prompt}
+    def _build_message(self, query: Query, document: Document) -> list[dict[str, str]]:
+        system_prompt_msg = {"role": "system", "content": self.system_prompt}
         messages = [system_prompt_msg] + self.__build_few_shot_samples()
         formatters = {
             self.config.query_placeholder: query.query,
             self.config.document_placeholder: document.text,
         }
-        user_message = self.prompt.format(**formatters)
+        user_message = self.user_prompt.format(**formatters)
 
         messages.append({"role": "user", "content": user_message})
         return messages
 
-    def __build_few_shot_samples(self) -> List[Dict[str, str]]:
+    def __build_few_shot_samples(self) -> list[dict[str, str]]:
         few_shot_messages = []
         for few_shot in self.few_shots:
             formatters = {
@@ -46,7 +48,7 @@ def __build_few_shot_samples(self) -> List[Dict[str, str]]:
             }
             user_message = {
                 "role": "user",
-                "content": self.prompt.format(**formatters),
+                "content": self.user_prompt.format(**formatters),
             }
             formatters = {
                 self.config.reasoning_placeholder: few_shot.reasoning,
diff --git a/ragelo/evaluators/retrieval_evaluators/rdnam_evaluator.py b/ragelo/evaluators/retrieval_evaluators/rdnam_evaluator.py
index f9fea37..80b05fa 100644
--- a/ragelo/evaluators/retrieval_evaluators/rdnam_evaluator.py
+++ b/ragelo/evaluators/retrieval_evaluators/rdnam_evaluator.py
@@ -3,7 +3,7 @@
 https://arxiv.org/abs/2309.10621
 """
 
-import json
+from __future__ import annotations
 
 import numpy as np
 
@@ -12,8 +12,11 @@
     RetrievalEvaluatorFactory,
 )
 from ragelo.llm_providers.base_llm_provider import BaseLLMProvider
-from ragelo.types import Document, Query, RetrievalEvaluatorTypes
 from ragelo.types.configurations import RDNAMEvaluatorConfig
+from ragelo.types.evaluables import Document
+from ragelo.types.formats import LLMResponseType
+from ragelo.types.query import Query
+from ragelo.types.types import RetrievalEvaluatorTypes
 
 
 @RetrievalEvaluatorFactory.register(RetrievalEvaluatorTypes.RDNAM)
@@ -44,19 +47,19 @@ class RDNAMEvaluator(BaseRetrievalEvaluator):
 Consider the underlying intent of the search.
 {aspects}
 Consider the aspects above and relative importance of each, and decide on a final \
-score (O).
+overall score.
 {multiple}
-Produce a JSON array of scores without providing any reasoning. Example: {example}
+Produce a JSON with the scores without providing any reasoning. Example: {example}
 
-# Results
 """.strip()
 
     NARRATIVE_DESCRIPTION_PROMPT = "They were looking for: {description}\n{narrative}"
     ASPECTS_NARRATIVE = """Measure how well the content matches a likely intent \
-of the query (M).
-Measure how trustworthy the web page is (T).""".strip()
-    ASPECTS_EXAMPLE = """[{"M": 2, "T": 1, "O": 1}, {"M": 1..."""
-    DEFAULT_EXAMPLE = """[{"O": 1}, {"O": 2}, {"O": 0..."""
+of the query.
+Measure how trustworthy the web page is.""".strip()
+
+    ASPECTS_EXAMPLE = """{"intent_match": 2, "trustworthiness": 1, "overall": 1}"""
+    DEFAULT_EXAMPLE = """{"overall": 1}"""
     MULTIPLE_PROMPT = """We asked five search engine raters to evaluate \
 the relevance of the web page for the query.
 Each rater used their own independent judgement."""
@@ -69,20 +72,35 @@ def __init__(
     ):
         """Initializes an evaluator based on RDNAM framework."""
         super().__init__(config, llm_provider)
-
-        self.__role = self.config.annotator_role if self.config.annotator_role else ""
-
-        self.__aspects_prompt = (
-            self.ASPECTS_NARRATIVE if self.config.use_aspects else ""
-        )
-        self.__multiple_prompt = (
-            self.MULTIPLE_PROMPT if self.config.use_multiple_annotators else ""
-        )
+        self.config.llm_response_schema = {
+            "overall": "An integer between 0 and 2 representing the score of the document."
+        }
+
+        self._role = self.config.annotator_role if self.config.annotator_role else ""
+
+        if self.config.use_aspects:
+            self._aspects_prompt = self.ASPECTS_NARRATIVE
+            self.config.llm_response_schema["intent_match"] = (
+                "An integer between 0 and 2 representing the match of the document to the query intent."
+            )
+            self.config.llm_response_schema["trustworthiness"] = (
+                "An integer between 0 and 2 representing the trustworthiness of the document."
+            )
+        else:
+            self._aspects_prompt = ""
         if self.config.use_multiple_annotators:
-            self.prompt += "\n[{{"
+            self._multiple_prompt = self.MULTIPLE_PROMPT
+            self.config.llm_response_schema = {
+                "annotator_1": self.config.llm_response_schema,
+                "annotator_2": self.config.llm_response_schema,
+                "annotator_3": self.config.llm_response_schema,
+                "annotator_4": self.config.llm_response_schema,
+                "annotator_5": self.config.llm_response_schema,
+            }
+            self.multiple = True
         else:
-            self.prompt += "\n{{"
-        self.multiple = self.config.use_multiple_annotators
+            self._multiple_prompt = ""
+            self.multiple = False
 
     def _build_message(self, query: Query, document: Document) -> str:
         narrative_description_str = ""
@@ -94,50 +112,41 @@ def _build_message(self, query: Query, document: Document) -> str:
                     narrative=narrative, description=description
                 )
 
-        example = (
-            self.ASPECTS_EXAMPLE if self.__aspects_prompt else self.DEFAULT_EXAMPLE
-        )
+        example = self.ASPECTS_EXAMPLE if self._aspects_prompt else self.DEFAULT_EXAMPLE
 
         formatted_prompt = self.prompt.format(
-            role=self.__role,
+            role=self._role,
             query=query.query,
             doc_content=document,
             narrative_description=narrative_description_str,
-            aspects=self.__aspects_prompt,
-            multiple=self.__multiple_prompt,
+            aspects=self._aspects_prompt,
+            multiple=self._multiple_prompt,
             example=example,
         )
         return formatted_prompt
 
-    def _process_answer(self, answer: str) -> int:
-        if answer.startswith("[") or answer.startswith("{"):
-            # Sometimes the answer is a valid JSON object, even if the prompt ends if "{" or "[{"
-            try:
-                ans = json.loads(answer)
-            except json.decoder.JSONDecodeError:
-                raise ValueError(f"Failed to parse answer: {answer}")
-        else:
-            if self.multiple:
-                answer = "[{" + answer
+    def _process_answer(self, llm_response: LLMResponseType) -> LLMResponseType:
+        assert isinstance(llm_response.parsed_answer, dict)
+        if not self.multiple:
+            return llm_response
+        answer: dict[str, list[float]] = {"overall": []}
+        if self.config.use_aspects:
+            answer["intent_match"] = []
+            answer["trustworthiness"] = []
+
+        for v in llm_response.parsed_answer.values():
+            if self.config.use_aspects:
+                answer["intent_match"].append(int(v["intent_match"]))
+                answer["trustworthiness"].append(int(v["trustworthiness"]))
+                answer["overall"].append(int(v["overall"]))
             else:
-                answer = "{" + answer
-            try:
-                ans = json.loads(answer)
-            except json.decoder.JSONDecodeError:
-                raise ValueError(f"Failed to parse answer: {answer}")
-
-        if self.multiple or isinstance(ans, list):
-            # Sometimes the model returns a list of answers, even if self.multiple is False
-            try:
-                scores = [int(a["O"]) for a in ans]
-                return int(np.mean(scores))
-            except (KeyError, ValueError):
-                raise ValueError(f"Failed to parse answer: {answer}")
-        try:
-            return int(ans["O"])
-        except KeyError:
-            # As a last try, try to simply get whatever is after "O"
-            try:
-                return int(answer.split('"O":')[1].split(",")[0])
-            except IndexError:
-                raise ValueError(f"Failed to parse answer: {answer}")
+                answer["overall"].append(int(v["overall"]))
+        if self.config.use_aspects:
+            return LLMResponseType(
+                raw_answer=llm_response.raw_answer,
+                parsed_answer={k: float(np.mean(v)) for k, v in answer.items()},
+            )
+        return LLMResponseType(
+            raw_answer=llm_response.raw_answer,
+            parsed_answer=float(np.mean(answer["overall"])),
+        )
diff --git a/ragelo/evaluators/retrieval_evaluators/reasoner_evaluator.py b/ragelo/evaluators/retrieval_evaluators/reasoner_evaluator.py
index 1c0502a..9db6393 100644
--- a/ragelo/evaluators/retrieval_evaluators/reasoner_evaluator.py
+++ b/ragelo/evaluators/retrieval_evaluators/reasoner_evaluator.py
@@ -1,9 +1,14 @@
+from __future__ import annotations
+
 from ragelo.evaluators.retrieval_evaluators.base_retrieval_evaluator import (
     BaseRetrievalEvaluator,
     RetrievalEvaluatorFactory,
 )
-from ragelo.types import Document, Query, RetrievalEvaluatorTypes
 from ragelo.types.configurations import ReasonerEvaluatorConfig
+from ragelo.types.evaluables import Document
+from ragelo.types.formats import LLMResponseType
+from ragelo.types.query import Query
+from ragelo.types.types import RetrievalEvaluatorTypes
 
 
 @RetrievalEvaluatorFactory.register(RetrievalEvaluatorTypes.REASONER)
@@ -20,7 +25,7 @@ class ReasonerEvaluator(BaseRetrievalEvaluator):
 Please act as an impartial relevance annotator for a search engine. \
 Your goal is to evaluate the relevancy of the documents given a user question.
 
-You should write one sentence reasoning wether the document is relevant or not for \
+You should write only one sentence reasoning wether the document is relevant or not for \
 the user question. A document can be:
     - Not relevant: The document is not on topic.
     - Somewhat relevant: The document is on topic but does not fully answer the \
@@ -30,7 +35,8 @@ class ReasonerEvaluator(BaseRetrievalEvaluator):
     {query}
 
     [document content]
-    {document}"""  # noqa: E501
+    {document}
+""".strip()
 
     def _build_message(self, query: Query, document: Document) -> str:
         formatters = {
@@ -39,5 +45,6 @@ def _build_message(self, query: Query, document: Document) -> str:
         }
         return self.prompt.format(**formatters)
 
-    def _process_answer(self, answer: str) -> str:
-        return answer
+    def _process_answer(self, llm_response: LLMResponseType) -> LLMResponseType:
+        assert isinstance(llm_response.parsed_answer, str)
+        return llm_response
diff --git a/ragelo/llm_providers/__init__.py b/ragelo/llm_providers/__init__.py
index 593b0b5..d240398 100644
--- a/ragelo/llm_providers/__init__.py
+++ b/ragelo/llm_providers/__init__.py
@@ -1,3 +1,15 @@
-from ragelo.llm_providers.base_llm_provider import BaseLLMProvider, LLMProviderFactory
+from ragelo.llm_providers.base_llm_provider import (
+    BaseLLMProvider,
+    LLMProviderFactory,
+    get_llm_provider,
+)
 from ragelo.llm_providers.ollama_client import OllamaProvider
 from ragelo.llm_providers.openai_client import OpenAIProvider
+
+__all__ = [
+    "BaseLLMProvider",
+    "LLMProviderFactory",
+    "OllamaProvider",
+    "OpenAIProvider",
+    "get_llm_provider",
+]
diff --git a/ragelo/llm_providers/base_llm_provider.py b/ragelo/llm_providers/base_llm_provider.py
index b104e53..0a3bb59 100644
--- a/ragelo/llm_providers/base_llm_provider.py
+++ b/ragelo/llm_providers/base_llm_provider.py
@@ -1,25 +1,16 @@
-"""A LLM provider is a class that can be called with a string and returns with another string as an answer from an LLM model."""
+from __future__ import annotations
 
-import logging
 import os
 from abc import ABC, abstractmethod
-from typing import Dict, List, Optional, Type, Union, get_type_hints
+from typing import Any, Type, get_type_hints
 
-from ragelo.types import LLMProviderConfig, LLMProviderTypes
-from ragelo.types.configurations.base_configs import _PYDANTIC_MAJOR_VERSION
+from pydantic import BaseModel as PydanticBaseModel
 
-
-def set_credentials_from_file(credentials_file: str, split_char: str = "="):
-    """Read credentials from a file and add them to the environment"""
-    logging.info(f"Loading credentials from {credentials_file}")
-    if not os.path.isfile(credentials_file):
-        raise FileNotFoundError(f"Credentials file {credentials_file} not found")
-    with open(credentials_file) as f:
-        for line in f:
-            key, value = line.strip().split(split_char, 1)
-            logging.debug(f"Setting {key} from file")
-            os.environ[key] = value
-            os.environ[key] = value
+from ragelo.types.configurations import LLMProviderConfig
+from ragelo.types.formats import AnswerFormat, LLMResponseType
+from ragelo.types.pydantic_models import _PYDANTIC_MAJOR_VERSION
+from ragelo.types.types import LLMProviderTypes
+from ragelo.utils import call_async_fn
 
 
 class BaseLLMProvider(ABC):
@@ -29,16 +20,22 @@ class BaseLLMProvider(ABC):
     def __init__(self, config: LLMProviderConfig):
         self.config = config
 
-    @abstractmethod
-    def __call__(self, prompt: Union[str, List[Dict[str, str]]]) -> str:
+    def __call__(
+        self,
+        prompt: str | list[dict[str, str]],
+        answer_format: AnswerFormat = AnswerFormat.TEXT,
+        response_schema: Type[PydanticBaseModel] | dict[str, Any] | None = None,
+    ) -> LLMResponseType:
         """Submits a single query-document pair to the LLM and returns the answer."""
-        raise NotImplementedError
+        return call_async_fn(self.call_async, prompt, answer_format, response_schema)
 
     @abstractmethod
     async def call_async(
         self,
-        prompt: Union[str, List[Dict[str, str]]],
-    ) -> str:
+        prompt: str | list[dict[str, str]],
+        answer_format: AnswerFormat = AnswerFormat.TEXT,
+        response_schema: Type[PydanticBaseModel] | dict[str, Any] | None = None,
+    ) -> LLMResponseType:
         """Submits a single query-document pair to the LLM and returns the answer."""
         raise NotImplementedError
 
@@ -56,7 +53,7 @@ def get_config_class(cls) -> Type[LLMProviderConfig]:
 
 
 class LLMProviderFactory:
-    registry: Dict[LLMProviderTypes, Type[BaseLLMProvider]] = {}
+    registry: dict[LLMProviderTypes, Type[BaseLLMProvider]] = {}
 
     @classmethod
     def register(cls, name: LLMProviderTypes):
@@ -74,15 +71,12 @@ def inner_wrapper(
     def create(
         cls,
         name: LLMProviderTypes,
-        config: Optional[LLMProviderConfig] = None,
-        credentials_file: Optional[str] = None,
+        config: LLMProviderConfig | None = None,
         **kwargs,
     ) -> BaseLLMProvider:
         """Creates a new LLM provider"""
         if name not in cls.registry:
             raise ValueError(f"LLM provider {name} not found")
-        if credentials_file and os.path.isfile(credentials_file):
-            set_credentials_from_file(credentials_file)
         if config is None:
             class_ = cls.registry[name]
             type_config = class_.get_config_class()
@@ -97,9 +91,7 @@ def create(
                     else:
                         is_required = api_key_field.required  # type: ignore
                     if is_required:
-                        raise ValueError(
-                            f"API key not found in environment variable {class_.api_key_env_var}"
-                        )
+                        raise ValueError(f"API key not found in environment variable {class_.api_key_env_var}")
                     else:
                         api_key = api_key_field.default
                 kwargs["api_key"] = api_key
@@ -109,12 +101,11 @@ def create(
 
 
 def get_llm_provider(
-    name: Union[LLMProviderTypes, str],
-    config: Optional[LLMProviderConfig] = None,
-    credentials_file: Optional[str] = None,
+    name: LLMProviderTypes | str,
+    config: LLMProviderConfig | None = None,
     **kwargs,
 ) -> BaseLLMProvider:
     """Creates a new LLM provider"""
     if isinstance(name, str):
         name = LLMProviderTypes(name)
-    return LLMProviderFactory.create(name, config, credentials_file, **kwargs)
+    return LLMProviderFactory.create(name, config, **kwargs)
diff --git a/ragelo/llm_providers/ollama_client.py b/ragelo/llm_providers/ollama_client.py
index d369d20..6e656fb 100644
--- a/ragelo/llm_providers/ollama_client.py
+++ b/ragelo/llm_providers/ollama_client.py
@@ -1,22 +1,23 @@
-import asyncio
-from concurrent.futures import ThreadPoolExecutor
-from typing import Dict, List, Union
+from __future__ import annotations
+
+import json
+from typing import Any, Type
 
 from openai import AsyncOpenAI
+from pydantic import BaseModel as PydanticBaseModel
 from tenacity import retry, stop_after_attempt, wait_random_exponential
 
 from ragelo.llm_providers.base_llm_provider import BaseLLMProvider, LLMProviderFactory
-from ragelo.types import LLMProviderTypes
 from ragelo.types.configurations import OllamaConfiguration
+from ragelo.types.formats import AnswerFormat, LLMResponseType
+from ragelo.types.types import LLMProviderTypes
 
 
-# TODO: Change client to use tools instead of processing the raw files
 @LLMProviderFactory.register(LLMProviderTypes.OLLAMA)
 class OllamaProvider(BaseLLMProvider):
     """A Wrapper over the Ollama client."""
 
     config: OllamaConfiguration
-    api_key_env_var: str = "OPENAI_API_KEY"
 
     def __init__(
         self,
@@ -26,77 +27,53 @@ def __init__(
         self.__ollama_client = self.__get_ollama_client(config)
 
     @retry(wait=wait_random_exponential(min=1, max=120), stop=stop_after_attempt(1))
-    def __call__(
+    async def call_async(
         self,
-        prompt: Union[str, List[Dict[str, str]]],
-    ) -> str:
-        """Calls the OpenAI API.
+        prompt: str | list[dict[str, str]],
+        answer_format: AnswerFormat = AnswerFormat.TEXT,
+        response_schema: Type[PydanticBaseModel] | dict[str, Any] | None = None,
+    ) -> LLMResponseType:
+        """Calls the Ollama Local API asynchronously.
 
         Args:
             prompt: The prompt to use. Either a list of messages or a string.
         """
-
-        def run(coroutine):
-            return asyncio.run(coroutine)
-
         if isinstance(prompt, str):
             prompt = [{"role": "system", "content": prompt}]
-        try:
-            asyncio.get_running_loop()
-            with ThreadPoolExecutor(max_workers=1) as executor:
-                future = executor.submit(
-                    run,
-                    self.__ollama_client.chat.completions.create(
-                        model=self.config.model,
-                        messages=prompt,  # type: ignore
-                        temperature=self.config.temperature,
-                        max_tokens=self.config.max_tokens,
-                    ),
-                )
-                answers = future.result()
-        except RuntimeError:
-            answers = asyncio.run(
-                self.__ollama_client.chat.completions.create(
-                    model=self.config.model,
-                    messages=prompt,  # type: ignore
-                    temperature=self.config.temperature,
-                    max_tokens=self.config.max_tokens,
+        if answer_format == AnswerFormat.STRUCTURED:
+            raise NotImplementedError("Structured answer format is not supported on Ollama.")
+        if answer_format == AnswerFormat.JSON:
+            if isinstance(response_schema, dict):
+                schema = json.dumps(response_schema, indent=4)
+                prompt[0]["content"] += (
+                    "\n\nYour output should be a JSON string that STRICTLY "
+                    f"adheres to the following schema:\n{schema}"
                 )
+            answers = await self.__ollama_client.chat.completions.create(
+                model=self.config.model,
+                messages=prompt,  # type: ignore
+                temperature=self.config.temperature,
+                max_tokens=self.config.max_tokens,
+                response_format={"type": "json_object"},
+            )
+        else:
+            answers = await self.__ollama_client.chat.completions.create(
+                model=self.config.model,
+                messages=prompt,  # type: ignore
+                temperature=self.config.temperature,
+                max_tokens=self.config.max_tokens,
             )
-        if (
-            not answers.choices
-            or not answers.choices[0].message
-            or not answers.choices[0].message.content
-        ):
+        if not answers.choices or not answers.choices[0].message or not answers.choices[0].message.content:
             raise ValueError("Ollama did not return any completions.")
-        return answers.choices[0].message.content
-
-    async def call_async(
-        self,
-        prompt: Union[str, List[Dict[str, str]]],
-    ) -> str:
-        """Calls the Ollama Local API asynchronously.
-
-        Args:
-            prompt: The prompt to use. Either a list of messages or a string.
-            temperature : The temperature to use. Defaults to 0.1.
-            max_tokens: The maximum number of tokens to retrieve. Defaults 512.
-        """
-        if isinstance(prompt, str):
-            prompt = [{"role": "system", "content": prompt}]
-        answers = await self.__ollama_client.chat.completions.create(
-            model=self.config.model,
-            messages=prompt,  # type: ignore
-            temperature=self.config.temperature,
-            max_tokens=self.config.max_tokens,
+        if answer_format == AnswerFormat.JSON:
+            return LLMResponseType(
+                raw_answer=answers.choices[0].message.content,
+                parsed_answer=json.loads(answers.choices[0].message.content),
+            )
+        return LLMResponseType(
+            raw_answer=answers.choices[0].message.content,
+            parsed_answer=answers.choices[0].message.content,
         )
-        if (
-            not answers.choices
-            or not answers.choices[0].message
-            or not answers.choices[0].message.content
-        ):
-            raise ValueError("OpenAI did not return any completions.")
-        return answers.choices[0].message.content
 
     @staticmethod
     def __get_ollama_client(ollama_config: OllamaConfiguration) -> AsyncOpenAI:
diff --git a/ragelo/llm_providers/openai_client.py b/ragelo/llm_providers/openai_client.py
index eafefa0..e55104e 100644
--- a/ragelo/llm_providers/openai_client.py
+++ b/ragelo/llm_providers/openai_client.py
@@ -1,16 +1,20 @@
-import asyncio
-from concurrent.futures import ThreadPoolExecutor
-from typing import Dict, List, Union
+from __future__ import annotations
+
+import json
+from typing import Any, Callable, Type
 
 from openai import AsyncAzureOpenAI, AsyncOpenAI
+from pydantic import BaseModel as PydanticBaseModel
 from tenacity import retry, stop_after_attempt, wait_random_exponential
 
 from ragelo.llm_providers.base_llm_provider import BaseLLMProvider, LLMProviderFactory
-from ragelo.types import LLMProviderTypes
+from ragelo.logger import logger
 from ragelo.types.configurations import OpenAIConfiguration
+from ragelo.types.formats import AnswerFormat, LLMResponseType
+from ragelo.types.pydantic_models import _PYDANTIC_MAJOR_VERSION
+from ragelo.types.types import LLMProviderTypes
 
 
-# TODO: Change client to use tools instead of processing the raw files
 @LLMProviderFactory.register(LLMProviderTypes.OPENAI)
 class OpenAIProvider(BaseLLMProvider):
     """A Wrapper over the OpenAI client."""
@@ -25,82 +29,84 @@ def __init__(
         super().__init__(config)
         self.__openai_client = self.__get_openai_client(config)
 
-    @retry(wait=wait_random_exponential(min=1, max=120), stop=stop_after_attempt(1))
-    def __call__(
-        self,
-        prompt: Union[str, List[Dict[str, str]]],
-    ) -> str:
-        """Calls the OpenAI API.
-
-        Args:
-            prompt: The prompt to use. Either a list of messages or a string.
-        """
-
-        def run(coroutine):
-            return asyncio.run(coroutine)
-
-        if isinstance(prompt, str):
-            prompt = [{"role": "system", "content": prompt}]
-        try:
-            asyncio.get_running_loop()
-            with ThreadPoolExecutor(max_workers=1) as executor:
-                future = executor.submit(
-                    run,
-                    self.__openai_client.chat.completions.create(
-                        model=self.config.model,
-                        messages=prompt,  # type: ignore
-                        temperature=self.config.temperature,
-                        max_tokens=self.config.max_tokens,
-                    ),
-                )
-                answers = future.result()
-        except RuntimeError:
-            answers = asyncio.run(
-                self.__openai_client.chat.completions.create(
-                    model=self.config.model,
-                    messages=prompt,  # type: ignore
-                    temperature=self.config.temperature,
-                    max_tokens=self.config.max_tokens,
-                )
-            )
-        if (
-            not answers.choices
-            or not answers.choices[0].message
-            or not answers.choices[0].message.content
-        ):
-            raise ValueError("OpenAI did not return any completions.")
-        return answers.choices[0].message.content
-
+    @retry(wait=wait_random_exponential(min=1, max=120), stop=stop_after_attempt(5))
     async def call_async(
         self,
-        prompt: Union[str, List[Dict[str, str]]],
-    ) -> str:
+        prompt: str | list[dict[str, str]],
+        answer_format: AnswerFormat = AnswerFormat.TEXT,
+        response_schema: Type[PydanticBaseModel] | dict[str, Any] | None = None,
+    ) -> LLMResponseType:
         """Calls the OpenAI API asynchronously.
 
         Args:
             prompt: The prompt to use. Either a list of messages or a string.
+            answer_format: The format of the answer to return. Either TEXT, JSON, or STRUCTURED.
+            response_format: The format of the response to expect. If the answer_format is STRUCTURED,
+                this should be a PydanticBaseModel class. If the answer_format is JSON, this should be a dictionary
+                with the desired format the answer should be in
+        Returns:
+            The response from the OpenAI API, formatted according to the answer_format. or a string.
         """
         if isinstance(prompt, str):
             prompt = [{"role": "system", "content": prompt}]
-        answers = await self.__openai_client.chat.completions.create(
-            model=self.config.model,
-            messages=prompt,  # type: ignore
-            temperature=self.config.temperature,
-            max_tokens=self.config.max_tokens,
-        )
-        if (
-            not answers.choices
-            or not answers.choices[0].message
-            or not answers.choices[0].message.content
-        ):
+        call_kwargs = {
+            "messages": prompt,  # type: ignore
+            "model": self.config.model,
+            "temperature": self.config.temperature,
+            "max_tokens": self.config.max_tokens,
+        }
+        call_fn: Callable
+        if answer_format == AnswerFormat.STRUCTURED:
+            if not isinstance(response_schema, type(PydanticBaseModel)):
+                raise ValueError("response_schema must be a PydanticBaseModel Class when using structured output.")
+            call_fn = self.__openai_client.beta.chat.completions.parse
+            call_kwargs["response_format"] = response_schema
+        else:
+            call_fn = self.__openai_client.chat.completions.create
+            if answer_format == AnswerFormat.JSON:
+                if isinstance(response_schema, type(PydanticBaseModel)):
+                    logger.info(
+                        "You provided a PydanticBaseModel class for the response_schema. "
+                        "Using JSON as the desired answer format. Dumping the schema to JSON."
+                    )
+                    if _PYDANTIC_MAJOR_VERSION >= 2:
+                        response_schema = response_schema.model_json_schema()  # type: ignore
+                    else:
+                        response_schema = response_schema.schema()  # type: ignore
+                elif isinstance(response_schema, dict):
+                    schema = json.dumps(response_schema, indent=4)
+                    prompt[0]["content"] += (
+                        "\n\nYour output should be a JSON string that STRICTLY "
+                        f"adheres to the following schema:\n{schema}"
+                    )
+                call_kwargs["response_format"] = {"type": "json_object"}
+        answers = await call_fn(**call_kwargs)
+
+        if not answers.choices or not answers.choices[0].message or not answers.choices[0].message.content:
             raise ValueError("OpenAI did not return any completions.")
-        return answers.choices[0].message.content
+        if answer_format == AnswerFormat.STRUCTURED:
+            parsed_answer = answers.choices[0].message.parsed
+            if not isinstance(parsed_answer, PydanticBaseModel):
+                raise ValueError(f"OpenAI did not return a valid structured answer: {parsed_answer}")
+        elif answer_format == AnswerFormat.JSON:
+            try:
+                parsed_answer = json.loads(answers.choices[0].message.content)
+            except json.JSONDecodeError as e:
+                raise ValueError(
+                    f"Failed to parse raw JSON answer {answers.choices[0].message.content} as JSON: {e}"
+                ) from e
+        else:
+            parsed_answer = answers.choices[0].message.content
+        return LLMResponseType(
+            raw_answer=answers.choices[0].message.content,
+            parsed_answer=parsed_answer,
+        )
 
     @staticmethod
     def __get_openai_client(openai_config: OpenAIConfiguration) -> AsyncOpenAI:
         if openai_config.api_type == "azure":
             if openai_config.api_base is None:
-                raise ValueError("Azure-OpenAI base url not found in configuration.")
+                raise ValueError("Azure-OpenAI base url (api_base) not found in configuration.")
             return AsyncAzureOpenAI(
                 azure_endpoint=openai_config.api_base,
                 api_key=openai_config.api_key,
diff --git a/ragelo/logger.py b/ragelo/logger.py
index 6cf4174..30a27f8 100644
--- a/ragelo/logger.py
+++ b/ragelo/logger.py
@@ -1,5 +1,6 @@
 import logging
-import re
+
+from rich import print as rich_print
 
 logger = logging.getLogger(__name__)
 
@@ -7,34 +8,13 @@
 class CLILogHandler(logging.Handler):
     """A logging handler for the CLI use case"""
 
-    def __init__(self):
-        super().__init__()
-        try:
-            import rich
-
-            self.print = rich.print
-
-            self.rich = True
-        except ImportError:
-            self.rich = False
-
     def emit(self, record):
-        try:
-            msg = self.format(record)
-            if self.rich:
-                if record.levelno == logging.WARNING:
-                    self.print(f"[yellow]{msg}[/yellow]")
-                elif record.levelno == logging.DEBUG:
-                    self.print(f"[blue bold]{msg}[/blue bold]")
-                elif record.levelno == logging.ERROR:
-                    self.print(f"[bold red]{msg}[/bold red]")
-                else:
-                    self.print(msg)
-            else:
-                self.print(self.__clean_formatting(msg))
-        except Exception:
-            self.handleError(record)
-
-    def __clean_formatting(self, msg: str) -> str:
-        clean_text = re.sub(r"\[.*?\]", "", msg).strip()
-        return clean_text
+        msg = self.format(record)
+        if record.levelno == logging.WARNING:
+            rich_print(f"[yellow]{msg}[/yellow]")
+        elif record.levelno == logging.DEBUG:
+            rich_print(f"[blue bold]{msg}[/blue bold]")
+        elif record.levelno == logging.ERROR:
+            rich_print(f"[bold red]{msg}[/bold red]")
+        else:
+            rich_print(msg)
diff --git a/ragelo/types/__init__.py b/ragelo/types/__init__.py
index 8765006..6e776f1 100644
--- a/ragelo/types/__init__.py
+++ b/ragelo/types/__init__.py
@@ -1,6 +1,5 @@
 from ragelo.types.configurations import (
     AgentRankerConfig,
-    AllConfig,
     BaseAnswerEvaluatorConfig,
     BaseConfig,
     EloAgentRankerConfig,
@@ -10,18 +9,45 @@
     PairwiseEvaluatorConfig,
     ReasonerEvaluatorConfig,
 )
+from ragelo.types.configurations.cli_configs import CLIConfig
+from ragelo.types.evaluables import AgentAnswer, Document, PairwiseGame
+from ragelo.types.experiment import Experiment
+from ragelo.types.formats import AnswerFormat
+from ragelo.types.query import Query
+from ragelo.types.results import (
+    AnswerEvaluatorResult,
+    EvaluatorResult,
+    RetrievalEvaluatorResult,
+)
 from ragelo.types.types import (
-    AgentAnswer,
     AgentRankerTypes,
-    AnswerEvaluatorResult,
     AnswerEvaluatorTypes,
-    AnswerFormat,
-    Document,
-    Evaluable,
-    EvaluatorResult,
     LLMProviderTypes,
-    PairwiseGame,
-    Query,
-    RetrievalEvaluatorResult,
     RetrievalEvaluatorTypes,
 )
+
+__all__ = [
+    "AgentAnswer",
+    "AgentRankerConfig",
+    "AgentRankerTypes",
+    "CLIConfig",
+    "BaseAnswerEvaluatorConfig",
+    "BaseConfig",
+    "EloAgentRankerConfig",
+    "LLMProviderConfig",
+    "OllamaConfiguration",
+    "OpenAIConfiguration",
+    "PairwiseEvaluatorConfig",
+    "PairwiseGame",
+    "Query",
+    "ReasonerEvaluatorConfig",
+    "RetrievalEvaluatorResult",
+    "RetrievalEvaluatorTypes",
+    "AnswerEvaluatorResult",
+    "AnswerEvaluatorTypes",
+    "EvaluatorResult",
+    "Document",
+    "AnswerFormat",
+    "LLMProviderTypes",
+    "Experiment",
+]
diff --git a/ragelo/types/configurations/__init__.py b/ragelo/types/configurations/__init__.py
index 7ed6c86..56645a0 100644
--- a/ragelo/types/configurations/__init__.py
+++ b/ragelo/types/configurations/__init__.py
@@ -1,5 +1,48 @@
-from ragelo.types.configurations.agent_ranker_configs import *
-from ragelo.types.configurations.answer_evaluator_configs import *
-from ragelo.types.configurations.base_configs import *
-from ragelo.types.configurations.llm_provider_configs import *
-from ragelo.types.configurations.retrieval_evaluator_configs import *
+from ragelo.types.configurations.agent_ranker_configs import (
+    AgentRankerConfig,
+    EloAgentRankerConfig,
+)
+from ragelo.types.configurations.answer_evaluator_configs import (
+    BaseAnswerEvaluatorConfig,
+    CustomPairwiseEvaluatorConfig,
+    CustomPromptAnswerEvaluatorConfig,
+    PairwiseDomainExpertEvaluatorConfig,
+    PairwiseEvaluatorConfig,
+)
+from ragelo.types.configurations.base_configs import BaseConfig, BaseEvaluatorConfig
+from ragelo.types.configurations.cli_configs import CLIConfig
+from ragelo.types.configurations.llm_provider_configs import (
+    LLMProviderConfig,
+    OllamaConfiguration,
+    OpenAIConfiguration,
+)
+from ragelo.types.configurations.retrieval_evaluator_configs import (
+    BaseRetrievalEvaluatorConfig,
+    CustomPromptEvaluatorConfig,
+    DomainExpertEvaluatorConfig,
+    FewShotEvaluatorConfig,
+    RDNAMEvaluatorConfig,
+    ReasonerEvaluatorConfig,
+)
+
+__all__ = [
+    "CLIConfig",
+    "BaseConfig",
+    "BaseEvaluatorConfig",
+    "LLMProviderConfig",
+    "OpenAIConfiguration",
+    "OllamaConfiguration",
+    "AgentRankerConfig",
+    "EloAgentRankerConfig",
+    "BaseAnswerEvaluatorConfig",
+    "CustomPairwiseEvaluatorConfig",
+    "CustomPromptAnswerEvaluatorConfig",
+    "PairwiseEvaluatorConfig",
+    "PairwiseDomainExpertEvaluatorConfig",
+    "BaseRetrievalEvaluatorConfig",
+    "FewShotEvaluatorConfig",
+    "RDNAMEvaluatorConfig",
+    "ReasonerEvaluatorConfig",
+    "CustomPromptEvaluatorConfig",
+    "DomainExpertEvaluatorConfig",
+]
diff --git a/ragelo/types/configurations/agent_ranker_configs.py b/ragelo/types/configurations/agent_ranker_configs.py
index a5cd1cf..bc81e03 100644
--- a/ragelo/types/configurations/agent_ranker_configs.py
+++ b/ragelo/types/configurations/agent_ranker_configs.py
@@ -1,4 +1,4 @@
-from typing import List
+from __future__ import annotations
 
 from pydantic import Field
 
@@ -7,7 +7,7 @@
 
 class AgentRankerConfig(BaseConfig):
     ranker_name: str = Field(default="elo", description="The name of the agent ranker")
-    output_columns_agent_ranker: List[str] = Field(
+    output_columns_agent_ranker: list[str] = Field(
         default=["agent", "score"],
         description="The columns to output in the CSV file",
     )
@@ -23,13 +23,11 @@ class AgentRankerConfig(BaseConfig):
 
 
 class EloAgentRankerConfig(AgentRankerConfig):
-    elo_k: int = Field(
-        default=32, description="The K factor for the Elo ranking algorithm"
+    elo_k: int = Field(default=32, description="The K factor for the Elo ranking algorithm")
+    initial_score: int = Field(default=1000, description="The initial score for each agent")
+    agents_evaluations_file: str = Field(default="elo_ranking.csv", description="Path to the output file")
+    tournaments: int = Field(default=10, description="The number of Elo tournaments to play")
+    score_mapping: dict[str, float] = Field(
+        default={"A": 1, "B": 0, "C": 0.5},
+        description="The mapping of the evaluation answers to scores",
     )
-    initial_score: int = Field(
-        default=1000, description="The initial score for each agent"
-    )
-    agents_evaluations_file: str = Field(
-        default="elo_ranking.csv", description="Path to the output file"
-    )
-    rounds: int = Field(default=10, description="The number of rounds to play")
diff --git a/ragelo/types/configurations/answer_evaluator_configs.py b/ragelo/types/configurations/answer_evaluator_configs.py
index 72905f9..259febd 100644
--- a/ragelo/types/configurations/answer_evaluator_configs.py
+++ b/ragelo/types/configurations/answer_evaluator_configs.py
@@ -1,23 +1,23 @@
-from typing import Callable, List, Optional, Union
+from __future__ import annotations
 
+from typing import Any, Callable, Type
+
+from pydantic import BaseModel as PydanticBaseModel
 from pydantic import Field
 
-from ragelo.types.configurations.base_configs import AnswerFormat, BaseEvaluatorConfig
+from ragelo.types.configurations.base_configs import BaseEvaluatorConfig
+from ragelo.types.formats import AnswerFormat
+from ragelo.types.pydantic_models import ValidationError, validator
 from ragelo.types.types import AnswerEvaluatorTypes
 
 
 class BaseAnswerEvaluatorConfig(BaseEvaluatorConfig):
-    evaluator_name: Union[str, AnswerEvaluatorTypes] = ""
-    answer_placeholder: str = Field(
-        default="answer", description="The placeholder for the answer in the prompt"
-    )
+    answer_placeholder: str = Field(default="answer", description="The placeholder for the answer in the prompt")
     documents_placeholder: str = Field(
         default="documents",
         description="The placeholder for the documents in the prompt",
     )
-    pairwise: bool = Field(
-        default=False, description="Whether or not to the evaluator is pairwise"
-    )
+    pairwise: bool = Field(default=False, description="Whether or not to the evaluator is pairwise")
     document_template: str = Field(
         default="[{did}] {doc}",
         description="The template to format each individual document in the prompt",
@@ -30,96 +30,141 @@ class BaseAnswerEvaluatorConfig(BaseEvaluatorConfig):
         ),
     )
     include_annotations: bool = Field(
-        default=True,
+        default=False,
         description="Whether or not to include the document relevance annotations in the prompt",
     )
     include_raw_documents: bool = Field(
-        default=False,
+        default=True,
         description="Whether or not to include the raw documents in the prompt",
     )
-    document_filter: Optional[Callable[[str], bool]] = Field(
+    document_filter: Callable[[str], bool] | None = Field(
         default=None, description="A function to filter the documents"
     )
-    document_relevance_threshold: Optional[int] = Field(
+    document_relevance_threshold: int | None = Field(
         default=None,
-        description="The minimum relevance score for a document to be included in the prompt. By default, all documents are included.",
+        description=(
+            "The minimum relevance score for a document to be included in the prompt. "
+            "By default, all documents are included."
+        ),
     )
 
 
 class PairwiseEvaluatorConfig(BaseAnswerEvaluatorConfig):
     """Configuration for the pairwise evaluator."""
 
-    evaluator_name: Union[str, AnswerEvaluatorTypes] = AnswerEvaluatorTypes.PAIRWISE
-    output_columns_pairwise_evaluator: List[str] = Field(
-        default=["qid", "agent_a", "agent_b", "raw_answer", "answer"],
-        description="The columns to output in the CSV file",
-    )
-    bidirectional: bool = Field(
-        default=False, description="Whether or not to run each game in both directions"
-    )
-    n_games_per_query: int = Field(
-        default=100, description="Maximum number of games to generate for each query"
-    )
-    games_evaluations_path: str = Field(
-        default="pairwise_answers_evaluations.csv",
-        description="Path to the output file",
+    evaluator_name: AnswerEvaluatorTypes = AnswerEvaluatorTypes.PAIRWISE
+    bidirectional: bool = Field(default=False, description="Whether or not to run each game in both directions")
+    n_games_per_query: int = Field(default=100, description="Maximum number of games to generate for each query")
+    pairwise: bool = Field(default=True, description="Whether or not to the evaluator is pairwise")
+    include_annotations: bool = Field(
+        default=True,
+        description="Whether or not to include the document relevance annotations in the prompt",
     )
-    pairwise: bool = Field(
-        default=True, description="Whether or not to the evaluator is pairwise"
+    include_raw_documents: bool = Field(
+        default=False,
+        description="Whether or not to include the raw documents in the prompt",
     )
+
     document_template: str = Field(
         default="[{did}] {annotation}",
         description="The template to format each individual document in the prompt",
     )
-    prompt: Optional[str] = Field(
+    prompt: str | None = Field(
         default=None,
         description="Prompt to use for the evaluator. If not provided, a default prompt will be used",
     )
     factors: str = Field(
         default=(
-            "the correctness, helpfulness, completeness, accuracy, depth, and "
-            "level of detail of their responses"
+            "the correctness, helpfulness, completeness, accuracy, depth, and " "level of detail of their responses"
         ),
         description=(
             "A string containing the factors to be used when evaluating an answer. "
             "If not provided, a default string will be used"
         ),
     )
+    llm_answer_format: AnswerFormat = Field(
+        default=AnswerFormat.JSON,
+        description="The format of the answer returned by the LLM",
+    )
+    llm_response_schema: Type[PydanticBaseModel] | dict[str, Any] | None = Field(
+        default={
+            "analysis_assistant_a": "A string with your analysis of assistant A's answer",
+            "analysis_assistant_b": "A string with your analysis of assistant B's answer",
+            "differences": "A string with your comparison between the two answers and their differences",
+            "winner": (
+                "The winner of the comparison. "
+                "'A' if assistant A is better, 'B' if assistant B is better, and 'C' for a tie"
+            ),
+        },
+        description=(
+            "The response schema for the LLM. "
+            "Required if the llm_answer_format is structured and recommended for JSON."
+        ),
+    )
+
+    @validator
+    @classmethod
+    def check_annotations_or_raw_documents(cls, v):
+        if v is None and cls.include_annotations is False and cls.include_raw_documents is False:
+            raise ValidationError("At least one of include_annotations or include_raw_documents must be True")
+        return v
+
+
+class CustomPairwiseEvaluatorConfig(BaseAnswerEvaluatorConfig):
+    """Configuration for a custom pairwise evaluator."""
+
+    evaluator_name: AnswerEvaluatorTypes = AnswerEvaluatorTypes.CUSTOM_PAIRWISE
+    system_prompt: str = Field(description="System prompt to use for the evaluator")
+    user_prompt: str = Field(description="User prompt to use for the evaluator.")
+    bidirectional: bool = Field(default=False, description="Whether or not to run each game in both directions")
+    n_games_per_query: int = Field(default=100, description="Maximum number of games to generate for each query")
+    pairwise: bool = Field(default=True, description="Whether or not to the evaluator is pairwise")
+    llm_answer_format: AnswerFormat = Field(
+        default=AnswerFormat.JSON,
+        description="The format of the answer returned by the LLM",
+    )
+    llm_response_schema: Type[PydanticBaseModel] | dict[str, Any] | None = Field(
+        default={
+            "analysis_assistant_a": "A string with your analysis of assistant A's answer",
+            "analysis_assistant_b": "A string with your analysis of assistant B's answer",
+            "differences": "A string with your comparison between the two answers and their differences",
+            "winner": (
+                "The winner of the comparison. "
+                "'A' if assistant A is better, 'B' if assistant B is better, and 'C' for a tie"
+            ),
+        },
+        description=(
+            "The response schema for the LLM. "
+            "Required if the llm_answer_format is structured and recommended for JSON."
+        ),
+    )
 
 
 class CustomPromptAnswerEvaluatorConfig(BaseAnswerEvaluatorConfig):
-    evaluator_name: Union[
-        str, AnswerEvaluatorTypes
-    ] = AnswerEvaluatorTypes.CUSTOM_PROMPT
+    evaluator_name: AnswerEvaluatorTypes = AnswerEvaluatorTypes.CUSTOM_PROMPT
     prompt: str = Field(
         default="retrieved documents: {documents} query: {query} answer: {answer}",
-        description="The prompt to be used to evaluate the documents. It should contain a {query} and a {document} placeholder",
-    )
-    answers_evaluations_path: str = Field(
-        default="custom_prompt_answers_evaluations.csv",
-        description="Path to the output file",
-    )
-    scoring_keys_answer_evaluator: List[str] = Field(
-        default=["quality", "trustworthiness", "originality"],
-        description="The fields to extract from the answer",
+        description=(
+            "The prompt to be used to evaluate the documents. "
+            "It should contain a {query} and a {document} placeholder"
+        ),
     )
-    answer_format_answer_evaluator: Union[str, AnswerFormat] = Field(
-        default=AnswerFormat.MULTI_FIELD_JSON,
+    llm_answer_format: AnswerFormat = Field(
+        default=AnswerFormat.JSON,
         description="The format of the answer returned by the LLM",
     )
+    include_annotations: bool = False
+    include_raw_documents: bool = True
 
 
 class PairwiseDomainExpertEvaluatorConfig(PairwiseEvaluatorConfig):
-    evaluator_name: Union[
-        str, AnswerEvaluatorTypes
-    ] = AnswerEvaluatorTypes.DOMAIN_EXPERT
-    expert_in: str = Field(
-        default="",
-        description="What the LLM should mimic being an expert in.",
-    )
-    company: Optional[str] = Field(
+    evaluator_name: AnswerEvaluatorTypes = AnswerEvaluatorTypes.DOMAIN_EXPERT
+    expert_in: str = Field(description="What the LLM should mimic being an expert in.")
+    company: str | None = Field(
         default=None,
         description="Name of the company or organization that the user that "
         "submitted the query works for. that the domain belongs to. "
         "(e.g.: ChemCorp, CS Inc.)",
     )
+    include_annotations: bool = True
+    include_raw_documents: bool = False
diff --git a/ragelo/types/configurations/base_configs.py b/ragelo/types/configurations/base_configs.py
index bb0f8b8..8e1743b 100644
--- a/ragelo/types/configurations/base_configs.py
+++ b/ragelo/types/configurations/base_configs.py
@@ -1,11 +1,13 @@
-from importlib import metadata
-from typing import List, Optional, Union
+from __future__ import annotations
 
-from pydantic import Field
+from typing import Any, Type
 
-from ragelo.types.types import AnswerFormat, BaseModel
+from pydantic import BaseModel as PydanticBaseModel
+from pydantic import Field
 
-_PYDANTIC_MAJOR_VERSION: int = int(metadata.version("pydantic").split(".")[0])
+from ragelo.types.formats import AnswerFormat
+from ragelo.types.pydantic_models import BaseModel
+from ragelo.types.types import AnswerEvaluatorTypes
 
 
 class BaseConfig(BaseModel):
@@ -13,148 +15,39 @@ class BaseConfig(BaseModel):
         default=False,
         description="Force the execution of the commands and overwrite any existing files.",
     )
-    rich_print: bool = Field(
-        default=False, description="Use rich to print colorful outputs."
-    )
+    rich_print: bool = Field(default=False, description="Use rich to print colorful outputs.")
     verbose: bool = Field(
         default=False,
         description="Whether or not to be verbose and print all intermediate steps.",
     )
-    credentials_file: Optional[str] = Field(
-        default=None,
-        description="Path to a txt file with the credentials for the different LLM providers.",
-    )
-    data_dir: Optional[str] = Field(
-        default=None,
-        description="Path to the data folder. Defaults to the root folder where the script is being executed. Every file path in the configuration will be relative to this path.",
-    )
-
-    llm_provider_name: str = Field(
-        default="openai", description="The name of the LLM provider to be used."
-    )
-    write_output: bool = Field(
-        default=True, description="Whether or not to write the outputs to a file."
-    )
+    llm_provider_name: str = Field(default="openai", description="The name of the LLM provider to be used.")
     use_progress_bar: bool = Field(
         default=True,
         description="Whether or not to show a progress bar while running the evaluations.",
     )
-
-
-class BaseEvaluatorConfig(BaseConfig):
-    queries_file: str = Field(
-        default="queries.csv",
-        description="Input CSV file with the user queries answered by the agents",
-    )
-    documents_file: str = Field(
-        default="documents.csv",
-        description="Input CSV file with the documents retrieved by the agents",
-    )
-    answers_file: str = Field(
-        default="answers.csv", description="Input CSV file with agents answers."
-    )
-    document_evaluations_file: str = Field(
-        default="reasonings.csv",
-        description="CSV file to write (or read) the output of the retrieval evaluators on the retrieved documents.",
-    )
-    answers_evaluations_file: str = Field(
-        default="answers_evaluations.csv",
-        description="CSV file to write (or read) the evaluations of each agent's answers.",
-    )
-    games_evaluations_file: str = Field(
-        default="pairwise_answers_evaluations.csv",
-        description="CSV file to write (or read) the evaluations of the pairwise games between agents answers to queries.",
-    )
-    query_placeholder: str = Field(
-        default="query",
-        description="The placeholder for the query in the prompt.",
-    )
-    scoring_key_retrieval_evaluator: str = Field(
-        default="relevance",
-        description="When using answer_format=json, the key to extract from the answer for the retrieval evaluator.",
-    )
-    scoring_key_answer_evaluator: str = Field(
-        default="relevance",
-        description="When using answer_format=json, the key to extract from the answer for the answer evaluator.",
-    )
-    scoring_keys_answer_evaluator: List[str] = Field(
-        default=["relevance"],
-        description="When using answer_format=multi_field_json, the keys to extract from the answer for the answer evaluator.",
-    )
-    scoring_keys_retrieval_evaluator: List[str] = Field(
-        default=["relevance"],
-        description="When using answer_format=multi_field_json, the keys to extract from the answer for the retrieval evaluator.",
-    )
-    answer_format_retrieval_evaluator: Union[str, AnswerFormat] = Field(
-        default="text",
-        description="The format of the answer returned by the LLM for the retrieval evaluator.",
-    )
-    answer_format_answer_evaluator: Union[str, AnswerFormat] = Field(
-        default="json",
-        description="The format of the answer returned by the LLM for the answer evaluator.",
-    )
-    output_columns_retrieval_evaluator: List[str] = Field(
-        default=["qid", "did", "raw_answer", "answer"],
-        description="The columns to write in the --document_evaluations_file csv.",
-    )
-    output_columns_answer_evaluator: List[str] = Field(
-        default=["qid", "agent", "raw_answer", "answer"],
-        description="The columns to write in the --answers_evaluations_file csv.",
-    )
-    output_columns_pairwise_evaluator: List[str] = Field(
-        default=["qid", "agent_a", "agent_b", "raw_answer", "answer"],
-        description="The columns to write in the --games_evaluations_file csv.",
-    )
     n_processes: int = Field(
         default=1,
         description="The number of parallel LLM calls to use for the evaluation.",
     )
 
 
-class AllConfig(BaseEvaluatorConfig):
-    reasoning_file: str = Field(
-        default="reasonings.csv",
-        description="CSV file to write (or read) the reasonings generated by the `Reasoner` retrieval evaluator on the relevance of the documents retrieved by the agents.",
-    )
-    answers_evaluations_file: str = Field(
-        default="answers_evaluations.csv",
-        description="CSV file to write (or read) the evaluations of each agent's answers, generated by an `AnswerEvaluator`",
-    )
-    agents_evaluations_file: str = Field(
-        default="agents_ranking.csv", description="Path to the output file"
-    )
-    retrieval_evaluator_name: str = Field(
-        default="reasoner",
-        description="The name of the retrieval evaluator to use",
-    )
-    answer_evaluator_name: str = Field(
-        default="pairwise",
-        description="The name of the answer evaluator to use",
-    )
-    answer_ranker_name: str = Field(
-        default="elo", description="The name of the answer ranker to use"
-    )
-    scoring_key_retrieval_evaluator: str = Field(
-        default="answer",
-        description="When using answer_format=json, the key to extract from the answer for the retrieval evaluator.",
-    )
-    scoring_key_answer_evaluator: str = Field(
-        default="answer",
-        description="When using answer_format=json, the key to extract from the answer for the answer evaluator.",
-    )
-    k: int = Field(default=100, description="Number of games to generate")
-    initial_score: int = Field(
-        default=1000, description="The initial score for each agent"
-    )
-    elo_k: int = Field(
-        default=32, description="The K factor for the Elo ranking algorithm"
+class BaseEvaluatorConfig(BaseConfig):
+    evaluator_name: str | AnswerEvaluatorTypes | None = Field(
+        default=None,
+        description="The name of the evaluator to use.",
     )
-    bidirectional: bool = Field(
-        default=False, description="Wether or not to run each game in both directions"
+    query_placeholder: str = Field(
+        default="query",
+        description="The placeholder for the query in the prompt.",
     )
-    model: str = Field(
-        default="gpt-4o-mini", description="The model to use for the LLM"
+    llm_answer_format: AnswerFormat = Field(
+        default=AnswerFormat.JSON,
+        description="The format of the answer returned by the LLM.",
     )
-    rich_print: bool = Field(
-        default=True, description="Use rich to print colorful outputs."
+    llm_response_schema: Type[PydanticBaseModel] | dict[str, Any] | None = Field(
+        default=None,
+        description=(
+            "The response schema for the LLM. "
+            "Required if the llm_answer_format is structured and recommended for JSON."
+        ),
     )
diff --git a/ragelo/types/configurations/cli_configs.py b/ragelo/types/configurations/cli_configs.py
new file mode 100644
index 0000000..908ab54
--- /dev/null
+++ b/ragelo/types/configurations/cli_configs.py
@@ -0,0 +1,109 @@
+from __future__ import annotations
+
+from pydantic import Field
+
+from ragelo.types.configurations.answer_evaluator_configs import (
+    PairwiseDomainExpertEvaluatorConfig,
+    PairwiseEvaluatorConfig,
+)
+from ragelo.types.configurations.base_configs import BaseConfig
+from ragelo.types.configurations.retrieval_evaluator_configs import (
+    DomainExpertEvaluatorConfig,
+    RDNAMEvaluatorConfig,
+    ReasonerEvaluatorConfig,
+)
+
+
+class BaseCLIConfig(BaseConfig):
+    experiment_name: str = Field(
+        default="experiment",
+        description="The name of the experiment to run. This is also used as the local cache file name",
+    )
+    data_dir: str = Field(
+        default="data",
+        description="The directory where the data is stored.",
+    )
+    queries_csv_file: str = Field(
+        default="queries.csv",
+        description="The path to the queries CSV file. The file should contain at least the following columns: "
+        "qid, query. Any additional columns will be considered as metadata.",
+    )
+    verbose: bool = Field(
+        default=True,
+        description="Whether or not to be verbose and print all intermediate steps.",
+    )
+    output_file: str | None = Field(
+        default=None,
+        description="The path to the output file where the results will be saved.",
+    )
+    save_results: bool = Field(
+        default=True,
+        description="Whether or not to save the results to disk.",
+    )
+    rich_print: bool = Field(
+        default=True,
+        description="Use rich to print colorful outputs.",
+    )
+
+
+class CLIEvaluatorConfig(BaseCLIConfig):
+    documents_csv_file: str = Field(
+        default="documents.csv",
+        description=(
+            "The path to the documents CSV file. The file should contain at least the following columns: "
+            "qid, did, document. Any additional columns will be considered as metadata."
+        ),
+    )
+    answers_csv_file: str = Field(
+        default="answers.csv",
+        description="The path to the answers CSV file. The file should contain at least the following columns: "
+        "qid, agent, answer. Any additional columns will be considered as metadata. Ignored on Retrieval Evaluators.",
+    )
+
+
+class CLIDomainExpertEvaluatorConfig(CLIEvaluatorConfig, DomainExpertEvaluatorConfig):
+    expert_in: str = " "
+
+
+class CLIReasonerEvaluatorConfig(CLIEvaluatorConfig, ReasonerEvaluatorConfig):
+    pass
+
+
+class CLIRDNAMEvaluatorConfig(CLIEvaluatorConfig, RDNAMEvaluatorConfig):
+    pass
+
+
+class CLIPairwiseDomainExpertEvaluatorConfig(CLIEvaluatorConfig, PairwiseDomainExpertEvaluatorConfig):
+    add_reasoning: bool = Field(
+        default=False,
+        description="If set to True, a reasoning retrieval evaluator will run, and the reasoning of the quality "
+        "  of the retrieved results will be included in the prompt for the pairwise games.",
+    )
+    expert_in: str = " "
+
+
+class CLIPairwiseEvaluatorConfig(CLIEvaluatorConfig, PairwiseEvaluatorConfig):
+    add_reasoning: bool = Field(
+        default=False,
+        description="If set to True, a reasoning retrieval evaluator will run, and the reasoning of the quality "
+        "  of the retrieved results will be included in the prompt for the pairwise games.",
+    )
+
+
+class CLIConfig(BaseCLIConfig):
+    documents_csv_file: str = Field(
+        default="documents.csv",
+        description="The path to the documents CSV file. The file should contain at least the following columns: "
+        "qid, did, document. Any additional columns will be considered as metadata.",
+    )
+    answers_csv_file: str = Field(
+        default="answers.csv",
+        description="The path to the answers CSV file. The file should contain at least the following columns: "
+        "qid, did, answer. Any additional columns will be considered as metadata.",
+    )
+
+    k: int = Field(default=100, description="Number of pairwise games to generate")
+    initial_score: int = Field(default=1000, description="The initial Elo score for each agent")
+    elo_k: int = Field(default=32, description="The K factor for the Elo ranking algorithm")
+    bidirectional: bool = Field(default=False, description="Wether or not to run each game in both directions")
+    model: str = Field(default="gpt-4o-mini", description="The model to use for the LLM")
diff --git a/ragelo/types/configurations/llm_provider_configs.py b/ragelo/types/configurations/llm_provider_configs.py
index 12ac9cf..1abc175 100644
--- a/ragelo/types/configurations/llm_provider_configs.py
+++ b/ragelo/types/configurations/llm_provider_configs.py
@@ -1,27 +1,25 @@
-from typing import Optional
+from __future__ import annotations
 
 from ragelo.types.configurations.base_configs import BaseModel
 
 
 class LLMProviderConfig(BaseModel):
-    api_key: Optional[str]
-    max_retries: int = 3
-    sleep_time: int = 2
+    api_key: str | None
     temperature: float = 0.1
-    max_tokens: int = 512
+    max_tokens: int = 2048
 
 
 class OpenAIConfiguration(LLMProviderConfig):
-    org: Optional[str] = None
-    api_type: Optional[str] = None
-    api_base: Optional[str] = None
-    api_version: Optional[str] = None
+    org: str | None = None
+    api_type: str | None = None
+    api_base: str | None = None
+    api_version: str | None = None
     model: str = "gpt-4o-mini"
 
 
 class OllamaConfiguration(LLMProviderConfig):
-    org: Optional[str] = None
-    api_base: Optional[str] = "http://localhost:11434/v1/"
-    api_version: Optional[str] = None
-    api_key: Optional[str] = "NoKeyNeeded"
+    org: str | None = None
+    api_base: str | None = "http://localhost:11434/v1/"
+    api_version: str | None = None
+    api_key: str | None = "NoKeyNeeded"
     model: str = "gemma2:27b"
diff --git a/ragelo/types/configurations/retrieval_evaluator_configs.py b/ragelo/types/configurations/retrieval_evaluator_configs.py
index 0e8e01d..c3a5486 100644
--- a/ragelo/types/configurations/retrieval_evaluator_configs.py
+++ b/ragelo/types/configurations/retrieval_evaluator_configs.py
@@ -1,15 +1,33 @@
-from typing import List, Optional, Union
+from __future__ import annotations
 
+from typing import Any, Type
+
+from pydantic import BaseModel as PydanticBaseModel
 from pydantic import Field
 
+from ragelo.logger import logger
 from ragelo.types.configurations.base_configs import AnswerFormat, BaseEvaluatorConfig
-from ragelo.types.types import FewShotExample, RetrievalEvaluatorTypes
+from ragelo.types.pydantic_models import BaseModel, post_validator
+from ragelo.types.types import RetrievalEvaluatorTypes
+
+
+class FewShotExample(BaseModel):
+    """A few-shot example used in the few-shot retrieval evaluator.
+    Args:
+        passage str: The passage of the example.
+        query str: The query of the example.
+        relevance int: The relevance of the example.
+        reasoning str: The reasoning behind the relevance
+    """
+
+    passage: str
+    query: str
+    relevance: int
+    reasoning: str
 
 
 class BaseRetrievalEvaluatorConfig(BaseEvaluatorConfig):
-    evaluator_name: Union[
-        str, RetrievalEvaluatorTypes
-    ] = RetrievalEvaluatorTypes.CUSTOM_PROMPT
+    evaluator_name: str | RetrievalEvaluatorTypes = RetrievalEvaluatorTypes.CUSTOM_PROMPT
     document_placeholder: str = Field(
         default="document",
         description="The placeholder for the document in the prompt",
@@ -17,116 +35,86 @@ class BaseRetrievalEvaluatorConfig(BaseEvaluatorConfig):
 
 
 class ReasonerEvaluatorConfig(BaseRetrievalEvaluatorConfig):
-    answer_format_retrieval_evaluator: Union[str, AnswerFormat] = Field(
-        default="text",
+    evaluator_name: str | RetrievalEvaluatorTypes = RetrievalEvaluatorTypes.REASONER
+    llm_answer_format: AnswerFormat = Field(
+        default=AnswerFormat.TEXT,
         description="The format of the answer returned by the LLM",
     )
-    scoring_key_retrieval_evaluator: str = Field(
-        default="answer",
-        description="When using answer_format=json, the key to extract from the answer",
-    )
-    scoring_keys_retrieval_evaluator: List[str] = Field(
-        default=["relevance"],
-        description="When using answer_format=multi_field_json, the keys to extract from the answer",
-    )
-    document_evaluations_file: str = Field(
-        default="reasonings.csv",
-        description="Path to write (or read) the evaluations of the retrieved documents",
-    )
-    output_columns_retrieval_evaluator: List[str] = Field(
-        default=["qid", "did", "raw_answer", "answer"],
-        description="The columns to output in the CSV file",
-    )
-    evaluator_name: Union[
-        str, RetrievalEvaluatorTypes
-    ] = RetrievalEvaluatorTypes.REASONER
+
+    @post_validator
+    @classmethod
+    def check_answer_format(cls, values):
+        if values.llm_answer_format != AnswerFormat.TEXT:
+            logger.warning("We are using the Reasoner Evaluator config. Forcing the LLM answer format to TEXT.")
+            values.llm_answer_format = AnswerFormat.TEXT
+        return values
 
 
 class DomainExpertEvaluatorConfig(BaseRetrievalEvaluatorConfig):
-    expert_in: str = Field(
-        default="",
-        description="What the LLM should mimic being an expert in.",
-    )
-    domain_short: Optional[str] = Field(
+    evaluator_name: str | RetrievalEvaluatorTypes = RetrievalEvaluatorTypes.DOMAIN_EXPERT
+    expert_in: str = Field(description="What the LLM should mimic being an expert in.")
+    domain_short: str | None = Field(
         default=None,
-        description="A short or alternative name of the domain. "
-        "(e.g., Chemistry, CS, etc.)",
+        description="A short or alternative name of the domain. " "(e.g., Chemistry, CS, etc.)",
     )
-    company: Optional[str] = Field(
+    company: str | None = Field(
         default=None,
         description="Name of the company or organization that the user that "
         "submitted the query works for. that the domain belongs to. "
         "(e.g.: ChemCorp, CS Inc.)",
     )
-    extra_guidelines: Optional[List[str]] = Field(
+    extra_guidelines: list[str] | None = Field(
         default=None,
-        description="A list of extra guidelines to be used when reasoning about the "
-        "relevancy of the document.",
+        description="A list of extra guidelines to be used when reasoning about the " "relevancy of the document.",
     )
-    document_evaluations_file: str = Field(
-        default="domain_expert_evaluations.csv",
-        description="Path to write (or read) the evaluations of the retrieved documents",
-    )
-    answer_format_retrieval_evaluator: str = Field(
-        default="text",
+    llm_answer_format: AnswerFormat = Field(
+        default=AnswerFormat.JSON,
         description="The format of the answer returned by the LLM",
     )
-    scoring_key_retrieval_evaluator: str = Field(
-        default="score",
-        description="The field to use when parsing the llm answer",
-    )
-    evaluator_name: Union[
-        str, RetrievalEvaluatorTypes
-    ] = RetrievalEvaluatorTypes.DOMAIN_EXPERT
-    output_columns_retrieval_evaluator: List[str] = Field(
-        default=["qid", "did", "reasoning", "score"],
-        description="The columns to output in the CSV file",
+    llm_response_schema: Type[PydanticBaseModel] | dict[str, Any] | None = Field(
+        default={
+            "score": (
+                "An integer between 0 and 2 representing the score of the document, "
+                "where 0 means the document is not relevant to the query, 1 means the document is somewhat relevant, "
+                "and 2 means the document is highly relevant."
+            )
+        },
     )
 
 
 class CustomPromptEvaluatorConfig(BaseRetrievalEvaluatorConfig):
+    evaluator_name: str | RetrievalEvaluatorTypes = RetrievalEvaluatorTypes.CUSTOM_PROMPT
     prompt: str = Field(
         default="query: {query} document: {document}",
-        description="The prompt to be used to evaluate the documents. It should contain a {query} and a {document} placeholder",
-    )
-    scoring_keys_retrieval_evaluator: List[str] = Field(
-        default=["quality", "trustworthiness", "originality"],
-        description="The fields to use when parsing the llm answer",
-    )
-    answer_format_retrieval_evaluator: Union[str, AnswerFormat] = Field(
-        default="multi_field_json",
-        description="The format of the answer returned by the LLM",
+        description=(
+            "The prompt to be used to evaluate the documents. "
+            "It should contain a {query} and a {document} placeholder"
+        ),
     )
-    document_evaluations_file: str = Field(
-        default="custom_prompt_evaluations.csv",
-        description="Path to write (or read) the evaluations of the retrieved documents",
-    )
-    evaluator_name: Union[
-        str, RetrievalEvaluatorTypes
-    ] = RetrievalEvaluatorTypes.CUSTOM_PROMPT
 
 
 class FewShotEvaluatorConfig(BaseRetrievalEvaluatorConfig):
+    evaluator_name: str | RetrievalEvaluatorTypes = RetrievalEvaluatorTypes.FEW_SHOT
     system_prompt: str = Field(
         default="You are a helpful assistant.",
         description="The system prompt to be used to evaluate the documents.",
     )
-    few_shots: List[FewShotExample] = Field(
-        default=[], description="A list of few-shot examples to be used in the prompt"
+    few_shots: list[FewShotExample] = Field(
+        default_factory=list,
+        description="A list of few-shot examples to be used in the prompt",
     )
     few_shot_user_prompt: str = Field(
         default="Query: {query}\n\nPassage:{passage}",
-        description="The individual prompt to be used to evaluate the documents. It should contain a {query} and a {passage} placeholder",
+        description=(
+            "The individual prompt to be used to evaluate the documents. "
+            "It should contain a {query} and a {passage} placeholder"
+        ),
     )
     few_shot_assistant_answer: str = Field(
         default='{reasoning}\n\n{{"relevance": {relevance}}}',
         description="The expected answer format from the LLM for each evaluated document "
         "It should contain a {reasoning} and a {relevance} placeholder",
     )
-    document_evaluations_file: str = Field(
-        default="few_shot_evaluations.csv",
-        description="Path to the output file",
-    )
     reasoning_placeholder: str = Field(
         default="reasoning",
         description="The placeholder for the reasoning in the prompt",
@@ -135,13 +123,11 @@ class FewShotEvaluatorConfig(BaseRetrievalEvaluatorConfig):
         default="relevance",
         description="The placeholder for the relevance in the prompt",
     )
-    evaluator_name: Union[
-        str, RetrievalEvaluatorTypes
-    ] = RetrievalEvaluatorTypes.FEW_SHOT
 
 
 class RDNAMEvaluatorConfig(BaseRetrievalEvaluatorConfig):
-    annotator_role: Optional[str] = Field(
+    evaluator_name: str | RetrievalEvaluatorTypes = RetrievalEvaluatorTypes.RDNAM
+    annotator_role: str | None = Field(
         default=None,
         description="A String defining the type of user the LLM should mimic. "
         "(e.g.: 'You are a search quality rater evaluating the relevance "
@@ -157,11 +143,15 @@ class RDNAMEvaluatorConfig(BaseRetrievalEvaluatorConfig):
         default=False,
         description="Should the prompt ask the LLM to mimic multiple annotators?",
     )
-    document_evaluations_file: str = Field(
-        default="rdnam_evaluations.csv", description="Path to the output file"
-    )
-    scoring_key_retrieval_evaluator: str = Field(
-        default="answer",
-        description="The field to use when parsing the llm answer",
+    llm_answer_format: AnswerFormat = Field(
+        default=AnswerFormat.JSON,
+        description="The format of the answer returned by the LLM",
     )
-    evaluator_name: Union[str, RetrievalEvaluatorTypes] = RetrievalEvaluatorTypes.RDNAM
+
+    @post_validator
+    @classmethod
+    def check_answer_format(cls, values):
+        if values.llm_answer_format != AnswerFormat.JSON:
+            logger.warning("We are using the RDNAM Evaluator config. Forcing the LLM answer format to JSON.")
+            values.llm_answer_format = AnswerFormat.JSON
+        return values
diff --git a/ragelo/types/evaluables.py b/ragelo/types/evaluables.py
new file mode 100644
index 0000000..b7b9c44
--- /dev/null
+++ b/ragelo/types/evaluables.py
@@ -0,0 +1,163 @@
+from __future__ import annotations
+
+from typing import Any
+
+from ragelo.logger import logger
+from ragelo.types.pydantic_models import BaseModel, validator
+from ragelo.types.results import EvaluatorResult
+
+
+class ChatMessage(BaseModel):
+    sender: str
+    content: str
+
+    def __str__(self) -> str:
+        return f"{self.sender}: {self.content}"
+
+
+class Evaluable(BaseModel):
+    """A base class for objects that can be evaluated. Either a retrieved document or an agent answer.
+    Args:
+        evaluation Optional[EvaluatorResult]: The result of the evaluation.
+        metadata Optional[dict[str, Any]]: Metadata that can be templated in the prompt.
+    """
+
+    qid: str
+    evaluation: EvaluatorResult | None = None
+    metadata: dict[str, Any] | None = None
+
+    def add_metadata(self, metadata: dict[str, Any] | None):
+        if not metadata:
+            return
+        if self.metadata is None:
+            self.metadata = {}
+        for k in metadata:
+            if k in self.metadata:
+                logger.warning(
+                    f"Metadata {k} for {self.__class__.__name__}"
+                    " is being overwritten!\n"
+                    f"Old metadata: {self.metadata[k]}\n"
+                    f"New metadata: {metadata[k]}\n"
+                )
+            self.metadata[k] = metadata[k]
+
+
+class Document(Evaluable):
+    """A document retrieved by an agent in response to a query.
+    Args:
+        did str: The document ID.
+        text str: The text of the document.
+        retrieved_by dict[str, float]: If a document was retrieved by multiple agents,
+            the score attributed by each agent for this document.
+    """
+
+    did: str
+    text: str
+    retrieved_by: dict[str, float] = {}
+
+    def add_retrieved_by(self, agent: str, score: float | None = None, force: bool = False):
+        """Adds the score of an agent that retrieved the document."""
+        if agent in self.retrieved_by and not force:
+            logger.info(f"Document with did {self.did} already retrieved by agent {agent}")
+            return
+        if score is None:
+            score = 1.0
+        self.retrieved_by[agent] = score
+
+    def __str__(self) -> str:
+        if len(self.text) > 100:
+            return f"{self.did}: {self.text[:100]}..."
+        return f"{self.did}: {self.text}"
+
+    @classmethod
+    def assemble_document(
+        cls,
+        document: Document | str,
+        qid: str | None = None,
+        metadata: dict[str, Any] | None = None,
+    ) -> Document:
+        """Assembles a Document object from a string or a Document object."""
+        if isinstance(document, str):
+            if qid is None:
+                raise ValueError("qid must be provided if document is a string")
+            did = "<no_did>"
+            if metadata:
+                valid_id_fields = ["did", "doc_id", "document_id", "id", "_id"]
+                valid_id_fields = [f for f in valid_id_fields if f in metadata]
+                if valid_id_fields:
+                    did = metadata[valid_id_fields[0]]
+            document = cls(qid=qid, did=did, text=document)
+        document.add_metadata(metadata)
+        return document
+
+    @staticmethod
+    def assemble_documents(
+        documents: list[Document | str],
+        qid: str,
+        metadata: list[dict[str, Any]] | list[None] | None = None,
+    ) -> dict[str, Document]:
+        """Assembles a list of Document objects from a list of strings or Document objects."""
+        assembled_docs: dict[str, Document] = {}
+        if metadata and len(documents) != len(metadata):
+            raise ValueError("The number of documents and document metadata do not match")
+        if not metadata:
+            metadata = [None] * len(documents)
+
+        for idx, (doc, m) in enumerate(zip(documents, metadata)):
+            doc_obj = Document.assemble_document(doc, qid, m)
+            if doc_obj.did == "<no_did>":
+                doc_obj.did = f"doc_{idx}"
+            assembled_docs[doc_obj.did] = doc_obj
+        return assembled_docs
+
+
+class AgentAnswer(Evaluable):
+    """An answer generated by an agent in response to a query.
+    Args:
+        agent str: The agent that provided the answer.
+        text str: The text of the answer.
+        conversation Optional[list[ChatMessage]]: The conversation between the user and the agent.
+    """
+
+    agent: str
+    text: str | None = None
+    conversation: list[ChatMessage] | None = None
+
+    @validator
+    @classmethod
+    def one_of_text_or_conversation(cls, values):
+        text = values.get("text")
+        conversation = values.get("conversation")
+        if text is None and conversation is None:
+            raise ValueError("Either text or conversation must be provided")
+
+        if text is not None and conversation is not None:
+            raise ValueError("Only one of text or conversation must be provided")
+        return values
+
+    @classmethod
+    def assemble_answer(
+        cls,
+        answer: AgentAnswer | str,
+        qid: str,
+        agent: str | None = None,
+        metadata: dict[str, Any] | None = None,
+    ) -> AgentAnswer:
+        """Assembles an AgentAnswer object from a string or an AgentAnswer object."""
+        if isinstance(answer, str):
+            if agent is None:
+                raise ValueError("agent must be provided if answer is a string")
+            answer = cls(agent=agent, qid=qid, text=answer)
+        answer.add_metadata(metadata)
+        return answer
+
+
+class PairwiseGame(Evaluable):
+    """A game to be played between two agent answers.
+    Args:
+        agent_a_answer AgentAnswer: The answer provided by the first agent.
+        agent_b_answer AgentAnswer: The answer provided by the second agent.
+    """
+
+    agent_a_answer: AgentAnswer
+    agent_b_answer: AgentAnswer
diff --git a/ragelo/types/experiment.py b/ragelo/types/experiment.py
new file mode 100644
index 0000000..bfbb1b9
--- /dev/null
+++ b/ragelo/types/experiment.py
@@ -0,0 +1,903 @@
+"""
+An Experiment is a collection of queries, and manages their retrieved documents, agent answers and their respective
+evaluations created by Evaluators.
+For most use cases, this class should be the main entry point to interact with RAGElo.
+"""
+
+from __future__ import annotations
+
+import csv
+import json
+import os
+import warnings
+from pathlib import Path
+from typing import Any, Literal
+
+import rich
+from pydantic import BaseModel as PydanticBaseModel
+
+from ragelo.logger import logger
+from ragelo.types.evaluables import AgentAnswer, Document
+from ragelo.types.pydantic_models import _PYDANTIC_MAJOR_VERSION
+from ragelo.types.query import Query
+from ragelo.types.results import (
+    AnswerEvaluatorResult,
+    EloTournamentResult,
+    EvaluatorResult,
+    RetrievalEvaluatorResult,
+)
+
+
+class Experiment:
+    """
+    A class to manage and evaluate an experiment. An experiment is composed of multiple queries and their evaluations.
+    This is the main class for interacting with RAGElo, and manages all the queries to which agents retrieve documents
+      and answer questions.
+    Attributes:
+        experiment_name (str): An unique name for your experiment.
+        queries: (dict[str, Query]): A dictionary with the queries for this experiment.
+            Can also be accessed directly with __getitem__.
+        elo_tournaments: (list[EloTournamentResult]): A list with the results for Elo Tournaments
+            played over this set of queries.
+    Methods:
+        add_query(query, query_id, doc_id, metadata, force): Adds a query
+            (either a Query object or a string) to the current experiment. Returns the id of the added query.
+        add_retrieved_doc(doc, query_id, doc_id, score, agent, force): Adds a retrieved document to a query
+        add_agent_answer(answer, agent, query_id, force): Adds an agent's answer to a query.
+        add_evaluation(evaluation, should_save, force): Adds an evaluation to a retrieved document,
+            an answer or a pairwise game.
+        evaluate_retrieval(metrics, relevance_threshold): Evaluates the retrieval performance of the agents
+            using the specified metrics and the evaluations generated by a `RetrievalEvaluator`.
+            This requires the ir_measures package to be installed.
+        get_qrels(relevance_key, relevance_threshold, output_path, output_format): Gets the qrels
+            (dictionary of relevance) for the queries. Documents with a score above relevance_threshold are considered
+            relevant. Optionally saves the result to output_path with output_format.
+        get_runs(agents, output_path, output_format): Gets the runs (retrieval results) for the provided agents for
+            all the queries. Optionally saves the result to output_path with output_format.
+        save(): Saves the current state of the queries object to disk as a JSON object to `cache_path`.
+        save_results(result): Saves an evaluation result to the `evaluations_cache_path` file.
+        add_retrieved_docs_from_runfile(run_file_path, corpus, top_k): Adds retrieved documents from a trec-formatted
+            run file to the queries. Will create the Document objects as needed.
+
+        __len__(): Returns the number of queries.
+        __iter__(): Returns an iterator over the queries.
+        __getitem__(key: str) -> Query: Gets a query by its key.
+        keys(): Returns the keys (query_ids) of all queries.
+    """
+
+    def __init__(
+        self,
+        experiment_name: str,
+        save_path: str | None = None,
+        evaluations_cache_path: str | None = None,
+        cache_evaluations: bool = True,
+        save_on_disk: bool = True,
+        verbose: bool = False,
+        rich_print: bool = True,
+        clear_evaluations: bool = False,
+        queries_csv_path: str | None = None,
+        csv_query_id_col: str | None = None,
+        csv_query_text_col: str = "query",
+        documents_csv_path: str | None = None,
+        csv_document_id_col: str | None = None,
+        csv_document_text_col: str = "document",
+        answers_csv_path: str | None = None,
+        csv_agent_col: str = "agent",
+        csv_answer_text_col: str = "answer",
+    ):
+        """
+        Initialize the experiment object with optional data and set up paths and caches.
+
+        Args:
+            experiment_name (str): The name of your experiment
+            save_path (Optional[str]): A JSON file path to persist the experiment on disk.
+                If not set, results will not be saved locally. If it already exists,
+                we will try to load the state from that file.
+            evaluations_cache_path (Optional[str]): A JSON Lines file path to persist the evaluators result on disk,
+                to avoid re-computing evaluations. If not set, will create one based on experiment_name.
+            cache_evaluations (bool, defaults to True): Whether or not to cache the evaluations on evaluations_cache_path,
+            save_on_disk (bool, defaults to True): Whether to save the experiment to disk using the experiment_name.
+            verbose (bool, defaults to False): Whether to print information about the experiment.
+            rich_print (bool, defaults to True): Whether to print information about the experiment using the rich library.
+            clear_evaluations (bool, defaults to False): If set to True, will clear all existing evaluations and
+                re-compute as needed.
+            queries_csv_path (Optional[str]): An optional path to a CSV file with the queries to be used.
+            csv_query_id_col (Optional[str]): The column name for query ID in the CSV files.
+            csv_query_text_col (str): The column name for query text in the queries CSV file.
+            documents_csv_path (Optional[str]): An optional path to a CSV file with the retrieved documents for
+                the queries.
+            csv_document_id_col (Optional[str]): The column name for document ID in the documents CSV file.
+            csv_document_text_col (str): The column name for document text in the documents CSV file.
+            answers_csv_path (Optional[str]): An optional path to a CSV file with the answers for the queries.
+            csv_agent_col (str, defaults to "agent"): The column name for agent name in the documents and answers
+                CSV files.
+            csv_answer_text_col (str, defaults to "answer"): The column name for answer text in the answers CSV file.
+
+
+
+        The initialization process involves setting up paths and loading queries from different sources:
+        1. If `csv_path` is provided, queries are loaded from the specified CSV file.
+        2. If `csv_path` is not provided but `save_path` exists and points to a valid file, queries are loaded from
+            the cache file.
+        3. If both `csv_path` and `save_path` are provided, the method ensures that the queries loaded from the CSV
+            file match those in the cache file.
+        The method prioritizes loading queries from the CSV file over the cache file. If neither source is available,
+            it relies on the `queries` attribute if it has been set.
+        """
+        self.cache_evaluations = cache_evaluations
+        self.evaluations_cache_path = evaluations_cache_path
+        self.experiment_name = experiment_name
+        self.queries: dict[str, Query] = {}
+        self.elo_tournaments: list[EloTournamentResult] = []
+        self.verbose = verbose
+        self.rich_print = rich_print
+        self.save_on_disk = save_on_disk
+        self.save_path = save_path
+
+        if self.save_on_disk and not self.save_path:
+            logger.info("Creating a save path for the experiment")
+            os.makedirs("ragelo_cache", exist_ok=True)
+            self.save_path = f"ragelo_cache/{self.experiment_name}.json"
+        elif self.save_path is None and not self.save_on_disk:
+            logger.info("No save path provided. Will not save the experiment to disk.")
+
+        if self.cache_evaluations and not self.evaluations_cache_path:
+            logger.info("Creating a cache path for the evaluations cache")
+            if self.save_path:
+                self.evaluations_cache_path = self.save_path.replace(".json", "_results.jsonl")
+            else:
+                os.makedirs("ragelo_cache", exist_ok=True)
+                self.evaluations_cache_path = f"ragelo_cache/{self.experiment_name}_results.jsonl"
+            Path(self.evaluations_cache_path).touch()
+
+        if self.save_path and os.path.isfile(self.save_path):
+            self.queries = self._load_from_cache(self.save_path)
+        else:
+            self.queries = {}
+        if queries_csv_path:
+            self.add_queries_from_csv(queries_csv_path, csv_query_id_col, csv_query_text_col)
+        if documents_csv_path:
+            self.add_documents_from_csv(
+                documents_csv_path,
+                csv_document_id_col,
+                csv_document_text_col,
+                csv_query_id_col,
+                csv_agent_col,
+            )
+        if answers_csv_path:
+            self.add_agent_answers_from_csv(answers_csv_path, csv_agent_col, csv_answer_text_col, csv_query_id_col)
+
+        if self.evaluations_cache_path and os.path.isfile(self.evaluations_cache_path):
+            self._load_results_from_cache(self.evaluations_cache_path)
+        if clear_evaluations:
+            self.__clear_all_evaluations()
+        self.save()
+
+    def add_query(
+        self,
+        query: Query | str,
+        query_id: str | None = None,
+        metadata: dict | None = None,
+        force: bool = False,
+    ) -> str:
+        """
+        Adds a query to the collection of queries.
+        Args:
+            query (Query | str): The query to be added. It can be an instance of the Query class or a string.
+            query_id (str | None, optional): The identifier for the query.
+                If not provided, a default ID will be generated.
+            metadata (dict | None, optional): Additional metadata for the query. Defaults to None.
+            force (bool, optional): Whether to overwrite the query if it already exists. Defaults to False.
+        """
+
+        if isinstance(query, Query):
+            if query_id is not None and query.qid != query_id:
+                warnings.warn(f"Query ID mismatch. Using ID from query object: {query.qid}")
+            query_id = query.qid
+            query_obj = query
+        elif query_id is None:
+            query_id = f"query_{len(self.queries) + 1}"
+            warnings.warn(f"Query ID not provided. Using default query ID: {query_id}.")
+            query_obj = Query(qid=query_id, query=query, metadata=metadata)
+        else:
+            query_obj = Query(qid=query_id, query=query, metadata=metadata)
+        if query_id in self.queries and not force:
+            warnings.warn(f"Query {query_id} already exists. Use force=True to overwrite")
+            return query_id
+        if query_id in self.queries and force:
+            logger.info(f"Query {query_id} already exists, but force was set to True. Overwriting.")
+        self.queries[query_id] = query_obj
+        return query_id
+
+    def add_retrieved_doc(
+        self,
+        doc: Document | str,
+        query_id: str | None = None,
+        doc_id: str | None = None,
+        score: float | None = None,
+        agent: str | None = None,
+        force: bool = False,
+    ):
+        """
+        Adds a retrieved document to a query.
+        Args:
+            doc (Document | str): The document to be added. It can be an instance of the Document class or a string.
+            query_id (Optional[str]): The query used to retrieve the document.
+            doc_id (Optional[str]): The identifier for the document. If not provided, a default ID will be generated.
+            score (Optional[float]): The score of the document. Defaults to None.
+            agent (Optional[str]): The agent that retrieved the document. Defaults to None.
+            force (bool): Whether to overwrite the document if it already exists. Defaults to False.
+        """
+        if isinstance(doc, Document):
+            if doc.qid != query_id and query_id is not None:
+                logger.warning(f"Query ID mismatch. Using ID from document object: {doc.qid}")
+                query_id = doc.qid
+            if doc.did != doc_id and doc_id is not None:
+                logger.warning(f"Document ID mismatch. Using ID from document object: {doc.did}")
+            doc_id = doc.did
+            query_id = doc.qid
+        elif doc_id is None:
+            raise ValueError("Document was provided as string, but Document ID was not provided")
+        if query_id is None:
+            raise ValueError(f"ID of the query to which document {doc_id} was retrieved was not provided.")
+        if isinstance(doc, str):
+            doc = Document(qid=query_id, did=doc_id, text=doc)
+        if query_id not in self.queries:
+            raise ValueError(f"Query {query_id} not found in queries")
+        self.queries[query_id].add_retrieved_doc(doc, score, agent, force)
+
+    def add_agent_answer(
+        self,
+        answer: AgentAnswer | str,
+        agent: str | None = None,
+        query_id: str | None = None,
+        force: bool = False,
+    ):
+        if isinstance(answer, AgentAnswer):
+            if answer.qid != query_id and query_id is not None:
+                logger.warning(f"Query ID mismatch. Using ID from answer object: {answer.qid}")
+            query_id = answer.qid
+            if answer.agent != agent and agent is not None:
+                logger.warning(f"Agent mismatch. Using agent from answer object: {answer.agent}")
+        elif query_id is None:
+            raise ValueError("Query ID not provided")
+        else:
+            if agent is None:
+                agent = f"agent_{len(self.queries[query_id].answers) + 1}"
+                logger.warning(f"Agent not provided. Using default agent name: {agent}.")
+            if query_id is None:
+                raise ValueError("Query ID not provided")
+            answer = AgentAnswer(qid=query_id, agent=agent, text=answer)
+        if query_id not in self.queries:
+            raise ValueError(f"Query {query_id} not found in queries")
+        self.queries[query_id].add_agent_answer(answer, force=force)
+
+    def add_evaluation(
+        self,
+        evaluation: EvaluatorResult | EloTournamentResult,
+        should_save: bool = True,
+        should_print: bool = True,
+        force: bool = False,
+        exist_ok: bool = False,
+    ):
+        """
+        Adds an evaluation to the queries and optionally saves the result.
+
+        Args:
+            evaluation (RetrievalEvaluatorResult | AnswerEvaluatorResult): The evaluation result to be added.
+            should_save (bool): Flag indicating whether the result should be persist to disk. Defaults to True.
+            force (bool): Whether to overwrite an evaluation if it already exists.
+        """
+        if isinstance(evaluation, EloTournamentResult):
+            self.elo_tournaments.append(evaluation)
+            added = True
+        else:
+            qid = evaluation.qid
+            if qid not in self.queries:
+                raise ValueError(f"Query {qid} not found in queries")
+            added = self.queries[qid].add_evaluation(evaluation, force=force, exist_ok=exist_ok)
+        if not added:
+            return
+        if should_save:
+            self.save_results(evaluation)
+        if not self.verbose:
+            return
+        if not should_print:
+            return
+        if isinstance(evaluation, EloTournamentResult):
+            self._print_elo_tournament_result(evaluation)
+        else:
+            self._print_response(evaluation)
+
+    def _print_elo_tournament_result(self, result: EloTournamentResult):
+        scores = sorted(result.scores.items(), key=lambda x: x[1], reverse=True)
+        if self.rich_print:
+            rich.print("-------[bold white] Agents Elo Ratings [/bold white]-------")
+        else:
+            print("------- Agents Elo Ratings -------")
+        for agent, rating in scores:
+            std_dev = result.std_dev.get(agent, 0)
+            if self.rich_print:
+                rich.print(f"[bold white]{agent:<15}[/bold white]: {rating:.1f}(±{std_dev:.1f})")
+            else:
+                print(f"{agent:<15}: {rating:.1f} (±{std_dev:.1f})")
+
+    def _print_response(
+        self,
+        response: EvaluatorResult,
+    ):
+        answer: float | str | dict | int
+        if isinstance(response.answer, dict):
+            # Print the answer in a more readable format
+            answer = json.dumps(response.answer, indent=4, ensure_ascii=False)
+        elif isinstance(response.answer, PydanticBaseModel):
+            if _PYDANTIC_MAJOR_VERSION >= 2:
+                answer = response.answer.model_dump_json(indent=4)  # type: ignore
+            else:
+                answer = response.answer.schema_json(indent=4)  # type: ignore
+        else:
+            answer = str(response.answer)
+        response_dict = response.model_dump()
+        agent_a = response_dict.get("agent_a")
+        agent_b = response_dict.get("agent_b")
+        agent = response_dict.get("agent")
+        qid = response_dict.get("qid")
+        did = response_dict.get("did")
+        raw_answer = response_dict.get("raw_answer")
+
+        if self.rich_print:
+            rich.print(f"[bold blue]🔎 Query ID[/bold blue]: {qid}")
+            did = response_dict.get("did")
+            if did:
+                rich.print(f"[bold blue]📜 Document ID[/bold blue]: {did}")
+            if agent_a and agent_b:
+                rich.print(
+                    f"[bold bright_cyan] {agent_a:<18} [/bold bright_cyan] 🆚  " f"[bold red] {agent_b}[/bold red]"
+                )
+            elif agent:
+                rich.print(f"[bold bright_cyan]🕵️ Agent[/bold bright_cyan]: {agent}")
+                if raw_answer != answer:
+                    rich.print(f"[bold blue]Raw Answer[/bold blue]: {raw_answer}")
+            rich.print(f"[bold blue]Parsed Answer[/bold blue]: {answer}")
+            rich.print("")
+            return
+        print(f"Query ID: {qid}")
+        if did:
+            print(f"Document ID: {did}")
+        if agent_a and agent_b:
+            print(f"{agent_a} vs {agent_b}")
+        elif agent:
+            print(f"Agent: {agent}")
+        print(f"Raw Answer: {raw_answer}")
+        print(f"Parsed Answer: {answer}")
+        print("")
+
+    def evaluate_retrieval(
+        self,
+        metrics: list[str] = ["Precision@10", "nDCG@10", "Judged@10"],
+        relevance_threshold: int = 0,
+    ) -> dict[str, dict[str, float]]:
+        """
+        Evaluate the retrieval performance of agents using specified metrics.
+        Args:
+            metrics (list[str]): A list of metrics to use. defaults to ["Precision@10", "nDCG@10", "Judged@10"].
+        relevance_threshold (int): The threshold above which a document is considered relevant (default is 0).
+        Returns:
+            dict[str, dict[str, float]]: A dictionary where keys are agent names and values are dictionaries of scores.
+        Raises:
+            ImportError: If the `ir_measures` package is not installed.
+            ValueError: If a specified metric is not found in the `ir_measures` registry.
+        Notes:
+        - The function uses the `ir_measures` package to calculate the metrics.
+        - If the `rich` package is installed, the results are printed in a formatted table.
+        - If the `rich` package is not installed, the results are printed using plain `print`.
+
+        Example:
+        >>> results = evaluate_retrieval(metrics=["Precision@10", "nDCG@10"], relevance_threshold=1)
+        >>> print(results)
+        {
+            "agent1": {
+                "Precision@10": 0.5,
+                "nDCG@10": 0.6
+            },
+            "agent2": {
+                "Precision@10": 0.4,
+                "nDCG@10": 0.5
+            }
+        }
+        """
+
+        try:
+            import ir_measures
+            from ir_measures import parse_measure
+        except ImportError:
+            raise ImportError("ir_measures is not installed. Please install it with `pip install ir-measures`")
+        qrels = self.get_qrels(relevance_threshold=relevance_threshold)
+        runs = self.get_runs()
+        measures = []
+        for metric in metrics:
+            try:
+                measure = parse_measure(metric)
+            except NameError:
+                valid_metrics = list(ir_measures.measures.registry.keys())
+                raise ValueError(f"Metric {metric} not found. Valid metrics are: {valid_metrics}")
+            measures.append(measure)
+        metrics = [str(m) for m in measures]
+        results = {}
+        for agent, run in runs.items():
+            # Transform the keys of the results back to strings
+            results[agent] = {
+                str(k): v
+                for k, v in ir_measures.calc_aggregate(measures, qrels, run).items()  # type: ignore
+            }
+
+        key_metric = metrics[0]
+        max_agent_len = max([len(agent) for agent in results.keys()]) + 3
+        max_metric_len = max([len(metric) for metric in metrics])
+        sorted_agents = sorted(results.items(), key=lambda x: x[1][key_metric], reverse=True)
+        if self.rich_print:
+            rich.print("---[bold cyan] Retrieval Scores [/bold cyan] ---")
+            if relevance_threshold > 0:
+                rich.print(f"[bold yellow]Relevance threshold: {relevance_threshold}[/bold yellow]")
+            header = f"[bold magenta]{'Agent Name':<{max_agent_len}}"
+            header += "\t".join([f"{m:<{max_metric_len}}" for m in metrics])
+            header += "[/bold magenta]"
+            rich.print(f"[bold cyan]{header}[/bold cyan]")
+            for agent, scores in sorted_agents:
+                row = f"[bold white]{agent:<{max_agent_len}}[/bold white]"
+                row += "\t".join([f"{scores[metric]:<{max_metric_len},.4f}" for metric in metrics])
+                rich.print(row)
+
+        else:
+            print(results)
+        return results
+
+    def get_qrels(
+        self,
+        relevance_key: str | None = "answer",
+        relevance_threshold: int = 0,
+        output_path: str | None = None,
+        output_format: Literal["trec", "json"] = "trec",
+    ) -> dict[str, dict[str, int]]:
+        """
+        Retrieve the qrels (query relevance judgments) for the queries.
+        Args:
+            relevance_key (str | None): The key to use for determining relevance. Defaults to "answer".
+            relevance_threshold (int): The threshold value for relevance. Defaults to 0.
+        Returns:
+            dict[str, dict[str, int]]: A dictionary where each key is a query ID and the value is another dictionary
+                                       mapping document IDs to their relevance scores.
+        """
+
+        qrels = {}
+        for qid, query in self.queries.items():
+            qrels[qid] = query.get_qrels(relevance_key, relevance_threshold)
+        if output_path:
+            with open(output_path, "w") as f:
+                if output_format.lower() == "trec":
+                    for qid, qrel in qrels.items():
+                        for did, rel in qrel.items():
+                            f.write(f"{qid} Q0 {did} {rel}\n")
+                elif output_format.lower() == "json":
+                    json.dump(qrels, f)
+                else:
+                    raise ValueError(
+                        f"Invalid output format for QRELS: {output_format}" "Valid options are 'trec' and 'json'"
+                    )
+        return qrels
+
+    def get_runs(
+        self,
+        agents: list[str] | None = None,
+        output_path: str | None = None,
+        output_format: str = "trec",
+    ) -> dict[str, dict[str, dict[str, float]]]:
+        """
+        Retrieve runs for specified agents.
+        This method aggregates runs from all queries and organizes them by agent.
+        Args:
+            agents (list[str] | None): A list of agent names to filter the runs.
+                If None, runs for all agents are retrieved.
+            output_path (str | None): The path to save the runs to disk. If None, the runs are not saved.
+            output_format (str): The format to save the runs. Defaults to "trec". If TREC, output_path should be a
+                directory, and each agent will have a separate file.
+        Returns:
+            dict[str, dict[str, dict[str, float]]]: A nested dictionary where the first level keys are agent names,
+            the second level keys are run identifiers, and the third level keys are metric names with their
+                corresponding float values.
+        """
+
+        runs_by_agent: dict[str, dict[str, dict[str, float]]] = {}
+        for query in self.queries.values():
+            runs = query.get_runs(agents)
+            for agent, run in runs.items():
+                runs_by_agent[agent] = runs_by_agent.get(agent, {}) or run
+        if output_path:
+            if output_format.lower() == "trec":
+                if not os.path.isdir(output_path):
+                    os.makedirs(output_path, exist_ok=True)
+                for agent, run in runs_by_agent.items():
+                    with open(f"{output_path}/{agent}.run", "w") as f:
+                        for qid, docs in run.items():
+                            sorted_scores = sorted(docs.items(), key=lambda x: x[1], reverse=True)
+                            for idx, (did, score) in enumerate(sorted_scores):
+                                f.write(f"{qid} Q0 {did} {idx+1} {score} {agent}\n")
+            elif output_format.lower() == "json":
+                with open(output_path, "w") as f:
+                    json.dump(runs_by_agent, f)
+
+        return runs_by_agent
+
+    def save(self, output_path: str | None = None):
+        """
+        Persist the current state of the model to disk if caching is enabled.
+        This method checks if caching is enabled by evaluating the `save_cache` attribute.
+        If caching is enabled, it attempts to write the model's current state to the file
+        specified by `cache_path`. If `cache_path` is not set, a ValueError is raised.
+        Raises:
+            ValueError: If `cache_path` is None and caching is enabled.
+        """
+        if not self.save_on_disk:
+            return
+        if output_path is None:
+            assert self.save_path is not None, "Cannot save experiment without a save path"
+            output_path = self.save_path
+        else:
+            assert self.save_path is not None, "Cannot save experiment without a save path"
+        output_dict: dict[str, Any] = {}
+
+        output_dict["queries"] = {qid: query.model_dump() for qid, query in self.queries.items()}
+        output_dict["experiment_name"] = self.experiment_name
+        output_dict["elo_tournaments"] = [tournament.model_dump() for tournament in self.elo_tournaments]
+
+        with open(output_path, "w") as f:
+            json.dump(output_dict, f, indent=4, ensure_ascii=False)
+
+    def save_results(
+        self,
+        result: EvaluatorResult | EloTournamentResult,
+    ):
+        """
+        Persist the evaluation result to a cache file.
+        This method writes the given evaluation result to a cache file specified by
+        `self.evaluations_cache_path`. The result is serialized to JSON format and appended
+        to the file. The type of the result (either "answer" or "retrieval") is included
+        in the serialized data.
+        Args:
+            result (EvaluatorResult | EloTournamentResult): The evaluation result
+                to be persisted to the cache.
+        Raises:
+            ValueError: If `self.evaluations_cache_path` is not set or if the type of `result`
+                is not recognized.
+        """
+
+        if not self.cache_evaluations:
+            return
+        if self.evaluations_cache_path is None:
+            raise ValueError("Results cache path not set. Cannot dump result")
+        with open(self.evaluations_cache_path, "a+") as f:
+            if isinstance(result, AnswerEvaluatorResult):
+                result_type = "answer"
+            elif isinstance(result, RetrievalEvaluatorResult):
+                result_type = "retrieval"
+            elif isinstance(result, EloTournamentResult):
+                result_type = "elo_tournament"
+            else:
+                raise ValueError(f"Cannot save evaluation of type {type(result)} to cache")
+            f.write(json.dumps({result_type: result.model_dump()}, ensure_ascii=False) + "\n")
+
+    def add_queries_from_csv(
+        self,
+        file_path: str,
+        query_id_column: str | None = None,
+        query_text_column: str = "query",
+    ):
+        if not os.path.isfile(file_path):
+            raise FileNotFoundError(f"CSV file with queries {file_path} not found")
+        read_queries = set()
+        try:
+            query_id_column = self.__infer_query_id_column(file_path)
+        except ValueError:
+            query_id_column = None
+
+        for idx, row in enumerate(csv.DictReader(open(file_path))):
+            if query_id_column is None:
+                query_id_column = f"query_{idx}"
+            else:
+                qid = row.get(query_id_column, f"query_{idx}")
+            if qid in read_queries or qid in self.queries:
+                logger.warning(f"Query with ID {qid} already read. Skipping")
+                continue
+            query_text = row[query_text_column].strip()
+            metadata = {k: v for k, v in row.items() if k not in [query_id_column, query_text_column]}
+            self.queries[qid] = Query(qid=qid, query=query_text, metadata=metadata)
+            read_queries.add(qid)
+        logger.info(f"Loaded {len(self.queries)} queries from {file_path}")
+
+    def add_documents_from_csv(
+        self,
+        file_path: str,
+        document_id_col: str | None = "did",
+        document_text_col: str = "document",
+        query_id_col: str | None = "qid",
+        agent_col: str | None = None,
+    ):
+        """
+        Loads a list of retrieved documents from a csv file. The csv file should have the following columns:
+        query_id, document_id, document_text, [agent], [metadata columns]
+        Args:
+            file_path (str): Path to the CSV file with the retrieved documents.
+            query_id_col (str): Name of the column with the query id to which the document was retrieved.
+                Defaults to 'qid'.
+            document_id_col (str): Name of the column with the document id. Defaults to 'did'.
+            document_text_col (str): Name of the column with the document text. Defaults to 'document_text'.
+            agent (str): Name of the column with the agent name. If None, will not add the agent name.
+                Defaults to None.
+        """
+        documents_read = 0
+        if not os.path.isfile(file_path):
+            raise FileNotFoundError(f"CSV file {file_path} not found")
+        query_id_col = self.__infer_query_id_column(file_path, query_id_col)
+        document_id_col = self.__infer_document_id_column(file_path, document_id_col)
+
+        for line in csv.DictReader(open(file_path)):
+            qid = line[query_id_col].strip()
+            did = line[document_id_col].strip()
+            text = line[document_text_col].strip()
+            agent = line.get(agent_col)
+            metadata = {k: v for k, v in line.items() if k not in [query_id_col, document_id_col, document_text_col]}
+            if qid not in self.queries:
+                logger.warning(f"Query {qid} found in {file_path} but not found in queries. Skipping")
+                continue
+            doc_obj = Document(qid=qid, did=did, text=text)
+            doc_obj.add_metadata(metadata)
+            if agent is not None:
+                doc_obj.add_retrieved_by(agent)
+            self.add_retrieved_doc(doc_obj)
+            documents_read += 1
+        logger.info(f"Loaded {documents_read} documents from {file_path}")
+
+    def add_retrieved_docs_from_runfile(
+        self,
+        run_file_path: str,
+        corpus: dict[str, Document] | dict[str, str],
+        top_k: int = 1000,
+    ):
+        """
+        Adds the retrieved documents from a run file to the queries.
+
+        A run file has, traditionally, the following format:
+            qid, Q0, docid, rank, score, run_name
+        The run name will be considered as the agent that retrieved the documents.
+            The second column is usually ignored.
+        Args:
+            run_file_path (str): Path to the run file.
+            corpus (dict[str, Document] | dict[str, str]): A dictionary with the documents.
+                If a dictionary with just docid and text is provided, we will use it as a corpus.
+                Otherwise, we will assume it's a dictionary with docid as key and Document object as value.
+            top_k (int): The maximum number of documents to add for each query.
+                Assumes that the run file is ordered by rank.
+        """
+        documents_read = set()
+        missing_docs = set()
+        docs_per_query = {}
+        warned_queries = set()
+
+        if isinstance(list(corpus.values())[0], str):
+            corpus = {k: str(v) for k, v in corpus.items()}
+            # corpus = {k: Document(did=k, text=str(v)) for k, v in corpus.items()}
+
+        if not os.path.isfile(run_file_path):
+            raise FileNotFoundError(f"Run file {run_file_path} not found")
+        for line in open(run_file_path):
+            qid, _, did, _, score, agent = line.strip().split()
+            if did not in corpus:
+                missing_docs.add(did)
+                continue
+            doc = corpus[did]
+            if isinstance(doc, str):
+                doc = Document(qid=qid, did=did, text=doc)
+            if qid not in self.queries:
+                if qid not in warned_queries:
+                    warned_queries.add(qid)
+                    logger.warning(f"Query {qid} not found in queries. Skipping")
+                continue
+            if qid not in docs_per_query:
+                docs_per_query[qid] = 0
+            if docs_per_query[qid] < top_k:
+                self.add_retrieved_doc(doc, qid, score=float(score), agent=agent)
+                docs_per_query[qid] += 1
+                documents_read.add(did)
+        if len(missing_docs) > 0:
+            logger.warning(f"Loaded {len(documents_read)} documents. {len(missing_docs)} missing docs")
+
+    def add_agent_answers_from_csv(
+        self,
+        file_path: str,
+        agent_col: str = "agent",
+        answer_col: str = "answer",
+        query_id_col: str | None = None,
+    ):
+        """
+        Adds agent answers from a CSV file to the queries.
+        The CSV file should have the following columns:
+        query_id, agent, answer, [metadata columns]
+        Args:
+            csv_path (str): Path to the CSV file with the agent answers.
+            agent_col (str): Name of the column with the agent name. Defaults to 'agent'.
+            answer_col (str): Name of the column with the answer text. Defaults to 'answer'.
+            query_id_col (str): Name of the column with the query id.
+                If None, will try to infer the query id column. Defaults to None.
+        """
+        if not os.path.isfile(file_path):
+            raise FileNotFoundError(f"Answers file {file_path} not found")
+        answers_read = 0
+        query_id_col = self.__infer_query_id_column(file_path, query_id_col)
+        for line in csv.DictReader(open(file_path)):
+            qid = line[query_id_col].strip()
+            agent = line[agent_col].strip()
+            answer = line[answer_col].strip()
+            metadata = {k: v for k, v in line.items() if k not in [query_id_col, agent_col, answer_col]}
+            if qid not in self.queries:
+                logger.warning(f"Query {qid} found in {file_path} but not found in queries. Skipping")
+                continue
+            answer_obj = AgentAnswer(qid=qid, agent=agent, text=answer)
+            answer_obj.add_metadata(metadata)
+            self.add_agent_answer(answer_obj)
+            answers_read += 1
+
+    def add_answers_from_multiple_csv(
+        self,
+        answers_files: list[str],
+        query_id_col: str | None = None,
+        query_text_col: str = "query",
+        answer_text_col: str = "answer",
+    ):
+        """Loads queries and answers from a list of CSVs with queries and answers generated by agents.
+        We assume that each csv was generated by a single agent, and name each agent according to the file name.
+        Args:
+            answers_files (list[str]): List of paths to the CSV files with the answers.
+            query_id_col (str): The column name with the query ID.
+                If None, will try to infer the query ID column. Defaults to None.
+            query_text_col (str): The column name with the query text. Defaults to 'query'.
+            answer_text_col (str): The column name with the answer text. Defaults to 'answer'.
+        """
+        for f in answers_files:
+            if query_id_col is None:
+                query_id_col = self.__infer_query_id_column(f)
+            if query_id_col is None:
+                query_id_col = query_text_col
+            p = Path(f)
+            agent_name = p.stem
+
+            for line in csv.DictReader(open(p)):
+                qid = line[query_id_col].strip()
+                query = line[query_text_col].strip()
+                answer = line[answer_text_col].strip()
+                if qid not in self.queries:
+                    self.add_query(query, qid)
+                self.add_agent_answer(answer, agent_name, qid)
+
+    def __clear_all_evaluations(self, should_save: bool = False):
+        logger.warning(f"Clearing all evaluations for {len(self)} queries")
+        doc_eval_count = 0
+        game_eval_count = 0
+        answer_eval_count = 0
+        for query in self.queries.values():
+            for doc in query.retrieved_docs.values():
+                if doc.evaluation is not None:
+                    doc_eval_count += 1
+                doc.evaluation = None
+            for game in query.pairwise_games:
+                if game.evaluation is not None:
+                    game_eval_count += 1
+                game.evaluation = None
+            for answer in query.answers.values():
+                if answer.evaluation is not None:
+                    answer_eval_count += 1
+                answer.evaluation = None
+        self.elo_tournaments = []
+        if doc_eval_count > 0 or game_eval_count > 0 or answer_eval_count > 0:
+            logger.info(
+                f"Cleared {doc_eval_count} document evaluations, {game_eval_count} game evaluations, "
+                f"and {answer_eval_count} answer evaluations"
+            )
+
+        if should_save:
+            if self.evaluations_cache_path and os.path.isfile(self.evaluations_cache_path):
+                with open(self.evaluations_cache_path, "w"):
+                    pass
+
+            self.save()
+
+    def _load_from_cache(self, cache_path: str) -> dict[str, Query]:
+        if not os.path.isfile(cache_path):
+            raise FileNotFoundError(f"Cache file {cache_path} not found")
+        with open(cache_path) as f:
+            data = json.load(f)
+        read_experiment_name = data.get("experiment_name")
+        if read_experiment_name and read_experiment_name != self.experiment_name:
+            logger.warning(f"Experiment name mismatch. Expected {self.experiment_name}. Found {read_experiment_name}")
+        queries = {k: Query(**v) for k, v in data.get("queries").items()}
+        if "elo_tournaments" in data:
+            self.elo_tournaments = [EloTournamentResult(**t) for t in data["elo_tournaments"]]
+        return queries
+
+    def _load_results_from_cache(self, cache_path: str):
+        assert os.path.isfile(cache_path)
+        for line in open(cache_path):
+            result = json.loads(line)
+            result_type = list(result.keys())[0]
+            if result_type == "answer":
+                result = AnswerEvaluatorResult(**result["answer"])
+                if result.qid not in self.queries:
+                    continue
+            elif result_type == "retrieval":
+                result = RetrievalEvaluatorResult(**result["retrieval"])
+                if result.qid not in self.queries:
+                    continue
+            elif result_type == "elo_tournament":
+                result = EloTournamentResult(**result["elo_tournament"])
+            self.add_evaluation(result, should_save=False, should_print=False)
+
+    def __len__(self):
+        return len(self.queries)
+
+    def __iter__(self):
+        return iter(self.queries.values())
+
+    def __getitem__(self, key: str) -> Query:
+        return self.queries[key]
+
+    def keys(self):
+        return self.queries.keys()
+
+    @staticmethod
+    def __infer_query_id_column(file_path: str, query_id_col: str | None = None) -> str:
+        """Infer the column name with the query id from a CSV file."""
+        if query_id_col is not None:
+            return query_id_col
+        possible_qid_columns = [
+            "query_id",
+            "qid",
+            "question_id",
+            "q_id",
+            "queryid",
+            "questionid",
+        ]
+        with open(file_path) as f:
+            reader = csv.DictReader(f)
+            columns = reader.fieldnames
+            if columns is not None:
+                for col in possible_qid_columns:
+                    if col in columns:
+                        return col
+        raise ValueError(f"Could not identify Query ID column for CSV file {file_path}")
+
+    @staticmethod
+    def __infer_document_id_column(file_path: str, doc_id_col: str | None = None) -> str:
+        """Infer the column name with the doc id from a CSV file."""
+        if doc_id_col is not None:
+            return doc_id_col
+        possible_did_columns = [
+            "document_id",
+            "did",
+            "d_id",
+            "documentid",
+            "passage_id",
+            "pid",
+            "p_id",
+            "passageid",
+        ]
+        with open(file_path) as f:
+            reader = csv.DictReader(f)
+            columns = reader.fieldnames
+            if columns is not None:
+                for col in possible_did_columns:
+                    if col in columns:
+                        return col
+        raise ValueError(f"Could not identify Document ID column for CSV file {file_path}")
+
+    @classmethod
+    def load(cls, experiment_name: str, path: str):
+        return cls(experiment_name=experiment_name, save_path=path)
diff --git a/ragelo/types/formats.py b/ragelo/types/formats.py
new file mode 100644
index 0000000..02b82b1
--- /dev/null
+++ b/ragelo/types/formats.py
@@ -0,0 +1,19 @@
+from __future__ import annotations
+
+from enum import Enum
+from typing import Any
+
+from ragelo.types.pydantic_models import BaseModel, PydanticBaseModel
+
+
+class AnswerFormat(str, Enum):
+    """Enum that contains the names of the available answer formats"""
+
+    TEXT = "text"
+    STRUCTURED = "structured"
+    JSON = "json"
+
+
+class LLMResponseType(BaseModel):
+    raw_answer: str
+    parsed_answer: float | str | dict[str, Any] | PydanticBaseModel | None
diff --git a/ragelo/types/pydantic_models.py b/ragelo/types/pydantic_models.py
new file mode 100644
index 0000000..bbff1b1
--- /dev/null
+++ b/ragelo/types/pydantic_models.py
@@ -0,0 +1,38 @@
+from __future__ import annotations
+
+from importlib import metadata
+
+from pydantic import BaseModel as PydanticBaseModel
+
+_PYDANTIC_MAJOR_VERSION: int = int(metadata.version("pydantic").split(".")[0])
+ValidationError: type[TypeError] | type[ValueError]
+if _PYDANTIC_MAJOR_VERSION == 1:
+    from pydantic import root_validator
+
+    validator = root_validator(pre=True)  # type: ignore
+    post_validator = root_validator(pre=False)  # type: ignore
+    ValidationError = TypeError
+else:
+    from pydantic import (
+        ValidationError,  # type: ignore
+        model_validator,  # type: ignore
+    )
+
+    validator = model_validator(mode="before")  # type: ignore
+    post_validator = model_validator(mode="after")  # type: ignore
+    ValidationError = ValidationError
+
+
+class BaseModel(PydanticBaseModel):
+    @classmethod
+    def get_model_fields(cls):
+        if _PYDANTIC_MAJOR_VERSION == 1:
+            return cls.__fields__  # type: ignore
+        else:
+            return cls.model_fields  # type: ignore
+
+    def model_dump(self, *args, **kwargs):
+        if _PYDANTIC_MAJOR_VERSION == 1:
+            return self.dict()  # type: ignore
+        else:
+            return super().model_dump(*args, **kwargs)  # type: ignore
diff --git a/ragelo/types/query.py b/ragelo/types/query.py
new file mode 100644
index 0000000..a519aeb
--- /dev/null
+++ b/ragelo/types/query.py
@@ -0,0 +1,305 @@
+from __future__ import annotations
+
+import warnings
+from collections.abc import Iterator
+from typing import Any
+
+from ragelo.logger import logger
+from ragelo.types.evaluables import AgentAnswer, Document, PairwiseGame
+from ragelo.types.pydantic_models import BaseModel
+from ragelo.types.results import (
+    AnswerEvaluatorResult,
+    EvaluatorResult,
+    RetrievalEvaluatorResult,
+)
+
+
+class Query(BaseModel):
+    """A user query that can have retrieved documents and agent answers.
+    Args:
+        qid str: The query ID.
+        query str: The query text.
+        metadata Optional[dict[str, Any]]: Metadata that can be templated in the prompt.
+        retrieved_docs dict[str, Document]: A dictionary of retrieved documents, where the key is the document ID.
+        answers list[AgentAnswer]: The list of agent answers.
+        pairwise_games list[PairwiseGame]: The list games to be played between agent's answers.
+            Generated by the evaluator.
+    """
+
+    qid: str
+    query: str
+    metadata: dict[str, Any] | None = None
+    retrieved_docs: dict[str, Document] = {}
+    answers: dict[str, AgentAnswer] = {}
+    pairwise_games: list[PairwiseGame] = []
+
+    def add_metadata(self, metadata: dict[str, Any] | None):
+        """Adds metadata to the query that may be templated in the prompt.
+        Args:
+            metadata dict[str, Any]: The metadata to add to the query.
+        """
+        if not metadata:
+            return
+        if self.metadata is None:
+            self.metadata = {}
+        for k in metadata:
+            if k in self.metadata:
+                logger.warning(
+                    f"Metadata {k} for query {self.qid} is being overwritten!\n"
+                    f"Old metadata: {self.metadata[k]}\n"
+                    f"New metadata: {metadata[k]}\n"
+                )
+            self.metadata[k] = metadata[k]
+
+    def add_retrieved_docs(
+        self,
+        docs: list,
+        agent: str | None = None,
+        force: bool = False,
+    ):
+        """Add a list of retrieved documents to the query.
+        Args:
+            docs (list[Document | str] | list[Tuple[Document | str], float]): The list of documents
+                (either the objects or the texts) to add to the query.
+                Optionally, a score can be provided for each document.
+            agent (str): The agent that retrieved the documents.
+        """
+        # First, transform all of the strings into Document objects.
+        # If an agent name was provided, but no score information exists,
+        # assume the list is ranked and assign a score of 1/(n+1) to each document.
+        for idx, doc in enumerate(docs):
+            if isinstance(doc, tuple):
+                doc, score = doc
+                if agent is None:
+                    raise ValueError(
+                        "If you provide a score for each retrieved document, you should also provide an agent name."
+                    )
+            elif agent is not None:
+                score = 1 / (idx + 1)
+            else:
+                score = None
+            self.add_retrieved_doc(doc, score=score, agent=agent, force=force)
+
+    def add_retrieved_doc(
+        self,
+        doc: Document,
+        score: float | None = None,
+        agent: str | None = None,
+        force: bool = False,
+    ):
+        """Add a retrieved document to the query.
+        Args:
+            doc Document: The document object retrieved for the query.
+            score float: The score of the document when retrieved.
+            agent str: The agent that retrieved the document.
+        """
+        doc = self.retrieved_docs.get(doc.did, doc.model_copy(deep=True))
+        if agent is not None:
+            doc.add_retrieved_by(agent, score, force)
+        if doc.did in self.retrieved_docs and not force:
+            logger.info(f"Query {self.qid} already have a document {doc.did} retrieved.")
+            return
+
+        self.retrieved_docs[doc.did] = doc
+
+    def add_agent_answer(self, answer: AgentAnswer, force: bool = False):
+        """Add an answer generated by an agent to the query.
+        Args:
+            answer AgentAnswer | str: The answer (either the object or the text) to add to the query.
+            agent str: The id or name of the agent that generated the answer.
+                If not provided, it will be autogenerated.
+        """
+        agent = answer.agent
+        answer = self.answers.get(agent, answer)
+        if answer.agent in self.answers and not force:
+            warnings.warn(f"Answer from agent {answer.agent} already exists in query {self.qid}")
+            return
+        if answer.agent in self.answers:
+            logger.info(f"Answer from agent {answer.agent} already exists in query {self.qid}. Overwriting.")
+        self.answers[agent] = answer
+
+    def add_evaluation(
+        self,
+        evaluation: EvaluatorResult,
+        force: bool = False,
+        exist_ok: bool = False,
+    ) -> bool:
+        if not isinstance(evaluation, RetrievalEvaluatorResult) and not isinstance(evaluation, AnswerEvaluatorResult):
+            raise ValueError("Evaluation must be a RetrievalEvaluatorResult or an AnswerEvaluatorResult")
+        if isinstance(evaluation, RetrievalEvaluatorResult):
+            did = evaluation.did
+            if did not in self.retrieved_docs:
+                logger.warning(f"Trying to add evaluation for non-retrieved document {did} in query {self.qid}")
+            if self.retrieved_docs[did].evaluation is not None and exist_ok:
+                return False
+            if self.retrieved_docs[did].evaluation is not None and not force:
+                logger.warning(f"Document {did} in query {self.qid} already has an evaluation.")
+                return False
+            if self.retrieved_docs[did].evaluation is not None:
+                logger.info(f"Document {did} in query {self.qid} already has an evaluation. Overwriting.")
+            self.retrieved_docs[did].evaluation = evaluation
+            return True
+        if evaluation.pairwise:
+            agent_a = evaluation.agent_a
+            agent_b = evaluation.agent_b
+            if agent_a not in self.answers:
+                logger.warning(
+                    "Trying to add a pairwise evaluation for a comparison between agents "
+                    f"{agent_a} and {agent_b}, but {agent_a} does not have an answer for query {self.qid}"
+                )
+            if agent_b not in self.answers:
+                logger.warning(
+                    "Trying to add a pairwise evaluation for a comparison between agents "
+                    f"{agent_a} and {agent_b}, but {agent_b} does not have an answer for query {self.qid}"
+                )
+            for game in self.pairwise_games:
+                if game.agent_a_answer.agent == agent_a and game.agent_b_answer.agent == agent_b:
+                    if game.evaluation is not None and exist_ok:
+                        return False
+                    if game.evaluation is not None and not force:
+                        logger.warning(
+                            f"Query {self.qid} already has an evaluation for agents {agent_a} and {agent_b}."
+                        )
+                        return False
+                    if game.evaluation is not None:
+                        logger.info(
+                            f"Query {self.qid} already has an evaluation for agents {agent_a} and {agent_b}. "
+                            "Overwriting."
+                        )
+                    game.evaluation = evaluation
+                    return True
+            logger.info(
+                f"Trying to add a pairwise evaluation for a comparison between agents {agent_a} and {agent_b},"
+                f" but no game was found for query {self.qid}. Creating a new game."
+            )
+            agent_a = evaluation.agent_a
+            agent_b = evaluation.agent_b
+            if agent_a not in self.answers:
+                logger.warning(
+                    f"Trying to add a pairwise evaluation for a comparison between agents {agent_a} and {agent_b},"
+                    f" but {agent_a} does not have an answer for query {self.qid}"
+                )
+                return False
+            if agent_b not in self.answers:
+                logger.warning(
+                    f"Trying to add a pairwise evaluation for a comparison between agents {agent_a} and {agent_b},"
+                    f" but {agent_b} does not have an answer for query {self.qid}"
+                )
+                return False
+            self.pairwise_games.append(
+                PairwiseGame(
+                    qid=self.qid,
+                    agent_a_answer=self.answers[agent_a],
+                    agent_b_answer=self.answers[agent_b],
+                    evaluation=evaluation,
+                )
+            )
+            return True
+        if evaluation.agent is None:
+            raise ValueError("A pointwise AnswerEvaluatorResult must have an agent assigned to it.")
+        agent = evaluation.agent
+        if agent not in self.answers:
+            logger.warning(
+                f"Trying to add evaluation for agent {agent} in query {self.qid}, but {agent} does not have an answer."
+            )
+        if self.answers[agent].evaluation is not None and exist_ok:
+            return False
+        if self.answers[agent].evaluation is not None and not force:
+            logger.warning(f"Agent {agent} in query {self.qid} already has an evaluation.")
+            return False
+        if self.answers[agent].evaluation is not None:
+            logger.info(f"Agent {agent} in query {self.qid} already has an evaluation. Overwriting.")
+        self.answers[agent].evaluation = evaluation
+        return True
+
+    def get_qrels(
+        self,
+        relevance_key: str | None = "relevance",
+        relevance_threshold: int = 0,
+    ) -> dict[str, int]:
+        """Get a qrels-formatted dictionary with the relevance of the retrieved documents.
+        Args:
+            relevance_key str: The key in the answer object that contains an integer with the relevance of the
+                document.
+            relevance_threshold int: The minimum relevance value to consider a document as relevant.
+                Documents with a relevance lower than this value will be considered as 0.
+        """
+        qrels = {}
+        if len(self.retrieved_docs) == 0:
+            logger.warning(f"Query {self.qid} does not have any retrieved documents. Returning empty qrels.")
+        docs_without_relevance = 0
+        for did, document in self.retrieved_docs.items():
+            if document.evaluation is None:
+                docs_without_relevance += 1
+                continue
+            if isinstance(document.evaluation.answer, int):
+                relevance = document.evaluation.answer
+            elif isinstance(document.evaluation.answer, str):
+                try:
+                    relevance = int(document.evaluation.answer)
+                except ValueError:
+                    logger.warning(
+                        f"Document {did} has a relevance key ({relevance_key})"
+                        f" that cannot be converted to an int ({document.evaluation.answer})."
+                        " Skipping."
+                    )
+                    continue
+            elif isinstance(document.evaluation.answer, dict):
+                if relevance_key is None or relevance_key not in document.evaluation.answer:
+                    logger.warning(
+                        f"Document {did} does not have a relevance key ({relevance_key})"
+                        " in the evaluation. Skipping."
+                    )
+                    continue
+                # check if the relevance is a number or a str that can be converted to an int
+                if not isinstance(document.evaluation.answer[relevance_key], int):
+                    try:
+                        relevance = int(document.evaluation.answer[relevance_key])
+                    except ValueError:
+                        logger.warning(
+                            f"Document {did} has a relevance key ({relevance_key})"
+                            f" that cannot be converted to an int ({document.evaluation.answer[relevance_key]})."
+                            " Skipping."
+                        )
+                        continue
+                relevance = int(document.evaluation.answer[relevance_key])
+            qrels[did] = 0 if relevance < relevance_threshold else relevance
+        if docs_without_relevance > 0:
+            logger.warning(f"Query {self.qid} has {docs_without_relevance} documents without relevance.")
+        if docs_without_relevance == len(self.retrieved_docs):
+            logger.error(f"Query {self.qid} has no documents without relevance.")
+        return qrels
+
+    def get_runs(self, agents: list[str] | None = None) -> dict[str, dict[str, dict[str, float]]]:
+        """Get a trec-style run dictionary with the documents retrieved by the agents provided.
+        If no agents are provided, all agents will be included.
+        Args:
+            agents list[str]: The list of agent IDs to include in the run.
+        """
+        runs: dict[str, dict[str, dict[str, float]]] = {}
+        for did, document in self.retrieved_docs.items():
+            for agent, score in document.retrieved_by.items():
+                if agents is not None and agent not in agents:
+                    continue
+                if agent not in runs:
+                    runs[agent] = {self.qid: {}}
+                runs[agent][self.qid][did] = float(score)
+
+        if len(runs) == 0:
+            logger.warning(f"Query {self.qid} does not have any retrieved documents with an agent assigned.")
+        return runs
+
+    def retrieved_docs_iter(self) -> Iterator[Document]:
+        return iter(self.retrieved_docs.values())
+
+    @classmethod
+    def assemble_query(cls, query: Query | str, metadata: dict[str, Any] | None = None) -> Query:
+        """Assembles a Query object from a Query object or a query text.
+        Args:
+            query Query | str: The query object or the query text.
+            metadata dict[str, Any]: Metadata to add to the query.
+        """
+        if isinstance(query, Query):
+            query.add_metadata(metadata)
+            return query
+        return cls(qid="<no_qid>", query=query, metadata=metadata)
diff --git a/ragelo/types/results.py b/ragelo/types/results.py
new file mode 100644
index 0000000..c3b15f4
--- /dev/null
+++ b/ragelo/types/results.py
@@ -0,0 +1,86 @@
+from __future__ import annotations
+
+from typing import Any
+
+from pydantic import BaseModel as PydanticBaseModel
+
+from ragelo.types.pydantic_models import BaseModel, ValidationError, validator
+
+
+class EvaluatorResult(BaseModel):
+    """Generic class with the results of an evaluator.
+    Args:
+        qid str: The query ID to which the result corresponds.
+        agent str | None: The agent that provided the answer or retrieved the document.
+        raw_answer str | None: The raw answer provided by the LLMProvider used in the evaluator.
+        answer Optional[Union[int, str, dict[str, Any]]]: The processed answer provided by the evaluator.
+            If the evaluator uses "multi_field_json" as answer_format, this will be a dictionary with evaluation keys.
+    """
+
+    qid: str
+    agent: str | None = None
+    raw_answer: str | None = None
+    answer: float | str | dict[str, Any] | PydanticBaseModel | None = None
+    exception: str | None = None
+
+    @validator
+    @classmethod
+    def check_agents(cls, v):
+        exception = v.get("exception")
+        raw_answer = v.get("raw_answer")
+        answer = v.get("answer")
+        if (raw_answer is None or answer is None) and exception is None:
+            raise ValidationError(
+                "Either answer or raw_answer must be provided. Otherwise, an exception must be provided."
+            )  # type: ignore
+        return v
+
+
+class AnswerEvaluatorResult(EvaluatorResult):
+    """The results of an answer evaluator.
+    Args:
+        agent str | None: The agent that provided the answer. Only used if pairwise=False.
+        agent_a str | None: The first agent that provided the answer. Only used if the evaluator is pairwise.
+        agent_b str | None: The second agent that provided the answer. Only used if the evaluator is pairwise.
+        pairwise bool: Whether the evaluation is pairwise or not.
+    """
+
+    agent: str | None = None
+    agent_a: str | None = None
+    agent_b: str | None = None
+    pairwise: bool = False
+
+    @validator
+    @classmethod
+    def check_agents(cls, v):
+        agent = v.get("agent")
+        agent_a = v.get("agent_a")
+        agent_b = v.get("agent_b")
+        if agent is None and agent_a is None and agent_b is None:
+            raise ValidationError("Either agent or agent_a and agent_b must be provided")  # type: ignore
+        if agent_a is not None and agent_b is not None:
+            v["pairwise"] = True
+        return v
+
+
+class RetrievalEvaluatorResult(EvaluatorResult):
+    """The results of a retrieval evaluator.
+    Args:
+        did str: The document ID to which the result corresponds.
+    """
+
+    did: str
+
+
+class EloTournamentResult(BaseModel):
+    """A class to store the results of an Elo tournament between multiple agents."""
+
+    agents: list[str]
+    scores: dict[str, float]
+    games_played: dict[str, int]
+    wins: dict[str, int]
+    loses: dict[str, int]
+    ties: dict[str, int]
+    std_dev: dict[str, float]
+    total_games: int
+    total_tournaments: int
diff --git a/ragelo/types/types.py b/ragelo/types/types.py
index 759cd6e..f0a1960 100644
--- a/ragelo/types/types.py
+++ b/ragelo/types/types.py
@@ -1,45 +1,6 @@
-from enum import Enum
-from importlib import metadata
-from typing import Any, Dict, List, Optional, Union
-
-from pydantic import BaseModel as PydanticBaseModel
-
-from ragelo.logger import logger
-
-_PYDANTIC_MAJOR_VERSION: int = int(metadata.version("pydantic").split(".")[0])
-if _PYDANTIC_MAJOR_VERSION == 1:
-    from pydantic import root_validator
-
-    validator = root_validator(pre=True)  # type: ignore
-    ValidationError = TypeError
-else:
-    from pydantic import ValidationError  # type: ignore
-    from pydantic import model_validator  # type: ignore
-
-    validator = model_validator(mode="before")  # type: ignore
-
+from __future__ import annotations
 
-class BaseModel(PydanticBaseModel):
-    @classmethod
-    def get_model_fields(cls):
-        if _PYDANTIC_MAJOR_VERSION == 1:
-            return cls.__fields__  # type: ignore
-        else:
-            return cls.model_fields  # type: ignore
-
-    def model_dump(self):
-        if _PYDANTIC_MAJOR_VERSION == 1:
-            return self.dict()  # type: ignore
-        else:
-            return super().model_dump()  # type: ignore
-
-
-class AnswerFormat(str, Enum):
-    """Enum that contains the names of the available answer formats"""
-
-    JSON = "json"
-    TEXT = "text"
-    MULTI_FIELD_JSON = "multi_field_json"
+from enum import Enum
 
 
 class RetrievalEvaluatorTypes(str, Enum):
@@ -63,6 +24,7 @@ class AnswerEvaluatorTypes(str, Enum):
     """Enum that contains the names of the available answer evaluators"""
 
     PAIRWISE = "pairwise"
+    CUSTOM_PAIRWISE = "custom_pairwise"
     CUSTOM_PROMPT = "custom_prompt"
     DOMAIN_EXPERT = "domain_expert"
     CHAT_PAIRWISE = "chat_pairwise"
@@ -70,376 +32,3 @@ class AnswerEvaluatorTypes(str, Enum):
 
 class AgentRankerTypes(str, Enum):
     ELO = "elo"
-
-
-class EvaluatorResult(BaseModel):
-    """Generic class with the results of an evaluator.
-    Args:
-        qid str: The query ID to which the result corresponds.
-        agent Optional[str]: The agent that provided the answer or retrieved the document.
-        raw_answer Optional[str]: The raw answer provided by the LLMProvider used in the evaluator.
-        answer Optional[Union[int, str, Dict[str, Any]]]: The processed answer provided by the evaluator.
-            If the evaluator uses "multi_field_json" as answer_format, this will be a dictionary with evaluation keys.
-    """
-
-    qid: str
-    agent: Optional[str] = None
-    raw_answer: Optional[str]
-    answer: Optional[Union[int, str, Dict[str, Any]]]
-    exception: Optional[str] = None
-
-    @validator
-    @classmethod
-    def check_agents(cls, v):
-        exception = v.get("exception")
-        raw_answer = v.get("raw_answer")
-        answer = v.get("answer")
-        if (raw_answer is None or answer is None) and exception is None:
-            raise ValidationError(
-                "Either answer or raw_answer must be provided. Otherwise, an exception must be provided."
-            )  # type: ignore
-        return v
-
-
-class AnswerEvaluatorResult(EvaluatorResult):
-    """The results of an answer evaluator.
-    Args:
-        did str: The document ID to which the result corresponds.
-        agent Optional[str]: The agent that provided the answer. Only used if pairwise=False.
-        agent_a Optional[str]: The first agent that provided the answer. Only used if the evaluator is pairwise.
-        agent_b Optional[str]: The second agent that provided the answer. Only used if the evaluator is pairwise.
-        pairwise bool: Whether the evaluation is pairwise or not.
-    """
-
-    agent: Optional[str] = None
-    agent_a: Optional[str] = None
-    agent_b: Optional[str] = None
-    pairwise: bool = False
-
-    @validator
-    @classmethod
-    def check_agents(cls, v):
-        agent = v.get("agent")
-        agent_a = v.get("agent_a")
-        agent_b = v.get("agent_b")
-        if agent is None and agent_a is None and agent_b is None:
-            raise ValidationError(
-                "Either agent or agent_a and agent_b must be provided"
-            )  # type: ignore
-        if agent_a is not None and agent_b is not None:
-            v["pairwise"] = True
-        return v
-
-
-class RetrievalEvaluatorResult(EvaluatorResult):
-    """The results of a retrieval evaluator.
-    Args:
-        did str: The document ID to which the result corresponds.
-    """
-
-    did: str
-
-
-class FewShotExample(BaseModel):
-    """A few-shot example used in the few-shot retrieval evaluator.
-    Args:
-        passage str: The passage of the example.
-        query str: The query of the example.
-        relevance int: The relevance of the example.
-        reasoning str: The reasoning behind the relevance
-    """
-
-    passage: str
-    query: str
-    relevance: int
-    reasoning: str
-
-
-class Evaluable(BaseModel):
-    """A base class for objects that can be evaluated. Either a document or an agent answer.
-    Args:
-        evaluation Optional[EvaluatorResult]: The result of the evaluation.
-        metadata Optional[Dict[str, Any]]: Metadata that can be templated in the prompt.
-    """
-
-    evaluation: Optional[EvaluatorResult] = None
-    metadata: Optional[Dict[str, Any]] = None
-
-    def add_metadata(self, metadata: Optional[Dict[str, Any]]):
-        if not metadata:
-            return
-        if self.metadata is None:
-            self.metadata = {}
-        for k in metadata:
-            if k in self.metadata:
-                logger.warning(
-                    f"Metadata {k} for {self.__class__.__name__}"
-                    " is being overwritten!\n"
-                    f"Old metadata: {self.metadata[k]}\n"
-                    f"New metadata: {metadata[k]}\n"
-                )
-            self.metadata[k] = metadata[k]
-
-
-class Document(Evaluable):
-    """A document retrieved by an agent in response to a query.
-    Args:
-        did str: The document ID.
-        text str: The text of the document.
-        retrieved_by Dict[str, float]: If a document was retrieved by multiple agents, the score attributed by each agent for this document.
-    """
-
-    did: str
-    text: str
-    retrieved_by: Dict[str, float] = {}
-
-    def add_retrieved_by(
-        self, agent: str, score: Optional[float] = None, overwrite: bool = False
-    ):
-        """Adds the score of an agent that retrieved the document."""
-        if agent in self.retrieved_by and not overwrite:
-            logger.info(
-                f"Document with did {self.did} already retrieved by agent {agent}"
-            )
-            return
-        if score is None:
-            score = 1.0
-        self.retrieved_by[agent] = score
-
-
-class ChatMessage(BaseModel):
-    sender: str
-    content: str
-
-
-class AgentAnswer(Evaluable):
-    """An answer generated by an agent in response to a query.
-    Args:
-        agent str: The agent that provided the answer.
-        text str: The text of the answer.
-        conversation Optional[List[ChatMessage]]: The conversation between the user and the agent.
-    """
-
-    agent: str
-    text: Optional[str] = None
-    conversation: Optional[List[ChatMessage]] = None
-
-    @validator
-    @classmethod
-    def one_of_text_or_conversation(cls, values):
-        text = values.get("text")
-        conversation = values.get("conversation")
-        if text is None and conversation is None:
-            raise ValueError("Either text or conversation must be provided")
-
-        if text is not None and conversation is not None:
-            raise ValueError("Only one of text or conversation must be provided")
-        return values
-
-
-class PairwiseGame(Evaluable):
-    """A game to be played between two agent answers.
-    Args:
-        agent_a_answer AgentAnswer: The answer provided by the first agent.
-        agent_b_answer AgentAnswer: The answer provided by the second agent.
-    """
-
-    agent_a_answer: AgentAnswer
-    agent_b_answer: AgentAnswer
-
-
-class Query(BaseModel):
-    """A user query that can have retrieved documents and agent answers.
-    Args:
-        qid str: The query ID.
-        query str: The query text.
-        metadata Optional[Dict[str, Any]]: Metadata that can be templated in the prompt.
-        retrieved_docs Dict[str, Document]: A dictionary of retrieved documents, where the key is the document ID.
-        answers List[AgentAnswer]: The list of agent answers.
-        pairwise_games List[PairwiseGame]: The list games to be played between agent's answers. Generated by the evaluator.
-    """
-
-    qid: str
-    query: str
-    metadata: Optional[Dict[str, Any]] = None
-    retrieved_docs: Dict[str, Document] = {}
-    answers: Dict[str, AgentAnswer] = {}
-    pairwise_games: List[PairwiseGame] = []
-
-    def add_metadata(self, metadata: Optional[Dict[str, Any]]):
-        """Adds metadata to the query that may be templated in the prompt.
-        Args:
-            metadata Dict[str, Any]: The metadata to add to the query.
-        """
-        if not metadata:
-            return
-        if self.metadata is None:
-            self.metadata = {}
-        for k in metadata:
-            if k in self.metadata:
-                logger.warning(
-                    f"Metadata {k} for query {self.qid} is being overwritten!\n"
-                    f"Old metadata: {self.metadata[k]}\n"
-                    f"New metadata: {metadata[k]}\n"
-                )
-            self.metadata[k] = metadata[k]
-
-    def add_retrieved_docs(
-        self,
-        docs: List,
-        agent: Optional[str] = None,
-        overwrite: bool = False,
-    ):
-        """Add a list of retrieved documents to the query.
-        Args:
-            docs List[Document | str] | List[Tuple[Document | str], float]: The list of documents (either the objects or the texts) to add to the query. Optionally, a score can be provided for each document.
-            agent str: The agent that retrieved the documents.
-        """
-        # First, transform all of the strings into Document objects.
-        # If an agent name was provided, but no score information exists,
-        # assume the list is ranked and assign a score of 1/(n+1) to each document.
-        for idx, doc in enumerate(docs):
-            if isinstance(doc, tuple):
-                doc, score = doc
-                if agent is None:
-                    raise ValueError(
-                        "If you provide a score for each retrieved document, you should also provide an agent name."
-                    )
-            elif agent is not None:
-                score = 1 / (idx + 1)
-            else:
-                score = None
-            self.add_retrieved_doc(doc, score=score, agent=agent, overwrite=overwrite)
-
-    def add_retrieved_doc(
-        self,
-        doc: Union[Document, str],
-        doc_id: Optional[str] = None,
-        score: Optional[float] = None,
-        agent: Optional[str] = None,
-        overwrite: bool = False,
-    ):
-        """Add a retrieved document to the query.
-        Args:
-            doc Document | str: The document (either the object or the text) to add to the query.
-            doc_id str: The document ID. If not provided, it will be autogenerated.
-            score float: The score of the document when retrieved.
-            agent str: The agent that retrieved the document.
-        """
-        if doc_id is None:
-            if isinstance(doc, Document):
-                doc_id = doc.did
-            else:
-                logger.info(f"No doc_id provided. Using '{doc_id}' as doc_id")
-                doc_id = f"doc_{len(self.retrieved_docs)+1}"
-        if isinstance(doc, str):
-            doc = Document(did=doc_id, text=doc)
-        doc = self.retrieved_docs.get(doc_id, doc)
-        if agent is not None:
-            doc.add_retrieved_by(agent, score, overwrite)
-        if doc_id in self.retrieved_docs and not overwrite:
-            logger.info(f"Query {self.qid} already have a document {doc_id} retrieved.")
-            return
-
-        self.retrieved_docs[doc_id] = doc
-
-    def add_agent_answer(
-        self, answer: Union[AgentAnswer, str], agent: Optional[str] = None
-    ):
-        """Add an answer generated by an agent to the query.
-        Args:
-            answer AgentAnswer | str: The answer (either the object or the text) to add to the query.
-            agent str: The id or name of the agent that generated the answer. If not provided, it will be autogenerated.
-        """
-        if agent is None:
-            if isinstance(answer, AgentAnswer):
-                agent = answer.agent
-            else:
-                agent = f"agent_{len(self.answers) + 1}"
-                logger.info(f"No agent provided. Using '{agent}' as agent")
-        if isinstance(answer, str):
-            answer = AgentAnswer(agent=agent, text=answer)
-        answer = self.answers.get(agent, answer)
-        if answer.agent in self.answers:
-            logger.warning(
-                f"Answer from agent {answer.agent} already exists in query {self.qid}"
-            )
-            return
-        self.answers[agent] = answer
-
-    def get_qrels(
-        self,
-        relevance_key: Optional[str] = "relevance",
-        relevance_threshold: int = 0,
-    ) -> Dict[str, Dict[str, int]]:
-        """Get a qrels-formatted dictionary with the relevance of the retrieved documents.
-        Args:
-            relevance_key str: The key in the answer object that contains an integer with the relevance of the document.
-            relevance_threshold int: The minimum relevance value to consider a document as relevant.
-                Documents with a relevance lower than this value will be considered as 0.
-        """
-        qrels = {}
-        if len(self.retrieved_docs) == 0:
-            logger.warning(
-                f"Query {self.qid} does not have any retrieved documents. Returning empty qrels."
-            )
-        docs_without_relevance = 0
-        for did, document in self.retrieved_docs.items():
-            if document.evaluation is None:
-                docs_without_relevance += 1
-                continue
-            if isinstance(document.evaluation.answer, int):
-                qrels[did] = document.evaluation.answer
-            elif isinstance(document.evaluation.answer, dict):
-                if (
-                    relevance_key is None
-                    or relevance_key not in document.evaluation.answer
-                ):
-                    logger.warning(
-                        f"Document {did} does not have a relevance key ({relevance_key})"
-                        " in the evaluation. Skipping."
-                    )
-                    continue
-                # check if the relevance is a number or a str that can be converted to an int
-                if not isinstance(document.evaluation.answer[relevance_key], int):
-                    try:
-                        relevance = int(document.evaluation.answer[relevance_key])
-                    except ValueError:
-                        logger.warning(
-                            f"Document {did} has a relevance key ({relevance_key})"
-                            f" that cannot be converted to an int ({document.evaluation.answer[relevance_key]})."
-                            " Skipping."
-                        )
-                        continue
-                relevance = int(document.evaluation.answer[relevance_key])
-                qrels[did] = 0 if relevance < relevance_threshold else relevance
-        if docs_without_relevance > 0:
-            logger.warning(
-                f"Query {self.qid} has {docs_without_relevance} documents without relevance."
-            )
-        if docs_without_relevance == len(self.retrieved_docs):
-            logger.error(f"Query {self.qid} has no documents without relevance.")
-        return {self.qid: qrels}
-
-    def get_runs(
-        self, agents: Optional[List[str]] = None
-    ) -> Dict[str, Dict[str, Dict[str, float]]]:
-        """Get a trec-style run dictionary with the documents retrieved by the agents provided.
-        If no agents are provided, all agents will be included.
-        Args:
-            agents List[str]: The list of agent IDs to include in the run.
-        """
-        runs: Dict[str, Dict[str, Dict[str, float]]] = {}
-        for did, document in self.retrieved_docs.items():
-            for agent, score in document.retrieved_by.items():
-                if agents is not None and agent not in agents:
-                    continue
-                if agent not in runs:
-                    runs[agent] = {self.qid: {}}
-                runs[agent][self.qid][did] = score
-
-        if len(runs) == 0:
-            logger.warning(
-                f"Query {self.qid} does not have any retrieved documents with an agent assigned."
-            )
-        return runs
diff --git a/ragelo/utils.py b/ragelo/utils.py
index e31f4c2..abf88c9 100644
--- a/ragelo/utils.py
+++ b/ragelo/utils.py
@@ -1,342 +1,44 @@
-"""Utils functions for RAGElo"""
+from __future__ import annotations
 
-import csv
-import logging
-import os
-from collections import defaultdict
-from pathlib import Path
-from typing import List, Optional, Union
+import asyncio
+import warnings
+from collections.abc import Callable
+from concurrent.futures import ThreadPoolExecutor
+from typing import Any, Coroutine
 
-from ragelo.types import AgentAnswer, Document, EvaluatorResult, PairwiseGame, Query
+from tqdm import TqdmExperimentalWarning
+from tqdm.auto import tqdm
+from tqdm.rich import tqdm_rich
 
 
-def infer_query_id_column(file_path: str) -> Union[str, None]:
-    """Infer the column name with the query id from a CSV file."""
-    possible_qid_columns = [
-        "query_id",
-        "qid",
-        "question_id",
-        "q_id",
-        "queryid",
-        "questionid",
-    ]
-    with open(file_path) as f:
-        reader = csv.DictReader(f)
-        columns = reader.fieldnames
-        if columns is not None:
-            for col in possible_qid_columns:
-                if col in columns:
-                    return col
-    return None
-
-
-def infer_doc_id_column(file_path: str) -> Union[str, None]:
-    """Infer the column name with the document id from a CSV file."""
-    possible_did_columns = [
-        "document_id",
-        "doc_id",
-        "did",
-        "docid",
-        "d_id",
-        "documentid",
-        "passageid",
-        "pid",
-        "p_id",
-    ]
-    with open(file_path) as f:
-        reader = csv.DictReader(f)
-        columns = reader.fieldnames
-        if columns is not None:
-            for col in possible_did_columns:
-                if col in columns:
-                    return col
-    return None
-
-
-def load_queries_from_csv(
-    queries_path: str,
-    query_text_col: str = "query",
-    query_id_col: Optional[str] = None,
-    infer_metadata_fields: bool = True,
-    metadata_fields: Optional[List[str]] = None,
-) -> List[Query]:
-    """Loads the queries from a CSV file and returns a dictionary with the queries.
-        The CSV file should have a header with the columns 'query_id' and 'query'.
-        The key is the query id and the value is the query object.
-
-    Args:
-        queries_path (str): Path to the CSV file with the queries.
-        query_text_col (str): Name of the column with the query text. Defaults to 'query'.
-        query_id_col (str): Name of the column with the query id. If not passed, try to infer from the csv.
-            Uses the query_text_col as last resource.
-        infer_metadata_fields (bool): Wether to infer the metadata fields in the csv.
-        metadata_fields (Optional[List[str]]): The list of columns to be used as metadata.
-            If infer_metadata_fields is True, will use all the fields if not explicitly set.
-    Returns:
-        List[Query]: List of queries.
-
+def run(coroutine: Coroutine[Any, Any, Any]) -> Any:
     """
-    queries = []
-    read_queries = set()
-    if not os.path.isfile(queries_path):
-        raise FileNotFoundError(f"Queries file {queries_path} not found")
-    if query_id_col is None:
-        query_id_col = infer_query_id_column(queries_path)
-    if query_id_col is None:
-        query_id_col = query_text_col
-
-    with open(queries_path) as f:
-        reader = csv.DictReader(f)
-        headers = reader.fieldnames
-        if headers and infer_metadata_fields:
-            if metadata_fields is None:
-                metadata_fields = [
-                    x for x in headers if x not in [query_id_col, query_text_col]
-                ]
-            else:
-                # Use only the metadata fields that actually exist
-                metadata_fields = [x for x in metadata_fields if x in headers]
-                if not metadata_fields:
-                    logging.warning(
-                        "No metadata fields found in the csv. Ignoring metadata fields"
-                    )
-                    metadata_fields = None
-        for line in reader:
-            qid = line[query_id_col].strip()
-            if qid in read_queries:
-                logging.warning(f"Query {qid} already read. Skipping")
-                continue
-            query_text = line[query_text_col].strip()
-            if metadata_fields is not None:
-                extra_metadata = {k: v for k, v in line.items() if k in metadata_fields}
-                queries.append(
-                    Query(qid=qid, query=query_text, metadata=extra_metadata)
-                )
-            else:
-                queries.append(Query(qid=qid, query=query_text))
-
-            read_queries.add(qid)
-    logging.info(f"Loaded {len(queries)} queries")
-
-    return queries
-
-
-def add_documents_from_csv(
-    documents_file: str,
-    queries_file: Optional[str] = None,
-    queries: Optional[List[Query]] = None,
-    query_id_col: str = "qid",
-    document_id_col: str = "did",
-    document_text_col: str = "document_text",
-) -> List[Query]:
-    """Loads a list of retrieved documents for each query. If queries is a string, will load the queries from the csv file.
-
-    Args:
-        documents_path (str): Path to the CSV file with the documents.
-        queries (Optional[Dict[str, Query]]): Dictionary with the queries.
-            If provided, only documents for the queries in the dictionary will be
-            loaded. Defaults to None.
-        query_id_col (str): Name of the column with the query id. Defaults to 'query_id'.
-        document_id_col (str): Name of the column with the document id.
-            Defaults to 'did'.
-        document_text_col (str): Name of the column with the document text.
-            Defaults to 'document_text'.
-
-    Returns:
-        List[Query]: A list of queries with the retrieved documents.
-
+    Runs the given coroutine and returns its result.
     """
-    documents_read = 0
-    if not os.path.isfile(documents_file):
-        raise FileNotFoundError(f"Documents file {documents_file} not found")
-
-    if queries is None and queries_file is None:
-        raise ValueError("Either queries or queries_file must be provided")
-    if queries is None:
-        assert queries_file is not None  # Should not happen. but mypy complains
-        queries = load_queries_from_csv(queries_file)
-    queries_dict = {q.qid: q for q in queries}
-
-    for line in csv.DictReader(open(documents_file)):
-        qid = line[query_id_col].strip()
-        did = line[document_id_col].strip()
-        text = line[document_text_col].strip()
-        extra_metadata = {
-            k: v
-            for k, v in line.items()
-            if k not in [query_id_col, document_id_col, document_text_col]
-        }
-        if qid not in queries_dict:
-            logging.info(f"Query {qid} not in the provided queries. Skipping")
-            continue
-        doc = Document(did=did, text=text, metadata=extra_metadata or None)
-        queries_dict[qid].add_retrieved_doc(doc)
-        documents_read += 1
-    logging.info(f"Loaded {documents_read} documents")
-    return list(queries_dict.values())
-
-
-def add_documents_from_run_file(
-    run_file_path: str,
-    documents_path: str,
-    queries_file: Optional[str] = None,
-    queries: Optional[List[Query]] = None,
-    document_id_col: str = "document_id",
-    document_text_col: str = "document_text",
-) -> List[Query]:
-    """Loads documents based on a TREC-formatted run file and a CSV with the documents."""
+    return asyncio.run(coroutine)
 
-    documents_read = 0
-    if queries is None and queries_file is None:
-        raise ValueError("Either queries or queries_file must be provided")
-    if queries is None:
-        assert queries_file is not None  # Should not happen. but mypy complains
-        queries = load_queries_from_csv(queries_file)
-    queries_dict = {q.qid: q for q in queries}
 
-    if not os.path.isfile(run_file_path):
-        raise FileNotFoundError(f"Run file {run_file_path} not found")
-    if not os.path.isfile(documents_path):
-        raise FileNotFoundError(f"Documents file {documents_path} not found")
-    # A dictionary with all queries associated with each document
-    query_per_doc = defaultdict(set)
-
-    for _line in open(run_file_path, "r"):
-        qid, _, did, _, _, _ = _line.strip().split()
-        if qid not in queries_dict:
-            logging.info(f"Query {qid} not in the provided queries. Skipping")
-            continue
-        query_per_doc[did].add(qid)
-
-    for line in csv.DictReader(open(documents_path)):
-        did = line[document_id_col].strip()
-        if did not in query_per_doc:
-            continue
-        text = line[document_text_col].strip()
-        for qid in query_per_doc[did]:
-            queries_dict[qid].add_retrieved_doc(Document(did=did, text=text))
-            documents_read += 1
-    logging.info(f"Loaded {documents_read} documents")
-    return list(queries_dict.values())
-
-
-def load_answers_from_multiple_csvs(
-    answer_files: List[str],
-    queries: Optional[Union[List[Query], str]] = None,
-    query_text_col: str = "query",
-    answer_text_col: str = "answer",
-    query_id_col: Optional[str] = None,
-):
-    """Loads queries and answers from a list of CSVs with queries and answers generated by agents.
-    We load all queries from all csvs first, and add the answers for all agents to the query objects.
-    The agents will be named according to the csv file name.
-    Args:
-        answer_files (List[str]): A list of CSVs files with agent answers.
-        queries (optional(Dict[str, Query] | str)): Either a dictionary with
-            existing queries, a csv file with the queries or None. If none, will
-            assume the queries also exist in the answers file.
-        query_text_col (str): Name of the column with the query.
-        answer_text_col (str): Name of the column with the agent's answer.
-        query_id_col (str): Name of the column with the query id. If not defined, will try to infer it from the csv headers
+def call_async_fn(fn: Callable[..., Coroutine[Any, Any, Any]], *args: Any, **kwargs: Any) -> Any:
     """
-    queries = []
-    queries_dict = {}
-
-    for f in answer_files:
-        if query_id_col is None:
-            query_id_col = infer_query_id_column(f)
-        if query_id_col is None:
-            query_id_col = query_text_col
-        p = Path(f)
-        agent_name = p.stem
-
-        for line in csv.DictReader(open(p)):
-            qid = line[query_id_col]
-            query_text = line[query_text_col].strip()
-            if qid not in queries_dict:
-                query = Query(qid=qid, query=query_text)
-                queries_dict[qid] = query
-                queries.append(query)
-                logging.info(f'added query: "{qid}"')
-            agent_answer = line[answer_text_col]
-            answer = AgentAnswer(agent=agent_name, text=agent_answer)
-            queries_dict[qid].add_agent_answer(answer)
-
-    return queries
-
-
-def add_answers_from_csv(
-    answers_file: str,
-    queries_file: Optional[str] = None,
-    queries: Optional[List[Query]] = None,
-    agent_col: str = "agent",
-    answer_col: str = "answer",
-    query_id_col: Optional[str] = None,
-) -> List[Query]:
-    """Loads all answers and agents from an answers CSV file.
-    Args:
-        answers_path (str): Path to the CSV file with the answers.
-        queries (Dict[str, Query]): Dictionary with the queries.
-        query_id_col (str): Name of the column with the query id. If not defined, will try to infer it from the csv headers
-        agent_col (str): Name of the column with the agent. Defaults to 'agent'.
-        answer_col (str): Name of the column with the answer. Defaults to 'answer'.
-    Returns:
-        List[Query]: A list of queries with the answers.
+    Calls an asynchronous function, either in the current event loop
+    or in a separate thread if no loop is running.
     """
-
-    if not os.path.isfile(answers_file):
-        raise FileNotFoundError(f"Answers file {answers_file} not found")
-
-    if queries is None and queries_file is None:
-        raise ValueError("Either queries or queries_file must be provided")
-    if queries is None:
-        assert queries_file is not None  # Should not happen. but mypy complains
-        queries = load_queries_from_csv(queries_file)
-
-    queries_dict = {q.qid: q for q in queries}
-    query_id_col = query_id_col or infer_query_id_column(answers_file)
-    for line in csv.DictReader(open(answers_file)):
-        qid = line[query_id_col]
-        if qid not in queries_dict:
-            raise ValueError(f"Unknown query id {qid}")
-        agent = line[agent_col].strip()
-        answer = line[answer_col].strip()
-        extra_metadata = {
-            k: v
-            for k, v in line.items()
-            if k not in [query_id_col, agent_col, answer_col]
-        }
-        answer = AgentAnswer(agent=agent, text=answer, metadata=extra_metadata or None)
-        queries_dict[qid].add_agent_answer(answer)
-
-    return list(queries_dict.values())
-
-
-# TODO: Replace all this loading by JSON serialization of Pydantic models
-def load_answer_evaluations_from_csv(
-    evaluations_file: str,
-) -> List[Query]:
-    """Loads all evaluations produced by an answer evaluator from a CSV file."""
-
-    queries_dict = {}
-    for row in csv.DictReader(open(evaluations_file)):
-        qid = row["qid"]
-        agent_a = row["agent_a"]
-        agent_b = row["agent_b"]
-        raw_answer = row["raw_answer"]
-        answer = row["answer"]
-
-        if qid not in queries_dict:
-            queries_dict[qid] = Query(qid=qid, query="<unknown>")
-        query = queries_dict[qid]
-        query.pairwise_games.append(
-            PairwiseGame(
-                agent_a_answer=AgentAnswer(agent=agent_a, text=""),
-                agent_b_answer=AgentAnswer(agent=agent_b, text=""),
-                evaluation=EvaluatorResult(
-                    raw_answer=raw_answer, answer=answer, qid=qid
-                ),
-            )
-        )
-    queries = list(queries_dict.values())
-    return queries
+    try:
+        asyncio.get_running_loop()
+        with ThreadPoolExecutor(max_workers=1) as executor:
+            # Execute the coroutine using the `run` function in a thread pool
+            future = executor.submit(run, fn(*args, **kwargs))
+            return future.result()
+    except RuntimeError:
+        # If no running loop is detected, run the coroutine directly
+        return asyncio.run(fn(*args, **kwargs))
+
+
+def get_pbar(
+    total: int, use_rich: bool = True, ncols: int = 100, desc: str = "Evaluating", disable: bool = False
+) -> tqdm | tqdm_rich:
+    if use_rich:
+        warnings.filterwarnings("ignore", category=TqdmExperimentalWarning)
+        return tqdm_rich(total=total, ncols=ncols, desc=desc, disable=disable)
+    else:
+        return tqdm(total=total, ncols=ncols, desc=desc, disable=disable)


Test patch:
diff --git a/tests/cli/test_cli.py b/tests/cli/test_cli.py
index 62914c0..b2f029e 100644
--- a/tests/cli/test_cli.py
+++ b/tests/cli/test_cli.py
@@ -1,6 +1,8 @@
-import csv
+from __future__ import annotations
+
 import os
 
+import pytest
 from typer.testing import CliRunner
 
 from ragelo.cli.cli import app
@@ -8,6 +10,7 @@
 runner = CliRunner()
 
 
+@pytest.mark.requires_openai
 def test_run_all_cli():
     result = runner.invoke(
         app,
@@ -19,12 +22,30 @@ def test_run_all_cli():
             "--verbose",
             "--data-dir",
             "tests/data/",
+            "--experiment-name",
+            "test",
+            "--output-file",
+            "output.json",
+            "--force",
         ],
     )
     assert result.exit_code == 0
-    assert "Agent Scores by elo" in result.stdout
+    assert "Agents Elo Ratings" in result.stdout
+    assert result.stdout.startswith("🔎 Query ID: 0\n📜 Document ID: 0")
+    assert "🔎 Query ID: 0\n agent1              🆚   agent2\nParsed Answer: A" in result.stdout
+    assert "Total evaluations: 4" in result.stdout
+    assert "Total evaluations: 2" in result.stdout
+    assert "Evaluating Retrieved documents" in result.stdout
+    assert "Evaluating Agent Answers" in result.stdout
+    assert len(result.stdout.split("✅ Done!")) == 3
+    assert os.path.exists("tests/data/output.json")
+    assert os.path.exists("tests/data/output_results.jsonl")
+    assert not os.path.exists("ragelo_cache/test.json")
+    os.remove("tests/data/output.json")
+    os.remove("tests/data/output_results.jsonl")
 
 
+@pytest.mark.requires_openai
 def test_run_reasoner_cli():
     result = runner.invoke(
         app,
@@ -33,84 +54,54 @@ def test_run_reasoner_cli():
             "reasoner",
             "queries.csv",
             "documents.csv",
-            "output.csv",
             "--verbose",
             "--data-dir",
             "tests/data/",
+            "--experiment-name",
+            "test",
+            "--output-file",
+            "test-output.json",
+            "--no-save-results",
+            "--force",
         ],
     )
     assert result.exit_code == 0
     assert "✅ Done!" in result.stdout
-    assert "Failed evaluations: 0" in result.stdout
+    assert result.stdout.startswith("🔎 Query ID: 0\n📜 Document ID: 0")
     assert "Total evaluations: 4" in result.stdout
-    # Make sure that the output file was created
-    with open("tests/data/reasonings.csv", "r") as f:
-        reader = csv.DictReader(f)
-        assert reader.fieldnames == ["qid", "did", "raw_answer", "answer"]
-        assert len(list(reader)) == 4
+    assert os.path.exists("tests/data/test-output.json")
+    os.remove("tests/data/test-output.json")
 
 
+@pytest.mark.requires_openai
 def test_run_answer_cli():
     result = runner.invoke(
         app,
         [
             "answer-evaluator",
-            "pairwise-reasoning",
+            "pairwise",
             "queries.csv",
             "documents.csv",
             "answers.csv",
-            "--games-evaluations-file",
-            "pairwise_answers_evaluations.csv",
             "--verbose",
             "--data-dir",
             "tests/data/",
+            "--experiment-name",
+            "test",
+            "--output-file",
+            "test-output.json",
+            "--no-save-results",
+            "--force",
+            "--bidirectional",
+            "--add-reasoning",
         ],
     )
     assert result.exit_code == 0
-    assert "✅ Done!" in result.stdout
-    assert "Failed evaluations: 0" in result.stdout
-    assert "Total evaluations: 2" in result.stdout
-    # Make sure that the output file was created
-    with open("tests/data/pairwise_answers_evaluations.csv", "r") as f:
-        reader = csv.DictReader(f)
-        assert reader.fieldnames == [
-            "qid",
-            "agent_a",
-            "agent_b",
-            "raw_answer",
-            "answer",
-            "relevance",
-        ]
-        assert len(list(reader)) == 2
-
-
-def test_run_agents_ranker_cli():
-    # remove the output file if it exists
-    try:
-        os.remove("tests/data/agents_ranking.csv")
-    except FileNotFoundError:
-        pass
-    result = runner.invoke(
-        app,
-        [
-            "agents-ranker",
-            "elo",
-            "pairwise_answers_evaluations.csv",
-            "--agents-evaluations-file",
-            "agents_ranking.csv",
-            "--verbose",
-            "--data-dir",
-            "tests/data/",
-        ],
-    )
-    assert result.exit_code == 0
-    lines = result.stdout.split("\n")
-    assert "Agent Scores by elo" in lines[0]
-    assert "agent1" in lines[1]
-    assert "agent2" in lines[2]
-
-    # Make sure that the output file was created
-    with open("tests/data/agents_ranking.csv", "r") as f:
-        reader = csv.DictReader(f)
-        assert reader.fieldnames == ["agent", "score"]
-        assert len(list(reader)) == 2
+    assert len(result.stdout.split("✅ Done!")) == 3
+    assert len(result.stdout.split("Total evaluations: 4")) == 3
+    assert result.stdout.startswith("🔎 Query ID: 0\n📜 Document ID: 0")
+    assert "Evaluating Retrieved documents" in result.stdout
+    assert "🔎 Query ID: 0\n agent2              🆚   agent1\nParsed Answer: B" in result.stdout
+    assert "🔎 Query ID: 1\n agent1              🆚   agent2\nParsed Answer: A" in result.stdout
+    assert os.path.exists("tests/data/test-output.json")
+    os.remove("tests/data/test-output.json")
diff --git a/tests/conftest.py b/tests/conftest.py
index 957b85c..a374ff4 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1,73 +1,66 @@
+from __future__ import annotations
+
 import json
-from unittest.mock import AsyncMock, Mock
+from typing import Any, Type
+from unittest.mock import AsyncMock
 
 import pytest
 from openai import AsyncOpenAI
-from openai.resources.chat import AsyncChat
+from openai.resources.beta import AsyncBeta
+from openai.resources.beta.chat import AsyncChat
 from openai.types.chat.chat_completion import ChatCompletion, Choice
 from openai.types.chat.chat_completion_message import ChatCompletionMessage
+from openai.types.chat.parsed_chat_completion import (
+    ParsedChatCompletion,
+    ParsedChatCompletionMessage,
+    ParsedChoice,
+)
+from pydantic import BaseModel as PydanticBaseModel
 
 from ragelo.llm_providers.base_llm_provider import BaseLLMProvider
 from ragelo.llm_providers.openai_client import OpenAIConfiguration
 from ragelo.types.configurations import (
+    BaseAnswerEvaluatorConfig,
     BaseEvaluatorConfig,
     BaseRetrievalEvaluatorConfig,
     CustomPromptAnswerEvaluatorConfig,
     CustomPromptEvaluatorConfig,
     DomainExpertEvaluatorConfig,
     FewShotEvaluatorConfig,
-    FewShotExample,
     LLMProviderConfig,
+    PairwiseDomainExpertEvaluatorConfig,
     PairwiseEvaluatorConfig,
     RDNAMEvaluatorConfig,
 )
-from ragelo.utils import (
-    add_answers_from_csv,
-    add_documents_from_csv,
-    load_queries_from_csv,
+from ragelo.types.configurations.retrieval_evaluator_configs import FewShotExample
+from ragelo.types.evaluables import ChatMessage
+from ragelo.types.experiment import Experiment
+from ragelo.types.formats import AnswerFormat, LLMResponseType
+from ragelo.types.results import (
+    AnswerEvaluatorResult,
+    EloTournamentResult,
+    RetrievalEvaluatorResult,
 )
+from ragelo.types.types import AnswerEvaluatorTypes, RetrievalEvaluatorTypes
 
 
-class MockLLMProvider(BaseLLMProvider):
-    def __init__(self, config):
-        self.config = config
-        self.call_mocker = Mock(
-            side_effect=lambda prompt: f"Received prompt: {prompt}."
-        )
-        self.async_call_mocker = AsyncMock(
-            side_effect=lambda prompt: f"Async prompt: {prompt}."
-        )
-
-    @classmethod
-    def from_configuration(cls, config: LLMProviderConfig):
-        return cls(config)
-
-    async def call_async(self, prompt):
-        return await self.async_call_mocker(prompt)
-
-    def __call__(self, prompt) -> str:
-        return self.call_mocker(prompt)
-
-
-@pytest.fixture
-def queries_test():
-    return load_queries_from_csv("tests/data/queries.csv")
-
-
-@pytest.fixture
-def qs_with_docs(queries_test):
-    return add_documents_from_csv("tests/data/documents.csv", queries=queries_test)
-
-
-@pytest.fixture
-def rdnam_queries():
-    queries = load_queries_from_csv("tests/data/rdnam_queries.csv")
-    return add_documents_from_csv("tests/data/documents.csv", queries=queries)
+def pytest_addoption(parser):
+    parser.addoption(
+        "--runopenai",
+        action="store_true",
+        default=False,
+        help="run tests that requires an OpenAI API key",
+    )
 
 
-@pytest.fixture
-def answers_test(queries_test):
-    return add_answers_from_csv("tests/data/answers.csv", queries=queries_test)
+def pytest_collection_modifyitems(config, items):
+    if config.getoption("--runopenai"):
+        # --runopenai given in cli: do not skip tests that will call OpenAI
+        return
+    skip_openai = pytest.mark.skip(reason="need --runopenai option to run")
+    for item in items:
+        if "requires_openai" in item.keywords:
+            item.add_marker(skip_openai)
 
 
 @pytest.fixture
@@ -89,26 +82,106 @@ def llm_provider_config():
     )
 
 
+class AnswerModel(PydanticBaseModel):
+    keyA: str
+    keyB: str
+
+
+def answer_model_factory(prompt, answer_format, response_schema, **kwargs):
+    if answer_format == AnswerFormat.STRUCTURED:
+        return LLMResponseType(
+            raw_answer='{"keyA": "valueA", "keyB": "valueB"}',
+            parsed_answer=AnswerModel(keyA="valueA", keyB="valueB"),
+        )
+    elif answer_format == AnswerFormat.JSON:
+        return LLMResponseType(
+            raw_answer='{"relevance": 1}',
+            parsed_answer={"relevance": 1},
+        )
+    elif answer_format == AnswerFormat.TEXT:
+        return LLMResponseType(
+            raw_answer='{"keyA": "valueA", "keyB": "valueB"}',
+            parsed_answer='{"keyA": "valueA", "keyB": "valueB"}',
+        )
+    else:
+        raise ValueError(f"Unsupported answer format: {answer_format}")
+
+
+class MockLLMProvider(BaseLLMProvider):
+    def __init__(self, config):
+        self.config = config
+
+        def side_effect(*args, **kwargs):
+            return answer_model_factory(*args, **kwargs)
+
+        self.async_call_mocker = AsyncMock(side_effect=side_effect)
+
+    @classmethod
+    def from_configuration(cls, config: LLMProviderConfig):
+        return cls(config)
+
+    async def call_async(
+        self,
+        prompt: str | list[dict[str, str]],
+        answer_format: AnswerFormat = AnswerFormat.TEXT,
+        response_schema: Type[PydanticBaseModel] | dict[str, Any] | None = None,
+    ) -> LLMResponseType:
+        return await self.async_call_mocker(prompt, answer_format, response_schema)
+
+
 @pytest.fixture
-def chat_completion_mock():
-    fake_response = ChatCompletion(
+def chat_completion_mock(answer_format):
+    _cls: Type[ChatCompletion | ParsedChatCompletion]
+    if answer_format == AnswerFormat.STRUCTURED:
+        _cls = ParsedChatCompletion
+    else:
+        _cls = ChatCompletion
+
+    async_mock = AsyncMock()
+    fake_response = _cls(
         id="fake id",
-        choices=[
+        choices=[],
+        created=0,
+        model="fake model",
+        object="chat.completion",
+    )
+    if answer_format == AnswerFormat.TEXT:
+        fake_response.choices.append(
+            Choice(
+                finish_reason="stop",
+                index=0,
+                logprobs=None,
+                message=ChatCompletionMessage(content="fake response", role="assistant"),
+            )  # type: ignore
+        )
+        async_mock.create.return_value = fake_response
+    elif answer_format == AnswerFormat.JSON:
+        fake_response.choices.append(
             Choice(
                 finish_reason="stop",
                 index=0,
                 logprobs=None,
-                message=ChatCompletionMessage(
-                    content="fake response", role="assistant"
+                message=ChatCompletionMessage(content='{"keyA": "valueA", "keyB": "valueB"}', role="assistant"),
+            )  # type: ignore
+        )
+        async_mock.create.return_value = fake_response
+    elif answer_format == AnswerFormat.STRUCTURED:
+        fake_response.choices.append(
+            ParsedChoice(
+                finish_reason="stop",
+                index=0,
+                logprobs=None,
+                message=ParsedChatCompletionMessage(
+                    content='{"keyA": "valueA", "keyB": "valueB"}',
+                    parsed=AnswerModel(keyA="valueA", keyB="valueB"),
+                    role="assistant",
                 ),
-            )
-        ],
-        created=0,
-        model="fake model",
-        object="chat.completion",
-    )
-    async_mock = AsyncMock()
-    async_mock.create.return_value = fake_response
+            )  # type: ignore
+        )
+        async_mock.parse.return_value = fake_response
+    else:
+        raise ValueError(f"Unsupported answer format: {answer_format}")
+
     return async_mock
 
 
@@ -116,53 +189,175 @@ def chat_completion_mock():
 def openai_client_mock(mocker, chat_completion_mock):
     openai_client = mocker.AsyncMock(AsyncOpenAI)
     type(openai_client).chat = mocker.AsyncMock(AsyncChat)
-    type(openai_client.chat).completions = mocker.PropertyMock(
-        return_value=chat_completion_mock
-    )
+    type(openai_client.chat).completions = mocker.PropertyMock(return_value=chat_completion_mock)
+    type(openai_client).beta = mocker.AsyncMock(AsyncBeta)
+    type(openai_client.beta).chat = mocker.AsyncMock(AsyncChat)
+    type(openai_client.beta.chat).completions = mocker.PropertyMock(return_value=chat_completion_mock)
     return openai_client
 
 
 @pytest.fixture
-def base_eval_config():
-    return BaseEvaluatorConfig(
-        documents_file="tests/data/documents.csv",
-        queries_file="tests/data/queries.csv",
-        force=True,
-        verbose=True,
-        write_output=False,
+def base_experiment_config():
+    config = {
+        "experiment_name": "test_experiment",
+        "save_on_disk": False,
+        "cache_evaluations": False,
+        "queries_csv_path": "tests/data/queries.csv",
+        "documents_csv_path": "tests/data/documents.csv",
+        "answers_csv_path": "tests/data/answers.csv",
+        "rich_print": True,
+        "verbose": True,
+    }
+    return config
+
+
+@pytest.fixture
+def experiment(base_experiment_config):
+    return Experiment(**base_experiment_config)
+
+
+@pytest.fixture
+def experiment_with_conversations_and_reasonings(experiment):
+    experiment.queries["0"].answers["agent1"].conversation = [
+        ChatMessage(sender="user", content="What is the capital of Brazil?"),
+        ChatMessage(
+            sender="agent1",
+            content="Brasília is the capital of Brazil, according to [0].",
+        ),
+    ]
+    experiment.queries["0"].answers["agent2"].conversation = [
+        ChatMessage(sender="user", content="What is the capital of Brazil?"),
+        ChatMessage(
+            sender="agent2",
+            content="According to [1], Rio de Janeiro used to be the capital of Brazil, until the 60s.",
+        ),
+    ]
+    experiment.queries["1"].answers["agent1"].conversation = [
+        ChatMessage(sender="user", content="What is the capital of France?"),
+        ChatMessage(sender="agent1", content="Paris is the capital of France, according to [2]."),
+    ]
+    experiment.queries["1"].answers["agent2"].conversation = [
+        ChatMessage(sender="user", content="What is the capital of France?"),
+        ChatMessage(
+            sender="agent2",
+            content="According to [3], Lyon is the second largest city in France. Meanwhile, Paris is its capital [2].",
+        ),
+    ]
+    experiment.queries["0"].retrieved_docs["0"].evaluation = RetrievalEvaluatorResult(
+        qid=experiment.queries["0"].qid,
+        did="0",
+        raw_answer="The document is very relevant as it directly answers the user's question about the capital of Brazil",
+        answer="The document is very relevant as it directly answers the user's question about the capital of Brazil",
+    )
+    experiment.queries["0"].retrieved_docs["1"].evaluation = RetrievalEvaluatorResult(
+        qid=experiment.queries["0"].qid,
+        did="1",
+        raw_answer="The document is somewhat relevant as it provides historical information about the capital of Brazil, but it does not provide the current capital.",
+        answer="The document is somewhat relevant as it provides historical information about the capital of Brazil, but it does not provide the current capital.",
+    )
+    experiment.queries["1"].retrieved_docs["2"].evaluation = RetrievalEvaluatorResult(
+        qid=experiment.queries["1"].qid,
+        did="2",
+        raw_answer="The document is very relevant as it directly answers the user's question about the capital of France.",
+        answer="The document is very relevant as it directly answers the user's question about the capital of France.",
     )
+    experiment.queries["1"].retrieved_docs["3"].evaluation = RetrievalEvaluatorResult(
+        qid=experiment.queries["1"].qid,
+        did="3",
+        raw_answer="The document is not relevant to the user question as it does not provide information about the capital of France.",
+        answer="The document is not relevant to the user question as it does not provide information about the capital of France.",
+    )
+    return experiment
 
 
 @pytest.fixture
-def base_retrieval_eval_config(base_eval_config):
-    base_config = base_eval_config.model_dump()
-    del base_config["answer_format_retrieval_evaluator"]
-    return BaseRetrievalEvaluatorConfig(
-        answer_format_retrieval_evaluator="json",
-        **base_config,
+def experiment_with_retrieval_scores(experiment):
+    experiment.queries["0"].retrieved_docs["0"].retrieved_by["agent1"] = 1.0
+    experiment.queries["0"].retrieved_docs["0"].retrieved_by["agent2"] = 0.5
+    experiment.queries["0"].retrieved_docs["1"].retrieved_by["agent1"] = 0.5
+    experiment.queries["0"].retrieved_docs["1"].retrieved_by["agent2"] = 1.0
+    experiment.queries["1"].retrieved_docs["2"].retrieved_by["agent1"] = 1.0
+    experiment.queries["1"].retrieved_docs["2"].retrieved_by["agent2"] = 0.5
+    experiment.queries["1"].retrieved_docs["3"].retrieved_by["agent1"] = 0.5
+    experiment.queries["1"].retrieved_docs["3"].retrieved_by["agent2"] = 1.0
+    return experiment
+
+
+@pytest.fixture
+def empty_experiment(base_experiment_config):
+    base_config = base_experiment_config.copy()
+    base_config.pop("queries_csv_path")
+    base_config.pop("documents_csv_path")
+    base_config.pop("answers_csv_path")
+    return Experiment(**base_config)
+
+
+@pytest.fixture
+def retrieval_evaluation():
+    return RetrievalEvaluatorResult(
+        qid="0",
+        did="0",
+        raw_answer="Document is relevant. Score: 1.0",
+        answer=1,
+    )
+
+
+@pytest.fixture
+def answer_evaluation():
+    return AnswerEvaluatorResult(
+        qid="0",
+        agent="agent1",
+        pairwise=False,
+        raw_answer="Answer is good. Scores: {'quality': 1.0, 'relevance': 0.8}",
+        answer={"quality": 1.0, "relevance": 0.8},
+    )
+
+
+@pytest.fixture
+def pairwise_answer_evaluation():
+    return AnswerEvaluatorResult(
+        qid="0",
+        agent_a="agent1",
+        agent_b="agent2",
+        pairwise=True,
+        raw_answer="Answer [[A]] is better than [[B]]",
+        answer="A",
+    )
+
+
+@pytest.fixture
+def elo_tournament_result():
+    return EloTournamentResult(
+        agents=["agent1", "agent2"],
+        scores={"agent1": 1200, "agent2": 1000},
+        games_played={"agent1": 1, "agent2": 1},
+        wins={"agent1": 1, "agent2": 0},
+        loses={"agent1": 0, "agent2": 1},
+        ties={"agent1": 0, "agent2": 0},
+        total_games=2,
+        total_tournaments=1,
+        std_dev={"agent1": 0, "agent2": 0},
     )
 
 
 @pytest.fixture
-def pairwise_answer_eval_config(base_eval_config):
+def base_eval_config():
+    return BaseEvaluatorConfig(force=True, verbose=True, llm_answer_format=AnswerFormat.JSON)
+
+
+@pytest.fixture
+def base_retrieval_eval_config(base_eval_config):
     base_config = base_eval_config.model_dump()
-    base_config["document_evaluations_file"] = "tests/data/reasonings.csv"
-    config = PairwiseEvaluatorConfig(
-        bidirectional=True,
+    base_config["evaluator_name"] = RetrievalEvaluatorTypes.REASONER
+    return BaseRetrievalEvaluatorConfig(
         **base_config,
     )
-    return config
 
 
 @pytest.fixture
 def custom_answer_eval_config(base_eval_config):
     base_config = base_eval_config.model_dump()
-    base_config["answer_format_answer_evaluator"] = "multi_field_json"
-    base_config["scoring_keys_answer_evaluator"] = [
-        "quality",
-        "trustworthiness",
-        "originality",
-    ]
+    base_config["evaluator_name"] = AnswerEvaluatorTypes.CUSTOM_PROMPT
     config = CustomPromptAnswerEvaluatorConfig(
         prompt="""
 You are an useful assistant for evaluating the quality of the answers generated \
@@ -185,8 +380,7 @@ def custom_answer_eval_config(base_eval_config):
 @pytest.fixture
 def expert_retrieval_eval_config(base_eval_config):
     base_eval_config = base_eval_config.model_dump()
-    del base_eval_config["scoring_keys_retrieval_evaluator"]
-    del base_eval_config["answer_format_retrieval_evaluator"]
+    base_eval_config["evaluator_name"] = AnswerEvaluatorTypes.DOMAIN_EXPERT
     return DomainExpertEvaluatorConfig(
         expert_in="Computer Science",
         domain_short="computer scientists",
@@ -200,9 +394,11 @@ def expert_retrieval_eval_config(base_eval_config):
 def rdnam_config(base_eval_config):
     base_config = base_eval_config.model_dump()
     base_config["query_file"] = "tests/data/rdnam_queries.csv"
+    base_config["evaluator_name"] = RetrievalEvaluatorTypes.RDNAM
     return RDNAMEvaluatorConfig(
         annotator_role="You are a search quality rater evaluating the relevance of web pages. ",
         use_multiple_annotators=True,
+        use_aspects=True,
         **base_config,
     )
 
@@ -210,19 +406,79 @@ def rdnam_config(base_eval_config):
 @pytest.fixture
 def custom_prompt_retrieval_eval_config(base_eval_config):
     base_eval_config = base_eval_config.model_dump()
-    del base_eval_config["scoring_keys_retrieval_evaluator"]
-    del base_eval_config["answer_format_retrieval_evaluator"]
+    base_eval_config["evaluator_name"] = RetrievalEvaluatorTypes.CUSTOM_PROMPT
     config = CustomPromptEvaluatorConfig(
         prompt="query: {query} doc: {document}",
-        scoring_keys_retrieval_evaluator=["relevance"],
         **base_eval_config,
     )
     return config
 
 
+@pytest.fixture
+def llm_provider_mock(llm_provider_config):
+    return MockLLMProvider(llm_provider_config)
+
+
+@pytest.fixture
+def llm_provider_mock_rdnam(llm_provider_config):
+    mocked_scores = {
+        "annotator_1": {"intent_match": 2, "trustworthiness": 1, "overall": 1},
+        "annotator_2": {"intent_match": 1, "trustworthiness": 1, "overall": 2},
+    }
+    LLM_response = LLMResponseType(raw_answer=json.dumps(mocked_scores), parsed_answer=mocked_scores)
+    provider = MockLLMProvider(llm_provider_config)
+
+    def side_effect(*args, **kwargs):
+        return LLM_response
+
+    provider.async_call_mocker = AsyncMock(side_effect=side_effect)
+    return provider
+
+
+@pytest.fixture
+def llm_provider_reasoner_mock(llm_provider_config):
+    provider = MockLLMProvider(llm_provider_config)
+    answer = LLMResponseType(
+        raw_answer="The document is very relevant",
+        parsed_answer="The document is very relevant",
+    )
+
+    def side_effect(*args, **kwargs):
+        return answer
+
+    provider.async_call_mocker = AsyncMock(side_effect=side_effect)
+    return provider
+
+
+@pytest.fixture
+def base_answer_eval_config(base_eval_config):
+    base_config = base_eval_config.model_dump()
+    return BaseAnswerEvaluatorConfig(**base_config)
+
+
+@pytest.fixture
+def pairwise_answer_eval_config(base_answer_eval_config):
+    base_config = base_answer_eval_config.model_dump()
+    base_config["pairwise"] = True
+    base_config["evaluator_name"] = AnswerEvaluatorTypes.PAIRWISE
+    return PairwiseEvaluatorConfig(bidirectional=True, **base_config)
+
+
+@pytest.fixture
+def domain_expert_answer_eval_config(base_answer_eval_config):
+    base_config = base_answer_eval_config.model_dump()
+    base_config["pairwise"] = True
+    base_config["expert_in"] = "Computer Science"
+    base_config["include_annotations"] = True
+    base_config["include_raw_documents"] = False
+    base_config["evaluator_name"] = AnswerEvaluatorTypes.DOMAIN_EXPERT
+    return PairwiseDomainExpertEvaluatorConfig(**base_config)
+
+
 @pytest.fixture
 def few_shot_retrieval_eval_config(base_eval_config):
     base_eval_config = base_eval_config.model_dump()
+    base_eval_config["evaluator_name"] = RetrievalEvaluatorTypes.FEW_SHOT
     few_shot_samples = [
         FewShotExample(
             passage="Few shot example 1",
@@ -237,7 +493,7 @@ def few_shot_retrieval_eval_config(base_eval_config):
             reasoning="This is a bad document",
         ),
     ]
-    del base_eval_config["answer_format_retrieval_evaluator"]
+
     return FewShotEvaluatorConfig(
         system_prompt="System prompt",
         few_shot_user_prompt="query: {query} doc: {document}",
@@ -245,70 +501,39 @@ def few_shot_retrieval_eval_config(base_eval_config):
         reasoning_placeholder="reasoning",
         relevance_placeholder="relevance",
         few_shots=few_shot_samples,
-        answer_format_retrieval_evaluator="json",
         **base_eval_config,
     )
 
 
 @pytest.fixture
-def llm_provider_mock(llm_provider_config):
-    return MockLLMProvider(llm_provider_config)
-
-
-@pytest.fixture
-def llm_provider_json_mock(llm_provider_config):
+def llm_provider_answer_mock(llm_provider_config):
     provider = MockLLMProvider(llm_provider_config)
-    provider.call_mocker = Mock(
-        side_effect=lambda _: 'LLM JSON response\n{"relevance": 0}'
-    )
-    provider.async_call_mocker = AsyncMock(
-        side_effect=lambda _: 'Async LLM JSON response\n{"relevance": 1}'
+    mocked_answer = LLMResponseType(
+        raw_answer='{"quality": 1, "trustworthiness": 0, "originality": 0}',
+        parsed_answer={"quality": 1, "trustworthiness": 0, "originality": 0},
     )
-    return provider
 
+    def side_effect(*args, **kwargs):
+        return mocked_answer
 
-@pytest.fixture
-def llm_provider_pairwise_answer_mock(llm_provider_config):
-    provider = MockLLMProvider(llm_provider_config)
-    provider.call_mocker = Mock(
-        side_effect=[
-            "Agent [[A]] is better",
-            "Agent [[B]] is better",
-            "A tie. Therefore, [[C]]",
-            "I don't know. [[C]]",
-        ]
-    )
-    provider.async_call_mocker = AsyncMock(
-        side_effect=[
-            "Async Agent [[A]] is better",
-            "Async Agent [[B]] is better",
-            "Async A tie. Therefore, [[C]]",
-            "Async I don't know. [[C]]",
-        ]
-    )
+    provider.async_call_mocker = AsyncMock(side_effect=side_effect)
     return provider
 
 
 @pytest.fixture
-def llm_provider_answer_mock(llm_provider_config):
+def llm_provider_pairwise_answer_mock(llm_provider_config):
     provider = MockLLMProvider(llm_provider_config)
-    provider.call_mocker = Mock(
-        side_effect=lambda prompt: f"Answer for {prompt}\n"
-        '{"quality": 2, "trustworthiness": 1, "originality": 1}',
-    )
-    provider.async_call_mocker = AsyncMock(
-        side_effect=lambda prompt: f"Async answer for {prompt}\n"
-        '{"quality": 1, "trustworthiness": 0, "originality": 0}',
-    )
-    return provider
+    answer = {
+        "answer_a_reasoning": "Answer A is good",
+        "answer_b_reasoning": "Answer B is bad",
+        "comparison_reasoning": "Answer A is better than Answer B",
+        "winner": "A",
+    }
+    LLM_response = LLMResponseType(raw_answer=json.dumps(answer), parsed_answer=answer)
 
+    def side_effect(*args, **kwargs):
+        return LLM_response
+
+    provider.async_call_mocker = AsyncMock(side_effect=side_effect)
 
-@pytest.fixture
-def llm_provider_mock_rdnam(llm_provider_config):
-    mocked_scores = [{"M": 2, "T": 1, "O": 1}, {"M": 1, "T": 1, "O": 2}]
-    provider = MockLLMProvider(llm_provider_config)
-    provider.call_mocker = Mock(side_effect=lambda _: json.dumps(mocked_scores)[2:])
-    provider.async_call_mocker = AsyncMock(
-        side_effect=lambda _: json.dumps(mocked_scores)[2:]
-    )
     return provider
diff --git a/tests/data/documents.csv b/tests/data/documents.csv
index eaeef9e..696fda1 100644
--- a/tests/data/documents.csv
+++ b/tests/data/documents.csv
@@ -1,4 +1,4 @@
-qid,did,document_text
+qid,did,document
 0,0, Brasília is the capital of Brazil.
 0,1, Rio de Janeiro used to be the capital of Brazil.
 1,2, Paris is the capital of France.
diff --git a/tests/unit/test_answer_evaluators.py b/tests/unit/test_answer_evaluators.py
index 884f22a..ff33417 100644
--- a/tests/unit/test_answer_evaluators.py
+++ b/tests/unit/test_answer_evaluators.py
@@ -1,111 +1,296 @@
+from __future__ import annotations
+
+import json
+import warnings
+
 import pytest
 
 from ragelo import get_answer_evaluator
 from ragelo.evaluators.answer_evaluators import (
+    BaseAnswerEvaluator,
+    ChatPairwiseEvaluator,
+    CustomPairwiseEvaluator,
     CustomPromptEvaluator,
     PairwiseAnswerEvaluator,
+    PairwiseDomainExpertEvaluator,
 )
-from ragelo.types.types import AnswerEvaluatorResult
+from ragelo.types.evaluables import AgentAnswer, PairwiseGame
+from ragelo.types.query import Query
+from ragelo.types.results import AnswerEvaluatorResult
+
+
+class AnswerEvaluator(BaseAnswerEvaluator):
+    def _build_message(self, query: Query, answer: AgentAnswer) -> str:
+        return f"Query: {query.query}\nAnswer: {answer.text}"
+
+    def _build_message_pairwise(self, query: Query, game: PairwiseGame) -> str:
+        return f"Query: {query.query}\nAnswer A: {game.agent_a_answer.text}\nAnswer B: {game.agent_b_answer.text}"
+
+    def test_get_by_name(self, llm_provider_mock):
+        pairwise_evaluator = get_answer_evaluator(
+            "pairwise",
+            llm_provider_mock,
+        )
+        assert isinstance(pairwise_evaluator, PairwiseAnswerEvaluator)
+        custom_evaluator = get_answer_evaluator(
+            "custom_prompt",
+            llm_provider_mock,
+        )
+        assert isinstance(custom_evaluator, CustomPromptEvaluator)
+        custom_pairwise_evaluator = get_answer_evaluator(
+            "custom_pairwise",
+            llm_provider=llm_provider_mock,
+            system_prompt="system prompt",
+            user_prompt="user prompt",
+        )
+        assert isinstance(custom_pairwise_evaluator, CustomPairwiseEvaluator)
+        domain_expert_evaluator = get_answer_evaluator(
+            "domain_expert",
+            expert_in="computer science",
+            llm_provider=llm_provider_mock,
+        )
+        assert isinstance(domain_expert_evaluator, PairwiseDomainExpertEvaluator)
+
+        chat_pairwise_evaluator = get_answer_evaluator(
+            "chat_pairwise",
+            llm_provider=llm_provider_mock,
+        )
+        assert isinstance(chat_pairwise_evaluator, ChatPairwiseEvaluator)
 
 
-class TestPairwiseWithReasoningEvaluator:
-    @pytest.mark.asyncio
-    async def test_batch_eval(
+class TestAnswerEvaluator:
+    def test_evaluate_single_answer(self, llm_provider_answer_mock, experiment, base_answer_eval_config):
+        pointwise_evaluator = AnswerEvaluator.from_config(
+            config=base_answer_eval_config, llm_provider=llm_provider_answer_mock
+        )
+        query = experiment["0"]
+        answer = query.answers["agent1"]
+        result = pointwise_evaluator.evaluate(query, answer)
+        assert isinstance(result, AnswerEvaluatorResult)
+        assert isinstance(result.answer, dict)
+        assert result.answer.keys() == {"quality", "trustworthiness", "originality"}
+        assert result.raw_answer == '{"quality": 1, "trustworthiness": 0, "originality": 0}'
+        assert result.exception is None
+        assert result.pairwise is False
+        assert result.qid == query.qid
+        assert result.agent == answer.agent
+        call_args = llm_provider_answer_mock.async_call_mocker.call_args_list
+        assert len(call_args) == 1
+        expected_prompt = f"Query: {query.query}\nAnswer: {answer.text}"
+        assert call_args[0][0][0] == expected_prompt
+
+    def test_evaluate_single_game(self, llm_provider_pairwise_answer_mock, experiment, pairwise_answer_eval_config):
+        evaluator = PairwiseAnswerEvaluator.from_config(
+            config=pairwise_answer_eval_config, llm_provider=llm_provider_pairwise_answer_mock
+        )
+        query = experiment["0"]
+        result = evaluator.evaluate(query, answer_a=query.answers["agent1"], answer_b=query.answers["agent2"])
+
+        assert isinstance(result, AnswerEvaluatorResult)
+        assert isinstance(result.raw_answer, str)
+        assert isinstance(result.answer, str)
+        raw_answer = json.loads(result.raw_answer)
+        assert raw_answer.keys() == {"answer_a_reasoning", "answer_b_reasoning", "comparison_reasoning", "winner"}
+        assert result.answer == "A"
+        assert result.exception is None
+        assert result.pairwise is True
+        assert result.qid == query.qid
+        assert result.agent_a == query.answers["agent1"].agent
+        assert result.agent_b == query.answers["agent2"].agent
+
+    def test_evaluate_experiment(self, llm_provider_answer_mock, experiment, base_answer_eval_config):
+        evaluator = AnswerEvaluator.from_config(config=base_answer_eval_config, llm_provider=llm_provider_answer_mock)
+        evaluator.evaluate_experiment(experiment)
+        for query in experiment:
+            for answer in query.answers.values():
+                assert isinstance(answer.evaluation, AnswerEvaluatorResult)
+                assert isinstance(answer.evaluation.answer, dict)
+                assert isinstance(answer.evaluation.raw_answer, str)
+                assert answer.evaluation.exception is None
+                assert answer.evaluation.pairwise is False
+                assert answer.evaluation.qid == query.qid
+                assert answer.evaluation.agent == answer.agent
+
+    def test_evaluate_pairwise_experiment(
+        self, llm_provider_pairwise_answer_mock, experiment, pairwise_answer_eval_config
+    ):
+        evaluator = PairwiseAnswerEvaluator.from_config(
+            config=pairwise_answer_eval_config, llm_provider=llm_provider_pairwise_answer_mock
+        )
+        evaluator.evaluate_experiment(experiment)
+        for query in experiment:
+            assert len(query.pairwise_games) == 2
+            for game in query.pairwise_games:
+                assert isinstance(game.evaluation, AnswerEvaluatorResult)
+                assert isinstance(game.evaluation.answer, str)
+                assert isinstance(game.evaluation.raw_answer, str)
+                assert game.evaluation.exception is None
+                assert game.evaluation.pairwise is True
+                assert game.evaluation.qid == query.qid
+                assert game.evaluation.agent_a == game.agent_a_answer.agent
+                assert game.evaluation.agent_b == game.agent_b_answer.agent
+
+
+class TestPairwiseAnswerEvaluator:
+    def test_evaluate_single_game(
         self,
         llm_provider_pairwise_answer_mock,
+        experiment_with_conversations_and_reasonings,
         pairwise_answer_eval_config,
-        answers_test,
     ):
         evaluator = PairwiseAnswerEvaluator.from_config(
             config=pairwise_answer_eval_config,
             llm_provider=llm_provider_pairwise_answer_mock,
         )
-        queries = await evaluator._async_batch_evaluate(answers_test)
-        flat_answers = [(q, a) for q in queries for a in q.pairwise_games]
-        evaluations = [a.evaluation for (_, a) in flat_answers]
-        assert len(evaluations) == 4
-        expected_answers = ["A", "B", "C", "C"]
-        for e, expected_ans in zip(evaluations, expected_answers):
-            assert isinstance(e, AnswerEvaluatorResult)
-            assert isinstance(e.answer, str)
-            assert isinstance(e.raw_answer, str)
-            assert e.answer == expected_ans
-            assert f"[[{expected_ans}]]" in e.raw_answer
-
-        llm_call_args = (
-            llm_provider_pairwise_answer_mock.async_call_mocker.call_args_list
-        )
-        assert len(llm_call_args) == 4
+        query = experiment_with_conversations_and_reasonings["0"]
+        result = evaluator.evaluate(query, answer_a=query.answers["agent1"], answer_b=query.answers["agent2"])
+        assert isinstance(result, AnswerEvaluatorResult)
+        assert isinstance(result.answer, str)
+        assert isinstance(result.raw_answer, str)
+        assert result.answer == "A"
+
+        llm_call_args = llm_provider_pairwise_answer_mock.async_call_mocker.call_args_list
+        assert len(llm_call_args) == 1
         assert isinstance(llm_call_args[0][0][0], str)
-        assert llm_call_args[0][0][0] != llm_call_args[1][0][0]
         # Make sure that no games with the same agent were called
-        for call_arg in llm_call_args:
-            agent_a_answer = (
-                call_arg[0][0]
-                .split("[The Start of Assistant A's Answer]")[1]
-                .split("[The End of Assistant A's Answer]")[0]
-            ).strip()
-            agent_b_answer = (
-                call_arg[0][0]
-                .split("[The Start of Assistant B's Answer]")[1]
-                .split("[The End of Assistant B's Answer]")[0]
-            ).strip()
-            assert agent_a_answer != agent_b_answer
+        prompt = llm_call_args[0][0][0]
+        agent_a_answer = (
+            prompt.split("[The Start of Assistant A's Answer]")[1].split("[The End of Assistant A's Answer]")[0]
+        ).strip()
+        agent_b_answer = (
+            prompt.split("[The Start of Assistant B's Answer]")[1]
+            .split("[The End of Assistant B's Answer]")[0]
+            .strip()
+        )
+        assert agent_a_answer != agent_b_answer
+        assert agent_a_answer == query.answers["agent1"].text
+        assert agent_b_answer == query.answers["agent2"].text
+
+    def test_evaluation_no_documents(
+        self, llm_provider_pairwise_answer_mock, empty_experiment, pairwise_answer_eval_config
+    ):
+        evaluator = PairwiseAnswerEvaluator.from_config(
+            config=pairwise_answer_eval_config, llm_provider=llm_provider_pairwise_answer_mock
+        )
+        empty_experiment.add_query("empty_query", "0")
+        empty_experiment.add_agent_answer("answer_a", "agent1", "0")
+        empty_experiment.add_agent_answer("answer_b", "agent2", "0")
+        query = empty_experiment["0"]
+        with pytest.warns(UserWarning):
+            _ = evaluator.evaluate(query, answer_a=query.answers["agent1"], answer_b=query.answers["agent2"])
 
 
 class TestCustomPromptEvaluator:
-    @pytest.mark.asyncio
-    async def test_batch_eval(
-        self,
-        llm_provider_answer_mock,
-        custom_answer_eval_config,
-        answers_test,
-    ):
+    def test_evaluate_single_answer(self, llm_provider_answer_mock, experiment, custom_answer_eval_config):
         evaluator = CustomPromptEvaluator.from_config(
             config=custom_answer_eval_config,
             llm_provider=llm_provider_answer_mock,
         )
-        queries = await evaluator._async_batch_evaluate(answers_test)
-        flat_answers = [(q, a) for q in queries for a in q.answers.values()]
-        assert len(flat_answers) == 4
-
+        query = experiment["0"]
+        answer = query.answers["agent1"]
+        result = evaluator.evaluate(query, answer)
+        assert isinstance(result, AnswerEvaluatorResult)
+        assert isinstance(result.answer, dict)
+        assert isinstance(result.raw_answer, str)
+        assert isinstance(result.answer["quality"], int)
+        assert isinstance(result.answer["trustworthiness"], int)
+        assert isinstance(result.answer["originality"], int)
         llm_call_args = llm_provider_answer_mock.async_call_mocker.call_args_list
-        assert (
-            len(
-                llm_call_args[0][0][0]
-                .split("DOCUMENTS RETRIEVED:")[1]
-                .split("User Query")[0]
-                .strip()
-                .split("\n")
-            )
-            == 2
-        )
-        for (q, a), args in zip(flat_answers, llm_call_args):
-            evaluation = a.evaluation
-            assert isinstance(evaluation, AnswerEvaluatorResult)
-            assert isinstance(evaluation.answer, dict)
-            assert isinstance(evaluation.raw_answer, str)
-            assert isinstance(evaluation.answer["quality"], int)
-            assert isinstance(evaluation.answer["trustworthiness"], int)
-            assert isinstance(evaluation.answer["originality"], int)
-
-            assert evaluation.answer["quality"] == 1
-            assert evaluation.answer["trustworthiness"] == 0
-            assert evaluation.answer["originality"] == 0
-
-            submitted_query = args[0][0].split("User Query: ")[1].split("\n")[0].strip()
-            submitted_answer = (
-                args[0][0].split("Agent answer: ")[1].split("\n")[-1].strip()
-            )
-            expected_query = q.query
-            expected_answer = a.text
-            assert submitted_query == expected_query
-            assert submitted_answer == expected_answer
-
-
-def test_get_by_name(llm_provider_pairwise_answer_mock, pairwise_answer_eval_config):
-    evaluator = get_answer_evaluator(
-        "pairwise",
+        documents_text = []
+        for did, d in query.retrieved_docs.items():
+            documents_text.append(f"[{did}] {d.text}")
+        expected_prompt = evaluator.prompt.format(
+            query=query.query,
+            answer=answer.text,
+            documents="\n".join(documents_text),
+        )
+        assert llm_call_args[0][0][0] == expected_prompt
+
+
+class TestChatPairwiseEvaluator:
+    def test_evaluate_single_game(
+        self,
+        llm_provider_pairwise_answer_mock,
+        experiment_with_conversations_and_reasonings,
+        pairwise_answer_eval_config,
+    ):
+        evaluator = ChatPairwiseEvaluator.from_config(
+            config=pairwise_answer_eval_config,
+            llm_provider=llm_provider_pairwise_answer_mock,
+        )
+        query = experiment_with_conversations_and_reasonings["0"]
+
+        result = evaluator.evaluate(query, answer_a=query.answers["agent1"], answer_b=query.answers["agent2"])
+        assert isinstance(result, AnswerEvaluatorResult)
+        assert isinstance(result.answer, str)
+        assert isinstance(result.raw_answer, str)
+        assert result.answer == "A"
+        llm_call_args = llm_provider_pairwise_answer_mock.async_call_mocker.call_args_list
+        assert len(llm_call_args) == 1
+        prompt = llm_call_args[0][0][0]
+        ans_a = (
+            prompt.split("[The Start of Conversation with Assistant A]")[1]
+            .split("[The End of Conversation with Assistant A]")[0]
+            .strip()
+        )
+        ans_b = (
+            prompt.split("[The Start of Conversation with Assistant B]")[1]
+            .split("[The End of Conversation with Assistant B]")[0]
+            .strip()
+        )
+        assert ans_a == "\n".join([str(msg) for msg in query.answers["agent1"].conversation])
+        assert ans_b == "\n".join([str(msg) for msg in query.answers["agent2"].conversation])
+
+
+class TestDomainExpertEvaluator:
+    def test_evaluate_single_answer(
+        self,
+        llm_provider_pairwise_answer_mock,
+        experiment_with_conversations_and_reasonings,
+        domain_expert_answer_eval_config,
+    ):
+        evaluator = PairwiseDomainExpertEvaluator.from_config(
+            config=domain_expert_answer_eval_config, llm_provider=llm_provider_pairwise_answer_mock
+        )
+        query = experiment_with_conversations_and_reasonings["0"]
+        with warnings.catch_warnings():
+            warnings.simplefilter("error")
+            result = evaluator.evaluate(query, answer_a=query.answers["agent1"], answer_b=query.answers["agent2"])
+        assert isinstance(result, AnswerEvaluatorResult)
+        prompt = llm_provider_pairwise_answer_mock.async_call_mocker.call_args_list[0][0][0]
+        assert isinstance(prompt, str)
+        assert "You work for" not in prompt
+
+    def test_evaluate_single_answer_with_company(
+        self,
         llm_provider_pairwise_answer_mock,
-    )
-    assert isinstance(evaluator, PairwiseAnswerEvaluator)
+        experiment_with_conversations_and_reasonings,
+        domain_expert_answer_eval_config,
+    ):
+        domain_expert_answer_eval_config.company = "Zeta Alpha"
+        evaluator = PairwiseDomainExpertEvaluator.from_config(
+            config=domain_expert_answer_eval_config, llm_provider=llm_provider_pairwise_answer_mock
+        )
+        query = experiment_with_conversations_and_reasonings["0"]
+        with warnings.catch_warnings():
+            warnings.simplefilter("error")
+            result = evaluator.evaluate(query, answer_a=query.answers["agent1"], answer_b=query.answers["agent2"])
+        assert isinstance(result, AnswerEvaluatorResult)
+        prompt = llm_provider_pairwise_answer_mock.async_call_mocker.call_args_list[0][0][0]
+        assert "You work for" in prompt
+
+    def test_evaluate_single_answer_no_documents(
+        self,
+        llm_provider_pairwise_answer_mock,
+        experiment,
+        domain_expert_answer_eval_config,
+    ):
+        evaluator = PairwiseDomainExpertEvaluator.from_config(
+            config=domain_expert_answer_eval_config, llm_provider=llm_provider_pairwise_answer_mock
+        )
+        query = experiment["0"]
+        with pytest.warns(UserWarning):
+            result = evaluator.evaluate(query, answer_a=query.answers["agent1"], answer_b=query.answers["agent2"])
+        assert isinstance(result, AnswerEvaluatorResult)
+        prompt = llm_provider_pairwise_answer_mock.async_call_mocker.call_args_list[0][0][0]
+        assert "You work for" not in prompt
diff --git a/tests/unit/test_experiment.py b/tests/unit/test_experiment.py
new file mode 100644
index 0000000..2b89d7d
--- /dev/null
+++ b/tests/unit/test_experiment.py
@@ -0,0 +1,321 @@
+from __future__ import annotations
+
+import json
+import os
+
+import pytest
+
+from ragelo.types.evaluables import AgentAnswer, Document
+from ragelo.types.experiment import Experiment
+from ragelo.types.query import Query
+from ragelo.types.results import AnswerEvaluatorResult, RetrievalEvaluatorResult
+
+
+class TestExperiment:
+    def test_experiment_initialization(self, experiment):
+        assert len(experiment) == 2
+        assert "0" in experiment.keys()
+        assert "1" in experiment.keys()
+
+        # Check queries were loaded correctly
+        assert experiment["0"].query == "What is the capital of Brazil?"
+        assert experiment["1"].query == "What is the capital of France?"
+
+        # Check documents were loaded correctly
+        assert len(experiment["0"].retrieved_docs) == 2
+        assert experiment["0"].retrieved_docs["0"].text == "Brasília is the capital of Brazil."
+
+        # Check answers were loaded correctly
+        assert len(experiment["0"].answers) == 2
+        assert "agent1" in experiment["0"].answers
+        assert "agent2" in experiment["0"].answers
+
+    def test_add_query(self, empty_experiment):
+        """Test adding queries manually"""
+        # Add query as string
+        qid = empty_experiment.add_query("Test query 1", query_id="test1")
+        assert qid in empty_experiment.queries
+        assert empty_experiment[qid].query == "Test query 1"
+
+        # Add query as Query object
+        query = Query(qid="test2", query="Test query 2")
+        qid2 = empty_experiment.add_query(query)
+        assert qid2 == "test2"
+        assert empty_experiment[qid2].query == "Test query 2"
+
+        # Test force parameter
+        with pytest.warns(UserWarning):
+            empty_experiment.add_query("New query", qid, force=False)
+        empty_experiment.add_query("Forced query", qid, force=True)
+        assert empty_experiment[qid].query == "Forced query"
+
+    def test_add_retrieved_doc(self, empty_experiment):
+        """Test adding retrieved documents manually"""
+        qid = empty_experiment.add_query("Test query", query_id="test1")
+
+        # Add document as string
+        empty_experiment.add_retrieved_doc("Test document", query_id=qid, doc_id="doc1", score=0.5, agent="agent1")
+        assert "doc1" in empty_experiment[qid].retrieved_docs
+        assert empty_experiment[qid].retrieved_docs["doc1"].text == "Test document"
+
+        # Add document as Document object
+        doc = Document(qid=qid, did="doc2", text="Test document 2")
+        empty_experiment.add_retrieved_doc(doc, agent="agent2")
+        assert "doc2" in empty_experiment[qid].retrieved_docs
+
+        # Test adding to non-existent query
+        with pytest.raises(ValueError):
+            empty_experiment.add_retrieved_doc("Test", "invalid_qid", "doc3")
+
+    def test_add_agent_answer(self, empty_experiment):
+        """Test adding agent answers manually"""
+        qid = empty_experiment.add_query("Test query", query_id="test1")
+
+        # Add answer as string
+        empty_experiment.add_agent_answer("Test answer", agent="agent1", query_id=qid)
+        assert "agent1" in empty_experiment[qid].answers
+        assert empty_experiment[qid].answers["agent1"].text == "Test answer"
+
+        # Add answer as AgentAnswer object
+        answer = AgentAnswer(qid=qid, agent="agent2", text="Test answer 2")
+        empty_experiment.add_agent_answer(answer)
+        assert "agent2" in empty_experiment[qid].answers
+
+        # Test adding to non-existent query
+        with pytest.raises(ValueError):
+            empty_experiment.add_agent_answer("Test", "agent3", "invalid_qid")
+
+    def test_save_and_load(self, tmp_path, base_experiment_config):
+        """Test saving and loading experiment state"""
+        # Save experiment
+        save_path = tmp_path / "test_experiment.json"
+        base_experiment_config["save_on_disk"] = True
+        base_experiment_config["save_path"] = str(save_path)
+        experiment = Experiment(**base_experiment_config)
+        experiment.save()
+
+        # Load experiment
+        loaded_experiment = Experiment(
+            experiment_name="test_experiment",
+            save_path=str(save_path),
+            cache_evaluations=False,
+            save_on_disk=True,
+        )
+
+        # Verify contents
+        assert len(loaded_experiment) == len(experiment)
+        assert list(loaded_experiment.keys()) == list(experiment.keys())
+        for qid in experiment.keys():
+            assert loaded_experiment[qid].query == experiment[qid].query
+            assert len(loaded_experiment[qid].retrieved_docs) == len(experiment[qid].retrieved_docs)
+            assert len(loaded_experiment[qid].answers) == len(experiment[qid].answers)
+
+    def test_get_qrels(self, tmp_path, experiment):
+        """Test getting relevance judgments"""
+        qrels = experiment.get_qrels()
+
+        # Basic structure check
+        assert len(qrels) == 2  # Two queries
+        assert "0" in qrels
+        assert "1" in qrels
+
+        # Test saving qrels
+        output_path = tmp_path / "test_qrels.txt"
+        experiment.get_qrels(output_path=str(output_path), output_format="trec")
+        assert os.path.exists(output_path)
+
+    def test_add_retrieval_evaluation(self, experiment, retrieval_evaluation, caplog):
+        """Test adding retrieval evaluation"""
+        # Add evaluation
+        experiment.add_evaluation(retrieval_evaluation, should_save=False)
+
+        # Verify evaluation was added
+        query = experiment["0"]
+        doc = query.retrieved_docs["0"]
+        assert doc.evaluation is not None
+        assert doc.evaluation.answer == 1
+        assert doc.evaluation.raw_answer == "Document is relevant. Score: 1.0"
+
+        # Test adding duplicate evaluation without force
+        added = experiment.add_evaluation(retrieval_evaluation, should_save=False)
+        assert "Document 0 in query 0 already has an evaluation." in caplog.text
+        assert not added
+
+        # Test adding duplicate evaluation with force
+        modified_eval = RetrievalEvaluatorResult(qid="0", did="0", raw_answer="Modified evaluation", answer=2)
+        experiment.add_evaluation(modified_eval, should_save=False, force=True)
+        assert doc.evaluation.answer == 2
+
+        # Test adding evaluation for non-existent query
+        invalid_eval = RetrievalEvaluatorResult(qid="999", did="0", raw_answer="Invalid", answer=1)
+        with pytest.raises(ValueError):
+            experiment.add_evaluation(invalid_eval)
+
+    def test_add_answer_evaluation(self, experiment, answer_evaluation, caplog):
+        """Test adding answer evaluation"""
+        # Add evaluation
+        experiment.add_evaluation(answer_evaluation, should_save=False)
+
+        # Verify evaluation was added
+        query = experiment["0"]
+        answer = query.answers["agent1"]
+        assert answer.evaluation is not None
+        assert answer.evaluation.answer == {"quality": 1.0, "relevance": 0.8}
+        assert answer.evaluation.raw_answer == "Answer is good. Scores: {'quality': 1.0, 'relevance': 0.8}"
+        assert not answer.evaluation.pairwise
+
+        # Test adding duplicate evaluation without force
+        added = experiment.add_evaluation(answer_evaluation, should_save=False)
+        assert not added
+        assert "Agent agent1 in query 0 already has an evaluation." in caplog.text
+
+        # Test adding duplicate evaluation with force
+        modified_eval = AnswerEvaluatorResult(
+            qid="0",
+            agent="agent1",
+            pairwise=False,
+            raw_answer="Modified evaluation",
+            answer={"quality": 2.0, "relevance": 1.0},
+        )
+        experiment.add_evaluation(modified_eval, should_save=False, force=True)
+        assert answer.evaluation.answer == {"quality": 2.0, "relevance": 1.0}
+
+    def test_add_pairwise_evaluation(self, experiment, pairwise_answer_evaluation):
+        """Test adding pairwise answer evaluation"""
+        # Add evaluation
+        experiment.add_evaluation(pairwise_answer_evaluation, should_save=False)
+
+        # Verify evaluation was added
+        query = experiment["0"]
+        assert len(query.pairwise_games) == 1
+        game = query.pairwise_games[0]
+        assert game.evaluation is not None
+        assert game.evaluation.answer == "A"
+        assert game.evaluation.raw_answer == "Answer [[A]] is better than [[B]]"
+        assert game.evaluation.pairwise
+        assert game.evaluation.agent_a == "agent1"
+        assert game.evaluation.agent_b == "agent2"
+
+    def test_add_elo_tournament(self, experiment, elo_tournament_result):
+        """Test adding Elo tournament results"""
+        # Add evaluation
+        experiment.add_evaluation(elo_tournament_result, should_save=False)
+
+        # Verify tournament was added
+        assert len(experiment.elo_tournaments) == 1
+        tournament = experiment.elo_tournaments[0]
+
+        # Check tournament results
+        assert tournament.agents == ["agent1", "agent2"]
+        assert tournament.scores == {"agent1": 1200, "agent2": 1000}
+        assert tournament.games_played == {"agent1": 1, "agent2": 1}
+        assert tournament.wins == {"agent1": 1, "agent2": 0}
+        assert tournament.loses == {"agent1": 0, "agent2": 1}
+        assert tournament.ties == {"agent1": 0, "agent2": 0}
+        assert tournament.total_games == 2
+        assert tournament.total_tournaments == 1
+
+    def test_save_evaluations(
+        self,
+        tmp_path,
+        base_experiment_config,
+        retrieval_evaluation,
+        answer_evaluation,
+        pairwise_answer_evaluation,
+        elo_tournament_result,
+    ):
+        """Test saving evaluations to disk"""
+        # Set up save paths
+        results_path = tmp_path / "test_results.jsonl"
+        base_experiment_config["evaluations_cache_path"] = str(results_path)
+        base_experiment_config["cache_evaluations"] = True
+        experiment = Experiment(**base_experiment_config)
+
+        # Add and save evaluations
+        experiment.add_evaluation(retrieval_evaluation, should_save=True)
+        experiment.add_evaluation(answer_evaluation, should_save=True)
+        experiment.add_evaluation(pairwise_answer_evaluation, should_save=True)
+        experiment.add_evaluation(elo_tournament_result, should_save=True)
+
+        # Verify results were saved
+        assert results_path.exists()
+
+        # Load and verify saved results
+        with open(results_path) as f:
+            lines = f.readlines()
+            assert len(lines) == 4
+
+            # Parse each line and verify content types
+            results = [json.loads(line) for line in lines]
+            result_types = [list(r.keys())[0] for r in results]
+            assert "retrieval" in result_types
+            assert "answer" in result_types
+            assert "elo_tournament" in result_types
+
+    def test_clear_evaluations(
+        self,
+        experiment,
+        retrieval_evaluation,
+        answer_evaluation,
+        pairwise_answer_evaluation,
+        elo_tournament_result,
+    ):
+        """Test clearing all evaluations"""
+        # Add evaluations
+        experiment.add_evaluation(retrieval_evaluation, should_save=False)
+        experiment.add_evaluation(answer_evaluation, should_save=False)
+        experiment.add_evaluation(pairwise_answer_evaluation, should_save=False)
+        experiment.add_evaluation(elo_tournament_result, should_save=False)
+
+        # Clear evaluations
+        experiment._Experiment__clear_all_evaluations()
+
+        # Verify evaluations were cleared
+        query = experiment["0"]
+        assert query.retrieved_docs["0"].evaluation is None
+        assert query.answers["agent1"].evaluation is None
+        assert len(experiment.elo_tournaments) == 0
+
+    def test_get_runs(self, tmp_path, experiment_with_retrieval_scores):
+        """Test getting retrieval runs"""
+        runs = experiment_with_retrieval_scores.get_runs()
+
+        # Basic structure check
+        assert len(runs) > 0  # At least one agent
+
+        # Test saving runs
+        output_dir = tmp_path / "test_runs"
+        experiment_with_retrieval_scores.get_runs(output_path=str(output_dir), output_format="trec")
+        assert os.path.exists(output_dir)
+        import shutil
+
+        shutil.rmtree(output_dir)  # Cleanup
+
+    def test_iteration(self, experiment):
+        """Test iteration over queries"""
+        queries = list(experiment)
+        assert len(queries) == 2
+        assert all(isinstance(q, Query) for q in queries)
+
+    def test_add_queries_from_csv(self, empty_experiment):
+        """Test adding queries from CSV"""
+        empty_experiment.add_queries_from_csv("tests/data/queries.csv")
+        assert len(empty_experiment) == 2
+        assert "0" in empty_experiment.keys()
+        assert "1" in empty_experiment.keys()
+
+    def test_add_documents_from_csv(self, empty_experiment):
+        """Test adding documents from CSV"""
+        empty_experiment.add_queries_from_csv("tests/data/queries.csv")
+        empty_experiment.add_documents_from_csv("tests/data/documents.csv")
+
+        assert len(empty_experiment["0"].retrieved_docs) == 2
+        assert len(empty_experiment["1"].retrieved_docs) == 2
+
+    def test_add_agent_answers_from_csv(self, empty_experiment):
+        """Test adding answers from CSV"""
+        empty_experiment.add_queries_from_csv("tests/data/queries.csv")
+        empty_experiment.add_agent_answers_from_csv("tests/data/answers.csv")
+
+        assert len(empty_experiment["0"].answers) == 2
+        assert len(empty_experiment["1"].answers) == 2
diff --git a/tests/unit/test_llm_providers.py b/tests/unit/test_llm_providers.py
index 26f9fe8..c82c3ef 100644
--- a/tests/unit/test_llm_providers.py
+++ b/tests/unit/test_llm_providers.py
@@ -1,73 +1,78 @@
+from __future__ import annotations
+
+import json
 from unittest.mock import AsyncMock
 
 import pytest
+from pydantic import BaseModel as PydanticBaseModel
 
-from ragelo import get_llm_provider
 from ragelo.llm_providers.openai_client import OpenAIProvider
+from ragelo.types.formats import AnswerFormat, LLMResponseType
+
+
+class AnswerModel(PydanticBaseModel):
+    keyA: str
+    keyB: str
 
 
 class TestOpenAIProvider:
+    @pytest.mark.parametrize(
+        "answer_format,response_schema,raw_answer,parsed_answer",
+        [
+            (AnswerFormat.TEXT, None, "fake response", "fake response"),
+            (
+                AnswerFormat.JSON,
+                {"keyA": "Value of key a", "KeyB": "Value of key b"},
+                '{"keyA": "valueA", "keyB": "valueB"}',
+                {"keyA": "valueA", "keyB": "valueB"},
+            ),
+            (
+                AnswerFormat.STRUCTURED,
+                AnswerModel,
+                '{"keyA": "valueA", "keyB": "valueB"}',
+                AnswerModel(keyA="valueA", keyB="valueB"),
+            ),
+        ],
+    )
     def test_response(
         self,
-        chat_completion_mock,
+        answer_format,
+        response_schema,
+        raw_answer,
+        parsed_answer,
         openai_client_mock,
         openai_client_config,
-        monkeypatch,
-        mocker,
-    ):
-        mocker.patch("openai.types.chat.chat_completion", new_callable=AsyncMock)
-
-        openai_client = OpenAIProvider(config=openai_client_config)
-        monkeypatch.setattr(
-            openai_client, "_OpenAIProvider__openai_client", openai_client_mock
-        )
-
-        prompt = "hello world"
-        prompts = [
-            {"role": "system", "content": "hello world"},
-            {"role": "user", "content": "hello openai"},
-        ]
-        result = openai_client(prompt)
-        assert result == "fake response"
-        result = openai_client(prompts)
-        assert result == "fake response"
-
-        call_args = chat_completion_mock.create.call_args_list
-        assert call_args[0][1]["model"] == "fake model"
-        assert call_args[0][1]["messages"] == [{"role": "system", "content": prompt}]
-        assert call_args[1][1]["messages"] == prompts
-
-    @pytest.mark.asyncio
-    async def test_parallel_call(
-        self,
         chat_completion_mock,
-        openai_client_mock,
-        openai_client_config,
         monkeypatch,
         mocker,
     ):
         mocker.patch("openai.types.chat.chat_completion", new_callable=AsyncMock)
+        mocker.patch("openai.types.chat.parsed_chat_completion", new_callable=AsyncMock)
 
         openai_client = OpenAIProvider(config=openai_client_config)
-        monkeypatch.setattr(
-            openai_client, "_OpenAIProvider__openai_client", openai_client_mock
-        )
-
+        monkeypatch.setattr(openai_client, "_OpenAIProvider__openai_client", openai_client_mock)
         prompt = "hello world"
         prompts = [
             {"role": "system", "content": "hello world"},
             {"role": "user", "content": "hello openai"},
         ]
-        result = await openai_client.call_async(prompt)
-        assert result == "fake response"
-        result = await openai_client.call_async(prompts)
-        assert result == "fake response"
+        result = openai_client(prompt, answer_format=answer_format, response_schema=response_schema)
+        assert isinstance(result, LLMResponseType)
+        assert result.raw_answer == raw_answer
+        if answer_format == AnswerFormat.STRUCTURED:
+            assert isinstance(result.parsed_answer, PydanticBaseModel)
+            assert AnswerModel(**result.parsed_answer.model_dump()) == parsed_answer
+        else:
+            assert result.parsed_answer == parsed_answer
 
-        call_args = chat_completion_mock.create.call_args_list
+        result = openai_client(prompts, answer_format=answer_format, response_schema=response_schema)
+        if answer_format == AnswerFormat.STRUCTURED:
+            call_args = chat_completion_mock.parse.call_args_list
+        else:
+            call_args = chat_completion_mock.create.call_args_list
         assert call_args[0][1]["model"] == "fake model"
-        assert call_args[0][1]["messages"] == [{"role": "system", "content": prompt}]
+        if answer_format == AnswerFormat.JSON:
+            assert call_args[0][1]["messages"][0]["content"].endswith(json.dumps(response_schema, indent=4))
+        else:
+            assert call_args[0][1]["messages"] == [{"role": "system", "content": prompt}]
         assert call_args[1][1]["messages"] == prompts
-
-    def test_get_by_name(self, openai_client_config, openai_client_mock):
-        provider = get_llm_provider("openai", api_key="fake key")
-        assert isinstance(provider, OpenAIProvider)
diff --git a/tests/unit/test_retrieval_evaluators.py b/tests/unit/test_retrieval_evaluators.py
index 68c0330..01e6835 100644
--- a/tests/unit/test_retrieval_evaluators.py
+++ b/tests/unit/test_retrieval_evaluators.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 import json
 from unittest.mock import AsyncMock
 
@@ -11,7 +13,8 @@
     ReasonerEvaluator,
 )
 from ragelo.types import Document, Query
-from ragelo.types.types import RetrievalEvaluatorResult
+from ragelo.types.formats import LLMResponseType
+from ragelo.types.results import RetrievalEvaluatorResult
 
 
 class RetrievalEvaluator(BaseRetrievalEvaluator):
@@ -20,71 +23,57 @@ def _build_message(self, query: Query, document: Document) -> str:
 
 
 class TestRetrievalEvaluator:
-    def test_evaluate_single_answer(
-        self,
-        llm_provider_json_mock,
-        base_retrieval_eval_config,
-        qs_with_docs,
-    ):
-        evaluator = RetrievalEvaluator.from_config(
-            config=base_retrieval_eval_config, llm_provider=llm_provider_json_mock
-        )
-        query = qs_with_docs[0]
+    def test_evaluate_single_answer(self, llm_provider_mock, experiment, base_retrieval_eval_config):
+        evaluator = RetrievalEvaluator.from_config(config=base_retrieval_eval_config, llm_provider=llm_provider_mock)
+        query = experiment["0"]
         doc = query.retrieved_docs["0"]
-        raw_answer, answer = evaluator.evaluate(query, doc)
-        assert raw_answer == 'Async LLM JSON response\n{"relevance": 1}'
-        assert answer == 1
+        result = evaluator.evaluate(query, doc)
+        assert result.raw_answer == '{"relevance": 1}'
+        assert result.answer == {"relevance": 1}
 
         expected_prompt = f"Query: {query.query}\nDocument: {doc.text}"
-        call_args = llm_provider_json_mock.async_call_mocker.call_args_list
+        call_args = llm_provider_mock.async_call_mocker.call_args_list
         assert call_args[0][0][0] == expected_prompt
 
-    def test_batch_eval(
+    def test_evaluate_experiment(
         self,
-        llm_provider_json_mock,
+        llm_provider_mock,
+        experiment,
         base_retrieval_eval_config,
-        qs_with_docs,
     ):
-        evaluator = RetrievalEvaluator.from_config(
-            config=base_retrieval_eval_config, llm_provider=llm_provider_json_mock
-        )
-        queries = evaluator.batch_evaluate(qs_with_docs)
-        evaluations = [d.evaluation for q in queries for d in q.retrieved_docs.values()]
+        evaluator = RetrievalEvaluator.from_config(config=base_retrieval_eval_config, llm_provider=llm_provider_mock)
+        evaluator.evaluate_experiment(experiment)
         doc_ids = ["0", "1", "2", "3"]
         qids = ["0", "0", "1", "1"]
-        for e, did, qid in zip(evaluations, doc_ids, qids):
-            assert isinstance(e, RetrievalEvaluatorResult)
-            assert isinstance(e.raw_answer, str)
-            assert e.raw_answer.startswith("Async")
-            assert isinstance(e.answer, int)
-            assert e.did == did
-            assert e.qid == qid
-
-    def test_evaluate_with_text(
-        self, llm_provider_json_mock, base_retrieval_eval_config
-    ):
-        evaluator = RetrievalEvaluator.from_config(
-            config=base_retrieval_eval_config, llm_provider=llm_provider_json_mock
-        )
-        raw_answer, processed_answer = evaluator.evaluate(
-            document="This is a query", query="This is a document"
-        )
-        assert raw_answer == 'Async LLM JSON response\n{"relevance": 1}'
-        assert processed_answer == 1
+        for query in experiment:
+            for doc in query.retrieved_docs_iter():
+                assert isinstance(doc.evaluation, RetrievalEvaluatorResult)
+                assert isinstance(doc.evaluation.raw_answer, str)
+                assert isinstance(doc.evaluation.answer, dict)
+                assert doc.did == doc_ids.pop(0)
+                assert doc.qid == qids.pop(0)
+
+    def test_evaluate_with_text(self, llm_provider_mock, base_retrieval_eval_config):
+        evaluator = RetrievalEvaluator.from_config(config=base_retrieval_eval_config, llm_provider=llm_provider_mock)
+        result = evaluator.evaluate(document="This is a document", query="This is a query")
+        call_args = llm_provider_mock.async_call_mocker.call_args_list
+        assert call_args[0][0][0] == "Query: This is a query\nDocument: This is a document"
+        assert isinstance(result.answer, dict)
+        assert result.answer["relevance"] == 1
 
     def test_rich_printing(
         self,
-        llm_provider_json_mock,
+        llm_provider_mock,
         base_eval_config,
-        qs_with_docs,
+        experiment,
         capsys,
     ):
         base_eval_config.rich_print = True
         evaluator = RetrievalEvaluator.from_config(
             config=base_eval_config,
-            llm_provider=llm_provider_json_mock,
+            llm_provider=llm_provider_mock,
         )
-        _ = evaluator.batch_evaluate(qs_with_docs)
+        evaluator.evaluate_experiment(experiment)
         captured = capsys.readouterr()
         assert "🔎" in captured.out
 
@@ -111,83 +100,72 @@ def test_get_by_name(self, llm_provider_mock, expert_retrieval_eval_config):
 
 
 class TestRDNAMEvaluator:
-    def test_process_single_answer(
-        self, llm_provider_mock_rdnam, rdnam_config, rdnam_queries
-    ):
-        evaluator = RDNAMEvaluator.from_config(
-            config=rdnam_config, llm_provider=llm_provider_mock_rdnam
-        )
-        query = rdnam_queries[0]
+    def test_process_single_answer(self, llm_provider_mock_rdnam, rdnam_config, experiment):
+        evaluator = RDNAMEvaluator.from_config(config=rdnam_config, llm_provider=llm_provider_mock_rdnam)
+        query = experiment["0"]
         doc = query.retrieved_docs["0"]
-        raw_answer, answer = evaluator.evaluate(query, doc)
-        assert raw_answer == '"M": 2, "T": 1, "O": 1}, {"M": 1, "T": 1, "O": 2}]'
-        assert answer == 1
+        result = evaluator.evaluate(query, doc)
+        assert isinstance(result.answer, dict)
+        assert len(result.answer) == 3
+        assert result.answer["intent_match"] == 1.5
+        assert result.answer["trustworthiness"] == 1.0
+        assert result.answer["overall"] == 1.5
+        assert result.raw_answer is not None
+        assert "annotator_1" in result.raw_answer
+        assert "annotator_2" in result.raw_answer
 
         call_args = llm_provider_mock_rdnam.async_call_mocker.call_args_list
-        assert call_args[0][0][0].startswith(
-            "You are a search quality rater evaluating"
-        )
+        assert call_args[0][0][0].startswith("You are a search quality rater evaluating")
 
 
 class TestReasonerEvaluator:
-    def test_process_single_answer(
-        self,
-        llm_provider_mock,
-        base_retrieval_eval_config,
-        qs_with_docs,
-    ):
+    def test_process_single_answer(self, llm_provider_reasoner_mock, base_retrieval_eval_config, experiment):
         evaluator = ReasonerEvaluator.from_config(
-            config=base_retrieval_eval_config,
-            llm_provider=llm_provider_mock,
+            config=base_retrieval_eval_config, llm_provider=llm_provider_reasoner_mock
         )
-        query = qs_with_docs[0]
+        query = experiment["0"]
         doc = query.retrieved_docs["0"]
-        raw_answer, answer = evaluator.evaluate(query, doc)
+        result = evaluator.evaluate(query, doc)
         formatted_prompt = evaluator.prompt.format(query=query.query, document=doc.text)
-        call_args = llm_provider_mock.async_call_mocker.call_args_list
+        call_args = llm_provider_reasoner_mock.async_call_mocker.call_args_list
 
-        assert raw_answer == answer
-        assert formatted_prompt in raw_answer
+        assert result.raw_answer == result.answer
         assert call_args[0][0][0] == formatted_prompt
 
 
 class TestCustomPromptEvaluator:
     def test_process_single_answer(
         self,
-        llm_provider_json_mock,
+        llm_provider_mock,
         custom_prompt_retrieval_eval_config,
-        qs_with_docs,
+        experiment,
     ):
-        query = qs_with_docs[0]
+        query = experiment["0"]
         doc = query.retrieved_docs["0"]
 
         evaluator = CustomPromptEvaluator.from_config(
             config=custom_prompt_retrieval_eval_config,
-            llm_provider=llm_provider_json_mock,
+            llm_provider=llm_provider_mock,
         )
         formatter = {
             custom_prompt_retrieval_eval_config.query_placeholder: query.query,
             custom_prompt_retrieval_eval_config.document_placeholder: doc.text,
         }
-        formatted_prompt = custom_prompt_retrieval_eval_config.prompt.format(
-            **formatter
-        )
+        formatted_prompt = custom_prompt_retrieval_eval_config.prompt.format(**formatter)
 
-        _, answer = evaluator.evaluate(query, doc)
-        call_args = llm_provider_json_mock.async_call_mocker.call_args_list
+        response = evaluator.evaluate(query, doc)
+        call_args = llm_provider_mock.async_call_mocker.call_args_list
 
-        assert answer == {"relevance": 1}
+        assert response.answer == {"relevance": 1}
         assert call_args[0][0][0] == formatted_prompt
 
-    def test_process_with_custom_fields(
-        self, llm_provider_json_mock, custom_prompt_retrieval_eval_config
-    ):
-        custom_prompt_retrieval_eval_config.prompt = "query: {query} doc: {document} q_metadata: {q_metadata} d_metadata: {d_metadata}"
-        custom_prompt_retrieval_eval_config.query_placeholder = "query"
-        custom_prompt_retrieval_eval_config.document_placeholder = "document"
+    def test_process_with_custom_fields(self, llm_provider_mock, custom_prompt_retrieval_eval_config):
+        custom_prompt_retrieval_eval_config.prompt = (
+            "query: {query} doc: {document} q_metadata: {q_metadata} d_metadata: {d_metadata}"
+        )
         evaluator = get_retrieval_evaluator(
             "custom_prompt",
-            llm_provider_json_mock,
+            llm_provider_mock,
             config=custom_prompt_retrieval_eval_config,
         )
         evaluator.evaluate(
@@ -197,7 +175,7 @@ def test_process_with_custom_fields(
             doc_metadata={"d_metadata": "d_1"},
         )
         assert (
-            llm_provider_json_mock.async_call_mocker.call_args_list[0][0][0]
+            llm_provider_mock.async_call_mocker.call_args_list[0][0][0]
             == "query: this is a query doc: this is a document q_metadata: q_1 d_metadata: d_1"
         )
 
@@ -207,26 +185,23 @@ def test_sys_prompt(self, llm_provider_mock, expert_retrieval_eval_config):
         evaluator = DomainExpertEvaluator.from_config(
             config=expert_retrieval_eval_config, llm_provider=llm_provider_mock
         )
-        assert expert_retrieval_eval_config.expert_in in evaluator.sys_prompt
-        assert expert_retrieval_eval_config.domain_short in evaluator.sys_prompt
-        assert (
-            f"You work for {expert_retrieval_eval_config.company}"
-            in evaluator.sys_prompt
-        )
+        assert expert_retrieval_eval_config.expert_in in evaluator.system_prompt
+        assert expert_retrieval_eval_config.domain_short in evaluator.system_prompt
+        assert f"You work for {expert_retrieval_eval_config.company}" in evaluator.system_prompt
 
     def test_process_single_answer(
         self,
         llm_provider_mock,
         expert_retrieval_eval_config,
-        qs_with_docs,
+        experiment,
     ):
         evaluator = DomainExpertEvaluator.from_config(
             config=expert_retrieval_eval_config,
             llm_provider=llm_provider_mock,
         )
-        query = qs_with_docs[0]
+        query = experiment["0"]
         doc = query.retrieved_docs["0"]
-        _, _ = evaluator.evaluate(query, doc)
+        _ = evaluator.evaluate(query, doc)
 
         assert llm_provider_mock.async_call_mocker.call_count == 2
         call_args = llm_provider_mock.async_call_mocker.call_args_list
@@ -240,58 +215,60 @@ def test_process_single_answer(
         assert prompts_score[2]["role"] == "assistant"
         assert prompts_score[1]["role"] == prompts_score[3]["role"] == "user"
         assert prompts_reasoning[0]["content"].startswith("You are a domain expert in")
-        assert prompts_score[1]["content"].endswith(
-            expert_retrieval_eval_config.extra_guidelines[0]
-        )
+        assert prompts_score[1]["content"].endswith(expert_retrieval_eval_config.extra_guidelines[0])
         assert prompts_score[3]["content"].startswith("Given the previous reasoning")
 
 
 class TestFewShotEvaluator:
     def test_process_single_answer(
         self,
-        llm_provider_json_mock,
+        llm_provider_mock,
         few_shot_retrieval_eval_config,
-        qs_with_docs,
+        experiment,
     ):
         evaluator = FewShotEvaluator.from_config(
             config=few_shot_retrieval_eval_config,
-            llm_provider=llm_provider_json_mock,
+            llm_provider=llm_provider_mock,
         )
-        query = qs_with_docs[0]
+        query = experiment["0"]
         doc = query.retrieved_docs["0"]
 
-        raw_answer, answer = evaluator.evaluate(query, doc)
-        assert raw_answer == 'Async LLM JSON response\n{"relevance": 1}'
-        assert answer == 1
-        call_args = llm_provider_json_mock.async_call_mocker.call_args_list
+        _ = evaluator.evaluate(query, doc)
+        call_args = llm_provider_mock.async_call_mocker.call_args_list
         call_messages = call_args[0][0][0]
         assert len(call_messages) == 6
         assert call_messages[0]["role"] == "system"
-        assert (
-            call_messages[1]["role"]
-            == call_messages[3]["role"]
-            == call_messages[5]["role"]
-            == "user"
-        )
+        assert call_messages[1]["role"] == call_messages[3]["role"] == call_messages[5]["role"] == "user"
         assert call_messages[2]["role"] == call_messages[4]["role"] == "assistant"
 
 
 class TestReadmeExamples:
     def test_rdnam_example(self, llm_provider_mock_rdnam):
-        llm_provider_mock_rdnam.async_call_mocker = AsyncMock(
-            side_effect=lambda _: '"O": 0\n}'
-        )
-        evaluator = get_retrieval_evaluator(
-            "RDNAM", llm_provider=llm_provider_mock_rdnam, write_output=False
-        )
-        raw_answer, processed_answer = evaluator.evaluate(
+        def side_effect(*args, **kwargs):
+            return LLMResponseType(raw_answer='{"overall": 1}', parsed_answer={"overall": 1})
+
+        llm_provider_mock_rdnam.async_call_mocker = AsyncMock(side_effect=side_effect)
+        evaluator = get_retrieval_evaluator("RDNAM", llm_provider=llm_provider_mock_rdnam, write_output=False)
+        result = evaluator.evaluate(
             query="What is the capital of France?",
             document="Lyon is the second largest city in France.",
         )
-        assert raw_answer == '"O": 0\n}'
-        assert processed_answer == 0
+        assert isinstance(result.answer, dict)
+        assert result.answer["overall"] == 1
+        assert result.answer == {"overall": 1}
+        assert result.raw_answer == '{"overall": 1}'
 
     def test_custom_prompt_evaluator_example(self, llm_provider_mock):
+        answer_dict = {
+            "relevance": 0,
+            "recency": 0,
+            "truthfulness": 0,
+            "reasoning": (
+                "The document is outdated and incorrect. Rio de Janeiro was the capital of "
+                "Brazil until 1960 when it was changed to Brasília."
+            ),
+        }
+
         prompt = """You are a helpful assistant for evaluating the relevance of a retrieved document to a user query.
 You should pay extra attention to how **recent** a document is. A document older than 5 years is considered outdated.
 
@@ -310,39 +287,36 @@ def test_custom_prompt_evaluator_example(self, llm_provider_mock):
 - "truthfulness": 0 if the document is false, 1 if it is true.
 - "reasoning": A short explanation of why you think the document is relevant or irrelevant.
 """
-        answer_dict = {
-            "relevance": 0,
-            "recency": 0,
-            "truthfulness": 0,
-            "reasoning": "The document is outdated and incorrect. Rio de Janeiro was the capital of Brazil until 1960 when it was changed to Brasília.",
+        response_schema = {
+            "relevance": "An integer, either 0 or 1. 0 if the document is irrelevant, 1 if it is relevant.",
+            "recency": "An integer, either 0 or 1. 0 if the document is outdated, 1 if it is recent.",
+            "truthfulness": "An integer, either 0 or 1. 0 if the document is false, 1 if it is true.",
+            "reasoning": "A short explanation of why you think the document is relevant or irrelevant.",
         }
-        mocked_llm_answer = json.dumps(answer_dict)
-        llm_provider_mock.async_call_mocker = AsyncMock(
-            side_effect=lambda _: mocked_llm_answer
-        )
+        response_format = "json"
+
         evaluator = get_retrieval_evaluator(
-            "custom_prompt",
-            llm_provider=llm_provider_mock,
-            prompt=prompt,
-            query_placeholder="q",
-            document_placeholder="d",
-            scoring_keys_retrieval_evaluator=[
-                "relevance",
-                "recency",
-                "truthfulness",
-                "reasoning",
-            ],
-            answer_format_retrieval_evaluator="multi_field_json",
-            write_output=False,
+            "custom_prompt",  # name of the retrieval evaluator
+            llm_provider=llm_provider_mock,  # Which LLM provider to use
+            prompt=prompt,  # your custom prompt
+            query_placeholder="q",  # the placeholder for the query in the prompt
+            document_placeholder="d",  # the placeholder for the document in the prompt
+            llm_answer_format=response_format,  # The format of the answer. Can be either TEXT, if you expect plain text to be returned, JSON if the answer should be in JSON format, or STRUCTURED, if you provide a Pydantic BaseModel as the response_schema.
+            llm_response_schema=response_schema,  # The response schema for the LLM. Required if the llm_answer_format is structured and recommended for JSON.
         )
 
-        raw_answer, answer = evaluator.evaluate(
-            query="What is the capital of Brazil?",
-            document="Rio de Janeiro is the capital of Brazil.",
-            query_metadata={"today_date": "08-04-2024"},
-            doc_metadata={"document_date": "04-03-1950"},
+        def side_effect(*args, **kwargs):
+            return LLMResponseType(raw_answer=json.dumps(answer_dict), parsed_answer=answer_dict)
+
+        llm_provider_mock.async_call_mocker = AsyncMock(side_effect=side_effect)
+
+        result = evaluator.evaluate(
+            query="What is the capital of Brazil?",  # The user query
+            document="Rio de Janeiro is the capital of Brazil.",  # The retrieved document
+            query_metadata={"today_date": "08-04-2024"},  # Some metadata for the query
+            doc_metadata={"document_date": "04-03-1950"},  # Some metadata for the document
         )
         call_args = llm_provider_mock.async_call_mocker.call_args_list
-        assert raw_answer == mocked_llm_answer
-        assert answer == answer_dict
+        assert result.raw_answer == json.dumps(answer_dict)
+        assert result.answer == answer_dict
         assert len(call_args) == 1