Repository: All-Hands-AI/OpenHands
Pull request #8177: specify condenser config for evals
Author: login: enyst
Created: 2025-04-30T13:56:58Z | Merged: 2025-05-21T20:08:57Z

- [ ] This change is worth documenting at https://docs.all-hands.dev/
- [ ] Include this change in the Release Notes. If checked, you **must** provide an **end-user friendly** description for your change below

**End-user friendly description of the problem this fixes or functionality that this introduces.**


---
**Give a summary of what the PR does, explaining any non-trivial design decisions.**

This PR proposes to add named condenser configurations as cli parameter in SWE-bench evals.
We add it like:
- define `[condenser.custom_condenser_for_eval]` with the desired options
- specify on command line `run_infer.sh llm.... condenser.custom_condenser_for_eval`
- alternatively, `--condenser-config` for the python script

---

When specified (llm-based):
```
openhands:INFO: shared.py:210 - Metadata: {"agent_class":"CodeActAgent" ...
...
"condenser_config":{"type":"llm","llm_config":{"model":"litellm_proxy/anthropic/claude-3-7-sonnet-20250219","api_key":"**********","base_url":"https://llm-proxy.eval.all-hands.dev/","api_version":null,"aws_access_key_id":null,"aws_secret_access_key":null,"aws_region_name":null,"openrouter_site_url":"https://docs.all-hands.dev/","openrouter_app_name":"OpenHands","num_retries":4,"retry_multiplier":2.0,"retry_min_wait":5,"retry_max_wait":30,"timeout":null,"max_message_chars":30000,"temperature":0.0,"top_p":1.0,"custom_llm_provider":null,"max_input_tokens":null,"max_output_tokens":null,"input_cost_per_token":null,"output_cost_per_token":null,"ollama_base_url":null,"drop_params":true,"modify_params":true,"disable_vision":null,"caching_prompt":true,"log_completions":true,"custom_tokenizer":null,"native_tool_calling":null,"reasoning_effort":"high","seed":null},"keep_first":4,"max_size":20,"max_event_length":10000}}
```

If not specified, it defaults to NoOp:
```
openhands:INFO: shared.py:190 - Metadata: {"agent_class":... 
...
"condenser_config":{"type":"noop"}}
```

---

To run this PR locally, use the following command:
```
docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:acb8ec0-nikolaik   --name openhands-app-acb8ec0   docker.all-hands.dev/all-hands-ai/openhands:acb8ec0
```


Diff:
diff --git a/evaluation/README.md b/evaluation/README.md
index cfaf1ba36c4d..91d608d8f531 100644
--- a/evaluation/README.md
+++ b/evaluation/README.md
@@ -42,6 +42,37 @@ api_key = "XXX"
 temperature = 0.0
 ```
 
+### Configuring Condensers for Evaluation
+
+For benchmarks that support condenser configuration (like SWE-Bench), you can define multiple condenser configurations in your `config.toml` file. A condenser is responsible for managing conversation history to maintain context while staying within token limits - you can learn more about how it works [here](https://www.all-hands.dev/blog/openhands-context-condensensation-for-more-efficient-ai-agents):
+
+```toml
+# LLM-based summarizing condenser for evaluation
+[condenser.summarizer_for_eval]
+type = "llm"
+llm_config = "haiku"  # Reference to an LLM config to use for summarization
+keep_first = 2        # Number of initial events to always keep
+max_size = 100        # Maximum size of history before triggering summarization
+
+# Recent events condenser for evaluation
+[condenser.recent_for_eval]
+type = "recent"
+keep_first = 2        # Number of initial events to always keep
+max_events = 50       # Maximum number of events to keep in history
+```
+
+You can then specify which condenser configuration to use when running evaluation scripts, for example:
+
+```bash
+EVAL_CONDENSER=summarizer_for_eval \
+./evaluation/benchmarks/swe_bench/scripts/run_infer.sh llm.eval_gpt4_1106_preview HEAD CodeActAgent 500 100 1 princeton-nlp/SWE-bench_Verified test
+```
+
+The name is up to you, but should match a name defined in your `config.toml` file. The last argument in the command specifies the condenser configuration to use. In this case, `summarizer_for_eval` is used, which refers to the LLM-based summarizing condenser as defined above.
+
+If no condenser configuration is specified, the 'noop' condenser will be used by default, which keeps the full conversation history.
+```
+
 For other configurations specific to evaluation, such as `save_trajectory_path`, these are typically set in the `get_config` function of the respective `run_infer.py` file for each benchmark.
 
 ## Supported Benchmarks
diff --git a/evaluation/benchmarks/swe_bench/README.md b/evaluation/benchmarks/swe_bench/README.md
index a3e354b5c454..239df12e9690 100644
--- a/evaluation/benchmarks/swe_bench/README.md
+++ b/evaluation/benchmarks/swe_bench/README.md
@@ -45,7 +45,7 @@ For example, for instance ID `django_django-11011`, it will try to pull our pre-
 This image will be used create an OpenHands runtime image where the agent will operate on.
 
 ```bash
-./evaluation/benchmarks/swe_bench/scripts/run_infer.sh [model_config] [git-version] [agent] [eval_limit] [max_iter] [num_workers] [dataset] [dataset_split]
+./evaluation/benchmarks/swe_bench/scripts/run_infer.sh [model_config] [git-version] [agent] [eval_limit] [max_iter] [num_workers] [dataset] [dataset_split] [n_runs] [mode]
 
 # Example
 ./evaluation/benchmarks/swe_bench/scripts/run_infer.sh llm.eval_gpt4_1106_preview HEAD CodeActAgent 500 100 1 princeton-nlp/SWE-bench_Verified test
@@ -69,13 +69,20 @@ default, it is set to 1.
 - `dataset`, a huggingface dataset name. e.g. `princeton-nlp/SWE-bench`, `princeton-nlp/SWE-bench_Lite`, `princeton-nlp/SWE-bench_Verified`, or `princeton-nlp/SWE-bench_Multimodal`, specifies which dataset to evaluate on.
 - `dataset_split`, split for the huggingface dataset. e.g., `test`, `dev`. Default to `test`.
 
+- `n_runs`, e.g. `3`, is the number of times to run the evaluation. Default is 1.
+- `mode`, e.g. `swt`, `swt-ci`, or `swe`, specifies the evaluation mode. Default is `swe`.
+
 > [!CAUTION]
 > Setting `num_workers` larger than 1 is not officially tested, YMMV.
 
-There is also one optional environment variable you can set.
+There are also optional environment variables you can set:
 
 ```bash
-export USE_HINT_TEXT=true # if you want to use hint text in the evaluation. Default to false. Ignore this if you are not sure.
+# Use hint text in the evaluation (default: false)
+export USE_HINT_TEXT=true # Ignore this if you are not sure.
+
+# Specify a condenser configuration for memory management (default: NoOpCondenser)
+export EVAL_CONDENSER=summarizer_for_eval # Name of the condenser config group in config.toml
 ```
 
 Let's say you'd like to run 10 instances using `llm.eval_gpt4_1106_preview` and CodeActAgent,
diff --git a/evaluation/benchmarks/swe_bench/run_infer.py b/evaluation/benchmarks/swe_bench/run_infer.py
index 8e23125945f6..c7846055f957 100644
--- a/evaluation/benchmarks/swe_bench/run_infer.py
+++ b/evaluation/benchmarks/swe_bench/run_infer.py
@@ -44,6 +44,8 @@
     get_llm_config_arg,
     get_parser,
 )
+from openhands.core.config.utils import get_condenser_config_arg
+from openhands.core.config.condenser_config import NoOpCondenserConfig
 from openhands.core.logger import openhands_logger as logger
 from openhands.core.main import create_runtime, run_controller
 from openhands.critic import AgentFinishedCritic
@@ -756,6 +758,7 @@ def filter_dataset(dataset: pd.DataFrame, filter_column: str) -> pd.DataFrame:
         choices=['swe', 'swt', 'swt-ci'],
         help="mode to run the evaluation, either 'swe', 'swt', or 'swt-ci'",
     )
+
     args, _ = parser.parse_known_args()
 
     # NOTE: It is preferable to load datasets from huggingface datasets and perform post-processing
@@ -792,6 +795,19 @@ def filter_dataset(dataset: pd.DataFrame, filter_column: str) -> pd.DataFrame:
     if llm_config is None:
         raise ValueError(f'Could not find LLM config: --llm_config {args.llm_config}')
 
+    # Get condenser config from environment variable
+    condenser_name = os.environ.get('EVAL_CONDENSER')
+    if condenser_name:
+        condenser_config = get_condenser_config_arg(condenser_name)
+        if condenser_config is None:
+            raise ValueError(
+                f'Could not find Condenser config: EVAL_CONDENSER={condenser_name}'
+            )
+    else:
+        # If no specific condenser config is provided via env var, default to NoOpCondenser
+        condenser_config = NoOpCondenserConfig()
+        logger.debug('No Condenser config provided via EVAL_CONDENSER, using NoOpCondenser.')
+
     details = {'mode': args.mode}
     _agent_cls = openhands.agenthub.Agent.get_cls(args.agent_cls)
 
@@ -806,6 +822,7 @@ def filter_dataset(dataset: pd.DataFrame, filter_column: str) -> pd.DataFrame:
         args.eval_note,
         args.eval_output_dir,
         details=details,
+        condenser_config=condenser_config,
     )
 
     output_file = os.path.join(metadata.eval_output_dir, 'output.jsonl')
diff --git a/evaluation/benchmarks/swe_bench/scripts/run_infer.sh b/evaluation/benchmarks/swe_bench/scripts/run_infer.sh
index ea86976fc368..c89ad72c4888 100755
--- a/evaluation/benchmarks/swe_bench/scripts/run_infer.sh
+++ b/evaluation/benchmarks/swe_bench/scripts/run_infer.sh
@@ -14,6 +14,7 @@ SPLIT=$8
 N_RUNS=$9
 MODE=${10}
 
+
 if [ -z "$NUM_WORKERS" ]; then
   NUM_WORKERS=1
   echo "Number of workers not specified, use default $NUM_WORKERS"
@@ -51,6 +52,12 @@ if [ -z "$MODE" ]; then
   echo "MODE not specified, use default $MODE"
 fi
 
+if [ -n "$EVAL_CONDENSER" ]; then
+  echo "Using Condenser Config: $EVAL_CONDENSER"
+else
+  echo "No Condenser Config provided via EVAL_CONDENSER, use default (NoOpCondenser)."
+fi
+
 export RUN_WITH_BROWSING=$RUN_WITH_BROWSING
 echo "RUN_WITH_BROWSING: $RUN_WITH_BROWSING"
 
@@ -65,6 +72,7 @@ echo "MAX_ITER: $MAX_ITER"
 echo "NUM_WORKERS: $NUM_WORKERS"
 echo "COMMIT_HASH: $COMMIT_HASH"
 echo "MODE: $MODE"
+echo "EVAL_CONDENSER: $EVAL_CONDENSER"
 
 # Default to NOT use Hint
 if [ -z "$USE_HINT_TEXT" ]; then
@@ -88,6 +96,10 @@ fi
 if [ "$MODE" != "swe" ]; then
   EVAL_NOTE="${EVAL_NOTE}-${MODE}"
 fi
+# Add condenser config to eval note if provided
+if [ -n "$EVAL_CONDENSER" ]; then
+  EVAL_NOTE="${EVAL_NOTE}-${EVAL_CONDENSER}"
+fi
 
 function run_eval() {
   local eval_note="${1}"
@@ -101,6 +113,8 @@ function run_eval() {
     --split $SPLIT \
     --mode $MODE"
 
+
+
   if [ -n "$EVAL_LIMIT" ]; then
     echo "EVAL_LIMIT: $EVAL_LIMIT"
     COMMAND="$COMMAND --eval-n-limit $EVAL_LIMIT"
diff --git a/openhands/core/config/utils.py b/openhands/core/config/utils.py
index 27393acafaa8..e9f3b357397b 100644
--- a/openhands/core/config/utils.py
+++ b/openhands/core/config/utils.py
@@ -16,7 +16,11 @@
 from openhands.core import logger
 from openhands.core.config.agent_config import AgentConfig
 from openhands.core.config.app_config import AppConfig
-from openhands.core.config.condenser_config import condenser_config_from_toml_section
+from openhands.core.config.condenser_config import (
+    CondenserConfig,
+    condenser_config_from_toml_section,
+    create_condenser_config,
+)
 from openhands.core.config.config_utils import (
     OH_DEFAULT_AGENT,
     OH_MAX_ITERATIONS,
@@ -491,6 +495,118 @@ def get_llm_config_arg(
     return None
 
 
+def get_condenser_config_arg(
+    condenser_config_arg: str, toml_file: str = 'config.toml'
+) -> CondenserConfig | None:
+    """Get a group of condenser settings from the config file by name.
+
+    A group in config.toml can look like this:
+
+    ```
+    [condenser.my_summarizer]
+    type = 'llm'
+    llm_config = 'gpt-4o' # References [llm.gpt-4o]
+    max_size = 50
+    ...
+    ```
+
+    The user-defined group name, like "my_summarizer", is the argument to this function.
+    The function will load the CondenserConfig object with the settings of this group,
+    from the config file.
+
+    Note that the group must be under the "condenser" group, or in other words,
+    the group name must start with "condenser.".
+
+    Args:
+        condenser_config_arg: The group of condenser settings to get from the config.toml file.
+        toml_file: Path to the configuration file to read from. Defaults to 'config.toml'.
+
+    Returns:
+        CondenserConfig: The CondenserConfig object with the settings from the config file, or None if not found/error.
+    """
+    # keep only the name, just in case
+    condenser_config_arg = condenser_config_arg.strip('[]')
+
+    # truncate the prefix, just in case
+    if condenser_config_arg.startswith('condenser.'):
+        condenser_config_arg = condenser_config_arg[10:]
+
+    logger.openhands_logger.debug(
+        f'Loading condenser config [{condenser_config_arg}] from {toml_file}'
+    )
+
+    # load the toml file
+    try:
+        with open(toml_file, 'r', encoding='utf-8') as toml_contents:
+            toml_config = toml.load(toml_contents)
+    except FileNotFoundError as e:
+        logger.openhands_logger.error(f'Config file not found: {toml_file}. Error: {e}')
+        return None
+    except toml.TomlDecodeError as e:
+        logger.openhands_logger.error(
+            f'Cannot parse condenser group [{condenser_config_arg}] from {toml_file}. Exception: {e}'
+        )
+        return None
+
+    # Check if the condenser section and the specific config exist
+    if (
+        'condenser' not in toml_config
+        or condenser_config_arg not in toml_config['condenser']
+    ):
+        logger.openhands_logger.error(
+            f'Condenser config section [condenser.{condenser_config_arg}] not found in {toml_file}'
+        )
+        return None
+
+    condenser_data = toml_config['condenser'][
+        condenser_config_arg
+    ].copy()  # Use copy to modify
+
+    # Determine the type and handle potential LLM dependency
+    condenser_type = condenser_data.get('type')
+    if not condenser_type:
+        logger.openhands_logger.error(
+            f'Missing "type" field in [condenser.{condenser_config_arg}] section of {toml_file}'
+        )
+        return None
+
+    # Handle LLM config reference if needed, using get_llm_config_arg
+    if (
+        condenser_type in ('llm', 'llm_attention', 'structured')
+        and 'llm_config' in condenser_data
+        and isinstance(condenser_data['llm_config'], str)
+    ):
+        llm_config_name = condenser_data['llm_config']
+        logger.openhands_logger.debug(
+            f'Condenser [{condenser_config_arg}] requires LLM config [{llm_config_name}]. Loading it...'
+        )
+        # Use the existing function to load the specific LLM config
+        referenced_llm_config = get_llm_config_arg(llm_config_name, toml_file=toml_file)
+
+        if referenced_llm_config:
+            # Replace the string reference with the actual LLMConfig object
+            condenser_data['llm_config'] = referenced_llm_config
+        else:
+            # get_llm_config_arg already logs the error if not found
+            logger.openhands_logger.error(
+                f"Failed to load required LLM config '{llm_config_name}' for condenser '{condenser_config_arg}'."
+            )
+            return None
+
+    # Create the condenser config instance
+    try:
+        config = create_condenser_config(condenser_type, condenser_data)
+        logger.openhands_logger.info(
+            f'Successfully loaded condenser config [{condenser_config_arg}] from {toml_file}'
+        )
+        return config
+    except (ValidationError, ValueError) as e:
+        logger.openhands_logger.error(
+            f'Invalid condenser configuration for [{condenser_config_arg}]: {e}.'
+        )
+        return None
+
+
 # Command line arguments
 def get_parser() -> argparse.ArgumentParser:
     """Get the argument parser."""


Reviews:
user: login: xingyaoww
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-01T01:18:57Z
user: login: csmith49
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-01T01:26:19Z
user: login: enyst
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-01T20:02:30Z
user: login: enyst
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-01T20:03:11Z
user: login: csmith49
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-01T20:17:18Z
user: login: enyst
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-01T20:21:47Z
user: login: openhands-agent
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T20:29:40Z
user: login: enyst
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-01T20:41:06Z
user: login: openhands-agent
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T20:46:53Z
user: login: xingyaoww
body: LGTM - thanks!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-03T02:11:53Z
user: login: enyst
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-03T23:08:10Z
user: login: openhands-agent
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-03T23:13:36Z

Review comments:
diff_hunk: @@ -68,6 +68,9 @@ default, it is set to 60.
 default, it is set to 1.
 - `dataset`, a huggingface dataset name. e.g. `princeton-nlp/SWE-bench`, `princeton-nlp/SWE-bench_Lite`, `princeton-nlp/SWE-bench_Verified`, or `princeton-nlp/SWE-bench_Multimodal`, specifies which dataset to evaluate on.
 - `dataset_split`, split for the huggingface dataset. e.g., `test`, `dev`. Default to `test`.
+- `condenser_config`, e.g. `summarizer_for_eval`, is the name of the condenser configuration group to use for memory management, as defined in your `config.toml`. If not specified, it defaults to `NoOpCondenser`.
path: evaluation/benchmarks/swe_bench/README.md
user: login: xingyaoww
body: since this is probably going to be less frequently used, can we make it an environment variable?
So we don't keep adding args to run_infer.sh which already have a lot of arguments 😢 
created_at: 2025-05-01T01:18:56Z
author_association: COLLABORATOR
diff_hunk: @@ -780,6 +788,18 @@ def filter_dataset(dataset: pd.DataFrame, filter_column: str) -> pd.DataFrame:
     if llm_config is None:
         raise ValueError(f'Could not find LLM config: --llm_config {args.llm_config}')
 
+    condenser_config = None
+    if args.condenser_config:
+        condenser_config = get_condenser_config_arg(args.condenser_config)
+        if condenser_config is None:
+            raise ValueError(
+                f'Could not find Condenser config: --condenser-config {args.condenser_config}'
+            )
+    else:
+        # If no specific condenser config is provided via args, default to NoOpCondenser
+        condenser_config = NoOpCondenserConfig()
+        logger.warning('No Condenser config provided via --condenser-config, using NoOpCondenser.')
path: evaluation/benchmarks/swe_bench/run_infer.py
user: login: csmith49
body: `warning` feels a bit strong here -- I'd expect no condenser provided is still the normal use case for these eval scripts. Maybe `info` or `debug`?
created_at: 2025-05-01T01:26:19Z
author_association: COLLABORATOR
diff_hunk: @@ -780,6 +788,18 @@ def filter_dataset(dataset: pd.DataFrame, filter_column: str) -> pd.DataFrame:
     if llm_config is None:
         raise ValueError(f'Could not find LLM config: --llm_config {args.llm_config}')
 
+    condenser_config = None
+    if args.condenser_config:
+        condenser_config = get_condenser_config_arg(args.condenser_config)
+        if condenser_config is None:
+            raise ValueError(
+                f'Could not find Condenser config: --condenser-config {args.condenser_config}'
+            )
+    else:
+        # If no specific condenser config is provided via args, default to NoOpCondenser
+        condenser_config = NoOpCondenserConfig()
+        logger.warning('No Condenser config provided via --condenser-config, using NoOpCondenser.')
path: evaluation/benchmarks/swe_bench/run_infer.py
user: login: enyst
body: ```suggestion
        logger.debug('No Condenser config provided via --condenser-config, using NoOpCondenser.')
```
created_at: 2025-05-01T20:02:30Z
author_association: COLLABORATOR
diff_hunk: @@ -780,6 +788,18 @@ def filter_dataset(dataset: pd.DataFrame, filter_column: str) -> pd.DataFrame:
     if llm_config is None:
         raise ValueError(f'Could not find LLM config: --llm_config {args.llm_config}')
 
+    condenser_config = None
+    if args.condenser_config:
+        condenser_config = get_condenser_config_arg(args.condenser_config)
+        if condenser_config is None:
+            raise ValueError(
+                f'Could not find Condenser config: --condenser-config {args.condenser_config}'
+            )
+    else:
+        # If no specific condenser config is provided via args, default to NoOpCondenser
+        condenser_config = NoOpCondenserConfig()
+        logger.warning('No Condenser config provided via --condenser-config, using NoOpCondenser.')
path: evaluation/benchmarks/swe_bench/run_infer.py
user: login: enyst
body: Done
created_at: 2025-05-01T20:03:11Z
author_association: COLLABORATOR
diff_hunk: @@ -68,6 +68,9 @@ default, it is set to 60.
 default, it is set to 1.
 - `dataset`, a huggingface dataset name. e.g. `princeton-nlp/SWE-bench`, `princeton-nlp/SWE-bench_Lite`, `princeton-nlp/SWE-bench_Verified`, or `princeton-nlp/SWE-bench_Multimodal`, specifies which dataset to evaluate on.
 - `dataset_split`, split for the huggingface dataset. e.g., `test`, `dev`. Default to `test`.
+- `condenser_config`, e.g. `summarizer_for_eval`, is the name of the condenser configuration group to use for memory management, as defined in your `config.toml`. If not specified, it defaults to `NoOpCondenser`.
path: evaluation/benchmarks/swe_bench/README.md
user: login: enyst
body: Sure! 😅

@openhands-agent-exp Understand this PR in depth, and fix xingyaoww's feedback.

Name the new env var `EVAL_CONDENSER`. Make sure to read the diff of this PR and fix the necessary files to:
- remove the command line argument, including from the .py file, and
- read the env var, and use it instead, only in run_infer.sh and in __main__ of the run_infer.py.

Adapt accordingly the right places in the documentation files changed by this PR.

< IMPORTANT >

We replace the command line arg with an env var, to specify the user-defined condenser's name to choose, the user still needs to define that name in the toml file.

Do NOT modify other files than those already modified in this PR. Really.

You don't need to add tests, I took care of that in CI.

< /IMPORTANT >
created_at: 2025-05-01T20:21:47Z
author_association: COLLABORATOR
diff_hunk: @@ -68,6 +68,9 @@ default, it is set to 60.
 default, it is set to 1.
 - `dataset`, a huggingface dataset name. e.g. `princeton-nlp/SWE-bench`, `princeton-nlp/SWE-bench_Lite`, `princeton-nlp/SWE-bench_Verified`, or `princeton-nlp/SWE-bench_Multimodal`, specifies which dataset to evaluate on.
 - `dataset_split`, split for the huggingface dataset. e.g., `test`, `dev`. Default to `test`.
+- `condenser_config`, e.g. `summarizer_for_eval`, is the name of the condenser configuration group to use for memory management, as defined in your `config.toml`. If not specified, it defaults to `NoOpCondenser`.
path: evaluation/benchmarks/swe_bench/README.md
user: login: openhands-agent
body: Openhands fix success summary


The feedback has been successfully incorporated through a complete transition from command-line argument to environment variable for the condenser configuration. Specifically:

1. The command-line argument `--condenser-config` was completely removed from both the argument parsing in `run_infer.py` and the parameter list in `run_infer.sh`

2. A new environment variable `EVAL_CONDENSER` was introduced to replace it, maintaining the same functionality but reducing the number of command-line arguments

3. The documentation in README.md was properly updated to:
   - Remove the condenser_config from the command-line arguments section
   - Add clear documentation about the new `EVAL_CONDENSER` environment variable
   - Maintain the important note that the condenser name still needs to be defined in the toml file

4. The functionality remains identical - the only change is how the configuration is passed (environment variable instead of command-line argument), which directly addresses the feedback about reducing the number of arguments to `run_infer.sh`

The changes are focused only on the files that were already modified in the PR, and the core functionality of selecting a condenser configuration is preserved while making the interface cleaner.
created_at: 2025-05-01T20:29:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,6 +42,36 @@ api_key = "XXX"
 temperature = 0.0
 ```
 
+### Configuring Condensers for Evaluation
+
+For benchmarks that support condenser configuration (like SWE-Bench), you can define multiple condenser configurations in your `config.toml` file:
+
+```toml
+# LLM-based summarizing condenser for evaluation
+[condenser.summarizer_for_eval]
+type = "llm"
+llm_config = "haiku"  # Reference to an LLM config to use for summarization
+keep_first = 2        # Number of initial events to always keep
+max_size = 100        # Maximum size of history before triggering summarization
+
+# Recent events condenser for evaluation
+[condenser.recent_for_eval]
+type = "recent"
+keep_first = 2        # Number of initial events to always keep
+max_events = 50       # Maximum number of events to keep in history
+```
+
+You can then specify which condenser configuration to use when running evaluation scripts, for example:
+
+```bash
+./evaluation/benchmarks/swe_bench/scripts/run_infer.sh llm.eval_gpt4_1106_preview HEAD CodeActAgent 500 100 1 princeton-nlp/SWE-bench_Verified test summarizer_for_eval
path: evaluation/README.md
user: login: enyst
body: @openhands-agent-exp Fix this line in the README to add at its beginning the new EVAL_CONDENSER=summarizer_for_eval instead of a cli arg. Split on a new line too, with \

Make sure you read the whole diff to understand what the PR is doing and why you need to fix this detail.
created_at: 2025-05-01T20:41:06Z
author_association: COLLABORATOR
diff_hunk: @@ -42,6 +42,36 @@ api_key = "XXX"
 temperature = 0.0
 ```
 
+### Configuring Condensers for Evaluation
+
+For benchmarks that support condenser configuration (like SWE-Bench), you can define multiple condenser configurations in your `config.toml` file:
+
+```toml
+# LLM-based summarizing condenser for evaluation
+[condenser.summarizer_for_eval]
+type = "llm"
+llm_config = "haiku"  # Reference to an LLM config to use for summarization
+keep_first = 2        # Number of initial events to always keep
+max_size = 100        # Maximum size of history before triggering summarization
+
+# Recent events condenser for evaluation
+[condenser.recent_for_eval]
+type = "recent"
+keep_first = 2        # Number of initial events to always keep
+max_events = 50       # Maximum number of events to keep in history
+```
+
+You can then specify which condenser configuration to use when running evaluation scripts, for example:
+
+```bash
+./evaluation/benchmarks/swe_bench/scripts/run_infer.sh llm.eval_gpt4_1106_preview HEAD CodeActAgent 500 100 1 princeton-nlp/SWE-bench_Verified test summarizer_for_eval
path: evaluation/README.md
user: login: openhands-agent
body: Openhands fix success summary


The changes successfully address the feedback by:

1. Moving the condenser configuration from being a CLI argument to an environment variable by adding `EVAL_CONDENSER=summarizer_for_eval` at the start of the command
2. Breaking the long command into two lines using a backslash (`\`) for better readability
3. Removing `summarizer_for_eval` from the end of the command since it's now being set via the environment variable

The changes maintain the same functionality while improving the command structure. The environment variable approach is consistent with the PR's intent (as mentioned in the feedback about understanding the whole diff), and the line splitting improves readability. The example command in the README now correctly demonstrates the new way to specify the condenser configuration.
created_at: 2025-05-01T20:46:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,6 +42,37 @@ api_key = "XXX"
 temperature = 0.0
 ```
 
+### Configuring Condensers for Evaluation
+
+For benchmarks that support condenser configuration (like SWE-Bench), you can define multiple condenser configurations in your `config.toml` file:
path: evaluation/README.md
user: login: xingyaoww
body: I'd suggest maybe we include a link to https://www.all-hands.dev/blog/openhands-context-condensensation-for-more-efficient-ai-agents to briefly explain what is this "condenser doing" :)
created_at: 2025-05-03T02:11:46Z
author_association: COLLABORATOR
diff_hunk: @@ -42,6 +42,37 @@ api_key = "XXX"
 temperature = 0.0
 ```
 
+### Configuring Condensers for Evaluation
+
+For benchmarks that support condenser configuration (like SWE-Bench), you can define multiple condenser configurations in your `config.toml` file:
path: evaluation/README.md
user: login: enyst
body: @openhands-agent-exp Fix xingyaoww 's feedback.

created_at: 2025-05-03T23:08:10Z
author_association: COLLABORATOR
diff_hunk: @@ -42,6 +42,37 @@ api_key = "XXX"
 temperature = 0.0
 ```
 
+### Configuring Condensers for Evaluation
+
+For benchmarks that support condenser configuration (like SWE-Bench), you can define multiple condenser configurations in your `config.toml` file:
path: evaluation/README.md
user: login: openhands-agent
body: Openhands fix success summary


The changes directly address the feedback by adding both the requested link (https://www.all-hands.dev/blog/openhands-context-condensensation-for-more-efficient-ai-agents) and a brief explanation of what a condenser does. Specifically, the changes:

1. Added the link to the blog post in a contextually appropriate location where condensers are being introduced
2. Included a clear explanation that condensers "manage conversation history to maintain context while staying within token limits"
3. Integrated these additions naturally into the existing documentation flow

The changes provide readers with both immediate context about condensers' purpose and a way to learn more detailed information through the blog post link, which fully satisfies the original feedback request to explain what the condenser is doing.
created_at: 2025-05-03T23:13:36Z
author_association: CONTRIBUTOR

PR comments:
user: login: openhands-agent
created_at: 2025-05-01T20:22:03Z
body: [OpenHands](https://github.com/All-Hands-AI/OpenHands) started fixing the pr! You can monitor the progress [here](https://github.com/All-Hands-AI/OpenHands/actions/runs/14782839092).
author_association: CONTRIBUTOR
user: login: openhands-agent
created_at: 2025-05-01T20:29:39Z
body: Overview of Changes:
- Successfully transitioned from command-line argument (`--condenser-config`) to environment variable (`EVAL_CONDENSER`)
- Updated all relevant files: `run_infer.py`, `run_infer.sh`, and README.md
- Maintained core functionality while simplifying the interface
- Documentation properly reflects the new implementation

Status: ✅ All issues appear to be successfully resolved
- The change addresses the original feedback about reducing command-line arguments
- The implementation is consistent across all affected files
- Documentation has been appropriately updated
- Core functionality has been preserved

No remaining issues are apparent from the provided information.
author_association: CONTRIBUTOR
user: login: openhands-agent
created_at: 2025-05-01T20:41:20Z
body: [OpenHands](https://github.com/All-Hands-AI/OpenHands) started fixing the pr! You can monitor the progress [here](https://github.com/All-Hands-AI/OpenHands/actions/runs/14783149901).
author_association: CONTRIBUTOR
user: login: openhands-agent
created_at: 2025-05-01T20:46:52Z
body: OVERVIEW:
All issues have been successfully resolved. The changes effectively:
- Moved condenser configuration to an environment variable
- Improved command readability through line splitting
- Removed redundant parameter
- Maintained intended functionality
- Aligned with the PR's overall goals

No remaining issues to address.
author_association: CONTRIBUTOR
user: login: openhands-agent
created_at: 2025-05-03T23:08:22Z
body: [OpenHands](https://github.com/All-Hands-AI/OpenHands) started fixing the pr! You can monitor the progress [here](https://github.com/All-Hands-AI/OpenHands/actions/runs/14815402428).
author_association: CONTRIBUTOR
user: login: openhands-agent
created_at: 2025-05-03T23:13:35Z
body: OVERVIEW:
The changes successfully address all identified issues by:
- Adding the requested blog post link
- Including a clear definition of condensers' functionality
- Integrating the new content smoothly into existing documentation

STATUS: ✓ All issues resolved, no remaining concerns.

The implementation appears complete and effectively addresses the original feedback while maintaining documentation quality and readability.
author_association: CONTRIBUTOR