Repository: EleutherAI/lm-evaluation-harness
Pull request: `higher_is_better` tickers in output table
# What does this PR do?

This PR adds ticker symbols (↑/↓) for each metric in the output table.

Addresses #1153 

# How does it work?

On the task level, for each metric I take the specified `higher_is_better` value, and append it to the output table.

On the group level, I first check whether the metrics for all tasks have consistent `higher_is_better` values. 
1. If yes (99% of scenarios), then the group also inherits the consistent value. 
2. If not, the symbol for that particular metric within the group is left out and a warning is printed to the console. This handles the very rare cases when the same metric name could have different `higher_is_better` values for different tasks within a group. 

# Example outputs
Suppose we have two tasks `dummy1` and `dummy2` from the group `dummygroup`.
1. In the case the `higher_is_better` values are consistent, i.e.:

`dummy1.yaml`:
```yaml
group:
  - dummygroup
task: dummytask1
dataset_path: hellaswag
dataset_name: null
output_type: multiple_choice
training_split: train
validation_split: validation
test_split: null
process_docs: !function utils.process_docs
doc_to_text: "{{query}}"
doc_to_target: "{{label}}"
doc_to_choice: "choices"
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
  - metric: brier_score
    higher_is_better: false
metadata:
  version: 1.0
```

`dummy2.yaml`:
```yaml
group:
  - dummygroup
task: dummytask2
dataset_path: hellaswag
dataset_name: null
output_type: multiple_choice
training_split: train
validation_split: validation
test_split: null
process_docs: !function utils.process_docs
doc_to_text: "{{query}}"
doc_to_target: "{{label}}"
doc_to_choice: "choices"
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
  - metric: brier_score
    higher_is_better: false
metadata:
  version: 1.0
```

we get the following output:

|    Tasks    |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|-------------|-------|------|-----:|-----------|---|-----:|---|------|
|dummygroup   |N/A    |none  |     0|acc        |↑  |0.4000|±  |0.1732|
|             |       |none  |     0|brier_score|↓  |0.9913|±  |N/A   |
| - dummytask1|      1|none  |     0|acc        |↑  |0.4000|±  |0.2449|
|             |       |none  |     0|brier_score|↓  |0.9913|±  |N/A   |
| - dummytask2|      1|none  |     0|acc        |↑  |0.4000|±  |0.2449|
|             |       |none  |     0|brier_score|↓  |0.9913|±  |N/A   |

|  Groups  |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|----------|-------|------|-----:|-----------|---|-----:|---|------|
|dummygroup|N/A    |none  |     0|acc        |↑  |0.4000|±  |0.1732|
|          |       |none  |     0|brier_score|↓  |0.9913|±  |N/A   |


2. In the case the `higher_is_better` values are consistent, i.e.: 

`dummy1.yaml`:
```yaml
group:
  - dummygroup
task: dummytask1
dataset_path: hellaswag
dataset_name: null
output_type: multiple_choice
training_split: train
validation_split: validation
test_split: null
process_docs: !function utils.process_docs
doc_to_text: "{{query}}"
doc_to_target: "{{label}}"
doc_to_choice: "choices"
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
  - metric: brier_score
    higher_is_better: false  # <--------- THIS IS SET TO FALSE  
metadata:
  version: 1.0
```

`dummy2.yaml`:
```yaml
group:
  - dummygroup
task: dummytask2
dataset_path: hellaswag
dataset_name: null
output_type: multiple_choice
training_split: train
validation_split: validation
test_split: null
process_docs: !function utils.process_docs
doc_to_text: "{{query}}"
doc_to_target: "{{label}}"
doc_to_choice: "choices"
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
  - metric: brier_score
    higher_is_better: true  # <--------- THIS IS SET TO TRUE  
metadata:
  version: 1.0
```
We get the following output:

|    Tasks    |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|-------------|-------|------|-----:|-----------|---|-----:|---|------|
|dummygroup   |N/A    |none  |     0|brier_score|   |0.9913|±  |N/A   |
|             |       |none  |     0|acc        |↑  |0.4000|±  |0.1732|
| - dummytask1|      1|none  |     0|acc        |↑  |0.4000|±  |0.2449|
|             |       |none  |     0|brier_score|↓  |0.9913|±  |N/A   |
| - dummytask2|      1|none  |     0|acc        |↑  |0.4000|±  |0.2449|
|             |       |none  |     0|brier_score|↑  |0.9913|±  |N/A   |

|  Groups  |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|----------|-------|------|-----:|-----------|---|-----:|---|------|
|dummygroup|N/A    |none  |     0|brier_score|   |0.9913|±  |N/A   |
|          |       |none  |     0|acc        |↑  |0.4000|±  |0.1732|


Problem statement:
Print "higher_is_better" in results table
We should add a column to the printed results table, that displays an arrow pointing up if the metric's `"higher_is_better"` value is True, arrow pointing down otherwise. 

Currently `higher_is_better` is not really used in the code.

Hints from issue discussion:
This can be a part of #1158

Commit message: add extra check for `higher_is_better` not being None already

Patch to review:
diff --git a/lm_eval/evaluator.py b/lm_eval/evaluator.py
index 95e018a508..ae269315ea 100644
--- a/lm_eval/evaluator.py
+++ b/lm_eval/evaluator.py
@@ -4,6 +4,7 @@
 import random
 import time
 from collections import defaultdict
+from copy import deepcopy
 from typing import TYPE_CHECKING, List, Optional, Union
 
 import numpy as np
@@ -503,9 +504,14 @@ def evaluate(
         # aggregate results ; run bootstrap CIs
         for task_output in eval_tasks:
             task_output.calculate_aggregate_metric(bootstrap_iters=bootstrap_iters)
-        results, samples, configs, versions, num_fewshot = consolidate_results(
-            eval_tasks
-        )
+        (
+            results,
+            samples,
+            configs,
+            versions,
+            num_fewshot,
+            higher_is_better,
+        ) = consolidate_results(eval_tasks)
 
         ### Calculate group metrics ###
         if bool(results):
@@ -516,6 +522,21 @@ def evaluate(
                     # or `task_name: []`.
                     # we only want to operate on groups here.
                     continue
+
+                _higher_is_better = deepcopy(higher_is_better[task_list[0]])
+                for task in task_list:
+                    for m, h in higher_is_better[task].items():
+                        if (
+                            m in _higher_is_better
+                            and _higher_is_better[m] is not None
+                            and _higher_is_better[m] != h
+                        ):
+                            eval_logger.warning(
+                                f"Higher_is_better values for metric {m} in group {group} are not consistent. Defaulting to None."
+                            )
+                            _higher_is_better[m] = None
+                higher_is_better[group] = _higher_is_better
+
                 metric_list = list(
                     {
                         key
@@ -591,6 +612,7 @@ def evaluate(
             "configs": dict(sorted(configs.items())),
             "versions": dict(sorted(versions.items())),
             "n-shot": dict(sorted(num_fewshot.items())),
+            "higher_is_better": dict(sorted(higher_is_better.items())),
             "n-samples": {
                 task_output.task_name: {
                     "original": len(task_output.task.eval_docs),
diff --git a/lm_eval/evaluator_utils.py b/lm_eval/evaluator_utils.py
index 4eb3d94ff5..82197b5d57 100644
--- a/lm_eval/evaluator_utils.py
+++ b/lm_eval/evaluator_utils.py
@@ -253,6 +253,9 @@ def consolidate_results(
     configs = collections.defaultdict(dict)
     # Tracks each task's version.
     versions = collections.defaultdict(dict)
+    # Track `higher_is_better` for each metric
+    higher_is_better = collections.defaultdict(dict)
+
     for task_output in eval_tasks:
         if "task_alias" in (task_config := task_output.task_config):
             results[task_output.task_name]["alias"] = task_config["task_alias"]
@@ -263,6 +266,7 @@ def consolidate_results(
         configs[task_output.task_name] = task_output.task_config
         versions[task_output.task_name] = task_output.version
         samples[task_output.task_name] = task_output.logged_samples
+        higher_is_better[task_output.task_name] = task_output.task.higher_is_better()
         for (metric, filter_key), items in task_output.sample_metrics.items():
             metric_key = f"{metric},{filter_key}"
             results[task_output.task_name][metric_key] = task_output.agg_metrics[
@@ -272,7 +276,7 @@ def consolidate_results(
             results[task_output.task_name][
                 f"{metric}_stderr,{filter_key}"
             ] = task_output.agg_metrics[f"{metric}_stderr,{filter_key}"]
-    return results, samples, configs, versions, num_fewshot
+    return results, samples, configs, versions, num_fewshot, higher_is_better
 
 
 @positional_deprecated
diff --git a/lm_eval/utils.py b/lm_eval/utils.py
index 2e71e38c99..54de16dd7a 100644
--- a/lm_eval/utils.py
+++ b/lm_eval/utils.py
@@ -26,6 +26,11 @@
 
 SPACING = " " * 47
 
+HIGHER_IS_BETTER_SYMBOLS = {
+    True: "↑",
+    False: "↓",
+}
+
 
 def hash_string(string: str) -> str:
     return hashlib.sha256(string.encode("utf-8")).hexdigest()
@@ -257,6 +262,7 @@ def make_table(result_dict, column: str = "results", sort_results: bool = True):
         "Filter",
         "n-shot",
         "Metric",
+        "",
         "Value",
         "",
         "Stderr",
@@ -277,6 +283,7 @@ def make_table(result_dict, column: str = "results", sort_results: bool = True):
         dic = result_dict[column][k]
         version = result_dict["versions"].get(k, "N/A")
         n = str(result_dict["n-shot"][k])
+        higher_is_better = result_dict.get("higher_is_better", {}).get(k, {})
 
         if "alias" in dic:
             k = dic.pop("alias")
@@ -286,13 +293,15 @@ def make_table(result_dict, column: str = "results", sort_results: bool = True):
             if m.endswith("_stderr"):
                 continue
 
+            hib = HIGHER_IS_BETTER_SYMBOLS.get(higher_is_better.get(m), "")
+
             if m + "_stderr" + "," + f in dic:
                 se = dic[m + "_stderr" + "," + f]
                 if se != "N/A":
                     se = "%.4f" % se
-                values.append([k, version, f, n, m, "%.4f" % v, "±", se])
+                values.append([k, version, f, n, m, hib, "%.4f" % v, "±", se])
             else:
-                values.append([k, version, f, n, m, "%.4f" % v, "", ""])
+                values.append([k, version, f, n, m, hib, "%.4f" % v, "", ""])
             k = ""
             version = ""
     md_writer.value_matrix = values


Code review comments:
diff_hunk: @@ -516,6 +522,21 @@ def evaluate(
                     # or `task_name: []`.
                     # we only want to operate on groups here.
                     continue
+
+                _higher_is_better = deepcopy(higher_is_better[task_list[0]])
+                for task in task_list:
+                    for m, h in higher_is_better[task].items():
+                        if (
+                            m in _higher_is_better
+                            and _higher_is_better[m] is not None
+                            and _higher_is_better[m] != h
+                        ):
path: lm_eval/evaluator.py
text: @user1:
```suggestion
                _higher_is_better = {}
                for task in task_list:
                    for m, h in higher_is_better[task].items():
                    	if m not in _higher_is_better.keys():
                    	    _higher_is_better[m] = h
                        if (
                            m in _higher_is_better
                            and _higher_is_better[m] is not None
                            and _higher_is_better[m] != h
                        ):
```

@user1:
May not be safe to reference `task_list[0]` to obtain all of a group's metrics. 

#1741 should make this cleaner but for now, I believe this should do the trick

@user1:
tested with group `pythia` and this change appears to work!