Repository: optuna/optuna

Issue:
`study.trials_dataframe` call fails for multi-objective studies
When using the new multi-objective study framework in v2.4.0 (`optuna.create_study(..., directions=[...])`), calling the `trials_dataframe` message raises an exception caused by some incorrect column spec type comparison in the _dataframe framework. It appears the column aggregation for contains a bug when used with multiple objectives, which adds an extra `values` Tuple to the `values` Set during the initial mapping of `'value' -> 'values'` for a multi-objective study.

## Expected behavior

Accessing the `trials_dataframe` method of a study works the same for both single and multi-objective studies.

## Environment

- Optuna version: 2.3.0 -> 2.4.0
- Python version: 3.8
- OS: Ubuntu 20.04.1, Linux Kernel v5.4.0-56

## Error messages, stack traces, or logs

```
Traceback (most recent call last):
  File "optuna_vis.py", line 12, in <module>
    study_df = study.trials_dataframe()
  File "/home/user.name/anaconda3/lib/python3.8/site-packages/optuna/study.py", line 499, in trials_dataframe
    return _trials_dataframe(self, attrs, multi_index)
  File "/home/user.name/anaconda3/lib/python3.8/site-packages/optuna/_dataframe.py", line 80, in _trials_dataframe
    columns: List[Tuple[str, str]] = sum(
  File "/home/user.name/anaconda3/lib/python3.8/site-packages/optuna/_dataframe.py", line 81, in <genexpr>
    (sorted(column_agg[k]) for k in attrs if k in column_agg), []
TypeError: '<' not supported between instances of 'str' and 'int'
```

By inspecting the `attr` field in the `_trials_dataframe` method (line 23, _dataframe.py), reveals an extra `values` item:

```
{'number': {('number', '')}, 'values': {('values', 1), ('values', ''), ('values', 0), ('values', 3), ('values', 2)}, 'datetime_start': {('datetime_start', '')}, 'datetime_complete': {('datetime_complete', '')}, 'duration': {('duration', '')}, 'params': {(<param_tuples>), ...}, 'user_attrs': {('user_attrs', 'total_x'), ('user_attrs', 'total_y'), ('user_attrs', '_time_'), ('user_attrs', 'value_total')}, 'system_attrs': {('system_attrs', 'nsga2:generation')}, 'state': {('state', '')}}
```

For context, I have defined 4 objectives in this study. After the first trial, there appears this `('values', '')` anomaly which causes the above exception.

## Steps to reproduce

1. Create a MPO study with 4 objectives.
2. Start your study and after at least 2 trials, try to call the `study.trials_dataframe()` method.
3. The above exception will be raised.



Discussion:
Thanks for raising an issue. I have tested with the following code snippets but had no error with the following outputs. The version of Optuna is v2.4.0.
```python
import optuna


def objective(trial):
    return 0, 1, 2, 3

study = optuna.create_study(directions=["minimize"]*4)
study.optimize(objective, n_trials=2)
print(study.trials_dataframe())
```
<img width="1182" alt="スクリーンショット 2021-01-19 8 16 08" src="https://user-images.githubusercontent.com/38826298/104970387-9af9f500-5a2e-11eb-97ef-e1a779ea7760.png">

Could you share the reproducible codes or snippet?
Sure! After investigation, I found the cause of the issue. It seems that if, for some reason, the first trial fails or returns a value which is inconsistent with the multi-objective value intended for the trial, the mapping of `value -> values` adds this erroneous item to the `values` set. Then, in the internal `_trials_dataframe` method, the `sort` comparison fails.

In my case, the distributed nature of the study would sometimes fail during initialization on the first trial or a subsequent trial would finish before the first trial had finished, causing the value for the first trial to remain unset during the first call of `study.trials_dataframe`.

Here is a modified version of your snippet which reproduces the issue:

```
import optuna

def objective(trial):
    return 0, 1, 2, 3 if trial.number else None

study = optuna.create_study(study_name="test", directions=["minimize"]*4, storage="sqlite:///motpe.db", load_if_exists=True)
study.optimize(objective, n_trials=10)
print(study.trials_dataframe())
```

Perhaps it would make more sense to create the `values` set on the initial creation of the study from the `directions` passed during to `optuna.create_study` rather than trying to parse the first trial's returned values? This would certainly be more consistent.

Resolving patch:
diff --git a/optuna/_dataframe.py b/optuna/_dataframe.py
--- a/optuna/_dataframe.py
+++ b/optuna/_dataframe.py
@@ -68,6 +68,12 @@ def _create_record_and_aggregate_column(
                 for nested_attr, nested_value in enumerate(value):
                     record[(df_column, nested_attr)] = nested_value
                     column_agg[attr].add((df_column, nested_attr))
+            elif attr == "values":
+                # trial.values should be None when the trial's state is FAIL or PRUNED.
+                assert value is None
+                for nested_attr in range(len(study.directions)):
+                    record[(df_column, nested_attr)] = None
+                    column_agg[attr].add((df_column, nested_attr))
             else:
                 record[(df_column, non_nested_attr)] = value
                 column_agg[attr].add((df_column, non_nested_attr))


Test patch:
diff --git a/tests/study_tests/test_dataframe.py b/tests/study_tests/test_dataframe.py
--- a/tests/study_tests/test_dataframe.py
+++ b/tests/study_tests/test_dataframe.py
@@ -4,9 +4,11 @@
 import pytest
 
 from optuna import create_study
+from optuna import create_trial
 from optuna import Trial
 from optuna.testing.storage import STORAGE_MODES
 from optuna.testing.storage import StorageSupplier
+from optuna.trial import TrialState
 
 
 def test_study_trials_dataframe_with_no_trials() -> None:
@@ -183,3 +185,29 @@ def f(trial: Trial) -> Tuple[float, float]:
         for i in range(3):
             assert df.values_0[i] == 3
             assert df.values_1[i] == 5
+
+
+@pytest.mark.parametrize(
+    "attrs",
+    [
+        ("value",),
+        ("values",),
+    ],
+)
+@pytest.mark.parametrize("multi_index", [True, False])
+def test_trials_dataframe_with_multi_objective_optimization_with_fail_and_pruned(
+    attrs: Tuple[str, ...], multi_index: bool
+) -> None:
+    study = create_study(directions=["minimize", "maximize"])
+    study.add_trial(create_trial(state=TrialState.FAIL))
+    study.add_trial(create_trial(state=TrialState.PRUNED))
+    df = study.trials_dataframe(attrs=attrs, multi_index=multi_index)
+
+    if multi_index:
+        for i in range(2):
+            assert df.get("values")[0][i] is None
+            assert df.get("values")[1][i] is None
+    else:
+        for i in range(2):
+            assert df.values_0[i] is None
+            assert df.values_1[i] is None