Repository: optuna/optuna
Pull request: Fix TensorBoard error on categorical choices of mixed types
<!-- Thank you for creating a pull request! In general, we merge your pull request after it gets two or more approvals. To proceed to the review process by the maintainers, please make sure that the PR meets the following conditions: (1) it passes all CI checks, and (2) it is neither in the draft nor WIP state. If you wish to discuss the PR in the draft state or need any other help, please mention the Optuna development team in the PR. -->

## Motivation
<!-- Describe your motivation why you will submit this PR. This is useful for reviewers to understand the context of PR. -->

The current version of TensorBoard integration fails when a choice of categorical distribution is not type of `int`, `float`, `bool`, or `str`. This is caused by a limitation of TensorBoard.

https://github.com/tensorflow/tensorboard/blob/80ccab8c8729ff7f8f5e17c132af808b0efe5f5b/tensorboard/plugins/hparams/summary_v2.py#L477-L499

## Description of the changes
<!-- Describe the changes in this PR. -->

This PR fixes the issue by converting choices to strings when needed.

Fixes #4761.

Problem statement:
TensorBoard integration fails if None is suggested
### Expected behavior

`Optuna` with `TensorBoardCallback` should allow `None` as a suggestion.

### Environment

- Optuna version:3.2.0
- Python version:3.9.13
- OS:Linux-5.4.228-132.418.amzn2.x86_64-x86_64-with-glibc2.31

### Error messages, stack traces, or logs

```shell
File /opt/python3.9/site-packages/tensorboard/plugins/hparams/summary_v2.py:507, in Discrete.__init__(self, values, dtype)
    505         raise ValueError("Empty domain with no dtype specified")
    506 if dtype not in (int, float, bool, str):
--> 507     raise ValueError("Unknown dtype: %r" % (dtype,))
    508 self._dtype = dtype
    509 for value in self._values:

ValueError: Unknown dtype: <class 'NoneType'>
```
```


### Steps to reproduce

1. If I run this
```python
import optuna
from optuna.integration.tensorboard import TensorBoardCallback

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import f1_score

X, y = make_classification(n_samples=1000, n_features=4,
                           n_informative=2, n_redundant=0,
                           random_state=0, shuffle=False)

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.33, random_state=0)

def objective(trial):
    params = {
        'max_leaf_nodes': trial.suggest_categorical('max_leaf_nodes', [None, 2, 5, 10, 20]),
    }
    clf = RandomForestClassifier()
    clf.fit(X_train, y_train)

    y_val_pred = clf.predict(X_val)
    f1 = f1_score(y_val, y_val_pred)
    return f1

tensorboard_callback = TensorBoardCallback("logs/", metric_name="f1")
study = optuna.create_study(direction="minimize")

study.optimize(objective, n_trials=10, timeout=600, callbacks=[tensorboard_callback])
```

It would return an error above.

2. If I don't have `None` in the suggestion or not use `TensorBoardCallback` then it would work fine.

### Additional context (optional)

_No response_

Hints from issue discussion:
I think it would be sufficient to add a code that converts None to str. This is because after converting to str, it is used only for recording or visualization purposes.
@saiwing-yeung Thank you for taking the time to submit this bug report. Are you interested in contributing to fix this issue?
@ciffelia is working on this issue.

Commit message: Fix TensorBoard error on categorical suggestions
fixes #4761

Patch to review:
diff --git a/optuna/integration/tensorboard.py b/optuna/integration/tensorboard.py
index 19ede07a27a..c9b278ed5a7 100644
--- a/optuna/integration/tensorboard.py
+++ b/optuna/integration/tensorboard.py
@@ -4,12 +4,15 @@
 import optuna
 from optuna._experimental import experimental_class
 from optuna._imports import try_import
+from optuna.logging import get_logger
 
 
 with try_import() as _imports:
     from tensorboard.plugins.hparams import api as hp
     import tensorflow as tf
 
+_logger = get_logger(__name__)
+
 
 @experimental_class("2.0.0")
 class TensorBoardCallback:
@@ -46,7 +49,11 @@ def __call__(self, study: optuna.study.Study, trial: optuna.trial.FrozenTrial) -
         for param_name, param_value in trial.params.items():
             if param_name not in self._hp_params:
                 self._add_distributions(trial.distributions)
-            hparams[self._hp_params[param_name]] = param_value
+            param = self._hp_params[param_name]
+            if isinstance(param.domain, hp.Discrete):
+                hparams[param] = param.domain.dtype(param_value)
+            else:
+                hparams[param] = param_value
         run_name = "trial-%d" % trial.number
         run_dir = os.path.join(self._dirname, run_name)
         with tf.summary.create_file_writer(run_dir).as_default():
@@ -74,9 +81,24 @@ def _add_distributions(
                     hp.IntInterval(param_distribution.low, param_distribution.high),
                 )
             elif isinstance(param_distribution, optuna.distributions.CategoricalDistribution):
+                choices = param_distribution.choices
+                dtype = type(choices[0])
+                if any(not isinstance(choice, dtype) for choice in choices):
+                    _logger.warning(
+                        "Choices contains mixed types, which is not supported by TensorBoard."
+                        "Converting all choices to strings."
+                    )
+                    choices = tuple(map(str, choices))
+                elif dtype not in (int, float, bool, str):
+                    _logger.warning(
+                        f"Choices are of type {dtype}, which is not supported by TensorBoard."
+                        "Converting all choices to strings."
+                    )
+                    choices = tuple(map(str, choices))
+
                 self._hp_params[param_name] = hp.HParam(
                     param_name,
-                    hp.Discrete(param_distribution.choices),
+                    hp.Discrete(choices, dtype),
                 )
             else:
                 distribution_list = [
diff --git a/tests/integration_tests/test_tensorboard.py b/tests/integration_tests/test_tensorboard.py
index 317a3552a75..de53ffbda1e 100644
--- a/tests/integration_tests/test_tensorboard.py
+++ b/tests/integration_tests/test_tensorboard.py
@@ -63,6 +63,36 @@ def objective(trial: optuna.trial.Trial) -> float:
     study.optimize(objective, n_trials=1, callbacks=[tbcallback])
 
 
+def test_categorical() -> None:
+    def objective(trial: optuna.trial.Trial) -> float:
+        x = trial.suggest_categorical("x", [1, 2, 3])
+        assert isinstance(x, int)
+        return x
+
+    dirname = tempfile.mkdtemp()
+    metric_name = "target"
+    study_name = "test_tensorboard_integration"
+
+    tbcallback = TensorBoardCallback(dirname, metric_name)
+    study = optuna.create_study(study_name=study_name)
+    study.optimize(objective, n_trials=1, callbacks=[tbcallback])
+
+
+def test_categorical_mixed_types() -> None:
+    def objective(trial: optuna.trial.Trial) -> float:
+        x = trial.suggest_categorical("x", [None, 1, 2, 3, True, "foo"])
+        assert isinstance(x, str)
+        return len(x)
+
+    dirname = tempfile.mkdtemp()
+    metric_name = "target"
+    study_name = "test_tensorboard_integration"
+
+    tbcallback = TensorBoardCallback(dirname, metric_name)
+    study = optuna.create_study(study_name=study_name)
+    study.optimize(objective, n_trials=1, callbacks=[tbcallback])
+
+
 def test_experimental_warning() -> None:
     with pytest.warns(optuna.exceptions.ExperimentalWarning):
         TensorBoardCallback(dirname="", metric_name="")


Code review comments:
diff_hunk: @@ -63,6 +63,36 @@ def objective(trial: optuna.trial.Trial) -> float:
     study.optimize(objective, n_trials=1, callbacks=[tbcallback])
 
 
+def test_categorical() -> None:
+    def objective(trial: optuna.trial.Trial) -> float:
+        x = trial.suggest_categorical("x", [1, 2, 3])
+        assert isinstance(x, int)
+        return x
+
+    dirname = tempfile.mkdtemp()
+    metric_name = "target"
+    study_name = "test_tensorboard_integration"
+
+    tbcallback = TensorBoardCallback(dirname, metric_name)
+    study = optuna.create_study(study_name=study_name)
+    study.optimize(objective, n_trials=1, callbacks=[tbcallback])
+
+
+def test_categorical_mixed_types() -> None:
+    def objective(trial: optuna.trial.Trial) -> float:
+        x = trial.suggest_categorical("x", [None, 1, 2, 3, True, "foo"])
+        assert isinstance(x, str)
path: tests/integration_tests/test_tensorboard.py
text: @user1:
I'm not sure why we can ensure that `x` is str. In my understanding, the value is chosen by `RandomSampler`, and it depends on the environment/implementation.