Repository: optuna/optuna
Pull request: Add docstrings to `optuna.termintor`
## Motivation
Resolve #4598

## Description of the changes
- Add docstrings to
  - `TerminatorCallback` (`optuna/terminator/callback.py`)
  - `CrossValidationErrorEvaluator` (`optuna/terminator/erroreval.py`)
  - `report_cross_validation_scores` (`optuna/terminator/erroreval.py`)
  - `StaticErrorEvaluator` (`optuna/terminator/erroreval.py`)
- Fix docstring of `Terminator` (`optuna/terminator/terminator.py`)
- Add `.rst` files for terminator documentation

Problem statement:
Add docstrings for Optuna Terminator APIs
### What is an issue?

The goal of this issue is to write docstrings for the following Optuna Terminator APIs:

- [x] `Terminator` (`optuna/terminator/terminator.py`)
- [ ] `TerminatorCallback` (`optuna/terminator/callback.py`)
- [ ] `CrossValidationErrorEvaluator` (`optuna/terminator/erroreval.py`)
- [ ] `report_cross_validation_scores` (`optuna/terminator/erroreval.py`)
- [ ] `StaticErrorEvaluator` (`optuna/terminator/erroreval.py`)

It is expected that the docstrings will include usage examples.

Hints from issue discussion:
The docstring for the `Terminator` class is being added in #4596

Commit message: Add all objects to docs under terminator module

Patch to review:
diff --git a/docs/source/reference/index.rst b/docs/source/reference/index.rst
index 26ca4012de5..a88394fed75 100644
--- a/docs/source/reference/index.rst
+++ b/docs/source/reference/index.rst
@@ -16,5 +16,6 @@ API Reference
     search_space
     storages
     study
+    terminator
     trial
     visualization/index
diff --git a/docs/source/reference/terminator.rst b/docs/source/reference/terminator.rst
new file mode 100644
index 00000000000..a3e76cabe91
--- /dev/null
+++ b/docs/source/reference/terminator.rst
@@ -0,0 +1,16 @@
+.. module:: optuna.terminator
+
+optuna.terminator
+=================
+
+The :mod:`~optuna.terminator` module implements a mechanism for automatically terminating the optimization process, accompanied by a callback class for the termination and evaluators for the estimated room for improvement in the optimization and statistical error of the objective function. The terminator determines to terminate the optimization process when the improvement is estimated to be smaller than the statistical error.
+
+.. autosummary::
+   :toctree: generated/
+   :nosignatures:
+
+   optuna.terminator.Terminator
+   optuna.terminator.TerminatorCallback
+   optuna.terminator.CrossValidationErrorEvaluator
+   optuna.terminator.StaticErrorEvaluator
+   optuna.terminator.report_cross_validation_scores
\ No newline at end of file
diff --git a/optuna/terminator/callback.py b/optuna/terminator/callback.py
index 51d4866d6b7..7e549de85de 100644
--- a/optuna/terminator/callback.py
+++ b/optuna/terminator/callback.py
@@ -13,6 +13,55 @@
 
 @experimental_class("3.2.0")
 class TerminatorCallback:
+    """A callback that terminates the optimization using Terminator.
+
+    This class implements a callback which wraps :class:`~optuna.terminator.Terminator`
+    so that it can be used with the :func:`~optuna.study.Study.optimize` method.
+
+    Args:
+        terminator:
+            A terminator object which determines whether to terminate the optimization by
+            assessing the room for optimization and statistical error. Defaults to a
+            :class:`~optuna.terminator.Terminator` object with default
+            improvement_evaluator and error_evaluator.
+
+    Example:
+
+        .. testcode::
+
+            from sklearn.datasets import load_wine
+            from sklearn.ensemble import RandomForestClassifier
+            from sklearn.model_selection import cross_val_score
+            from sklearn.model_selection import KFold
+
+            import optuna
+            from optuna.terminator import TerminatorCallback
+            from optuna.terminator import report_cross_validation_scores
+
+
+            def objective(trial):
+                X, y = load_wine(return_X_y=True)
+
+                clf = RandomForestClassifier(
+                    max_depth=trial.suggest_int("max_depth", 2, 32),
+                    min_samples_split=trial.suggest_float("min_samples_split", 0, 1),
+                    criterion=trial.suggest_categorical("criterion", ("gini", "entropy")),
+                )
+
+                scores = cross_val_score(clf, X, y, cv=KFold(n_splits=5, shuffle=True))
+                report_cross_validation_scores(trial, scores)
+                return scores.mean()
+
+
+            study = optuna.create_study(direction="maximize")
+            terminator = TerminatorCallback()
+            study.optimize(objective, n_trials=50, callbacks=[terminator])
+
+    .. seealso::
+        Please refer to :class:`~optuna.terminator.Terminator` for the details of
+        the terminator mechanism.
+    """
+
     def __init__(
         self,
         terminator: Optional[BaseTerminator] = None,
diff --git a/optuna/terminator/erroreval.py b/optuna/terminator/erroreval.py
index 02b1e159b30..b790d0cc54e 100644
--- a/optuna/terminator/erroreval.py
+++ b/optuna/terminator/erroreval.py
@@ -27,11 +27,33 @@ def evaluate(
 
 @experimental_class("3.2.0")
 class CrossValidationErrorEvaluator(BaseErrorEvaluator):
+    """An error evaluator for objective functions based on cross-validation.
+
+    This evaluator evaluates objective functions' statistical error, which comes from the
+    randomness of dataset. This evaluator assumes that the objective function is evaluated by
+    cross-validation and uses the scaled variance of the cross-validation scores as the
+    statistical error.
+
+    """
+
     def evaluate(
         self,
         trials: list[FrozenTrial],
         study_direction: StudyDirection,
     ) -> float:
+        """Evaluate the statistical error of the objective function based on cross-validation.
+
+        Args:
+            trials:
+                A list of trials to evaluate the error of the best trial.
+
+            study_direction:
+                The direction of the study.
+
+        Returns:
+            A float representing the statistical error of the objective function.
+
+        """
         trials = [trial for trial in trials if trial.state == TrialState.COMPLETE]
         assert len(trials) > 0
 
@@ -62,6 +84,19 @@ def evaluate(
 
 @experimental_class("3.2.0")
 def report_cross_validation_scores(trial: Trial, scores: list[float]) -> None:
+    """A function to report cross-validation scores of a trial.
+
+    This function should be called after cross-validation step within the trial to report its
+    scores. The reported scores are used to evaluate the statistical error for termination
+    judgement.
+
+    Args:
+        trial:
+            A :class:`~optuna.trial.Trial` object to report the cross-validation scores.
+        scores:
+            The cross-validation scores of the trial.
+
+    """
     if len(scores) <= 1:
         raise ValueError("The length of `scores` is expected to be greater than one.")
     trial.storage.set_trial_system_attr(trial._trial_id, _CROSS_VALIDATION_SCORES_KEY, scores)
@@ -69,6 +104,16 @@ def report_cross_validation_scores(trial: Trial, scores: list[float]) -> None:
 
 @experimental_class("3.2.0")
 class StaticErrorEvaluator(BaseErrorEvaluator):
+    """An error evaluator that always returns a constant value.
+
+    This evaluator supports the cases where a fixed regret bound threshold to achieve is specified.
+
+    Args:
+        constant:
+            A user-specified constant value to always return as an error estimate.
+
+    """
+
     def __init__(self, constant: float) -> None:
         self._constant = constant
 
diff --git a/optuna/terminator/terminator.py b/optuna/terminator/terminator.py
index ec8290f1d4c..fd95dc1e842 100644
--- a/optuna/terminator/terminator.py
+++ b/optuna/terminator/terminator.py
@@ -19,7 +19,7 @@ def should_terminate(self, study: Study) -> bool:
 
 @experimental_class("3.2.0")
 class Terminator(BaseTerminator):
-    """Automatic stopping mechanism for Optuna studies
+    """Automatic stopping mechanism for Optuna studies.
 
     This class implements an automatic stopping mechanism for Optuna studies, aiming to prevent
     unnecessary computation. The study is terminated when the statistical error, e.g.
@@ -36,7 +36,7 @@ class Terminator(BaseTerminator):
             :class:`~optuna.terminator.improvement.evaluator.RegretBoundEvaluator` object.
         error_evaluator:
             An evaluator for calculating the statistical error, e.g. cross-validation error.
-            Defaults to a :class:`~optuna.terminator.erroreval.CrossValidationErrorEvaluator`
+            Defaults to a :class:`~optuna.terminator.CrossValidationErrorEvaluator`
             object.
         min_n_trials:
             The minimum number of trials before termination is considered. Defaults to ``20``.
@@ -45,45 +45,52 @@ class Terminator(BaseTerminator):
         ValueError: If ``min_n_trials`` is not a positive integer.
 
     Example:
-        from sklearn.datasets import load_wine
-        from sklearn.ensemble import RandomForestClassifier
-        from sklearn.model_selection import cross_val_score
-        from sklearn.model_selection import KFold
 
-        import optuna
-        from optuna.terminator.terminator import Terminator
-        from optuna.terminator.serror import report_cross_validation_scores
+        .. testcode::
 
+            import logging
+            import sys
 
-        study = optuna.create_study(direction="maximize")
-        terminator = Terminator()
-        min_n_trials = 20
+            from sklearn.datasets import load_wine
+            from sklearn.ensemble import RandomForestClassifier
+            from sklearn.model_selection import cross_val_score
+            from sklearn.model_selection import KFold
 
-        while True:
-            trial = study.ask()
+            import optuna
+            from optuna.terminator import Terminator
+            from optuna.terminator import report_cross_validation_scores
 
-            X, y = load_wine(return_X_y=True)
 
-            clf = RandomForestClassifier(
-                max_depth=trial.suggest_int("max_depth", 2, 32),
-                min_samples_split=trial.suggest_float("min_samples_split", 0, 1),
-                criterion=trial.suggest_categorical("criterion", ("gini", "entropy")),
-            )
+            study = optuna.create_study(direction="maximize")
+            terminator = Terminator()
+            min_n_trials = 20
 
-            scores = cross_val_score(clf, X, y, cv=KFold(n_splits=5, shuffle=True))
-            report_cross_validation_scores(trial, scores)
+            while True:
+                trial = study.ask()
 
-            value = scores.mean()
-            print(f"Trial #{trial.number} finished with value {value}.")
-            study.tell(trial, value)
+                X, y = load_wine(return_X_y=True)
 
-            if trial.number > min_n_trials and terminator.should_terminate(study):
-                print("Terminated by Optuna Terminator!")
-                break
+                clf = RandomForestClassifier(
+                    max_depth=trial.suggest_int("max_depth", 2, 32),
+                    min_samples_split=trial.suggest_float("min_samples_split", 0, 1),
+                    criterion=trial.suggest_categorical("criterion", ("gini", "entropy")),
+                )
+
+                scores = cross_val_score(clf, X, y, cv=KFold(n_splits=5, shuffle=True))
+                report_cross_validation_scores(trial, scores)
+
+                value = scores.mean()
+                logging.info(f"Trial #{trial.number} finished with value {value}.")
+                study.tell(trial, value)
+
+                if trial.number > min_n_trials and terminator.should_terminate(study):
+                    logging.info("Terminated by Optuna Terminator!")
+                    break
 
     .. seealso::
-        Please refer to :class:`~optuna.terminator.callbacks.TerminationCallback` for to use the
-        terminator mechanism with the :func:`~optuna.study.Study.optimize` method.
+        Please refer to :class:`~optuna.terminator.TerminationCallback` for how to use
+        the terminator mechanism with the :func:`~optuna.study.Study.optimize` method.
+
     """
 
     def __init__(
@@ -100,6 +107,7 @@ def __init__(
         self._min_n_trials = min_n_trials
 
     def should_terminate(self, study: Study) -> bool:
+        """Judge whether the study should be terminated based on the reported values."""
         trials = study.get_trials(states=[TrialState.COMPLETE])
 
         if len(trials) < self._min_n_trials:


Code review comments:
diff_hunk: @@ -0,0 +1,16 @@
+.. module:: optuna.terminator
+
+optuna.terminator
+=================
+
+The :mod:`~optuna.terminator` module implements a mechanism for automatically terminating the optimization process, accompanied by a callback class for the termination and evaluators for the estimated room for improvement in the optimization and statistical error of the objective function. The terminator determines to terminate the optimization process when the improvement is estimated to be smaller than the statistical error.
path: docs/source/reference/terminator.rst
text: @user1:
```suggestion
The :mod:`~optuna.terminator` module implements a mechanism for automatically terminating the optimization process, accompanied by a callback class for the termination and evaluators for the estimated room for improvement in the optimization and statistical error of the objective function. The terminator stops the optimization process when the estimated potential improvement is smaller than the statistical error.
```

diff_hunk: @@ -27,11 +27,33 @@ def evaluate(
 
 @experimental_class("3.2.0")
 class CrossValidationErrorEvaluator(BaseErrorEvaluator):
+    """An error evaluator for objective functions based on cross-validation.
+
+    This evaluator evaluates objective functions' statistical error, which comes from the
+    randomness of dataset. This evaluator assumes that the objective function is evaluated by
+    cross-validation and uses the scaled variance of the cross-validation scores as the
path: optuna/terminator/erroreval.py
text: @user1:
```suggestion
    cross-validation and uses the scaled variance of the reported cross-validation scores in the best trial as the
```