Repository: optuna/optuna

Issue:
[RFC] Show progress bar based on `timeout`
<!-- Please write a clear and concise description of the feature proposal. -->

## Motivation

The current [`study.optimize`](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.Study.html#optuna.study.Study.optimize) shows the progress bar when we specify the number of trials. We can show the progress bar based on the `timeout` value as well.

Note that before implementing this feature, we need to decide the priority of `n_trials` and `timeout`. When we specify both values as not `None` values, there are two ways to show the progress bar. 

Moreover, we need to discuss the case when both `None`. I suppose not showing the progress bar makes sense when both are `None`.

## Description

<!-- Please write a detailed description of the new feature. -->


- Show progress bar using `timeout` value.
- Add test by following https://github.com/optuna/optuna/pull/2900

## Alternatives (optional)


<!-- Please write a clear and concise description of any alternative solutions or features you've considered. -->

## Additional context (optional)

We might need to support the `n_jobs!=1` case depending on https://github.com/optuna/optuna/issues/2957.

<!-- Please add any other context or screenshots about the feature request here. -->



Discussion:
> When we specify both values as not None values, there are two ways to show the progress bar.

We could always show `n_trials` early on, and check after first iteration (on nth, depending on precision we want) if completing all trials will take longer than `timeout`. If so, we would switch progbar from trials left to time left.

> Moreover, we need to discuss the case when both None. I suppose not showing the progress bar makes sense when both are None.

Sounds reasonable.

I'll open a PR on this if you have capacity to review it.
Thank you for your comment! 

Let me summarise candidates if both `n_trials` and `timeout` are not `None`:

1. prioritise `n_trials`
2. prioritise `timeout`
3. during the first trial, use `n_trials` (or maybe `timeout`). After the first trial, decide which to choose `n_trials` or `timeout` for the progress bar.
4. (more accurate version of 3) during the first trial, use `n_trials` (or maybe `timeout`). After _each_ trial, decide which to choose `n_trials` or `timeout` for the progress bar. 
IMO +1 to the last candidate. Informative and will work well when trial execution times are uneven.
Me too! I'll ask other committers about this.
@toshihikoyanase suggested using only `n_trials`: the first candidate. A user wants to perform optimisation `n_trials` times ideally but, to avoid taking too much to perform optimisation, he/she specifies `timeout` at the same time. Indeed, I saw such a setting in a machine learning paper, so his suggestion is supposed to be natural.

So how about implementing candidate 1 at first? If the optuna community prefers the last candidates to the first one or sends feedback to us, we can change the logic. 

Thanks, @toshihikoyanase for your comment.
Sure, I'll prioritize `n_trials` then! Will come back with implementation after the weekend.

Resolving patch:
diff --git a/optuna/progress_bar.py b/optuna/progress_bar.py
--- a/optuna/progress_bar.py
+++ b/optuna/progress_bar.py
@@ -39,9 +39,11 @@ class _ProgressBar(object):
     def __init__(
         self, is_valid: bool, n_trials: Optional[int] = None, timeout: Optional[float] = None
     ) -> None:
-        self._is_valid = is_valid
+
+        self._is_valid = is_valid and (n_trials or timeout) is not None
         self._n_trials = n_trials
         self._timeout = timeout
+        self._last_elapsed_seconds = 0.0
 
         if self._is_valid:
             self._init_valid()
@@ -50,7 +52,21 @@ def __init__(
     # longer experimental.
     @experimental("1.2.0", name="Progress bar")
     def _init_valid(self) -> None:
-        self._progress_bar = tqdm(range(self._n_trials) if self._n_trials is not None else None)
+
+        if self._n_trials is not None:
+            self._progress_bar = tqdm(total=self._n_trials)
+
+        else:
+            fmt = "{percentage:3.0f}%|{bar}| {elapsed}/{desc}"
+            self._progress_bar = tqdm(total=self._timeout, bar_format=fmt)
+
+            # Using description string instead postfix string
+            # to display formatted timeout, since postfix carries
+            # extra comma space auto-format.
+            # https://github.com/tqdm/tqdm/issues/712
+            total = tqdm.format_interval(self._timeout)
+            self._progress_bar.set_description_str(total)
+
         global _tqdm_handler
 
         _tqdm_handler = _TqdmLoggingHandler()
@@ -59,22 +75,37 @@ def _init_valid(self) -> None:
         optuna_logging.disable_default_handler()
         optuna_logging._get_library_root_logger().addHandler(_tqdm_handler)
 
-    def update(self, elapsed_seconds: Optional[float]) -> None:
+    def update(self, elapsed_seconds: float) -> None:
         """Update the progress bars if ``is_valid`` is :obj:`True`.
 
         Args:
             elapsed_seconds:
                 The time past since :func:`~optuna.study.Study.optimize` started.
         """
+
         if self._is_valid:
-            self._progress_bar.update(1)
-            if self._timeout is not None and elapsed_seconds is not None:
-                self._progress_bar.set_postfix_str(
-                    "{:.02f}/{} seconds".format(elapsed_seconds, self._timeout)
-                )
+            if self._n_trials is not None:
+                self._progress_bar.update(1)
+                if self._timeout is not None:
+                    self._progress_bar.set_postfix_str(
+                        "{:.02f}/{} seconds".format(elapsed_seconds, self._timeout)
+                    )
+
+            elif self._timeout is not None:
+                time_diff = elapsed_seconds - self._last_elapsed_seconds
+                if elapsed_seconds > self._timeout:
+                    # Clip elapsed time to avoid tqdm warnings.
+                    time_diff -= elapsed_seconds - self._timeout
+
+                self._progress_bar.update(time_diff)
+                self._last_elapsed_seconds = elapsed_seconds
+
+            else:
+                assert False
 
     def close(self) -> None:
         """Close progress bars."""
+
         if self._is_valid:
             self._progress_bar.close()
             assert _tqdm_handler is not None


Test patch:
diff --git a/tests/study_tests/test_study.py b/tests/study_tests/test_study.py
--- a/tests/study_tests/test_study.py
+++ b/tests/study_tests/test_study.py
@@ -708,6 +708,77 @@ def test_optimize_without_progbar(capsys: _pytest.capture.CaptureFixture) -> Non
     assert "100%" not in err
 
 
+def test_optimize_with_progbar_timeout(capsys: _pytest.capture.CaptureFixture) -> None:
+
+    study = create_study()
+    study.optimize(lambda _: 1.0, timeout=2.0, show_progress_bar=True)
+    _, err = capsys.readouterr()
+
+    assert "00:02/00:02" in err
+    assert "100%" in err
+
+
+@pytest.mark.parametrize(
+    "timeout,expected",
+    [
+        (59.0, "/00:59"),
+        (60.0, "/01:00"),
+        (60.0 * 60, "/1:00:00"),
+        (60.0 * 60 * 24, "/24:00:00"),
+        (60.0 * 60 * 24 * 10, "/240:00:00"),
+    ],
+)
+def test_optimize_with_progbar_timeout_formats(
+    timeout: float, expected: str, capsys: _pytest.capture.CaptureFixture
+) -> None:
+    def _objective(trial: Trial) -> float:
+        if trial.number == 5:
+            trial.study.stop()
+        return 1.0
+
+    study = create_study()
+    study.optimize(_objective, timeout=timeout, show_progress_bar=True)
+    _, err = capsys.readouterr()
+    assert expected in err
+
+
+def test_optimize_without_progbar_timeout(capsys: _pytest.capture.CaptureFixture) -> None:
+
+    study = create_study()
+    study.optimize(lambda _: 1.0, timeout=2.0)
+    _, err = capsys.readouterr()
+
+    assert "00:02/00:02" not in err
+    assert "100%" not in err
+
+
+def test_optimize_progbar_n_trials_prioritized(capsys: _pytest.capture.CaptureFixture) -> None:
+
+    study = create_study()
+    study.optimize(lambda _: 1.0, n_trials=10, timeout=10.0, show_progress_bar=True)
+    _, err = capsys.readouterr()
+
+    assert "10/10" in err
+    assert "100%" in err
+    assert "it" in err
+
+
+def test_optimize_progbar_no_constraints(capsys: _pytest.capture.CaptureFixture) -> None:
+    def _objective(trial: Trial) -> float:
+        if trial.number == 5:
+            trial.study.stop()
+        return 1.0
+
+    study = create_study()
+    study.optimize(_objective, show_progress_bar=True)
+    _, err = capsys.readouterr()
+
+    # We can't simply test if stderr is empty, since we're not sure
+    # what else could write to it. Instead, we are testing for a character
+    # that forms progress bar borders.
+    assert "|" not in err
+
+
 def test_optimize_with_progbar_parallel(capsys: _pytest.capture.CaptureFixture) -> None:
 
     study = create_study()