Repository: optuna/optuna
Pull request: Fix `--verbose` and `--quiet` options in CLI
## Motivation
To address the issue #3485.

## Description of the changes
`_OptunaApp` will set verbosity with the same value specified by its super `App` in `cliff.app`. Also test code is added.
Ref. https://github.com/openstack/cliff/blob/master/cliff/app.py#L211-L219

Problem statement:
`--verbose` and `--quiet` option in CLI is not working
### Expected behavior

`--verbose` and `--quiet` options in CLI should change the logging level of the Optuna's logger.

### Environment

- Optuna version: 3.0.0b1.dev
- Python version: 3.8.2
- OS: macOS-10.16-x86_64-i386-64bit
- (Optional) Other libraries and their versions:


### Error messages, stack traces, or logs

```shell
See steps to reproduce.
```


### Steps to reproduce

```sh
~/optuna$ optuna create-study --storage sqlite:///test.db --study-name test-study -q
[I 2022-04-22 20:36:11,192] A new study created in RDB with name: test-study
test-study
```

The INFO level logging message should be suppressed by `-q`.

### Additional context (optional)

_No response_

Hints from issue discussion:
@nyanhi is working on debugging this issue.

Commit message: add test for verbosity

Patch to review:
diff --git a/optuna/cli.py b/optuna/cli.py
index 254626d7924..9d729866303 100644
--- a/optuna/cli.py
+++ b/optuna/cli.py
@@ -919,6 +919,7 @@ def configure_logging(self) -> None:
         assert len(stream_handlers) == 1
         stream_handler = stream_handlers[0]
         stream_handler.setFormatter(optuna.logging.create_default_formatter())
+        optuna.logging.set_verbosity(stream_handler.level)
 
     def clean_up(self, cmd: Command, result: int, err: Optional[Exception]) -> None:
 
diff --git a/tests/test_cli.py b/tests/test_cli.py
index 04a0e95caf4..c97a1a4cc69 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -1542,3 +1542,25 @@ def test_tell_with_nan() -> None:
         assert len(study.trials) == 1
         assert study.trials[0].state == TrialState.FAIL
         assert study.trials[0].values is None
+
+
+@pytest.mark.skip_coverage
+@pytest.mark.parametrize(
+    "verbosity, expected",
+    [
+        ("--verbose", True),
+        ("--quiet", False),
+    ],
+)
+def test_configure_logging_verbosity(verbosity: str, expected: bool) -> None:
+
+    with StorageSupplier("sqlite") as storage:
+        assert isinstance(storage, RDBStorage)
+        storage_url = str(storage.engine.url)
+
+        # Create study.
+        args = ["optuna", "create-study", "--storage", storage_url, verbosity]
+        # If `--verbose`, the INFO level log message should be generated.
+        result = subprocess.run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        error_message = result.stderr.decode()
+        assert ("A new study created in RDB with name" in error_message) == expected


Code review comments:
diff_hunk: @@ -1542,3 +1542,25 @@ def test_tell_with_nan() -> None:
         assert len(study.trials) == 1
         assert study.trials[0].state == TrialState.FAIL
         assert study.trials[0].values is None
+
+
+@pytest.mark.skip_coverage
+@pytest.mark.parametrize(
+    "verbosity, expected",
+    [
+        ("--verbose", True),
+        ("--quiet", False),
+    ],
+)
+def test_configure_logging_verbosity(verbosity: str, expected: bool) -> None:
+
+    with StorageSupplier("sqlite") as storage:
+        assert isinstance(storage, RDBStorage)
+        storage_url = str(storage.engine.url)
+
+        # Create study.
+        args = ["optuna", "create-study", "--storage", storage_url, verbosity]
+        # If `--verbose`, the INFO level log message should be generated.
path: tests/test_cli.py
text: @user1:
Your unit tests seem OK, but could you clarify the logging level on the comment? The logging level with the `--verbose` option is `DEBUG`. I'm not sure it is easy to test `DEBUG` level logging messages because Optuna has a small number of debug messages.

@author:
This unit test confirms the necessary condition that the behavior changes depending on the value of the `--verbose` option. As you point out, it doesn't fully test for the `DEBUG` level - sorry I couldn't think of simple way to generate `DEBUG` logging message via CLI.

@user1:
> sorry I couldn't think of simple way to generate DEBUG logging message via CLI.

I agree it is not easy to test.

> If `--verbose`, the INFO level log message should be generated.

This comment seems confusing about the log level with the `--verbose` option. How about

```python
# `--verbose` makes the log level DEBUG.
# `--quiet` makes the log level WARNING.
```

@author:
Thank you so much for the suggestion, I have updated the comment. Kindly let me know if anything else.