Repository: optuna/optuna

Issue:
`time_budget` option on `lightgbm_tuner.train()` is not working well
`time_budget` option on `lightgbm_tuner.train()`  is not working well with stepwise optimization.
IMHO, Isn't the direction of the comparison operator reversed the following?

https://github.com/pfnet/optuna/blob/v0.18.0/optuna/integration/lightgbm_tuner/optimize.py#L378,L407

## Expected behavior

I expected `time_budget` to behave like `timeout` option of `Study#optimize()`.
In other words, it is specified for the time allowed for optimization.
But current `time_budget` does not behave that so.
If large value is set, only `feature_fraction` is optimized and exited.
NOTE: `feature_fraction` is the first parameter of lightgbm_tuner step-wise optimization, I expect.

Sorry if the above expectation is fundamentally wrong...

## Environment

- Optuna version: 0.18.0
- Python version: 3.7.5
- OS: macOS Mojave (10.14.6)
- LightGBM version: 2.3.0

## Error messages, stack traces, or logs

- Output of step 1 (set a large value to `time_budget`)

```
$ python3 lgbtune.py
...
tune_feature_fraction, val_score: 0.109507:  86%|########5 | 6/7 [00:00<00:00, 17.63it/s][I 2019-11-08 13:03:10,975] Finished trial#6 resulted in value: 0.11621424018474308. Current best value is 0.10950685788917311 with parameters: {'feature_fraction': 0.7}.
tune_feature_fraction, val_score: 0.109507: 100%|##########| 7/7 [00:00<00:00, 16.58it/s]

(Process finished with exit code 0)
```

- Output of step 3 (comment out `time_budget`)

```
$ python3 lgbtune.py
...
...
tune_min_child_samples, val_score: 0.066915:  80%|########  | 4/5 [00:00<00:00, 21.78it/s][I 2019-11-08 13:27:24,351] Finished trial#4 resulted in value: 1.4095717272810777. Current best value is 0.08811113641629516 with parameters: {'min_child_samples': 10}.
tune_min_child_samples, val_score: 0.066915: 100%|##########| 5/5 [00:00<00:00, 19.24it/s]

(Process finished with exit code 0)
```

## Steps to reproduce

1. run the following code (set large value to`time_budget`)
2. edit the following code. (comment out or set to zero: `time_budget`)
3. re-run

## Reproducible examples (optional)

```python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-

from sklearn import datasets
from sklearn.model_selection import train_test_split
from optuna.integration import lightgbm_tuner
import lightgbm as lgb


def main():
    dataset = datasets.load_iris()
    X, y = dataset.data, dataset.target
    X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                        shuffle=True,
                                                        random_state=42)

    lgb_train = lgb.Dataset(X_train, y_train)
    lgb_valid = lgb.Dataset(X_test, y_test, reference=lgb_train)

    lgb_params = {
        'objective': 'regression',
        'metric': 'rmse',
    }

    lightgbm_tuner.train(lgb_params, lgb_train,
                         valid_sets=lgb_valid,
                         num_boost_round=1000,
                         early_stopping_rounds=100,
                         verbose_eval=10,
                         time_budget=10000,
                         )


if __name__ == '__main__':
    main()
```

`time_budget` option on `lightgbm_tuner.train()` is not working well
`time_budget` option on `lightgbm_tuner.train()`  is not working well with stepwise optimization.
IMHO, Isn't the direction of the comparison operator reversed the following?

https://github.com/pfnet/optuna/blob/v0.18.0/optuna/integration/lightgbm_tuner/optimize.py#L378,L407

## Expected behavior

I expected `time_budget` to behave like `timeout` option of `Study#optimize()`.
In other words, it is specified for the time allowed for optimization.
But current `time_budget` does not behave that so.
If large value is set, only `feature_fraction` is optimized and exited.
NOTE: `feature_fraction` is the first parameter of lightgbm_tuner step-wise optimization, I expect.

Sorry if the above expectation is fundamentally wrong...

## Environment

- Optuna version: 0.18.0
- Python version: 3.7.5
- OS: macOS Mojave (10.14.6)
- LightGBM version: 2.3.0

## Error messages, stack traces, or logs

- Output of step 1 (set a large value to `time_budget`)

```
$ python3 lgbtune.py
...
tune_feature_fraction, val_score: 0.109507:  86%|########5 | 6/7 [00:00<00:00, 17.63it/s][I 2019-11-08 13:03:10,975] Finished trial#6 resulted in value: 0.11621424018474308. Current best value is 0.10950685788917311 with parameters: {'feature_fraction': 0.7}.
tune_feature_fraction, val_score: 0.109507: 100%|##########| 7/7 [00:00<00:00, 16.58it/s]

(Process finished with exit code 0)
```

- Output of step 3 (comment out `time_budget`)

```
$ python3 lgbtune.py
...
...
tune_min_child_samples, val_score: 0.066915:  80%|########  | 4/5 [00:00<00:00, 21.78it/s][I 2019-11-08 13:27:24,351] Finished trial#4 resulted in value: 1.4095717272810777. Current best value is 0.08811113641629516 with parameters: {'min_child_samples': 10}.
tune_min_child_samples, val_score: 0.066915: 100%|##########| 5/5 [00:00<00:00, 19.24it/s]

(Process finished with exit code 0)
```

## Steps to reproduce

1. run the following code (set large value to`time_budget`)
2. edit the following code. (comment out or set to zero: `time_budget`)
3. re-run

## Reproducible examples (optional)

```python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-

from sklearn import datasets
from sklearn.model_selection import train_test_split
from optuna.integration import lightgbm_tuner
import lightgbm as lgb


def main():
    dataset = datasets.load_iris()
    X, y = dataset.data, dataset.target
    X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                        shuffle=True,
                                                        random_state=42)

    lgb_train = lgb.Dataset(X_train, y_train)
    lgb_valid = lgb.Dataset(X_test, y_test, reference=lgb_train)

    lgb_params = {
        'objective': 'regression',
        'metric': 'rmse',
    }

    lightgbm_tuner.train(lgb_params, lgb_train,
                         valid_sets=lgb_valid,
                         num_boost_round=1000,
                         early_stopping_rounds=100,
                         verbose_eval=10,
                         time_budget=10000,
                         )


if __name__ == '__main__':
    main()
```



Discussion:

Resolving patch:
diff --git a/optuna/integration/lightgbm_tuner/optimize.py b/optuna/integration/lightgbm_tuner/optimize.py
--- a/optuna/integration/lightgbm_tuner/optimize.py
+++ b/optuna/integration/lightgbm_tuner/optimize.py
@@ -377,32 +377,32 @@ def run(self):
         self.start_time = time.time()
         with _timer() as t:
             self.tune_feature_fraction()
-            if time_budget is not None and time_budget > t.elapsed_secs():
+            if time_budget is not None and time_budget < t.elapsed_secs():
                 self.best_params.update(self._get_params())
                 return self.best_booster
 
             self.tune_num_leaves()
-            if time_budget is not None and time_budget > t.elapsed_secs():
+            if time_budget is not None and time_budget < t.elapsed_secs():
                 self.best_params.update(self._get_params())
                 return self.best_booster
 
             self.tune_bagging()
-            if time_budget is not None and time_budget > t.elapsed_secs():
+            if time_budget is not None and time_budget < t.elapsed_secs():
                 self.best_params.update(self._get_params())
                 return self.best_booster
 
             self.tune_feature_fraction_stage2()
-            if time_budget is not None and time_budget > t.elapsed_secs():
+            if time_budget is not None and time_budget < t.elapsed_secs():
                 self.best_params.update(self._get_params())
                 return self.best_booster
 
             self.tune_regularization_factors()
-            if time_budget is not None and time_budget > t.elapsed_secs():
+            if time_budget is not None and time_budget < t.elapsed_secs():
                 self.best_params.update(self._get_params())
                 return self.best_booster
 
             self.tune_min_data_in_leaf()
-            if time_budget is not None and time_budget > t.elapsed_secs():
+            if time_budget is not None and time_budget < t.elapsed_secs():
                 self.best_params.update(self._get_params())
                 return self.best_booster
 


Test patch: