Repository: autogluon/autogluon
Pull request: Various HPO Cleanup / Fixes
*Issue #, if available:*
#1031

*Description of changes:*

This PR introduces numerous code cleanup / quality of life improvements and major bug fixes to HPO logic.

- **Major Bug Fix**: HPO with bagging did not work correctly. In fact, the hyperparameters found during HPO were completely ignored for all fold models except the first, and instead the default hyperparameters were used. The first fold used the correct hyperparameters, and all other folds used the default hyperparameters. This was detected after the other fixes were made due to more stringent sanity checks on the hyperparameters. It is unknown how long this bug was present, but at minimum since v0.1. This has been fixed by avoiding the copying of the model entirely and instead initializing a new model with the originals parameters after updating based on the search hyperparameters.
- When search space is exhausted, no longer produces a confusing message and instead logs that all possible search spaces have been tried.
- Automatically early stop HPO if 80%+ of trials are failures to avoid hundreds of HPO trials failing in quick succession.
- HPO Searcher code for `self._results` incorrectly pickled configs in a way that the same config could have two different pickles: `{'a': 1, 'b': 2}` vs `{'b': 2, 'a': 1}`. This led to 2 results per config (didn't impact users, but would have impacted anyone extending the searchers). Refactored the pickling process to ensure a config will have a single unique pickle key.
- RandomForest, Linear, KNN, FastText models errored if a user specified a search space, now these models (all models) are able to perform HPO. By default, they have no search space, and I have further added logic to fallback to normal model training if no search space is provided when `hyperparameter_tune` is called. This means RandomForest and KNN will not do HPO by default if HPO is enabled, since their default search space is empty.
- Incremented trial ID's in model names by 1, so the first trial is `\T1` instead of `\T0`. This aligns the indexing with child folds, which start at `F1`.
- Fixed bug in grid searcher caused by returning np.float64 and np.int64 types instead of float and int types. This caused downstream models that did not support np.int64 and np.float64 to crash / spam warnings (XGBoost).
- Trial failures were not being properly tracked, and even if an exception was raised in the trial, the `is_failed` variable would be False. This has been fixed as tracking was necessary for failure based HPO early stop.
- Refactored usage of `_set_cpu_params` in models causing an update to `self.params`. Now this is done in the local variable `params` rather than updating `self.params`.
- Cleaned old HPO search space / parameter specification in LightGBM. The new configs are identical in functionality, while being less cluttered.
- Removed max_reward defaults for ImagePredictor and ObjectDetector that would have caused HPO to stop far earlier than necessary.
- Cleaned ImagePredictor and ObjectDetector HPO logic.
- Fixed CatBoost custom metric support.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.


Problem statement:
Confusing error message: HPO on Random Forest
Hi,

When tuning RF with the following:

```
    rf_options = {
        'max_depth': ag.space.Int(5, 15),
    }
```

I get the following error:

```
Hyperparameter tuning model: RandomForest_BAG_L1 ...
Warning: Exception caused RandomForest_BAG_L1 to fail during hyperparameter tuning... Skipping this model.
Traceback (most recent call last):
  File "D:\Projects\tactory\autogluon\tabular\trainer\abstract_trainer.py", line 1075, in _train_single_full
    hpo_models, hpo_model_performances, hpo_results = model.hyperparameter_tune(X=X, y=y, k_fold=k_fold, scheduler_options=hyperparameter_tune_kwargs, **model_fit_kwargs)
  File "D:\Projects\tactory\autogluon\core\models\abstract\abstract_model.py", line 683, in hyperparameter_tune
    return self._hyperparameter_tune(scheduler_options=scheduler_options, **kwargs)
  File "D:\Projects\tactory\autogluon\core\models\ensemble\stacker_ensemble_model.py", line 166, in _hyperparameter_tune
    return super()._hyperparameter_tune(X=X, y=y, k_fold=k_fold, scheduler_options=scheduler_options, preprocess_kwargs=preprocess_kwargs, **kwargs)
  File "D:\Projects\tactory\autogluon\core\models\ensemble\bagged_ensemble_model.py", line 640, in _hyperparameter_tune
    hpo_models, hpo_model_performances, hpo_results = self.model_base.hyperparameter_tune(X=X_fold, y=y_fold, X_val=X_val_fold, y_val=y_val_fold, scheduler_options=scheduler_options, **kwargs)
  File "D:\Projects\tactory\autogluon\core\models\abstract\abstract_model.py", line 683, in hyperparameter_tune
    return self._hyperparameter_tune(scheduler_options=scheduler_options, **kwargs)
  File "D:\Projects\tactory\autogluon\tabular\models\rf\rf_model.py", line 166, in _hyperparameter_tune
    return skip_hpo(self, **kwargs)
  File "D:\Projects\tactory\autogluon\core\models\abstract\model_trial.py", line 90, in skip_hpo
    fit_and_save_model(model=model, params=dict(), fit_args=fit_model_args, predict_proba_args=predict_proba_args, y_val=y_val, time_start=time.time(), time_limit=time_limit)
  File "D:\Projects\tactory\autogluon\core\models\abstract\model_trial.py", line 73, in fit_and_save_model
    model.fit(**fit_args, time_limit=time_left, reporter=reporter)
  File "D:\Projects\tactory\autogluon\core\models\abstract\abstract_model.py", line 405, in fit
    self._fit(**kwargs)
  File "D:\Projects\tactory\autogluon\tabular\models\rf\rf_model.py", line 117, in _fit
    self.model = self.model.fit(X, y, sample_weight=sample_weight)
  File "C:\Users\john\AppData\Roaming\Python\Python37\site-packages\sklearn\ensemble\_forest.py", line 392, in fit
    for i, t in enumerate(trees))
  File "D:\Anaconda3\envs\tactory\lib\site-packages\joblib\parallel.py", line 1054, in __call__
    self.retrieve()
  File "D:\Anaconda3\envs\tactory\lib\site-packages\joblib\parallel.py", line 933, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "D:\Anaconda3\envs\tactory\lib\multiprocessing\pool.py", line 657, in get
    raise self._value
  File "D:\Anaconda3\envs\tactory\lib\multiprocessing\pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "D:\Anaconda3\envs\tactory\lib\site-packages\joblib\_parallel_backends.py", line 595, in __call__
    return self.func(*args, **kwargs)
  File "D:\Anaconda3\envs\tactory\lib\site-packages\joblib\parallel.py", line 263, in __call__
    for func, args, kwargs in self.items]
  File "D:\Anaconda3\envs\tactory\lib\site-packages\joblib\parallel.py", line 263, in <listcomp>
    for func, args, kwargs in self.items]
  File "C:\Users\john\AppData\Roaming\Python\Python37\site-packages\sklearn\ensemble\_forest.py", line 168, in _parallel_build_trees
    tree.fit(X, y, sample_weight=curr_sample_weight, check_input=False)
  File "C:\Users\john\AppData\Roaming\Python\Python37\site-packages\sklearn\tree\_classes.py", line 894, in fit
    X_idx_sorted=X_idx_sorted)
  File "C:\Users\john\AppData\Roaming\Python\Python37\site-packages\sklearn\tree\_classes.py", line 276, in fit
    if max_depth <= 0:
TypeError: '<=' not supported between instances of 'Int' and 'int'
'<=' not supported between instances of 'Int' and 'int'
```

When trying to tune min_samples_leaf:

```
    rf_options = {
        # 'max_depth': ag.space.Int(5, 15),
        'min_samples_leaf': ag.space.Int(5, 25),
    }
```

I get this error:

```
  File "C:\Users\john\AppData\Roaming\Python\Python37\site-packages\sklearn\tree\_classes.py", line 220, in fit
    if not 0. < self.min_samples_leaf <= 0.5:
TypeError: '<' not supported between instances of 'float' and 'Int'
'<' not supported between instances of 'float' and 'Int'
```

When trying for Real (float):

```
    rf_options = {
        # 'max_depth': ag.space.Int(5, 15),
        'min_samples_leaf': ag.space.Real(5.0, 25.0),
    }
```

I get this error:

```
  File "C:\Users\john\AppData\Roaming\Python\Python37\site-packages\sklearn\tree\_classes.py", line 220, in fit
    if not 0. < self.min_samples_leaf <= 0.5:
TypeError: '<' not supported between instances of 'float' and 'Int'
'<' not supported between instances of 'float' and 'Int'
```

Am i doing something wrong?  Here is how I am calling autogluon:

 ```
   hyperparameters = {
        'RF': rf_options,
        # 'CAT': {},
        # 'XGB': {},
        # 'GBM': gbm_options,
        # 'XT': {},
        # 'KNN': {},
        # 'LR': {},
        # 'NN': {},
        # 'FASTAI': {},
    }

    # HPO is not performed unless hyperparameter_tune_kwargs is specified
    hyperparameter_tune_kwargs = {
        'num_trials': 5,
        'searcher': 'auto',
    }

    predictor = TabularPredictor(
        label='label',
        eval_metric='accuracy',
        path='state/autogluon').fit(X,
        # auto_stack=True,
        num_bag_folds=5,
        num_bag_sets=2, 
        num_stack_levels=3,
        hyperparameters=hyperparameters,
        hyperparameter_tune_kwargs=hyperparameter_tune_kwargs,
        time_limit=300)
```

Autogluon version:

```
autogluon==0.1.0
autogluon-contrib-nlp==0.0.1b20210201
autogluon.core==0.1.0
autogluon.extra==0.1.0
autogluon.features==0.1.0
autogluon.mxnet==0.1.0
autogluon.tabular==0.1.0
autogluon.text==0.1.0
autogluon.vision==0.1.0


```

Hints from issue discussion:
Ah, good catch!

HPO is currently disabled for RF/XT/KNN models as most users are better off tuning the other model types (GBM, CAT, XGB, NN). Tuning RF/XT/KNN has a major risk of taking up many GB or TB of disk space. It seems the error message users get when trying to do HPO on them is not clear however, so this is a bug.

Commit message: Cleaned HPO code of Vision models, removed default 'max_reward' limit

Patch to review:
diff --git a/core/src/autogluon/core/models/abstract/abstract_model.py b/core/src/autogluon/core/models/abstract/abstract_model.py
index af789dde2e75..e190887e1e74 100644
--- a/core/src/autogluon/core/models/abstract/abstract_model.py
+++ b/core/src/autogluon/core/models/abstract/abstract_model.py
@@ -18,11 +18,11 @@
 from autogluon.common.utils.pandas_utils import get_approximate_df_mem_usage
 from autogluon.common.utils.utils import setup_outputdir
 
-from ....core.data.label_cleaner import LabelCleaner, LabelCleanerMulticlassToBinary
+from .model_trial import model_trial, skip_hpo
 from ._tags import _DEFAULT_TAGS
-from .model_trial import model_trial
 from ... import metrics, Space
 from ...constants import AG_ARGS_FIT, BINARY, REGRESSION, QUANTILE, REFIT_FULL_SUFFIX, OBJECTIVES_TO_NORMALIZE
+from ...data.label_cleaner import LabelCleaner, LabelCleanerMulticlassToBinary
 from ...scheduler import LocalSequentialScheduler
 from ...utils import get_cpu_count, get_pred_from_proba, normalize_pred_probas, infer_eval_metric, infer_problem_type, \
     compute_permutation_feature_importance, compute_weighted_metric
@@ -247,7 +247,7 @@ def _get_default_searchspace(self) -> dict:
         """
         return {}
 
-    def _set_default_searchspace(self):
+    def _get_search_space(self):
         """ Sets up default search space for HPO. Each hyperparameter which user did not specify is converted from
             default fixed value to default search space.
         """
@@ -255,8 +255,9 @@ def _set_default_searchspace(self):
         # Note: when subclassing AbstractModel, you must define or import get_default_searchspace() from the appropriate location.
         for key in self.nondefault_params:  # delete all user-specified hyperparams from the default search space
             def_search_space.pop(key, None)
-        if self.params is not None:
-            self.params.update(def_search_space)
+        params = self._get_params()
+        params.update(def_search_space)
+        return params
 
     # TODO: v0.1 Change this to update path_root only, path change to property
     def set_contexts(self, path_context):
@@ -874,8 +875,18 @@ def _hyperparameter_tune(self, X, y, X_val, y_val, scheduler_options, **kwargs):
         # verbosity = kwargs.get('verbosity', 2)
         time_start = time.time()
         logger.log(15, "Starting generic AbstractModel hyperparameter tuning for %s model..." % self.name)
-        self._set_default_searchspace()
-        params_copy = self._get_params()
+        search_space = self._get_search_space()
+
+        if not any(isinstance(search_space[hyperparam], Space) for hyperparam in search_space):
+            logger.warning(f"\tNo hyperparameter search space specified for {self.name}. Skipping HPO. "
+                           f"Will train one model based on the provided hyperparameters.")
+            return skip_hpo(self, X=X, y=y, X_val=X_val, y_val=y_val, **kwargs)
+        else:
+            logger.log(15, f"\tHyperparameter search space for {self.name}: ")
+            for hyperparam in search_space:
+                if isinstance(search_space[hyperparam], Space):
+                    logger.log(15, f"{hyperparam}:   {search_space[hyperparam]}")
+
         directory = self.path  # also create model directory if it doesn't exist
         # TODO: This will break on S3. Use tabular/utils/savers for datasets, add new function
         scheduler_cls, scheduler_params = scheduler_options  # Unpack tuple
@@ -889,19 +900,17 @@ def _hyperparameter_tune(self, X, y, X_val, y_val, scheduler_options, **kwargs):
         val_path = directory + dataset_val_filename
         save_pkl.save(path=val_path, object=(X_val, y_val))
 
-        if not any(isinstance(params_copy[hyperparam], Space) for hyperparam in params_copy):
-            logger.warning("Attempting to do hyperparameter optimization without any search space (all hyperparameters are already fixed values)")
-        else:
-            logger.log(15, "Hyperparameter search space for %s model: " % self.name)
-            for hyperparam in params_copy:
-                if isinstance(params_copy[hyperparam], Space):
-                    logger.log(15, f"{hyperparam}:   {params_copy[hyperparam]}")
+        model_cls = self.__class__
+        init_params = self.get_params()
 
         fit_kwargs = scheduler_params['resource'].copy()
+        fit_kwargs['feature_metadata'] = self.feature_metadata
+        fit_kwargs['num_classes'] = self.num_classes
         fit_kwargs['sample_weight'] = kwargs.get('sample_weight', None)
         fit_kwargs['sample_weight_val'] = kwargs.get('sample_weight_val', None)
         train_fn_kwargs = dict(
-            model=self,
+            model_cls=model_cls,
+            init_params=init_params,
             time_start=time_start,
             time_limit=scheduler_params['time_out'],
             fit_kwargs=fit_kwargs,
@@ -909,7 +918,7 @@ def _hyperparameter_tune(self, X, y, X_val, y_val, scheduler_options, **kwargs):
             val_path=val_path,
         )
 
-        scheduler: LocalSequentialScheduler = scheduler_cls(model_trial, search_space=params_copy, train_fn_kwargs=train_fn_kwargs, **scheduler_params)
+        scheduler: LocalSequentialScheduler = scheduler_cls(model_trial, search_space=search_space, train_fn_kwargs=train_fn_kwargs, **scheduler_params)
 
         scheduler.run()
         scheduler.join_jobs()
@@ -933,7 +942,7 @@ def _get_hpo_results(self, scheduler, scheduler_params: dict, time_start):
         hpo_model_performances = {}
         for trial in sorted(hpo_results['config_history'].keys()):
             # TODO: ignore models which were killed early by scheduler (eg. in Hyperband). How to ID these?
-            file_id = f"T{trial}"  # unique identifier to files from this trial
+            file_id = f"T{trial+1}"  # unique identifier to files from this trial
             trial_model_name = self.name + os.path.sep + file_id
             trial_model_path = self.path_root + trial_model_name + os.path.sep
             trial_reward = scheduler.searcher.get_reward(hpo_results['config_history'][trial])
diff --git a/core/src/autogluon/core/models/abstract/model_trial.py b/core/src/autogluon/core/models/abstract/model_trial.py
index 4bbb1e85c190..19c50ee54821 100644
--- a/core/src/autogluon/core/models/abstract/model_trial.py
+++ b/core/src/autogluon/core/models/abstract/model_trial.py
@@ -10,27 +10,20 @@
 
 def model_trial(args,
                 reporter,
-                model,
+                model_cls,
+                init_params,
                 train_path,
                 val_path,
                 time_start,
                 time_limit=None,
                 fit_kwargs=None,
                 ):
-    """ Training script for hyperparameter evaluation of an arbitrary model that subclasses AbstractModel.
-
-        Notes:
-            - Model object itself must be passed as kwarg: model
-            - All model hyperparameters must be stored in model.params dict that may contain special keys such as:
-                'seed_value' to ensure reproducibility
-                'num_threads', 'num_gpus' to set specific resources in model.fit()
-            - model.save() must have return_filename, file_prefix, directory options
-    """
+    """ Training script for hyperparameter evaluation of an arbitrary model that subclasses AbstractModel."""
     try:
         if fit_kwargs is None:
             fit_kwargs = dict()
 
-        model, args = prepare_inputs(args=args, model=model)
+        model = init_model(args=args, model_cls=model_cls, init_params=init_params)
 
         X, y = load_pkl.load(train_path)
         X_val, y_val = load_pkl.load(val_path)
@@ -39,7 +32,6 @@ def model_trial(args,
         predict_proba_args = dict(X=X_val)
         model = fit_and_save_model(
             model=model,
-            params=args,
             fit_args=fit_model_args,
             predict_proba_args=predict_proba_args,
             y_val=y_val,
@@ -55,16 +47,19 @@ def model_trial(args,
         reporter(epoch=1, validation_performance=model.val_score)
 
 
-def prepare_inputs(args, model):
+def init_model(args, model_cls, init_params):
+    args = args.copy()
     task_id = args.pop('task_id')
 
-    file_prefix = f"T{task_id}"  # append to all file names created during this trial. Do NOT change!
-    model.name = model.name + os.path.sep + file_prefix
-    model.set_contexts(path_context=model.path_root + model.name + os.path.sep)
-    return model, args
+    file_prefix = f"T{task_id+1}"  # append to all file names created during this trial. Do NOT change!
 
+    init_params['hyperparameters'].update(args)
+    init_params['name'] = init_params['name'] + os.path.sep + file_prefix
 
-def fit_and_save_model(model, params, fit_args, predict_proba_args, y_val, time_start, time_limit=None, reporter=None):
+    return model_cls(**init_params)
+
+
+def fit_and_save_model(model, fit_args, predict_proba_args, y_val, time_start, time_limit=None, reporter=None):
     time_current = time.time()
     time_elapsed = time_current - time_start
     if time_limit is not None:
@@ -74,7 +69,6 @@ def fit_and_save_model(model, params, fit_args, predict_proba_args, y_val, time_
     else:
         time_left = None
 
-    model.params.update(params)
     time_fit_start = time.time()
     model.fit(**fit_args, time_limit=time_left, reporter=reporter)
     time_fit_end = time.time()
@@ -97,11 +91,18 @@ def fit_and_save_model(model, params, fit_args, predict_proba_args, y_val, time_
     return model
 
 
-def skip_hpo(model, X, y, X_val, y_val, scheduler_options=None, time_limit=None, **kwargs):
+def skip_hpo(model, X, y, X_val, y_val, time_limit=None, **kwargs):
     """Skips HPO and simply trains the model once with the provided HPO time budget. Returns model artifacts as if from HPO."""
     fit_model_args = dict(X=X, y=y, **kwargs)
     predict_proba_args = dict(X=X_val)
-    fit_and_save_model(model=model, params=dict(), fit_args=fit_model_args, predict_proba_args=predict_proba_args, y_val=y_val, time_start=time.time(), time_limit=time_limit)
+    fit_and_save_model(
+        model=model,
+        fit_args=fit_model_args,
+        predict_proba_args=predict_proba_args,
+        y_val=y_val,
+        time_start=time.time(),
+        time_limit=time_limit
+    )
     hpo_results = {'total_time': model.fit_time}
     hpo_model_performances = {model.name: model.val_score}
     hpo_models = {model.name: model.path}
diff --git a/core/src/autogluon/core/models/ensemble/bagged_ensemble_model.py b/core/src/autogluon/core/models/ensemble/bagged_ensemble_model.py
index 64b365e4d63d..46a47d5c6497 100644
--- a/core/src/autogluon/core/models/ensemble/bagged_ensemble_model.py
+++ b/core/src/autogluon/core/models/ensemble/bagged_ensemble_model.py
@@ -961,7 +961,7 @@ def _hyperparameter_tune(self, X, y, k_fold, scheduler_options, preprocess_kwarg
 
             # TODO: Create new Ensemble Here
             bag = copy.deepcopy(self)
-            bag.rename(f"{bag.name}{os.path.sep}T{i}")
+            bag.rename(f"{bag.name}{os.path.sep}T{i+1}")
             bag.set_contexts(self.path_root + bag.name + os.path.sep)
 
             oof_pred_proba, oof_pred_model_repeats = self._construct_empty_oof(X=X, y=y)
diff --git a/core/src/autogluon/core/models/greedy_ensemble/greedy_weighted_ensemble_model.py b/core/src/autogluon/core/models/greedy_ensemble/greedy_weighted_ensemble_model.py
index 2d4a31ee1d45..e164056abe78 100644
--- a/core/src/autogluon/core/models/greedy_ensemble/greedy_weighted_ensemble_model.py
+++ b/core/src/autogluon/core/models/greedy_ensemble/greedy_weighted_ensemble_model.py
@@ -4,7 +4,6 @@
 
 from .ensemble_selection import EnsembleSelection, SimpleWeightedEnsemble
 from ..abstract.abstract_model import AbstractModel
-from ..abstract.model_trial import skip_hpo
 from ...constants import MULTICLASS, SOFTCLASS, QUANTILE
 
 logger = logging.getLogger(__name__)
@@ -132,9 +131,6 @@ def _get_default_ag_args(cls) -> dict:
     def _get_default_stopping_metric(self):
         return self.eval_metric
 
-    def _hyperparameter_tune(self, **kwargs):
-        return skip_hpo(self, **kwargs)
-
 
 class SimpleWeightedEnsembleModel(GreedyWeightedEnsembleModel):
     def __init__(self, model_base=SimpleWeightedEnsemble, **kwargs):
diff --git a/core/src/autogluon/core/scheduler/seq_scheduler.py b/core/src/autogluon/core/scheduler/seq_scheduler.py
index 03b638aa63d8..1696af3970e5 100644
--- a/core/src/autogluon/core/scheduler/seq_scheduler.py
+++ b/core/src/autogluon/core/scheduler/seq_scheduler.py
@@ -5,11 +5,11 @@
 from copy import deepcopy
 from typing import Tuple
 
-import numpy as np
 from tqdm.auto import tqdm
 
 from .reporter import FakeReporter
 from ..searcher import searcher_factory
+from ..searcher.exceptions import ExhaustedSearchSpaceError
 from ..searcher.local_searcher import LocalSearcher
 
 logger = logging.getLogger(__name__)
@@ -147,7 +147,11 @@ def run(self, **kwargs):
         self.training_history = OrderedDict()
         self.config_history = OrderedDict()
 
-        trial_run_times = []
+        failure_count = 0
+        trial_count = 0
+        trial_run_times = 0
+        min_failure_threshold = 5
+        failure_rate_threshold = 0.8
         time_start = time.time()
 
         r = range(self.num_trials)
@@ -155,23 +159,36 @@ def run(self, **kwargs):
             trial_start_time = time.time()
             try:
                 is_failed, result = self.run_trial(task_id=i)
+            except ExhaustedSearchSpaceError:
+                break
             except Exception:
                 # TODO: Add special exception type when there are no more new configurations to try (exhausted search space)
                 logger.log(30, f'\tWARNING: Encountered unexpected exception during trial {i}, stopping HPO early.')
                 logger.exception('Detailed Traceback:')  # TODO: Avoid logging if verbosity=0
                 break
             trial_end_time = time.time()
-            trial_run_times.append(np.NaN if is_failed else (trial_end_time - trial_start_time))
+
+            trial_count += 1
+            if is_failed:
+                failure_count += 1
+            else:
+                trial_run_times += trial_end_time - trial_start_time
 
             if self.max_reward and self.get_best_reward() >= self.max_reward:
-                logger.log(20, f'\tMax reward is reached')
+                logger.log(20, f'\tStopping HPO: Max reward reached')
+                break
+
+            if failure_count >= min_failure_threshold and (failure_count / trial_count) >= failure_rate_threshold:
+                logger.warning(f'Warning: Detected a large trial failure rate: '
+                               f'{failure_count}/{trial_count} attempted trials failed ({round((failure_count / trial_count) * 100, 1)}%)! '
+                               f'Stopping HPO early due to reaching failure threshold ({round(failure_rate_threshold*100, 1)}%).\n'
+                               f'\tFailures may be caused by invalid configurations within the provided search space.')
                 break
 
             if self.time_out is not None:
-                avg_trial_run_time = np.nanmean(trial_run_times)
-                avg_trial_run_time = 0 if np.isnan(avg_trial_run_time) else avg_trial_run_time
+                avg_trial_run_time = 0 if trial_count == failure_count else trial_run_times / (trial_count - failure_count)
                 if not self.has_enough_time_for_trial_(self.time_out, time_start, trial_start_time, trial_end_time, avg_trial_run_time):
-                    logger.log(20, f'\tTime limit exceeded')
+                    logger.log(20, f'\tStopping HPO to satisfy time limit...')
                     break
 
     @classmethod
@@ -244,6 +261,8 @@ def run_trial(self, task_id=0) -> Tuple[bool, dict]:
     def run_job_(self, task_id, searcher_config, reporter):
         args = dict()
         if self.train_fn_kwargs is not None:
+            # TODO: Consider avoiding deepcopy and just do shallow copy,
+            #  Risk is that it will allow values in self.train_fn_kwargs to be altered by HPO trials, causing early trials to alter later trials.
             train_fn_kwargs = deepcopy(self.train_fn_kwargs)
         else:
             train_fn_kwargs = dict()
@@ -254,8 +273,11 @@ def run_job_(self, task_id, searcher_config, reporter):
         is_failed = False
         try:
             result = self.train_fn(args, reporter=reporter, **train_fn_kwargs)
-            if type(reporter) is not FakeReporter and reporter.last_result:
-                self.searcher.update(config=searcher_config, **reporter.last_result)
+            if type(reporter) is not FakeReporter:
+                if reporter.last_result:
+                    self.searcher.update(config=searcher_config, **reporter.last_result)
+                else:
+                    is_failed = True
         except Exception as e:
             logger.error(f'Exception during a trial: {e}')
             self.searcher.evaluation_failed(config=searcher_config)
diff --git a/core/src/autogluon/core/searcher/__init__.py b/core/src/autogluon/core/searcher/__init__.py
index 064b8b2c9fab..5e39c9471918 100644
--- a/core/src/autogluon/core/searcher/__init__.py
+++ b/core/src/autogluon/core/searcher/__init__.py
@@ -1,3 +1,4 @@
+from .dummy_searcher import DummySearcher
 from .local_grid_searcher import LocalGridSearcher
 from .local_random_searcher import LocalRandomSearcher
 from .searcher_factory import searcher_factory
diff --git a/core/src/autogluon/core/searcher/dummy_searcher.py b/core/src/autogluon/core/searcher/dummy_searcher.py
new file mode 100644
index 000000000000..8565a743296b
--- /dev/null
+++ b/core/src/autogluon/core/searcher/dummy_searcher.py
@@ -0,0 +1,21 @@
+import logging
+
+from .local_searcher import LocalSearcher
+from .exceptions import ExhaustedSearchSpaceError
+
+__all__ = ['DummySearcher']
+
+logger = logging.getLogger(__name__)
+
+
+class DummySearcher(LocalSearcher):
+    """Searcher which only returns the default config."""
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        self._exhausted = False
+
+    def get_config(self, **kwargs) -> dict:
+        if self._exhausted:
+            raise ExhaustedSearchSpaceError('Default config already provided. Search space is exhausted!')
+        self._exhausted = True
+        return self._params_default
diff --git a/core/src/autogluon/core/searcher/exceptions.py b/core/src/autogluon/core/searcher/exceptions.py
new file mode 100644
index 000000000000..ff19dee475db
--- /dev/null
+++ b/core/src/autogluon/core/searcher/exceptions.py
@@ -0,0 +1,4 @@
+
+
+class ExhaustedSearchSpaceError(Exception):
+    pass
diff --git a/core/src/autogluon/core/searcher/local_grid_searcher.py b/core/src/autogluon/core/searcher/local_grid_searcher.py
index 4d15b2cc477d..1b800e362887 100644
--- a/core/src/autogluon/core/searcher/local_grid_searcher.py
+++ b/core/src/autogluon/core/searcher/local_grid_searcher.py
@@ -69,4 +69,10 @@ def get_config(self):
             raise AssertionError(f'No configs left to get. All {self._grid_length} configs have been accessed already.')
         config = self._params_grid[self._grid_index]
         self._grid_index += 1
+        for key, val in config.items():
+            # If this isn't done, warnings are spammed in XGBoost
+            if isinstance(val, np.int64):
+                config[key] = int(val)
+            elif isinstance(val, np.float64):
+                config[key] = float(val)
         return config
diff --git a/core/src/autogluon/core/searcher/local_random_searcher.py b/core/src/autogluon/core/searcher/local_random_searcher.py
index cd5d32fa7c74..d771cf4f38f8 100644
--- a/core/src/autogluon/core/searcher/local_random_searcher.py
+++ b/core/src/autogluon/core/searcher/local_random_searcher.py
@@ -1,11 +1,11 @@
 import logging
-import pickle
 
 import numpy as np
 from sklearn.model_selection import ParameterSampler
 
 from .local_searcher import LocalSearcher
-from ..space import Space
+from .exceptions import ExhaustedSearchSpaceError
+from ..space import DiscreteSpace, Space
 
 __all__ = ['LocalRandomSearcher']
 
@@ -22,6 +22,7 @@ def __init__(self, *, first_is_default=True, random_seed=0, **kwargs):
         # We use an explicit random_state here, in order to better support checkpoint and resume
         self.random_state = np.random.RandomState(random_seed)
         self._params_space = self._get_params_space()
+        self._num_configs = self._get_num_configs()
 
     def _get_params_space(self) -> dict:
         param_space = dict()
@@ -31,12 +32,24 @@ def _get_params_space(self) -> dict:
                 param_space[key] = sk
         return param_space
 
+    def _get_num_configs(self) -> int:
+        num_unique = 1
+        for key, val in self.search_space.items():
+            if isinstance(val, Space):
+                if isinstance(val, DiscreteSpace):
+                    num_unique *= len(val)
+                else:
+                    num_unique = None
+                    break
+        return num_unique
+
     def _sample_config(self) -> dict:
         params = list(ParameterSampler(self._params_space, n_iter=1, random_state=self.random_state))[0]
         for key in params:
             if isinstance(params[key], np.float64):
                 # Fix error in FastAI, can't handle np.float64
                 params[key] = float(params[key])
+        params.update(self._params_static)
         return params
 
     def get_config(self, **kwargs) -> dict:
@@ -52,9 +65,14 @@ def get_config(self, **kwargs) -> dict:
         else:
             new_config = self._sample_config()
         num_tries = 1
-        while pickle.dumps(new_config) in self._results:
-            assert num_tries <= self.MAX_RETRIES, f"Cannot find new config in LocalRandomSearcher, even after {self.MAX_RETRIES} trials"
+        while self._pickle_config(new_config) in self._results:
+            if num_tries > self.MAX_RETRIES:
+                if self._num_configs is not None:
+                    num_results = len(self._results)
+                    logger.log(30, f'Stopping HPO due to exhausted search space: {num_results} of {self._num_configs} possible configs ran.')
+                    raise ExhaustedSearchSpaceError
+                assert num_tries <= self.MAX_RETRIES, f"Cannot find new config in LocalRandomSearcher, even after {self.MAX_RETRIES} trials"
             new_config = self._sample_config()
             num_tries += 1
-        self._results[pickle.dumps(new_config)] = self._reward_while_pending()
+        self._add_result(new_config, self._reward_while_pending())
         return new_config
diff --git a/core/src/autogluon/core/searcher/local_searcher.py b/core/src/autogluon/core/searcher/local_searcher.py
index 130938f0b22b..8a4362695722 100644
--- a/core/src/autogluon/core/searcher/local_searcher.py
+++ b/core/src/autogluon/core/searcher/local_searcher.py
@@ -18,32 +18,20 @@ class LocalSearcher(object):
         The configuration space to sample from. It contains the full
         specification of the Hyperparameters with their priors
     """
-    def __init__(self, search_space, reward_attribute=None, **kwargs):
+    def __init__(self, search_space: dict, reward_attribute: str = 'reward', **kwargs):
         """
         :param search_space: Configuration space to sample from or search in
         :param reward_attribute: Reward attribute passed to update.
-            Default: 'accuracy'
+            Default: 'reward'
 
         """
         self.search_space = search_space
         self._results = OrderedDict()
-        if reward_attribute is None:
-            reward_attribute = 'accuracy'
         self._reward_attribute = reward_attribute
-        self._params_default = self._get_params_default()
-
-    def _get_params_default(self) -> dict:
-        params_default = dict()
-        for key, val in self.search_space.items():
-            if isinstance(val, Space):
-                if isinstance(val, Categorical):
-                    # FIXME: Don't do this, fix the outer code to not require this
-                    d = val.data[0]
-                else:
-                    d = val.default
-                params_default[key] = d
-        return params_default
-
+        self._params_static = self._get_params_static()
+        self._params_default = self._get_params_default(self._params_static)
+        self._params_order = list(self._params_default.keys())
+        self._params_cat_dict = self._get_params_cat_dict()
 
     # FIXME: Consider removing
     def configure_scheduler(self, scheduler):
@@ -82,12 +70,14 @@ def get_config(self, **kwargs):
         """
         raise NotImplementedError(f'This function needs to be overwritten in {self.__class__.__name__}.')
 
-    def update(self, config, **kwargs):
-        """Update the searcher with the newest metric report"""
+    def update(self, config: dict, **kwargs):
+        """
+        Update the searcher with the newest metric report.
+        Will error if config contains unknown parameters, values outside the valid search space, or is missing parameters.
+        """
         reward = kwargs.get(self._reward_attribute, None)
         assert reward is not None, "Missing reward attribute '{}'".format(self._reward_attribute)
-        config_pkl = pickle.dumps(config)
-        self._results[config_pkl] = reward
+        self._add_result(config=config, result=reward)
 
     def register_pending(self, config, milestone=None):
         """
@@ -121,15 +111,102 @@ def get_best_reward(self):
     def get_reward(self, config):
         """Calculates the reward (i.e. validation performance) produced by training with the given configuration.
         """
-        k = pickle.dumps(config)
-        assert k in self._results
-        return self._results[k]
+        config_pkl = self._pickle_config(config=config)
+        assert config_pkl in self._results
+        return self._results[config_pkl]
 
     def get_best_config(self):
         """Returns the best configuration found so far.
         """
         if self._results:
             config_pkl = max(self._results, key=self._results.get)
-            return pickle.loads(config_pkl)
+            return self._unpickle_config(config_pkl=config_pkl)
         else:
             return dict()
+
+    def _get_params_static(self) -> dict:
+        """
+        Gets a dictionary of static key values, where no search space is used and therefore the values are always the same in all configs.
+        """
+        params_static = dict()
+        for key, val in self.search_space.items():
+            if not isinstance(val, Space):
+                params_static[key] = val
+        return params_static
+
+    def _get_params_default(self, params_static: dict) -> dict:
+        """
+        Gets the default config by calling `val.default` on every search space parameter, plus the static key values.
+        """
+        params_default = dict()
+        for key, val in self.search_space.items():
+            if isinstance(val, Space):
+                params_default[key] = val.default
+        params_default.update(params_static)
+        return params_default
+
+    def _get_params_cat_dict(self) -> dict:
+        """
+        Gets the dictionary of pickled category value -> index mapping for Category search spaces.
+        This is used in `self._pickle_config` to map values to idx when pickling the config. This compresses the size of the pkl file.
+        When being later unpickled via `self._unpickle_config`, the idx can be used to get the key value via `self.search_space[key][idx]`.
+        """
+        params_cat_dict = dict()
+        for key, val in self.search_space.items():
+            if isinstance(val, Categorical):
+                cat_map = dict()
+                for i, cat in enumerate(val.data):
+                    cat_pkl = pickle.dumps(cat)
+                    cat_map[cat_pkl] = i
+
+                params_cat_dict[key] = cat_map
+        return params_cat_dict
+
+    def _add_result(self, config: dict, result: float):
+        assert isinstance(result, (float, int)), f"result must be a float or int! Was instead {type(result)} | Value: {result}"
+        config_pkl = self._pickle_config(config=config)
+        self._results[config_pkl] = result
+
+    def _pickle_config(self, config: dict) -> bytes:
+        assert isinstance(config, dict), f"config must be a dict! Was instead {type(config)} | Value: {config}"
+        assert len(config) == len(self._params_order), f'Config length does not match expected params count!\n' \
+                                                       f'Expected: {self._params_order}\n' \
+                                                       f'Actual:   {list(config.keys())}'
+
+        # Note: This code is commented out because it can be computationally and memory expensive if user sends large objects in search space, such as datasets.
+        """
+        for key in self._params_static:
+            assert pickle.dumps(config[key]) == pickle.dumps(self._params_static[key]), \
+                f'Invalid config value for search space parameter "{key}" | Invalid Value: {config[key]} | Expected Value: {self._params_static[key]}'
+        """
+        config_to_pkl = []
+        for key in self._params_order:
+            if key in self._params_static:
+                pass
+            elif key in self._params_cat_dict:
+                try:
+                    cat_idx = self._params_cat_dict[key][pickle.dumps(config[key])]
+                except KeyError:
+                    raise AssertionError(f'Invalid config value for search space parameter "{key}" | '
+                                         f'Invalid Value: {config[key]} | Valid Values: {self.search_space[key].data}')
+                config_to_pkl.append(cat_idx)
+            else:
+                config_to_pkl.append(config[key])
+        return pickle.dumps(config_to_pkl)
+
+    def _unpickle_config(self, config_pkl: bytes) -> dict:
+        assert isinstance(config_pkl, bytes), f"config_pkl must be a bytes object! Was instead {type(config_pkl)} | Value: {config_pkl}"
+        config_compressed = pickle.loads(config_pkl)
+        config = dict()
+        i = -1
+        for key in self._params_order:
+            if key in self._params_static:
+                config[key] = self._params_static[key]
+            else:
+                i += 1
+                val = config_compressed[i]
+                if key in self._params_cat_dict:
+                    config[key] = self.search_space[key][val]
+                else:
+                    config[key] = val
+        return config
diff --git a/core/src/autogluon/core/space.py b/core/src/autogluon/core/space.py
index 92b57b9dd709..b46dbf91b8c6 100644
--- a/core/src/autogluon/core/space.py
+++ b/core/src/autogluon/core/space.py
@@ -36,7 +36,16 @@ def default(self, value):
         self._default = value
 
 
-class Categorical(SimpleSpace):
+class DiscreteSpace(SimpleSpace):
+    """
+    Search space with the requirement of having a discrete number of options, such that it is possible to exhaust the search space.
+    """
+    def __len__(self) -> int:
+        """Returns the number of unique spaces within the discrete search space."""
+        raise NotImplementedError
+
+
+class Categorical(DiscreteSpace):
     """Nested search space for hyperparameters which are categorical. Such a hyperparameter takes one value out of the discrete set of provided options. 
        The first value in the list of options will be the default value that gets tried first during HPO.
 
@@ -115,7 +124,7 @@ def convert_to_sklearn(self):
         return sampler
 
 
-class Int(SimpleSpace):
+class Int(DiscreteSpace):
     """Search space for numeric hyperparameter that takes integer values.
 
     Parameters
@@ -143,6 +152,9 @@ def convert_to_sklearn(self):
         from scipy.stats import randint
         return randint(self.lower, self.upper+1)
 
+    def __len__(self):
+        return self.upper - self.lower + 1
+
 
 class Bool(Int):
     """Search space for hyperparameter that is either True or False. 
diff --git a/core/src/autogluon/core/task/base/base_task.py b/core/src/autogluon/core/task/base/base_task.py
index 0dfd9e588a7a..a8387fc6e6d2 100644
--- a/core/src/autogluon/core/task/base/base_task.py
+++ b/core/src/autogluon/core/task/base/base_task.py
@@ -19,25 +19,25 @@
 logger = logging.getLogger(__name__)
 
 
-def create_scheduler(train_fn, search_space, scheduler, scheduler_options):
+def create_scheduler(train_fn, search_space, scheduler, scheduler_options, train_fn_kwargs=None):
     if isinstance(scheduler, str):
         scheduler_cls = schedulers[scheduler.lower()]
     else:
         assert callable(scheduler)
         scheduler_cls = scheduler
         scheduler_options = copy.copy(scheduler_options)
-    return scheduler_cls(train_fn, search_space=search_space, **scheduler_options)
+    return scheduler_cls(train_fn, search_space=search_space, train_fn_kwargs=train_fn_kwargs, **scheduler_options)
 
 
 # FIXME: REMOVE THIS, first GluonCV needs to stop depending on AG, as it imports this class
 class BaseTask(object):
     """BaseTask for AutoGluon applications"""
     @classmethod
-    def run_fit(cls, train_fn, search_space, search_strategy, scheduler_options,
+    def run_fit(cls, train_fn, search_space, search_strategy, scheduler_options, train_fn_kwargs=None,
                 plot_results=False):
         start_time = time.time()
         # create scheduler and schedule tasks
-        scheduler = create_scheduler(train_fn, search_space, search_strategy, scheduler_options)
+        scheduler = create_scheduler(train_fn, search_space, search_strategy, scheduler_options, train_fn_kwargs=train_fn_kwargs)
         scheduler.run()
         scheduler.join_jobs()
         # gather the best configuration
@@ -47,7 +47,7 @@ def run_fit(cls, train_fn, search_space, search_strategy, scheduler_options,
         best_config_run['final_fit'] = True
         if hasattr(best_config_run, 'epochs') and hasattr(best_config_run, 'final_fit_epochs'):
             best_config_run['epochs'] = best_config_run['final_fit_epochs']
-        scheduler_final = create_scheduler(train_fn, search_space, search_strategy, scheduler_options)
+        scheduler_final = create_scheduler(train_fn, search_space, search_strategy, scheduler_options, train_fn_kwargs=train_fn_kwargs)
         results = scheduler_final.run_with_config(best_config_run)
         total_time = time.time() - start_time
         if plot_results or in_ipynb():
diff --git a/core/tests/unittests/searcher/test_local_grid_searcher.py b/core/tests/unittests/searcher/test_local_grid_searcher.py
index a3bd42654536..2791d3851bf4 100644
--- a/core/tests/unittests/searcher/test_local_grid_searcher.py
+++ b/core/tests/unittests/searcher/test_local_grid_searcher.py
@@ -19,7 +19,7 @@ def test_local_grid_searcher_categorical():
     expected_config_6 = {'b': 15, 'a': ['hello', 2]}
 
     config1 = searcher.get_config()
-    searcher.update(config1, accuracy=0.2)
+    searcher.update(config1, reward=0.2)
     assert searcher.get_reward(config1) == 0.2
     assert searcher.get_best_reward() == 0.2
     assert searcher.get_best_config() == config1
@@ -29,11 +29,11 @@ def test_local_grid_searcher_categorical():
     config3 = searcher.get_config()
 
     config4 = searcher.get_config()
-    searcher.update(config4, accuracy=0.1)
+    searcher.update(config4, reward=0.1)
     assert searcher.get_reward(config4) == 0.1
     assert searcher.get_best_reward() == 0.2
     assert searcher.get_best_config() == config1
-    searcher.update(config4, accuracy=0.5)
+    searcher.update(config4, reward=0.5)
     assert searcher.get_reward(config4) == 0.5
     assert searcher.get_best_reward() == 0.5
     assert searcher.get_best_config() == config4
@@ -73,7 +73,7 @@ def test_local_grid_searcher_numeric():
             try:
                 cfg = searcher.get_config()
                 actual_values.append(cfg)
-                searcher.update(cfg, accuracy=0.1)
+                searcher.update(cfg, reward=0.1)
             except AssertionError as e:
                 assert expected_values == actual_values
                 break
@@ -98,7 +98,7 @@ def test_local_grid_searcher_numeric_grid_settings():
             try:
                 cfg = searcher.get_config()
                 actual_values.append(cfg)
-                searcher.update(cfg, accuracy=0.1)
+                searcher.update(cfg, reward=0.1)
             except AssertionError as e:
                 assert expected_values == actual_values
                 break
diff --git a/core/tests/unittests/searcher/test_local_random_searcher.py b/core/tests/unittests/searcher/test_local_random_searcher.py
index 6d44ae97867f..9dd8ec452700 100644
--- a/core/tests/unittests/searcher/test_local_random_searcher.py
+++ b/core/tests/unittests/searcher/test_local_random_searcher.py
@@ -23,7 +23,7 @@ def test_local_random_searcher():
     config1 = searcher.get_config()
     assert searcher.get_reward(config1) == float("-inf")
     assert searcher.get_best_reward() == float("-inf")
-    searcher.update(config1, accuracy=0.2)
+    searcher.update(config1, reward=0.2)
     assert searcher.get_reward(config1) == 0.2
     assert searcher.get_best_reward() == 0.2
     assert searcher.get_best_config() == config1
@@ -33,11 +33,11 @@ def test_local_random_searcher():
     config3 = searcher.get_config()
 
     config4 = searcher.get_config()
-    searcher.update(config4, accuracy=0.1)
+    searcher.update(config4, reward=0.1)
     assert searcher.get_reward(config4) == 0.1
     assert searcher.get_best_reward() == 0.2
     assert searcher.get_best_config() == config1
-    searcher.update(config4, accuracy=0.5)
+    searcher.update(config4, reward=0.5)
     assert searcher.get_reward(config4) == 0.5
     assert searcher.get_best_reward() == 0.5
     assert searcher.get_best_config() == config4
diff --git a/core/tests/unittests/searcher/test_local_searcher.py b/core/tests/unittests/searcher/test_local_searcher.py
index c215acaf83f2..477f5fe32f0d 100644
--- a/core/tests/unittests/searcher/test_local_searcher.py
+++ b/core/tests/unittests/searcher/test_local_searcher.py
@@ -1,23 +1,71 @@
 
+import unittest
+
 from autogluon.core.searcher.local_searcher import LocalSearcher
+from autogluon.core.space import Categorical
+
+
+class TestLocalSearcher(unittest.TestCase):
+    def test_local_searcher(self):
+        search_space = {'param1': Categorical('hello', 'world'), 7: 42}
+        searcher = LocalSearcher(search_space=search_space)
+
+        config1 = {'param1': 'hello', 7: 42}
+        config2 = {'param1': 'world', 7: 42}
+        config_edge_case_1 = {'param1': 'world', 7: 0}
+        config_invalid_2 = {'param1': 'invalid', 7: 42}
+        config_invalid_3 = {'param1': 'hello'}
+        config_invalid_4 = {7: 42}
+        config_invalid_5 = {}
+        config_invalid_6 = {'param1': 'hello', 7: 42, 'unknown_param': 7}
+        config_invalid_7 = 0
+
+        assert searcher.get_best_reward() == float("-inf")
+        searcher.update(config1, reward=0.2)
+        assert searcher.get_best_reward() == 0.2
+        assert searcher.get_best_config() == config1
+
+        searcher.update(config1, reward=0.1)
+        assert searcher.get_best_reward() == 0.1
+        assert searcher.get_best_config() == config1
+
+        searcher.update(config2, reward=0.7)
+        assert searcher.get_best_reward() == 0.7
+        assert searcher.get_best_config() == config2
 
+        assert len(searcher._results) == 2
+        # This config is technically invalid, but for performance reasons is allowed to avoid having to pickle compare static parameters.
+        # Since the static parameter should be fixed, this config is treated as being equivalent to config2
+        searcher.update(config_edge_case_1, reward=0)
+        assert searcher.get_best_reward() == 0.1
+        assert len(searcher._results) == 2
+        self.assertRaises(AssertionError, searcher.update, config_invalid_2, reward=0)
+        self.assertRaises(AssertionError, searcher.update, config_invalid_3, reward=0)
+        self.assertRaises(AssertionError, searcher.update, config_invalid_4, reward=0)
+        self.assertRaises(AssertionError, searcher.update, config_invalid_5, reward=0)
+        self.assertRaises(AssertionError, searcher.update, config_invalid_6, reward=0)
+        self.assertRaises(AssertionError, searcher.update, config_invalid_7, reward=0)
+        self.assertRaises(AssertionError, searcher.update, config1, reward='invalid_reward')
+        assert len(searcher._results) == 2
 
-def test_local_searcher():
-    search_space = {}
-    searcher = LocalSearcher(search_space=search_space)
+    def test_local_searcher_pickle(self):
+        search_space = {1: Categorical(1, 2), 2: Categorical(1, 2)}
+        searcher = LocalSearcher(search_space=search_space)
 
-    config1 = 'hello'
-    config2 = 7
+        # Identical configs should have same pkl key, different configs should have different pkl keys
+        config_1 = {1: 1, 2: 2}
+        config_2 = {2: 2, 1: 1}
+        config_diff = {1: 2, 2: 1}
+        config_pkl_1 = searcher._pickle_config(config_1)
+        config_pkl_2 = searcher._pickle_config(config_2)
+        config_pkl_diff = searcher._pickle_config(config_diff)
 
-    assert searcher.get_best_reward() == float("-inf")
-    searcher.update(config1, accuracy=0.2)
-    assert searcher.get_best_reward() == 0.2
-    assert searcher.get_best_config() == config1
+        assert config_pkl_1 == config_pkl_2
+        assert config_pkl_1 != config_pkl_diff
 
-    searcher.update(config1, accuracy=0.1)
-    assert searcher.get_best_reward() == 0.1
-    assert searcher.get_best_config() == config1
+        config_unpkl = searcher._unpickle_config(config_pkl_1)
+        config_unpkl_diff = searcher._unpickle_config(config_pkl_diff)
 
-    searcher.update(config2, accuracy=0.7)
-    assert searcher.get_best_reward() == 0.7
-    assert searcher.get_best_config() == config2
+        assert config_unpkl == config_1
+        assert config_unpkl == config_2
+        assert config_unpkl_diff == config_diff
diff --git a/tabular/src/autogluon/tabular/models/catboost/callbacks.py b/tabular/src/autogluon/tabular/models/catboost/callbacks.py
index d739d7aa47b1..3f195158276a 100644
--- a/tabular/src/autogluon/tabular/models/catboost/callbacks.py
+++ b/tabular/src/autogluon/tabular/models/catboost/callbacks.py
@@ -123,7 +123,7 @@ def __init__(self, stopping_rounds, eval_metric, compare_key='validation'):
         else:
             is_max_optimal = eval_metric.is_max_optimal()
             # FIXME: Unsure if this works for custom metrics!
-            eval_metric_name = str(eval_metric)
+            eval_metric_name = eval_metric.__class__.__name__
 
         self.eval_metric_name = eval_metric_name
         self.is_max_optimal = is_max_optimal
diff --git a/tabular/src/autogluon/tabular/models/fasttext/fasttext_model.py b/tabular/src/autogluon/tabular/models/fasttext/fasttext_model.py
index b6b55c2b30dd..7f40eda6a544 100644
--- a/tabular/src/autogluon/tabular/models/fasttext/fasttext_model.py
+++ b/tabular/src/autogluon/tabular/models/fasttext/fasttext_model.py
@@ -13,7 +13,6 @@
 from autogluon.common.features.types import S_TEXT
 from autogluon.core.constants import BINARY, MULTICLASS
 from autogluon.core.models import AbstractModel
-from autogluon.core.models.abstract.model_trial import skip_hpo
 
 from .hyperparameters.parameters import get_param_baseline
 
@@ -185,7 +184,3 @@ def load(cls, path: str, reset_paths=True, verbose=True):
 
     def get_memory_size(self) -> int:
         return self._model_size_estimate
-
-    # TODO: Add HPO
-    def _hyperparameter_tune(self, **kwargs):
-        return skip_hpo(self, **kwargs)
diff --git a/tabular/src/autogluon/tabular/models/knn/knn_model.py b/tabular/src/autogluon/tabular/models/knn/knn_model.py
index 6ec46f0c4aab..8d355308455b 100644
--- a/tabular/src/autogluon/tabular/models/knn/knn_model.py
+++ b/tabular/src/autogluon/tabular/models/knn/knn_model.py
@@ -8,7 +8,6 @@
 from autogluon.common.features.types import R_BOOL, R_CATEGORY, R_OBJECT, S_BOOL, S_TEXT_NGRAM, S_TEXT_SPECIAL, S_DATETIME_AS_INT
 from autogluon.core.constants import REGRESSION
 from autogluon.core.utils.exceptions import NotEnoughMemoryError
-from autogluon.core.models.abstract.model_trial import skip_hpo
 from autogluon.core.models import AbstractModel
 from autogluon.core.utils.utils import normalize_pred_probas
 
@@ -53,7 +52,6 @@ def _preprocess(self, X, **kwargs):
     def _set_default_params(self):
         default_params = {
             'weights': 'uniform',
-            'n_jobs': -1,
         }
         for param, val in default_params.items():
             self._set_default_param_value(param, val)
@@ -86,9 +84,6 @@ def _get_default_searchspace(self):
         spaces = {}
         return spaces
 
-    def _set_cpu_params(self, num_cpus):
-        self.params['n_jobs'] = num_cpus
-
     def _fit(self,
              X,
              y,
@@ -98,16 +93,18 @@ def _fit(self,
              **kwargs):
         time_start = time.time()
         X = self.preprocess(X)
-        self._set_cpu_params(num_cpus)
+        params = self._get_model_params()
+        if 'n_jobs' not in params:
+            params['n_jobs'] = num_cpus
         if sample_weight is not None:  # TODO: support
             logger.log(15, "sample_weight not yet supported for KNNModel, this model will ignore them in training.")
 
         num_rows_max = len(X)
         # FIXME: v0.1 Must store final num rows for refit_full or else will use everything! Worst case refit_full could train far longer than the original model.
         if time_limit is None or num_rows_max <= 10000:
-            self.model = self._get_model_type()(**self._get_model_params()).fit(X, y)
+            self.model = self._get_model_type()(**params).fit(X, y)
         else:
-            self.model = self._fit_with_samples(X=X, y=y, time_limit=time_limit - (time.time() - time_start))
+            self.model = self._fit_with_samples(X=X, y=y, model_params=params, time_limit=time_limit - (time.time() - time_start))
 
     def _estimate_memory_usage(self, X, **kwargs):
         model_size_bytes = 4 * X.shape[0] * X.shape[1]  # Assuming float32 types
@@ -167,6 +164,7 @@ def _get_oof_pred_proba(self, X, **kwargs):
     def _fit_with_samples(self,
                           X,
                           y,
+                          model_params,
                           time_limit,
                           start_samples=10000,
                           max_samples=None,
@@ -241,7 +239,7 @@ def sample_func(chunk, frac):
                 X_samp = X
                 y_samp = y
                 idx = None
-            self.model = model_type(**self._get_model_params()).fit(X_samp, y_samp)
+            self.model = model_type(**model_params).fit(X_samp, y_samp)
             time_limit_left_prior = time_limit_left
             time_fit_end_sample = time.time()
             time_limit_left = time_limit - (time_fit_end_sample - time_start)
@@ -256,10 +254,6 @@ def sample_func(chunk, frac):
             self._X_unused_index = [i for i in range(num_rows_max) if i not in idx]
         return self.model
 
-    # TODO: Add HPO
-    def _hyperparameter_tune(self, **kwargs):
-        return skip_hpo(self, **kwargs)
-
     def _more_tags(self):
         return {'valid_oof': True}
 
diff --git a/tabular/src/autogluon/tabular/models/lgb/hyperparameters/parameters.py b/tabular/src/autogluon/tabular/models/lgb/hyperparameters/parameters.py
index 75471d41fe25..06dab10928cb 100644
--- a/tabular/src/autogluon/tabular/models/lgb/hyperparameters/parameters.py
+++ b/tabular/src/autogluon/tabular/models/lgb/hyperparameters/parameters.py
@@ -5,6 +5,15 @@
 DEFAULT_NUM_BOOST_ROUND = 10000  # default for single training run
 
 
+def get_lgb_objective(problem_type):
+    return {
+        BINARY: 'binary',
+        MULTICLASS: 'multiclass',
+        REGRESSION: 'regression',
+        SOFTCLASS: 'multiclass',
+    }[problem_type]
+
+
 def get_param_baseline_custom(problem_type):
     if problem_type == BINARY:
         return get_param_binary_baseline_custom()
@@ -44,39 +53,21 @@ def get_param_multiclass_baseline_custom():
 
 def get_param_binary_baseline():
     params = {
-        'num_boost_round': DEFAULT_NUM_BOOST_ROUND,
-        'num_threads': -1,
         'learning_rate': 0.05,
-        'objective': 'binary',
-        'verbose': -1,
-        'boosting_type': 'gbdt',
-        'two_round': True,
     }
     return params
 
 
 def get_param_multiclass_baseline():
     params = {
-        'num_boost_round': DEFAULT_NUM_BOOST_ROUND,
-        'num_threads': -1,
         'learning_rate': 0.05,
-        'objective': 'multiclass',
-        'verbose': -1,
-        'boosting_type': 'gbdt',
-        'two_round': True,
     }
     return params
 
 
 def get_param_regression_baseline():
     params = {
-        'num_boost_round': DEFAULT_NUM_BOOST_ROUND,
-        'num_threads': -1,
         'learning_rate': 0.05,
-        'objective': 'regression',
-        'verbose': -1,
-        'boosting_type': 'gbdt',
-        'two_round': True,
     }
     return params
 
diff --git a/tabular/src/autogluon/tabular/models/lgb/hyperparameters/searchspaces.py b/tabular/src/autogluon/tabular/models/lgb/hyperparameters/searchspaces.py
index 1703e300ea13..ba4260fa3c78 100644
--- a/tabular/src/autogluon/tabular/models/lgb/hyperparameters/searchspaces.py
+++ b/tabular/src/autogluon/tabular/models/lgb/hyperparameters/searchspaces.py
@@ -2,34 +2,24 @@
 from autogluon.core import Real, Int
 from autogluon.core.constants import BINARY, MULTICLASS, REGRESSION
 
-DEFAULT_NUM_BOOST_ROUND = 10000  # default for HPO
 
-
-def get_default_searchspace(problem_type, num_classes=None):
+def get_default_searchspace(problem_type):
     if problem_type == BINARY:
         return get_searchspace_binary_baseline()
     elif problem_type == MULTICLASS:
-        return get_searchspace_multiclass_baseline(num_classes=num_classes)
+        return get_searchspace_multiclass_baseline()
     elif problem_type == REGRESSION:
         return get_searchspace_regression_baseline()
     else:
         return get_searchspace_binary_baseline()
 
 
-def get_searchspace_multiclass_baseline(num_classes):
+def get_searchspace_multiclass_baseline():
     params = {
-        'objective': 'multiclass',
-        'num_classes': num_classes,
         'learning_rate': Real(lower=5e-3, upper=0.2, default=0.05, log=True),
         'feature_fraction': Real(lower=0.75, upper=1.0, default=1.0),
         'min_data_in_leaf': Int(lower=2, upper=60, default=20),  # TODO: Use size of dataset to set upper, if row count is small upper should be small
         'num_leaves': Int(lower=16, upper=96, default=31),  # TODO: Use row count and feature count to set this, the higher feature count the higher num_leaves upper
-        'num_boost_round': DEFAULT_NUM_BOOST_ROUND,
-        'boosting_type': 'gbdt',
-        'verbose': -1,
-        'two_round': True,
-        'seed_value': None,
-        # 'device': 'gpu'  # needs GPU-enabled lightGBM build
         # TODO: Bin size max increase
     }
     return params
@@ -37,31 +27,19 @@ def get_searchspace_multiclass_baseline(num_classes):
 
 def get_searchspace_binary_baseline():
     params = {
-        'objective': 'binary',
         'learning_rate': Real(lower=5e-3, upper=0.2, default=0.05, log=True),
         'feature_fraction': Real(lower=0.75, upper=1.0, default=1.0),
         'min_data_in_leaf': Int(lower=2, upper=60, default=20),
         'num_leaves': Int(lower=16, upper=96, default=31),
-        'num_boost_round': DEFAULT_NUM_BOOST_ROUND,
-        'boosting_type': 'gbdt',
-        'verbose': -1,
-        'two_round': True,
-        'seed_value': None,
     }
     return params
 
 
 def get_searchspace_regression_baseline():
     params = {
-        'objective': 'regression',
         'learning_rate': Real(lower=5e-3, upper=0.2, default=0.05, log=True),
         'feature_fraction': Real(lower=0.75, upper=1.0, default=1.0),
         'min_data_in_leaf': Int(lower=2, upper=60, default=20),
         'num_leaves': Int(lower=16, upper=96, default=31),
-        'num_boost_round': DEFAULT_NUM_BOOST_ROUND,
-        'boosting_type': 'gbdt',
-        'verbose': -1,
-        'two_round': True,
-        'seed_value': None,
     }
     return params
diff --git a/tabular/src/autogluon/tabular/models/lgb/lgb_model.py b/tabular/src/autogluon/tabular/models/lgb/lgb_model.py
index 7fdc02527824..5a4a2741da44 100644
--- a/tabular/src/autogluon/tabular/models/lgb/lgb_model.py
+++ b/tabular/src/autogluon/tabular/models/lgb/lgb_model.py
@@ -18,10 +18,9 @@
 from autogluon.core.utils import try_import_lightgbm
 
 from . import lgb_utils
-from .hyperparameters.parameters import get_param_baseline
+from .hyperparameters.parameters import get_param_baseline, get_lgb_objective, DEFAULT_NUM_BOOST_ROUND
 from .hyperparameters.searchspaces import get_default_searchspace
 from .lgb_utils import construct_dataset
-from ..utils import fixedvals_from_searchspaces
 
 warnings.filterwarnings("ignore", category=UserWarning, message="Starting from version")  # lightGBM brew libomp warning
 logger = logging.getLogger(__name__)
@@ -50,7 +49,7 @@ def _set_default_params(self):
             self._set_default_param_value(param, val)
 
     def _get_default_searchspace(self):
-        return get_default_searchspace(problem_type=self.problem_type, num_classes=self.num_classes)
+        return get_default_searchspace(problem_type=self.problem_type)
 
     # Use specialized LightGBM metric if available (fast), otherwise use custom func generator
     def _get_stopping_metric_internal(self):
@@ -84,8 +83,6 @@ def _fit(self,
         start_time = time.time()
         ag_params = self._get_ag_params()
         params = self._get_model_params()
-        params['num_threads'] = num_cpus
-        params = fixedvals_from_searchspaces(params)
 
         if verbosity <= 1:
             log_period = False
@@ -98,10 +95,7 @@ def _fit(self,
 
         stopping_metric, stopping_metric_name = self._get_stopping_metric_internal()
 
-        if self.problem_type in [MULTICLASS, SOFTCLASS] and 'num_classes' not in params:
-            params['num_classes'] = self.num_classes
-
-        num_boost_round = params.pop('num_boost_round', 1000)
+        num_boost_round = params.pop('num_boost_round', DEFAULT_NUM_BOOST_ROUND)
         dart_retrain = params.pop('dart_retrain', False)  # Whether to retrain the model to get optimal iteration if model is trained in 'dart' mode.
         if num_gpus != 0:
             if 'device' not in params:
@@ -110,9 +104,16 @@ def _fit(self,
                 #  GPU training heavily alters accuracy, often in a negative manner. We will have to be careful about when to use GPU.
                 params['device'] = 'gpu'
                 logger.log(20, f'\tTraining {self.name} with GPU, note that this may negatively impact model quality compared to CPU training.')
-        logger.log(15, f'Training Gradient Boosting Model for {num_boost_round} rounds...')
-        logger.log(15, "with the following hyperparameter settings:")
-        logger.log(15, params)
+        logger.log(15, f"\tFitting {num_boost_round} rounds... Hyperparameters: {params}")
+
+        if 'num_threads' not in params:
+            params['num_threads'] = num_cpus
+        if 'objective' not in params:
+            params['objective'] = get_lgb_objective(problem_type=self.problem_type)
+        if self.problem_type in [MULTICLASS, SOFTCLASS] and 'num_classes' not in params:
+            params['num_classes'] = self.num_classes
+        if 'verbose' not in params:
+            params['verbose'] = -1
 
         num_rows_train = len(X)
         dataset_train, dataset_val = self.generate_datasets(
@@ -272,7 +273,7 @@ def _preprocess_nonadaptive(self, X, is_train=False, **kwargs):
             return X
 
     def generate_datasets(self, X: DataFrame, y: Series, params, X_val=None, y_val=None, sample_weight=None, sample_weight_val=None, save=False):
-        lgb_dataset_params_keys = ['objective', 'two_round', 'num_threads', 'num_classes', 'verbose']  # Keys that are specific to lightGBM Dataset object construction.
+        lgb_dataset_params_keys = ['two_round']  # Keys that are specific to lightGBM Dataset object construction.
         data_params = {key: params[key] for key in lgb_dataset_params_keys if key in params}.copy()
 
         X = self.preprocess(X, is_train=True)
diff --git a/tabular/src/autogluon/tabular/models/lr/lr_model.py b/tabular/src/autogluon/tabular/models/lr/lr_model.py
index 6bbb559bc31b..e6f1267ced3b 100644
--- a/tabular/src/autogluon/tabular/models/lr/lr_model.py
+++ b/tabular/src/autogluon/tabular/models/lr/lr_model.py
@@ -13,7 +13,6 @@
 from .hyperparameters.parameters import get_param_baseline, INCLUDE, IGNORE, ONLY, _get_solver, preprocess_params_set
 from .hyperparameters.searchspaces import get_default_searchspace
 from .lr_preprocessing_utils import NlpDataPreprocessor, OheFeaturesGenerator, NumericDataPreprocessor
-from autogluon.core.models.abstract.model_trial import skip_hpo
 from autogluon.core.models import AbstractModel
 
 logger = logging.getLogger(__name__)
@@ -132,7 +131,7 @@ def _preprocess_train(self, X, feature_types, vect_max_features):
     def _set_default_params(self):
         default_params = {'random_state': 0, 'fit_intercept': True}
         if self.problem_type != REGRESSION:
-            default_params.update({'solver': _get_solver(self.problem_type), 'n_jobs': -1})
+            default_params.update({'solver': _get_solver(self.problem_type)})
         default_params.update(get_param_baseline())
         for param, val in default_params.items():
             self._set_default_param_value(param, val)
@@ -152,8 +151,10 @@ def _fit(self,
         if self.problem_type == BINARY:
             y = y.astype(int).values
 
-        self._set_cpu_params(num_cpus)
         params = {k: v for k, v in self.params.items() if k not in preprocess_params_set}
+        if 'n_jobs' not in params:
+            if self.problem_type != REGRESSION:
+                params['n_jobs'] = num_cpus
 
         # Ridge/Lasso are using alpha instead of C, which is C^-1
         # https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html#sklearn.linear_model.Ridge
@@ -174,10 +175,6 @@ def _fit(self,
         else:
             self.model = model.fit(X, y, sample_weight=sample_weight)
 
-    # TODO: Add HPO
-    def _hyperparameter_tune(self, **kwargs):
-        return skip_hpo(self, **kwargs)
-
     def _select_features_handle_text_include(self, df, types_of_features, categorical_featnames, language_featnames, continuous_featnames):
         # continuous = numeric features to rescale
         # skewed = features to which we will apply power (ie. log / box-cox) transform before normalization
@@ -227,7 +224,3 @@ def _get_default_auxiliary_params(self) -> dict:
         )
         default_auxiliary_params.update(extra_auxiliary_params)
         return default_auxiliary_params
-
-    def _set_cpu_params(self, num_cpus):
-        if self.problem_type != REGRESSION:
-            self.params['n_jobs'] = num_cpus
diff --git a/tabular/src/autogluon/tabular/models/rf/rf_model.py b/tabular/src/autogluon/tabular/models/rf/rf_model.py
index b2b2a81df06a..97390a57bbb1 100644
--- a/tabular/src/autogluon/tabular/models/rf/rf_model.py
+++ b/tabular/src/autogluon/tabular/models/rf/rf_model.py
@@ -12,7 +12,6 @@
 from autogluon.core.utils.exceptions import NotEnoughMemoryError, TimeLimitExceeded
 from autogluon.core.utils.utils import normalize_pred_probas
 
-from autogluon.core.models.abstract.model_trial import skip_hpo
 from autogluon.core.models import AbstractModel
 from autogluon.features.generators import LabelEncoderFeatureGenerator
 
@@ -130,8 +129,9 @@ def _fit(self,
         model_cls = self._get_model_type()
 
         max_memory_usage_ratio = self.params_aux['max_memory_usage_ratio']
-        self._set_cpu_params(num_cpus)
         params = self._get_model_params()
+        if 'n_jobs' not in params:
+            params['n_jobs'] = num_cpus
         n_estimators_final = params['n_estimators']
 
         n_estimators_minimum = min(40, n_estimators_final)
@@ -313,10 +313,6 @@ def _get_oof_pred_proba(self, X, y, **kwargs):
 
         return self._convert_proba_to_unified_form(y_oof_pred_proba)
 
-    # TODO: Add HPO
-    def _hyperparameter_tune(self, **kwargs):
-        return skip_hpo(self, **kwargs)
-
     def _get_default_auxiliary_params(self) -> dict:
         default_auxiliary_params = super()._get_default_auxiliary_params()
         extra_auxiliary_params = dict(
@@ -333,9 +329,6 @@ def _get_default_ag_args_ensemble(cls, problem_type=None, **kwargs) -> dict:
             default_ag_args_ensemble.update(extra_ag_args_ensemble)
         return default_ag_args_ensemble
 
-    def _set_cpu_params(self, num_cpus):
-        self.params['n_jobs'] = num_cpus
-
     def _more_tags(self):
         if self.problem_type == QUANTILE:
             return {'valid_oof': False} # not supported in quantile regression
diff --git a/tabular/src/autogluon/tabular/models/tabular_nn/mxnet/tabular_nn_mxnet.py b/tabular/src/autogluon/tabular/models/tabular_nn/mxnet/tabular_nn_mxnet.py
index 26bfcae59f01..8726d4406866 100644
--- a/tabular/src/autogluon/tabular/models/tabular_nn/mxnet/tabular_nn_mxnet.py
+++ b/tabular/src/autogluon/tabular/models/tabular_nn/mxnet/tabular_nn_mxnet.py
@@ -18,7 +18,6 @@
 import pandas as pd
 
 from autogluon.common.features.types import R_OBJECT, S_TEXT_NGRAM, S_TEXT_AS_CATEGORY
-from autogluon.core import Space
 from autogluon.core.constants import BINARY, MULTICLASS, REGRESSION, SOFTCLASS
 from autogluon.core.utils import try_import_mxboard, try_import_mxnet
 from autogluon.core.utils.exceptions import TimeLimitExceeded
@@ -28,7 +27,6 @@
 from ..hyperparameters.searchspaces import get_default_searchspace
 from ..utils.data_preprocessor import create_preprocessor, get_feature_arraycol_map, get_feature_type_map
 from ..utils.nn_architecture_utils import infer_y_range, get_default_layers, default_numeric_embed_dim
-from ...utils import fixedvals_from_searchspaces
 
 warnings.filterwarnings("ignore", module='sklearn.preprocessing')  # sklearn processing n_quantiles warning
 logger = logging.getLogger(__name__)
@@ -140,7 +138,6 @@ def _fit(self, X, y, X_val=None, y_val=None,
             logger.log(15, "sample_weight not yet supported for TabularNeuralNetModel, this model will ignore them in training.")
 
         params = self._get_model_params()
-        params = fixedvals_from_searchspaces(params)
         if num_cpus is not None:
             self.num_dataloading_workers = max(1, int(num_cpus/2.0))
         else:
diff --git a/tabular/src/autogluon/tabular/models/tabular_nn/torch/tabular_nn_torch.py b/tabular/src/autogluon/tabular/models/tabular_nn/torch/tabular_nn_torch.py
index ea46b05094a2..7bd2c08e498a 100644
--- a/tabular/src/autogluon/tabular/models/tabular_nn/torch/tabular_nn_torch.py
+++ b/tabular/src/autogluon/tabular/models/tabular_nn/torch/tabular_nn_torch.py
@@ -17,7 +17,6 @@
 from ..hyperparameters.searchspaces import get_default_searchspace
 from ..utils.data_preprocessor import create_preprocessor, get_feature_arraycol_map, get_feature_type_map
 from ..utils.nn_architecture_utils import infer_y_range
-from ...utils import fixedvals_from_searchspaces
 
 logger = logging.getLogger(__name__)
 
@@ -144,7 +143,6 @@ def _fit(self, X, y, X_val=None, y_val=None,
         start_time = time.time()
 
         params = self._get_model_params()
-        params = fixedvals_from_searchspaces(params)
 
         processor_kwargs, optimizer_kwargs, fit_kwargs, loss_kwargs, params = self._prepare_params(params=params)
 
diff --git a/tabular/src/autogluon/tabular/models/utils.py b/tabular/src/autogluon/tabular/models/utils.py
deleted file mode 100644
index 76615b1b4415..000000000000
--- a/tabular/src/autogluon/tabular/models/utils.py
+++ /dev/null
@@ -1,28 +0,0 @@
-
-import logging
-
-from autogluon.core import Space
-
-logger = logging.getLogger(__name__)
-
-
-# Methods useful for all models:
-def fixedvals_from_searchspaces(params):
-    """Converts any search space hyperparams in params dict into fixed default values."""
-    if any(isinstance(params[hyperparam], Space) for hyperparam in params):
-        logger.warning("Attempting to fit model without HPO, but search space is provided. fit() will only consider default hyperparameter values from search space.")
-        bad_keys = [hyperparam for hyperparam in params if isinstance(params[hyperparam], Space)][:]  # delete all keys which are of type autogluon Space
-        params = params.copy()
-        for hyperparam in bad_keys:
-            params[hyperparam] = hp_default_value(params[hyperparam])
-        return params
-    else:
-        return params
-
-
-def hp_default_value(hp_value):
-    """Extracts default fixed value from hyperparameter search space hp_value to use a fixed value instead of a search space."""
-    if not isinstance(hp_value, Space):
-        return hp_value
-    else:
-        return hp_value.default
diff --git a/vision/src/autogluon/vision/_gluoncv/image_classification.py b/vision/src/autogluon/vision/_gluoncv/image_classification.py
index 2e068b9e7a7a..f1335c74d04d 100644
--- a/vision/src/autogluon/vision/_gluoncv/image_classification.py
+++ b/vision/src/autogluon/vision/_gluoncv/image_classification.py
@@ -17,10 +17,9 @@
 import autogluon.core as ag
 from autocfg import dataclass
 from autogluon.core.constants import BINARY, MULTICLASS, REGRESSION
-from autogluon.core.scheduler.reporter import FakeReporter
 from autogluon.core.utils import get_cpu_count, get_gpu_count_all
 from autogluon.core.task.base import BaseTask
-from autogluon.core.searcher import LocalRandomSearcher
+from autogluon.core.searcher import DummySearcher
 
 from gluoncv.auto.estimators.base_estimator import BaseEstimator
 from gluoncv.auto.estimators import ImageClassificationEstimator, TorchImageClassificationEstimator
@@ -73,7 +72,13 @@ class DefaultConfig:
     dist_ip_addrs : Union[type(None), list, Tuple] = None
 
 
-def _train_image_classification(args, reporter):
+def _train_image_classification(args,
+                                train_data,
+                                val_data,
+                                problem_type,
+                                wall_clock_tick,
+                                log_dir,
+                                reporter=None):
     """
     Parameters
     ----------
@@ -85,24 +90,11 @@ def _train_image_classification(args, reporter):
         task_id = int(args['task_id'])
     except:
         task_id = 0
-    problem_type = args.pop('problem_type', MULTICLASS)
     final_fit = args.pop('final_fit', False)
-    # train, val data
-    train_data = args.pop('train_data')
-    val_data = args.pop('val_data')
-    # wall clock tick limit
-    wall_clock_tick = args.pop('wall_clock_tick')
-    log_dir = args.pop('log_dir', os.getcwd())
     # exponential batch size for Int() space batch sizes
     exp_batch_size = args.pop('exp_batch_size', False)
     if exp_batch_size and 'batch_size' in args:
         args['batch_size'] = 2 ** args['batch_size']
-    try:
-        task = args.pop('task')
-        dataset = args.pop('dataset')
-        num_trials = args.pop('num_trials')
-    except KeyError:
-        task = None
 
     # mxnet and torch dispatcher
     dispatcher = None
@@ -187,17 +179,6 @@ def _train_image_classification(args, reporter):
             result = estimator.fit(train_data=train_data, val_data=val_data, time_limit=wall_clock_tick-tic)
             with open(os.path.join(trial_log_dir, valid_summary_file), 'w') as f:
                 json.dump(result, f)
-            # save config and result
-            if task is not None:
-                trial_log = {}
-                trial_log.update(args)
-                trial_log.update(result)
-                json_str = json.dumps(trial_log)
-                time_str = time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime())
-                json_file_name = task + '_dataset-' + dataset + '_trials-' + str(num_trials) + '_' + time_str + '.json'
-                with open(json_file_name, 'w') as json_file:
-                    json_file.write(json_str)
-                logging.info('Config and result in this trial have been saved to %s.', json_file_name)
     except:
         import traceback
         return {'traceback': traceback.format_exc(), 'args': str(args),
@@ -282,9 +263,9 @@ def __init__(self, config=None, logger=None, problem_type=None):
         self._config = config
 
         # scheduler options
-        self.scheduler = config.get('scheduler', 'local')
-        self.search_strategy = config.get('search_strategy', 'random')
-        self.search_options = config.get('search_options', {})
+        self.scheduler = config.pop('scheduler', 'local')
+        self.search_strategy = config.pop('search_strategy', 'random')
+        self.search_options = config.pop('search_options', {})
         self.scheduler_options = {
             'resource': {'num_cpus': nthreads_per_trial, 'num_gpus': ngpus_per_trial},
             'checkpoint': config.get('checkpoint', 'checkpoint/exp1.ag'),
@@ -297,7 +278,7 @@ def __init__(self, config=None, logger=None, problem_type=None):
             'dist_ip_addrs': config.get('dist_ip_addrs', None),
             'searcher': self.search_strategy,
             'search_options': self.search_options,
-            'max_reward': config.get('max_reward', 0.95)}
+            'max_reward': config.get('max_reward', None)}
 
     def fit(self, train_data, val_data=None, train_size=0.9, random_state=None, time_limit=None):
         """Fit auto estimator given the input data.
@@ -365,26 +346,30 @@ def fit(self, train_data, val_data=None, train_size=0.9, random_state=None, time
                 raise ValueError('Unable to determine the estimator for fit function.')
 
         # register args
-        config['train_data'] = train_data
-        config['val_data'] = val_data
-        config['wall_clock_tick'] = wall_clock_tick
-        config['log_dir'] = os.path.join(config.get('log_dir', os.getcwd()), str(uuid.uuid4())[:8])
-        self.scheduler_options['checkpoint'] = os.path.join(config['log_dir'], 'exp1.ag')
-        config['problem_type'] = self._problem_type
+        log_dir = os.path.join(config.pop('log_dir', os.getcwd()), str(uuid.uuid4())[:8])
+        self.scheduler_options['checkpoint'] = os.path.join(log_dir, 'exp1.ag')
+
+        train_fn_kwargs = dict(
+            train_data=train_data,
+            val_data=val_data,
+            wall_clock_tick=wall_clock_tick,
+            log_dir=log_dir,
+            problem_type=self._problem_type,
+        )
 
         start_time = time.time()
         self._fit_summary = {}
         self._results = {}
-        if config.get('num_trials', 1) < 2:
-            reporter = FakeReporter()
-            rand_config = LocalRandomSearcher(search_space=config).get_config()
+
+        num_trials = config.pop('num_trials', 1)
+        hpo_enabled = num_trials > 1
+        if not hpo_enabled:
+            default_config = DummySearcher(search_space=config).get_config()
             self._logger.info("Starting fit without HPO")
             cur_config = {**config}
-            cur_config.update(rand_config)
-            results = _train_image_classification({**cur_config}, reporter)
+            cur_config.update(default_config)
+            results = _train_image_classification({**cur_config}, **train_fn_kwargs)
             best_config = cur_config
-            best_config.pop('train_data', None)
-            best_config.pop('val_data', None)
             self._fit_summary.update({'train_acc': results.get('train_acc', -1),
                                       'valid_acc': results.get('valid_acc', -1),
                                       'total_time': results.get('time', time.time() - start_time),
@@ -393,13 +378,15 @@ def fit(self, train_data, val_data=None, train_size=0.9, random_state=None, time
         else:
             self._logger.info("Starting HPO experiments")
             results = self.run_fit(_train_image_classification, config, self.scheduler,
-                                   self.scheduler_options, plot_results=False)
+                                   self.scheduler_options, plot_results=False,
+                                   train_fn_kwargs=train_fn_kwargs
+                                   )
             if isinstance(results, dict):
                 ks = ('best_reward', 'best_config', 'total_time', 'config_history', 'reward_attr')
                 self._results.update({k: v for k, v in results.items() if k in ks})
         end_time = time.time()
         self._logger.info("Finished, total runtime is %.2f s", end_time - start_time)
-        if config.get('num_trials', 1) > 1:
+        if hpo_enabled:
             best_config = {**config}
             best_config.update(results['best_config'])
             best_config.update({'estimator': results['estimator']})
@@ -414,7 +401,7 @@ def fit(self, train_data, val_data=None, train_size=0.9, random_state=None, time
         self._logger.info(pprint.pformat(self._fit_summary, indent=2))
 
         if self._cleanup_disk:
-            shutil.rmtree(config['log_dir'], ignore_errors=True)
+            shutil.rmtree(log_dir, ignore_errors=True)
         model_checkpoint = results.get('model_checkpoint', None)
         if model_checkpoint is None:
             if results.get('traceback', '') == 'timeout':
diff --git a/vision/src/autogluon/vision/_gluoncv/object_detection.py b/vision/src/autogluon/vision/_gluoncv/object_detection.py
index 43cc9be1e11e..33674dd6c210 100644
--- a/vision/src/autogluon/vision/_gluoncv/object_detection.py
+++ b/vision/src/autogluon/vision/_gluoncv/object_detection.py
@@ -16,10 +16,9 @@
 import pandas as pd
 from autocfg import dataclass
 import autogluon.core as ag
-from autogluon.core.scheduler.reporter import FakeReporter
 from autogluon.core.utils import get_cpu_count, get_gpu_count_all
 from autogluon.core.task.base import BaseTask
-from autogluon.core.searcher import LocalRandomSearcher
+from autogluon.core.searcher import DummySearcher
 
 from gluoncv.auto.estimators.base_estimator import BaseEstimator
 from gluoncv.auto.estimators import SSDEstimator, FasterRCNNEstimator, YOLOv3Estimator, CenterNetEstimator
@@ -57,7 +56,12 @@ class DefaultConfig:
     dist_ip_addrs : Union[type(None), list, Tuple] = None
 
 
-def _train_object_detection(args, reporter):
+def _train_object_detection(args,
+                            train_data,
+                            val_data,
+                            wall_clock_tick,
+                            log_dir,
+                            reporter=None):
     """
     Parameters
     ----------
@@ -70,22 +74,10 @@ def _train_object_detection(args, reporter):
     except:
         task_id = 0
     final_fit = args.pop('final_fit', False)
-    # train, val data
-    train_data = args.pop('train_data')
-    val_data = args.pop('val_data')
-    # wall clock tick limit
-    wall_clock_tick = args.pop('wall_clock_tick')
-    log_dir = args.pop('log_dir', os.getcwd())
     # exponential batch size for Int() space batch sizes
     exp_batch_size = args.pop('exp_batch_size', False)
     if exp_batch_size and 'batch_size' in args:
         args['batch_size'] = 2 ** args['batch_size']
-    try:
-        task = args.pop('task')
-        dataset = args.pop('dataset')
-        num_trials = args.pop('num_trials')
-    except KeyError:
-        task = None
 
     # convert user defined config to nested form
     args = config_to_nested(args)
@@ -138,17 +130,6 @@ def _train_object_detection(args, reporter):
             result = estimator.fit(train_data=train_data, val_data=val_data, time_limit=wall_clock_tick-tic)
             with open(os.path.join(trial_log_dir, valid_summary_file), 'w') as f:
                 json.dump(result, f)
-            # save config and result
-            if task is not None:
-                trial_log = {}
-                trial_log.update(args)
-                trial_log.update(result)
-                json_str = json.dumps(trial_log)
-                time_str = time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime())
-                json_file_name = task + '_dataset-' + dataset + '_trials-' + str(num_trials) + '_' + time_str + '.json'
-                with open(json_file_name, 'w') as json_file:
-                    json_file.write(json_str)
-                logging.info('Config and result in this trial have been saved to %s.', json_file_name)
     except:
         import traceback
         return {'traceback': traceback.format_exc(), 'args': str(args),
@@ -245,12 +226,12 @@ def __init__(self, config=None, logger=None):
         config['gpus'] = [int(i) for i in range(ngpus_per_trial)]
         config['seed'] = config.get('seed', np.random.randint(32,767))
         config['final_fit'] = False
-        self._cleanup_disk = config.get('cleanup_disk', True)
+        self._cleanup_disk = config.pop('cleanup_disk', True)
         self._config = config
 
         # scheduler options
-        self.search_strategy = config.get('search_strategy', 'random')
-        self.search_options = config.get('search_options', {})
+        self.search_strategy = config.pop('search_strategy', 'random')
+        self.search_options = config.pop('search_options', {})
         self.scheduler_options = {
             'resource': {'num_cpus': nthreads_per_trial, 'num_gpus': ngpus_per_trial},
             'checkpoint': config.get('checkpoint', 'checkpoint/exp1.ag'),
@@ -263,7 +244,7 @@ def __init__(self, config=None, logger=None):
             'dist_ip_addrs': config.get('dist_ip_addrs', None),
             'searcher': self.search_strategy,
             'search_options': self.search_options,
-            'max_reward': config.get('max_reward', 0.9)}
+            'max_reward': config.get('max_reward', None)}
 
     def fit(self, train_data, val_data=None, train_size=0.9, random_state=None, time_limit=None):
         """Fit auto estimator given the input data.
@@ -327,24 +308,27 @@ def fit(self, train_data, val_data=None, train_size=0.9, random_state=None, time
             config.pop('train_dataset')
 
         # register args
-        config['train_data'] = train_data
-        config['val_data'] = val_data
-        config['wall_clock_tick'] = wall_clock_tick
-        config['log_dir'] = os.path.join(config.get('log_dir', os.getcwd()), str(uuid.uuid4())[:8])
+        log_dir = os.path.join(config.pop('log_dir', os.getcwd()), str(uuid.uuid4())[:8])
+        train_fn_kwargs = dict(
+            train_data=train_data,
+            val_data=val_data,
+            wall_clock_tick=wall_clock_tick,
+            log_dir=log_dir,
+        )
 
         start_time = time.time()
         self._fit_summary = {}
         self._results = {}
-        if config.get('num_trials', 1) < 2:
-            reporter = FakeReporter()
-            rand_config = LocalRandomSearcher(search_space=config).get_config()
+
+        num_trials = config.pop('num_trials', 1)
+        hpo_enabled = num_trials > 1
+        if not hpo_enabled:
+            default_config = DummySearcher(search_space=config).get_config()
             self._logger.info("Starting fit without HPO")
             cur_config = {**config}
-            cur_config.update(rand_config)
-            results = _train_object_detection({**cur_config}, reporter)
+            cur_config.update(default_config)
+            results = _train_object_detection({**cur_config}, **train_fn_kwargs)
             best_config = cur_config
-            best_config.pop('train_data', None)
-            best_config.pop('val_data', None)
             self._fit_summary.update({'train_map': results.get('train_map', -1),
                                       'valid_map': results.get('valid_map', -1),
                                       'total_time': results.get('time', time.time() - start_time),
@@ -353,13 +337,13 @@ def fit(self, train_data, val_data=None, train_size=0.9, random_state=None, time
         else:
             self._logger.info("Starting HPO experiments")
             results = self.run_fit(_train_object_detection, config, self.search_strategy,
-                                   self.scheduler_options)
+                                   self.scheduler_options, train_fn_kwargs=train_fn_kwargs)
             if isinstance(results, dict):
                 ks = ('best_reward', 'best_config', 'total_time', 'config_history', 'reward_attr')
                 self._results.update({k: v for k, v in results.items() if k in ks})
         end_time = time.time()
         self._logger.info("Finished, total runtime is %.2f s", end_time - start_time)
-        if config.get('num_trials', 1) > 1:
+        if hpo_enabled:
             best_config = {**config}
             best_config.update(results['best_config'])
             # convert best config to nested form
@@ -373,7 +357,7 @@ def fit(self, train_data, val_data=None, train_size=0.9, random_state=None, time
         self._logger.info(pprint.pformat(self._fit_summary, indent=2))
 
         if self._cleanup_disk:
-            shutil.rmtree(config['log_dir'], ignore_errors=True)
+            shutil.rmtree(log_dir, ignore_errors=True)
         model_checkpoint = results.get('model_checkpoint', None)
         if model_checkpoint is None:
             if results.get('traceback', '') == 'timeout':


Code review comments:
diff_hunk: @@ -147,31 +147,48 @@ def run(self, **kwargs):
         self.training_history = OrderedDict()
         self.config_history = OrderedDict()
 
-        trial_run_times = []
+        failure_count = 0
+        trial_count = 0
+        trial_run_times = 0
+        min_failure_threshold = 5
+        failure_rate_threshold = 0.8
         time_start = time.time()
 
         r = range(self.num_trials)
         for i in (tqdm(r) if self.num_trials < 1000 else r):
             trial_start_time = time.time()
             try:
                 is_failed, result = self.run_trial(task_id=i)
+            except ExhaustedSearchSpaceError:
+                break
             except Exception:
                 # TODO: Add special exception type when there are no more new configurations to try (exhausted search space)
                 logger.log(30, f'\tWARNING: Encountered unexpected exception during trial {i}, stopping HPO early.')
                 logger.exception('Detailed Traceback:')  # TODO: Avoid logging if verbosity=0
                 break
             trial_end_time = time.time()
-            trial_run_times.append(np.NaN if is_failed else (trial_end_time - trial_start_time))
+
+            trial_count += 1
+            if is_failed:
+                failure_count += 1
+            else:
+                trial_run_times += trial_end_time - trial_start_time
path: core/src/autogluon/core/scheduler/seq_scheduler.py
text: @user1:
1. If we want to keep individual times, then this might be broken - should be something like this: `trial_run_times += [trial_end_time - trial_start_time]` ? If we don't then naming is misleading as it can be read as if it contains multiple values. Suggestion: `trials_total_time`
2. This will break in parallel setting.

@author:
1. Updated to trials_total_time
2. This is the LocalSequentialScheduler so we don't need to write logic with parallel in mind yet.