Repository: autogluon/autogluon
Pull request: BasePredictor skeleton outlined
TODOs needed to get predictor object working with this BasePredictor object:

    - Base Task should implement @classmethod load(output_directory) method which restores a Predictor object from file (previously saved using Predictor.save(output_directory)). task.load(output_directory) can simply: return Predictor.load(output_directory)
    
    - Base Task.fit() needs to create Results object and assign it to predictor.results before returning predictor. The only thing task.fit should return is a single Predictor object.
    
    - Right before task.fit() returns predictor, it should call: predictor.save(outputdir) so that training progress is not accidentally lost.
    
    - task.fit() needs to have an output_directory argument where to store all outputs.
    
    - Delete line "Results = collections.namedtuple('Results', 'model reward config time metadata')" from task.fit(), and store all info in self.results dict object defined below instead.
    
    - This code assumes trial_ids are sortable such that lower trial_id indicates trial was scheduled earlier than trial with higher trial_id
    
    - Base Dataset object needs to have get_labels() method. In general, it's currently unclear what is the difference between a dataset with labels and one for which no labels are avialable (ie. test data).
      Also to make sure predictors are not cheating, there should be a way to erase all labels from the Dataset object after separately extracting them via get_labels()

*Issue #, if available:*

fixes https://github.com/awslabs/autogluon/issues/29

*Description of changes:*


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.


Problem statement:
Predictor Object for AutoGluon Fit API
Issue tracker for discussion with @jwmueller @cgraywang today.

`results` returned by `fit` may not be a good way to saving state and handling multiple experiments.

We decide to make a `Predictor` object which contains the following method:

```python
save()
load()
predict()
evaluate()
```

Hints from issue discussion:
I think this object should also support a `fit_summary()` method which returns something like the current `Results` object, with more informative/comprehensive instance-variable names as described here:  

https://github.com/awslabs/autogluon/issues/32 
Also I'd make sure all `task.fit()` calls automatically call `save` at the end before returning. We do not want to burden the user with having to explicitly call `save`.

By the way, `load()` should be a class method, so the user can call `task.load(filepath)` without any predictor object.
In contrast, `save()` should be an instance method, so the user calls `predictor.save(filepath)`.
Just to be clear, we should also try to ensure `predictor.predict()` is overloaded to be able to handle: 
1) single example (e.g. single image, single document) 
2) batch of many examples
3) MXNet Dataset object (ie. already preprocessed examples with a defined iterator)

For classification tasks, the sklearn convention is to use: `predict_proba` underneath the hood to produce predicted class probabilities, and then `predict` is just a wrapper around this that computes the argmax of the class probabilities.
> Just to be clear, we should also try to ensure `predictor.predict()` is overloaded to be able to handle:
> 
> 1. single example (e.g. single image, single document)
> 2. batch of many examples
> 3. MXNet Dataset object (ie. already preprocessed examples with a defined iterator)
> 
> For classification tasks, the sklearn convention is to use: `predict_proba` underneath the hood to produce predicted class probabilities, and then `predict` is just a wrapper around this that computes the argmax of the class probabilities.

This is great suggestion. I will work on it 👍

Commit message: added plotting functions to summarize fit() process

Patch to review:
diff --git a/autogluon/task/base/base_predictor.py b/autogluon/task/base/base_predictor.py
new file mode 100644
index 000000000000..c9e74abf1a5e
--- /dev/null
+++ b/autogluon/task/base/base_predictor.py
@@ -0,0 +1,180 @@
+""" TODOs needed to get predictor object working for any task:
+
+    - Base Task should implement @classmethod load(output_directory) method which restores a Predictor object from file (previously saved using Predictor.save(output_directory)). task.load(output_directory) can simply: return Predictor.load(output_directory)
+    
+    - Base Task.fit() needs to create Results object and assign it to predictor.results before returning predictor. The only thing task.fit should return is a single Predictor object.
+    
+    - Right before task.fit() returns predictor, it should call: predictor.save(outputdir) so that training progress is not accidentally lost.
+    
+    - task.fit() needs to have an output_directory argument where to store all outputs
+    
+    - Delete line "Results = collections.namedtuple('Results', 'model reward config time metadata')" from task.fit(), and store all info in self.results dict object defined below instead.
+    
+    - This code assumes trial_ids are sortable such that lower trial_id indicates trial was scheduled earlier than trial with higher trial_id
+    
+    - task object should have get_labels(Dataset) method
+"""
+
+import pickle, json
+from ...utils import plot_performance_vs_trials, plot_summary_of_models
+
+logger = logging.getLogger(__name__)
+
+__all__ = ['BasePredictor']
+
+PREDICTOR_FILENAME = "predictor.pkl" # Filename in which predictor object is stored. Should be hardcoded so that user only needs to specify directory where to store all training-related output files.
+RESULTS_FILENAME = "results.json" # Filename in which FitResults object is stored. Should be hardcoded so that user only needs to specify directory where to store all training-related output files.
+
+class BasePredictor(object):
+    """ 
+    Base object returned by task.fit() for each task implemented in AutoGluon.
+    
+    Example user workflow for say image classification applications:
+        # Training time:
+        >>> from autogluon import image_classification as task
+        >>> train_data = task.Dataset(traindata_filepath) 
+        >>> output_directory = '~/temp/' # any directory name specifying where to store all results
+        >>> predictor = task.fit(train_data=train_data, output_directory=output_directory)
+        >>> # To instead specify train/val split, do: predictor = task.fit(train_data=train_data, val_data=task.Dataset(valdata_filepath), output_directory=output_directory)
+        >>> results = predictor.fit_summary() # will also print out summary and create plots
+        # Inference time (may be a new Python session):
+        >>> test_data = task.Dataset(testdata_filepath)
+        >>> test_labels = task.get_labels(test_data)
+        >>> predictor = None  # We delete predictor here just to demonstrate how to load previously-trained predictor from file
+        >>> predictor = task.load(output_directory)
+        >>> batch_predictions = predictor.predict(test_data)
+        >>> performance = predictor.evaluate_predictions(y_true=test_labels, y_pred=batch_predictions) 
+        # or can instead just use equivalent shorthand: performance = predictor.evaluate(test_data)
+        # Can also do inference on just a single test example: x_i = single datapoint, eg. x_i = test_data[i]
+        >>> single_prediction = predictor.predict(x_i)
+        >>> print((x_i, single_prediction))
+    """
+    def __init__(self, loss_func, eval_func, model=None, **kwargs):
+        self.model = model # MXnet model or list of multiple models / ensemble. Each model should have its own loading/saving functionality.
+        self.loss_func = loss_func # Loss function (or string name) minimized during training
+        self.eval_func = eval_func # Evaluation function / metric applied on validation/test data to gauge predictive performance.
+        # Note: we may save a lot of headache if higher values of this eval_func metric = better, consistently across all tasks.
+        self.results = self.createResultsDict() # dict object to store all information during task.fit().
+    
+    @classmethod
+    def load(output_directory):
+        """ Load Predictor object from given directory.
+            Make sure to also load any models from files that exist in output_directory and set them = predictor.model.
+        """ 
+        filepath = output_directory + PREDICTOR_FILENAME
+        results_file = output_directory + RESULTS_FILENAME
+        predictor = pickle.load(open(filepath,"rb"))
+        predictor.results = json.load(open(results_file,'r'))
+        pass
+    
+    def save(self, output_directory):
+        """ Saves this object to file. Don't forget to save the models and the Results objects if they exist.
+            Before returning a Predictor, task.fit() should call predictor.save()
+        """
+        filepath = output_directory + PREDICTOR_FILENAME
+        self.save_model(output_directory)
+        self.save_results(output_directory)
+        self.model = None # Save model separately from Predictor object
+        self.results = None # Save results separately from Predictor object
+        pickle.dump(self, open(filepath,'wb'))
+        logger.info("Predictor saved to file: " % filepath)
+    
+    def save_results(self, output_directory):
+        """ Save results in human-readable file JSON format """
+        results_file = output_directory + RESULTS_FILENAME
+        json.dump(self.results, open(results_file, 'w'))
+    
+    @abstractmethod
+    def save_model(self, output_directory):
+        """ Save self.model object to file located in output_directory. 
+            For example, if self.model is MXNet model, can simply call self.model.save(output_directory+filename)
+        """
+        pass
+    
+    @abstractmethod
+    def predict(X):
+        """ This method should be able to produce predictions regardless if:
+            X = single data example (e.g. single image, single document), 
+            X = batch of many examples, X = task.Dataset object 
+        """
+        pass
+    
+    @abstractmethod
+    def predict_proba(self, X):
+        """ Produces predicted class probabilities if we are dealing with a classification task.
+            In this case, predict() should just be a wrapper around this method to convert predicted probabilties to predicted class labels.
+        """
+        pass
+        
+    @abstractmethod
+    def evaluate_predictions(self, y_true, y_pred):
+        """ Evaluate the provided list of predictions against list of ground truth labels according to the task-specific evaluation metric (self.eval_func). """
+        pass
+    
+    @abstractmethod
+    def evaluate(self, dataset):
+        """ Use self.model to produce predictions from the given Dataset object, and then compute task-specific evaluation metric (self.eval_func)
+            comparing these predictions against ground truth labels stored in the Dataset object.
+        """
+        pass
+    
+    def fit_summary(self, output_directory, verbosity = 2):
+        """ 
+            Returns a summary of the fit process. 
+            Args:
+                verbosity (int): how much output to print:
+                <= 0 for no output printing, 1 for just high-level summary, 2 for summary and plot, >= 3 for all information contained in results object. 
+        """
+        if verbosity > 0:
+            summary = {}
+            for k in self.results.keys():
+                if k not in ['metadata', 'trial_info']:
+                    summary[k] = self.results[k]
+            logger.info("Summary of Fit Process:  ")
+            logger.info(json.dumps(summary, indent=2))
+            if len(self.results['metadata']) > 0:
+                logger.info(json.dumps(self.results['metadata'], indent=2))
+        
+        if len(results['trial_info']) > 0 and  verbosity > 1:
+            ordered_trials = sorted(self.results['trial_info'].keys())
+            if verbosity > 2:
+                for trial_id in ordered_trials:
+                    logger.info("Information about each trial:  ")
+                    logger.info("Trial ID: %s" % trial_id)
+                    logger.info(json.dumps(self.results['trial_info'][trial_id], indent=2))
+            
+            # Create plot summaries:
+            plot_summary_of_models(results, output_directory)
+            plot_performance_vs_trials(results, output_directory)
+        return self.results
+    
+    def createResults(self):
+        """ Dict object to store all relevant information produced during task.fit(). 
+            Empty for now, but should be updated during task.fit().
+            All tasks should adhere to this same template for consistency.
+        """
+        results = {}
+        results['time'] = None # run-time of task.fit()
+        results['num_trials_completed'] = None # number of trials completed during task.fit() 
+        results['best_hyperparameters'] = None # hyperparameter values corresponding to the chosen model in self.model
+        results['validation_perf'] = None # validation performance achieved by the chosen model (what is currently called 'reward')
+        results['training_loss'] = None # training loss value achieved by the chosen model (on the training data)
+        results['search_space'] = None # hyperparameter search space considered in task.fit()
+        results['search_strategy'] = None # HPO algorithm used (ie. Hyperband, random, BayesOpt). If the HPO algorithm used kwargs, then this should be tuple (HPO_algorithm_string, HPO_kwargs)
+        
+        results['metadata'] = {} # dict containing other optional metadata with keys. For example:
+        # latency = inference-time of self.model (time for feedforward pass)
+        # memory = amount of memory required by self.model
+        
+        results['trial_info'] = {} # dict with keys = trial_IDs, values = dict of information about each individual trial (length = results['num_trials_completed'])
+        """ Example of what one element of this dict must look like: 
+        
+        results['trial_info'][trial_id] =  {
+            'config' : hyperparameter configuration tried in this trial
+            'validation_perf' : validation performance achieved by the model from this trial
+            'training_loss' : training loss value achieved by the model from this trial (on the training data)
+            'metadata' : dict of various optional metadata with keys such as: latency, memory, time, early_stopped, etc.
+        }
+        
+        """
+        return results
diff --git a/autogluon/utils/__init__.py b/autogluon/utils/__init__.py
index d427fbfa4ed5..fcd1338d6e0d 100644
--- a/autogluon/utils/__init__.py
+++ b/autogluon/utils/__init__.py
@@ -7,5 +7,6 @@
 from .edict import EasyDict
 from .serialization import *
 from .data_analyzer import *
+from plots import *
 from .custom_queue import Queue
 from .defaultdict import keydefaultdict
diff --git a/autogluon/utils/plots.py b/autogluon/utils/plots.py
new file mode 100644
index 000000000000..54c11c36aee2
--- /dev/null
+++ b/autogluon/utils/plots.py
@@ -0,0 +1,168 @@
+""" Methods to create various plots used throughout AutoGluon.
+    If matplotlib or bokeh are not installed, simply will print warning message that plots cannot be shown.
+"""
+
+import warnings
+import numpy as np
+from collections import OrderedDict
+import pandas as pd
+
+__all__ = ['plot_performance_vs_trials', 'plot_summary_of_models', 'mousover_plot']
+
+
+def plot_performance_vs_trials(results, output_directory, filename="PerformanceVsTrials.png"):
+    try:
+        import matplotlib.pyplot as plt
+        matplotlib_imported = True
+    except ImportError:
+        matplotlib_imported = False
+    
+    if not matplotlib_imported:
+        warnings.warn('AutoGluon summary plots cannot be created because matplotlib is not installed. Please do: "pip install matplotlib"')
+        return None
+    
+    ordered_val_perfs = [self.results['trial_info'][trial_id]['validation_perf'] for trial_id in ordered_trials]
+    x = range(len(ordered_trials))
+    y = []
+    for i in x:
+        y.append(max([ordered_val_perfs[j] for j in range(i)])) # best validation performance in trials up until ith one (assuming higher = better)
+    fig, ax = plt.subplots()
+    ax.plot(x, y)
+    ax.set(xlabel='Completed Trials', ylabel='Best Validation Performance')
+    if output_directory is not None:
+        fig.savefig(output_directory + filename)
+    plt.show()
+
+def plot_summary_of_models(results, output_directory, save_file='SummaryOfModels.html', plot_title="Models produced during fit()"):
+    """ Plot dynamic scatterplot summary of each model encountered during fit(), based on the returned Results object. 
+    """
+    num_trials = len(results['trial_info'])
+    attr_color = None
+    attr_size = None
+    datadict = {'trial_id': sorted(results['trial_info'].keys())}
+    datadict['validation_performance'] = [results['trial_info'][trial_id]['validation_perf'] for trial_id in datadict['trial_id']]
+    datadict['hyperparameters'] = [str(results['trial_info'][trial_id]['config']) for trial_id in datadict['trial_id']]
+    
+    # Determine x-axis attribute:
+    if 'latency' in results['metadata']:
+        datadict['latency'] = [results['trial_info'][trial_id]['metadata']['latency'] for trial_id in datadict['trial_id']]
+        attr_x = 'latency'
+    
+    # Determine size attribute:
+    if 'memory' in results['metadata']:
+        datadict['memory'] = [results['trial_info'][trial_id]['metadata']['memory'] for trial_id in datadict['trial_id']]
+        attr_size = 'memory'
+    
+    # Determine color attribute:
+    if 'training_loss' in results:
+        datadict['training_loss'] = [results['trial_info'][trial_id]['training_loss'] for trial_id in datadict['trial_id']]
+        attr_color = 'training_loss'
+    
+    mousover_plot(datadict, attr_x=attr_x, attr_y='validation_perf', attr_color=attr_color, 
+                  attr_size=attr_size, save_file=output_directory+save_file, plot_title=plot_title)
+
+
+def mousover_plot(datadict, attr_x, attr_y, attr_color=None, attr_size=None, save_file=None, plot_title="",
+                  point_transparency = 0.5, point_size=20, default_color="#2222aa", hidden_keys = []):
+    """ Produces dynamic scatter plot that can be interacted with by mousing over each point to see its label
+        Args:
+            datadict (dict): keys contain attributes, values of lists of data from each attribute to plot (each list index corresponds to datapoint).
+                             The values of all extra keys in this dict are considered (string) labels to assign to datapoints when they are moused over.
+            attr_x (str): name of column in dataframe whose values are shown on x-axis (eg. 'latency'). Can be categorical or numeric values
+            attr_y (str): name of column in dataframe whose values are shown on y-axis (eg. 'validation performance'). Must be numeric values.
+            attr_size (str): name of column in dataframe whose values determine size of dots (eg. 'memory consumption'). Must be numeric values.
+            attr_color (str): name of column in dataframe whose values determine color of dots  (eg. one of the hyperparameters). Can be categorical or numeric values
+            point_labels (list): list of strings describing the label for each dot (must be in same order as rows of dataframe)
+            save_file (str): where to save plot to (html) file (if None, plot is not saved)
+            plot_title (str): Title of plot and html file
+            point_transparency (float): alpha value of points, lower = more transparent
+            point_size (int): size of points, higher = larger
+            hidden keys (list[str]): which keys of datadict NOT to show labels for.
+    """
+    try:
+        import bokeh
+        from bokeh.plotting import output_file, ColumnDataSource, show, figure
+        from bokeh.models import HoverTool, CategoricalColorMapper, LinearColorMapper, Legend, LegendItem, ColorBar
+        from bokeh.palettes import Category20
+        bokeh_imported = True
+    except ImportError:
+        bokeh_imported = False
+    
+    if not bokeh_imported:
+        warnings.warn('AutoGluon summary plots cannot be created because bokeh is not installed. Please do: "pip install bokeh"')
+        return None
+    
+    n = len(datadict[attr_x])
+    for key in datadict.keys(): # Check lengths are all the same
+        if len(datadict[key]) != n:
+            raise ValueError("Key %s in datadict has different length than %s" % (key, attr_x))
+    
+    attr_x_is_string = any([type(val)==str for val in datadict[attr_x]])
+    if attr_x_is_string:
+        attr_x_levels = list(set(datadict[attr_x])) # use this to translate between int-indices and x-values
+        og_x_vals = datadict[attr_x][:]
+        attr_x2 = attr_x + "___"
+        hidden_keys.append(attr_x2)
+        datadict[attr_x2] = [attr_x_levels.index(category) for category in og_x_vals] # convert to ints
+    
+    legend = None
+    if attr_color is not None:
+        attr_color_is_string = any([type(val)==str for val in datadict[attr_color]])
+        color_datavals = datadict[attr_color]
+        if attr_color_is_string:
+            attr_color_levels = list(set(color_datavals))
+            colorpalette = Category20[20]
+            color_mapper = CategoricalColorMapper(factors=attr_color_levels, palette=[colorpalette[2*i % len(colorpalette)] for i in range(len(attr_color_levels))])
+            legend = attr_color
+        else:
+            color_mapper = LinearColorMapper(palette='Magma256', low=min(datadict[attr_color]), high=max(datadict[attr_color])*1.25)
+        default_color = {'field': attr_color, 'transform': color_mapper}
+    
+    if attr_size is not None: # different size for each point, ensure mean-size == point_size
+        attr_size2 = attr_size + "____"
+        hidden_keys.append(attr_size2)
+        og_sizevals = np.array(datadict[attr_size])
+        sizevals = point_size + (og_sizevals - np.mean(og_sizevals))/np.std(og_sizevals) * (point_size/2)
+        if np.min(sizevals) < 0:
+            sizevals = -np.min(sizevals) + sizevals + 1.0
+        datadict[attr_size2] = list(sizevals)
+        point_size = attr_size2
+    
+    if save_file is not None:
+        output_file(save_file, title=plot_title)
+    source = ColumnDataSource(datadict)
+    TOOLS="crosshair,pan,wheel_zoom,box_zoom,reset,hover,previewsave"
+    p = figure(title=plot_title, tools=TOOLS)
+    if attr_x_is_string:
+        circ = p.circle(attr_x2, attr_y, line_color=default_color, line_alpha = point_transparency,
+                fill_color = default_color, fill_alpha=point_transparency, size=point_size, source=source)
+    else:
+        circ = p.circle(attr_x, attr_y, line_color=default_color, line_alpha = point_transparency,
+                fill_color = default_color, fill_alpha=point_transparency, size=point_size, source=source)
+    hover =p.select(dict(type=HoverTool))
+    hover.tooltips = OrderedDict([(key,'@'+key) for key in datadict.keys() if key not in hidden_keys])
+    # Format axes:
+    p.xaxis.axis_label = attr_x
+    p.yaxis.axis_label = attr_y
+    if attr_x_is_string: # add x-ticks:
+        p.xaxis.ticker = range(len(attr_x_levels))
+        p.xaxis.major_label_overrides = {i: attr_x_levels[i] for i in range(len(attr_x_levels))}
+    
+    # Legend additions:
+    if attr_color is not None and attr_color_is_string:
+        legend_it = []
+        for i in range(len(attr_color_levels)):
+            legend_it.append(LegendItem(label=attr_color_levels[i], renderers = [circ], index=datadict[attr_color].index(attr_color_levels[i])))
+        legend = Legend(items=legend_it, location=(0, 0))
+        p.add_layout(legend, 'right')
+    
+    if attr_color is not None and not attr_color_is_string: 
+        color_bar = ColorBar(color_mapper=color_mapper, title = attr_color, 
+                             label_standoff=12, border_line_color=None, location=(0,0))
+        p.add_layout(color_bar, 'right')
+    
+    if attr_size is not None:
+        p.add_layout(Legend(items=[LegendItem(label='Size of points based on "'+attr_size + '"')]), 'below')
+    
+    show(p)
+


Code review comments:
diff_hunk: @@ -0,0 +1,180 @@
+""" TODOs needed to get predictor object working for any task:
+
+    - Base Task should implement @classmethod load(output_directory) method which restores a Predictor object from file (previously saved using Predictor.save(output_directory)). task.load(output_directory) can simply: return Predictor.load(output_directory)
+    
+    - Base Task.fit() needs to create Results object and assign it to predictor.results before returning predictor. The only thing task.fit should return is a single Predictor object.
+    
+    - Right before task.fit() returns predictor, it should call: predictor.save(outputdir) so that training progress is not accidentally lost.
+    
+    - task.fit() needs to have an output_directory argument where to store all outputs
+    
+    - Delete line "Results = collections.namedtuple('Results', 'model reward config time metadata')" from task.fit(), and store all info in self.results dict object defined below instead.
+    
+    - This code assumes trial_ids are sortable such that lower trial_id indicates trial was scheduled earlier than trial with higher trial_id
+    
+    - task object should have get_labels(Dataset) method
+"""
+
+import pickle, json
+from ...utils import plot_performance_vs_trials, plot_summary_of_models
+
+logger = logging.getLogger(__name__)
+
+__all__ = ['BasePredictor']
+
+PREDICTOR_FILENAME = "predictor.pkl" # Filename in which predictor object is stored. Should be hardcoded so that user only needs to specify directory where to store all training-related output files.
+RESULTS_FILENAME = "results.json" # Filename in which FitResults object is stored. Should be hardcoded so that user only needs to specify directory where to store all training-related output files.
+
+class BasePredictor(object):
+    """ 
+    Base object returned by task.fit() for each task implemented in AutoGluon.
+    
+    Example user workflow for say image classification applications:
+        # Training time:
+        >>> from autogluon import image_classification as task
+        >>> train_data = task.Dataset(traindata_filepath) 
+        >>> output_directory = '~/temp/' # any directory name specifying where to store all results
+        >>> predictor = task.fit(train_data=train_data, output_directory=output_directory)
+        >>> # To instead specify train/val split, do: predictor = task.fit(train_data=train_data, val_data=task.Dataset(valdata_filepath), output_directory=output_directory)
+        >>> results = predictor.fit_summary() # will also print out summary and create plots
+        # Inference time (may be a new Python session):
+        >>> test_data = task.Dataset(testdata_filepath)
+        >>> test_labels = task.get_labels(test_data)
+        >>> predictor = None  # We delete predictor here just to demonstrate how to load previously-trained predictor from file
+        >>> predictor = task.load(output_directory)
+        >>> batch_predictions = predictor.predict(test_data)
+        >>> performance = predictor.evaluate_predictions(y_true=test_labels, y_pred=batch_predictions) 
+        # or can instead just use equivalent shorthand: performance = predictor.evaluate(test_data)
+        # Can also do inference on just a single test example: x_i = single datapoint, eg. x_i = test_data[i]
+        >>> single_prediction = predictor.predict(x_i)
+        >>> print((x_i, single_prediction))
+    """
+    def __init__(self, loss_func, eval_func, model=None, **kwargs):
+        self.model = model # MXnet model or list of multiple models / ensemble. Each model should have its own loading/saving functionality.
+        self.loss_func = loss_func # Loss function (or string name) minimized during training
+        self.eval_func = eval_func # Evaluation function / metric applied on validation/test data to gauge predictive performance.
+        # Note: we may save a lot of headache if higher values of this eval_func metric = better, consistently across all tasks.
+        self.results = self.createResultsDict() # dict object to store all information during task.fit().
+    
+    @classmethod
+    def load(output_directory):
+        """ Load Predictor object from given directory.
+            Make sure to also load any models from files that exist in output_directory and set them = predictor.model.
+        """ 
+        filepath = output_directory + PREDICTOR_FILENAME
+        results_file = output_directory + RESULTS_FILENAME
+        predictor = pickle.load(open(filepath,"rb"))
+        predictor.results = json.load(open(results_file,'r'))
+        pass
+    
+    def save(self, output_directory):
+        """ Saves this object to file. Don't forget to save the models and the Results objects if they exist.
+            Before returning a Predictor, task.fit() should call predictor.save()
+        """
+        filepath = output_directory + PREDICTOR_FILENAME
+        self.save_model(output_directory)
+        self.save_results(output_directory)
+        self.model = None # Save model separately from Predictor object
+        self.results = None # Save results separately from Predictor object
+        pickle.dump(self, open(filepath,'wb'))
+        logger.info("Predictor saved to file: " % filepath)
+    
+    def save_results(self, output_directory):
+        """ Save results in human-readable file JSON format """
+        results_file = output_directory + RESULTS_FILENAME
+        json.dump(self.results, open(results_file, 'w'))
+    
+    @abstractmethod
+    def save_model(self, output_directory):
path: autogluon/task/base/base_predictor.py
text: @user1:
why we need `save_model` in predictor? `save` should already handle the model saving part.

@author:
The point here is that `save` pickles `self` (ie. the `Predictor` object).  Suppose `Predictor.model` is some huge ConvNet. We definitely don't want to be pickling the convnet object.  Similarly, we do not want to be pickling the `Results` dict object (since the file will not be human-readable).

This is why `Predictor.save()` first saves the model & Results objects to separate files, and then sets these `=None` before pickling itself.  I really think this sort of design should be followed across all tasks, but agree it can be abstract method to allow flexibility.  

@author:
I can also change: `save_model` -> `_save_model` and `save_results` -> `_ save_results` to make it clear these are just supposed to be internal helper methods, not user-facing?  The user is only supposed to call `Predictor.save()`

(However, note that `save_results` should not be an abstract method since we agreed the Results object should always be a dict for all tasks).

@user1:
Change them to internal method is good to me.