Repository: mlflow/mlflow
Pull request: Add autologging support for Fastai
## What changes are proposed in this pull request?

This PR includes a new flavor for Fastai v1. It supports model exporting and autologging.
The autologging functionality keeps track of most used callbacks: EarlyStopping and OneCycleScheduler. 

It's also included an improvement of log_fn_args_as_params function that replaces the deprecated function getargspec in favor of getfullargspec which supports keyword-only parameters.

Closes #2087

> Note: Pyfunc wrapper only works for tabular learners

## How is this patch tested?

Some tests similar to the Keras ones have been written to ensure model exporting and autologging works in most cases.

## Release Notes

### Is this a user-facing change?

- [ ] No. You can skip the rest of this section.
- [x] Yes. Give a description of this change to be included in the release notes for MLflow users.

Added support for Fastai models.

### What component(s) does this PR affect?

- [ ] UI
- [ ] CLI
- [x] API
- [ ] REST-API
- [ ] Examples
- [] Docs
- [ ] Tracking
- [ ] Projects
- [ ] Artifacts
- [x] Models
- [ ] Scoring
- [ ] Serving
- [ ] R
- [ ] Java
- [x] Python

### How should the PR be classified in the release notes? Choose one:

- [] `rn/breaking-change` - The PR will be mentioned in the "Breaking Changes" section
- [ ] `rn/none` - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
- [x] `rn/feature` - A new user-facing feature worth mentioning in the release notes
- [ ] `rn/bug-fix` - A user-facing bug fix worth mentioning in the release notes
- [ ] `rn/documentation` - A user-facing documentation change worth mentioning in the release notes


Problem statement:
[FR] Autologging functionality for PyTorch/fastai
## Describe the proposal

It'd be nice to add an mlflow.pytorch.autolog() API for automatically logging metrics, params & models generated via PyTorch. In particular, I'd like this to be incorporated with the fastai library (built on PyTorch), which also generates artifacts such as learning rate finder plots, and autolog those as well. 

### Motivation
PyTorch is heavily used in research for Deep Learning and is becoming very prevalent in industry as well with the fastai library. The fastai community has over 450 contributors, and it incorporates practical deep learning best practices + cutting edge algorithms in the library. In the fast.ai course that is run by USF, they use Excel spreadsheets to keep track of their runs. If we could just autolog everything once they import mlflow, it will help bring the adoption of MLflow to the fastai community. 

### Proposed Changes
See above - we propose adding a new mlflow.pytorch.autolog API

Hints from issue discussion:
@shalomma @LarsHill would either of you be interested in working on this feature? If so I'm happy to write up / help collaborate on an API proposal :)
@smurching sorry, but in the near future I won't have time working on it. I just thought it would be a useful feature.
@smurching are you working on it ? I will be happy to help :)
@pierrelandry I haven't worked on it, that'd be awesome! Before writing code & opening a PR, a good first step would be to write up a description of the proposed API & its behavior. Happy to review a proposal if you're able to write one & share it here.

I'm new to Pytorch, but I'd recommend focusing on adding autologging for the most common Pytorch APIs first (will spend some time tomorrow learning more about Pytorch). Also, I'd suggest `mlflow.pytorch.autolog()` as the API name, to match the existing Keras & TF autologging APIs, but open to alternatives. Thanks for being willing to contribute :)!

@pierrelandry hope you enjoyed the holidays! Thanks again for your interest in helping with this - are you still interested in collaborating?
Hi, I'd love to work on the fastai integration, is there any proposed API?
I thought of adding support for logging metrics during training and saving artifacts generated from
[SaveModelCallback](https://docs.fast.ai/callbacks.html#SaveModelCallback) or [Learner.save](https://docs.fast.ai/basic_train.html#Learner.save) method.
@antoniomdk sorry for the delayed response, that'd be awesome - one strawman API proposal would be to add an `mlflow.pytorch.autolog()` API, and an `mlflow.fastai.autolog()` if appropriate - AFAICT fastai & PyTorch are analogous to Keras and TensorFlow (fastai seems to be a higher-level API implemented using PyTorch), so seems reasonable to have separate APIs for the two libraries. 

That being said, it's probably worth considering whether implementing PyTorch autologging could indirectly solve fastai autologging (i.e. could we reuse the implementation of PyTorch autologging for fastai autologging?). My guess is that it might be possible, but the right way to implement fastai autologging would be instead be to use public `fastai` APIs, e.g. by adding callbacks to save & log the user's model as you described - I don't know too much about either PyTorch or fastai so curious to hear your opinion.


Yes, you're right, Fastai is a higher-level API on top of Pytorch as Keras is on top of Tensorflow. Adding model saving autologging for Pytorch would solve it also for Fastai, as the latter relies on torch.save
https://github.com/fastai/fastai/blob/master/fastai/basic_train.py#L246
So, monkey patching [torch.save](https://pytorch.org/docs/stable/torch.html?highlight=save#torch.save) would work for both libraries. It'd also work for model checkpoints along epochs. 

On the other hand, I think we should go for adding a callback to handle metric logging on Fastai.
There is also a proposal from the Fastai community to add a callback for this: https://forums.fast.ai/t/mlflow-tracking-support/38893

For Pytorch I'm not sure how we should proceed because the training cycle is done manually and the evaluation depends on the user implementing it.
Adding autologging to PyTorch is much harder inherently because PyTorch's training cycle can be very flexible and can take many forms, like @antoniomdk mentioned. I took a look at what MLFlow autologging does for other frameworks (https://www.mlflow.org/docs/latest/tracking.html#automatic-logging-from-tensorflow-and-keras-experimental), and I think we might be able to do the following:

- Metrics

| Logging Item | Possible Implementation |
| ------------- | ------------- |
| training loss  | Monkey-patch [Tensor.backward](https://pytorch.org/docs/stable/autograd.html#torch.Tensor.backward) method and log the tensor |
| validation loss  | (This is difficult to track because we can't know programmatically which variable is the validation loss. We might have to resort to manual logging.) |
| user-specified metrics | (Not sure how we should map this concept from Keras to PyTorch.) |

- Parameters

| Logging Item | Possible Implementation |
| ------------- | ------------- |
| fit() parameters  | (Not sure how we should map this concept from Keras to PyTorch.) |
| optimizer name  | Find instantiations of [torch.optim.Optimizer](https://pytorch.org/docs/stable/optim.html#torch.optim.Optimizer) subclasses in the code, and log the optimizer name |
| learning rate | Log the `lr` value of the optimizer found above |
| epsilon | Log the `eps` value of the optimizer found above |

- Artifacts

| Logging Item | Possible Implementation |
| ------------- | ------------- |
| model summary on training start | Monkey-patch `Tensor.backward` or `Optimizer.step` method and log the model summary on the first method call |
| MLFlow model (PyTorch model) on training end | The end of training is hard to detect, but maybe we can monkey-patch `Optimizer.step` or `torch.nn.Module.eval` method to trigger MLFlow model saving after every epoch |

> AFAICT fastai & PyTorch are analogous to Keras and TensorFlow (fastai seems to be a higher-level API implemented using PyTorch), so seems reasonable to have separate APIs for the two libraries.

+1 I think it's easier to add autologging to fastai as a separate API because callbacks are a cleaner way to implement autologging.

Commit message: Fix linting issues for fastai.py

Patch to review:
diff --git a/.travis.yml b/.travis.yml
index 6ec39db277109..66d88514e089f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -83,7 +83,7 @@ matrix:
         - pip install -r travis/small-requirements.txt
         - pip install -e .
       script:
-        - pytest --verbose --ignore=tests/h2o --ignore=tests/keras --ignore=tests/pytorch --ignore=tests/pyfunc --ignore=tests/sagemaker --ignore=tests/sklearn  --ignore=tests/spark --ignore=tests/tensorflow --ignore=tests/keras_autolog --ignore=tests/tensorflow_autolog --ignore tests/azureml --ignore tests/onnx --ignore tests/projects --ignore=tests/xgboost --ignore=tests/lightgbm --ignore=tests/spark_autologging tests
+        - pytest --verbose --ignore=tests/h2o --ignore=tests/keras --ignore=tests/pytorch --ignore=tests/pyfunc --ignore=tests/sagemaker --ignore=tests/sklearn  --ignore=tests/spark --ignore=tests/tensorflow --ignore=tests/keras_autolog --ignore=tests/tensorflow_autolog --ignore tests/azureml --ignore tests/onnx --ignore tests/projects --ignore=tests/xgboost --ignore=tests/lightgbm --ignore=tests/spark_autologging --ignore=tests/fastai tests
     - language: python
       python: 3.6
       name: "Docs (rsthtml, javadocs)"
diff --git a/docs/source/python_api/mlflow.fastai.rst b/docs/source/python_api/mlflow.fastai.rst
new file mode 100644
index 0000000000000..6737531d8c0bf
--- /dev/null
+++ b/docs/source/python_api/mlflow.fastai.rst
@@ -0,0 +1,7 @@
+mlflow.fastai
+===============
+
+.. automodule:: mlflow.fastai
+    :members:
+    :undoc-members:
+    :show-inheritance:
diff --git a/mlflow/fastai.py b/mlflow/fastai.py
new file mode 100644
index 0000000000000..5b3f27396adf3
--- /dev/null
+++ b/mlflow/fastai.py
@@ -0,0 +1,340 @@
+"""
+The ``mlflow.fastai`` module provides an API for logging and loading fast.ai models. This module
+exports fast.ai models with the following flavors:
+
+fastai (native) format
+    This is the main flavor that can be loaded back into fastai.
+:py:mod:`mlflow.pyfunc`
+    Produced for use by generic pyfunc-based deployment tools and batch inference.
+
+.. _fastai.Learner:
+    https://docs.fast.ai/basic_train.html#Learner
+.. _fastai.Learner.export:
+    https://docs.fast.ai/basic_train.html#Learner.export
+"""
+
+from __future__ import absolute_import
+
+import os
+import yaml
+import gorilla
+import tempfile
+import shutil
+
+from mlflow import pyfunc
+from mlflow.models import Model
+import mlflow.tracking
+from mlflow.exceptions import MlflowException
+from mlflow.tracking.artifact_utils import _download_artifact_from_uri
+from mlflow.utils.environment import _mlflow_conda_env
+from mlflow.utils.model_utils import _get_flavor_configuration
+from mlflow.utils.annotations import experimental
+from mlflow.utils.autologging_utils import try_mlflow_log, log_fn_args_as_params
+
+from fastai.tabular import TabularList
+from fastai.basic_data import DatasetType
+
+
+FLAVOR_NAME = "fastai"
+
+
+def get_default_conda_env(include_cloudpickle=False):
+    """
+    :return: The default Conda environment for MLflow Models produced by calls to
+             :func:`save_model()` and :func:`log_model()`.
+    """
+    import fastai
+    pip_deps = None
+    if include_cloudpickle:
+        import cloudpickle
+        pip_deps = ["cloudpickle=={}".format(cloudpickle.__version__)]
+    return _mlflow_conda_env(
+        additional_conda_deps=[
+            "fastai={}".format(fastai.__version__),
+        ],
+        additional_pip_deps=pip_deps,
+        additional_conda_channels=None
+    )
+
+
+def save_model(fastai_learner, path, conda_env=None, mlflow_model=Model(), **kwargs):
+    """
+    Save a fastai Learner to a path on the local file system.
+
+    :param fastai_learner: fastai Learner to be saved.
+    :param path: Local path where the model is to be saved.
+    :param conda_env: Either a dictionary representation of a Conda environment or the path to a
+                      Conda environment yaml file. If provided, this describes the environment
+                      this model should be run in. At minimum, it should specify the
+                      dependencies contained in :func:`get_default_conda_env()`. If
+                      ``None``, the default :func:`get_default_conda_env()` environment is
+                      added to the model. The following is an *example* dictionary
+                      representation of a Conda environment::
+
+                        {
+                            'name': 'mlflow-env',
+                            'channels': ['defaults'],
+                            'dependencies': [
+                                'python=3.7.0',
+                                'fastai=1.0.60',
+                            ]
+                        }
+    :param mlflow_model: MLflow model config this flavor is being added to.
+    :param kwargs: kwargs to pass to ``Learner.save`` method.
+    """
+    import fastai
+    from pathlib import Path
+
+    path = os.path.abspath(path)
+    if os.path.exists(path):
+        raise MlflowException("Path '{}' already exists".format(path))
+    model_data_subpath = "model.fastai"
+    model_data_path = os.path.join(path, model_data_subpath)
+    model_data_path = Path(model_data_path)
+    os.makedirs(path)
+
+    # Save an Learner
+    fastai_learner.export(model_data_path, **kwargs)
+
+    conda_env_subpath = "conda.yaml"
+
+    if conda_env is None:
+        conda_env = get_default_conda_env()
+    elif not isinstance(conda_env, dict):
+        with open(conda_env, "r") as f:
+            conda_env = yaml.safe_load(f)
+    with open(os.path.join(path, conda_env_subpath), "w") as f:
+        yaml.safe_dump(conda_env, stream=f, default_flow_style=False)
+
+    pyfunc.add_to_model(mlflow_model, loader_module="mlflow.fastai",
+                        data=model_data_subpath, env=conda_env_subpath)
+    mlflow_model.add_flavor(FLAVOR_NAME, fastai_version=fastai.__version__, data=model_data_subpath)
+    mlflow_model.save(os.path.join(path, "MLmodel"))
+
+
+def log_model(fastai_learner, artifact_path, conda_env=None, registered_model_name=None, **kwargs):
+    """
+    Log a fastai model as an MLflow artifact for the current run.
+
+    :param fastai_learner: Fastai model (an instance of `fastai.Learner`_) to be saved.
+    :param artifact_path: Run-relative artifact path.
+    :param conda_env: Either a dictionary representation of a Conda environment or the path to a
+                      Conda environment yaml file. If provided, this describes the environment
+                      this model should be run in. At minimum, it should specify the dependencies
+                      contained in :func:`get_default_conda_env()`. If ``None``, the default
+                      :func:`get_default_conda_env()` environment is added to the model.
+                      The following is an *example* dictionary representation of a Conda
+                      environment::
+
+                        {
+                            'name': 'mlflow-env',
+                            'channels': ['defaults'],
+                            'dependencies': [
+                                'python=3.7.0',
+                                'fastai=1.0.60',
+                            ]
+                        }
+    :param registered_model_name: Note:: Experimental: This argument may change or be removed in a
+                                  future release without warning. If given, create a model
+                                  version under ``registered_model_name``, also creating a
+                                  registered model if one with the given name does not exist.
+    :param kwargs: kwargs to pass to `fastai.Learner.export`_ method.
+    """
+    Model.log(artifact_path=artifact_path, flavor=mlflow.fastai,
+              registered_model_name=registered_model_name,
+              fastai_learner=fastai_learner, conda_env=conda_env, **kwargs)
+
+
+def _load_model(path):
+    from fastai.basic_train import load_learner
+    abspath = os.path.abspath(path)
+    path, file = os.path.split(abspath)
+    return load_learner(path, file)
+
+
+class _FastaiModelWrapper:
+    def __init__(self, learner):
+        self.learner = learner
+
+    def predict(self, dataframe):
+        test_data = TabularList.from_df(dataframe, cont_names=self.learner.data.cont_names)
+        self.learner.data.add_test(test_data)
+        return self.learner.get_preds(DatasetType.Test)[0].numpy()
+
+
+def _load_pyfunc(path):
+    """
+    Load PyFunc implementation. Called by ``pyfunc.load_pyfunc``.
+
+    :param path: Local filesystem path to the MLflow Model with the ``fastai`` flavor.
+    """
+    return _FastaiModelWrapper(_load_model(path))
+
+
+def load_model(model_uri):
+    """
+    Load a fastai model from a local file or a run.
+
+    :param model_uri: The location, in URI format, of the MLflow model. For example:
+
+                      - ``/Users/me/path/to/local/model``
+                      - ``relative/path/to/local/model``
+                      - ``s3://my_bucket/path/to/model``
+                      - ``runs:/<mlflow_run_id>/run-relative/path/to/model``
+
+                      For more information about supported URI schemes, see
+                      `Referencing Artifacts <https://www.mlflow.org/docs/latest/tracking.html#
+                      artifact-locations>`_.
+
+    :return: A fastai model (an instance of `fastai.Learner`_).
+    """
+    local_model_path = _download_artifact_from_uri(artifact_uri=model_uri)
+    flavor_conf = _get_flavor_configuration(model_path=local_model_path, flavor_name=FLAVOR_NAME)
+    model_file_path = os.path.join(local_model_path, flavor_conf.get("data", "model.fastai"))
+    return _load_model(path=model_file_path)
+
+
+@experimental
+def autolog():
+    """
+    Enable automatic logging from Fastai to MLflow.
+    Logs loss and any other metrics specified in the fit
+    function, and optimizer data as parameters. Model checkpoints
+    are logged as artifacts to a 'models' directory.
+
+    MLflow will also log the parameters of the EarlyStopping and OneCycleScheduler callbacks
+    """
+    from fastai.basic_train import LearnerCallback, Learner
+    from fastai.callbacks.hooks import model_summary, layers_info
+    from fastai.callbacks import EarlyStoppingCallback, OneCycleScheduler
+
+    class __MLflowFastaiCallback(LearnerCallback):
+        """
+        Callback for auto-logging metrics and parameters.
+        Records model structural information as params when training begins
+        """
+        def __init__(self, learner, ):
+            super().__init__(learner)
+            self.learner = learner
+            self.opt = self.learn.opt
+            self.metrics_names = ['train_loss', 'valid_loss'] + \
+                                 [o.__name__ for o in learner.metrics]
+
+        def on_epoch_end(self, **kwargs):
+            """
+            Log loss and other metrics values after each epoch
+            """
+            if kwargs['smooth_loss'] is None or kwargs["last_metrics"] is None:
+                return
+            epoch = kwargs['epoch']
+            metrics = [kwargs['smooth_loss']] + kwargs["last_metrics"]
+            metrics = map(float, metrics)
+            metrics = dict(zip(self.metrics_names, metrics))
+            try_mlflow_log(mlflow.log_metrics, metrics, step=epoch)
+
+        def on_train_begin(self, **kwargs):
+            info = layers_info(self.learner)
+            try_mlflow_log(mlflow.log_param, 'num_layers', len(info))
+            try_mlflow_log(mlflow.log_param, 'opt_func', self.opt_func.func.__name__)
+
+            if hasattr(self.opt, 'true_wd'):
+                try_mlflow_log(mlflow.log_param, 'true_wd', self.opt.true_wd)
+
+            if hasattr(self.opt, 'bn_wd'):
+                try_mlflow_log(mlflow.log_param, 'bn_wd', self.opt.bn_wd)
+
+            if hasattr(self.opt, 'train_bn'):
+                try_mlflow_log(mlflow.log_param, 'train_bn', self.train_bn)
+
+            summary = model_summary(self.learner)
+            try_mlflow_log(mlflow.set_tag, 'model_summary', summary)
+
+            tempdir = tempfile.mkdtemp()
+            try:
+                summary_file = os.path.join(tempdir, "model_summary.txt")
+                with open(summary_file, 'w') as f:
+                    f.write(summary)
+                try_mlflow_log(mlflow.log_artifact, local_path=summary_file)
+            finally:
+                shutil.rmtree(tempdir)
+
+        def on_train_end(self, **kwargs):
+            try_mlflow_log(log_model, self.learner, artifact_path='model')
+
+    def _find_callback_of_type(callback_type, callbacks):
+        for callback in callbacks:
+            if isinstance(callback, callback_type):
+                return callback
+        return None
+
+    def _log_early_stop_callback_params(callback):
+        if callback:
+            try:
+                earlystopping_params = {'early_stop_monitor': callback.monitor,
+                                        'early_stop_min_delta': callback.min_delta,
+                                        'early_stop_patience': callback.patience,
+                                        'early_stop_mode': callback.mode}
+                try_mlflow_log(mlflow.log_params, earlystopping_params)
+            except Exception:  # pylint: disable=W0703
+                return
+
+    def _log_one_cycle_callback_params(callback):
+        if callback:
+            try:
+                params = {
+                    'lr_max': callback.lr_max,
+                    'div_factor': callback.div_factor,
+                    'pct_start': callback.pct_start,
+                    'final_div': callback.final_div,
+                    'tot_epochs': callback.tot_epochs,
+                    'start_epoch': callback.start_epoch,
+                    'moms': callback.moms,
+                }
+                try_mlflow_log(mlflow.log_params, params)
+            except Exception:  # pylint: disable=W0703
+                return
+
+    def _run_and_log_function(self, original, args, kwargs, unlogged_params, callback_arg_index):
+        if not mlflow.active_run():
+            try_mlflow_log(mlflow.start_run)
+            auto_end_run = True
+        else:
+            auto_end_run = False
+
+        log_fn_args_as_params(original, [self] + list(args), kwargs, unlogged_params)
+
+        callbacks = [cb(self) for cb in self.callback_fns] + (self.callbacks or [])
+
+        # Checking if the 'callback' argument of the function is set
+        if len(args) > callback_arg_index:
+            tmp_list = list(args)
+            callbacks += list(args[callback_arg_index])
+            tmp_list[callback_arg_index] += [__MLflowFastaiCallback(self)]
+            args = tuple(tmp_list)
+        elif 'callbacks' in kwargs:
+            callbacks += list(kwargs['callbacks'])
+            kwargs['callbacks'] += [__MLflowFastaiCallback(self)]
+        else:
+            kwargs['callbacks'] = [__MLflowFastaiCallback(self)]
+
+        early_stop_callback = _find_callback_of_type(EarlyStoppingCallback, callbacks)
+        one_cycle_callback = _find_callback_of_type(OneCycleScheduler, callbacks)
+
+        _log_early_stop_callback_params(early_stop_callback)
+        _log_one_cycle_callback_params(one_cycle_callback)
+
+        result = original(self, *args, **kwargs)
+
+        if auto_end_run:
+            try_mlflow_log(mlflow.end_run)
+
+        return result
+
+    @gorilla.patch(Learner)
+    def fit(self, *args, **kwargs):
+        original = gorilla.get_original_attribute(Learner, 'fit')
+        unlogged_params = ['self', 'callbacks', 'learner']
+        return _run_and_log_function(self, original, args, kwargs, unlogged_params, 3)
+
+    settings = gorilla.Settings(allow_hit=True, store_hit=True)
+    gorilla.apply(gorilla.Patch(Learner, 'fit', fit, settings=settings))
diff --git a/mlflow/utils/autologging_utils.py b/mlflow/utils/autologging_utils.py
index 1468687129c25..3fe28c930cabb 100644
--- a/mlflow/utils/autologging_utils.py
+++ b/mlflow/utils/autologging_utils.py
@@ -57,7 +57,12 @@ def log_fn_args_as_params(fn, args, kwargs, unlogged=[]):  # pylint: disable=W01
     """
     # all_default_values has length n, corresponding to values of the
     # last n elements in all_param_names
-    all_param_names, _, _, all_default_values = inspect.getargspec(fn)  # pylint: disable=W1505
+    pos_params, _, _, pos_defaults, kw_params, kw_defaults, _ = inspect.getfullargspec(fn)
+
+    kw_params = list(kw_params) if kw_params else []
+    pos_defaults = list(pos_defaults) if pos_defaults else []
+    all_param_names = pos_params + kw_params
+    all_default_values = pos_defaults + [kw_defaults[param] for param in kw_params]
 
     # Checking if default values are present for logging. Known bug that getargspec will return an
     # empty argspec for certain functions, despite the functions having an argspec.
@@ -71,9 +76,10 @@ def log_fn_args_as_params(fn, args, kwargs, unlogged=[]):  # pylint: disable=W01
 
     # Logging the arguments passed by the user
     args_dict = dict((param_name, param_val) for param_name, param_val
-                     in zip(all_param_names[:len(args)], args)
+                     in zip(all_param_names, args)
                      if param_name not in unlogged)
-    if len(args_dict.keys()) > 0:
+
+    if args_dict:
         try_mlflow_log(mlflow.log_params, args_dict)
 
     # Logging the kwargs passed by the user
diff --git a/tests/fastai/test_fastai_autolog.py b/tests/fastai/test_fastai_autolog.py
new file mode 100644
index 0000000000000..bc3d548ff8c3c
--- /dev/null
+++ b/tests/fastai/test_fastai_autolog.py
@@ -0,0 +1,258 @@
+import pytest
+import numpy as np
+from tests.projects.utils import tracking_uri_mock  # pylint: disable=W0611
+
+import pandas as pd
+import sklearn.datasets as datasets
+from fastai.tabular import tabular_learner, TabularList
+from fastai.metrics import accuracy
+import mlflow  # noqa
+import mlflow.fastai  # noqa
+from fastai.callbacks import EarlyStoppingCallback, SaveModelCallback
+
+np.random.seed(1337)
+
+LARGE_EPOCHS = 10
+
+
+@pytest.fixture(params=[True, False])
+def manual_run(request, tracking_uri_mock):
+    if request.param:
+        mlflow.start_run()
+    yield
+    mlflow.end_run()
+
+
+def iris_dataframe():
+    iris = datasets.load_iris()
+    return pd.DataFrame(iris.data[:, :2], columns=iris.feature_names[:2])
+
+
+@pytest.fixture(scope="session")
+def iris_data():
+    iris = datasets.load_iris()
+    X = pd.DataFrame(iris.data[:, :2], columns=iris.feature_names[:2])
+    y = pd.Series(iris.target, name='label')
+    return (TabularList.from_df(pd.concat([X, y], axis=1), cont_names=list(X.columns))
+            .split_by_rand_pct(valid_pct=0.1, seed=42)
+            .label_from_df(cols='label')
+            .databunch())
+
+
+def fastai_model(data, **kwargs):
+    return tabular_learner(data, metrics=accuracy, layers=[5, 3, 2], **kwargs)
+
+
+@pytest.mark.large
+@pytest.mark.parametrize('fit_variant', ['fit', 'fit_one_cycle'])
+def test_fastai_autolog_ends_auto_created_run(iris_data, fit_variant):
+    mlflow.fastai.autolog()
+    model = fastai_model(iris_data)
+    if fit_variant == 'fit_one_cycle':
+        model.fit_one_cycle(1)
+    else:
+        model.fit(1)
+    assert mlflow.active_run() is None
+
+
+@pytest.mark.large
+@pytest.mark.parametrize('fit_variant', ['fit', 'fit_one_cycle'])
+def test_fastai_autolog_persists_manually_created_run(iris_data, fit_variant):
+    mlflow.fastai.autolog()
+
+    with mlflow.start_run() as run:
+        model = fastai_model(iris_data)
+
+        if fit_variant == 'fit_one_cycle':
+            model.fit_one_cycle(LARGE_EPOCHS)
+        else:
+            model.fit(LARGE_EPOCHS)
+
+        assert mlflow.active_run()
+        assert mlflow.active_run().info.run_id == run.info.run_id
+
+
+@pytest.fixture
+def fastai_random_data_run(iris_data, fit_variant, manual_run):
+    mlflow.fastai.autolog()
+
+    model = fastai_model(iris_data)
+
+    if fit_variant == 'fit_one_cycle':
+        model.fit_one_cycle(LARGE_EPOCHS)
+    else:
+        model.fit(LARGE_EPOCHS)
+
+    client = mlflow.tracking.MlflowClient()
+    return model, client.get_run(client.list_run_infos(experiment_id='0')[0].run_id)
+
+
+@pytest.mark.large
+@pytest.mark.parametrize('fit_variant', ['fit', 'fit_one_cycle'])
+def test_fastai_autolog_logs_expected_data(fastai_random_data_run, fit_variant):
+    model, run = fastai_random_data_run
+    data = run.data
+
+    # Testing metrics are logged
+    assert 'train_loss' in data.metrics
+    assert 'valid_loss' in data.metrics
+
+    for o in model.metrics:
+        assert o.__name__ in data.metrics
+
+    client = mlflow.tracking.MlflowClient()
+    metric_history = client.get_metric_history(run.info.run_id, 'valid_loss')
+    assert np.array_equal([m.value for m in metric_history], model.recorder.val_losses)
+
+    # Testing explicitly passed parameters are logged correctly
+    assert 'epochs' in data.params
+    assert data.params['epochs'] == str(LARGE_EPOCHS)
+
+    # Testing implicitly passed parameters are logged correctly
+    assert 'wd' in data.params
+
+    # Testing unwanted parameters are not logged
+    assert 'callbacks' not in data.params
+    assert 'learn' not in data.params
+
+    # Testing optimizer parameters are logged
+    assert 'opt_func' in data.params
+    assert data.params['opt_func'] == 'Adam'
+    assert 'model_summary' in data.tags
+
+    # Testing model_summary.txt is saved
+    client = mlflow.tracking.MlflowClient()
+    artifacts = client.list_artifacts(run.info.run_id)
+    artifacts = map(lambda x: x.path, artifacts)
+    assert 'model_summary.txt' in artifacts
+
+
+@pytest.mark.large
+@pytest.mark.parametrize('fit_variant', ['fit', 'fit_one_cycle'])
+def test_fastai_autolog_logs_default_params(fastai_random_data_run, fit_variant):
+    _, run = fastai_random_data_run
+    if fit_variant == 'fit':
+        assert 'lr' in run.data.params
+        assert run.data.params['lr'] == 'slice(None, 0.003, None)'
+    else:
+        assert 'pct_start' in run.data.params
+        assert run.data.params['pct_start'] == '0.3'
+
+
+@pytest.mark.large
+@pytest.mark.parametrize('fit_variant', ['fit', 'fit_one_cycle'])
+def test_fastai_autolog_model_can_load_from_artifact(fastai_random_data_run):
+    run_id = fastai_random_data_run[1].info.run_id
+    client = mlflow.tracking.MlflowClient()
+    artifacts = client.list_artifacts(run_id)
+    artifacts = map(lambda x: x.path, artifacts)
+    assert 'model' in artifacts
+    model = mlflow.fastai.load_model("runs:/" + run_id + "/model")
+    model_wrapper = mlflow.fastai._FastaiModelWrapper(model)
+    model_wrapper.predict(iris_dataframe())
+
+
+@pytest.fixture
+def fastai_random_data_run_with_callback(iris_data, fit_variant, manual_run, callback, patience):
+    mlflow.fastai.autolog()
+    callbacks = []
+
+    if callback == 'early':
+        # min_delta is set as such to guarantee early stopping
+        callbacks.append(lambda learn: EarlyStoppingCallback(
+            learn, patience=patience, min_delta=99999999))
+
+    model = fastai_model(iris_data, callback_fns=callbacks)
+
+    if fit_variant == 'fit_one_cycle':
+        model.fit_one_cycle(LARGE_EPOCHS)
+    else:
+        model.fit(LARGE_EPOCHS)
+
+    client = mlflow.tracking.MlflowClient()
+    return model, client.get_run(client.list_run_infos(experiment_id='0')[0].run_id)
+
+
+@pytest.mark.large
+@pytest.mark.parametrize('fit_variant', ['fit', 'fit_one_cycle'])
+@pytest.mark.parametrize('callback', ['early'])
+@pytest.mark.parametrize('patience', [0, 1, 5])
+def test_fastai_autolog_early_stop_logs(fastai_random_data_run_with_callback, patience):
+    model, run = fastai_random_data_run_with_callback
+    metrics = run.data.metrics
+    params = run.data.params
+    assert 'early_stop_patience' in params
+    assert params['early_stop_patience'] == str(patience)
+    assert 'early_stop_monitor' in params
+    assert params['early_stop_monitor'] == 'valid_loss'
+    assert 'early_stop_mode' in params
+    assert params['early_stop_mode'] == 'auto'
+    assert 'early_stop_min_delta' in params
+    assert params['early_stop_min_delta'] == f'-{99999999}'
+
+    """
+    assert 'stopped_epoch' in metrics
+    assert 'restored_epoch' in metrics
+    restored_epoch = int(metrics['restored_epoch'])
+    assert int(metrics['stopped_epoch']) - max(1, callback.patience) == restored_epoch
+    """
+
+    client = mlflow.tracking.MlflowClient()
+    metric_history = client.get_metric_history(run.info.run_id, 'valid_loss')
+    num_of_epochs = len(model.recorder.val_losses)
+    # Check the test epoch numbers are correct
+    # Check that MLflow has logged the metrics of the "best" model
+    assert len(metric_history) == num_of_epochs
+    # Check that MLflow has logged the correct data
+    # assert history.history['loss'][restored_epoch] == metric_history[-1].value
+
+
+@pytest.mark.large
+@pytest.mark.parametrize('fit_variant', ['fit', 'fit_one_cycle'])
+@pytest.mark.parametrize('callback', ['early'])
+@pytest.mark.parametrize('patience', [11])
+def test_fastai_autolog_early_stop_no_stop_does_not_log(
+        fastai_random_data_run_with_callback, patience):
+    model, run, = fastai_random_data_run_with_callback
+    metrics = run.data.metrics
+    params = run.data.params
+    assert 'early_stop_patience' in params
+    assert params['early_stop_patience'] == str(patience)
+    assert 'early_stop_monitor' in params
+    assert params['early_stop_monitor'] == 'valid_loss'
+    assert 'early_stop_mode' in params
+    assert 'early_stop_min_delta' in params
+    assert params['early_stop_min_delta'] == f'-{99999999}'
+    """
+    assert 'stopped_epoch' in metrics
+    assert metrics['stopped_epoch'] == 0
+    assert 'restored_epoch' not in metrics
+    """
+    num_of_epochs = len(model.recorder.val_losses)
+    client = mlflow.tracking.MlflowClient()
+    metric_history = client.get_metric_history(run.info.run_id, 'valid_loss')
+    # Check the test epoch numbers are correct
+    assert num_of_epochs == LARGE_EPOCHS
+    assert len(metric_history) == num_of_epochs
+
+
+@pytest.mark.large
+@pytest.mark.parametrize('fit_variant', ['fit', 'fit_one_cycle'])
+@pytest.mark.parametrize('callback', ['not-early'])
+@pytest.mark.parametrize('patience', [5])
+def test_fastai_autolog_non_early_stop_callback_does_not_log(fastai_random_data_run_with_callback):
+    model, run, = fastai_random_data_run_with_callback
+    metrics = run.data.metrics
+    params = run.data.params
+    assert 'early_stop_patience' not in params
+    assert 'early_stop_monitor' not in params
+    assert 'early_stop_mode' not in params
+    assert 'stopped_epoch' not in metrics
+    assert 'restored_epoch' not in metrics
+    assert 'early_stop_min_delta' not in params
+    num_of_epochs = len(model.recorder.val_losses)
+    client = mlflow.tracking.MlflowClient()
+    metric_history = client.get_metric_history(run.info.run_id, 'valid_loss')
+    # Check the test epoch numbers are correct
+    assert num_of_epochs == LARGE_EPOCHS
+    assert len(metric_history) == num_of_epochs
diff --git a/tests/fastai/test_fastai_model_export.py b/tests/fastai/test_fastai_model_export.py
new file mode 100644
index 0000000000000..9f8d6bc1159f3
--- /dev/null
+++ b/tests/fastai/test_fastai_model_export.py
@@ -0,0 +1,287 @@
+from __future__ import print_function
+
+import mock
+import os
+import pytest
+import yaml
+import json
+from collections import namedtuple
+
+import numpy as np
+import pandas as pd
+import pandas.testing
+import sklearn.datasets as datasets
+from fastai.tabular import tabular_learner, TabularList
+from fastai.metrics import accuracy
+
+import mlflow.fastai
+import mlflow.utils
+import mlflow.pyfunc.scoring_server as pyfunc_scoring_server
+from mlflow import pyfunc
+from mlflow.models import Model
+from mlflow.store.artifact.s3_artifact_repo import S3ArtifactRepository
+from mlflow.tracking.artifact_utils import _download_artifact_from_uri
+from mlflow.utils.environment import _mlflow_conda_env
+from mlflow.utils.file_utils import TempDir
+from mlflow.utils.model_utils import _get_flavor_configuration
+
+from tests.helper_functions import set_boto_credentials  # pylint: disable=unused-import
+from tests.helper_functions import mock_s3_bucket  # pylint: disable=unused-import
+from tests.helper_functions import score_model_in_sagemaker_docker_container
+from tests.projects.utils import tracking_uri_mock  # pylint: disable=unused-import
+
+ModelWithData = namedtuple("ModelWithData", ["model", "inference_dataframe"])
+
+
+@pytest.fixture(scope="session")
+def fastai_model():
+    iris = datasets.load_iris()
+    X = pd.DataFrame(iris.data[:, :2], columns=iris.feature_names[:2])
+    y = pd.Series(iris.target, name='label')
+    data = (TabularList.from_df(pd.concat([X, y], axis=1), cont_names=list(X.columns))
+            .split_by_rand_pct(valid_pct=0.1, seed=42)
+            .label_from_df(cols='label')
+            .databunch())
+    model = tabular_learner(data, metrics=accuracy, layers=[3])
+    model.fit(1)
+    return ModelWithData(model=model, inference_dataframe=X)
+
+
+@pytest.fixture
+def model_path(tmpdir):
+    return os.path.join(str(tmpdir), "model")
+
+
+@pytest.fixture
+def fastai_custom_env(tmpdir):
+    conda_env = os.path.join(str(tmpdir), "conda_env.yml")
+    _mlflow_conda_env(
+            conda_env,
+            additional_pip_deps=["fastai", "pytest"])
+    return conda_env
+
+
+@pytest.mark.large
+def test_model_save_load(fastai_model, model_path):
+    model = fastai_model.model
+
+    mlflow.fastai.save_model(fastai_learner=model, path=model_path)
+    reloaded_model = mlflow.fastai.load_model(model_uri=model_path)
+    reloaded_pyfunc = pyfunc.load_model(model_uri=model_path)
+
+    model_wrapper = mlflow.fastai._FastaiModelWrapper(model)
+    reloaded_model_wrapper = mlflow.fastai._FastaiModelWrapper(reloaded_model)
+
+    np.testing.assert_array_almost_equal(
+            model_wrapper.predict(fastai_model.inference_dataframe),
+            reloaded_model_wrapper.predict(fastai_model.inference_dataframe))
+
+    np.testing.assert_array_almost_equal(
+            reloaded_model_wrapper.predict(fastai_model.inference_dataframe),
+            reloaded_pyfunc.predict(fastai_model.inference_dataframe))
+
+
+@pytest.mark.large
+def test_model_load_from_remote_uri_succeeds(fastai_model, model_path, mock_s3_bucket):
+    model = fastai_model.model
+
+    mlflow.fastai.save_model(fastai_learner=fastai_model.model, path=model_path)
+    artifact_root = "s3://{bucket_name}".format(bucket_name=mock_s3_bucket)
+    artifact_path = "model"
+    artifact_repo = S3ArtifactRepository(artifact_root)
+    artifact_repo.log_artifacts(model_path, artifact_path=artifact_path)
+
+    model_uri = artifact_root + "/" + artifact_path
+    reloaded_model = mlflow.fastai.load_model(model_uri=model_uri)
+
+    model_wrapper = mlflow.fastai._FastaiModelWrapper(model)
+    reloaded_model_wrapper = mlflow.fastai._FastaiModelWrapper(reloaded_model)
+
+    np.testing.assert_array_almost_equal(
+            model_wrapper.predict(fastai_model.inference_dataframe),
+            reloaded_model_wrapper.predict(fastai_model.inference_dataframe))
+
+
+@pytest.mark.large
+def test_model_log(fastai_model, model_path):
+    old_uri = mlflow.get_tracking_uri()
+    model = fastai_model.model
+    with TempDir(chdr=True, remove_on_exit=True) as tmp:
+        for should_start_run in [False, True]:
+            try:
+                mlflow.set_tracking_uri("test")
+                if should_start_run:
+                    mlflow.start_run()
+
+                artifact_path = "model"
+                conda_env = os.path.join(tmp.path(), "conda_env.yaml")
+                _mlflow_conda_env(conda_env, additional_pip_deps=["fastai"])
+
+                mlflow.fastai.log_model(
+                        fastai_learner=model,
+                        artifact_path=artifact_path,
+                        conda_env=conda_env)
+
+                model_uri = "runs:/{run_id}/{artifact_path}".format(
+                    run_id=mlflow.active_run().info.run_id,
+                    artifact_path=artifact_path)
+
+                reloaded_model = mlflow.fastai.load_model(model_uri=model_uri)
+
+                model_wrapper = mlflow.fastai._FastaiModelWrapper(model)
+                reloaded_model_wrapper = mlflow.fastai._FastaiModelWrapper(reloaded_model)
+
+                np.testing.assert_array_almost_equal(
+                        model_wrapper.predict(fastai_model.inference_dataframe),
+                        reloaded_model_wrapper.predict(fastai_model.inference_dataframe))
+
+                model_path = _download_artifact_from_uri(artifact_uri=model_uri)
+                model_config = Model.load(os.path.join(model_path, "MLmodel"))
+                assert pyfunc.FLAVOR_NAME in model_config.flavors
+                assert pyfunc.ENV in model_config.flavors[pyfunc.FLAVOR_NAME]
+                env_path = model_config.flavors[pyfunc.FLAVOR_NAME][pyfunc.ENV]
+                assert os.path.exists(os.path.join(model_path, env_path))
+
+            finally:
+                mlflow.end_run()
+                mlflow.set_tracking_uri(old_uri)
+
+
+def test_log_model_calls_register_model(tracking_uri_mock, fastai_model):
+    artifact_path = "model"
+    register_model_patch = mock.patch("mlflow.register_model")
+    with mlflow.start_run(), register_model_patch, TempDir(chdr=True, remove_on_exit=True) as tmp:
+        conda_env = os.path.join(tmp.path(), "conda_env.yaml")
+        _mlflow_conda_env(conda_env, additional_pip_deps=["fastai"])
+        mlflow.fastai.log_model(fastai_learner=fastai_model.model,
+                                artifact_path=artifact_path,
+                                conda_env=conda_env,
+                                registered_model_name="AdsModel1")
+        model_uri = "runs:/{run_id}/{artifact_path}".format(run_id=mlflow.active_run().info.run_id,
+                                                            artifact_path=artifact_path)
+        mlflow.register_model.assert_called_once_with(model_uri, "AdsModel1")
+
+
+def test_log_model_no_registered_model_name(tracking_uri_mock, fastai_model):
+    artifact_path = "model"
+    register_model_patch = mock.patch("mlflow.register_model")
+    with mlflow.start_run(), register_model_patch, TempDir(chdr=True, remove_on_exit=True) as tmp:
+        conda_env = os.path.join(tmp.path(), "conda_env.yaml")
+        _mlflow_conda_env(conda_env, additional_pip_deps=["fastai"])
+        mlflow.fastai.log_model(fastai_learner=fastai_model.model,
+                                artifact_path=artifact_path,
+                                conda_env=conda_env)
+        mlflow.register_model.assert_not_called()
+
+
+@pytest.mark.large
+def test_model_save_persists_specified_conda_env_in_mlflow_model_directory(
+        fastai_model, model_path, fastai_custom_env):
+
+    mlflow.fastai.save_model(
+            fastai_learner=fastai_model.model, path=model_path, conda_env=fastai_custom_env)
+
+    pyfunc_conf = _get_flavor_configuration(model_path=model_path, flavor_name=pyfunc.FLAVOR_NAME)
+    saved_conda_env_path = os.path.join(model_path, pyfunc_conf[pyfunc.ENV])
+    assert os.path.exists(saved_conda_env_path)
+    assert saved_conda_env_path != fastai_custom_env
+
+    with open(fastai_custom_env, "r") as f:
+        fastai_custom_env_parsed = yaml.safe_load(f)
+    with open(saved_conda_env_path, "r") as f:
+        saved_conda_env_parsed = yaml.safe_load(f)
+    assert saved_conda_env_parsed == fastai_custom_env_parsed
+
+
+@pytest.mark.large
+def test_model_save_accepts_conda_env_as_dict(fastai_model, model_path):
+    conda_env = dict(mlflow.fastai.get_default_conda_env())
+    conda_env["dependencies"].append("pytest")
+    mlflow.fastai.save_model(fastai_learner=fastai_model.model,
+                             path=model_path, conda_env=conda_env)
+
+    pyfunc_conf = _get_flavor_configuration(model_path=model_path, flavor_name=pyfunc.FLAVOR_NAME)
+    saved_conda_env_path = os.path.join(model_path, pyfunc_conf[pyfunc.ENV])
+    assert os.path.exists(saved_conda_env_path)
+
+    with open(saved_conda_env_path, "r") as f:
+        saved_conda_env_parsed = yaml.safe_load(f)
+    assert saved_conda_env_parsed == conda_env
+
+
+@pytest.mark.large
+def test_model_log_persists_specified_conda_env_in_mlflow_model_directory(
+        fastai_model, fastai_custom_env):
+    artifact_path = "model"
+    with mlflow.start_run():
+        mlflow.fastai.log_model(fastai_learner=fastai_model.model,
+                                artifact_path=artifact_path,
+                                conda_env=fastai_custom_env)
+        model_uri = "runs:/{run_id}/{artifact_path}".format(
+            run_id=mlflow.active_run().info.run_id,
+            artifact_path=artifact_path)
+
+    model_path = _download_artifact_from_uri(artifact_uri=model_uri)
+    pyfunc_conf = _get_flavor_configuration(model_path=model_path, flavor_name=pyfunc.FLAVOR_NAME)
+    saved_conda_env_path = os.path.join(model_path, pyfunc_conf[pyfunc.ENV])
+    assert os.path.exists(saved_conda_env_path)
+    assert saved_conda_env_path != fastai_custom_env
+
+    with open(fastai_custom_env, "r") as f:
+        fastai_custom_env_parsed = yaml.safe_load(f)
+    with open(saved_conda_env_path, "r") as f:
+        saved_conda_env_parsed = yaml.safe_load(f)
+    assert saved_conda_env_parsed == fastai_custom_env_parsed
+
+
+@pytest.mark.large
+def test_model_save_without_specified_conda_env_uses_default_env_with_expected_dependencies(
+        fastai_model, model_path):
+    mlflow.fastai.save_model(fastai_learner=fastai_model.model,
+                             path=model_path, conda_env=None)
+
+    pyfunc_conf = _get_flavor_configuration(model_path=model_path, flavor_name=pyfunc.FLAVOR_NAME)
+    conda_env_path = os.path.join(model_path, pyfunc_conf[pyfunc.ENV])
+    with open(conda_env_path, "r") as f:
+        conda_env = yaml.safe_load(f)
+
+    assert conda_env == mlflow.fastai.get_default_conda_env()
+
+
+@pytest.mark.large
+def test_model_log_without_specified_conda_env_uses_default_env_with_expected_dependencies(
+        fastai_model):
+    artifact_path = "model"
+    with mlflow.start_run():
+        mlflow.fastai.log_model(fastai_learner=fastai_model.model,
+                                artifact_path=artifact_path, conda_env=None)
+        model_uri = "runs:/{run_id}/{artifact_path}".format(
+            run_id=mlflow.active_run().info.run_id,
+            artifact_path=artifact_path)
+
+    model_path = _download_artifact_from_uri(artifact_uri=model_uri)
+    pyfunc_conf = _get_flavor_configuration(model_path=model_path, flavor_name=pyfunc.FLAVOR_NAME)
+    conda_env_path = os.path.join(model_path, pyfunc_conf[pyfunc.ENV])
+    with open(conda_env_path, "r") as f:
+        conda_env = yaml.safe_load(f)
+
+    assert conda_env == mlflow.fastai.get_default_conda_env()
+
+
+@pytest.mark.release
+def test_sagemaker_docker_model_scoring_with_default_conda_env(fastai_model, model_path):
+    mlflow.fastai.save_model(fastai_learner=fastai_model.model, path=model_path, conda_env=None)
+    reloaded_pyfunc = pyfunc.load_model(model_uri=model_path)
+
+    scoring_response = score_model_in_sagemaker_docker_container(
+            model_uri=model_path,
+            data=fastai_model.inference_dataframe,
+            content_type=pyfunc_scoring_server.CONTENT_TYPE_JSON_SPLIT_ORIENTED,
+            flavor=mlflow.pyfunc.FLAVOR_NAME)
+    deployed_model_preds = pd.DataFrame(json.loads(scoring_response.content))
+
+    pandas.testing.assert_frame_equal(
+        deployed_model_preds,
+        pd.DataFrame(reloaded_pyfunc.predict(fastai_model.inference_dataframe)),
+        check_dtype=False,
+        check_less_precise=6)
diff --git a/travis/large-requirements.txt b/travis/large-requirements.txt
index 152db414a2796..944b8a0861a38 100644
--- a/travis/large-requirements.txt
+++ b/travis/large-requirements.txt
@@ -35,6 +35,7 @@ pytest-localserver==0.5.0
 sqlalchemy==1.3.0
 kubernetes==9.0.0
 pystan==2.17.1.0
+fastai==1.0.60
 # TODO (harupy): Unpin fbprophet once https://github.com/facebook/prophet/issues/1339 is fixed.
 fbprophet==0.5
 # test plugin
diff --git a/travis/run-large-python-tests.sh b/travis/run-large-python-tests.sh
index 31287abe4cd4b..14449f586886a 100755
--- a/travis/run-large-python-tests.sh
+++ b/travis/run-large-python-tests.sh
@@ -21,7 +21,7 @@ pytest tests --large --ignore=tests/examples --ignore=tests/h2o --ignore=tests/k
   --ignore=tests/spark --ignore=tests/tensorflow --ignore=tests/azureml --ignore=tests/onnx \
   --ignore=tests/keras_autolog --ignore=tests/tensorflow_autolog --ignore=tests/gluon \
   --ignore=tests/gluon_autolog --ignore=tests/xgboost --ignore=tests/lightgbm \
-  --ignore tests/spark_autologging --ignore=tests/models
+  --ignore tests/spark_autologging --ignore=tests/models --ignore=tests/fastai
 # Run ML framework tests in their own Python processes to avoid OOM issues due to per-framework
 # overhead
 pytest --verbose tests/pytorch --large
@@ -38,6 +38,7 @@ pytest --verbose tests/azureml --large
 pytest --verbose tests/models --large
 pytest --verbose tests/xgboost --large
 pytest --verbose tests/lightgbm --large
+pytest --verbose tests/fastai --large
 # TODO(smurching) Unpin TensorFlow dependency version once test failures with TF 2.1.0 have been
 # fixed
 pip install 'tensorflow==2.0.0'
diff --git a/travis/run-small-python-tests.sh b/travis/run-small-python-tests.sh
index 4ed60ccbf0cc2..e1c5d13374bce 100755
--- a/travis/run-small-python-tests.sh
+++ b/travis/run-small-python-tests.sh
@@ -11,6 +11,6 @@ pytest --cov=mlflow --verbose --ignore=tests/h2o --ignore=tests/keras \
   --ignore=tests/pytorch --ignore=tests/pyfunc --ignore=tests/sagemaker --ignore=tests/sklearn \
   --ignore=tests/spark --ignore=tests/tensorflow --ignore=tests/keras_autolog \
   --ignore=tests/tensorflow_autolog --ignore tests/azureml --ignore tests/onnx \
-  --ignore=tests/xgboost --ignore=tests/lightgbm tests --ignore=tests/spark_autologging
+  --ignore=tests/xgboost --ignore=tests/lightgbm --ignore=tests/spark_autologging --ignore=tests/fastai tests
 
 test $err = 0


Code review comments:
diff_hunk: @@ -0,0 +1,340 @@
+"""
+The ``mlflow.fastai`` module provides an API for logging and loading fast.ai models. This module
+exports fast.ai models with the following flavors:
+
+fastai (native) format
+    This is the main flavor that can be loaded back into fastai.
+:py:mod:`mlflow.pyfunc`
+    Produced for use by generic pyfunc-based deployment tools and batch inference.
+
+.. _fastai.Learner:
+    https://docs.fast.ai/basic_train.html#Learner
+.. _fastai.Learner.export:
+    https://docs.fast.ai/basic_train.html#Learner.export
+"""
+
+from __future__ import absolute_import
+
+import os
+import yaml
+import gorilla
+import tempfile
+import shutil
+
+from mlflow import pyfunc
+from mlflow.models import Model
+import mlflow.tracking
+from mlflow.exceptions import MlflowException
+from mlflow.tracking.artifact_utils import _download_artifact_from_uri
+from mlflow.utils.environment import _mlflow_conda_env
+from mlflow.utils.model_utils import _get_flavor_configuration
+from mlflow.utils.annotations import experimental
+from mlflow.utils.autologging_utils import try_mlflow_log, log_fn_args_as_params
+
+from fastai.tabular import TabularList
+from fastai.basic_data import DatasetType
+
+
+FLAVOR_NAME = "fastai"
+
+
+def get_default_conda_env(include_cloudpickle=False):
+    """
+    :return: The default Conda environment for MLflow Models produced by calls to
+             :func:`save_model()` and :func:`log_model()`.
+    """
+    import fastai
+    pip_deps = None
+    if include_cloudpickle:
+        import cloudpickle
+        pip_deps = ["cloudpickle=={}".format(cloudpickle.__version__)]
+    return _mlflow_conda_env(
+        additional_conda_deps=[
+            "fastai={}".format(fastai.__version__),
+        ],
+        additional_pip_deps=pip_deps,
+        additional_conda_channels=None
+    )
+
+
+def save_model(fastai_learner, path, conda_env=None, mlflow_model=Model(), **kwargs):
+    """
+    Save a fastai Learner to a path on the local file system.
+
+    :param fastai_learner: fastai Learner to be saved.
+    :param path: Local path where the model is to be saved.
+    :param conda_env: Either a dictionary representation of a Conda environment or the path to a
+                      Conda environment yaml file. If provided, this describes the environment
+                      this model should be run in. At minimum, it should specify the
+                      dependencies contained in :func:`get_default_conda_env()`. If
+                      ``None``, the default :func:`get_default_conda_env()` environment is
+                      added to the model. The following is an *example* dictionary
+                      representation of a Conda environment::
+
+                        {
+                            'name': 'mlflow-env',
+                            'channels': ['defaults'],
+                            'dependencies': [
+                                'python=3.7.0',
+                                'fastai=1.0.60',
+                            ]
+                        }
+    :param mlflow_model: MLflow model config this flavor is being added to.
+    :param kwargs: kwargs to pass to ``Learner.save`` method.
+    """
+    import fastai
+    from pathlib import Path
+
+    path = os.path.abspath(path)
+    if os.path.exists(path):
+        raise MlflowException("Path '{}' already exists".format(path))
+    model_data_subpath = "model.fastai"
+    model_data_path = os.path.join(path, model_data_subpath)
+    model_data_path = Path(model_data_path)
+    os.makedirs(path)
+
+    # Save an Learner
+    fastai_learner.export(model_data_path, **kwargs)
+
+    conda_env_subpath = "conda.yaml"
+
+    if conda_env is None:
+        conda_env = get_default_conda_env()
+    elif not isinstance(conda_env, dict):
+        with open(conda_env, "r") as f:
+            conda_env = yaml.safe_load(f)
+    with open(os.path.join(path, conda_env_subpath), "w") as f:
+        yaml.safe_dump(conda_env, stream=f, default_flow_style=False)
+
+    pyfunc.add_to_model(mlflow_model, loader_module="mlflow.fastai",
+                        data=model_data_subpath, env=conda_env_subpath)
+    mlflow_model.add_flavor(FLAVOR_NAME, fastai_version=fastai.__version__, data=model_data_subpath)
+    mlflow_model.save(os.path.join(path, "MLmodel"))
+
+
+def log_model(fastai_learner, artifact_path, conda_env=None, registered_model_name=None, **kwargs):
+    """
+    Log a fastai model as an MLflow artifact for the current run.
+
+    :param fastai_learner: Fastai model (an instance of `fastai.Learner`_) to be saved.
+    :param artifact_path: Run-relative artifact path.
+    :param conda_env: Either a dictionary representation of a Conda environment or the path to a
+                      Conda environment yaml file. If provided, this describes the environment
+                      this model should be run in. At minimum, it should specify the dependencies
+                      contained in :func:`get_default_conda_env()`. If ``None``, the default
+                      :func:`get_default_conda_env()` environment is added to the model.
+                      The following is an *example* dictionary representation of a Conda
+                      environment::
+
+                        {
+                            'name': 'mlflow-env',
+                            'channels': ['defaults'],
+                            'dependencies': [
+                                'python=3.7.0',
+                                'fastai=1.0.60',
+                            ]
+                        }
+    :param registered_model_name: Note:: Experimental: This argument may change or be removed in a
+                                  future release without warning. If given, create a model
+                                  version under ``registered_model_name``, also creating a
+                                  registered model if one with the given name does not exist.
+    :param kwargs: kwargs to pass to `fastai.Learner.export`_ method.
+    """
+    Model.log(artifact_path=artifact_path, flavor=mlflow.fastai,
+              registered_model_name=registered_model_name,
+              fastai_learner=fastai_learner, conda_env=conda_env, **kwargs)
+
+
+def _load_model(path):
+    from fastai.basic_train import load_learner
+    abspath = os.path.abspath(path)
+    path, file = os.path.split(abspath)
+    return load_learner(path, file)
+
+
+class _FastaiModelWrapper:
+    def __init__(self, learner):
+        self.learner = learner
+
+    def predict(self, dataframe):
+        test_data = TabularList.from_df(dataframe, cont_names=self.learner.data.cont_names)
+        self.learner.data.add_test(test_data)
+        return self.learner.get_preds(DatasetType.Test)[0].numpy()
path: mlflow/fastai.py
text: @user1:
Would you be able to clarify why we take the first element of `get_preds` here? In what cases can [get_preds](https://docs.fast.ai/basic_train.html#get_preds) return multiple elements / would users be surprised if we only output the first one as the model's prediction?

@author:
The method get_preds returns a tuple of two tensors, both are the predictions but the second tensor contains classes (0, 1, 2, ..., N) in case of classification, while the first one contains the actual probabilities.

https://docs.fast.ai/basic_train.html#Learner.get_preds

diff_hunk: @@ -0,0 +1,258 @@
+import pytest
+import numpy as np
+from tests.projects.utils import tracking_uri_mock  # pylint: disable=W0611
+
+import pandas as pd
+import sklearn.datasets as datasets
+from fastai.tabular import tabular_learner, TabularList
+from fastai.metrics import accuracy
+import mlflow  # noqa
+import mlflow.fastai  # noqa
+from fastai.callbacks import EarlyStoppingCallback, SaveModelCallback
+
+np.random.seed(1337)
+
+LARGE_EPOCHS = 10
+
+
+@pytest.fixture(params=[True, False])
+def manual_run(request, tracking_uri_mock):
+    if request.param:
+        mlflow.start_run()
+    yield
+    mlflow.end_run()
+
+
+def iris_dataframe():
+    iris = datasets.load_iris()
+    return pd.DataFrame(iris.data[:, :2], columns=iris.feature_names[:2])
+
+
+@pytest.fixture(scope="session")
+def iris_data():
+    iris = datasets.load_iris()
+    X = pd.DataFrame(iris.data[:, :2], columns=iris.feature_names[:2])
+    y = pd.Series(iris.target, name='label')
+    return (TabularList.from_df(pd.concat([X, y], axis=1), cont_names=list(X.columns))
+            .split_by_rand_pct(valid_pct=0.1, seed=42)
+            .label_from_df(cols='label')
+            .databunch())
+
+
+def fastai_model(data, **kwargs):
+    return tabular_learner(data, metrics=accuracy, layers=[5, 3, 2], **kwargs)
+
+
+@pytest.mark.large
+@pytest.mark.parametrize('fit_variant', ['fit', 'fit_one_cycle'])
+def test_fastai_autolog_ends_auto_created_run(iris_data, fit_variant):
+    mlflow.fastai.autolog()
+    model = fastai_model(iris_data)
+    if fit_variant == 'fit_one_cycle':
+        model.fit_one_cycle(1)
+    else:
+        model.fit(1)
+    assert mlflow.active_run() is None
+
+
+@pytest.mark.large
+@pytest.mark.parametrize('fit_variant', ['fit', 'fit_one_cycle'])
+def test_fastai_autolog_persists_manually_created_run(iris_data, fit_variant):
+    mlflow.fastai.autolog()
+
+    with mlflow.start_run() as run:
+        model = fastai_model(iris_data)
+
+        if fit_variant == 'fit_one_cycle':
+            model.fit_one_cycle(LARGE_EPOCHS)
+        else:
+            model.fit(LARGE_EPOCHS)
+
+        assert mlflow.active_run()
+        assert mlflow.active_run().info.run_id == run.info.run_id
+
+
+@pytest.fixture
+def fastai_random_data_run(iris_data, fit_variant, manual_run):
+    mlflow.fastai.autolog()
+
+    model = fastai_model(iris_data)
+
+    if fit_variant == 'fit_one_cycle':
+        model.fit_one_cycle(LARGE_EPOCHS)
+    else:
+        model.fit(LARGE_EPOCHS)
+
+    client = mlflow.tracking.MlflowClient()
+    return model, client.get_run(client.list_run_infos(experiment_id='0')[0].run_id)
+
+
+@pytest.mark.large
+@pytest.mark.parametrize('fit_variant', ['fit', 'fit_one_cycle'])
+def test_fastai_autolog_logs_expected_data(fastai_random_data_run, fit_variant):
+    model, run = fastai_random_data_run
+    data = run.data
+
+    # Testing metrics are logged
+    assert 'train_loss' in data.metrics
+    assert 'valid_loss' in data.metrics
+
+    for o in model.metrics:
+        assert o.__name__ in data.metrics
+
+    client = mlflow.tracking.MlflowClient()
+    metric_history = client.get_metric_history(run.info.run_id, 'valid_loss')
+    assert np.array_equal([m.value for m in metric_history], model.recorder.val_losses)
+
+    # Testing explicitly passed parameters are logged correctly
+    assert 'epochs' in data.params
+    assert data.params['epochs'] == str(LARGE_EPOCHS)
+
+    # Testing implicitly passed parameters are logged correctly
+    assert 'wd' in data.params
+
+    # Testing unwanted parameters are not logged
+    assert 'callbacks' not in data.params
+    assert 'learn' not in data.params
+
+    # Testing optimizer parameters are logged
+    assert 'opt_func' in data.params
+    assert data.params['opt_func'] == 'Adam'
+    assert 'model_summary' in data.tags
+
+    # Testing model_summary.txt is saved
+    client = mlflow.tracking.MlflowClient()
+    artifacts = client.list_artifacts(run.info.run_id)
+    artifacts = map(lambda x: x.path, artifacts)
+    assert 'model_summary.txt' in artifacts
+
+
+@pytest.mark.large
+@pytest.mark.parametrize('fit_variant', ['fit', 'fit_one_cycle'])
+def test_fastai_autolog_logs_default_params(fastai_random_data_run, fit_variant):
+    _, run = fastai_random_data_run
+    if fit_variant == 'fit':
+        assert 'lr' in run.data.params
+        assert run.data.params['lr'] == 'slice(None, 0.003, None)'
+    else:
+        assert 'pct_start' in run.data.params
+        assert run.data.params['pct_start'] == '0.3'
+
+
+@pytest.mark.large
+@pytest.mark.parametrize('fit_variant', ['fit', 'fit_one_cycle'])
+def test_fastai_autolog_model_can_load_from_artifact(fastai_random_data_run):
+    run_id = fastai_random_data_run[1].info.run_id
+    client = mlflow.tracking.MlflowClient()
+    artifacts = client.list_artifacts(run_id)
+    artifacts = map(lambda x: x.path, artifacts)
+    assert 'model' in artifacts
+    model = mlflow.fastai.load_model("runs:/" + run_id + "/model")
+    model_wrapper = mlflow.fastai._FastaiModelWrapper(model)
+    model_wrapper.predict(iris_dataframe())
+
+
+@pytest.fixture
+def fastai_random_data_run_with_callback(iris_data, fit_variant, manual_run, callback, patience):
+    mlflow.fastai.autolog()
+    callbacks = []
+
+    if callback == 'early':
+        # min_delta is set as such to guarantee early stopping
+        callbacks.append(lambda learn: EarlyStoppingCallback(
+            learn, patience=patience, min_delta=99999999))
+
+    model = fastai_model(iris_data, callback_fns=callbacks)
+
+    if fit_variant == 'fit_one_cycle':
+        model.fit_one_cycle(LARGE_EPOCHS)
+    else:
+        model.fit(LARGE_EPOCHS)
+
+    client = mlflow.tracking.MlflowClient()
+    return model, client.get_run(client.list_run_infos(experiment_id='0')[0].run_id)
+
+
+@pytest.mark.large
+@pytest.mark.parametrize('fit_variant', ['fit', 'fit_one_cycle'])
+@pytest.mark.parametrize('callback', ['early'])
+@pytest.mark.parametrize('patience', [0, 1, 5])
+def test_fastai_autolog_early_stop_logs(fastai_random_data_run_with_callback, patience):
+    model, run = fastai_random_data_run_with_callback
+    metrics = run.data.metrics
+    params = run.data.params
+    assert 'early_stop_patience' in params
+    assert params['early_stop_patience'] == str(patience)
+    assert 'early_stop_monitor' in params
+    assert params['early_stop_monitor'] == 'valid_loss'
+    assert 'early_stop_mode' in params
+    assert params['early_stop_mode'] == 'auto'
+    assert 'early_stop_min_delta' in params
+    assert params['early_stop_min_delta'] == f'-{99999999}'
+
+    """
path: tests/fastai/test_fastai_autolog.py
text: @user1:
(for the follow-up PR) Uncomment this?

diff_hunk: @@ -0,0 +1,258 @@
+import pytest
+import numpy as np
+from tests.projects.utils import tracking_uri_mock  # pylint: disable=W0611
+
+import pandas as pd
+import sklearn.datasets as datasets
+from fastai.tabular import tabular_learner, TabularList
+from fastai.metrics import accuracy
+import mlflow  # noqa
+import mlflow.fastai  # noqa
+from fastai.callbacks import EarlyStoppingCallback, SaveModelCallback
+
+np.random.seed(1337)
+
+LARGE_EPOCHS = 10
path: tests/fastai/test_fastai_autolog.py
text: @user1:
For the follow-up PR containing the autologging functionality, might make sense to pick a smaller number here to help ensure the test runs quickly.