Repository: ludwig-ai/ludwig

Issue:
k-fold cross validation for training models with CLI
**Describe the use case**
Training multiple models with shuffled training data (k-folds) can reveals information about our data sets. For example, one fold may be much less accurate than other folds telling us that we might need to increase the total number of samples.

**Describe the solution you'd like**
For the CLI, it would be useful if we could have a parameter to train with k-fold cross validation.

**Describe alternatives you've considered**
k-fold can be achieved with the python API but not as convenient as the CLI.

**Additional context**
I can provide the sample code I've been using for the API.



Discussion:
@danielduckworth Thanks for the suggestion. This is a good idea. 

Feel free to create a pr if you have time to work on it :)

@msaisumanth I'll work on it with my team and submit something in the next few weeks. We have already benefited greatly from Ludwig so happy to contribute.
Thank you so much @danielduckworth . I would suggest you do do a draft minimal not working pr so that we can give you guidance on the design first, so that when you end up implementing it it's more likely there won't be any misalignment.
Hello @w4nderlust , I am working with @danielduckworth on this k-fold contrib. I want to know what do you expect as the structure of the minimal draft over here. Should I right it as a standalone function that will be imported or should it be added as an extra parameter in the full_train() function and add the parameter in cli() function. Sorry if I am asking something obvious but this is my first time contributing :)
@the-learner Thanks a lot for working on this! I think what @w4nderlust was suggesting was to outline the high level flow so that we're all on the same page as to how to implement it. 

You can write the logic in full_train first and then adding the cli stuff should be fairly easy. So please tell us how you would implement this in full_train at a high level and then you can start coding. If you think it's straightforward, go right ahead and we can review the code. 
@msaisumanth The change seems very straight forward but I will tell high level approach anyway.

K-fold k value will be taken input into full_train. There will be different combinations of data (k precisely), based on those data combinations, the training will be done. I might not have to change much of a functionality in the main  training function as well. 
@the-learner there are a bunch of additional considerations: if you have more than one split, the HDF5 and JSON files will be saved for all the splits? Will they be skipped? What about the results? Will you have k different results directories or just one containing all 3 of them? And what about aggregate results (with mean and standard deviation)?
@w4nderlust That was indeed a question on my mind for some time. I think the function of k-fold splits should be to let the user know that the data size needs to be increased if the accuracy values difference will be considerably large. Because saving aggregate results, or most accurate results will just be an accuracy hack. I think it will just store models and final results for one of the splits. Functionality of k-fold can be to inform user about the nature of data. So the results on the test data will be displayed for the k-splits but final model should just be one as it is now.
I've discussed this with @the-learner and I think what makes the most sense is to stay in keeping with the way ludwig already works. So that means storing separate HDF5 and JSON files for each split.

The results of each k-fold would also be saved separately so they can be reported for each fold and the mean calculated and reported.

I think the final step would then be to train using all the data (no split) as the final model. The user can then decide on the basis of the k-fold results whether they want to use the final model.

I'm not sure about the convention for the directories. Do you have any preference @w4nderlust?
This makes sense. A few more considerations:
- in case the k fold is specified I would train just k models and not a final model on all the splits, as that can always be done anyway by calling ludwig again without the split parameter
- when ludwig creates the HDF5 and JSON files, they are placed in the same directory of the original data with the same name, so in order to make the k fold work, you will probably have to either name the files created for each for differently (there's no way to doing it at the moment) or not saving HDF5 and JSON at all
- code-wise, the way I'm thinking about this is a `kfold_cross_validate()` function in `train..py` that calls `full_train()` `k` times. In the main `cli()` function there would be an if that checks if a `kfold` command line parameter is psecified, if not, `full_train` will be called, otherwise `kfold_cross_validate` will be called. But I'm also open to other designs if you think there's a better way.
- after all the k fold models have been trained and the results have been obtained, I would create either an additional JSON file containing the k fold summarized statistics (something like `kfold_training_statistics.json`) so that we could later also build visualizations, like violin plots of the like
- as the different k fold training processes are entirely independent of each other, it would be interesting to find a way to run them in parallel on separate processes. The problem I see here is the fact that, in particular when using GPUs, TensorFlow tries to grab all the resources possible, so running multiple things on the same GPU is kinda tricky, you would have to reduce the percentage of ram used for each process for instance, but if you have as many GPUs as folds, that would be pretty easy to do. Anyway, this could be done also later on, but I wanted to put it here as I believe it would be pretty useful.

Resolving patch:
diff --git a/ludwig/experiment.py b/ludwig/experiment.py
--- a/ludwig/experiment.py
+++ b/ludwig/experiment.py
@@ -22,6 +22,10 @@
 import logging
 import os
 import sys
+import tempfile
+
+import numpy as np
+import pandas as pd
 import yaml
 
 from ludwig.contrib import contrib_command
@@ -31,12 +35,12 @@
 from ludwig.predict import print_test_results
 from ludwig.predict import save_prediction_outputs
 from ludwig.predict import save_test_statistics
-from ludwig.train import full_train
-from ludwig.utils.defaults import default_random_seed
+from ludwig.train import full_train, logger
+from ludwig.utils.data_utils import generate_kfold_splits, save_json
+from ludwig.utils.defaults import default_random_seed, merge_with_defaults
 from ludwig.utils.print_utils import logging_level_registry
 from ludwig.utils.print_utils import print_ludwig
 
-
 logger = logging.getLogger(__name__)
 
 
@@ -282,6 +286,156 @@ def experiment(
     return experiment_dir_name
 
 
+def kfold_cross_validate(
+        k_fold,
+        model_definition=None,
+        model_definition_file=None,
+        data_csv=None,
+        output_directory='results',
+        random_seed=default_random_seed,
+        skip_save_k_fold_split_indices=False,
+        **kwargs
+):
+    """Performs k-fold cross validation.
+
+    # Inputs
+    :param k_fold: (int) number of folds to create for the cross-validation
+    :param model_definition: (dict, default: None) a dictionary containing
+            information needed to build a model. Refer to the [User Guide]
+           (http://ludwig.ai/user_guide/#model-definition) for details.
+    :param model_definition_file: (string, optional, default: `None`) path to
+           a YAML file containing the model definition. If available it will be
+           used instead of the model_definition dict.
+    :param data_csv: (string, default: None)
+    :param output_directory: (string, default: 'results')
+    :param random_seed: (int) Random seed used k-fold splits.
+    :param skip_save_k_fold_split_indices: (boolean, default: False) Disables
+            saving k-fold split indices
+
+    :return: None
+    """
+
+    # check for model_definition and model_definition_file
+    if model_definition is None and model_definition_file is None:
+        raise ValueError(
+            'Either model_definition of model_definition_file have to be'
+            'not None to initialize a LudwigModel'
+        )
+    if model_definition is not None and model_definition_file is not None:
+        raise ValueError(
+            'Only one between model_definition and '
+            'model_definition_file can be provided'
+        )
+
+    # check for k_fold
+    if k_fold is None:
+        raise ValueError(
+            'k_fold parameter must be specified'
+        )
+
+    logger.info('starting {:d}-fold cross validation'.format(k_fold))
+
+    # create output_directory if not available
+    if not os.path.isdir(output_directory):
+        os.mkdir(output_directory)
+
+    # read in data to split for the folds
+    data_df = pd.read_csv(data_csv)
+
+    # place each fold in a separate directory
+    data_dir = os.path.dirname(data_csv)
+    kfold_training_stats = {}
+    kfold_split_indices = {}
+    for train_indices, test_indices, fold_num in \
+            generate_kfold_splits(data_df, k_fold, random_seed):
+        with tempfile.TemporaryDirectory(dir=data_dir) as temp_dir_name:
+            curr_train_df = data_df.iloc[train_indices]
+            curr_test_df = data_df.iloc[test_indices]
+
+            if not skip_save_k_fold_split_indices:
+                kfold_split_indices['fold_' + str(fold_num)] = {
+                    'training_indices': train_indices,
+                    'test_indices': test_indices
+                }
+
+            # train and validate model on this fold
+            if model_definition_file is not None:
+                with open(model_definition_file, 'r') as def_file:
+                    model_definition = \
+                        merge_with_defaults(yaml.safe_load(def_file))
+            logger.info("training on fold {:d}".format(fold_num))
+            (model,
+             preprocessed_data,
+             _,
+             train_stats,
+             model_definition) = full_train(
+                model_definition,
+                data_train_df=curr_train_df,
+                data_test_df=curr_test_df,
+                experiment_name='cross_validation',
+                model_name='fold_' + str(fold_num),
+                output_directory=os.path.join(temp_dir_name, 'results')
+            )
+
+            # score on hold out fold
+            eval_batch_size = model_definition['training']['eval_batch_size']
+            batch_size = model_definition['training']['batch_size']
+            preds = model.predict(
+                preprocessed_data[2],
+                eval_batch_size if eval_batch_size != 0 else batch_size
+            )
+
+            # augment the training statistics with scoring metric fron
+            # the hold out fold
+            train_stats['fold_metric'] = {}
+            for metric_category in preds:
+                train_stats['fold_metric'][metric_category] = {}
+                for metric in preds[metric_category]:
+                    train_stats['fold_metric'][metric_category][metric] = \
+                        preds[metric_category][metric]
+
+            # collect training statistics for this fold
+            kfold_training_stats['fold_' + str(fold_num)] = train_stats
+
+    # consolidate raw fold metrics across all folds
+    raw_kfold_stats = {}
+    for fold_name in kfold_training_stats:
+        for category in kfold_training_stats[fold_name]['fold_metric']:
+            if category not in raw_kfold_stats:
+                raw_kfold_stats[category] = {}
+            category_stats = \
+                kfold_training_stats[fold_name]['fold_metric'][category]
+            for metric in category_stats:
+                if metric not in {'predictions', 'probabilities'}:
+                    if metric not in raw_kfold_stats[category]:
+                        raw_kfold_stats[category][metric] = []
+                    raw_kfold_stats[category][metric] \
+                        .append(category_stats[metric])
+
+    # calculate overall kfold statistics
+    overall_kfold_stats = {}
+    for category in raw_kfold_stats:
+        overall_kfold_stats[category] = {}
+        for metric in raw_kfold_stats[category]:
+            mean = np.mean(raw_kfold_stats[category][metric])
+            std = np.std(raw_kfold_stats[category][metric])
+            overall_kfold_stats[category][metric + '_mean'] = mean
+            overall_kfold_stats[category][metric + '_std'] = std
+
+    kfold_training_stats['overall'] = overall_kfold_stats
+
+    # save k-fold cv statistics
+    save_json(os.path.join(output_directory, 'kfold_training_statistics.json'),
+              kfold_training_stats)
+
+    # save k-fold split indices
+    if not skip_save_k_fold_split_indices:
+        save_json(os.path.join(output_directory, 'kfold_split_indices.json'),
+                  kfold_split_indices)
+
+    logger.info('completed {:d}-fold cross validation'.format(k_fold))
+
+
 def cli(sys_argv):
     parser = argparse.ArgumentParser(
         description='This script trains and tests a model',
@@ -378,6 +532,26 @@ def cli(sys_argv):
         default=False
     )
 
+    # -----------------
+    # K-fold parameters
+    # -----------------
+    parser.add_argument(
+        '-kf',
+        '--k_fold',
+        type=int,
+        default=None,
+        help='number of folds for a k-fold cross validation run '
+    )
+    parser.add_argument(
+        '-skfsi',
+        '--skip_save_k_fold_split_indices',
+        action='store_true',
+        default=False,
+        help='disables saving indices generated to split training data set '
+             'for the k-fold cross validation run, but if it is not needed '
+             'turning it off can slightly increase the overall speed'
+    )
+
     # ----------------
     # Model parameters
     # ----------------
@@ -526,7 +700,10 @@ def cli(sys_argv):
     if is_on_master():
         print_ludwig('Experiment', LUDWIG_VERSION)
 
-    experiment(**vars(args))
+    if args.k_fold is None:
+        experiment(**vars(args))
+    else:
+        kfold_cross_validate(**vars(args))
 
 
 if __name__ == '__main__':
diff --git a/ludwig/train.py b/ludwig/train.py
--- a/ludwig/train.py
+++ b/ludwig/train.py
@@ -21,6 +21,7 @@
 import argparse
 import logging
 import os
+import os.path
 import sys
 from pprint import pformat
 
@@ -96,17 +97,17 @@ def full_train(
     :param model_definition_file: The file that specifies the model definition.
            It is a yaml file.
     :type model_definition_file: filepath (str)
-    :param data_csv: A CSV file contanining the input data which is used to
+    :param data_csv: A CSV file containing the input data which is used to
            train, validate and test a model. The CSV either contains a
            split column or will be split.
     :type data_csv: filepath (str)
-    :param data_train_csv: A CSV file contanining the input data which is used
+    :param data_train_csv: A CSV file containing the input data which is used
            to train a model.
     :type data_train_csv: filepath (str)
-    :param data_validation_csv: A CSV file contanining the input data which is used
+    :param data_validation_csv: A CSV file containing the input data which is used
            to validate a model..
     :type data_validation_csv: filepath (str)
-    :param data_test_csv: A CSV file contanining the input data which is used
+    :param data_test_csv: A CSV file containing the input data which is used
            to test a model.
     :type data_test_csv: filepath (str)
     :param data_hdf5: If the dataset is in the hdf5 format, this is used instead
@@ -146,7 +147,7 @@ def full_train(
     :param skip_save_model: Disables saving model weights
            and hyperparameters each time the model
            improves. By default Ludwig saves model weights after each epoch
-           the validation measure imrpvoes, but if the model is really big
+           the validation measure improves, but if the model is really big
            that can be time consuming if you do not want to keep
            the weights and just find out what performance can a model get
            with a set of hyperparameters, use this parameter to skip it,
@@ -169,8 +170,8 @@ def full_train(
            is not needed turning it off can slightly increase the
            overall speed..
     :type skip_save_progress: Boolean
-    :param output_directory: The directory that will contanin the training
-           statistics, the saved model and the training procgress files.
+    :param output_directory: The directory that will contain the training
+           statistics, the saved model and the training progress files.
     :type output_directory: filepath (str)
     :param gpus: List of GPUs that are available for training.
     :type gpus: List
diff --git a/ludwig/utils/data_utils.py b/ludwig/utils/data_utils.py
--- a/ludwig/utils/data_utils.py
+++ b/ludwig/utils/data_utils.py
@@ -26,6 +26,7 @@
 import numpy as np
 import pandas as pd
 from pandas.errors import ParserError
+from sklearn.model_selection import KFold
 
 logger = logging.getLogger(__name__)
 
@@ -52,16 +53,17 @@ def read_csv(data_fp, header=0):
     :param header: header argument for pandas to read the csv
     :return: Pandas dataframe with the data
     """
-    
-    separator=','
+
+    separator = ','
     with open(data_fp, 'r') as csvfile:
         try:
-            dialect = csv.Sniffer().sniff(csvfile.read(1024*100), delimiters=[',' , '\t', '|', ' '])
-            separator=dialect.delimiter
+            dialect = csv.Sniffer().sniff(csvfile.read(1024 * 100),
+                                          delimiters=[',', '\t', '|', ' '])
+            separator = dialect.delimiter
         except csv.Error:
             # Could not conclude the delimiter, defaulting to comma
             pass
-    
+
     try:
         df = pd.read_csv(data_fp, sep=separator, header=header)
     except ParserError:
@@ -395,6 +397,7 @@ def add_sequence_feature_column(df, col_name, seq_length):
     df[new_col_name] = new_data
     df[new_col_name] = df[new_col_name].fillna(method='backfill')
 
+
 def override_in_memory_flag(input_features, override_value):
     num_overrides = 0
     for feature in input_features:
@@ -404,6 +407,7 @@ def override_in_memory_flag(input_features, override_value):
                 num_overrides += 1
     return num_overrides
 
+
 class NumpyEncoder(json.JSONEncoder):
     def default(self, obj):
         if isinstance(obj, set):
@@ -418,3 +422,11 @@ def default(self, obj):
             return obj.tolist()
         else:
             return json.JSONEncoder.default(self, obj)
+
+
+def generate_kfold_splits(data_df, num_folds, random_state):
+    kf = KFold(n_splits=num_folds, shuffle=True, random_state=random_state)
+    fold_num = 0
+    for train_indices, test_indices in kf.split(data_df):
+        fold_num += 1
+        yield train_indices, test_indices, fold_num


Test patch:
diff --git a/tests/integration_tests/test_kfold_cv.py b/tests/integration_tests/test_kfold_cv.py
new file mode 100644
--- /dev/null
+++ b/tests/integration_tests/test_kfold_cv.py
@@ -0,0 +1,80 @@
+import logging
+import os
+import os.path
+import tempfile
+
+import yaml
+
+from ludwig.experiment import kfold_cross_validate
+from ludwig.utils.data_utils import load_json
+from tests.integration_tests.utils import category_feature
+from tests.integration_tests.utils import generate_data
+from tests.integration_tests.utils import numerical_feature
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+logging.getLogger("ludwig").setLevel(logging.INFO)
+
+
+def test_kfold_cv():
+    num_folds = 3
+
+    # setup temporary directory to run test
+    with tempfile.TemporaryDirectory() as tmpdir:
+
+        training_data_fp = os.path.join(tmpdir, 'train.csv')
+        model_definition_fp = os.path.join(tmpdir, 'model_definition.yaml')
+        results_dir = os.path.join(tmpdir, 'results')
+        statistics_fp = os.path.join(results_dir,
+                                     'kfold_training_statistics.json')
+        indices_fp = os.path.join(results_dir, 'kfold_split_indices.json')
+
+        # generate synthetic data for the test
+        input_features = [
+            numerical_feature(normalization='zscore'),
+            numerical_feature(normalization='zscore')
+        ]
+
+        output_features = [
+            category_feature(vocab_size=2, reduce_input='sum')
+        ]
+
+        generate_data(input_features, output_features, training_data_fp)
+
+        # generate model definition file
+        model_definition = {
+            'input_features': input_features,
+            'output_features': output_features,
+            'combiner': {'type': 'concat', 'fc_size': 14},
+            'training': {'epochs': 2}
+        }
+
+        with open(model_definition_fp, 'w') as f:
+            yaml.dump(model_definition, f)
+
+        # run k-fold cv
+        kfold_cross_validate(
+            k_fold=num_folds,
+            model_definition_file=model_definition_fp,
+            data_csv=training_data_fp,
+            output_directory=results_dir,
+            logging_level='warn'
+        )
+
+        # check for expected results
+        # check for existence and structure of statistics file
+        assert os.path.isfile(statistics_fp)
+
+        # check for required keys
+        cv_statistics = load_json(statistics_fp)
+        for key in ['fold_' + str(i + 1)
+                    for i in range(num_folds)] + ['overall']:
+            assert key in cv_statistics
+
+        # check for existence and structure of split indices file
+        assert os.path.isfile(indices_fp)
+
+        # check for required keys
+        cv_indices = load_json(indices_fp)
+        for key in ['fold_' + str(i + 1) for i in range(num_folds)]:
+            assert key in cv_indices