Repository: ludwig-ai/ludwig
Pull request: Add k fold cv cli feature
# Code Pull Requests

Resolves #462.  

#### Summary of changes:
```
Added function kfold_cross_validate() function in train..py that calls full_train() k times. 

Added function generate_kfold_splits() function to utils/data_utils.py

Create `kfold_training_statistics.json` statistics file to hold results from the k-fold cv run

```
A new parameter `--k_fold` is now available for the `ludwig train` command.  This is the help output for the new parameter:
```
 -kf K_FOLD, --k_fold K_FOLD
                        number of folds for a k-fold cross validation run
``` 

#### Work to be completed:
* Add `'aggregate': {...}` node to the `fold_training_statistics.json` file
* Parallel processing for k-fold cv.  Note: I have a working version of parallel processing in a separate branch.  I figure it is easier to debug issues with the serial version.  Once we are comfortable with how k-fold cv processing occurs, I'll add the parallel processing functionality.  Parallel processing is implemented using the `joblib` package.
* add unit test for k-fold cv function
* update documentation



Problem statement:
k-fold cross validation for training models with CLI
**Describe the use case**
Training multiple models with shuffled training data (k-folds) can reveals information about our data sets. For example, one fold may be much less accurate than other folds telling us that we might need to increase the total number of samples.

**Describe the solution you'd like**
For the CLI, it would be useful if we could have a parameter to train with k-fold cross validation.

**Describe alternatives you've considered**
k-fold can be achieved with the python API but not as convenient as the CLI.

**Additional context**
I can provide the sample code I've been using for the API.

Hints from issue discussion:
@danielduckworth Thanks for the suggestion. This is a good idea. 

Feel free to create a pr if you have time to work on it :)

@msaisumanth I'll work on it with my team and submit something in the next few weeks. We have already benefited greatly from Ludwig so happy to contribute.
Thank you so much @danielduckworth . I would suggest you do do a draft minimal not working pr so that we can give you guidance on the design first, so that when you end up implementing it it's more likely there won't be any misalignment.
Hello @w4nderlust , I am working with @danielduckworth on this k-fold contrib. I want to know what do you expect as the structure of the minimal draft over here. Should I right it as a standalone function that will be imported or should it be added as an extra parameter in the full_train() function and add the parameter in cli() function. Sorry if I am asking something obvious but this is my first time contributing :)
@the-learner Thanks a lot for working on this! I think what @w4nderlust was suggesting was to outline the high level flow so that we're all on the same page as to how to implement it. 

You can write the logic in full_train first and then adding the cli stuff should be fairly easy. So please tell us how you would implement this in full_train at a high level and then you can start coding. If you think it's straightforward, go right ahead and we can review the code. 
@msaisumanth The change seems very straight forward but I will tell high level approach anyway.

K-fold k value will be taken input into full_train. There will be different combinations of data (k precisely), based on those data combinations, the training will be done. I might not have to change much of a functionality in the main  training function as well. 
@the-learner there are a bunch of additional considerations: if you have more than one split, the HDF5 and JSON files will be saved for all the splits? Will they be skipped? What about the results? Will you have k different results directories or just one containing all 3 of them? And what about aggregate results (with mean and standard deviation)?
@w4nderlust That was indeed a question on my mind for some time. I think the function of k-fold splits should be to let the user know that the data size needs to be increased if the accuracy values difference will be considerably large. Because saving aggregate results, or most accurate results will just be an accuracy hack. I think it will just store models and final results for one of the splits. Functionality of k-fold can be to inform user about the nature of data. So the results on the test data will be displayed for the k-splits but final model should just be one as it is now.
I've discussed this with @the-learner and I think what makes the most sense is to stay in keeping with the way ludwig already works. So that means storing separate HDF5 and JSON files for each split.

The results of each k-fold would also be saved separately so they can be reported for each fold and the mean calculated and reported.

I think the final step would then be to train using all the data (no split) as the final model. The user can then decide on the basis of the k-fold results whether they want to use the final model.

I'm not sure about the convention for the directories. Do you have any preference @w4nderlust?
This makes sense. A few more considerations:
- in case the k fold is specified I would train just k models and not a final model on all the splits, as that can always be done anyway by calling ludwig again without the split parameter
- when ludwig creates the HDF5 and JSON files, they are placed in the same directory of the original data with the same name, so in order to make the k fold work, you will probably have to either name the files created for each for differently (there's no way to doing it at the moment) or not saving HDF5 and JSON at all
- code-wise, the way I'm thinking about this is a `kfold_cross_validate()` function in `train..py` that calls `full_train()` `k` times. In the main `cli()` function there would be an if that checks if a `kfold` command line parameter is psecified, if not, `full_train` will be called, otherwise `kfold_cross_validate` will be called. But I'm also open to other designs if you think there's a better way.
- after all the k fold models have been trained and the results have been obtained, I would create either an additional JSON file containing the k fold summarized statistics (something like `kfold_training_statistics.json`) so that we could later also build visualizations, like violin plots of the like
- as the different k fold training processes are entirely independent of each other, it would be interesting to find a way to run them in parallel on separate processes. The problem I see here is the fact that, in particular when using GPUs, TensorFlow tries to grab all the resources possible, so running multiple things on the same GPU is kinda tricky, you would have to reduce the percentage of ram used for each process for instance, but if you have as many GPUs as folds, that would be pretty easy to do. Anyway, this could be done also later on, but I wanted to put it here as I believe it would be pretty useful.

Commit message: Merge branch 'master' into add_k-fold_cv_cli_feature

Patch to review:
diff --git a/ludwig/train.py b/ludwig/train.py
index cd090ac1edb..51b2e228f49 100644
--- a/ludwig/train.py
+++ b/ludwig/train.py
@@ -21,10 +21,13 @@
 import argparse
 import logging
 import os
+import os.path
 import sys
 from pprint import pformat
 
 import yaml
+import tempfile
+import pandas as pd
 
 from ludwig.contrib import contrib_command
 from ludwig.data.preprocessing import preprocess_for_training
@@ -36,6 +39,7 @@
 from ludwig.models.model import load_model_and_definition
 from ludwig.models.modules.measure_modules import get_best_function
 from ludwig.utils.data_utils import save_json
+from ludwig.utils.data_utils import generate_kfold_splits
 from ludwig.utils.defaults import default_random_seed
 from ludwig.utils.defaults import merge_with_defaults
 from ludwig.utils.misc import get_experiment_description, \
@@ -96,17 +100,17 @@ def full_train(
     :param model_definition_file: The file that specifies the model definition.
            It is a yaml file.
     :type model_definition_file: filepath (str)
-    :param data_csv: A CSV file contanining the input data which is used to
+    :param data_csv: A CSV file containing the input data which is used to
            train, validate and test a model. The CSV either contains a
            split column or will be split.
     :type data_csv: filepath (str)
-    :param data_train_csv: A CSV file contanining the input data which is used
+    :param data_train_csv: A CSV file containing the input data which is used
            to train a model.
     :type data_train_csv: filepath (str)
-    :param data_validation_csv: A CSV file contanining the input data which is used
+    :param data_validation_csv: A CSV file containing the input data which is used
            to validate a model..
     :type data_validation_csv: filepath (str)
-    :param data_test_csv: A CSV file contanining the input data which is used
+    :param data_test_csv: A CSV file containing the input data which is used
            to test a model.
     :type data_test_csv: filepath (str)
     :param data_hdf5: If the dataset is in the hdf5 format, this is used instead
@@ -146,7 +150,7 @@ def full_train(
     :param skip_save_model: Disables saving model weights
            and hyperparameters each time the model
            improves. By default Ludwig saves model weights after each epoch
-           the validation measure imrpvoes, but if the model is really big
+           the validation measure improves, but if the model is really big
            that can be time consuming if you do not want to keep
            the weights and just find out what performance can a model get
            with a set of hyperparameters, use this parameter to skip it,
@@ -169,8 +173,8 @@ def full_train(
            is not needed turning it off can slightly increase the
            overall speed..
     :type skip_save_progress: Boolean
-    :param output_directory: The directory that will contanin the training
-           statistics, the saved model and the training procgress files.
+    :param output_directory: The directory that will contain the training
+           statistics, the saved model and the training progress files.
     :type output_directory: filepath (str)
     :param gpus: List of GPUs that are available for training.
     :type gpus: List
@@ -396,6 +400,68 @@ def full_train(
     )
 
 
+def kfold_cross_validate(
+                model_definition,
+                model_definition_file=None,
+                data_csv=None,
+                data_train_csv=None,
+                data_validation_csv=None,
+                output_directory='results',
+                k_fold=None,
+                **kwargs
+):
+    logger = logging.getLogger('ludwig')
+    logger.info('starting {:d}-fold cross validation'.format(k_fold))
+
+    # create output_directory if not available
+    if not os.path.isdir(output_directory):
+        os.mkdir(output_directory)
+
+    # read in data to split for the folds
+    data_df = pd.read_csv(data_train_csv)
+
+    # place each fold in a separate directory
+    data_dir = os.path.dirname(data_train_csv)
+    kfold_training_stats = {}
+    for train_index, test_index, fold_num in generate_kfold_splits(data_train_csv, k_fold):
+        with tempfile.TemporaryDirectory(dir=data_dir) as temp_dir_name:
+            # save training and validation subset for the fold into a temporary directory
+            train_csv_fp = os.path.join(temp_dir_name, 'train_fold.csv')
+            test_csv_fp = os.path.join(temp_dir_name, 'test_fold.csv')
+            logger.info("\n\n>>>>> for fold {:d} created temporary directory: {}".format(fold_num, temp_dir_name))
+            data_df.iloc[train_index].to_csv(train_csv_fp, index=False)
+            data_df.iloc[test_index].to_csv(test_csv_fp, index=False)
+
+            # train and validate model on this fold
+            logger.info("training on fold {:d}".format(fold_num))
+            (model,
+             preprocessed_data,
+             _,
+             train_stats,
+             model_definition) = full_train({},
+                                            model_definition_file=model_definition_file,
+                                            data_train_csv = train_csv_fp,
+                                            data_test_csv = test_csv_fp,
+                                            experiment_name='cross_validation',
+                                            model_name='fold_' + str(fold_num),
+                                            output_directory=os.path.join(temp_dir_name,'results'))
+
+            # score on hold out fold
+            preds = model.predict(preprocessed_data[2],
+                                  model_definition['training']['batch_size'])
+
+            # augment the training statistics with scoring metric fron the hold out fold
+            train_stats['fold_metric'] = preds['combined']
+
+            # collect training statistics for this fold
+            kfold_training_stats['fold_'+str(fold_num)] = train_stats
+
+    # save consolidated training statistics from k-fold cv runs
+    save_json(os.path.join(output_directory,'kfold_training_statistics.json'), kfold_training_stats)
+
+    logger.info('completed {:d}-fold cross validation'.format(k_fold))
+
+
 def train(
         training_set,
         validation_set,
@@ -736,6 +802,13 @@ def cli(sys_argv):
              'to a random number generator: data splitting, parameter '
              'initialization and training set shuffling'
     )
+    parser.add_argument(
+        '-kf',
+        '--k_fold',
+        type=int,
+        default=None,
+        help='number of folds for a k-fold cross validation run '
+    )
     parser.add_argument(
         '-g',
         '--gpus',
@@ -782,7 +855,10 @@ def cli(sys_argv):
     if is_on_master():
         print_ludwig('Train', LUDWIG_VERSION)
 
-    full_train(**vars(args))
+    if args.k_fold is None:
+        full_train(**vars(args))
+    else:
+        kfold_cross_validate(**vars(args))
 
 
 if __name__ == '__main__':
diff --git a/ludwig/utils/data_utils.py b/ludwig/utils/data_utils.py
index 760dd519881..2227f54336c 100644
--- a/ludwig/utils/data_utils.py
+++ b/ludwig/utils/data_utils.py
@@ -26,6 +26,7 @@
 import numpy as np
 import pandas as pd
 from pandas.errors import ParserError
+from sklearn.model_selection import KFold
 
 logger = logging.getLogger(__name__)
 
@@ -418,3 +419,11 @@ def default(self, obj):
             return obj.tolist()
         else:
             return json.JSONEncoder.default(self, obj)
+
+def generate_kfold_splits(data_train_df_fp, k_fold):
+    kf = KFold(n_splits=k_fold, shuffle=True)
+    data_train_df = pd.read_csv(data_train_df_fp)
+    fold_num = 0
+    for train_index, test_index in kf.split(data_train_df):
+        fold_num += 1
+        yield train_index, test_index, fold_num


Code review comments:
diff_hunk: @@ -396,6 +400,68 @@ def full_train(
     )
 
 
+def kfold_cross_validate(
+                model_definition,
+                model_definition_file=None,
+                data_csv=None,
+                data_train_csv=None,
+                data_validation_csv=None,
+                output_directory='results',
+                k_fold=None,
+                **kwargs
+):
+    logger = logging.getLogger('ludwig')
+    logger.info('starting {:d}-fold cross validation'.format(k_fold))
+
+    # create output_directory if not available
+    if not os.path.isdir(output_directory):
+        os.mkdir(output_directory)
+
+    # read in data to split for the folds
+    data_df = pd.read_csv(data_train_csv)
+
+    # place each fold in a separate directory
+    data_dir = os.path.dirname(data_train_csv)
+    kfold_training_stats = {}
+    for train_index, test_index, fold_num in generate_kfold_splits(data_train_csv, k_fold):
+        with tempfile.TemporaryDirectory(dir=data_dir) as temp_dir_name:
+            # save training and validation subset for the fold into a temporary directory
+            train_csv_fp = os.path.join(temp_dir_name, 'train_fold.csv')
+            test_csv_fp = os.path.join(temp_dir_name, 'test_fold.csv')
+            logger.info("\n\n>>>>> for fold {:d} created temporary directory: {}".format(fold_num, temp_dir_name))
+            data_df.iloc[train_index].to_csv(train_csv_fp, index=False)
+            data_df.iloc[test_index].to_csv(test_csv_fp, index=False)
+
+            # train and validate model on this fold
+            logger.info("training on fold {:d}".format(fold_num))
+            (model,
+             preprocessed_data,
+             _,
+             train_stats,
+             model_definition) = full_train({},
+                                            model_definition_file=model_definition_file,
+                                            data_train_csv = train_csv_fp,
+                                            data_test_csv = test_csv_fp,
+                                            experiment_name='cross_validation',
+                                            model_name='fold_' + str(fold_num),
+                                            output_directory=os.path.join(temp_dir_name,'results'))
+
+            # score on hold out fold
+            preds = model.predict(preprocessed_data[2],
+                                  model_definition['training']['batch_size'])
path: ludwig/train.py
text: @user1:
this should be `eval_batch_size`. If `==0` then `batch_size`.

@author:
Question related to this statement.  On line 450, I've coded this construct `preprocessed_data[2]`.   Is this the best way to refer to the particular data structure?  I'm usually concerned by hard coding a specific numeric value.

@author:
implemented `eval_batch_size if eval_batch_size != 0 else batch_size`

diff_hunk: @@ -396,6 +400,68 @@ def full_train(
     )
 
 
+def kfold_cross_validate(
+                model_definition,
+                model_definition_file=None,
+                data_csv=None,
+                data_train_csv=None,
+                data_validation_csv=None,
+                output_directory='results',
+                k_fold=None,
+                **kwargs
+):
+    logger = logging.getLogger('ludwig')
+    logger.info('starting {:d}-fold cross validation'.format(k_fold))
+
+    # create output_directory if not available
+    if not os.path.isdir(output_directory):
+        os.mkdir(output_directory)
+
+    # read in data to split for the folds
+    data_df = pd.read_csv(data_train_csv)
+
+    # place each fold in a separate directory
+    data_dir = os.path.dirname(data_train_csv)
+    kfold_training_stats = {}
+    for train_index, test_index, fold_num in generate_kfold_splits(data_train_csv, k_fold):
+        with tempfile.TemporaryDirectory(dir=data_dir) as temp_dir_name:
+            # save training and validation subset for the fold into a temporary directory
+            train_csv_fp = os.path.join(temp_dir_name, 'train_fold.csv')
+            test_csv_fp = os.path.join(temp_dir_name, 'test_fold.csv')
+            logger.info("\n\n>>>>> for fold {:d} created temporary directory: {}".format(fold_num, temp_dir_name))
+            data_df.iloc[train_index].to_csv(train_csv_fp, index=False)
+            data_df.iloc[test_index].to_csv(test_csv_fp, index=False)
+
+            # train and validate model on this fold
+            logger.info("training on fold {:d}".format(fold_num))
+            (model,
+             preprocessed_data,
+             _,
+             train_stats,
+             model_definition) = full_train({},
+                                            model_definition_file=model_definition_file,
+                                            data_train_csv = train_csv_fp,
+                                            data_test_csv = test_csv_fp,
+                                            experiment_name='cross_validation',
+                                            model_name='fold_' + str(fold_num),
+                                            output_directory=os.path.join(temp_dir_name,'results'))
+
+            # score on hold out fold
+            preds = model.predict(preprocessed_data[2],
+                                  model_definition['training']['batch_size'])
+
+            # augment the training statistics with scoring metric fron the hold out fold
+            train_stats['fold_metric'] = preds['combined']
+
+            # collect training statistics for this fold
+            kfold_training_stats['fold_'+str(fold_num)] = train_stats
+
path: ludwig/train.py
text: @user1:
before saving i would add an additional `'overall'` key to the kfold stats that for each output feature takes the overall and adds mean and std.
Something like:
```
overall_kfold_stats = {}
for fold_stats in kfold_training_stats:
    for of in fold_stats:
        if OVERALL in of:
            for metric in of[OVERALL]:
                if metric not in overall_kfold_stats:
                    overall_kfold_stats[metric] = []
                overall_kfold_stats[metric].append(of[OVERALL][metric])
for metric in overall_kfold_stats:
    mean = np.mean(overall_kfold_stats[metric])
    std = np.std(overall_kfold_stats[metric])
    del overall_kfold_stats[metric]
    overall_kfold_stats[metric + "_mean"] = mean
    overall_kfold_stats[metric + "_std"] = std
kfold_training_stats[OVERALL] = overall_kfold_stats
```

@author:
Added `'overall'` key.  This contains the mean and standard deviation for the various metrics (loss, accuracy) from the hold out fold.  Here is an example:
```
   "overall": {
        "Survived": {
            "accuracy_mean": 0.39955106621773284,
            "accuracy_std": 0.015632310075403076,
            "loss_mean": 6.96722408684282,
            "loss_std": 0.5409600467679053
        },
        "combined": {
            "accuracy_mean": 0.39955106621773284,
            "accuracy_std": 0.015632310075403076,
            "loss_mean": 6.96722408684282,
            "loss_std": 0.5409600467679053
        }
    }

```
The structure of the `fold_training_statistics.json` file is now
```
{
  "fold_1": {...}
 "fold_2": {...}
 "fold_3": {...}
 "overall": {...}
}
```

@user1:
looks good

diff_hunk: @@ -418,3 +419,11 @@ def default(self, obj):
             return obj.tolist()
         else:
             return json.JSONEncoder.default(self, obj)
+
+def generate_kfold_splits(data_train_df_fp, k_fold):
path: ludwig/utils/data_utils.py
text: @user1:
this should probably get the DataFrame directly as input

@author:
made modification to directly use DataFram as input to the `generate_kfold_splits()`

diff_hunk: @@ -396,6 +400,68 @@ def full_train(
     )
 
 
+def kfold_cross_validate(
+                model_definition,
+                model_definition_file=None,
+                data_csv=None,
+                data_train_csv=None,
+                data_validation_csv=None,
+                output_directory='results',
+                k_fold=None,
+                **kwargs
+):
+    logger = logging.getLogger('ludwig')
+    logger.info('starting {:d}-fold cross validation'.format(k_fold))
+
+    # create output_directory if not available
+    if not os.path.isdir(output_directory):
+        os.mkdir(output_directory)
+
+    # read in data to split for the folds
+    data_df = pd.read_csv(data_train_csv)
+
+    # place each fold in a separate directory
+    data_dir = os.path.dirname(data_train_csv)
+    kfold_training_stats = {}
+    for train_index, test_index, fold_num in generate_kfold_splits(data_train_csv, k_fold):
+        with tempfile.TemporaryDirectory(dir=data_dir) as temp_dir_name:
+            # save training and validation subset for the fold into a temporary directory
+            train_csv_fp = os.path.join(temp_dir_name, 'train_fold.csv')
+            test_csv_fp = os.path.join(temp_dir_name, 'test_fold.csv')
+            logger.info("\n\n>>>>> for fold {:d} created temporary directory: {}".format(fold_num, temp_dir_name))
+            data_df.iloc[train_index].to_csv(train_csv_fp, index=False)
+            data_df.iloc[test_index].to_csv(test_csv_fp, index=False)
+
+            # train and validate model on this fold
+            logger.info("training on fold {:d}".format(fold_num))
+            (model,
+             preprocessed_data,
+             _,
+             train_stats,
+             model_definition) = full_train({},
+                                            model_definition_file=model_definition_file,
+                                            data_train_csv = train_csv_fp,
+                                            data_test_csv = test_csv_fp,
+                                            experiment_name='cross_validation',
+                                            model_name='fold_' + str(fold_num),
+                                            output_directory=os.path.join(temp_dir_name,'results'))
+
+            # score on hold out fold
+            preds = model.predict(preprocessed_data[2],
+                                  model_definition['training']['batch_size'])
+
+            # augment the training statistics with scoring metric fron the hold out fold
+            train_stats['fold_metric'] = preds['combined']
path: ludwig/train.py
text: @user1:
why only combined and not all output features?

@author:
Picking `combined` was MVP to illustrate functionality of saving k-fold statistics to disk.  I can update to collect all the metric sets to save.

@user1:
Yeah all of them would be better

diff_hunk: @@ -396,6 +400,68 @@ def full_train(
     )
 
 
+def kfold_cross_validate(
+                model_definition,
+                model_definition_file=None,
+                data_csv=None,
+                data_train_csv=None,
+                data_validation_csv=None,
+                output_directory='results',
+                k_fold=None,
+                **kwargs
+):
+    logger = logging.getLogger('ludwig')
+    logger.info('starting {:d}-fold cross validation'.format(k_fold))
+
+    # create output_directory if not available
+    if not os.path.isdir(output_directory):
+        os.mkdir(output_directory)
+
+    # read in data to split for the folds
+    data_df = pd.read_csv(data_train_csv)
+
+    # place each fold in a separate directory
+    data_dir = os.path.dirname(data_train_csv)
+    kfold_training_stats = {}
+    for train_index, test_index, fold_num in generate_kfold_splits(data_train_csv, k_fold):
+        with tempfile.TemporaryDirectory(dir=data_dir) as temp_dir_name:
+            # save training and validation subset for the fold into a temporary directory
+            train_csv_fp = os.path.join(temp_dir_name, 'train_fold.csv')
+            test_csv_fp = os.path.join(temp_dir_name, 'test_fold.csv')
+            logger.info("\n\n>>>>> for fold {:d} created temporary directory: {}".format(fold_num, temp_dir_name))
+            data_df.iloc[train_index].to_csv(train_csv_fp, index=False)
+            data_df.iloc[test_index].to_csv(test_csv_fp, index=False)
+
+            # train and validate model on this fold
+            logger.info("training on fold {:d}".format(fold_num))
+            (model,
+             preprocessed_data,
+             _,
+             train_stats,
+             model_definition) = full_train({},
path: ludwig/train.py
text: @user1:
the {} should not be needed

@author:
removed `{}`

@author:
Looks like I was a bit pre-mature in calling this resolved.

There may be an issue in removing `{}` in line 441.  The function definition for `full_train()` looks like this:
```
def full_train(
        model_definition,
        model_definition_file=None,
        data_df=None,
        data_train_df=None,
```

When I remove `{}` from the call to `full_train()` an error is raised.

It appears to remove `{}` from line 441, we have to modify `full_train()`.  

Do you want me to take a stab at modifying `full_train()` as well?

@author:
Ignore my earlier comment.  I just realized what you were asking me to do. This issues was resolved by this coding sequence:
```
           # train and validate model on this fold
            with open(model_definition_file, 'r') as def_file:
                model_definition = merge_with_defaults(yaml.safe_load(def_file))
            logger.info("training on fold {:d}".format(fold_num))
            (model,
             preprocessed_data,
             _,
             train_stats,
             model_definition) = full_train(
                model_definition,
                data_train_csv=train_csv_fp,
                data_test_csv=test_csv_fp,
                experiment_name='cross_validation',
```

@user1:
You are right, I added that check from API also in full_train: https://github.com/uber/ludwig/commit/7826a1b4f594cf760d5403929fa1ab41302b53de
now you can pass only the file path and it will work.

diff_hunk: @@ -396,6 +400,68 @@ def full_train(
     )
 
 
+def kfold_cross_validate(
+                model_definition,
+                model_definition_file=None,
+                data_csv=None,
+                data_train_csv=None,
+                data_validation_csv=None,
+                output_directory='results',
+                k_fold=None,
+                **kwargs
+):
+    logger = logging.getLogger('ludwig')
+    logger.info('starting {:d}-fold cross validation'.format(k_fold))
+
+    # create output_directory if not available
+    if not os.path.isdir(output_directory):
+        os.mkdir(output_directory)
+
+    # read in data to split for the folds
+    data_df = pd.read_csv(data_train_csv)
+
+    # place each fold in a separate directory
+    data_dir = os.path.dirname(data_train_csv)
+    kfold_training_stats = {}
+    for train_index, test_index, fold_num in generate_kfold_splits(data_train_csv, k_fold):
path: ludwig/train.py
text: @user1:
maybe an option to save the train and test indices for each fold would be useful for reproducibility.

@author:
Any particular preference on data format for storage?  json?  pickle?  

@author:
Decided to use json format to make it consistent with other saved information. Saved split indices saved to a data set called `kfold_split_indices.json` in the `output_directory`.  Format of data set looks like this
```
{
   "fold_1": {"test_index": [1, 3, 10,  ...], "training_index": [2, 4, 5,...]  },
   "fold_2": {"test_index": [...], "training_index": [...]},
   "fold_3": {"test_index": [...], "training_index": [...]}
}
```

added command line parameter to skip saving kfold indices. 
```
    parser.add_argument(
        '-skfsi',
        '--skip_save_k_fold_split_indices',
        action='store_true',
        default=False,
        help='disables saving indices generated to split training data set ' 
             'for the k-fold cross validation run, but if it is not needed '
             'turning it off can slightly increase the overall speed'
    )
```

diff_hunk: @@ -396,6 +400,68 @@ def full_train(
     )
 
 
+def kfold_cross_validate(
+                model_definition,
+                model_definition_file=None,
+                data_csv=None,
+                data_train_csv=None,
+                data_validation_csv=None,
+                output_directory='results',
+                k_fold=None,
path: ludwig/train.py
text: @user1:
I would make this parameter mandatory

@author:
Couple of questions:

1) I just want to confirm that when you write, "this parameter" you mean the "k_fold" parameter.  Is this correct?

2) Re: the function signature.  Couple of major issues called out by deepsource code quality check identified unused parameters `data_csv` and `data_validation_csv`.  To resolve those issues, I removed them from the function definition.  Right now I'm assuming that it does not make sense to duplicate all the parameters from `full_train()` into `kfold_cross_validate()`

@author:
This is the new def function:
```
def kfold_cross_validate(
                k_fold,
                model_definition_file=None,
                data_train_csv=None,
                output_directory='results',
                **kwargs
):
```

A follow-up question, should `model_definition_file` and `data_train_csv` be mandatory as well?

@user1:
In the current API you have model_definition and model_definition_file both None and then there's some logic to chack tha at least one of them is not none: https://github.com/uber/ludwig/blob/master/ludwig/api.py#L141-L156
You could do the same.

Regarding data_train_csv, I would actually call it data_csv. The rationale being that that data is going to be split in folds and used for both training and test, so I'd rather be more generic.

@author:
Made change to function signature and added check for parameters.  This new function:
```
def kfold_cross_validate(
                model_definition=None,
                model_definition_file=None,
                k_fold=None,
                data_csv=None,
                output_directory='results',
                random_seed=default_random_seed,
                skip_save_k_fold_split_indices=False,
                **kwargs
):

    # check for model_definition and model_definition_file
    if model_definition is None and model_definition_file is None:
        raise ValueError(
            'Either model_definition of model_definition_file have to be'
            'not None to initialize a LudwigModel'
        )
    if model_definition is not None and model_definition_file is not None:
        raise ValueError(
            'Only one between model_definition and '
            'model_definition_file can be provided'
        )
    
    # check for k_fold
    if k_fold is None:
        raise ValueError(
            'k_fold parameter must be specified'
        )
```