# Create the data handler #

Each party in a Federated Learning experiment must get a data handler to process their data\. You or a data scientist must create the data handler\. A data handler is a Python class that loads and transforms data so that all data for the experiment is in a consistent format\.

## About the data handler class ##

The data handler performs the following functions:

<!-- <ul> -->

 *  Accesses the data that is required to train the model\. For example, reads data from a CSV file into a Pandas data frame\.
 *  Pre\-processes the data so data is in a consistent format across all parties\. Some example cases are as follows:
    
    <!-- <ul> -->
    
     *  The **Date** column might be stored as a time epoch or timestamp.
     *  The **Country** column might be encoded or abbreviated.
    
    <!-- </ul> -->
    
 *  The data handler ensures that the data formatting is in agreement\.
    
    <!-- <ul> -->
    
     *  *Optional:* feature engineer as needed.
    
    <!-- </ul> -->
    
<!-- </ul> -->

The following illustration shows how a data handler is used to process data and make it consumable by the experiment:

![A use case of the data handler unifying data formats](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/images/fl-data-handler.svg)

## Data handler template ##

A general data handler template is as follows:

    # your import statements
    
    from ibmfl.data.data_handler import DataHandler
    
    class MyDataHandler(DataHandler):
        """
        Data handler for your dataset.
        """
        def __init__(self, data_config=None):
            super().__init__()
            self.file_name = None
            if data_config is not None:
                # This can be any string field.
                # For example, if your data set is in `csv` format,
                # <your_data_file_type> can be "CSV", ".csv", "csv", "csv_file" and more.
                if '<your_data_file_type>' in data_config:
                    self.file_name = data_config['<your_data_file_type>']
                # extract other additional parameters from `info` if any.
    
            # load and preprocess the training and testing data
            self.load_and_preprocess_data()
    
            """
            # Example:
            # (self.x_train, self.y_train), (self.x_test, self.y_test) = self.load_dataset()
            """
    
        def load_and_preprocess_data(self):
            """
            Loads and pre-processeses local datasets, 
            and updates self.x_train, self.y_train, self.x_test, self.y_test.
    
            # Example:
            # return (self.x_train, self.y_train), (self.x_test, self.y_test)
            """
    
            pass
    
        def get_data(self):
            """
            Gets the prepared training and testing data.
    
            :return: ((x_train, y_train), (x_test, y_test)) # most build-in training modules expect data is returned in this format
            :rtype: `tuple` 
    
            This function should be as brief as possible. Any pre-processing operations should be performed in a separate function and not inside get_data(), especially computationally expensive ones.
    
            # Example:
            # X, y = load_somedata()
            # x_train, x_test, y_train, y_test = \
            # train_test_split(X, y, test_size=TEST_SIZE, random_state=RANDOM_STATE)
            # return (x_train, y_train), (x_test, y_test)
            """
            pass
    
        def preprocess(self, X, y):
            pass

**Parameters**

<!-- <ul> -->

 *  `your_data_file_type`: This can be any string field\. For example, if your data set is in `csv` format, `your_data_file_type` can be "CSV", "\.csv", "csv", "csv\_file" and more\.

<!-- </ul> -->

### Return a data generator defined by Keras or Tensorflow ###

The following is a code example that needs to be included as part of the `get_data` function to return a data generator defined by Keras or Tensorflow:

    train_gen = ImageDataGenerator(rotation_range=8,
                                    width_sht_range=0.08,
                                    shear_range=0.3,
                                    height_shift_range=0.08,
                                    zoom_range=0.08)
    
    train_datagenerator = train_gen.flow(
        x_train, y_train, batch_size=64)
    
    return train_datagenerator

## Data handler examples ##

<!-- <ul> -->

 *  [MNIST Keras data handler](https://github.com/IBMDataScience/sample-notebooks/blob/master/Files/mnist_keras_data_handler.py)
 *  [Adult XGBoost data handler](https://github.com/IBMDataScience/sample-notebooks/blob/master/Files/adult_sklearn_data_handler.py)

<!-- </ul> -->

**Parent topic:**[Creating a Federated Learning experiment](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fl-start.html)

<!-- </article "role="article" "> -->