Repository: mindsdb/lightwood

Issue:
Add `export` method that saves both `Predictor` object and its code.
Respectively, `import` should load the predictor+code.


Discussion:
This is what I'm doing to save and load predictors and their code.  I'm working with several predictors at once.
 
    def save_predictors(self,tag=""):
        for i, predictor in enumerate(self.predictors):
            predictor.save("predictors/"+str(self.targets[i])+tag)
        for i, code in enumerate(self.codes):
            with open("code/"+str(self.targets[i])+"_code"+tag, "wb") as fp:
                dill.dump(self.codes[i], fp)

    def load_predictors(self,tag=""):
        predictors = []
        predictor_files = sorted(glob.glob('predictors/'+'*'+tag))
        code_files = sorted(glob.glob('code/'+'*'+tag))
        print('predictors:', predictor_files)
        print('predictors loaded')
        for i, predictor in enumerate(predictor_files):
            with open(code_files[i], 'rb') as fp:
                code = dill.load(fp)
            predictors.append(predictor_from_state(predictor, code))
        self.predictors = predictors
        self.tag = tag

Nice, that certainly works. 

We've discussed previously with other contributors that the ideal format for this method would be to `dill.dump()` an object (e.g. a dictionary) that contains both code and predictor instance (which includes the weights, analysis, etc.) in the same binary.

An implementation would be very similar to the current `.save()` method, but calling `code_from_json_ai()` previously to generate that artifact on demand. _However_, this also means we should be saving the actual JsonAI within the predictor instance first.

Are you interested in contributing with a Pull Request @gwild?

 
Is this still on demand @paxcema ?
Hi @TalaatHasanin, yes it is :)

If you want to try your hand at it, we can assign you. Feel free to ask any questions regarding design or implementation of this.
Now I'm installing the package to try it and try to understand.
if there is anything to learn or read about the implementation I would be vey grateful ?
Hey was thinking about looking into this issue. But I was wondering if I can get a reference to the relevant file where this should be implemented? 
Would it be predictor.py?
@paxcema  are you wanting something like this?????

I just started doing some basic outlining of the idea.

````

import json
import dill

class Predictor:
    # Existing code ...

    def save(self, file_path: str) -> None:
        """
        With a provided file path, saves the Predictor instance for later use.

        :param file_path: Location to store your Predictor Instance.

        :returns: Saves Predictor instance.
        """
        predictor_dict = {}
        predictor_dict['predictor'] = self.__dict__
        predictor_dict['code'] = json.loads(self.to_json())

        with open(file_path, "wb") as fp:
            dill.dump(predictor_dict, fp)

    def to_json(self):
        """
        Serializes the Predictor to JSON format.

        :returns: Serialized JSON string.
        """
        return json.dumps(self.to_dict())

    def to_dict(self):
        """
        Converts the Predictor object into a dictionary.

        :returns: Dictionary representation of the Predictor object.
        """
        predictor_dict = {}
        predictor_dict['config'] = self.config.to_dict()
        predictor_dict['model'] = self.model.to_dict()
        predictor_dict['tokenizer'] = self.tokenizer.to_dict()
        predictor_dict['analyzer'] = self.analyzer.to_dict()
        return predictor_dict

Hi @noraa-july-stoke,

Yes, that `save()` method is close to what we're after, although not exactly. Please open a pull request and we can provide specific feedback on what's missing or should change.
@paxcema ok will do! thank you for clarifying!!  
im assuming you want me to use that code_to_json_ai() function  and use the value returned from it also?
i'll go ahead and make that change and submit PR, and await further comments, thank you!

Resolving patch:
diff --git a/lightwood/api/predictor.py b/lightwood/api/predictor.py
--- a/lightwood/api/predictor.py
+++ b/lightwood/api/predictor.py
@@ -11,7 +11,7 @@ class PredictorInterface:
     Abstraction of a Lightwood predictor. The ``PredictorInterface`` encompasses how Lightwood interacts with the full ML pipeline. Internally,
 
     The ``PredictorInterface`` class must have several expected functions:
-    
+
     - ``analyze_data``: Peform a statistical analysis on the unprocessed data; this helps inform downstream encoders and mixers on how to treat the data types.
     - ``preprocess``: Apply cleaning functions to each of the columns within the dataset to prepare them for featurization
     - ``split``: Split the input dataset into a train/dev/test set according to your splitter function
@@ -24,7 +24,7 @@ class PredictorInterface:
     For simplification, we offer an end-to-end approach that allows you to input raw data and follow every step of the process until you reach a trained predictor with the ``learn`` function:
 
         - ``learn``: An end-to-end technique specifying how to pre-process, featurize, and train the model(s) of interest. The expected input is raw, untrained data. No explicit output is provided, but the Predictor object will "host" the trained model thus.
-    
+
     You can also use the predictor to now estimate new data:
 
     - ``predict``: Deploys the chosen best model, and evaluates the given data to provide target estimates.
@@ -85,7 +85,7 @@ def featurize(self, split_data: Dict[str, pd.DataFrame]):
 
     def fit(self, enc_data: Dict[str, pd.DataFrame]) -> None:
         """
-        Fits "mixer" models to train predictors on the featurized data. Instantiates a set of trained mixers and an ensemble of them. 
+        Fits "mixer" models to train predictors on the featurized data. Instantiates a set of trained mixers and an ensemble of them.
 
         :param enc_data: Pre-processed and featurized data, split into the relevant train/test splits. Keys expected are "train", "dev", and "test"
         """  # noqa
@@ -103,7 +103,7 @@ def learn(self, data: pd.DataFrame) -> None:
         """
         Trains the attribute model starting from raw data. Raw data is pre-processed and cleaned accordingly. As data is assigned a particular type (ex: numerical, categorical, etc.), the respective feature encoder will convert it into a representation useable for training ML models. Of all ML models requested, these models are compiled and fit on the training data.
 
-        This step amalgates ``preprocess`` -> ``featurize`` -> ``fit`` with the necessary splitting + analyze_data that occurs. 
+        This step amalgates ``preprocess`` -> ``featurize`` -> ``fit`` with the necessary splitting + analyze_data that occurs.
 
         :param data: (Unprocessed) Data used in training the model(s).
 
@@ -116,7 +116,7 @@ def adjust(self, new_data: pd.DataFrame, old_data: Optional[pd.DataFrame] = None
         """
         Adjusts a previously trained model on new data. Adopts the same process as ``learn`` but with the exception that the `adjust` function expects the best model to have been already trained.
 
-        .. warning:: This is experimental and subject to change. 
+        .. warning:: This is experimental and subject to change.
         :param new_data: New data used to adjust a previously trained model.
         :param old_data: In some situations, the old data is still required to train a model (i.e. Regression mixer) to ensure the new data doesn't entirely override it.
         :param adjust_args: Optional dictionary with parameters to customize the finetuning process.
@@ -139,10 +139,25 @@ def predict(self, data: pd.DataFrame, args: Dict[str, object] = {}) -> pd.DataFr
     def save(self, file_path: str) -> None:
         """
         With a provided file path, saves the Predictor instance for later use.
+        :param file_path: Location to store your Predictor Instance.
+        :returns: Saves Predictor instance.
+        """
+        with open(file_path, "wb") as fp:
+            dill.dump(self, fp)
+
+    def export(self, file_path: str, json_ai_code: str) -> None:
+        """
+        Exports both the predictor object and its code to a single binary file for later usage.
 
         :param file_path: Location to store your Predictor Instance.
+        :param json_ai_code: The code generated by the user's specification.
 
         :returns: Saves Predictor instance.
         """
+
+        predictor_dict = {}
+        predictor_dict['predictor'] = self
+        predictor_dict['code'] = json_ai_code
+
         with open(file_path, "wb") as fp:
-            dill.dump(self, fp)
+            dill.dump(predictor_dict, fp)


Test patch: