Repository: mindsdb/mindsdb

Issue:
Add consistency check for model data object in the CI tests
We should take a snapshot of a "correct" model data object (as returned by `get_model_data()`) for the CI testing dataset and always compare the json-parsed outputs of `get_model_data` with it.

Might be a bit tedious to alter every time when we run `get_model_data` ,but otherwise there are certain situation where we can get silent failures. Since certain parts of the Model Analyzer have more general exception catching built in (since certain data simply can't be produced in various situations) and `get_model_data` itself is made to work with incomplete data in order to return this data during training.

If we want to be really scrupulous we should check the model data object at each phase of training, but I think that'd be overdoing it.


Discussion:

Resolving patch:
diff --git a/mindsdb/__about__.py b/mindsdb/__about__.py
--- a/mindsdb/__about__.py
+++ b/mindsdb/__about__.py
@@ -1,6 +1,6 @@
 __title__ = 'MindsDB'
 __package_name__ = 'mindsdb'
-__version__ = '1.7.11'
+__version__ = '1.7.12'
 __description__ = "MindsDB's goal is to make it very simple for developers to use the power of artificial neural networks in their projects."
 __email__ = "jorge@mindsdb.com"
 __author__ = 'MindsDB Inc'


Test patch:
diff --git a/tests/ci_tests/tests.py b/tests/ci_tests/tests.py
--- a/tests/ci_tests/tests.py
+++ b/tests/ci_tests/tests.py
@@ -10,8 +10,9 @@ def basic_test(backend='ludwig',use_gpu=True,ignore_columns=[], run_extra=False,
             os.system(f'python3 ../functional_testing/{py_file}')
 
     # Create & Learn
+    to_predict = 'rental_price'
     mdb = mindsdb.Predictor(name='home_rentals_price')
-    mdb.learn(to_predict='rental_price',from_data="https://s3.eu-west-2.amazonaws.com/mindsdb-example-data/home_rentals.csv",backend=backend, stop_training_in_x_seconds=20,use_gpu=use_gpu)
+    mdb.learn(to_predict=to_predict,from_data="https://s3.eu-west-2.amazonaws.com/mindsdb-example-data/home_rentals.csv",backend=backend, stop_training_in_x_seconds=20,use_gpu=use_gpu)
 
     # Reload & Predict
     model_name = 'home_rentals_price'
@@ -49,4 +50,30 @@ def basic_test(backend='ludwig',use_gpu=True,ignore_columns=[], run_extra=False,
     # See if we can get the adapted metadata
     amd = mdb.get_model_data(model_name)
     # Make some simple assertions about it
-    assert(5 < len(list(amd.keys())))
+
+    # @TODO: Sometimes are None, not sure why: [, validation_set_accuracy, accuracy]
+    for k in ['status', 'name', 'version', 'data_source', 'current_phase', 'updated_at', 'created_at', 'train_end_at']:
+        assert(type(amd[k]) == str)
+    assert(type(amd['predict']) == list or type(amd['predict']) == str)
+    assert(type(amd['is_active']) == bool)
+
+    for k in ['validation_set_accuracy', 'accuracy']:
+        assert(type(amd[k]) == float)
+
+    for k in amd['data_preparation']:
+        assert(type(amd['data_preparation'][k]) == int or type(amd['data_preparation'][k]) == float)
+
+
+    for k in amd['data_analysis']:
+        assert(len(amd['data_analysis'][k]) > 0)
+        assert(type(amd['data_analysis'][k][0]) == dict)
+
+    assert(len(amd['model_analysis']) > 0)
+    assert(type(amd['model_analysis'][0]) == dict)
+
+    for k in amd['force_vectors'][to_predict]['normal_data_distribution']:
+        assert(len(amd['force_vectors'][to_predict]['normal_data_distribution'][k]) > 0)
+
+    for k in amd['force_vectors'][to_predict]['missing_data_distribution']:
+        for sk in amd['force_vectors'][to_predict]['missing_data_distribution'][k]:
+            assert(len(amd['force_vectors'][to_predict]['missing_data_distribution'][k][sk]) > 0)