Repository: mindsdb/mindsdb

Issue:
Datasource analysing refactoring
Couple thoughts about analysing:
History of analysing in 'upload ds' route:
1. Let come back to time when we work just with 'file' datasources. We dont run analysis after file uploaded. We do it if user want see analysis, or learn predictor.
2. We want add new feature: filter/sort for data in datasource in Scout 'data preview' screen. Doing in pandas.DataFrame is slow, so we decide use sqllite database. But how we can create database, if we dont know file columns types? If just copy csv to sqllite-db, then all  columns will be text, and all filters in Scout will be text. So we need analyse datasource right after it uploaded.
3. We add database-based datasources without any changes in datasource upload flow. How this flow looks now: route get from user query string and 'integration' name. Analyse process run. After it done, sqllite db created. 
Problems in this flow:
1. analysis can take long time
2. we save in sqllite db copy of data from real db.
3. if user will train predictor from that datasource after some time, predictor will be trained from current database data, not from sqllite data. But in 'datasource preview' will be still data from sqllite.
4. each time (not each really, it has some timeout), when user open 'datasource analysis', datasource analysis will be re-run.

How i see solve:
1. add statues for datasources. In this case, we can set status='analysing' for uploading datasource and make analysis in background. For datasources with this status should be disabled all interactions in Scout.
2. save datasource analysis in file in datasource folder. User should see date/time of last analysis and button 're-analyse' (probably just for db-based datasource, for file-based is make no sense). In this case we can open 'datasource analysis' screen without waiting.
3. dont make sqllite db for db-based datasources. User already have his data in database, we dont want make full copy of it.
4. point above mean we should disable 'datasource preview' screen for db-based datasources. Why: it will be difficult to make filter/sorting/pagination for that data. At least at this moment.



Discussion:
1. On upload, run an analysis on very little data, and don't cache.
2. We split analysis into two endpoints `analyze_refresh` and `analyze`
3. `analyze_refresh` will run an analysis on the whole datasource, and then dump that together with the rest of the datasource info.
4. `analyze` will check if the datasource has an analysis associated with it, and in that case return said analysis, otherwise it will call `analyze_refresh`
Store time when analysis was done (and return it together with the analysis)

Resolving patch:
diff --git a/mindsdb/api/http/namespaces/config.py b/mindsdb/api/http/namespaces/config.py
--- a/mindsdb/api/http/namespaces/config.py
+++ b/mindsdb/api/http/namespaces/config.py
@@ -59,6 +59,8 @@ def put(self, name):
         if integration is not None:
             abort(400, f"Integration with name '{name}' already exists")
         try:
+            if 'enabled' in params:
+                params['publish'] = params['enabled']
             ca.config_obj.add_db_integration(name, params)
 
             mdb = ca.mindsdb_native
@@ -92,6 +94,8 @@ def post(self, name):
         if integration is None:
             abort(400, f"Nothin to modify. '{name}' not exists.")
         try:
+            if 'enabled' in params:
+                params['publish'] = params['enabled']
             ca.config_obj.modify_db_integration(name, params)
             DatabaseWrapper(ca.config_obj)
         except Exception as e:
diff --git a/mindsdb/interfaces/database/database.py b/mindsdb/interfaces/database/database.py
--- a/mindsdb/interfaces/database/database.py
+++ b/mindsdb/interfaces/database/database.py
@@ -24,7 +24,7 @@ def _get_integrations(self):
         # @TODO Once we have a presistent state sorted out this should be simplified as to not refresh the existing integrations every single time
         integration_arr = []
         for db_alias in self.config['integrations']:
-            if self.config['integrations'][db_alias]['enabled']:
+            if self.config['integrations'][db_alias]['publish']:
                 db_type = self.config['integrations'][db_alias]['type']
                 if db_type == 'clickhouse':
                     integration_arr.append(Clickhouse(self.config, db_alias))
diff --git a/mindsdb/utilities/wizards.py b/mindsdb/utilities/wizards.py
--- a/mindsdb/utilities/wizards.py
+++ b/mindsdb/utilities/wizards.py
@@ -28,28 +28,28 @@ def auto_config(python_path, pip_path, storage_dir):
         },
         "integrations": {
             "default_clickhouse": {
-                "enabled": False,
+                "publish": False,
                 "type": 'clickhouse'
             },
             "default_mariadb": {
-                "enabled": False,
+                "publish": False,
                 "type": 'mariadb'
             },
             "default_mysql": {
-                "enabled": False,
+                "publish": False,
                 "type": 'mysql'
             },
             "default_postgres": {
-                "enabled": False,
+                "publish": False,
                 "type": 'postgres',
                 "database": 'postgres'
             },
             "default_mssql": {
-                "enabled": False,
+                "publish": False,
                 "type": 'mssql'
             },
             "default_mongodb": {
-                "enabled": False,
+                "publish": False,
                 "type": 'mongodb'
             }
         },
@@ -138,7 +138,7 @@ def cli_config(python_path, pip_path, storage_dir, config_dir, use_default=False
 
     clickhouse = _in('Connect to clickhouse ? [Y/N]', 'Y', use_default)
     if clickhouse in ['Y', 'y']:
-        config['integrations']['default_clickhouse']['enabled'] = _in('Enable Clickhouse integration?: ', False, use_default)
+        config['integrations']['default_clickhouse']['publish'] = _in('Enable Clickhouse integration?: ', False, use_default)
         config['integrations']['default_clickhouse']['host'] = _in('Clickhouse host: ', '127.0.0.1', use_default)
         config['integrations']['default_clickhouse']['port'] = _in('Clickhouse port: ', 8123, use_default)
         config['integrations']['default_clickhouse']['user'] = _in('Clickhouse user: ', 'default', use_default)
@@ -147,7 +147,7 @@ def cli_config(python_path, pip_path, storage_dir, config_dir, use_default=False
 
     mariadb = _in('Connect to Mariadb ? [Y/N]', 'Y', use_default)
     if mariadb in ['Y', 'y']:
-        config['integrations']['default_mariadb']['enabled'] = _in('Enable Mariadb integration?: ', False, use_default)
+        config['integrations']['default_mariadb']['publish'] = _in('Enable Mariadb integration?: ', False, use_default)
         config['integrations']['default_mariadb']['host'] = _in('Mariadb host: ', '127.0.0.1', use_default)
         config['integrations']['default_mariadb']['port'] = _in('Mariadb port: ', 3306, use_default)
         config['integrations']['default_mariadb']['user'] = _in('Mariadb user: ', 'root', use_default)
@@ -156,7 +156,7 @@ def cli_config(python_path, pip_path, storage_dir, config_dir, use_default=False
 
     mysql = _in('Connect to MySQL ? [Y/N]', 'Y', use_default)
     if mysql in ['Y', 'y']:
-        config['integrations']['default_mysql']['enabled'] = _in('Enable MySQL integration?: ', False, use_default)
+        config['integrations']['default_mysql']['publish'] = _in('Enable MySQL integration?: ', False, use_default)
         config['integrations']['default_mysql']['host'] = _in('MySQL host: ', '127.0.0.1', use_default)
         config['integrations']['default_mysql']['port'] = _in('MySQL port: ', 3306, use_default)
         config['integrations']['default_mysql']['user'] = _in('MySQL user: ', 'root', use_default)
@@ -165,7 +165,7 @@ def cli_config(python_path, pip_path, storage_dir, config_dir, use_default=False
 
     mysql = _in('Connect to PostgreSQL ? [Y/N]', 'Y', use_default)
     if mysql in ['Y', 'y']:
-        config['integrations']['default_postgres']['enabled'] = _in('Enable PostgreSQL integration?: ', False, use_default)
+        config['integrations']['default_postgres']['publish'] = _in('Enable PostgreSQL integration?: ', False, use_default)
         config['integrations']['default_postgres']['host'] = _in('PostgreSQL host: ', '127.0.0.1', use_default)
         config['integrations']['default_postgres']['port'] = _in('PostgreSQL port: ', 5432, use_default)
         config['integrations']['default_postgres']['user'] = _in('PostgreSQL user: ', 'postgres', use_default)
@@ -175,7 +175,7 @@ def cli_config(python_path, pip_path, storage_dir, config_dir, use_default=False
 
     mssql = _in('Connect to MSSQL ? [Y/N]', 'Y', use_default)
     if mssql in ['Y', 'y']:
-        config['integrations']['default_mssql']['enabled'] = _in('Enable MSSQL integration?: ', False, use_default)
+        config['integrations']['default_mssql']['publish'] = _in('Enable MSSQL integration?: ', False, use_default)
         config['integrations']['default_mssql']['host'] = _in('MSSQL host: ', '127.0.0.1', use_default)
         config['integrations']['default_mssql']['port'] = _in('MSSQL port: ', 1433, use_default)
         config['integrations']['default_mssql']['user'] = _in('MSSQL user: ', 'sa', use_default)
@@ -185,7 +185,7 @@ def cli_config(python_path, pip_path, storage_dir, config_dir, use_default=False
 
     mongodb = _in('Connect to MongoDB ? [Y/N]', 'Y', use_default)
     if mongodb in ['Y', 'y']:
-        config['integrations']['default_mongodb']['enabled'] = _in('Enable MongoDB integration?: ', False, use_default)
+        config['integrations']['default_mongodb']['publish'] = _in('Enable MongoDB integration?: ', False, use_default)
         config['integrations']['default_mongodb']['host'] = _in('MongoDB host: ', '127.0.0.1', use_default)
         config['integrations']['default_mongodb']['port'] = _in('MongoDB port: ', 27017, use_default)
         config['integrations']['default_mongodb']['user'] = _in('MongoDB user: ', '', use_default)
@@ -193,7 +193,7 @@ def cli_config(python_path, pip_path, storage_dir, config_dir, use_default=False
         config['integrations']['default_mongodb']['type'] = 'mongodb'
 
     for db_name in list(config['integrations'].keys()):
-        if not config['integrations'][db_name]['enabled']:
+        if not config['integrations'][db_name]['publish']:
             del config['integrations'][db_name]
 
 


Test patch:
diff --git a/tests/integration_tests/api/test_http.py b/tests/integration_tests/api/test_http.py
--- a/tests/integration_tests/api/test_http.py
+++ b/tests/integration_tests/api/test_http.py
@@ -32,10 +32,10 @@ def setUpClass(cls):
             apis=['http'],
             override_integration_config={
                 'default_mariadb': {
-                    'enabled': True
+                    'publish': True
                 },
                 'default_clickhouse': {
-                    'enabled': True
+                    'publish': True
                 }
             },
             mindsdb_database=common.MINDSDB_DATABASE
@@ -60,7 +60,7 @@ def test_1_config(self):
         for integration_name in integration_names['integrations']:
             assert integration_name in self.initial_integrations_names
 
-        test_integration_data = {'enabled': False, 'host': 'test', 'type': 'clickhouse'}
+        test_integration_data = {'publish': False, 'host': 'test', 'type': 'clickhouse'}
         res = requests.put(f'{root}/config/integrations/test_integration', json={'params': test_integration_data})
         assert res.status_code == 200
 
@@ -85,7 +85,7 @@ def test_1_config(self):
             assert res.status_code == 200
 
             integration = res.json()
-            for k in ['enabled', 'host', 'port', 'type', 'user']:
+            for k in ['publish', 'host', 'port', 'type', 'user']:
                 assert k in integration
                 assert integration[k] is not None
             assert integration['password'] is None
diff --git a/tests/integration_tests/flows/common.py b/tests/integration_tests/flows/common.py
--- a/tests/integration_tests/flows/common.py
+++ b/tests/integration_tests/flows/common.py
@@ -101,7 +101,7 @@
 
 def prepare_config(config, mindsdb_database='mindsdb', override_integration_config={}, override_api_config={}, clear_storage=True):
     for key in config._config['integrations']:
-        config._config['integrations'][key]['enabled'] = False
+        config._config['integrations'][key]['publish'] = False
 
     if USE_EXTERNAL_DB_SERVER:
         with open(EXTERNAL_DB_CREDENTIALS, 'rt') as f:
diff --git a/tests/integration_tests/flows/config/config.json b/tests/integration_tests/flows/config/config.json
--- a/tests/integration_tests/flows/config/config.json
+++ b/tests/integration_tests/flows/config/config.json
@@ -20,7 +20,7 @@
     "debug": true,
     "integrations": {
         "default_clickhouse": {
-            "enabled": true,
+            "publish": true,
             "host": "localhost",
             "password": "",
             "port": 8123,
@@ -28,7 +28,7 @@
             "user": "default"
         },
         "default_mariadb": {
-            "enabled": true,
+            "publish": true,
             "host": "localhost",
             "password": "root",
             "port": 3306,
@@ -36,7 +36,7 @@
             "user": "root"
         },
         "default_mongodb": {
-            "enabled": false,
+            "publish": false,
             "host": "127.0.0.1",
             "password": "",
             "port": 27001,
@@ -44,7 +44,7 @@
             "user": ""
         },
         "default_mssql": {
-            "enabled": true,
+            "publish": true,
             "host": "127.0.0.1",
             "odbc_driver_name": "MySQL ODBC 8.0 Unicode Driver",
             "password": "password",
@@ -53,7 +53,7 @@
             "user": "sa"
         },
         "default_mysql": {
-            "enabled": true,
+            "publish": true,
             "host": "localhost",
             "password": "root",
             "port": 3307,
@@ -62,7 +62,7 @@
         },
         "default_postgres": {
             "database": "postgres",
-            "enabled": true,
+            "publish": true,
             "host": "localhost",
             "password": "",
             "port": 5432,
@@ -70,7 +70,7 @@
             "user": "postgres"
         },
         "default_snowflake": {
-            "enabled": false,
+            "publish": false,
             "type": "snowflake"
         }
     },
diff --git a/tests/integration_tests/flows/test_clickhouse.py b/tests/integration_tests/flows/test_clickhouse.py
--- a/tests/integration_tests/flows/test_clickhouse.py
+++ b/tests/integration_tests/flows/test_clickhouse.py
@@ -98,7 +98,7 @@ def setUpClass(cls):
             apis=['mysql'],
             override_integration_config={
                 'default_clickhouse': {
-                    'enabled': True
+                    'publish': True
                 }
             },
             override_api_config={
diff --git a/tests/integration_tests/flows/test_custom_model.py b/tests/integration_tests/flows/test_custom_model.py
--- a/tests/integration_tests/flows/test_custom_model.py
+++ b/tests/integration_tests/flows/test_custom_model.py
@@ -80,7 +80,7 @@ def setUpClass(cls):
             apis=['http', 'mysql'],
             override_integration_config={
                 'default_mariadb': {
-                    'enabled': True
+                    'publish': True
                 }
             },
             mindsdb_database=MINDSDB_DATABASE
diff --git a/tests/integration_tests/flows/test_mariadb.py b/tests/integration_tests/flows/test_mariadb.py
--- a/tests/integration_tests/flows/test_mariadb.py
+++ b/tests/integration_tests/flows/test_mariadb.py
@@ -83,7 +83,7 @@ def setUpClass(cls):
             apis=['mysql'],
             override_integration_config={
                 'default_mariadb': {
-                    'enabled': True
+                    'publish': True
                 }
             },
             mindsdb_database=MINDSDB_DATABASE
diff --git a/tests/integration_tests/flows/test_mistakes.py b/tests/integration_tests/flows/test_mistakes.py
--- a/tests/integration_tests/flows/test_mistakes.py
+++ b/tests/integration_tests/flows/test_mistakes.py
@@ -46,7 +46,7 @@
 class UserFlowTest_1(unittest.TestCase):
     def test_1_wrong_integration(self):
         '''
-        start mindsdb with enabled integration with wrong password
+        start mindsdb with publish integration with wrong password
         try create ds
         change password to correct
         '''
@@ -56,7 +56,7 @@ def test_1_wrong_integration(self):
             apis=['mysql', 'http'],
             override_integration_config={
                 'default_mariadb': {
-                    'enabled': True,
+                    'publish': True,
                     'password': 'broken'
                 }
             },
@@ -117,7 +117,7 @@ def test_2_broke_analisys(self):
             apis=['mysql', 'http'],
             override_integration_config={
                 'default_mariadb': {
-                    'enabled': True
+                    'publish': True
                 }
             },
             mindsdb_database=MINDSDB_DATABASE,
@@ -157,7 +157,7 @@ def test_3_wrong_predictor(self):
             apis=['mysql', 'http'],
             override_integration_config={
                 'default_mariadb': {
-                    'enabled': True
+                    'publish': True
                 }
             },
             mindsdb_database=MINDSDB_DATABASE,
@@ -199,7 +199,7 @@ def test_4_wrong_prediction(self):
             apis=['mysql', 'http'],
             override_integration_config={
                 'default_mariadb': {
-                    'enabled': True
+                    'publish': True
                 }
             },
             mindsdb_database=MINDSDB_DATABASE,
diff --git a/tests/integration_tests/flows/test_mongo.py b/tests/integration_tests/flows/test_mongo.py
--- a/tests/integration_tests/flows/test_mongo.py
+++ b/tests/integration_tests/flows/test_mongo.py
@@ -53,7 +53,7 @@ def setUpClass(cls):
             apis=['mongodb'],
             override_integration_config={
                 'default_mongodb': {
-                    'enabled': True,
+                    'publish': True,
                     'port': 27002,
                     'host': '127.0.0.1',
                     'type': 'mongodb',
diff --git a/tests/integration_tests/flows/test_mssql.py b/tests/integration_tests/flows/test_mssql.py
--- a/tests/integration_tests/flows/test_mssql.py
+++ b/tests/integration_tests/flows/test_mssql.py
@@ -59,7 +59,7 @@ def setUpClass(cls):
             apis=['mysql'],
             override_integration_config={
                 'default_mssql': {
-                    'enabled': True
+                    'publish': True
                 }
             },
             mindsdb_database=MINDSDB_DATABASE
diff --git a/tests/integration_tests/flows/test_mysql.py b/tests/integration_tests/flows/test_mysql.py
--- a/tests/integration_tests/flows/test_mysql.py
+++ b/tests/integration_tests/flows/test_mysql.py
@@ -83,7 +83,7 @@ def setUpClass(cls):
             apis=['mysql'],
             override_integration_config={
                 'default_mysql': {
-                    'enabled': True
+                    'publish': True
                 }
             },
             mindsdb_database=MINDSDB_DATABASE
diff --git a/tests/integration_tests/flows/test_postgres.py b/tests/integration_tests/flows/test_postgres.py
--- a/tests/integration_tests/flows/test_postgres.py
+++ b/tests/integration_tests/flows/test_postgres.py
@@ -89,7 +89,7 @@ def setUpClass(cls):
             apis=['mysql'],
             override_integration_config={
                 'default_postgres': {
-                    'enabled': True
+                    'publish': True
                 }
             },
             mindsdb_database=MINDSDB_DATABASE
diff --git a/tests/integration_tests/flows/test_user_flow_1.py b/tests/integration_tests/flows/test_user_flow_1.py
--- a/tests/integration_tests/flows/test_user_flow_1.py
+++ b/tests/integration_tests/flows/test_user_flow_1.py
@@ -123,7 +123,7 @@ def test_1_create_integration_via_http(self):
 
         test_integration_data = {}
         test_integration_data.update(config['integrations']['default_mariadb'])
-        test_integration_data['enabled'] = True
+        test_integration_data['publish'] = True
         test_integration_data['database_name'] = TEST_INTEGRATION
         res = requests.put(f'{HTTP_API_ROOT}/config/integrations/{TEST_INTEGRATION}', json={'params': test_integration_data})
         assert res.status_code == 200
@@ -132,7 +132,7 @@ def test_1_create_integration_via_http(self):
         assert res.status_code == 200
         test_integration = res.json()
         assert test_integration['password'] is None
-        for key in ['user', 'port', 'host', 'enabled']:
+        for key in ['user', 'port', 'host', 'publish']:
             assert test_integration[key] == test_integration_data[key]
 
     def test_3_create_ds_from_sql_by_http(self):
diff --git a/tests/integration_tests/flows/test_user_flow_2.py b/tests/integration_tests/flows/test_user_flow_2.py
--- a/tests/integration_tests/flows/test_user_flow_2.py
+++ b/tests/integration_tests/flows/test_user_flow_2.py
@@ -137,7 +137,7 @@ def test_1_upload_ds(self):
     def test_2_add_integration(self):
         test_integration_data = {}
         test_integration_data.update(config['integrations']['default_mariadb'])
-        test_integration_data['enabled'] = True
+        test_integration_data['publish'] = True
         test_integration_data['database_name'] = TEST_INTEGRATION
         res = requests.put(f'{HTTP_API_ROOT}/config/integrations/{TEST_INTEGRATION}', json={'params': test_integration_data})
         assert res.status_code == 200
@@ -150,7 +150,7 @@ def test_3_restart_and_connect(self):
             apis=['mysql'],
             override_integration_config={
                 'default_mariadb': {
-                    'enabled': True
+                    'publish': True
                 }
             },
             mindsdb_database=MINDSDB_DATABASE,