Repository: mindsdb/mindsdb

Issue:
Datasource analysing refactoring
Couple thoughts about analysing:
History of analysing in 'upload ds' route:
1. Let come back to time when we work just with 'file' datasources. We dont run analysis after file uploaded. We do it if user want see analysis, or learn predictor.
2. We want add new feature: filter/sort for data in datasource in Scout 'data preview' screen. Doing in pandas.DataFrame is slow, so we decide use sqllite database. But how we can create database, if we dont know file columns types? If just copy csv to sqllite-db, then all  columns will be text, and all filters in Scout will be text. So we need analyse datasource right after it uploaded.
3. We add database-based datasources without any changes in datasource upload flow. How this flow looks now: route get from user query string and 'integration' name. Analyse process run. After it done, sqllite db created. 
Problems in this flow:
1. analysis can take long time
2. we save in sqllite db copy of data from real db.
3. if user will train predictor from that datasource after some time, predictor will be trained from current database data, not from sqllite data. But in 'datasource preview' will be still data from sqllite.
4. each time (not each really, it has some timeout), when user open 'datasource analysis', datasource analysis will be re-run.

How i see solve:
1. add statues for datasources. In this case, we can set status='analysing' for uploading datasource and make analysis in background. For datasources with this status should be disabled all interactions in Scout.
2. save datasource analysis in file in datasource folder. User should see date/time of last analysis and button 're-analyse' (probably just for db-based datasource, for file-based is make no sense). In this case we can open 'datasource analysis' screen without waiting.
3. dont make sqllite db for db-based datasources. User already have his data in database, we dont want make full copy of it.
4. point above mean we should disable 'datasource preview' screen for db-based datasources. Why: it will be difficult to make filter/sorting/pagination for that data. At least at this moment.



Discussion:
1. On upload, run an analysis on very little data, and don't cache.
2. We split analysis into two endpoints `analyze_refresh` and `analyze`
3. `analyze_refresh` will run an analysis on the whole datasource, and then dump that together with the rest of the datasource info.
4. `analyze` will check if the datasource has an analysis associated with it, and in that case return said analysis, otherwise it will call `analyze_refresh`
Store time when analysis was done (and return it together with the analysis)

Resolving patch:
diff --git a/mindsdb/api/http/namespaces/datasource.py b/mindsdb/api/http/namespaces/datasource.py
--- a/mindsdb/api/http/namespaces/datasource.py
+++ b/mindsdb/api/http/namespaces/datasource.py
@@ -152,8 +152,6 @@ def get(self, name):
         if name in ds_analysis:
             if ds_analysis[name] is None:
                 return {'status': 'analyzing'}, 200
-            elif (datetime.datetime.utcnow() - ds_analysis[name]['created_at']) > datetime.timedelta(seconds=3600):
-                del ds_analysis[name]
             else:
                 analysis = ds_analysis[name]['data']
                 return analysis, 200
@@ -167,6 +165,27 @@ def get(self, name):
         x.start()
         return {'status': 'analyzing'}, 200
 
+@ns_conf.route('/<name>/analyze_refresh')
+@ns_conf.param('name', 'Datasource name')
+class Analyze(Resource):
+    @ns_conf.doc('analyze_refresh_dataset')
+    def get(self, name):
+        global ds_analysis
+        if name in ds_analysis:
+            if ds_analysis[name] is None:
+                return {'status': 'analyzing'}, 200
+            else:
+                del ds_analysis[name]
+
+        ds = ca.default_store.get_datasource(name)
+        if ds is None:
+            print('No valid datasource given')
+            abort(400, 'No valid datasource given')
+
+        x = threading.Thread(target=analyzing_thread, args=(name, ca.default_store))
+        x.start()
+        return {'status': 'analyzing'}, 200
+
 
 @ns_conf.route('/<name>/analyze_subset')
 @ns_conf.param('name', 'Datasource name')


Test patch: