internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mindsdb/mindsdb

Issue:
Cant get GUI if `storage_dir` is relative path in config
If specify in config.json storage_dir as relative path, then `GET /index.html` return 404. With absolute paths all work fine.


Discussion:
If the relative path in config is  e.g "storage_dir": "storage/" /index.html will return 404. Adding the full path as     "storage_dir": "/home/zoran/mindsdb-examples/mysql/storage/" it will work. Note that you need to run MindsDB with config `python -m mindsdb --api=http,mysql --config=config.json`


Resolving patch:
diff --git a/mindsdb/utilities/config.py b/mindsdb/utilities/config.py
--- a/mindsdb/utilities/config.py
+++ b/mindsdb/utilities/config.py
@@ -57,7 +57,7 @@ def __init__(self, config_path):
         self._config_hash = None
         self._config = None
         if isinstance(config_path, str):
-            self.config_path = config_path
+            self.config_path = os.path.abspath(config_path)
             self._read()
             self._config_hash = self._gen_hash()
 
@@ -65,7 +65,7 @@ def __init__(self, config_path):
             if os.path.isabs(storage_dir) is False:
                 storage_dir = os.path.normpath(
                     os.path.join(
-                        os.path.dirname(config_path),
+                        os.path.dirname(self.config_path),
                         storage_dir
                     )
                 )


Test patch:
Repository: mindsdb/mindsdb

Issue:
Route to check connection
At this moment in GUI possible check connection only for already exists integration. Route need for checking connection before integration created.


Discussion:
This could just be a parameter, e.g. `test` on the current integration endpoint, since you still have to send all the integration-creation data anyway.
if `test=True` => create integration => check connection => delete integration
Also make sure *not* to have this running at the same time for the same integration due to the create+delete logic (maybe just append a random string to the id being sent to avoid this?)

Resolving patch:
diff --git a/mindsdb/api/http/namespaces/config.py b/mindsdb/api/http/namespaces/config.py
--- a/mindsdb/api/http/namespaces/config.py
+++ b/mindsdb/api/http/namespaces/config.py
@@ -55,6 +55,11 @@ def put(self, name):
         params = request.json.get('params')
         if not isinstance(params, dict):
             abort(400, "type of 'params' must be dict")
+
+        is_test = params.get('test', False)
+        if is_test:
+            del params['test']
+
         integration = get_integration(name)
         if integration is not None:
             abort(400, f"Integration with name '{name}' already exists")
@@ -72,6 +77,12 @@ def put(self, name):
         except Exception as e:
             print(traceback.format_exc())
             abort(500, f'Error during config update: {str(e)}')
+
+        if is_test:
+            cons = dbw.check_connections()
+            ca.config_obj.remove_db_integration(name)
+            return {'success': cons[name]}, 200
+
         return '', 200
 
     @ns_conf.doc('delete_integration')


Test patch:
Repository: mindsdb/mindsdb

Issue:
~/.local/etc/mindsdb and ~/.local/var/lib/mindsdb have been created in current working directory
**Your Environment**

* Python version: 3.7.6
* Pip version: 19.2.3
* Operating system: MacOS 10.15.7
* Python environment used (e.g. venv, conda):  venv
* Mindsdb version you tried to install:2.13.8 (staging)

**Bug description**
~/.local/etc/mindsdb and ~/.local/var/lib/mindsdb have been created in current working directory instead of creating
.local/etc/mindsdb and .local/var/lib/mindsdb in user home directory


**To Reproduce**
1. Create next directories and revoke `write` permissions from current user for all of these dirs:
            f'{mindsdb_path}/etc/',
            f'{mindsdb_path}/var/'
            '/etc/mindsdb',
             '/var/lib/mindsdb'

**Expected behavior**
.local/etc/mindsdb and .local/var/lib/mindsdb created in user home directory

**Actual behavior**
~/.local/etc/mindsdb and ~/.local/var/lib/mindsdb created in current working directory:


**Additional context**
```
mindsdb_venv) ~/repos/MindsDB/mindsdb itsyplen$ python -m mindsdb
Athena Datasource is not available by default. If you wish to use it, please install mindsdb_native[extra_data_sources]
SnowflakeDS Datasource is not available by default. If you wish to use it, please install mindsdb_native[snowflake]
Google Cloud Storage Datasource is not available by default. If you wish to use it, please install mindsdb_native[extra_data_sources]
Configuration file:
   ~/.local/etc/mindsdb/config.json
Storage path:
   /Users/itsyplen/repos/MindsDB/mindsdb/~/.local/etc/mindsdb/~/.local/var/lib/mindsdb
Versions:
```



Discussion:

Resolving patch:
diff --git a/mindsdb/utilities/fs.py b/mindsdb/utilities/fs.py
--- a/mindsdb/utilities/fs.py
+++ b/mindsdb/utilities/fs.py
@@ -40,8 +40,8 @@ def get_paths():
                 '/var/lib/mindsdb'
             ),
             (
-                '~/.local/etc/mindsdb',
-                '~/.local/var/lib/mindsdb'
+                '{}/.local/etc/mindsdb'.format(Path.home()),
+                '{}/.local/var/lib/mindsdb'.format(Path.home())
             )
         ])
 
@@ -51,9 +51,9 @@ def get_paths():
 def get_or_create_dir_struct():
     for tup in get_paths():
         try:
-            for dir in tup:
-                assert(os.path.exists(dir))
-                assert(os.access(dir, os.W_OK) is True)
+            for _dir in tup:
+                assert os.path.exists(_dir)
+                assert os.access(_dir, os.W_OK) is True
 
             config_dir = tup[0]
             if 'DEV_CONFIG_PATH' in os.environ:
@@ -65,9 +65,9 @@ def get_or_create_dir_struct():
 
     for tup in get_paths():
         try:
-            for dir in tup:
-                create_directory(dir)
-                assert(os.access(dir, os.W_OK) is True)
+            for _dir in tup:
+                create_directory(_dir)
+                assert os.access(_dir, os.W_OK) is True
 
             config_dir = tup[0]
             if 'DEV_CONFIG_PATH' in os.environ:


Test patch:
Repository: mindsdb/mindsdb

Issue:
Endpoint to disable telemtry
fairly self explainatory, add and an endpoint to the HTTP API that can be used to disable/enable the telemtry.

Endpoint to disable telemtry
fairly self explainatory, add and an endpoint to the HTTP API that can be used to disable/enable the telemtry.



Discussion:
interested in taking this!
@George3d6 quick question-- what exactly is the telemetry you're looking to display on the endpoint? i tried searching through the repo but there isn't really anything that exists with that name yet that I can call in the function. however i've implemented the endpoint as a skeleton as below:
![image](https://user-images.githubusercontent.com/33303128/102041584-d81ef780-3d9d-11eb-8922-8c4591abc4aa.png)
![image](https://user-images.githubusercontent.com/33303128/102041596-dfde9c00-3d9d-11eb-8ce2-8a9f008fd58d.png)


interested in taking this!
@George3d6 quick question-- what exactly is the telemetry you're looking to display on the endpoint? i tried searching through the repo but there isn't really anything that exists with that name yet that I can call in the function. however i've implemented the endpoint as a skeleton as below:
![image](https://user-images.githubusercontent.com/33303128/102041584-d81ef780-3d9d-11eb-8922-8c4591abc4aa.png)
![image](https://user-images.githubusercontent.com/33303128/102041596-dfde9c00-3d9d-11eb-8ce2-8a9f008fd58d.png)



Resolving patch:
diff --git a/mindsdb/api/http/namespaces/util.py b/mindsdb/api/http/namespaces/util.py
--- a/mindsdb/api/http/namespaces/util.py
+++ b/mindsdb/api/http/namespaces/util.py
@@ -1,3 +1,4 @@
+import os
 from flask import request
 from flask_restx import Resource, abort
 from flask import current_app as ca
@@ -5,6 +6,8 @@
 from mindsdb.api.http.namespaces.configs.util import ns_conf
 from mindsdb import __about__
 
+TELEMETRY_FILE = 'telemetry.lock'
+
 @ns_conf.route('/ping')
 class Ping(Resource):
     @ns_conf.doc('get_ping')
@@ -21,3 +24,34 @@ def get(self):
         return {
             'report_uuid': predictor.report_uuid
         }
+
+@ns_conf.route('/telemetry')
+class Telemetry(Resource):
+    @ns_conf.doc('get_telemetry_status')
+    def get(self):
+        status = "enabled" if is_telemetry_active() else "disabled"
+        return {"status": status}
+
+    @ns_conf.doc('set_telemetry')
+    def post(self):
+        data = request.json
+        action = data['action']
+        if str(action).lower() in ["true", "enable", "on"]:
+            enable_telemetry()
+        else:
+            disable_telemetry()
+
+
+def enable_telemetry():
+    path = os.path.join(ca.config_obj['storage_dir'], TELEMETRY_FILE)
+    if os.path.exists(path):
+        os.remove(path)
+
+def disable_telemetry():
+    path = os.path.join(ca.config_obj['storage_dir'], TELEMETRY_FILE)
+    with open(path, 'w') as _:
+        pass
+
+def is_telemetry_active():
+    path = os.path.join(ca.config_obj['storage_dir'], TELEMETRY_FILE)
+    return not os.path.exists(path)


Test patch:
Repository: mindsdb/mindsdb

Issue:
Error if try create datasource from not existing integration
If send query `PUT api/datasources/ds_name` with data:
```
{
    "integration_id": 'unexists_integration',
    "name": 'ds_name',
    "query": f"select * from test_data.any_data limit 100;"
}
```
then in response will be error:
```
{
   "message": 'TypeError: expected str, bytes or os.PathLike object, not dict'
}
```


Discussion:
I wouldn't call this a bug, yes, we need better logs for these type of things, but in principle this issue shouldn't happen to being with if people are using this via Scout or the database.

If they are using it via the API directly, I assume they are "expert" users. Still, something we want better logging for, but not a bug.

Resolving patch:
diff --git a/mindsdb/api/http/namespaces/datasource.py b/mindsdb/api/http/namespaces/datasource.py
--- a/mindsdb/api/http/namespaces/datasource.py
+++ b/mindsdb/api/http/namespaces/datasource.py
@@ -129,6 +129,10 @@ def on_file(file):
 
         if 'query' in data:
             source_type = request.json['integration_id']
+            if source_type not in ca.default_store.config['integrations']:
+                # integration doens't exist
+                abort(400, f"{source_type} integration doesn't exist")
+
             ca.default_store.save_datasource(name, source_type, request.json)
             os.rmdir(temp_dir_path)
             return ca.default_store.get_datasource(name)


Test patch:
diff --git a/tests/integration_tests/api/test_http.py b/tests/integration_tests/api/test_http.py
--- a/tests/integration_tests/api/test_http.py
+++ b/tests/integration_tests/api/test_http.py
@@ -1,15 +1,16 @@
 import os
+import time
+import unittest
+import importlib.util
 from random import randint
 from pathlib import Path
-import unittest
-import requests
-import time
+from uuid import uuid1
 
+import requests
 import psutil
 
 from mindsdb.utilities.config import Config
 
-import importlib.util
 common_path = Path(__file__).parent.parent.absolute().joinpath('flows/common.py').resolve()
 spec = importlib.util.spec_from_file_location("common", str(common_path))
 common = importlib.util.module_from_spec(spec)
@@ -235,13 +236,17 @@ def test_9_gui_is_served(self):
         assert response.status_code == 200
         assert response.content.decode().find('<head>') > 0
 
-    def test_10_telemetry_enabled(self):
+    def test_10_ds_from_unexist_integration(self):
         """
         Call telemetry enabled
         then check the response is status 200
         """
-        response = requests.get(f'{root}/config/telemetry/true')
-        assert response.status_code == 200
+        ds_name = f"ds_{uuid1()}"
+        data = {"integration_id": f'unexists_integration_{uuid1()}',
+                "name": ds_name,
+                "query": "select * from test_data.any_data limit 100;"}
+        response = requests.put(f'{root}/datasources/{ds_name}', json=data)
+        assert response.status_code == 400, f"expected 400 but got {response.status_code}, {response.text}"
 
 if __name__ == '__main__':
     unittest.main(failfast=True)
Repository: mindsdb/mindsdb

Issue:
Close learn process gracefuly
`close_api_gracefully` only closes the API, but not any ongoing learn processes, which are left orphaned and consuming loads of memory.


Discussion:

Resolving patch:
diff --git a/mindsdb/__main__.py b/mindsdb/__main__.py
--- a/mindsdb/__main__.py
+++ b/mindsdb/__main__.py
@@ -5,7 +5,7 @@
 import time
 import asyncio
 import datetime
-import platform
+import signal
 
 import torch.multiprocessing as mp
 
@@ -16,7 +16,7 @@
 from mindsdb.api.http.start import start as start_http
 from mindsdb.api.mysql.start import start as start_mysql
 from mindsdb.api.mongo.start import start as start_mongo
-from mindsdb.utilities.ps import is_pid_listen_port
+from mindsdb.utilities.ps import is_pid_listen_port, get_child_pids
 from mindsdb.interfaces.database.database import DatabaseWrapper
 from mindsdb.utilities.functions import args_parse, get_all_models_meta_data
 from mindsdb.utilities.log import log
@@ -26,6 +26,9 @@ def close_api_gracefully(apis):
     try:
         for api in apis.values():
             process = api['process']
+            childs = get_child_pids(process.pid)
+            for p in childs:
+                os.kill(p, signal.SIGTERM)
             sys.stdout.flush()
             process.terminate()
             process.join()
diff --git a/mindsdb/interfaces/native/learn_process.py b/mindsdb/interfaces/native/learn_process.py
--- a/mindsdb/interfaces/native/learn_process.py
+++ b/mindsdb/interfaces/native/learn_process.py
@@ -26,8 +26,8 @@ def run(self):
         '''
         import mindsdb_native
 
-        fs_store = FsSotre()
         config = Config()
+        fs_store = FsSotre()
         company_id = os.environ.get('MINDSDB_COMPANY_ID', None)
         name, from_data, to_predict, kwargs, datasource_id = self._args
 
diff --git a/mindsdb/utilities/ps.py b/mindsdb/utilities/ps.py
--- a/mindsdb/utilities/ps.py
+++ b/mindsdb/utilities/ps.py
@@ -4,6 +4,11 @@
 import psutil
 
 
+def get_child_pids(pid):
+    p = psutil.Process(pid=pid)
+    return p.children(recursive=True)
+
+
 def net_connections():
     """Cross-platform psutil.net_connections like interface"""
     if sys.platform.lower().startswith('linux'):


Test patch:
Repository: mindsdb/mindsdb

Issue:
Flask serializer that can use python datetimes/date/timestamp types
We need to introduce a json serializer for flask that can handle python datetime/date/timestamp type.

Note: Figure out commonly used timestamp/datetime types other than the "standard" ones from the `datetime` library that might be yielded from our database clients and make it into the dataframe. 

See example in db.py for how to do this, but maybe there's a easier flask specific way of doing this.

Format should be a string that's `yyyy-MM-dd'T'HH:mm:ss.SSS`, so from years to miliseconds with no timezeone/offset.


Discussion:

Resolving patch:
diff --git a/mindsdb/api/http/initialize.py b/mindsdb/api/http/initialize.py
--- a/mindsdb/api/http/initialize.py
+++ b/mindsdb/api/http/initialize.py
@@ -7,10 +7,13 @@
 from zipfile import ZipFile
 from pathlib import Path
 import traceback
+from datetime import datetime, date, timedelta
 #import concurrent.futures
 
-from flask import Flask, url_for
+from flask import Flask, url_for, make_response
+from flask.json import dumps
 from flask_restx import Api
+from flask.json import JSONEncoder
 
 from mindsdb.__about__ import __version__ as mindsdb_version
 from mindsdb.interfaces.datastore.datastore import DataStore
@@ -34,6 +37,24 @@ def specs_url(self):
         return url_for(self.endpoint("specs"), _external=False)
 
 
+class CustomJSONEncoder(JSONEncoder):
+    def default(self, obj):
+        if isinstance(obj, date):
+            return obj.strftime("%Y-%m-%d")
+        if isinstance(obj, datetime):
+            return obj.strftime("%Y-%m-%dT%H:%M:%S.%f")
+        if isinstance(obj, timedelta):
+            return str(obj)
+
+        return JSONEncoder.default(self, obj)
+
+
+def custom_output_json(data, code, headers=None):
+    resp = make_response(dumps(data), code)
+    resp.headers.extend(headers or {})
+    return resp
+
+
 def initialize_static(config):
     ''' Update Scout files basing on compatible-config.json content.
         Files will be downloaded and updated if new version of GUI > current.
@@ -214,11 +235,13 @@ def initialize_flask(config, init_static_thread, no_studio):
 
     app.config['SEND_FILE_MAX_AGE_DEFAULT'] = 60
     app.config['SWAGGER_HOST'] = 'http://localhost:8000/mindsdb'
+    app.json_encoder = CustomJSONEncoder
+
     authorizations = {
         'apikey': {
-            'type': 'apiKey',
+            'type': 'session',
             'in': 'query',
-            'name': 'apikey'
+            'name': 'session'
         }
     }
 
@@ -231,6 +254,8 @@ def initialize_flask(config, init_static_thread, no_studio):
         doc='/doc/'
     )
 
+    api.representations['application/json'] = custom_output_json
+
     port = config['api']['http']['port']
     host = config['api']['http']['host']
 


Test patch:
Repository: mindsdb/mindsdb

Issue:
Can not connect to mysql api by last mariadb shell.
Last mariadb client (v.10.5.9) can not connect to mindsdb with default auth plugin. With last mysql client all ok.
To reproduce:
```
docker pull mariadb:10.5.9
docker run -it --network host --rm mariadb:10.5.9 mysql -h 127.0.0.1 --port 47335 -uuser -ppassword --default-auth=caching_sha2_password
```
with `--default-auth=mysql_native_password` mariadb client can connect to mindsdb.


Discussion:

Resolving patch:
diff --git a/mindsdb/api/mysql/mysql_proxy/mysql_proxy.py b/mindsdb/api/mysql/mysql_proxy/mysql_proxy.py
--- a/mindsdb/api/mysql/mysql_proxy/mysql_proxy.py
+++ b/mindsdb/api/mysql/mysql_proxy/mysql_proxy.py
@@ -274,7 +274,10 @@ def get_fast_auth_password():
             if password == b'\x00':
                 password = ''
             else:
-                password = get_fast_auth_password()
+                if self.session.is_ssl:
+                    password = get_fast_auth_password()
+                else:
+                    password = switch_auth()
         elif 'mysql_native_password' in client_auth_plugin:
             log.debug(f'Check auth, user={username}, ssl={self.session.is_ssl}, auth_method={client_auth_plugin}: '
                       'check auth using mysql_native_password')


Test patch:
Repository: mindsdb/mindsdb

Issue:
error in parsing  mysql insert
Error will cause if try to parse such request:
```
insert into predictors(name, predict, select_data_query) ...
```
reason is missed space before `(`. That valid in sql, so we should support it.


Discussion:

Resolving patch:
diff --git a/mindsdb/api/mysql/mysql_proxy/classes/sql_statement_parser.py b/mindsdb/api/mysql/mysql_proxy/classes/sql_statement_parser.py
--- a/mindsdb/api/mysql/mysql_proxy/classes/sql_statement_parser.py
+++ b/mindsdb/api/mysql/mysql_proxy/classes/sql_statement_parser.py
@@ -445,7 +445,7 @@ def parse_as_insert(self) -> dict:
         from_value = (
             QuotedString('`')
             | originalTextFor(
-                Word(printables, excludeChars='.`')
+                Word(printables, excludeChars='().`')
             )
         )
 
@@ -597,7 +597,7 @@ def test():
                 ' START transaction',
                 {'keyword': 'start'}
             ], [
-                "insert into a.b (col1, col2) values ('val1', 'val2');",
+                "insert into a.b(col1, col2) values ('val1', 'val2');",
                 {
                     'keyword': 'insert',
                     'struct': {


Test patch:
Repository: mindsdb/mindsdb

Issue:
Reject predict(POST) API call if JSON payload is not properly wrapped by WHEN 
**Is your feature request related to a problem? Please describe.**
The MindsDB Predict API returns predictions even when the JSON payload is not properly following the specification.
(https://apidocs.mindsdb.com/#acaf5684-c1bb-4df7-bae0-3a673ac1dd11) . 
For instance, the payload should follow this rule but the API also accepts payload without 'when' 
```
--data-raw '{
	"when": {
		"number_of_rooms": 2,
		"sqft": 1700
	}
}'
```
```
--data-raw '{
		"number_of_rooms": 2,
		"sqft": 1700
}'
```
In this case, the API cannot recognize input variables in the JSON request, thus return them as 'missing' in the response text. I am not sure if the prediction itself is not reliable in this case, but we might as well block such wrong API requests. 

**Describe the solution you'd like**
Return 'error' message if 'predict when' specification is not met 




Discussion:
Thanks for reporting @shawn-domuso, we are already working on a fix.

Resolving patch:
diff --git a/mindsdb/api/http/namespaces/predictor.py b/mindsdb/api/http/namespaces/predictor.py
--- a/mindsdb/api/http/namespaces/predictor.py
+++ b/mindsdb/api/http/namespaces/predictor.py
@@ -182,12 +182,12 @@ class PredictorPredict2(Resource):
     def post(self, name):
         '''Queries predictor'''
         data = request.json
-        when = data.get('when', {})
+        when = data.get('when')
         format_flag = data.get('format_flag', 'explain')
         kwargs = data.get('kwargs', {})
 
-        if when is None:
-            return 'No data provided for the predictions', 500
+        if isinstance(when, dict) is False or len(when) == 0:
+            return 'No data provided for the predictions', 400
 
         results = request.native_interface.predict(name, format_flag, when_data=when, **kwargs)
 


Test patch: