internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mindsdb/mindsdb

Issue:
Mess in integrations setup
Currently we have two issues with integrations setup:
1. 'setup' is call for all integrations, regardless 'publish' flag.
2. if integration exists in db and in config file with same name, then only integration from db will be setup. That bad, because any integration from config file will be copied to db right after mindsdb start, and any changes in config file after that will have no effect.


Discussion:
@StpMax What do you suggest to fix this mess?
How that must be:
1. setup must call only once at mindsdb start
2. setup must call only for integrations with `publish: true`
3. setup should not be called if mindsdb work in cloud
What about this note:
```hat bad, because any integration from config file will be copied to db right after mindsdb start ```
Do you think that we shouldn't allow to have (or merge) integrations in config if integrations with the same names already exist in database?

Resolving patch:
diff --git a/mindsdb/__main__.py b/mindsdb/__main__.py
--- a/mindsdb/__main__.py
+++ b/mindsdb/__main__.py
@@ -82,17 +82,20 @@ def close_api_gracefully(apis):
     if not is_cloud:
         for integration_name in get_db_integrations(COMPANY_ID, sensitive_info=True):
             print(f"Setting up integration: {integration_name}")
-            dbw.setup_integration(integration_name)
+            if get_db_integration(integration_name, COMPANY_ID)['publish']:
+                # do setup and register only if it is 'publish' integration
+                dbw.setup_integration(integration_name)
+                dbw.register_predictors(model_data_arr, integration_name=integration_name)
 
     for integration_name in config.get('integrations', {}):
         print(f'Adding: {integration_name}')
         try:
             it = get_db_integration(integration_name, None)
-            if it is None:
+            if it is None:      # register and setup it only if it doesn't conflict with records in db
                 add_db_integration(integration_name, config['integrations'][integration_name], None)            # Setup for user `None`, since we don't need this for cloud
-            if config['integrations'][integration_name].get('publish', False) and not is_cloud:
-                dbw.setup_integration(integration_name)
-                dbw.register_predictors(model_data_arr, integration_name=integration_name)
+                if config['integrations'][integration_name].get('publish', False) and not is_cloud:
+                    dbw.setup_integration(integration_name)
+                    dbw.register_predictors(model_data_arr, integration_name=integration_name)
         except Exception as e:
             log.error(f'\n\nError: {e} adding database integration {integration_name}\n\n')
 


Test patch:
Repository: mindsdb/mindsdb

Issue:
Can`t upload big files with waitress
* Mindsdb version you tried to install: latest
* Additional info if applicable: waitress==2.0.0

Waitress consumes too much memory when uploading a file. 16Gb of memory is not enough to upload 1.1Gb file 


Discussion:

Resolving patch:
diff --git a/mindsdb/api/http/namespaces/datasource.py b/mindsdb/api/http/namespaces/datasource.py
--- a/mindsdb/api/http/namespaces/datasource.py
+++ b/mindsdb/api/http/namespaces/datasource.py
@@ -3,6 +3,8 @@
 import tempfile
 import re
 import multipart
+import zipfile
+import tarfile
 
 from dateutil.parser import parse
 from flask import request, send_file
@@ -10,6 +12,7 @@
 from flask import current_app as ca
 
 from mindsdb.utilities.log import log
+from mindsdb.api.http.utils import http_error
 from mindsdb.api.http.namespaces.configs.datasources import ns_conf
 from mindsdb.api.http.namespaces.entitites.datasources.datasource import (
     datasource_metadata,
@@ -147,6 +150,24 @@ def on_file(file):
 
         if source_type == 'file':
             file_path = os.path.join(temp_dir_path, data['file'])
+            lp = file_path.lower()
+            if lp.endswith(('.zip', '.tar.gz')):
+                if lp.endswith('.zip'):
+                    with zipfile.ZipFile(file_path) as f:
+                        f.extractall(temp_dir_path)
+                elif lp.endswith('.tar.gz'):
+                    with tarfile.open(file_path) as f:
+                        f.extractall(temp_dir_path)
+                os.remove(file_path)
+                files = os.listdir(temp_dir_path)
+                if len(files) != 1:
+                    os.rmdir(temp_dir_path)
+                    return http_error(400, 'Wrong content.', 'Archive must contain only one data file.')
+                file_path = os.path.join(temp_dir_path, files[0])
+                source = files[0]
+                if not os.path.isfile(file_path):
+                    os.rmdir(temp_dir_path)
+                    return http_error(400, 'Wrong content.', 'Archive must contain data file in root.')
         else:
             file_path = None
 
@@ -174,7 +195,6 @@ def get(self, name):
         if analysis is not None:
             return analysis, 200
 
-
         ds = request.default_store.get_datasource(name)
         if ds is None:
             log.error('No valid datasource given')


Test patch:
Repository: mindsdb/mindsdb

Issue:
Update MongoDB connector to sample data when dataset is too large.
Too large: undefined, maybe 100,000 rows?
Use $sample https://docs.mongodb.com/manual/reference/operator/aggregation/sample/





Discussion:

Resolving patch:
diff --git a/mindsdb/interfaces/datastore/datastore.py b/mindsdb/interfaces/datastore/datastore.py
--- a/mindsdb/interfaces/datastore/datastore.py
+++ b/mindsdb/interfaces/datastore/datastore.py
@@ -163,7 +163,6 @@ def save_datasource(self, name, source_type, source, file_path=None, company_id=
             )
             session.add(datasource_record)
             session.commit()
-            datasource_record = session.query(Datasource).filter_by(company_id=company_id, name=name).first()
 
             ds_meta_dir = os.path.join(self.dir, f'{company_id}@@@@@{name}')
             os.mkdir(ds_meta_dir)
@@ -327,9 +326,20 @@ def save_datasource(self, name, source_type, source, file_path=None, company_id=
                     'kwargs': {}
                 }
 
-            df = ds.df
+            if hasattr(ds, 'get_columns') and hasattr(ds, 'get_row_count'):
+                try:
+                    column_names = ds.get_columns()
+                    row_count = ds.get_row_count()
+                except Exception:
+                    df = ds.df
+                    column_names = list(df.keys())
+                    row_count = len(df)
+            else:
+                df = ds.df
+                column_names = list(df.keys())
+                row_count = len(df)
 
-            if '' in df.columns or len(df.columns) != len(set(df.columns)):
+            if '' in column_names or len(column_names) != len(set(column_names)):
                 shutil.rmtree(ds_meta_dir)
                 raise Exception('Each column in datasource must have unique non-empty name')
 
@@ -337,8 +347,8 @@ def save_datasource(self, name, source_type, source, file_path=None, company_id=
             datasource_record.data = json.dumps({
                 'source_type': source_type,
                 'source': source,
-                'row_count': len(df),
-                'columns': [dict(name=x) for x in list(df.keys())]
+                'row_count': row_count,
+                'columns': [dict(name=x) for x in column_names]
             })
 
             self.fs_store.put(f'{company_id}@@@@@{name}', f'datasource_{company_id}_{datasource_record.id}', self.dir)


Test patch:
Repository: mindsdb/mindsdb

Issue:
Create predictor error
I have a few datasources connected as PostgreSQL, Singlestore, Mongo. If I try to train a new model from PostgreSQL as:
```
CREATE PREDICTOR insurance1 FROM TESTPSQL (SELECT * FROM data.insurance) as ds_insurance_data PREDICT charges as charges_target;
```
where TESTPSQL is the datasource name of PostgreSQL, MindsDB returns bellow error related to singlestore:

```
SQL Error [1149] [42000]: 2005 (HY000): Unknown MySQL server host 'svc-eccc27ae-d275-4168-aabb-585cbd648dcf-ddl.aws-oregon-1.svc.singlestore.com' (0)
```


Discussion:

Resolving patch:
diff --git a/mindsdb/api/mysql/mysql_proxy/mysql_proxy.py b/mindsdb/api/mysql/mysql_proxy/mysql_proxy.py
--- a/mindsdb/api/mysql/mysql_proxy/mysql_proxy.py
+++ b/mindsdb/api/mysql/mysql_proxy/mysql_proxy.py
@@ -531,7 +531,7 @@ def answer_create_predictor(self, struct):
         company_id = self.session.company_id
 
         if get_db_integration(struct['integration_name'], company_id) is None:
-            struct['integration_name'] = list(get_db_integrations(company_id).keys())[0]
+            raise Exception(f"Unknown integration: {struct['integration_name']}")
 
         is_temp_ds = False
         ds_name = struct.get('datasource_name')
diff --git a/mindsdb/interfaces/database/integrations.py b/mindsdb/interfaces/database/integrations.py
--- a/mindsdb/interfaces/database/integrations.py
+++ b/mindsdb/interfaces/database/integrations.py
@@ -1,8 +1,10 @@
-from copy import deepcopy
 import os
 import shutil
-from pathlib import Path
 import tempfile
+from pathlib import Path
+from copy import deepcopy
+
+from sqlalchemy import func
 
 from mindsdb.interfaces.storage.db import session
 from mindsdb.interfaces.storage.db import Integration
@@ -156,8 +158,14 @@ def _get_integration_record_data(integration_record, sensitive_info=True):
     return data
 
 
-def get_db_integration(name, company_id, sensitive_info=True):
-    integration_record = session.query(Integration).filter_by(company_id=company_id, name=name).first()
+def get_db_integration(name, company_id, sensitive_info=True, case_sensitive=False):
+    if case_sensitive:
+        integration_record = session.query(Integration).filter_by(company_id=company_id, name=name).first()
+    else:
+        integration_record = session.query(Integration).filter(
+            (Integration.company_id == company_id)
+            & (func.lower(Integration.name) == func.lower(name))
+        ).first()
     return _get_integration_record_data(integration_record, sensitive_info)
 
 


Test patch:
Repository: mindsdb/mindsdb

Issue:
Error if remove ds which is being analyse
That happen only locally, issue related to sqlalchemy 'features'.


Discussion:

Resolving patch:
diff --git a/mindsdb/interfaces/datastore/datastore.py b/mindsdb/interfaces/datastore/datastore.py
--- a/mindsdb/interfaces/datastore/datastore.py
+++ b/mindsdb/interfaces/datastore/datastore.py
@@ -123,6 +123,9 @@ def delete_datasource(self, name, company_id=None):
             linked_models = Predictor.query.filter_by(company_id=company_id, datasource_id=datasource_record.id).all()
             if linked_models:
                 raise Exception("Can't delete {} datasource because there are next models linked to it: {}".format(name, [model.name for model in linked_models]))
+        session.query(Semaphor).filter_by(
+            company_id=company_id, entity_id=datasource_record.id, entity_type='datasource'
+        ).delete()
         session.delete(datasource_record)
         session.commit()
         self.fs_store.delete(f'datasource_{company_id}_{datasource_record.id}')


Test patch:
Repository: mindsdb/mindsdb

Issue:
Columns_to_ignore is ignored
need add converting for `columns_to_ignore` to acceptable for new lightwood name


Discussion:

Resolving patch:
diff --git a/mindsdb/interfaces/model/model_controller.py b/mindsdb/interfaces/model/model_controller.py
--- a/mindsdb/interfaces/model/model_controller.py
+++ b/mindsdb/interfaces/model/model_controller.py
@@ -92,6 +92,9 @@ def _unpack_old_args(self, from_data: dict, kwargs: dict, to_predict: Optional[U
         if 'stop_training_in_x_seconds' in kwargs:
             problem_definition['time_aim'] = kwargs['stop_training_in_x_seconds']
 
+        if 'ignore_columns' in kwargs:
+            problem_definition['ignore_features'] = kwargs['ignore_columns']
+
         ds_cls = getattr(mindsdb_datasources, from_data['class'])
         ds = ds_cls(*from_data['args'], **from_data['kwargs'])
         df = ds.df


Test patch:
Repository: mindsdb/mindsdb

Issue:
Ignore column doesn't work for `USING` keyword
Example query from home rentals dataset:

```sql
CREATE PREDICTOR home_rentals_model FROM postgres (SELECT * from test_data.home_rentals)
 as ds_name PREDICT rental_price USING {"ignore_columns": "sqft"};
```
The ignored column will still be used for the model training and visible in model column importance.


Discussion:

Resolving patch:
diff --git a/mindsdb/interfaces/model/model_controller.py b/mindsdb/interfaces/model/model_controller.py
--- a/mindsdb/interfaces/model/model_controller.py
+++ b/mindsdb/interfaces/model/model_controller.py
@@ -97,6 +97,8 @@ def _unpack_old_args(self, from_data: dict, kwargs: dict, to_predict: Optional[U
 
         if 'ignore_columns' in kwargs:
             problem_definition['ignore_features'] = kwargs['ignore_columns']
+            if isinstance(problem_definition['ignore_features'], list) is False:
+                problem_definition['ignore_features'] = [problem_definition['ignore_features']]
 
         ds_cls = getattr(mindsdb_datasources, from_data['class'])
         ds = ds_cls(*from_data['args'], **from_data['kwargs'])
@@ -145,7 +147,6 @@ def learn(self, name: str, from_data: dict, to_predict: str, datasource_id: int,
             data['status'] = 'complete'
         else:
             data['status'] = 'error'
-        print(f'!!!!===== {name} learn finished status={ data["status"]}')
 
     @mark_process(name='predict')
     def predict(self, name: str, when_data: Union[dict, list, pd.DataFrame], pred_format: str, company_id: int):
@@ -172,7 +173,6 @@ def predict(self, name: str, when_data: Union[dict, list, pd.DataFrame], pred_fo
                     'pickle': str(os.path.join(self.config['paths']['predictors'], fs_name))
                 }
             else:
-                print(f'===== {name} predict={ predictor_record.data is None}')
                 raise Exception(f'Trying to predict using predictor {original_name} with status: {predictor_data["status"]}')
 
         if isinstance(when_data, dict) and 'kwargs' in when_data and 'args' in when_data:


Test patch:
Repository: mindsdb/mindsdb

Issue:
Create multiple predictors
Currently, users can create multiple predictors with the same name using the `CREATE Predictor` statement. After that none of them will be valid.


Discussion:

Resolving patch:
diff --git a/mindsdb/interfaces/model/model_controller.py b/mindsdb/interfaces/model/model_controller.py
--- a/mindsdb/interfaces/model/model_controller.py
+++ b/mindsdb/interfaces/model/model_controller.py
@@ -11,7 +11,6 @@
 import lightwood
 from lightwood.api.types import ProblemDefinition
 from lightwood import __version__ as lightwood_version
-from packaging import version
 import numpy as np
 import pandas as pd
 import mindsdb_datasources
@@ -108,6 +107,10 @@ def _unpack_old_args(self, from_data: dict, kwargs: dict, to_predict: Optional[U
 
     @mark_process(name='learn')
     def learn(self, name: str, from_data: dict, to_predict: str, datasource_id: int, kwargs: dict, company_id: int) -> None:
+        predictor_record = db.session.query(db.Predictor).filter_by(company_id=company_id, name=name).first()
+        if predictor_record is not None:
+            raise Exception('Predictor name must be unique.')
+
         df, problem_definition, join_learn_process = self._unpack_old_args(from_data, kwargs, to_predict)
 
         problem_definition = ProblemDefinition.from_dict(problem_definition)
@@ -321,7 +324,12 @@ def update_model(self, name: str, company_id: int):
         return 'Updated in progress'
 
     @mark_process(name='learn')
-    def generate_predictor(self, name: str, from_data: dict, datasource_id, problem_definition_dict: dict, join_learn_process: bool, company_id: int):
+    def generate_predictor(self, name: str, from_data: dict, datasource_id, problem_definition_dict: dict,
+                           join_learn_process: bool, company_id: int):
+        predictor_record = db.session.query(db.Predictor).filter_by(company_id=company_id, name=name).first()
+        if predictor_record is not None:
+            raise Exception('Predictor name must be unique.')
+
         df, problem_definition, _ = self._unpack_old_args(from_data, problem_definition_dict)
 
         problem_definition = ProblemDefinition.from_dict(problem_definition)
@@ -389,6 +397,7 @@ def fit_predictor(self, name: str, from_data: dict, join_learn_process: bool, co
             if not IS_PY36:
                 p.close()
 
+
 '''
 Notes: Remove ray from actors are getting stuck
 try:


Test patch:
Repository: mindsdb/mindsdb

Issue:
Can't connect to MindsDB from MySQL Workbench
MySQL Workbench(8) returns the bellow error when connecting to MindsDB

![244619631_5020372154645537_5393427741065503105_n](https://user-images.githubusercontent.com/7192539/136210031-a869b84e-21f3-44ea-8450-1fd38a88f527.png)
Note that all the parameters are correct


Discussion:

Resolving patch:
diff --git a/mindsdb/api/mysql/mysql_proxy/mysql_proxy.py b/mindsdb/api/mysql/mysql_proxy/mysql_proxy.py
--- a/mindsdb/api/mysql/mysql_proxy/mysql_proxy.py
+++ b/mindsdb/api/mysql/mysql_proxy/mysql_proxy.py
@@ -1119,6 +1119,13 @@ def queryAnswer(self, sql):
                 table = sql[sql.rfind('.') + 1:].strip(' .;\n\t').replace('`', '')
                 self.answer_show_create_table(table)
                 return
+            elif 'show character set where charset =' in sql_lower:
+                # show character set where charset = 'utf8mb4';
+                charset = sql_lower.replace('show character set where charset = ', '').strip("'")
+                self.answer_show_charset(charset)
+                return
+
+
 
         if keyword == 'start':
             # start transaction
@@ -1276,6 +1283,37 @@ def queryAnswer(self, sql):
             print(sql)
             raise NotImplementedError('Action not implemented')
 
+    def answer_show_charset(self, charset):
+        # TODO answer based on 'charset' arg
+        packages = []
+        packages += self.getTabelPackets(
+            columns=[{
+                'table_name': '',
+                'name': 'Charset',
+                'type': TYPES.MYSQL_TYPE_VAR_STRING
+            }, {
+                'table_name': '',
+                'name': 'Description',
+                'type': TYPES.MYSQL_TYPE_VAR_STRING
+            }, {
+                'table_name': '',
+                'name': 'Default collation',
+                'type': TYPES.MYSQL_TYPE_VAR_STRING
+            }, {
+                'table_name': '',
+                'name': 'Default Maxlen',
+                'type': TYPES.MYSQL_TYPE_LONG,
+                'charset': CHARSET_NUMBERS['binary']
+            }],
+            data=[['utf8mb4', 'UTF-8 Unicode', 'utf8mb4_general_ci', 4]]
+        )
+        if self.client_capabilities.DEPRECATE_EOF is True:
+            packages.append(self.packet(OkPacket, eof=True))
+        else:
+            packages.append(self.packet(EofPacket))
+        self.sendPackageGroup(packages)
+        return
+
     def answer_show_create_table(self, table):
         packages = []
         packages += self.getTabelPackets(


Test patch:
Repository: mindsdb/mindsdb

Issue:
Add new method to count number of rows for PostgreSQL datasources :electric_plug: :1234: 
When MindsDB creates a new PostgreSQL datasource we get information for row counts by fetching all datasources. The problem here is that if datasource is big it takes a lot of time. We need a new get_row_count method to return the number of rows per datasource. The PR should include this method inside the PostgreSQL class .

## Steps :male_detective: :female_detective: 

- Implement in https://github.com/mindsdb/mindsdb/blob/stable/mindsdb/integrations/postgres/postgres.py#L37
- Example method:
```py
def get_row_count(self, query):
    result = conn.execute(query)
    return len(query)
```
- Push to staging branch

## Additional rewards :1st_place_medal: 

Each code PR brings  :three:  point for entry into the draw for a :computer:  Deep Learning Laptop powered by the NVIDIA RTX 3080 Max-Q GPU or other swag :shirt: :bear: .  For more info check out https://mindsdb.com/hacktoberfest/
 




Discussion:
I can work on this

Sure, if you have any questions feel free to post them. Note that we are not assigning issues so just make the PR on staging.
ok sure
@ZoranPandovski Raised a PR for this. Can you please check?
@ZoranPandovski I was wondering that how will adding a `get_row_count` method in PostgreSQL would help in finding the number of rows per datasource. In the existing logic, the `row_count` for each datasource is calculated using `DataSource.get_row_count()` (https://github.com/mindsdb/mindsdb/blob/227f56902a60b97c0f149cce85974562f34e93e9/mindsdb/interfaces/datastore/datastore.py#L337) and that function is defined in `mindsdb/datasources` repo (https://github.com/mindsdb/datasources/blob/staging/mindsdb_datasources/datasources/data_source.py#L287-L290). So, in order to make the changes work, won't we need to submit a PR on the `mindsdb/datasources` repo?

Resolving patch:
diff --git a/mindsdb/integrations/postgres/postgres.py b/mindsdb/integrations/postgres/postgres.py
--- a/mindsdb/integrations/postgres/postgres.py
+++ b/mindsdb/integrations/postgres/postgres.py
@@ -192,3 +192,10 @@ def unregister_predictor(self, name):
             DROP FOREIGN TABLE IF EXISTS {self.mindsdb_database}.{self._escape_table_name(name)};
         """
         self._query(q)
+
+    def get_row_count(self, query):
+        q = f""" 
+            SELECT COUNT(*) as count
+            FROM ({query}) as query;"""
+        result = self._query(q)
+        return result[0]['count']


Test patch:
Repository: mindsdb/mindsdb

Issue:
Add new method to count number of rows for MySQL datasources :electric_plug: :1234: 
When MindsDB creates a new MySQL datasource we get information for row counts by fetching all datasources. The problem here is that if datasource is big it takes a lot of time. We need a new get_row_count method to return the number of rows per datasource. The PR should include this method inside the PostgreSQL class .

## Steps :male_detective: :female_detective: 

- Implement in https://github.com/mindsdb/mindsdb/blob/stable/mindsdb/integrations/mysql/mysql.py#L51
- Example method:
```py
def get_row_count(self, query):
    result = conn.execute(query)
    return len(query)
```
- Push to staging branch

## Additional rewards :1st_place_medal: 

Each code PR brings  :three:  point for entry into the draw for a :computer:  Deep Learning Laptop powered by the NVIDIA RTX 3080 Max-Q GPU or other swag :shirt: :bear: .  For more info check out https://mindsdb.com/hacktoberfest/
 




Discussion:
Hi @ZoranPandovski,

I've raised https://github.com/mindsdb/mindsdb/pull/1533 to close this issue. I decided to add it to the base class instead as it is really needed by all implementing classes. Let me know what you think.
Hey @ZoranPandovski , I made a PR #1562 to resolve this issue. Please have a look at it.



Resolving patch:
diff --git a/mindsdb/integrations/mysql/mysql.py b/mindsdb/integrations/mysql/mysql.py
--- a/mindsdb/integrations/mysql/mysql.py
+++ b/mindsdb/integrations/mysql/mysql.py
@@ -190,3 +190,10 @@ def unregister_predictor(self, name):
             drop table if exists {self.mindsdb_database}.{self._escape_table_name(name)};
         """
         self._query(q)
+
+    def get_row_count(self, query):
+        q = f""" 
+            SELECT COUNT(*) as count
+            FROM ({query}) as query;"""
+        result = self._query(q)
+        return result[0]['count']


Test patch: