internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mindsdb/mindsdb

Issue:
Error when csv header has empty space
### Discussed in https://github.com/mindsdb/mindsdb/discussions/2343

<div type='discussions-op-text'>

<sup>Originally posted by **Ashok0** June 13, 2022</sup>
Hi,

I have a question regarding usage of imported data with MindsDB.  I registered a MindsDB account and selected "Add Data" > "Import File" and uploaded some sample CSV data to a table.   I can view the data just fine as shown in the following screencap:  https://imgur.com/a/X6mrzHt

If I attempt to select an individual column i.e. 'sensitivity' with the following statement:  **SELECT sensitivity FROM files.bedcheck**, 
I receive the following error message:  **Binder Error: Referenced column "sensitivity" not found in FROM clause! Candidate bindings: "df_table. sensitivity" LINE 1: SELECT sensitivity AS sensitivity**

Any assistance in how I can resolve this error would be greatly appreciated.  

Thank you!</div>


Discussion:
Looking at the error message `Candidate bindings: "df_table. rval", "df_table.peak", "df_table.p_id", "df_table.diff", "df_table.sent"` it looks like somehow we didn't parse the empty space in the header so sensitivity can't be found. For now, can you please just exclude the empty space and I will post an issue so we can strip() headers.



Resolving patch:
diff --git a/mindsdb/integrations/handlers/file_handler/file_handler.py b/mindsdb/integrations/handlers/file_handler/file_handler.py
--- a/mindsdb/integrations/handlers/file_handler/file_handler.py
+++ b/mindsdb/integrations/handlers/file_handler/file_handler.py
@@ -128,6 +128,7 @@ def _handle_source(file_path, clean_rows=True, custom_parser=None):
         else:
             file_list_data = file_data
 
+        header = [x.strip() for x in header]
         col_map = dict((col, col) for col in header)
         return pd.DataFrame(file_list_data, columns=header), col_map
 
@@ -287,7 +288,7 @@ def get_columns(self, table_name) -> Response:
             RESPONSE_TYPE.TABLE,
             data_frame=pd.DataFrame([
                 {
-                    'Field': x,
+                    'Field': x.strip(),
                     'Type': 'str'
                 } for x in file_meta['columns']
             ])


Test patch:
Repository: mindsdb/mindsdb

Issue:
[Bug]: `select abc` does not raise error
`select abc` returns 'abc' as text, but should raise error instead


Discussion:

Resolving patch:
diff --git a/mindsdb/api/mysql/mysql_proxy/executor/executor_commands.py b/mindsdb/api/mysql/mysql_proxy/executor/executor_commands.py
--- a/mindsdb/api/mysql/mysql_proxy/executor/executor_commands.py
+++ b/mindsdb/api/mysql/mysql_proxy/executor/executor_commands.py
@@ -915,8 +915,8 @@ def answer_single_row_select(self, statement):
                 column_alias = target.alias or column_name
                 result = SERVER_VARIABLES.get(column_name)
                 if result is None:
-                    log.warning(f'Unknown variable: {column_name}')
-                    result = ''
+                    log.error(f'Unknown variable: {column_name}')
+                    raise Exception(f"Unknown variable '{var_name}'")
                 else:
                     result = result[0]
             elif target_type == Function:
@@ -945,8 +945,7 @@ def answer_single_row_select(self, statement):
                 column_alias = 'NULL'
             elif target_type == Identifier:
                 result = '.'.join(target.parts)
-                column_name = str(result)
-                column_alias = '.'.join(target.alias.parts) if type(target.alias) == Identifier else column_name
+                raise Exception(f"Unknown column '{result}'")
             else:
                 raise Exception(f'Unknown constant type: {target_type}')
 


Test patch:
Repository: mindsdb/mindsdb

Issue:
Need to store rows and columns count for training dataset
That need to collect statistic


Discussion:

Resolving patch:
diff --git a/mindsdb/interfaces/model/model_controller.py b/mindsdb/interfaces/model/model_controller.py
--- a/mindsdb/interfaces/model/model_controller.py
+++ b/mindsdb/interfaces/model/model_controller.py
@@ -240,6 +240,8 @@ def learn(self, name: str, training_data: DataFrame, to_predict: str,
             to_predict=problem_definition.target,
             learn_args=problem_definition.to_dict(),
             data={'name': name},
+            training_data_columns_count=len(training_data.columns),
+            training_data_rows_count=len(training_data)
         )
 
         db.session.add(predictor_record)
diff --git a/mindsdb/interfaces/storage/db.py b/mindsdb/interfaces/storage/db.py
--- a/mindsdb/interfaces/storage/db.py
+++ b/mindsdb/interfaces/storage/db.py
@@ -90,6 +90,8 @@ class Predictor(Base):
     is_custom = Column(Boolean)
     learn_args = Column(Json)
     update_status = Column(String, default='up_to_date')
+    training_data_columns_count = Column(Integer)
+    training_data_rows_count = Column(Integer)
 
     json_ai = Column(Json, nullable=True)
     code = Column(String, nullable=True)
diff --git a/mindsdb/migrations/versions/2022-07-15_b5b53e0ea7f8_training_data_rows_columns_count.py b/mindsdb/migrations/versions/2022-07-15_b5b53e0ea7f8_training_data_rows_columns_count.py
new file mode 100644
--- /dev/null
+++ b/mindsdb/migrations/versions/2022-07-15_b5b53e0ea7f8_training_data_rows_columns_count.py
@@ -0,0 +1,29 @@
+"""training_data_rows_columns_count
+
+Revision ID: b5b53e0ea7f8
+Revises: 999bceb904df
+Create Date: 2022-07-15 12:21:40.523039
+
+"""
+from alembic import op
+import sqlalchemy as sa
+import mindsdb.interfaces.storage.db
+
+
+# revision identifiers, used by Alembic.
+revision = 'b5b53e0ea7f8'
+down_revision = '999bceb904df'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    with op.batch_alter_table('predictor', schema=None) as batch_op:
+        batch_op.add_column(sa.Column('training_data_columns_count', sa.Integer(), nullable=True))
+        batch_op.add_column(sa.Column('training_data_rows_count', sa.Integer(), nullable=True))
+
+
+def downgrade():
+    with op.batch_alter_table('predictor', schema=None) as batch_op:
+        batch_op.drop_column('training_data_rows_count')
+        batch_op.drop_column('training_data_columns_count')


Test patch:
Repository: mindsdb/mindsdb

Issue:
[BUG] Fix scylladb error when connecting with secure bundle
When connecting with `secure_connect_bundle` users got unknown secure_connect_bundle path error.


Discussion:

Resolving patch:
diff --git a/mindsdb/integrations/handlers/scylla_handler/scylla_handler.py b/mindsdb/integrations/handlers/scylla_handler/scylla_handler.py
--- a/mindsdb/integrations/handlers/scylla_handler/scylla_handler.py
+++ b/mindsdb/integrations/handlers/scylla_handler/scylla_handler.py
@@ -41,24 +41,21 @@ def connect(self):
         connection_props = {
             'auth_provider': auth_provider
         }
-
-        if self.connection_args['protocol_version'] is not None:
-            connection_props['protocol_version'] = self.connection_args['protocol_version']
- 
+        connection_props['protocol_version'] = self.connection_args.get('protocol_version', 4)
         secure_connect_bundle = self.connection_args.get('secure_connect_bundle')
 
         if secure_connect_bundle is not None:
-            if os.path.isfile(self.secure_connect_bundle) is False:
+            if os.path.isfile(secure_connect_bundle) is False:
                 raise Exception("Secure_connect_bundle' must be path to the file")
             connection_props['cloud'] = {
-                'secure_connect_bundle': self.secure_connect_bundle
+                'secure_connect_bundle': secure_connect_bundle
             }
         else:
             connection_props['contact_points'] = [self.connection_args['host']]
             connection_props['port'] = int(self.connection_args['port'])
 
         cluster = Cluster(**connection_props)
-        session = cluster.connect(self.connection_args['keyspace'])
+        session = cluster.connect(self.connection_args.get('keyspace'))
 
         self.is_connected = True
         self.session = session


Test patch:
Repository: mindsdb/mindsdb

Issue:
Endpoint to return handler's icons
At the moment we return icons for handlers by general `GET /handlers` route. Icons are return in svg or base64, which is not effective. We need new endpoint to return handler icon:
`GET /handlers/{name}/icon/{icon_file_name}`



Discussion:

Resolving patch:
diff --git a/mindsdb/api/http/namespaces/handlers.py b/mindsdb/api/http/namespaces/handlers.py
--- a/mindsdb/api/http/namespaces/handlers.py
+++ b/mindsdb/api/http/namespaces/handlers.py
@@ -1,4 +1,8 @@
-from flask import request
+import os
+import importlib
+from pathlib import Path
+
+from flask import request, send_file, abort
 from flask_restx import Resource
 
 from mindsdb.api.http.utils import http_error
@@ -20,6 +24,24 @@ def get(self):
         return result
 
 
+@ns_conf.route('/<handler_name>/icon')
+class HandlerIcon(Resource):
+    @ns_conf.param('handler_name', 'Handler name')
+    def get(self, handler_name):
+        try:
+            handlers_import_status = request.integration_controller.get_handlers_import_status()
+            icon_name = handlers_import_status[handler_name]['icon']['name']
+            handler_folder = handlers_import_status[handler_name]['import']['folder']
+            mindsdb_path = Path(importlib.util.find_spec('mindsdb').origin).parent
+            icon_path = mindsdb_path.joinpath('integrations/handlers').joinpath(handler_folder).joinpath(icon_name)
+            if icon_path.is_absolute() is False:
+                icon_path = Path(os.getcwd()).joinpath(icon_path)
+        except Exception:
+            return abort(404)
+        else:
+            return send_file(icon_path)
+
+
 @ns_conf.route('/<handler_name>/install')
 class InstallDependencies(Resource):
     @ns_conf.param('handler_name', 'Handler name')


Test patch:
Repository: mindsdb/mindsdb

Issue:
[CUSTOMER REQUEST]: Show class predicted probabilities as columns for classification problems
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Is your feature request related to a problem? Please describe.

https://www.loom.com/share/19d256063ac944e3b129a3ed381d1c42

### Describe the solution you'd like.

A column for each class in the tabular output of the predictor.

Possible column naming convention: probability_[class_name]

### Describe an alternate solution.

_No response_

### Anything else? (Additional Context)

Acceptance criteria: quick loom video showing outputs of a classification predictor

Demo to use: https://docs.google.com/document/d/1PWjsDzbTsKRODgGxLff0o0DEyFatdyNf-CPIOaGyxbU/edit


Discussion:
SELECT .. .FROM PREDICTOR ... USING ... params = {}? 

== SELECT prediction, prediction.data. , prediction.columnimportce FORM PREDICTOR predictor. 

== SELECT prediction, prediction.data. , prediction.columnimportce FORM PREDICTOR predictor. WHERE predictor.bounds = X 

Resolving patch:
diff --git a/mindsdb/integrations/handlers/lightwood_handler/lightwood_handler/lightwood_handler.py b/mindsdb/integrations/handlers/lightwood_handler/lightwood_handler/lightwood_handler.py
--- a/mindsdb/integrations/handlers/lightwood_handler/lightwood_handler/lightwood_handler.py
+++ b/mindsdb/integrations/handlers/lightwood_handler/lightwood_handler/lightwood_handler.py
@@ -429,14 +429,17 @@ def predict(self, model_name: str, data: list, pred_format: str = 'dict') -> pd.
         explain_arr = []
         pred_dicts = []
         for i, row in enumerate(predictions):
-            obj = {
-                target: {
-                    'predicted_value': row['prediction'],
-                    'confidence': row.get('confidence', None),
-                    'anomaly': row.get('anomaly', None),
-                    'truth': row.get('truth', None)
-                }
+            values = {
+                'predicted_value': row['prediction'],
+                'confidence': row.get('confidence', None),
+                'anomaly': row.get('anomaly', None),
+                'truth': row.get('truth', None)
             }
+            if predictor.supports_proba:
+                for cls in predictor.statistical_analysis.train_observed_classes:
+                    if row.get(f'__mdb_proba_{cls}', False):
+                        values[f'probability_class_{cls}'] = round(row[f'__mdb_proba_{cls}'], 4)
+            obj = { target: values }
             if 'lower' in row:
                 obj[target]['confidence_lower_bound'] = row.get('lower', None)
                 obj[target]['confidence_upper_bound'] = row.get('upper', None)


Test patch:
Repository: mindsdb/mindsdb

Issue:
Move predictors 'tables' to Lightwood 'db'
Need to move predictors tables from 'mindsdb' to 'ligthwood' database


Discussion:

Resolving patch:
diff --git a/mindsdb/__main__.py b/mindsdb/__main__.py
--- a/mindsdb/__main__.py
+++ b/mindsdb/__main__.py
@@ -112,7 +112,7 @@ def close_api_gracefully(apis):
 
     if not is_cloud:
         # region creating permanent integrations
-        for integration_name in ['files', 'views']:
+        for integration_name in ['files', 'views', 'lightwood']:
             integration_meta = integration_controller.get(name=integration_name)
             if integration_meta is None:
                 integration_record = db.Integration(
diff --git a/mindsdb/api/mysql/mysql_proxy/executor/executor_commands.py b/mindsdb/api/mysql/mysql_proxy/executor/executor_commands.py
--- a/mindsdb/api/mysql/mysql_proxy/executor/executor_commands.py
+++ b/mindsdb/api/mysql/mysql_proxy/executor/executor_commands.py
@@ -735,9 +735,17 @@ def answer_drop_view(self, statement):
         return ExecuteAnswer(answer_type=ANSWER_TYPE.OK)
 
     def answer_create_predictor(self, statement):
-        lw_handler = self.session.integration_controller.get_handler('lightwood')
+        ml_integration_name = self.session.database
+        if len(statement.name.parts) == 2:
+            ml_integration_name = statement.name.parts[0]
+            statement.name.parts = [statement.name.parts[-1]]
+        ml_integration_name = ml_integration_name.lower()
+        if ml_integration_name == 'mindsdb':
+            ml_integration_name = 'lightwood'
 
-        result = lw_handler.query(statement)
+        ml_handler = self.session.integration_controller.get_handler(ml_integration_name)
+
+        result = ml_handler.query(statement)
         if result.type == RESPONSE_TYPE.ERROR:
             raise Exception(result.error_message)
 


Test patch:
Repository: mindsdb/mindsdb

Issue:
[BUG] Error if upload file with not valid symbol
Here is example of such file: https://www.kaggle.com/datasets/fedesoriano/covid19-effect-on-liver-cancer-prediction-dataset?select=covid-liver.csv
Column `Type_of_incidental_finding` contains not valid symbols. 


Discussion:

Resolving patch:
diff --git a/mindsdb/integrations/handlers/file_handler/file_handler.py b/mindsdb/integrations/handlers/file_handler/file_handler.py
--- a/mindsdb/integrations/handlers/file_handler/file_handler.py
+++ b/mindsdb/integrations/handlers/file_handler/file_handler.py
@@ -203,11 +203,13 @@ def _get_data_io(file_path):
                     explain=False
                 )
                 best_meta = file_encoding_meta.best()
+                errors = 'strict'
                 if best_meta is not None:
                     encoding = file_encoding_meta.best().encoding
                 else:
                     encoding = 'utf-8'
-                data = StringIO(byte_str.decode(encoding))
+                    errors = 'replace'
+                data = StringIO(byte_str.decode(encoding, errors))
         except Exception:
             print(traceback.format_exc())
             print('Could not load into string')


Test patch:
Repository: pallets/click

Issue:
Automatically dedent help text of options?
```
import click

@click.command()
@click.option('--foo', help="""
    heyho

    i am

    multiline
""")
def cli(foo):
    click.echo(foo)

cli()
```

This currently does not remove the leading whitespace from each paragraph in the help text for `--foo`:

```
untitaker@untibox:~/projects/click, branch master
$ python lol.py --help
Usage: lol.py [OPTIONS]

Options:
  --foo TEXT
                  heyho

                  i am

                  multiline
  --help      Show this message and exit.
```

Although i am not sure if help texts for options are even supposed to get that
complex.



Discussion:
From what i can see in `write_dl`, this seems to be a bug.


Resolving patch:
diff --git a/click/decorators.py b/click/decorators.py
--- a/click/decorators.py
+++ b/click/decorators.py
@@ -150,6 +150,8 @@ def option(*param_decls, **attrs):
     :attr:`Command.params` list.
     """
     def decorator(f):
+        if 'help' in attrs:
+            attrs['help'] = inspect.cleandoc(attrs['help'])
         _param_memo(f, Option(param_decls, **attrs))
         return f
     return decorator


Test patch:
diff --git a/tests/test_options.py b/tests/test_options.py
--- a/tests/test_options.py
+++ b/tests/test_options.py
@@ -169,3 +169,23 @@ def cmd(foo):
     assert result.exit_code == 2
     assert 'Error: Missing option "--foo".  Choose from foo, bar.' \
         in result.output
+
+
+def test_multiline_help(runner):
+    @click.command()
+    @click.option('--foo', help="""
+        hello
+
+        i am
+
+        multiline
+    """)
+    def cmd(foo):
+        click.echo(foo)
+
+    result = runner.invoke(cmd, ['--help'])
+    assert result.exit_code == 0
+    out = result.output.splitlines()
+    assert '  --foo TEXT  hello' in out
+    assert '              i am' in out
+    assert '              multiline' in out
Repository: pallets/click

Issue:
Make click.echo()'s 'nl=True' thread-safe



Discussion:

Resolving patch:
diff --git a/click/utils.py b/click/utils.py
--- a/click/utils.py
+++ b/click/utils.py
@@ -275,6 +275,13 @@ def echo(message=None, file=None, nl=True, err=False, color=None):
     if message is not None and not isinstance(message, echo_native_types):
         message = text_type(message)
 
+    if nl:
+        message = message or u''
+        if isinstance(message, text_type):
+            message += u'\n'
+        else:
+            message += b'\n'
+
     # If there is a message, and we're in Python 3, and the value looks
     # like bytes, we manually need to find the binary stream and write the
     # message in there.  This is done separately so that most stream
@@ -285,8 +292,6 @@ def echo(message=None, file=None, nl=True, err=False, color=None):
         if binary_file is not None:
             file.flush()
             binary_file.write(message)
-            if nl:
-                binary_file.write(b'\n')
             binary_file.flush()
             return
 
@@ -307,8 +312,6 @@ def echo(message=None, file=None, nl=True, err=False, color=None):
 
     if message:
         file.write(message)
-    if nl:
-        file.write(u'\n')
     file.flush()
 
 


Test patch:
Repository: pallets/click

Issue:
BadOptionUsage exception no longer takes option_name argument
During [this change](https://github.com/mitsuhiko/click/commit/5f98f570c08a11e607cccfd9c1b787c69a6a67b7#diff-f391f677ae94d58e508e7af268fdb718R168), `BadOptionUsage`'s `__init__` signature was changed to remove the `option_name` parameter. This seems to be an undocumented breaking change that makes the class less useful programmatically (it removes programmatic access to the attribute that's being used incorrectly). If this was an intentional change, it should be documented in the changelog; otherwise, it's a bug and should be fixed. I'm happy to submit a pull request with the correct change. Can a maintainer clarify what the correct behavior should be?



Discussion:
That's probably not ideal. Willing to accept a pull request here.


Resolving patch:
diff --git a/click/exceptions.py b/click/exceptions.py
--- a/click/exceptions.py
+++ b/click/exceptions.py
@@ -164,10 +164,13 @@ class BadOptionUsage(UsageError):
     for an option is not correct.
 
     .. versionadded:: 4.0
+
+    :param option_name: the name of the option being used incorrectly.
     """
 
-    def __init__(self, message, ctx=None):
+    def __init__(self, option_name, message, ctx=None):
         UsageError.__init__(self, message, ctx)
+        self.option_name = option_name
 
 
 class BadArgumentUsage(UsageError):
diff --git a/click/parser.py b/click/parser.py
--- a/click/parser.py
+++ b/click/parser.py
@@ -74,8 +74,8 @@ def _fetch(c):
 
 def _error_opt_args(nargs, opt):
     if nargs == 1:
-        raise BadOptionUsage('%s option requires an argument' % opt)
-    raise BadOptionUsage('%s option requires %d arguments' % (opt, nargs))
+        raise BadOptionUsage(opt, '%s option requires an argument' % opt)
+    raise BadOptionUsage(opt, '%s option requires %d arguments' % (opt, nargs))
 
 
 def split_opt(opt):
@@ -342,7 +342,7 @@ def _match_long_opt(self, opt, explicit_value, state):
                 del state.rargs[:nargs]
 
         elif explicit_value is not None:
-            raise BadOptionUsage('%s option does not take a value' % opt)
+            raise BadOptionUsage(opt, '%s option does not take a value' % opt)
 
         else:
             value = None


Test patch:
Repository: pallets/click

Issue:
Argument.make_metavar ignores ParamType.get_metavar ; is not configurable to fix
If you do the following, you get bad behavior:

``` python
class MyType(click.ParamType):
    def get_metavar(self, param):
        return "MY_TYPE"

@click.command("foo")
@click.help_option()
@click.argument("param", type=MyType())
def foo_command(param):
    pass
```

`foo --help` produces a metavar of `PARAM` instead of the more logically correct `MY_TYPE`.

This is because `Argument.make_metavar` never looks at `self.type.get_metavar(self)`.

Either this should be fixed so that `Argument.make_metavar` uses the type metavar by default, or `Argument.make_metavar` should take a parameter that lets you toggle between the default "name" metavar and the type's default metavar.



Discussion:

Resolving patch:
diff --git a/click/core.py b/click/core.py
--- a/click/core.py
+++ b/click/core.py
@@ -1719,7 +1719,9 @@ def human_readable_name(self):
     def make_metavar(self):
         if self.metavar is not None:
             return self.metavar
-        var = self.name.upper()
+        var = self.type.get_metavar(self)
+        if not var:
+            var = self.name.upper()
         if not self.required:
             var = '[%s]' % var
         if self.nargs != 1:


Test patch:
Repository: pallets/click

Issue:
Dynamic bash autocompletion should escape blanks
Thanks for #755, wonderful feature! I found an issue when my completion callback returns entries that have (legitimate) blanks in them. In this case, entries get split into separate arguments, whereas I would have expected that blanks are escaped with a backslash (as bash does by default).


Discussion:

Resolving patch:
diff --git a/click/_bashcomplete.py b/click/_bashcomplete.py
--- a/click/_bashcomplete.py
+++ b/click/_bashcomplete.py
@@ -12,6 +12,7 @@
 
 COMPLETION_SCRIPT = '''
 %(complete_func)s() {
+    local IFS=$'\n'
     COMPREPLY=( $( env COMP_WORDS="${COMP_WORDS[*]}" \\
                    COMP_CWORD=$COMP_CWORD \\
                    %(autocomplete_var)s=complete $1 ) )


Test patch: