Repository: modin-project/modin

Issue:
Incorrect syntax error on read_sql
I am getting the following error in case of msssql.

Connection query details:

```
params1 = urllib.parse.quote_plus("driver={ODBC Driver 17 for SQL Server};server=server;database=Kepler;Uid=uid;Pwd=password;Encrypt=yes;")
engine2 = 'mssql+pyodbc:///?odbc_connect={}'.format(params1)
```
Error:
```
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near ')'. (102) (SQLExecDirectW)")
[SQL: SELECT COUNT(*) FROM (select * from cashflow_data as foo)]
```

Minimal code is as follows:

```
import modin.experimental.pandas as pd
query = f'''select * from cashflow_data'''
file_data = pd.read_sql(sql = query, con=engine2,max_sessions = 10)
y = datetime.datetime.now() - x
```

Dataset example:

instu_id | cashflow_amount | currency_code | cf_type | data_date | import_source | entity | maturity_date
-- | -- | -- | -- | -- | -- | -- | --
6.03675E+11 | 7500 | KWD | P | 30-Sep-19 | MANUAL | RFOL | 1-May-20
6.03468E+11 | 635.143 | KWD | P | 30-Sep-19 | MANUAL | RFOL | 30-Sep-19
6.03474E+11 | 20000 | KWD | P | 30-Sep-19 | MANUAL | RFOL | 31-Oct-19
6.03E+11 | 5149 | KWD | P | 30-Sep-19 | MANUAL | RFOL | 18-Feb-20
6.03511E+11 | 16635 | KWD | P | 30-Sep-19 | MANUAL | RFOL | 30-Oct-19



@devin-petersohn : Any way to fix it



Discussion:
Hi @Harshitg, thanks for posting! This looks like something we're going to have to fix internally.

It will take me some time to set up a SQLServer environment to test this locally. I have a suspicion that the way we are aliasing our subqueries for postgres does not translate, so it's going to take some time to find a way to detect the type of database and fix the issue. I will make sure it is out for the next release. 
SQL SERVER requires aliasing the inner query. For instance, the following should work
```
SELECT COUNT(*) FROM (select * from cashflow_data) as foo
```
I believe this was the intent from the start, but the alias snuck into the inner query.
or am I missing something bigger here?
@eavidan I believe you are correct. I wanted to set up a testing environment to verify this. Postgres also has this requirement, so we need to make sure that the fix doesn't break the postgres functionality.
Thinking about it, Microsoft SQL does not support LIMIT and OFFSET as well. 
So this means that in order to support Microsoft SQL we would probably need to generalize the way we construct our queries better.
@devin-petersohn After diving into SQL Alchemy and pandas implementation, I believe we would need to created different query templates for different SQL engines. we can add this to `sql_reader.py`. 
I am not sure there is a simple way to implement the distribution of the query in a unified way across all SQL engines.
does this make sense? any other directions you had in mind?
Thanks @eavidan that makes sense, I think that's the best approach. As a part of this, we will need to add the different SQL engines to the CI. There's going to be a little setup required to do that.
Since this is a slightly larger rewrite, I'm pushing to the next release, which should be in 3-4 weeks, possibly sooner.
@devin-petersohn sounds reasonable.
I have tried working with SQL Alchemy to create a query generator that would wrap the given `sql` query, while being agnostic to the SQL engine. Unfortunately, I was not able to produce a complete solution.
Going back to the last approach, do we want to simply check the connection string or engine dialect and choose a query template accordingly. something like this?
```python
row_cnt_template = "SELECT COUNT(*) FROM ({}) as foo"
cols_names_template = "SELECT * FROM ({0}) as foo LIMIT 0"
partition_template = "SELECT * FROM ({0}) as foo LIMIT {1} OFFSET {2}"

engine = sa.create_engine(con)
if engine.driver == "pymssql":
    partition_template = "SELECT * FROM ({0}) as foo ORDER BY(SELECT NULL) OFFSET {2} ROWS FETCH NEXT {1} ROWS ONLY"
    cols_names_template = "SELECT TOP 0 FROM ({0}) as foo"
```
If so, would a generator for the templates be more suited and should we start only with a `Base` template and a `MSSQL` templates?
Yes, that should be good. Ideally this code block would go in a `util` of some sort outside of the implementation so it's easy to find/add new templates if necessary.

Ideally, as a part of this effort we can add better testing for the various SQL systems. I have set up a Windows test in the CI so we can add mssql to the testing environment there.

Resolving patch:
diff --git a/modin/config/envvars.py b/modin/config/envvars.py
--- a/modin/config/envvars.py
+++ b/modin/config/envvars.py
@@ -446,6 +446,13 @@ def get(cls):
         return min_partition_size
 
 
+class TestReadFromSqlServer(EnvironmentVariable, type=bool):
+    """Set to true to test reading from SQL server."""
+
+    varname = "MODIN_TEST_READ_FROM_SQL_SERVER"
+    default = False
+
+
 def _check_vars():
     """
     Check validity of environment variables.
diff --git a/modin/core/io/sql/sql_dispatcher.py b/modin/core/io/sql/sql_dispatcher.py
--- a/modin/core/io/sql/sql_dispatcher.py
+++ b/modin/core/io/sql/sql_dispatcher.py
@@ -53,8 +53,9 @@ def _read(cls, sql, con, index_col=None, **kwargs):
         BaseQueryCompiler
             Query compiler with imported data for further processing.
         """
-        is_modin_db_connection = isinstance(con, ModinDatabaseConnection)
-        if not (is_modin_db_connection or isinstance(con, str)):
+        if isinstance(con, str):
+            con = ModinDatabaseConnection("sqlalchemy", con)
+        if not isinstance(con, ModinDatabaseConnection):
             warnings.warn(
                 "To use parallel implementation of `read_sql`, pass either "
                 + "the SQL connection string or a ModinDatabaseConnection "
@@ -63,13 +64,12 @@ def _read(cls, sql, con, index_col=None, **kwargs):
                 + "https://modin.readthedocs.io/en/latest/supported_apis/io_supported.html#connecting-to-a-database-for-read-sql"
             )
             return cls.single_worker_read(sql, con=con, index_col=index_col, **kwargs)
-        row_cnt_query = "SELECT COUNT(*) FROM ({}) as foo".format(sql)
-        connection_for_pandas = con.get_connection() if is_modin_db_connection else con
-        row_cnt = pandas.read_sql(row_cnt_query, connection_for_pandas).squeeze()
+        row_count_query = con.row_count_query(sql)
+        connection_for_pandas = con.get_connection()
+        colum_names_query = con.column_names_query(sql)
+        row_cnt = pandas.read_sql(row_count_query, connection_for_pandas).squeeze()
         cols_names_df = pandas.read_sql(
-            "SELECT * FROM ({}) as foo LIMIT 0".format(sql),
-            connection_for_pandas,
-            index_col=index_col,
+            colum_names_query, connection_for_pandas, index_col=index_col
         )
         cols_names = cols_names_df.columns
         num_partitions = NPartitions.get()
@@ -79,9 +79,7 @@ def _read(cls, sql, con, index_col=None, **kwargs):
         limit = math.ceil(row_cnt / num_partitions)
         for part in range(num_partitions):
             offset = part * limit
-            query = "SELECT * FROM ({}) as foo LIMIT {} OFFSET {}".format(
-                sql, limit, offset
-            )
+            query = con.partition_query(sql, limit, offset)
             partition_id = cls.deploy(
                 cls.parse,
                 num_partitions + 2,
diff --git a/modin/db_conn.py b/modin/db_conn.py
--- a/modin/db_conn.py
+++ b/modin/db_conn.py
@@ -28,6 +28,8 @@
 
 
 class UnsupportedDatabaseException(Exception):
+    """Modin can't create a particular kind of database connection."""
+
     pass
 
 
@@ -52,6 +54,30 @@ def __init__(self, lib, *args, **kwargs):
         self.lib = lib
         self.args = args
         self.kwargs = kwargs
+        self._dialect_is_microsoft_sql_cache = None
+
+    def _dialect_is_microsoft_sql(self):
+        """
+        Tell whether this connection requires Microsoft SQL dialect.
+
+        If this is a sqlalchemy connection, create an engine from args and
+        kwargs. If that engine's driver is pymssql or pyodbc, this
+        connection requires Microsoft SQL. Otherwise, it doesn't.
+
+        Returns
+        -------
+        Boolean
+        """
+        if self._dialect_is_microsoft_sql_cache is None:
+            self._dialect_is_microsoft_sql_cache = False
+            if self.lib == _SQLALCHEMY_LIB_NAME:
+                from sqlalchemy import create_engine
+
+                self._dialect_is_microsoft_sql_cache = create_engine(
+                    *self.args, **self.kwargs
+                ).driver in ("pymssql", "pyodbc")
+
+        return self._dialect_is_microsoft_sql_cache
 
     def get_connection(self):
         """
@@ -76,4 +102,62 @@ def get_connection(self):
 
             return create_engine(*self.args, **self.kwargs).connect()
 
-        raise Exception("Unsupported database library")
+        raise UnsupportedDatabaseException("Unsupported database library")
+
+    def column_names_query(self, query):
+        """
+        Get a query that gives the names of columns that `query` would produce.
+
+        Parameters
+        ----------
+        query : str
+            The SQL query to check.
+
+        Returns
+        -------
+        str
+        """
+        # This query looks odd, but it works in both PostgreSQL and Microsoft
+        # SQL, which doesn't let you use a "limit" clause to select 0 rows.
+        return f"SELECT * FROM ({query}) AS _ WHERE 1 = 0"
+
+    def row_count_query(self, query):
+        """
+        Get a query that gives the names of rows that `query` would produce.
+
+        Parameters
+        ----------
+        query : str
+            The SQL query to check.
+
+        Returns
+        -------
+        str
+        """
+        return f"SELECT COUNT(*) FROM ({query}) AS _"
+
+    def partition_query(self, query, limit, offset):
+        """
+        Get a query that partitions the original `query`.
+
+        Parameters
+        ----------
+        query : str
+            The SQL query to get a partition.
+        limit : int
+            The size of the partition.
+        offset : int
+            Where the partition begins.
+
+        Returns
+        -------
+        str
+        """
+        return (
+            (
+                f"SELECT * FROM ({query}) AS _ ORDER BY(SELECT NULL)"
+                + f" OFFSET {offset} ROWS FETCH NEXT {limit} ROWS ONLY"
+            )
+            if self._dialect_is_microsoft_sql()
+            else f"SELECT * FROM ({query}) LIMIT {limit} OFFSET {offset}"
+        )


Test patch:
diff --git a/modin/pandas/test/test_io.py b/modin/pandas/test/test_io.py
--- a/modin/pandas/test/test_io.py
+++ b/modin/pandas/test/test_io.py
@@ -19,8 +19,17 @@
 from pandas.core.dtypes.common import is_list_like
 from pathlib import Path
 from collections import OrderedDict
-from modin.db_conn import ModinDatabaseConnection, UnsupportedDatabaseException
-from modin.config import TestDatasetSize, Engine, StorageFormat, IsExperimental
+from modin.db_conn import (
+    ModinDatabaseConnection,
+    UnsupportedDatabaseException,
+)
+from modin.config import (
+    TestDatasetSize,
+    Engine,
+    StorageFormat,
+    IsExperimental,
+    TestReadFromSqlServer,
+)
 from modin.utils import to_pandas
 from modin.pandas.utils import from_arrow
 from modin.test.test_utils import warns_that_defaulting_to_pandas
@@ -1834,6 +1843,32 @@ def test_read_sql(self, make_sql_connection):
         pandas_df = pandas.read_sql(sql=query, con=sqlalchemy_connection)
         df_equals(modin_df, pandas_df)
 
+    @pytest.mark.skipif(
+        not TestReadFromSqlServer.get(),
+        reason="Skip the test when the test SQL server is not set up.",
+    )
+    def test_read_sql_from_sql_server(self):
+        table_name = "test_1000x256"
+        query = f"SELECT * FROM {table_name}"
+        sqlalchemy_connection_string = (
+            "mssql+pymssql://sa:Strong.Pwd-123@0.0.0.0:1433/master"
+        )
+        pandas_df_to_read = pandas.DataFrame(
+            np.arange(
+                1000 * 256,
+            ).reshape(1000, 256)
+        ).add_prefix("col")
+        pandas_df_to_read.to_sql(
+            table_name, sqlalchemy_connection_string, if_exists="replace"
+        )
+        modin_df = pd.read_sql(
+            query,
+            ModinDatabaseConnection("sqlalchemy", sqlalchemy_connection_string),
+        )
+        pandas_df = pandas.read_sql(query, sqlalchemy_connection_string)
+        df_equals(modin_df, pandas_df)
+
+    def test_invalid_modin_database_connections(self):
         with pytest.raises(UnsupportedDatabaseException):
             ModinDatabaseConnection("unsupported_database")