Repository: dbcli/mycli (Python)

Issue:
PyMySQL version 0.10 produces ImportError crash on mycli execution
After updating PyMySQL from 0.9.3 to 0.10.0 the following crash occurs when executing mycli (version 1.12.1):

```
Traceback (most recent call last):
  File "/usr/bin/mycli", line 33, in <module>
    sys.exit(load_entry_point('mycli==1.21.1', 'console_scripts', 'mycli')())
  File "/usr/bin/mycli", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/<user>/.local/lib/python3.8/site-packages/mycli/main.py", line 41, in <module>
    from .sqlexecute import FIELD_TYPES, SQLExecute
  File "/home/<user>/.local/lib/python3.8/site-packages/mycli/sqlexecute.py", line 6, in <module>
    from pymysql.converters import (convert_mysql_timestamp, convert_datetime,
ImportError: cannot import name 'convert_mysql_timestamp' from 'pymysql.converters' (/home/<user>/.local/lib/python3.8/site-packages/pymysql/converters.py)
```

I downgraded to PyMySQL 0.9.3 to continue using mycli
cannot import name 'convert_mysql_timestamp' from 'pymysql.converters'
while typing `mycli` command in the terminal, it displays a lot errers:
```
PS C:\Users\leon> mycli
Traceback (most recent call last):
  File "D:\Python\Python38-32\Scripts\mycli-script.py", line 11, in <module>
    load_entry_point('mycli==1.21.1', 'console_scripts', 'mycli')()
  File "d:\python\python38-32\lib\site-packages\pkg_resources\__init__.py", line 490, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "d:\python\python38-32\lib\site-packages\pkg_resources\__init__.py", line 2862, in load_entry_point
    return ep.load()
  File "d:\python\python38-32\lib\site-packages\pkg_resources\__init__.py", line 2462, in load
    return self.resolve()
  File "d:\python\python38-32\lib\site-packages\pkg_resources\__init__.py", line 2468, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "d:\python\python38-32\lib\site-packages\mycli-1.21.1-py3.8.egg\mycli\main.py", line 41, in <module>
    from .sqlexecute import FIELD_TYPES, SQLExecute
  File "d:\python\python38-32\lib\site-packages\mycli-1.21.1-py3.8.egg\mycli\sqlexecute.py", line 6, in <module>
    from pymysql.converters import (convert_mysql_timestamp, convert_datetime,
ImportError: cannot import name 'convert_mysql_timestamp' from 'pymysql.converters' (d:\python\python38-32\lib\site-packages\pymysql\converters.py)
```

### enviroment:
win10 pro
```
PS C:\Users\leon> python -V
Python 3.8.4
```
```
mysql> status
--------------
C:\Program Files\MySQL\MySQL Server 8.0\bin\mysql.exe  Ver 8.0.21 for Win64 on x86_64 (MySQL Community Server - GPL)

Connection id:          8
Current database:
Current user:           root@localhost
SSL:                    Cipher in use is TLS_AES_256_GCM_SHA384
Using delimiter:        ;
Server version:         8.0.21 MySQL Community Server - GPL
Protocol version:       10
Connection:             localhost via TCP/IP
Server characterset:    utf8mb4
Db     characterset:    utf8mb4
Client characterset:    gbk
Conn.  characterset:    gbk
TCP port:               3306
Binary data as:         Hexadecimal
Uptime:                 4 hours 34 min 42 sec

Threads: 2  Questions: 5  Slow queries: 0  Opens: 115  Flush tables: 3  Open tables: 36  Queries per second avg: 0.000
--------------

mysql>
```

### what I have tried
I've use pip command to uninstall and reinstall `mycli`, but theres' still the same error
not sure what to do now...






Discussion:
Here's the commit that removed the convert_mysql_timestamp function:
https://github.com/PyMySQL/PyMySQL/commit/fe0cd60e6f9e3bc0fb81f76f7e4fa30a1e1b34cc#diff-40496cc4a0f4db6907de60f7ce470b95
solution 1)  Install PyMySQL 0.9.2 (this is last version that you can use with mycli)

or

solution 2)  
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/mycli/sqlexecute.py

**Open the above file and modify it as follows:**

line 6) from pymysql.converters import (convert_datetime, 
line 102) FIELD_TYPE.TIMESTAMP: lambda obj: (convert_datetime(obj) or obj),

If you want to use the last version of mycli, you can use mycli with docker https://hub.docker.com/repository/docker/dbcliorg/mycli
I was new to doing this, so in case it helps anyone else, these are the commands I used to get it working based on @lunatiee 's solution ➊

```bash
$ pip3 install mycli
$ pip3 install -I PyMySQL==0.9.2
```


We have a sprint this weekend and we'll see if we can tackle this issue during the sprint. 

If anyone is interested in participating, we're doing a virtual sprint through the PyOhio conference. https://www.pyohio.org/2020/events/sprints




> solution 1) Install PyMySQL 0.9.2 (this is last version that you can use with mycli)
> 
> or
> 
> solution 2)
> /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/mycli/sqlexecute.py
> 
> **Open the above file and modify it as follows:**
> 
> line 6) from pymysql.converters import (convert_datetime,
> line 102) FIELD_TYPE.TIMESTAMP: lambda obj: (convert_datetime(obj) or obj),

This is an effective solution. Thank you!
It's fixed in the latest version. Please update.

pip install -U mycli

On Sun, Jul 26, 2020, 20:49 ZDSDESIRE <notifications@github.com> wrote:

> solution 1) Install PyMySQL 0.9.2 (this is last version that you can use
> with mycli)
>
> or
>
> solution 2)
>
> /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/mycli/sqlexecute.py
>
> *Open the above file and modify it as follows:*
>
> line 6) from pymysql.converters import (convert_datetime,
> line 102) FIELD_TYPE.TIMESTAMP: lambda obj: (convert_datetime(obj) or obj),
>
> This is an effective solution.
>
> —
> You are receiving this because you modified the open/close state.
> Reply to this email directly, view it on GitHub
> <https://github.com/dbcli/mycli/issues/882#issuecomment-664090873>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAAMA3FA42WZUYZMZ6Y5VTDR5TTLLANCNFSM4PAC2OGQ>
> .
>

yay, it works again. THANKS ALL!
not working even with the latest version in Cent Os
Not working even with the latest version in Win10: 
```text
Successfully installed mycli-1.22.1 prompt-toolkit-3.0.5

C:\Users\z_zon>mycli --help
Traceback (most recent call last):
  File "e:\python\python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "e:\python\python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "E:\Python\Python38\Scripts\mycli.exe\__main__.py", line 4, in <module>
  File "e:\python\python38\lib\site-packages\mycli\main.py", line 9, in <module>
    from pwd import getpwuid
ModuleNotFoundError: No module named 'pwd'
```
I returned to mycli-1.21.1 and get it working based on @lunatiee 's solution ②: https://github.com/dbcli/mycli/issues/882#issuecomment-660885896
@verajosemanuel Can you tell me what is the version you're using? If you installed it via `yum` the latest version is probably not available via yum. 

You need to install it via `pip`. 
@ZDSDESIRE I can't reproduce it. Looks like the error you're seeing is very different from the original error reported. 

Looks like there is an issue with your python installation. The module not found error is complaining about a module called `pwd` but this is a standard library module that is shipped with python. https://docs.python.org/3.8/library/pwd.html

You might want to reinstall python on your machine and try again. 
@amjith `pwd` is Unix-only:
> This module provides access to the Unix user account and password database. It is available on all Unix versions.
Looks like the developers are only testing on Linux or MacOS. On Windows `mycli` is completely broken because of the import of `pwd`.
You're right. I'm sorry. I didn't know that pwd is Unix only. I'll have to
address this.

On Fri, Jul 31, 2020, 10:40 Thorsten Kampe <notifications@github.com> wrote:

> Looks like the developers are only testing on Linux or MacOS. On Windows
> mycli is completely broken because of the import of pwd.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/dbcli/mycli/issues/882#issuecomment-667219791>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAAMA3BDQOPULFTI2ZGLOE3R6LXZFANCNFSM4PAC2OGQ>
> .
>

@thorstenkampe @ZDSDESIRE I'v created a branch that fixes the `pwd` import issue. 

Can you try it out? 

```
pip install -U https://github.com/dbcli/mycli/archive/pwd_optional.zip
```
@amjith seems to work fine
ERROR: Package 'mycli' requires a different Python: 2.7.5 not in '>=3.6'

The above command should be:

`pip3 install -U https://github.com/dbcli/mycli/archive/pwd_optional.zip
`

@amjith 
Yay, it's working, thanks! Have you pulled out the pwd module?

It's the same problem as in #882
PyMySQL released a new version that breaks some API that we use. Hopefully will be fixed soon, for now you can downgrade   PyMYSQL:
```
pip3 install -I PyMySQL==0.9.2
```



Patch:
diff --git a/changelog.md b/changelog.md
index d8b3c4d6..1709d718 100644
--- a/changelog.md
+++ b/changelog.md
@@ -1,6 +1,10 @@
 TBD
 ===
 
+Bug Fixes:
+----------
+* Fix the breaking change introduced in PyMySQL 0.10.0. (Thanks: [Amjith]).
+
 Features:
 ---------
 * Add an option `--ssh-config-host` to read ssh configuration from OpenSSH configuration file.
diff --git a/mycli/sqlexecute.py b/mycli/sqlexecute.py
index 035d98d1..c68af0fa 100644
--- a/mycli/sqlexecute.py
+++ b/mycli/sqlexecute.py
@@ -3,7 +3,7 @@
 import sqlparse
 from .packages import special
 from pymysql.constants import FIELD_TYPE
-from pymysql.converters import (convert_mysql_timestamp, convert_datetime,
+from pymysql.converters import (convert_datetime,
                                 convert_timedelta, convert_date, conversions,
                                 decoders)
 try:
@@ -99,7 +99,7 @@ def connect(self, database=None, user=None, password=None, host=None,
         )
         conv = conversions.copy()
         conv.update({
-            FIELD_TYPE.TIMESTAMP: lambda obj: (convert_mysql_timestamp(obj) or obj),
+            FIELD_TYPE.TIMESTAMP: lambda obj: (convert_datetime(obj) or obj),
             FIELD_TYPE.DATETIME: lambda obj: (convert_datetime(obj) or obj),
             FIELD_TYPE.TIME: lambda obj: (convert_timedelta(obj) or obj),
             FIELD_TYPE.DATE: lambda obj: (convert_date(obj) or obj),


Test patch:
diff --git a/test/features/steps/auto_vertical.py b/test/features/steps/auto_vertical.py
index 6a67106a..974740d7 100644
--- a/test/features/steps/auto_vertical.py
+++ b/test/features/steps/auto_vertical.py
@@ -41,5 +41,5 @@ def step_see_large_results(context):
                 '***************************\r\n' +
                 '{}\r\n'.format('\r\n'.join(rows) + '\r\n'))
 
-    wrappers.expect_pager(context, expected, timeout=5)
+    wrappers.expect_pager(context, expected, timeout=10)
     wrappers.expect_exact(context, '1 row in set', timeout=2)
diff --git a/test/test_sqlexecute.py b/test/test_sqlexecute.py
index e445166f..c2d38be7 100644
--- a/test/test_sqlexecute.py
+++ b/test/test_sqlexecute.py
@@ -82,7 +82,7 @@ def test_invalid_syntax(executor):
 
 @dbtest
 def test_invalid_column_name(executor):
-    with pytest.raises(pymysql.InternalError) as excinfo:
+    with pytest.raises(pymysql.err.OperationalError) as excinfo:
         run(executor, 'select invalid command')
     assert "Unknown column 'invalid' in 'field list'" in str(excinfo.value)
 
diff --git a/test/test_tabular_output.py b/test/test_tabular_output.py
index fe7e5da3..7d7d000c 100644
--- a/test/test_tabular_output.py
+++ b/test/test_tabular_output.py
@@ -56,17 +56,18 @@ def description(self):
         [(None, None, None, 'Changed table format to sql-update')]
     mycli.formatter.query = ""
     output = mycli.format_output(None, FakeCursor(), headers)
-    assert "\n".join(output) == dedent('''\
+    actual = "\n".join(output)
+    assert actual == dedent('''\
             UPDATE `DUAL` SET
               `number` = 1
             , `optional` = NULL
-            , `float` = 10
+            , `float` = 10.0e0
             , `binary` = X'aa'
             WHERE `letters` = 'abc';
             UPDATE `DUAL` SET
               `number` = 456
             , `optional` = '1'
-            , `float` = 0.5
+            , `float` = 0.5e0
             , `binary` = X'aabb'
             WHERE `letters` = 'd';''')
     # Test sql-update-2 output format
@@ -77,12 +78,12 @@ def description(self):
     assert "\n".join(output) == dedent('''\
             UPDATE `DUAL` SET
               `optional` = NULL
-            , `float` = 10
+            , `float` = 10.0e0
             , `binary` = X'aa'
             WHERE `letters` = 'abc' AND `number` = 1;
             UPDATE `DUAL` SET
               `optional` = '1'
-            , `float` = 0.5
+            , `float` = 0.5e0
             , `binary` = X'aabb'
             WHERE `letters` = 'd' AND `number` = 456;''')
     # Test sql-insert output format (without table name)
@@ -92,8 +93,8 @@ def description(self):
     output = mycli.format_output(None, FakeCursor(), headers)
     assert "\n".join(output) == dedent('''\
             INSERT INTO `DUAL` (`letters`, `number`, `optional`, `float`, `binary`) VALUES
-              ('abc', 1, NULL, 10, X'aa')
-            , ('d', 456, '1', 0.5, X'aabb')
+              ('abc', 1, NULL, 10.0e0, X'aa')
+            , ('d', 456, '1', 0.5e0, X'aabb')
             ;''')
     # Test sql-insert output format (with table name)
     assert list(mycli.change_table_format("sql-insert")) == \
@@ -102,8 +103,8 @@ def description(self):
     output = mycli.format_output(None, FakeCursor(), headers)
     assert "\n".join(output) == dedent('''\
             INSERT INTO `table` (`letters`, `number`, `optional`, `float`, `binary`) VALUES
-              ('abc', 1, NULL, 10, X'aa')
-            , ('d', 456, '1', 0.5, X'aabb')
+              ('abc', 1, NULL, 10.0e0, X'aa')
+            , ('d', 456, '1', 0.5e0, X'aabb')
             ;''')
     # Test sql-insert output format (with database + table name)
     assert list(mycli.change_table_format("sql-insert")) == \
@@ -112,6 +113,6 @@ def description(self):
     output = mycli.format_output(None, FakeCursor(), headers)
     assert "\n".join(output) == dedent('''\
             INSERT INTO `database`.`table` (`letters`, `number`, `optional`, `float`, `binary`) VALUES
-              ('abc', 1, NULL, 10, X'aa')
-            , ('d', 456, '1', 0.5, X'aabb')
+              ('abc', 1, NULL, 10.0e0, X'aa')
+            , ('d', 456, '1', 0.5e0, X'aabb')
             ;''')