Repository: kedro-org/kedro

Issue:
ExcelDataSet not working after pandas 1.4.0 release (python 3.8)
## Description

On a fresh new project, after installing the extra dependency `kedro[pandas.ExcelDataSet]`, the user receives a runtime error when trying to load a xslx file dataset via the data catalog. The cause of the issue seems to be an incompatibility with the new pandas `v1.4.0` and `xlrd~=1.0`, the latter being enforced by kedro `17.6.0`.

## Context

The error occurred while I was following the exact steps of the spaceflights tutorial. In order to load the `shuttles` dataset (stored as a xslx file), I was instructed to install the extra dependency `kedro[pandas.ExcelDataSet]`. After installing the dependency and trying to load the dataset via the catalog, I received the following error: `Pandas requires version '2.0.1' or newer of 'xlrd' (version '1.2.0' currently installed).`

After some debugging, I discovered that pandas released version `1.4.0` two days ago, and it bumped the mininum version of optional dependency xlrd to `2.0.1` ([source](https://pandas.pydata.org/docs/whatsnew/v1.4.0.html#increased-minimum-versions-for-dependencies), [source2](https://github.com/pandas-dev/pandas/blob/v1.4.0/pandas/compat/_optional.py#L40)). As kedro enforces `xlrd~=1.0` via extra dependency `kedro[pandas.ExcelDataSet]` ([source](https://github.com/kedro-org/kedro/blob/0.17.6/setup.py#L75)), pandas checks that the currently installed version of xlrd is `1.2.0` and raises the error before importing the package ([source](https://github.com/pandas-dev/pandas/blob/v1.4.0/pandas/compat/_optional.py#L133-L152)).

I repeated the same steps using python 3.7, but because pandas 1.4.0 only supports python 3.8+ ([source](https://github.com/pandas-dev/pandas/blob/v1.4.0/setup.cfg#L40)), the installed version of pandas was 1.3.5. When I tried to load the dataset, I received no errors (although I got a future warning saying xlrd will not support xlsx files in version >= 2.0)

## Steps to Reproduce
1. Create a virtualenv with python 3.8 and activate it
2. Run `pip install kedro==0.17.6`
3. Run `kedro new`
4. `cd` to project dir
5. Run `kedro install`
6. Replace the line `kedro==0.17.6` with `kedro[pandas.ExcelDataSet]==0.17.6`  in src/requirements.in
7. Run `kedro build-reqs && kedro install`
8. Add file `shuttles.xlsx` (from spaceflights tutorial) to folder `data/01_raw/`
9. Add the dataset information to `catalog.yml`:
    ```yml
    shuttles:
      type: pandas.ExcelDataSet
      filepath: data/01_raw/shuttles.xlsx
    ```
10. Run `kedro ipython`
11. Run python code: `shuttles = catalog.load("shuttles")`

## Expected Result
The xlsx file should've been loaded into memory as a pandas Dataframe

## Actual Result
I received the following error:

```
DataSetError: Failed while loading data from data set ExcelDataSet(filepath=/.../data/01_raw/shuttles.xlsx, load_args={'engine': xlrd}, protocol=file, save_args={'index': False}, writer_args={'engine': xlsxwriter}).
Pandas requires version '2.0.1' or newer of 'xlrd' (version '1.2.0' currently installed).
```

## Your Environment

* Kedro version used (`pip show kedro` or `kedro -V`): 0.17.6
* Python version used (`python -V`): 3.8.12
* Operating system and version: WSL: Ubuntu 20.04
* Pandas version: 1.4.0
* xlrd version: 1.2.0



Discussion:
Hi @amaralbf we're working on a patch fix for this since one of our upstream dependencies has started causing this.

Quick hack solution for you to get you working:

`pip install openpyxl # which will be the default engine in 0.18.0`

then in you catalog entry:

```yaml
shuttles:
  type: pandas.ExcelDataSet
  filepath: data/01_raw/shuttles.xlsx
  load_args:
    engine: openpyxl
```
Hello @datajoely, thanks for the quick reply!

<p>There is a typo in the yml snippet: <code><del>enigne</del> -> engine</code></p>

Could you edit the snippet, please?
> Hello @datajoely, thanks for the quick reply!
> 
> There is a typo in the yml snippet: `~enigne~ -> engine`
> 
> Could you edit the snippet, please?

Done 🙈
Thank you for your very thorough bug report @amaralbf, much appreciated. I had noticed the same thing also, since our CI tests are now failing on Python 3.8...
Confirming same issue with pandas.SQLTableDataSet
> Confirming same issue with pandas.SQLTableDataSet

@lucasjamar can you provide a stack trace - is there a newer SQLAlchemy version required?
Here  is what i get in a new conda python 3.8 environment. Pandas and kedro are there but not SQLAlchemy.

```
(base) C:\Users\JAMAR_L\Documents>conda activate test-kedro

(test-kedro) C:\Users\JAMAR_L\Documents>pip install kedro[pandas.SQLTableDataSet]
WARNING: Ignoring invalid distribution -lotly (c:\users\jamar_l\appdata\roaming\python\python38\site-packages)
WARNING: Ignoring invalid distribution -ffi (c:\users\jamar_l\appdata\roaming\python\python38\site-packages)
WARNING: Ignoring invalid distribution -lotly (c:\users\jamar_l\appdata\roaming\python\python38\site-packages)
WARNING: Ignoring invalid distribution -ffi (c:\users\jamar_l\appdata\roaming\python\python38\site-packages)
Requirement already satisfied: kedro[pandas.SQLTableDataSet] in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (0.17.6)
Requirement already satisfied: PyYAML<6.0,>=4.2 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (5.4.1)
Requirement already satisfied: pip-tools~=6.4 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (6.4.0)
Requirement already satisfied: dynaconf<4.0.0,>=3.1.2 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (3.1.7)
Requirement already satisfied: anyconfig~=0.10.0 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (0.10.1)
Requirement already satisfied: fsspec<2022.01,>=2021.04 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (2021.11.1)
Requirement already satisfied: cachetools~=4.1 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (4.2.4)
Requirement already satisfied: python-json-logger~=2.0 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (2.0.2)
Requirement already satisfied: setuptools>=38.0 in c:\users\jamar_l\anaconda3\envs\test-kedro\lib\site-packages (from kedro[pandas.SQLTableDataSet]) (58.0.4)
Requirement already satisfied: toposort~=1.5 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (1.6)
Requirement already satisfied: pluggy~=0.13.0 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (0.13.1)
Requirement already satisfied: toml~=0.10 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (0.10.2)
Requirement already satisfied: rope~=0.21.0 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (0.21.1)
Requirement already satisfied: jmespath<1.0,>=0.9.5 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (0.10.0)
Requirement already satisfied: gitpython~=3.0 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (3.1.26)
Requirement already satisfied: jupyter-client<7.0,>=5.1 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (6.1.12)
Requirement already satisfied: click<8.0 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (7.1.2)
Requirement already satisfied: cookiecutter~=1.7.0 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (1.7.3)
Requirement already satisfied: SQLAlchemy~=1.2 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (1.4.25)
Requirement already satisfied: pandas>=0.24 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from kedro[pandas.SQLTableDataSet]) (1.4.0)
Requirement already satisfied: poyo>=0.5.0 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from cookiecutter~=1.7.0->kedro[pandas.SQLTableDataSet]) (0.5.0)
Requirement already satisfied: binaryornot>=0.4.4 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from cookiecutter~=1.7.0->kedro[pandas.SQLTableDataSet]) (0.4.4)
Collecting six>=1.10
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: requests>=2.23.0 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from cookiecutter~=1.7.0->kedro[pandas.SQLTableDataSet]) (2.27.1)
Requirement already satisfied: jinja2-time>=0.2.0 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from cookiecutter~=1.7.0->kedro[pandas.SQLTableDataSet]) (0.2.0)
Collecting Jinja2<4.0.0,>=2.7
  Using cached Jinja2-3.0.3-py3-none-any.whl (133 kB)
Requirement already satisfied: python-slugify>=4.0.0 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from cookiecutter~=1.7.0->kedro[pandas.SQLTableDataSet]) (5.0.2)
Requirement already satisfied: chardet>=3.0.2 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from binaryornot>=0.4.4->cookiecutter~=1.7.0->kedro[pandas.SQLTableDataSet]) (4.0.0)
Requirement already satisfied: gitdb<5,>=4.0.1 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from gitpython~=3.0->kedro[pandas.SQLTableDataSet]) (4.0.9)
Requirement already satisfied: smmap<6,>=3.0.1 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from gitdb<5,>=4.0.1->gitpython~=3.0->kedro[pandas.SQLTableDataSet]) (5.0.0)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from Jinja2<4.0.0,>=2.7->cookiecutter~=1.7.0->kedro[pandas.SQLTableDataSet]) (2.0.1)
Requirement already satisfied: arrow in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from jinja2-time>=0.2.0->cookiecutter~=1.7.0->kedro[pandas.SQLTableDataSet]) (1.2.1)
Requirement already satisfied: traitlets in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from jupyter-client<7.0,>=5.1->kedro[pandas.SQLTableDataSet]) (5.1.1)
Requirement already satisfied: jupyter-core>=4.6.0 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from jupyter-client<7.0,>=5.1->kedro[pandas.SQLTableDataSet]) (4.7.1)
Requirement already satisfied: tornado>=4.1 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from jupyter-client<7.0,>=5.1->kedro[pandas.SQLTableDataSet]) (6.1)
Requirement already satisfied: pyzmq>=13 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from jupyter-client<7.0,>=5.1->kedro[pandas.SQLTableDataSet]) (22.3.0)
Requirement already satisfied: python-dateutil>=2.1 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from jupyter-client<7.0,>=5.1->kedro[pandas.SQLTableDataSet]) (2.8.2)
Requirement already satisfied: pywin32>=1.0 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from jupyter-core>=4.6.0->jupyter-client<7.0,>=5.1->kedro[pandas.SQLTableDataSet]) (303)
Collecting pytz>=2020.1
  Using cached pytz-2021.3-py2.py3-none-any.whl (503 kB)
Requirement already satisfied: numpy>=1.18.5 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from pandas>=0.24->kedro[pandas.SQLTableDataSet]) (1.22.1)
Requirement already satisfied: wheel in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from pip-tools~=6.4->kedro[pandas.SQLTableDataSet]) (0.36.2)
Requirement already satisfied: pep517 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from pip-tools~=6.4->kedro[pandas.SQLTableDataSet]) (0.12.0)
Requirement already satisfied: pip>=21.2 in c:\users\jamar_l\anaconda3\envs\test-kedro\lib\site-packages (from pip-tools~=6.4->kedro[pandas.SQLTableDataSet]) (21.2.2)
Requirement already satisfied: text-unidecode>=1.3 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from python-slugify>=4.0.0->cookiecutter~=1.7.0->kedro[pandas.SQLTableDataSet]) (1.3)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\jamar_l\anaconda3\envs\test-kedro\lib\site-packages (from requests>=2.23.0->cookiecutter~=1.7.0->kedro[pandas.SQLTableDataSet]) (2021.10.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from requests>=2.23.0->cookiecutter~=1.7.0->kedro[pandas.SQLTableDataSet]) (1.26.8)
Requirement already satisfied: idna<4,>=2.5 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from requests>=2.23.0->cookiecutter~=1.7.0->kedro[pandas.SQLTableDataSet]) (3.3)
Requirement already satisfied: charset-normalizer~=2.0.0 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from requests>=2.23.0->cookiecutter~=1.7.0->kedro[pandas.SQLTableDataSet]) (2.0.10)
Requirement already satisfied: greenlet!=0.4.17 in c:\users\jamar_l\appdata\roaming\python\python38\site-packages (from SQLAlchemy~=1.2->kedro[pandas.SQLTableDataSet]) (1.1.2)

(test-kedro) C:\Users\JAMAR_L\Documents>python
Python 3.8.12 (default, Oct 12 2021, 03:01:40) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import SQLAlchemy as sq
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'SQLAlchemy'
>>> import pandas as pd
>>> pd.Timestamp.now()
Timestamp('2022-01-27 09:52:59.264089')
```
Thanks @lucasjamar - can confirm this is the pandas 1.4.0 release, cross referencing with this list of changes: https://pandas.pydata.org/docs/dev/whatsnew/v1.4.0.html#backwards-incompatible-api-changes

| Package | Pandas Version | Kedro Version (0.17.6) | Status |
|---|---|---|---|
| matplotlib | 3.3.2 | matplotlib>=3.0.3, <4.0 | ✅ |
| openpyxl | 3.0.3 | openpyxl>=3.0.3 | ✅ |
| pandas-gbq | 0.14.0 | pandas-gbq>=0.12.0, <1.0 | ✅ |
| pyarrow | 1.0.1 | pyarrow>=1.0, <7.0 | ✅ |
| sqlalchemy | 1.4.0 | SQLAlchemy~=1.2 | 🚨 |
| xlrd | 2.0.1 | xlrd~=1.0 | 🚨 |
| xlsxwriter | 1.2.2 | xlsxwriter~=1.0 | 🚨 |

Looking into this now and I think the only problem is with `xlrd`. Those are currently the only unit tests that are failing, and I don't see what the problem would be with other dependencies.

@lucasjamar - I think this is just an issue that you are doing `import SQLAlchemy` rather than `import sqlalchemy`. SQLAlchemy is one of those packages like scikit-learn where the distribution name on PyPI (what you `pip install`) doesn't match the package name (what you `import`). I've tried and this works ok for me. 

@datajoely - `SQLAlchemy=1.4.0` is compatible with the version specification `SQLAlchemy~=1.2`, since that gives `SQLAlchemy>=1.2, == 1.*`. You can see from the above logs that the latest version is being installed
```
Requirement already satisfied: SQLAlchemy~=1.2 ... (from kedro[pandas.SQLTableDataSet]) (1.4.25)
Requirement already satisfied: pandas>=0.24 ... (from kedro[pandas.SQLTableDataSet]) (1.4.0)
```
Same goes for `xlsxwriter`. So I think the only problem here is `xlrd`.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -10,7 +10,7 @@
 here = path.abspath(path.dirname(__file__))
 
 
-PANDAS = "pandas>=0.24"
+PANDAS = "pandas>=0.24, <1.4"
 SPARK = "pyspark>=2.2, <4.0"
 HDFS = "hdfs>=2.5.8, <3.0"
 S3FS = "s3fs>=0.3.0, <0.5"
@@ -77,8 +77,11 @@ def _collect_requirements(requires):
     "pandas.FeatherDataSet": [PANDAS],
     "pandas.GBQTableDataSet": [PANDAS, "pandas-gbq>=0.12.0, <1.0"],
     "pandas.GBQQueryDataSet": [PANDAS, "pandas-gbq>=0.12.0, <1.0"],
-    "pandas.HDFDataSet": [PANDAS, "tables~=3.6.0; platform_system == 'Windows'",
-                          "tables~=3.6; platform_system != 'Windows'"],
+    "pandas.HDFDataSet": [
+        PANDAS,
+        "tables~=3.6.0; platform_system == 'Windows'",
+        "tables~=3.6; platform_system != 'Windows'",
+    ],
     "pandas.JSONDataSet": [PANDAS],
     "pandas.ParquetDataSet": [PANDAS, "pyarrow>=1.0, <7.0"],
     "pandas.SQLTableDataSet": [PANDAS, "SQLAlchemy~=1.2"],


Test patch:
diff --git a/test_requirements.txt b/test_requirements.txt
--- a/test_requirements.txt
+++ b/test_requirements.txt
@@ -27,7 +27,7 @@ nbformat~=4.4
 networkx~=2.4
 openpyxl>=3.0.3, <4.0
 pandas-gbq>=0.12.0, <1.0
-pandas>=0.24.0  # Needs to be at least 0.24.0 to make use of `pandas.DataFrame.to_numpy` (recommended alternative to `pandas.DataFrame.values`)
+pandas>=0.24, <1.4  # Needs to be at least 0.24.0 to make use of `pandas.DataFrame.to_numpy` (recommended alternative to `pandas.DataFrame.values`); needs to be less than 1.4 for compatiblity with xlrd~=1.0.
 Pillow~=8.0
 plotly>=4.8.0, <6.0
 pre-commit~=1.17