Repository: Snowflake-Labs/schemachange (language: python)

Issue:
Update of Numpy to 2.0 has broken schemachange
**Describe the bug**
A ValueError occurs when attempting to run SchemaChange, this did not occur last Friday (6/14) but is occurring this Monday (6/17).

**To Reproduce**
Steps to reproduce the behavior:
1. pip install schemachange
2. run schemachange

**Expected behavior**
Application of a versioned migration and repeatable files.

**Screenshots**
2024-06-17T15:57:56.5457992Z Traceback (most recent call last):
2024-06-17T15:57:56.5458535Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/bin/schemachange", line 5, in <module>
2024-06-17T15:57:56.5458837Z     from schemachange.cli import main
2024-06-17T15:57:56.5459520Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/schemachange/cli.py", line 15, in <module>
2024-06-17T15:57:56.5459791Z     import snowflake.connector
2024-06-17T15:57:56.5460319Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/snowflake/connector/__init__.py", line 19, in <module>
2024-06-17T15:57:56.5460671Z     from .connection import SnowflakeConnection
2024-06-17T15:57:56.5461162Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/snowflake/connector/connection.py", line 35, in <module>
2024-06-17T15:57:56.5461731Z     from . import errors, proxy
2024-06-17T15:57:56.5462137Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/snowflake/connector/errors.py", line 15, in <module>
2024-06-17T15:57:56.5462514Z     from .compat import BASE_EXCEPTION_CLASS
2024-06-17T15:57:56.5462948Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/snowflake/connector/compat.py", line 19, in <module>
2024-06-17T15:57:56.5463285Z     from . import constants
2024-06-17T15:57:56.5463719Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/snowflake/connector/constants.py", line 12, in <module>
2024-06-17T15:57:56.5463989Z     from .options import pyarrow as pa
2024-06-17T15:57:56.5464397Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/snowflake/connector/options.py", line 135, in <module>
2024-06-17T15:57:56.5464708Z     pandas, pyarrow, installed_pandas = _import_or_missing_pandas_option()
2024-06-17T15:57:56.5465295Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/snowflake/connector/options.py", line 77, in _import_or_missing_pandas_option
2024-06-17T15:57:56.5465683Z     pandas = importlib.import_module("pandas")
2024-06-17T15:57:56.5465951Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/importlib/__init__.py", line 126, in import_module
2024-06-17T15:57:56.5515033Z     return _bootstrap._gcd_import(name[level:], package, level)
2024-06-17T15:57:56.5516339Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pandas/__init__.py", line 22, in <module>
2024-06-17T15:57:56.5516827Z     from pandas.compat import is_numpy_dev as _is_numpy_dev  # pyright: ignore # noqa:F401
2024-06-17T15:57:56.5517385Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pandas/compat/__init__.py", line 18, in <module>
2024-06-17T15:57:56.5518015Z     from pandas.compat.numpy import (
2024-06-17T15:57:56.5518522Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pandas/compat/numpy/__init__.py", line 4, in <module>
2024-06-17T15:57:56.5518894Z     from pandas.util.version import Version
2024-06-17T15:57:56.5519393Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pandas/util/__init__.py", line 2, in <module>
2024-06-17T15:57:56.5519762Z     from pandas.util._decorators import (  # noqa:F401
2024-06-17T15:57:56.5520269Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pandas/util/_decorators.py", line 14, in <module>
2024-06-17T15:57:56.5520620Z     from pandas._libs.properties import cache_readonly
2024-06-17T15:57:56.5521222Z   File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pandas/_libs/__init__.py", line 13, in <module>
2024-06-17T15:57:56.5521536Z     from pandas._libs.interval import Interval
2024-06-17T15:57:56.5521835Z   File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
2024-06-17T15:57:56.5522229Z ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

**Schemachange (please complete the following information):**
- Version: 3.6.1


Discussion / hints:
I have the same issue but the current workaround that I have is to run this after the schemachange install:
pip install --force-reinstall numpy==1.26.0
> I have the same issue but the current workaround that I have is to run this after the schemachange install: pip install --force-reinstall numpy==1.26.0

I am doing that as well, but feel this is more of a band aid and anyone looking to use schemachange will run into the same issue until it is resolved.
> I have the same issue but the current workaround that I have is to run this after the schemachange install: pip install --force-reinstall numpy==1.26.0

After seeing this, I was able to use the same technique as a workaround. Thank you for mentioning this.
this looks to be numpy issue https://github.com/numpy/numpy/issues/26710 and adding `numpy<2` to your requirements file might provide a relief until this is sorted
installing schemachange with -c option with numpy==1.26.4 in the constraints.txt also works. 
also, next release will probably remove the pandas dependency (https://github.com/Snowflake-Labs/schemachange/issues/229) which I think removes numpy from the dependencies here.
For what it's worth, my team is also running into this issue. We are currently unable to resolve it, due to the nature in which we are attempting to run.

We are running via Gitlab, their docker-in-docker service. We are running a pre-compiled docker image (google/cloud-sdk:slim), from there we are (pipx) installing schemachange. After installing schemachange, we tried force installing a lowered version of numpy, but to no avail. 

We will continue to try different solutions, but this is our version of the problem. We have temporarily disabled use of schemachange for our project until this package gets updated appropriately.
For those of us who have installed schemachange with pipx be aware pipx creates venvs which is going to isolate the installed packages, so you need to "inject" the package change:

```
$ pipx inject --force schemachange "numpy<2"
  injected package numpy into venv schemachange
done! ✨ 🌟 ✨
```

schemachange should now work but this is a bandaid.
> For those of us who have installed schemachange with pipx be aware pipx creates venvs which is going to isolate the installed packages, so you need to "inject" the package change:
> 
> ```
> $ pipx inject --force schemachange "numpy<2"
>   injected package numpy into venv schemachange
> done! ✨ 🌟 ✨
> ```
> 
> schemachange should now work but this is a bandaid.

Wow, thanks for this pro-tip. I'll give this a try -- but yes, this is a band-aid! The schemachange package needs to be updated appropriately

Fix patch:
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
index e6a49e39..3ded2980 100644
--- a/.github/CONTRIBUTING.md
+++ b/.github/CONTRIBUTING.md
@@ -4,7 +4,7 @@
 
 When reporting issues please include as much detail as possible about your
 operating environment, schemachange version and python version. Whenever possible, please
-also include a brief, self-contained code example that demonstrates the problem. 
+also include a brief, self-contained code example that demonstrates the problem.
 
 We have included [issue templates for reporting bugs, requesting features and seeking clarifications.](https://github.com/Snowflake-Labs/schemachange/issues/new/choose)
 Choose the appropriate issue template to contribute to the repository.
@@ -21,50 +21,50 @@ Thanks for your interest in contributing code to schemachange!
 1. If you are a first-time contributor
     + Go to [Snowflake-Labs/Schemachange](https://github.com/Snowflake-Labs/schemachange) and click the "fork" button to create your own copy of the project.
     + [Clone](https://github.com/git-guides/git-clone) the project to your local computer
-  
+
     ```shell
     git clone https://github.com/your-username/schemachange.git
-    
+
     ```
-    
+
     + Change the directory
-   
+
     ```shell
     cd schemachange
-    
+
     ```
     + Add upstream repository:
 
     ```shell
     git remote add upstream https://github.com/Snowflake-Labs/schemachange
-    
+
     ```
-    
+
     + Now, `git remote -v` will show two [remote](https://github.com/git-guides/git-remote) repositories named:
       + `upstream`, which refers to the `schemachange` repository
-      + `origin`, which refers to your personal fork 
+      + `origin`, which refers to your personal fork
     + [Pull](https://github.com/git-guides/git-pull) the latest changes from upstream, including tags:
-      
+
     ```shell
     git checkout main
     git pull upstream main --tags
-    
+
     ```
-    
+
 3. Develop your contribution
     + Create a branch for the features you want to work on. Since the branch name will appear in the merge message, use a sensible name such as 'update-build-library-dependencies':
-      
+
     ```shell
     git checkout -b update-build-library-dependencies
     ```
-    
+
     + Commit locally as you progress ( [git add](https://github.com/git-guides/git-add) and [git commit](https://github.com/git-guides/git-commit) ). Use a properly formatted commit message. Be sure to document any changed behavior.
 4. To submit your contribution
     + [Push](https://github.com/git-guides/git-push) your changes back to your fork on GitHub
 
     ```shell
     git push origin update-build-library-dependencies
-    
+
     ```
 
     + Go to GitHub. The new branch will show up with a green [Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests#initiating-the-pull-request) button. Make sure the title and message are clear, concise and self explanatory. Then click the button to submit it.
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 63e4b8fe..c707b5ef 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,6 +3,11 @@ All notable changes to this project will be documented in this file.
 
 *The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).*
 
+## [3.6.2] - 2024-06-10
+### Changed
+- Updated pandas version dependency
+- Pinned NumPy version dependency
+
 ## [3.6.1] - 2023-11-15
 ### Added
 - Allow passing snowflake schema as config or CLI parameter
diff --git a/requirements.txt b/requirements.txt
index 5e8d0440..161d51b6 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,4 +1,5 @@
 Jinja2~=3.0
-pandas~=1.3
+numpy=>=1.23.2,<2
+pandas>=1.5.3,<3
 PyYAML~=6.0
 snowflake-connector-python>=2.8,<4.0
diff --git a/setup.cfg b/setup.cfg
index b3f8ec01..66a0f9bc 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,6 +1,6 @@
 [metadata]
 name = schemachange
-version = 3.6.1
+version = 3.6.2
 description = A Database Change Management tool for Snowflake
 long_description = file: README.md
 long_description_content_type = text/markdown
@@ -18,7 +18,8 @@ classifiers =
 packages = schemachange
 install_requires =
     jinja2~=3.0
-    pandas~=1.3
+    numpy>=1.23.2,<2
+    pandas>=1.5.3,<3
     pyyaml~=6.0
     snowflake-connector-python>=2.8,<4.0
 python_requires = >=3.8


Test patch:
diff --git a/.github/workflows/dev-pytest.yml b/.github/workflows/dev-pytest.yml
index 177d2f6f..215e3b67 100644
--- a/.github/workflows/dev-pytest.yml
+++ b/.github/workflows/dev-pytest.yml
@@ -4,9 +4,9 @@ on:
   push:
     branches:
       - dev
-  
+
   workflow_dispatch:
-    
+
 
 jobs:
   build: