Repository: hackingmaterials/matminer (language: python)

Issue:
compatibility request: pandas-2+
**The reason I am are unable to use the Discourse forum in my ticket:
got message "Oops! That page doesn’t exist or is private" by following link https://discuss.matsci.org/c/matminer/automatminer**

Could you please make matminer compatible with pandas newer than 1.5.3? Its not fun to create another 1.2 Gb environment just because one package only compatible with  1.5.3.
Missing compatibility with pandas v2
In the current state of `master` we are allowing people to install matminer with pandas v2, yet many featurizers do not work in this state. I made as much progress as I can in #912 but cannot justify spending any time on this myself right now. This should be addressed before the next release.


Discussion / hints:
Unfortunately its not as simple as bumping the version number and work (and someone to do the work) is required to make it compatible (see https://github.com/hackingmaterials/matminer/issues/915) 

Depending on what you're using matminer for you might be able to get away with installing directly from this repo (which allows pandas 2) but you'll have to test yourself if featurizers are consistent (if you just using this to access datasets then you'll be fine).

Closing as dupe of #915 
If only to add a public voice that this is an issue...

I have also found that this dependency makes matminer difficult to use alongside other modern packages.
Proposed solution: deprecate the featurizers that are in some sense "broken" by this pandas upgrade and be more strict in our pinning of dependencies (in order to match the current sustainable release cadence of this package without dedicated maintainers) in preparation for a v0.10.0 release. 

In the future, when things break due to upstream packages, people are more than welcome to submit fixes (and I am happy to review/merge/release) but I cannot sign up for the ongoing maintenance of keeping deps up to date.
Any opinions? @ardunn @tschaume ? I guess it could break subsequent code.
I would love to keep on using matminer but in it's current state that's probably hard to do.
@JaGeo matminer used to have a pin on `pandas~=1.5` which is keeping MP from staying up-to-date with pandas (This would be true with any upward pins that `matminer` enforces in its dependencies). I use present tense here because there hasn't been a release of matminer since the pandas requirement was removed in https://github.com/hackingmaterials/matminer/commit/237603cdf5bd43675af45109ee3b7baf904cdda9. As @ml-evs mentioned, we're waiting for 0.10.0 to be released. We're also looking into how `matminer` enters MP's dependency stack as a required dependency and how we could make that dependency optional.

I agree with @ml-evs that featurizers that don't support recent versions of pandas should probably be deprecated. Alternatively, they could run if a compatible version of pandas happens to be installed and throw a warning otherwise.

HTH

 
If it's not that urgent then can we please just reintroduce the pandas pin so that this package actually works for the majority of its userbase? i.e. people who don't mind having a separate virtualenv for matminer do its featurising. I can't see what MP are using it for in any of the open repos but I'm sure it would be less overhead for everyone if you could just vendor the bit you need (or contribute back pandas V2 support yourselves, as it seems like no-one needs it enough to implement it). 

We'll have exactly the same problem with pandas 3 soon too. 
As an aside, it's likely that it's actually numpy that's causing most of the problems, a side effect of bumping pandas
@ml-evs We were able to remove the matminer dependency from the MP stack entirely (other than our builders that depend on `robocrys`). Feel free to manage matminer's pandas dependency as you, @ardunn and @computron think is best for your user community.
I think I've resolved the remaining issues with numpy 1.24+ support in #925, after merging I plan to do a v0.9.1 release unless there are any objections. Hopefully it will then be easier to upgrade pandas etc in the future.
Release is made -- if people run into issues then we can reopen this and consider steps to support pandas v2 (it may not be as difficult as described above, as numpy compatibility has now been fixed [at least as far as it is tested]).

Fix patch:
diff --git a/setup.py b/setup.py
index e087f8f93..c0e79f2de 100644
--- a/setup.py
+++ b/setup.py
@@ -56,7 +56,7 @@ def local_version(version):
         install_requires=[
             "numpy >= 1.23, < 2",
             "requests ~= 2.31",
-            "pandas ~= 1.5",
+            "pandas >= 1.5, < 3",
             "tqdm ~= 4.66",
             "pymongo ~= 4.5",
             "scikit_learn ~= 1.3",


Test patch:
diff --git a/requirements/macos-latest_py3.10.txt b/requirements/macos-latest_py3.10.txt
deleted file mode 100644
index ed418ea7d..000000000
--- a/requirements/macos-latest_py3.10.txt
+++ /dev/null
@@ -1,354 +0,0 @@
-#
-# This file is autogenerated by pip-compile with Python 3.10
-# by the following command:
-#
-#    pip-compile --all-extras --output-file=requirements/macos-latest_py3.10_extras.txt
-#
-aflow==0.0.11
-    # via matminer (setup.py)
-alabaster==0.7.13
-    # via sphinx
-ase==3.22.1
-    # via
-    #   aflow
-    #   dscribe
-    #   matminer (setup.py)
-astroid==2.15.6
-    # via pylint
-attrs==23.1.0
-    # via
-    #   jsonschema
-    #   referencing
-babel==2.12.1
-    # via sphinx
-beautifulsoup4==4.12.2
-    # via aflow
-black==23.7.0
-    # via matminer (setup.py)
-certifi==2023.7.22
-    # via requests
-cffi==1.15.1
-    # via cryptography
-charset-normalizer==3.2.0
-    # via requests
-citrination-client==6.5.1
-    # via matminer (setup.py)
-click==8.1.7
-    # via black
-contourpy==1.1.0
-    # via matplotlib
-coverage[toml]==6.5.0
-    # via
-    #   coveralls
-    #   matminer (setup.py)
-    #   pytest-cov
-coveralls==3.3.1
-    # via matminer (setup.py)
-cryptography==41.0.3
-    # via
-    #   globus-sdk
-    #   pyjwt
-cycler==0.11.0
-    # via matplotlib
-dill==0.3.7
-    # via pylint
-dnspython==2.4.2
-    # via pymongo
-docopt==0.6.2
-    # via coveralls
-docutils==0.20.1
-    # via sphinx
-dscribe==2.1.0
-    # via matminer (setup.py)
-emmet-core==0.67.5
-    # via mp-api
-exceptiongroup==1.1.3
-    # via pytest
-fair-research-login==0.3.1
-    # via mdf-toolbox
-flake8==6.1.0
-    # via matminer (setup.py)
-fonttools==4.42.1
-    # via matplotlib
-future==0.18.3
-    # via
-    #   matminer (setup.py)
-    #   uncertainties
-globus-nexus-client==0.4.1
-    # via mdf-toolbox
-globus-sdk==3.28.0
-    # via
-    #   fair-research-login
-    #   globus-nexus-client
-    #   mdf-forge
-    #   mdf-toolbox
-httplib2==0.22.0
-    # via matminer (setup.py)
-idna==3.4
-    # via requests
-imagesize==1.4.1
-    # via sphinx
-iniconfig==2.0.0
-    # via pytest
-isort==5.12.0
-    # via pylint
-jinja2==3.1.2
-    # via
-    #   aflow
-    #   sphinx
-jmespath==1.0.1
-    # via matminer (setup.py)
-joblib==1.3.2
-    # via
-    #   dscribe
-    #   pymatgen
-    #   scikit-learn
-jsonschema==4.3.0
-    # via
-    #   matminer (setup.py)
-    #   mdf-toolbox
-kiwisolver==1.4.5
-    # via matplotlib
-latexcodec==2.0.1
-    # via pybtex
-lazy-object-proxy==1.9.0
-    # via astroid
-llvmlite==0.40.1
-    # via numba
-markupsafe==2.1.3
-    # via jinja2
-matplotlib==3.7.2
-    # via
-    #   ase
-    #   pymatgen
-mccabe==0.7.0
-    # via
-    #   flake8
-    #   pylint
-mdf-forge==0.8.0
-    # via matminer (setup.py)
-mdf-toolbox==0.6.0
-    # via mdf-forge
-monty==2023.9.5
-    # via
-    #   emmet-core
-    #   matminer (setup.py)
-    #   mp-api
-    #   pymatgen
-mp-api==0.35.1
-    # via pymatgen
-mpmath==1.3.0
-    # via sympy
-msgpack==1.0.5
-    # via mp-api
-mypy-extensions==1.0.0
-    # via black
-networkx==3.1
-    # via pymatgen
-numba==0.57.1
-    # via sparse
-numpy==1.24.4
-    # via
-    #   aflow
-    #   ase
-    #   contourpy
-    #   dscribe
-    #   matminer (setup.py)
-    #   matplotlib
-    #   numba
-    #   pandas
-    #   pymatgen
-    #   pypif
-    #   scikit-learn
-    #   scipy
-    #   sparse
-    #   spglib
-packaging==23.1
-    # via
-    #   black
-    #   matplotlib
-    #   plotly
-    #   pytest
-    #   sphinx
-palettable==3.3.3
-    # via pymatgen
-pandas==2.1.0
-    # via
-    #   matminer (setup.py)
-    #   pymatgen
-pathspec==0.11.2
-    # via black
-pillow==10.0.0
-    # via matplotlib
-platformdirs==3.10.0
-    # via
-    #   black
-    #   pylint
-plotly==5.16.1
-    # via pymatgen
-pluggy==1.3.0
-    # via pytest
-pybind11==2.11.1
-    # via dscribe
-pybtex==0.24.0
-    # via
-    #   emmet-core
-    #   pymatgen
-pycodestyle==2.11.0
-    # via flake8
-pycparser==2.21
-    # via cffi
-pydantic==1.10.12
-    # via
-    #   emmet-core
-    #   pymatgen
-pyflakes==3.1.0
-    # via flake8
-pygments==2.16.1
-    # via sphinx
-pyjwt[crypto]==2.8.0
-    # via globus-sdk
-pylint==2.17.5
-    # via matminer (setup.py)
-pymatgen==2023.9.2
-    # via
-    #   emmet-core
-    #   matminer (setup.py)
-    #   mp-api
-pymongo==4.5.0
-    # via matminer (setup.py)
-pyparsing==3.0.9
-    # via
-    #   httplib2
-    #   matplotlib
-pypif==2.1.2
-    # via citrination-client
-pyrsistent==0.19.3
-    # via jsonschema
-pytest==7.4.1
-    # via
-    #   matminer (setup.py)
-    #   pytest-cov
-    #   pytest-timeout
-pytest-cov==4.1.0
-    # via matminer (setup.py)
-pytest-timeout==2.1.0
-    # via matminer (setup.py)
-python-dateutil==2.8.2
-    # via
-    #   matplotlib
-    #   pandas
-pytz==2023.3.post1
-    # via pandas
-pyyaml==6.0.1
-    # via
-    #   citrination-client
-    #   pybtex
-requests==2.31.0
-    # via
-    #   citrination-client
-    #   coveralls
-    #   globus-sdk
-    #   matminer (setup.py)
-    #   mdf-forge
-    #   mdf-toolbox
-    #   mp-api
-    #   pymatgen
-    #   sphinx
-ruamel-yaml==0.17.32
-    # via pymatgen
-ruamel-yaml-clib==0.2.7
-    # via ruamel-yaml
-scikit-learn==1.3.0
-    # via
-    #   dscribe
-    #   matminer (setup.py)
-scipy==1.11.2
-    # via
-    #   ase
-    #   dscribe
-    #   pymatgen
-    #   scikit-learn
-    #   sparse
-six==1.16.0
-    # via
-    #   aflow
-    #   citrination-client
-    #   latexcodec
-    #   pybtex
-    #   pypif
-    #   python-dateutil
-snowballstemmer==2.2.0
-    # via sphinx
-soupsieve==2.5
-    # via beautifulsoup4
-sparse==0.14.0
-    # via dscribe
-spglib==2.0.2
-    # via
-    #   emmet-core
-    #   pymatgen
-sphinx==7.2.5
-    # via
-    #   matminer (setup.py)
-    #   sphinxcontrib-applehelp
-    #   sphinxcontrib-devhelp
-    #   sphinxcontrib-htmlhelp
-    #   sphinxcontrib-qthelp
-    #   sphinxcontrib-serializinghtml
-sphinxcontrib-applehelp==1.0.7
-    # via sphinx
-sphinxcontrib-devhelp==1.0.5
-    # via sphinx
-sphinxcontrib-htmlhelp==2.0.4
-    # via sphinx
-sphinxcontrib-jsmath==1.0.1
-    # via sphinx
-sphinxcontrib-qthelp==1.0.6
-    # via sphinx
-sphinxcontrib-serializinghtml==1.1.9
-    # via sphinx
-sympy==1.12
-    # via
-    #   matminer (setup.py)
-    #   pymatgen
-tabulate==0.9.0
-    # via pymatgen
-tenacity==8.2.3
-    # via plotly
-termcolor==2.3.0
-    # via aflow
-threadpoolctl==3.2.0
-    # via scikit-learn
-tomli==2.0.1
-    # via
-    #   black
-    #   coverage
-    #   pylint
-    #   pytest
-tomlkit==0.12.1
-    # via pylint
-tqdm==4.66.1
-    # via
-    #   matminer (setup.py)
-    #   mdf-forge
-    #   pymatgen
-typing-extensions==4.7.1
-    # via
-    #   astroid
-    #   emmet-core
-    #   mp-api
-    #   pydantic
-tzdata==2023.3
-    # via pandas
-ujson==5.8.0
-    # via matminer (setup.py)
-uncertainties==3.1.7
-    # via pymatgen
-urllib3==2.0.4
-    # via requests
-wrapt==1.15.0
-    # via astroid
-
-# The following packages are considered to be unsafe in a requirements file:
-# setuptools
diff --git a/requirements/macos-latest_py3.9.txt b/requirements/macos-latest_py3.9.txt
deleted file mode 100644
index b0cc9e97c..000000000
--- a/requirements/macos-latest_py3.9.txt
+++ /dev/null
@@ -1,365 +0,0 @@
-#
-# This file is autogenerated by pip-compile with Python 3.9
-# by the following command:
-#
-#    pip-compile --all-extras --output-file=requirements/macos-latest_py3.9_extras.txt
-#
-aflow==0.0.11
-    # via matminer (setup.py)
-alabaster==0.7.13
-    # via sphinx
-ase==3.22.1
-    # via
-    #   aflow
-    #   dscribe
-    #   matminer (setup.py)
-astroid==2.15.6
-    # via pylint
-attrs==23.1.0
-    # via
-    #   jsonschema
-    #   referencing
-babel==2.12.1
-    # via sphinx
-beautifulsoup4==4.12.2
-    # via aflow
-black==23.7.0
-    # via matminer (setup.py)
-certifi==2023.7.22
-    # via requests
-cffi==1.15.1
-    # via cryptography
-charset-normalizer==3.2.0
-    # via requests
-citrination-client==6.5.1
-    # via matminer (setup.py)
-click==8.1.7
-    # via black
-contourpy==1.1.0
-    # via matplotlib
-coverage[toml]==6.5.0
-    # via
-    #   coveralls
-    #   matminer (setup.py)
-    #   pytest-cov
-coveralls==3.3.1
-    # via matminer (setup.py)
-cryptography==41.0.3
-    # via
-    #   globus-sdk
-    #   pyjwt
-cycler==0.11.0
-    # via matplotlib
-dill==0.3.7
-    # via pylint
-dnspython==2.4.2
-    # via pymongo
-docopt==0.6.2
-    # via coveralls
-docutils==0.20.1
-    # via sphinx
-dscribe==2.1.0
-    # via matminer (setup.py)
-emmet-core==0.67.5
-    # via mp-api
-exceptiongroup==1.1.3
-    # via pytest
-fair-research-login==0.3.1
-    # via mdf-toolbox
-flake8==6.1.0
-    # via matminer (setup.py)
-fonttools==4.42.1
-    # via matplotlib
-future==0.18.3
-    # via
-    #   matminer (setup.py)
-    #   uncertainties
-globus-nexus-client==0.4.1
-    # via mdf-toolbox
-globus-sdk==3.28.0
-    # via
-    #   fair-research-login
-    #   globus-nexus-client
-    #   mdf-forge
-    #   mdf-toolbox
-httplib2==0.22.0
-    # via matminer (setup.py)
-idna==3.4
-    # via requests
-imagesize==1.4.1
-    # via sphinx
-importlib-metadata==6.8.0
-    # via sphinx
-importlib-resources==6.0.1
-    # via matplotlib
-iniconfig==2.0.0
-    # via pytest
-isort==5.12.0
-    # via pylint
-jinja2==3.1.2
-    # via
-    #   aflow
-    #   sphinx
-jmespath==1.0.1
-    # via matminer (setup.py)
-joblib==1.3.2
-    # via
-    #   dscribe
-    #   pymatgen
-    #   scikit-learn
-jsonschema==4.3.0
-    # via
-    #   matminer (setup.py)
-    #   mdf-toolbox
-kiwisolver==1.4.5
-    # via matplotlib
-latexcodec==2.0.1
-    # via pybtex
-lazy-object-proxy==1.9.0
-    # via astroid
-llvmlite==0.40.1
-    # via numba
-markupsafe==2.1.3
-    # via jinja2
-matplotlib==3.7.2
-    # via
-    #   ase
-    #   pymatgen
-mccabe==0.7.0
-    # via
-    #   flake8
-    #   pylint
-mdf-forge==0.8.0
-    # via matminer (setup.py)
-mdf-toolbox==0.6.0
-    # via mdf-forge
-monty==2023.9.5
-    # via
-    #   emmet-core
-    #   matminer (setup.py)
-    #   mp-api
-    #   pymatgen
-mp-api==0.35.1
-    # via pymatgen
-mpmath==1.3.0
-    # via sympy
-msgpack==1.0.5
-    # via mp-api
-mypy-extensions==1.0.0
-    # via black
-networkx==3.1
-    # via pymatgen
-numba==0.57.1
-    # via sparse
-numpy==1.24.4
-    # via
-    #   aflow
-    #   ase
-    #   contourpy
-    #   dscribe
-    #   matminer (setup.py)
-    #   matplotlib
-    #   numba
-    #   pandas
-    #   pymatgen
-    #   pypif
-    #   scikit-learn
-    #   scipy
-    #   sparse
-    #   spglib
-packaging==23.1
-    # via
-    #   black
-    #   matplotlib
-    #   plotly
-    #   pytest
-    #   sphinx
-palettable==3.3.3
-    # via pymatgen
-pandas==2.1.0
-    # via
-    #   matminer (setup.py)
-    #   pymatgen
-pathspec==0.11.2
-    # via black
-pillow==10.0.0
-    # via matplotlib
-platformdirs==3.10.0
-    # via
-    #   black
-    #   pylint
-plotly==5.16.1
-    # via pymatgen
-pluggy==1.3.0
-    # via pytest
-pybind11==2.11.1
-    # via dscribe
-pybtex==0.24.0
-    # via
-    #   emmet-core
-    #   pymatgen
-pycodestyle==2.11.0
-    # via flake8
-pycparser==2.21
-    # via cffi
-pydantic==1.10.12
-    # via
-    #   emmet-core
-    #   pymatgen
-pyflakes==3.1.0
-    # via flake8
-pygments==2.16.1
-    # via sphinx
-pyjwt[crypto]==2.8.0
-    # via globus-sdk
-pylint==2.17.5
-    # via matminer (setup.py)
-pymatgen==2023.9.2
-    # via
-    #   emmet-core
-    #   matminer (setup.py)
-    #   mp-api
-pymongo==4.5.0
-    # via matminer (setup.py)
-pyparsing==3.0.9
-    # via
-    #   httplib2
-    #   matplotlib
-pypif==2.1.2
-    # via citrination-client
-pyrsistent==0.19.3
-    # via jsonschema
-pytest==7.4.1
-    # via
-    #   matminer (setup.py)
-    #   pytest-cov
-    #   pytest-timeout
-pytest-cov==4.1.0
-    # via matminer (setup.py)
-pytest-timeout==2.1.0
-    # via matminer (setup.py)
-python-dateutil==2.8.2
-    # via
-    #   matplotlib
-    #   pandas
-pytz==2023.3.post1
-    # via pandas
-pyyaml==6.0.1
-    # via
-    #   citrination-client
-    #   pybtex
-requests==2.31.0
-    # via
-    #   citrination-client
-    #   coveralls
-    #   globus-sdk
-    #   matminer (setup.py)
-    #   mdf-forge
-    #   mdf-toolbox
-    #   mp-api
-    #   pymatgen
-    #   sphinx
-ruamel-yaml==0.17.32
-    # via pymatgen
-ruamel-yaml-clib==0.2.7
-    # via ruamel-yaml
-scikit-learn==1.3.0
-    # via
-    #   dscribe
-    #   matminer (setup.py)
-scipy==1.11.2
-    # via
-    #   ase
-    #   dscribe
-    #   pymatgen
-    #   scikit-learn
-    #   sparse
-six==1.16.0
-    # via
-    #   aflow
-    #   citrination-client
-    #   latexcodec
-    #   pybtex
-    #   pypif
-    #   python-dateutil
-snowballstemmer==2.2.0
-    # via sphinx
-soupsieve==2.5
-    # via beautifulsoup4
-sparse==0.14.0
-    # via dscribe
-spglib==2.0.2
-    # via
-    #   emmet-core
-    #   pymatgen
-sphinx==7.2.5
-    # via
-    #   matminer (setup.py)
-    #   sphinxcontrib-applehelp
-    #   sphinxcontrib-devhelp
-    #   sphinxcontrib-htmlhelp
-    #   sphinxcontrib-qthelp
-    #   sphinxcontrib-serializinghtml
-sphinxcontrib-applehelp==1.0.7
-    # via sphinx
-sphinxcontrib-devhelp==1.0.5
-    # via sphinx
-sphinxcontrib-htmlhelp==2.0.4
-    # via sphinx
-sphinxcontrib-jsmath==1.0.1
-    # via sphinx
-sphinxcontrib-qthelp==1.0.6
-    # via sphinx
-sphinxcontrib-serializinghtml==1.1.9
-    # via sphinx
-sympy==1.12
-    # via
-    #   matminer (setup.py)
-    #   pymatgen
-tabulate==0.9.0
-    # via pymatgen
-tenacity==8.2.3
-    # via plotly
-termcolor==2.3.0
-    # via aflow
-threadpoolctl==3.2.0
-    # via scikit-learn
-tomli==2.0.1
-    # via
-    #   black
-    #   coverage
-    #   pylint
-    #   pytest
-tomlkit==0.12.1
-    # via pylint
-tqdm==4.66.1
-    # via
-    #   matminer (setup.py)
-    #   mdf-forge
-    #   pymatgen
-typing-extensions==4.7.1
-    # via
-    #   astroid
-    #   black
-    #   emmet-core
-    #   globus-sdk
-    #   mp-api
-    #   pydantic
-    #   pylint
-tzdata==2023.3
-    # via pandas
-ujson==5.8.0
-    # via matminer (setup.py)
-uncertainties==3.1.7
-    # via pymatgen
-urllib3==2.0.4
-    # via requests
-wrapt==1.15.0
-    # via astroid
-zipp==3.16.2
-    # via
-    #   importlib-metadata
-    #   importlib-resources
-
-# The following packages are considered to be unsafe in a requirements file:
-# setuptools
diff --git a/requirements/ubuntu-latest_py3.10.txt b/requirements/ubuntu-latest_py3.10.txt
index 8a6c4ac8a..e3a1d42ab 100644
--- a/requirements/ubuntu-latest_py3.10.txt
+++ b/requirements/ubuntu-latest_py3.10.txt
@@ -159,7 +159,7 @@ packaging==24.0
     #   sphinx
 palettable==3.3.3
     # via pymatgen
-pandas==1.5.3
+pandas==2.2.1
     # via
     #   matminer (setup.py)
     #   pymatgen
diff --git a/requirements/ubuntu-latest_py3.11.txt b/requirements/ubuntu-latest_py3.11.txt
index 26af6a703..af2c2ba8b 100644
--- a/requirements/ubuntu-latest_py3.11.txt
+++ b/requirements/ubuntu-latest_py3.11.txt
@@ -157,7 +157,7 @@ packaging==24.0
     #   sphinx
 palettable==3.3.3
     # via pymatgen
-pandas==1.5.3
+pandas==2.2.1
     # via
     #   matminer (setup.py)
     #   pymatgen
diff --git a/requirements/ubuntu-latest_py3.9.txt b/requirements/ubuntu-latest_py3.9.txt
index 8d05b6369..82c5a0e30 100644
--- a/requirements/ubuntu-latest_py3.9.txt
+++ b/requirements/ubuntu-latest_py3.9.txt
@@ -163,7 +163,7 @@ packaging==24.0
     #   sphinx
 palettable==3.3.3
     # via pymatgen
-pandas==1.5.3
+pandas==2.2.1
     # via
     #   matminer (setup.py)
     #   pymatgen