Repository: MAVENSDC/cdflib

Issue:
Release a version compatible with numpy >2.0.0rc1
:wave: hello,

As you may or may not be aware numpy 2.0 is an upcoming major release, SunPy has recently gone through and made our code 2.0 compatible in preparation for a compatible release, and we found that cdflib wouldn't work with 2.0.

Thanks!

Discussion:
Cadair: @bryan-harter Thanks for the review :smile:

Do you have any idea about the online build where it's downloading data?
bryan-harter: > @bryan-harter Thanks for the review 😄
> 
> Do you have any idea about the online build where it's downloading data?

Yeah sorry I was trying to figure out what tox was so I could try to run the tests its running locally.  Even though it's in the project, I wasn't super familiar with it, but it seems handy and I'd be happy if this is how we run the unit tests in the future!  I'm going to try checking out this commit and figure out why its throwing errors.  
bryan-harter: So two things so far:
1) For unit tests we'll need netcdf4 installed, so we'll need to add that to the [test] installs in setup.cfg
2) At least one of the unit tests was failing due to a bug which was just fixed in main, so at least that one will be fixed if you pull main back into this branch

Theres still another bug that I'm working, but just in case I don't get to it within the next day or two, that's what I have so far

Patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index b1b113c..674fa41 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -2,30 +2,36 @@ name: Run tests
 
 on:
   push:
-    branches: [ main ]
+    branches:
+      - 'main'
   pull_request:
-    branches: [ main ]
+  # Allow manual runs through the web UI
+  workflow_dispatch:
 
-jobs:
-  tests:
-    runs-on: ${{ matrix.os }}
-    strategy:
-      matrix:
-        os: [ubuntu-latest, windows-latest, macos-latest]
-        python-version: ['3.9', '3.10', '3.11']
-    steps:
-    - uses: actions/checkout@v3
-
-    - uses: actions/setup-python@v4
-      with:
-        python-version: ${{ matrix.python-version }}
-    - name: Install package and test requirements
-      run: |
-        pip install -e .[tests]
-        pip check
+# Only allow one run per git ref at a time
+concurrency:
+  group: '${{ github.workflow }}-${{ github.ref }}'
+  cancel-in-progress: true
 
-    - name: Run tests
-      run: pytest --cov=cdflib
-
-    - name: Upload code coverage
-      uses: codecov/codecov-action@v3
+jobs:
+  core:
+    uses: OpenAstronomy/github-actions-workflows/.github/workflows/tox.yml@v1
+    with:
+      submodules: false
+      coverage: codecov
+      envs: |
+        - linux: py38-oldestdeps
+        - linux: py39-online
+        #- linux: py310
+        #- linux: py311
+        - linux: py312-devdeps
+        #- windows: py38
+        #- windows: py39
+        - windows: py310
+        #- windows: py311
+        #- windows: py312
+        #- macos: py38
+        #- macos: py39
+        #- macos: py310
+        - macos: py311
+        #- macos: py312
diff --git a/cdflib/cdfread.py b/cdflib/cdfread.py
index ced85ee..e3669b4 100644
--- a/cdflib/cdfread.py
+++ b/cdflib/cdfread.py
@@ -1739,7 +1739,7 @@ class CDF:
 
         # Put the data into system byte order
         if self._convert_option() != "=":
-            ret = ret.byteswap().newbyteorder()
+            ret = ret.view(ret.dtype.newbyteorder()).byteswap()
 
         if self._majority == "Column_major":
             if dimensions is not None:
diff --git a/cdflib/cdfwrite.py b/cdflib/cdfwrite.py
index 90897bd..213c5f9 100644
--- a/cdflib/cdfwrite.py
+++ b/cdflib/cdfwrite.py
@@ -2285,7 +2285,6 @@ class CDF:
         odata : byte stream
             The stream of bytes to write to the CDF file
         """
-
         recSize = self._datatype_size(data_type, num_elems) * num_values
         # List or Tuple data
         if isinstance(indata, list) or isinstance(indata, tuple):
@@ -2393,10 +2392,13 @@ class CDF:
                         complex_data.append(indata.imag)
                     indata = complex_data
                 form = tofrom + str(recs * num_values * num_elems) + dt_string
-                if recs * num_values * num_elems > 1:
-                    return recs, struct.pack(form, *indata)
-                else:
-                    return recs, struct.pack(form, indata)
+                try:
+                    if recs * num_values * num_elems > 1:
+                        return recs, struct.pack(form, *indata)
+                    else:
+                        return recs, struct.pack(form, indata)
+                except struct.error:
+                    raise ValueError("Unable to convert data to CDF format, data " "object cannot be of type string.")
 
     def _num_values(self, zVar: bool, varNum: int) -> int:
         """
diff --git a/cdflib/xarray/xarray_to_cdf.py b/cdflib/xarray/xarray_to_cdf.py
index 5175591..10d9465 100644
--- a/cdflib/xarray/xarray_to_cdf.py
+++ b/cdflib/xarray/xarray_to_cdf.py
@@ -164,7 +164,7 @@ def _dtype_to_cdf_type(var: xr.DataArray, terminate_on_warning: bool = False) ->
         cdf_data_type = "CDF_UINT4"
     elif numpy_data_type == np.uint64:
         cdf_data_type = "CDF_UINT8"
-    elif numpy_data_type == np.complex_:
+    elif numpy_data_type == np.complex128:
         cdf_data_type = "CDF_EPOCH16"
     elif numpy_data_type.type in (np.str_, np.bytes_):
         element_size = int(numpy_data_type.str[2:])  # The length of the longest string in the numpy array
diff --git a/setup.cfg b/setup.cfg
index b6258b1..c2f4551 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -15,10 +15,11 @@ classifiers =
   Environment :: Console
   Intended Audience :: Science/Research
   Operating System :: OS Independent
-  Programming Language :: Python :: 3.6
-  Programming Language :: Python :: 3.7
   Programming Language :: Python :: 3.8
   Programming Language :: Python :: 3.9
+  Programming Language :: Python :: 3.10
+  Programming Language :: Python :: 3.11
+  Programming Language :: Python :: 3.12
   Programming Language :: Python :: 3 :: Only
   Topic :: Utilities
 license_file = LICENSE
@@ -26,11 +27,11 @@ long_description = file: README.md
 long_description_content_type = text/markdown
 
 [options]
-python_requires = >= 3.6
+python_requires = >= 3.8
 include_package_data = True
 packages = find:
 install_requires =
-  numpy
+  numpy >= 1.21
 
 [options.extras_require]
 tests =
@@ -40,6 +41,7 @@ tests =
   pytest-cov
   pytest-remotedata
   xarray
+  h5netcdf
   netcdf4
 docs =
   astropy
diff --git a/tox.ini b/tox.ini
index 3a67265..05820eb 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,8 +1,69 @@
 [tox]
-envlist = py36, py37, py38, py39, py310
+min_version = 4.0
+envlist =
+    py{38,39,310,311,312}{,-online}
+    py38-oldestdeps
+    py312-devdeps
+    build_docs
 
 [testenv]
-setenv =
-    PYTHONPATH = {toxinidir}
+# tox environments are constructed with so-called 'factors' (or terms)
+# separated by hyphens, e.g. test-devdeps-cov. Lines below starting with factor:
+# will only take effect if that factor is included in the environment name. To
+# see a list of example environments that can be run, along with a description,
+# run:
+#
+#     tox -l -v
+#
+description =
+    run tests
+    oldestdeps: with the oldest supported version of key dependencies
+    devdeps: with the latest developer version of key dependencies
+    online: that require remote data (as well as the offline ones)
 
-commands = python setup.py test
+# Run the tests in a temporary directory to not pollute the working directory with files or other downloads
+changedir = .tmp/{envname}
+
+# Pass through the following environment variables which may be needed for the CI
+pass_env =
+    # A variable to tell tests we are on a CI system
+    CI
+    # Custom compiler locations (such as ccache)
+    CC
+    # Location of locales (needed by sphinx on some systems)
+    LOCALE_ARCHIVE
+    # If the user has set a LC override we should follow it
+    # (note LANG is automatically passed through by tox)
+    LC_ALL
+
+set_env =
+    PYTEST_COMMAND = pytest --cov=cdflib {toxinidir}/tests/ {toxinidir}/doc/
+    devdeps: PIP_EXTRA_INDEX_URL = https://pypi.anaconda.org/astropy/simple https://pypi.anaconda.org/scientific-python-nightly-wheels/simple
+
+deps =
+    oldestdeps: minimum_dependencies
+    devdeps: astropy>=0.0.dev0
+    devdeps: numpy>=0.0.dev0
+    devdeps: pandas>=0.0.dev0  # needed to prevent pandas pulling in numpy<2, this can be removed after pandas does a release supporting 2.0
+
+# The following indicates which extras_require from setup.cfg will be installed
+extras =
+    tests
+
+commands_pre =
+    oldestdeps: minimum_dependencies cdflib --filename requirements-min.txt
+    oldestdeps: pip install -r requirements-min.txt
+    pip freeze
+
+commands =
+    !online: {env:PYTEST_COMMAND} {posargs}
+    online: {env:PYTEST_COMMAND} --remote-data=any {posargs}
+
+[testenv:build_docs]
+description = invoke sphinx-build to build the HTML docs
+change_dir =
+    doc
+extras =
+    docs
+commands =
+    sphinx-build -j auto --color -W --keep-going -b html -d _build/.doctrees . _build/html {posargs}


Test patch:
diff --git a/tests/test_cdfwrite.py b/tests/test_cdfwrite.py
index 33ad087..6e44b5e 100755
--- a/tests/test_cdfwrite.py
+++ b/tests/test_cdfwrite.py
@@ -2,6 +2,7 @@ from pathlib import Path
 from typing import Any, Dict
 
 import numpy as np
+import pytest
 
 from cdflib import cdfread, cdfwrite
 
@@ -630,3 +631,11 @@ def test_create_zvariables_with_data_to_convert(tmp_path):
     # Test CDF info
     var = reader.varget("Variable1")
     assert var[3] == 3
+
+
+def test_convert_data_error(tmp_path):
+    indata = int(-9223372036854775808)
+    cdf = cdfwrite.CDF(tmp_path / "test.cdf", cdf_spec={"rDim_sizes": [1]})
+    with pytest.raises(ValueError):
+        # Data from list of strings with dimension "epoch"
+        cdf._convert_data(51, 1, 1, indata)