Repository: astropy/astropy

Issue:
Going forward after numpy 2.0's change of copy semantics
This issue is to track and discuss the follow up of #16142

Here's the context in a nutshell:
- up to now astropy has been following numpy's semantics regarding the meaning of the `copy` keyword argument in `numpy.array`'s signature
- numpy 2.0 *changes* this semantic (see https://github.com/numpy/numpy/pull/25168)

Here's a summary of old/new behaviors

|  `copy`  | numpy 1.x                                     | numpy 2.0    |
|----------|-----------------------------------------------|--------------|
|  `True`  | always copy                                   | always copy (unchanged)                                                  |
|  `False` | copy if and only if it cannot be avoided      | do not perform a copy, raise an error if it cannot be avoided            |
|  `None`  |  `ValueError` |   copy if and only if it cannot be avoided (same a `False` in numpy 1.x) |


It follows that we cannot keep both the following implicit, and now conflicting, promises 
1) future astropy APIs (e.g. `Quantity(..., copy=False)`) will always match their past behavior
2) astropy APIs follow numpy's

So we need to choose which one we care about most, and apply necessary transformations in that direction.
Another, equivalent, way to think about this choice is whether, going forward, we should support numpy 1 semantics or numpy 2 semantics.

I'm personally inclined to think `2)` (follow numpy 2.0) should be preferred because
- the new numpy API offers more flexibility to users (or in other words, numpy 1.x is too restrictive)
- at some point in the future, no one will be running numpy 1.x any more so matching *that* API is guaranteed to surprise some users

So, assuming my preference is also consensual, let me draft out a plan for this pivoting. In the following,
I'll be referring to compatibility shims introduced in #16142 as "band aids"
1) make the band aids redundant by refactoring our internal usage of `numpy.array(..., copy=False)` (this was originally part of #16142, and is re-rolled in #16166). I think this part is probably non-controversial as long as we don't attempt to backport it, so as it stands, it a pure refactor for astropy 6.1.0. This is needed in preparation to step 2 (avoid self-triggered warnings)
2) deprecate band aids: warn users that `copy=False` will raise an error in the future if a copy cannot be avoided. The details of this deprecation will require further discussion[^1], but I think we can start working on it as soon as #16166 is dealt with.
3) eventually, remove all band aids (this would be a major API change, so it can only happen in a x.0 release ?)


This discussion originated in https://github.com/astropy/astropy/pull/16142#pullrequestreview-1914490965 and https://github.com/astropy/astropy/pull/16142#issuecomment-1977156309

[^1]: when should it end ? do we want to warn all users or just the ones that are already on numpy >= 2.0 ? should we support some way to opt-in future behavior, and how ?



Discussion:

Patch:
diff --git a/astropy/coordinates/baseframe.py b/astropy/coordinates/baseframe.py
index fbb935a308..658474b3a3 100644
--- a/astropy/coordinates/baseframe.py
+++ b/astropy/coordinates/baseframe.py
@@ -352,7 +352,14 @@ class BaseCoordinateFrame(ShapedLikeNDArray):
 
         # Broadcast the data if necessary and set it
         if data is not None and data.shape != self._shape:
-            data = data._apply(np.broadcast_to, shape=self._shape, subok=True)
+            try:
+                # if broadcasting isn't strictly needed, avoid it
+                # see https://github.com/astropy/astropy/issues/16219
+                data = data.reshape(self._shape)
+            except Exception:
+                data = data._apply(np.broadcast_to, shape=self._shape, subok=True)
+                if copy:
+                    data = data.copy()
         self._data = data
         # Broadcast the attributes if necessary by getting them again
         # (we now know the shapes will be OK).
diff --git a/astropy/coordinates/matching.py b/astropy/coordinates/matching.py
index aa219ab69e..585f2cb28f 100644
--- a/astropy/coordinates/matching.py
+++ b/astropy/coordinates/matching.py
@@ -455,23 +455,11 @@ def _get_cartesian_kdtree(coord, attrname_or_kdt="kdtree", forceunit=None):
 
     Returns
     -------
-    kdt : `~scipy.spatial.cKDTree` or `~scipy.spatial.KDTree`
+    kdt : `~scipy.spatial.KDTree`
         The KD-Tree representing the 3D cartesian representation of the input
         coordinates.
     """
-    from warnings import warn
-
-    # without scipy this will immediately fail
-    from scipy import spatial
-
-    try:
-        KDTree = spatial.cKDTree
-    except Exception:
-        warn(
-            "C-based KD tree not found, falling back on (much slower) "
-            "python implementation"
-        )
-        KDTree = spatial.KDTree
+    from scipy.spatial import KDTree
 
     if attrname_or_kdt is True:  # backwards compatibility for pre v0.4
         attrname_or_kdt = "kdtree"
@@ -503,15 +491,9 @@ def _get_cartesian_kdtree(coord, attrname_or_kdt="kdtree", forceunit=None):
         # There should be no NaNs in the kdtree data.
         if np.isnan(flatxyz.value).any():
             raise ValueError("Catalog coordinates cannot contain NaN entries.")
-        try:
-            # Set compact_nodes=False, balanced_tree=False to use
-            # "sliding midpoint" rule, which is much faster than standard for
-            # many common use cases
-            kdt = KDTree(flatxyz.value.T, compact_nodes=False, balanced_tree=False)
-        except TypeError:
-            # Python implementation does not take compact_nodes and balanced_tree
-            # as arguments.  However, it uses sliding midpoint rule by default
-            kdt = KDTree(flatxyz.value.T)
+        # Not obvious if compact_nodes=False, balanced_tree=False is still needed but
+        # we stay backwards-compatible with previous versions of `astropy` for now.
+        kdt = KDTree(flatxyz.value.T, compact_nodes=False, balanced_tree=False)
 
     if attrname_or_kdt:
         # cache the kdtree in `coord`
diff --git a/astropy/io/votable/tree.py b/astropy/io/votable/tree.py
index 0a1c2fdfd4..bec5411555 100644
--- a/astropy/io/votable/tree.py
+++ b/astropy/io/votable/tree.py
@@ -2740,7 +2740,10 @@ class TableElement(
                 try:
                     colnumbers = [names.index(x) for x in columns]
                 except ValueError:
-                    raise ValueError(f"Columns '{columns}' not found in fields list")
+                    missing_columns = [name for name in columns if name not in names]
+                    raise ValueError(
+                        f"Columns {missing_columns} were not found in fields list"
+                    ) from None
             else:
                 raise TypeError("Invalid columns list")
 
diff --git a/astropy/table/operations.py b/astropy/table/operations.py
index 8da3f6cf4a..95a9ddb6dd 100644
--- a/astropy/table/operations.py
+++ b/astropy/table/operations.py
@@ -234,7 +234,7 @@ def join_distance(distance, kdtree_args=None, query_args=None):
     "fuzzy" match can apply to 1-D or 2-D columns, where in the latter case
     the distance is a vector distance.
 
-    The distance cross-matching is done using `scipy.spatial.cKDTree`. If
+    The distance cross-matching is done using `scipy.spatial.KDTree`. If
     necessary you can tweak the default behavior by providing ``dict`` values
     for the ``kdtree_args`` or ``query_args``.
 
@@ -243,9 +243,9 @@ def join_distance(distance, kdtree_args=None, query_args=None):
     distance : float or `~astropy.units.Quantity` ['length']
         Maximum distance between points to be considered a join match
     kdtree_args : dict, None
-        Optional extra args for `~scipy.spatial.cKDTree`
+        Optional extra args for `~scipy.spatial.KDTree`
     query_args : dict, None
-        Optional extra args for `~scipy.spatial.cKDTree.query_ball_tree`
+        Optional extra args for `~scipy.spatial.KDTree.query_ball_tree`
 
     Returns
     -------
@@ -275,7 +275,7 @@ def join_distance(distance, kdtree_args=None, query_args=None):
 
     """
     try:
-        from scipy.spatial import cKDTree
+        from scipy.spatial import KDTree
     except ImportError as exc:
         raise ImportError("scipy is required to use join_distance()") from exc
 
@@ -306,8 +306,8 @@ def join_distance(distance, kdtree_args=None, query_args=None):
             col2.shape = col2.shape + (1,)
 
         # Cross-match col1 and col2 within dist using KDTree
-        kd1 = cKDTree(col1, **kdtree_args)
-        kd2 = cKDTree(col2, **kdtree_args)
+        kd1 = KDTree(col1, **kdtree_args)
+        kd2 = KDTree(col2, **kdtree_args)
         nears = kd1.query_ball_tree(kd2, r=dist, **query_args)
 
         # Output of above is nears which is a list of lists, where the outer
diff --git a/docs/changes/16252.other.rst b/docs/changes/16252.other.rst
new file mode 100644
index 0000000000..8b6ea58907
--- /dev/null
+++ b/docs/changes/16252.other.rst
@@ -0,0 +1,3 @@
+astropy is now compiled against NumPy 2.0, enabling runtime compatibility
+with this new major release. Compatibility with NumPy 1.23 and newer
+versions of NumPy 1.x is preserved through this change.
diff --git a/docs/changes/coordinates/16241.bugfix.rst b/docs/changes/coordinates/16241.bugfix.rst
new file mode 100644
index 0000000000..fe07d21d45
--- /dev/null
+++ b/docs/changes/coordinates/16241.bugfix.rst
@@ -0,0 +1,2 @@
+Fix a bug where ``SkyCoord.spherical_offsets_by`` would crash when a wrap
+was needed.
diff --git a/docs/changes/io.votable/15956.bugfix.rst b/docs/changes/io.votable/15956.bugfix.rst
new file mode 100644
index 0000000000..87c5f4da14
--- /dev/null
+++ b/docs/changes/io.votable/15956.bugfix.rst
@@ -0,0 +1,3 @@
+When reading a VOTable, if some user-requested columns were not present then the
+resulting error message previously listed all the requested column names.
+Now only columns that are actually missing are shown.
diff --git a/docs/table/operations.rst b/docs/table/operations.rst
index 5e494569f5..85e9a4835a 100644
--- a/docs/table/operations.rst
+++ b/docs/table/operations.rst
@@ -1186,8 +1186,8 @@ a generic distance between column values using the
 :func:`~astropy.table.join_distance` join function. This can apply to 1D or 2D
 (vector) columns. This will look very similar to the coordinates example, but
 here there is a bit more flexibility. The matching is done using
-:class:`scipy.spatial.cKDTree` and
-:meth:`scipy.spatial.cKDTree.query_ball_tree`, and the behavior of these can be
+:class:`scipy.spatial.KDTree` and
+:meth:`scipy.spatial.KDTree.query_ball_tree`, and the behavior of these can be
 controlled via the ``kdtree_args`` and ``query_args`` arguments, respectively.
 
 .. _unique-rows:
diff --git a/pyproject.toml b/pyproject.toml
index 9d88f86b09..0cb5dfe389 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -41,7 +41,7 @@ keywords = [
 dependencies = [
     "numpy>=1.23",
     "pyerfa>=2.0.1.1",
-    "astropy-iers-data>=0.2024.2.26.0.28.55",
+    "astropy-iers-data>=0.2024.4.1.0.33.14",
     "PyYAML>=3.13",
     "packaging>=19.0",
 ]
@@ -126,7 +126,7 @@ wcslint = "astropy.wcs.wcslint:main"
 requires = ["setuptools",
             "setuptools_scm>=6.2",
             "cython>=3.0.0,<3.1.0",
-            "numpy>=1.25",
+            "numpy>=2.0.0rc1", # see https://github.com/astropy/astropy/issues/16257
             "extension-helpers==1.*"]
 build-backend = "setuptools.build_meta"
 


Test patch:
diff --git a/astropy/coordinates/tests/test_frames.py b/astropy/coordinates/tests/test_frames.py
index 3a9aa62ba4..c42d6f65b6 100644
--- a/astropy/coordinates/tests/test_frames.py
+++ b/astropy/coordinates/tests/test_frames.py
@@ -1690,3 +1690,14 @@ def test_spherical_offsets_by_broadcast():
     assert SkyCoord(
         ra=np.array([123, 134, 145]), dec=np.array([45, 56, 67]), unit=u.deg
     ).spherical_offsets_by(2 * u.deg, 2 * u.deg).shape == (3,)
+
+
+@pytest.mark.parametrize("shape", [(1,), (2,)])
+def test_spherical_offsets_with_wrap(shape):
+    # see https://github.com/astropy/astropy/issues/16219
+    sc = SkyCoord(ra=np.broadcast_to(123.0, shape), dec=90.0, unit=u.deg)
+    scop = sc.spherical_offsets_by(+2 * u.deg, 0 * u.deg)
+    assert scop.shape == shape
+
+    scom = sc.spherical_offsets_by(-2 * u.deg, 0 * u.deg)
+    assert scom.shape == shape
diff --git a/astropy/coordinates/tests/test_matching.py b/astropy/coordinates/tests/test_matching.py
index b642cac437..91b6731a6c 100644
--- a/astropy/coordinates/tests/test_matching.py
+++ b/astropy/coordinates/tests/test_matching.py
@@ -119,20 +119,6 @@ def test_kdtree_storage(functocheck, args, defaultkdtname, bothsaved):
     assert "KD" in e.value.args[0]
 
 
-@pytest.mark.skipif(not HAS_SCIPY, reason="Requires scipy.")
-def test_python_kdtree(monkeypatch):
-    from astropy.coordinates import ICRS
-
-    cmatch = ICRS([4, 2.1] * u.degree, [0, 0] * u.degree, distance=[1, 2] * u.kpc)
-    ccatalog = ICRS(
-        [1, 2, 3, 4] * u.degree, [0, 0, 0, 0] * u.degree, distance=[1, 2, 3, 4] * u.kpc
-    )
-
-    monkeypatch.delattr("scipy.spatial.cKDTree")
-    with pytest.warns(UserWarning, match=r"C-based KD tree not found"):
-        matching.match_coordinates_sky(cmatch, ccatalog)
-
-
 @pytest.mark.skipif(not HAS_SCIPY, reason="Requires scipy.")
 def test_matching_method():
     from astropy.coordinates import ICRS, SkyCoord
diff --git a/astropy/io/votable/tests/test_vo.py b/astropy/io/votable/tests/test_vo.py
index fae7ac66d2..2fbde75df1 100644
--- a/astropy/io/votable/tests/test_vo.py
+++ b/astropy/io/votable/tests/test_vo.py
@@ -8,6 +8,7 @@ import difflib
 import gzip
 import io
 import pathlib
+import re
 import sys
 from unittest import mock
 
@@ -252,6 +253,27 @@ class TestReferences:
         assert len(list(self.votable.iter_coosys())) == 1
 
 
+@pytest.mark.parametrize(
+    "columns, expected_missing",
+    [
+        # a single non-existent column
+        pytest.param(["c1"], ["c1"], id="basic"),
+        # multiple missing columns (checking that order is preserved)
+        pytest.param(["c1", "c2", "c3"], ["c1", "c2", "c3"], id="check-ordering"),
+        # mixing existing with missing columns
+        pytest.param(["c1", "string_test", "c2"], ["c1", "c2"], id="list-only-missing"),
+    ],
+)
+def test_select_missing_columns_error_message(columns, expected_missing):
+    # see https://github.com/astropy/astropy/pull/15956
+    filename = get_pkg_data_filename("data/regression.xml")
+    with pytest.raises(
+        ValueError,
+        match=re.escape(f"Columns {expected_missing!r} were not found in fields list"),
+    ):
+        parse_single_table(filename, columns=columns)
+
+
 def test_select_columns_by_index():
     columns = [0, 5, 13]
     table = parse(