Issue 1:
Describe your issue.

This comes from failures seen in SymPy CI:

sympy/sympy#23847

The release of Cython 0.29.31 means that building SciPy from source with pip fails both with the latest release SciPy 1.8.1 and also the main branch.

It looks like pyproject.toml on the 1.8.x branch needs to have a better upper bound for the Cython version since it only requires <3.0 which does not prevent new releases in Cython's 0.29.x series from breaking the build:

scipy/pyproject.toml

Lines 1 to 14
in
b76137f

# Note that in maintenance branches, all build dependencies should

# have an upper bound equal to the most recent already-released version

# of the dependency. This to prevent that a future backwards-incompatible

# release will break the source build of a SciPy release.

# Do accept micro (bug-fix) releases. So for example, if pybind11 2.4.3 is

# the most recent version on PyPI:

#

# "pybind11>=2.4.3,<2.5.0",

[build-system]

requires = [

"wheel<0.38.0",

"setuptools<60.0.0",

"Cython>=0.29.18,<3.0",

Reproducing Code Example

$ pip install -U pip setuptools wheel cython numpy
Collecting pip
Downloading pip-22.2.1-py3-none-any.whl (2.0 MB)
|████████████████████████████████| 2.0 MB 4.8 MB/s
Collecting setuptools
Downloading setuptools-63.2.0-py3-none-any.whl (1.2 MB)
|████████████████████████████████| 1.2 MB 28.2 MB/s
Collecting wheel
Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Collecting cython
Downloading Cython-0.29.31-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
|████████████████████████████████| 1.9 MB 2.8 MB/s
Collecting numpy
Using cached numpy-1.23.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
Installing collected packages: pip, setuptools, wheel, cython, numpy
Attempting uninstall: pip
Found existing installation: pip 20.2.3
Uninstalling pip-20.2.3:
Successfully uninstalled pip-20.2.3
Attempting uninstall: setuptools
Found existing installation: setuptools 49.2.1
Uninstalling setuptools-49.2.1:
Successfully uninstalled setuptools-49.2.1
Successfully installed cython-0.29.31 numpy-1.23.1 pip-22.2.1 setuptools-63.2.0 wheel-0.37.1
$ pip install --no-binary scipy scipy
Collecting scipy
Downloading scipy-1.8.1.tar.gz (38.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.2/38.2 MB 8.0 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error

Error message

$ pip install --no-binary scipy scipy
Collecting scipy
Downloading scipy-1.8.1.tar.gz (38.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.2/38.2 MB 8.0 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [102 lines of output]
Running scipy/special/_generate_pyx.py
Running scipy/stats/_generate_pyx.py
Running scipy/linalg/_generate_pyx.py
Processing scipy/special/_ufuncs.pyx
Processing scipy/special/_test_round.pyx
Processing scipy/special/_ufuncs_cxx.pyx
Processing scipy/special/_ellip_harm_2.pyx
Processing scipy/special/cython_special.pyx
Processing scipy/special/_comb.pyx
Processing scipy/spatial/_voronoi.pyx
Processing scipy/spatial/_hausdorff.pyx
Processing scipy/spatial/_qhull.pyx
Processing scipy/spatial/_ckdtree.pyx
Processing scipy/spatial/transform/_rotation.pyx
Processing scipy/interpolate/_ppoly.pyx
Processing scipy/interpolate/_bspl.pyx
Processing scipy/interpolate/interpnd.pyx
Processing scipy/cluster/_vq.pyx
Processing scipy/cluster/_hierarchy.pyx
Processing scipy/cluster/_optimal_leaf_ordering.pyx
Processing scipy/_lib/messagestream.pyx
Processing scipy/_lib/_test_deprecation_call.pyx
Processing scipy/_lib/_test_deprecation_def.pyx
Processing scipy/_lib/_ccallback_c.pyx
Processing scipy/signal/_upfirdn_apply.pyx
Processing scipy/signal/_max_len_seq_inner.pyx
Processing scipy/signal/_sosfilt.pyx
Processing scipy/signal/_spectral.pyx
Processing scipy/signal/_peak_finding_utils.pyx
Processing scipy/io/matlab/_streams.pyx
Processing scipy/io/matlab/_mio5_utils.pyx
Processing scipy/io/matlab/_mio_utils.pyx
Processing scipy/stats/_stats.pyx
Processing scipy/stats/_qmc_cy.pyx
Processing scipy/stats/_biasedurn.pyx
Processing scipy/stats/_sobol.pyx
Processing scipy/stats/_boost/src/nbinom_ufunc.pyx
Processing scipy/stats/_boost/src/beta_ufunc.pyx

Error compiling Cython file:
------------------------------------------------------------
...
return np.array(result, dtype=np.int64)

@cython.wraparound(False)
@cython.boundscheck(False)
def _weightedrankedtau(ordered[:] x, ordered[:] y, intp_t[:] rank, weigher, bool additive):
^
------------------------------------------------------------

_stats.pyx:173:0: Referring to a memoryview typed argument directly in a nested closure function is not supported in Cython 0.x. Either upgrade to Cython 3, or assign the argument to a local variable and use that in the nested function.
Processing scipy/stats/_boost/src/binom_ufunc.pyx
Traceback (most recent call last):
File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 317, in <module>
main()
File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 313, in main
find_process_files(root_dir)
File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 302, in find_process_files
for result in pool.imap_unordered(lambda args: process(*args), jobs):
File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
raise value
File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 302, in <lambda>
for result in pool.imap_unordered(lambda args: process(*args), jobs):
File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 236, in process
processor_function(fromfile, tofile, cwd=path)
File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 102, in process_pyx
raise Exception('Cython failed')
Exception: Cython failed
setup.py:486: UserWarning: Unrecognized setuptools command ('dist_info --egg-base /tmp/pip-modern-metadata-tcs7khbn'), proceeding with generating Cython sources and expanding templates
warnings.warn("Unrecognized setuptools command ('{}'), proceeding with "
Running from SciPy source directory.
Cythonizing sources
Traceback (most recent call last):
File "setup.py", line 350, in generate_cython
import pip
ModuleNotFoundError: No module named 'pip'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/oscar/current/sympy/sympy.git/tmpvenv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/home/oscar/current/sympy/sympy.git/tmpvenv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/oscar/current/sympy/sympy.git/tmpvenv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/tmp/pip-build-env-nvuzm8iv/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 174, in prepare_metadata_for_build_wheel
self.run_setup()
File "/tmp/pip-build-env-nvuzm8iv/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 267, in run_setup
super(_BuildMetaLegacyBackend,
File "/tmp/pip-build-env-nvuzm8iv/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 158, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 628, in <module>
setup_package()
File "setup.py", line 612, in setup_package
generate_cython()
File "setup.py", line 352, in generate_cython
raise RuntimeError("Running cythonize failed!")
RuntimeError: Running cythonize failed!
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

SciPy/NumPy/Python version information

SciPy 1.8.1 and current main, numpy 1.23.1, Any version of Python or PyPy (I think), Cython 0.29.31

Issue 2:
Cython 0.29.31 was release a few hours ago and it corresponds to our CI going red.

https://github.com/cython/cython/releases/tag/0.29.31

See for instance: https://github.com/scipy/scipy/runs/7551272609?check_suite_focus=true

Error compiling Cython file:
1453
------------------------------------------------------------
1454
...
1455
return np.array(result, dtype=np.int64)
1456

1457

1458
@cython.wraparound(False)
1459
@cython.boundscheck(False)
1460
def _weightedrankedtau(ordered[:] x, ordered[:] y, intp_t[:] rank, weigher, bool additive):
1461
^
1462
------------------------------------------------------------
1463

1464
_stats.pyx:173:0: Referring to a memoryview typed argument directly in a nested closure function is not supported in Cython 0.x. Either upgrade to Cython 3, or assign the argument to a local variable and use that in the nested function.