Repository: nilearn/nilearn

Issue:
numpy2 compatibility
This workflow has been failing for more than 2 weeks and I had not realized because I did not get any notifications.

https://github.com/nilearn/nilearn/actions/workflows/numpy2_compatibility.yml

Failing tests: https://github.com/nilearn/nilearn/actions/runs/7665540512/job/20891619867#step:5:18881

```
FAILED nilearn/regions/tests/test_region_extractor.py::test_remove_small_regions
FAILED nilearn/regions/tests/test_region_extractor.py::test_connected_label_regions
FAILED nilearn/regions/tests/test_region_extractor.py::test_connected_label_regions_return_empty_for_large_min_size
FAILED nilearn/regions/tests/test_region_extractor.py::test_connected_label_regions_check_labels
```

To run locally:

```bash
tox run -e test_numpy2 -- nilearn/regions/tests/test_region_extractor.py
```

Also realising that numpy 2 is actually not installed by this workflow:

```
test_numpy2: freeze> python -m pip freeze --all
test_numpy2: certifi==2023.11.17,charset-normalizer==3.3.2,contourpy==1.2.0,coverage==7.4.0,cycler==0.12.1,fonttools==4.47.2,idna==3.6,iniconfig==2.0.0,joblib==1.3.2,kaleido==0.2.1,kiwisolver==1.4.5,lxml==5.1.0,matplotlib==3.8.2,nibabel==5.2.0,nilearn @ file:///home/runner/work/nilearn/nilearn/.tox/.tmp/package/1/nilearn-0.1.dev1%2Bgd0cbdcd.tar.gz#sha256=694501acd0e789448bcc3a688a97f8ccbf04f8895f5e684780597fc1a8d83d31,numpy==1.26.3,packaging==23.2,pandas==2.2.0,pillow==10.2.0,pip==23.3.1,plotly==5.18.0,pluggy==1.4.0,pyparsing==3.1.1,pytest==7.4.4,pytest-cov==4.1.0,python-dateutil==2.8.2,pytz==2023.3.post1,requests==2.31.0,scikit-learn==1.4.0,scipy==1.12.0,six==1.16.0,tenacity==8.2.3,threadpoolctl==3.2.0,tzdata==2023.4,urllib3==2.1.0
```


Potentially useful:

- numpy 2 migration guide: https://numpy.org/devdocs/numpy_2_0_migration_guide.html
- [numpy/numpy#24300](https://github.com/numpy/numpy/issues/24300)
- scikit learn issue about it: https://github.com/scikit-learn/scikit-learn/issues/27075
- https://pythonspeed.com/articles/numpy-2/


Discussion:
scipy dependencies:
- https://github.com/scipy/scipy/blob/19b4fe03c939c7b6becb1757cff9c38da929aee7/pyproject.toml#L33

pandas dependencies: 
- https://github.com/pandas-dev/pandas/blob/d928a5cc222be5968b2f1f8a5f8d02977a8d6c2d/pyproject.toml#L9

nibabel dependenicies:
- https://github.com/nipy/nibabel/blob/5d884bda234a22ce7035997c090449851a82c0be/pyproject.toml#L14
Yes, I can see the notifications in my spam... Did you undertsand what causes the failure ?
Weird I also never received any notifications not even in my spam.

> Also realising that numpy 2 is actually not installed by this workflow:

You can see at least in the logs of the Action that it is installed https://github.com/nilearn/nilearn/actions/runs/7677387501/job/20926068575#step:5:112 (first a stable version is installed as a dependency and then it gets uninstalled and replaced with the dev). At the moment I am not able to check locally

This thread is also good to follow: https://github.com/numpy/numpy/issues/24300


> Weird I also never received any notifications not even in my spam.

Yeah not sure how come @bthirion is getting them but not us.

> > Also realising that numpy 2 is actually not installed by this workflow:
> 
> You can see at least in the logs of the Action that it is installed https://github.com/nilearn/nilearn/actions/runs/7677387501/job/20926068575#step:5:112 (first a stable version is installed as a dependency and then it gets uninstalled and replaced with the dev). At the moment I am not able to check locally

Ah indeed! Sorry I had totally missed that part. Thanks for the pointer.

> This thread is also good to follow: [numpy/numpy#24300](https://github.com/numpy/numpy/issues/24300)

Adding this one to the top message.


> Did you undertsand what causes the failure ?

not looked at it in details yet: will get the release over with and then move on to it
More info on who gets notified about failed runs: https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/notifications-for-workflow-runs

TL;DR: the last person who modified the workflow (or the person who manually triggered it)
For the record here is a tree of nilearn dependencies on main at the moment obtained with `pipdeptree`.

Just to keep track of what direct or transitive dependencies have pinned numpy to < 2.0

```
kaleido==0.2.1
matplotlib==3.8.2
├── contourpy [required: >=1.0.1, installed: 1.2.0]
│   └── numpy [required: >=1.20,<2.0, installed: 1.26.3]
├── cycler [required: >=0.10, installed: 0.12.1]
├── fonttools [required: >=4.22.0, installed: 4.47.2]
├── kiwisolver [required: >=1.3.1, installed: 1.4.5]
├── numpy [required: >=1.21,<2, installed: 1.26.3]
├── packaging [required: >=20.0, installed: 23.2]
├── pillow [required: >=8, installed: 10.2.0]
├── pyparsing [required: >=2.3.1, installed: 3.1.1]
└── python-dateutil [required: >=2.7, installed: 2.8.2]
    └── six [required: >=1.5, installed: 1.16.0]
nilearn==0.10.4.dev3+g29ae5f12f
├── joblib [required: >=1.0.0, installed: 1.3.2]
├── lxml [required: Any, installed: 5.1.0]
├── nibabel [required: >=4.0.0, installed: 5.2.0]
│   ├── numpy [required: >=1.20, installed: 1.26.3]
│   └── packaging [required: >=17, installed: 23.2]
├── numpy [required: >=1.19.0, installed: 1.26.3]
├── packaging [required: Any, installed: 23.2]
├── pandas [required: >=1.1.5, installed: 2.2.0]
│   ├── numpy [required: >=1.23.2,<2, installed: 1.26.3]
│   ├── python-dateutil [required: >=2.8.2, installed: 2.8.2]
│   │   └── six [required: >=1.5, installed: 1.16.0]
│   ├── pytz [required: >=2020.1, installed: 2023.4]
│   └── tzdata [required: >=2022.7, installed: 2023.4]
├── requests [required: >=2.25.0, installed: 2.31.0]
│   ├── certifi [required: >=2017.4.17, installed: 2023.11.17]
│   ├── charset-normalizer [required: >=2,<4, installed: 3.3.2]
│   ├── idna [required: >=2.5,<4, installed: 3.6]
│   └── urllib3 [required: >=1.21.1,<3, installed: 2.1.0]
├── scikit-learn [required: >=1.0.0, installed: 1.4.0]
│   ├── joblib [required: >=1.2.0, installed: 1.3.2]
│   ├── numpy [required: >=1.19.5,<2.0, installed: 1.26.3]
│   ├── scipy [required: >=1.6.0, installed: 1.12.0]
│   │   └── numpy [required: >=1.22.4,<1.29.0, installed: 1.26.3]
│   └── threadpoolctl [required: >=2.0.0, installed: 3.2.0]
└── scipy [required: >=1.8.0, installed: 1.12.0]
    └── numpy [required: >=1.22.4,<1.29.0, installed: 1.26.3]
pip==23.3.2
pipdeptree==2.13.2
plotly==5.18.0
├── packaging [required: Any, installed: 23.2]
└── tenacity [required: >=6.2.0, installed: 8.2.3]
setuptools==69.0.3
wheel==0.42.0
```

```
tox run -e test_numpy2 -- nilearn/regions/tests/test_region_extractor.py
```

cannot be run twice in a row because pip cannot find a version of contourpy

```
ERROR: Cannot install contourpy==1.2.0, matplotlib==3.9.0.dev0, pandas==3.0.0.dev0+168.g622f31c9c4, scikit-learn==1.5.dev0 and scipy==1.13.0.dev0 because these package versions have conflicting dependencies.

The conflict is caused by:
    pandas 3.0.0.dev0+168.g622f31c9c4 depends on numpy>=1.23.5; python_version < "3.12"
    scipy 1.13.0.dev0 depends on numpy>=1.22.4
    scikit-learn 1.5.dev0 depends on numpy>=1.19.5
    matplotlib 3.9.0.dev0 depends on numpy>=1.21
    contourpy 1.2.0 depends on numpy<2.0 and >=1.20

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
```

environment needs to be recreated on every run... 😭 
debugging is going to be slow... 
@Remi-Gau does it work to explicitly include an install of a dev version of contourpy in the tox commands of this environment? Otherwise what might work is just to create your own environment with whatever environment manager you use, activate that and debug in there. That should be more stable.
> install of a dev version of contourpy in the tox commands of this environment

did not work somehow
will modify the recipe to reinstall on every run

you are right, probably a better way to debug

may still need to modify the tox recipe

Resolving patch:
diff --git a/nilearn/regions/region_extractor.py b/nilearn/regions/region_extractor.py
--- a/nilearn/regions/region_extractor.py
+++ b/nilearn/regions/region_extractor.py
@@ -116,7 +116,7 @@ def _remove_small_regions(input_data, index, affine, min_size):
     # np.unique and then use np.bincount to count the region sizes.
 
     _, region_indices = np.unique(input_data, return_inverse=True)
-    region_sizes = np.bincount(region_indices)
+    region_sizes = np.bincount(region_indices.ravel())
     size_in_vox = min_size / np.abs(np.linalg.det(affine[:3, :3]))
     labels_kept = region_sizes > size_in_vox
     if not np.all(labels_kept):


Test patch: