CI Issues regarding conda lock files

Opening a new issue regarding some of the discussions around https://github.com/scikit-learn/scikit-learn/pull/27448#issuecomment-1733374337

@lesteve these are maybe what I have in mind:

- they're generated files and usually it's a good idea not to have generated files in the repo
- the files are generated, and their generation depends on the exact time when the conda repo data was downloaded in the user's machine. So there's basically no way to review those changes. Which means we have a ton of changed lines which we cannot review.
- since they're generated and we don't review them, we can't really make sure somebody hasn't manually changed anything in the changed file and that the files are indeed generated and not touched afterwards.
- a significant amount of changed lines has been coming from these files
- we are not even consistent on the use of the same snapshot for different CI. We might be using a generated file from last week for circleCI, and one from a month ago for some of the Azure stuff, and one from two weeks ago for other Azure configs.
- we don't really have good docs on how contributors could use the lock files or if they should. Those are only used in the CI, and for that they don't need to be in the same repo.

A few things we could do:
- only allow changes to those files via PRs done by a bot that we write, and generate them periodically, like weekly maybe, and daily for some which change often (like scipy-dev)
- move them out of the main repo.
- we could also move to a docker based system where our CI doesn't have to download and install for each job for each commit. This would also save us a bunch of time, and it would make it even easer to reproduce CI locally.

COMMENT:
I think the point is that all "latest" CI jobs should be updated more frequently, e.g. every week, ideally by an automated system that would auto-merge the lockfile update PRs if they are green.

There is nothing to review there: if the lock update is green, then its fine. If it's not green, it means that one of our direct or indirect dependency has introduced a regression, either in scikit-learn or in another dependency of scikit-learn.

If the breakage is not automatically resolved after a few days by attempting to bump up again the dependencies (e.g. the updated library that caused the breakage issuing a quick bugfix release because it broke half of github), then we should spend time investigating and reporting the breakage upstream.

The point is that all of our CI statuses of our PRs do not have to be red for a week because something completely unrelated in setuptools or anywhere else has changed and is causing our CI builds to fail.

Since we have introduced the lock files I thing we have significantly reduced the disruption on contributors PRs.

> only allow changes to those files via PRs done by a bot that we write, and generate them periodically, like weekly maybe, and daily for some which change often (like scipy-dev)

I agree. This was the original goal to have such a bot.

> move them out of the main repo.

I don't see how that would help. That would probably make things more complex by having to deal with synchronization of the state of different repos. Sometimes we need to experiment with upgrading a specific version of a library in a given PR and we will loose that flexibility.

> we could also move to a docker based system where our CI doesn't have to download and install for each job for each commit. This would also save us a bunch of time, and it would make it even easer to reproduce CI locally.

That does not work for macOS: as far as I know there no guest image for docker that run a macOS runtime: it's only possible to run a linux runtime on a macOS docker host.

For windows, it might be possible but I have the feeling that this is an exotic setup that very few people have expertise with.

Labels: Build / CI, Needs Decision