> Have you tried to run this locally? Tests shouldn't only be able to be run on the CI server, relying on this is a severe limitation.

Of course. We don't have to remove that folder, but I generally do it locally just to make sure I'm not using old binaries.