Backport a fix for concurrent requires

Backport 16fc8e8b to 2.0 branch.
see #640 and https://bugs.ruby-lang.org/issues/9224