Backport a fix for concurrent requires

Backport 16fc8e8 to 2.0 branch.
see #640 and https://bugs.ruby-lang.org/issues/9224