Revert #20027

The introduced issue https://github.com/rust-lang/rust/issues/20127 seems serious enough that we may want to enqueue a reversion for now. Feel free to beat me with a fix though @michaelwoerister!