Revert part of 466ce5fc88546baf75dc9e9dead5e1d6af6fd721

We no longer need it to speed things up in trunk, and this way is
actually a bit faster now.