Avoid needless spec double load

Per #2454, the (rather expensive) refresh may no longer be necessary.

Without a concrete idea of the problem it was originally introduced to solve, I don't have a better idea than "let's remove it, and see what happens".

The refresh is present back to 1.1.0; I propose merging to 1-3-stable as a perf fix, but not further back. Or is this master-only material?

/cc @tenderlove @indirect