revert incorrect performance change

This didn't turn to be correct after all.

Added a test to ensure we don't break this again.

Closes #3963.
Closes #4119.