Fix optimizer performance regression

This was a subtle one.