Only scroll if we need to