Only call refresh when necessary