On stage, physical movement is often imposed on verbal actions to give audiences something to watch and keep them visually involved. Audio does the opposite. Characters move only when they must to complete an action. Arbitrary movement confuses the listener. A sound location cannot suddenly change from stereo left to stereo right.
Listeners have to hear the pan, or cross.
Avoid small crosses, they will not be heard. When they move, characters should move broadly across the stereo field. Block to facilitate clean, uncluttered, unambiguous pannings and crosses. The same character ends one scene and begins the next. Events are growing too big and changing two fast for a character to keep up with.