Stirring agitates the sugar crystals, which have jagged edges. When those edges bump into each other, they get unhappy and can make your caramel grainy — not a good thing. As the sugar begins to brown, you may notice some spots beginning to get darker than others. If this happens, rather than stirring, just give the pot a gentle swirl to redistribute the hot spots without agitating it too much.