You might be swaying because you’re assuming your swing is parallel to your target. If you’re swinging back and forth to try and get your ball to be straight down the target line it’s easy to accidentally sway your body.