Baths are typically stressful events for a cavy, and stressed guinea pigs are more susceptible to bacterial and viral infections. Dusting powders are available for rabbits and guinea pigs, and can pretty much be used interchangeably. As you brush the powder out, it should pick up any dirt particles and condition the coat. For dirty bottoms, hydrogen peroxide is good at removing urine stains. The water should be slightly warm, but not hot.
Small mammals have very sensitive skin, so powerful shampoos can be harmful.
Most owners use a baby shampoo for general baths. Generally, you should only use a medicated shampoo when instructed by your vet. Lather your guinea pig from the rump up to the back of the neck. Do not get any soap around the face. Be careful when rinsing to make sure no soap runs over their head and into their eyes or face.