Dusting powders are available for rabbits and guinea pigs, and can pretty much be used interchangeably. As you brush the powder out, it should pick up any dirt particles and condition the coat. For dirty bottoms, hydrogen peroxide is good at removing urine stains. The water should be slightly warm, but not hot. Small mammals have very sensitive skin, so powerful shampoos can be harmful.
Most owners use a baby shampoo for general baths.
Generally, you should only use a medicated shampoo when instructed by your vet. Lather your guinea pig from the rump up to the back of the neck. Do not get any soap around the face. Be careful when rinsing to make sure no soap runs over their head and into their eyes or face. Dry them gently with a towel, and then let them air-dry the rest of the way.