You can use water and baking soda to clean just about anything, including a water cooler. Use a ratio of around 1 cup to 2 tablespoons of baking soda.
After you’ve poured it into the cooler, scrub it with a clean cloth. Then empty what you can through the spigots, and run it under the running water a few times until the residue is gone.