Air! Well, technically it's really just the oxygen component that does the most damage to coffee (side note: carbon dioxide is actually somewhat protective in fact). And oxygen isn't just cold brew's enemy, but more generally it's coffee's biggest enemy in all its various forms (beans, grounds, or ready-to-drink). The reason why it's especially destructive with cold brew is that cold brew is often brewed in a batch and stored for days. But if you're not storing it in an airtight container, it will quickly oxidize and get nasty. So you need to keep oxygen out of the picture. This is a big reason we built our original Rumble Jar filter to work inside a Mason jar. As you probably know, Mason jars have a ridiculously good airtight seal. Without a good seal, your cold brew could start oxidizing the minute you put it in your fridge.
Pro-tip: use a spare Mason jar to store your coffee (preferably in whole bean form before you grind them), and just store that jar in a pantry away from sunlight. That will keep your beans as fresh as just about any other specialized storage solution we've come across.