You don't need to refrigerate your cold brew coffee, as the nitrogen stabilizes the drink and helps to minimize issues. You should keep your brew out of direct sunlight and minimize how long you keep your keg open, as this may end up impacting the overall quality of your brew. Limit yourself to two weeks of non-refrigeration and then put it in the fridge to avoid any complications.