No, you don't. BUT, I feel that the flavor and texture is best when the chill has come off. The ganache softens and becomes really luscious. Eating it straight from the fridge has more of a fudgy texture. It comes down to preference, honestly!