Yes. All of that moisture in the whoopie pie is inherently going to make them a little sticky. And the longer they sit out, the stickier they may get. I refrigerate my whoopie pies to preserve the filling and reduce the stickiness.