You can't thaw Cool Whip in the oven, but you can bake with it. Cool Whip replaces the oil in these cake mix cookies, making them lighter than regular cake mix cookies.