For that just-baked flavor, you can reheat muffins in a microwave or oven until warm. Adding the cream cheese glaze is best done after thawing or reheating for the freshest taste and texture.