Yes, although using whole milk will result in a richer and creamier pastry cream due to its higher fat content, while skim milk will produce a lighter cream that is less rich in flavor.