Yes, you can use lower fat or skim milk to make this dish, however whole milk really gives this gravy a rich and creamy flavor.