If you want rich and silky mashed potatoes, you should use double cream (heavy cream). Milk will not provide the same texture but it will give you lighter mashed potatoes with less amount of fat.