No! The yeast will work fine with room temperature butter and milk as well. It might take a little longer to rise, but the slower rise will actually make both the consistency and taste so much better - and you also don't run the risk of heating your milk & butter too much and "killing" the yeast.