Yes, you should always rinse mushrooms in cold water and rub off any dirt with your fingers before slicing them. Since mushrooms grow close to the ground and are delicate, they don't get as clean as other vegetables before they reach the grocery store.