You don't have too, especially if using smaller mushrooms. If you are stuffing portobello mushrooms, you will want to remove the gills first.