No! I always leave them on and assuming you will be filtering your milk, then the small pieces of skin will be removed to ensure the milk is not grainy.