I personally did not not, but if you don't like the slightly bitter taste orange peels can have, you definitely can remove them first.