I personally don't mind the orange peel, especially when it's baked in honey, like in this recipe. It still has a slightly bitter flavor to it, but I enjoy it a lot. Feel free to remove the peel if you like. Removing the peel will also make the cake easier to slice.