Yes, I recommend peeling them. If you like, you can scrub the skin well and keep the skin on before grating and making the fritters.