Yes you can, and you should! The skin of the sweet potato holds most of the fiber and antioxidants.