Add an uninstall profile to remove configlet. Inspiration taken from collective.portlets.lineage