Add an uninstall profile to remove configlet. 

See http://blog.keul.it/2013/05/how-to-make-your-plone-add-on-products.html

Code taken from example at collective.portlets.lineage