Remove config directory on destroy