Support removing certificates from store and CA