Remove certificates by alias when we add new ones to the keystore