config: clean up extra certs only once
v0.19.4

Signed-off-by: Fedor Indutny <fedor@indutny.com>