smarter removal of keys

Signed-off-by: Stephen Gran <steve@lobefin.net>