Yes, we have an existing bug for this.  I intend to fix it by adding an active/inactive flag to contacts, so we can deactivate them instead of removing them.  It's a slightly worrying change, though, so I haven't tried it yet.  In any case, this change is a helpful precursor, since the deactivation/activation code will be synchronous too.