Don't attempt to remove associated objects that are already deleted.  Fixes #36