Don't attempt to remove associated objects that are already deleted. 

 Fixes #36