Properly unlink a failed insertion