Fix bad access crashes by properly removing connected neighbors from their lists when before they are deleted from memory.