Rebuild neighbors when a block is broken.  This needs to be cleaned up