Prevent multiple neighbors from being created on the same line/direction