added tolerance around a dot product calculation in visibility to handle some floating point errors