detecting if a ray hits the inside or outside of a corner