Replaced all the scene geometry with normals. Clearly the problem isn't with the intersection test itself but with
using the intersection point / normal correctly.