Remove possibly premature optimization that is causing artifacts in light raytracing.

Might be able to re-add some kind of optimization like this later, but for now it seems to be causing problems.