I realize you've just disabled this, but this looks wrong; since direction is a vector (as opposed to a position), it should be glm::vec4(direction, 0.0f).  Otherwise, you're including the translation component of the matrix.