I'm so stupid. Goddamnit. I totally forgot glm is the transpose of normal matrices so that meant ALL my matrix multiplications were backwards.