Simple fix. I normalised the vector pointing at the player before getting a rotation quat.