The following comments are derived from my experiences in the physics department, but in general, they also hold true for math students. I will point out some major differences between the two disciplines at the end. I would say that the first year is generally the toughest because that's when you are saddled with taking core courses, teaching undegraduates, and preparing for the PhD Qualifying exam. Basically, the transition from an undergraduate to a graduate student is pretty discontinuous, and most people will admit that the experience is generally much tougher than they expected. Of course, all is not doom and gloom.
Times do get much better once a student passes the first year and the Qualifying exam.
I think what gets most people through the tough times is the friendship and camaraderie they develop with their fellow grad students. As usual, your mileage may vary, but you get the picture.