If you had enough disk space and memory available when you installed Linux, you undoubtedly already have X installed. I'm going to assume you have X up and running. This increases stability. If a graphical app goes berserk, for instance, it doesn't take the entire operating system down with it. On the minus side, because X is less integrated into the OS there are occasional challenges in getting it installed properly.
X is also a bit slower, overall, than an integrated graphical environment.
Everything in system design, as in life, is a series of tradeoffs. The X Window System, like Linux, is an open-source product. Most Linux distributions provide binaries so you don't have to compile the source. X uses some confusing terminology. When you run an X program on your Linux box, it is the X Window client and it connects itself to an X Window server.