Now imagine a hardware architecture that could protect against that. So instead of having user space, with limited privileges, and kernel space, with full privileges, you can have multiple spaces, each with full privileges, yet each protected from the others. You don't want a rogue X client trashing other clients or even taking the X server down. The price for this safety is high. You have to make Xlib calls, which have to create protocol, which has to be pushed into pipes or sockets.
Data has to be reformatted and copied from here to there.
On the server side, you have to scan the header, decode the major an minor protocol numbers. Hundreds of thousands of CPU cycles just to perform some silly graphics operation safely. If you're into SQL, you get the same raw deal, hundreds of thousands of CPU cycles to query some puny 100 byte record. Imagine instead being able to get the same function, with the same amount of safety, but this time with only the overhead of a subroutine call. Traditionally, in Unix, there is only one privileged process on the hardware, and its called the kernel.