The amount of memory resources now being thrown at graphic devices is simply astounding. The following report was recently seen on Usenet, from Jim Gettys, one of the original designers of X, and is quite pertinent. While running, the server was reporting its size was a couple of megabytes. It doesn't take all that many apps, particularly with some fragmentatation of the memory pool in the server, to cause the virtual address space of an X server to get large. Which is unlikely.
X is platform independent, which is a crucial property.
Moreover, as we are talking about applications where the X protocol is the bottleneck, it should be noted that this is likely to only affect a somewhat special class of programs. Many interactive applications are mostly bottlenecked by the user's ability to read and respond. If the system is waiting for you to respond, you're obviously not waiting on X. Netscape may pump out a lot of X updates, but the bottleneck that affects the user is normally the speed of the connection across the Internet, and not the X protocols. Speed is often bound by the efficiency of the application itself, which only a truly magical graphical infrastructure can help. The only major class of applications where this is liable to be a common and significant bottleneck is that of games and animation.