You may of course redirect X11 to a remote display server, but you will still need the client libraries and header files. Time and patience. Compiling all this stuff takes several hours. Be prepared to run into problems. The host type , used by GNU packages to name particular operating systems, is often not recognized by the config.guess and config.sub scripts.
Shared libraries are something of a problem.
Most packages use GNU libtool. See the libtool page for more information and patches. It is solved by a small compatibility library called dlcompat. This is a small compatibility library that provides dynamic loading. A new compression program and library.