From: rick@jaguar.msc.cornell.edu (Rick Cochran)
Subject: Re: XDM, Ksh, mwm, and nothing works
Organization: Cornell Materials Science Center
Date: Fri, 18 Dec 1992 17:19:24 GMT

I have finally managed to get an AFS authenticating version of MIT R5
xdm working on my AIX 3.2.2+ machines.  The below also applies to
MIT R5 xdm without AFS authentication.

I had to use a wrapper to start the X server.  "-sdebug" DID NOT WORK!
The wrapper sets up /dev/hft as its controlling terminal, then execs
/usr/bin/X11/X.

I also had to patch X11R5/mit/lib/X/XConnDis.c to get unix domain sockets
to work properly under AIX 3.2.2+.  Without this patch, you have to use
"host:0" rather than ":0" as your display, thus using TCP sockets, thus
running more slowly.

I also had to start xdm in such a way as to get root's limits instead of
the limits which the init process has.

I include the details below:

excerpt from /etc/rc.local:
if [ -f /usr/local/X11R5/bin/xdm -a -f /usr/local/xdmR5/xdm-config ] ; then
	su root "-c /usr/local/X11R5/bin/xdm -config /usr/local/xdmR5/xdm-config"
fi


Xservers file:
:0      local   /usr/local/xdmR5/xwrap -D /usr/lib/X11/rgb -x dps -T


Patch to X11R5/mit/lib/X/XConnDis.c:
*** XConnDis.c.orig     Fri Nov  6 14:13:13 1992
--- XConnDis.c  Thu Nov 19 13:19:09 1992
***************
*** 450,458 ****

      unaddr.sun_family = AF_UNIX;
      sprintf (unaddr.sun_path, "%s%d", X_UNIX_PATH, idisplay);

      addr = (struct sockaddr *) &unaddr;
!     addrlen = strlen(unaddr.sun_path) + sizeof(unaddr.sun_family);

  #ifdef hpux /* this is disgusting */
      ounaddr.sun_family = AF_UNIX;
--- 450,459 ----

      unaddr.sun_family = AF_UNIX;
      sprintf (unaddr.sun_path, "%s%d", X_UNIX_PATH, idisplay);
+     unaddr.sun_len = strlen(unaddr.sun_path);

      addr = (struct sockaddr *) &unaddr;
!     addrlen = (sizeof(unaddr) - sizeof(unaddr.sun_path) + unaddr.sun_len);

  #ifdef hpux /* this is disgusting */
      ounaddr.sun_family = AF_UNIX;


Source code for xwrap:
/*
 *	X wrapper for AIX 3.2 server by Drew Eckhardt
 *
 * 	The Xserver needs to run with a hft as a controlling tty.
 *	This wrapper does just that.
 */

#include <stdio.h>
#include <sys/file.h>
#include <sys/ioctl.h>

/*
 * 	I tried moving the server to /usr/lpp/X11/bin/X.real, 
 *	it puked all over - some shared library problem.
 */

#ifndef SERVER
#define SERVER "/usr/lpp/X11/bin/X"
#endif

/*
 *	/dev/hft  means the "first free virtual terminal".
 */

#ifndef HFT
#define HFT "/dev/hft"
#endif

void main (int argc, char **argv, char **envp) {
	int pgrp, f;

/*
 *	First, we must disconnect from the controlling terminal 
 *	(if there is one)
 */
	
	if ((f = open("/dev/tty", O_WRONLY)) >= 0) {

		if (ioctl(f, TIOCNOTTY, 0) < 0) {
			fprintf(stderr, "%s : TIOCNOTTY ioctl failed ", 
				argv[0]);
			perror("");
			exit(1);
		}

		close(f);
	}

	close(0);
	close(1);

/*
 *	Now, we get our own process group, and session.  This lets 
 *	us associate with a new controlling tty.
 */

	if ((pgrp = setpgrp()) < 0) {
		fprintf(stderr, "%s : setpgrp() failed ", argv[0]);
		perror("");
		exit(1);
	}

/*
 *	Since we have no controlling tty, and the HFT we want 
 *	has no session / pgrp associated with it, when we open it,
 *	it will become our controlling tty.
 */

	if (open(HFT, O_RDONLY, 0) < 0 ||
	    open(HFT, O_WRONLY, 0) < 0) {
		fprintf(stderr, "%s : open failed on %s ", argv[0], HFT);
		perror("");
		exit(1);
	}

	execv(SERVER, argv, envp);
	fprintf(stderr, "%s : exec %s failed ", argv[0], SERVER);
	perror("");
	exit(1);
}

-- 
|Rick Cochran		  607-255-7223		     rick@msc.cornell.edu|
|Cornell Materials Science Center		     rick@crnlmsc2.bitnet|
|E20 Clark Hall, Ithaca, N.Y. 14853	     cornell!msc.cornell.edu!rick|