From: robin@pencom.com (Robin D. Wilson)
Subject: Re: High-speed modem at ... (Very LONG)
Reply-To: robin@pencom.com
Organization: Pencom Systems Incorporated
Date: Thu, 10 Sep 1992 17:09:08 GMT

Well, since there has been soooooo much interest in it (and because I got  
tired of sending off individual responses, here is my modem setup guide for  
the RS/6000 and AIX systems.

Disclaimer: This is provided without warrantee, it was written while I worked  
at IBM Defect Support, and that was 1.5 yrs ago.  It still seems to apply to  
3.2 code (for the most part), but don't hold me to that.

BTW, If I calim something as fact, it is really only MY interpretation of the  
the facts.  So take it all with a grain of salt... Hope this helps.

Here it is:

------------------------------CUT HERE---------------------------------------
To set up modems in general requires an understanding of several important
concepts:

    1) (D)CD (Data) Carrier Detect.  This is a voltage signal placed
    on pin 8 of the RS232 DB25 serial connector.  It is used to tell
    the DTE (Data Terminal Equipment - in this case the computer to
    which the modem is attached) that the DCE (Data Circuit terminating
    Equipment - the modem) has established a logical 'wire' from point-
    to-point.  In other words, we are connected from the serial port on
    your computer to the serial port on the other device to which you are
    attempting to communicate.

    2) DTR (Data Terminal Ready).  The DTE device is ready to talk.  
    This is a voltage on pin 20 of the RS232 DB25 serial connector.  It
    is intended to alert the DCE device to the 'existance' of the DTE
    device.  (Sometimes this is used for a rudementary form of hardware
    flow control also... Turn off DTR == stop talking, Turn on DTR ==
    start talking again.)  Basically with a modem this is used to 
    signal to the modem that the DTE (in our case the computer) is 
    ready to talk.  Usually this means that the tty has been opened
    (by either a "getty" process or some other program - like "cu") and
    the modem should enable "Auto-Answer" and prepare to accept commands.
    When DTR drops, the modem should interpret this as "the DTE device
    has been shutdown... time for me to hang up".

    3) Flow control.  This can either be accomplished through hardware
    or software.  Hardware flow control can take many forms, but for 
    modems there is really only one standard: RTS/CTS.  This form of
    flow control uses a voltage on pins 4 and 5 (RTS and CTS pins) to 
    signal to both sides of the connection when either side can accept
    data.  If the voltage goes away, the other side should stop sending
    data.  Software flow control usually takes the form of XON/XOFF 
    flow control.  This is a character that is placed into the stream
    of data going from one side to the other.  When the other side 
    receives the XOFF character, it should stop sending, when it 
    receives an XON character it can start back up again.  (There are
    more serious limitations to the software flow control methods because
    some programs generate binary data that could equal either an XON
    or an XOFF character.  This means that the serial device will strip
    those bits out of the data assuming that they were intended for 
    flow control, when in reality they were part of the data.  The other
    limitation is: if the remote side has sent an XOFF to the local side
    (or visa versa) then the local side is not allowed to send any data
    back to the remote side... including XOFF or other flow control 
    characters.)

Enough with the basics.  Now you want to know how to set up your IBM RS/6000
or your IBM RT to use a modem.  There are two sides to the setup: the tty 
setup and the modem setup.  The tty is fairly easy.  For the most part your
choice will be limited to the defaults from SMIT (for the RS/6000) or to
the defaults from Devices (for the RT).  There is; however, one area to which
you will need to pay particular attention: do I intend for this modem to be
used for dial-in, dial-out, or bi-directionally?  One ofthe following 
settings for "Enable LOGIN" (on the RS/6000) or "Automatic Enable" (on the  
RT) will need to selected based on what you want to do:

    DISABLE: Use this setting if you only want to use the port for dial-out
    type connections.  Basically, this setting does nothing but configure
    the serial port.  No getty process is started, nobody can login on this
    port (unless you decide to "penable, or pshare, or pdelay" the port at
    some later point).

    ENABLE: This setting is best used for terminals (ie. not for modems), 
    but it can work for "dial-in" only modem connections.  This setting
    starts a "getty" process on the port.  This getty started in this
    manner immediately locks the port, preventing any other process from
    using the port.  Then, when getty sees the CD signal come up on the
    port, it sends a login herald, allowing the remote user to login.
    (IT IS SIGNIFICANT that the login herald is sent AFTER CD comes up.
    Think about it like this... "the modem brings up carrier AFTER it has
    negotiated the speed and connection parameters -- a "carrier frequency"
    -- with the remote modem, so getty doesn't send the login herald until
    the remote system/terminal is attached.")

    SHARE: This setting works well for "bi-directional" modem use.  This
    setting causes a "getty -u" process to be started on the port.  The 
    "getty -u" is the classic AT&T "uugetty" process.  Basically this 
    type of getty starts on the port, and then waits for carrier to come
    up before attempting to lock the port.  This allows other programs
    to open the port, create a lock, and prevent the getty from locking
    the port.  As an example, the port is "share'd", but I want to use it
    to dial-out using the "cu" program.  I issue the cu command, and cu
    opens the tty, and locks the port.  I connect to the remote modem, 
    CD comes up.  (Remember that "getty" is still running, and just waiting
    for CD to come up.)  Getty attempts to lock the port now (because CD 
    came up), but "cu" has already locked the port so getty can't get a
    lock.  The "getty" process then goes into a loop, "Do I have CD?
    Yes.  Can I get the lock?  No.  Do I have CD? ..."  When the "cu"
    process exits, CD is down, so the "Do I have CD?" part of the loop
    fails, and getty just loops on that function.  When CD comes up due
    to someone dialing-in, and no other process has the lock, getty says:
    "Do I have CD?  Yes.  Can I get the lock?  Yes.  Great! issue a 
    'login:' herald."  (Imbellishments added by the writer - you get the
    general idea.)

    DELAY: This setting is nearly identical to the "SHARE" setting, except
    the getty is called as: "getty -r" (which is the same as the AT&T
    uugetty -r) and this version of getty is waiting to see a character 
    on the serial input buffer (read() on the tty doesn't return '0'). 
    before it attempts to lock the port.  So in the paragraph above, just
    replace all occurances of "CD" and "Carrier" with "Character on the 
    serial input buffer".

The modem setup can get a great deal more complicated.  Since there are so
many different brands of modems, each with their own setup parameters and 
commands, I will attempt to address them in a "generic" HAYES compatible
manner, and let the reader find the equivalent settings for his particular
brand modem.  Try to remember that most modems have settings for all of
these parameters, but some modems call them different things, and some 
divide them into "sub-categories" that are not discussed here.  I will try
to explain why the settings are necessary, so that confusion can be limited.

You will need to make the following configurations for your modem:

    CD (Carrier, DCD, Data Carrier Detect, even CO) should be set to 
        follow "true" carrier.  This means that it is not "strapped
        high" on the modem, and that it only goes high when the 
        remote modem is connected to the local modem.  The reason
        this is important relates back to the 'getty' settings 
        described above.  If getty is waiting for carrier to lock
        the port and/or send out the login herald; strapping carrier
        high will defeat the purpose of having distinct settings for
        "ENABLE" and "SHARE" (ie. they will work functionally the
        same).  Also, having getty echo a login herald to an 
        unconnected modem can cause a loopback condition.  The modem
        should usually be set to "echo" back the commands you type;
        but when you send the "login:" herald, it echos that back too.
        Getty will echo everything you type, so the login herald is 
        echoed back to the modem, which echos it back, ad infinitum.
        If there is no carrier until a connection is made, then; when
        getty sends the "login:" herald, it is not echoed back, it is
        sent on to the remote terminal.  Finally, if you strap carrier
        high on the modem, the serial port device driver will not 
        know when the connection has gone down.  For example, if the
        remote modem is powered off during a connection, the local
        modem will sense this, and hang up... BUT the local shell
        process on the computer will see carrier up (because the modem
        is set to "ALWAYS" present CD) and never log the user off.
        The next person who calls in, will be able to use the previous
        user's shell.  This is (needless to say) a severe security risk.
        On the HAYES "V-Series" modems, the correct setting is:
        AT&C1.

        NOTE:  If you use the "tip" program for dialing out, you NEED to
        strap carrier high on the modem.  This is because "tip" is designed
        to work with carrier on the port only.  (The port is opened, but
        IO to the port is blocked until carrier detect is high.  This has
        to do with the way the port is opened using the "open()" system 
        call.)  To strap carrier "ALWAYS" high on a HAYES "V-Series" modem,
        enter the command: AT&C0.

    DTR should be set to "hang up" and "disable auto-answer" when the DTE
        device drops DTR.  Most modems will come with the factory
        default set to ignore DTR signals from the DTE.  This way,
        the modem will "auto-answer" right out of the box.  This is 
        not what you really want.  On the RS/6000 the DTR signal is
        raised when the tty device is opened with the 'open()' 
        system call.  This means that "getty" will cause DTR to be
        to be raised and the modem will be able to "auto-answer"
        only when there is a getty running on the port.  The other
        part of the equation is that the modem should hang up when 
        DTR drops.  On most SysV type machines, the "stty" setting
        of "hupcl" (Hang UP on last CLose) will cause DTR to drop on
        the last "close()" of the open tty file descriptor (in other
        words, when the last process using the port releases it, 
        the device driver will stop asserting DTR).  This is the way
        the computer signals to the modem to "hang up" the connection.
        On HAYES "V-Series" modems, there are 2 possible settings:
        AT&D2 (hang-up, disable auto-answer, and enter command state
        when DTR is not present); and AT&D3 (hang up, disable auto-
        answer, "RESET - reload operating parameters", and enter 
        command mode when DTR is not present).  The big difference
        is: "RESET".  This means that if you made changes to the 
        software switch settings, and then dropped DTR without saving
        the changes, they would be lost.  On the other hand, if someone
        else made changes to the software switches, and you had the
        AT&D2 setting, those changes would remain "in-force" until
        you reset them.  Most people prefer AT&D3.

    Auto-Answer should be enabled if you intend to allow other people
        to dial-in to your machine.  On the HAYES "V-Series" modems
        this is software switch "S0=001".  (Which sets the number of
        times to let the phone ring to "001".)  You can set this
        with the command: ATS0=<num> (where <num> equals the number
        of times you want the modem to let the phone ring before
        it answers).  Try to remember if you intend to set this to 
        something more than "1 ring", that most modems have a default
        "timeout" value that is set around 30 seconds.  If the modem
        calling you doesn't get a "carrier" tone within that timeout
        it will hang-up, so setting a very large number of rings could
        make your connection rate somewhat unreliable.

    Command Response should be set to respond to "LOCAL" commands only.
        This one is a little tricky to describe.  Most modems try
        to alert the local DTE device that there is activity coming
        in from the outside world.  Normally, when an incoming call
        is detected the modem sends the string "RING" to the local
        DTE device.  This means that string will be echo'd back to 
        the modem (because getty will turn on the "echo" on the tty
        device).  When the modem sees any characters coming into it
        on the receive serial line (the one coming from the local
        machine), it immediately "HANGS UP" and enters command state.
        This means that this "RING" string will cause your modem to 
        hang up before making a connection to the remote modem.  Also,
        if your modem doesn't exhibit the strange behavior of "hanging
        up" right as the connection is getting started, then it will
        still have the problem of the first user to the "getty"
        process running on the port will have the name "RING".  So
        the long answer is... set command response to only respond to
        local commands.  On the HAYES "V-Series" modems the "Q"
        register handles this.  I have found that a setting of:
        "ATQ2" works fine.

    Flow control should be set according to the desired programs
        that you will be using.  For most instances, if you are going
        to be doing any kind of file transfers, you will need to make
        sure that you are NOT using "XON/XOFF" flow control (the
        same caveat applies to the serial port setup as well).  UUCP
        and xmodem for example use a form of checksumming or CRC 
        error control, and these "checksums" or "CRC" values are 
        binary numbers -- meaning they can take the form of ANY 
        ASCII character sequence -- including the ^S and/or ^Q used
        for XON/XOFF flow control.  UUCP and xmodem will not usually 
        work with XON/XOFF (if they do work, it just means that the
        files you happened to transfer didn't cause a "checksum" or
        "CRC" value that matched the XON or XOFF characters).  What 
        happens is this: one system (either modem or serial port) is
        receiving data, and responding with a CRC.  The receiver gets
        a packet of information that CRC's to include the binary value
        of '^Q'.  This CRC is sent to the remote system.  The device 
        that expects "XON/XOFF" flow control sees the '^Q' as a flow 
        control character, and assumes that it was not intended to be
        in the data stream; so it strips it out of the incomming data.
        Then, the system that was sending the packet receives the CRC
        (minus the bits that made up the '^Q') and thinks that it has
        received an invalid CRC, so it resends the packet (which will
        still have the SAME CRC value).  For most connections you will
        either need to use RTS/CTS flow control, or "NONE" for flow
        control.  (On UUCP and Xmodem file transfers flow control is
        not required, the size of the packets that each program sends is
        small enough that you don't have to worry about overrunning the
        receive speed of the remote systems serial line).  On the IBM RT
        product "RTS/CTS" flow is not supported, so you must use "NONE"
        for any type of file transfer software.  On the RS/6000 the "RTS"
        line discipline can be set, but setting it is cumbersome (at
        least on the current release of AIX v.3.1), so it is usually
        easier to use "NONE" on this machine as well.  Use the following
        list for the HAYES "V-series" modems to determine how to set up
        specific flow control methods:

        XON/XOFF    AT&K4

        RTS/CTS        AT&K3

        NONE        AT&K0

        (By the way, it doesn't hurt to set "RTS/CTS" flow control
        on the modem even if the computer is not using any flow 
        control.)
        

By and large, these settings should allow you setup reliable  
uni/bi-directional
modem connections for the IBM RS/6000 and RT products.  If they don't help,  
or
if I have left something out, please mail me a note and I will add the info  
to
the next revision of this guide. 

NOTE: The "SHARE" and "DELAY" settings for the tty setup on the RS/6000 don't
work correctly until the 3003 update.  Make sure that you have at least the
3003 update on the RS/6000 before attempting to use "bi-directional" modem
lines, or you will have alot of headaches.

Additional NOTE: Versions of AIX prior to 3.1 update 3008 still have MANY  
known serial problems.  I suggest you update to 3.2 before declaring anything  
as "broken".

--
+---------------------------------------------------------------------------+
|These are MY views, nobody where I work even knows I have views  ;-)       |
|Internet: robin@pencom.com                                                 |
|US Mail:  8-6 Brooke Club Dr.                Home Phone: (914) 923-4093    |
|          Ossining, NY  10562                Work Phone: (212) 513-7777    |
+---------------------------------------------------------------------------+
Newsgroups: comp.unix.aix,comp.sys.novell