From: exudnw@exu.ericsson.se (Dave Williams)
Newsgroups: comp.protocols.tcp-ip,comp.protocols.nfs
Subject: backbone routers and topology
Keywords: FDDI ethernet router backbone topology performance routed
Date: 20 Jan 93 18:16:56 GMT
Reply-To: exudnw@exu.ericsson.se (Dave Williams)
Organization: Ericsson Network Systems
X-Disclaimer: This article was posted by a user at Ericsson.
              Any opinions expressed are strictly those of the
              user and not necessarily those of Ericsson.

This may be in some FAQ somewhere, but I can't find it.

I'd like to hear from any large sites that have routers configured as 
backbones.  There's lots of marketing hype from the router companies
on why you "need" to do this, but very little in the way of hard info
on configuration, performance and pitfalls.

Our current configuration consists of about 400 SPARC clients fed by 13 Sun
4/490 servers. Each server has multiple NC400 Network Co-processors (one per
client network), 2 IPI controllers and 6-8 IPI disks (sime Saber's, some Elites).

We are currently setup in a straight hierarchical topology that looks something 
like this: 

                                                                           
                            
                   server backbone (ethernet)                                               
     +-----------------------+----------------------+-----> to existing router 
     |                       |                      |      (3270, x.25, etc.)
     |                       |                      |        
 +-------+               +-------+              +-------+   
 | 490   |               |  490  |              |  490  |
 |  1    |               |   2   |              |   n   |
 +-+-+-+-+               +-+-+-+-+              +-+-+-+-+ 
   | | |  NC400 I/F's      | | |                  | | |    

  client                   client                 client                         
  networks                 networks               networks                                    
                                                                           
                                                                            
                                                                               
We typically put a maximum of 25 diskless w/s on each client network and get
about 60 clients per server.  We run NIS, DNS, automounter, etc. and the whole
thing works pretty well. We are currenly in the process of adding about 200 
disks to make about half of our client population dataless, the remainder is 
Sun SLC's :~(

We started questioning this layout when we noticed our "server backbone" was
hitting high % of possible lan bandwidth at peak hours of the day. We already
had common data replicated on multiple servers but it's too big to copy to
all of them. We thought about this for a bit and came up with the following 
possible solutions:

1) Put a large router (Cisco AGS+, Alantec, Wellfleet, etc.) in place of the 
   server backbone. These typically have near-gigabit backplanes and have *got* 
   to be faster than our current Ethernet 10 MB/sec "governor".

2) Connect the server's client networks to both the clients and the router
   and turn off routing in the servers. We did some informal tests and found
   that a single client could cause 10% server CPU load just routing a single 
   "find" command. I've always heard that Suns made poor routers, but could 
   never find any hard data on the subject.

3) Use the router as an FDDI connection to a single server containing all 
   replicated filesystems. We thought this could be a SS10/41 with multiple
   fast SCSI disks. This would free up all the space currently taken up by
   replicated data spread over all the servers while providing for faster 
   access and better configuration control.

   One question that looms is whether the replicated (read mostly) data server
   should use multiple ethernet I/F's, one to each client network (using more 
   than one server) or use FDDI straight into the router.  We don't have a lot
   of FDDI experience to guide us.

   Is the answer the same for database (read-write, non-NFS) servers as it is for 
   "read mostly" (NFS) nodes?  Which is better (for NFS and/or socket DB access), 
   multiple ethernet segments in a "matrix" topology spread over each client 
   network or single FDDI interfaces from each server into the router?

We would end up with something like this:

                            
                         "old" server backbone 
           (Left intact for sanity and backup if router ever blows)                                               
      +--------------------------+-------------------------+-------------+                                                                      
      |                          |                         |             | 
      |                          |                         |             |
   +-------+                  +-------+                 +-------+        | 
   | 490   |                  |  490  |                 |  490  |        |
   |  1    |                  |   2   |                 |   n   |        |
   +-+-+-+-+                  +-+-+-+-+                 +-+-+-+-+        |
     | | |   NC400 I/F's        | | |                     | | |          |  
    client                     client                    client          | 
    networks                   networks                  networks        | 
     | | |                      | | |                     | | |          |  
 +---+-+-+----------------------+-+-+---------------------+-+-+----+     | 
 |    enet                       enet                      enet    |     |
 |                                                             enet+-----+
 |                         router "backbone"                       |
 |                                                      FDDI/CDDI  |
 +-----------------------------+----------------------------++-----+ 
                               |                            ||       
                           X.25,3270                  +-----++----+ 
                              etc.                    | replicated|
                                                      |   data    |
                                                      | server(s) |
                                                      |           |
                                                      +-----------+


Are we on the trail of something good here or have we all been locked up in
the hacketorium too long?  

Specifically:

1) Is turning off routing on the servers a *good* thing?

2) What will break?

3) One question that looms is whether the replicated (read mostly) data server
   should use multiple ethernet I/F's, one to each client network (using more 
   than one server to cover all of the client networks) or use FDDI straight 
   into the router.  We don't have a lot of FDDI experience to guide us.

   Is the answer the same for database (read-write) servers as it is for "read 
   mostly" nodes?  Which is better (for NFS and/or socket DB access), multiple 
   ethernet segments in a "matrix" topology spread over each client network
   or a single FDDI interface into the router for each type of server?

4) What are we overlooking?

5) Are there any other large sites doing this kind of stuff?


Any comments/suggestions appreciated.


= exudnw@exu.ericsson.se       (214)907-7928                            =
= David Williams              "You can't win, you can't break even,     =
= Ericsson Network Systems     and you can't quit"                      =
= Richardson, TX 75081                                  my opinions...  =    
--
= exudnw@exu.ericsson.se                                                =
= David Williams              "You can't win, you can't break even,     =
= Ericsson Network Systems     and you can't quit"                      =
= Richardson, TX 75081                                  my opinions...  =