# Presentation: 676818

## SNS Control Systems

**A new Tool to study Network Stack Exhaustion in**
**VxWorks**
*** ******Epics Collaboration Meeting***
***Dec. 8, 2004***
***Sheng Peng***
***Ernest L. Williams Jr.***
***David Thompson***

## The Story

**When we were dealing with “IOC Disease” earlier this year we got pretty good at using vxWorks diagnostics tools, mBufShow, inetStatShow, and a few that WRS gave us like ifQValuesShow.**
**We got pretty good at “tuning” by setting mbufs, driver queues, and the “if_Q length”.**
**We found and fixed several causes of depleted buffers.**
**We still have errors!****  Diagnostics like ifShow indicates txErrors and we still get white screens.**
**The end driver with debugging turned on also reports txErrors.**

## The first round of cures

**inetstatShow**
**Active Internet connections (including servers) ****PCB Proto Recv-Q Send-Q Local Address Foreign Address (state) ****-------- ----- ------ ------ ------------------ ------------------ ------- ****... ****1b4a990 TCP 0 ****8184**** 172.31.124.20.5064 172.31.124.107.51553 << Archive server**
**...****.****mbufShow**
**CLUSTER POOL TABLE ****_______________________________________________________________________________ ****size clusters free usage ****------------------------------------------------------------------------------- ****64 800 ****772**** 9859 ****128 1600 ****1531**** 105147601 ****256 800 800 2138545 ****512 400 400 34635 ****1024 200 200 1913 ****2048 300 300 27947 ****4096 20 20 6197 **
**Eventually the archive server would consume all of the buffers because daily restarts never closed the old sockets.  It usually took several days to a week for the IOC to crash, especially with large buffer configurations.**
**Other clients, were problems as well.  Edm with a stuck mouse would do the same thing!**
**The net result is that we understand this and have fixed the problems with clients for the most part.**

## The second round

**Now what?  We still have problems and the IOCs have plenty of free buffers in the network stack.**
**Maybe it is time to look at traffic patterns.**
**Bring in etherreal!**

## Network Traffic Analysis (Setup)

**IOC Under Study:**
**Scl-hprf-ioc05 (without Beckhoff driver)**
**Connected to CISCO 2950 layer2 switch**
**lin-ics-netsw3b1 ---- port 1**
**Port 1 is mirrored for observation via a linux-based packet capture and analysis system.**
**Tools used:**
**Laptop with “Ethereal” packet capture software**
**NIC 1 (eth0) ---- used for remote access to the packet capture station**
**NIC 2 (eth1) --- connected to the CISCO port mirror.**

## What we will study?

**What we will study:**
**Network Memory Resources Model**
**What are mBufs?**
**What are mBlks?**
**What are cBlks?**
**What are clusters?**
**Flow diagram of the vxWorks Network Stack**
**How are packets moved in out with respect to the OSI model?**
**The journey of a network packets as seen through the eyes of a network sniffer in an EPICS environment.**
**We will make a timeline of events from the time an IOC is booted to when it is “open for business”**
**CISCO port auto-negotiation and turn-on**
**Loading of vxWorks image from boot server**
**Re-setting of IOC's network hardware by vxWorks**
**Loading startup file common to all vxWorks IOCs (i.e. common.cmd)**
**Loading application specific startup file (i.e. st.cmd)**
**IocInit**
**What protocols are showing up?**
**Needed in the context of EPICS**
**Nuisance Protocols/traffic**

## VxWorks Network Stack Flow

**Notes:**
Explain how this works,  but keep it up on a different projector for reference.   Or you could hand this out.

However, please don’t spend too much time on this. Or you won’t make it through the other slides.

## Real-Time OS Considerations

**Buffer management**
**Pre-allocated buffers as opposed to dynamic from the global heap at run-time**
**Timers**
**Connection management**
**Timeouts**
**Retries**
**Latency**
**Fast and deterministic interrupt handling interfaces**
**Small thread context switch times**
**Concurrency**
**Smart use of semaphores**
**Minimized Data Copying**
**The TCP/IP implementation should minimize the amount of data copying. The data within each ****frame**** can be maintained in the *****same buffer***** so it doesn't need to be copied and re-copied by the CPU at each stage of the protocol. The networking chip's ****DMA**** places the packets directly in the *****managed buffer pool***** where the packet is passed up through the stack by manipulating ****pointers**** and not by copying data. The mbuf mechanism has been extended to allow the data to be shared between mbufs and mblocks where there are STREAMS protocols also present in the system.**

## Protocols we deal with in EPICS

**UDP port 5065**
**CA beacons (“I am here” Heartbeat)**
**Used to re-establish CA TCP virtual circuits**
**CA beacons do not expect any replies**
**The CA Beacon Daemon is listening on UDP port 5065.**
**A.K.A  caRepeater**
**UDP port 5064**
**CA search message **
**A response is expected within some timeout interval**
**TCP port 5064**
**CA server establishes a virtual circuit on port 5064**
**NFS**
**UDP port 111**
**Loading up IOC application**
**Running autosave/restore**
**Re-directing IOC files to boot server**
**NTP**
**UDP port 123**
**Keep IOCs time in synch**
**At SNS we should see this about every 10 seconds in our current configuration.**
**RSH**
**UDP port 514**
**Remote login support**
**Cat in vxWorks image**

## Network Traffic Analysis

***Ethereal Packet Analysis Timeline***

**Bring NIC online**

**T + 3 sec**

**PowerOn IOC**

**T - 0 sec**

**EPICS neighbors come**

**T + 3.02 sec**

**Notes:**
Since we know what protocols to look for, we can put TCPDUMP like filters on the capture file to help determine where the IOC is in the boot process.

We also used ping markers in the startup file to help initially.

The few slides will be nothing more than the results of the ethereal capture file with different filters based on protocols.   The goal is to build a timeline and correlate heavy traffic versus protocol.

Note: On this slide as soon as the NIC is up, the IOC begings receiving EPICS broadcasts from everywhere!!!

Remember
We have IOCs on various subnets and use directed broadcasts to deliver CA search and CA beacons.

## Network Traffic Analysis (Cont’d)

**Network Traffic Analysis (Cont’d)**

***Ethereal Packet Analysis Timeline***

**Warning!!**

**NFS is heavy**

**Notes:**
AH HA.
NFS is a heavy hitter. Especially with respect the “Network Driver’s Memory Pool”

We have to think about what autosave/restore is doing here..

Can we do this a better way, hmmm?   I think Dave Thompson has some ideas about this.   However, let’s not get to hasty yet; more to come later

## Network Traffic Analysis (Annotated)

**Load vxWorks**

**NIC restart**

**Startup.cmd**

**Load EPICS**
**Heavy NFS**

**Still loading EPICS**
**More NFS**

**iocInit is ready**
**EPICS is running**

**AutoSave/Restore**
**Heavy NFS**

**Normal Work**

**Reboot IOC**

**Notes:**
Now the job is done!!!
We have built a nice correlation of packets versus time along with the corresponding event.

This picture clearly shows that we have a problem with NFS.
Why?  Well when the NFS spikes occur we get transmit errors.
This means that the Network Driver’s Memory Pool is depleted.

In this current study the IOC always recovers.  So we do not have a fatal case for the IOC.  However, if this kind of activity continued after the IOC is fully up and running we could have intermittent white screens!!

## Network Analysis (Packet Size Distribution)

**Scl-hprf-ioc05**

**Notes:**
We decided to take a look at a commercial network sniffer:
EtherPeek (Win32-based)  
Works really well but expensive (~$5K)

We need to get an idea of packet sizes, hmmm?

## Network Analysis (NFS/RPC statistics)

**Scl-hprf-ioc05**

**Notes:**
This again is the commercial package.

Well, some useful info is the statistics of the NFS packet sizes that seem to be hitting is us hard in this particular study.

## Network Analysis:Data Collection on Network Queues

**Healthy:**
**dtl-llrf-ioc1a> protocolQValuesShow**
**IP receive queue max size = 50**
**IP receive queue drops = 0**
**ARP receive queue max size = 50**
**ARP receive queue drops = 0**
**value = 28 = 0x1c**
**Unhealthy:**
**scl-hprf-ioc05> protocolQValuesShow**
**IP receive queue max size = 50**
**IP receive queue drops = 107**
**ARP receive queue max size = 50**
**ARP receive queue drops = 0**
**value = 28 = 0x1c**

***PROTOCOL RECEIVE QUEUES***

**Notes:**
These show routines do not come with the vxWorks distribution.
They are available from WRS as an unsupported diagnostic tool.

They call it  “utility.c”. Maybe SNS should provide a link to it on our web site?
==================================================
Network Stack Queues are also very important.

Drop packets in the queues may indicated clients not reading the sockets, hmmm?

Something must be stuck if packets are dropped from the queue?

## Network Analysis:Data Collection on Network Queues

**Healthy:**
**dtl-llrf-ioc1a> ifQValuesShow("dc0")**
**dc0 drops = 0 queue length = 0 max_len = 100 **
**value = 46 = 0x2e = '.'**
**Unhealthy:**
**scl-hprf-ioc05> ifQValuesShow("dc0")**
**dc0 drops = 200 queue length = 0 max_len = 100 **
**value = 48 = 0x30 = '0'**

***IP SEND QUEUES***

**Notes:**
These show routines do not come with the vxWorks distribution.
They are available from WRS as an unsupported diagnostic tool.

They call it  “utility.c”. Maybe SNS should provide a link to it on our web site?

Now, there is something stuck in the driver pool.
Could be that tNetTask is too busy, hmmmm?

## What can go wrong with the Network Stack?

**Disruption of tNetTask via deadlock causing sockets not to be read.**
**User tasks in general should have a priority lower than tNetTask. (i.e. greater than 50)**
**Do not create and then take SEM_INVERSION_SAFE semaphores before making a socket call or your task could be promoted to run at tNetTask level**
**tNetTask netTask 1cee480 0+I PEND**** **

## What can go wrong with the Network Stack?

**Application may have deadlock conditions which prevent them from reading sockets.  **
**If inetstatShow (or equivalent in other systems) displays data backed up on the send side and on the receive side of the peer, most likely there is a deadlock situation within the client/server application code.**
**Running both server and client in the target by sending to****127.0.0.l or to the target's own IP address is a good way to****detect this kind of problem.**
**Heavy NFS traffic may require an increase in driver memory pool.**

## Results/Conclusions

**The Network Analysis allows tuning of the network stack from apriori information as well as empirical data collected from the real environment.**
**We have discovered some devices on our network that have improper configurations and hence cause unnecessary traffic.**
**We have discovered that NFS is really a heavy hitter and that autosave/restore request files should be stored in one location.**
**We have discovered that IGMP snooping must be supported on the CISCO edge switches to contain Allen Bradley Control Logix PLC multicast traffic.  Multicast traffic should be contained in general.**
**We moved from the CISCO 3500 series to the CISCO 2950 series**
**CISCO 3500 series only supported CGMP snooping**
**We learned that sometimes IOC application errors are the main cause of Network Stack Exhaustion and/or failure.**
**We have added an “open-source” network sniffer (Ethereal) to our EPICS Network trouble-shooting ToolKit.**
**We have built in the Network diagnostics show routines from WRS in to our IOC’s common support library.**

## Outline

**Introduction**
**Implementing a network stack in the context of a Real-Time OS (RTOS)**
**Basic Definitions and Memory Pools**
**Network Stack Flow Diagram**
**Network Traffic Analysis (w/ethereal)**
**What can go wrong with the Network Stack?**
**Results/Conclusion**

## Basic Definitions

**Mbufs (deprecated):**
**stores small stack data structures such as socket addresses, and packet data.  Mbufs were designed to facilitate passing data between network drivers and the network stack, and contain pointers that can be adjusted as protocol headers are added or****stripped.  Mbufs contain space within them to store small amounts of data.  Larger amounts of data were stored in fixed-sized clusters (typically 2048 bytes), which could be referenced and shared by more than one mbuf.**
**Clusters:**
**Network Data containers of various sizes in bytes**
**Data containers must be a power of two**
**cBlks:**
**The cBlk is a structure that contains a pointer to the cluster data, the cluster size, and an external reference count.****the "cluster block" was added, supporting the zbuf sockets ****interface and multiple network pools in addition to cluster****sharing.**
**One cluster block is required for each cluster **
**mBlks:**
**The mBlk is a structure that contains a pointer to a cBlk or another mBlk. mBlks are basically a modified version of the BSD style mbufs.  The difference is that they now reference external clusters rather than carrying data directly. They are now called "mblocks.“**

**Fundamental Data Structures**

## The 3 main Network Memory Pools

**Network Stack “Data” Pool:**
**Data pools are used for packet send data with extra space for protocol headers. Clusters from the pool are allocated in the socket layer. The function which offers info about it is ****netStackDataPoolShow().**** You can configure it with the definition of NUM_64, ... , NUM_2048.   **
**application layer ******** network stack layer ********network driver**
**Network Stack “System” Pool:**
**System pools are used for network structures (sockets, routes, etc). The function to offer info about it is ****netStackSysPoolShow().**** You can configure it with the definition of NUM_SYS_64, ... , NUM_SYS_512 **
**Network “Driver” Interface Pool:**
**Buffer pool for each network interface. Data from the wire is received in clusters from a network device pool. These buffers are then passed up to the network stack. This pool is also used for staging packets to be transmitted by the target. The driver pool can be shown with the following utility routine: ****endPoolShow(“dc”,0)**** for our MVME2101 boards.  Call ****muxShow()**** to show network driver info.**

**Notes:**
The endPoolShow routine does not come with the vxWorks distribution.
The routine is part of a suite of show routines available from WRS as an unsupported network diagnostic tool.

They call it  “utility.c”. Maybe SNS should provide a link to it on our web site?

## More on the Driver’s Pool

**Cluster size for ethernet is ****1520**
**Cluster size has to be big enough to receive or transmit the maximum packet size allowed by the link layer. In this case that is 1518 bytes**
**Two extra bytes are required to align the IP header on a 4 byte boundary for incoming data.**
**Default number of clusters is 80.**
**END network drivers lend all their clusters. **
**Clusters = numRds + numTds + NUM_LOAN **
**Where numRds (32) is the number of receive descriptors**
**Where numTds (64) is the number of transmit descriptors**
**NUM_LOAN (16) is the number of loan buffers**
**mBlks = 4(numRds + NUM_LOAN)**
**Currently in the field for SNS06a and SNS06c we have:**
**numRds = 32,  numTds = 64, and NUM_LOAN = 16**
**mBlks = 192, Clusters = 112**
**Should this be increased for some Apps? If yes, we need a configuration parameter in the “other” field.**
**Driver Pool for END drivers are configured in:**
**$(WIND_BASE)/target/config/<bsp>/configNet.h **

## Basic Definitions (Cont’d)

**Queues are used to hold data waiting to be processed**
**Queues are implemented as a linked-list.**
**Clusters are chained the to the queue's linked list**
**Types of Queues:**
**Receive Queues:**
**IP PROTOCOL RECEIVE QUEUE **
**FRAGMENT REASSEMBLY QUEUE **
**ARP RECEIVE QUEUE **
**TCP REASSEMBLY QUEUE **
**SOCKET RECEIVE QUEUES **
**Send Queues:**
**SOCKET SEND QUEUES **
**IP NETWORK INTERFACE SEND QUEUES **

**Network Stack Queues**

**Notes:**
2.2 RECEIVE QUEUES     --------------   1. IP PROTOCOL RECEIVE QUEUE      -------------------------      This queue stores DRIVER or DATA cluster containing incoming      packets passed to IP from all configured interfaces.      IP processes the packets and passes them to the protocols      (TCP, UDP...).  Protocols process the packets and queue      data in the destination socket receive buffers.  If packets      come in bursts and the queue is full, the excess is      dropped.      The default value for this queue is 50 packets.   2. IP FRAGMENT REASSEMBLY QUEUE      ----------------------------      This queue stores DATA or DRIVER clusters that contain      fragments of a larger IP datagram waiting to be reassembled.            The maximum size for an IP packet is 65535 bytes.  Packets      exceeding the maximum transmission unit (MTU) for a link      layer protocol must be fragmented.      For example, for Ethernet 10BaseT, the maximum packet size      is 1518 bytes. Subtracting the Ethernet header and checksum      (18 bytes), the IP header (20 bytes), and the UDP header      (8 bytes) leaves 1472 bytes.  A UDP message consisting of      1473 bytes arrives in 2 separate IP packets.      There is a default timeout to drop the fragments if they      cannot be reassembled in time.  There is no explicit      queue size.  IP fragments are dropped also when a pool      is out of clusters and the system drains the protocols.    3. ARP RECEIVE QUEUE       -----------------             This queue holds DATA or DRIVER buffers containing ARP       packets which are used for hardware address resolution.       If packets come in bursts and the queue is full, the       excess is dropped.      The default value for this queue is 50 packets.          4. TCP REASSEMBLY QUEUE       --------------------      This queue contains DATA or DRIVER clusters containing TCP      segments that arrive out of order.      For example, the maximum TCP message size over Ethernet      10BaseT is 1460 bytes (assuming no TCP options).      TCP avoids IP fragmentation, so each segment is 1460      bytes maximum.      The socket receive/send buffer sizes determine the      maximum send and receive window sizes for the connection.      If socket receive buffers are large, the peer TCP will be      able to send a large number of segments within the window.      TCP has to pass data to the application in the order in      which it was sent.  When packets arrive out of order, or      are lost, TCP stores these segments in a reassembly queue      until the missing segment arrives.      There is no limit on the queue size nor is there a timeout      period to drop the fragments.    5. SOCKET RECEIVE QUEUES      ---------------------      Each socket's receive buffer stores the DATA or DRIVER      clusters until the application reads the socket, at which      point, the clusters would be freed.  This is important:      Driver or DATA clusters are freed IF AND WHEN applications      read the sockets.  If they don't, data flow may stop      eventually.      The default value is 8192 bytes for TCP sockets and 41600      bytes for UDP sockets.2.3 SEND QUEUES    -----------   1. SOCKET SEND QUEUES      ------------------       Each TCP socket's send buffer holds DATA clusters      containing packets waiting to be acknowledged.  The default      value for TCP sockets send queues is 8192 bytes.      UDP sockets send queues do not hold any data, but the stack      checks that the message size does not exceed the send socket      buffer size (9216 bytes). A DATA cluster is freed as soon      as the driver copies the data to its own clusters prior to      transmission.   2. IP NETWORK INTERFACE SEND QUEUES      --------------------------------      IP has a send queue for each configured interface which      stores the outgoing packets, DATA buffers, that IP needs to      pass to the driver for transmission.  If a driver is busy      transmitting, and this queue is full, the packet is dropped      and ENOBUFS error is returned to the application.  The default      value is 50 packets.             FIGURE 1: MEMORY POOLS AND QUEUES DEFAULT VALUES                       --------------------------------------             RAW SOCK SEND BUFF                          SYSTEM POOL      ******************                      ******************      *    DATA        *----------->          *    SOCKETS     *      *   CLUSTERS     *           |          *    ROUTES      *      *  8192 bytes    *           |          *  PCBs, TCPCB   *      ******************           |          ******************                              IP SEND QUEUE      UDP SOCK SEND BUFF    FOR "FEI" END DRIVER      ******************    ******************      *  DATA CLUSTERS *--->*  DATA CLUSTERS *---------->      *   9216 bytes   *    *   50 PACKETS   *          |      ******************    ******************          |                                                        |                               IP  SEND  QUEUE           |      TCP SOCK SEND BUFF    FOR "ELT" BSD 4.4 DRIVER    |      ******************    ******************          |      *  DATA CLUSTERS *--->* DATA CLUSTERS  *----------|--------      *  8192 bytes    *    *    50 PACKETS  *          |       |      ******************    ******************          |       |                                                        |       |                         ********************           |       |                         * ARP RECEIVE QUEUE*<-<--------|-------|<                         **** 50 PACKETS ***  |         |       ||                                              |         V       ||      UDP SOCK RCV BUFF    IP FRAGMENT QUEUE  | FEI END DRIVER  ||      ******************    ***************   | **************  ||      * DRIVER, DATA   *    * DRIVER, DATA*   |<*END DRIVER  *  ||      *  41600 bytes   *<---*CLUSTERS/FRAM*     *            *  ||      ******************    ***************     *RECV:       *  ||             |                     ^          --*CLUSTERS: 64*  ||             <-----------------    |          | *CLBLKs:  64 *  ||                              |    |          | *MBLK:   128 *  ||      RAW SOCK RECV BUFF    IP RECEIVE QUEUE  | *TX FRAMES:32*  ||      ******************    ***************   | **************  ||      * DRIVER, DATA   *    *DRIVER,DATA  *<--|                 ||      *CLUSTERS/FRAMES *<---*CLUSTERS/FRAM*                     ||      *   8192 bytes   *    *  50 PACKETS *<-- ELT BSD4.4 DRIVER||      ******************    ***************   | **************  ||                              |    |          | *BSD44 DRIVER*  ||             <--------------- |    |          | *            *  ||             V                     V          | *FRAME       *  ||      TCP SOCK RECV BUFF   TCP REASSEMBLY Q.  |-*DESCRIPTORS:*<- |      ******************    ***************     *  Rx: 20    *   |      * DRIVER, DATA   *<---* DRIVER,DATA *     *  Tx: 1     *--->      *   8192 bytes   *    *CLUSTERS/FRAM*     *  Loan: 16  *      ******************    ***************     **************

## Network Traffic Analysis (Cont’d)

***Ethereal Packet Analysis Timeline***

**Load vxWorks**

**T + 3.4 sec**

**Load startup.cmd**

**T + 23.5 sec**

**Notes:**
We filter on RSH since we know this is how the SNS loads up vxWorks,

The time above is consistent on all of our IOCs since they all boot in a standard way.

## Network Traffic Analysis (Cont’d)

***Ethereal Packet Analysis Timeline***

**vxWorks initialize**
**Restart NIC**

**T + 5.240 sec**

**NIC is ready again**

**T + 20.736 sec**

**EPICS neighbors**
**come knocking**

**Notes:**
What was a discovery for me is that the NIC is initialize twice.
Once by the code in the Boot Flash and again after vxWorks is loaded over the network and initialized.

Then of course, the EPICS neighbors come knocking at the door again.

## Network Traffic Analysis (Cont’d)

***Ethereal Packet Analysis Timeline***

**Do NFS**

**T + 30.7 sec**

**Notes:**
We definitely use NFS with our IOCs!!

Pay close attention to the NFS “LOOKUPS”

There is more to come.

## Network Traffic Analysis (Cont’d)

***Ethereal Packet Analysis Timeline***

**Do EtherIp**

**T + 87.16 sec**

**RSTs from a previous**
**connection**

**Notes:**
Notice the RST from a previous connection.  Remember that when the IOC is rebooted while CA clients have active connections a TCP RST will be sent by the IOC once it is back up.  You will see this behavior on the next few slides as well.

This IOC uses the etherNet IP driver to communicate with Allen Bradley RSLogix PLCs.

Let’s see when ether_ip was doing its business.   What kind of traffic load does it present to the IOC’s network driver memory pool?

## Network Traffic Analysis (Cont’d)

***Ethereal Packet Analysis Timeline***

**IocInit is running**

**T + 88.711 sec**

**Note:**
**EPICS ready after it sends**
**out its CA beacons**

**Notes:**
Okay, IocInit starts here.  This means that the IOC is “open for business”

Note our clue is that the IOC sends beacons out!!!  Cool, eh!!

## Network Traffic Analysis (Cont’d)

***Ethereal Packet Analysis Timeline***

**Talk to Archiver**

**T + 90.73 sec**

**RSTs from a previous**
**connection**

**Notes:**
The Channel Archiver is one of our biggest CA clients.  So we would like to see what traffic impact results.