Newsgroups: bit.listserv.ibmtcp-l
Date:         Sat, 3 Oct 1992 09:12:00 PDT
From:         Michael Stein <CSYSMAS@UCLAMVS.BITNET>
Subject:      Re: selectex parameters

> Michael Stein writes:
> > Eeeek!  Overhead**2.  ECBLISTS are *NOT* a goodness.

> Really? The "platform" uses all sorts of PC/SRB/IRB stuff to do
> IUCV -

Yup. Ever time it?  I see about 500 usec of *TASK* CPU for a
(small) send or receive TCP socket call.  That's for an ES/9000
M900, which has a 9 ns cycle time.

I'm timing user task time (TIMEUSED), so it ONLY includes the
IUCV overhead done by the user task (no SRBs, nothing in other
address spaces).  Thus it does NOT include the TCP checksum nor
*any* TCP/IP protocol processing.  It doesn't even include the
IUCV SRB time in the user address space.

Since IUCV is used to communicate between the user and TCPIP
address spaces it makes sense to assume that the TCPIP address
space consumes an equal amount of time in it's calls of IUCV
(just a guess, no measurements).  So that's 1000 usec of CPU.

Now ignoring any other processing this implies that 1000 calls a
second will use 100% of a whole CPU.  This puts an upper bound on
a single CPU data transfer (depending on the size of the send or
receive) something like this:

   size          max rate (bytes/sec)

  512             512K
 1024            1024K
 2048            2048K
 4096            4096K

The original 500 usec number was for a very small send and
receives, much less than 512 bytes.  I'd assume the above numbers
are very optimistic since the overhead probably increases with
the size of the send -- no measurements though.

Since the above ignored all the CPU time used by IUCV SRBs, TCP
and IP protocols, TCP checksums, and CPU used for actual I/O
processing, an actual TCP connection *has* to be slower.

Does data to/from the "offload" box travel via IUCV?  If so
the these limits apply to it too!

A few comparisons:

A MVS QSAM PUT macro (MACRF=PM) takes about 5 usec of CPU, but
doesn't cross address spaces nor privilege boundaries.

A locally written general cross address space service can send
10,000 2048 byte records between address spaces in 2 seconds of
CPU time counting *all* the time in both address spaces
(including SRB time).  It's about 50% SRB time, so the one side
task time would be:

   2 sec / (10000 times) ->  200 usec for both size all time
   200 usec / 2 sides    ->  100 usec for one side
   100 usec * 50% task time ->  50 usec one side task time

This local service is about 9 years old now and was NOT written
for performance.  It can do 10000K bytes/sec with one CPU at the
2K send/receive size (real measured data).

> scanning a list of ECBs can't be a high overhead.

I'm overly sensitive to n**2 things, they don't scale well.
Things like bubble sorts and linear searches.  The ECBs used in
IUCV are probably involved in a very small part of it's overhead.
But every little bit does add up...

> We use 40 in our X.25 GATE task and it's not a bottleneck.

If you will never need more than 40 and the current overhead
is acceptable then it's not and won't be a problem.

If the number of ECBs is related to a "number of sessions" then
it might be.

It's the n**2 factor which is the problem.  For 40 it's 1600.  At
100 it's 10,000.  1000 is usually near the "breaking point".  At
1000 it's 1,000,000 and a 1 usec cost will chew up a whole second
of CPU.  Don't even think of 10,000.

In this day, 1000 sessions isn't impossible and over the life of
some software is even likely.

> If you use "Test all ECBs before WAIT" (or even "test the next
> n ECBs before WAIT") and "Compare-and-swap POST", the more you
> are overloaded, the less you use WAIT and POST SVCs at all.

You are only saving time relative to the system wait and post
services.  This is a significant savings, however the algorthm is
still n**2 since you are doing linear searchs of the ECBs.

> What alternative are you suggesting?

If you are posting yourself then you have a chance to save *lots*
of overhead.  You could add an element to a "to do" queue instead
of posting a separate ECB for each event.  Then just post (or
quick post) a single work ECB. When your mainline wakes, it then
just processes the work queue.  This results in NO ecb list
searching for the events where you control the posting.  A side
effect is that the code is simpler!

> > Since "sockets" seem to be some sort of standard (and as such
> > can't be replaced), it'd be nice to see a select without more
> > than one ECB in it's wait...
>
> select _does_ normally have only one ECB in its wait, if you
> are only using TCP/IP sockets, and have no ECBs of your own.
> The multiple ECBs come from AF_IUCV sockets, which are a
> "platform" thing and aren't managed by TCP/IP, and from the
>  optional user ECB.

Well, select is a "polling" wait service.  So it's designed to do
linear searches to start with.  Adding yet another thing for it
to search didn't strike me as great.

I can see the usefulness of having select accept a list of ECBs,
it just seems to me that "there has to be a better way".

Michael Stein  (Still counting the nanoseconds...)