From: rec@aim.com (Bob Cousins)
Subject: Re: Sun 600MP Benchmark Anomaly
Organization: /usr/lib/news/organization
Date: Mon, 27 Jul 1992 21:19:27 GMT

In article <1992Jul27.155559.19995@crd.ge.com> davidsen@crd.ge.com (bill davidsen) writes:
>In article <JIM.92Jul24155432@hunter.cs.strath.ac.uk>, jim@cs.strath.ac.uk (Jim Reid) writes:
>
>| So does this mean that this "symmetrical multi-processor SunOS" isn't
>| really what it claims to be? If only one CPU at a time can be executing 
>| interrupt service code, what we've got is an OS with one dedicated
>| (but not fixed) I/O processor.
>There are tradeoffs to having multiple processors handle interrupts, and
>if the latency is low enough with one processor you are possibly better
>off with using only one and avoiding all the context switches. If one
>can't keep up, then you will have to take the CS overhead and use them
>all. I'm reassured that Sun has left this open to software change, and
>I'm sure if people complain about how it works in Solaris 2.0 Sun will
>change it.
>bill davidsen, GE Corp. R&D Center; Box 8; Schenectady NY 12345

There are a number of issues at play here. One can create a mendacity sequence
for MP interrupts as follows:

1.	Only one CPU (the master) ever receives interrupts except for
	cross-processor interrupts. (Few will argue that this is symmetrical.)
	This is a common technique in master/slave systems. AIM's results
	for these machines are often disappointing when one CPU cannot handle
	the burden.

2.	Only one CPU at a time can receive interrupts except for cross-
	processor interrupts. (More will argue that this is symmetrical.)
	This is a common technique in early UNIX MP implementations and
	a number of proprietary OS's. It's a reasonable "first step."
	These machines suffer from the above limitation, also.

3.	Only one CPU at a time can be executing interrupts. (This limit
	is often from the historic requirement that UNIX executes interrupts
	on the current task's kernel stack, can't allow task switching,
	etc.) This *may* be symmetrical or not depending on implementation
	and your definition. Same performance problems.

4.	Only one interrupt dispatch at a time (multiple outstanding
	interrupts, but the dispatch is single threaded). This is typically
	implemented in a "symmetric" fashion. (This is often used on
	systems which have heavy kernel based support for FPUs, etc. where
	a large amount of kernel state must be maintained for each CPU but
	where it isn't efficient to use "standard" techniques.) These machines
	often outperform machines of classes 1-3 because of their more
	efficient interrupt handling. Interrupt dispatch is seldom a limiting
	factor except in situations with high interrupt count peripherals
	such as LANs, early dumb SCSI controllers, massive numbers of UARTs.

5.	Any CPU can field an interrupt at any time. This is considered the
	*most* symmetrical. In actuality, we see little benchmark difference
	between this and #4. This is often because of the relative infrequency
	of #4 delaying interrupt service sufficiently to cause lost performance
	(an extra disk rotation, lost character or packets, etc.). For those
	machines where this is a problem, there are usually other bottlenecks
	which limit performance gained by interrupt services. (This is not
	intrinsic, this is just an emperical conclusion.)

My personal favorite method is #4 with a twist: use a microcontroller to
"single thread" interrupt dispatch. This microcontroller can be programmed
with "interrupt policy" code during OS initialization. Since the microcontroller
knows which interrupts are outstanding and which CPUs are busy (and how long
since the last interrupt), it is able to make some intelligent choices as
to the "optimal" CPU to interrupt. This also avoids the traditional spin-lock
over shared interrupt masks common in #5. The first implementation could be
simple code which forces mode #1 or #2. Eventually the system could alternate
between modes depending upon load, timing, interrupt mix and a host of other
parameters. Fortunately, most fast microcontrollers today can execute policy
code and choose a processor in a few tens of microseconds, which is typically
well under the variance of interrupt latencies for current machines.
-- 
________________________________________________________________________________
Robert Cousins                    rec@aim.com                  (408)748-8649x212
Consultant to Aim Technology                          Speaking for myself alone.