From: sung@ceco.ceco.com (Sung Han)
Newsgroups: comp.realtime
Subject: Summary of opinions and info on realtime kernels (long)
Keywords: more subjective than factual
Date: 3 Jun 91 22:29:47 GMT
Organization: Commonwealth Edison Co., Chicago, IL


A While ago, I solicited responses from people on their experiences with three
real-time kernels for 680x0-based processor boards, namely pSOS+, VRTX, and
VxWorks.  And by request, I'm posting this summary of the responses I've
received, along with my (humble) personal observations on these kernels. 

A word of caution: if you believe in objectivity, don't bother reading this
article, because the material here is entirely subjective.


/*---------------------------------------------------------------------------*/
/*                                                                           */
/*         ############ D I S C L A I M E R S #############                  */
/*         ############ D I S C L A I M E R S #############                  */
/*         ############ D I S C L A I M E R S #############                  */
/*                                                                           */
/*   1)  All opinions expressed here are solely those of the individual      */
/*       respondents, and do not necessarily reflect those of their          */
/*       employers.  This goes for myself also.                              */
/*									     */
/*   2)	If there are factual errors in the following material, I will        */
/*	welcome corrections; however, flames on subjective matters will be   */
/*	ignored.                                                             */
/*                                                                           */
/*   3)	I have no affiliation with any of the vendors listed in this post.   */
/*                                                                           */
/*---------------------------------------------------------------------------*/



I also received performance data on the three kernels, plus LynxOS, courtesy
of the people at the Superconducting Supercollider Lab in Texas.  I've inserted
the results later in this article. 

In addition, I've received some responses that suggested alternatives to the
three kernels that I requested info on.  These responses are listed later.

Here then are the responses, minus names.  I've edited parts of the responses
for clarity (apologies to the authors).

And REMEMBER! please don't associate these responses with me, or my 
organization - they're simply other peoples' opinions, which I'm reprinting.



-------------------------------------------------------------------------------

vxWorks is a solid product; it does what they (Wind River Systems)
say. It has an especially strong communications layer, wide and deep
and rich. Three years ago I compared vxWorks and VRTX and chose
vxWorks because of its superior networking. A colleague conducted a
similar procurement within the last year and told me that vxWorks is
still superior to VRTX on communications. This comm support is
*essential* if you intend to build a modern distributed RT system. If
you are building a standalone embedded application the vxWorks RT
kernel is fine, arguably even excellent, but there are a number of
other RT kernels available that would probably do about as well
overall. *All* RT applications here are distributed, so vxWorks is
being chosen again and again...

-------------------------------------------------------------------------------

Hi.

I went through a similar process about one year ago. I considered pSOS+,
VRTX, VxWorks, and a message-passing OS called Unison.
I ended up favoring VxWorks by a large margin. My main reasons:

1) Wind River was very quick and efficient when I asked for a demo version
  (30 days etc.) The others just seemed confused.

2) VxWorks is the most complete package. The kernel is just a small part of
   the larger system, which Wind River realizes more than anybody. Things
   like the ram disk driver and run-time loader are really great for my 
   application.

3) VxWorks networking is very solid, and certainly more mature than any
   competitor's. Their I/O system in general is very well set up 
   (network, serial, and RT-11 (ram-disk) devices all mapped into the file
   name space).

4) Building different systems, boot PROMS, etc. is well planned out and
   relatively easy. This is paying off now that we are moving to stand-alone
   oepration.

Until version 5.0, VxWorks did not really have "source-level" debugging,
just "symbolic" (assembly-level) debugging, but this should be improved with
VxGDB in 5.0.

Good luck,

-------------------------------------------------------------------------------

I haven't used any other commercial real-time OS., but I'm quite happy
with VxWorks.  Using the Unix development environment and downloading
over ethernet is quite nice.  We went thru an unpleasnt time where the
remote source level debugger wouldn't work with operating system
updates, and we had to get a new version from Sun Consulting.  Now that
VxWorks uses gdb as a base, everthing seems to work quite well.
We tie our real-time system to workstations using Remote Procedure Calls and
sockets, which seem to work just as advertised.

The technical support is reasonably good, and the product seems fairly bug
free.

-------------------------------------------------------------------------------

I have been working with VRTX32 (without Velocity) for a little over two years
now.  I have found it to be reliable and reasonably well documented; the only
bug I ever found in it was fixed in the latest release.

On balance, I would say we have had good luck with VRTX, but I haven't had
much experience with other Real Time OS's so I can't say whether the others
are as easy to use or not.

-------------------------------------------------------------------------------

I have been using VRTX/ Velocity for the past 2 years. The RTSource
debugger is very good, but the rest of their stuff I have found to be
"buggy" (esp. the TCP/IP package) and difficult to install. Ready
Systems' support is disgusting, unless you get friendly with a good FAE
(who are meant to do pre-sales only).

-------------------------------------------------------------------------------

We have recently done an evaluation of the same products.  We got on-sight
demos of each, then got evaluation copies of each system except Uniflex.
Uniflex is a Unix "work-a-like".  I would think it should be compared with
LynxOS.  The other three are considered hard real time.

Without making any criticism's about the systems, you will find that they
(pSOS, VRTX32, VxWorks) all provide a fast real time kernel, UNIX compatible
sockets, and a standard ANSI run time library.  All are downloadable, and
ROM-able systems.  Development is done on a host system with code cross
compiled for the target and downloaded.  VRTX and VxWorks have a user
shell that can be used to invoke functions for debugging.  In addition,
VRTX and VxWorks provide NFS facilities to the target.

All the systems could be hosted on SUN.  pSOS+ could also be hosted on HP.
Its debugger uses X Window which is nice for exporting the display to any
workstation on your network.  The XRAY+ debugger is very useful.

We had three people on the evaluation.  I actually worked primarily with the
pSOS+ software.  Incidentally, I'll tell you up front that pSOS+ is the one we
choose.  The decision was based on the fact that pSOS+ is hosted on more
workstations, and targeted on more targets than the others.  

The debugger seems a bit more sophisticated.  For instance, it is a board
level debugger (as is with VRTX), that can break on any task in the system.
VxWorks bebugs one task at a time.  When it breaks, the others continue to
run.  Come to think of it, that may also be the case with VRTX.

The SCG interface (on HP) is X Window.  We have successfully exported this
screen to other X Window displays.  I.E., The debugger runs on one machine
and displays on an X terminal.  This is useful in the following context.
We have a number of HP's to run the development software.  Then, we can
actually log into those machines and run the display to another terminal.
This allows more users.

VxWorks is probably a bit easier to start up since it comes installed (as
I understand it).  pSOS+ and VRTX each took some work from the factory to
understand how to use what you are given.

I think pSOS+ has the best documentation, then probably VRTX and VxWorks.
Of course, I can't say that I have spent much time with the latter two.

>From the performance tests that we ran, they all seem comparable.  We devised
a series of tests, and implemented it accross the three platforms.  Then
measurements were made using the same hardware in each case.



###############################################################################

/*
 *
 * The following responses were pointers to other real-time systems:
 *
 */

###############################################################################


-------------------------------------------------------------------------------

You've mentioned VRTX, pSOS, and VxWorks -- What about PDOS?  
You are missing a great operating system that provides full native and
cross development, posix file system, networking, tape and hard
disk support, and good customer support.  PDOS can also claim excellent
real-time response with 0 interrupt latency on high level interrupts.

[PDOS is marketed by Eyring; contact either the author of this response, at

    rick@Eyring.COM  uunet!lanai!rick
    
    or try the following person:

    Walt C. Jones
    Director of Marketing & Sales
    Eyring
    1455 West 820 North
    Provo, Utah 84601
    (801) 375-2434
    
S.H.]

-------------------------------------------------------------------------------

I would suggest using OS-9 for realtime application on 680x0 CPU's. It is
availible from MicroWare. There are ports of it for several variants around
too. (such as Signetics 68070 - a suped up 68000 clone)(also, for Mac's, 
Amiga's, and Atari ST's, as well as some 680x0 boxes designed just for OS-9)

[I saw a demo of OS/9 on a new MVME165/Motorola 68040 port, and it looked quite
 nice actually.  It provides all necessary development tools on the target
 system, including a C compiler, source debugger, etc., along with a full
 Unix-like file system.  Plus there's a pretty good OS-9 user's group too.
 
 Contact Microware at the following address:
 
     MicroWare Systems Corporation
     1900 N.W. 114th Street
     Des Moines, Iowa 50322
     (515) 224-1929

S.H.]

-------------------------------------------------------------------------------

You should also check out Precise/MPX from:

              Precise Software Technologies Inc
              Suite 308, The Mallorn Centre
              301 Moodie Dr.
              NEPEAN, Ontario Canada
              K2H 9C4

              Tel:    (613) 596-2251
              Fax:    (613) 596-6713

Precise/MPX is a commercial product based on the Harmony
Realtime, Multitasking, Multiprocessing Operating System, an
ongoing (8 years) research project at the National Research
Council of Canada. Precise/MPX has been ported to Intel 80x86 and
Motorola 680x0 and 88K microprocessor families. The host
development tools are available on the PC AT, Sun 3, Data General
Aviion, Apple Macintosh and others.

I can supply technical details about Harmony, via email or
regular mail (we have a Harmony bibliography, technical
description and short manual set available for free). For
questions of availability for particular host/target
configurations, pricing and licensing contact Jeremy James,
President of Precise Software at the above address or phone
number.

Hope this helps.

[Author of response is:
    Stephen MacKay  <mackay@iit.nrc.ca>
    Harmony Research Project
    Software Engineering Lab.
    Inst. for Information Technology
    National Research Council
S.H.]

-------------------------------------------------------------------------------

I'm a bit biased since I work on the following product, but I thought you might
like to know about REAL/IX:

Modcomp's REAL/IX is a fully preemtive, low-latency, realtime UNIX operating
system (System V.3).  It conforms to AT&T's System V Interface Definition
(SVID) and passes the System V Verification Suite (SVVS).  It runs on a 68030
based VME system (with SCSI)  (Motorola 147 & 141 cards).  Features include:

o FULLY Preemptive Kernel (no pre-emption windows)
o FAST context switch time (Best Case context switch time: approx. 30 microsec)
o Directly Connected Interrupts
o Loosely Connected Interrupts
o Low Interrupt Latency (Maximum Interrupt latency: approx. 100 microsec)
o Event Driven Priority Scheduler
o Flexible Task Control
o Shared Memory Facilities
o Preallocation of Resources
o Process lock in Memory
o High Performance File System
o Asynchronous I/O
o Priority Based I/O
o Direct I/O
o High Resolution Timers
o Fast Interprocess Communications
o Fast Process Synchronization
o Common Event Notification
o User Extensible System Services and Interrupt Handlers

There's also a book about REAL/IX:

REAL-TIME UNIX SYSTEMS
Design and Application Guide
Furht, Grostick, Gluch, Rabbat, Parker, McRoberts
Kluwer Academic Publishers

It describes REAL/IX in detail, and even includes two case studies.

[If you'd like to inquire about REAL/IX, contact the author of this response 
 (Steve R. Pietrowicz) at:
 
     uunet!modcomp!rlxdev!srp
 or  ...!uunet!modcomp!srp
 
S.H.]

-------------------------------------------------------------------------------

LynxOS is UNIX System V (with BSD extensions).  LynxOS is realtime.  It
contains no ATT code.  It is POSIX compliant.  It was selected as the
runtime system for the NASA space station.  You can buy it to run in an
embedded system, e.g., a Motorola MVME 147 board.  Or you can buy it to do
C development, e.g., on a 386 or 486 PC.  Or you can do both, e.g., do your
development on the 147 board as well as run your real-time application on
the 147 board.  I believe it supports everything you asked for, and more.
You should call them and find out.  They're hot!

[Actually, as I understand it, Lynx may not be appropriate for 'hard' realtime
 systems, as the Unix compatibility exacts a performance penalty.  The 
 performance results shown later support this.  However, this is not meant to
 knock LynxOS; it still looks like a fine system, if Unix is what you'd like 
 in a real-time system.  You might also look at Uniflex, described later.
 
 Lynx's address is:

    Lynx Real-Time Systems, Inc.
    16780 Lark Ave
    Los Angeles, CA 95030
    (408) 354-7770
    
S.H.]

-------------------------------------------------------------------------------

I like PolyFORTH. 
[this was the entire text of the message - I don't know much about this
particular system - S.H.]

-------------------------------------------------------------------------------

[Another alternative may be Uniflex, a Unix-like hybrid system.  It manages two
parallel lists of tasks - a realtime list, and a Unix list.  The realtime
tasks always have priority over the Unix tasks.  Most of the development tools
(compiler, etc.) reside on the target, and are variants of GNU tools.  That's
about all I know about Uniflex - their address is:

    UniFLEX Computing Ltd.
    111 Providence Road
    Chapel Hill, NC 27514
    (919) 493-1451
    sales: (800) 486-1000

S.H.]


###############################################################################


As I mentioned, the people at the SSC Project ran a series of performance tests
on pSOS+, VRTX, and VxWorks, along with LynxOs.  I've printed some excerpts
below.  Thanks to Mike Allen and Carl Kalbfleich for sharing this data.

[taken from the paper "Overview of Real-Time Kernels at the Superconducting
Super Collider Laboratory"]
[The platform used for the tests was an MVME147S-1 VME-based, single board
computer with a 25MHz 68030 cpu]


    Throughput measurements are tabulated in Table 1 and what follows is a briefdescription of each test as it appears in the table.
    
  1) Create/Delete Task: This test measure the time it takes to create and
     delete a task.  A task deletes itself as soon as it is created.  the 
     created task has a higher priority than its creator, so the time quoted
     actually includes a create, start, delete, and two context switches.
  2) Ping Suspend/Resume Task: A low priority task resumes a suspended high
     priority task.  The high priority task immediately suspends itself.  This
     measurement includes two task context switches and the time it takes to
     suspend and resume a task.  There is no facility to suspend and resume a
     task on LynxOS apart from signals.  So this test was not performed under
     LynxOs.
  3) Suspend/Resume Task: This is identical to the previous test except that a
     high priority task suspends and resumes a suspended lower priority task so
     that there is no context switching.
  4) Ping Semaphore: Two tasks of the same priority communicate with each other 
     through semaphores.  Task A creates a semaphore, gets the semaphore then
     creates Task B which blocks when it attempts to get the semaphore.  Task A
     then releases the semaphore which immediately unblocks Task B.  Task A then
     attempts to get the semaphore which causes it to block until Task B 
     releases it.  The two tasks then alternate ownership of the semaphore
     thereby causing conbtext switches.  In our version of VxWorks, two separate
     semaphores are required because round-robin scheduling is not supported.
     [this was version 4.0.2 of VxWorks; round-robin mode, whileit may be
     possible in version 5.0.1, is still difficult to implement - S.H.]
  5) Getting/Releasing Semaphore: The time reporterd includes the time it takes
     to get and immediately release a semaphore within the same task context.
  6) Queue Fill, Drain, Fill Urgent:  We first time how long it takes to fill
     a queue with messages and then we tim how long it takes to drain the queue.
     Finally we repeat the two tests with priority messages i.e. messages are
     sent to the head of the queue.  VxWorks 4.0.2 does not support message 
     queues but ring buffers with semaphores gives the functionality of a 
     message queue. [VxWorks 5.0 now has message queues - S.H.]  LynxOS uses
     SystV message queues with priority messages handled differently.
  7) Queue Fill/Drain: A single task sends a message to a queue which the task
     immediately receives on the same queue.  There is no task switch nor are 
     there any pending queue operations.  The next test consists of two tasks
     with two queues.  The two tasks alternate execution by sending to the
     queue that the other is blocked waiting to receive from.  The total time
     now includes context switches, queue pends and sending plus receiving a
     message.
  8) Allocating/Deallocating Memory: We measure the time it takes to allocate a 
     number of buffers from a memory partition and the time it takes to return
     those buffers to the partition.
     
  9) Real-Time Response: We quantify the real-time response of the kernels by
     measuring the interrupt service response and the interrupt task response.
     The interrupt service response is the time it takes to execute the first
     instruction of an interrupt service routine (ISR) from when the interrupt
     occurrs.  The Task response is the time it takes for a user task to resume 
     excution from when the interrupt occurrs.


[ Note: there are two entries in the below table for pSOS+.  This is because
the pSOS+ programming interface comes in two forms: through a C Interface
Library (CILxxx.s), and through a Kernel Jump Module (KJMxxx.s).  Without going 
into specifics, suffice it to say that the standard pSOS+ distribution includes 
the KJM package, and this was the original configuration tested.   The CIL
package is faster, but it is not yet available, although I understand that SCG
will come out with it very soon. - S.H.]   
  

[Each column lists the average time for each test, in microseconds.  The
asterisk(*) marks the fastest kernel for each test.  Note that the message
queue times were fastest with VxWorks, but remember that these were not TRUE
message queues that were tested for VxWorks - S.H.]

                             pSOS+    pSOS+    VRTX32     LynxOS    VxWorks  
                             CILxxx   KJMxxx
Create/Delete Task           ---      591       371*       ---       1423
Ping Suspend                 114*     128       142        ---        177
Suspend Resume                71       83        87        ---         69*
Ping Semaphore               193*     219       239        397        232
Get/Release Semaphore         55       63        55         74         33*
Queue Fill                    38       46        26        140         20*
Queue Drain                   36       43        29        132         22*
Queue Fill Urgent             38       47        27*       170         72
Queue Fill/Drain              76       91        59        278         44*
Alternate Qs Fill/Drain      210*     238       252        867        371
Alloc Memory                  29       40        27*        57         68
Dealloc Memory                29*      38        33         20         83

Interrupt Service Response   ---        6         6         13          6
Interrupt Task Response      ---      163       169        175        125



[the following displays the range of times for the above tests.  Originally
this data was in the same chart as above, but I've split them because of space
restriuctions.  Anyway, they might provide an indication of the determinism of
these systems.  pSOS+ with the CIL interface is not included here - S.H.]

                        pSOS+(KJM)      VRTX32        LynxOS        VxWorks  
                        min/max/avg   min/max/avg   min/max/avg   min/max/avg
Create/Delete Task      540/600/591   370/380/371       ---      1378/1446/1423
Ping Suspend            120/130/128   140/150/142       ---       174/182/177
Suspend Resume           80/ 90/ 83    80/ 90/ 87       ---        68/ 74/ 69
Ping Semaphore          210/220/219   230/250/239   390/400/397   228/234/232
Get/Release Semaphore    63/ 64/ 63    55/ 56/ 55    73/ 76/ 74    33/ 34/ 33
Queue Fill               40/ 50/ 46    20/ 30/ 26   136/146/140    19/ 21/ 20
Queue Drain              40/ 50/ 43    20/ 40/ 29   126/136/132    21/ 25/ 22
Queue Fill Urgent        40/ 50/ 47    20/ 30/ 27   166/175/170    70/ 76/ 72
Queue Fill/Drain         90/ 93/ 91    50/ 70/ 59   280/290/278    43/ 48/ 44
Alt., Qs Fill/Drain     230/240/238   250/260/252   860/900/867   366/376/371
Alloc Memory             40/ 40/ 40    20/ 30/ 27    34/ 79/ 57    67/ 71/ 68
Dealloc Memory           30/ 40/ 38    30/ 40/ 33    20/ 21/ 20    82/ 86 /83

Interr. Svc Response      6/  6/ 6      6/  6/  6    13/ 88/ 13     6/ 56/ 6
Interr. Task Response   100/169/163   179/343/169   163/262/175   119/319/125



###############################################################################

Now for my personal observations on these kernels:

I evaluated pSOS+, VRTX Velocity, and VxWorks 5.0 each for about 30 days.  I
was mainly interested in the suitability of these systems for a networked
realtime system.  What I found was that while these systems were remarkably
similar, each one still had a character all its own.  I will therefore try to
point out the differences among them, instead of dwelling on the similarities.

Each system that I got for evaluation consisted of at least the following:

    1) a realtime kernel
    2) a board-level monitor/debugger
    3) a source-level remote debugger
    4) a networking module
    5) a compiler package

VRTX Velocity and VxWorks also include a target shell.
Notes on the above systems follow.


pSOS+:
------

    pSOS+ is the follow-on to the venerable pSOS kernel, from Software
Components Group.  The review package included the pSOS+ kernel, pROBE+ 
board-level monitor/debugger, and the pNA+ Network manager.  Also included was
a compiler package from Microtec, and XRAY+, which is a joint Microtec/SCG
product.

    The pSOS+ kernel is highly flexible, and its feature set is competitive
with the others.  However, the approach used to interface pSOS+ components to
user code is different from the other systems.  While the other systems link
the user code with the system code to resolve referneces, all of SCG's
components are called through software traps, which vector directly into the
pSOS+ kernel.  The kernel then determines which pSOS+ component to call to
service the request.  And while this provides position independence and
eliminates the need to link user code with pSOS+ routines, it also creates some 
overhead, since every call to a pSOS+ system component means a service trap.
This is reflected in the performance figures provided later.  Note, however,
that this applies to the KJMxxx.s (Kernel Jump Module) interface; an 
alternative interface, CILxxx.s (C Interface Library), apparently provides a
better way of calling pSOS+ system modules, and provided a better showing in
the same tests.

    The pSOS+ kernel's messaging facility allows up to four long words of user
data to be passed between tasks.  VRTX only allows a single word, which usually
means that memory has to be allocated in order to send any meaningful data
through messages (the length of VxWorks messages is unlimited).  Also, event
flags in pSOS+ can be directed either at a specific task, or be global.  Events 
in VRTX are always global, and VxWorks doies not have event flags.  pSOS+ also
allows each task to have an asynchronous signal handler.  This signal facility
is not entirely like Unix signals however, as the signal handler will not be
called until the user task makes a pSOS+ system call.

    The pNA+ networking package worked flawlessly, providing full Berkely 4.3 
sockets support.  The competing network packages provide similar functionality,
but only pNA+ has a socket-sharing mechanism, which can be convenient if
you're dealing with numerous independent connections on a single board.

    The XRAY+ source level debugger proved to be a versatile, reliable
debugger.  Its user interface was a bit awkward, however.  The debugging format
used is IEEE695; this format is fully supported by the accompanied MRI compiler
tools package, but I've heard of compatibility problems in using IEEE695 output
from other compilers.

    SCG has just announced the pNFS package.  This provides NFS and RPC support,
and is available in conjunction with their pHILE+ file management system.

    SCG also offers pSOS+/M, which is a multiprocessor version of the pSOS+
kernel.  It provides (nearly) seamless usage of standard pSOS+ calls over
multiple processors connected through various types of media, including
backplane and ethernet.


VRTX Velocity:
-------------- 

    VRTX Velocity is a combination of target software components and host-based
tools.  The target software evaluated consisted of the VRTX32 kernel, RTScope
board-level monitor/debugger, TNX TCP/IP Network manager, RTL runtime C
library, and RTShell target shell.  The host-based tools included the RTSource
remote source debugger, Hyperlink ethernet downloader/command center, and an
Oasys Compiler tools package.  The host side of the Velocity package runs only
on Sun3/Sun4 workstations in SunView.

    VRTX Velocity offers two approaches to development.  One is where tftp is
used to load the system software and RTshell at boot time using TFTP.  The
shell then can be used to load software onto the board off a network through
NFS.  An incremental linking loader is provided, which accepts standard Unix
format relocatable object files.  The shell allows funtions to be called, and
can evaluate most C expressions.  The alternative approach is to manually
download system software to the target board using Hyperlink, an intelligent
downloading program.  Hyperlink can also be used to launch other Velocity
programs, such as RTScope and RTSource.

    The VRTX32 kernel provides a unique approach to identifying tasks.  Each
task has a numeric ID, from 0..255.  They cannot have names, which are allowed
in pSOS+ and VxWorks.  If more tasks are needed, there can be multiple tasks
with an ID of zero, up to 16K of them.  These tasks are referred to as
'anonymous' tasks, and they usually cannot be directly referenced by other
tasks.  For instance, they cannot be deleted by ID, since an ID of zero used in 
system calls usually refers to the caller.  Fortunately, VRTX32 allows tasks to 
be referenced by priority groups, so this is one way to deal with anonymous
tasks.
    
    The VRTX kernel has some features which are lacking in the other kernels;
specifically, it provides character I/O operations at the kernel level, and it
provides mailboxes, which are similar to single-slot message queues.  However,
it is also lacking in other areas - for instance, there are no built-in time/
calendar functions, aside from a tick counter (this is also true with VxWorks);
in addition, when creating a new task, there is no easy way to pass parameters
to it.  Also, creation of tasks is always a single-step process; i.e., create
the task, and if its priority is higher than the caller, pre-empt the caller
and run the task.  This is different from pSOS+, where task creation is
a two-step process - i.e., create, then activate.  VxWorks allows the use of
either approach.  Also, there is the odd restriction that once a message queue
is created, it cannot be deleted.  Ready Systems claims that this is to
prevent fragmentation of system memory.  Finally, unlike pSOS+ and VxWorks,
VRTX32 lacks asynchronous signal handling.

    The RTScope board level monitor/debugger is highly flexible, and provides
two modes of operation: command mode and task mode.  In command mode, all user
tasks are halted, interrupts are disabled, and the standard suite of memory
operations and task control commands can be used.  In tasking mode, RTScope
runs as a task, alongside user tasks, with interrupts enabled.  The system can
therefore be monitored in action without impeding on user tasks.  Note that 
commands which infringe on normal system operations cannot be used in this mode 
- e.g., memory patching, task suspension, task creation, etc., are not
available.  Also, RTScope, unlike pROBE+, will not automatically halt all tasks 
if a single task crashes (usually).  The other user tasks will continue to run
normally, if possible (although whether this is a safe practice may be 
questionable).

    RTSource is a good, user-friendly source debugger.  It was easier to use
than XRAY+; however, reliability seemed to be inferior, as it would more easily 
get lost in the code.  RTSource also uses a proprietary debugging format, so
the Oasys compiler tools have to be used.  Ready Systems currently supports
version 1.8.3 of the compiler and rev 5.07 of the assembler/linker; however,
we use the Oasys tools in house here, and we're currently up to revs 1.8.5 and
5.11, so Ready Systems is several months behind in their support of the latest
Oasys revisions.

    Also available from Ready Systems is MPV, a multiprocessing version of the
VRTX kernel, which is similar in approach to pSOS+/M.  Also available is the
IFX manager, along with NFS and RPC support.  In addition, unique to Ready
Systems is the availability of VRTX Designer, which is a CASE tool for real-time
system design.  It uses known VRTX timing data to project the performance of
user designs and spot potential bottlenecks.  These products were not evaluated.

    VRTX Velocity provides automated scripts to build eproms, applications, or
even makefiles.  The scripts are interactive, and the user chooses from a list
what components he/she wants to include in the target.  The appropriate
makefile is then created that builds both the VRTX system software and the
user application.

    Installing VRTX Velocity is a mighty task, especially for the beginner.  It
required a kernel mod on the host workstation, and I had to spend some time
with the Field Application Engineer to get parts of Velocity working.  In
addition, RTShell doesn't like workstations that don't have a local disk
attached.  But aside form these problems, VRTX Velocity is a sophistaced,
highly integrated development environment.


VxWorks:
--------

    The current version of VxWorks is 5.0.1a, from Wind River Systems.  It
includes the WIND kernel, a target shell, a TCP/IP networking manager, an
extensive C runtime library, and the GNU compiler toolkit.  The GNU tools
include the C compiler, assembler, linker, and VxGDB, which is the GNU
debugger modified by WRS to work in the realtime environement.

    VxWorks, like VRTX, normally boots by loading its system code off the
network.  Whereas VRTX uses TFTP to do this, VxWorks uses rsh on a host 
workstation.

    As an interesting note, prior to version 4.0, VxWorks utilized the VRTX
kernel from Ready Systems.  Eventually however, WRS packaged their own WIND 
kernel in VxWorks.
    
    The WIND kernel provides comparable functionality to the other kernels.
However, it lacks true roundrobin scheduling, and event flags are absent.  In
addition, the WIND kernel does not offer any real-time clock functions, only a
tick counter.  On the other hand, its semaphore support is excellent - binary, 
counting, and mutual exclusion semaphores (with priority inversion) are
included.  Both pSOS+ and VRTX offer only counting semaphores.  Likewise,
pipes are absent in both pSOS+ and VRTX32, but they're present in VxWorks. 
VxWorks also features Unix-style signals, which are truly asynchronous - i.e.,
a task will stop whatever it's doing at the moment, and service the signal.

    WIND insists on routing interrupt routines through its kernel; however,
this can be bypassed.  Handling interrupts yourself should not be a problem
with any of these kernels; however, VxWorks tries to convince the user to let
it handle interrupts, and thus provides several interrupt-related commands in
the kernel.

    Unlike pSOS+ and VRTX, there is no separate mulitprocessing version of the
WIND kernel.  However, WRS does provide a loosely coupled backplane protocol
where boards on the same bus can communicate via a socket interface.

   VxWorks does not have a true board-level monitor/debugger like pSOS+ or
RTScope.  It does however have a target shell that is a cross between the
board level debugger and the RTShell type of shell.  It provides an incremental
linking loader that accepts Unix object files, and it provides all of the
memory & task oriented examine/modify commands provided in the board debuggers.
In addition, it also offers extensive symbolic disassembly and  debugging
facilities.

    The level of networking support provided by VxWorks is excellent.  NFS and
RPC support is of course included; in addition, rsh, telnet, and ftp are
available to and from the target.

    Unique to VxWorks is the availability of a source license.  You may have
already heard about the person who posted in this group on how he ported
VxWorks over to SunOS.

    The VxGDB debugger has its origins in the Unix GNU GDB debugger.  People
have mixed opinions about GNU tools, but I found VxGDB to be a serviceable, if
somewhat unremarkable debugger.  It's not a windowed, SunView application like
XRAY+ and RTSource are, so it lacks their flash and glamour.  In addition, it's 
more of a single threaded debugger, so it's more difficult to use when
debugging several tasks in one system.

    One area in which VxWorks suffers seriously is documentation.  The basic
documentation is a single programmer's guide, which is very skimpy and does not 
offer enough solid information to build and use a VxWorks based system.  The
necessary functions are listed, but without parameters or error codes.  WRS
does also provide Unix-style online man pages; however, these are no better
than their Unix counterparts, and provide only a minimum of information.  After 
dealing with the excellent documentation that came with pSOS+ and VRTX
Velocity, the VxWorks documentation was a real disappointment.



Conclusions:
============

    Draw whatever conclusions you will - the fact is that all three systems are
competitive; otherwise, the vendors would not be where they are today (in
business, that is).  However, each of them has its strenghts and weakensses.

 o  pSOS+ is a solid, reliable system, and with the CILxxx interface its
    performance is tops among the three kernels.  However, with the currently
    distributed KJMxxx interface, its components utilize the software trap
    interface and performance suffers as a result.  Also, the level of manual
    configuration work required of the user is quite high.  In addition, it
    should be noted that Software Components has been slow in bringing out
    products in support of new technology - e.g., they've only now announced
    NFS & RPC support.
    
 o  VRTX Velocity is a sophisticated development environment, and provides
    an unmatched level of flexibility, automation and ease of use.  However,
    its complexity can be daunting at times, especially to the beginner.  Also,
    VRTX32 is a decent kernel, but it falls short of pSOS+ and WIND in terms of
    both features and performance.  Finally, sources both on and off the net
    confirm the somewhat buggy nature of VRTX Velocity components.

 o  VxWorks enjoys a good reputation among its customers.  Its networking
    support is clearly superior, and it has a flexible kernel, with strong
    interprocess communication facilities.  It also provides the most
    similarity to Unix of all three systems, without sacrificing performance.
    On the down side, however, its remote source level debugging facility is
    not as sophisticated as the competition, and its documentation leaves much
    to be desired.  In addition, VxWorks is conspicuously void in the area ofa
    multiprocessor kernel, which the others provide.  Finally, I have to wonder 
    about the level of support that would be available for the GNU tools.
 
 
Again, remember that I have no affiliation with any of the above companies, and
the opinions I express are mine alone and do not necessarily reflect those
of my organization.


###############################################################################

If you have any factual questions about these products, don't ask me; go 
straight for the horse's mouth.

    Software Components Group
    1731 Technology Drive
    San Jose, CA 95110
    (408) 437-0700
    
    Ready Systems
    470 Potrero Avenue
    P.O. Box 60217
    Sunnyvale, CA 94086
    (408)736-2600
    
    Wind River Systems
    [I'm not sure of their California address, as I think they've moved; so
     here's their Boston sales office - S.H.]
    Paul Kusiak, Sales Rep
    77 North Washington Street
    Boston, MA 02114
    (617) 367-6567
    

Informed discussions on the relative merits of these systems will be welcome.
Biased flames, however, will be redirected to /dev/null.

================================================================================
================================================================================
Keith Smith
keith@das.harvard.edu

Another OS that you should consider is VENIX.  It is a real-time
UNIX variant sold by VenuturCom, Inc.  They start with the original
AT&T (oops, I mean USL) source code, and add additional, system calls
to support a variety of real-time features including:

	-Preemptive priority scheduling
	-Asynchronous and direct I/O
	-High resolution alarms (<= 1ms I think)
	-Device control from user level.

Their system runs on Intel 286, 386, and 486 processors.  They have
both a workstation version, which is meant to run on PC's, and an
embedded product, which is much smaller (and cheaper) and can be
run in a operatorless environment.  

Because their system is built from USL sources, they have full 
compatability with the standard UNIX API's.  They also have 
support for X-windows and NFS.

You can reach them by phone at (617) 661-1230 or via mail at:

	VenturCom, Inc.
	215 First Street
	Cambridge, MA 02142

================================================================================
================================================================================

-- 
Enrico Badella				Internet: eb@felix.sublink.org
Soft*Star s.r.l.			          eb@icnucevx.cnuce.cnr.it
Via Camburzano 9			Phone:    +39-11-746092
10143 Torino, ITALY			Fax:      +39-11-746487
Newsgroups: comp.realtime