From: rob@array.UUCP (Rob Marchand)
Newsgroups: comp.realtime
Subject: Re: Where to stick the watchdog ?
Date: 11 Sep 92 15:00:55 GMT
Organization: Array Systems Computing, Inc., Toronto, Ontario, CANADA

In article <Bu9two.2DA@hatch.socal.com> mike@hatch.socal.com (Mike Bernadett) writes:
>In article <1992Sep8.111139.1@wombat.newcastle.edu.au> eepjm@wombat.newcastle.edu.au (Peter Moylan) writes:
>>In article <1992Sep03.034229.13095@crs-sys.uucp>, crs@crs-sys.uucp (Chris Gregors) writes:
>>> Recently I had an argument (discussion) with some co-workers regarding where
>>> is an appropriate place to kick a watchdog timer inside an application program.
>>> 
>>> Their side was 'stick it in the lowest level routine in a non-kernal based
>>> application, or stick it in the null task in a real-time kernal.'
>>This assumes that the main job of the watchdog is to handle rare
>>conditions which expose software bugs.  I know that most people think
>>of watchdogs as things to look out for hardware bugs, but in practice
>>the hardware is likely to be far more reliable than the software.
>>In any case, hardware problems are just as likely to be caused by
>>the watchdog circuitry itself (Murphy's law again).
>
>At Hughes, we're putting together a remote sensing system
>using pSOS+ on an i960MC.  
>
>Yes, we are looking for latent bugs in both hardware and software.

	We were in a somewhat different position, with some known problems
	in the implementation of operating system extensions on top of
	a real-time kernel in a custom military signal processor.  For
	some time, the only way to keep things working was with a user-level
	watchdog task.

	[some stuff deleted]

>We are taking the above approach and adding a little more complexity
>at the task level.  The watchdog kicker is periodic at a high
>priority and keeps an array of flags (timestamps) of who's kicked
>it lately.  The added complexity is that the kicker
>actively solicits (via message queue) those kicks from every task in
>the system periodically, and has a worst case response time defined
>for every task.  If the kick from a task doesn't come back within
>its assigned response time, that task is warm-started.

	This is exactly what we did to solve our problem.  A watchdog task
	was available for use by whoever needed it.  On startup, a task
	could `subscribe' to watchdog management, and could `unsubscribe'
	at some point in the future (i.e., the watchdog usage was optional)
	It would provide a watchdog interval, and a routine to be called in
	the event of a watchdog timeout.  The subscribing task also
	supplies a mailbox, or some method of informing the watchdog
	task that it is still alive.  It is then responsible for
	notifying the watchdog via this channel periodically.  The
	watchdog wakes up periodically, and, if any of the timers for
	the subscribing tasks have expired, a subscriber-task-specified
	routine is executed.  In our case, this routine would restart a
	blocked I/O that was subject to a race condition failure in the
	extensions to the RT kernel.

>So the kicker evolves into a 'task management' executive task.
>This task-by-task monitoring lets the system continue in a degraded
>mode if a non-critical task dies.  When the task that died is
>warm-started (immediately), it will dump its previous

	There are a lot of ways to do this, and the requirements are going
	to specify what you need.  Probably one of the primary drivers of
	this will be the safety issues surrounding the real-time platform.
	In our case, we could afford to lose a block of data occasionally.
	There also were no safety issues, and most problems we encountered
	could be dealt with by a restart or reset of the task.

	Cheers!
	Rob Marchand
-- 
Rob Marchand                   UUCP  : uunet!attcan!lsuc!array!rob
Array Systems Computing        ARPA  : rob%array.UUCP@uunet.UU.NET
401 Magnetic Drive, Unit 24    Phone : +1(416)736-0900   Fax: (416)736-4715
Downsview, Ont CANADA M3J 3H9  Telex : 063666 (CNCP EOS TOR) .TO 21:ARY001