From: mike@hatch.socal.com (Mike Bernadett)
Subject: Re: Where to stick the watchdog ?
Organization: Hatch Usenet and E-mail.  Playa del Rey, CA
Date: Tue, 8 Sep 1992 17:57:59 GMT

In article <1992Sep8.111139.1@wombat.newcastle.edu.au> eepjm@wombat.newcastle.edu.au (Peter Moylan) writes:
>In article <1992Sep03.034229.13095@crs-sys.uucp>, crs@crs-sys.uucp (Chris Gregors) writes:
>> Recently I had an argument (discussion) with some co-workers regarding where
>> is an appropriate place to kick a watchdog timer inside an application program.
>> 
>> Their side was 'stick it in the lowest level routine in a non-kernal based
>> application, or stick it in the null task in a real-time kernal.'

[theory deleted]
...
>This assumes that the main job of the watchdog is to handle rare
>conditions which expose software bugs.  I know that most people think
>of watchdogs as things to look out for hardware bugs, but in practice
>the hardware is likely to be far more reliable than the software.
>In any case, hardware problems are just as likely to be caused by
>the watchdog circuitry itself (Murphy's law again).

At Hughes, we're putting together a remote sensing system
using pSOS+ on an i960MC.  

Yes, we are looking for latent bugs in both hardware and software.
We want to know (and management 3 levels up want to know),
"Howcum the system died?"  Therefore we need an automatic warm-start
mechanism that can bring the processor back up and at least dump
a 'core image' -- something that will tell us what software was doing
what to whom, and roughly what state the hardware was in.
 
>
>My own experiences have left me wondering whether a watchdog is ever
>worth using.  As one might have predicted from Murphy's law, the usual
>failure mode seems to be that one or more tasks misbehave, but the
>watchdog-kicker never crashes.
>
>In particular, the null task tends to be the most stable task in the
>entire system.  Everything else can fall in a heap, and the null task
>merrily keeps running.
>
>One solution I like, but which is rather complicated, is as follows.
>Choose a low-level, high-priority, periodic task as the one which
>kicks the watchdog, but also adopt the rule that every other task in
>the system - or, at least, every periodic task - must periodically
>kick the watchdog-kicker.  The watchdog-kicker can keep an array of
>flags showing who has kicked it lately, and it should refrain from
>kicking the watchdog if any of these flags is not set.
>

We are taking the above approach and adding a little more complexity
at the task level.  The watchdog kicker is periodic at a high
priority and keeps an array of flags (timestamps) of who's kicked
it lately.  The added complexity is that the kicker
actively solicits (via message queue) those kicks from every task in
the system periodically, and has a worst case response time defined
for every task.  If the kick from a task doesn't come back within
its assigned response time, that task is warm-started.

So the kicker evolves into a 'task management' executive task.
This task-by-task monitoring lets the system continue in a degraded
mode if a non-critical task dies.  When the task that died is
warm-started (immediately), it will dump its previous
data area to non-volatile media and stand by idly.  By default we'll
assume this task is now useless, but an override command could put
it back in service, whereby it would re-initialize its data area,
listen to the control message stream for a 'synchronizing' type of 
message, and sync up with the rest of the system.

And of course if the kicker dies, the WDT gives the whole system a
warm-start.  On system warm-start, the OS startup code does a
'core dump' of the previous OS and kicker (executive task) data
areas to to non-volatile media for post-mortem.

>
>-- 
>Peter Moylan                      eepjm@wombat.newcastle.edu.au


-- 
Mike Bernadett                                        mike@hatch.socal.com
8635 Falmouth Ave. #105, Playa del Rey, CA 90293      (310) 305-8758