# Presentation: 026987

## Using Ranger as a Better Swatch

- Chuck Boeheim
- SLAC Computing Services

## What is Ranger?

- Ranger is a monitoring system used at SLAC for > 10 years
- Written entirely in Perl
- Implements language constructs to make it easy to
- Collect monitoring observations
- Write rules to test for conditions
- Take actions triggered by rules

## What is Swatch?

- Swatch is the Simple Watcher originally written by Todd Atkins of Stanford
- It had a simple pattern-action syntax and limited actions
- /(larry|moe|curly)/&&/panic/  mail=action,exec="/etc/page” 05:00   0:16
- /(larry|moe|curly)/&&/reboot/ mail=action,exec="/etc/page" 05:00   0:16

## Limitations

- Swatch was very useful but
- It had a rigid syntax
- There was no nesting or complex logic
- It had few actions
- It had no state mechanism
- It only considered one line at a time, without context

## A Ranger Collector

- Collectors are objects that
- Monitor some aspect of the system
- Create observation records
- Save state between observations and compute deltas, etc.
- apply watch(‘/var/syslog.pipe’, ‘slacmsg -r rawcons|’) =>
- ruleset
- {
-   ...
- };

## Rulesets in Ranger

- A ruleset contains perl statements, including rule/action pairs
- A rule evaluates a perl expression to true or false
- The first rule to evaluate true executes its action and terminates the ruleset

## A Watch Ruleset

- apply watch(‘/var/syslog.pipe’) =>
- ruleset
- {
-    	rule { /Fatal Reset/           } action { mail STD, 'admin' };
-    	rule { /Psycho/                } action { mail STD, 'admin' };
- 	rule { /RED State Exception/   } action { dup 1*HR; mail STD, ‘admin' };
-    	rule { /(SIR|Watchdog) Reset/  } action { dup 1*HR; mail STD, 'admin' };
-    # Handle alerts from T3s
-    rule { / t3\-\d+ / }
-    action {
-       rule { / .*(failed|error).*/i } action { mail STD, 'sys-raid' };
-       rule { TRUE } action { mcons };
-    };
- };

## State and Context

- Each observation can save and test an arbitrary state string
- A collector can have multiple rulesets
- A ruleset can have a context for evalation
- context { state eq ‘’ } => ruleset {
-    rule { /Unable to handle kernel paging request/ or
-           /Unable to handle kernel NULL pointer)/ }
-    action { state('wedged') };
- }
- context { state eq ‘wedged’ } => ruleset {
-    rule { /.* Linux version .*/ } action { mcons; state('') };
- };

## Multi-line Messages

- Message DRIVERPT <<‘EOF’;
- The following drive error was reported by $host:
- $state
- $msg
- EOF
- apply watch(‘/var/syslog.pipe’) =>
- context { state =~ /Error occurred on drive / } ruleset {
-    rule { TRUE } action { mail DRIVERPT, ‘admin’; state(‘’) };
- }
- context { state eq ‘’ } ruleset {
-    rule { /Error occurred on drive (\S+)/ } action { state(“$_”) };
- };

## Duplicates and Floods

- The dup action detects repeat events
- If *same* host triggers the *same* rule
- Unless a threshold level is reached
- At end of interval a summary is generated
- rule { /RED State Exception/   }
-    action { dup 1*HR; mail STD, 'admin'; mcons };
- rule { /^[0-9:]* (\S+)/ }
-    action { dup 5*MIN,500; mcons FLOODING };

## Conclusions

- Watch collector provides more powerful mechanism for reacting to log entries
- State and context allow more intelligent decisions to be made
- Duplicate message detection helps deal with information floods
- Available at
- 	_[ftp://ftp.slac.stanford.edu/software/sysadmin/ranger.5.0beta1.tgz](ftp://ftp.slac.stanford.edu/software/sysadmin/ranger.5.0beta1.tgz)_