[Overview 1](#overview)

[Architecture 1](#architecture)

[Display 2](#display)

[Plugin framework 3](#plugin-framework)

[Monitoring plugin context instantiation
3](#monitoring-plugin-context-instantiation)

[Plugins 4](#plugins)

[Execution control 4](#execution-control)

[Deliverables 4](#deliverables)

[Dcron 4](#dcron)

[samgrid_mon 5](#samgrid_mon)

[Installation steps 5](#installation-steps)

[NGOP adaptation, service state reports
6](#ngop-adaptation-service-state-reports)

# Overview

Let's define computing production as activity that involves repletion of
similar or same tasks over extended periods of time. Such tasks may have
wide range of responsibilities: from asserting system state to supplying
input that triggers operations flow. (i.e. log file cleanups, disk space
checks, server status checks, CPU utilization monitors, plotting tools,
automatic testing / validation, computing request submission).
Irrespective of the task size , a computing time needs to be dispensed
in order to organize all activities into coherent workflow.

More often than not production support solutions resort to using low
level UNIX tools (cron jobs) as a way to express "periodic" nature of
the process. While seemingly convenient on a small scale, need to
support large sets of different production activities over many
computing nodes leads to hardly maintainable computing infrastructure.
Specifically in case where there is no structure in the task codes and
task execution control.

Thus, our goal is not only to provide monitoring and alarming
infrastructure but also to organize production support code and
computing activities into a well isolated, documented and supported
subject. Primarily we are concerned with grouping, scheduling,
referencing, and execution control of the production maintenance tasks
in as seamless and as organized fashion as possible.

# Architecture

One of the major goals of the program is to formalize presentation of
existing deployment components as well as provide operators fine grain
per-SAMgrid component state validation features. The paradigm described
in the paragraph above has been used to deliver framework which is
structured to support modules responsible for periodic and independent
monitoring activities. The general idea of the framework is to provide
driver and set of plugins as a standalone package which encapsulates
operational traits of the SAMGrid workflow. Such package addresses tasks
that require ambiguous handling or human involvement. Also, through
preconfigured set of threshold values, each component of the package can
streamline validation of the newly released software or configuration.
The overview of the software and machine deployment architecture is
depicted below.

Health and alarming deployment consists of display, monitoring plugin
framework and execution control modules. All modules are related through
a well defined interface that will be described at the end of this
document. Each module is deployed as part of the UPD package that
contains all tools necessary for its installation and self
configuration.

## Display

Display module interfaces operators and shifters to changes in state of
the monitored resources. These changes are propagated in two ways.

-   NGOP display\
    NGOP display offers drill down overview of all services and hardware
    resources involved in D0 production through SAMGrid. The display
    tree is built to assist system troubleshooting by providing
    categorized hierarchy of monitored elements each representing state
    of the particular component of the SAMGrid production deployment.
    The particular choice of the current NGOP setup has been influenced
    by default NGOP alarm propagation rules that can aggregate display
    tree element status information in a way to allow navigation of the
    troubleshooting effort from top of the tree to its bottom.
    <http://samgrid.fnal.gov:8080/cgi-bin/ngop-mon.cgi>

    -   NGOP display deployment\
        Apache HTTP service is required. Configured and Running NGOP
        server is required. User interacts with display cgi script that
        proxies requests to ngop web_cgi command as follows:

> #!/bin/sh
>
> . /samgrid/products/ups/etc/setups.sh
>
> setup ngop -q server 2\>/tmp/log.ng
>
> exec ngop web_fcgi -c /data/products/apache/samgrid/cgi-bin/cfg.xml
> \"\$@\"
>
> Where configuration file is :
>
> \<?xml version=\'1.0\'?\>
>
> \<!DOCTYPE webmonitor_cfg SYSTEM \"monitor.dtd\"\>
>
> \<webmonitor_cfg\>
>
> \<WebGui Type=\"-fcgi\"/\>
>
> \<LS Port=\"3111\" Host=\"d0sammon.fnal.gov\"/\>
>
> \</webmonitor_cfg\>
>
> At the time of writing, the ngop web_cgi command must be modified to
> use --fill argument for the "convert" program to work. Also, imgpath
> variable must be repointed to the existing directory under HTTP
> server.

-   <Sam-oncall@fnal.gov>\
    Beyond state change message sent to the NGOP server, state
    validation modules may report resource status via email directly to
    the list. Each notification is sent no often than once in 8 hours
    for any monitored instance only in case monitored resource is
    "down".

## **Plugin framework**

Each SAMGrid service deployment is been periodically monitored for one
or more of its state properties. This monitoring activity is carried out
by set of plugins that work within established framework. Each plugin is
split in two parts: context instantiation and state validation. In
addition to organizing the code, the framework takes care of common
tasks such as plugin grouping, NGOP adaption, polling interval
management as well as it interfaces code execution control system. The
framework and plugins are packaged into samgrid_mon product.

## **Monitoring plugin context instantiation**

Plugin context is information nessasary to contact service for which
plugin is reporting some property of service state. That information is
put together by a standalone code piece that analyzes user input and
environment of the forwarding node. Such method of isolating context
information into a separately managed object instance ensures integrity
and consistency of the monitoring activity. In particular, after
instantiation, established context can only change in response to inputs
from the plugin itself. Such changes typically reflect the new resource
property state or data which may help detailing problem troubleshooting.
Hence, evolution of the configuration and environment of the forwarding
node will not have an impact on state of the plugin.

For further convenience, plugin instantiation module may accept default
inputs from user (i.e. resource polling interval for ex. ) and program
environment. For example: to instantiate fcp monitoring modules, sam_fcp
and jim_job_managers products need to be set in order to produce all
data nodes and corresponding fcp port number tuples needed by fcp
monitoring module.

## Plugins

Plugin is the code piece that contacts SAMgrid service to evaluate
particular criteria of the monitored service state. For example:
gridftpserver may be running but may not be configured with valid
certificate. Thus, the same service needs to be evaluated by two
different criteria to certify that service is in operational state. By
our design, each criterion is supported by independent plugin instance
with its own unique context. The end result of plugin work is statement
on whether such monitored criteria does or does not match some preset
expectation. That result, along with accompanying information is written
back to the plugin context. By reading that context information, any
external program can gain access to the state of the plugin.

## **Execution control**

Scheduling of periodic monitoring activities is delegated to a
standalone program - dcron. dCron task is to schedule the process which
loads persistently stored context into memory and calls user defined
function supplying with that object. The tuple (the function and the
context) together defines a unit of work that is periodically scheduled
by dcron using internal policies, the context itself and available CPU
resources. Monitoring plugins are written using program API for the
dcron "algorithm" while monitoring plugin context is defined though API
of the algorithm context.

# Deliverables 

## Dcron

dCron is the product provides functions of execution control and context
management for plugin framework.

Usage: dcron \<command\> \<requiredOptions\> \<otherOptions\> \<args\>

Where \<command\> is one of the following, sorted by category:

Category Miscellaneous:

\_generate \_documentation

Category dcron.Commands:

list modules

register module

register shell module

start

stop

unregister module

dcron provides both command line and python APIs to register ,
unregister and list context and algorithm tuples (modules). Along with
initial context information, Python api allows setting of the desired
time interval that suggests how often a given context should be updated
by its algorithm. The same interval is used to avoid collision of
execution requests scheduled by simultaneous registration of many
modules. That is achieved by uniform randomization of time for the first
update which is set values that are no earlier than now and no later
than in time specified by original polling interval.

## samgrid_mon

samgrid_mon product is driver and code for all monitoring pluging
delivered as part of SAMGrid health and alarm solution. Samgrid_mon
product is distribute tgtough FNAL upd and provides user interface
though self documented samGridMon.py command. The command expands into
options that deploy and configure respective service monitoring objects.
Each option supports list of optional parameters that can override
existing environment settings.

Usage: samGridMon \<command\> \<requiredOptions\> \<otherOptions\>
\<args\>

Where \<command\> is one of the following, sorted by category:

Category Miscellaneous:

\_generate \_documentation

Category samGridMonLib.samGridMonCommands:

make snapshot caches

make snapshot durable locations

make snapshot fcp queues

make snapshot gridftp gsi servers

make snapshot gridftp servers

make snapshot station

Category samGridMonLib.samGridMonCommandsForwardingNode:

make snapshot fwdfcp queues

make snapshot fwdgridftp gsi servers

make snapshot fwdgridftp servers

make snapshot jim advertize

make snapshot ngop

make snapshot sandbox

make snapshot xmldb space

# Installation steps

Upd install samgrid_mon --G-c \# installs dcron and samgrid_mon packages

Setup samgrid_mon

By default, Samgrid_mon supports two deployment patterns -- deployment
on the primary and deployment on the secondary forwarding nodes.

Primary forwarding node is assumed responsible for monitoring of local
(deployed at the node itself) and all remote resources participating in
the production workflow. Because all forwarding nodes must be maintained
identical we have no need in duplicate running of the monitoring plugins
for resources that are common to all forwarding nodes. Hence, monitoring
deployment on the secondary forwarding node is limited to resources that
are local of the node itself.

Primary forwarding node: use installPrimaryMon.sh

Secondary forwarding node: use installSecondaryMon.sh

samGridMon.py command also supports manipulation of the individual
plugins through options which can customize plugins individual
parameters i.e. setting polling interval or versions of the monitored
products.

### NGOP adaptation, service state reports

NGOP alarm and email notifications are sent using program that has been
developed as part of the monitoring framework. NGOP reporting agent is a
special type of the plugin which periodically furnish all service status
information to both NGOP and email notification systems. samGridMon.py
make snapshot ngop is the command that deploys that plugin. Two main
configuration options of the command are: \--polling-interval and
--email. The value for polling interval option should establish interval
reasonably close to polling intervals of all services which are been
reported by the collection of plugins deployed at node. Optional Email
parameter sets the destination for notification messages if
corresponding resource transitions to "down" state.