**NGOP Users Guide**

[Chapter 1: Introduction 5](#chapter-1-introduction)

[Chapter 2: Requirements 7](#chapter-2-requirements)

[Chapter 3: NGOP Terminology 8](#chapter-3-ngop-terminology)

[3.1 Host 8](#host)

[3.2 Cluster 8](#cluster)

[3.3 Monitored Element 8](#monitored-element)

[3.4 System 8](#system)

[3.5 System View 8](#system-view)

[3.6 Monitored Object 8](#monitored-object)

[3.7 State 8](#state)

[3.8 Severity Level 9](#severity-level)

[3.9 Event 9](#event)

[3.10 Status 9](#status)

[3.11 Status Rules 9](#status-rules)

[3.12 Action 9](#action)

[Chapter 4: Common Configuration Language
10](#chapter-4-common-configuration-language)

[4.1 Expansion Mechanism: \<For\> tag 10](#expansion-mechanism-for-tag)

[Example: 10](#example)

[Example: 10](#example-1)

[4.2 Expression \<Apply\> tag 11](#expression-apply-tag)

[4.2.1: Examples 11](#examples)

[4.3 System \<System\> tag 12](#system-system-tag)

[Example: 13](#example-3)

[4.4 Monitored Element \<MonitoredElement\> tag
13](#monitored-element-monitoredelement-tag)

[Example: 13](#example-4)

[4.5 Action \<Action\> tag 14](#action-action-tag)

[4.6 Conditional Mechanism: \<If\> tag
14](#conditional-mechanism-if-tag)

[Chapter 5: NGOP Central Server 15](#chapter-5-ngop-central-server)

[5.1 NCS Overview 15](#ncs-overview)

[5.2 NCS Starting/Stopping 15](#ncs-startingstopping)

[5.3 NCS Configuration 16](#ncs-configuration)

[Chapter 6: NGOP Monitor 17](#chapter-6-ngop-monitor)

[6.1 Overview 17](#overview)

[6.2 Starting the Monitor. 17](#starting-the-monitor.)

[6.3 Monitor Startup Configuration 17](#monitor-startup-configuration)

[6.4 NGOP Monitor Usage 18](#ngop-monitor-usage)

[6.4.1 CONFIGURE 18](#configure)

[6.4.2 DISPLAY 20](#display)

[6.5 Monitor Configuration 22](#monitor-configuration)

[6.5.1 Common Configuration Files 22](#common-configuration-files)

[6.5.2 File service_class.xml 22](#file-service_class.xml)

[6.5.3 File hosts_in_clusters.xml 23](#file-hosts_in_clusters.xml)

[Example: 24](#example-6)

[6.5.4 File known_status.xml 24](#file-known_status.xml)

[Examples: 25](#examples-1)

[6.6 NGOP Hierarchy Definition 25](#ngop-hierarchy-definition)

[6.6.1 System View 26](#system-view-1)

[6.6.2 System 26](#system-1)

[6.6.3 Monitored Element 27](#monitored-element-1)

[6.7 Status Rule Sets 27](#status-rule-sets)

[6.7.1 Dependent List 28](#dependent-list)

[6.7.2 Rule 29](#rule)

[6.7.3 Generic Rule Example 30](#generic-rule-example)

[6.7.4 Dependent Rule 31](#dependent-rule)

[Chapter 7: Configuration File Manager
33](#chapter-7-configuration-file-manager)

[7.1: Librarian 33](#librarian)

[7.2: Indexer 33](#indexer)

[7.3 Broker 33](#broker)

[7.4 CFMS Configuration File 33](#cfms-configuration-file)

[7.5 CFMS Starting/Stopping 34](#cfms-startingstopping)

[7.6 Administrative Client 34](#administrative-client)

[7.7 Admin Starting/Stopping 35](#admin-startingstopping)

[![](media/image1.png){width="3.2291666666666665in"
height="3.379861111111111in"} 35](#__RefHeading___Toc532195471)

[Chapter 8: Archive Server 36](#chapter-8-archive-server)

[8.1 Archive Server Overview 36](#archive-server-overview)

[8.2 Starting and Stopping 36](#starting-and-stopping)

[8.3: Archiver Configuration 36](#archiver-configuration)

[Chapter 9: Monitoring Agents 37](#chapter-9-monitoring-agents)

[9.1: Overview 37](#overview-1)

[9.2 Plugin Agent 37](#plugin-agent)

[9.2.1 Starting/Stopping Plugins Agent
41](#startingstopping-plugins-agent)

[9.3 Swatch Agents 41](#swatch-agents)

[9.3.1 Starting/Stopping Swatch Agent
43](#startingstopping-swatch-agent)

[9.4 Monitoring Agent API 43](#monitoring-agent-api)

[9.4.1: API Description 43](#api-description)

[9.4.2: MA API Example 46](#ma-api-example)

[9.4.3 Starting/Stopping Your Agent 47](#startingstopping-your-agent)

[Chapter 10: Action Client 49](#chapter-10-action-client)

[10.1 Action Client Configuration File
49](#action-client-configuration-file)

[10.2 Starting/Stopping Action Client
49](#startingstopping-action-client)

[10.3 File authorized.xml 49](#file-authorized.xml)

[10.4 File action.xml 50](#file-action.xml)

[Chapter 11: Controlling the NGOP Daemons
52](#chapter-11-controlling-the-ngop-daemons)

[11.1: The /var/ngop Directory. 52](#the-varngop-directory.)

[11.2 Starting the Agents 52](#starting-the-agents)

[11.3 Monitoring the Agents 52](#monitoring-the-agents)

[11.4 Stopping the Agents 53](#stopping-the-agents)

[11.5 Disabling/Enabling Agents 53](#disablingenabling-agents)

[11.6 Controlling Agents on Remote Hosts
54](#controlling-agents-on-remote-hosts)

[11.7 Starting/Stopping Individual Agents
54](#startingstopping-individual-agents)

[Appendix A 55](#appendix-a)

[\<For\> DTD 55](#for-dtd)

[\<Apply\> DTD 55](#apply-dtd)

[\<Action\> DTD 56](#action-dtd)

[\<If\> DTD 56](#if-dtd)

[NCS Configuration File DTD 56](#ncs-configuration-file-dtd)

[Monitor Configuration File DTD 57](#monitor-configuration-file-dtd)

[PlugIns Agent DTD 57](#plugins-agent-dtd)

[Swatch Agent DTD 58](#swatch-agent-dtd)

[\<ServiceClass\> DTD 59](#serviceclass-dtd)

[\<HostsInClusters\> DTD 59](#hostsinclusters-dtd)

[\<KnownStatus\> DTD 60](#knownstatus-dtd)

[Monitored Hierarchy DTD 60](#monitored-hierarchy-dtd)

[\<StatusRulesSet\> DTD 61](#statusrulesset-dtd)

[CFMS Configuration File DTD 61](#cfms-configuration-file-dtd)

[Archiver Configuration File DTD 62](#archiver-configuration-file-dtd)

[Action Client Configuration File DTD
62](#action-client-configuration-file-dtd)

[\<Autorization_File\> DTD 63](#autorization_file-dtd)

[\<Action_File\> DTD 63](#action_file-dtd)

###########  

########### 

# Chapter 1: Introduction

NGOP is a distributed monitoring system that provides active monitoring
of software and hardware, customizable service-level reporting, early
error detection, and problem prevention. NGOP provides persistent
storage of collected data and is capable of executing corrective actions
and sending notifications. NGOP is a framework for developing monitoring
tools.

The target audience for this document is wide ranging. It is intended
that users will go directly to chapters that interest them, rather than
reading the document from cover to cover.

NGOP uses a centralized collection scheme. The *NGOP central server*
(NCS) collects and stores information from various *monitoring agents*
running on remote machines. The NCS is passive, simply listening for
messages from the monitoring agents which communicate with the NCS using
a well-defined protocol. NGOP provides a "plug-in" monitoring agent,
which is a template that is used to easily create monitoring agents for
many common tasks. A full API is also provided allowing users to create
any type of monitoring agent.

Typically, a monitoring agent will monitor a piece of hardware or
software and generate alarms to the NCS when appropriate. For example, a
monitoring agent could be written to look for the presence of an
important daemon and report when the daemon has died to the NCS.

The *Archive Server* is a component of NGOP that provides persistent
storage. The NCS forwards all messages received from monitoring agents
to the Archive Server. The Archive Server stores the messages in an
Oracle database, and also provides a web based report generator as well
as maintaining the database (rolling old records out to minimize the
overhead for example).

The *Configuration Server* is the component that handles all of the
configuration files in NGOP. The configuration files are written using
XML.

Information from the NGOP system is made available through the NGOP
Monitor. The Monitor is a GUI that displays the data collected by the
NCS in a customizable way. Although the NCS is collecting information
from potentially many systems, the monitor can watch a subset of the
clusters being monitored. In addition, the monitor is configured in such
a way that one persons definition of "broke" may be different than
another. For example, an operations staff interested in the overall
service of a system has a different view than a systems administrator
who is interested in every detail. To the operations staff, having 80%
of the cluster available is sufficient to provide the service, therefore
they want their monitor to tell them the system is fine. The systems
administrator wants to know when anything has happened in the cluster.

Below is a pictorial view of the entire NGOP system:

![](media/image2.png){width="6.352083333333334in"
height="4.758333333333334in"}

# Chapter 2: Requirements

This chapter discusses the various system requirements of various
components of the NGOP system.

+---------------+----------+-------------+--------------------+------+
| Subsystem     | Status   | Available   | Requirements       | Nu   |
| Name          |          | on          |                    | mber |
|               |          | Platforms   |                    | of   |
|               |          |             |                    | n    |
|               |          |             |                    | odes |
+---------------+----------+-------------+--------------------+------+
| NCS           | Required | Linux       | \- Python 1.5.2    | 1    |
|               |          |             | and higher         |      |
|               |          |             |                    |      |
|               |          |             | \- FCSlib v2_0 and |      |
|               |          |             | higher             |      |
+---------------+----------+-------------+--------------------+------+
| CFMS Broker   | Optional | Linux       | \- Python 1.5.2    | 1    |
| /Indexer      |          |             | built with tcl/tk  |      |
|               |          |             | support module.    |      |
| Admin GUI     |          |             | CVS                |      |
|               |          |             |                    |      |
|               |          |             | \- Tcl v8_0_2,     |      |
|               |          |             | tkv8_0_2           |      |
|               |          |             |                    |      |
|               |          |             | \- Blt v2_3, xed   |      |
|               |          |             | b1_0(opt)          |      |
+---------------+----------+-------------+--------------------+------+
| Archiver      | Optional | Linux,IRIX, | \- Python 1.5.2    | 1    |
|               |          | SunOS       | and higher         |      |
|               |          |             |                    |      |
|               |          |             | \- dcoracle python |      |
|               |          |             | package            |      |
|               |          |             |                    |      |
|               |          |             | (available from    |      |
|               |          |             | ups as             |      |
|               |          |             | python_dcoracle)   |      |
|               |          |             |                    |      |
|               |          |             | \- Oracle client   |      |
|               |          |             | license            |      |
|               |          |             |                    |      |
|               |          |             | \- Shared file     |      |
|               |          |             | space for message  |      |
|               |          |             | requests.          |      |
+---------------+----------+-------------+--------------------+------+
| Action Client | Optional | Linux       | \- Python v.1.5.2  | N    |
|               |          |             | and higher         |      |
+---------------+----------+-------------+--------------------+------+
| Monitorint    | Required | Linux,      | \- Python v1.5.2   | N    |
| Agents        |          | OSF1,       | and higher         |      |
|               |          | SunOS, IRIX |                    |      |
| (             |          |             |                    |      |
| PluginsSwatch |          |             |                    |      |
|               |          |             |                    |      |
| MA Api Ping)  |          |             |                    |      |
+---------------+----------+-------------+--------------------+------+
| Monitoring    | Required | Linux,      | \- Python 1.5.2    | N    |
| Client        |          | SunOS, IRIX | and higher built   |      |
|               |          |             | with tcl/tk        |      |
|               |          |             | support module.    |      |
|               |          |             |                    |      |
|               |          |             | \- Tcl v8_0_2,     |      |
|               |          |             |                    |      |
|               |          |             | \- tk v8_0_2       |      |
|               |          |             |                    |      |
|               |          |             | \- blt v2_3        |      |
|               |          |             |                    |      |
|               |          |             | \- 64MB Ram for    |      |
|               |          |             | 6000 elements      |      |
+---------------+----------+-------------+--------------------+------+

#  

# Chapter 3: NGOP Terminology

This chapter describes the terminology used when discussing the NGOP
system.

## 3.1 Host 

A Host is a computer or an entity with an assigned IP address,
identified by its name.

## 3.2 Cluster 

A Cluster is a collection of Hosts that have a common usage or purpose.
Clusters may overlap. A Cluster may consist of only one Host. A Cluster
is uniquely identified by its name.

## 3.3 Monitored Element 

A Monitored Element (**ME**) is an atomic entity that is monitored by
NGOP. It has a well-defined behavior, which is characterized by its
state and is associated with some quantitative measurements. This entity
is derived from several parts; each of them contributing to the overall
State of the monitored element. A ME is located on a particular Host and
belongs to a particular System. Each ME has a unique id that consists of
the ME name, the Host name, the System name and the Cluster name.
(Examples of Monitored Element: files system, tape drive, system daemon,
and memory utilization.)

## 3.4 System 

A System is a set of software components (ME) that are logically
integrated into one unit monitored by NGOP. A System is defined on a
Cluster and may be distributed across multiple Hosts. It is
characterized by its State and Status. A System has a unique id that
consists of the System name and the Cluster name. (Examples: LSF Batch ,
OS "Health" System that could contain system daemons, critical file
systems, etc.)

## 3.5 System View

A System View is a logical collection of Systems, Monitored Elements and
System Views. A System View is created by a user/administrator in order
to create hierarchical structure in the NGOP Monitor. It is
characterized by its Status.

## 3.6 Monitored Object 

A Monitored Object is a System or a ME monitored by NGOP.

## 3.7 State

A State is a characteristic of a Monitored Object defined by either a
Monitoring Agent, or the NCS. A Monitored Object could be in four
different states:

-   1 (Up) - the Monitored Object is operational

-   0 (Down) - the Monitored Object is not operational

-   -1 (Undefined) - NGOP was not able to determine the state of the
    monitored element. This is usually set by the NCS when no
    information has been obtained about this object since the NCS had
    started.

-   -2 (Unknown) - NGOP failed to determine the current state of the
    monitored element but was able to do it earlier. This is set by the
    NCS when the connection with the MA has been lost. This state
    indicates that at some point the NCS was communicating with the MA.

## 3.8 Severity Level

A Severity Level is a characteristic of an event defined by a MA. It
could assume the integer values from 0 - "OK" to 6 -- "Bad". It is used
to describe events when the monitored object is still operational, but a
change in the monitored object's behavior or quantitative
characteristics could indicate a potential problem. The severity level
of the occurred event is redefined by the status rule in the NGOP
Monitor configuration.

## 3.9 Event

Events are generated by MAs and describe a detected condition. An Event
includes the following fields:

-   System name

```{=html}
<!-- -->
```
-   Cluster name

```{=html}
<!-- -->
```
-   Monitored Element name

```{=html}
<!-- -->
```
-   Host name

```{=html}
<!-- -->
```
-   Date/Time

```{=html}
<!-- -->
```
-   Event Name (an aspect of the monitored element that contributed to
    event initiation).

```{=html}
<!-- -->
```
-   Event Value (the current measurements that are associated with that
    aspect of the monitored element).

```{=html}
<!-- -->
```
-   State

```{=html}
<!-- -->
```
-   Severity Level

```{=html}
<!-- -->
```
-   Source -- the id of the Monitoring Agent (MA_name.host)

```{=html}
<!-- -->
```
-   Description (human readable explanation of the occurred event)

## 3.10 Status

A Status is a characteristic of a Monitored Object or System View
defined by the NGOP Monitor based on the Status Rules and events. A
Status of a monitored object/system view may assume the following
values: "Good", "NotInService", "Undefined", "Unknown", "Warning",
"Error", "Bad". Status defines the color of the icons that represents
system views or monitored objects in monitor.

## 3.11 Status Rules 

Status Rules are a set of rules defined by a user/administrator that are
used by the NGOP Monitor to determine the Status of the Monitored
Objects and System Views.

## 3.12 Action 

Actions are associated with monitored objects. An event could trigger
the NGOP application to send the request to the NCS to perform an
action**.** NGOP generates zero or more actions depending on the event,
NGOP configuration, current day/time, and requester's authorization.
Examples of Actions are:

-   Display a message on the Operator console

-   Send an e-mail message

-   Send a message to a pager

-   Run a script

# Chapter 4: Common Configuration Language

This chapter describes the NGOP configuration language that allows the
creation of hierarchies of monitored components, describes rules to
determine the status of components, and defines when and what kind of
actions should be performed. The NGOP configuration language provides a
framework for creating monitoring tools ("PlugIns", "Swatch" Monitoring
Agents).

The NGOP configuration files are stored in a central repository. All
NGOP configuration files are written in XML. XML stands for eXtensible
Markup Language (see <http://www.w3.org/XML> for details). XML makes use
of *tags* (words bracketed by '\<' and '\>') and *attributes* (of the
form name="value"). XML uses the tags only to delimit pieces of data,
and leaves the interpretation of the data completely to the application
that reads it. All configuration files should conform to a corresponding
DTD (Document Type Definition). A DTD is a set of rules for constructing
of valid XML documents.

## 4.1 Expansion Mechanism: \<For\> tag

The NGOP applications (such as NGOP Monitor, CFMS, PlugIns and Swatch
Agent) use an expansion mechanism that allows the replication of a
particular fragment of an XML document. This fragment refers to a
hierarchy and is repeated for every element of this hierarchy. The
hierarchy should be defined in the same XML document, or in some other
XML documents referred to by name. The hierarchy consists of XML tags
where each tag has at least one attribute: *Name*. There is just one
outermost tag of hierarchy. This tag contains multiple tags that could
be the same. This XML fragment should conform to [the DTD
rules](#for-dtd).

### Example:

This is a hierarchy of \<Cluster\> tags that contains \<Host\> tags:

\<Cluster Name="CDFFarm"\>

\<Cluster Name="CDFFarmIO"\>

\<Host Name="cdffarm1"/\>

\</Cluster\>

\<Cluster Name="CDFFarmWorker"\>

\<Host Name="fncdf1"/\>

.

.

.

\<Host Name="fncdf90"/\>

\</Cluster\>

\</Cluster\>

Each fragment of the XML document that needs to be replicated should be
placed within \<For\> \</For\> tags. A \<For\> tag has the following
attributes:

-   Each (required) -- refers to the child element within the hierarchy

-   Var (required) -- name of the variable that will be replaced every
    time when this name is encountered in the XML construction;
    Var="{%PlaceHolder}"

-   In (required) -- refers to the parent element

-   Name (required) -- refers to the attribute Name of the particular
    parent element

-   Filename (optional) -- the name of the file where the hierarchy is
    described

### Example:

\<For Each="Host" Var="{%Host}" In="Cluster" Name="CDFFarm"

Filename="CDFFarmCluster.xml "\>

\<System Name="OSHealth" Cluster= "{%Host}" \>

\<MonitoredElement Name="ypbind" Host="{%Host}" Type="Daemon" /\>

\<MonitoredElement Name="syslogd" Host="{%Host}" Type="Daemon" /\>

\</System\>

\</For\>

The fragment of the XML document will be repeated for every Host tag
within the Cluster tag with attribute Name="CDFFarm". These tags are
listed in the file CDFFarmCluster.xml. The values of the Cluster
attribute of a \<System\> tag and the Host attribute of a
\<MonitoredElement\> tag will be replaced with the corresponding value
of the {%Host} variable. The resulting configuration will look like:

\<System Cluster="cdffarm1" Name="OSHealth"\>

\<MonitoredElement Host="cdffarm1" Name="ypbind" Type="Daemon"/\>

\<MonitoredElement Host="cdffarm1" Name="syslogd" Type="Daemon"/\>

\</System\>

\<System Cluster="fncdf1" Name="OSHealth"\>

\<MonitoredElement Host="fncdf1" Name="ypbind" Type="Daemon"/\>

\<MonitoredElement Host="fncdf1" Name="syslogd" Type="Daemon"/\>

\</System\>

.

.

.

\<System Cluster="fncdf90" Name="OSHealth"\>

\<MonitoredElement Host="fncdf90" Name="ypbind" Type="Daemon"/\>

\<MonitoredElement Host="fncdf90" Name="syslogd" Type="Daemon"/\>

\</System\>

## 4.2 Expression \<Apply\> tag

An \<apply\> tag is used in various places in the NGOP configuration
(Rules, Service Type, and Condition definitions). It defines a
mathematical expression ("logical brackets") (see MathML for details).
This expression is evaluated by the NGOP applications and if it is true
some specific operations are carried out by the applications. For
example, if at some point an expression, defined within a \<Condition\>
tag in a PlugIns agent configuration file becomes true, an agent will
generate an event; if an expression within a \<GenericRule\> tag becomes
true, the NGOP Monitor will apply this rule to define the status of the
monitored object associated with this rule.

An \<apply\> tag can contain other \<apply\> tags. It also could contain
logical operators (\<and\>, \<or\>, \<eq\>, \<neq\>, \<lt\>, \<leq\>,
\<gt\>, \<geq\>, \<in\>, \<notin\>. ) or functions (\<plus\>,
\<times\>,\<minus\>,\<divide\>,\<sum\>,\<min\>,\<max\>). An \<apply\>
element includes a number token element (\<cn\>) and identifier token
element (\<ci\>). One of the operators or functions should be the first
element within \<apply\> tag.

This XML fragment should conform to [the DTD rules](#apply-dtd).

## 4.2.1: Examples

######################################### **Example 1**

Evaluate the following expression:

*2y+ 4x + 1\> 3z.*

\<apply\>

\<gt/\>

\<apply\>

\<plus/\>

\<apply\>

\<times/\>

\<ci\>y\</ci\>

\<cn\>2\</cn\>

\</apply\>

\<!---2y\--!\>

\<apply\>

\<times/\>

\<cn\>4\</cn\>

\<ci\>x\</ci\>

\</apply\>

\<!---4x\--!\>

\<cn\>1\</cn\>

\</apply\>

\<!---2y+4x+1\--!\>

\<apply\>

\<times/\>

\<cn\>3\</cn\>

\<ci\>z\</ci\>

\</apply\>

\<!---3z\--!\>

\</apply\>

\<!\--2y+4x+1\>3z\--!\>

The \<sum\>,\<min\> and \<max\> tags should have the following
construction:

\<sum\>

\<bvar\>i\</bvar\>

\<lowlimit\> \<cn\>N1\</cn\> \</lowlimit\>

\<uplimit\> \<cn\>N2\</cn\> \</uplimit\>

\<ci\>element\[i\]\</ci\>

\</sum\>

This represents the following expression:

(element\[i\])=element\[N1\]+....element\[N2\]

i=N1

#### Example 2 

\<apply\>

\<gt/\>

\<apply\>

\<sum\>

\<bvar\>I\</bvar\>

\<lowlimit\> \<cn\>0\</cn\> \</lowlimit\>

\<uplimit\> \<cn\>10\</cn\> \</uplimit\>

\<ci\>element\[i\]\</ci\>

\</sum\>

\<apply/\>

\<cn\>20\</cn\>

\</apply\>

This defines the following expression:

(element\[i\])\>20

i=0

## 4.3 System \<System\> tag

A system definition is used in various places in the NGOP configuration
(Swatch and PlugIns Agent configuration, NGOP Monitor hierarchy, status
rules set and known status definition). A System is uniquely defined by
two tuple:

*(System_Name,Cluster_Name)*

A \<System\> tag indicates the beginning of the system definition and
requires two attributes:

*Name* -- defines the system name

*Cluster* - defines the cluster name for this system

A \<System\> tag contains multiple *\<*MonitoredElement*\>* tags.

This XML fragment should conform to [the DTD
rules](#monitored-hierarchy-dtd).

### Example:

\<System Name="OSHealth" Cluster="Fnalu"/\>

This defines the system "OsHealth.Fnalu". The \<For\> tag is used to
define multiple systems:

\<Unix Name="UnixFlavor"\>

\<Flavor Name="Irix"/\>

\<Flavor Name="Solaris"/\>

\<Flavor Name="OSF1"/\>

\<Flavor Name="Linux"/\>

\</Unix\>

\<For Each="Flavor" Var="{%F}" In="Unix" Name="UnixFlavor"\>

\<System Name="OSHealth\_{%F}" Cluster="Fnalu"/\>

\</For\>

The code above is equivalent to the following XML fragment:

\<System Name="OSHealth_Irix" Cluster="Fnalu" /\>

\<System Name="OSHealth_Solaris" Cluster="Fnalu" /\>

\<System Name="OSHealth_OSF1" Cluster="Fnalu" /\>

\<System Name="OSHealth_Linix" Cluster="Fnalu" /\>

## 4.4 Monitored Element \<MonitoredElement\> tag

A monitored element definition is used in various places in the NGOP
configuration (Swatch and PlugIns Agent configuration, NGOP Monitor
hierarchy, status rules set and known status definition). An ME is
uniquely defined by four tuple:

(ME_Name,Host_Name,System_Name,Cluster_Name)

A \<MonitoredElement\> tag can be encountered only within a \<System\>
tag. It has the following required attributes**:**

Name *--* defines the monitored element name

Host -- defines the physical location of monitored element. (Instances
of ***"***localhost***"*** in this value will be replaced by the local
host name in MA.)

Type -- defines the type of monitored element (see Event for details)

This XML fragment should conform to [the DTD
rules](#monitored-hierarchy-dtd).

### Example:

\<System Name="OSHealth" Cluster="Fnalu"\>

\<MonitoredElement Name="cpuLoad" Host="fnsfo" Type="sysUsage"/\>

\</System\>

This defines the monitored element with
id="cpuLoad.fnsfo.OSHealth.Fnalu" and type="sysUsage" .

The \<For\> tag is used to define multiple monitored elements:

\<List Name="Scratch"\>

\<Item Name="1"/\>

\<Item Name="2"/\>

\<Item Name="3"/\>

\</List\>

\<System Name="OsHelath_Irix" Cluster="Fnalu"\>

\<For Each="Item" Var="{%I}" In="List" Name="Scratch"\>

\<MonitoredElement Name="/local/stage\_{%I}" Host="fnsfo"
Type="fileSystem"/\>

\</For\>

\</System\>

The above code fragment is equivalent to the following XML code
fragment:

\<System Name="OSHealth_Irix" Cluster="Fnalu" \>

\<MonitoredElement Name="/local/stage_1" Host="fnsfo"
Type="fileSystem"/\>

\<MonitoredElement Name="/local/stage_2" Host="fnsfo"
Type="fileSystem"/\>

\<MonitoredElement Name="/local/stage_3" Host="fnsfo"
Type="fileSystem"/\>

\</System**\>**

## 4.5 Action \<Action\> tag

An \<Action\> tag is used in various places in the NGOP configuration
(Rules, Condition definition). \<Action\> tags require two attributes;
ID and execution Host. Several optional attributes may be provided as
well:

Method - perform a manual or automatic action (default method is
automatic)

Type - execute an action locally or send request to NCS (default type is
local)

Gap - time before attempt to repeat the same action in case of action
failure or reoccurrence of the same event

Repeat - how many times the same action is repeated

Counter - the thershold that allows Agent to generate an action if the
number of ocurrences of the same event exceeded this threshold within
"Gap" period

An \<Action\> contains just one other tag \<Exec\> that describes actual
executable and its arguments in two required attributes:

Name

Argument

Special parameters are included in an *argument*; these parameters
always start with % sign. Every application has a list of parameters
that are used in configuration. This XML fragment should conform to [the
DTD rules](#action-dtd).

######################################### **Example**

\<Action ID="email" Host="ndem" Type="central" Method="automatic"\>

\<Exec Name="send_email" Argument="%Mail,Something_awful_just_happened!"
/\>

\</Action\>

This defines the action with ID="email" that should be started
automatically on the host ndem. The arguments that will be passed to the
script send_mail will contain user e-mail address, and some description.

So far we have discussed the XML constructions that are common to the
all NGOP subsystems, now we will concentrate on XML constructions
specific to each module.

## 4.6 Conditional Mechanism: \<If\> tag 

The If construct is used as a conditional operation in the NGOP. The
only attribute is Cond, which specified the condition. An optional
\<Else\> tag can be used. For the time being the value of Cond attribute
should consist of variable placeholder "'{%Role}'", logical operarator
("==","!=") and role name. This XML fragment should conform to [the DTD
rules](#if-dtd).

**Example 1**

\<ItemList Name="CMS"\>

\<Item Name="CMSPROD"/\>

\<If Cond="'{%Role}'=='cmsadmin'"\>

> \<Item Name="CMSREF"/\>
>
> \</If\>

\</ItemList\>

**Example 2**

\<If Cond="'{%Role}'!='default'"\>

\<For Each="Host" Var="{%Host}" In="Cluster" Name="{%C}Worker"

Filename="hosts_files/hostsInClusters.xml"\>

.

.

\</For\>

\<Else\>

.

.

\</Else\>

\</If\>

# Chapter 5: NGOP Central Server

This chapter discusses the role of the NGOP Central Server(NCS), how to
start and stop it, and it's configuration.

## 5.1 NCS Overview

The NGOP Central Server (NCS) is a process that collects messages from
multiple monitoring agents and provides clients with requested
information. In particular, the NCS performs the following tasks:

-   Accepts requests from a monitoring client (GUI for example) to
    provide monitoring information.

```{=html}
<!-- -->
```
-   The monitoring client can instruct the NCS to perform certain
    actions based on a condition. For example, if a node is being
    monitored by a ping command then the NCS is told to send email to
    the systems administrator if the node is failing the ping request.

```{=html}
<!-- -->
```
-   Allows for the connection of monitoring agents. The monitoring
    agents will send events to the NCS.

```{=html}
<!-- -->
```
-   Forwards all messages sent by monitoring agents to an Archive
    Server.

```{=html}
<!-- -->
```
-   Once a monitoring agent has connected to the NCS, the NCS will note
    when the monitoring agent had died. In affect, the NCS monitors the
    monitoring agents.

## 5.2 NCS Starting/Stopping

The NCS is started with the other daemons running on a host by issuing
the ups start ngop. If UPS is not installed, then the command ngop start
must be issued after the \$PATH environment variable has been set to
point to the NGOP directories. This command starts all of the daemons
that have configuration files defined in /var/ngop**.**

To start only the NCS, the following command must be issued.

ngop start server

or

ngop start "ngop server --c /var/ngop/server/ncs.xml"

Conversely, to stop the NCS issue the following:

ngop stop server

or

ngop stop "ngop server --c /var/ngop/server/ncs.xml"

## 5.3 NCS Configuration

The NCS configuration file is written using XML. The following is a
sample configuration file that is used as a template:

\<NCS_cfg\>

\<NCS tcpPort="19996" udpPort="19997" /\>

\<Admin adminName="ngopadmin" actnHost="ndem"

adminMail="<ngopadmin@fnal.gov>"/\>

> \<Client name="Archiver" port="7001" host="fncduh1"
> local_log="/tmp/log.log"/\>

\</NCS_cfg\>

The NCS tag has two entries, tcpPort and udpPort. These two values must
be assigned an unused port number. The Admin tag has three entries, the
account of the NGOP administrator (adminName**)**, the host that the NCS
is running on (actnHost), and the email address of the NGOP
administrator (adminMail)**.** The Client tag is used to locate the host
that the archive server is running on and the port that it is listening
on. The final tag in the template is the local_log tag, which specifies
where the logging information is to be stored. The default is to store
the logging information in the file log.log in the current working
directory. This configuration file should conform to [the DTD
rules](#ncs-configuration-file-dtd).

# Chapter 6: NGOP Monitor

The NGOP monitor gets information from the NCS and allows clients to
display the information to the end user through a GUI. This chapter
gives an overview of the monitor, how to start and stop it, discusses
the configuration file, and explains how to use the display.

## 6.1 Overview

The NGOP monitor provides graphical display of information collected by
the NCS. The NGOP monitor performs the following functions:

-   Gives users ability to display custom built system views. This
    allows different people to interpret the data collected by NGOP in
    different ways. For example, a systems administrator may be
    interested in every small detail of the system, whereas a computer
    operator may only be interested in being notified about major
    events, which cause the system to be unusable.

```{=html}
<!-- -->
```
-   Uses rules that define the status of a system. For example, the
    system may be considered "ok" if 75% of the nodes are operable.

```{=html}
<!-- -->
```
-   Provides a way to initiate a request to the NCS to perform some
    action.

## 6.2 Starting the Monitor.

The NGOP monitor is started as follows:

\$setup ngop

\$ngop start monitor &

This will start the monitor using the configuration file
/var/ngop/monitor. To use another monitor, you can supply the --c
option:

\$ngop monitor --c your_configuration_file &

## 6.3 Monitor Startup Configuration

In order to run the NGOP monitor, there must be a startup configuration
file. Below is an example that can be used as a template:

\<client_cfg\>

\<Client port='19996' host='ndem' name='NCSClnt'/\>

\<Client port='8080' host='ndem' name='CnfgClnt'/\>

\<CfgXml cvsRep='configxml' wrkDir='.ngop'

> cvsRoot=' :pserver :ngop@ndem.fnal.gov:/home/ngop/Repository'
>
> role='default'/\>

\<CfgGui Mail="[%User@fnal.gov](mailto:%25User@fnal.gov)" LogLength="10"
Upadte="1"/\>

\</client_cfg\>

The first \<Client\> tag has the host and port of the NGOP Central
Server. The next \<Client\> tag defines the port and host of the
Configuration Server.

The \<CfgXml\> tag defines the cvs repository where the configuration
files are checked out from (cvsRep**),** the directory where the files
are checkout into (wrkDir), the CVSROOT environment variable needed for
CVS (cvsRoot), the chosen set of NGOP configuration (role), the version
of NGOP(version), and the name.

The \<CfgGui\> tag specifies an email address for receiving notifictaion
(Mail), the number of events to store for each monitored object
(LogLength), and if the object "known status" modification should be
automaticaly propagate to display (Update). This configuration file
should conform to [the DTD rules.](#monitor-configuration-file-dtd)

## 

## ![](media/image3.png){width="2.138888888888889in" height="0.5020833333333333in"}6.4 NGOP Monitor Usage

When the ngop monitor command is issued, a popup window appears as
follows:

### 6.4.1 CONFIGURE 

When the user invokes the NGOP Monitor for the first time, she/he will
be prompted to select a desirable configuration. The selection request
will be sent to the CFMS and the corresponding configuration files will
be downloaded into a specific user area . In order to perform this
function, you must have access to the CVS repository where configuration
files are located. In addition, the CVSROOT environment variable must be
set, and possibly a cvs login should be performed.

You can skip this step if you already have a configuration in place and
choose DISPLAY. Also, you can choose to create a configuration yourself.
If you decide to do so, you have to place all of the configuration files
in the directory specified in the monitor configuration file. You must
then copy the following files into this directory:

-   \$NGOP_DIR/templates/monitor/defME.rules.xml

-   \$NGOP_DIR/templates/monitor/defSys.rules.xml

-   \$NGOP_DIR/templates/monitor/kn_st.xml

-   \$NGOP_DIR/templates/monitor/hostsInClusters.xml

-   \$NGOP_DIR/templates/monitor/service_class.xml

The CONFIGURE Menu has several options:

![](media/image3.png){width="2.1389763779527557in"
height="0.5019685039370079in"}![](media/image4.png){width="1.1700787401574804in"
height="0.9535433070866142in"}

#### 6.4.1.1 Components

To dowload a configuration, select Components. All monitored objects
known to the CFMS will be displayed in the left hand side window. Using
the - button, you can select systems and system views that you want to
download from the central repository.

![](media/image5.png){width="5.429861111111111in"
height="2.484722222222222in"}

The Refresh button is used when you want to replace all modified files.
Update will checkout all new files from CVS, but will not change the
files that you have modified. When you are finished with the selections,
click the Checkout button. The following popup window will appear:

![](media/image6.png){width="2.7in" height="1.132638888888889in"}

If you confirm your selection by clicking the OK button, the cvs
checkout will start and you will be able to see the progress:

![](media/image7.png){width="2.2395833333333335in"
height="1.1201388888888888in"}

***6.4.1.2* Icons For Objects/Alarms**

The icons that the NGOP display uses are customized by clicking on the
Icons For Objects and Icons For Alarms buttons. Click on the icon that
you want to replace. A window will be displayed with all of the
available icons. Simply click on the icon from this window that you want
to use. You can add your own icons by placing the files in bmp format in
\$NGOP_DIR/icons directory

![](media/image8.png){width="4.729166666666667in"
height="2.0104166666666665in"}

***6.4.1.3* Colors**

This option allows you to select colors that represents various statuses
of monitored objects. When you select Colors, a window will popup that
displays the mapping between a status and it's color. To modify a color,
click on the colored tile that you wish to change. A popup window will
be displayed that allows you to modify this color.

## ![](media/image9.png){width="1.2602362204724409in" height="2.4374015748031495in"}![](media/image10.png){width="2.7291338582677165in" height="1.7291338582677165in"}

## 

## 

## 

## 

## 

#### 6.4.1.4 Mailing Address

You can modify your mailing address by choosing the Mailing
Address![](media/image11.png){width="2.9895833333333335in"
height="0.6041666666666666in"} button. The following window will be
displayed after choosing this option.

## 

## 

### 6.4.2 DISPLAY

To start monitoring, click the DISPLAY button from the main menu. If a
monitor configuration file CFMS is defined (\<Client port='8080'
host='ndem' name='CnfgClnt'/\>), but you didn't configure your system, a
popup window will be displayed asking if you want to the CFMS.

The status of the system views and monitored objects is represented by
the color of the icon. In addition, a set of colored arrows that will be
displayed over the icon exists to indicate *potential* problems. There
is a hierarchy of color (based on severity level of the occurred event)
to represent the proximity to the possible problem. The first level of
monitored hierarchy could look as follows:

![](media/image12.png){width="3.290277777777778in"
height="3.504861111111111in"}

## 

## 

## 

## 

## 

By default the window is divided into three sections. The uppermost part
displays current level of hierarchy. The middle section(not shown in
example below) displays the action log. The bottommost section displays
the event log. The displaying of the lower two sections is controlled
from within the Options menu.

When an icon is clicked with Button-1 of the mouse, the next level in
the hierarchy is displayed. Button-2 will allow you to see the alarms
that are present, and Button-3 allows you to see object status rule
sets, and change current status or request an update.

> ![](media/image13.png){width="3.3in"
> height="4.005511811023622in"}![](media/image14.png){width="2.2901574803149605in"
> height="2.340944881889764in"}![](media/image15.png){width="1.2102362204724408in"
> height="0.6838582677165355in"}![](media/image16.png){width="1.3299212598425196in"
> height="0.9102362204724409in"}![](media/image17.png){width="3.6251968503937007in"
> height="2.0519685039370077in"}

Clicking on a log message will display a window with the message details
and allow you to acknowledge the
![](media/image18.png){width="1.8402777777777777in"
height="2.502083333333333in"}message.

## 

## 

## 

## 6.5 Monitor Configuration

NGOP uses XML to describe the hierarchy and status rules, as well as
information about "out of service" monitored objects, available service
classes, existing hosts and clusters. The description of the NGOP object
hierarchy is located in one or multiple configuration files. Each status
rule set should be placed in a separate file.

### 6.5.1 Common Configuration Files 

There are several configuration files that contain general information
needed for the NGOP Monitor. These files include data about "out of
service" monitored objects, available service classes, existing hosts
and clusters.

These files will be downloaded into user configuration area regardless
of which monitored objects are selected by the user. These are
considered the default configuration files.

### 6.5.2 File service_class.xml 

The service_class.xml configuration file contains information about
defined types of service. The service type is associated with the hosts
and monitored objects. By default, a monitored element, located on a
host has the same service type as this host. A service type defines the
time period of active monitoring.

This file has the following required declarations and tags:

\<?xml version='1.0'?\>

\<!DOCTYPE NGOPConfig SYSTEM "service_class.dtd"\>

\<NGOPConfig\>

\<Default_File/\>

\<ServiceClass\>

......... - definition of service type should be placed here

\</ServiceClass\>

\</NGOPConfig\>

A \<ServiceClass\> tag contains definition of the several service types
(tag \<ServiceType\> , such as "9x5" or "24x7"("9x5" means that a
particular monitored object/host will be actively monitored from 8 am to
5 pm on weekdays). The default service type is "24x7". A service type is
described by a mathematical expression by using an \<apply\> tag. If the
expression is evaluated to be false, all events occurred with the
corresponding monitored object/host will be ignored. Within an \<apply\>
tag, a \<ci\> tag could assume only two values: hour or day_of_the_week.
Days of the week are represented by an array of integers, where 0
corresponds Monday. Hour is represented by an integer value within 0 --
24 range. This configuration file should conform the to [the DTD
rules](#_Service_Class_DTD).

***Example*:**

\<ServiceType name="9x5"\>

\<apply\>

\<and/\>

\<apply\>

\<geq/\>

\<ci\>hour\</ci\>

\<cn\>8\</cn\>

\</apply\>

\<!---(hour\>=8)\--!\>

\<apply\>

\<leq/\>

\<ci\>hour\</ci\>

\<cn\>17\</cn\>

\</apply\>

\<!---(hour\<=17)\--!\>

\<apply\>

\<notin/\>

\<ci\>day_of_the_week\</ci\>

\<cn\>\[5,6\]\</cn\>

\</apply\>

\<!---(day_of_the_week not in \[Saturday,Sunday\])\--!\>

\</apply\>

\<!---this just means that "9x5" service type is defined between
8:00-17:00 every day except Saturday and Sunday\--!\>

\<!---see apply for details\--!\>

\</ServiceType\>

### 6.5.3 File hosts_in_clusters.xml 

The hosts_in_clusters.xml configuration file contains clusters and hosts
that exist in the system. The service type of each host is defined in
this configuration. If a service type is not defined, the default
service type is assumed for a host. This file has the following required
declaration and tags:

\<?xml version='1.0'?\>

\<!DOCTYPE NGOPConfig SYSTEM "hosts_in_clusters.dtd"\>

\<NGOPConfig\>

\<Default_File/\>

\<HostsInClusters\>

......... - known status definition should be placed here

\</HostsInClusters\>

\</NGOPConfig\>

A \<HostsInClusters\> tag contains multiple \<Cluster\> tags. A
\<Cluster\> tag has one required attribute (Name).

A \<Cluster\> tag contains other \<Cluster\> or \<Host\> tags. A
\<Host\> tag also has Name as the only required attribute.

A \<ServiceType\> tag is placed anywhere within a \<HostsInClusters\>
tag. It is defined the service type for all clusters and hosts it
contains. A \<ServiceType\> tag has \<Name\> as the one required
attribute. Name contains the name of the service type defined in
service_class.xml.

This configuration file should conform to [the DTD
rules](#hostsinclusters-dtd).

### Example:

\<ServiceType Name="24x7"\>

\<Cluster Name="FNALU_BATCH"\>

\<Cluster Name=" FNALU_BATCH_OSF1"\>

\<Host Name="fdei01"/\>

\</Cluster\>

\<Cluster Name=" FNALU_BATCH_IRIX"\>

\<Host Name="fsgb02"/\>

\<Host Name="fsgb03"/\>

\<Host Name="fsgi02"/\>

\<Host Name="fsgi03"/\>

\</Cluster\>

\<Cluster Name=" FNALU_BATCH_Solaris"\>

\<Host Name="fsub01"/\>

\<Host Name="fsui02"/\>

\<Host Name="fsui03"/\>

\</Cluster\>

\</Cluster\>

\</ServiceType\>

This example describes the cluster FNALU_BATCH. It has three sub
clusters:

-   FNALU_BATCH_IRIX with hosts:

```{=html}
<!-- -->
```
-   fsgb02

-   fsgb03

-   fsgi02

-   fsgi03

```{=html}
<!-- -->
```
-   FNALU_BATCH_OSF1 with host:

```{=html}
<!-- -->
```
-   fdei01

```{=html}
<!-- -->
```
-   FNALU_BATCH_Solaris with hosts:

```{=html}
<!-- -->
```
-   fsub01

-   fsui02

-   fsui03

All hosts that belong to the FNALU_BATCH cluster require 24x7
maintenance support.

### 6.5.4 File known_status.xml 

The known_status.xml configuration file contains references to the
monitored objects or hosts that are known to be out of service for a
significant period of time. A monitored object/host is marked as "bad",
"in repair" or "test". If a monitored object/host is not listed in this
file, its status is working. This file has the following required
declaration and tags:

\<?xml version='1.0'?\>

\<!DOCTYPE NGOPConfig SYSTEM "known_status.dtd"\>

\<NGOPConfig\>

\<Default_File/\>

\<KnownStatus\>

......... - known status definition should be placed here

\</KnownStatus\>

\</NGOPConfig\>

A *\<*KnownStatus*\>* tag contains multiple *\<*Status*\>* tags. A
\<Status\> tag has one required attribute; Name that can assume the
values "bad", "in_repair", or "test".

You can specify the "out of service" time interval
(*\<*OutOfServiceInterval*\>* tag) within the *\<*Status*\>* tag. It
includes one optional attribute Description and the following required
attributes:

StartDateTime -- "yyyy-mm-dd hh:mm"

EndDateTime -- "yyyy-mm-dd hh:mm"

If an object/host needs to be marked "out of service" for periodic
maintenance, it is specified using a *\<*MaintenancePeriod*\>* tag. This
tag has three required attributes:

DaysInterval -- the maintenance reoccurrence interval in days (the count
starts from the day defined in *\<*StartDate*\>)*

FromTime -- the maintenance start time (hh:mm)

HoursDuration - maintenance duration in hours

Out of service monitored objects and hosts are listed within the
corresponding *\<*Status*\>* tag. This configuration file should conform
to [the DTD rules.](#knownstatus-dtd)

### Examples:

\<Status Name="bad"\>

\<Host Name="fnpc110"/\>

\<System Name="LSF" Cluster="fsgb02" /\>

\</Status\>

This declares host fnpc110 and system LSF.fsgb02 to be in a known bad
condition.

\<Status Name="in_repair"\>

\<OutOfServiceInterval StartDateTime="2001-05-01 12:30"\>

\<System Name="OCS" Cluster="FixTarget"/\>

\</OutOfServiceInterval\>

\</Status\>

\<Status Name="test"\>

\<OutOfServiceInterval StartDateTime="2001-05-04 08:30"\>

\<MaintenancePeriod DaysInterval="7" FromTime="08:30"

> HoursDuration="4"\> \<Host Name="fnpc107"/\>

\</ MaintenancePeriod\>

\</OutOfServiceInterval\>

\</Status\>

This declares the system OCS.FixTarget to be in repair since May 1, 2001
12:30 and host "fnpc107" being used for testing purpose weekly from 8 am
to 12 pm since May 4, 2001

## 6.6 NGOP Hierarchy Definition 

An NGOP monitored hierarchy consists of system views, systems, and
monitored elements. The system and system view definitions are placed in
one or multiple configuration files. The monitored element definitions
should be always placed within the system definition. Every
configuration file describing the NGOP monitored hierarchy has the
following required declaration and tags:

\<?xml version='1.0'?\>

\<!DOCTYPE NGOPConfig SYSTEM "hierarchy.dtd"\>

\<NGOPConfig\>

......... - definition of system view, system, and monitored elements
should be placed here

\</NGOPConfig\>

The following XML tags are used to describe the monitored hierarchy:

\<SystemView\>

\<System\>

\<Monitoried Element\>

A \<For\> tag can be used anywhere in the monitored hierarchy definition
in order to replicate some XML fragments.

### 6.6.1 System View

A System View is uniquely defined by its id. A system view contains only
references to the other system views and monitored objects.
(**Important**: all components of the hierarchy should be defined
elsewhere!)

A *\<*SystemView*\>* tag has the following attributes:

ID (required)

RefRule - a reference to the status rule set, describing the status
rules for this system view, the default value is "SystemViewDefRuleSet"

This configuration file should conform [the DTD
rules](#monitored-hierarchy-dtd).

#### 

#### Example 1:

\<SytemView ID="LSF_Fnalu_Batch"\>

\<SystemView ID="Fnalu_Batch_Irix"/\>

\<SystemView ID="Fnalu_Batch_Solaris"\>

\<!---references to the system views\-\--!\>

\<System Name=Ping Cluster="Fnalu_Batch"/\>

\<!---reference to the system\--!\>

\<System Name="OSHealth" Cluster="Fnalu_Batch"\>

\<MonitoredElement Name="/tmp" Host= Host="fsgb02"/\>

\<MonitoredElement Name="/tmp" Host="fsgb03" /\>

....

\</System\>

\<!---references to monitored elements\--!\>

\</SystemView\>

This example defines a system view LSF_Fnalu_Batch that contains two
other system views (Fnalu_Batch_Solaris and Fnalu_Batch_Irix), one
system (Ping.Fnalu_Batch), and several monitored elements
(/tmp.fsgb03.OSHealth.Fnalu_Batch for example).

***Example 2:***

The following example defines system views Fnalu_Batch_Irix that
contains three LSF systems running on nodes named fsgb02, fsgb03, and
fsgi02.

\<SystemView ID="Fnalu_Batch_Irix"\>

\<System Name="LSF" Cluster="fsgb02"/\>

\<System Name="LSF" Cluster="fsgb03"/\>

\<System Name="LSF" Cluster="fsgi02"/\>

\</SystemView\>

### 6.6.2 System

A *\<*System*\>* tag contains multiple *\<*MonitoredElement*\>* tags and
should be referenced at least once within *\<*SystemView*\>* tag. A
definition of a system hierarchy should be placed outside system view
scope. In the NGOP hierarchy definition a \<System\> tag has two
additional optional attributes:

ServiceType -- default "24x7"

RefRule - a reference to the status rules set, describing the status
rules for this system, the default value is "SystemDefRuleSet"

This configuration file should conform [the DTD
rules](#monitored-hierarchy-dtd).

***Example:***

The following example defines a system called OSHealth.Fnalu that is
monitored around on a 24x7 basis. The status rule set defining the
status of this system is described in SGIHealthRuleSet. The system
consists of several monitored elements ("ping.fsgb02.Ping.Fnalu_batch"
for example).

\<System Name="Ping" Cluster="Fnalu_Batch" ServiceType="24x7"

RefRule="SGIHealthRuleSet"\>

\<MonitoredElement Name="ping" Host="fsgb02" Type="Hardware"/\>

\<MonitoredElement Name="ping" Host="fsgb03" Type="Hardware"/\>

\<MonitoredElement Name="ping" Host="fsub02" Type="Hardware"/\>

\</System\>

### 6.6.3 Monitored Element

A \<Monitored Element\> tag is encountered only within \<System\> tags
and has two additional optional attributes:

ServiceType *--* default is service type of the host

RefRule -- a reference to the status rule set, describing the status
rules for this monitored element, the default value is "MEDefRuleSet"

This configuration file should conform [the DTD
rules](#monitored-hierarchy-dtd).

***Example:***

The following example defines the monitored elements with an id of
cpuLoad.fnsfo.OSHealth. Fnalu and a Type of sysUsage. The status rule
set defining the status of this monitored element is described in
MEDefRuleSet and the service type is the service type of the host fnsfo.

\<System Name="OSHealth" Cluster="Fnalu"\>

\<MonitoredElement Name="cpuLoad" Host="fnsfo" Type="sysUsage"/\>

\</System\>

## 6.7 Status Rule Sets

Every set of status rules is associated with some systems view or
monitored objects. When the NGOP Monitor receives an event regarding an
object, it uses set of status rules associated with this object to
define its status and severity level. It also applies the corresponding
rules to every component of the hierarchy this object belongs. In the
NGOP configuration, a \<StatusRuleSet\> tag with required attribute ID
represents the set of status rules. Every set of status rules definition
is located in a separate file and has the following required declaration
and tags:

\<?xml version='1.0'?\>

\<!DOCTYPE NGOPRules SYSTEM "rules.dtd"\>

\<NGOPRules\>

\<StatusRuleSet ID="MEDefRuleSet\>

......depenent list could be placed here

...... rules

\</StatusRuleSet\>

\</NGOPRules\>

The content of the set of status rules definition is divided into two
parts:

-   Dependent list - list of all objects that this particular monitored
    object depends on

```{=html}
<!-- -->
```
-   Rules

A Dependent list is omitted if a monitored object doesn't depend on any
other object. This configuration file should conform to [the DTD
rules](#statusrulesset-dtd).

### 6.7.1 Dependent List

A dependent list contains a list of the references to monitored objects
and system views. In the NGOP configuration, a \<DependList\> tag
represents a dependent list*.* In a dependent list, monitored
objects/system views are arranged in groups. A group may contain other
groups and is represented by a \<Group\> tag that has one required
attribute Name (it should be unique only within this \<SatusRulesSet\>
definition). Every group has a parameter "%GroupLen" that is equal to
the total number of monitored objects in the group. A system may contain
one special empty group with the attribute Name set to "{self}". It
means that this system depends on all monitored elements that it
contains. All objects in a dependent list are ordered by their
appearance relative to a particular group. A \<For\> tag may be used in
a dependent list. This XML fragment should conform to [the DTD
rules](#_Status_Rule_Set_1).

######################################### **Example**:

This is an example of dependent list that consist of the "self" group:

\<DependList\>

\<Group Name = "{self}"/\>

\</DependList\>

The FBS system is a batch system developed at Fermilab. FBS depends on a
bmgr and logd process running on a central node. FBS depends on the
central node being up. FBS also depends on a process called a *launcher*
to be running on all nodes in the system that can run a batch process.
FBS runs on a cluster. In this example, the clusters CDFFarm and D0Farm
(defined in the HostsInClusters.xml file) are running the FBS system.

HostsInClusters.xml:

\<?xml version='1.0'?\>

\<!DOCTYPE NGOPConfig SYSTEM "hosts_in_clusters.dtd"\>

\<NGOPConfig\>

\<Default_File/\>

\<HostsInClusters\>

\<Cluster Name="CDFFarm"\>

\<Cluster Name="CDFFarmIO"\>

\<Host Name="cdffarm1"/\>

\</Cluster\>

\<Cluster Name="CDFFarmWorker"\>

\<Host Name="fncdf1"/\>

....

\<Host Name="fncdf90"/\>

\</Cluster\>

\</Cluster\>

\<Cluster Name="D0Farm"\>

\<Cluster Name="D0FarmIO"\>

\<Host Name="d0bbin"/\>

\</Cluster\>

\<Cluster Name="D0FarmWorker"\>

\<Host Name="fnd01"/\>

....

\<Host Name="fnd100"/\>

\</Cluster\>

\</Cluster\>

....

\</HostsInClusters\>

\</NGOPConfig\>

\<?xml version='1.0'?\>

\<!DOCTYPE NGOPRules SYSTEM "statusruleset.dtd"\>

\<NGOPRules\>

\<FBSInstance Name="FBS"\>

\<Instance Name="D0"/\>

\<Instance Name="CDFFarm"/\>

\</FBSInstance\>

\<For Each="Instance" Var="{%I}" In="FBSInstance" Name="FBS"\>

\<StatusRuleSet ID="FBS{%I}RuleSet\>

\<DependList\>

\<Group Name="fbs_daemon/\>

\<System ID="FBS" Cluster="{%I}Farm" \>

\<For Each="Host" Var="{%H}" In="Cluster" Name="{%I}FarmIO"

> Filename="HostsInClusters.xml"\>

\<MonitoredElement Name="bmgr" Host="{%H}" /\>

\<MonitoredElement Name="logd" Host="{%H}" /\>

\</For\>

\</System\>

\</Group\>

\<!---logd could be referenced in DependRule as fbs_daemon\[1\]\--!\>

\<Group Name="launcher"\>

\<System ID="FBS" Cluster="{%I}Farm" \>

\<For Each="Host" Var="{%H}" In="Cluster" Name="{%I}FarmWorker"\>

\<MonitoredElement Name="launcher Host="{%H}"/\>

\</For\>

\</System\>

\</Group\>

\<!---launcher on fncdf1 could be referenced in DependRule as
launcher\[0\]\--!\>

\<Group Name="hostUP"/\>

\<System Name="Ping" Cluster="{%I}FarmIO"\>

\<For Each="Host" Var="{%H}" In="Cluster" Name="{%I}FarmIO"\>

\<MonitoredElement Name="ping" Host="{%H}"/\>

\</For\>

\</System\>

\</Group\>

\</DependList\>

...

\</StatusRuleSet\>

\</NGOPRules\>

### 6.7.2 Rule

When the NGOP Monitor receives an event it performs the following steps:

1.  Finds the monitored object associated with this event

2.  Finds the status rule set that defined rules for this monitored
    object

3.  Evaluates an expression defined in every rule

4.  Applies the rule (sets status and severity level) if an evaluated
    expression is true. The worst status/severity level of the
    corresponding rule with the highest priority will determine ultimate
    object's status/severity level.

5.  Identifies all the members of hierarchy that is affected by the
    change of this monitored object status.

6.  Repeats steps 2-6 until there is no more affected members of
    hierarchy (step 5).

There are two implemented rule types.

-   A Generic Rule ( *\<*GenericRule*\>* tag ) sets the monitored object
    status and severity level based on the event received from the NCS.

-   A Dependent Rule (*\<*DependRule*\>* tag) sets the monitored element
    status and severity level based on the event received from the NCS
    and the status of each dependent monitored object in some group.

All these rules have three required attributes:

-   Status -- This can assume a special value "None" indicates that this
    rule will not change an existing status. In a dependant rule the
    Status of dependent list members is used in the expression.

-   Prio (Priority) -- This indicates the importance of the particular
    rule. It can assume any integer value greater than or equal to 0.
    The lower the value, the less important the rule is. If several
    rules are satisfied, the status and severity level of the monitored
    object will be the one associated with the rule with the highest
    priority.

-   SevLevel (Severity Level) -- This can assume a special value of
    "None" that indicates that this rule will not change the existing
    severity level.

There is one optional attribute:

-   Dsc (Description)*.* -- Description is an explanation of the
    condition of a rule. Special parameters may be included in a
    description such as %ID , %Host or %Event. These parameters will be
    replaced by the corresponding values of the monitored object
    associated with this rule.

Every rule contains an expression that has to be evaluated upon receipt
of an event. In an expression any particular field of the event is
referred by its name. An Action is attached to any of the rules.

***Example***:

Let's assume that the agent "LinuxHealth" is monitoring the file system
"/export/home" on the worker node "fnd01". This file system should be
mounted from the I/O node "d0bbin". The LinuxHealth Agent can generate
events in three cases:

1.  The file system is not mounted

2.  Automount program is not running

3.  The file system is more than 95% full

The status of the monitored element should change upon receiving any of
these events unless the I/O node is down.

In order to do so the set of status rules (FileSystemRuleSet) should
include the following:

  ----------------- ----------------- ----------------- -----------------
  Rule Type         Status            Priority          Evaluated
                                                        Expression

  Dependent         Good              1                 d0bbin is down

  Generic           Bad               0                 File system is
                                                        not mounted

  Generic           Error             0                 Automount is not
                                                        running

  Generic           Warning           0                 File system is
                                                        95% full
  ----------------- ----------------- ----------------- -----------------

If at some point we receive the event (1.), the status becomes "Bad" if
the I/O node is up and "Good" if the I/O node is down.

### 6.7.3 Generic Rule Example

This rule is applied to a particular monitored object if the event
associated with this object has a "State" of 1 ("Up"). The severity
level remains unchanged. This XML fragment should conform the DTD rules.

\<GenericRule Status="Good" Prio="0" SevLevel="None" \>

\<apply\>

\<eq/\>

\<ci\> State \</ci\>

\<cn\> 1 \</cn\>

\</apply\>

\<!---if expression (State==1) is true , rule is applied\--!\>

\</GenericRule\>

This rule is applied to a particular monitored object if the event
associated with this object has "State" value equal to 0 ("Down"). The
severity level remains unchanged.

\<GenericRule Status="BAD" Prio="0" SevLevel="None" \>

\<apply\>

\<eq/\>

\<ci\>State\</ci\>

\<cn\>0\</cn\>

\</apply\>

\<!-if exression (State == 0) is true then the rule is applied \--!\>

\</GenericRule\>

This rule is applied to a particular monitored object if the event
associated with this object has "State" value equal to 1 ("Up") and
"SevLevel" value equal to 6 ("Bad"). It set status to "Error".

\<GenericRule Status="Error" Prio="0" SevLevel="None"\>

\<apply\>

\<and/\>

\<apply\>

\<eq/\>

\<ci\>State\</ci\>

\<cn\>1\</cn\>

\</apply\>

\<apply\>

\<eq/\>

\<ci\>SevLevel\</ci\>

\<cn\>6\</cn\>

\</apply\>

\</apply\>

\<!---if expression ((State==1) && (SevLevel==6)) is true , rule is
applied\--!\>

### 6.7.4 Dependent Rule

A Dependent Rule allows for the use of objects from a dependent list in
an expression. These objects are indexed by their position within a
specific group of a dependent list. For example, the object that is
listed third in the group named "fbs_daemon" is referred as
"fbs_deamon\[2\]" (indexing starts with 0) in an expression. If a
dependent list of a system status rule set has a group with
Name="{self}", the i-th monitored object that belongs to this system is
referred as "{self\[i-1\]}". This XML fragment should conform to [the
DTD rules](#statusrulesset-dtd).

######################################### 

######################################### **Example**

This rule is applied to the "FBS" system when NGOP reports that the bmgr
daemon is not running. bmgr is the first element (fbs_daemon\[0\]) of
the fbs_daemon group in the dependent list of the FBS rule (see
dependent list Example)

\<DependRule Status="Bad" Prio="1" SevLevel="None"
Dsc="Batch_Manager_is_down"\>

\<apply\>

\<and/\>

\<apply\>

\<eq/\>

\<ci\>fbs_daemon\[0\].EventType\</ci\>

\<cn\>"Daemon"\</cn\>

\</apply\>

\<apply\>

\<eq/\>

\<ci\>fbs_deamon\[0\].State\</ci\>

\<cn\>0\</cn\>

\</apply\>

\</apply\>

\<!---if expression ((bmgr.EventType=="Daemon") && (bmgr.State==0)) is
true then rule is applied\--!\>

This rule is applied to the FBS system when NGOP reports that the FBS
central machine is down. ping is the first element of the "hostUp" group
(hostUp\[0\])in the dependent list of the FBS rule (see dependent list
Example).

\<DependRule Status="Unknown" Prio="1" Dsc="%Host_is_down"\>

\<apply\>

\<eq/\>

\<ci\>hostUp\[0\].State\</ci\>

\<cn\>0\</cn\>

\</apply\>

\</DependRule\>

# Chapter 7: Configuration File Manager

The configuration files for the NGOP system are monitored by a separate
set of processes referred to as the Configuration File Manager System
(CFMS) which cooperate:

-   Configuration file Librarian

```{=html}
<!-- -->
```
-   Configuration file Indexer

```{=html}
<!-- -->
```
-   Configuration file Broker

```{=html}
<!-- -->
```
-   Administrative client

```{=html}
<!-- -->
```
-   Monitoring Client

Each of these packages has distinct responsibilities and is described
below.

## 7.1: Librarian

The librarian is responsible for maintaining the master copies of the
configuration files. In addition, the librarian is responsible for:

-   Authenticating that users have permissions for file modification

```{=html}
<!-- -->
```
-   Maintaining revision history of files to allow checkpointing,
    rollback, commit, and full revision history.

```{=html}
<!-- -->
```
-   Delivering the contents of particular configuration files to
    Monitoring Clients, the NCS, and the CFM Indexer.

The librarian uses CVS to store configuration files.

## 7.2: Indexer

The Indexer reads a CVS tagged set of configuration files from the
Librarian, generates an index listing of the files needed for each
component of the system. The indexer also performs syntax and basic
sanity checks of the configuration files, as well as finding
dependencies.

## 7.3 Broker

The Broker communicates with two types of clients, and has distinct
responsibilities for each:

*Monitoring /Action Clients*

> When the monitoring client connects to the Broker, it sends the broker
> a subscription list of components. The Broker uses the indices
> generated by the Indexer to repeatedly send a revision tag and list of
> configuration files to the Monitoring Client---once initially, and
> then again as new indices are created. The monitoring client then
> requests those configuration files directly from the Librarian.

*Administrative Clients*

> The Broker accepts requests from the Administrative Client including a
> version control/rollback tag after the admin client has run the
> indexer and checked the new index is with the Librarian

## 7.4 CFMS Configuration File

CFMS configuration file contains the following information:

\<client_cfg\>

\<Client port="8080" host="ngop" name="CnfgClnt"/\>

\<CfgXml cvsRep="configxml"

> cvsRoot=":pserver:ngop@ngop.fnal.gov:/home/ngop/Repository"

version="ngop_v1_1" name="CfgXmlAC"/\>

\</client_cfg\>

The \<client_cfg\> tag client tag defines parameters that are required
to start CFMS. This tag is required and it includes the following
attributes: TCP Port to connect to CFMS, host name where CFMS is
running, and the name of CFMS (CnfgClnt).

The CfgXml tag is required. It defines the parameters that will be used
to create local configuration and connect to CVS repository. The CfgXml
tag includes the following attributes: name of ngop configuration cvs
repository, CVSROOT definition, tag of current configuration version and
the name. This configuration file should conform to [the DTD
rules](#cfms-configuration-file-dtd).

## 7.5 CFMS Starting/Stopping

The CFMS is started in several ways. If the CFMS configuration file is
located in

/var/ngop/cfms directory, it is started issuing the following command:

ngop start cfms

To start the CFMS with your own configuration file, use the following
command:

ngop cfms --c config_file

## 7.6 Administrative Client

The administrative client allows one to:

-   modify/create one or more configuration files (via the Librarian,
    and an appropriate editor)

```{=html}
<!-- -->
```
-   Run consistency checks on the files (by invoking Indexer)

```{=html}
<!-- -->
```
-   Commit a set of changed files (possibly yielding a CVS tag)

```{=html}
<!-- -->
```
-   Notify the Broker that the new taged version of configuration files
    are available.

## 7.7 Admin Starting/Stopping

The Admin GUI is started issuing the following command:

ngop admin --c config_file

The Admin GUI uses the same configuration file as CFMS. Below is a
screen sample from the Admin GUI.

# ![](media/image1.png)

# 

# 

# 

# 

# 

# 

# 

# Chapter 8: Archive Server

## 8.1 Archive Server Overview

The Archive/History Server System is responsible for storing and
retrieving messages generated by the NGOP system. Each message sent to
the archive server is stored in an Oracle database. There are four major
components of the Archive Server:

-   Server: This process runs on an Oracle client machine and accepts
    messages from the NCS. It immediately caches the requests to local
    disk.

```{=html}
<!-- -->
```
-   Database Interface: This process takes the requests that have been
    cached by the server and stores them in the database. Having a
    separate process to store the data in the database allows the server
    to continue to run even if there are problems with the Oracle
    database.

```{=html}
<!-- -->
```
-   Web Interface: The information in the database is retrieved using a
    web-based interface.

```{=html}
<!-- -->
```
-   Cleanup Process: This process processes records in the Oracle
    database and rolls messages off that are more than 15 days old.

## 8.2 Starting and Stopping

The script to start the Archiver is located in the
\$NGOP_DIR/prototype/archsrv/src/server directory. \$NGOP_DIR is set
with UPS by issuing setup ngop**.** To start the archiver daemons:

setup ngop

cd \$NGOP_DIR/prototype/archsrv/src/server

start_daemons

The start_daemons script launches two other scripts: start_archiver and
start_dbinter**.** These scripts sit in a loop and periodically check to
make sure the daemons are running. If for some reason the daemons die,
the scripts will restart them.

## 8.3: Archiver Configuration

Below is an example configuration file for the archive server:

\<ArchiverConfig\>

\<Port\>7001\</Port\>

\<ArchiverHost\>fncduh1.fnal.gov\</ArchiverHost\>

\<OraUser\>oracle_user\</OraUser\>

\<OraPW\>oracle_pw\</OraPW\>

\<OraInstance\>procdev\</OraInstance\>

\<LogPath\>/home/fncduh/ngop/serverlog/log.out\</LogPath\>

\<DBInterSleepInterval\>15\</DBInterSleepInterval\>

\<RequestDirectory\>/home/fncduh/ngop/scratch\</RequestDirectory\>

\<ErrorDirectory\>/home/fncduh/ngop/errors\</ErrorDirectory\>

\</ArchiverConfig\>

The Port and ArchiverHost are the port and host that the archive server
is listening on for requests. The OraUser/OraPW is the Oracle userid and
password of an owner that can write into the archive tables. OraInstance
is the Oracle instance that the tables reside in. LogPath points to the
file that contains the log files created by the archiver.
DBInterSleepInterval is the time in seconds that the database interface
program will look in RequestDirectory to process new messages.
ErrorDirectory is the pathname where requests that could not be
processed are placed. This configuration file should conform to [the DTD
rules](#archiver-configuration-file-dtd).

# Chapter 9: Monitoring Agents

## 9.1: Overview

Monitoring Agents (MA) are processes that monitor some entity and report
a status to the NCS. NGOP provides a basic set of MA's, but users are
free to write their own. The MA is the element that gives NGOP a great
deal of flexibility.

A Monitoring Agent(MA) includes the following features:

-   interfaces to NCS

-   monitors the characteristics of a particular monitored object

-   sends events to the NCS when characteristic of the object meets
    specific conditions (An MA doesn't send an event when the monitored
    object doesn't meet any conditions. In this case the State of the
    monitored object is assumed to be UP. A MA will send an event if a
    monitored object satisfies some condition)

-   performs local actions

-   sends requests to perform centrilized actions

-   sends heartbeats to the NCS

-   resends events and configuration when the connection with the NCS is
    interrupted

-   MA configuration, conditions and actions associated with conditions
    are described in the configuration file using XML. This file should
    be located on the node where MA is running

NGOP provides a framework for creation of the MAs**:** either by using
the MA API or the PlugIns Agent.

## 9.2 Plugin Agent 

A PlugIn Agent provides the monitoring of software or hardware
components utilizing existing scripts or executables (plug-ins). These
plug-ins should be able to measure and print some quantitative
characteristics of the monitored objects. A configuration file
describing the monitored hierarchy, plug-ins and set of conditions is
required in order to use a PlugIns Agent. This configuration file should
conform to specific DTD rules. A configuration file should start with
the following XML declarations:

\<?xml version="1.0"?\>

\<!DOCTYPE MA-Config SYSTEM "agent.dtd"\>

The first tag of a PlugIns Agent XML document is a *\<*MA_config*\>*
tag, which defines the MA configuration. This tag is required, and
requires to additional attributes:

-   Name -- the name of the PlugIn Agents.

```{=html}
<!-- -->
```
-   Update -- specifies the time interval in seconds between running the
    plug-in agent.

-   Type -- specifies the type of the MA. There are two possible types:

    -   Daemon (default) - Monitoring Agent that should be always
        present

    -   Cron -- Monitoring Agent that will run for a short period of
        time and then reaper within the time interval specified in the
        heartbeat attribute

An \<NCS*\>* tag (required) defines the NCS parameters and includes the
following required attributes:

-   Port **-** the NCS UDP port

-   Host **-** the NCS host

-   Heartbeat **-** specifies the heartbeat interval in seconds

A system description should follow the *\<*NCS*\>* tag. Several systems
are described in the same XML document. A *\<*System*\>* tag indicates
the beginning of the system definition. It contains multiple monitored
elements.

A *\<*ConditionSet*\>* tag indicates the beginning of the condition set
definition. A \<ConditionSet\> tag could be placed within the
*\<*MonitoredElement*\>* or *\<*System*\>* tags. It contains the
description of a plug-in and at least one condition. The *\<*fn*\>* tag
describes plug-in that should be executed in order to define the state
of a monitored. It has the following required attributes:

-   Name - the name of the operation ("plug_in" for all the PlugIns
    Agents)

-   Arg -- the full path to the plug-in that needs to be executed to
    verify the state of monitored object (Parameters %ID, %Name,
    %Cluster and %Host is used in an attribute Arg and will be
    substituted with the corresponding values of monitored object)

-   RetVal -- the description of return values. It has the following
    format:

"type:var_name, type:var_name..."

> where type is float**,** int**,** string, array int, array float or
> array string

***Important***: in case of int, float or string types the return values
should be returned in the standard output of a plug-in, and separated by
a newline character. In case of an "array ..." type the return values
should be returned in the standard output of a plug-in, and separated by
space (see Example for more details). If the plug-in exits with non-zero
exit code then the return value is set to Error and the following event
will be generated:

"Date=... ID=... EventType=" executable" EventName="plug-ins" State=-2
Description="Failed to execute command"

The *\<*Condition*\>* tag indicates the beginning of the condition
definition and has the following attributes:

-   State (required) - defines the monitored object state if the
    occurred event satisfied this condition

-   SevLevel (required) - severity level of the event that satisfied
    this condition

-   Description (required) - readable description of the
    event(Parameters %ID, %Name, %Cluster ,%Host and %Event is used in
    an attribute *Description* and will be substituted with the
    corresponding values of monitored object)

-   EventName *(*optional) - defines the event name if it is different
    from the monitored object name (see Event)

-   EventType (optional) - defines the event type if it is different
    from the monitored object type (see Event)

The \<apply\> tag indicates the beginning of a mathematical expression
that should be evaluated in order to determine if the condition is
satisfied. If an expression is evaluated to be true a PlugIn Agent will
generate an event. A special variable %len(retValue_array_name) is used
in a \<ci\> tag. It refers to the length of the array in plug-ins return
value and is used in \<sum\>, \<min\> and \<max\> function operators
(see Example for more details).

The *\<*Action*\>* tag is optional and it indicates the beginning of
action definition. If the condition is satisfied and the action is
defined, then the PlugIns Agent will perform this action locally or send
request to the NCS to execute this action.

The general structure of a PlugIns Agent configuration file should look
like the following:

\<?xml version="1.0" ?\>

\<!DOCTYPE MA-Config SYSTEM "agent.dtd" \>

\<MA-Config Name....\>

\<NCS Host=.../\>

\<System Name=..\>

\<MonitoredElement Name=...\>

\<ConditionSet\>

\<fn Name="plug-ins" Arg=.../\>

\<Condition State=...\>

\<apply\>

> \<!---expression\--!\>

\</apply\>

\<Action\>

\<!---action\--!\>

\</Action\>

\</Condition\>

\<!---more conditions\--!\>

\</ConditionSet\>

\</MonitoredElement\>

\<!---more monitored elements\--!\>

\</System\>

\<!---more systems\--!\>

\</MA-Config\>

In the PlugIns Agent configuration localhost instances will be replaced
by the local host name. This configuration file should conform to [the
DTD rules](#plugins-agent-dtd).

***Example 1***:

Let's assume that we want to monitor the system load averages for the
past 1, 5, and 15 minutes using the following command as a "plugin":

uptime\|awk '{print \$(NF-2),\$(NF-1),\$NF}'\|awk -F',' '{print
\$1,\$2,\$3}'

Assume that we want the PlugIns agent to generate an event when the
minimum of cpu load averages exceeds 12.0. The fragment of the PlugIns
configuration file to perform this would look like the following:

\<MonitoredElement Name="cpuLoad" Host="localhost" Type="sysUsage"\>

\<ConditionSet\>

\<fn Name="plug_ins" Arg=" uptime\|awk '{print \$(NF-2),

> \$(NF-1),\$NF}'\|awk -F',' '{print \$1,\$2,\$3}' "
>
> RetVal="array float:load"/\>

\<Condition State="UP" SevLevel="6" Description="Cpu load too high"\>

\<apply\>

\<geq/\>

\<apply\>

\<min\>

> \<bvar\>i\<bvar\>
>
> \<lowlimit\>\<cn\>0\<cn\>\</lowlimit\>
>
> \<uperlimit\>\<ci\>%len(load)\</ci\>\</uperlimit\>
>
> \<ci\>load\</ci\>

\</min\>

\</apply\>

\<cn\>12.0\</cn\>

\</apply\>

\<!---checking for condition : min(load\[i\]) \>=12.0, where i = 0,
len(load)\--!\>

\</Condition\>

\</ConditionSet\>

#### Example 2:

Let's assume that we want to monitor OS "Health" on an SGI node named
fnsfo. We want to check some components using the following Unix
commands:

Number of cpu off --line:

> mpadmin -n\|wc -l

Cpu load during last 15 min:

> uptime \| awk -F',' '{print \$NF}'

/dev/root file system size

> df /dev/root \| grep -v Filesystem\|awk '{print \$6}'

Inetd daemon presence

> ps -ef \| grep inetd \| grep -v grep\|wc --l

The agent's configuration file will look like the following:

\<?xml version="1.0"?\>

\<!DOCTYPE MA-conifg System "agent.dtd"\>

\<MA-config Name="SGI_Health" Update="180"\>

\<NCS Heartbeat="600" Port="19997" Host="ndem.fnal.gov"/\>

\<System Name="OSHealth" Cluster="localhost"\>

\<!---system id is "OSHealth.fnsfo" \--!\>

\<MonitoredElement Name="cpuStatus" Host="localhost" Type="Hardware"\>

\<!---monitored element id is "cpuStatus.fnsfo.OSHealth.fnsfo" \--!\>

\<ConditionSet\>

\<fn Name="plug_ins" Arg="mpadmin -n\|wc -l" RetVal="int:onlinecount"/\>

\<Condition State="Down" SevLevel="6" Description="At least one cpu is
off-line"\>

\<apply\>

\<eq/\>

\<ci\>onlinecount\</ci\>

\<cn\>4\</cn\>

\</apply\>

\<!---checking for condition : (onlinecount == 4) , where number of

> processors on fnsfo is equal to 4\--!\>

\</Condition\>

\</ConditionSet\>

\</MonitoredElement\>

\<MonitoredElement Name="cpuLoad" Host="localhost" Type="sysUsage"\>

\<!---monitored element id is "cpuLoad.fnsfo.OSHealth.fnsfo" \--!\>

\<ConditionSet\>

\<fn Name="plug_ins" Arg="uptime \| awk -F',' '{print \$NF}'"
RetVal="float:load"/\>

\<Condition State="UP" SevLevel="4" Description="Cpu load is between 8
and 15 during

> last 15 minutes"\>

\<apply\>

\<and/\>

\<apply\>

\<geq/\>

\<ci\>load\</ci\>

\<cn\>8.0\</cn\>

\</apply\>

\<apply\>

\<lt/\>

\<ci\>load\</ci\>

\<cn\>15.0\</cn\>

\</apply\>

\</apply\>

\<!---checking for condition : ((load\>=8)&& (load\<15)) \--!\>

\</Condition\>

\<Condition State="UP" SevLevel="6" Description="Cpu load is greater
than 15 during last 15 minutes"\>

\<apply\>

\<geq/\>

\<ci\>load\</ci\>

\<cn\>15.0\</cn\>

\</apply\>

\<!---checking for condition : (load\>=15) \--!\>

\</Condition\>

\</ConditionSet\>

\</MonitoredElement\>

\<MonitoredElement Name="/dev/root" Host="localhost" Type="FileSystem"\>

\<!---monitored element id is "/dev/root.fnsfo.OSHealth.fnsfo" \--!\>

\<ConditionSet\>

\<fn Name="plug_ins" Arg="df /dev/root \| grep -v Filesystem\|awk
'{print \$6}'"

RetVal="int:size"/\>

\<Condition State="UP" SevLevel="6" Description="file system is more
then 95% full"\>

\<apply\>

\<gt/\>

\<ci\>size\</ci\>

\<cn\>95\</cn\>

\</apply\>

\<!---checking for condition : (size\>95%) \--!\>

\</Condition\>

\</ConditionSet\>

\</MonitoredElement\>

\<MonitoredElement Name="inetd" Host="localhost" Type="Daemon"\>

\<!---monitored element id is "inetd.fnsfo.OSHealth.fnsfo" \--!\>

\<ConditionSet\>

\<fn Name="plug_ins" Arg="ps -ef \| grep inetd \| grep -v grep\|wc -l"

> RetVal="int:ifExist"/\>

\<Condition State="Down" SevLevel="6" Description="inetd daemons is not
running"\>

\<apply\>

\<neq/\>

\<ci\>ifExists\</ci\>

\<cn\>1\</cn\>

\</apply\>

\</Condition\>

\</ConditionSet\>

\</MonitoredElement\>

\</System\>

\</MA-config\>

######################################### 

### 9.2.1 Starting/Stopping Plugins Agent

You can start Plugins Agent in several ways. Multiple Agents
configuration files are placed in /var/ngop/plugins_agent directory and
could start (stop) simultaneously by issuing

ngop start/stop plugins_agent

#### In order to start individual agents, the following commands are used:

ngop start/stop "ngop plugins_agent --c
/ngop/var/plugins_agent/cfg_file"

or

ngop plugins_agent --c cfg_file

(you have to manually kill an agent if started it this way)

## 9.3 Swatch Agents

A Swatch Agent is an agent that watches a log file for lines matching a
regular expression, and takes some action when this occurs (similar to
swatch). Like the other agents in NGOP, an XML configuration file
controls the Swatch Agent's behavior. A XML document for each Swatch
Agent is placed in a separate file and should begins with the following
XML declarations:

\<?xml version="1.0"?\>

\<!DOCTYPE SwatchAgentConfig SYSTEM "swatchagent.dtd" \>

The outermost tag of the file is \<SwatchAgentConfig\> , which includes
the required name attribute. name specifies the name of the monitoring
agent.

The second outermost tag of the file is \<NCS\>, which includes the
following attributes:

Heartbeat - specifies the hearbeat interval in seconds

Host - specifies the host name of the NCS to send events

Port - specifies the port number on the above host

The third outermost tag of the file is \<File\>, which includes the
following attributes:

file - This lists the file the agent should watch for messages

filetype -- The valid values for filetype are:

"multihost" -- This indicates that a hostname match should be prepended
to regular expressions when expanding HostType lists.

*"*plain*"* -- This indicates that all regular expressions are to be
used verbatim

A system description should follow the\<File\> tag. Several systems are
described in the same XML document. The \<System\> tag indicates the
beginning of the system definition. It contains multiple monitored
elements.

Once we are in the context of a given \<MonitoredElement\>, we can
specify rules about log file lines, which will trigger events about that
monitored element, with an **\<**ReRule**\>** tag. \<ReRule\> tag has
the following required attributes:

Regexp -- defines a regular expression

*State*

SevLevel

EventName

EventValue

and one optional attribute:

ActionID -- defines action that should be executed when pattern is
matched

An \<Action\> tag that should be within a \<SwatchAgentConfig\> tag
describes an action.

In the Swatch Agent configuration instances of "localhost" will be
replaced by the local host name. This configuration file should conform
to [the DTD rules](#swatch-agent-dtd).

***Example***:

Let's assume that we want to monitor a syslogd log file on a Linux
machine. We want to watch for the following patterns:

'kernel: nfs: server.\*not responding'

'ypbind.\*failed'

'shutdown succeeded'

'startup succeeded'

'kernel:.\*irq

'kernel:.\*reset: success'

'kernel:.\*status timeout:'

'kernel:.\*drive not ready for command'

\<?xml version='1.0'?\>

\<!DOCTYPE SwatchAgentConfig SYSTEM "swatchagent.dtd" \>

\<SwatchAgentConfig name="SwatchAgent"\>

\<NCS Heartbeat="600" Host='ndem.fnal.gov' Port='19997'\>

\<File file='/var/log/messages' filetype='plain'\>

\<System ID='OSHealth' Cluster='localhost'\>

\<MonitoredElement Name='syslogd' Type='Daemon' Host='localhost'\>

\<ReRule Regexp='kernel: nfs: server.\*not responding' EventName='nfs'

> State='UP' SevLevel='6'/\>

\<ReRule Regexp='ypbind.\*failed' EventName='ypbind' State='UP'
SevLevel='4'/\>

\<ReRule Regexp='shutdown succeeded' State='UP' SevLevel ='5'/\>

\<ReRule Regexp='startup succeeded' State='UP' SevLevel= '0'/\>

\<ReRule Regexp='kernel:.\*irq timeout' State='UP' SevLevel= '6'/\>

\<ReRule Regexp='kernel:.\*reset: success' State='UP' SevLevel= '6'/\>

\<ReRule Regexp='kernel:.\*status timeout:' State='UP' SevLevel= '6'/\>

\<ReRule Regexp='kernel:.\*drive not ready for command' State='1'

> SevLevel= '6'/\>

\<ReRule Regexp='kernel:.\*Unable to load interpreter
/lib/ld-linux.so.2'

> State='1' SevLevel= '6'/\>

\</MonitoredElement\>

\</System\>

\</File\>

\</SwatchAgentConfig\>

### 9.3.1 Starting/Stopping Swatch Agent

You can start Swatch Agent in several ways. Multiple Agents
configuration files are placed in /var/ngop/swatch_agent directory and
could start (stop) simultaneously by issuing

ngop start/stop swatch_agent

#### In order to start individual agents , the following commands are used:

ngop start/stop "ngop swatch_agent --c /ngop/var/swatch_agent/cfg_file"

or

ngop swatch_agent --c cfg_file

The agent must be killed manually if started by the latter.

## 9.4 Monitoring Agent API

Users can write their own monitoring agents using the supplied
monitoring agent API that comes with the NGOP product. This chapter
discussed the monitoring agent API and gives examples.

### 9.4.1: API Description

NGOP Monitoring Agent API provides way for users to write their own
Monitoring Agent that will communicate with NGOP Central Server.

MAClient Class performs all the communication between Monitoring Agent
and NGOP Central Server. User has to create the object of this class.
The MAClient Class provides the following methods:

-   Setting MA attributes

-   Describing configuration

-   Registering with NGOP Central Server

-   Sending Events to NGOP Central server

-   Performing Action

-   Sending request to NCS to perform Action

Only a Python binding API is provided in prototype version.

#### 9.4.1.1: MAClient Class 

In order to use the MA API, user applications should import the MAClient
class from the MA_API module:

from MA_API import MAClient

#### 

#### 9.4.1.2: MAClient methods

This section describes the methods available for the MAClient class.

######################################### 

######################################### 9.4.1.2.1 MAClient()

The constructor MAClient() creates an MAClient object and establishes
communication with the NGOP Central Server.

Synopsis: MAClient()

Arguments: None

Return Value: MAClient object.

######################################### 9.4.1.2.2 setMAAttrib()

This method sets the monitoring agent attributes such as name, heartbeat
rate, central server host and port.

Synopsis: setMAAttrib(maName,heartbeat,ncsHost,ncsPort,type)

Arguments:

maName: String; Monitoring Agent name

heartbeat: String; interval in seconds to sent a heartbeat message to
the NCS.

ncsHost: String; NCS host

ncsPort: String; NCS port

type: Sring,MA type (Cron or Daemon)

Return Value: None.

######################################### 9.4.1.2.3 addSystem()

This method adds system information to the list of monitored objects.

Synopsis: addSystem(sysName, clusterName)

Arguments:

sysName: String; name of the system.

clusterName: String; name of the cluster.

Return Value: None.

######################################### 9.4.1.2.4 addME()

This method adds monitored elements to the system configuration.

Synopsis: addME(sysName, clusterName, meName, meType, host)

Arguments:

sysName: String; name of the system.

clusterName: String; name of the cluster.

meName: String: monitored element name.

meType: String: monitored element type.

host: String: host name where the monitored element is located.

Return Value: None.

######################################### 9.4.1.2.5 register()

This method sends the initial configuratio to the NGOP Central Server.

Synopsis: register()

Arguments: None

Return Value: None

######################################### 9.4.2.6 send_event()

This method sends an event message to the NGOP Central Server.

Synopsis: send_event(eventDict,sysName,clusterName,meName,meHost)

Arguments:

eventDict: Dictionary: Describes the event with the following dictionary
keys:

EventType -- String

EventName -- String

EventValue -- String

State -- Integer (-1,0,1)

1 - undefined

0 - up

1 - down

SevLevel -- Integer (0-6).

0 -- None

1 - NotInService

2 - Unknown

3 - Undefined

4 - Warning

5 - Error

6 - Alert

sysName: String: name of the system

clusterName: String: name of the cluster

> meName: String: name of the monitored element

meHost: String: name of the host where the monitored element is located.

meName,meHost are set to None if event is related to

system state

Return Value: 2-type (status,reason)

status: Integer

0 -- failure

1 -- success

reason: String; Reason for failure or NULL.

######################################### 

######################################### 9.4.1.2.6 do_action()

This method sends an event message to the NGOP Central Server.

Synopsis:
do_action(sysName,clusterName,meName,meHost,eventDict,actionDict)

Arguments:

eventDict: Dictionary: Describes the event with the following dictionary
keys:

EventType -- String

EventName -- String

EventValue -- String

State -- Integer (-1,0,1)

1 - undefined

0 - up

1 - down

SevLevel -- Integer (0-6).

0 -- None

1 - NotInService

2 - Unknown

3 - Undefined

4 - Warning

5 - Error

6 - Alert

actionDict: Dictionary: Describes the action with the following keys:
ActionID -- String: The action id.

ExecName -- String: Name of the command to be executed.

ArgList -- String: The argument list to ExecName.

ActionType -- String: Either "local" or "central".

sysName: String: name of the system

clusterName: String: name of the cluster

meName: String: name of the monitored element.

meHost: String: name of the host where the monitored element is located.

meName,meHost are set to None if event is related to

system state

Return Value: None.

######################################### 9.4.1.2.7 stop()

This method notify NCS that is ended normaly .

Synopsis: stop()

Arguments:

None

Return Value: None

### 9.4.2: MA API Example

This section details a monitoring agent written using the API. In this
example, let's assume that we want to monitor the system "mySystem" on
the cluster "myCluster". Let's say the cluster consists of 100 nodes
named myWorker1, myWorker2, \..., myWorker100. A monitored element
called myDaemon is running on each the node in the cluster. When
myDaemon dies or restarts we would like to send an event message to the
NGOP Central Server.

Here is the code to perform this task:

import MA_API\
import time\
import sys\
DOWN=0\
UP=1\
UNKNOWN=-1\
def isDaemonAlive(self,name,node):\
   #user provides way to verify that the daemon is alive on the node\
  \...\...\
   return state,description\
   #state could be Down, Up, Unknown\
   #description should not have blanks\
if \_\_name\_\_=="\_\_main\_\_":\
    checkTime=myCheck\
    \# monitoring interval\
    maName="myAgent" i\
    #name of the monitoring agent\
    sysName="mySystem"\
    #system name\
    clusterName="myCluster"\
    #cluster name\
    nodeName="myWorker"\
    #common node name\
    minIdx=1\
    #node number starts with this index\
    maxIdx=100\
    #node number ends with this index\
    meName="myDaemon"\
    #name of monitored element\
    meType="Daemon"\
    #type of monitored element\
    heartbeat="300"\
    #heartbeat rate in sec\
    serverHost='ngop'\
    serverPort="19997"\
    #NGOP Central Server host and port\
\
    cl=MA_API.MAClient()\
    #creates MAClient object\
\
    cl.setMAAttrib(maName,heartbeat,serverHost,serverPort)\
    #sets MA attributes\
\
    cl.addSystem(sysName,clusterName)\
    #configures the system\
    oldStateList=\[\]\
    #hold previous state of the monitred element\
    for i in range(minIdx,maxIdx):\
        cl.addME(sysName,clusterName,meName,meType,nodeName+repr(i))\
        #configures system monitored elements list\
        oldStateList.append(UP)\
        \# sets all state to UP\
\
    cl.register()\
    #registers monitoring agent with NGOP Central Server\
\
    while 1:\
\
        for i in range(minIdx,maxIdx):\
\
            state,description=isDaemonAlive(meName, nodeName+repr(i))\
            if oldStateList\[i\]==state:\
              continue #nothing has changed\
            eventDict={'EventType':meType, 'EventName':meName, \\\
              'State':state,'SevLevel':0}\
            eventDict\['Description'\]=description\
           

status,reason=cl.sendEvent(eventDict,sysName,clusterName,

> meName,nodeName)

            #sends event to NGOP CS

if not status: print "Error:",reason

> oldStateList\[i\]=state\
> \
>      time.sleep(checkTime)

### 9.4.3 Starting/Stopping Your Agent

#### You can start your Agent issuing the following command:

ngop your_python_code.py &

# 

# Chapter 10: Action Client

An Action Client has the following features:

-   It gets configuration information from the CFMS

```{=html}
<!-- -->
```
-   It gets action requests from the NCS

```{=html}
<!-- -->
```
-   It verifies user authorization to request the actions

```{=html}
<!-- -->
```
-   It performs actions

```{=html}
<!-- -->
```
-   It notifies the NCS about success/failure of a performed actions

There are several configuration files that contain general information
needed for the action clients. These files will be downloaded into a
designated configuration area during the NGOP action client startup.

## 10.1 Action Client Configuration File

Action Client configuration file contains the following information:

\<AC_cfg\>

\<Client port=\"19986\" host=\"localhost\" name=\"NCSClnt\"/\>

\<Client port=\"8080\" host=\"localhost\" name=\"CnfgClnt\"/\>

\<CfgXml cvsRep=\"configxml\" wrkDir=\".ngop_action\" excDir=\"scripts\"
cvsRoot=\":pserver:ngop@ngop.fnal.gov:/home/ngop/Repository\"
role=\"farmadmin\"/\>

\</AC_cfg\>

The \<AC_cfg\> tag client tag defines parameters that are required to
start Action Client. Action Client should be connected to NCS, so first
\<Client\> tag is required. The second \<Client\> tag is optional and is
needed if Action Client connects to CFMS. The \<Client\> tag has the
following attributes: service tcp port, host name of the node where
service is running and service name (NCSClnt/CnfgClnt). The CfgXml tag
is required. It defines the parameters that will be used to create local
configuration and connect to CVS repository. CfgXml tag includes the
following attributes: name of the root dirctory (required), name of ngop
configuration cvs repository (requred), CVSROOT definition (optional),
tag of current configuration role.

This configuration file should conform to [the DTD
rules](#_Action_Client_Configuration).

## 10.2 Starting/Stopping Action Client

An administrator can start Action Client in several ways. Action Client
configuration files are placed in /var/ngop/action directory and could
start (stop) simultaneously by issuing

ngop start/stop action

or

ngop action --c cfg_file &

If the agent is started by the latter command, the agent can only be
killed manually.

## 10.3 File authorized.xml

The authorized.xml configuration file contains information about the
users who are authorized to perform certain actions via an action
client. Each user belongs to an authorization group.

If a user has requested an action but is not listed in the
authorized.xml configuration file, the request will be denied. The
authorized.xml file requires the following declaration and tags:

\<?xml version='1.0'?\>

\<!DOCTYPE NGOPAction SYSTEM "action.dtd"\>

\<NGOPAction\>

\<Authorization_File\>

\<AuthorizedGroup ID=\"ngop_admin\"\>

\<User Name=\"user_name\"/\>

\...

\</AuthorizedGroup\>

\</Authorization_File\>

\</NGOPAction\>

An \<Authorization_File\> tag contains zero or more \<AuthorizedGroup\>
tags. These tags have a required attribute of ID and contain zero or
more user names (\<User\>). This configuration file should conform to
[the DTD rules](#autorization_file-dtd).

######################################### **Example**: 

Two groups (ngop_admin and oss_admin) are described in this example. A
list of authorized users is attached to each group.

\<AuthorizationFile\>

\<AuthorizedGroup ID="ngop_admin"\>

\<User Name="smith"/\>

\<User Name="jones"/\>

\</AuthorizedGroup\>

\<AuthorizedGroup ID="oss_admin"\>

\<User Name="brown"/\>

\<User Name="johnson"/\>

\</AuthorizedGroup\>

\</AuthorizationFile\>

## 10.4 File action.xml

The action.xml configuration file describes actions which consist of
executables or scripts, a host where they are located, and the groups
that are authorized to perform this action. The action.xml file requires
the following declaration and tags:

\<?xml version='1.0'?\>

\<!DOCTYPE NGOPAction System "action.dtd"\>

\<NGOPAction\>

\<Action_File\>

\<NGOPAction\>

\<Action_File\>

\<Action ID=\"action_name\"\>

\<Host Name=\"host_name\"\>

\<AuthorizedGroup ID=\"group_name\"/\>

\...

\<Exec Path=\"command_name\"/\>

\...

\</Host\>

\</Action\>

\</Action_File\>

An \<Action\> tag has one required attribute (ID) and contains several
\<Host\> tags. A \<Host\> tag has a Name attribute and contains one or
more \<AuthorizedGroup\> tags (with an ID attribute) and \<Exec\> tags
(with a Path attribute). This configuration file should conform to [the
DTD rules](#action_file-dtd).

######################################### **Example**:

Two actions are defined in this example. The first action allows
operator and oss_admin groups to send email via an action client running
on the host ndem..

\<ActionFile\>

\<ActionID="email"\>

\<Host Name="ndem"\>

> \<AuthorizedGroup ID="operator"/\>
>
> \<AuthorizedGroup ID="oss_admin"/\>
>
> \<Exec Path="scripts/email"\>

\</Host\>

\</Action\>

\</ActionFile\>

# Chapter 11: Controlling the NGOP Daemons

The NGOP package requires multiple processes to be running on multiple
systems:

-   The NCS, Broker, and Action Client on a central service machine.

-   Ping agents on some machines which watch over other machines.

-   Monitoring agents local to various systems.

To facilitate this, NGOP provides a simple mechanism for an
administrator to write down what NGOP processes should be running on a
given system, and to start them, stop them, and make sure that they are
still running. This mechanism is also integrated with the UPS packaging
system which has an umbrella mechanism to start processes needed for
various UPS products at system startup.

## 11.1: The /var/ngop Directory. 

The start/stop mechanism by default uses a directory tree under
/var/ngop on each system to record what processes should be running on
that system. The location of this directory can be changed by setting
the environment variable NGOP_START_DIR.

As an example, suppose that you wanted to have two swatch_agent
processes each running a different configuration file, and one
plugins_agent process:

ngop swatch_agent --c cfg1.xml

ngop swatch_agent --c cfg2.xml

ngop plugins_agent --c cfg3.xml

To configure this you would place the configuration files under
/var/ngop as follows:

/var/ngop/swatch_agent/cfg1.xml

/var/ngop/swatch_agent/cfg2.xml

/var/ngop/plugins_agent/cfg3.xml

The directory tree should have r+w permissions for the uid who will be
running the NGOP processes.

## 11.2 Starting the Agents

Once the directory structure has been setup under /var/ngop, the agents
are started by doing one of the following:

ups start ngop

or

setup ngop

ngop start

When the command is issued, an informational message will be displayed
to the screen for each process started. The start/stop mechanism also
records which agents have been started with their process ID numbers in
/var/ngop/.pids.\<hostname\>.

## 11.3 Monitoring the Agents

The start/stop mechanism provides a means to monitor the agents:

ups status ngop

or

setup ngop

ngop status

Below is an example of the output produced by the status command:

\$ ngop status

Running:

PID COMMAND

9707 ngop swatch_agent --c /var/ngop/swatch_agent/cfg1.xml

9710 ngop swatch_agent --c /var/ngop/swatch_agent/cfg2.xml

9713 ngop plugins_agent --c /var/ngop/plugins_agent/cfg3.xml

If one of the agents has died, it will still show in the listing:

\$ ngop status

Running:

PID COMMAND

9707 ngop swatch_agent --c /var/ngop/swatch_agent/cfg1.xml

died ngop swatch_agent --c /var/ngop/swatch_agent/cfg2.xml

9713 ngop plugins_agent --c /var/ngop/plugins_agent/cfg3.xml

There is also a check command which will obtain the status of the agents
and restart those that have died:

\$ ngop check

Running:

PID COMMAND

9707 ngop swatch_agent --c /var/ngop/swatch_agent/cfg1.xml still running

9710 ngop swatch_agent --c /var/ngop/swatch_agent/cfg2.xml died,
restarting \...

9713 ngop plugins_agent --c /var/ngop/plugins_agent/cfg3.xml still
running

## 11.4 Stopping the Agents

Once the directory structure under /var/ngop has been setup and the
agents have been started as in the previous section, stopping the agents
is accomplished with one of the following commands:

ups stop ngop

or

setup ngop

ngop stop

The stopping mechanism looks under /var/ngop/.pids.\<hostname\> for
agents that have been started. Informational messages are displayed as
each agent is stopped:

\$ngop stop

Stopping: ngop swatch_agent --c /var/ngop/swatch_agent/cfg1.xml

Stopping: ngop swatch_agent --c /var/ngop/swatch_agent/cfg2.xml

Stopping: ngop plugins_agent --c /var/ngop/plugins_agent/cfg2.xml

## 11.5 Disabling/Enabling Agents

It is sometimes desirable to disable an agent, but not to discard it's
configuration. This is accomplished with the ngop disable command. The
agent is enabled with ngop enable. The argument to these commands is
either the full NGOP command (as listed by ngop status) in quotes, or
the base name of the configuration file (cfg2 for example):

\$ ngop status

Running:

PID COMMAND

9707 ngop swatch_agent --c /var/ngop/swatch_agent/cfg1.xml

9710 ngop swatch_agent --c /var/ngop/swatch_agent/cfg2.xml

9713 ngop plugins_agent --c /var/ngop/plugins_agent/cfg3.xml

\$ ngop disable "cfg2"

Stopping: ngop swatch_agent --c /var/ngop/swatch_agent/cfg2.xml

\$ngop stop

Stopping: ngop swatch_agent --c /var/ngop/swatch_agent/cfg1.xml

Stopping: ngop plugins_agent --c /var/ngop/plugins_agent/cfg2.xml

\$ngop start

Starting: ngop swatch_agent --c /var/ngop/swatch_agent/cfg1.xml

Disabled: ngop swatch_agent --c /var/ngop/swatch_agent/cfg2.xml

Starting: ngop plugins_agent --c /var/ngop/swatch_agent/cfg3.xml

\$ngop enable "cfg2"

Starting: ngop swatch_agent --c /var/ngop/swatch_agent/cfg2.xml

## 11.6 Controlling Agents on Remote Hosts

NGOP agents often run on multiple hosts. The ngop remote command is used
to stop, start, or modify the behaviour of NGOP agents on remote hosts
provided that the user issuing the command has permission to rsh to
those hosts.

The format of the remote command is:

ngop remote \[-l user\] \<host\>\|\<cluster\> \|

ngop remote \[-l user\] \<host_prefix\>:\<start_range\>-\<end_range\>

For example, to start nodes fcdf09, fcdf10, fcdf11, fcdf12, and fcdf13,
the following command is used:

ngop remote fcdf:09-13

If the above command should be executed as the user ngopuser, the
following command would be used:

ngop remote --l ngopuser fcdf:09-13

If the "tictac" tools for the farms are being used, the tictac cluster
name can be used:

ngop remote --l ngop start --c fcdf_cluster

## 11.7 Starting/Stopping Individual Agents

The start and stop commands can be given a string to match that will
pick servers to start or stop. For example, to stop a swatch_agent using
the configuration file /var/ngop/swatch_agent/ cfg1.xml:

\$ngop stop "cfg1"

Stopping: ngop swatch_agent --c /var/ngop/swatch_agent/cfg1.xml

\$ngop status

Running:

PID COMMAND

died ngop swatch_agent --c /var/ngop/swatch_agent/cfg1.xml

9708 ngop swatch_agent --c /var/ngop/swatch_agent/cfg2.xml

9713 ngop plugins_agent --c /var/ngop/plugins_agent/cfg3.xml

With multiple hosts, this can result in a more complicated string. For
example, to kill the deamon in the above example on hosts fcdf09-13, you
would issue the following command:

ngop remote 'stop cfg1' fcdf09-13

# Appendix A

## \<For\> DTD

*\<!ELEMENT For ( For\* \| MonitoredElement+ ) \>*

*\<!ATTLIST For*

*Each CDATA #REQUIRED*

*In CDATA #REQUIRED*

*Name CDATA #REQUIRED*

*Var CDATA #REQUIRED*

*File CDATA #IMPLIED*

*\>*

## \<Apply\> DTD

*\<!ELEMENT apply ( ( sum \| min \| max \| divide \| times \| plus \|
minus \| and \| or \| eq \| neq \| gt \| geq \| lt \| leq \| in \| notin
), ( apply \| cn \| ci )\* ) \>*

*\<!ELEMENT sum ( bvar, uplimit, lowlimit, ( apply \| ci \| cn )\* ) \>*

*\<!ELEMENT min ( bvar, uplimit, lowlimit, ( apply \| ci \| cn )\* ) \>*

*\<!ELEMENT max ( bvar, uplimit, lowlimit, ( apply \| ci \| cn )\* ) \>*

*\<!ELEMENT bvar EMPTY \>*

*\<!ELEMENT uplimit ( apply \| cn ) \>*

*\<!ELEMENT lowlimit ( apply \| cn ) \>*

*\<!ELEMENT divide EMPTY \>*

*\<!ELEMENT times EMPTY \>*

*\<!ELEMENT plus EMPTY \>*

*\<!ELEMENT minus EMPTY \>*

*\<!ELEMENT and EMPTY \>*

*\<!ELEMENT or EMPTY \>*

*\<!ELEMENT eq EMPTY \>*

*\<!ELEMENT neq EMPTY \>*

*\<!ELEMENT gt EMPTY \>*

*\<!ELEMENT geq EMPTY \>*

*\<!ELEMENT lt EMPTY \>*

*\<!ELEMENT leq EMPTY \>*

*\<!ELEMENT in EMPTY \>*

*\<!ELEMENT notin EMPTY \>*

*\<!ELEMENT ci ( #PCDATA ) \>*

*\<!ELEMENT cn ( #PCDATA ) \>*

## \<Action\> DTD

*\<!ELEMENT Action ( Exec ) \>*

*\<!ATTLIST Action*

*ID CDATA #REQUIRED*

*Host CDATA #REQUIRED*

*Method ( manual \| automatic ) \'automatic\'*

*Type ( local \| central ) \'central\'*

*Repeat CDATA #IMPLIED*

*Gap CDATA #IMPLIED*

*\>*

*\<!ELEMENT Exec EMPTY \>*

*\<!ATTLIST Exec*

*Name CDATA #REQUIRED*

*Argument CDATA #REQUIRED*

*\>*

## \<If\> DTD

*\<!ELEMENT If ( PCDATA\|Else?) \>*

*\<!ATTLIST If*

*Cond CDATA #REQUIRED "'{%Role}'==(!=)'role_name'"*

*\>*

*\<!ELEMENT Else (PCDATA)\>*

## NCS Configuration File DTD

*\<!ELEMENT NCS_cfg ( NCS, Client?, Admin ) \>*

*\<!ELEMENT NCS EMPTY \>*

*\<!ATTLIST NCS*

*tcpPort CDATA #REQUIRED*

*udpPort CDATA #REQUIRED*

*\>*

*\<!ELEMENT Client EMPTY \>*

*\<!ATTLIST Client*

*name (Archiver) #IMPLIED*

*port CDATA #REQUIRED*

*local_log CDATA \'log.log\'*

*host CDATA \'localhost\'*

*\>*

*\<!ELEMENT Admin EMPTY \>*

*\<!ATTLIST Admin*

*actnHost CDATA \'localhost\'*

*adminMail CDATA \'%User@fnal.gov\'*

*adminName CDATA \'%User\'*

*\>*

## Monitor Configuration File DTD

*\<!ELEMENT client_cfg ( Client\*, CfgXml, CfgGui ) \>*

*\<!ELEMENT Client EMPTY \>*

*\<!ATTLIST Client*

*port CDATA #REQUIRED*

*host CDATA #REQUIRED*

*name ( NCSClnt \| CnfgClnt ) \'NCSClnt\'*

*\>*

*\<!ELEMENT CfgXml EMPTY \>*

*\<!ATTLIST CfgXml*

*wrkDir CDATA #REQUIRED*

*cvsRep CDATA #REQUIRED*

*cvsRoot CDATA #IMPLIED*

*role CDATA \'default\'*

*\>*

*\<!ELEMENT CfgGui EMPTY \>*

*\<!ATTLIST CfgGui*

*LogLength CDATA \'10\'*

*Mail CDATA #IMPLIED*

*Update ("0"\|"1") "1"*

*\>*

## PlugIns Agent DTD

*\<!ELEMENT MA-config ( NCS, ( For \| System \| ItemList )\* ) \>*

*\<!ATTLIST MA-config*

*Name CDATA #REQUIRED*

*Update CDATA #REQUIRED*

*\>*

*\<!ELEMENT NCS EMPTY \>*

*\<!ATTLIST NCS*

*Heartbeat CDATA #REQUIRED*

*Host CDATA #REQUIRED*

*Port CDATA #REQUIRED*

*\>*

*\<!ELEMENT System ( ConditionSet \| For \| MonitoredElement )\* \>*

*\<!ATTLIST System*

*Name CDATA #REQUIRED*

*Cluster CDATA \'localhost\'*

*\>*

*\<!ELEMENT MonitoredElement ( ConditionSet )\* \>*

*\<!ATTLIST MonitoredElement*

*Name CDATA #REQUIRED*

*Type CDATA \'Element\'*

*Host CDATA \'localhost\'*

*\>*

*\<!ELEMENT ConditionSet ( fn, Condition\* ) \>*

*\<!ELEMENT fn EMPTY \>*

*\<!ATTLIST fn*

*Name ( plug_ins ) #REQUIRED*

*Arg CDATA #REQUIRED*

*RetVal CDATA #REQUIRED*

*\>*

*\<!ELEMENT Condition ( apply\*, Action? ) \>*

*\<!ATTLIST Condition*

*Description CDATA #REQUIRED*

*SevLevel CDATA #REQUIRED*

*State CDATA #REQUIRED*

*EventType CDATA #IMPLIED*

*EventName CDATA #IMPLIED*

*\>*

*\<!ELEMENT Item EMPTY \>*

*\<!ATTLIST Item*

*Name CDATA #REQUIRED*

*\>*

*\<!ELEMENT ItemList ( Item )+ \>*

*\<!ATTLIST ItemList*

*Name CDATA #REQUIRED*

*\>*

*\<!\-- see for dtd \--\>*

*\<!\-- see apply dtd \--\>*

*\<!\-- see action dtd \--\>*

## Swatch Agent DTD

*\<!ELEMENT SwatchAgentConfig ( File )\* \>*

*\<!ATTLIST SwatchAgentConfig*

*name CDATA #REQUIRED*

*\>*

*\<!ELEMENT NCS EMPTY \>*

*\<!ATTLIST NCS*

*Port CDATA #REQUIRED*

*Host CDATA #REQUIRED*

*Heartbeat CDATA #REQUIRED*

*\>*

*\<!ELEMENT File ( For \| ( System, Action? )\* )\* ) \>*

*\<!ATTLIST File*

*file CDATA #REQUIRED*

*filetype ( plain \| multihost ) #REQUIRED*

*\>*

*\<!ELEMENT System ( For \| MonitoredElement )\* \>*

*\<!ATTLIST System*

*Name CDATA #REQUIRED*

*Cluster CDATA #REQUIRED*

*\>*

*\<!ELEMENT MonitoredElement ( ReRule )\* \>*

*\<!ATTLIST MonitoredElement*

*Name CDATA #REQUIRED*

*Type CDATA #REQUIRED*

*Host CDATA #REQUIRED*

*\>*

*\<!ELEMENT ReRule EMPTY \>*

*\<!ATTLIST ReRule*

*Regexp CDATA #REQUIRED*

*SevLevel CDATA #REQUIRED*

*State CDATA #REQUIRED*

*EventType CDATA #IMPLIED*

*EventName CDATA #IMPLIED*

*EventValue CDATA #IMPLIED*

*ActionID CDATA #IMPLIED*

*\>*

*\<!\--see for dtd \--\>*

*\<!\--see apply dtd \--\>*

*\<!\--see action dtd \--\>*

## \<ServiceClass\> DTD

*\<!ELEMENT NGOPConfig ( Default_File, ServiceClass ) \>*

*\<!ELEMENT Default_File EMPTY \>*

*\<!ELEMENT ServiceClass ( ServiceType )\* \>*

*\<!ELEMENT ServiceType ( apply )\* \>*

*\<!ATTLIST ServiceType*

*Name CDATA #REQUIRED*

*\>*

*\<!-see apply dtd \--!\>*

## \<HostsInClusters\> DTD

*\<!ELEMENT NGOPConfig (Default_File, HostByCluster ) \>*

*\<!ELEMENT Default_File EMPTY \>*

*\<!ELEMENT HostByCluster ( ServiceType \| Cluster )\* \>*

*\<!ELEMENT ServiceType ( Cluster \| Host )\* \>*

*\<!ATTLIST ServiceType*

*Name CDATA #REQUIRED*

*\>*

*\<!ELEMENT Cluster ( Host )\* \>*

*\<!ATTLIST Cluster*

*Name CDATA #REQUIRED*

\>

*\<!ELEMENT Host EMPTY \>*

*\<!ATTLIST Host*

*Name CDATA #REQUIRED*

\>

## \<KnownStatus\> DTD

*\<!ELEMENT NGOPConfig ( Default_File, KnownStatus ) \>*

*\<!ELEMENT Default_File EMPTY \>*

*\<!ELEMENT KnownStatus ( Status )\* \>*

*\<!ELEMENT Status ( OutOfServiceInterval \| System \| Host )\* \>*

*\<!ATTLIST Status*

*Name ( bad \| test \| in_repair ) #REQUIRED*

*\>*

*\<!ELEMENT OutOfServiceInterval ( MaintenancePeriod )\* \>*

*\<!ATTLIST OutOfServiceInterval*

*StartDateTime ( CDATA \| None ) \"None\"*

*EndDateTime ( CDATA \| None ) \"None\"*

*\>*

*\<!ELEMENT MaintenancePeriod EMPTY \>*

*\<!ATTLIST MaintenancePeriod*

*DaysInterval CDATA #REQUIRED*

*FromTime CDATA #REQUIRED*

*HoursDuration CDATA #REQUIRED*

*\>*

## Monitored Hierarchy DTD

*\<!ELEMENT NGOPConfig (For \| SystemView)\* \>*

*\<!ELEMENT SystemView ( For \| SystemView \| System )\* \>*

*\<!ATTLIST SystemView*

*ID CDATA #REQUIRED*

*RefRule ( CDATA \| SystemViewDefRuleSet ) \"SystemViewDefRuleSet\"*

*\>*

*\<!ELEMENT System ( For \| MonitoredElement )\* \>*

*\<!ATTLIST System*

*Name CDATA #REQUIRED*

*ServiceType ( CDATA \| 9x5 \| 24x7 ) \"24x7\"*

*\<!\-- new type should be first introduced in ServiceClass definition
before it could be used here \--\>*

*RefRule ( CDATA \| SystemDefRuleSet ) \"SystemDeftRuleSet\"*

*\>*

*\<!ELEMENT MonitoredElement EMPTY \>*

*\<!ATTLIST MonitoredElement*

*Name CDATA #REQUIRED*

*Type CDATA #REQUIRED*

*RefRule ( CDATA \| MEDeftRuleSet ) \"MEDefRuleSet\"*

*ServiceType ( CDATA \| 9x5 \| 24x7 ) \"24x7\"*

\>

*\<!\--see for dtd \--\>*

## \<StatusRulesSet\> DTD

*\<!ELEMENT NGOPRule ( StatusRulesSet )\* \>*

*\<!ELEMENT StatusRulesSet ( DependList?, ( GenricRule \| DependRule )\*
) \>*

*\<!ATTLIST StatusRulesSet*

*ID CDATA #REQUIRED*

*\>*

*\<!ELEMENT DependList (Group )\* \>*

*\<!ELEMENT Group ( For \| System )\* \>*

*\<!ATTLIST Name*

*Name CDATA #REQUIRED*

*\>*

*\<!\-- see system dtd in hierarchy \--\>*

*\<!ELEMENT GenericRule ( apply, Action ) \>*

*\<!ATTLIST GenericRule*

*Prio CDATA #REQUIRED*

*Status ( None\|Good\|Unknown\|Undefined\|Warning\|Error\|Bad )
#REQUIRED*

*SevLevel ( None\|Good\|Unknown\|Undefined\|Warning\|Error\|Bad )
\'Good\'*

*Dsc CDATA #IMPLIED*

*\>*

#### 

*\<!ELEMENT DependRule ( apply, Action ) \>*

*\<!ATTLIST DependRule*

*Prio CDATA #REQUIRED*

*Status (None\|Good\|Unknown\|Undefined\|Warning\|Error\|Bad) #REQUIRED*

*SevLevel (None\|Good\|Unknown\|Undefined\|Warning\|Error\|Bad)
\'Good\'*

*Dsc CDATA #IMPLIED*

*\>*

*\<!\--see for dtd \--\>*

*\<!\--see apply dtd \--\>*

*\<!\--see action dtd \--\>*

## CFMS Configuration File DTD

*\<!ELEMENT client_cfg ( Client \| CfgXml ) \>*

*\<!ELEMENT Client EMPTY \>*

*\<!ATTLIST Client*

*host CDATA #REQUIRED*

*port CDATA #REQUIRED*

*name CDATA #REQUIRED*

*\>*

*\<!ELEMENT CfgXml EMPTY \>*

*\<!ATTLIST CfgXml*

*cvsRoot CDATA #REQUIRED*

*name CDATA #REQUIRED*

*version CDATA #REQUIRED*

*cvsRep CDATA #REQUIRED*

*\>*

## Archiver Configuration File DTD

*\<!ELEMENT ArchiverConfig ( Port, ArchiverHost, LogPath,
RequestDirectory, ErrorDirectory, DBInterSleepInterval ) \>*

*\<!ELEMENT Port EMPTY \>*

*\<!ELEMENT ArchiverHost ( OraPW, OraUser, OraInstance ) #REQUIRED \>*

*\<!ELEMENT LogPath EMPTY \>*

*\<!ELEMENT RequestDirectory EMPTY \>*

*\<!ELEMENT ErrorDirectory EMPTY \>*

*\<!ELEMENT DBInterSleepInterval EMPTY \>*

*\<!ELEMENT OraPW EMPTY \>*

*\<!ELEMENT OraUser EMPTY \>*

*\<!ELEMENT OraInstance EMPTY \>*

## Action Client Configuration File DTD

*\<!ELEMENT AC_cfg ( Client+, CfgXml ) \>*

*\<!ELEMENT Client EMPTY \>*

*\<!ATTLIST Client*

*port CDATA #REQUIRED*

*name ( NCSClnt \| CnfgClnt ) \'NCSClnt\'*

*host CDATA \'localhost*\'

*\>*

*\<!ELEMENT CfgXml EMPTY \>*

*\<!ATTLIST CfgXml*

*excDir CDATA #REQUIRED*

*wrkDir CDATA #REQUIRED*

*cvsRep CDATA #IMPLIED*

*cvsRoot CDATA #IMPLIED*

*role CDATA \'default\'*

*\>*

## \<Autorization_File\> DTD

*\<!ELEMENT NGOPAction ( Authorization_File ) \>*

*\<!ELEMENT Authorization_File ( AuthorizedGroup )\* \>*

*\<!ELEMENT AuthorizedGroup ( User )\* \>*

*\<!ATTLIST AuthorizedGroup*

*ID CDATA #REQUIRED*

*\>*

*\<!ELEMENT User EMPTY \>*

*\<!ATTLIST User*

*Name CDATA #REQUIRED*

*\>*

## \<Action_File\> DTD

*\<!ELEMENT NGOPAction ( Action_File ) \>*

*\<!ELEMENT Action_File ( Action )\* \>*

*\<!ELEMENT Action ( Host )\* \>*

*\<!ATTLIST Action*

*ID CDATA #REQUIRED*

*\>*

*\<!ELEMENT Host ( AuthorizedGroup , Exec )\* \>*

*\<!ATTLIST Host*

*Name CDATA #REQUIRED*

*\>*

*\<!ELEMENT Exec EMPTY \>*

*\<!ATTLIST Exec*

*Path CDATA #REQUIRED*

*\>*

*\<!ELEMENT AuthorizedGroup EMPTY \>*

*\<!ATTLIST AuthorizedGroup*

*ID CDATA #REQUIRED*

*\>*

#