# Monitoring user case: Use Linux Farm Monitoring Information For Job Submission

## Description

A physicist wants to submit his data challenge program to tier 1 site,
which has a Linux farm consisting of several hundred nodes. He wants to
get several light-loaded farm nodes. The Linux farm already collects the
system load information for each node at a certain time interval. This
statistics can be used for monitoring the usage of computing resources
for grid organizations. Based on the physicist's request and the
monitoring information, the monitoring software should return the nodes
which meet the user requirements.

## Contact: Dantong Yu, Jason Smith 

## Performance events required: 

> Configuration information: node name, domain, IP address, Hardware
> type, Network card type, OS version, Kernel version, memory size, swap
> space, home directory, local disk space.
>
> Node status: Machine up time, CPU load, (15 minutes), memory load,
> disk load, and network load

### How the performance information will be used: 

> A user sends out the request: give me 40 least loaded nodes which have
> Redhat 2.4 kernel and Giga bit Ethernet card installed.
>
> The monitoring system will select the nodes with the required hardware
> and software. A prediction module will select the 40 lease loaded node
> from the pre-select resource pool based on the system load information
> of the last three days. The result will be returned to the user.
>
> To prevent the calculation overhead of prediction, the prediction
> software should predict the system load at a pre-defined time interval
> and save it to database also.

### Access needed:

> Last value will be needed.
>
> Streaming of data.
>
> Summary of the data stream.
>
> Requires access to historical information.
>
> Archive database should be published.

### Size of data to be gathered:

> A 600-byte data will be sampled every 15 minutes from each node in
> farm.
>
> Historical archives may become large after years of monitoring.

### Overhead constraints:

> Daemon needs to run on each node to collect machine status.
>
> Machine status will be sent through local network.
>
> Large amount of disk space is needed to save the log information.
>
> All these activities should not interface the normal system running.

### Frequency data will be updated:

> The current testing system gets data sample every 15 minutes from each
> node.

### Frequency data will be accessed:

> The frequency will be determined by the number of farm users. Each
> time when a user submits a job, the monitoring information will be
> accessed.

### How timely does data need to be:

> The last data should be available. The data of last two weeks should
> be desirable.
>
> Data need to be archived. The data will be compressed every six
> months.

### Scale issues: 

> Tier1, Tier2 and Tier3 should be able to deploy the monitoring tools.
> During the initial test period, BNL and Fermi lab should be able to
> use these tools.

### Security requirements 

> Every user who has proper grid certificate can use it.

### Consistency or failure concerns: 

> The sensitive data will be saved the database and mirror site in case
> of failure.

## Duration of the logging 

> If cumulative measurements are taken daily, logging can continue for
> several years before removing old data from the archives. Due to the
> space limitation, every half year, the log data will be compressed
> into half size, i.e. only one data sample will be picked from every
> two data samples in the database.

## Platforms: 

> Globus MDS2.1 on Linux and Relational Databases.