Comprehensive Grid and Job
Monitoring with Fifemon
Kevin Retzke
User Support for Distributed Computing @ Fermilab
HTCondor Week 2016

FIFE Project
FabrIc for Frontier Experiments:
Common computing for “notCMS” experiments at Fermilab

●●
●
●
●
●

O(10) experiments
O(100) users
O(10 000) simultaneous jobs
O(1 000 000) jobs per week
O(1 PB) data collected permonth
One global HTCondor pool (viaGlideinWMS)○~ ⅔ jobs run on dedicated localcluster○~ ⅓ opportunistic through OpenScience GridWhy Do We Need Monitoring?
Grid admins want to know:●●●●

Overall health of the batch system
Worker node status andavailability
Efficiency in matching jobs toresources
Identify and fix problems quickly(before users and stakeholders
notice… and open tickets)

Users want to know:●●●

State of their jobs
Availability of resources
WHY ISN’T MY JOB RUNNING‽

Stakeholders want to know:●●

Each group is getting the resources
it needs
Resources are being usedeffectively

Fermigrid Monitor (ca. 2004)
Monitoring for local HTCondor cluster(GPGrid).
●●●

Aggregate metrics for grid andVOs.
No offsite information, no user job
information.
Difficult to alter or expand.

OK for grid admins, good forstakeholders, bad for users.

Fifemon v1 (ca. 2014)
Growing usage of offsite resources
through OSG; needed new monitoring.
●●●

Aggregate metrics for users and VOs.
No cluster information.
Cumbersome to maintain andexpand.

OK for grid admins, bad for stakeholders,good for users.

Fifemon v2+ (ca. 2015)
Landscape Program: develop
comprehensive monitoring for FIFE,HEP Cloud, and beyond.
●●●Leverage open-source monitoring
technology
Focus on incorporating new datasources and new dashboards
Rapid development and iteration oftailored views for each target
audience.

Good for grid admins, stakeholders, andusers alike.

Fifemon Backend
Data collection:
●
●
●
●
●
●

●

Generic HTCondor probe collecting
daemon, machine, and job status
Logstash collecting live HTCondor Events
Several other centrally-run probes querying
other resources
Some services directly reporting to Graphite
Most probes report stats every five minutes
Graphite:○
250K individual metrics○~80GB○
10 year history
Elasticsearch: ~8GB per day

Graphite:●●●

Time-series database, stores data in files
similar to RRD with caching layer.
Simple line protocol
Powerful query manipulations andaggregations

Elasticsearch:●●“NoSQL” document database, powered byApache Lucene.
Store full details on current jobs, batch slots,and logs.

Fifemon Frontend
Grafana:
●
●
●
●
●
●

Time-series (primarily) visualization
dashboard platform.
Supports numerous data sources (Graphite,InfluxDB, Elasticsearch, etc).
Several auth methods (LDAP, OAuth,proxy).
Rich user interface for graphing metrics andcomposing dashboards.
Scripted and templated dashboards and raw
HTML panels allow extensive customization.
V3 (released last week) introduces new
plugin system to support custom datasources
and panels.

Kibana:●Elasticsearch data only○○●●●

Current jobs and machine status
Event logs

Explore data, create ad-hoc visualizations,combine into dashboards
Used for analytics and troubleshooting
Access limited to grid admins and powerusers

Fifemon Architecture
GrafanaFifebatch
GPGridCMS Tier 1
CMS LPCHEP CloudData handling
dCacheBlueArc
Postgresmore...

Time-series
Aggregations

Graphite

Logstash &Probes
Collect:● Job Details● Slot Details● System Metrics● Event Logs
Raw
Documents

KibanaElasticsearchNext Steps
Fifemon is constantly evolving:●●Adding new data sources and metrics
New dashboards:○○●●Tailored views based on user request
Discovering new ways of looking at the data

New Grafana panels
Further leverage HTCondor event logs &gangliad/metricsd for true real-time monitoringCase Studies“There’s a dashboard for that...”Case Study:
Grid Admin“Is the batch system healthy?”

CinnamonPhoto:Public DomainWoof, all green!

Throughput
is Good.

Grid utilizationis OK.

Let’s checkanyways… what
happened here?

Hmm, we couldn’t
query a CE for a
few minutes. I’ll
check the probelogs.

Case Study:
Stakeholder“Is my experiment getting theresources it needs and using themeffectively?”
Hazel

Photo: Claudio Gennari (CC-BY-2.0)

We’re well above
our quota, but
efficiency could bebetter.

Grrr… time to sendsome emails!
Let’s check on resource
requests.

Disk and Memory requests look
good, lots of users exceeding
request time though.

Otherwise job throughput isgood both onsite and offsite.

Case Study:
User“What’s the status of my jobs?”

Cocoa

Yay my Jobs arestarting, but myefficiency is dropping!

This cluster haspoor efficiency, let’s
take a look at it.

A few failed
processes, and abunch aredisconnected.

Some of my jobs
are held… I need toincrease memory
request.

Case Study:
UpperManagement“What does the computing
division do again?”
Sage (and minions)Photo: Public DomainComprehensive grid monitoringwith Fifemon has improved
resource utilization, job
throughput, and computing
visibility at Fermilab.
Probes, dashboards, and docs at:https://github.com/fifemon