Package: colmet
Colmet - Collecting metrics about jobs running in a distributed environnementIntroduction:Colmet is a monitoring tool to collect metrics about jobs running in a
distributed environnement, especially for gathering metrics on clusters and
grids. It provides currently several backends :Input backends:taskstats: fetch task metrics from the linux kernelrapl: intel processors realtime consumption metricsperfhw: perf_event countersjobproc: get infos from /procipmipower: get power metrics from ipmitemperature: get temperatures from /sys/class/thermalinfiniband: get infiniband/omnipath network metricslustre: get lustre FS statsOutput backends:elasticsearch: store the metrics on elasticsearch indexeshdf5: store the metrics on the filesystemstdout: display the metrics on the terminalIt uses zeromq to transport the metrics across the network.It is currently bound to theOARRJMS.A Grafanasample dashboardis provided for the elasticsearch backend. Here are some snapshots:Installation:Requirementsa Linux kernel that supportsTaskstatsintel_rapl (for RAPL backend)perf_event (for perfhw backend)ipmi_devintf (for ipmi backend)Python Version 2.7 or newerpython-zmq 2.2.0 or newerpython-tables 3.3.0 or newerpython-pyinotify 0.9.3-2 or newerpython-requestsFor the Elasticsearch output backend (recommended for sites with > 50 nodes)An Elasticsearch serverA Grafana server (for visu)For the RAPL input backend:libpowercap, powercap-utils (https://github.com/powercap/powercap)For the infiniband backend:perfquerycommand line toolfor the ipmipower backend:ipmi-oemcommand line tool (freeipmi) or other configurable commandInstallationYou can install, upgrade, uninstall colmet with these commands::$ pip install [--user] colmet
$ pip install [--user] --upgrade colmet
$ pip uninstall colmetOr from git (last development version)::$ pip install [--user] git+https://github.com/oar-team/colmet.gitOr if you already pulled the sources::$ pip install [--user] path/to/sourcesUsage:for the nodes :sudo colmet-node -vvv --zeromq-uri tcp://127.0.0.1:5556for the collector :# Simple local HDF5 file collect:
colmet-collector -vvv --zeromq-bind-uri tcp://127.0.0.1:5556 --hdf5-filepath /data/colmet.hdf5 --hdf5-complevel 9# Collector with an Elasticsearch backend:
  colmet-collector -vvv \
    --zeromq-bind-uri tcp://192.168.0.1:5556 \
    --buffer-size 5000 \
    --sample-period 3 \
    --elastic-host http://192.168.0.2:9200 \
    --elastic-index-prefix colmet_dahu_ 2>>/var/log/colmet_err.log >> /var/log/colmet.logYou will see the number of counters retrieved in the debug log.For more information, please refer to the help of theses scripts (--help)Notes about backendsSome input backends may need external libraries that need to be previously compiled and installed:# For the perfhw backend:
cd colmet/node/backends/lib_perf_hw/ && make && cp lib_perf_hw.so /usr/local/lib/
# For the rapl backend:
cd colmet/node/backends/lib_rapl/ && make && cp lib_rapl.so /usr/local/lib/Here's acomplete colmet-node start-up process, with perfw, rapl and more backends:export LIB_PERFHW_PATH=/usr/local/lib/lib_perf_hw.so
export LIB_RAPL_PATH=/applis/site/colmet/lib_rapl.so

colmet-node -vvv --zeromq-uri tcp://192.168.0.1:5556 \
   --cpuset_rootpath /dev/cpuset/oar \
   --enable-infiniband --omnipath \
   --enable-lustre \
   --enable-perfhw --perfhw-list instructions cache_misses page_faults cpu_cycles cache_references \
   --enable-RAPL \
   --enable-jobproc \
   --enable-ipmipower >> /var/log/colmet.log 2>&1RAPL - Running Average Power Limit (Intel)RAPL is a feature on recent Intel processors that makes possible to know the power consumption of cpu in realtime.Usage : start colmet-node with option--enable-RAPLA file named RAPL_mapping.[timestamp].csv is created in the working directory. It established the correspondence betweencounter_1,counter_2, etc from collected data and the actual name of the metric as well as the package and zone (core / uncore / dram) of the processor the metric refers to.If a given counter is not supported by harware the metric name will be "counter_not_supported_by_hardware" and0values will appear in the collected data;-1values in the collected data means there is no counter mapped to the column.PerfhwThis provides metrics collected using  interfaceperf_event_open.Usage : start colmet-node with option--enable-perfhwOptionnaly choose the metrics you want (max 5 metrics) using options--perfhw-listfollowed by space-separated list of the metrics/Example :--enable-perfhw --perfhw-list instructions cpu_cycles cache_missesA file named perfhw_mapping.[timestamp].csv is created in the working directory. It establishes the correspondence betweencounter_1,counter_2, etc from collected data and the actual name of the metric.Available metrics (refers to perf_event_open documentation for signification) :cpu_cycles 
instructions 
cache_references 
cache_misses 
branch_instructions
branch_misses
bus_cycles 
ref_cpu_cycles 
cache_l1d 
cache_ll
cache_dtlb 
cache_itlb 
cache_bpu 
cache_node 
cache_op_read 
cache_op_prefetch 
cache_result_access 
cpu_clock 
task_clock 
page_faults 
context_switches 
cpu_migrations
page_faults_min
page_faults_maj
alignment_faults 
emulation_faults
dummy
bpf_outputTemperatureThis backend gets temperatures from/sys/class/thermal/thermal_zone*/tempUsage : start colmet-node with option--enable-temperatureA file named temperature_mapping.[timestamp].csv is created in the working directory. It establishes the correspondence betweencounter_1,counter_2, etc from collected data and the actual name of the metric.Colmet CHANGELOGVersion 0.6.9Fix for newer pyzmq versionsVersion 0.6.8Added nvidia GPU supportVersion 0.6.7bugfix: glob import missing into procstatsVersion 0.6.6Added --no-check-certificates option for elastic backendAdded involved jobs and new metrics into jobprocstatsVersion 0.6.4Added http auth support for elasticsearch backendVersion 0.6.3Released on September 4th 2020Bugfixes into lustrestats and jobprocstats backendVersion 0.6.2Released on September 3rd 2020Python package fixVersion 0.6.1Released on September 3rd 2020New input backends: lustre, infiniband, temperature, rapl, perfhw, impipower, jobprocNew ouptut backend: elasticsearchExample Grafana dashboard for Elasticsearch backendAdded "involved_jobs" value for metrics that are global to a node (job 0)Bugfix for "dictionnary changed size during iteration"Version 0.5.4Released on January 19th 2018hdf5 extractor script for OAR RESTFUL APIAdded infiniband backendAdded lustre backendFixed cpuset_rootpath default always appendedVersion 0.5.3Released on April 29th 2015Removed unnecessary lock from the collector to avoid colmet to wait foreverRemoved (async) zmq eventloop and added--sample-periodto the collector.Fixed some bugs about hdf fileVersion 0.5.2Released on Apr 2nd 2015Fixed python syntax errorVersion 0.5.1Released on Apr 2nd 2015Fixed error about missingrequirements.txtfile in the sdist packageVersion 0.5.0Released on Apr 2nd 2015Don't run colmet as a daemon anymoreMaintained compatibility with zmq 3.x/4.xDropped--zeromq-swap(swap was dropped from zmq 3.x)Handled zmq name change from HWM to SNDHWM and RCVHWMFixed requirementsDropped python 2.6 supportVersion 0.4.0Saved metrics in new HDF5 file if colmet is reloaded in order to avoid HDF5 data corruptionHandled HUP signal to reloadcolmet-collectorRemovedhiwater_rssandhiwater_vmcollected metrics.Version 0.3.1New metricshiwater_rssandhiwater_vmfor taskstatsWorked with pyinotify 0.8Added--disable-procstatsoption to disable procstats backend.Version 0.3.0Divided colmet package into three partscolmet-node : Retrieve data from taskstats and procstats and send to
collectors with ZeroMQcolmet-collector : A collector that stores data received by ZeroMQ in a
hdf5 filecolmet-common : Common colmet part.Added some parameters of ZeroMQ backend to prevent a memory overflowSimplified the command line interfaceDropped rrd backend because it is not yet workingAdded--buffer-sizeoption for collector to define the maximum number of
counters that colmet should queue in memory before pushing it to output
backendHandled SIGTERM and SIGINT to terminate colmet properlyVersion 0.2.0Added options to enable hdf5 compressionSupport for multiple job by cgroup path scanningUsed Inotify events for job list updateDon't filter packets if no job_id range was specified, especially with zeromq
backendWaited the cgroup_path folder creation before scanning the list of jobsAdded procstat for node monitoring through fictive job with 0 as identifierUsed absolute time take measure and not delay between measure, to avoid the
drift of measure timeAdded workaround when a newly cgroup is created without process in it
(monitoring is suspended upto one process is launched)Version 0.0.1Conception