Package: pollect
pollect - python data collection daemonpollect is a daemon for collecting system and application metrics in periodical intervals.
(similar to collectd). It's designed to require very little dependencies and to be easily customizable.Architecture------------                           ----------
 | executor |  -- result of sources --> | writer |
 ------------                           ----------
       1
       | Calls sources to probe data
       n
 ---------------                                   
 |   source    |   
 ---------------pollect usesexecutorswhich containsourcesfor probing the data. The data is exported using thecollectionname.For persisting the datawritersare used. They can be defined globally (for all executors)
or on a per executor level.By default, the tick time is defined globally, but can be changed on a executor level.Usagepipinstallpollect
pollect--configconfig.json[--dry-run]Note: You can use eitherjsonorymlfiles for configuration.DockerPlace yourconfig.[json|yml]and any customsourcesinto your working directory and rundockerrun-v$(pwd):/pollect-p8000:8000davidgiga1993/pollect:latestConfigHere is an example for collecting the load average every 30 seconds as well as sampling the response time of google
every 2 min and exporting it as prometheus metrics---
tickTime: 30
threads: 4
writer:
  type: Prometheus
  port: 8000
executors:
  - collection: pollect
    sources:
      - type: LoadAvg
  - collection: slowerMetrics
    tickTime: 120
    sources:
      - type: Http # See Http below for more details
        url: https://google.com
        labels: # Additional labels/tags for the metrics
          # It is also possible to access env variables anywhere
          # in the config
          system: prod
          home: ${HOME}A more advanced configuration sample can be found in thepollect.[json|yml]file.Metric namesThe metric names are automatically build out of the collection, source and value name. Example:Pattern:${collection}_${sourceType}[_${sourceName}][_$valueName}]CollectionSource typeSource name (optional)Resulting metric namepollectHttppollect_HttppollectHttptestpollect_Http_testpollectProcesspollect_Process_load_percentpollect_Process_virtual_memorySourcesA source collects data in regular intervals. Depending on the source there are multiple configuration parameters
available.Certain sources might need additional dependencies.The following parameters are available for all sources:ParamDescnameName of the metric (prefix)labelsDict of static labelsHttp response timeHttpMeasures the http response time in millisecondsParamDescurlUrl to the web service. Can be a list of strings as well (the url will be added as label)timeoutTimeout in seconds (default 15)statusCodeThe expected status code (default any non error)proxyHttp proxy which should be used (defaults to none respecting environment variables. Set to '' to use no proxy regardless of environment)Disk usageDiskUsageDisk usage statistics. RequiresshutilpackageLoad averageLoadAvgSystem load average. Linux onlyMemory usageMemoryUsageSystem memory usage. RequirespsutilpackageProcess statsProcessInformation about one or more processesParamDescnameName of the metricprocRegexName of the process(es) - RegexmemoryTrue to enable memory metrics (default true)loadTrue to enable cpu load metrics (default true)InterfaceInterfaceCollects NIC statistics. RequirespsutilpackageParamDescincludeTotalEnables incremental counter dataincludeExplicitly includes nics. Can be emptyexcludeExcludes nics. Can be emptyIOIOCollects IO statistics. RequirespsutilpackageParamDescincludeExplicitly includes disks. Can be emptyexcludeExcludes disks. Can be emptyHDD smart dataSmartCtlWrapper for the linuxsmartctltool. Collects SMART dataParamDescattributesName of smart attributes which should be includeddevicesList of regex for matching disks which should be includedSensorsSensorsWrapper for the linuxsensorsutility. Collects sensor data such as temps and voltagesParamDescincludeName of chips which should be included. Can be emptyexcludeName of chips which should be excluded. Can be emptyuseBaseUnitsSet toFalseto report the raw values in their reported unit (for example mV instead of V)DNS server statisticsBindBind DNS server statistics.ParamDescurlURL to the bind statisticsviewsViews which should be includedSNMPSnmpGetWrapper for the snmpget binary.type:SnmpGetname:Procurve# Host which should be contactedhost:10.1.100.1# Community string (public by default)communityString:public# Metrics which should be collected# Each metric can query one or more oidsmetrics:# Name of the metric-name:throughput# Processing mode:# - undefined (default): No processing# - rate: Computes the change per second, compensating for value overflowsmode:rate# OID which should be probedoid:iso.3.6.1.2.1.16.1.1.1.4.1# It is also possible to define ranges-name:interface_link_state# The parameter in the oid will be replaced with the range numberoid:iso.3.6.1.2.1.2.2.1.5.${if}range:from:1to:10# From and to are inclusivelabel:"if"# The label will be attached to the metricPlex serverPlexCollects plex mediaserver statistics. This requires local IP addresses to be allowed without authentication.ParamDescurlURL to plex. Use the IP of the NIC instead oflocalhostPlex serverZfsProvides simple ZFS pool sizing and performance metrics.Fritzbox WANFritzboxConnects to the fritzbox api and collects WAN statistics. Requires
thefritzconnectionpackage.EVCCEvccExposes the values shown in the EVCC web-ui as metrics.
Requires thewebsocket-clientpackage.ParamDeschostHost and port of the EVCC instance (e.g.localhost:7070)Viessmann APIViessmannCollects sensor data from viessmann heatpumpsParamDescuserUsername of viessmann accountpasswordPassword of accountThese information are only required for the first data collection. Afterwards aviessmann_token.jsonfile is created
to cache the oauth credentials.Homematic IPHomematicIpCollects temperature and humidity data from homematic IP. Requires
thehomematicippackage.ParamDescauthTokenAuth tokenaccessPointAccess point idSMA Energy MeterSmaEnergyMeterCollects data from the SMA Home Manager 2.0. The data is received via multicast from the meter. If this doesn't work for
you, there is also a possibility to configure the meter to unicast the values directly to your machine.ParamDeschostIpThe primary IP address of the host running pollect.Required for multicastSMA PV ModbusSmaPvModbusCollects data from SMA Photovoltaik inverters. Requires tcp modbus to be enabled on the inverted.Requires thepymodbusdependency.ParamDeschostIP or hostname of the invertedportOptional: modbus port, 502 by defaultTP-LINK EAPTpLinkEapCollects wifi statistics of the TP-LINK EAP series devices. This uses the rest api of the device. Note that the devices
only support one session at a time, meaning you will be logged out from the web UI in regular intervals.ParamDescurlURL to the admin UIuserUsernamepasswordPasswordOpenhabOpenhabCollects the values of all channels of all items in openhab. This contains more data than the original metrics exporter
of openhab (since it doesn't include all items).ParamDescurlURL to openhabAudi MMIMMIConnects to the audi MMI backend and collects data. Requires theaudi apipackage. Note: This pacakge is currently broken due to API changes.ParamDesccredentialsPath to the credentials.jsonvinVIN of the car that should be crawledGoogle Play Developer ConsoleGdcProvides app statistics from the google play developer console. Requires the google-cloud-storage package.Importanteach fetch will call the google cloud storage api to check for updates so make sure to call is less
frequent (every 30min or so).Sample config:{
  "type": "Gdc",
  // Name of the bucket - this can be found at the bottom of the bulk export page in the GDC
  "bucketName": "pubsite_prod_rev_1234",
  "apps": [
	{ // All apps for which the statistics should be crawled
	  "package": "my.app.package",
	  "name": "My App"
	}
  ],
  // Key file for the service account which has read access to the GDC
  "keyFile": "api-project-1234.json",
  // Name of the folder where the crawled files should be stored
  "dbDir": "db"
}Apple appstore connectAppStoreConnectCollects download statistics from apple{"type":"AppStoreConnect","keyId":"ASDH123123","keyFile":"my_api_key.p8","issuerId":"asdasd-123123asd-asd12312-asd","vendorNumber":"882223","dbDir":"db"}Http Ingress sourceHttpIngressRequires thegeventpackage
This source starts a simple http webserver and where you can post metrics to.
It's intended if you want to push metrics to pollect, instead of using the default pull probes.- type: HttpIngress
  name: Ingress
  port: 9005 # Listener port
  metrics: # You can define multiple metrics
    sample_metric: # Name of the metric
      type: counter # Optional, gauge by default, counter will cause the value to increment by X every time
      labels: # Labels for this metric
        - hostYou can update the metrics using a simple http json post:curl-XPOSThttp://pollect:9005\-H'Content-Type: application/json'\--data-binary@-<< EOF{"metrics": {"sample_metric": {"value": 124"labels": {"host": "my-hostname"}}}}EOFCertificate sourceCertificateReturns the expiry date of a https certificate. Requiresopensslbinary andpyOpenSSL.- type: Certificate
  name: cert
  url: https://google.comWritersA writer represents the destination where the collected data is written to.Dry runDryRunPrints the collected data to the stdoutPrometheus http exporterPrometheusExports the data via a prometheus endpoint. The port can be configured usingportas configuration:writer:type:Prometheusport:9001Https supportPollect has a custom prometheus exporter which supports https.
This requires thegeventpackage.writer:type:PrometheusSslport:8000key:key.keycert:cert.pemMultithreadingBy default, pollect executes all sources of a collection in parallel with 5 threads.
Different collections are executed in separate threads as well, meaning that multiple long-running probes in one
collection can't block another collection.
If the writer supports partial writes (for exampleprometheus) the result of each source will be immediately
available.
Writers which do not support partial writes will receive the data once all probes have completed.ExtensionsThis example shows how to add your own collectorsSourceextensions/SingleRandom.py:# Single random value with parameterclassSingleRandomSource(Source):def__init__(self,data):super().__init__(data)self.max=data.get('max')def_probe(self):returnrandom()*self.maxextensions/MultiRandomSource.py:# Multiple random valuesclassMultiRandomSource(Source):def_probe(self):return{'a':random(),'b':random()}config.json:{"sources":[{"type":"extensions.SingleRandom","max":100},{"type":"extensions.MultiRandom"}]}A similar principle is used for the writers. Take a look at thesourcesandwritersfolders for more examples.