Project: Vector (Observability and Analysis / Observability)
File: 2020-07-28-3642-jmx_rfc.md

This RFC is to introduce a new metrics source to consume metrics from Java Virtual Machines (JVM) using the JMX protocol. The high level plan is to implement one source that collects metrics from JVM servers. Background reading on JVM/JMX monitoring: https://sysdig.com/blog/jmx-monitoring-custom-metrics/ This RFC will cover: A new source for JVM-based metrics using JMX. Users want to collect, transform, and forward metrics to better observe how their JVM-based applications are performing. Build a single source called `jmx_metrics` (name to be confirmed) to collect JVM metrics. The recommended implementation is to use a/the Rust JMX client to connect to a JVM server by an address specified in configuration. https://docs.rs/jmx/0.2.1/jmx/index.html https://github.com/stefano-pogliani/jmx-rust This will require the end user to configure JMX on their JVM instances bound to an external port that can be queried by Vector. Both the Telegraf and Prometheus JMX exporter provide a Java-based agent. Prometheus uses a self-maintained agent and Telegraf uses the agent through . The Prometheus JMX exporter states: ```text This exporter is intended to be run as a Java Agent, exposing a HTTP server and serving metrics of the local JVM. It can be also run as an independent HTTP server and scrape remote JMX targets, but this has various disadvantages, such as being harder to configure and being unable to expose process metrics (e.g., memory and CPU usage). Running the exporter as a Java Agent is thus strongly encouraged. ``` This may require some testing as we

The query will return these metrics by parsing the query results and converting them into metrics: `jmx_up` Used as an uptime metric (0/1) `jmxconfigreloadsuccesstotal` (counter) `jmxprocesscpusecondstotal` (counter) `jmxprocessstarttimeseconds` (gauge) `jmxprocessopen_fds` (gauge) `jmxprocessmax_fds` (gauge) `jmxjvmthreads_current` (gauge) `jmxjvmthreads_daemon` (gauge) `jmxjvmthreads_peak` (gauge) `jmxjvmthreadsstartedtotal` (counter) `jmxjvmthreads_deadlocked` (gauge) `jmxjvmthreadsdeadlockedmonitor` (gauge) `jmxjvmthreads_state` tagged with `state` (gauge) `jmxconfigreloadfailuretotal` (counter) `jmxjvmbufferpoolused_bytes` tagged with `pool` (gauge) `jmxjvmbufferpoolcapacity_bytes` tagged with `pool` (gauge) `jmxjvmbufferpoolused_buffers` tagged with `pool` (gauge) `jmxjvmclasses_loaded` (gauge) `jmxjvmclassesloadedtotal` (counter) `jmxjvmclassesunloadedtotal` (counter) `jmxjavalangMemoryPoolUsageThresholdSupported` tagged with `name` (gauge) `jmxjavalangThreadingThreadContentionMonitoringEnabled` (gauge) `jmxjavalangOperatingSystemCommittedVirtualMemorySize` (gauge) `jmxjavalangGarbageCollectorLastGcInfomemoryUsageAfterGcused` tagged with `name`, `key` (counter?) `jmxjavalangThreadingThreadContentionMonitoringSupported` (gauge) `jmxjvmgccollectionseconds` tagged with `gc` (summary) `jmxjvmmemorybytesused` tagged with `area` (gauge) `jmxjvmmemorybytescommitted` tagged with `area` (gauge) `jmxjvmmemorybytesmax` tagged with `area` (gauge) `jmxjvmmemorybytesinit` tagged with `area` (gauge) `jmxjvmmemorypoolbytes_used` tagged with `pool` (gauge) `jmxjvmmemorypoolbytes_committed` tagged with `pool` (gauge) `jmxjvmmemorypoolbytes_max` tagged with `pool` (gauge) `jmxjvmmemorypoolbytes_init` tagged with `pool` (gauge) `jmxjvminfo` tagged with version, vendor, runtime (gauge) `jmxjvmmemorypoolallocatedbytestotal` tagged with `pool` (counter) `jmxjavalangMemoryHeapMemoryUsage_committed` maybe gauge? TBD `jmxjavalangOperatingSystemTotalSwapSpaceSize` maybe gauge? TBD `jmxjavalangMemoryPoolCollectionUsage_max` tagged with `name` (untyped) `jmxjavalangRuntimeStartTime` (untyped) `jmxjavalangGarbageCollectorLastGcInfo_endTime` tagged with `name` (untyped) `jmxjavalangMemoryHeapMemoryUsage_max` (untyped) `jmxjavalangMemoryPoolUsageThreshold` tagged with `name` (untyped) `jmxjavalangMemoryPoolCollectionUsageThresholdCount` tagged with `name` (untyped) `jmxjavalangMemoryNonHeapMemoryUsage_used` (untyped) `jmxjavalangThreadingPeakThreadCount` (untyped) `jmxjavalangMemoryPoolPeakUsage_used` tagged with `name` (untyped) `jmxjavalangClassLoadingTotalLoadedClassCount` (untyped) `jmxjavalangOperatingSystemMaxFileDescriptorCount` (untyped) `jmxjavalangClassLoadingVerbose` (untyped) `jmxjavalangGarbageCollectorLastGcInfo_id` tagged with `name` (untyped) `jmxjavalangThreadingCurrentThreadUserTime` (untyped) `jmxjavalangGarbageCollectorLastGcInfomemoryUsageAfterGccommitted` tagged with `name` and `key` (untyped) `jmxjavalangThreadingThreadCount` (untyped) `jmxjavalangMemoryPoolPeakUsage_committed` tagged with `name` (untyped) `jmxjavalangMemoryObjectPendingFinalizationCount` (untyped) `jmxjavalangMemoryPoolUsage_used` tagged with `name` (untyped) `jmxjavalangGarbageCollectorCollectionCount` tagged with `name` (untyped) `jmxjavalangThreadingSynchronizerUsageSupported` (untyped) `jmxjavalangRuntimeBootClassPathSupported` (untyped) `jmxjavanioBufferPoolCount` tagged with `name` (untyped) `jmxjavalangGarbageCollectorLastGcInfo_GcThreadCount` tagged with `name` (untyped) `jmxjavalangGarbageCollectorLastGcInfomemoryUsageBeforeGccommitted` tagged with `name` and `key` (untyped) `jmxjavalangThreadingCurrentThreadCpuTimeSupported` (untyped) `jmxjavalangClassLoadingLoadedClassCount` (untyped) `jmxjavalangMemoryPoolCollectionUsage_init` tagged with `name` (untyped) `jmxjavalangMemoryPoolPeakUsage_max` tagged with `name` (untyped) `jmxjavalangMemoryPoolUsage_max` tagged with `name` (untyped) `jmxjavalangGarbageCollectorValid` tagged with `name` (untyped) `jmxjavalangGarbageCollectorLastGcInfomemoryUsageBeforeGcused` tagged with `name` and `key` (untyped) `jmxjavalangThreadingThreadAllocatedMemoryEnabled` (untyped) `jmxjavalangMemoryManagerValid` tagged with `name` (untyped) `jmxjavalangMemoryPoolUsage_init` tagged with `name` (untyped) `jmxjavalangOperatingSystemProcessCpuLoad` (untyped) `jmxjavalangMemoryPoolCollectionUsage_committed` tagged with `name` (untyped) `jmxjavalangOperatingSystemTotalPhysicalMemorySize` (untyped) `jmxjavalangMemoryNonHeapMemoryUsage_committed` (untyped) `jmxjavalangCompilationTotalCompilationTime` (untyped) `jmxjavalangMemoryVerbose` (untyped) `jmxjavalangMemoryPoolValid` tagged with `name` (untyped) `jmxjavalangOperatingSystemFreeSwapSpaceSize` (untyped) `jmxjavalangMemoryPoolUsageThresholdExceeded` tagged with `name` (untyped) `jmxjavalangThreadingCurrentThreadCpuTime` (untyped) `jmxjavalangMemoryPoolCollectionUsageThreshold` tagged with `name` (untyped) `jmxjavalangGarbageCollectorCollectionTime` tagged with `name` (untyped) `jmxjavalangCompilationCompilationTimeMonitoringSupported` (untyped) `jmxjavalangMemoryPoolUsage_committed` tagged with `name` (untyped) `jmxjavalangMemoryNonHeapMemoryUsage_init` (untyped) `jmxjavalangMemoryPoolPeakUsage_init` tagged with `name` (untyped) `jmxjavalangGarbageCollectorLastGcInfo_startTime` tagged with `name` (untyped) `jmxjavalangOperatingSystemAvailableProcessors` (untyped) `jmxjavalangMemoryPoolCollectionUsageThresholdSupported` tagged with `name` (untyped) `jmxjavalangGarbageCollectorLastGcInfomemoryUsageBeforeGcmax` tagged with `name` and `key` (untyped) `jmxjavalangClassLoadingUnloadedClassCount` (untyped) `jmxjavanioBufferPoolMemoryUsed` tagged with `name` (untyped) `jmxjavanioBufferPoolTotalCapacity` tagged with `name` (untyped) `jmxjavalangMemoryHeapMemoryUsage_used` (untyped) `jmxjavalangMemoryPoolCollectionUsage_used` tagged with `name` (untyped) `jmxjavalangMemoryHeapMemoryUsage_init` (untyped) `jmxjavalangOperatingSystemSystemCpuLoad` (untyped) `jmxjavalangGarbageCollectorLastGcInfomemoryUsageAfterGcinit` tagged with `name` and `keys` (untyped) `jmxjavalangGarbageCollectorLastGcInfomemoryUsageBeforeGcinit` tagged with `name` and `key` (untyped) `jmxjavalangThreadingThreadAllocatedMemorySupported` (untyped) `jmxjavalangMemoryNonHeapMemoryUsage_max` (untyped) `jmxjavalangThreadingDaemonThreadCount` (untyped) `jmxjavalangThreadingThreadCpuTimeSupported` (untyped) `jmxjavalangOperatingSystemSystemLoadAverage` (untyped) `jmxjavalangThreadingTotalStartedThreadCount` (untyped) `jmxjavalangOperatingSystemProcessCpuTime` (untyped) `jmxjavalangOperatingSystemFreePhysicalMemorySize` (untyped) `jmxjavalangRuntimeUptime` (untyped) `jmxjavalangMemoryPoolCollectionUsageThresholdExceeded` tagged with `name` (untyped) `jmxjavalangGarbageCollectorLastGcInfo_duration` tagged with `name` (untyped) `jmxjavalangThreadingObjectMonitorUsageSupported` (untyped) `jmxjavalangGarbageCollectorLastGcInfomemoryUsageAfterGcmax` tagged with `name` and `key` (untyped) `jmxjavalangMemoryPoolUsageThresholdCount` tagged with `name` (untyped) `jmxjavalangThreadingThreadCpuTimeEnabled` (untyped) `jmxjavalangOperatingSystemOpenFileDescriptorCount` (untyped) The type of metric for any metric marked `untyped` is unclear but likely gauges. We'll need to do deeper research to determine: Whether we want to keep them. What type they are if we retain them. Naming of metrics is determined via: `jmx metricname` This is in line with JMX's internal naming. All metrics will be tagged with the `endpoint` and `host`. This list is the default JVM metrics available via JMX. Individual applications (Kafka, Cassandra, Tomcat, etc) will expose their own MBeans and metrics and as the Prometheus jmx_exporter does. The Prometheus exporter contains accept/deny patterns for specific MBean objects and a rule-based rewrite engine to match and construct metrics from specific objects. Both of these seems like reasonable future work items, albeit we may want to look at whether a transform might be more effective? The following additional source configuration will be added: ```toml [sources.mysourceid] type = "jmx_metrics" # required endpoint =