Package: metREx
metRExSQL query and monitoring system metrics exporter forPrometheus. A product ofReliability EngineeringatThe Home Depot.Supported database engines include:Google BigQueryIBM DB2IBM Informix(experimental)Microsoft SQL ServerMySQLOraclePostgreSQLSQLitePrometheus metrics can also be generated from the following monitoring systems:AppDynamicsExtraHopNew RelicTable of ContentsInstallationDatabase EnginesLocal Environment SetupENV VariablesConfiguration: ServicesDatabase Connection ParametersAPI Connection ParametersEncrypting CredentialsConfiguration: JobsDefining Jobs as ServicesDefining Jobs in a GitHub RepositoryJob ParametersAbout AggregationPushing MetricsExposing Metrics to a PushgatewayPushing Metrics to a Graphite BridgePushing Metrics to WavefrontActivating Push ServicesRunning the ApplicationRunning in DockerSwaggerTestingLicenseInstallation$pipinstallmetRExDatabase EnginesTheSQLAlchemypackage used by this application enables connectivity to a variety of database engines. However, additional packages may need to be installed, depending on which engines are used.A variation on the abovepip installcommand can be used to include specific SQLAlchemy driver packages as needed.*In the following example, the dialect identifiersmysqlandpostgres, supplied in brackets, will install thepymysqlandpg8000driver packages for MySQL and PostgreSQL database connectivity, respectively:$pipinstallmetREx"[mysql,postgresql]"For convenience, a selection of predefined dialects are available for commonly used SQLAlchemy driver packages:*DialectDriverbigquerypybigquerydb2ibm_dbinformixifx_jdbcmssqlpymssqlmysqlpymysqloraclecx_oraclepostgresqlpg8000sqlite**pysqliteTo install any combination of these SQLAlchemy drivers, substitute their dialect identifiers, delimited by commas, within the brackets shown in the previous example(seeSupported Driversfor details about other SQLAlchemy drivers not mapped to the predefined dialects above).*Additional client software may also be required for certain database engines. Installation steps will vary depending on the host operating system.**SQLite support is included by default;sqlitedialect does not need to be installed as an extra.Local Environment SetupA.envfile can optionally be used to define environment variables available to the application at runtime. This can be useful when running in a local development environment.The.envfile should be placed in the installed package root. Any of the.env.*files included in theexamplesfolder can be renamed to.env, as shown below:$cpexamples/.env.macos-oraclemetREx/.envENV VariablesSeeSupported ENV Variables.Configuration: ServicesThis application was originally designed to run onCloud Foundry (CF). However, it can also be configured for other environments.When running in Cloud Foundry, the connection parameters used by metric exporter "jobs" are defined viauser-provided service instances.In non-CF environments, services can be defined via a YAML configuration file (default:env/services.yml), as shown in the following example:---METREX_DB_MYSQL:name:database_namehostname:mysql.mydomain.comport:3306username:userpassword:passwordencrypted:falsedialect:mysqldriver:pymysqlMETREX_DB_BIGQUERY:project:gcp-project-namelocation:UScredentials_path:/path/to/service/account/credentials.jsondialect:bigqueryMETREX_DB_SQLITE:path:/path/to/database/file.dbdialect:sqliteMETREX_API_APPD:hostname:appdynamics.mydomain.comaccount:accountusername:userpassword:passwordencrypted:falsevendor:appdynamicsMETREX_API_EXTRAHOP:hostname:extrahop.mydomain.comapikey:extrahop_apikeyencrypted:falsevendor:extrahopMETREX_API_GITHUB:apikey:github_apikeyencrypted:falsevendor:githubMETREX_API_NEWRELIC:hostname:api.newrelic.comaccount_id:account_idapikey:newrelic_apikeyencrypted:falsevendor:newrelicDatabase Connection ParametersThe names assigned to services containing database connection details must begin with the prefixMETREX_DB_(or the value of theDB_PREFIXenvironment variable). The text that appears after this prefix will be prepended to the names of all Prometheus metrics generated from this service.For database connections (except BigQuery and SQLite), the following parameters must be included:name: Database or service namehostname: Server name or IP addressport: Port numberusername: User namepassword: Passwordencrypted: "true" (recommended) ifpasswordvalue is encrypted or "false" if it is not.dialect: Dialect identifier(refer to table of Dialect/Driver mappings inDatabase Enginessection above)driver: SQLAlchemy driver(refer to table of Dialect/Driver mappings inDatabase Enginessection above)timezone: Localizeddatabase timezoneto apply to "naive" (non-TZ-aware)timestamp_columnvalues returned by metric exporter jobs (optional, to override theDB_DEFAULT_LOCAL_TZenv variable)In addition to the above, the following parameters for establishing SSL connections are supported for MySQL and PostgreSQL databases:ssl_ca: Full path to a file of concatenated CA certificates in PEM formatssl_cert: Full path to a single file in PEM format containing the certificate as well as any number of CA certificates needed to establish the certificate’s authenticityssl_key: Full path to a file containing the private keyFor BigQuery connections, the following parameters are used:project: Name of the GCP project (defaults to the project specified in the credentials JSON file)location: Specifies the dataset location (optional)credentials_path: Full path to the service account credentials JSON file (optional)*credentials_info: Contents of the service account credentials JSON file (optional)*encrypted: "true" (recommended) if thecredentials_infovalue is encrypted or "false" if it is not (optional, for use only withcredentials_info)dialect: "bigquery"*NOTE: Eithercredentials_pathorcredentials_infocan be used instead of or to override theGOOGLE_APPLICATION_CREDENTIALSENV variable on a per-service basis.credentials_pathtakes precedence if provided in combination withGOOGLE_APPLICATION_CREDENTIALSorcredentials_info.For SQLite connections, the following parameters are used:path: Full path to the database file (optional)*dialect: "sqlite"*NOTE: Ifpathis not provided, an in-memory database will be used.API Connection ParametersThe names assigned to services containing API connection details must begin with the prefixMETREX_API_(or the value of theAPI_PREFIXenvironment variable). Presently, connectivity to the following APIs is supported:AppDynamicsExtraHopGitHub*New RelicPrometheus-to-Graphite bridge**Prometheus Pushgateway**Wavefront***GitHub connectivity is provided only as a means to access metric exporter job definitions in a source repository(seeDefining Jobs in a GitHub Repositorysection below). It cannot be used as aserviceparameter in individual exporter job definitions.**Prometheus-to-Graphite bridge, Pushgateway and Wavefront connectivity is provided to allow exporter job metrics to be sent to other services(seePushing Metricssection below). These cannot be used as aserviceparameter in individual exporter job definitions.The parameters to be included for each API connection type are shown below.AppDynamics API connections:hostname: AppD controller server name or IP addressport: Port number (optional)account: Account name (optional, default: "customer1")username: User namepassword: Passwordencrypted: "true" (recommended) ifpasswordvalue is encrypted or "false" if it is notvendor: "appdynamics"ExtraHop API connections:hostname: ExtraHop server name or IP addressport: Port number (optional)apikey: ExtraHop API keyencrypted: "true" (recommended) ifapikeyvalue is encrypted or "false" if it is notvendor: "extrahop"GitHub API connections:hostname: Custom GitHub Enterprise server name or IP address (only include if not using standard GitHub.com)port: Port number (optional)apikey: GitHub API keyencrypted: "true" (recommended) ifapikeyvalue is encrypted or "false" if it is notvendor: "github"New Relic API connections:hostname: New Relic API server name ("api.newrelic.com" in most cases) or IP addressport: Port number (optional)apikey: New Relic user keyencrypted: "true" (recommended) ifapikeyvalue is encrypted or "false" if it is notvendor: "newrelic"Prometheus-to-Graphite bridge API connections:hostname: Graphite bridge server name or IP addressport: Port number (optional)prefix: Prefix to use for all metric names - e.g., "dev.", "prod.system.", etc. (optional)vendor: "graphite"Prometheus Pushgateway API connections:hostname: Pushgateway server name or IP address (include "https://" for secure server connections)port: Port number (optional)username: User name (optional, for connections requiring Basic auth)password: Password (optional, for connections requiring Basic auth)encrypted: "true" (recommended) ifpasswordvalue is encrypted or "false" if it is not (optional, for connections requiring Basic auth)vendor: "pushgateway"Wavefront API connections:hostname: Wavefront server name or IP address (include "http://" or "https://" for direct ingestion, or "proxy://" for proxy server)port: Port number (optional)apikey: API token (optional)encrypted: "true" (recommended) ifapikeyvalue is encrypted or "false" if it is notprefix: Prefix to use for all metric names - e.g., "dev.", "prod.system.", etc. (optional)vendor: "wavefront"Encrypting CredentialsFor security purposes, it is recommended that database passwords and API keys be encrypted, especially when stored in CF. This application supports the decryption of credentials encrypted via thecryptofypackage.To configure the application to decrypt a password or API key for a given connection at runtime, set theencryptedproperty to "true", as shown in the following example:METREX_DB_MYSQL:name:database_namehostname:mysql.mydomain.comport:3306username:userpassword:Zq4B/mjYV9a4sVBzWiCxz5+tjcXHK2yc4UDVhWJLP2g=encrypted:truedialect:mysqldriver:pymysqlMETREX_API_EXTRAHOP:hostname:extrahop.mydomain.comapikey:V37REDWZithWkr9Qu2d/p6uurhpj6kShcYjX0Y6yBWI=encrypted:truevendor:extrahopA "secret" must be supplied to the application in order to unlock encrypted credentials at runtime. This secret must be the same one that was used to encrypt each of the passwords/API keys(refer to the cryptofy documentation for instructions on generating secrets and encrypting credentials).Secrets may be supplied either by means of theSECRET_KEYenvironment variable or via a YAML-formatted file, the location of which is referenced in the ENV variableSECRET_PATH.*Method 1: Environment variableSECRET_KEY=my_secretMethod 2: Secrets file (with file path environment variable)ENV variable:SECRET_PATH=secrets.ymlContents ofsecrets.yml:secret-key:my_secret*WARNING: Due to their sensitive nature, care should be taken to protect secret keys from being exposed. If defined via an ENV variable, make sure it is not visible to others; else if it is supplied as a secrets file, do NOT store the file in source code!Configuration: JobsMetric exporter jobs can be defined via:Service definitions(user-provided services in CF or the services file)GitHub repositoryDefining Jobs as ServicesIf running in Cloud Foundry, metric exporter jobs can be defined as user-provided services. All user-provided services containing exporter job definitions must begin with the prefixMETREX_JOB_(or the value of theJOB_PREFIXenvironment variable).In non-CF environments, jobs can be added to the same YAML-formatted configuration file containing the connection services, as shown below:METREX_JOB_QUERY_EXAMPLE:services:-METREX_DB_MYSQLinterval_minutes:5statement:SELECT col1, col2, col3 FROM my_tablevalue_columns:-col1-col2static_labels:label1:valuelabel2:valueMETREX_JOB_APPD_EXAMPLE:services:-METREX_API_APPDinterval_minutes:15application:Application_Namemetric_path:Business Transaction Performance|Business Transactions|*|*|*METREX_JOB_EXTRAHOP_EXAMPLE:services:-METREX_API_EXTRAHOPinterval_minutes:60metric_params:cycle:autoobject_type:deviceobject_ids:-9363metric_category:http_clientmetric_specs:-reqaggregation:funcs:-count-sum-95threshold:operator:">"value:10METREX_JOB_NEWRELIC_EXAMPLE:services:-METREX_API_NEWRELICinterval_minutes:15statement:FROM Transaction SELECT average(duration) AS 'avg_duration', appName FACET appNamevalue_attrs:-avg_durationDefining Jobs in a GitHub RepositoryMetric exporter jobs can also be defined via a YAML-formatted config file hosted in GitHub like the one shown below:---QUERY_EXAMPLE:services:-METREX_DB_MYSQLinterval_minutes:5statement:SELECT col1, col2, col3 FROM my_tablevalue_columns:-col1-col2static_labels:label1:valuelabel2:valueAPPD_EXAMPLE:services:-METREX_API_APPDinterval_minutes:15application:Application_Namemetric_path:Business Transaction Performance|Business Transactions|*|*|*EXTRAHOP_EXAMPLE:services:-METREX_API_EXTRAHOPinterval_minutes:60metric_params:cycle:autoobject_type:deviceobject_ids:-9363metric_category:http_clientmetric_specs:-reqaggregation:funcs:-count-sum-95threshold:operator:">"value:10NEWRELIC_EXAMPLE:services:-METREX_API_NEWRELICinterval_minutes:15statement:FROM Transaction SELECT average(duration) AS 'avg_duration', appName FACET appNamevalue_attrs:-avg_durationThere is no prefix requirement for GitHub-hosted job names. However, the name referenced in the job'sserviceparameter must match the full service name (prefix included) defined in CF.In order for jobs defined in a GitHub repository to be consumed, the following environment variables must be provided:JOBS_SOURCE_SERVICE=METREX_API_GITHUB
JOBS_SOURCE_ORG=org_name
JOBS_SOURCE_REPO=repo_name
JOBS_SOURCE_PATH=path/to/jobs.ymlChanges to the jobs config file referenced by these env variables will be picked up everyxminutes, per the value assigned toJOBS_SOURCE_REFRESH_INTERVAL, and incorporated into the job scheduler. Unlike changes to services, no restart of the application is required for GitHub-hosted job file changes to take effect.Job ParametersThe parameters to be included for each metric exporter job type are shown below.Database queries:services: A list of one or more service names referencing database connections, from which the job metrics will be sourcedpush_services: (optional) A list of one or more service names referencing push service API connections, to which the job metrics will be sentinterval_minutes: The interval (in minutes) to wait between each execution of the jobstatement: SELECT queryvalue_columns: A list of one or more returned column names representing numeric metric values (any columns returned via the query which do not match the names in this list will be used as metric "labels")static_labels: (optional) One or morekey: valuepairs to apply as static labels for all metrics (static label names must not conflict with returned column names)timestamp_column: (optional) The name of a column returned by the SQL query to use as the metric timestamp (ignored when metrics are exposed via Pushgateway)AppDynamics metrics:services: A list of one or more service names referencing AppDynamics API connections, from which the job metrics will be sourcedpush_services: (optional) A list of one or more service names referencing push service API connections, to which the job metrics will be sentinterval_minutes: The interval (in minutes) to wait between each execution of the jobapplication: The Application name as it appears in AppD (for Database metrics, value is always "Database Monitoring")metric_path: The AppD metrics path; may include wildcards (*)static_labels: (optional) One or morekey: valuepairs to apply as static labels for all metricsExtraHop metrics:services: A list of one or more service names referencing ExtraHop API connections, from which the job metrics will be sourcedpush_services: (optional) A list of one or more service names referencing push service API connections, to which the job metrics will be sentinterval_minutes: The interval (in minutes) to wait between each execution of the jobmetric_params: The list of parameters passed to the ExtraHop/metricsAPI endpointmetric_name: The identifier to be used in the Prometheus metric collector nameaggregation: The list of parameters used to aggregate the values returned from the ExtraHop API(seeAbout Aggregationbelow)static_labels: (optional) One or morekey: valuepairs to apply as static labels for all metricsNew Relic metrics:services: A list of one or more service names referencing New Relic API connections, from which the job metrics will be sourcedpush_services: (optional) A list of one or more service names referencing push service API connections, to which the job metrics will be sentinterval_minutes: The interval (in minutes) to wait between each execution of the jobstatement: NRQL query (excluding SINCE/UNTIL and TIMESERIES clauses)value_attrs: A list of one or more returned attribute labels representing numeric metric values (any attributes returned via the query which do not match the names in this list will be used as metric "labels")static_labels: (optional) One or morekey: valuepairs to apply as static labels for all metrics (static label names must not conflict with returned attribute names)About AggregationCertain job types(see "ExtraHop metrics" above)provide the ability to define aggregation functions to produce metrics from the result data returned by the API.Theaggregationparameter group consists of two subgroups:funcsandthreshold(see example inDefining Jobs as Servicesabove)Thefuncssubgroup accepts any combination of the following aggregation functions:avgcount(default)minmaxsumIt can also take integer values between 0 and 100, representing the nth percentile.The optionalthresholdsubgroup works much in the same way as aHAVINGclause in a SQLGROUP BY: it filters by values matching defined criteria.*  Here, the criteria are expressed via theoperatorandvalueparams.Supportedoperatorvalues are:><>=<==<>!=Thevalueparam must be an integer.Example: Match all values greater than 10JOB_EXAMPLE:aggregation:threshold:operator:">"value:10*NOTE: If nothresholdis specified, the aggregation function will include all returned values.Pushing MetricsBy default, each metric exporter job exposes its own registry endpoint at/metrics/<job_id>, which can be scraped by a Prometheus process. In addition, exporter job metrics can optionally be sent to one or more of the services detailed below.Exposing Metrics to a PushgatewayMetrics can be exposed to one or morePrometheus Pushgatewayservices, which may simplify the process of managing jobs in Prometheus.Defining a Pushgateway service:METREX_API_PUSHGATEWAY:hostname:https://pushgateway.mydomain.comvendor:pushgatewayPushing Metrics to a Graphite BridgeMetrics can also be pushed over TCP to aGraphite bridgeservice, which provides an option to expose metrics to systems other than Prometheus. If a Graphite bridge service is defined, all new exporter jobs will automatically be pushed to it.Defining a Graphite bridge service:METREX_API_GRAPHITE:hostname:graphite.mydomain.comport:2003prefix:my.prefix.vendor:graphitePushing Metrics to WavefrontAs an alternative to Prometheus, metrics can be sent to one or moreWavefrontservices. If a Wavefront service is defined, all new exporter jobs will automatically be pushed to it.Defining a Wavefront service:METREX_API_WAVEFRONT:hostname:mydomain.wavefront.comport:2878prefix:my.prefix.vendor:wavefrontActivating Push ServicesMetrics can be configured to be sent to one or more push services on a per-job basis or on a global basis for all jobs.To configure a specific job to send metrics to one or more push services, include thepush_servicesattribute in the job definition to list the services.Example: Send metrics to push services for a jobJOB_EXAMPLE:services:-METREX_DB_MYSQLpush_services:-METREX_API_GRAPHITEinterval_minutes:5statement:SELECT col1 FROM my_tablevalue_columns:-col1To send all exporter job metrics to one or more push services, list them (comma-delimited, if more than one) via the ENV variableSEND_ALL_JOBS_TO_SERVICES:SEND_ALL_JOBS_TO_SERVICES=METREX_API_PUSHGATEWAY,METREX_API_WAVEFRONTRunning the ApplicationExecute the following command:$metrexRunning in DockerThis application can be run inside aDockercontainer. A prepackaged image is available onDocker Hub.It was built with support for the following databases:Google BigQueryMicrosoft SQL ServerMySQLPostgreSQLSQLiteSampledocker-compose.ymlfile:version:"3.8"services:metrex:image:homedepottech/metrex:latestports:-5000:5000environment:SECRET_PATH:/secrets.ymlSERVICES_PATH:/services.ymlGOOGLE_APPLICATION_CREDENTIALS:/gcp_credentials.jsonvolumes:-/path/to/secrets.yml:/secrets.yml-/path/to/services.yml:/services.yml-/path/to/gcp_credentials.json:/gcp_credentials.jsonCreate and start the Docker container:$docker-composeup-dmetrexSwaggerOpen the URL shown in the resulting output following application startup (default:http://127.0.0.1:5000) to access the Swagger UI.TestingExecute the following command from the top-level directory of the cloned repository:$pythonsetup.pytestLicenseDistributed under theApache, version 2.0 license.