Package: prometheus-http-sd
prometheus-http-sdThis is aPrometheus HTTP SDframework.FeaturesInstallationUsageThe Python Target GeneratorPython Target Generator Cache and ThrottleManage prometheus-http-sd by systemdAdmin PageServe under a different root pathSentry APMDefine your targetsYour target generatorThe Target PathOverwritingjob_namelabelsCheck and Validate your TargetsScript DependenciesUpdate Your ScriptsBest PracticeFeaturesSupport static targets from Json file;Support static targets from Yaml file;Support generating target list using Python script;Supportcheckcommand, to testing the generated target is as expected, and
counting the targets;You can monitoring your target generator via/metrics, seemetrics;Admin page to list all target paths;Auto reload when generator or targets changed;Support managing targets in a hierarchy way;Throttle parallel execution and cache the result for Python script;Support Sentry APM.Installationpipinstallprometheus-http-sdUsageFirst, you need a directory, everything in this directory will be used to
generate targets for prometheus-http-sd.$mkdirtargetsIn this directory, every file is called a target "generator":Filename that ending with.jsonwill be exposed directlyFilename that ending with.yamlwill be exposed directlyFilename that ending with.pymust include agenerate_targets()function,
the function will be run, and it must return aTargetList(Type helper inprometheus_http_sd.targets.)Filename that starts with_will be ignored, so you can have some python
utils there, for e.g._utils/__init__.pythat you can import in yougenerate_targets()Filename that starts with.(hidden file in Linux) will also be ignoredLet write our first target generator by yaml, put this into yourtargets/first_target.yaml:----targets:-"10.1.1.9:9100"-"10.1.1.10:9100"labels:job:nodedatacenter:nycgroup:g1-targets:-"10.2.1.9:9100"-"10.2.1.10:9100"labels:job:nodedatacenter:sggroup:g2If you use json, the data structure is the same, just in Json format.The Python Target GeneratorLet's put another generator using Python:Put this into yourtargets/by_python.py:defgenerate_targets(**extra_parameters):return[{"targets":["10.1.1.22:2379"],"labels":{"app":"etcd"}}]Then you can runprometheus-http-sd serve -h 0.0.0.0 -p 8080 /tmp/targets,
prometheus-http-sd will start to expose targets at:http://0.0.0.0:8080/targetsThe-hand-pis optional, defaults to127.0.0.1and8080.$prometheus-http-sdserve/tmp/targets# replace this to your target path[2022-07-2400:52:03,896]{wasyncore.py:486}INFO-Servingonhttp://127.0.0.1:8080If you runcurl http://127.0.0.1:8080/targetsyou will get:{"targets":"10.1.1.22:2379","labels":{"app":"etcd"}}Finally, you can tell your Prometheus to find targets underhttp://127.0.0.1:8080/targets, by adding this into your Prometheus config:scrape_configs:-job_name:"etcd"http_sd_config:url:http://127.0.0.1:8080/targets/The Python target generator also support URL query params. You can check the
params in yourgenerate_targets()function.For example:defgenerate_targets(**params):cluster=params.get("cluster")return[{"targets":["10.1.1.22:2379"],"labels":{"app":"etcd","cluster":cluster}}]Thencurl http://127.0.0.1:8080/targets?cluster=us1you will get:{"targets":"10.1.1.22:2379","labels":{"app":"etcd","cluster":"us1"}}Python Target Generator Cache and ThrottleSupport you have 10 Prometheus instance request http-sd for targets every
minutes, for Python script target generator, it doesn't make sense that the same
script run 10 times in every minute, instead, it should run only once, and use
this result to respond for 10 Prometheus instances.prometheus-http-sd has cache and throttle by default, that means:At any time there is only one python script runningThe result will be cached for 1minute (This means that every script at max
will be only running one time per minute, and your target update will delay at
most 1 minute)Manage prometheus-http-sd by systemdJust put this file under/lib/systemd/system/http-sd.service(remember to
change your installation path and root_dir path):# /lib/systemd/system/http-sd.service
[Unit]
Description=Prometheus HTTP SD Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/opt/httpsd_env/bin/prometheus-http-sd serve \
    -h 0.0.0.0                                         \
    -p 8080                                            \
    /opt/httpsd_targets

Restart=always
RestartSec=90

[Install]
WantedBy=multi-user.targetAdmin PageYou can open the root path,http://127.0.0.1:8080/in this example, and you
will see all of the available paths list in the admin page.Serve under a different root pathIf you put prometheus-http-sd behind a reverse proxy like Nginx, like this:location /http_sd/ {
      proxy_pass http://prometheus_http_sd;
}Then you need to tell prometheus_http_sd to serve all HTTP requests under this
path, by using the--url_prefix /http_sdcli option, (or-r /http_sdfor
short).Change CertificateBy default,prometheus-http-sdhas caching capabilities for Python targets to avoid server crashes due to too many queries.+------------+
    |            |
    |            |
    |            |                  +-----------+
    |  Caller 1  +----+             |           |
    |            |    |             |           |
    |            |    |             |           |
    |            |    |             |           |
    +------------+    |             |           |
                      |             |           |
                      |             |           |
    +------------+    |             |           |                  +----------+
    |            |    |             |           |                  |          |
    |            |    | call at the |           | only single call |          |
    |            |    |  same time  |  Timeout  |    to the back   |          |
    |  Caller 2  +----|------------>+   Cache   +----------------->+ Function |
    |            |    |             |           |                  |          |
    |            |    |             |           |                  |          |
    |            |    |             |           |                  |          |
    +------------+    |             |           |                  +----------+
                      |             |           |
                      |             |           |
    +------------+    |             |           |
    |            |    |             |           |
    |            |    |             |           |
    |            |    |             |           |
    |  Caller 3  +----+             |           |
    |            |                  +-----------+
    |            |
    |            |
    +------------+To change this behavior, you can use the option--cache-typeto change the cache behavior.Also, you can use the option--cache-optto change the variable.
For example:prometheus-http-sdserve\-h0.0.0.0\-p8080\--cache-type="Timeout"\--cache-opt="timeout=360"\/opt/httpsd_targetsTimeoutThis is the default value, It will cache the result or exception from the target function.timeout=<seconds>:
function timeout. if exceed, raise TimeoutException (in sec).cache_time=<seconds>:
after function return normally, how long should we cache the result (in sec).cache_exception_time=<seconds>:
after function return incorrectly, how long should we cache the exception (in sec).name=<str>:
prometheus_client metrics prefixgarbage_collection_count=<seconds>:
garbage collection thresholdgarbage_collection_interval=<seconds>:
the second to avoid collection too often.copy_response=<bool>:
if true, use copy.deepcopy on the response from the target function.NoneThis is a dummy function if you don't need any cache method.Sentry APMYou can use the option--sentry-url <you-sentry-url>(or-s <your-sentry-url>)
to enable Sentry APM.The Exception from user's script will be sent to Sentry.Define your targetsYour target generatorPlease see theUsageto know how to define your generator.The Target Pathprometheus-http-sd support sub-pathes.For example, if we useprometheus-http-sd serve gateway, and thegatewaydirectory's structure is as follows:gateway
├──nginx
│├──edge.py
│└──targets.json
└──targets.jsonThen:/targets/gatewaywill return the targets from:gateway/nginx/edge.pygateway/nginx/targets.jsongateway/targets.json/targets/gateway/nginxwill return the targets from:gateway/nginx/edge.pygateway/nginx/targets.jsonThis is very useful when you use vertical scaling. Say you have 5 Prometheus
instances, and you want each one of them scrape for different targets, then you
can use the sub-path feature of prometheus-http-sd.For example, in one Prometheus's scrape config:scrape_configs:-job_name:"nginx"http_sd_config:url:http://prometheus-http-sd:8080/targets/nginx-job_name:"etcd"http_sd_config:url:http://prometheus-http-sd:8080/targets/etcdAnd in another one:scrape_configs:-job_name:"nginx"http_sd_config:url:http://prometheus-http-sd:8080/targets/database-job_name:"etcd"http_sd_config:url:http://prometheus-http-sd:8080/targets/applicationOverwritingjob_namelabelsYou may want to put all of etcd targets in one generator, including port 2379
for etcd metrics and 9100 for node_exporter metrics of the etcd server. But thejob_namesetting was based on per URL.The trick is that, you can overwrite thejoblabel in the target labels, like
this:----targets:-"10.1.1.9:9100"labels:job:nodedatacenter:nycgroup:g1-targets:-"10.1.1.9:2379"labels:job:etcddatacenter:nycgroup:g1Check and Validate your TargetsYou can useprometheus-http-sd checkcommand to test your targets dir. It will
run all of you generators, validate the targets, and print the targets count
that each generator generates.$prometheus-http-sdchecktest/test_generator/root[2022-08-0600:50:11,095]{validate.py:16}INFO-Rungeneratortest/test_generator/root/json/target.json,took0.0011398792266845703s,generated1targets.[2022-08-0600:50:11,100]{validate.py:16}INFO-Rungeneratortest/test_generator/root/yaml/target.yaml,took0.0043718814849853516s,generated2targets.[2022-08-0600:50:11,100]{validate.py:22}INFO-Done!Generated{total_targets}intotal.It's a good idea to useprometheus-http-sd checkin your CI system to validate
your targets generator scripts and target files.For Python script,prometheus-http-sd checkcommand will rungenerate_targetsin each script, without any params. However, you can
overwrite thechecklogic by providing a function calledtest_generate_targets()(without any function args), thencheckwill runtest_generate_targetsinstead. (So you can callgenerate_targets(foo="bar")to set the test logic of your own.Script DependenciesIf you want your scripts to use some other python library, just install them
into thesame virtualenvthat you install prometheus-http-sd, so that
prometheus-http-sd can import them.Update Your ScriptsIf you want to update your script file or target json file, just upload and
overwrite with your new version, it will take effect immediately after you
making changes,there is no need to restartprometheus-http-sd,
prometheus-http-sd will read the file (or reload the python script) every time
serving a request.It is worth noting that restarting is safe because if Prometheus failed to get
the target list via HTTP request, it won't update its current target list to
empty, instead,it will keep using the current list.Prometheus caches target lists. If an error occurs while fetching an updated
targets list, Prometheus keeps using the current targets list.For the same reason, if there are 3 scripts under/targets/mysystemand only
one failed for a request, prometheus-http-sd will return a HTTP 500 Error for
the whole request instead of returning the partial targets from the other two
scripts.Also for the same reason, if your script met any error, you should throw outExceptionall the way to the top instead of catch it in your script and return
a nullTargetList, if you return a nullTargetList, prometheus-http-sd will
think that your script run successfully and empty the target list as well.You can notice this error from stdout logs or/metricsfrom
prometheus-http-sd.Best PracticeYou can use a git repository to manage your target generator.