Package: hoststats
hoststatshoststatscaptures resource usage (CPU, memory, network, disk) on a set of
remote hosts or containers over a period of time, then lets you aggregate and
filter the results using Pandas.Usage:Start collection from a client host via HTTP orkubectlusing the Python
client, or directly over HTTP.Stop collection, at which pointhoststatspulls the results from all hosts
into a single CSV file on the client machine.Usehoststatsto load, aggregate and filter data from this CSV using
Pandas.ThehoststatsserverThehoststatsserver must run on each host/ container from which you wish to
collect metrics, and port5000must be accessible to the client if you wish to
use HTTP.Using Docker# Run container in backgrounddockerrun-d--rm-p5000:5000shillaker/hoststats:0.0.5# Checkcurlhttp://localhost:5000/pingUsing Pip# Installpip3installhoststats# Start the server in the background, e.g.nohuphoststatsstart>/var/log/hoststats.log2>&1&# Checkcurlhttp://localhost:5000/pingThehoststatsclientThehoststatsclient host can accesshoststatsservers in two ways.Python APIfromhostats.clientimportHostStats# Create list of IPs/ hostnames for target hostsip_list=["1.2.3.4","5.6.7.8"]# Set up the clienths=HostStats(ip_list)# Start collectionhs.start_collection()# Wait some time# Write stats to CSVhs.stop_and_write_to_csv("hoststats.csv")Directly via HTTP APINote that although the HTTP API works, the data that comes out requires more
processing.# Check a given host is running the server and accessiblecurlhttp://<target_host>:5000/ping# Start the recordingcurlhttp://<target_host>:5000/start# Wait some time# Get stats as JSONcurlhttp://<target_host>:5000/stop>/tmp/hoststats.jsonProxiesIf your client host can't directly access the target hosts, you can specify a
proxy server, which must also have thehoststatsserver running. This proxy
can also be included in the list of target hosts.To use a proxy, you just need to provide an extra argument to theHostStatsconstructor:fromhostats.clientimportHostStats# List of IPs/ hostnames accessible from the proxyip_list=["1.2.3.4","5.6.7.8"]# Proxy IP/ hostname accessible from the clientproxy_ip="9.8.7.6"# Set up the clienths=HostStats(ip_list,proxy=proxy_ip)KubectlIf running in a Kubernetes cluster, you can usekubectlrather than HTTP. This
is useful if your client host is outside the cluster and you don't have direct
ingress to each container you're gathering stats from.To usekubectl, you need to pass certain arguments to theHostStatsconstructor, and the list of hosts should be a list of pod names:fromhostats.clientimportHostStats# List of pod namespods=["pod-a","pod-b","pod-c"]# Set up the clienths=HostStats(pods,kubectl=True,kubectl_ns="my-namespace",# K8s namespace (optional)kubectl_container="my-container",# Container name within the pod (optional))Handling resultsIf the data has been written to CSV via the Python API, you can access the data
with theHostStatsResultsclass:fromhoststats.resultsimportHostStatsResultscsv_file="hoststats.csv"s=HostStatsResults(csv_file)# Get list of hosts in resultss.get_hosts()# Get list of available statss.get_stats()# Get timeseries of given stat per hosts.get_stat_per_host("CPU_PCT")# Get average stat across hostss.get_avg_stat("MEMORY_USED")DevelopmentEnsurepipandsetuptoolsare up to date and install requirements.To develop:pip3install-e.Run tests:./bin/tests.shDeveloping on a local clusterIf you want to run distributed tests against your local modifications, you can
run the following:# Start up some hoststats containers and enter the client container./bin/dev.shFrom within this container, run tests:# Non-distributed testsnosetestshoststats.tests--nocapture# Distributed testsnosetestshoststats.disttest--nocaptureYou can then edit files and restart the target containers with:./bin/dev_restart.shOnce restarted, you can rerun the tests against servers with your changes.See the scripts mentioned above anddocker-compose-dev.ymlfor more info.ReleasingTo push to PyPI, make sure you have set upTwine keyring
support, or apypirc.Then increment the version inVERSION.Then:# Tag the code./bin/tag.sh# Build the Docker image./bin/build.sh# Check the distributed tests passs./bin/dist_tests.sh# Push the package./bin/release.shOnce everything looks good, create a release manuallyon
Github.After that, merge the PR, then retag from the master branch.