Package: netflix-spectator-py
IntroductionPython thin-client metrics library for use withAtlasandSpectatorD.Supports Python >= 3.5. This version is chosen as the baseline, because it is the oldest system
Python available in our operating environments.Local DevelopmentInstallpyenv, possibly withHomebrew, and
install a recent Python version.makesetup-venv
maketestmakecoverageUsageInstallingInstall this library for your project as follows:pip3 install netflix-spectator-pyPublishing metrics requires aSpectatorDprocess running on your instance.ImportingStandard UsageAt Netflix, your initialization script should load the environment, to ensure that the standard
variables are available to the Python application.source/etc/nflx/environmentImporting theGlobalRegistryinstantiates aRegistrywith a default configuration that applies
process-specific common tags based on environment variables and opens a socket to theSpectatorDsidecar. The remainder of the instance-specific common tags are provided by SpectatorD.fromspectatorimportGlobalRegistryOnce theGlobalRegistryis imported, it is used to create and manage Meters.LoggingThis package provides the following loggers:spectator.MeterIdspectator.SidecarWriterTheMeterIdlogger is used to report invalid meters which have not-a-str tag keys or values.When troubleshooting metrics collection and reporting, you should set theSidecarWriterlogging
to theDEBUGlevel, before the first metric is recorded. For example:importlogging# record the human-readable time, name of the logger, logging level, thread id and messagelogging.basicConfig(level=logging.DEBUG,format='%(asctime)s-%(name)s-%(levelname)s-%(thread)d-%(message)s')logging.getLogger('spectator.SidecarWriter').setLevel(logging.DEBUG)There is approximately a 10% performance penalty in UDP write performance when debug logging is
enabled. It may be more, depending on the exact logging configuration (i.e. flushing to slow disk).Working with IDsThe IDs used for identifying a meter in theGlobalRegistryconsist of a name and a set of tags.
IDs will be consumed by users many times after the data has been reported, so they should be
chosen thoughtfully, while considering how they will be used. See thenaming conventionspage
for general guidelines.IDs are immutable, so they can be freely passed around and used in a concurrent context. Tags can
be added to an ID when it is created, to track the dimensionality of the metric.All tag keys
and values must be strings.For example, if you want to keep track of the number of successful
requests, you must cast integers to strings.fromspectatorimportGlobalRegistryrequests_id=GlobalRegistry.counter("server.numRequests",{"statusCode":str(200)})requests_id.increment()Meter TypesAge GaugesThe value is the time in seconds since the epoch at which an event has successfully occurred, or0to use the current time in epoch seconds. After an Age Gauge has been set, it will continue
reporting the number of seconds since the last time recorded, for as long as the SpectatorD
process runs. The purpose of this metric type is to enable users to more easily implement the
Time Since Last Success alerting pattern.To set a specific time as the last success:fromspectatorimportGlobalRegistryGlobalRegistry.age_gauge("time.sinceLastSuccess").set(1611081000)To setnow()as the last success:fromspectatorimportGlobalRegistryGlobalRegistry.age_gauge("time.sinceLastSuccess").set(0)By default, a maximum of1000Age Gauges are allowed perspectatordprocess, because there is no
mechanism for cleaning them up. This value may be tuned with the--age_gauge_limitflag on thespectatordbinary.CountersA Counter is used to measure the rate at which an event is occurring. Considering an API
endpoint, a Counter could be used to measure the rate at which it is being accessed.Counters are reported to the backend as a rate-per-second. In Atlas, the:per-stepoperator
can be used to convert them back into a value-per-step on a graph.Callincrement()when an event occurs:fromspectatorimportGlobalRegistryGlobalRegistry.counter("server.numRequests").increment()You can also pass a value toincrement(). This is useful when a collection of events happens
together:fromspectatorimportGlobalRegistryGlobalRegistry.counter("queue.itemsAdded").increment(10)Distribution SummariesA Distribution Summary is used to track the distribution of events. It is similar to a Timer, but
more general, in that the size does not have to be a period of time. For example, a Distribution
Summary could be used to measure the payload sizes of requests hitting a server.Always use base units when recording data, to ensure that the tick labels presented on Atlas graphs
are readable. If you are measuring payload size, then use bytes, not kilobytes (or some other unit).
This means that a4Ktick label will represent 4 kilobytes, rather than 4 kilo-kilobytes.Callrecord()with a value:fromspectatorimportGlobalRegistryGlobalRegistry.distribution_summary("server.requestSize").record(10)Percentile Distribution SummariesThe value tracks the distribution of events, with percentile estimates. It is similar to a
Percentile Timer, but more general, because the size does not have to be a period of time.For example, it can be used to measure the payload sizes of requests hitting a server or the
number of records returned from a query.In order to maintain the data distribution, they have a higher storage cost, with a worst-case of
up to 300X that of a standard Distribution Summary. Be diligent about any additional dimensions
added to Percentile Distribution Summaries and ensure that they have a small bounded cardinality.Callrecord()with a value:fromspectatorimportGlobalRegistryGlobalRegistry.pct_distribution_summary("server.requestSize").record(10)GaugesA gauge is a value that is sampled at some point in time. Typical examples for gauges would be
the size of a queue or number of threads in a running state. Since gauges are not updated inline
when a state change occurs, there is no information about what might have occurred between samples.Consider monitoring the behavior of a queue of tasks. If the data is being collected once a minute,
then a gauge for the size will show the size when it was sampled. The size may have been much
higher or lower at some point during interval, but that is not known.Callset()with a value:fromspectatorimportGlobalRegistryGlobalRegistry.gauge("server.queueSize").set(10)Gauges will report the last set value for 15 minutes. This done so that updates to the values do
not need to be collected on a tight 1-minute schedule to ensure that Atlas shows unbroken lines in
graphs. A custom TTL may be configured for gauges. SpectatorD enforces a minimum TTL of 5 seconds.fromspectatorimportGlobalRegistryGlobalRegistry.gauge("server.queueSize",ttl_seconds=120).set(10)TimersA Timer is used to measure how long (in seconds) some event is taking.Callrecord()with a value:fromspectatorimportGlobalRegistryGlobalRegistry.timer("server.requestLatency").record(0.01)Astopwatch()method is available which may be used as aContext Managerto automatically record the number of seconds that have elapsed while executing a block of code:importtimefromspectatorimportGlobalRegistryt=GlobalRegistry.timer("thread.sleep")witht.stopwatch():time.sleep(5)Internally, Timers will keep track of the following statistics as they are used:counttotalTimetotalOfSquaresmaxPercentile TimersThe value is the number of seconds that have elapsed for an event, with percentile estimates.This metric type will track the data distribution by maintaining a set of Counters. The
distribution can then be used on the server side to estimate percentiles, while still
allowing for arbitrary slicing and dicing based on dimensions.In order to maintain the data distribution, they have a higher storage cost, with a worst-case of
up to 300X that of a standard Timer. Be diligent about any additional dimensions added to Percentile
Timers and ensure that they have a small bounded cardinality.Callrecord()with a value:fromspectatorimportGlobalRegistryGlobalRegistry.pct_timer("server.requestLatency").record(0.01)Astopwatch()method is available which may be used as aContext Managerto automatically record the number of seconds that have elapsed while executing a block of code:importtimefromspectatorimportGlobalRegistryt=GlobalRegistry.pct_timer("thread.sleep")witht.stopwatch():time.sleep(5)asyncio SupportTheGlobalRegistryprovides aUdpWriterimplementation of theSidecarWriterby default. UDP
is a non-blocking, unordered and connectionless protocol, which is ideal for communicating with a
local SpectatorD process in a variety of circumstances. TheUdpWritershould be used in asyncio
applications.ThePrintWriterimplementation, which can be used to communicate with the SpectatorD Unix domain
socket, does not offer asyncio support at this time.IPv6 SupportBy default, SpectatorD will listen onIPv6 UDP *:1234, without setting thev6_only(true)flag. On dual-stacked systems, this means that it will receive packets from both IPv4 and IPv6,
and the IPv4 addresses will show up on the server as IPv4-mapped IPv6 addresses.By default, theGlobalRegistrywill write UDP packets to127.0.0.1:1234, which will allow
for communication with SpectatorD on dual-stacked systems.On IPv6-only systems, it may be necessary to change the default configuration using one of the
following methods:Configure the following environment variable, which will override the default configuration of
theGlobalRegistry:export SPECTATOR_OUTPUT_LOCATION="udp://[::1]:1234"Configure a custom Registry, instead of using theGlobalRegistry:from spectator import Registry
from spectator.sidecarconfig import SidecarConfig

r = Registry(config=SidecarConfig({"sidecar.output-location": "udp://[::1]:1234"}))
r.counter("test").increment()Writing TestsTo write tests against this library, instantiate a test instance of the Registry and configure it
to use theMemoryWriter,
which stores all updates in a List. Use thewriter()method on the Registry to access the writer,
then inspect thelast_line()orget()all messages to verify your metrics updates.importunittestfromspectatorimportRegistryfromspectator.sidecarconfigimportSidecarConfigclassMetricsTest(unittest.TestCase):deftest_counter(self):r=Registry(config=SidecarConfig({"sidecar.output-location":"memory"}))c=r.counter("test")self.assertTrue(r.writer().is_empty())c.increment()self.assertEqual("c:test:1",r.writer().last_line())If you need to override the default output location (udp) of theGlobalRegistry, then you can
set aSPECTATOR_OUTPUT_LOCATIONenvironment variable to one of the following values supported
by theSidecarConfigclass:none- Disable output.memory- Write to memory.stdout- Write to standard out for the process.stderr- Write to standard error for the process.file://$path_to_file- Write to a file (e.g.file:///tmp/foo/bar).udp://$host:$port- Write to a UDP socket.If you want to disable metrics publishing from theGlobalRegistry, then you can set:exportSPECTATOR_OUTPUT_LOCATION=noneIf you want to validate the metrics that will be published through theGlobalRegistryin tests, then you can set:exportSPECTATOR_OUTPUT_LOCATION=memoryTheMemoryWritersubclass offers a few methods to inspect the values that it captures:clear()- Delete the contents of the internal list.get()- Return the internal list.is_empty()- Is the internal list empty?last_line()- Return the last element of the internal list.Lastly, a SpectatorD line protocol parser is available, which is intended to be used for validating
the results captured by aMemoryWriter. It may be used as follows:importunittestfromspectator.counterimportCounterfromspectator.protocolparserimportparse_protocol_lineclassProtocolParserTest(unittest.TestCase):deftest_parse_counter_with_multiple_tags(self):meter_class,meter_id,value=parse_protocol_line("c:test,foo=bar,baz=quux:1")self.assertEqual(Counter,meter_class)self.assertEqual("test",meter_id.name)self.assertEqual({"foo":"bar","baz":"quux"},meter_id.tags())self.assertEqual("1",value)Migrating from 0.1.X to 0.2.XThis library no longer publishes directly to the Atlas backends. It now publishes to theSpectatorDsidecar which is bundled with all standard AMIs and containers. If you must
have the previous direct publishing behavior, because SpectatorD is not yet available on the
platform where your code runs, then you can pin to version0.1.18.The internal Netflix configuration companion library is no longer required and this dependency
may be dropped from your project.The API surface area remains unchanged to avoid breaking library consumers, and standard uses ofGlobalRegistryhelper methods for publishing metrics continue to work as expected. Several helper
methods on meter classes are now no-ops, always returning values such as0ornan. If you want
to write tests to validate metrics publication, take a look at the tests in this library for a few
examples of how that can be done. The core idea is to capture the lines which will be written out
to SpectatorD.Replace uses ofPercentileDistributionSummarywith direct use of the Registrypct_distribution_summarymethod.# before
from spectator import GlobalRegistry
from spectator.histogram import PercentileDistributionSummary

d = PercentileDistributionSummary(GlobalRegistry, "server.requestSize")
d.record(10)# after
from spectator import GlobalRegistry

GlobalRegistry.pct_distribution_summary("server.requestSize").record(10)Replace uses ofPercentileTimerwith direct use of the Registrypct_timermethod.# before
from spectator import GlobalRegistry
from spectator.histogram import PercentileTimer

t = PercentileTimer(GlobalRegistry, "server.requestSize")
t.record(0.01)# after
from spectator import GlobalRegistry

GlobalRegistry.pct_timer("server.requestSize").record(0.1)Implemented new meter types supported bySpectatorD:age_gauge,max_gaugeandmonotonic_counter. See the SpectatorD documentation or the class docstrings for
more details.