Package: cloudwatch-fluent-metrics
FluentMetricsIMPORTANT: When using unique stream IDs, you have the potential to create a large number of metrics. Please make sure to review thecurrent AWS CloudWatch Custom Metrics pricingbefore proceeding.OverviewFluentMetricsis an easy-to-use Python module that makes logging CloudWatch custom metrics a breeze. The goal is to provide a framework for logging detailed metrics with a minimal footprint. When you look at your code logic, you want to see your actual code logic, not line after line of metrics logging.FluentMetricslets you maximize your metrics footprint while minimizing your metrics code footprint.InstallationYou can install directly from PyPI:pipinstallcloudwatch-fluent-metrics'Fluent' . . . what is that?Fluent describes an easy-to-read programming style. The goal of fluent development is to make code easier to read and reduce the amount of code required to build objects. It's easier to take a look a comparison between fluent and non-fluent style.Non-Fluent Exampleg=Game()f=Frame(Name='Tom')f.add_score(7)f.add_score(3)g.add_frame(f)f=Frame(Name='Tom')f.add_strike()g.add_frame(f)Non-Fluent Example with Constructorg=Game()g.add_frame(Frame(Name='Tom',Score1=7,Score2=3)g.add_frame(Frame(Name='Tom',Score1=10)Fluent Exampleg=Game()g.add_frame(Frame().with_name('Tom').score(3).spare())g.add_frame(Frame().with_name('Tom').strike())While the difference may seem to be nitpicking, a frame is really just a constructed object. In the first example, we're taking up three lines of code to create the object--there's nothing wrong with that. However, in the second example, we're using constructors. This is slightly more readable, but there's a great deal of logic bulked up in our constructor. In the third example, we're using fluent-style code as it starts at creating the frame andfluentlycontinues until it's created the entire frame in a single line. And more importantly,it's readable.We're not just creating an object with a massive constructor or spending several lines of code just to create a single object.Terminology QuickstartNamespacesEvery metric needs to live in a namespace. Since you are logging your own custom metrics, you need to provide a custom namespace for your metric. Clickherefor a list of the standard AWS namespaces.Example:
In this example, we're creating a simpleFluentMetricin a namespace calledPerformance. This means that every time we log a metric withm, we will automatically log it to thePerformancenamespace.fromfluentmetricsimportFluentMetricm=FluentMetric().with_namespace('Performance')Metric NamesThe metric name is the thing you are actually logging. Each value that you log must be tied to a metric name. When you log a custom metric with a new metric name, the name will automatically be created if it doesn't already exist. Clickhereto see existing metrics that can help you define names for your custom metrics.Example:
In this example, we're logging two metrics calledStartupTimeandStuffTimeto thePerformancenamespace (we only needed to define the namespace once).m=FluentMetric().with_namespace('Performance')m.log(MetricName='StartupTime',Value=27,Unit='Seconds')do_stuff()m.log(MetricName='StuffTime',Value=12000,Unit='Milliseconds')ValuesObviously we need to log a value with each metric. This needs to be a number since we convert this value to afloatbefore sending to CloudWatch.IMPORTANT: When logging multiple values for the same custom metric within a minute, CloudWatch aggregates an average over a minute. Clickherefor more details.Storage ResolutionThe PutMetricData function now accepts an optional StorageResolution parameter. Set this parameter to 1 to publish high-resolution metrics; omit it (or set it to 60) to publish at standard 1-minute resolution.Example:
In this example, we're logging metric at one-second resolution:m=FluentMetric().with_namespace('Application/MyApp').with_storage_resolution(1)m.log(MetricName='Transactions/Sec',Value=trans_count,Unit='Count/Sec')```sh#### DimensionsAdimensiondefineshowyouwanttosliceanddicethemetric.Thesearesimplyname-valuepairsandyoucandefineupto10permetric.Click[here](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html#usingDimensions)formoredetailsonusingdimensions.
**IMPORTANT:**Whenyoudefinemultipledimensions,CloudMetricsattachesallofthosedimensionstothemetricasasinglecombineddimensionset--thinkofthemasanaggregateprimarykey.Forexample,ifyoulogametricwiththedimensions`os='linux'`and`flavor='ubunutu'`youwillonlybeabletoaggregateby**both**`os`and`flavor`.You**cannot**aggregateonlybyjust`os`orjust`flavor`.`FluentMetrics`solvesthisproblembyautomaticallyloggingthreemetrics--onefor`os`,onefor`flavor`andthenoneforthecombieddimensions,givingyoumaximumflexibility.
*Example*:
Inthisexample,we're logging boot/restart time metrics. When this code executes, we will end up with 6 metrics:* `BootTime` and `RestartTime` for `os`* `BootTime` and `RestartTime` for `instance-id`* `BootTime` and `RestartTime` for 'os`and`instance-id````shm=FluentMetric().with_namespace('Performance/EC2')\.with_dimension('os','linux').\.with_dimension('instance-id','i-123456')boot_time=start_instance()m.log(MetricName='BootTime',Value=boot_time,Unit='Milliseconds')restart_time=restart_instance()m.log(MetricName='RestartTime',Value=restart_time,Unit='Milliseconds')UnitsCloudWatch has built-in logic to provide meaning to the metric values. We're not just logging a value--we're logging a value of some unit. By defining the unit type, CloudWatch will know how to properly present, aggregate and compare that value with other values. For example, if you submit a value with unitMilliseconds, then it can properly aggregate it up to seconds, minutes or hours. This is a list of the most current valid list of units. A more up-to-date list should be availablehereunder theUnitsection,."Seconds"|"Microseconds"|"Milliseconds"|"Bytes"|"Kilobytes"|"Megabytes"|"Gigabytes"|"Terabytes"|"Bits"|"Kilobits"|"Megabits"|"Gigabits"|"Terabits"|"Percent"|"Count"|"Bytes/Second"|"Kilobytes/Second"|"Megabytes/Second"|"Gigabytes/Second"|"Terabytes/Second"|"Bits/Second"|"Kilobits/Second"|"Megabits/Second"|"Gigabits/Second"|"Terabits/Second"|"Count/Second"|"None"Unit Shortcut MethodsIf you don't want to type out the individual unit name, there are shortcut methods for each unit.m=FluentMetric().with_namespace('Performance/EC2')\.with_dimension('os','linux').\.with_dimension('instance-id','i-123456')m.seconds(MetricName='CompletionInSeconds',Value='1000')m.microseconds(MetricName='CompletionInMicroseconds',Value='1000')m.milliseconds(MetricName='CompletionInMilliseconds',Value='1000')m.bytes(MetricName='SizeInBytes',Value='1000')m.kb(MetricName='SizeInKb',Value='1000')m.mb(MetricName='SizeInMb',Value='1000')m.gb(MetricName='SizeInGb',Value='1000')m.tb(MetricName='SizeInTb',Value='1000')m.bits(MetricName='SizeInBits',Value='1000')m.kbits(MetricName='SizeInKilobits',Value='1000')m.mbits(MetricName='SizeInMegabits',Value='1000')m.gbits(MetricName='SizeInGigabits',Value='1000')m.tbits(MetricName='SizeInTerabits',Value='1000')m.pct(MetricName='Percent',Value='20')m.count(MetricName='ItemCount',Value='20')m.bsec(MetricName='BandwidthBytesPerSecond',Value='1000')m.kbsec(MetricName='BandwidthKilobytesPerSecond',Value='1000')m.mbsec(MetricName='BandwidthMegabytesPerSecond',Value='1000')m.gbsec(MetricName='BandwidthGigabytesPerSecond',Value='1000')m.tbsec(MetricName='BandwidthTerabytesPerSecond',Value='1000')m.bitsec(MetricName='BandwidthBitsPerSecond',Value='1000')m.kbitsec(MetricName='BandwidthKilobitsPerSecond',Value='1000')m.mbitsec(MetricName='BandwidthMegabitsPerSecond',Value='1000')m.gbitsec(MetricName='BandwidthGigabitsPerSecond',Value='1000')m.tbitsec(MetricName='BandwidthTerabitsPerSecond',Value='1000')m.countsec(MetricName='ItemCountsPerSecond',Value='1000')TimersOne of the most common uses of logging is measuring performance. FluentMetrics allows you to activate multiple built-in timers by name and log the elapsed time in a single line of code.NOTE:The elapsed time value is automatically stored as unitMilliseconds.Example:
In this example, we're starting timersworkflowandjob1at the same time. Timers start as soon as you create them and never stop running. When you callelapsed,FluentMetricswill log the number of elapsed milliseconds with theMetricName.m=FluentMetric()m.with_timer('workflow').with_timer('job1')do_job1()m.elapsed(MetricName='Job1CompletionTime',TimerName='job1')m.with_timer('job2')do_job2()m.elapsed(MetricName='Job2CompletionTime',TimerName='job2')finish_workflow()m.elapsed(MetricName='WorkflowCompletionTime',TimerName='workflow')Metric Stream IDA key feature ofFluentMetricsis the metric stream ID. This ID will be added as a dimension and logged with every metric. The benefit of this dimension is to provide a distinct stream of metrics for an end-to-end operation. When you create a new instance ofFluentMetric, you can either pass in your own value orFluentMetricswill generate a GUID. In CloudWatch, you can then see all of the metrics for a particular stream ID in chronological order. A metric stream can be a job, or a server or any way that you want to unique group a contiguous stream of metrics.Example:
In this example, we'll have two metrics in thePerformancenamespace, each with metric stream ID ofabc-123. We can then go to CloudWatch and filter by that stream ID to see the entire operation performance at a glance.m=FluentMetric().with_namespace('Performance').with_stream_id('abc-123')m.log(MetricName='StartupTime',Value=100,Unit='Seconds')do_work()m.log(MetricName='WorkCompleted',Value=1000,Unit='Milliseconds')Use Case Quickstart#1: Least Amount of Code Required to Log a MetricThis is the minimal amount of work you need to log--create aFluentMetricwith a namespace, then log a value.Result: This code will log a single value100forActiveServerCountin theStatsnamespace.fromfluentmetricsimportFluentMetricm=FluentMetric().with_namespace('Stats')m.log(MetricName='ActiveServerCount',Value='100',Unit='Count')#2: Logging Multiple Metrics to the Same NamespaceIf you are logging multiple metrics to the same namespace, this is a great use case forFluentMetrics. You only need to create one instance ofFluentMetricand specify a different metric name when you calllog.Result: This code will log a single value100forActiveServerCountin theStatsnamespace.fromfluentmetricsimportFluentMetricm=FluentMetric().with_namespace('Stats')m.log(MetricName='ActiveServerCount',Value='10',Unit='Count')\.log(MetricName='StoppedServerCount',Value='20',Unit='Count')\.log(MetricName='ActiveLinuxCount',Value='50',Unit='Count')\.log(MetricName='ActiveWindowsCount',Value='50',Unit='Count')#3: Logging CountsIn the previous example, we logged a metric and identified the unitCount. Instead of specifying the unit, you can specify the type of objectResult: This code will log a single value100forActiveServerCountin theStatsnamespace.fromfluentmetricsimportFluentMetricm=FluentMetric().with_namespace('Stats')m.count(MetricName='ActiveServerCount',Value='10')BufferedFluentMetricNormally, with FluentMetric, metrics are sent immediately whenlogis called (orcount,milliseconds, etc). This
can result in a lot ofput_metric_datacalls to CloudWatch that are not full. When you useBufferedFluentMetricinstead ofFluentMetric, it waits until it has the maximum (20) metrics before callingput_metric_data. This optimizes
traffic to cloudwatch.In general,BufferedFluentMetricbehaves identically toFluentMetric, except that now it is possible to "forget" to
send some metrics. TheBufferedFluentMetric.flush()method pushes out all metrics immediately (clears the buffer). It
is often best to do this at the end of a request (or some other obviously bounded interval).Here is an example of how it works in Flask:fromflaskimportgfromfluentmetricsimportBufferedFluentMetric@app.before_requestdefstart_request():g.metrics=BufferedFluentMetric()g.metrics.with_namespace('MyApp')g.metrics.with_timer('RequestLatency')@app.after_requestdefend_request(response):deferror_counter(hundred):ifresponse.status_code/100==hundred:return1else:return0g.metrics.count(MetricName='4xxError',Value=error_counter(400))g.metrics.count(MetricName='5xxError',Value=error_counter(500))g.metrics.count(MetricName='Availability',Value=(1-error_counter(500)))g.metrics.elapsed(MetricName='RequestLatency',TimerName='RequestLatency')# Finally, ensure that all metrics end up in CloudWatch before this request finally ends.g.metrics.flush()LicenseThis library is licensed under the Apache 2.0 License.