Package: collectington
Welcome toCollectingtonCollectingtoncalls 3rd party APIs to gather data/metrics and translates those metrics forPrometheusto monitorHow does it work?This application (Python3) can run in any environment as it can be installed usingpipcommandpip install collectingtonOnce the service is running, the service will make an API call every 60 sec to read data (metrics) from 3rd party services. The read data must be translated based on the logic you provide soPrometheusis able to monitor.CollectingtonusesPrometheus-Clientand we have automated a lot of steps that require writing many lines of code.Collectingtonremoves the process of:Having to define logic for calling APIsInstantiating prometheus client class many timesMapping prometheus metric to each function using the clientCalculating metrics to avoid double countingCaching API call response to minimize requestsAll you need to useCollectingtonis a singlejsonconfig and a custom API class.How do I start usingCollectington?InstallCollectingtonusing the below commandpip install collectingtonCreate your configuration file.Linke.g.config.jsonHere's an example config file:{
    "api_call_intervals" : 2,
    "log_level" : "INFO",
    "services" : {
        "splunk" : {
            "service_class" : "SplunkApi",
            "service_module" : "splunk_api",
            "port" : 8000,
            "api_url" : "https://api.victorops.com/api-reporting/v2/incidents",
            "prometheus_metrics_mapping" : {
                "counter" : [
                "number_of_incidents"
                ],
                "summary" : [
                "time_taken_to_acknowledge",
                "time_taken_to_resolve"
                ]
            },
            "secret_file_path" : "./splunk",
            "api_key" : "",
            "api_id" : ""
        }
    }
}Let's take a closer look at this config fileport(required): the port you want to run your service.api_call_intervals(required): the interval between each API call in secondslog_level(required): level of logging you want. It is currently under development.services(required):There are a number of components to pay attention to. This requires a dictionary withkeybeing the name of yourservice.
Inside yourservicedictionary, below is a detailed explanation of what these areservice_class(required): this is the name of your service class that you create to define your metrics.service_module(required): this is the name of your module or filename that contains yourservice_classapi_url(required): this is the API endpoint that you want to get data from.prometheus_metrics_mapping(required):Defineprometheus_metrics_mappingin adictionaryPrometheusclient includes 4 metrics -Counter,Summary,Histogram, andGauge:TheCountertype is useful for tracking increasing metrics (i.e. number of calls)Gaugeshould be used when you want to track the exact value of a metric. We've setupcollectionto report theobserve()value of the metric when sending to prometheus.Summaryshould be used for recording statistics on aggregated values. If you have many metrics perAPI_CALL_INTERVALS, you can use summary type to get the<METRIC>_sum,<METRIC>_count, and<METRIC>_createdon prometheus for free.Histogramcan be used similarly asSummaryexcept it has different metrics aggregations (it groups by percentile buckets).Please refer to theofficial Prometheus client documentationadditional details.This instantiation step is automated from theCollection Serviceand can be configured from theconfigfile as below:e.g.number_of_instancesmetric will be auto-instantiated withCounterPROMETHEUS_METRICS_MAPPING = {
    'counter': ['number_of_incidents'],
    'summary': ['time_taken_to_acknowledge', 'time_taken_to_resolve']
}The rest of the config fields are completely optional. If you want to add any custom config to read from it, that is completely fine. The only optinal config that is built-in withCollectingtonis theprometheus_metric_labels. This fields is reserved forcollectingonfor its own use.prometheus_metric_labels(optional):This adds labels to the given metrics. Based on the above examples the labels could look like:"prometheus_metrics_mapping" : {
        "counter" : [
        "number_of_incidents"
        ],
        "summary" : [
        "time_taken_to_acknowledge",
        "time_taken_to_resolve"
        ]
    }Whenever a metric is defined with labels there are a couple of changes thatneedto be done or else thecollectionservice will error. These are as follow:The user-defined function in the*_api.pythat gets metrics with labels (i.e.get_number_of_incidents) will need to return a different type of datastructure. It should return alistofdictthat has every label as keys and one extra key that stores the actual metric value.i.e. for a metric that has labels["team_name", "stack"]the returned metric must look like below (even if it's only one element):[
    {
        "team_name": "some_label_value",
        "stack": "prd",
        "metric_value": 123
    },
    {
        "team_name": "some_label_value",
        "stack": "sbx",
        "metric_value": 34
    }
]Notice that this is different from the normal user-defined functions which return a single numeric value.There should be onedictfor each unique combination of label values; that meansteam_name = some_label_valueandstack = prdcannot have appear a second time (from above example) since they already have a metric value.The keys that correspond to the labels must match exactly (case-sensitive).The key for the metric can be anything, but must be there.You cannot use the@enable_delta_metricdecorator with labeled metrics, instead change your function to calculate the delta or change the metric type to begauge.Create an API Class.LinkYou will find an example API Class below. Let's take a look at a closer look at this file.ImportsThe first task is to importcollectingtonlibrary. You would needcollectington.configin order to read config properlyYou would also need to import certain class and functions fromcollectington.collectington_apifrom the libraryenable_delta_metric(optional): This is optional but will be useful as metrics data collected is cumulative. This will avoid double counting any metricregister_metric_class(required): This is required as you would have to register your own class toCollectington'smetric.register_metric(required): This allows you to register your metrics functions to be used/called by the main application.Class initThere are certain fields that are required to usecollectington.def __init__(self):
    super(SplunkApi, self).__init__()

    self.config = get_config("config.json")
    self.service_name = "splunk"
    ...self.config(required): you have to provide your config path by using theget_configfunctionself.service_name(required): you have to provide your service name that matches the service name in theconfig fileself.headers(optional): this is required if you need to sendheaderinformation.Collectingtonuses therequestslibrary so it works the same way.self.params(optional): this is required if you need to addparamsto your API URL. If you need custom params for each metric method. You can simply override it from a method.self.name_of_datastore(required): this is required as this will ensure that your API is cached and not making unnecessary calls for every metricThe rest of the init attributes will be different for each case. This example uses the custom utils functions to get credentials for its API key.Class Metric MethodsThis is the core of your application.Collectingtonallows you to only worry about how to call 3rd party APIs and get the metrics. The rest is automated byCollectington. You just need to simply define your metric methods and register it using the imported decorator.Here's how to implement Metrics in detailIf you decide to add another service -new_api, you must do the following.Create a new module - e.g.new_api.pyCreate a subclass calledNewApiwhich inherits from theCollectingtonApiclass.Ensure that it is using@register_metric_classclass decorator to ensure metrics defined in your class are going to be registered to be used.@register_metric_class
class NewApi(AbstractApi):CollectingtonApiis an abstract class that includes common implementations which can be used across most services - read data, cache data, instantiatePrometheusmetrics and etc.Since the logic for each metric for each 3rd party service is different, user of the class must implement custom metrics logics from the newly created subclass.You may also override common methods if you require custom implementations.You must define and implement all metrics defined from theconfige.g.def get_number_of_incidents(self):
    response = self.get_data_from_store(self.name_of_datastore)
    total_incidents= response['total']
    return total_incidentsOnce you define a metric, you must register your method using a decorator. This will ensure that the name of the metric that you defined in your config will be mapped to a correct method defined in this class.e.g.@register_metric("time_taken_to_resolve")
def get_time_taken_to_resolve(self):
    response = self.get_data_from_store(self.name_of_datastore)
    list_of_time_triggered_and_resolved = []

    ...(Optional) Use@enable_delta_metricdecoratorThis is to ensure that the metric you are sending toPromteheusis not double counted.For example, let's say an API returns1,000which is a YTD total number of incidents. If an API is queried again after a minute and returns1,100, without@enable_delta_metric, you are sending1,100to Prometheus. This will result in double counting and total number of incidents will become2,100.e.g.@register_metric("number_of_incidents")
@enable_delta_metric
def get_number_of_incidents(self):
    response = self.get_data_from_store(self.name_of_datastore)
    total_incidents = response["total"]
    return total_incidents@enable_delta_metricwill send the difference between the previous data (1,000) and the latest data (1,100) to ensure double counting is avoided.(Optional) Override_update_metricThis method is to determine whichPrometheusmethod will be used for each metric. If you need custom behaviour, you can override this method.e.g.if isinstance(p_instance, Counter):
    # inc is a method from Prometheus client
    p_instance.inc(
        service_metric_dict[metric]
    )Once the above steps are complete, you can run the below command to spin upCollectington.Collectingtonhas its own command line so you don't have to run the service as a python modulecton -s <SERVICE_NAME> -c <CONFIG_PATH>Please note that your service name must match the name defined in your config file.Example Service UsageWe have in fact created a working service as an example usingSplunkAPI. You can go to theexample directoryto see it.cdinside of theexampledirectory and run the below command to see results.cton -s splunk -c ./config.jsonPlease note that since this service requires API credentials and requires to read fromsplunkfile that contains the secrets, you have to create the file to run it locally in below foramtAPI_KEY="<YOUR_API_KEY>"
API_ID="<YOUR_API_ID>"This is the example service class....

from collectington.config import *
from collectington.collectington_api import (
    CollectingtonApi,
    enable_delta_metric,
    register_metric_class,
    register_metric,
)


@register_metric_class
class SplunkApi(CollectingtonApi):
    """
    This class is the main class for calling Splunk On-Call API to generate
    custom metrics to be read by Prometheus. This class inherits from an abstract class
    to use common methods.
    The Splunk API returns data in a JSON format which requires iteration to retrieve
    desired data:
        - The Number of Incidents
        - The Number of Incidents Per team
        - Meantime to Acknowledge
        - Meantime to Resolve
    This class includes 3 major processes:
        1. Call an API to retrive data
        2. Define each metric logic as a method
        3. Implementing abstract methods to be called from the main run
    """

    def __init__(self):
        super(SplunkApi, self).__init__()

        self.config = get_config("config.json")
        self.service_name = "splunk"
        self.api_url = self.config["services"][self.service_name]["api_url"]

        dict_of_credentials = get_credentials_from_secret_file(
            self.config["services"][self.service_name]["secret_file_path"]
        )

        self.api_id = dict_of_credentials.get("API_ID", "")
        self.api_key = dict_of_credentials.get("API_KEY", "")

        self.headers = {
            "X-VO-Api-Key": self.api_key,
            "X-VO-Api-Id": self.api_id,
        }

        self.params = {"startedAfter": get_iso_timestamp_x_min_ago(1)}
        self.name_of_datastore = "splunk_datastore"

    ...

    @register_metric("number_of_incidents")
    @enable_delta_metric
    def get_number_of_incidents(self):
        self.params = {}  # override params to get all time total figure

        response = self.get_data_from_store(self.name_of_datastore)
        total_incidents = response["total"]

        return total_incidents

    ...Development & ContributionSince this is an open source project, anyone is welcome to contribute towardsCollectingtonTo do development work on this project make sure you follow these steps.Have your localvirtualenvsetup and add tests for each service.Since we want to ensure that new features are runnable from running below,pip install collectingtonit is recommended that you create a local library to test installation and run the service.You can run the below command to build awhlfilepython setup.py sdist bdist_wheelOnce the whl file is created, you can runpip install <PATH_TO_YOUR_WHL_FILE>to install the package locally.TestingTo run all the tests, use the virtual environment and runpython -m unittest discover