Package: websites-metrics-consumer
Websites Metrics ConsumerAuthor: Antonio Di Mariano -antonio.dimariano@gmail.comDescriptionThis service is part of the Websites Monitoring Application, a small distributed application
that aims to produce and collect metrics about the availability of one or more websites.
This service is responsible for consuming messages about metrics being produced to an Apache Kafka Avro Topic by a different service.
The main action of this service is to store the incoming message into a PostgreSQL database.Overview of the Websites Monitoring ApplicationThe application is made of three services that can run in different systems.There are main two services.The first onehttps://github.com/antoniodimariano/websites_metrics_collectoris responsible for fetching and collecting the information from a list of URLs. The information collected isHTTP Status returned HTTP response time regexp pattern that is expected to be found on the HTML content.
For each record, a message is produced to an Apache Kafka Topic. This service exposes a REST API Service with a POST method to accept a list of URLs to fetch and process.The second service ishttps://github.com/antoniodimariano/metrics_consumerthat is responsible for consuming messages about metrics being produced to an Apache Kafka Avro Topic by a different service. The main action of this service is to store the incoming message into a PostgreSQL database.The last one is a Celery Beat based servicehttps://github.com/antoniodimariano/crontab_requestthat periodically send a POST request with a list of URLS to thewebsites_metrics_collectorSo, to run all the whole application,  you don't need to clone the three repos,  You can use these two public python packageshttps://pypi.org/project/websites-metrics-consumer/https://pypi.org/project/websites-metrics-collector/Create twopythonapplications. One will consume messagespip3 install websites-metrics-consumerThe other will produce metricspip3 install websites-metrics-collectorIn order to produce metrics, thehttps://github.com/antoniodimariano/websites_metrics_collectorruns a REST Server with aPOST/api/v1/websites_metricsendpoint that accepts a list of URLs to fetch.
For the complete documentation go herehttps://github.com/antoniodimariano/websites_metrics_collector/blob/master/README.mdThe last application (https://github.com/antoniodimariano/crontab_request) uses Celery Beat to periodically run the task of reading a list or URLs from a localjsonfile and will send it to as payload.
of thePOSTrequest to/api/v1/websites_metricsIt requiresRedisas broker.You can decide not to usehttps://github.com/antoniodimariano/crontab_requestand implements your own way of requesting a list or URLs to monitor.
As long as you send a POST Request to the endpoint/api/v1/websites_metricsmetrics will be collected, messages will be produced and data will be stored.
I decided to use Celery and not, for instance, a simple timer implemented with Thread or 3-party libs because Celery is robust, scalable and production ready.
I know it comes at the price of having a broker, but I prefer to pay a small price for a significant advance. Not to mention, I am a big fan of Celery!How to Run this serviceIf you want to run from the source code, go to the directorywebsites_metrics_consumerand runpython mainIf you want to use it as package (suggested method) installpip3 install websites_metrics_consumerThenSet the ENV variables as show in this README.mdThen use it this waydefconsume_message():fromwebsites_metrics_consumer.mainimportstart_consumerstart_consumer()RequirementsPython >=3.8Dependenciesconfluent_kafka==1.7.0psycopg2==2.9.1Run testNOTE: for simplicity I am assuming you have a running local instance of PostgreSQL with the following informationdatabase='test'user='postgresql'password='test123'host='localhost'python -m unittest tests/test_db_operations.pypython -m unittest test/test_events_handler_class.pyService ENV configurationIf you are using a Broker that usesSSL, like Aiven.com, you need to download the following certificates files and copy them to a folderAccess KeyAccess CertificateCA Certificatefor more information about SSL and Kafka please readhttps://docs.confluent.io/3.0.0/kafka/ssl.htmlExample of your project foldermyprojectname/
|------configuration/
            __init__.py
            ca.pem
            service.cert
            service.key
|------venv
|main.py
|requirements.txtENV VariableVALUEDESCRIPTIONbrokersstringRequired. The FQDN of the Apache Kafka Brokers.schema_registrystringRequired. The FQDN of the Service Registry.ssl_ca_locationstringRequired. he relative path to your ca.pemssl_certificate_locationstringRequired. The relative path to your service.certssl_key_locationstringRequired. The relative path to your service.keysecurity_protocolstringRequired. SSL.persistence_confstringRequired. The full string for connecting to your PostgreSQL database.group_idstringOptional. The consumer group id to assign to the consumer(s) . The default value ismetrics_consumerconsumer_topicsstring or listOptional. The Kafka Avro Topic where to consume messages from. The defaul value iswebsites_metricslogging_levelstringOptional. The level of logging to use for the built-inloggingpackage. The default islogging.INFOExample of mandatory ENV variables to usebrokers=kafka-xxx-yyy-abc.com:12026schema_registry=https://user:password@kafka-xxx-yyy-abc.com:12029ssl_ca_location=configuration/ca.pemssl_certificate_location=configuration/service.certssl_key_location=configuration/service.keysecurity_protocol=SSLpersistence_conf=postgres://user:password@mydbhost.com:12024/metrics?sslmode=requireIf your broker requires SASL authentication, like Confluent Cloud, these are the ENV variables to includeENV VariableVALUEDESCRIPTIONbrokersstringRequired. The FQDN of the Apache Kafka Brokers.schema_registrystringRequired. The FQDN of the Service Registry.sasl_usernamestringRequired. YOUR USERNAME HEREsasl_passwordstringRequired. YOUR PASSWORD HEREschema_registry_basic_auth_user_infostringRequired. AUTH HEREschema_registry_basic_auth_credentials_sourcestringRequired. USER_INFO.sasl_mechanismsstringRequired. PLAIN.security_protocolstringRequired. SASL_SSL.persistence_confstringRequired. The full string for connecting to your PostgreSQL database.group_idstringOptional. The consumer group id to assign to the consumer(s) . The default value ismetrics_consumerconsumer_topicsstring or listOptional. The Kafka Avro Topic where to consume messages from. The defaul value iswebsites_metricslogging_levelstringOptional. The level of logging to use for the built-inloggingpackage. The default islogging.INFO