Package: websites-metrics-collector
Websites Metrics CollectorAuthor: Antonio Di Mariano -antonio.dimariano@gmail.comDescriptionThis service is part of the Websites Monitoring Application, a small distributed application that periodically produces and collects metrics about the availability of one or more websites.
This service is responsible for fetching and collecting the information from a list of URLs. The information collected isHTTP Status returnedHTTP response timeregexp pattern that is expected to be found on the HTML contentFor each record, a message is produced to an Apache Kafka Topic. This service exposes a REST API Service with aPOSTmethod to accept a list of URLs to fetch and process.Infra RequirementsThis service requires an Apache Kafka Broker with the Schema Registry and AVRO support.
In theterraformdirectory you will find the necessary scripts to create a token and a schema registry on Aiven (https://aiven.com)If you want to automate the creation of topics and schemas, you can use Terraform. Otherwise, you can do it yourself from the console.Steps to follow to use terraform:Install terraform (https://www.terraform.io/downloads.html)Edit the terraform.tfvars. You will need an API key from the Aiven console. Please follow instructions inhttps://help.aiven.io/en/articles/2059201-authentication-tokensto paste into the terraform.tfvars file.then run$terraforminit
$terraformplan
$terraformapply("init" sets up the working directory and downloads the Aiven provider, "plan" shows you what changes will be made and "apply" actually makes the changes.)
For the full documentation of Aiven Terraform gohttps://github.com/aiven/terraform-provider-aiven/blob/master/docs/index.md.
Go here for a lot of exampleshttps://github.com/aiven/terraform-provider-aiven/tree/master/examplesManually create your own Avro SchemaThe default topicwebsites_metricshas the following schemaswebsites_metrics-values{"type":"record","name":"example","namespace":"example","fields":[{"type":"string","name":"url","doc":"The checked URL"},{"type":"int","name":"http_status","doc":"The HTTP status "},{"type":"float","name":"elapsed_time","doc":"The elapsed_time of the request "},{"type":"boolean","name":"pattern_verified","doc":""}],"doc":"example"}websites_metrics-key{"doc":"example","fields":[{"name":"service_name","type":"string"}],"name":"example","namespace":"example","type":"record"}RequirementsPython >=3.8RunIf you want to run from the source code, go to the directory websites_metrics_collector and runpython mainIf you want to use it as package (suggested method) install pip3 install websites_metrics_collectorThenSet the ENV variables as show in this README.mdThen use it this waydefstart_service():fromwebsites_metrics_collector.mainimportstartstart()Dependenciesrequests==2.26.0confluent-kafka-producers-wrapper==0.0.6  ( More information onhttps://github.com/antoniodimariano/confluent_kafka_producers_wrapper)aiohttp==3.7.4.post0Run testNOTE: for simplicity, I am assuming you have a running Kafka broker for testing purposespython -m unittest tests/test_driver_class.pypython -m unittest test/test_fetching_info_from_websites.pypython -m unittest test/test_kafka_producer.pypython -m unittest test/test_patterns_in_text.pypython -m unittest test/test_rest_server.pyNOTEIn order to run thetest_rest_apiyou have to start the service firstRest APIsMethod to get metrics from a list of URLs aPOSThttp://SERVER\_URI:SERVER\_PORT/api/v1/websites_metricsHeader FieldsHEADER FIELDVALUEContent-TypeRequired. application/jsonPayload FieldsA concatenated list with the following format[['http://urltofetch.com'],['optional 1st pattern to verify in the html','optional 2nd pattern to verify in the html',''optional Nth pattern to verify in the html']]Example[["http://cloudbased.me",["Microservices","Antonio"]],["http://ferrari.com",["ferrari","italia"]]]ResponsesHTTP Status Code : 201{"urls":[["http://cloudbased.me",["Microservices","Antonio"]],["http://ferrari.com",["ferrari","italia"]]],"submitted":true}HTTP Status Code : 400 Bad or Wrong payloadResponse:{"message":"Bad format! You need to send me a list of URLS [['http://urltofetch.com'],['optional 1st pattern to verify in the html','optional 2nd pattern to verify in the html',''optional Nth pattern to verify in the html']]. See the example.","example":[["http://cloudbased.me",["Microservices","Antonio"]],["http://ferrari.com",["ferrari","italia"]],["http://motoguzzi.com",["italia"]]]}HTTP Status Code : 403 An error occurred.Response{"message":"An Error occurred and the request or the payload cannot be processed. =..= No squealing, remember that it's all in your head =..="}Service ENV configurationIf you are using a Broker that usesSSL, like Aiven.com, you need to download the following certificates files and
copy them to a folderAccess KeyAccess CertificateCA Certificatefor more information about SSL and Kafka please readhttps://docs.confluent.io/3.0.0/kafka/ssl.htmlENV VariableVALUEDESCRIPTIONSERVICE_HOSTstringOptional. The hostname where to bind the REST server. The default value is0.0.0.0SERVICE_LISTEN_PORTstringOptional. The port where to bind the REST server. The default value is8080brokersstringRequired. The FQDN of the Apache Kafka Brokers.schema_registrystringRequired. The FQDN of the Service Registry.ssl_ca_locationstringRequired. he relative path to your ca.pemssl_certificate_locationstringRequired. The relative path to your service.certssl_key_locationstringRequired. The relative path to your service.keysecurity_protocolstringRequired. SSL.topic_for_producing_metricsstringOptional. The Kafka Avro Topic where to produce messages to. The default value iswebsites_metricslogging_levelstringOptional. The level of logging to use for the built-inloggingpackage. The default islogging.INFOExample of mandatory ENV variables to usebrokers=kafka-xxx-yyy-abc.com:12026schema_registry=https://user:password@kafka-xxx-yyy-abc.com:12029ssl_ca_location=configuration/ca.pemssl_certificate_location=configuration/service.certssl_key_location=configuration/service.keysecurity_protocol=SSLpersistence_conf=postgres://user:password@mydbhost.com:12024/metrics?sslmode=requireIf your broker requires SASL authentication, like Confluent Cloud, these are the ENV variables to includeENV VariableVALUEDESCRIPTIONSERVICE_HOSTstringOptional. The hostname where to bind the REST server. The default value is0.0.0.0SERVICE_LISTEN_PORTstringOptional. The port where to bind the REST server. The default value is8080brokersstringRequired. The FQDN of the Apache Kafka Brokers.schema_registrystringRequired. The FQDN of the Service Registry.sasl_usernamestringRequired. YOUR USERNAME HEREsasl_passwordstringRequired. YOUR PASSWORD HEREschema_registry_basic_auth_user_infostringRequired. AUTH HEREschema_registry_basic_auth_credentials_sourcestringRequired. USER_INFO.sasl_mechanismsstringRequired. PLAIN.security_protocolstringRequired. SASL_SSL.persistence_confstringRequired. The full string for connecting to your PostgreSQL database.topic_for_producing_metricsstringOptional. The Kafka Avro Topic where to produce messages to. The default value iswebsites_metricslogging_levelstringOptional. The level of logging to use for the built-inloggingpackage. The default islogging.INFO