Package: kafka-scrapy-connect
🚀 Kafka Scrapy ConnectOverviewkafka_scrapy_connectis a custom Scrapy library that integrates Scrapy with Kafka.It consists of two main components: spiders and pipelines, which interact with Kafka for message consumption and item publishing.The library also comes with a custom extension that publishes log stats to a kafka topic at EoD, which allows the user to analyse offline how well the spider is performing!This project has been motivated by the great work undertaken in:https://github.com/dfdeshom/scrapy-kafka.kafka_scrapy_connectutilisesConfluent'sKafka Python client under the hood, to provide high-level producer and consumer features.Features1️⃣Integration with Kafka📈Enables communication between Scrapy spiders and Kafka topics for efficient data processing.Through partitions and consumer groups message processing can be parallelised across multiple spiders!Reduces overhead and improves throughput by giving the user the ability to consume messages in batches.2️⃣Customisable Settings🛠️Provides flexibility through customisable configuration for both consumers and producers.3️⃣Error Handling🚑Automatically handles network errors during crawling and publishes failed URLs to a designated output topic.4️⃣Serialisation Customisation🧬Allows users to customize how Kafka messages are deserializsd by overriding the process_kafka_message method.InstallationYou can installkafka_scrapy_connectvia pip:pip install kafka_scrapy_connectExampleA fullexampleusing thekafka_scrapy_connectlibrary can be found inside the repo.Theonly prerequisite for walking through the example is the installation ofDocker🐳 .This is needed because a kafka cluster will be created locally using containers sokafka_scrapy_connectcan communicate with a broker.If all set, follow the below steps 👇Create a virtual environment, clone the repo and install requirements:python3-mvenv.venvsource.venv/bin/activate
gitclonehttps://github.com/spicyparrot/kafka_scrapy_connect.git&&cdkafka_scrapy_connect
pipinstall-rrequirements.txtCreate a local kafka cluster with required topics:bash./examples/kafka/kafka_start.sh--input-topicScrapyInput,1--output-topicScrapyOutput,1--error-topicScrapyError,1Initiate the spider:cdexamples/quotes&&scrapycrawlquotesPublish a message to the input kafka topic and watch the spider consume and process the mesasge 🪄☝️ This will require some custom producer code to publish messages or go tohttps://localhost:8080and use the UI to publish some example messages in! (The UI was created when bringing up your local kafka cluster)When satisfied with testing, exit the spider and clean up the local kafka cluster:bash./examples/kafka/kafka_stop.shUsageCustom Settingskafka_scrapy_connectsupports the following custom settings:SCRAPY_KAFKA_HOSTS- A list of kafka broker hosts. (Default:localhost:29092)SCRAPY_INPUT_TOPIC- Topic from which the spider[s]consumesmessages from. (Default:ScrapyInput)SCRAPY_OUTPUT_TOPIC- Topic where scraped items are published. (Default:ScrapyOutput)SCRAPY_ERROR_TOPIC- Topic for publishing URLs that failed due tonetwork errors. (Default:ScrapyError)SCRAPY_CONSUMER_CONFIG- Additional configuration options for Kafka consumers (seehere)SCRAPY_PRODUCER_CONFIG- Additional configuration options for Kafka producers (seehere)SCRAPY_KAFKA_PRODUCER- Key used for partitioning messages in Kafka producer (Default:""Roundrobin)SCRAPY_KAFKA_PRODUCER_CALLBACKS- Enable or disable asynchronous message delivery callbacks. (Default:False)CustomisationCustomising deserialisationYou can customize how Kafka messages are deserialized by overriding the process_kafka_message method in your spider class.This allows for handling custom message formats or data transformations.classCustomSpider(KafkaListeningSpider):defprocess_kafka_message(self,message,meta={},headers={}):# Custom deserialization logic# Return URL, metadata or None if extraction failspass⚠️By default, if no customprocess_kafka_messagemethod is provided, the spider methodprocess_kafka_messagewill expecta JSON payload or a string containing a valid url. If it's a JSON object, it expectsurlin the K/V pair.Customising Producer & Consumer settingsYou can customize producer and consumer settings by providing a dictionary of configuration options in your Scrapy settings underSCRAPY_PRODUCER_CONFIGandSCRAPY_CONSUMER_CONFIG.# Example SCRAPY_PRODUCER_CONFIGSCRAPY_PRODUCER_CONFIG={'compression.type':'gzip','request.timeout.ms':5000}# Example SCRAPY_CONSUMER_CONFIGSCRAPY_CONSUMER_CONFIG={'fetch.wait.max.ms':10,'max.poll.interval.ms':600000,'auto.offset.reset':'latest'}Custom stats extensionskafka_scrapy_connectcomes with a custom Scrapy stats extension that:logs basic scraping statistics every minute (frequency can be configured by the scrapy settingKAFKA_LOGSTATS_INTERVAL)Atend-of-day, will publish logging statistics to a Kafka topic (specified by the scrapy settingSCRAPY_STATS_TOPIC).Each summary message will be published with a key specifying the summary date (2024-02-27) for easy identification.If the spider is shutdown or closed, due to a deployment etc, a summary payload will also be sent to a kafka topic (SCRAPY_STATS_TOPIC)To enable this custom extension, disable the standard LogStats extension and modify yoursettings.pyto include the below:# Kafka topic for capturing stats
SCRAPY_STATS_TOPIC = 'ScrapyStats'

# Disable standard logging extension (use custom kafka_scrapy_connect extension)
EXTENSIONS = {
  "scrapy.extensions.logstats.LogStats": None,
  "kafka_scrapy_connect.extensions.KafkaLogStats": 500
}An example payload sent to the statistics topic will look like:{"pages_crawled":3,"items_scraped":30,"avg pages/min":0.4,"avg pages/hour":23.78,"avg pages/day":570.63,"avg items/min":3.96,"avg items/hour":237.76,"avg items/day":5706.3,"successful_request_pct":100.0,"http_status_counts":"200: 3","max_memory":76136448,"elapsed_time":454.23}