Package: estela-requests
Estela RequestsIntroductionEstela Requests is a Python library that provides enhanced functionality for making HTTP requests and seamlessly integrates withestela, an open-source project that implements a platform for running spiders in-house or in the cloud, you can learn more about ithere. This documentation provides a comprehensive overview of the Estela Requests library, installation instructions, and useful usage examples.Table of ContentsInstallationUsageBasic usageExtend Estela RequestsMoreInstallationTo install Estela Requests, you can use pip, the Python package manager. Open your terminal or command prompt and run the following command:pipinstallestela-requests@git+https://github.com/bitmakerla/estela-requests.gitAlso, you can clone the repository and install from there, running the following command:gitclonegit@github.com:bitmakerla/estela-requests.gitcdestela-requests
pipinstall-e.UsageBasic UsageHere's an example of how to use Estela Requests to scrape the sitehttp://quotes.toscrape.comand send items to Estela:frombs4importBeautifulSoupfromestela_requestsimportEstelaRequestsfromestela_requests.estela_hubimportEstelaHubfromurllib.parseimporturljoinwithEstelaRequests.from_estela_hub(EstelaHub.create_from_settings())asrequests:spider_name="quotes_toscrape"# Send a GET request to the websitedefparse_quotes(url):response=requests.get(url)# Parse the HTML content using BeautifulSoupsoup=BeautifulSoup(response.content,"html.parser")# Extract the desired information from the parsed HTMLquotes=[]forquoteinsoup.find_all("div",class_="quote"):text=quote.find("span",class_="text").textauthor=quote.find("small",class_="author").texttags=[tag.textfortaginquote.find_all("a",class_="tag")]quotes.append({"text":text,"author":author,"tags":tags})# Print the extracted informationforquoteinquotes:item={"quote":quote["text"],"author":quote["author"],"tags":','.join(quote["tags"]),}requests.send_item(item)try:next=soup.find("li",class_="next").find("a").get("href")exceptAttributeError:next=Noneifnext:parse_quotes(urljoin(url,next))if__name__=="__main__":parse_quotes("http://quotes.toscrape.com/")First we need to import theEstelaRequestsandEstelaHubclasses:fromestela_requestsimportEstelaRequestsfromestela_requests.estela_hubimportEstelaHubOnce imported, you can create aEstelaRequestscontext manager:withEstelaRequests.from_estela_hub(EstelaHub.create_from_settings())asrequests:To assign a name for the spider in estela you should declare thespider_namewith the desired name, e.g.spider_name="quotes_toscrape"Finally if you want to yield items you should usesend_itemmethod:requests.send_item(item)Extend Estela Requests (BETA)Estela Requests can be easily customized by creating a settings.py file in the directory where you run your code:importloggingfromestela_requests.request_interfacesimportRequestsInterfacefromestela_queue_adapter.get_interfaceimportget_producer_interfacefromestela_requests.middlewares.requests_historyimportRequestsHistoryMiddlewarefromestela_requests.middlewares.spider_statusimportSpiderStatusMiddlewarefromestela_requests.middlewares.statsimportStatsMiddlewarefromestela_requests.log_helpers.handlersimportKafkaLogHandlerfromestela_requests.item_pipeline.exporterimportKafkaItemExporter,StdoutItemExporterESTELA_PRODUCER=get_producer_interface()# ESTELA_PRODUCER is a queue producer(e.g. kafka producer) that will be used to communicate estela-requestsESTELA_PRODUCER.get_connection()# with estelaHTTP_CLIENT=RequestsInterface()# HTTP Requests interface that will be used, at the moment we just have RequestsInterface(requests library)ESTELA_API_HOST=""# This code will be set by estela, you shouldn't move it at least you want to test thingsESTELA_SPIDER_JOB=""# Same as aboveESTELA_SPIDER_ARGS=""# Same as above, at the moment estela-requests doesn't support arguments.ESTELA_ITEM_PIPELINES=[]# Item Pipelines to use, i.e. a DateItemPipeline that will add the timestamp to the item.# Check ItemPipelineInterface to create a new item pipeline.ESTELA_ITEM_EXPORTERS=[KafkaItemExporter]# Item Exporter to use. Where to export, send the data. Check ItemExporterInterface to create a new                                                     # exporter.ESTELA_LOG_LEVEL=logging.DEBUG# Logging LevelESTELA_LOG_FLAG='kafka'# This will be removed in future releases.ESTELA_NOISY_LIBRARIES=[]# A list of noisy library that you want to turn off.ESTELA_MIDDLEWARES=[RequestsHistoryMiddleware,StatsMiddleware,SpiderStatusMiddleware]# Middlewares to use, check MiddlewareInterface to create a new one.JOB_STATS_TOPIC="job_stats"# Topic name for job stats.JOB_ITEMS_TOPIC="job_items"# Topic name for job items.JOB_REQUESTS_TOPIC="job_requests"# Topic name for job requests.JOB_LOGS_TOPIC="job_logs"# Topic name for job logsMoreFor more details and information about the Estela project, please refer to theEstela documentation. The documentation provides comprehensive information about the project and its functionalities.