Package: scrape-schema
Scrape-schemaThis library is designed to write structured, readable and
reusable parsers for html, raw text and is inspired by dataclasses and ORM libraries🚨 Scrape-schema is currently in Pre-Alpha. Please expect breaking changes.MotivationSimplifying parsers support, where it is difficult to use
or the complete absence of the API interfaces, decrease boilerplate code and
easy separate extraction logic from the crawlingStructuring, data serialization and use as an intermediate layer
for third-party serialization libraries: pydantic, json, dataclasses, attrs, etcFeaturesBuilt top onParsel.re, css, xpath, jmespath,chompjsfeatures.Fluent interfacesimular parsel.Selector API for easy to use.Decrease boilerplate code.Does not depend on the http client implementation, use any!Python 3.8+ support.Reusability, code consistency.Dataclass-like structure.Partial support auto type-casting from annotations (str, int, float, bool, list, dict, Optional)Detailed logging process to make it easier to write a parserInstallpipinstallscrape-schemaExampleThe fields interface is similar to the originalparsellibrary# Example from parsel documentationfromparselimportSelectortext="""<html><body><h1>Hello, Parsel!</h1><ul><li><a href="http://example.com">Link 1</a></li><li><a href="http://scrapy.org">Link 2</a></li></ul><script type="application/json">{"a": ["b", "c"]}</script></body></html>"""defschema(txt:str):selector=Selector(text=txt)return{"h1":selector.css('h1::text').get(),"words":selector.xpath('//h1/text()').re(r'\w+'),"urls":selector.css('ul > li').xpath('.//@href').getall(),"sample_jmespath_1":selector.css('script::text').jmespath("a").get(),"sample_jmespath_2":selector.css('script::text').jmespath("a").getall()}print(schema(text))# {'h1': 'Hello, Parsel!',# 'words': ['Hello', 'Parsel'],# 'urls': ['http://example.com', 'http://scrapy.org'],# 'sample_jmespath_1': 'b',# 'sample_jmespath_2': ['b', 'c']}fromscrape_schemaimportBaseSchema,ParselclassSchema(BaseSchema):h1:str=Parsel().css('h1::text').get()words:list[str]=Parsel().xpath('//h1/text()').re(r'\w+')urls:list[str]=Parsel().css('ul > li').xpath('.//@href').getall()sample_jmespath_1:str=Parsel().css('script::text').jmespath("a").get()sample_jmespath_2:list[str]=Parsel().css('script::text').jmespath("a").getall()text="""<html><body><h1>Hello, Parsel!</h1><ul><li><a href="http://example.com">Link 1</a></li><li><a href="http://scrapy.org">Link 2</a></li></ul><script type="application/json">{"a": ["b", "c"]}</script></body></html>"""print(Schema(text).dict())# {'h1': 'Hello, Parsel!',# 'words': ['Hello', 'Parsel'],# 'urls': ['http://example.com', 'http://scrapy.org'],# 'sample_jmespath_1': 'b',# 'sample_jmespath_2': ['b', 'c']}The scrape_schema example output looks like the previous one, why do you need this library?Easy to modifyEasy add additional methodsEasy to port to another project without having to deal with the logic and call stackIDE typing supportFor example, if you need to modify a parser, withscrape_schemait is easy and simple to do!fromuuidimportuuid4fromdatetimeimportdatetimefromscrape_schemaimportBaseSchema,Parsel,sc_param,Callbackfromscrape_schema.validatorimportmarkup_pre_validatorclassSchema(BaseSchema):# invoke simple functions to fields output# add uuid4 idid:str=Callback(lambda:str(uuid4()))# add parse datedate:str=Callback(lambda:str(datetime.today()))h1:str=Parsel().css('h1::text').get()# convert to upper caseh1_upper:str=Parsel().css('h1::text').get().upper()# convert to lower caseh1_lower:str=Parsel().css('h1::text').get().lower()words:list[str]=Parsel().xpath('//h1/text()').re(r'\w+')# alt solution split wordswords_2:list[str]=Parsel().xpath('//h1/text()').get().split()# join result by ' - ' stringwords_join:str=Parsel().xpath('//h1/text()').re(r'\w+').join(" AND ")urls:list[str]=Parsel().css('ul > li').xpath('.//@href').getall()# replace http protocol to httpsurls_https:list[str]=Parsel().css('ul > li').xpath('.//@href').getall().replace("http://","https://")# you can modify output keyssample_jmespath_1:str=Parsel(alias="jsn1").css('script::text').jmespath("a").get()sample_jmespath_2:list[str]=Parsel(alias="class").css('script::text').jmespath("a").getall()# or calc json count valuesjsn_len:int=Parsel(auto_type=False).css('script::text').jmespath("a").getall().count()# pre validation markup input before parse text.# if text from first h1 element != 'Hello, Parsel!' - throw `SchemaPreValidationError` exception@markup_pre_validator()defvalidate_markup(self)->bool:returnself.__selector__.css('h1::text').get()=='Hello, Parsel!'# or create fields with custom rule@sc_paramdefcustom(self)->str:return"hello world!"# you can add extra methods!defparse_urls(self):forurlinself.urls:print(f"parse{url}")text="""<html><body><h1>Hello, Parsel!</h1><ul><li><a href="http://example.com">Link 1</a></li><li><a href="http://scrapy.org">Link 2</a></li></ul><script type="application/json">{"a": ["b", "c"]}</script></body></html>"""schema=Schema(text)# invoke custom methodschema.parse_urls()# parse http://example.com# parse http://scrapy.orgprint(schema.dict())# !!!field from @sc_param decorator#   vvvvvvvvvvvvvvvvvvvvvv# {'custom': 'hello world!',#  !!!simple functions callbacks output#  vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv# 'id': '6b66de7b-5b5f-445a-b8a7-3b17332c1ff5',# 'date': '2023-09-29 18:47:03.638941',# 'h1': 'Hello, Parsel!', 'h1_upper': 'HELLO, PARSEL!', 'h1_lower': 'hello, parsel!',# 'words': ['Hello', 'Parsel'], 'words_2': ['Hello,', 'Parsel!'],# 'words_join': 'Hello AND Parsel',# 'urls': ['http://example.com', 'http://scrapy.org'],# 'urls_https': ['https://example.com', 'https://scrapy.org'],#         !!!changed key alias 'sample_jmespath_2' TO 'class'!!!#               vvvvvvvvvvvvvvvvv# 'jsn1': 'b', 'class': ['b', 'c'], 'jsn_len': 2}See moreexamplesanddocumentationfor get more information/examplesThis project is licensed under the terms of the MIT license.