Package: html-processor
HTML Processor - a package that provides a set of classes aroundBeautifulSoupfor
easy HTML modification.Well, what for?Sometimes it is necessary to make constant changes in HTML code
according to specified rules. For example, it is necessary to insert
links to thumbnails for images inserted into WYSIWYG editor. It would be
desirable to be able to describe changes in a more declarative and
structured way, rather than write simple scripts.ExampleIn the example we will do it (Inserting thumbnails in the code for
pictures). Let’s say we have html, with a description of the characters
of “Adventure Time”:<html><head><title>Heroes of Ooo</title></head><body><header><h1>Heroes of Ooo</h1><imgsrc=""/></header><main><imgalt="Delete me"src="#"/><article><figure><imgalt="Finn Mertens"src="/media/images/heroes/Finn.jpeg"/><figcaption>Finn Mertens</figcaption></figure><div><p>Finn Mertens (simply known as Finn the Human and formerly known as
            Pen in the original short) the main protagonist of the Cartoon
            Network series Adventure Time.</p><p>He was voiced by Jeremy Shada, who also voice as Lance from Voltron:
            Legendary Defender and Cody Maverick in Surf's Up: Wavemania.</p></div></article><article><figure><imgalt="Jake the Dog"src="/media/images/heroes/Jake.jpeg"/><figcaption>Jake the Dog</figcaption></figure><div><p>Jake is the deuteragonist of Adventure Time. He's a magical dog and
            Finn's constant companion, best friend and adoptive brother. Jake
            has shape shifting abilities so he can "stretch" into different
            objects.</p><p>He was voiced by John DiMaggio, who also voiced as Fu Dog from
            American Dragon: Jake Long.</p></div></article></main></body></html>We understand that we need to optimize the images.For example, we usenginx. We set up paths to change the image frame in the following template:/width/path, wherewidth- image width,path- image path.We need to replace that code:<imgsrc="/media/images/heroes/Jake.jpeg"/>with the next:<picture><sourcemedia="(min-width: 1024px)"srcset="/1280/media/images/heroes/Jake.jpeg 1x, /1920/media/images/heroes/Jake.jpeg 1.5x, /2560/media/images/heroes/Jake.jpeg 2x, /3840/media/images/heroes/Jake.jpeg 3x"><sourcemedia="(min-width: 768px)"srcset="/1024/media/images/heroes/Jake.jpeg 1x, /1536/media/images/heroes/Jake.jpeg 1.5x, /2048/media/images/heroes/Jake.jpeg 2x, /3072/media/images/heroes/Jake.jpeg 3x"><imgloading="lazy"src="/media/images/heroes/Jake.jpeg"srcset="/768/media/images/heroes/Jake.jpeg 1x, /1152/media/images/heroes/Jake.jpeg 1.5x, /1536/media/images/heroes/Jake.jpeg 2x, /2304/media/images/heroes/Jake.jpeg 3x"/></picture>We also need to remove images, the source of which is not a link.In doing so, we should not be tied specifically to this image and to
this location on the page.Let’s get started. First we need to create a basic rule that will work
for all images on the page:fromhtml_processorimport(HtmlProcessor,TagRule,)classImageRule(TagRule):tag='img'defprocess():source_html=open('heroes.html').read()processor=HtmlProcessor(source_html,rules=[ImageRule])withopen('enhanced-heroes.html','w')asfile:file.write(repr(processor))if__name__=='__main__':process()If we run the script now, you will see that nothing has changed (except
the formatting).That’s because we didn’t describe how we should change the image tags.
Let’s do this:...classImageRule(TagRule):tag='img'rotations=(1,1.5,2,3,)sources=((1024,1280),(768,1024),)default_width=768defget_new_tag(self,attributes,contents=None):src=attributes.get('src','')picture=self.create_tag('picture')formin_screen_width,widthinself.sources:source=self.create_sources(src,min_screen_width,width)picture.append(source)img=self.create_img(src)picture.append(img)returnpicturedefcreate_img(self,src):img=self.create_tag()img.attrs['src']=srcimg.attrs['srcset']=self.build_srcset(self.default_width,src)img.attrs['loading']='lazy'returnimgdefcreate_sources(self,src,min_screen_width,width):source=self.create_tag('source')source.attrs['media']='(min-width:{}px)'.format(min_screen_width)source.attrs['srcset']=self.build_srcset(width,src)returnsourcedefbuild_srcset(self,width,src):return', '.join(['/{}{}{}x'.format(int(width*rotate),src,rotate)forrotateinself.rotations])...We overridden the methodget_new_tag. This method is called for all
tags defined in the attributeTagRule.tag, from which you can return
a new tagbs4.Tag, which will replace the tag found. If we returnNone, the tag found does not change....<header><h1>Heroes of Ooo</h1><picture><sourcemedia="(min-width: 1024px)"srcset="/1280 1x, /1920 1.5x, /2560 2x, /3840 3x"/><sourcemedia="(min-width: 768px)"srcset="/1024 1x, /1536 1.5x, /2048 2x, /3072 3x"/><imgloading="lazy"src=""srcset="/768 1x, /1152 1.5x, /1536 2x, /2304 3x"/></picture></header>...<figure><picture><sourcemedia="(min-width: 1024px)"srcset="/1280/media/images/heroes/Finn.jpeg 1x, /1920/media/images/heroes/Finn.jpeg 1.5x, /2560/media/images/heroes/Finn.jpeg 2x, /3840/media/images/heroes/Finn.jpeg 3x"/><sourcemedia="(min-width: 768px)"srcset="/1024/media/images/heroes/Finn.jpeg 1x, /1536/media/images/heroes/Finn.jpeg 1.5x, /2048/media/images/heroes/Finn.jpeg 2x, /3072/media/images/heroes/Finn.jpeg 3x"/><imgloading="lazy"src="/media/images/heroes/Finn.jpeg"srcset="/768/media/images/heroes/Finn.jpeg 1x, /1152/media/images/heroes/Finn.jpeg 1.5x, /1536/media/images/heroes/Finn.jpeg 2x, /2304/media/images/heroes/Finn.jpeg 3x"/></picture><figcaption>Finn Mertens</figcaption></figure>...<figure><picture><sourcemedia="(min-width: 1024px)"srcset="/1280/media/images/heroes/Jake.jpeg 1x, /1920/media/images/heroes/Jake.jpeg 1.5x, /2560/media/images/heroes/Jake.jpeg 2x, /3840/media/images/heroes/Jake.jpeg 3x"/><sourcemedia="(min-width: 768px)"srcset="/1024/media/images/heroes/Jake.jpeg 1x, /1536/media/images/heroes/Jake.jpeg 1.5x, /2048/media/images/heroes/Jake.jpeg 2x, /3072/media/images/heroes/Jake.jpeg 3x"/><imgloading="lazy"src="/media/images/heroes/Jake.jpeg"srcset="/768/media/images/heroes/Jake.jpeg 1x, /1152/media/images/heroes/Jake.jpeg 1.5x, /1536/media/images/heroes/Jake.jpeg 2x, /2304/media/images/heroes/Jake.jpeg 3x"/></picture><figcaption>Jake the Dog</figcaption></figure>...You may notice that the images have been replaced with tags using media queries and tambneiles links. But we still have the tags of images that do not refer to the images themselves.Let’s fix this:fromurllib.parseimporturlparse...defget_new_tag(self,attributes,contents=None):src=attributes.get('src','')parsed_url=urlparse(src)ifparsed_url.path:picture=self.create_tag('picture')formin_screen_width,widthinself.sources:source=self.create_sources(src,min_screen_width,width)picture.append(source)img=self.create_img(src)picture.append(img)returnpicture...defis_extract(self,attributes,**kwargs):src=attributes.get('src','')parsed_url=urlparse(src)returnFalseifparsed_url.pathelseTrueWhat we’ve changed:We return a value from theget_new_tagmethod only if the link in
thesrcattribute contains a path.Override methodis_extract, which returnsTrueif there is no
path referenced in parametersrc. This method is responsible for
extracting the tag from html. If it returnsTruethe tag will be
extracted, ifFalse, no action will be taken with the tag.is_extractis only called if methodget_new_taghas not
returned anything.So, let’s run the script and get the next result:<html><head><title>Heroes of Ooo</title></head><body><header><h1>Heroes of Ooo</h1></header><main><article><figure><picture><sourcemedia="(min-width: 1024px)"srcset="/1280/media/images/heroes/Finn.jpeg 1x, /1920/media/images/heroes/Finn.jpeg 1.5x, /2560/media/images/heroes/Finn.jpeg 2x, /3840/media/images/heroes/Finn.jpeg 3x"/><sourcemedia="(min-width: 768px)"srcset="/1024/media/images/heroes/Finn.jpeg 1x, /1536/media/images/heroes/Finn.jpeg 1.5x, /2048/media/images/heroes/Finn.jpeg 2x, /3072/media/images/heroes/Finn.jpeg 3x"/><imgloading="lazy"src="/media/images/heroes/Finn.jpeg"srcset="/768/media/images/heroes/Finn.jpeg 1x, /1152/media/images/heroes/Finn.jpeg 1.5x, /1536/media/images/heroes/Finn.jpeg 2x, /2304/media/images/heroes/Finn.jpeg 3x"/></picture><figcaption>Finn Mertens</figcaption></figure><div><p>Finn Mertens (simply known as Finn the Human and formerly known as            Pen in the original short) the main protagonist of the Cartoon            Network series Adventure Time.</p><p>He was voiced by Jeremy Shada, who also voice as Lance from Voltron:            Legendary Defender and Cody Maverick in Surf's Up: Wavemania.</p></div></article><article><figure><picture><sourcemedia="(min-width: 1024px)"srcset="/1280/media/images/heroes/Jake.jpeg 1x, /1920/media/images/heroes/Jake.jpeg 1.5x, /2560/media/images/heroes/Jake.jpeg 2x, /3840/media/images/heroes/Jake.jpeg 3x"/><sourcemedia="(min-width: 768px)"srcset="/1024/media/images/heroes/Jake.jpeg 1x, /1536/media/images/heroes/Jake.jpeg 1.5x, /2048/media/images/heroes/Jake.jpeg 2x, /3072/media/images/heroes/Jake.jpeg 3x"/><imgloading="lazy"src="/media/images/heroes/Jake.jpeg"srcset="/768/media/images/heroes/Jake.jpeg 1x, /1152/media/images/heroes/Jake.jpeg 1.5x, /1536/media/images/heroes/Jake.jpeg 2x, /2304/media/images/heroes/Jake.jpeg 3x"/></picture><figcaption>Jake the Dog</figcaption></figure><div><p>Jake is the deuteragonist of Adventure Time. He's a magical dog and            Finn's constant companion, best friend and adoptive brother. Jake            has shape shifting abilities so he can "stretch" into different            objects.</p><p>He was voiced by John DiMaggio, who also voiced as Fu Dog from            American Dragon: Jake Long.</p></div></article></main></body></html>This is what we wanted. You can find out more about the example inexamples/insert_thumbnails.py.APIHtmlProcessorThe class of processor that starts the html processing rules.You can set the rules of html processing by creating a descendant class and overriding the attributerules, for example:classTextProcessor(HtmlProcessor):rules=[AdventureTextRule,]The same rules can be set through the constructor:init(html: string, rules: List[Rule] = None, unqoute: bool =
False) - конструтор принимает строку с html кодом. Так же в него
можно передать правила обработки, как список объектов классаRule, и флаг - стоит ли применять к html строке экранирование
через методurllib.parse.unqoute.Processed content can be obtained from the processor in 3 ways:Callprocessmethod. This method will return the objectbs4.BeautifulSoup.str(processor). This call will return a string with processed and
unformatted html code.repr(processor). This call will return a string with processed and
formatted html code.RuleBase class for describing the html code processing rule.Creating a custom ruleRuleobjects contain an attributecontentthat contains an
objectBeautifulSoupcreated from the source html code.To create its own rules, a class inherited fromRulethe method must
be overridden:process() - this method is called to process the objectRule.content.You can also override the following methods for convenience:get_area - returns the area where objects are searched for. The area
is selected from the attributecontent.select(area: BeautifulSoup) - returns the objects that we need to
process.select_element(element) - returnsTrueif the object is suitable
for processing andFalseif not.These methods are needed to make the methodRule.get_elementsreturned the elements needed for processing.The creation of rules can be seen in more detail on the example of
predefined rule classes, for exampleTagRuleandTextRule.Predetermined rulesTagRuleA rule to process a specific tag.To specify a rule, you need to create a class inherited fromTagRuleand define an attributetagthat takes the tag name as a string, for exampletag = 'img'.There are 2 methods for working with a tag that can be overridden:get_new_tag(self, attributes: dict, contents=None) - the method
accepts attribute dictionaryattributes, as well as the content
of the tagcontents. The method is called for each tag found. The
method must returnNoneif we do not want to change the tag, or a
new tagbs4.Tag, which will replace the current tag.is_extract(self, attributes: dict, contents=None) - The method
accepts attribute dictionaryattributes, as well as the content
of the tag incontents. The method returnsTrueif the tag
needs to be extracted from html, orFalseif nothing needs to be
done with the tag. The method is called only ifget_new_taghas
not returned anything for the given tag.TextRuleA rule for processing texts inside html.To set a rule, you should create a class inherited fromTextRule.The following methods are available for string processing.get_new_string(self, string: str) - takes a string and returns a
new string to replace the found one.is_extract(self, string: str) - accepts the string and returnsTrueif the item with this string must be removed from html, orFalseif left. Removed by the string itself, and the tag that
this string contains, as well as the content of this tag.