Package: html-diff
HTML-Diff: HTML-formatted diff'ing of HTML snippetsCompares two HTML snippets strings and returns the diff as a valid HTML snippet with changes wrapped in<del>and<ins>tags.Relies onBeautifulSoup4withhtml.parserbackend for HTML parsing and dumping.HTML-Diff focusses on providingvaliddiffs, that is:The returned strings represent valid HTML snippets;Provided that the input snippets represent valid HTML snippets containing no<del>or<ins>tags, they can be identically reconstructed from the diff output, by removing<ins>tags and their content and replacing<del>tags by their content for theoldsnippet, the other way round for thenewone. Functions are provided in thecheckmodule for those reconstructions and checking that the reconstructions match the inputs.UsageBasic usage>>>fromhtml_diffimportdiff>>>diff("<em>ABC</em>","<em>AB</em>C")'<em><del>ABC</del><ins>AB</ins></em><ins>C</ins>'Adding custom tags to be treated as insecable blocksExample use case: having MathJax elements wrapped into<span class="math-tex">\(...\)</span>and wanting to avoid<del>and<ins>tags inside the\(...\)(which would be badly rendered):>>>fromhtml_diff.configimportconfig>>>config.tags_fcts_as_blocks.append(lambdatag:tag.name=="span"and"math-tex"intag.attrs.get("class",[]))Without it (does not render correctly with MathJax):>>>diff(r'<span class="math-tex">\(\vec{v}\)</span>',r'<span class="math-tex">\(\vec{w}\)</span>')'<span class="math-tex">\\(\\vec{<del>v</del><ins>w</ins>}\\)</span>'With it:>>>fromhtml_diffimportclear_cache>>>clear_cache()>>>diff(r'<span class="math-tex">\(\vec{v}\)</span>',r'<span class="math-tex">\(\vec{w}\)</span>')'<del><span class="math-tex">\\(\\vec{v}\\)</span></del><ins><span class="math-tex">\\(\\vec{w}\\)</span></ins>'The functions inconfig.config.tags_fcts_as_blocksshould take abs4.element.Tagas input and return abool; the tags are tested against all functions in the list, and are considered insecable blocks if any call returnsTrue.Score for tagsHTML tags have a base score associated, which is added to there content score. This base score can be configured:>>>config.EMPTY_ELEMENT_SCORE# default: 2>>>config.OTHER_ELEMENT_SCORE# default: 2WARNING: some results are cached and changing the config does not invalidate the cache, thus the results may be wrong afterwards. Callclear_cache()to reset the cache.Reconstructingoldandnewfromdiff>>>old="old_string">>>new="new_string">>>d=diff(old,new)>>>fromhtml_diff.checkimportnew_from_diff>>>new==new_from_diff(d)True>>>fromhtml_diff.checkimportold_from_diff>>>old==old_from_diff(d)True>>>fromhtml_diff.checkimportis_diff_valid>>>is_diff_valid(old,new,d)TrueTestingAutomated testingRunpython-munittestat the root of the project.Manual testingRunpython-mhtml_diffand navigate to 127.0.0.1:8080 with your browser.You can specify further options:-aor--address: custom address of the server (default: 127.0.0.1)-por--port: custom port of the server (default: 8080)-bor--blocks: definitions of functions to be added toconfig.config.tags_fcts_as_blocks, e.g.:python-mhtml_diff-b'lambda tag: tag.name == "span" and "math-tex" in tag.attrs.get("class", [])'-cor--cuttable-words-mode: way of treating words cutting, see above for details; one of theconfig.Config.CuttableWordsModevalues (default: CUTTABLE)AlgorithmThe new implementation uses an algorithm that is closer todifflib.SequenceMatcher, although it does ironically not use it anymore.The algorithm is similar to the legacy implementation with theUNCUTTABLE_PRECISEconfiguration, with the difference that it uses a Ratcliff-Obershelp-like procedure (best-matching subsequence) on all levels rather than testing all combinations to find the optimum. It is thus faster.MatchingParse the inputs withBeautifulSoup4; this yields two iterables of elements, eitherbs4.element.NavigableStringorbs4.element.Tag.On the top level of the HTML structure, split thebs4.element.NavigableString's in words (usingre's\Wpattern), then find the best-matching subsequence, using a score:identical words: the length of the word,bs4.element.Tag's where thenameandattrsattributes match exactly:if the tags are considered as blocks (those that testTruewith a function ofconfig.config.tags_fcts_as_blocks):config.EMPTY_ELEMENT_SCOREif the tags areempty, elseconfig.OTHER_ELEMENT_SCOREplus the length of the string content of the tags,else,config.OTHER_ELEMENT_SCOREplus the sum of the scores of the tags' contents (calculated recursively).On the left and the right of the best-matching subsequence, repeat 2. If non-matchable subsequences remain, assign them a score of 0 and treat them as completely deleted/inserted.DumpingWith those tree match structures, dumping can be done directly, recursively, by dumping thenode_leftfirst, then the matched subsequence, finally thenode_right. Matches are always exact and thus can be dumped as-is, except for non-matchables subsequences that are first completely deleted and then completely reinserted.Dumping is done in aBeautifulSoup4soup, then output as a string.Legacy implementationThe legacy implementation is available inhtml_diff.legacy.Word cutting in diffBy default, the diff'ing algorithm for plain text parts does not care about words - if a word part is modified, that part gets<del>'ed and<ins>'ed, while the rest of the word remains untouched. It may however be more readable to have full words deleted and reinserted. To ensure this, switchconfig.config.cuttable_words_modeto eitherconfig.Config.CuttableWordsMode.UNCUTTABLE_SIMPLEorconfig.Config.CuttableWordsMode.UNCUTTABLE_PRECISE:config.config.cuttable_words_mode == config.Config.CuttableWordsMode.CUTTABLE(default):>>>fromhtml_diff.legacyimportdiffasldiff>>>ldiff("OlyExams","ExamTools")'<del>Oly</del>Exam<ins>Tool</ins>s'>>>ldiff("abcdef<br/>ghifjk","abcdef ghifjk")'abcdef<ins> ghifjk</ins><del><br/>ghifjk</del>'config.config.cuttable_words_mode == config.Config.CuttableWordsMode.UNCUTTABLE_SIMPLE(fast and gives acceptable results):>>>ldiff("OlyExams","ExamTools")'<del>OlyExams</del><ins>ExamTools</ins>'>>>ldiff("abcdef<br/>ghifjk","abcdef ghifjk")'abcdef<ins> ghifjk</ins><del><br/>ghifjk</del>'config.config.cuttable_words_mode == config.Config.CuttableWordsMode.UNCUTTABLE_PRECISE(quite slow, but uses early word breaking for better matching, in particular if plain string parts of the inputs were split or merged betweenoldandnew):>>>ldiff("OlyExams","ExamTools")'<del>OlyExams</del><ins>ExamTools</ins>'>>>ldiff("abcdef<br/>ghifjk","abcdef ghifjk")'abcdef<del><br/></del><ins> </ins>ghifjk'In uncuttable words modes, non-word characters correspond tore's\Wpattern.AlgorithmMatchingParse the inputs withBeautifulSoup4; this yields two iterables of elements, eitherbs4.element.NavigableStringorbs4.element.Tag.Compare each element of the first iterable with each element of the second one. A match is only allowed in two cases:Both elements arebs4.element.NavigableString's (depending on thecuttable words modeconfiguration, matching is done on the raw string, a list of words or on the raw string split in substrings);Both elements arebs4.element.Tag's and theirnameandattrsattributes exactly match.Each match is temporarily stored, together with a "score":Forbs4.element.NavigableStringmatches, their matching length as perdifflib.SequenceMatcher;Forbs4.element.Tagmatches that are treated as blocks (those that testTruewith a function ofconfig.config.tags_fcts_as_blocks), tags that differ have a matching length of0. Tags comparing equal are assigned following matching length: the length of the tag itself foremptytags (e.g.<br />) (this is a mostly arbitrary choice), else the length of the content of the tag (tag.string);For other, "conventional"bs4.element.Tagmatches, the matching length is the sum of matching lengths of their children using the same algorithm recursively.The highest scoring match is kept and the algorithm recursively repeated on the subiterables before and after the matching elements. Each match thus gets (maximally) amatch_beforeand amatch_afterassigned.Regions without match are stored as "no-matches". With them, both iterables are completely covered by matches and no-matches.DumpingWith those tree match structures, dumping can be done directly, recursively, by dumping thematch_beforefirst, then the matched element itself, finally thematch_after. Matchedbs4.element.NavigableString's are dumped parts by parts following the blocks (of words or of characters, depending on thecuttable words modeconfiguration) found bydifflib.SequenceMatcher. Matchedbs4.element.Tag's to be treated as blocks are either dumped without change if fully matching, else are first completely deleted and then completely reinserted. No-matches elements are dumped as completely deleted and completely inserted.Dumping is done in aBeautifulSoup4soup, then output as a string.