Package: diffplus
What is DiffPlus?What for?ProblemProposed solutionHOWTOInstallIndented config to dictSanitizingIncremental diffDiff using a third-party module(optional)Known limitationsOnly additions, no deletionsCode-like configOne character for indentationDoes DiffPlus reinvent the wheel?What is DiffPlus?A lightweight module to help in the comparison of config files.In particular, it computes an incremental diff between two indented config files whilst respecting the scope of the indented blocks (aka contextual diff).The module leverages then-ary tree data structure to achieve such a diff. Taking about a hundred lines of code, it only relies on Python builtins and has no extra dependencies.What for?The whole point of DiffPlus is to compare config filesbefore running them in production.Basically, we'd like to mergeAconfig file (the candidate config, partial or full) into anotherBconfig file (the running config). The next section states the problem and gives an insight of what we expect from the module.ProblemSay we have two indented configs:Config A (to merge into B)Config B!
no ip domain lookup
!
interface FastEthernet0/0
 no shutdown
!
interface FastEthernet0/0.10
 description LAN
 encapsulation dot1Q 10
 ip address 192.168.1.254 255.255.255.0
!
router bgp 64512
 neighbor 172.16.0.1 remote-as 100
 !
 address-family ipv4
  neighbor 172.16.0.1 activate
  neighbor 172.16.0.1 allowas-in 1
  neighbor 172.16.0.1 prefix-list IN in
  neighbor 172.16.0.1 prefix-list OUT out
  network 192.168.1.0 mask 255.255.255.0
!!
hostname R1
!
interface FastEthernet0/0
 description LAN
 no ip address
 shutdown
 duplex auto
 speed auto
!
router bgp 64512
 neighbor 172.16.0.1 remote-as 100
 !
 address-family ipv4
  neighbor 172.16.0.1 activate
  neighbor 172.16.0.1 prefix-list IN in
  neighbor 172.16.0.1 prefix-list OUT out
!
ip prefix-list IN seq 5 permit 192.168.2.0/24
ip prefix-list OUT seq 5 permit 192.168.1.0/24
!The example shows Cisco configs but it applies to any indented config (NOT necessarily network-related).Some items ofAare missing inB. How to find them?We are only looking for new items to be added inB— it is called anincremental diffThe comparison must respect the indented blocks scope — it is called acontextual diffA line-by-line diff will not help here sinceAis typically a partial config to merge into a full one beingB.We humans are able to compute that diff with ease because we visually identify blocks and items (though we make mistakes, sometimes). After some effort, we'd end up with the following result in mind:# items of A to be added in B (to be computed by diffplus)+no ip domain lookupinterface FastEthernet0/0+ no shutdown+interface FastEthernet0/0.10+ description LAN+ encapsulation dot1Q 10+ ip address 192.168.1.254 255.255.255.0router bgp 64512address-family ipv4+  neighbor 172.16.0.1 allowas-in 1+  network 192.168.1.0 mask 255.255.255.0As stated above, this is an incremental diff:there are only additions (hence the module name),no deletions.As simple as it seems, such a diff is not so trivial for an algorithm.Proposed solutionConvert each config to ann-ary treeDo a deep comparison of then-ary treesThen-ary tree data structure will help: to represent how nested the items are, to do the match between blocks.Because we deal with config files,each line is unique per indented block.Therefore, no need for a list of nodes (allowing for duplicates). We can directly use raw nested dicts asn-ary trees, keys being the lines.The deep comparison can then be achieved with either a dedicated third-party module or the lightweightIncrementalDiffhelper embedded in this module.HOWTOInstallDiffPlus is available onPyPI:pipinstalldiffplusIndented config to dictTheto_dict()method converts an indented config to ann-ary tree:fromdiffplusimportIndentedConfigfromjsonimportdumpsconfigA=open('configA.txt').read()# or it may be a stringconfigB=open('configB.txt').read()# or it may be a stringconfigA=IndentedConfig(configA,comment_char='!',sanitize=True)configB=IndentedConfig(configB,comment_char='!',sanitize=True)print(dumps(configA.to_dict(),indent=2))print(dumps(configB.to_dict(),indent=2))Output:Config A (to merge into B)Config B{"no ip domain lookup":{},"interface FastEthernet0/0":{"no shutdown":{}},"interface FastEthernet0/0.10":{"description LAN":{},"encapsulation dot1Q 10":{},"ip address 192.168.1.254 255.255.255.0":{}},"router bgp 64512":{"neighbor 172.16.0.1 remote-as 100":{},"address-family ipv4":{"neighbor 172.16.0.1 activate":{},"neighbor 172.16.0.1 allowas-in 1":{},"neighbor 172.16.0.1 prefix-list IN in":{},"neighbor 172.16.0.1 prefix-list OUT out":{},"network 192.168.1.0 mask 255.255.255.0":{}}}}{"hostname R1":{},"interface FastEthernet0/0":{"description LAN":{},"no ip address":{},"shutdown":{},"duplex auto":{},"speed auto":{}},"router bgp 64512":{"neighbor 172.16.0.1 remote-as 100":{},"address-family ipv4":{"neighbor 172.16.0.1 activate":{},"neighbor 172.16.0.1 prefix-list IN in":{},"neighbor 172.16.0.1 prefix-list OUT out":{}}},"ip prefix-list IN seq 5 permit 192.168.2.0/24":{},"ip prefix-list OUT seq 5 permit 192.168.1.0/24":{}}There is no list but only dicts. Each item may have child items. The nesting level is NOT limited.At this point, you are free to compare the dicts the way you want. However, to that end, you may be interested in existing modules likeDeepDiffor the lightweightIncrementalDiffhelper embedded in this module.ℹ Depending on your config format, bothindent_charandcomment_charcan be set at init:IndentedConfig(config,comment_char='#',indent_char=' ')# default values (e.g., for Huawei)IndentedConfig(config,comment_char='!')# '!' as comment_char (e.g., for Cisco)IndentedConfig(config,indent_char='\t')# tab as indent_charIndentedConfig(config,sanitize=True)# see next sectionSanitizingYour config may:have trailing spaceshave blank linescontain commentsnot be correctly indentedThesanitize()method has been made to address this. The last point especially would break the tree conversion. The other ones would make some parts of the diff irrelevant.fromdiffplusimportIndentedConfigconfig="""# a global command badly indentedsysname PE#interface GigabitEthernet0/0/1# badly indented as welldescription my-super-descriptionip address 1.1.1.1 255.255.255.0#"""config=IndentedConfig(config,comment_char='#')print(config)config.sanitize()print(config)Print before sanitizing ❌Print after sanitizing ✔# a global command badly indentedsysnamePE#interfaceGigabitEthernet0/0/1# badly indented as welldescriptionmy-super-descriptionipaddress1.1.1.1255.255.255.0#sysnamePE
interfaceGigabitEthernet0/0/1descriptionmy-super-descriptionipaddress1.1.1.1255.255.255.0ℹ It is recommended to alwayssanitizeat init:config=IndentedConfig(config,sanitize=True)Incremental diffDiff onlyTo compute the diff just as introduced in the#problemsection:fromdiffplusimportIndentedConfig,IncrementalDiffconfigA=open('configA.txt').read()configB=open('configB.txt').read()configA=IndentedConfig(configA,comment_char='!',sanitize=True)configB=IndentedConfig(configB,comment_char='!',sanitize=True)diff=IncrementalDiff(configA,configB)print(diff)Output:# items of A to be added in B (computed by IncrementalDiff)+no ip domain lookupinterface FastEthernet0/0+ no shutdown+interface FastEthernet0/0.10+ description LAN+ encapsulation dot1Q 10+ ip address 192.168.1.254 255.255.255.0router bgp 64512address-family ipv4+  neighbor 172.16.0.1 allowas-in 1+  network 192.168.1.0 mask 255.255.255.0MergingAlternatively, we can mergeAintoB. It is useful to get a preview of the full config before applying it:fromdiffplusimportIndentedConfig,IncrementalDiffconfigA=open('configA.txt').read()configB=open('configB.txt').read()configA=IndentedConfig(configA,comment_char='!',sanitize=True)configB=IndentedConfig(configB,comment_char='!',sanitize=True)diff=IncrementalDiff(configA,configB,merge=True)print(diff)Output:# items of A merged into B (computed by IncrementalDiff)hostname R1interface FastEthernet0/0description LANno ip addressshutdownduplex autospeed auto+ no shutdownrouter bgp 64512neighbor 172.16.0.1 remote-as 100address-family ipv4neighbor 172.16.0.1 activateneighbor 172.16.0.1 prefix-list IN inneighbor 172.16.0.1 prefix-list OUT out+  neighbor 172.16.0.1 allowas-in 1+  network 192.168.1.0 mask 255.255.255.0ip prefix-list IN seq 5 permit 192.168.2.0/24ip prefix-list OUT seq 5 permit 192.168.1.0/24+no ip domain lookup+interface FastEthernet0/0.10+ description LAN+ encapsulation dot1Q 10+ ip address 192.168.1.254 255.255.255.0Colored diffThis option is useful to better visualize the changes in the diff:fromdiffplusimportIndentedConfig,IncrementalDiffconfigA=open('configA.txt').read()configB=open('configB.txt').read()configA=IndentedConfig(configA,comment_char='!',sanitize=True)configB=IndentedConfig(configB,comment_char='!',sanitize=True)diff=IncrementalDiff(configA,configB,merge=True,colored=True)print(diff)Output:Not coloredColoredColorization is done through ANSI escape sequences:\033[32mfor green color and\033[mfor reset color.Under the hoodThe incremental diff is computed recursively as a dict you can access (if needed) through theto_dict()method:diff=IncrementalDiff(configA,configB,merge=False)# or merge=Trueprint(dumps(diff.to_dict(),indent=2))Output:{"+no ip domain lookup":{},"interface FastEthernet0/0":{"+no shutdown":{}},"+interface FastEthernet0/0.10":{"description LAN":{},"encapsulation dot1Q 10":{},"ip address 192.168.1.254 255.255.255.0":{}},"router bgp 64512":{"address-family ipv4":{"+neighbor 172.16.0.1 allowas-in 1":{},"+network 192.168.1.0 mask 255.255.255.0":{}}}}New items are marked with a+so that they can be pretty rendered recursively via the__str__()special method:print(diff)print(str(diff))# equivalentprint(diff.__str__())# equivalentDiff using a third-party moduleConvertingIndentedConfigto dict allows for deep comparison using existing modules likeDeepDiff. This way, you are not stuck with the rather simplisticIncrementalDiffhelper.fromdiffplusimportIndentedConfigfromdeepdiffimportDeepDiffconfigA=open('configA.txt').read()configB=open('configB.txt').read()configA=IndentedConfig(configA,comment_char='!',sanitize=True)configB=IndentedConfig(configB,comment_char='!',sanitize=True)diff=DeepDiff(configB.to_dict(),configA.to_dict())foritem_addedindiff['dictionary_item_added']:print(item_added)Output:root['no ip domain lookup']root['interface FastEthernet0/0.10']root['interface FastEthernet0/0']['no shutdown']root['router bgp 64512']['address-family ipv4']['neighbor 172.16.0.1 allowas-in 1']root['router bgp 64512']['address-family ipv4']['network 192.168.1.0 mask 255.255.255.0']But third-party modules are generally heavier (since they offer more features) and may not do exactly what you want. For example, the above output does not include (on purpose) the children of missing items, e.g.,interface FastEthernet0/0.10is missing fromBas well as its children yet NOT displayed:interfaceFastEthernet0/0.10# displayed in above output as root['interface FastEthernet0/0.10']descriptionLAN# not displayed in above outputencapsulationdot1Q10# not displayed in above outputipaddress192.168.1.254255.255.255.0# not displayed in above outputKnown limitationsOnly additions, no deletionsBecause DiffPlus focuses on merging a config into another one—and not just doing a line-by-line diff—deletions aren't as easy as additions to compute.How to know that an item ofAwill affect another one inB?Let's take an example in a network context:# computed by diffplus (only additions)interface FastEthernet0/0description Some interface+ no descriptionip address 10.0.0.1 255.255.255.0ip address 10.0.0.2 255.255.255.0 secondaryip address 10.0.0.3 255.255.255.0 secondary+ no ip address 10.0.0.3 255.255.255.0 secondary+ ip address 10.0.0.4 255.255.255.0 secondaryspeed 10+ speed 100Some of the new items will negate or change existing ones. So we'd like a smarter diff:# NOT computed by diffplus (additions and deletions)interface FastEthernet0/0- description Some interfaceip address 10.0.0.1 255.255.255.0ip address 10.0.0.2 255.255.255.0 secondary- ip address 10.0.0.3 255.255.255.0 secondary+ ip address 10.0.0.4 255.255.255.0 secondary- speed 10+ speed 100We humans are able to compute that diff because we visually identify items and are familiar with the config logic. From an algorithmic point of view, however, it is challenging. Not only it depends on the config grammar and syntax (i.e., what are considered keywords and values) but also on the semantic (e.g., adding an item won't necessarily replace a similar one as it is the case forsecondaryaddresses).Theclosest string match approachusing an helper likedifflib.get_close_matches()is an interesting lead, yet not 100% accurate and it would have over-complexified the module.Code-like configBy essence, DiffPlus is not suited for code diff. Each line is assumed to be unique per indented block. Therefore, the tree conversion won't work well with algorithms as they have statements which repeat in the code.Let's take an example with pseudocode:# algo.txtifsome_expressionthenfirst_if_contentelsefirst_else_contentfiifanoter_expressionthensecond_if_contentelsesecond_else_contentfiTree conversion:fromdiffplusimportIndentedConfigfromjsonimportdumpsalgo=open('algo.txt').read()algo=IndentedConfig(algo)print(dumps(algo.to_dict(),indent=4))Output:{"if some_expression then":{"first_if_content":{}},"else":{"second_else_content":{}},"fi":{},"if anoter_expression then":{"second_if_content":{}}}Inconsistencies:There is only onefiinstead of twos.Likewise, the firstelsehas been overwritten by the second one at the same indentation level.This is in accordance with how the module works: it has been designed for config diff, not code diff.One character for indentationFor now, DiffPlus does not support multiple characters as indentation symbol:IndentedConfig(config,indent_char='  ')# 2 spaces (NOT supported)IndentedConfig(config,indent_char='    ')# 4 spaces (NOT supported)IndentedConfig(config,indent_char='\t')# tab char (supported)The first two lines will raise an error:ValueError:"indent_char"mustbeachar,notanstrThe same limitation applies forcomment_char.A future version may support it if the module gains interest in the community.Does DiffPlus reinvent the wheel?I didn't find a suitable module providing such an incremental and contextual diff simply based on an indented config (NOT necessarily Cisco-based or network-based for the sake of genericity).difflib(Python builtin) does a line-by-line diffDeepDiffcompares dicts but does not convert an indented text to dictConfigTreeonly supports YAML and JSON formatsanytreedrops the support of the indented text formatdiffiosonly supports Cisco format and it is not clear what it does exactly (variable parsing or diff?)shconfparser(probably the closest one) only supports Cisco format and does not offer diff featuresconf_diffdoes a line-by-line diff