Package: netcov
NetCovNetCov is an open-source tool that can be used withBatfishto analyze test coverage for network configurations. Given a set of Batfish queries, it analyzes which lines of configurations has/has not been covered. The result can be used to assess the rigorousness of the test suite and help discover the blind spots. Please refer to ourpaperfor technical details.NetCov is written in Python and can be used in concert withpybatfish, Batfish's Python API.FeaturesNetCov supports coverage tracking for the followingBatfish questions:Reachability test viatraceroutequestionRouting policy evaluation viatestRoutePoliciesquestionDirect inspection of routing tables viaroutesandbgpRibquestionsNetCov supports configuration syntax of the following vendors:InterfaceBGP neighborRouting PolicyPrefix listCommunity listAs-path listCisco:heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark:Juniper:heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark:Arista:heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark:Palo Alto Networks:heavy_check_mark:SONiC:heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark:If you’d like support for additionalvendors(which are supported by Batfish but unsupported by NetCov) or currently-unsupported configuration features, let us know viaGitHub issue. We'll try to add support, and it would help if you share a configuration example. You may want to use ananonymizerbefore sharing.NetCov reports configuration coverage as the percentage of configuration lines that are covered, such as:NetCov can also report fine-grained coverage results as colored annotations on source configurations (lines in blue are covered, lines in red are not covered):You can find an example of the coverage reporthere.Installing NetCovInstall NetCov usingpip. A virtual environment and Python version 3.7 is recommended.$pipinstallnetcovNetCov leveragesLCOVto generate HTML report. If you would like to use this feature, install LCOV using Homebrew (MacOS) or apt (Ubuntu):Installlcovon MacOS using Homebrew:$brewinstalllcovInstalllcovon Ubuntu using apt:$sudoaptinstalllcovUsageNetCov can be used seamlessly withpybatfish. It provides a hooked version of pybatfish APIs that automatically tracks coverage during the execution of supported pybatfish questions.Using NetCov for an existing pybatfish script/notebookIt takes only two simple steps to measure coverage for your existing pybatfish scripts/notebooks.For import, replace pybatfish client session with the one provided by NetCov:#from pybatfish.client.session import SessionfromnetcovimportNetCovSessionasSessionGenerate coverage results at the end of your script. To print coverage metrics to the console, usebf.cov.result(). To generate HTML report, usebf.cov.html_report().Use NetCov for a new pybatfish script/notebookWe provide ademo videoand anexampleto help you get started. If you are new to pybatfish, we recommend reading thepybatfish docfirst.AdvancedSometimes not all information retrieved by Batfish questions are meant to be tested, for example, when you retrieve all RIB entries but only assert on a subset of them. To help NetCov model coverage more accurately, you can pause coverage tracking and add tested information use a NetCov API:# pause coverage tracking to avoid over-estimation
bf.cov.pause()
routes = bf.q.routes(nodes="edge-0000").answer().frame()
bf.cov.resume()

# filter RIB entries to test
tested = routes[routes["Network"] == '0.0.0.0/0'].head(1)

# add tested route to coverage trace
bf.cov.add_tested_routes(tested)bf.cov.result()prints coverage metrics usingloggingmodule and writes tostderrby default. To save the coverage report to file, you can customize logger by:importloggingfh=logging.FileHandler('cov.log')logging.getLogger('netcov').addHandler(fh)References@article{netcov,
  title={Test Coverage for Network Configurations},
  author={Xu, Xieyang and Deng, Weixin and Beckett, Ryan and Mahajan, Ratul and Walker, David},
  journal={arXiv preprint arXiv:2209.12870},
  year={2022}
}