Package: circuit-maintenance-parser
circuit-maintenance-parsercircuit-maintenance-parseris a Python library that parses circuit maintenance notifications from Network Service Providers (NSPs), converting heterogeneous formats to a well-defined structured format.ContextEvery network depends on external circuits provided by NSPs who interconnect them to the Internet, to office branches or to
external service providers such as Public Clouds.Obviously, these services occasionally require operation windows to upgrade or to fix related issues, and usually, they happen in the form ofcircuit maintenance periods.
NSPs generally notify customers of these upcoming events so that customers can take actions to minimize the impact on the regular usage of the related circuits.The challenge faced by many customers is that almost every NSP defines its own maintenance notification format, even though in the
end the relevant information is mostly the same across NSPs. This library is built to parse notification formats from
several providers and to return always the same object struct which will make it easier to process them afterwards.The format of this output follows theBCOPdefined
during a NANOG meeting that aimed to promote the usage of the iCalendar format. Indeed, if the NSP is using the
proposed iCalendar format, the parser is straightforward and there is no need to define custom logic, but this library
enables supporting other providers that are not using this proposed practice, getting the same outcome.You can leverage this library in your automation framework to process circuit maintenance notifications and use the standardizedMaintenancemodel to handle your received circuit maintenance notifications in a simple way. EveryMaintenanceobject contains the following attributes:provider: identifies the provider of the service that is the subject of the maintenance notification.account: identifies an account associated with the service that is the subject of the maintenance notification.maintenance_id: contains text that uniquely identifies (at least within the context of a specific provider) the maintenance that is the subject of the notification.circuits: list of circuits affected by the maintenance notification and their specific impact. Note that in a maintenance canceled or completed notification, some providers omit the circuit list, so this may be blank for maintenance notifications with a status of CANCELLED or COMPLETED.start: timestamp that defines the starting date/time of the maintenance in GMT.end: timestamp that defines the ending date/time of the maintenance in GMT.stamp: timestamp that defines the update date/time of the maintenance in GMT.organizer: defines the contact information included in the original notification.status: defines the overall status or confirmation for the maintenance.¹summary: human-readable details about this maintenance notification. May be an empty string.sequence: a sequence number for notifications involving this maintenance window. In practice, this is generally redundant with thestampfield and will be defaulted to1for most non-iCalendar parsed notifications.²uid: a unique (?) identifier for a thread of related notifications. In practice, this is generally redundant with themaintenance_idfield and will be defaulted to0for most non-iCalendar parsed notifications.Please, refer to theBCOPto more details about the standardized meaning of these attributes.¹ Per the BCOP, thestatus(X-MAINTNOTE_STATUS) is an optional field in iCalendar notifications. However, aMaintenanceobject will always contain astatusvalue; in the case where an iCalendar notification omits this field, thestatuswill be set to"NO-CHANGE", and it's up to the consumer of this library to determine how to appropriately handle this case. Parsers of other notification formats are responsible for setting an appropriate value for this field based on the notification contents, and may or may not include"NO-CHANGE"as one of the possible reported values.² Per the BCOP, thesequenceis a mandatory field in iCalendar notifications. However, some NSPs have been seen to send notifications which, while otherwise consistent with the BCOP, omit theSEQUENCEfield; in such cases, this library will report a sequence number of-1.WorkflowWe instantiate aProvider, directly or via theinit_providermethod, that depending on the selected type will return the corresponding instance.Get an instance of theNotificationDataclass. This instance groups togetherDataPartswhich each contain some content and a specific type (that will match a specificParser). For example, aNotificationDatamight describe a received email message, withDataPartscorresponding to the subject line and body of the email. There are factory methods to initialize aNotificationDatadescribing a single chunk of binary data, as well as others to initialize one directly from a raw email message oremail.message.EmailMessageinstance.EachProvideruses one or moreProcessorsthat will be used to buildMaintenanceswhen theProvider.get_maintenances(data)method is called.EachProcessorclass uses one or moreParsersto process each type of data that it handles. It can have custom logic to combine the parsed data from multipleParsersto create the finalMaintenanceobject.EachParserclass supports one or a set of related data types, and implements theParser.parse()method used to retrieve aDictwith the relevant keys/values.By default, there is aGenericProviderthat supports aSimpleProcessorusing the standardICalParser, being the easiest path to start using the library in case the provider uses the reference iCalendar standard.Supported ProvidersSupported providers using the BCOP standardArelion (previously Telia)EuNetworksNTTPacketFabricTelstraSupported providers based on other parsersAWSAquaCommsBSOCogentColtCrown Castle FiberEquinixEXA (formerly GTT)HGCGoogleLumenMegaportMomentumNetflix (AS2906 only)SeabornSparkleTelstraTurkcellVerizonZayoNote: Because these providers do not support the BCOP standard natively, maybe there are some gaps on the implemented parser that will be refined with new test cases. We encourage you to report relatedissues!LLM-powered ParsersThe library supports an optional parser option leveraging Large Language Models (LLM) to provide best-effort parsing when the specific parsers have not been successful.Warning: Some of these integrations, such as OpenAI, require of extras installations parameters. Check theextras sectionWhen the appropriate environment variable(s) are set (see below), these LLM parsers are automatically appended after all existing processors for each defined Provider.These integrations may involve some costs for API usage. Use it carefully! As an order of magnitude, a parsing of an email with OpenAI GPT gpt-3.5-turbo model costs $0.004.These are the currently supported LLM integrations:PARSER_LLM_QUESTION_STR(Optional), question to overwrite the default one. Change it carefully. It has precedence overPARSER_LLM_QUESTION_FILEPATHPARSER_LLM_QUESTION_FILEPATH(Optional), a path to a file that contains a question to overwrite the default one.OpenAI, these are the supported ENVs:PARSER_OPENAI_API_KEY(Required): OpenAI API Key.PARSER_OPENAI_MODEL(Optional): The LLM model to use, defaults to "gpt-3.5-turbo".MetadataEachMaintenancecomes with ametadataattribute to provide information about the provider used and the process and parsers used in the successful parsing of the maintenance.This information is relevant to validate the actual content of theMaintenancebecause it may be generated using an LLM-powered parser which means that the confidence level is lower than using a pre-defined parser. You can check theMetadata.generate_by_llmboolean to check it.InstallationThe library is available as a Python package in pypi and can be installed with pip:pip install circuit-maintenance-parserExtrasOpenAIpip install circuit-maintenance-parser[openai]How to use it?The library requires two things:Thenotificationdata: this is the data that the library will check to extract the maintenance notifications. It can be simple (only one data type and content, such as an iCalendar notification) or more complex (with multiple data parts of different types, such as from an email).Theprovideridentifier: used to select the properProviderwhich contains theprocessorlogic to take the properParsersand use the data that they extract. By default, theGenericProvider(used when no other provider type is defined) will support parsing ofiCalendarnotifications using the recommended format.Python LibraryThe first step is to define theProviderthat we will use to parse the notifications. As commented, there is aGenericProviderthat implements the gold standard format and can be reused for any notification matching the expectations.fromcircuit_maintenance_parserimportinit_providergeneric_provider=init_provider()type(generic_provider)<class'circuit_maintenance_parser.provider.GenericProvider'>However, usually someProvidersdon't fully implement the standard and maybe some information is missing, for example theorganizeremail or maybe a custom logic to combine information is required, so we allow customProviders:ntt_provider=init_provider("ntt")type(ntt_provider)<class'circuit_maintenance_parser.provider.NTT'>Once we have theProviderready, we need to initialize the data to process, we call itNotificationDataand can be initialized from a simple content and type or from more complex structures, such as an email.fromcircuit_maintenance_parserimportNotificationDataraw_data=b"""BEGIN:VCALENDARVERSION:2.0PRODID:-//Maint Note//https://github.com/maint-notification//BEGIN:VEVENTSUMMARY:Maint Note ExampleDTSTART;VALUE=DATE-TIME:20151010T080000ZDTEND;VALUE=DATE-TIME:20151010T100000ZDTSTAMP;VALUE=DATE-TIME:20151010T001000ZUID:42SEQUENCE:1X-MAINTNOTE-PROVIDER:example.comX-MAINTNOTE-ACCOUNT:137.035999173X-MAINTNOTE-MAINTENANCE-ID:WorkOrder-31415X-MAINTNOTE-IMPACT:OUTAGEX-MAINTNOTE-OBJECT-ID;X-MAINTNOTE-OBJECT-IMPACT=NO-IMPACT:acme-widgets-as-a-serviceX-MAINTNOTE-OBJECT-ID;X-MAINTNOTE-OBJECT-IMPACT=OUTAGE:acme-widgets-as-a-service-2X-MAINTNOTE-STATUS:TENTATIVEORGANIZER;CN="Example NOC":mailto:noone@example.comEND:VEVENTEND:VCALENDAR"""data_to_process=NotificationData.init_from_raw("ical",raw_data)type(data_to_process)<class'circuit_maintenance_parser.data.NotificationData'>Finally, with we retrieve the maintenances (it is aListbecause a notification can contain multiple maintenances) from the data calling theget_maintenancesmethod from theProviderinstance:maintenances=generic_provider.get_maintenances(data_to_process)print(maintenances[0].to_json()){"account":"137.035999173","circuits":[{"circuit_id":"acme-widgets-as-a-service","impact":"NO-IMPACT"},{"circuit_id":"acme-widgets-as-a-service-2","impact":"OUTAGE"}],"end":1444471200,"maintenance_id":"WorkOrder-31415","organizer":"mailto:noone@example.com","provider":"example.com","sequence":1,"stamp":1444435800,"start":1444464000,"status":"TENTATIVE","summary":"Maint Note Example","uid":"42"}Notice that, either with theGenericProviderorNTTprovider, we get the same result from the same data, because they are using exactly the sameProcessorandParser. The only difference is thatNTTnotifications come withoutorganizerandproviderin the notification, and this info is fulfilled with some default values for theProvider, but in this case, the original notification contains all the necessary information, so the defaults are not used.ntt_maintenances=ntt_provider.get_maintenances(data_to_process)assertmaintenances_ntt==maintenancesEvery maintenance contains themetadataattribute to understand how has been parsed:print(maintenances[0].metadata)provider='genericprovider'processor="SimpleProcessor"parsers=["ICal"],generated_by_llm=FalseCLIThere is also aclientry pointcircuit-maintenance-parserwhich offers easy access to the library using a few arguments:data-file: file storing the notification.data-type:ical,htmloremail, depending on the data type.provider-type: to choose the rightProvider. If empty, theGenericProvideris used.circuit-maintenance-parser--data-file"/tmp/___ZAYO TTN-00000000 Planned MAINTENANCE NOTIFICATION___.eml"--data-typeemail--provider-typezayo
CircuitMaintenanceNotification#0{"account":"some account","circuits":[{"circuit_id":"/OGYX/000000/ /ZYO /","impact":"OUTAGE"}],"end":1601035200,"maintenance_id":"TTN-00000000","organizer":"mr@zayo.com","provider":"zayo","sequence":1,"stamp":1599436800,"start":1601017200,"status":"CONFIRMED","summary":"Zayo will implement planned maintenance to troubleshoot and restore degraded span","uid":"0"}How to Extend the Library?Even though the library aims to include support for as many providers as possible, it's likely that not all the thousands of NSP are supported and you may need to add support for some new one. Adding a newProvideris quite straightforward, and in the following example we are adding support for an imaginary provider, ABCDE, that uses HTML notifications.The first step is creating a new file:circuit_maintenance_parser/parsers/abcde.py. This file will contain all the custom parsers needed for the provider and it will import the base classes for each parser type fromcircuit_maintenance_parser.parser. In the example, we only need to importHtmland in the child class implement the methods required by the class, in this caseparse_html()which will return adictwith all the data that thisParsercan extract. In this case, we have to helper methods,_parse_bsand_parse_tablesthat implement the logic to navigate the notification data.fromtypingimportDictimportbs4# type: ignorefrombs4.elementimportResultSet# type: ignorefromcircuit_maintenance_parser.parserimportHtmlclassHtmlParserABCDE1(Html):defparse_html(self,soup:ResultSet)->Dict:data={}self._parse_bs(soup.find_all("b"),data)self._parse_tables(soup.find_all("table"),data)return[data]def_parse_bs(self,btags:ResultSet,data:Dict):...def_parse_tables(self,tables:ResultSet,data:Dict):...The next step is to create the newProviderby defining a new class incircuit_maintenance_parser/provider.py. This class that inherits fromGenericProvideronly needs to define two attributes:_processors: is alistofProcessorinstances that uses several dataParsers. In this example, we don't need to create a new customProcessorbecause the combined logic serves well (the most likely case), and we only need to use the newly definedHtmlParserABCDE1and also the genericEmailDateParserthat extracts the email date. Also notice that you could have multipleProcessorswith differentParsersin this list, supporting several formats._default_organizer: This is a default helper to fill theorganizerattribute in theMaintenanceif the information is not part of the original notification.classABCDE(GenericProvider):_processors:List[GenericProcessor]=[CombinedProcessor(data_parsers=[EmailDateParser,HtmlParserABCDE1]),]_default_organizer="noc@abcde.com"And expose the newProviderincircuit_maintenance_parser/__init__.py:from.providerimport(GenericProvider,ABCDE,...)SUPPORTED_PROVIDERS=(GenericProvider,ABCDE,...)Last, but not least, you should update the tests!Test the newParserintests/unit/test_parsers.pyTest the newProviderlogic intests/unit/test_e2e.py... adding the necessary data samples intests/unit/data/abcde/.You can anonymize your IPv4 and IPv6 addresses using theinvoke anonymize-ips --local. Keep in mind that only IPv4 addresses for documentation purposes (RFC5737: "192.0.2.0/24", "198.51.100.0/24", "203.0.113.0/24") are preserved, in case you need to check these IPs in your test output (unlikely)ContributingPull requests are welcomed and automatically built and tested against multiple versions of Python through Travis CI.The project is following Network to Code software development guidelines and is leveraging:Black, Pylint, Mypy, Bandit and pydocstyle for Python linting and formatting.Unit and integration tests to ensure the library is working properly.Local DevelopmentRequirementsInstallpoetryInstall dependencies and library locally:poetry installRun CI tests locally:invoke tests --localHow to add a new Circuit Maintenance provider?Define theParsers(inheriting from some of the genericParsersor a new one) that will extract the data from the notification, which could contain multipleDataParts. Thedata_typeof theParserand theDataParthave to match. The customParserswill be placed in theparsersfolder.Update theunit/test_parsers.pywith the new parsers, providing some data to test and validate the extracted data.Define a newProviderinheriting from theGenericProvider, defining theProcessorsand the respectiveParsersto be used. Maybe you can reuse some of the genericProcessorsor maybe you will need to create a custom one. If this is the case, place it in theprocessorsfolder.TheProvideralso supports the definition of a_include_filterand a_exclude_filterto limit the notifications that are actually processed, avoiding false positive errors for notification that are not relevant.Update theunit/test_e2e.pywith the new provider, providing some data to test and validate the finalMaintenancescreated.Expose the newProviderclassupdating the mapSUPPORTED_PROVIDERSincircuit_maintenance_parser/__init__.pyto officially expose theProvider.You can run some tests here to verify that your new unit tests do not cause issues with existing tests, and in general they work as expected. You can do this by runningpytest --log-cli-level=DEBUG --capture=tee-sys. You can narrow down the tests that you want to execute with the-kflag. If successful, your results should look similar to the following:-> % pytest --log-cli-level=DEBUG --capture=tee-sys -k test_parsers
...omitted debug logs...
====================================================== 99 passed, 174 deselected, 17 warnings in 10.35s ======================================================Run some final CI tests locally to ensure that there is no linting/formatting issues with your changes. You should look to get a code score of 10/10. See the example below:invoke tests --local-> % invoke tests --local
LOCAL - Running command black --check --diff .
All done! ✨ 🍰 ✨
41 files would be left unchanged.
LOCAL - Running command flake8 .
LOCAL - Running command find . -name "*.py" | xargs pylint
************* Module tasks
tasks.py:4:0: W0402: Uses of a deprecated module 'distutils.util' (deprecated-module)

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)How to debug circuit-maintenance-parser library locallypoetry installupdates the library and its dependencies locally.circuit-maintenance-parseris now built with your recent local changes.If you were to add loggers or debuggers to one of the classes:classHtmlParserZayo1(Html):defparse_bs(self,btags:ResultSet,data:dict):"""Parse B tag."""raiseException('Debugging exception')After runningpoetry install:-> % circuit-maintenance-parser --data-file ~/Downloads/zayo.eml --data-type email --provider-type zayo
Provider processing failed: Failed creating Maintenance notification for Zayo.
Details:
- Processor CombinedProcessor from Zayo failed due to: Debugging exceptionNote:invoke buildwill result in an error due to no Dockerfile. This is expected as the library runs simple pytest testing without a container.-> % invoke build
Building image circuit-maintenance-parser:2.2.2-py3.8
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 2B done
#1 DONE 0.0s
WARNING: failed to get git remote url: fatal: No remote configured to list refs from.
ERROR: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount1243547759/Dockerfile: no such file or directoryQuestionsFor any questions or comments, please check theFAQfirst and feel free to swing by theNetwork to Code slack channel(channel #networktocode).
Sign uphereLicense notesThis library uses a Basic World Cities Database by Pareto Software, LLC, the owner of Simplemaps.com: The Provider offers a Basic World Cities Database free of charge. This database is licensed under the Creative Commons Attribution 4.0 license as described at:https://creativecommons.org/licenses/by/4.0/.