Package: ahbicht
A python package that parses condition expressions from EDI@Energy
Anwendungshandbücher (AHB). Since it’s based on lark, we named the
module AHBicht.What is this all about?The German energy market usesEDIFACTas an intercompany
data exchange format. The rules on how to structure and validate the
EDIFACT messages are written inoneMessageImplementationGuide (MIG) per EDIFACT
format (for example UTILMD or MSCONS)oneAnwendungshandbuch (AHB,
en. manual) per use case group (for exampleGPKEorWechselprozesse im Messwesen(WiM))According to the legislation for the German energy market, the
organisations in charge of maintaining the documents described above
(AHB and MIGs) are theBundesverbandderEnergie-
undWasserwirtschaft (BDEW) and theBundesnetzagentur (BNetzA). They form a working
group named “Arbeitsgruppe EDI@Energy”. This work group publishes the
MIGs and AHBs onedi-energy.de. The
documents are published as PDFs which is better than faxing them but far
from ideal.The AHBs contain information on how to structure single EDIFACT
messages. To create messages that are valid according to the respective
AHB, you have to process information of the kind:In this example:This library parses the string
``Muss [210] U ([182] X ([90] U [183]))`` and allows determining whether
“Details der Prognosegrundlage” is an obligatory field according to the
AHB, iff the individual status of the conditions is given.We call
this “expression evaluation”.Note that determining the individual status of[210],[182],[90]and[183]itself (the so called “content evaluation”, see
below) isnotwithin the scope of this parsing library.Note also, that this library also parses the new convention using logical operators that becomes effective 2022-04-01 (“MaKo2022”).Muss [210] ∧ ([182] ⊻ ([90] ∧[183])).Usage and ExamplesJupyter NotebookFor a minimal working example on how what the library is used, check outthis Jupyter notebook.Free to Use REST APIYou can also use our public REST API to parse condition expressions (other features will follow). Simply send a GET request with the condition expression as query parameter to:ahbicht.azurewebsites.net/api/ParseExpression?expression=[2] U ([3] O [4])[901] U [555]Easily Integrate AHBicht with Your SolutionIf you want to use AHBicht together with your own software, you can use theJSON Schema filesprovided to kick start the integration.Code Quality / Production ReadinessThe code has at least a 95% unit test coverage. ✔️The code is rated 10/10 in pylint and type checked with mypy. ✔️The code isMIT licensed. ✔️There are onlyfew dependencies. ✔️Expression Evaluation / Parsing the Condition StringEvaluating expressions likeMuss [59] U ([123] O [456])from the
AHBs by parsing it with theparsing library
larkand
combining the parsing result with information about the state of[59],[123],[456]is calledexpression evaluation.
Determining the state of each single condition (f.e.[59]is
fulfilled,[123]is not fulfilled,[456]is unknown) for a given
message is part of thecontent evaluation(see next chapter).If you’re new to this topic, please readedi-energy.de → Dokumente →
Allgemeine
Festlegungenfirst. This document contains German explanations, how the Bedingungen
are supposed to be read.FunctionalityExpressions can contain single numbers e.g.[47]or numbers
combined withU/O/Xor∧/∨/⊻respectively which are translated to boolean
operatorsand/or/exclusive or, e.g.[45]U[2]or they
can be combinedwithoutan operator, e.g.[930][5]in the
case of FormatConstraints.Expressions can contain random whitespaces.Input conditions are passed in form of aConditionNode, see
below.Bedingungen/RequirementConstraintswith a boolean value,
Hinweise/Hintsand Formatdefinitionen/FormatConstraintsare
so far functionally implemented as the result returns if the
condition expression is fulfilled and which Hints and
FormatConstraints are relevant.The boolean logic follows ‘brackets( )beforethen_alsobeforeandbeforeor’.Hints and UnevaluatedFormatConstraints are implemented asneutralelement, so not changing the boolean outcome of an expression for the
evaluation regarding the requirement constraints and raising errors
when there is no sensible logical outcome of the expression.Acondition_fulfilledattribute can also take the valueunknown.Brackets e.g.([43]O[4])U[5]Requirement indicators (i.eMuss,Soll,Kann,X,O,U) are seperated from the condition expressions and also
seperated into single requirement indicator expressions if there are
more than one (for modal marks).Format Constraint Expressionsthat are returned after the
requirement condition evaluation can now be parsed and evaluated.Evaluate several modal marks in one ahb_expression: the first one
that evaluates to fulfilled is the valid one.In planningEvaluate requirement indicators:Soll, Kann, Muss, X, O, U -> is_required, is_forbidden, etc…Definition of termsTermDescriptionExampleconditionsingle operand[53]condition_keyint or str, the number of the condition53operatorcombines two conditionsU, Ocompositiontwo parts of an expression combined by an operator([4]U[76])O[5] consists of an and_composition of [4] and [76] and an or_composition of [4]U[76] and [5]used in the context of the parsing and evaluation of the expressionahb expressionan expression as given from the ahbX[59]U[53]Consists of at least one single requirement indicator expression.Muss[59]U([123]O[456])Soll[53]In case of several model mark expressions the first one will be evaluated and if not fulfilled, it will be continued with the next one.single requirement indicator expressionAn expression consisting of exactly one requirement indicator and their respective condition expression.Soll[53]If there is only one requirement indicator in the ahb expression, then both expressions are identical.condition expressionone or multiple conditions combined with or (in case of FormatConstraints) also without operators[1]used as input for the condition parser[4]O[5]U[45]format constraint expressionIs returned after the evaluation of the RequirementConstraints[901]X[954]consist only of FormatConstraintsrequirement indicatorThe Merkmal/modal_mark or Operator/prefix_operator of the data element/data element group/segment/segment group.Muss, Soll, Kann, X, O, UMerkmal / modal_markas defined by the EDI Energy group (see edi-energy.de → Dokumente → Allgemeine Festlegungen)Muss, Soll, KannStands alone or before a condition expression, can be the start of several requirement indicator expressions in one ahb expressionprefix operatorOperator which does not function to combine conditions, but as requirement indicator.X, O, UStands alone or in front of a condition expression.tree, branches, tokenas used by larkConditionNodeDefines the nodes of the tree as they are passed, evaluated und returned.RequirementConstraint, FormatConstraint, Hint, EvaluatedComposition, RepeatabilityConstraintThere are different kinds of conditions (Bedingung, Hinweis, Format) as defined by the EDI Energy group (see edi-energy.de → Dokumente → Allgemeine Festlegungen) and also a EvaluatedComposition after a composition of two nodes is evaluated.Bedingung / RequirementConstraint (rc)are true or false, has to be determined“falls SG2+IDE+CCI == EHZ”keys between [1] and [499]Wiederholbarkeit / RepeatabilityConstraintgives minimum and maximum occurrence“Segmentgruppe ist mindestens einmal je SG4 IDE+24 (Vorgang) anzugeben”keys between [2000] and [2499]Hinweis / Hintjust a hint, even if it is worded like a condition“Hinweis: ‘ID der Messlokation’”keys from [500] onwards, starts with ‘Hinweis:’“Hinweis: ‘Es ist der alte MSB zu verwenden’”Formatdefinition / FormatConstraint (fc)a constraint for how the data should be given“Format: Muss größer 0 sein”keys between [901] and [999], starts with ‘Format:’“Format: max 5 Nachkommastellen”Format Constraints are “collected” while evaluating the rest of the tree, meaning the evaluated composition of the Mussfeldprüfung contains an expression that consists only of format constraints.UnevaluatedFormatConstraintA format constraint that is just “collected” during the requirement constraint evaluation. To have a clear separation of conditions that affect whether a field is mandatory or not and those that check the format of fields without changing their state it will become a part of the format_constraint_expression which is part of the EvaluatedComposition.EvaluatableFormatConstraintAn evaluatable FormatConstraint will (other than the UnevaluatedFormatConstraint) be evaluated by e.g. matching a regex, calculating a checksum etc. This happens after the Mussfeldprüfung. (details to be added upon implementing)EvaluatedCompositionis returned after a composition of two nodes is evaluatedPackage Resolvera package resolver is a class that replaces package nodes in a tree with a sub tree that is derived from a package definition. Replacing package nodes with sub trees is referred to as “package expansion”Example: “[123P]” is replaced with a tree for “[5]U[6]O[7]”neutralHints and UnevaluatedFormat Constraints are seen as neutral as they don’t have a condition to be fulfilled or unfulfilled and should not change the requirement outcome. See truth table below.unknownIf the condition can be fulfilled but we don’t know (yet) if it is or not. See truth table below.“Wenn vorhanden”The decision if a requirement constraint is met / fulfilled / true is
made in the content evaluation module.Program structureThe following diagram shows the structure of the condition check for
more than one condition. If it is only a single condition or just a
requirement indicator, the respective tree consists of just this token
and the result equals the input.The raw and updated data for this diagram can be found in thedraw_io_charts
repositoryand edited underapp.diagrams.netwith your GitHub Account.There is also anUML Diagramavailable (last updated 2022-01-29).Truth tablesAdditionally to the usual boolean logic we also haveneutralelements (e.g.Hints,UnevaluatedFormatConstraintsand in some
casesEvaluatedCompositions) orunknownrequirement constraints.
They are handled as follows:and_compositionABA U BNeutralTrueTrueNeutralFalseFalseNeutralNeutralNeutralUnknownTrueUnknownUnknownFalseFalseUnknownUnknownUnknownUnknownNeutralUnknownor_compositionABA O BnoteNeutralTruedoes not make senseNeutralFalsedoes not make senseNeutralNeutralNeutralno or_compositions of
hint and format
constraintUnknownTrueTrueUnknownFalseUnknownUnknownUnknownUnknownUnknownNeutraldoes not make sensexor_compositionABA X BnoteNeutralTruedoes not make senseNeutralFalsedoes not make senseNeutralNeutralNeutralno xor_compositions
of hint and format
constraintUnkownTrueUnknownUnkownFalseUnknownUnkownUnknownUnknownUnkownNeutraldoes not make senseLink to automatically generate HintsProvider Json content:https://regex101.com/r/za8pr3/5Content EvaluationEvaluation is the term used for the processing ofsingleunevaluated
conditions. The results of the evaluation of all relevant conditions
inside a message can then be used to validate a message. The latter isnotpart of the evaluation.This library doesnotprovide content evaluation code for all the
conditions used in the available AHBs. You can use the Content
Evaluation class stubs though. Please contact@JoschaMetzeif you’re interested in a
ready-to-use solution to validate your EDIFACT messages according to the
latest AHBs. We probably have you covered.EvaluatableData (Edifact Seed and others)For the evaluation of a condition (that is referenced by its key,
e.g. “17”) it is necessary to have a data basis that allows to decide
whether the respective condition is met or not met. This data basis that
is stable for all conditions that are evaluated in on evaluation run is
calledEvaluatableData. These data usually contain theedifact
seed(a JSON representation of the EDIFACT message) but may also hold
other information. TheEvaluatableDataclass acts a container for
these data.EvaluationContext (Scope and others)While the data basis is stable, the context in which a condition is
evaluated might change during on evaluation run. The same condition can
have different evaluation results depending on e.g. in which scope it is
evaluated. Ascopeis a (json) path that references a specific
subtree of the edifact seed. For example one “Vorgang” (SG4 IDE) in
UTILMD could be a scope. If a condition is described asThere has to be exactly one xyz per Vorgang (SG4+IDE) Then fornVorgänge there arenscopes:one scope for each Vorgang (pathes refer to an edifact seed):$["Dokument"][0]["Nachricht"][0]["Vorgang"][0]$["Dokument"][0]["Nachricht"][0]["Vorgang"][1]…$["Dokument"][0]["Nachricht"][0]["Vorgang"][<n-1>]Each of the single vorgang scopes can have a different evaluation
result. Those results are relevant for the user when entering data,
probably based in a somehow Vorgang-centric manner.TheEvaluationContextclass is a container for the scope and
other information that are relevant for a single condition and a single
evaluation only but (other thanEvaluatableData) might change within
an otherwise stable message.ReleasingThe version number has to be changed insetup.cfgfile.ContributingYou are very welcome to contribute to this repository by opening a pull
request against the main branch.How to use this Repository on Your Machine / Local SetupPlease follow the instructions in ourPython Template Repository.