Package: absynthe
Absynthe: A (branching) Behavior SynthesizerMotivationAbsynthe came about in response to the need for test data for analysizing the
performance and accuracy of log analysis algorithms. Even though plenty of real
life logs are available, e.g./var/log/in unix-based laptops, they do not
serve the purpose of test data. For that, we need to understand the core
application logic that is generating these logs.A more interesting situation arises while trying to test log analytic (and
anomaly detection) solutions for distributed applications where multiple
sources or modules emit their respective log messages in a single log queue or
stream. This means that consecutive log lines could have originated from
different, unrelated application components. Absynthe providesground truthmodels to simulate such situations.You need Absynthe if you wish to simulate the behavior of any well defined
process -- whether it's a computer application or a business process flow.OverviewEach business process or compuater application is modelled as acontrol flow
graph(orCFG), which typically has one or more roots (i.e. entry) nodes and
multiple leaf (i.e. end) nodes.Tree-like CFGAn example of a simple, tree-like CFG generated using Absynthe is shown below.
This is like a tree since nodes are laid out in levels, and nodes at levelihave outgoing edges only to nodes at leveli + 1.Eachbehavioris the sequence of nodes encountered while traversing this CFG
from a root to a leaf. Of course, a CFG might contain loops which could be
traversed multiple times before arriving at the leaf. Moreover, if there are
multiple CFGs, then Absynthe can synthesizeinterleavedbehaviors. This means
that a single sequence of nodes might contain nodes from multiple CFGs. We are
ultimately interested in this interleaving behavior, which is produced by
multiple CFGs.The above screenshot shows logs generated by Absynthe. Each log line starts
with a time stamp, followed by a session ID, CFG ID, and a log message. At
present, the log message is simply a random concatenation of the node ID to
which the log message corresponds. A single CFG might participate in multiple
sessions, where each session is a different traversal of the CFG. Therefore, we
maintain both session ID and CFG ID in the log line.Directed Cyclic CFGAn example of a more complex CFG, a directed cyclic graph, is shown in the
figure below. It expands the tree-like graph illustrated above by:attaching loops on some of the nodes,constructing skip-level edges, i.e. edges from a node at levelito a
node at level ≥(i + 2), andoptionally, upward edges (not shown here), i.e. edges from a node at
levelito a node at level ≤(i - 1).The identifiers of nodes appearing loops are helpfully prefixed with the
identifiers of nodes where these loops start and finish. Moreover, loops could
be traversed multiple times in a single behavior, as illustrated in the figure
below.InstallationThis package has been developed withPython 3.6.*and depends onscipy 1.2.1.
Things might not work withPython 3.7.*orscipy 1.3.*. Therefore, consider
creating a virtual environment if your default configuration differs.The latest release is available on PyPi, simplypip install absynthe. Themasterbranch of this repository will always provide the latest release.For the latest features not yet released, clone or download thedevelopbranch
and then:# Change dir to absynthecd/path/to/absynthe# Install dependenciespipinstall-rrequirements.txt# Install absynthepipinstall.UsageIt is possible to start using Absynthe with two classes:any concrete implementation of the abstractGraphBuilderclass, which
generates CFGs, andany concrete implementation of the abstractBehaviorclass, which
traverses the CFGs generated above and emits log messages.For instance, consider thebasicLogGenerationmethod in./examples/01_generateSimpleBehavior.py:fromabsynthe.graph_builderimportTreeBuilderfromabsynthe.behaviorimportMonospaceInterleavingdefbasicLogGeneration(numRoots:int=2,numLeaves:int=4,branching:int=2,numInnerNodes:int=16,loggerNodeTypes:str="SimpleLoggerNode"):# Capture all the arguments required by GraphBuilder classtree_kwargs={TreeBuilder.KW_NUM_ROOTS:str(numRoots),TreeBuilder.KW_NUM_LEAVES:str(numLeaves),TreeBuilder.KW_BRANCHING_DEGREE:str(branching),TreeBuilder.KW_NUM_INNER_NODES:str(numInnerNodes),TreeBuilder.KW_SUPPORTED_NODE_TYPES:loggerNodeTypes}# Instantiate a concrete GraphBuilder. Note that the# generateNewGraph() method of this class returns a# new, randomly generated graph that (more or less)# satisfies all the parameters provided to the# constructor, viz. tree_kwargs in the present case.simpleTreeBuilder=TreeBuilder(**tree_kwargs)# Instantiate a concrete behavior generator. Some# behavior generators do not print unique session ID# for each run, but it's nice to have those.wSessionID:bool=TrueexBehavior=MonospaceInterleaving(wSessionID)# Add multiple graphs to this behavior generator. The# behaviors that it will synthesize would essentially# be interleavings of simultaneous traversals of all# these graphs.exBehavior.addGraph(simpleTreeBuilder.generateNewGraph())exBehavior.addGraph(simpleTreeBuilder.generateNewGraph())exBehavior.addGraph(simpleTreeBuilder.generateNewGraph())exBehavior.addGraph(simpleTreeBuilder.generateNewGraph())# Specify how many behaviors are to be synthesized,# and get going.numTraversalsOfEachGraph:int=2forlogLineinexBehavior.synthesize(numTraversalsOfEachGraph):print(logLine)returnIn order to generate behaviors from a directed cyclic CFG, create a DCG as shown
in./examples/03_generateControlFlowDCG.pyand then generate behaviors after
adding the DCG to a behavior object as shown in the code snippet above.Note:When generating a behavior, i.e. when traversing a graph, successors
of nodes are chosen based on the probability distributions associated with those
nodes. Different nodes rely on different distributions and these nodes are
randomly assigned in the graphs that are constructed bygenerateNewGraph()methods, resulting in graphs with a mix of nodes.Release NotesNote:This tool is still in alpha stage, so backward compatibility is not
guaranteed between releases. However, inasmuch as users stick to graph builders'generateNewGraph()methods, they will stay away from compatibility problems.Major changes in v0.0.2Added new graph builders, viz.DAGBuilderandDCGBuilder, which build
CFGs with skip-level edges and loops respectively.Added new node, viz.BinomialNode, which exploits the binomial distribution
in order to select its successors at the time of graph traversal.Added a separate utility class calledUtilsinabsynthe.cfg.utils.pyto
create a newNodeobject from any of the concrete implementations ofNodeat
random. All concrete implementations ofNodetherefore transparently available
to graph builders (and everyone else) through this utility.Coming up in future releasesSophisticated interleaving behaviorsLogger nodes that emit morelife likelog messagesAnomalousbehaviors