Package: tfgen
DescriptionThe process log/event log will be used as input for the feature generator. The feature generator will generate transition matrices.How to useInstallationpipinstalltfgen# normal installpipinstall--upgradetfgen# update tfgenHow to useThe observable event classes are first required, and they can be acquired using the method “get observable ec()”.
The generated features will change if the event classes or the order of the event classes are changed, so it is
preferable to save the observable event classes for later use. A dataframe/list/array of attributes should be the
method’s first parameter. The datasets we’ll use for the following code examples are available in releasev0.2.1on Github.fromtfgen.observe_event_classesimportget_observable_ecdata_for_ec=pd.read_csv('test_data_for_ec.csv')ec=get_observable_ec(data_for_ec[['Flags','S/C']])# Flags and S/C are the attributesWe can now start creating the TFGen object. A list of observable event classes that we obtained in the previous step is
the first parameter. The window size is the second variable.fromtfgenimportTFGentfgen=TFGen(ec,window_size=500)We are currently loading the event log data to create features. Before sending the data to TFGen, make sure it is
already in chronological order. EOT marking is required at the ending of each case, and it must be done under each
attribute. Without EOT, the TET will continue to expand. Possibly like this:Example of input data.Case_IDFlagsS/C………13000.ACK.FIN.C13000.ACK.S14000.SYN.C13000.ACK.RST.S13EOTEOT14000.ACK.SYN.S………data_for_feature=pd.read_csv('test_data_with_eot.csv')We can load the offline dataset or load the dataset in an online streaming mode. The method for loading the dataset
in offline mode is:tfgen.load_from_dataframe(data_for_feature,case_id_col='Case_ID',attributes_cols=['Flags','S/C'])output=tfgen.get_output_list()# this will return a list of data.Note that the output is a list (or other iterable) of tuples where each tuple contains two variables
(case_id, transition_table). The case_id comes from the last processed event, and it can be used for labelling the data
for supervised learning or validation. “get_output_list()” can only be used in offline mode.The following example uses the generator as input for online streaming mode.# replace this generator with the actual generatordefreplace_with_the_actual_generator():whileTrue:forrowsindata_for_feature.values:case_id=rows[0]event_attrs=rows[[2,3]]yieldcase_id,event_attrs# event_attr is an iterable with multiple attributes.# Use the generator as an input for online streaming.tfgen.load_from_generator(replace_with_the_actual_generator)out=tfgen.get_output_generator()# this will return a generator as the output.Only the input methods “load_from_dataframe()” or “load_from_generator()” can be used with the output method
“get_output_generator()”.The data can be entered one at a time into TFGen. Due to the fact that TFGen requires several events to initialise,
the output is not guaranteed. To use this method, handle the InitialisingException exception.fromtfgenimportInitialisingExceptiondata_for_feature_array=data_for_feature.valuesforsampleindata_for_feature_array:case_id=sample[0]event_attrs=sample[[2,3]]# tfgen.load_next(<your data sample>). The sample is a tuple of (case_id, event_attrs)# and event_attrs is an iterable with multiple attributes.tfgen.load_next(case_id,event_attrs)try:print(tfgen.get_output_next())exceptInitialisingException:continueThe output method “get_output_next()” is compatible with all input methods.MethodsCurrently, the “Classic” and the “ClassicLargeSparse” methods for feature generation are available. The “Classic”method
is employed by default. The “ClassicLargeSparse” method can be used to output Scipy sparse matrices for event logs that
contain a larger number of event classes.fromtfgenimportTFGentfgen=TFGen(ec,window_size=500,method=TFGen.ClassicLargeSparse)Implementing New MethodsBy deriving from the “BaseMethod” located in “tfgen/methods/base method.py,” one can extend the existing methods by
creating new method classes. All classes must be placed under “tfgen/methods/” directory. The next event sample must
be obtained using method “self.get_next_data(),” and the generated feature must be sent to the output using
method “self.send_data()”.
“self.finished” will become “True” if the input stream reaches the end.fromtfgen.methods.base_methodimportBaseMethodclassNewMethod(BaseMethod):def__init__(self,ec_lookup_table,window_size,input_stream,output_stream):super().__init__(ec_lookup_table,window_size,input_stream,output_stream)# entrydefstart_processing(self):whileTrue:# event is a tuple of (case_id, event_attrs)event=self.get_next_data()# do somethingself.send_data(processed_data)ifself.finished:breakThen include the new method in the “TFGen” class found in “tfgen/tfgen.py”. Two locations are required to be modified.classTFGen:METHOD_CLASSIC=101METHOD_CLASSIC_LARGE_SPARSE=102METHOD_NEW_METHOD=103# The first location. New method classdef_select_method(self,method):ifself.method==TFGen.METHOD_CLASSIC:returnClassic(self.ec_lookup,self.window_size,self.input_stream,self.output_stream)elifself.method==TFGen.METHOD_CLASSIC_LARGE_SPARSE:returnClassicLargeSparse(self.ec_lookup,self.window_size,self.input_stream,self.output_stream)# The second location. The instance to the new method classelifself.method==TFGen.METHOD_NEW_METHOD:returnNewMethod(self.ec_lookup,self.window_size,self.input_stream,self.output_stream)else:raiseException("Method not supported")