Package: th2-data-services
Table of ContentsTable of Contents1. Introduction2. Getting started2.1. InstallationCoreData sources (providers)GRPC provider warningReasons for the restriction2.2. Example2.3. Short theoryTermsConceptStream operationsPipeliningInternal iterationData cachingForced cachingEventsTree and collectionsEventsTreeCollectionsHints2.4. Links3. API4. Examples4.1. Notebooks4.2. *.py1. IntroductionThis repository is a library for creating th2-data-services applications.The library used to analyze stream data usingaggregate operationsmainly from
the"Report Data Provider". Data Services allows you to manipulate
the stream data processing workflow usingpipelining.The library allows you:Natively connect to"Report Data Provider"viaProviderDataSourceclass and extract TH2 Events/Messages viacommandsWork with iterable objects (list, tuple, etc including files) viaData objectusing its featuresManipulate the workflow to make some analysis byData objectmethodsBuild Event Trees (EventsTreeCollectionclass)Workflow manipulation tools allows you:Filtering stream data (Data.filtermethod)Transforming stream data (Data.mapmethod)Limiting the number of processed streaming data (Data.limitmethod)There is also another part ofdata servicesth2-data-services-utils. It's a set of tools to perform the most
common analysis tasks.2. Getting started2.1. InstallationCoreFrom PyPI (pip)This package can be found onPyPI.pip install th2-data-servicesFrom Sourcegit clone https://github.com/th2-net/th2-data-services
pip install th2-data-services/Data sources (providers)Sincev1.3.0, the library doesn't provide data source dependencies.You should provide it manually during installation.
You just need to add square brackets after library name and put dependency name.pip install th2-data-services[dependency_name]Dependencies listdependency nameprovider versionRDP55RDP66Examplepip install th2-data-services[rdp5]GRPC provider warningThis library has ability to interact with several versions of grpc providers, but it's limited by installed version ofth2_grpc_data_providerpackage version. You can use only appropriate version of provider api, which is compatible with
installed version ofth2_grpc_data_provider.By default,th2_data_servicesuses the latest available version of provider api version.Reasons for the restrictionTwo different versions ofth2_grpc_data_providercan't be installed in the same virtual environment;Two different versions of packageth2_grpc_data_providermay depend on different versions of packagesth2_grpc_common;In the case of using another package in the process of usingth2_data_services(for exampleth2_common),
which also depends onth2_grpc_common, a version conflict may occur (both at the Python level and at the Protobuf level).2.2. ExampleA good, short example is worth a thousand words.This example works withEvents, but you also can do the same actions withMessages.The following example as a file.fromcollectionsimportGeneratorfromtypingimportTuple,List,Optionalfromdatetimeimportdatetimefromth2_data_servicesimportDatafromth2_data_services.events_treeimportEventsTreefromth2_data_services.provider.v5.data_source.httpimportHTTPProvider5DataSourcefromth2_data_services.provider.v5.commandsimporthttpascommandsfromth2_data_services.provider.v5.events_treeimportEventsTreeCollectionProvider5,ParentEventsTreeCollectionProvider5fromth2_data_services.provider.v5.filters.event_filtersimportNameFilter,TypeFilter,FailedStatusFilterfromth2_data_services.provider.v5.filters.message_filtersimportBodyFilter# [0] Lib configuration# [0.1] Interactive or Script mode# If you use the lib in interactive mode (jupyter, ipython) it's recommended to set the special# global parameter to True. It'll keep cache files if something went wrong.importth2_data_servicesth2_data_services.INTERACTIVE_MODE=True# [1] Create DataSource object to connect to rpt-data-provider.DEMO_HOST="10.100.66.66"# th2-kube-demo  Host port where rpt-data-provider is located.DEMO_PORT="30999"# Node port of rpt-data-provider.data_source=HTTPProvider5DataSource(f"http://{DEMO_HOST}:{DEMO_PORT}")START_TIME=datetime(year=2021,month=6,day=17,hour=9,minute=44,second=41,microsecond=692724)# Datetime in utc format.END_TIME=datetime(year=2021,month=6,day=17,hour=12,minute=45,second=50)# [2] Get events or messages from START_TIME to END_TIME.# [2.1] Get events.events:Data=data_source.command(commands.GetEvents(start_timestamp=START_TIME,end_timestamp=END_TIME,attached_messages=True,# Use Filter class to apply rpt-data-provider filters.# Do not use multiple classes of the same type.filters=[TypeFilter("Send message"),NameFilter(["ExecutionReport","NewOrderSingle"]),# You can use multiple values.FailedStatusFilter(),],))# [2.2] Get messages.messages:Data=data_source.command(commands.GetMessages(start_timestamp=START_TIME,end_timestamp=END_TIME,attached_events=True,stream=["demo-conn2"],filters=BodyFilter("195"),# Filter message if there is a substring '195' in the body.))# [3] Work with a Data object.# [3.1] Filter.filtered_events:Data=events.filter(lambdae:e["body"]!=[])# Filter events with empty body.# [3.2] Map.deftransform_function(record):return{"eventName":record["eventName"],"successful":record["successful"]}filtered_and_mapped_events=filtered_events.map(transform_function)# [3.3] Data pipeline.#       Instead of doing data transformations step by step you can do it in one line.filtered_and_mapped_events_by_pipeline=events.filter(lambdae:e["body"]!=[]).map(transform_function)# Content of these two Data objects should be equal.assertlist(filtered_and_mapped_events)==list(filtered_and_mapped_events_by_pipeline)# [3.4] Sift. Skip the first few items or limit them.events_from_11_to_end:Generator=events.sift(skip=10)only_first_10_events:Generator=events.sift(limit=10)# [3.5] Changing cache status.events.use_cache(True)# or justevents.use_cache()# If you want to activate cache.# [3.6] Walk through data.foreventinevents:# Do something with event (event is a dict).print(event)# After first iteration the events has a cache file.# Now they will be used in the cache in the next iteration.# [3.7] Get number of the elements in the Data object.number_of_events=events.len# [3.8] Check that Data object isn't empty.# The data source should be not empty.assertevents.is_emptyisFalse# [3.9] Convert Data object to the list of elements(events or messages).# Be careful, this can take too much memory.events_list=list(events)# [3.10] Get event/message by id.desired_event="9ce8a2ff-d600-4366-9aba-2082cfc69901:ef1d722e-cf5e-11eb-bcd0-ced60009573f"desired_events=["deea079b-4235-4421-abf6-6a3ac1d04c76:ef1d3a20-cf5e-11eb-bcd0-ced60009573f","a34e3cb4-c635-4a90-8f42-37dd984209cb:ef1c5cea-cf5e-11eb-bcd0-ced60009573f",]desired_message="demo-conn1:first:1619506157132265837"desired_messages=["demo-conn1:first:1619506157132265836","demo-conn1:first:1619506157132265833",]data_source.command(commands.GetEventById(desired_event))# Returns 1 event (dict).data_source.command(commands.GetEventsById(desired_events))# Returns 2 events list(dict).data_source.command(commands.GetMessageById(desired_message))# Returns 1 message (dict).data_source.command(commands.GetMessagesById(desired_messages))# Returns 2 messages list(dict).# [3.11] The cache inheritance.# Creates a new Data object that will use cache from the events Data object.events_filtered:Data=events.filter(lambdarecord:record.get("batchId"))# New Data objects don't use their own cache by default but use the cache of the parent Data object.# Use use_cache method to activate caching.# After that, the Data object will create its own cache file.events_filtered.use_cache()list(events_filtered)# Just to iterate Data object (cache file will be created).filtered_events_types=events_filtered.map(lambdarecord:{"eventType":record.get("eventType")})events_without_types_with_batch=filtered_events_types.filter(lambdarecord:notrecord.get("eventType"))events_without_types_with_batch.use_cache()# [3.12] Data objects joining.# You have the following 3 Data objects.d1=Data([1,2,3])d2=Data(["a",{"id":123},"c"])d3=Data([7,8,9])# You can join Data objects in following ways.data_via_init=Data([d1,d2,d3])data_via_add=d1+d2+d3data_with_non_data_obj_via_init=Data([d1,["a",{"id":123},"c"],d3])data_with_non_data_obj_via_add=d1+["a",{"id":123},"c"]+d3# [3.13] Build and read Data object cache files.events.build_cache("cache_filename_or_path")data_obj_from_cache=Data.from_cache_file("cache_filename_or_path")# [4] Working with EventsTree and EventsTreeCollection.# [4.1] Building the EventsTreeCollection.# If you don't specify data_source for the tree then it won't recover detached events.collection=EventsTreeCollectionProvider5(events)# Detached events isn't empty.assertcollection.detached_eventscollection=EventsTreeCollectionProvider5(events,data_source=data_source)# Detached events are empty because they were recovered.assertnotcollection.detached_events# The collection has EventsTrees each with a tree of events.# Using Collection and EventsTrees, you can work flexibly with events.# [4.1.1] Get leaves of all trees.leaves:Tuple[dict]=collection.get_leaves()# [4.1.2] Get roots ids of all trees.roots:List[str]=collection.get_roots_ids()# [4.1.3] Find an event in all trees.find_event:Optional[dict]=collection.find(lambdaevent:"Send message"inevent["eventType"])# [4.1.4] Find all events in all trees. There is also iterable version 'findall_iter'.find_events:List[dict]=collection.findall(lambdaevent:event["successful"]isTrue)# [4.1.5] Find an ancestor of the event.ancestor:Optional[dict]=collection.find_ancestor("8bbe3717-cf59-11eb-a3f7-094f904c3a62",filter=lambdaevent:"RootEvent"inevent["eventName"])# [4.1.6] Get children of the event. There is also iterable version 'get_children_iter'.children:Tuple[dict]=collection.get_children("814422e1-9c68-11eb-8598-691ebd7f413d")# [4.1.7] Get subtree for specified event.subtree:EventsTree=collection.get_subtree("8e23774d-cf59-11eb-a6e3-55bfdb2b3f21")# [4.1.8] Get full path to the event.# Looks like [ancestor_root, ancestor_level1, ancestor_level2, event]event_path:List[dict]=collection.get_full_path("8e2524fa-cf59-11eb-a3f7-094f904c3a62")# [4.1.9] Get parent of the event.parent=collection.get_parent("8e2524fa-cf59-11eb-a3f7-094f904c3a62")# [4.1.10] Append new event to the collection.collection.append_event(event={"eventId":"a20f5ef4-c3fe-bb10-a29c-dd3d784909eb","parentEventId":"8e2524fa-cf59-11eb-a3f7-094f904c3a62","eventName":"StubEvent",})# [4.1.11] Show the entire collection.collection.show()# [4.2] Working with the EventsTree.# EventsTree has the same methods as EventsTreeCollection, but only for its own tree.# [4.2.1] Get collection trees.trees:List[EventsTree]=collection.get_trees()tree:EventsTree=trees[0]# But EventsTree provides a work with the tree, but does not modify it.# If you want to modify the tree, use EventsTreeCollections.# [4.3] Working with ParentlessTree.# ParentlessTree is EventsTree which has detached events with stubs.parentless_trees:List[EventsTree]=collection.get_parentless_trees()# [4.4] Working with ParentEventsTreeCollection.# ParentEventsTreeCollection is a tree like EventsTreeCollection but it has only events that have references.collection=ParentEventsTreeCollectionProvider5(events,data_source=data_source)collection.show()2.3. Short theoryThe library provides tools for handling stream data. What’s a stream? It's a sequence of elements from a source that
supports aggregate operations.TermsData object: An instance ofDataclass which is wrapper under stream.Sequence of elements:
AData objectprovides an interface to a sequenced set of values of a specific element type. Stream inside theData
objectdon’t actually storeelements; they are computed on demand.DataSource:
Any source of data. E.g.Report Data Provider, collections,
arrays, or I/O resources.ProviderDataSource:
The DataSource object whose source isReport Data Provider.SourceAPI:
Each source has its own API to retrieve data. SourceAPI is a class that provide API for some data source.Commands:
Objects that provide user-friendly interfaces for getting some data from DataSource. Commands useSourceAPIto
achieve it.Adapters:
It's similar to function forData.mapmethod. Adoptable commands used it to update the data stream.Aggregate operations:
Common operations such as filter, map, limit and so on.Workflow: An ordered set ofAggregate operations.ConceptThe library describes the high-level interfacesISourceAPI,IDataSource,ICommand,IAdapter.Any data source must be described by theIDataSourceabstract class. These can beFileDataSource,CSVDataSource, _
DBDataSource_ and other.Usually, data sources have some kind of API. Databases - provide SQL language, when working with a file, you can read
line by line, etc. This API is described by theISourceAPIclass. Because different versions of the same data source
may have different API, it is better to create a class for each version.Generally, data source APIs are hidden behind convenient interfaces. The role of these interfaces is played
byICommandclasses.IAdapterclasses transform data stream like functions forData.mapmethod. Essentially it's the same thing but more
flexible.Thus, the nativeProviderDataSourceand the set of commands for it are described. This approach provides great
opportunities for extension. You can easily create your own unique commands forProviderDataSource, as well as entireDataSourceclasses.Stream operationsFurthermore, stream operations have two fundamental characteristics that make them very different from collection
operations:PipeliningandInternal iteration.PipeliningMany stream operations return a stream themselves. This allows operations to be chained to form a larger pipeline.Internal iterationIn contrast to collections, which are iterated explicitly (external iteration), stream operations do the iteration
behind the scenes for you. Note, it doesn't mean you cannot iterate theData object.Data cachingTheData objectprovides the ability to use the cache. The cache works for eachData object, that is, you choose
whichData objectyou want to save. TheData objectcache is saved after the first iteration, but the iteration
source may be different.If you don't use the cache, your source will be the data source you have in theData Object. But if you use the cache,
your source can be the data source, the parent cache, or own cache:The data source:
If theData Objectdoesn't have a parent cache and its cache.The parent cache:
If theData Objecthas a parent cache. It doesn't matter what position the parent cache has in inheritance.Data Objectunderstands whose cache it is and executes the part of the workflow that was not executed.The own cache:
If it is not the first iteration of this Data object.Note that the cache state of the Data object is not inherited.Forced cachingYou can tell DS to cache data to specific cache file, which won't be deleted after script end:importdatetimefromth2_data_servicesimportDatafromth2_data_services.provider.v5.commandsimporthttpfromth2_data_services.provider.v5.data_sourceimportHTTPProvider5DataSourcedata_source=HTTPProvider5DataSource("http://HOST:PORT")events:Data=data_source.command(http.GetEvents(start_timestamp=datetime.datetime.utcnow()-datetime.timedelta(minutes=5),end_timestamp=datetime.datetime.utcnow(),attached_messages=True,cache=True,))events.build_cache("my_cache.pickle")Later you can createDataobject from this cache file and use it as usual:fromth2_data_servicesimportDataevents=Data.from_cache_file("my_cache.pickle")forevent_idinevents.filter(lambdax:x["eventType"]=="Verification").map(lambdax:x["eventId"]):print(event_id)EventsTree and collectionsEventsTreeEventsTree is a tree-based data structure of events. It allows you get children and parents of event, display tree, get
full path to event etc.Details:EventsTree contains all events in memory.To reduce memory usage an EventsTreeCollection delete the 'body' field from events, but you can preserve it specify '
preserve_body'.Tree has some important terms:Ancestoris any relative of the event up the tree (grandparent, parent etc.).Parentis only the first relative of the event up the tree.Childis the first relative of the event down the tree.Take a look at the following HTML tree to understand them.<body> <!-- ancestor (grandparent), but not parent -->
     <div> <!-- parent & ancestor -->
         <p>Hello, world!</p> <!-- child -->
         <p>Goodbye!</p> <!-- sibling -->
     </div>
 </body>CollectionsEventsTreeCollectionis a collection of EventsTrees. The collection builds a fewEventsTreeby passedData
object. Although you can change the tree directly, it's better to do it through collections because they are aware ofdetached_eventsand can solve some events dependencies. The collection has similar features like a singleEventsTreebut applying them for all EventsTrees.ParentEventsTreeCollectionis a collection similar to EventsTreeCollection but containing only parent events that
are referenced in the data stream. It will be working data in the collection and trees of collection. The collection has
features similar to EventsTreeCollection.Details:The collection has a feature to recover events. All events that are not in the received data stream, but which are
referenced will be loaded from the data source.If you haven't passed aDataSource objectthen the recovery of events will not occur.You can takedetached_eventsto see which events are missing. It looks like{parent_id: [events are referenced]}If you want, you can build parentless trees where the missing events are stubbed instead. Just
useget_parentless_trees().Requirements:Events have to haveevent_name,event_id,parent_event_idfields, which are described in the
passedevent_structobject.HintsRemove all unnecessary fields from events before passing to acollectionto reduce memory usage.Useshow()method to print the tree in tree-like view.Note that theget_xmethods will raise an exception if you pass an unknown event id, unlike thefind_xmethods (
they return None).If you want to know that specified event exists, use the pythoninkeyword (e.g.'event-id' in events_tree).Use the pythonlenkeyword to get events number in the tree.2.4. LinksReport Data ProviderTh2 Data Services Utils3. APIIf you are looking for classes description see theAPI Documentation.4. Examples4.1. Notebooksnotebook_0.ipynb4.2. *.pyget_started_example.py