Package: nespy
NebulaStream Python ClientNebulaStream's python client is the first data science environment for NebulaStream.
It allows users to...use operators to execute queries in NebulaStream efficientlyperform data preparation tasks on data streamstrain machine learning models with easeintegrate any python libraryExampleAs an example, we want to perform a data exploration task.
First, we need to connect with NebulaStream and select the data stream "vehicles". This data stream contains data from all by NebulaStream registered vehicles in the berlin, including their current position, their speed, the total driven route, and vehicle type.>>>importnespyasnes>>>c=nes.Connection('127.0.0.1',8081)>>>vehicles=c.get_logical_stream("vehicles")>>>vehiclestypeidlatlontimestampspeed in km/hroute (in meters)6152.51312613.32834715127.0004252.51310913.3263881255.356352.51276313.32529116210.0009452.51267713.32596415110.0003552.51225913.3227781291.89What we want to do now is to find out the average speed over the last five minutes of all cars in berlin.
Therefore, we only filter cars (type = 6) and apply a sliding window on these cars. We are going to set the window size on five minutes and the sliding size on every second. This means that we are computing the average speed over the last five minutes every second.>>>vehicles=vehicles[vehicles["type"]==6]>>>vehicles.sliding(size=5,size_unit="min",slide=1,slide_unit="sec").avg(on="value",name="avg_speed")startendavg_speed052451026101525If we anaylize this data stream for an entire day we can find out how the speed changes throughout the day.
We can also filter other vehicle types, e.g., underground, bus, trucks, and compare them to each other.Additionally we can apply for example DBSCAN before using the window on each cluster.
This way we find out which areas are prone to traffic jams at a specific time of day.
We can write the DBSCAN algorithm by integrating other libaries. For this, we have to write a function like the following :>>>defmy_dbscan_function(current_tuple):# here comes the code for dbscanNote:Currently this function is the last function to be processed in the python pipeline. For this reason, we have to add the window functionality by hand right here too. Furthermore, we have to keep in mind that this function can only process one tuple at a time.We use.process(my_dbscan_function)to apply dbscan and the window on our data stream:>>>vehicles.process(my_dbscan_function)clusterstartstopavg_speed105522052520517RequirementsOS: Ubuntu 20.04+Jupyter NotebookPython 3+InstallationInstall the python client with:pip install nespyRun NebulaStream. There are two options: a) you start NebulaStream locally in CLion or b) you use docker.a) Run NebulaStream with CLionBuild NebulaStream. To build NebulaStream please followthis instruction.Start the Coordinator. If no parameters are configured the Coordinator starts with the default values. To configure the Coordinator please take a lookhere.Start the Worker. Just like the Coordinator the Worker starts with default values unless the parameters are configured. To configure the Worker please take a lookhereand to configure the sources take a lookhere.b) Run the NebulaStream docker imageInstall docker (https://docs.docker.com/get-docker/)Pull the latest docker image with:docker pull nebulastream/nes-executable-image:latestor a specific version with:docker pull nebulastream/nes-executable-image:<version_number>Run the latest docker image with:docker run --network host --name nes nebulastream/nes-executable-image:latestordocker run --network host --name nes nebulastream/nes-executable-image:<version_number>The following additional parameters are optional (source):Coordinatorparametersdefault valuedescription-serverIp127.0.0.1-coordinatorPort4000Set the RPC Server port of the Coordinator-restPort8081Set the port exposed for REST endpoints–numberOfSlots#processor in the hostSet the number of computing slots in the Coordinator–helpDisplay help messageWorkerparametersdefault valuedescription–coordinatorIp127.0.0.1Set the server IP of the Coordinator to which the Worker should connect–coordinatorPort0Set the RPC server Port of the Coordinator to which the Worker connected–rpcPort0Set the RPC server of the Worker–dataPort0Set the data port of the Worker–sourceTypeSet the type of the Source (must be either CSVSource or DefaultSource)–sourceConfigSet the configuration of the source (e.g., path to the CSV file)–sourceFrequencySet the sampling frequency of the source–physicalStreamNameSet the physical name of the stream–numberOfBuffersToProduceSet the number of buffers to produce–logicalStreamNameSet the logical stream name where this stream is added to–parentIdSet the parentId of the Worker node–localWorkerIp127.0.0.1Set worker ip–numberOfSlots#processor in the hostSet the number of computing slots in the Worker–helpDisplay help messageOpen Jupyter Notebook, import the python client with "import nespy as nes", and get startedLicenseApache License 2.0Documentation1. Connect with NebulaStream2. Manage Data StreamsRetrieving all Registered Logical StreamsRetrieving all Physical Streams for a Logical StreamSelecting a Specific Logical StreamAdd Logical StreamUpdate Logical StreamDelete Logical Stream3. Process Data StreamsFilterMapSelect/ProjectionRenameWindowsKeyed Window vs. Global WindowTumbling WindowsSliding WindowsInclude other Python librariesStream ProcessingBatch processing with DataFramesUnionJoinSinks4. Plotting Data Streams with Nespy5. Manage QueriesStop a Running QueryRetrieving All Registered QueriesRetrieving All Queries with a StatusRetrieve a Specific Query with a StatusRetrieving Execution PlanRetrieving Query PlanRetrieve Topology1.Connect with NebulaStreamConnection is a class that manages the overall connection to NebulaStream.Connection(host,port)Example:>>>c=ds.Connection('127.0.0.1',8081)2. Managing Data StreamsAfter setting up the connection with NebulaStream the user can call different functions to gather information about the data stream itself and manage data streams.Retrieving all Registered Logical StreamsConnection.get_all_logical_streams()Example:>>>c.get.all_logical_streams()Retrieving all Physical Streams for a Logical StreamConnection.get_all_physical_streams(logical_stream_name)Example:>>>c.get_all_physical_streams("default_logical")Selecting a Specific Logical StreamConnection.get_logical_stream(position_or_name)Example:>>>data_stream=c.get_logical_stream(0)or>>>data_stream=c.get_logical_stream("default_logical")>>>data_streamidvalue011121Add Logical StreamConnection.add_logical_stream(logical_stream_name,schema)Example:>>>schema="Schema::create()->addField(\"test\",INT32);">>>c.add_logical_stream("new_logical_stream",schema){Success:True}Update Logical StreamConnection.update_logical_stream(logical_stream_name,schema)Example:>>>schema="Schema::create()->addField(\"test\",INT32);">>>c.update_logical_stream("default_logical",schema):{Success:True}Delete Logical StreamConnection.delete_logical_stream(logical_stream_name)Example:>>>c.delete_logical_stream("default_logical"){Success:True}3. Process Data StreamsFilterdata_stream[filter_predicate]Examples:>>>sensor[sensor["temperature"]>3]>>>sensor[sensor["temperature"]<3]>>>sensor[sensor["temperature"]<=3]>>>sensor[sensor["temperature"]>=3]>>>sensor[sensor["temperature"]==3]>>>sensor[sensor["temperature"]!=3]>>>sensor[sensor["temperature"]>sensor["humidity"]>>>sensor[(sensor["temperature"]>3)&(sensor["temperature"]<15)]>>>sensor[(sensor["temperature"]<3)|(sensor["temperature"]==10)]Mapdata_stream[attribute_name]=new_value_for_this_attributeExample:>>>sensor["temperature"]=sensor["temperature"]+0.01>>>sensor["temperature"]=sensor["temperature"]-0.01>>>sensor["temperature"]=sensor["temperature"]*0.01>>>sensor["temperature"]=sensor["temperature"]/0.01>>>sensor["temperature"]=(sensor["sensor_a"]+sensor["sensor_b"])/2>>>sensor["temperature"]=0.01*sensor["temperature"]Select (in SQL)/Projection (in relational algebra)For a single selection you only have to write the name of the desired attribute as a string into the brackets.
For a multiple selection you write a list of attribute names as a string that you want to project.data_stream[data_stream[attribute_name]]data_stream[[attribute_name1,attribute_name2,attribute_name3]]Example:>>>cars[cars["car_id"]]>>>cars[["car_id"]]>>>cars[["car_id","speed","lat","lon"]]RenameRename only works if the operator before rename was a projection.
It gives a temporary (not persisted!) name for the attribute.data_stream.as({"old_name":"new_name"})Example:>>>cars[["car_id"]]>>>cars.rename({car_id": "id"})WindowsWindows allow to break a infinite data stream into finite data sets. As a result, we are then able to compute aggregations over these finite data sets.
The following aggregation functions are supported:sum(on, name)count(on, name)min(on, name)max(on, name)Warning: Do not try to rename the columns yet. This functionality isn't supported by NebulaStream yet!
The avg function is supported by NebulaStream soon! Please be patient :)parameterdescriptiondefaultonrequireddeclares on which attribute to aggregate over-nameoptionalname of the new column (not supported yet!)aggregation_attribute, e.g,  sum_valueIn this section, we will explain how to use keyed and non-keyed windows in our python client.Keyed Windows vs. Global WindowsKeyed WindowsA key for a window can be any attribute of your data stream.
With keyed windows we can run multiple different windows and window aggregatins on one data stream.
Therefore, a keyed window splits up a single data stream into multiple data streams to enable parallel computation.Global WindowsGlobal windows are non-keyed windows. These window do not split a data stream into multiple data streams and therefore, they cannot compute multiple different aggregation function in parallel.Tumbling WindowsDataStream.tumbling(on,event,event_unit,size,size_unit,lateness,lateness_unit)parameterdescriptiondefault valueonrequiredkey for the window, can be any attribute of the data stream, if not set this window is a global windowNoneeventrequiredvalue decides on which attribute in the data stream is the event time'timestamp'event_unitoptionalunit for the event, can only be 'min', 'sec', 'ms', and 'count''sec'sizerequiredsize of the window0 (meaning no window)size_unitrequiredunit of the parameter size, can only be 'min', 'sec', 'ms', and 'count''sec'latenessoptionalthe allowed lateness of a tuple0lateness_unitoptionalunit of the lateness, can only be 'min', 'sec', 'ms', and 'count''sec'Example:>>># keyed window>>>shop.tumbling(on="value",size=10,size_unit="sec").sum(on="sales")>>># global window>>>shop.tumbling(size=10,size_unit="sec").sum(on="sales")Sliding WindowsDataStream.sliding(on,event,event_unit,size=0,size_unit,slide,slide_unit,lateness,lateness_unit)parameterdescriptiondefault valueonoptionalkey for the window, can be any attribute of the data stream, if not set this window is a global windowNoneeventrequiredvalue decides on which attribute in the data stream is the event time'timestamp'event_unitoptionalunit for the event, can only be 'min', 'sec', 'ms', and 'count''sec'sizerequiredsize of the window0 (meaning no window)size_unitrequiredunit of the parameter size, can only be 'min', 'sec', 'ms', and 'count''sec'sliderequiredthe slide value or the update rate of the window0 (meaning no slide)slide_unitrequiredthe unit of the slide value, can only be 'min', 'sec', 'ms', and 'count''sec'latenessoptionalthe allowed lateness of a tuple0lateness_unitoptionalunit of the lateness, can only be 'min', 'sec', 'ms', and 'count''sec'Example:>>># keyed window>>>shop.sliding(on="sales",size=10,size_unit="sec",slide=5,slide_unit="sec").sum(on="sales")>>># global window>>>shop.sliding(size=10,size_unit="sec",slide=5,slide_unit="sec").sum(on="sales")Include other Python librariesStream Processing>>>defmy_function(data):>>>#do something with a single tuple>>>returndata>>>data_stream.process(my_function)Batch Processing with DataFramesThis function enables user to compute in batches over DataFrames. If on_time_over is set to True the batch function is triggered every time a tuple is added to the batch. Otherwise, we only trigger the batch function when the timeframe has reached it's time.
Currently, we only support 'sec', 'min', and 'h' as the timeframe_unit.>>>defmy_function(data):>>>#do something with a DataFrame>>>returndata>>>>>># We set the size of the buffer to 1 hour and only call the batch function my_function once after every hour.>>>data_stream.batch(my_function,timeframe=1,timeframe_unit='h, on_time_over=True)>>>data_stream.batch(my_function,timeframe=1,timeframe_unit='h, on_time_over=False)UnionUnion is the same as in relational algrebra and SQL. The schema of both data streams have to be the same.DataStream.union(other_stream)Example:>>>bus=c.get_logical_stream("bus")>>>cars=c.get_logical_stream("car")>>>bus.union(car)JoinJoin works just like JOIN in SQL or the theta join in relational algebra.
If both of the data streams have an attribute with the same name, we can set the parameter "on".
Otherwise, we have to set the parameter "left_on" and "right_on".DataStream.join(other_stream,on,left_on,right_on)Example:>>>position=get_logical_stream("car_position")>>>details=get_logical_stream("car_details")>>>position.join(details,left_on="id",right_on="car_id")SinksNespy offers different types of sources:ZMQ SinkKafka SinkFile SinkPrint SinkThe default source is using ZMQ.To select other sink use:DataStream.set_sink(set_sink(sink,host,port,topic,brokers,timeout,path)parameterdescriptionsinkrequiredcan only be "zmq", "kafka", "file" or "print"hostrequired for ZMQ SinkHost for ZMQ Sinkportrequired for ZMQ SinkPort for ZMQ Sinktopicrequired for Kafka SinkTopic for Kafka Sinkbrokersrequired for Kafka SinkBrokers for Kafka Sinktimoutrequired for Kafka SinkTimout for Kafka Sinkpathrequired for File SinkPath to where the result should be saved4. Plotting Data Streams with NespyTBD5. Manage QueriesStop a Running QueryDataStream.stop_query()Example:>>>data_stream.stop_query()Reset All Operators of a QueryThis function resets all operators saved for this variable and also stops the running query. Therefore, the user can reuse this variable for another query.DataStream.reset_operators()Example:>>>data_stream_reset_operators()Retrieving All Registered QueriesConnection.get_all_queries()Example:>>>c.get_all_queries(){"edges":[{"source":"UNDEFINED(OP-2)","target":"SINK(OP-3)"},{"source":"SOURCE(OP-1)","target":"UNDEFINED(OP-2)"}],"nodes":[{"id":"3","name":"SINK(OP-3)","nodeType":"SINK"},{"id":"2","name":"UNDEFINED(OP-2)","nodeType":"UNDEFINED"},{"id":"1","name":"SOURCE(OP-1)","nodeType":"SOURCE"}]}Retrieving All Queries with a StatusConnection.get_queries_with_status(status)Example:>>>registered=c.get_queries_with_status("Registered")>>>scheduling=c.get_queries_with_status("Scheduling")>>>running=c.get_queries_with_status("Running")>>>marked_for_stopped=c.get_queries_with_status("MarkedForStop")>>>stopped=c.get_queries_with_status("Stopped")>>>failed=c.get_queries_with_status("Failed")>>>running[null,"Query::from(\"default_logical\").project(Attribute(\"id\")).sink(PrintSinkDescriptor::create());"Retrieve a Specific Query with a StatusConnection.get_queries_with_status_at_pos(status,pos)Example:>>>c.get_queries_with_status_at_pos("Running",1)Query::from("default_logical").project(Attribute("id")).sink(PrintSinkDescriptor::create());Retrieving Execution PlanConnection.get_query_execution_plan(queryId)Example:>>>data_stream=c.get_logical_stream("default_logical")>>>data_stream["id"]id012>>>data_stream.get_query_execution_plan(){"executionNodes":[{"ScheduledQueries":[{"queryId":1,"querySubPlans":[{"operator":"SINK(5)\nPROJECTION(2, schema=default_logical$id:INTEGER )\n""SOURCE(1,default_logical)\n","querySubPlanId":1}]}],"executionNodeId":2,"topologyNodeId":2,"topologyNodeIpAddress":"127.0.0.1"},{"ScheduledQueries":[{"queryId":1,"querySubPlans":[{"operator":"SINK(3)\nSOURCE(4,)\n","querySubPlanId":2}]}],"executionNodeId":1,"topologyNodeId":1,"topologyNodeIpAddress":"127.0.0.1"}]}Retrieving Query PlanConnection.get_query_plan(queryId)Example:>>>data_stream=c.get_logical_stream("default_logical")>>>data_stream["id"]id012>>>data_stream.get_query_plan(queryId){"edges":[{"source":"UNDEFINED(OP-2)","target":"SINK(OP-3)"},{"source":"SOURCE(OP-1)","target":"UNDEFINED(OP-2)"}],"nodes":[{"id":"3","name":"SINK(OP-3)","nodeType":"SINK"},{"id":"2","name":"UNDEFINED(OP-2)","nodeType":"UNDEFINED"},{"id":"1","name":"SOURCE(OP-1)","nodeType":"SOURCE"}]}Retrieve TopologyConnection.get_nes_topology()Example:>>>c.get_nes_topology(){"edges":[{"source":2,"target":1}],"nodes":[{"available_resources":65535,"id":1,"ip_address":"127.0.0.1"},{"available_resources":8,"id":2,"ip_address":"127.0.0.1"}]}