Package: hyperstream
HyperStream logoHyperStreamHyperstream is a large-scale, flexible and robust software package for
processing streaming data.HyperStreamhomepageTutorialnotebooksGitterchat roomDeveloperdocumentationHyperstream overcomes the limitations of other computational engines and
provides high-level interfaces to execute complex nesting, fusion, and
prediction both in online and offline forms in streaming environments.
Although developed specifically for SPHERE, Hyperstream is a general
purpose tool that is well-suited for the design, development, and
deployment of algorithms and predictive models in a wide space of
sequential predictive problems.This software has been designed from the outset to be
domain-independent, in order to provide maximum value to the wider
community. Key aspects of the software include the capability to create
complex interlinked workflows, and a computational engine that is
designed to be “compute-on-request”, meaning that no unnecessary
resources are used.InstallationDocker imagesIf you do not want to install all the packages separately you can use
our Docker bundle availablehere.Local machineInstall via pippip install hyperstream
python -c 'from hyperstream import HyperStream'To get the latest versionpip install -U git+git://github.com/IRC-SPHERE/HyperStream.git#egg=hyperstream
pip install -r requirements.txtOr clone the repositorygitclonegit@github.com:IRC-SPHERE/HyperStream.gitcdHyperStreamvirtualenvvenv.venv/bin/activatepipinstall-rrequirements.txtpython-c'from hyperstream import HyperStream'Additionally, one of the requirements to run Hyperstream is a MongoDB
server. By default, Hyperstream tries to connect to the port 27017 on
the localhost.To access via a cloud provider, you might try mLab, MongoDB Atlas, or
search for a different one. If you are using mLab, here are a few notes:Set up an account via the mLab web site instructions. When asked to
pick a server type (e.g. Amazon, Google, etc) you can just choose
free option of 500MB. This is more than enough to get started.mLab will ask you to create a database; any name is fine, but make
sure you write down what it is.After creating a database, note that you’ll need to create at least
one database user in order to access the database.You can test your database connection using MongoDB’s built-in
command line toolsTo install MongoDB locally go to theofficial
documentation. E.g.
in a Debian OS it is possible to install with the following commandsudoapt-getinstallmongodbOnce the MongoDB server is installed, it can be started with the
following commandservicemongodstartIf installing MongoDB on OSXbrewinstallmongodbthenln-sfv/usr/local/opt/mongodb/*.plist~/Library/LaunchAgentsrun MongoDB service withlaunchctlload~/Library/LaunchAgents/homebrew.mxcl.mongodb.plistRunning testsRun the following commandnosetestsNote that for the MQTT logging test to succeed, you will need to have an
MQTT broker running (e.g. Mosquitto). For example:docker run -ti -p 1883:1883 -p 9001:9001 toke/mosquittoor on OSX you will need pidof and mosquitto:brew install pidof
brew install mosquitto
brew services start mosquittoTutorialsThe following tutorials show how to use HyperStream in a step-by-step
guide.Tutorial 1:
IntroductionTutorial 2: Creating
toolsTutorial 3: Stream
compositionTutorial 4: Real-time
streamsTutorial 5:
WorkflowsRunning the tutorials in a docker containerIt is possible to run all the tutorials in your own machine ussing
Docker containers defined inIRC-SPHERE/Hyperstream-Dockerfiles.
You can do that by running the following commands:gitclonehttps://github.com/IRC-SPHERE/Hyperstream-Dockerfiles.gitcdHyperstream-Dockerfilesdocker-compose-fdocker-compose-tutorials.yml-phyperstream-tutorialsupAnd then open the urlhttp://0.0.0.0:8888/treein a web-browserRunning the tutorials in the cloned folderTo run the tutorials in the cloned repository you will need to install
additional dependencies. First you should activate the virtual
environment and installed the general requirements to run HyperStream
following the instructions above. After that, install the dependencies
for the tutorial withpipinstall-rrequirements_tutorial.txtand go to the experiments foldercdexperimentsAnd run a Jupyter notebookjupyternotebookNow you can follow the instructions from the first tutorial.Simple use-casefromhyperstreamimportHyperStream,StreamId,TimeIntervalfromhyperstream.utilsimportutcnow,UTCfromdatetimeimporttimedeltahs=HyperStream(loglevel=20)M=hs.channel_manager.memoryT=hs.channel_manager.toolsclock=StreamId(name="clock")clock_tool=T[clock].window().last().value()ticker=M.get_or_create_stream(stream_id=StreamId(name="ticker"))now=utcnow()before=(now-timedelta(seconds=30)).replace(tzinfo=UTC)ti=TimeInterval(before,now)clock_tool.execute(sources=[],sink=ticker,interval=ti,alignment_stream=None)list(ticker.window().tail(5))The last list contains[StreamInstance(timestamp=datetime.datetime(2017,7,27,10,33,45,tzinfo=<UTC>),value=datetime.datetime(2017,7,27,10,33,45,tzinfo=<UTC>)),StreamInstance(timestamp=datetime.datetime(2017,7,27,10,33,46,tzinfo=<UTC>),value=datetime.datetime(2017,7,27,10,33,46,tzinfo=<UTC>)),StreamInstance(timestamp=datetime.datetime(2017,7,27,10,33,47,tzinfo=<UTC>),value=datetime.datetime(2017,7,27,10,33,47,tzinfo=<UTC>)),StreamInstance(timestamp=datetime.datetime(2017,7,27,10,33,48,tzinfo=<UTC>),value=datetime.datetime(2017,7,27,10,33,48,tzinfo=<UTC>)),StreamInstance(timestamp=datetime.datetime(2017,7,27,10,33,49,tzinfo=<UTC>),value=datetime.datetime(2017,7,27,10,33,49,tzinfo=<UTC>))]HyperStream ViewerTheHyperStream
Vieweris a
python/Flask web-app for interacting with HyperStream. In order to keep
HyperStream to a minimum, this web-app is released as a separate
repository that takes the core as a dependency.LicenseThis code is released under theMIT
license.AcknowledgementsThis work has been funded by the UK Engineering and Physical Sciences
Research Council (EPSRC) under GrantEP/K031910/1- “SPHERE Interdisciplinary Research Collaboration”.