Package: titanfe
Titan Flow EngineThe Titan Flow Engine provides the backend services for running data flows
on a node.The core flow engine is composed of two services:Control Peer: One peer per node that takes care of managing Brick Runners
on that node.Brick Runner: Virtual Machine for executing the Bricks that the Control
Peer has started on it's node. The Brick Runner is generic and can process all
types of Bricks.
Each runner will only execute one type of Brick at a time.Additionally Version 0.1.3 and onward requiretitan services(sources)
to run a full deployment of the titan platform.Choose How to InstallTitan Flow Enginerequires Python 3.7to be installed.Depending on whether you want to contribute to the Titan Flow Engine source code
you want to choose one of the following options to install:Installing via PyPiThe flow engine is available on the Python Package Index (PyPi). It is available
for Linux and Windows x64 systems. To install it without having to build it
yourself open a command line and run:pipinstalltitanfeBuilding the Titan Flow Engine (Advanced Install)Required packages for building or developing the project can be installed
via therequirements_dev.txtin the project's root folder:pipinstall-rrequirements_dev.txtTo build and install the Flow Engine open a command line and run:pythonsetup.pybuild
pipinstall.For those who also want to work on the documentation and build it locally,
please use therequirements_doc.txtfile to install the necessary python
packages.pipinstall-rrequirements_doc.txtRunning the flow engineThe flow engine is being started by starting the Control Peer. The Control Peer
takes care of starting Brick Runners as needed.Parameters for starting the Control Peer on the command line are:NamePurposebrick_folderFolder to store installed bricks in (default: ~/titanfe/bricks)config_filePath to ControlPeer configuration fileThe ControlPeers configuration file contains the following parameters:NameTypeDescriptionKafkastringNetwork address of Kafka instance for metrics and logging<hostname:port>(default: localhost:9092)GridManagerstringNetwork address of the GridManager<hostname:port>(default: localhost:8080)FlowManagerstringNetwork address of the FlowManager<hostname:port>(default: localhost:9002)EndpointProviderstringNetwork address of the EndpointProvider<hostname:port>(default: "tcp://127.0.0.1:9021")SecretKeystringsecret key for brick parameter decryption(default: None), alternatively use TITAN_SECRET_KEY environment variableIf no secret key is given, the ControlPeer will stop immediately. The secret key
needs to be identical to the one used in the FlowManager for encrypting
parameters.To try out the flow engine first download and start theservicesGridManager, Repository service, User service, PackageManager, and FlowManager.
These services are needed for the flow engine to receive flows to process.Adjust the ControlPeers configuration file
(titanfe/apps/control_peer/config.yaml) or create a new one.
Then, open a command line and run.python-mtitanfe.apps.control_peer-brick_folder~/titanfe/bricks-config_filepath_to_config_file/config.yamlOr run it from the root directory of the flow engine by providing the path to the example:python-mtitanfe.apps.control_peer-brick_folder~/titanfe/bricks-config_filetitanfe/apps/control_peer/config.yamlCode QualityCode quality within the project is checked using pylint and flake8.pylintLinting is performed withpylint. To define the
intended checks.pylintrcis used to configure linting for this project.Running pylint for the python code in this project the following commands are
used:pylint--rcfile=.pylintrctitanfeLinting the tests is done running the command:pylint--rcfile=.pylintrc--disable=duplicate-code./testflake8To make sure the PEP8 standard is applied to the code flake8 can be added to
the static tests.For this project we exclude various errors, warnings and notifications because
they do not make sense at this time. This may change while refactoring is
considered.You can run flake 8 with:flake8It finds all the python files in this project.
The configuration for this project is read from.flake8in the project's
root directory.