Package: ml4proflow
ml4proflow - A data flow oriented framework for ml applications in industryml4proflowis the acronym forDataflowofMachine Learningfor Production and Production Systems.It is a framework that manages the dataflow along the typical ml pipeline and is mainly used in industrial applications.
The ml pipeline is decribed by a graph, theDataFlowGraph. Typically, graphs describing ml pipelines are simple directed. However, since other types of graphs are possible, this frameworks can produce a dataflow for all types of graphs. Following the typical description of a graph, aDFGconsits of multiple nodes, which are connected along their edges. Nodes are represented bymodulesand edges bychannels.Modulescontain the ml algorithms and are implemented by experts.Channelsare created by theDFGand controlled by a configuration given by an non-expert.This enables all employees, from the machine operator to data scientists, to execute ml algorithms. Due to the framework's independence of execution platforms and execution architecture, it can be deployed anywhere in the production process, from edge devices to internal or public clouds.FeaturesOpen SourcePython basedModular & scalablePlatform independentInstallationBinary Installer from PyPiThe binaries are available from thePython Package Index.
Install this package withpipinstallml4proflowEnd User InstallationAsml4proflowis intended for developers and end users, we provide an installation script that sets up all the necessary dependencies for your operating system. It installs a Python instance and all available modules for the framework in a virtual environment. This installation method is intended for end users who are not familiar with Python.
This entry point to the framework is located inside the repositoryml4proflow-standalone.
Follow the steps given by the README.Installation from sourceThe source code is currently hosted onGitlab.Linuxgitclonehttps://gitlab.ub.uni-bielefeld.de/ml4proflow/ml4proflowcdml4proflow
pipinstall.Windowsgitclonehttps://gitlab.ub.uni-bielefeld.de/ml4proflow/ml4proflowcdml4proflow
pipinstall.Development installationFor further development, install the package in editable mode:pipinstall-e.UsageA DataFlowGraph is controlled by a Configuration-File. Theconfig.jsonis structured through a list of all appearing modules in the data pipeline. Every module is described by the path, the name and the configuration of the module.In most cases the order of execution is determined by the data flow defined through the DFG-configuration.
But since modules can decide for themselves whether they want to be executed (e.g. executables),it is necessary to arrange the modules in an intuitive way according to the order in the DFG.Example DFG-Config{"modules":[{"module_ident":"ml4proflow.mods.xxx.modules","module_name":"ModuleName","module_config":{"channels_pull":["src"]"channels_push":["src"],"moduleParam1":"xxx","moduleParam2":1.0}}]}CLI - Interface$ml4proflow-cli--[Options]For more documentation, seehere.Using ml4proflow for data analyticsBasic PrinciplesDataFlow in ml pipelnie represented as graph --> DataFlowGraphA node (Modules) in the graph is created by the DataFlowGraphNodes can have none to multiple inputs and outputsBasicModule : The basic class of the framework0-n inputs, 0-m outputsSources: Inherits from the BasicModule0 inputs, m outputsSinks: Inherits from the BasicModulen inputs, 0 outputsExecutable: Inherits from the BasicModule0 inputs, 0 outputsModules: Inherits from Sinks & Sourcesn inputs, m outputsDFG: Inherits from Executables0 inputs, 0 outputsAn edge between two nodes is created by the DataFlowManagercreate_channel(left side of edge = SourceModule)register_sink(right side of edge = SinkModule)Important: Everything is a node: Even a a complete graph can be a node of another graphversion: 1.1