Package: dp-cubed
Dynamic Profile Processing Platform (DP³)DP³ is a platform helps to keep a database of information (attributes) about individual
entities (designed for IP addresses and other network identifiers, but may be anything),
when the data constantly changes in time.You can read more about how it works in thedocumentation.This is a basis of CESNET's "Asset Discovery Classification and Tagging" (ADiCT) project,
focused on discovery and classification of network devices,
but the platform itself is general and should be usable for any kind of data.DP³ doesn't do much by itself, it must be supplemented by application-specific modules providing
and processing data.Repository structuredp3- Python package containing code of the processing core and the APIconfig- default/example configurationinstall- deployment configurationSee thedocumentationfor more details.InstallationSee thedocsfor more details.Installing for application developmentPre-requisites: Python 3.9 or higher,pip(withvirtualenvinstalled),git,DockerandDocker Compose.Create a virtualenv and install the DP³ platform using:python3-mvenvvenvsourcevenv/bin/activate
python-mpipinstall--upgradepip
pipinstallgit+https://github.com/CESNET/dp3.git#egg=dp-cubedCreating a DP³ applicationDP³ comes with adp3utility, which is used to create a new DP³ application and run it.
To create a new DP³ application we use thesetupcommand. Run:dp3setup<application_directory><your_application_name>So for example, to create an application calledmy_appin the current directory, run:dp3setup.my_appWhich produces a template DP3 application directory structure.Running the ApplicationTo run the application, we first need to setup the other services the platform depends on,
such as the MongoDB database, the RabbitMQ message distribution and the Redis database.
This can be done using the supplieddocker-compose.ymlfile. Simply run:dockercomposeup-d--buildThere are two main ways to run the application itself. First is a little more hand-on,
and allows easier debugging.
There are two main kinds of processes in the application: the API and the worker processes.To run the API, simply run:APP_NAME=my_appCONF_DIR=configdp3apiThe starting configuration sets only a single worker process, which you can run using:dp3workermy_appconfig0The second way is to use thedocker-compose.app.ymlfile, which runs the API and the worker processes
in separate containers. To run the API, simply run:dockercompose-fdocker-compose.app.ymlup-d--buildEither way, to test that everything is running properly, you can run:curl-X'GET''http://localhost:5000/'\-H'Accept: application/json'Which should return a JSON response with the following content:{"detail":"It works!"}Final note, to simplify the experience of adjusting the app configuration,
especially that of the DB entities, we provide thedp3 checkcommand.
The command simply loads the configuration and checks that it is valid, but if not,
it tries really hard to pinpoint where exactly you went wrong. This can be used as follows:dp3check<config_directory>You are now ready to start developing your application!Installing for platform developmentPre-requisites: Python 3.9 or higher,pip(withvirtualenvinstalled),git,DockerandDocker Compose.Pull the repository and install using:gitclone--branchmastergit@github.com:CESNET/dp3.gitdp3cddp3
python3-mvenvvenvsourcevenv/bin/activatepython-mpipinstall--upgradepippipinstall--editable".[dev]"pre-commitinstallRunning the dependencies and the platformThe DP³ platform is now installed and ready for development.
To run it, we first need to setup the other services the platform depends on,
such as the MongoDB database, the RabbitMQ message distribution and the Redis database.
This can be done using the supplieddocker-compose.ymlfile. Simply run:dockercomposeup-d--buildAfter the firstcompose upcommand, the images for RabbitMQ, MongoDB and Redis will be downloaded,
their images will be built according to the configuration and all three services will be started.
On subsequent runs, Docker will use the cache, so if the configuration does not change, the download
and build steps will not be repeated.The configuration is taken implicitly from thedocker-compose.ymlfile in the current directory.
Thedocker-compose.ymlconfiguration contains the configuration for the services,
as well as a testing setup of the DP³ platform itself.
The full configuration is intests/test_config.
The setup includes one worker process and one API process to handle requests.
The API process is exposed on port 5000, so you can send requests to it usingcurlor from your browser:curl-X'GET''http://localhost:5000/'\-H'Accept: application/json'curl-X'POST''http://localhost:5000/datapoints'\-H'Content-Type: application/json'\--data'[{"type": "test_entity_type", "id": "abc", "attr": "test_attr_int", "v": 123, "t1": "2023-07-01T12:00:00", "t2": "2023-07-01T13:00:00"}]'TestingWith the testing platform setup running, we can now run tests.
Tests are run using theunittestframework and can be run using:python-munittestdiscover\-stests/test_common\-vCONF_DIR=tests/test_config\python-munittestdiscover\-stests/test_api\-v