Package: fdi
Flexible Dataset Integrator (FDI) helps data producers and processors to build connections into isolated heterogeneous datasets. to assemble, organize, and integrate data into self-describing, modular, hierarchical, persistent, referenceableProducts, whose component datasets keep their own characteristicss and are easily accessible.FDI provides scripting-friendly  APIs  and
tools to define custom Products and generating Python class files. An integrated Product takes care of inter-platform compatibility, string representation, serialisation to simple exchange format, persistence to disk or server, and carrying references of other Products, enabling context-building and lazy-loading.FDI’s base data model is defined in sub-packagedataset. Persistent data
access using storage pools, referencing, and Universal Resource Names, and referencie-carrying product Context, are defined in sub-packagepal. A reference RESTful API server designed to communicate with a data processing server/docker using the data model, and a reference HTTP pool server are in sub-packagepns.Install/Uninstall<a href=”http://mercury.bao.ac.cn:9006/mh/fdi/-/commits/master”><img alt=”pipeline status” src=”http://mercury.bao.ac.cn:9006/mh/fdi/badges/master/pipeline.svg” /></a>TipIt is a good idea to add--userat the end or to use a virtualenv to avoid disturbing your system Python setup.For UsersIf you need to use fdi, install from PyPI:python3-mpipinstallfdior from git repo:python3-mpipinstallhttp://mercury.bao.ac.cn:9006/mh/fdi/-/archive/master/fdi-master.tar.gzIf you want to install thedevelopdependencies:python3-mpipinstallfdi[DEV]orpython3-mpipinstallhttp://mercury.bao.ac.cn:9006/mh/fdi/-/archive/develop/fdi-develop.tar.gz#egg=fdi[DEV]If you do not need to run tests, remove[DEV]to save time and disk space.To uninstall:python3-mpipuninstallfdiFor Developers  (or Those who are Not Sure which to Choose)To installFDIINSTDIR=/tmp# change this to your installation dircd$FDIINSTDIRgitclonehttp://mercury.bao.ac.cn:9006/mh/fdi.gitcdfdigitcheckoutdevelopmakeinstallEXT="[DEV]"If you want to install themasterbranch, remove thegit checkout developline above.To test your installationmaketestTipTo pass command-line arguments topytestdo, for example,maketestT='-k Bas'to testBaseProduct.To Generate Python Product Source CodeRe-generatebaseproduct.pyandproduct.pyfrom YAML schema files infdi/dataset/resources:makepyLearn/Try the HTTP Pool Server APIsThe APIs are documented infdi/httppool/schema/pools.ymlwith OpenAPI 3. Run this to see and try out with Swagger API Docs when the server is running:http://127.0.0.1:5000/apidocsModify/Generate DocumentsIf you plan to compile documents in thedocsdirectory, generate diagrams, API files, or HTML pages, run (in that order, respectively):First run this once to install necessary packages:python3-mpipinstall-U-e.[PUB]Then when you need to make new class diagrams, API docs, or HTML pages:makedocs_plotsmakedocs_apimakedocs_htmlThe generated HTML page is atdocs/html/index.html.Notehttps://readthedocs.iomakes web pages from sources indocs/sphinxin the repository. Locally generated HTML pages are not on RTD or in the repository. The API files and plots, however, are inapiand_staticsub-directpries, respectively.Run Servers and Quick TestsIf you plan to run thepnsand/or thehttp pool serverlocally,
first install the dependencies:python3-mpipinstall-e.[SERV]makeinstallpnsTo test yourpnsservers installation, in one window, run:makerunserverin another window run:maketestpnsTo test yourhttppoolservers installation, make sure to stop other server such aspnsserverabove, then in one window, run:makerunpoolserverin another window run:maketesthttpDocker ContainersfdiGet thefdidocker with running FDI environment:dockerpullmhastro/fdior build the image locally:makebuild_dockerLaunch and loginmakelaunch_dockermakeithttppoolAlso available is aHttpPoolserver made from Ubuntu and apache:dockerpullmhastro/httppoolTo build the image locally:makebuild_serverLaunch and connect:makelaunch_servercurl-ihttp://127.0.0.1:9884/v0.8/Run the above in the package root directory of fdi. A file named.secretis needed by the build and launch commands. This is an example:PNS_PORT=9884
PNS_USERNAME=...
PNS__PASSWORD=...
PNS_MQ_HOST=123.45.67.89
PNS_MQ_PORT=9876
PNS_MQ_USER=...
PNS_MQ_PASS=...For MoreFor more  examples seetests/test_*.py.Read more on package introduction, description, quick start, and API
documents onreadthedocs.io.