Package: mlcomp
Distributed directed acyclic graph framework for machine learning with UIThe goal of MLComp is to provide tools for training, inferencing, creating complex pipelines
(especially for computer vision) in a rapid, well manageable way.MLComp is compatible with: Python 3.6+, Unix operation system.Part ofCatalyst Ecosystem. Projectmanifest.FeaturesAmazing UICatalystsupportDistributed trainingSupervisor that controls computational resourcesSynchronization of both code and dataResource monitoringFull functionality of the pause and continue on UIAuto control of the requirementsCode dumping (with syntax highlight on UI)KaggleintegrationHierarchical loggingGrid searchExperiments comparisonCustomizing layout systemContentsScreenshotsInstallationUIUsageDocs and examplesEnvironment variablesScreenshotsDagsComputersReportsCodeGraphMore screenshotsInstallationInstall MLComp packagesudoapt-getinstall-y\libavformat-devlibavcodec-devlibavdevice-dev\libavutil-devlibswscale-devlibavresample-devlibavfilter-dev

pipinstallmlcomp
mlcompinit
mlcompmigrateSetup your environment. Please considerEnvironment variablessectionRun db, redis, mlcomp-server, mlcomp-workers:Variant 1: minimal (if you have 1 computer)Run all necessary (mlcomp-server, mlcomp-workers, redis-server), it uses SQLITE:mlcomp-serverstart--daemon=TrueVariant 2: fulla. Change yourEnvironment variablesto use PostgreSqlb. Install rsync on each work computersudo apt-get install rsyncEnsure that every computer is available by SSH protocol with IP/PORT you specified
in theEnvironment variablesfile.rsync will perform the following commands:to uploadrsync-vhru-e"ssh -p {target.port} -o StrictHostKeyChecking=no"\{folder}/{target.user}@{target.ip}:{folder}/--perms--chmod=777to downloadrsync -vhru -e "ssh -p {source.port} -o StrictHostKeyChecking=no" \
{source.user}@{source.ip}:{folder}/ {folder}/ --perms  --chmod=777c. Installapexfor distributed learningd. To Run postgresql, redis-server, mlcomp-server, execute on your server-computer:cd~/mlcomp/configs/
docker-compose-fserver-compose.ymlup-de. Run on each worker-computer:mlcomp-workerstartUIWeb site is available at http://{WEB_HOST}:{WEB_PORT}By default, it ishttp://localhost:4201The front is built with AngularJS.In case you desire to change it, please considerfront's Readme pageUsageRunmlcompdagPATH_TO_CONFIG.ymlThis command copies files of the directory to the database.Then, the server schedules the DAG considering free resources.For more information, please considerDocsDocs and examplesAPI documentation and an overview of the library can be
found hereYou can find advanced tutorials and MLComp best practices in theexamplesfolder of the repository.FileSync tutorialdescribes data synchronization mechanismEnvironment variablesThe single file to setup your computer environment is located at ~/mlcomp/configs/.envROOT_FOLDER - folder to save MLComp files: configs, db, tasks, etc.TOKEN - site security token. Please change it to any stringDB_TYPE. Either SQLITE or POSTGRESQLPOSTGRES_DB. PostgreSql db namePOSTGRES_USER. PostgreSql userPOSTGRES_PASSWORD. PostgreSql passwordPOSTGRES_HOST. PostgreSql hostPGDATA. PostgreSql db files locationREDIS_HOST. Redis hostREDIS_PORT. Redis portREDIS_PASSWORD. Redis passwordWEB_HOST. MLComp site host. 0.0.0.0 means it is available from everywhereWEB_PORT. MLComp site portCONSOLE_LOG_LEVEL. log level for output to the consoleDB_LOG_LEVEL. log level for output to the databaseIP. Ip of a work computer. The work computer must be accessible from other work computers by these IP/PORTPORT. Port of a work computer. The work computer must be accessible from other work computers by these IP/PORT (SSH protocol)MASTER_PORT_RANGE. distributed port range for a work computer. 29500-29510 means that if
this work computer is a master in a distributed learning, it will use the first free port
from this range. Ranges of different work computers must not overlap.NCCL_SOCKET_IFNAME. NCCL network interface.FILE_SYNC_INTERVAL. File sync interval in seconds. 0 means file sync is offWORKER_USAGE_INTERVAL. Interval in seconds of writing worker usage to DBINSTALL_DEPENDENCIES. True/False. Either install dependent libraries or notSYNC_WITH_THIS_COMPUTER. True/False. If False, all computers except that will not sync with that oneCAN_PROCESS_TASKS. True/False. If false, this computer does not process tasksYou can see your network interfaces withifconfigcommand.
Please considernvidia doc