Package: backend.ai-manager
Backend.AI Manager with API GatewayPackage Structureai.backend.manager: Computing resource and workload management with public APIsInstallationPlease visitthe installation guides.Kernel/system configurationRecommended resource limits:/etc/security/limits.confroot hard nofile 512000
root soft nofile 512000
root hard nproc 65536
root soft nproc 65536
user hard nofile 512000
user soft nofile 512000
user hard nproc 65536
user soft nproc 65536sysctlfs.file-max=2048000
net.core.somaxconn=1024
net.ipv4.tcp_max_syn_backlog=1024
net.ipv4.tcp_slow_start_after_idle=0
net.ipv4.tcp_fin_timeout=10
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_early_retrans=1
net.ipv4.ip_local_port_range="10000 65000"
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 12582912 16777216
net.ipv4.tcp_wmem=4096 12582912 16777216For developmentPrerequisitesPython 3.6 or higher withpyenvandpyenv-virtualenv(optional but recommneded)Docker 18.03 or later with docker-compose (18.09 or later is recommended)Common stepsClonethe meta repositoryand install a "halfstack"
configuration.  The halfstack configuration installs and runs several dependency daemons such as etcd in
the background.$gitclonehttps://github.com/lablup/backend.aihalfstack$cdhalfstack$docker-compose-fdocker-compose.halfstack.ymlup-dThen prepare the source clone of the agent as follows.
First install the current working copy.$gitclonehttps://github.com/lablup/backend.ai-managermanager$cdmanager$pyenvvirtualenvvenv-manager$pyenvlocalvenv-manager$pipinstall-Upipsetuptools$pipinstall-U-rrequirements/dev.txtFrom now on, let's assume all shell commands are executed inside the virtualenv.Halfstack (single-node development & testing)Recommended directory structurebackend.ai-devmanager(git clone from this repo)agent(git clone fromthe agent repo)common(git clone fromthe common repo)Installbackend.ai-commonas an editable package in the manager (and the agent) virtualenvs
to keep the codebase up-to-date.$cdmanager$pipinstall-U-e../common-rrequirements/dev.txtStepsCopy (or symlink) the halfstack configs:$cpconfig/halfstack.toml./manager.toml$cpconfig/halfstack.alembic.ini./alembic.iniSet up Redis:$backend.aimgretcdputconfig/redis/addr127.0.0.1:8110ℹ️ NOTE: You may replacebackend.ai mgrwithpython -m ai.backend.manager.cliin case yourPATHis unmodifiable.Set up the public Docker registry:$backend.aimgretcdputconfig/docker/registry/index.docker.io"https://registry-1.docker.io"$backend.aimgretcdputconfig/docker/registry/index.docker.io/username"lablup"$backend.aimgrimagerescanindex.docker.ioSet up the vfolder paths:$mkdir-p"$HOME/vfroot/local"$backend.aimgretcdputvolumes/_mount"$HOME/vfroot"$backend.aimgretcdputvolumes/_default_hostlocalSet up the allowed types of vfolder. Allowed values are "user" or "group".
If none is specified, "user" type is set implicitly:$backend.aimgretcdputvolumes/_types/user""# enable user vfolder$backend.aimgretcdputvolumes/_types/group""# enable group vfolderSet up the database:$backend.aimgrschemaoneshot$backend.aimgrfixturepopulatesample-configs/example-keypairs.json$backend.aimgrfixturepopulatesample-configs/example-resource-presets.jsonThen, run it (for debugging, append a--debugflag):$backend.aimgrstart-serverTo run tests:$python-mflake8srctests$python-mpytest-m'not integration'testsNow you are ready to install the agent.
Head tothe README of Backend.AI Agent.NOTE: To run tests including integration tests, you first need to install and run the agent on the same host.DeploymentConfigurationPut a TOML-formatted manager configuration (see the sample inconfig/sample.toml)
in one of the following locations:manager.toml(current working directory)~/.config/backend.ai/manager.toml(user-config directory)/etc/backend.ai/manager.toml(system-config directory)Only the first found one is used by the daemon.Also many configurations shared by both manager and agent are stored in etcd.
As you might have noticed above, the manager provides a CLI interface to access and manipulate the etcd
data.  Check out the help page of our etcd command set:$python-mai.backend.manager.clietcd--helpIf you run etcd as a Docker container (e.g., via halfstack), you may use the native client as well.
In this case, PLEASE BE WARNED that you must prefix the keys with "/sorna/{namespace}" manaully:$dockerexec-it${ETCD_CONTAINER_ID}/bin/ash-c'ETCDCTL_API=3 etcdctl ...'Running from a command lineThe minimal command to execute:python-mai.backend.gateway.serverFor more arguments and options, run the command with--helpoption.Writing a wrapper scriptTo use with systemd, crontab, and other system-level daemons, you may need to write a shell script
that executes specific CLI commands provided by Backend.AI modules.The following example shows how to set up pyenv and virtualenv for the script-local environment.
It runs the gateway server if no arguments are given, and execute the given arguments as a shell command
if any.
For instance, you may get/set configurations like:run-manager.sh python -m ai.backend.manager.etcd ...where the name of scripts isrun-manager.sh.#! /bin/bashif[-z"$HOME"];thenexportHOME="/home/devops"fiif[-z"$PYENV_ROOT"];thenexportPYENV_ROOT="$HOME/.pyenv"exportPATH="$PYENV_ROOT/bin:$PATH"fieval"$(pyenvinit-)"eval"$(pyenvvirtualenv-init-)"pyenvactivatevenv-bai-managerif["$#"-eq0];thenexecpython-mai.backend.gateway.serverelseexec"$@"fiNetworkingThe manager and agent should run in the same local network or different
networks reachable via VPNs, whereas the manager's API service must be exposed to
the public network or another private network that users have access to.The manager requires access to the etcd, the PostgreSQL database, and the Redis server.User-to-Manager TCP PortsUsagemanager:{80,443}Backend.AI API accessManager-to-X TCP PortsUsageetcd:2379etcd API accesspostgres:5432Database accessredis:6379Redis API accessThe manager must also be able to access TCP ports 6001, 6009, and 30000 to 31000 of the agents in default
configurations.  You can of course change those port numbers and ranges in the configuration.Manager-to-Agent TCP PortsUsage6001ZeroMQ-based RPC calls from managers to agents6009HTTP watcher API30000-31000Port pool for in-container servicesLICENSESGNU Lesser General Public LicenseDependencies