Package: backend.ai-agent
Backend.AI AgentThe Backend.AI Agent is a small daemon that does:Reports the status and available resource slots of a worker to the managerRoutes code execution requests to the designated kernel containerManages the lifecycle of kernel containers (create/monitor/destroy them)Package Structureai.backendagent: The agent packagedocker: A docker-based backend implementation for the kernel lifecycle interface.server: The agent daemon which communicates with the manager and the Docker daemonwatcher: A side-by-side daemon which provides a separate HTTP endpoint for accessing the status
information of the agent daemon and manipulation of the agent's systemd servicehelpers: A utility package that is available asai.backend.helpersinsidePython-based containerskernel: Language-specific runtimes (mostly ipykernel client adaptor) which runinsidecontainersrunner: Auxiliary components (usually self-contained binaries) mountedinsidecontainersInstallationPlease visitthe installation guides.Kernel/system configurationRecommended kernel parameters in the bootloader (e.g., Grub):cgroup_enable=memory swapaccount=1Recommended resource limits:/etc/security/limits.confroot hard nofile 512000
root soft nofile 512000
root hard nproc 65536
root soft nproc 65536
user hard nofile 512000
user soft nofile 512000
user hard nproc 65536
user soft nproc 65536sysctlfs.file-max=2048000
fs.inotify.max_user_watches=524288
net.core.somaxconn=1024
net.ipv4.tcp_max_syn_backlog=1024
net.ipv4.tcp_slow_start_after_idle=0
net.ipv4.tcp_fin_timeout=10
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_early_retrans=1
net.ipv4.ip_local_port_range=40000 65000
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 12582912 16777216
net.ipv4.tcp_wmem=4096 12582912 16777216
net.netfilter.nf_conntrack_max=10485760
net.netfilter.nf_conntrack_tcp_timeout_established=432000
net.netfilter.nf_conntrack_tcp_timeout_close_wait=10
net.netfilter.nf_conntrack_tcp_timeout_fin_wait=10
net.netfilter.nf_conntrack_tcp_timeout_time_wait=10Theip_local_port_rangeshould not overlap with the container port range pool
(default: 30000 to 31000).To apply netfilter settings during the boot time, you may need to addnf_conntrackto/etc/modulesso thatsysctlcould set thenet.netfilter.nf_conntrack_*values.For developmentPrerequisitesPython 3.6 or higher withpyenvandpyenv-virtualenv(optional but recommneded)Docker 18.03 or later with docker-compose (18.09 or later is recommended)First, you needa working manager installation.
For the detailed instructions on installing the manager, please referthe manager's READMEand come back here again.Preparing working copyInstall and activategit-lfsto work with pre-built binaries insrc/ai/backend/runner.$gitlfsinstallNext, prepare the source clone of the agent and install from it as follows.pyenvis just a recommendation; you may use other virtualenv management tools.$gitclonehttps://github.com/lablup/backend.ai-agentagent$cdagent$pyenvvirtualenvvenv-agent$pyenvlocalvenv-agent$pipinstall-Upipsetuptools$pipinstall-U-rrequirements/dev.txtLintingWe useflake8andmypyto statically check our code styles and type consistency.
Enable those linters in your favorite IDE or editor.Halfstack (single-node development & testing)With the halfstack, you can run the agent simply.
Note that you need a working manager running with the halfstack already!Recommended directory structurebackend.ai-devmanager(git clone fromthe manager repo)agent(git clone from here)common(git clone fromthe common repo)Installbackend.ai-commonas an editable package in the agent (and the manager) virtualenvs
to keep the codebase up-to-date.$cdagent$pipinstall-U-e../commonSteps$mkdir-p"./scratches"$cpconfig/halfstack.toml./agent.tomlIf you're running agent under linux, make sure you've set appropriate iptables rule
before starting agent. This can be done by executing scriptscripts/update-metadata-iptables.shbefore each agent start.Then, run it (for debugging, append a--debugflag):$python-mai.backend.agent.serverTo run the agent-watcher:$python-mai.backend.agent.watcherThe watcher shares the same configuration TOML file with the agent.
Note that the watcher is only meaningful if the agent is installed as a systemd service
namedbackendai-agent.service.To run tests:$python-mflake8srctests$python-mpytest-m'not integration'testsDeploymentConfigurationPut a TOML-formatted agent configuration (see the sample inconfig/sample.toml)
in one of the following locations:agent.toml(current working directory)~/.config/backend.ai/agent.toml(user-config directory)/etc/backend.ai/agent.toml(system-config directory)Only the first found one is used by the daemon.The agent reads most other configurations from the etcd v3 server where the cluster
administrator or the Backend.AI manager stores all the necessary settings.The etcd address and namespace must match with the manager to make the agent
paired and activated.
By specifying distinguished namespaces, you may share a single etcd cluster with multiple
separate Backend.AI clusters.By default the agent uses/var/cache/scratchesdirectory for making temporary
home directories used by kernel containers (the/home/workvolume mounted in
containers).  Note that the directory must exist in prior and the agent-running
user must have ownership of it.  You can change the location byscratch-rootoption inagent.toml.Running from a command lineThe minimal command to execute:python-mai.backend.agent.server
python-mai.backend.agent.watcherFor more arguments and options, run the command with--helpoption.Example config for systemd/etc/systemd/system/backendai-agent.service:[Unit]Description=Backend.AI AgentRequires=docker.serviceAfter=network.target remote-fs.target docker.service[Service]Type=simpleUser=rootGroup=rootEnvironment=HOME=/home/userExecStart=/home/user/backend.ai/agent/run-agent.shWorkingDirectory=/home/user/backend.ai/agentKillMode=processKillSignal=SIGTERMPrivateTmp=falseRestart=on-failureRestartSec=5[Install]WantedBy=multi-user.target/home/user/backend.ai/agent/run-agent.sh:#! /bin/shif[-z"$PYENV_ROOT"];thenexportPYENV_ROOT="$HOME/.pyenv"exportPATH="$PYENV_ROOT/bin:$PATH"fieval"$(pyenvinit-)"eval"$(pyenvvirtualenv-init-)"cd/home/user/backend.ai/agentif["$#"-eq0];thensh/home/user/backend.ai/agent/scripts/update-metadata-iptables.shexecpython-mai.backend.agent.serverelseexec"$@"fiNetworkingThe manager and agent should run in the same local network or different
networks reachable via VPNs, whereas the manager's API service must be exposed to
the public network or another private network that users have access to.The manager must be able to access TCP ports 6001, 6009, and 30000 to 31000 of the agents in default
configurations.  You can of course change those port numbers and ranges in the configuration.Manager-to-Agent TCP PortsUsage6001ZeroMQ-based RPC calls from managers to agents6009HTTP watcher API30000-31000Port pool for in-container servicesThe operation of agent itself does not require both incoming/outgoing access to
the public Internet, but if the user's computation programs need the Internet, the docker containers
should be able to access the public Internet (maybe via some corporate firewalls).Agent-to-X TCP PortsUsagemanager:5002ZeroMQ-based event push from agents to the manageretcd:2379etcd API accessredis:6379Redis API accessdocker-registry:{80,443}HTTP watcher API(Other hosts)Depending on user program requirementsLICENSESGNU Lesser General Public LicenseDependencies