Package: devcluster
devclusterA developer tool for running the Determined cluster.Configuring and RunningYou can run the code from the git root by runningpython -m devcluster, or
install it into your python environment withpip install -e .and then just
calldevclusterby itself.The first time you run it, it will give you the option to automatically install
the default configuration file (seedevcluster/example.yaml).By default, devcluster needs to run from the root of adeterminedsource
tree.  You can do this in one of a few ways:always rundevclusterfrom the root of yourdeterminedsource treesetcwd: /path/to/determinedto rundevclusterfrom anywhererundevclusterwith the-C /path/to/determinedoptionYou can either specify a config file via the--configor-coption, ordevclusterwill try to read the default config file at~/.devcluster.yaml.HowdevclusterWorksdevcluster runs a cluster with a linear dependency graph of stages.  With the
default config (devcluster/example.yaml), those stages are:DEADDBMASTERAGENTBecause of the linear dependency graph, theDBstage can run by itself, theMASTERstage is only started afterDBis already up, and theAGENTis only
started after theMASTERis up.  TheDEADstage indicates that nothing
is running.  The non-DEADstages correspond to thestagessetting in your
config file.The status bar has two rows.  The first row is the state row.  It shows all of
the stages in your dependency graph.  All active stages are highlighted orange.
There is also a 'target state', which is indicated by a less-than character
(<).  You set the target state by pressing the key corresponding to a
particular stage.When the target state is modified, devcluster will walk up or down the stage
list, starting or killing processes as needed to reach the target stage.  In
the default configuration, the MASTER and AGENT stages are rebuilt each time
they are started.A common pattern during development might be to:make a change to the master codepress1to set the target state toDB(killing theMASTERand theAGENT)press3to set the target state back toAGENT(rebuilding/restarting theMASTERandAGENTon the way).The second row in the status bar is the logging row.  Every stage collects a
separate stream of logs, and each stream can be toggled by pressing the
corresponding keys.Keybindings1-3set target states for the cluster (as shown in the status bar)!-#toggle logs for the corresponding stage (as shown in the status bar)qorctrl+conce to quit, or twice to force-quit (which may leave
dangling processes or docker containers laying around)u/dwill scroll up/down, andU/Dwill scroll up/down faster
Scrolling support is only partially line-aware; it will scroll by
literal log chunks, which may each be more or less than a linexwill reset scrolling to continue following logsSpacebar will emit a visual separator with a timestamp to the console stream.With the default configuration, the keysp,w, andcwill trigger
building the python harness, webui, and docs, respectively.Server/Client ModeYou can rundevclusteras a headless server with various configurations for
listening for clients:# listen on 0.0.0.0:1234
devcluster --quiet --listen 1234

# listen on localhost:1234
devcluster -q -l 127.0.0.1:1234

# listen on a unix socket
devcluster -q -l /path/to/socket

# listen on the default unix socket (/tmp/devcluster/sock)
devcluster -q

# configure multiple listeners
devcluster -q -l 1234 /path/to/socketSending aSIGINT(viactrl+c) or aSIGTERM(viakill) to the headless
server will close it.You can then connect the UI to a headless server:devcluster 1234
devcluster host:1234
devcluster /path/to/socketIn client mode, pressingqorctrl+cwill only close the UI; it will not
affect the server.Oneshot ModeYou can rundevclusterin a special oneshot mode (-1/--oneshot), suitable
for automated usage.  In automated mode, thestdoutfromdevclusterwill be
suitable for redirecting to a file, and thestderrwill emit the line:devcluster is upto indicate when the cluster is ready for use.Philosophydevclustertries as much as possible to be a standalone tool for running
arbitrary versions of the Determined cluster.  It is tested as far back as
v0.12.3.Some defaults are kept up-to-date with tip-of-main to simplify the config
file for the most-common use case, but in general preference was given to being
configurable and explicit.TODOconfig file reloading