Package: epic-lab
Epic lab — Opinionated research lab management toolsWhat is it?Theepic-labPython library and its additional resources serve to create a fully functional cloud lab environment
for organizational — or individual — research work.An opinionated libraryThis library makes specific choices in several major questions, not leaving them open for configuration.It currently holds the following non-negotiable opinions:Uses Google Cloud PlatformUses compute VM instances which...are for individual usageare namedusername-<suffix>(launch date suffix recommended)run on Ubuntu 20.04 LTSuse Python 3.10usecondafor Python environment managementrunJupyter Labcome pre-installed with some tools with pinned versionsgcc-10, nodejs v16come pre-installed with some Python libraries with mostly pinned versionsnumpy, pandas, matplotlib, scikit-learn, cython, pybind11, networkx, lz4, tqdm, dill, cytoolzcome pre-installed with the core epic-framework librariesprotect against OOM crashes by limiting Jupyter Lab service memory usage (85% soft, 92% hard)Uses shared cloud resources:One GCP project for everythingA shared GCP git repository for notebook storageA shared GCP git repository for user configuration storageA shared service account (the project's default) for VMsCode is...developed on a local machineuploaded to a shared staging areadownloaded on VMS for usageauto-downloaded whenever an IPython kernel is startedIt also strongly encourages you to:Use a single Cloud Run NGINX reverse proxy to access notebooks viahttps://<common-domain>/<instance-name>Use a single Google-managed domain for your base urlLimit online access through GCP authentication and access management (using IAP on a Load Balancer)Future major releases would likely update some of these opinions, most commonly by updating versions where applicable.Alternative opinions welcomeAll other opinions are just a fork away!While this project does not offer many formalized ways to parameterize its behavior, it does allow anyone to fork it and
modify any areas of the setup code to fit their specific needs. Every lab environment is different and so it is expected
to happen often. This is especially true for thevmsetupandproxyareas.Theepic-labPython library is somewhat more generic, and should hopefully be usable as-is for most situations. If it
isn't, we would certainly appreciate getting a pull request.UsagePreparationTypical users of an epic lab environment need to do two things:install theepic-lablibrarycreate a configuration file at~/.epic/labThis of course assumes that a cloud lab has been set up. See theCloud setup guidesection below
for details. This is a process that only needs to be done once.
Use the~/.epic/labconfiguration created during this setup process as a template for providing general users with
their own derivative configuration files.Working with a notebook instanceTo launch a new VM, run theepic-notebook launchcommand. For example:epic-notebooklaunchgooduser-20220704n2-standard-2To suspend a VM, run theepic-notebook suspendcommand. For example:epic-notebooksuspendgooduser-20220704To resume a suspended VM, run theepic-notebook resumecommand. For example:epic-notebookresumegooduser-20220704To terminate a VM, run theepic-notebook terminatecommand. For example:epic-notebookterminategooduser-20220704To connect to the machine usingssh, run theepic-notebook sshcommand.
This does not use a direct network connection, tunneling through the IAP service instead.
This requires users to have the following IAM roles assigned to them:roles/iap.tunnelResourceAccessorroles/compute.instanceAdmin.v1Note that any additional arguments will be passed tossh. So you can use it to e.g. forward a port by running:epic-notebooksshgooduser-20220704-L18080:localhost:8080Synchronizing working code into VMsCode synchronization works through a shared staging area. Repositories (really, folders) under a common parent directory
areuploadedto the staging area from the local working machine, and then later downloaded to a shared parent
directory (~/synccode) on cloud VMs for usage.To upload all changes to synchronized reposfrom your localworking machine, run theepic-synccode uploadcommand.You can optionally give it one or more repos as parameters to upload their changes exclusively.
For example:epic-synccodeuploadrepo1,repo2,repo3To download all changeson your cloud VM, run theepic-synccode downloadcommand.Note: Theepic-synccode downloadruns implicitly whenever an IPython kernel is started.If you want to stop synchronizing a previously synchronized repo, start by runningon your localworking machine
theepic-synccode deletecommand. For example:epic-synccodedeleterepo2After this command, VM downloaded copies ofrepo2will not be updated further, but are not auto-deleted. Remove these
copies from e.g.~/synccode/repo2manually.Finally, if you want to load all synccode paths into your python path, you can use theepic-synccode pathcommand.
This is useful for running modules withpython -m. For example:PYTHONPATH=$(epic-synccodepath)python-mrepo.pkg.moduleCloud setup guideThe setup process is made up of three parts.
Each part provides an additional layer of value for your setup, and relies on the previous ones.
You don't have to implement all of them - in fact you get most of the value (notebook machines and synccode) just from
the first part, which is also the simplest to set up.The three parts are:setting up cloud resources to allow launching notebook VM instances and synchronizing working code into themsetting up a reverse proxy to allow accessing the VMs through one base URL and the VM's namesetting up secure access from the Internet to the reverse proxy based on GCP authentication and access managementFollow the full guide incloud_setup.mdto create your lab today.