1[7] LPMX: A pure rootless composable container
system

トーゴーの日シンポジウム2021

楊 旭 （ヨウ キョク）
東京大学新領域創成科学研究科 メデイカル情報生命専攻
笠原研究室

Licensed under a Creative Commons 表示4.0国際ライセンス©2021 楊 旭（東京大学新領域創成科学研究科）

2[7] LPMX: A pure rootless composable container
system

トーゴーの日シンポジウム2021

With the advancement in sequencing technologies such as Next Generation Sequencing(NGS), huge amounts of genomic data is being generated
at a fast rate. Scientists have to use pipelines to automate the analysis procedure and use High-Performance Computing (HPC) infrastructures toaccelerate the analysisBioconductor: 1974+ packages
Bioconda:3500+ packages
Biocontainer: 1000+ images

Licensed under a Creative Commons 表示4.0国際ライセンス©2021 楊 旭（東京大学新領域創成科学研究科）

3[7] LPMX: A pure rootless composable container
systemTools are developed by students / researchers
often lack maintenance•For example: GeneWise•GeneWise was released in 2004 (ubuntu10.04), and stopped maintenance since version
2.2.3•Other tools, e.g. PASA2, CEGMA, depend on it•To compile GeneWise, it will cost researchers alarge amount of time to set up the environment

Dependency Hell Issue, arises when different
incompatible versions of shared libraries are required bydifferent packages•For example: Bioconda•When a package exist in multiple channels and isout of date in the channel with the highest priority.
Other packages depending on this package willreport incompatible error•Cannot automatically resolve conflicting packages inthe same namespace. For example, Illumina/manta
depends Python2, any other packages depending on
Python3 will be conflicted.

トーゴーの日シンポジウム2021

Difficult to reproduce the runtime environment,
users have to run software in compatible runtimes
across different machines•For example: Difference between OS, VM
platforms, Container platforms (Singularity[1])•It is often difficult to migrate tools from one
platform to another due to the requirements for
installing compatible platforms and runtimes

Licensed under a Creative Commons 表示4.0国際ライセンス©2021 楊 旭（東京大学新領域創成科学研究科）

4[7] LPMX: A pure rootless composable containersystem

Distribute source tar-ball, compiling source code eachtime•Pros•Convenient for developers•Optimized for current machines•If dependencies and libraries are alreadysatisfied, then no need to ask for theadmin’s help
•Cons
•Dependencies and libraries are usually notsatisfied•If packages are conflicting with host ones,
then they will fail•Weak isolation, libraries in the host have
impact on target compiling tools

Use VM to distribute pipelines•Pros•Best isolation, including an entire OS•Single file delivery•Works on old Linux kernel•Cons•Inflexibility, usually does not support layered filesystem, so changing a file will result in an entire
new package•Have to ask the server admin to install the same
VM platform•Performance overhead, e.g. I/O•Pre-allocated computing resources (CPU,memory, network bridges)•Security concern

トーゴーの日シンポジウム2021

Use container technology to distribute pipelines•Pros•Good isolation, lightweight•Support layered file system•Flexible, CGroup resource isolation•Many out-of-box images•Cons•Have to ask the server admin to installcompatible container platforms•Require modern Linux kernels•Maintenance cost
•Security concern (Docker Vulnerability)

Licensed under a Creative Commons 表示4.0国際ライセンス©2021 楊 旭（東京大学新領域創成科学研究科）

5[7] LPMX: A pure rootless composable containersystem

トーゴーの日シンポジウム2021•For genome analysis pipelines, researchers usually use a bunch of tools and scripts to shred data
files
Different tools and scripts require different runtime environments, larger chance to haveconflicting packages (Bioconda not work)•The input and output of these data files are trusted, and the output of previous step is usuallyused as the input of next steps
Mainly handle the file system isolation (Bioconda not work)•Dependencies and libraries required by these tools and scripts are varieties since they aredeveloped by different teams in past decades.
We need to separate tools into different containers to meet their runtime requirements,
meantime, we need to handle the communication among these tools (All containers notwork)•Many pipelines supports GPGPU and requires batch job engines
We have to support GPGPU as well as batch job systems (All containers not work on batchengine)•Reduce the cost for negotiation with the server administrator for installing container platforms(HPC)
Rootless support (Singularity not work)•Do modifications inside containers (optional)
Read-write container rather than read-only (Singularity not work)•Robust to old Linux kernel (HPC)
It is quite difficult to persuade the server administrator to upgrade Linux kernel, robust toLinux kernel (Docker, Podman not work)
Composability, refers to the ability that allows processes running inside containers•Support multiple image formats
to make calls to other processes in the host or other containers
Compatible with different implementations (Docker, Podman not work)

Licensed under a Creative Commons 表示4.0国際ライセンス©2021 楊 旭（東京大学新領域創成科学研究科）

6[7] LPMX: A pure rootless composable containersystem

トーゴーの日シンポジウム2021Local Package Manager X (LPMX)
A pure rootless composable container system
https://github.com/jasonyangshadow/lpmx

Licensed under a Creative Commons 表示4.0国際ライセンス©2021 楊 旭（東京大学新領域創成科学研究科）