Package: gvmkit-build
gvmkit-buildGolem VM Image builder used as companion app for Golem Registry:https://registry.golem.networkRequirementsRunning docker engine is required. Tool supports Linux, Windows and macOS.
Note that when using macOS ARM version use --platform linux/amd64 option for docker builds.InstallationYou can install gvmkit-build using pip (python3 with pip installer is required)pip install gvmkit-buildor install form npm (npm installation is required)npm install -g gvmkit-buildor install using cargo (Rust toolchain is required, a bit slow because it compiles from sources)cargo install gvmkit-buildor download prebuild from github releases page:https://github.com/golemfactory/gvmkit-build-rs/releasesor build from sources, you can find binary in target/release/gvmkit-build (or gvmkit-build.exe on Windows)cargo build --releaseImagesGolem Network is using gvmi images as base for creating VMs for tasks.
These images are basically squashfs images with some additional metadata.
They can be prepared from docker images using this (gvmkit-build) tool.Quick startMake sure your docker service is running and you have gvmkit-build installedGo to the folder with your dockerfile and rundocker build . -t my_imageCreate account on registry portalhttps://registry.golem.networkLet's assume your user name is golemCreate repository on registry portalLet's assume you created repository named my_exampleCreate and copy personal access token from registry portalRun (you will be asked for login and personal access token)gvmkit-build my_image --push-to golem/my_example:latestYour taggolem/my_example:latestis ready to use in one of Golem Network APIsNaming imageThe tool as main argument takes docker image name.Docker image name can be resolved using ImageId + tag or Repository name + tag.
Repository name can be composed of maximum two parts:<username>/<repository>Examples:pythonresolves topython:latestpython:3.8resolves topython:3.8golemfactory/blenderresolves togolemfactory/blender:latestYou can also use image id instead of name, usedocker image lsto find your image id.Following command will build image and create *.gvmi file in current directory.gvmkit-build <image_name>If image not exist locally the tool is trying to pull it from docker hub.To use it in Golem Network, you have to upload it to registry portal.To successfully add image to registry portal you have to name image accordingly or usegvmkit-build <user_name>/<image_name>:<tag> --pushor if your local image name is not compatible usegvmkit-build <docker_image_id> --push-to <user_name>/<image_name>:<tag>Build process explained a bitTool is creating new container and is copying data from given image to new container.
After copying is finished mksquashfs command is used inside tool container to create squashfs image.
After adding metadata *.gvmi file is created and tool container removed.
Note that tool image will stay downloaded on your machine for future use (it is quite small so no worry about disk space)[Optional] - building squashfs-tools imageIf you want to use your own tool without pulling from dockerhub:
Go to squashfs-tools directory and rundocker build . -t my_squash_fs_builderadd environment variable to .env in folder where you run gvmkit-buildSQUASHFS_IMAGE_NAME=my_squash_fs_builderFor managing docker images and containers bollard library is used.https://docs.rs/bollard/latest/bollard/Troubleshooting login to registry portalThe tool is usinghttps://registry.golem.networkas default registry portal.
You can change this behaviour by settingREGISTRY_URLenvironment variable.
You should create account on registry portal and generate access token for your user.
If you really don't want to create account you can use anonymous upload (see section below about anonymous upload).The tool will ask for login when --login --push or --push-to option is specified.For storing login information rpassword library is used:https://docs.rs/crate/rpassword/latestOnly one instance of login/token is kept saved at "gvmkit-build-rs/default". So if you login with new user/token old pair will be forgotten.This option will ask for login to registry portalgvmkit-build --loginYou can use command for check if your login information is correctgvmkit-build --login-checkIf you want to forget your login information you can use, it will clear your login informationgvmkit-build --logoutAbove command are optional, because you will be asked for login automatically if you use --push or --push-to option.On some systems when no secure store is provided you will be forced to use following method of keeping login/token pair.
You can optionally put them in .env file from convenience in your working directory.
Remember to set proper permissions to the file (chmod 600 .env) if using on shared machine.REGISTRY_USER=<username-in-registry-portal>
REGISTRY_TOKEN=<access-token-generated-in-registry-portal>When REGISTRY_USER/REGISTRY_TOKEN is set in environment variables, secure rpassword storage won't be used.Uploading to multiple tagsThe tool cannot upload to multiple tags, but you can call it multiple times with different tags.
All steps of operations are cached so no worry about re-uploading same file multiple times.Uploading large filesThe tool is using chunked upload with default chunk size 10MiB for images greater than 500MiB (changing not recommended).
Four upload workers are created by default (you can increase/decrease number of workers using --upload-workers argument depending on your network conditions).
If you think your upload is stuck you can always stop and run tool again to finish download. Only chunks that were not uploaded
will be uploaded again.Note: Total limit of chunks is set to 1000 (so around 10GB by default). If you want to upload larger file you have to set greater chunk size accordingly.Uploading image without loginYou can upload images anonymously. Note that lifetime of such images is limited,
and they can be removed from registry portal after some time without noticecargo run --release -- <image_name> --push --nologinUploading image without building partIf you are sure that you have proper *.gvmi file for example my-test.gvmi you can usegvmkit-build --direct-file-upload my-test.gvmi --push-to <user_name>/<image_name>:<tag>or anonymouslygvmkit-build --direct-file-upload my-test.gvmi --push --nologinChanging squashfs options when creating imageYou can change compression used in mksquashfs to produce more or less compact images.Look for help for more information. Note that currently zstd is not supported by Golem Network (you can use xz instead for extra compact images).gvmkit-build --help