Package: minipresto
MiniprestoA command line tool that makes it easy to run modular Presto environments
locally. Compatible with Starburst releases up to 350-e. After 350-e, the
refactored version of this project, Minitrino, will need to be used.Latest Stable Release: 1.0.2OverviewRequirementsInstallationCLITop-Level CLI OptionsProvisioning EnvironmentsRemoving ResourcesShutting Down EnvironmentsTaking Environment SnapshotsManage User ConfigurationInstall the LibraryDisplay Module MetadataDisplay CLI VersionPointing the CLI to the Minipresto LibraryMinipresto Configuration FileProject StructureAdding New Modules (Tutorial)TroubleshootingReporting Bugs and ContributingRequirementsDocker 19.03.0+Docker Compose (1.29.0+)Python 3.8+PipLinux or Mac OSInstallationEnd UsersMinipresto is available on PyPI and the library is available for public download
on GitHub. To install the Minipresto CLI, runpip install minipresto. To
install the library, runminipresto lib_install.DevelopersIn the project's root, run./install.shto install the Minipresto CLI. If you
encounter errors during installation, try runningsudo -H ./install.sh -v.CLIMinipresto is built withClick, a
popular, open-source toolkit used to build Python-based CLIs.All Minipresto commands/options are documented below. Note that many command
options can be specified with a shorthand alternative, which is the first letter
of each option, i.e.--modulecan be-m.Top-Level CLI OptionsYou can get help, enable verbose output, and change the runtime library
directory for any command.Usage: minipresto [OPTIONS] COMMAND [ARGS]...

Options:
  -v, --verbose   Enable verbose output.
  -e, --env TEXT  Add or override environment variables.

                  Environment variables are sourced from the Minipresto
                  library's root 'minipresto.env' file as well as the user 
                  config file in '~/.minipresto/minipresto.cfg'. Variables 
                  supplied by this option will override values from either 
                  of those sources. The variables will also be passed to the
                  environment of the shell executing commands during the
                  'provision' command.

  --help          Show this message and exit.Provisioning EnvironmentsYou can provision an environment via theprovisioncommand.Usage: minipresto provision [OPTIONS]

  Provision an environment based on specified modules. All options are
  optional and can be left empty.

Options:
  -m, --module TEXT         A specific module to provision.
  -n, --no-rollback         Do not rollback provisioned resources in the event
                            of an error.

  -d, --docker-native TEXT  Appends native docker-compose commands to the
                            generated docker-compose shell command. Run
                            `docker-compose up --help` to see all available
                            options.

                            Example: minipresto provision --docker-native
                            --build

                            Example: minipresto provision --docker-native '--
                            remove-orphans --force-recreate'

  --help                    Show this message and exit.Notes:If no options are passed in, the CLI will provision a standalone Presto
container.The command cannot currently be used to append additional modules to an active
environment. To modify an environment, first shut it down, then re-provision
with the needed modules.Sampleprovisioncommands:miniprestoprovision\--modulehive-s3\--moduleelasticsearch\--moduleldap\--docker-native'--build --force-recreate'miniprestoprovision-mhive-s3-melasticsearch-mldap

minipresto--envSTARBURST_VER=332-e.6provisionTheprovisioncommand constructs a Docker Compose command and executes it in
the host shell. The commands look similar to:ENV_VAR_1=SOMETHINGENV_VAR_2=SOMETHINGENV_VAR_3=${ENV_VAR_3}...\docker-compose-fdocker-compose.yml\-fmodules/catalog/elasticsearch/elasticsearch.yml\-fmodules/catalog/hive-s3/hive-s3.yml\-fmodules/security/ldap/ldap.yml\up-dUsing the structure of the Minipresto library, it is able to merge multiple
Docker Compose files together.Environment VariablesEnvironment variables passed to Docker containers are sourced through two
locations. The first is from theminipresto.envfile in the library root. These
variables define the versions of the provisioned Docker services. The second is
from from variables set in the[MODULES]section of theminipresto.cfgfile.
These variables can contain sensitive information like access credentials, so
their values are intentionally left out of library files.Any existing environment variable can be overridden with the top-level--envoption, and any unset variable can be set with it.Using Licensed Starburst FeaturesIf you are using licensed features, you will need to provide a path to a valid
Starburst license. This can be set viaminipresto configor provided via the--envoption at command runtime. The variable for this isSTARBURST_LIC_PATH.Additionally, you need to uncomment the volume mount in the library's rootdocker-compose.ymlfile:# Uncomment this to enable the volume mount. The variable should point to a# valid SEP license.volumes:-"${STARBURST_LIC_PATH}:/usr/lib/presto/etc/starburstdata.license:ro"Removing ResourcesYou can remove resources with theremovecommand.Usage: minipresto remove [OPTIONS]

  Remove Minipresto resources.

Options:
  -i, --images      Remove Minipresto images.
  -v, --volumes     Remove Minipresto container volumes.
  -l, --label TEXT  Target specific labels for removal (format: key-value
                    pair(s)).

  -f, --force       Force the removal of Minipresto resources. Normal Docker
                    removal restrictions apply.

  --help            Show this message and exit.Notes:Named volumes tied to anyexistingcontainer cannot be forcibly removed,
neither by Minipresto nor by the Docker CLI/SDK.Images tied to stopped containers can be forcibly removed, but any image tied
to a running container cannot be forcibly removed, neither by Minipresto nor
by the Docker CLI.Sampleremovecommand:minipresto-vremove\--volumes\--labelcom.starburst.tests.module.postgres=catalog-postgres\--forceThis will only remove volumes associated to the Postgres catalog module.Shutting Down EnvironmentsYou can shut down an active environment with thedowncommand.Usage: minipresto down [OPTIONS]

  Bring down running Minipresto containers. This command follows the
  behavior of `docker-compose down` where containers are both stopped and
  removed.

Options:
  -k, --keep  Does not remove containers; instead, containers will only be
              stopped.

  --sig-kill  Stop Minipresto containers without a grace period.
  --help      Show this message and exit.Sampledowncommand:minipresto-vdownTaking Environment SnapshotsYou can capture snapshots for both active and inactive environments with thesnapshotcommand.Usage: minipresto snapshot [OPTIONS]

  Create a snapshot of a Minipresto environment. A tarball is placed in the
  Minipresto `lib/snapshots/` directory.

  To take a snapshot of an active environment, leave the `--module` and
  option out of the command.

  To take a snapshot of modules whether they are active or not, specify the
  modules via the `--module` option.

Options:
  -m, --module TEXT     A specific module to snapshot.
  -n, --name TEXT       Basename of the resulting snapshot tarball file.
                        Allowed characters: alphanumerics, hyphens, and
                        underscores.  [required]

  -d, --directory PATH  Directory to save the resulting snapshot file in.
                        Defaults to the snapshots directory in the Minipresto
                        library.

  -f, --force           Overwrite the file if it already exists.
  --no-scrub            Do not scrub sensitive data from user config file.

                        WARNING: all sensitive information (passwords and
                        keys) will be kept in the user config file added to
                        the snapshot. Only use this if you are prepared to
                        share those secrets with another person.

  --help                Show this message and exit.Notes:Minipresto records the originalprovisioncommand and places it in the
snapshot file asprovision-snapshot.sh; this can be directly executed. This
makes it easier for others to reuse the environment and provision it
identically.Samplesnapshotcommands:# Take a snapshot of an active environment (this will create a tarball# called `snapshot-t2533.tar.gz` in the library's `snapshots/` directory):miniprestosnapshot--namet-2533# Take a snapshot of specific modules:miniprestosnapshot-nsuper-cool-env-mhive-s3-melasticsearch-mldapManage User ConfigurationYou can manage Minipresto configuration with theconfigcommand.Usage: minipresto config [OPTIONS]

  Edit the Minipresto user configuration file.

Options:
  -r, --reset  Reset the Minipresto user configuration file and create a new
               config file from a template.

               WARNING: This will remove your configuration file (if it
               exists) and replace it with a template.

  --help       Show this message and exit.Install the LibraryYou can install the Minipresto library with thelib_installcommand.Usage: minipresto lib_install [OPTIONS]

  Install the Minipresto library.

Options:
  -v, --version TEXT  The version of the library to install.
  --help              Show this message and exit.Display Module MetadataYou can see Minipresto module metadata with themodulescommand.Usage: minipresto modules [OPTIONS]

  Display module metadata.

Options:
  -m, --module TEXT  A specific module to display metadata for.
  -j, --json         Print the resulting metadata in JSON form (shows
                     additional module metadata).

  -r, --running      Print metadata for all running modules.
  --help             Show this message and exit.Display CLI VersionYou can display the Minipresto CLI version with theversioncommand.Usage: minipresto version [OPTIONS]

  Display the Minipresto version.

Options:
  --help  Show this message and exit.Pointing the CLI to the Minipresto LibraryThe Minipresto CLI should always point to a compatible library with the expected
structure. The library directory can be set one of four ways, listed below in
the order of precedence:Passing theLIB_PATHvariable to the CLI's--envoption sets the library
directory for the current command.Theminipresto.cfgfile'sLIB_PATHvariable sets the library directory if
present.The path~/.minipresto/lib/is used as the default lib path if theLIB_PATHvar is not found.As a last resort, Minipresto will check to see if the library exists in
relation to the positioning of thecomponents.pyfile and assumes the
project is being run out of a cloned repository.If you not running out of a cloned repository, it is advisable to provide a
pointer to the library in Minipresto's configuration via theLIB_PATHconfig.Minipresto Configuration FileSticky configuration is set in~/.minipresto/minipresto.cfg. The sections in
this file each serve a separate purpose.[CLI] SectionThese configs allow the user to customize the behavior of Minipresto.LIB_PATH: The filesystem path of the Minipresto library (specifically to thelib/directory).TEXT_EDITOR: The text editor to use with theconfigcommand, e.g. "vi",
"nano", etc. Defaults to the shell's default editor.[DOCKER] SectionThese configs allow the user to customize how Minipresto uses Docker.DOCKER_HOST: A URL pointing to an accessible Docker host. This is
automatically detected by Docker otherwise.[PRESTO] SectionThese configs allow the user to propagate config to the Presto container. Since
many modules can append to Presto's core files, the supported way to make
propagate changes to these Presto files is with these configs.CONFIG: Configuration for Presto'sconfig.propertiesfile.JVM_CONFIG: Configuration for Presto'sjvm.configfile.A multiline example of this section (note the indentation):[PRESTO]
CONFIG=
    query.max-memory-per-node=500MB
    query.max-total-memory-per-node=500MB
JVM_CONFIG=
    -Dsun.security.krb5.debug=true[MODULES] SectionThis section sets environment variables passed to containers provisioned by
Minipresto. Environment variables are only passed to a container if the variable
is specified in the module'sdocker-compose.ymlfile.Variables propagated to the Presto container are supported by Presto secrets.STARBURST_LIC_PATH: Required if using licensed Starburst Enterprise Presto
features. It can point to any valid license on your filesystem.S3_ENDPOINTS3_ACCESS_KEYS3_SECRET_KEYAWS_REGIONSNOWFLAKE_DIST_CONNECT_URLSNOWFLAKE_DIST_CONNECT_USERSNOWFLAKE_DIST_CONNECT_PASSWORDSNOWFLAKE_DIST_WAREHOUSESNOWFLAKE_DIST_DBSNOWFLAKE_DIST_STAGE_SCHEMASNOWFLAKE_JDBC_CONNECT_URLSNOWFLAKE_JDBC_CONNECT_USERSNOWFLAKE_JDBC_CONNECT_PASSWORDSNOWFLAKE_JDBC_WAREHOUSESNOWFLAKE_JDBC_DBSNOWFLAKE_JDBC_STAGE_SCHEMAProject StructureThe library is built around Docker Compose files and utilizes Docker's ability
toextend Compose files.The Starburst Presto service is defined in a Compose file at the library root,
and all other services look up in the directory tree to reference the parent
Presto service. In Compose files, the fully-qualified path––relative to the
library's rootdocker-compose.ymlfile––must be provided for Docker to locate
resources.A simplified library structure:lib
├── Dockerfile
├── docker-compose.yml
├── minipresto.env
├── modules
│   ├── catalog
│   │   └── postgres
│   │       ├── metadata.json
│   │       ├── postgres.yml
│   │       ├── readme.md
│   │       └── resources
│   │           ├── postgres
│   │           │   └── postgres.env
│   │           └── presto
│   │               └── postgres.properties
│   ├── resources
│   │   └── wait-for-it.sh
│   └── security
│       └── event-logger
│           ├── event-logger.yml
│           ├── metadata.json
│           ├── readme.md
│           └── resources
│               ├── event-logger
│               │   └── postgres.env
│               └── presto
│                   ├── event-listener.properties
│                   └── postgres_event_logger.properties
├── snapshots
└── versionAnd the contents of adocker-compose.ymlfile (postgres.yml):version:"3.7"services:presto:volumes:-"./modules/catalog/postgres/resources/presto/postgres.properties:/usr/lib/presto/etc/catalog/postgres.properties"postgres:image:"postgres:${POSTGRES_VER}"container_name:"postgres"labels:-"com.starburst.tests=minipresto"-"com.starburst.tests.module.postgres=catalog-postgres"env_file:-"./modules/catalog/postgres/resources/postgres/postgres.env"Notice that the volume mount is not relative to thelib/modules/catalog/postgres/directory––it is relative to the parent
directory which houses the top-leveldocker-compose.ymlfile. Also, notice the
labels––these labels will be used to identify Docker resources tied to
Minipresto modules so that the CLI commands actually work.Presto DockerfileMinipresto modifies the Starburst Presto Docker image by adding the Presto CLI
to the image as well as by providingsudoto theprestouser. This is
required for certain bootstrap scripts (i.e. usingyumto install packages in
a Presto container for a module). This image is compatible with Starburst Presto
images back to Starburst Presto version332-e.0.Adding New Modules (Tutorial)Adding new modules is relatively simple, but there are a few important
guidelines to follow to ensure compatibility with the Minipresto CLI. The design
rules are the same for both catalogs and security modules. The example below
demonstrates the process of creating a new catalog module for a Postgres
service.Create the Module DirectoryCreate the module's directory in thelib/modules/catalog/directory:mkdirlib/modules/catalog/postgres/cdlib/modules/catalog/postgres/Add Presto ResourcesAll resources for a module go inside of aresources/directory within the
module. Inside this directory, place Presto-specific resources into apresto/directory, then mount the resources to the Presto service defined in the rootdocker-compose.ymlfile.mkdir-presources/presto/In the newly-createdresources/presto/directory, add a properties file.bash-c"cat << EOF > postgres.propertiesconnector.name=postgresqlconnection-url=jdbc:postgresql://postgres:5432/miniprestoconnection-user=adminconnection-password=prestoRocks15EOF"Note: Passwords should always beprestoRocks15for consistency throughout
modules.Add the Docker Compose YAMLInlib/modules/catalog/postgres/, add a Docker Compose file:touchpostgres.ymlNotice the naming convention:postgres.yml. Giving the same root name of
"postgres" to both the parent directorypostgres/and to the Docker Compose
filepostgres.ymlwill allow Minipresto to find our new catalog module.Next, add an environment file for the Postgres service. Non-Presto resources
should go into their own directory, so create one for postgres:mkdirresources/postgres/In the newly-created directory, add an environment file which will register the
variables in the Postgres container when it is provisioned:bash-c"cat << EOF > postgres.envPOSTGRES_USER=adminPOSTGRES_PASSWORD=prestoRocks15POSTGRES_DB=miniprestoEOF"This file will initialize Postgres with a databaseminipresto, a userpresto, and a passwordprestoRocks15.Add a Metadata FileThis step is not required for personal development, but it is required to commit
a module to the Minipresto repository.Inlib/modules/catalog/postgres/, add themetadata.jsonfile:bash-c'cat << EOF > metadata.json{"description": "Creates a Postgres catalog using the standard Postgres connector.","incompatible_modules": []}EOF'The metadata file is presentable to the user via themodulescommand, and theincompatible_moduleskey restricts certain modules from being provisioned
alongside the given module. The*wildcard is a supported convention if the
module is incompatible with all other modules.Add a Readme FileThis step is not required for personal development, but it is required to commit
a module to the Minipresto repository.Inlib/modules/catalog/postgres/, add thereadme.mdfile:touchreadme.mdThis file should contain an overview of the module.Review ProgressThe resulting directory tree should look like this (from the/modules/catalog/directory):postgres
├── metadata.json
├── postgres.yml
├── readme.md
└── resources
    ├── postgres
    │   └── postgres.env
    └── presto
        └── postgres.propertiesConfigure the Docker Compose YAML FileWe will now define thepostgres.ymlDocker Compose file. Set it up as follows,
andread the important notes after:version:"3.7"services:presto:volumes:# Always place Presto files in `/usr/lib/presto/etc/` as symbolic links can change between versions-"./modules/catalog/postgres/resources/presto/postgres.properties:/usr/lib/presto/etc/catalog/postgres.properties"postgres:image:"postgres:${POSTGRES_VER}"container_name:"postgres"labels:-"com.starburst.tests=minipresto"-"com.starburst.tests.module.postgres=catalog-postgres"env_file:-"./modules/catalog/postgres/resources/postgres/postgres.env"Important Implementation Details: Paths and LabelsWe can observe a few things about the Compose file we just defined.Path References for Volumes and Build ContextsFirst, the volumes we mountare not relative to the Compose file itself, they
are relative to the basedocker-compose.ymlfile in the library root. This is
because the CLI extends Compose files, meaning that all path references in child
Compose files need to be relative to the positioning of the parent Compose file.The base Compose file is determined when you execute a Docker Compose
command––the first Compose file referenced in the command becomes the base file,
and that happens to be thedocker-compose.ymlfile in the library root. This
is how Minipresto constructs these commands.If this is confusing, you can read more about extending Compose files on theDocker docs.Minipresto Docker LabelsSecondly, notice how we applied sets of labels to the Postgres service. These
labels tell the CLI which resources to target when executing commands.In general, there is no need to apply labels to the Presto service since they
are already applied in the parent Compose fileunlessthe module is an
extension of the Presto service itself (i.e. the Snowflake modules). Labels
should always be applied to:Docker services (AKA the resulting container)Named volumesImages built from a DockerfileLabels should be defined in pairs of two. The convention is:The standard Minipresto resource label:com.starburst.tests=miniprestoA module-specific resource label:com.starburst.tests.module.<module-name>=<module-type>-<module-name>For this label, themodule-typeshould be eithercatalogorsecurityThis applies a unique label to the module, allowing it to be an isolated
component when necessary.In Compose files where multiple services are defined, all services should be
labeled with the same label sets (seehive-s3.ymlfor an example).Note: A named volume is defined explicitly in the Compose file, and these
should always have label sets applied to them. Below is an example of the
Compose file we just created with a named volume.version:"3.7"services:presto:volumes:-"./modules/catalog/postgres/resources/presto/postgres.properties:/usr/lib/presto/etc/catalog/postgres.properties"postgres:image:"postgres:${POSTGRES_VER}"container_name:"postgres"labels:# These labels are applied to the service/container-"com.starburst.tests=minipresto"-"com.starburst.tests.module.postgres=catalog-postgres"env_file:-"./modules/catalog/postgres/resources/postgres/postgres.env"volumes:postgres-data:labels:# These labels are applied to the named volume-"com.starburst.tests=minipresto"-"com.starburst.tests.module.postgres=catalog-postgres"Note: Certain modules will only extend the parent Presto service and do not
actually define any new services/containers. See the Snowflake catalog modules
for an example of this. For these modules, the only label requirement is to add
the module-specific label to the Presto service in the relevantdocker-compose.ymlfileTest the New CatalogWe are all finished up. We can test our new catalog through the Minipresto CLI:miniprestoprovision-mpostgresWe can now shell into theprestocontainer and run some tests:docker exec -it presto bash 
presto-cli
presto> show catalogs;Customizing ImagesIf you need to build an image from a local Dockerfile, you can do so and
structure the Compose file accordingly. See the library's rootdocker-compose.ymlfile for an example of this. Path references for volumes
and the image build context will follow the same convention as volume mount
paths described earlier.Bootstrap ScriptsMinipresto supports container bootstrap scripts. These scriptsdo not
replacethe entrypoint (or default command) for a given container. The script
is copied from the Minipresto library to the container, executed, and then
removed from the container. Containers are restarted after each bootstrap script
execution,so the bootstrap scripts themselves should not restart the
container's service.If a bootstrap script has already executed in a containerandthe volume
associated with the container still exists, Minipresto will not re-execute the
bootstrap scriptunless the contents of the script have changed. The is useful
after runningminipresto down --keep(which does not remove unnamed container
volumes), so that the subsequentprovisioncommand will not re-execute the
same bootstrap script(s).If a bootstrap script is updated, it is recommended to destroy the associated
container(s) viaminipresto downand then to re-provision.To add a bootstrap script, add aresources/bootstrap/directory in any given
module, create a shell script, and then reference the script name in the Compose
YAML file:version:"3.7"services:presto:environment:MINIPRESTO_BOOTSTRAP:"bootstrap.sh"Theelasticsearchmodule is a good example of this.Managing Presto'sconfig.propertiesFileMany modules can change the Prestoconfig.propertiesandjvm.configfiles.
Because of this, there are two supported ways to modify these files with
Minipresto.The first way is by setting theCONFIGvariable in yourminipresto.cfgfile.
This will propagate the config to the Presto container when it is provisioned.Generally speaking, this can be used for any type of configuration (i.e. memory
configuration) that is unlikely to be modified by any module. This also applies
to thejvm.configfile, which has identical support via theJVM_CONFIGvariable. If there are duplicate configs in either file, Minipresto will warn
the user.To set these configs, your configuration file should look like:[PRESTO]
CONFIG=
    query.max-memory-per-node=500MB
    query.max-total-memory-per-node=500MB
JVM_CONFIG=
    -Dsun.security.krb5.debug=trueThe second way to modify core Presto configuration is via module bootstrap
scripts. This method is utilized by modules that need to make module-specific
changes to Presto files. An example bootstrap snippet can be found below:#!/usr/bin/env bashset-euxopipefailecho"Adding Presto configs..."cat<<EOT >> /usr/lib/presto/etc/config.propertiesquery.max-stage-count=105query.max-execution-time=1hEOTTroubleshootingIf you experience issues executing a Minipresto command, re-run it with the-voption for verbose output. This will often reveal the issueIf you experience an issue with a particular Docker container, consider
running these commands:docker logs <container>: Print the logs for a given container to the
terminaldocker ps: Show all running Docker containers and associated statisticsIf you experience issues with a library module, check that that module is
structured correctly according to themodule
tutorialIf none of these troubleshooting tips help to resolve your issue,please file a
GitHub issueand provide as much information
as possible.Reporting Bugs and ContributingTo report bugs, please file a GitHub issue on theMinipresto
repository. Bug reports should:Contain any relevant log messages (if the bug is tied to a command, running
with the-vflag will make debugging easier)Describe what the expected outcome isDescribe the proposed code fix (optional)Contributors have two options:Fork the repository, then make a PR to merge your changesIf you have been added as a contributor, you can go with the method above or
you can create a feature branch, then submit a PR for that feature branch
when it is ready to be merged.In either case, please provide a comprehensive description of your changes with
the PR.