Package: preset-cli
A CLI to interact with Preset workspaces.This tool is a command line interface (CLI) to interact with your Preset workspaces. Currently it can be used to sync resources (databases, datasets, charts, dashboards) from source control, either in native format or from adbtproject. It can also be used to run SQL against any database in any workspace. In the future, the CLI will also allow you to manage your workspaces and users.InstallationInstall the CLI withpip:$pipinstall-Usetuptoolssetuptools_scmwheel# for good measure$pipinstallpreset-cliMake sure you’re using Python 3.8 or newer.You can also install frommain, if you need recent features:$pipinstall"git+https://github.com/preset-io/backend-sdk.git"AuthenticationThe CLI requires an API key for authentication, composed of token and an associated secret. Both can be defined in your environment asPRESET_API_TOKENandPRESET_API_SECRET, respectively, or can be stored in a configuration file. To store the credentials in a configuration file simply runpreset-cliauth. This should openhttps://manage.app.preset.io/app/userin your browser so you can generate a new token, and it will prompt you for the token and its secret:%preset-cliauthPleasegenerateanewtokenathttps://manage.app.preset.io/app/userifyoudon'thaveonealreadyAPItoken:35dac901-c775-43ff-8eb4-816edc061487APIsecret:[willnotecho]Credentialsstoredin/Users/beto/Library/ApplicationSupport/preset-cli/credentials.yamlThe credentials will be stored in a system-dependent location, in a file that’s readable only by you.This step is optional. If you try to run the CLI without the token/secret defined in your environment or stored in the expected location you will be prompted for them:%preset-clisupersetsqlYouneedtospecifyaJWTtokenoranAPIkey(nameandsecret)APItoken:35dac901-c775-43ff-8eb4-816edc061487APIsecret:[willnotecho]Storethecredentialsin/Users/beto/Library/ApplicationSupport/preset-cli/credentials.yaml?[y/N]You can also pass the token/secret (or even the JWT token) directly as an option:%preset-cli--api-token35dac901-c775-43ff-8eb4-816edc061487--api-secretXXXsupersetsql%preset-cli--jwt-tokenXXXsupersetsqlIn summary, there are 3 options for credentials:Stored in the environment asPRESET_API_TOKENandPRESET_API_SECRET.Stored in a user-readable file calledcredentials.yaml, with system-dependent location.Passed directly to the CLI via--api-tokenand--api-secret(or--jwt-token) options.WorkspacesThe CLI can run commands against one or more Preset workspaces (Apache Superset instances). When running a command you can specify the workspace(s) by passing a comma-separated list of URLs to the--workspacesoption:%preset-cli\>--workspaces=https://abcdef12.us1a.app.preset.io/,https://34567890.us1a.app.preset.io/\>supersetsqlIf you omit the--workspacesoption you will be prompted interactively:%preset-clisupersetsqlChooseoneormoreworkspaces(eg:1-3,5,8-):# Team 1 #✅(1)TheDataLab🚧(2)Newworkspace# Dev #⤴️(3)TestworkspaceEach workspace has an icon depicting its status:✅ ready📊 loading examples💾 creating/initializing metadata database🚧 migrating metadata database🕵️ migrating secrets⤴️ upgrading workspace❗️ error❓ unknown stateYou can specify one or more workspaces by using a comma-separated list of numbers and/or ranges:1: workspace 11,3: workspaces 1 and 31,3-5: workspaces 1, 3, 4, and 5-3: workspaces 1, 2, and 31-: all workspaces-: all workspacesCommandsThe following commands are currently available:preset-cliauth: store authentication credentials.preset-cliinvite-users: invite users to Preset.preset-cliimport-users: automatically add users to Preset.preset-clilist-group-membership: List SCIM groups from a team and their memberships.preset-clisuperset sql: run SQL interactively or programmatically against an analytical database.preset-clisupersetexport-assets(alternatively,preset-clisuperset export): export resources (databases, datasets, charts, dashboards) into a directory as YAML files.preset-clisupersetexport-ownership: export resource ownership (UUID -> email) into a YAML file.preset-clisupersetexport-rls: export RLS rules into a YAML file.preset-clisupersetexport-roles: export user roles into a YAML file.preset-clisupersetexport-users: export users (name, username, email, roles) into a YAML file.preset-clisuperset sync native(alternatively,preset-clisupersetimport-assets): synchronize the workspace from a directory of templated configuration files.preset-clisuperset syncdbt-core: synchronize the workspace from a dbt Core project.preset-clisuperset syncdbt-cloud: synchronize the workspace from a dbt Cloud project.All thesupersetsub-commands can also be executed against a standalone Superset instance, using thesuperset-clicommand. This means that if you are running an instance of Superset athttps://superset.example.org/you can export its resources with the command:%superset-clihttps://superset.example.org/export-assets/path/to/directoryAnd then import everything to a Preset workspace with:%preset-clisupersetsyncnative/path/to/directoryRunning SQLThe CLI offers an easy way to run SQL against an analytical database in a workspace. This can be done programmatically or interactively. For example, to run the querySELECTCOUNT(*)AS revenue FROM salesgiven a workspace URL and a database ID you can run:%preset-cli--workspaces=https://abcdef12.us1a.app.preset.io/supersetsql\>--database-id=1-e"SELECT COUNT(*) AS revenue FROM sales"https://abcdef12.us1a.app.preset.io/revenue---------42If you don’t specify the database ID you will be shown a list of available databases in order to choose one. If you don’t specify the SQL query via the-eoption the CLI will start a simple REPL (read-eval-print loop) where you can run queries interactively.Synchronizing from exportsYou can use the CLI to manage workspaces resources — databases, datasets, charts, and dashboards — from source control. The configuration should be stored as YAML files, using the same format the Apache Superset uses for importing and exporting resources.The easiest way to generate the configuration files is to build one or more dashboards in a Preset workspace, export them together, and unzip the generated file into a directory.After unzipping the directory should look like this:charts/dashboards/databases/datasets/metadata.yamlYou can see an examplehere.To synchronize these files to a Preset workspace you only need to run:%preset-cli--workspaces=https://abcdef12.us1a.app.preset.io/\>supersetsyncnative/path/to/directory/If any of the resources already exist you need to pass the--overwriteflag in order to replace them. The CLI will warn you of any resources that already exist if the flag is not passed:%preset-cli--workspaces=https://abcdef12.us1a.app.preset.io/\>supersetsyncnative/path/to/directory/ErrorimportingdatabaseThefollowingfile(s)alreadyexist.Pass--overwritetoreplacethem.-databases/Google_Sheets.yamlErrorimportingdatasetThefollowingfile(s)alreadyexist.Pass--overwritetoreplacethem.-datasets/Google_Sheets/country_cnt.yamlErrorimportingchartThefollowingfile(s)alreadyexist.Pass--overwritetoreplacethem.-charts/Total_count_134.yamlErrorimportingdashboardThefollowingfile(s)alreadyexist.Pass--overwritetoreplacethem.-dashboards/White_label_test.yamlSynchronized resources can be marked as “externally managed” by passing the--disallow-editsflag to the command. When the flag is passed users won’t be able to edit the resources. It’s also possible to provide a URL where the resource can be modified, eg, a link to a file in a Github repository. This can be done by passing the--external-url-prefixflag:%preset-cli--workspaces=https://abcdef12.us1a.app.preset.io/\>supersetsyncnative/path/to/directory/--disallow-edits\>--external-url-prefix=https://github.com/org/project/blob/master/This way, the filedashboards/White_label_test.yamlwould have an external URL pointing tohttps://github.com/org/project/blob/master/dashboards/White_label_test.yaml. Currently the URL is not displayed anywhere, but in the near future we should have affordances pointing users to it from the instance UI.Using templatesOne of the most powerful features of this command is that the YAML configuration files are treated asJinja2templates, allowing you to parametrize the synchronized files. For example, imagine a simple chart like this:slice_name:Total salesviz_type:big_number_totalparams:metric:sum__salesadhoc_filters:[]The chart shows the metricsum__sales, representing the total (unfiltered) sales of a given product. We can change the chart configuration to look like this instead:{%if country %}slice_name:Sales in {{ country }}{%else %}slice_name:Total sales{%endif %}viz_type:big_number_totalparams:metric:sum__sales{%if country %}adhoc_filters:-clause:WHEREexpressionType:SQLsqlExpression:country = '{{ country }}'subject:nulloperator:nullcomparator:null{%else %}adhoc_filters:[]{%endif %}Now, if thecountryparameter is set the chart will have a different title and an additional filter. Multiple parameters can be passed as optiona via the command line:%preset-cli--workspaces=https://abcdef12.us1a.app.preset.io/\>supersetsyncnative/path/to/directory/-ocountry=BRTemplates also have access to the workspace name through theinstancevariable (aURL object):params:metric:sum__salesadhoc_filters:-clause:WHEREexpressionType:SQL{%if instance.host == '//abcdef12.us1a.app.preset.io/ %}sqlExpression:warehouse_id = 1{%elif instance.host == '//34567890.us1a.app.preset.io/ %}sqlExpression:warehouse_id = 2{%else %}sqlExpression:warehouse_id = 3{%endif %}You can also load variables from the environment by passing the--load-env(or-e) flag:database_name:Postgressqlalchemy_uri:postgres://{{ env["POSTGRES_HOSTNAME"] }}Finally, as shown in the next section, templates can leverage user-defined functions.User defined functionsYou can create your own functions to be used in the configuration templates. Simply create a sub-directory calledfunctions/in the directory where the configuration files are stored, and add one or more Python files. As a simple example, imagine a file calledfunctions/demo.pywith the following content:# functions/demo.pydefhello_world()->str:return"Hello, world!"The function can then be called from any template the following way:slice_name:{{functions.demo.hello_world()}}viz_type:big_number_totalparams:...Disabling Jinja TemplatingBoth the CLI and Superset support Jinja templating. To prevent the CLI from loading Superset Jinja syntax, the export operation automatically escapes Jinja syntax from YAML files. As a consequence, this query:sql:'SELECTaction,count(*)astimesFROMlogs{%iffilter_values(''action_type'')|length%}WHEREactionisnull{%foractioninfilter_values(''action_type'')%}oraction=''{{action}}''{%endfor%}{%endif%}GROUPBYaction'Becomes this:sql:'SELECTaction,count(*)astimesFROMlogs{{'{%if'}}filter_values(''action_type'')|length {{ '%}' }}WHERE action is null{{ '{% for' }} action in filter_values(''action_type'') {{ '%}' }}or action = ''{{ '{{' }} action {{ '}}' }}''{{ '{% endfor %}' }}{{ '{% endif %}' }}GROUP BY action'When performing the import, the CLI would load any templating syntax that isn’t escaped, and remove escaping. However, this escaping syntax isn’t compatible with UI imports.
To avoid issues when running migrations using both the CLI and the UI, you can use:--disable-jinja-escapingflag with theexport-assetscommand to disable the escaping (so that exported assets can be imported via the UI)--disable-jinja-templatingflag with thesync nativecommand to disable jinja templating (so that assets exported via the UI can be imported via the CLI)Note that using these flags would remove the ability to dynamically modify the content through the CLI.Synchronizing to and from dbtThe CLI also allows you to synchronize models, and metrics from adbtproject.If you’re using dbt Core you can point the CLI to your compiled manifest and your profiles file, so that the database is automatically created, together with all the models and metrics. The full command is:%preset-cli--workspaces=https://abcdef12.us1a.app.preset.io/\>supersetsyncdbt-core/path/to/dbt/my_project/target/manifest.json\>--project=my_project--target=dev--profiles=${HOME}/.dbt/profiles.yml\>--exposures=/path/to/dbt/my_project/models/exposures.yaml\>--import-db\>--external-url-prefix=http://localhost:8080/Running this command will:Read the dbt profile and create the$targetdatabase for the specified project in the Preset workspace.Every source in the project will be created as a dataset in the Preset workspace.Every model in the project will be created as a dataset in the Preset workspace.Anymetricswill be added to the corresponding datasets.Every dashboard built on top of the dbt sources and/or models will be synchronized back to dbt as anexposure.Descriptions, labels and other metadata is also synced from dbt models to the corresponding fields in the dataset. It’s also possible to specify values for Superset-only fields directly in the model definition, undermodel.meta.superset.{{field_name}}. For example, to specify the cache timeout for a dataset:models:-name:my_dbt_modelmeta:superset:cache_timeout:250# Setting the dataset cache timeout to 250.The same is applied for metrics. For example, to specify the d3 format for a metric:-name:avg_revenuelabel:"AVGRevenue"model:ref('my_dbt_model')calculation_method:averageexpression:price_eachtimestamp:datemeta:superset:d3format:'%d'The--external-url-prefixshould point to your dbt docs, so that the resources in the workspace can point to the source of truth where they are being managed. Similar to the native sync, the dbt sync also supports the--disallow-editsflag.By default, the CLI sync would create a new database on the destination workspace using below name structure:f"{project_name}_{target_name}"If you want to sync data to an existing database connection on the workspace instead, you can specify the database connection name on the profiles YAML file. Add below structure under the<target-name>:meta:superset:database_name:my DB name# <= specify the database connection/display name used on the workspaceExample:jaffle_shop:outputs:dev:meta:superset:database_name:Postgres - ProductionIf--import-dbwas passed and a database connection was found on the workspace, the operation would update the connection configuration with the dbt connection settings.If you’re using dbt Cloud you can instead pass a job ID and aservice account access token:%preset-cli--workspaces=https://abcdef12.us1a.app.preset.io/\>supersetsyncdbt-cloud\>$TOKEN$JOB_ID\>--external-url-prefix=http://localhost:8080/The token only needs access to the “Metadata only” permission set for your project. You can see the job ID by going to the project URL in dbt Cloud and looking at the last ID in the URL. For example, if the URL ishttps://cloud.getdbt.com/#/accounts/12345/projects/567890/jobs/the job ID is 567890.When syncing from dbt Cloud, the database connection must already exist on the target workspace. The connection display name on the workspace must match the database name from dbt Cloud.Selecting modelsBy default all the models will be synchronized to the workspace. The CLI supports a subset of the syntax used by thedbtcommand line to select which models should be synchronized. Models that should be synchronized can be specified via the--selectflag:%preset-cli...--selectmy_model# sync only "my_model"%preset-cli...--selectmy_model+# sync "my_model" and its children%preset-cli...--selectmy_model+2# sync "my_model" and its children up to 2 degrees%preset-cli...--select+my_model# sync "my_model" and its parents%preset-cli...--select+my_model+# sync "my_model" with parents and childrenMultiple selectors can be passed by repeating the--selectflag (note that this is slightly different from dbt, which doesn’t require repeating the flag):%preset-cli...--selectmy_model--selectmy_other_modelThe CLI also support the intersection operator:%preset-cli...--selectmy_model+,tag:testThe command above will synchronizemy_modeland its children, as long as the models have the “test” tag.Finally, the CLI also supports the--excludeflag in a similar way:%preset-cli--selectmy_model+--excludetag:testThe command above synchronizes “my_model” and its children, as long as the models don’t have the “test” tag.Exporting resourcesThe CLI can also be used to export resources (databases, datasets, charts, and dashboards) from a given Preset workspace (usingpreset-cli) or Superset instance (usingsuperset-cli). This is useful for migrating resources between workspaces, from an existing Superset installation to Preset, or even from Preset to Superset (one of the advantages of Preset is no vendor lock in!).To export resources from a self-hosted Superset instance:%superset-clihttps://superset.example.org/export/path/to/directoryThis will create a nice directory structure in/path/to/directory, ready to be imported using thesync nativecommand.To export resources from a Preset workspace:%preset-cli--workspaces=https://abcdef12.us1a.app.preset.io/\>supersetexport/path/to/directoryIt’s also possible to use the CLI to export specific resources:Use--asset-typeto export all assets from a given type. Available options:dashboardchartdatasetdatabaseFor example, running below command would export all dashboards from this workspace (datasets, charts and DB connections wouldn’t be included):%preset-cli--workspaces=https://abcdef12.us1a.app.preset.io/\>supersetexport/path/to/directory--asset-type=dashboardUse--asset-idsto filter for specific assets. Available options:dashboard-idschart-idsdataset-idsdatabase-idsFor example, running below command would export specified dashboards from this workspace (datasets, charts and DB connections would be included):%preset-cli--workspaces=https://abcdef12.us1a.app.preset.io/\>supersetexport/path/to/directory--dashboard-ids=9,10To import the exported resources into a Preset workspace:%preset-cli--workspaces=https://abcdef12.us1a.app.preset.io/\>supersetsyncnative/path/to/directoryFinally, to import in a standalone Superset instance:%superset-clihttps://superset.example.org/syncnative/path/to/directoryNote that any existing Jinja2 template markers present will be escaped. For example, if you have a virtual dataset defined as:SELECTaction,count(*)astimesFROMlogsWHEREactionin{{filter_values('action_type')|where_in}}GROUPBYactionThe resulting YAML file will have the query defined as:SELECTaction,count(*)astimesFROMlogsWHEREactionin{{'{{'}}filter_values('action_type')|where_in}}'}}'}}GROUPBYactionSo that when processed bypreset-clisuperset sync nativethe original Jinja2 is reconstructed correctly.Exporting usersThepreset-clisupersetexport-userscommand can be used to export a list of users. These users can then be imported to Preset via thepreset-cliimport-userscommand.You can also export roles viapreset-clisupersetexport-roles, and import withimport-roles.Exporting RLS rulesThepreset-clisupersetexport-rlscommand can be used to export a list of RLS rules. Currently there’s no way to import this into a workspace, but work is in progress.Exporting ownershipThepreset-clisupersetexport-ownershipcommand generates a YAML file with information about ownership of different resources. The file maps resource UUIDs to user email address, and in the future will be used to recreate ownership on a different instance of Superset.Listing SCIM GroupsThepreset-clilist-group-membershipcommand prints all SCIM groups (including membership) associated with a Preset team. Instead of printing the results on the terminal (whcih can be useful for quick troubleshooting), it’s possible to use--save-report=yamlor--save-report=csvto write results to a file. The file name would be{TeamSlug}__user_group_membership.{FileExtension}.