Package: bigquery-views-manager
BigQuery Views ManagerUtility project to maintain BigQuery views. The main interface is the CLI.Main features:Synchronize BigQuery Views between GCP and local file systemMaterialize Views (by running a view and saving it to a table):BigQuery Materialized Viewsare now available as Pre-GAPre-requisitesPython 3Google Cloud SDKforgcloudInstallpipinstallbigquery-views-managerConfigurationViews SQL filesSQL code of the view queries is assumed to be in files with the.sqlfiles. By default they will be in theviewsdirectory.The files can contain placeholders (surrounded by curly brackets, e.g.{placeholder}) for the following variables:namedescriptionprojectThe GCP projectdatasetThe BigQuery datasetUsing the placeholders allows you to deploy the views to for example test, staging and production separately.Example:SELECT*FROM`{project}.{dataset}.view1`View List Configviews.ymlTheviews.ymlfile contains the list of views that should be processed. It is important that the list of views are in the correct insert order. i.e. ifv_view2depends onv_view1thenv_view1should appear first.The format is a yaml file. In the simplest case it will be the list of the views, e.g.:-v_view1-v_view2Additional parameters can be added, e.g. to materializev_view1:-v_view1:materialize:true-v_view2Or to materializev_view1to another table name:-v_view1:materialize:truematerialize_as:output_table1-v_view2The dataset could also be specified:-v_view1:materialize:truematerialize_as:output_dataset1.output_table1-v_view2When working with multiple datasets, this can also be conditional:-v_view1:materialize:trueconditions:-if:dataset:source_dataset1materialize_as:"output_dataset1.output_table1"-v_view2The condition will depend on the passed in--dataset.Config TablesConfig tables are tables loaded from CSV. They are meant to assist views with configuration data, rather than loading large data. Config tables are generally used by views to avoid having to hard-code certain values in the views.It is assumed that the filename is target table name with the.csvfile extension. By default in the./config-tables/tablesdirectory. A BigQuery table schema can be specified via a file with the_schema.jsonin the./config-tables/schemadirectory.Another directory can be specified via the--config-tables-base-dirCLI argument.Example DataSeeexample-data.BigQuery Views Manager CLITo get the command help:python-mbigquery_views_manager--helpOr:python-mbigquery_views_manager<sub-command>--helpCreate or Replace Viewspython-mbigquery_views_manager\create-or-replace-views\--dataset=my_dataset\[--view-list-config=/path/to/views.yml]\[<viewname>[<otherviewname>...]]Adding the--materializeflag will additionally materialize the views (where it has been enabled). In that case views will be materialized immediately after updating a view.Materialize Viewspython-mbigquery_views_manager\materialize-views\--dataset=my_dataset\[--view-list-config=/path/to/views.yml]\[<viewname>[<otherviewname>...]]Diff ViewsShow differences between local views and views within BigQuery.python-mbigquery_views_manager\diff-views\--dataset=my_dataset\[--view-list-config=/path/to/views.yml]\[<viewname>[<otherviewname>...]]Get ViewsCopy views from BigQuery to the local file system.To get all of the files listed inviews/views.yml:python-mbigquery_views_manager\get-views\--dataset=my_dataset\[--view-list-config=/path/to/views.yml]To get a particular view or views:python-mbigquery_views_manager\get-views\--dataset=my_dataset\[--view-list-config=/path/to/views.yml]\<viewname>[<otherviewname>...]When views are retrieved, the project name and dataset are replaced with placeholders.Update Config TablesCopy config tables (CSV) to BigQuery. The config tables are by default stored in./config-tables.python-mbigquery_views_manager\create-or-replace-config-tables\--dataset=my_dataset\[--config-tables-base-dir=/path/to/config-tables]\[<tablename>...]Adding a ViewAdd the view to theviewsdirectory with the view name and.sqlfile extension.The view name also needs to be added toviews/views.ymlin the correct order (i.e. if a view depends on another view, the other view should appear first).Cleanup Sub CommandsThe CLI also supports additional sub commands to delete views etc. Those are in particular use-ful in a CI environment.The following commands are supported:delete-config-tablesdelete-viewsdelete-materialized-tablesDocker ImageThere is also a Docker Image that can be used directly:dockerpullelifesciences/bigquery-views-manager:latest
dockerrun--rm\--volume~/.config/gcloud:/root/.config/gcloud\--volume$$PWD/views:/data/views\--envGOOGLE_CLOUD_PROJECT=my-gcp-project\elifesciences/bigquery-views-manager:latest\diff-views\--datasetmy_datasetRelated ProjectsBigQuery-DatasetManagerBigQuery View Analyzer