Project: StarRocks (App Definition and Development / Database)
File: authenticate_to_gcs.md

displayed_sidebar: "English" sidebar_label: Google Cloud Storage (GCS) description: "This topic describes how to authenticate with GCS" From v3.0 onwards, StarRocks supports using one of the following authentication methods to access Google Cloud Storage (GCS): VM-based authentication Use the credential attached to Google Cloud Compute Engine to authenticate GCS. Service account-based authentication Use a service account to authenticate GCS. Impersonation-based authentication Make a service account or virtual machine (VM) instance impersonate another service account. StarRocks can authenticate to GCS in the following scenarios: Batch load data from GCS. Back up data from and restore data to GCS. Query Parquet and ORC files in GCS. Query , , , and tables in GCS. In this topic, , , and are used as examples to show how StarRocks integrates with GCS in different scenarios. For information about `StorageCredentialParams` in the examples, see the "" section of this topic. NOTE StarRocks supports loading data or directly querying files from GCS only according to the gs protocol. Therefore, when you load data or query files from GCS, you must include `gs` as a prefix in the file path. Use the statement to create a Hive catalog named `hivecataloggcs` as follows, in order to query files from GCS: ```SQL CREATE EXTERNAL CATALOG hivecataloggcs PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", StorageCredentialParams ); ``` Use the statement to create a file external table named `externaltablegcs` as follows, in order to query a data file named `testfileexternal_tbl` from GCS without any metastore: ```SQL CREATE EXTERNAL TABLE externaltablegcs ( id varchar(65500), attributes map<varchar(100), varchar(2000)> ) ENGINE=FILE PROPERTIES ( "path" = "gs:////test-gcs/testfileexternal_tbl", "format" = "ORC", StorageCredentialParams ); ``` Use the statement to create a Broker Load job whose label is `testdb.label000`, in order to batch load data from GCS into the StarRocks table `targettable`: ```SQL LOAD LABEL test_db.label000 ( DATA INFILE("gs://bucketgcs/testbrokerload_ingestion/*") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( StorageCredentialParams ); ``` `StorageCredentialParams` represents a parameter set that describes how to authenticate to GCS with different authentication methods. If your StarRocks cluster is deployed on a VM instance hosted on Google Cloud Platform (GCP) and you want to use that VM instance to authenticate GCS, configure `StorageCredentialParams` as follows: ```Plain "gcp.gcs.usecomputeengineserviceaccount" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Default value | Value example | Description | | | -- | | | | gcp.gcs.usecomputeengineserviceaccount | false | true | Specifies whether to directly use the service account that is bound to your Compute

| If you directly use a service account to authenticate GCS, configure `StorageCredentialParams` as follows: ```Plain "gcp.gcs.serviceaccountemail" = "<googleserviceaccount_email>", "gcp.gcs.serviceaccountprivatekeyid" = "<googleserviceprivatekeyid>", "gcp.gcs.serviceaccountprivatekey" = "<googleserviceprivatekey>" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Default value | Value example | Description | | -- | -- | -- | | | gcp.gcs.serviceaccountemail | "" | "`user@hello.iam.gserviceaccount.com`" | The email address in the JSON file generated at the creation of the service account. | | gcp.gcs.serviceaccountprivatekeyid | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the service account. | | gcp.gcs.serviceaccountprivate_key | "" | "--BEGIN PRIVATE KEY-xxxx--END PRIVATE KEY--\n" | The private key in the JSON file generated at the creation of the service account. | If your StarRocks cluster is deployed on a VM instance hosted on GCP and you want to make that VM instance impersonate a service account, so as to make StarRocks inherit the privileges from the service account to access GCS, configure `StorageCredentialParams` as follows: ```Plain "gcp.gcs.usecomputeengineserviceaccount" = "true", "gcp.gcs.impersonationserviceaccount" = "<assumedgoogleserviceaccountemail>" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Default value | Value example | Description | | | -- | | | | gcp.gcs.usecomputeengineserviceaccount | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | | gcp.gcs.impersonationserviceaccount | "" | "hello" | The service account that you want to impersonate. | If you want to make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account) and make StarRocks inherit the privileges from the data service account to access GCS, configure `StorageCredentialParams` as follows: ```Plain "gcp.gcs.serviceaccountemail" = "<googleserviceaccount_email>", "gcp.gcs.serviceaccountprivatekeyid" = "<metagoogleserviceaccountemail>", "gcp.gcs.serviceaccountprivatekey" = "<metagoogleserviceaccount_email>", "gcp.gcs.impersonationserviceaccount" = "<datagoogleserviceaccountemail>" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Default value | Value example | Description | | -- | -- | -- | | | gcp.gcs.serviceaccountemail | "" | "`user@hello.iam.gserviceaccount.com`" | The email address in the JSON file generated at the creation of the meta service account. | | gcp.gcs.serviceaccountprivatekeyid | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the meta service account. | | gcp.gcs.serviceaccountprivate_key | "" | "--BEGIN PRIVATE KEY-xxxx--END PRIVATE KEY--\n" | The private key in the JSON file generated at the creation of the meta service account. | | gcp.gcs.impersonationserviceaccount | "" | "hello" | The data service account that you want to impersonate. |