Project: ClickHouse (App Definition and Development / Database)
File: gcs.md

slug: /en/sql-reference/table-functions/gcs sidebar_position: 70 sidebar_label: gcs keywords: [gcs, bucket] Provides a table-like interface to `SELECT` and `INSERT` data from . Requires the . This is an alias of the . If you have multiple replicas in your cluster, you can use the (which works with GCS) instead to parallelize inserts. Syntax ``` sql gcs(url [, NOSIGN | hmackey, hmacsecret] [,format] [,structure] [,compression_method]) gcs(named_collection[, option=value [,..]]) ``` :::tip GCS The GCS Table Function integrates with Google Cloud Storage by using the GCS XML API and HMAC keys. See the for more details about the endpoint and HMAC. ::: Parameters `url` Bucket path to file. Supports following wildcards in readonly mode: ``, `*`, `?`, `{abc,def}` and `{N..M}` where `N`, `M` numbers, `'abc'`, `'def'` strings. :::note GCS The GCS path is in this format as the endpoint for the Google XML API is different than the JSON API: ``` https://storage.googleapis.com/<bucket>/<folder>/<filename(s)> ``` and not https://storage.cloud.google.com. ::: `NOSIGN` If this keyword is provided in place of credentials, all the requests will not be signed. `hmackey` and `hmacsecret` Keys that specify credentials to use with given endpoint. Optional. `format` The of the file. `structure` Structure of the table. Format `'column1name column1type, column2name column2type, ...'`. `compression_method` Parameter is optional. Supported values: `none`, `gzip/gz`, `brotli/br`, `xz/LZMA`, `zstd/zst`. By default, it will autodetect compression method by file extension. Arguments can also be passed using . In this case `url`, `format`, `structure`, `compression_method` work in the same way, and some extra parameters are supported: `accesskeyid` `hmac_key`, optional. `secretaccesskey` `hmac_secret`, optional. `filename` appended to the url if specified. `useenvironmentcredentials` enabled by default, allows passing extra parameters using environment variables `AWSCONTAINERCREDENTIALSRELATIVEURI`, `AWSCONTAINERCREDENTIALSFULLURI`, `AWSCONTAINERAUTHORIZATIONTOKEN`, `AWSEC2METADATADISABLED`. `nosignrequest` disabled by default. `expirationwindowseconds` default value is 120. Returned value A table with the specified structure for reading or writing data in the specified file. Examples Selecting the first two rows from the table from GCS file `https://storage.googleapis.com/my-test-bucket-768/data.csv`: ``` sql SELECT * FROM gcs('https://storage.googleapis.com/my-test-bucket-768/data.csv', 'CSV', 'column1 UInt32, column2 UInt32, column3 UInt32') LIMIT 2; ``` ``` text column1column2column3 1 2 3 3 2 1 ``` The similar but from file with `gzip` compression method: ``` sql SELECT * FROM gcs('https://storage.googleapis.com/my-test-bucket-768/data.csv.gz', 'CSV', 'column1 UInt32, column2 UInt32, column3 UInt32', 'gzip') LIMIT 2; ``` ``` text column1column2column3 1 2 3 3 2 1 ``` Suppose that we have several files with following URIs on GCS: 'https://storage.googleapis.com/my-test-bucket-768/someprefix/somefile_1.csv' 'https://storage.googleapis.com/my-test-bucket-768/someprefix/somefile_2.csv' 'https://storage.googleapis.com/my-test-bucket-768/someprefix/somefile_3.csv' 'https://storage.googleapis.com/my-test-bucket-768/someprefix/somefile_4.csv' 'https://storage.googleapis.com/my-test-bucket-768/anotherprefix/somefile_1.csv' 'https://storage.googleapis.com/my-test-bucket-768/anotherprefix/somefile_2.csv' 'https://storage.googleapis.com/my-test-bucket-768/anotherprefix/somefile_3.csv'

Count the amount of rows in files ending with numbers from 1 to 3: ``` sql SELECT count(*) FROM gcs('https://storage.googleapis.com/my-test-bucket-768/{some,another}prefix/somefile_{1..3}.csv', 'CSV', 'name String, value UInt32') ``` ``` text count() 18 ``` Count the total amount of rows in all files in these two directories: ``` sql SELECT count(*) FROM gcs('https://storage.googleapis.com/my-test-bucket-768/{some,another}_prefix/*', 'CSV', 'name String, value UInt32') ``` ``` text count() 24 ``` :::warning If your listing of files contains number ranges with leading zeros, use the construction with braces for each digit separately or use `?`. ::: Count the total amount of rows in files named `file-000.csv`, `file-001.csv`, , `file-999.csv`: ``` sql SELECT count(*) FROM gcs('https://storage.googleapis.com/my-test-bucket-768/big_prefix/file-{000..999}.csv', 'CSV', 'name String, value UInt32'); ``` ``` text count() 12 ``` Insert data into file `test-data.csv.gz`: ``` sql INSERT INTO FUNCTION gcs('https://storage.googleapis.com/my-test-bucket-768/test-data.csv.gz', 'CSV', 'name String, value UInt32', 'gzip') VALUES ('test-data', 1), ('test-data-2', 2); ``` Insert data into file `test-data.csv.gz` from existing table: ``` sql INSERT INTO FUNCTION gcs('https://storage.googleapis.com/my-test-bucket-768/test-data.csv.gz', 'CSV', 'name String, value UInt32', 'gzip') SELECT name, value FROM existing_table; ``` Glob can be used for recursive directory traversal. Consider the below example, it will fetch all files from `my-test-bucket-768` directory recursively: ``` sql SELECT FROM gcs('https://storage.googleapis.com/my-test-bucket-768/*', 'CSV', 'name String, value UInt32', 'gzip'); ``` The below get data from all `test-data.csv.gz` files from any folder inside `my-test-bucket` directory recursively: ``` sql SELECT FROM gcs('https://storage.googleapis.com/my-test-bucket-768/*/test-data.csv.gz', 'CSV', 'name String, value UInt32', 'gzip'); ``` For production use cases it is recommended to use . Here is the example: ``` sql CREATE NAMED COLLECTION creds AS accesskeyid = '*', secretaccesskey = '*'; SELECT count(*) FROM gcs(creds, url='https://s3-object-url.csv') ``` If you specify `PARTITION BY` expression when inserting data into `GCS` table, a separate file is created for each partition value. Splitting the data into separate files helps to improve reading operations efficiency. Examples Using partition ID in a key creates separate files: ```sql INSERT INTO TABLE FUNCTION gcs('http://bucket.amazonaws.com/mybucket/file{partitionid}.csv', 'CSV', 'a String, b UInt32, c UInt32') PARTITION BY a VALUES ('x', 2, 3), ('x', 4, 5), ('y', 11, 12), ('y', 13, 14), ('z', 21, 22), ('z', 23, 24); ``` As a result, the data is written into three files: `filex.csv`, `filey.csv`, and `file_z.csv`. Using partition ID in a bucket name creates files in different buckets: ```sql INSERT INTO TABLE FUNCTION gcs('http://bucket.amazonaws.com/mybucket{partitionid}/file.csv', 'CSV', 'a UInt32, b UInt32, c UInt32') PARTITION BY a VALUES (1, 2, 3), (1, 4, 5), (10, 11, 12), (10, 13, 14), (20, 21, 22), (20, 23, 24); ``` As a result, the data is written into three files in different buckets: `mybucket1/file.csv`, `mybucket10/file.csv`, and `mybucket20/file.csv`. See Also