Project: ClickHouse (App Definition and Development / Database)
File: named-collections.md

slug: /en/operations/named-collections sidebar_position: 69 sidebar_label: "Named collections" title: "Named collections" Named collections provide a way to store collections of key-value pairs to be used to configure integrations with external sources. You can use named collections with dictionaries, tables, table functions, and object storage. Named collections can be configured with DDL or in configuration files and are applied when ClickHouse starts. They simplify the creation of objects and the hiding of credentials from users without administrative access. The keys in a named collection must match the parameter names of the corresponding function, table engine, database, etc. In the examples below the parameter list is linked to for each type. Parameters set in a named collection can be overridden in SQL, this is shown in the examples below. This ability can be limited using `[NOT] OVERRIDABLE` keywords and XML attributes and/or the configuration option `allownamedcollectionoverrideby_default`. :::warning If override is allowed, it may be possible for users without administrative access to figure out the credentials that you are trying to hide. If you are using named collections with that purpose, you should disable `allownamedcollectionoverrideby_default` (which is enabled by default). ::: ```sql CREATE NAMED COLLECTION name AS key_1 = 'value' OVERRIDABLE, key_2 = 'value2' NOT OVERRIDABLE, url = 'https://connection.url/' ``` In the above example: `key_1` can always be overridden. `key_2` can never be overridden. `url` can be overridden or not depending on the value of `allownamedcollectionoverrideby_default`. To manage named collections with DDL a user must have the `namedcontrolcollection` privilege. This can be assigned by adding a file to `/etc/clickhouse-server/users.d/`. The example gives the user `default` both the `accessmanagement` and `namedcollection_control` privileges: ```xml title='/etc/clickhouse-server/users.d/user_default.xml' <clickhouse> <users> <default> <passwordsha256hex>65e84be33532fb784c48129675f9eff3a682b27168c0ea744b2cf58ee02337c5</passwordsha256hex replace=true> <accessmanagement>1</accessmanagement> <!-- highlight-start --> <namedcollectioncontrol>1</namedcollectioncontrol> <!-- highlight-end --> </default> </users> </clickhouse> ``` :::tip In the above example the `passwordsha256hex` value is the hexadecimal representation of the SHA256 hash of the password. This configuration for the user `default` has the attribute `replace=true` as in the default configuration has a plain text `password` set, and it is not possible to have both plain text and sha256 hex passwords set for a user. ::: ```xml title='/etc/clickhouse-server/config.d/named_collections.xml' <clickhouse> <named_collections> <name> <key1 overridable="true">value</key1> <key2 overridable="false">value2</key_2> <url>https://connection.url/</url> </name> </named_collections> </clickhouse> ``` In the above example: `key_1` can always be overridden. `key_2` can never be overridden. `url` can be overridden or not depending on the value of `allownamedcollectionoverrideby_default`. Named collections that are created with DDL queries can be altered or dropped with DDL. Named collections created with XML files can be managed by editing or deleting the corresponding XML. Change or add the keys `key1` and `key3` of the collection `collection2` (this will not change the value of the `overridable` flag for those keys): ```sql ALTER NAMED COLLECTION collection2 SET key1=4, key3='value3' ``` Change or add the key `key1` and allow it to be always overridden: ```sql ALTER NAMED COLLECTION collection2 SET key1=4 OVERRIDABLE ``` Remove the key `key2` from `collection2`: ```sql ALTER NAMED COLLECTION collection2 DELETE key2 ``` Change or add the key `key1` and delete the key `key3` of the collection `collection2`: ```sql ALTER NAMED COLLECTION collection2 SET key1=4, DELETE key3 ``` To force a key to use the default settings for the `overridable` flag, you have to remove and re-add the key. ```sql ALTER NAMED COLLECTION collection2 DELETE key1; ALTER NAMED COLLECTION collection2 SET key1=4; ``` ```sql DROP NAMED COLLECTION collection2 ``` The description of parameters see

```sql CREATE NAMED COLLECTION s3_mydata AS accesskeyid = 'AKIAIOSFODNN7EXAMPLE', secretaccesskey = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY', format = 'CSV', url = 'https://s3.us-east-1.amazonaws.com/yourbucket/mydata/' ``` ```xml <clickhouse> <named_collections> <s3_mydata> <accesskeyid>AKIAIOSFODNN7EXAMPLE</accesskeyid> <secretaccesskey>wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY</secretaccesskey> <format>CSV</format> <url>https://s3.us-east-1.amazonaws.com/yourbucket/mydata/</url> </s3_mydata> </named_collections> </clickhouse> ``` Both of the following examples use the same named collection `s3_mydata`: ```sql INSERT INTO FUNCTION s3(s3mydata, filename = 'testfile.tsv.gz', format = 'TSV', structure = 'number UInt64', compression_method = 'gzip') SELECT * FROM numbers(10000); ``` :::tip The first argument to the `s3()` function above is the name of the collection, `s3_mydata`. Without named collections, the access key ID, secret, format, and URL would all be passed in every call to the `s3()` function. ::: ```sql CREATE TABLE s3enginetable (number Int64) ENGINE=S3(s3mydata, url='https://s3.us-east-1.amazonaws.com/yourbucket/mydata/testfile.tsv.gz', format = 'TSV') SETTINGS inputformatwithnamesuse_header = 0; SELECT * FROM s3enginetable LIMIT 3; number 0 1 2 ``` The description of parameters see . ```sql CREATE NAMED COLLECTION mymysql AS user = 'myuser', password = 'mypass', host = '127.0.0.1', port = 3306, database = 'test', connectionpoolsize = 8, replace_query = 1 ``` ```xml <clickhouse> <named_collections> <mymysql> <user>myuser</user> <password>mypass</password> <host>127.0.0.1</host> <port>3306</port> <database>test</database> <connectionpoolsize>8</connectionpoolsize> <replacequery>1</replacequery> </mymysql> </named_collections> </clickhouse> ``` The four following examples use the same named collection `mymysql`: ```sql SELECT count() FROM mysql(mymysql, table = 'test'); count() 3 ``` :::note The named collection does not specify the `table` parameter, so it is specified in the function call as `table = 'test'`. ::: ```sql CREATE TABLE mytable(A Int64) ENGINE = MySQL(mymysql, table = 'test', connectionpoolsize=3, replace_query=0); SELECT count() FROM mytable; count() 3 ``` :::note The DDL overrides the named collection setting for connectionpoolsize. ::: ```sql CREATE DATABASE mydatabase ENGINE = MySQL(mymysql); SHOW TABLES FROM mydatabase; name source test ``` ```sql CREATE DICTIONARY dict (A Int64, B String) PRIMARY KEY A SOURCE(MYSQL(NAME mymysql TABLE 'source')) LIFETIME(MIN 1 MAX 2) LAYOUT(HASHED()); SELECT dictGet('dict', 'B', 2); dictGet('dict', 'B', 2) two ``` The description of parameters see . ```sql CREATE NAMED COLLECTION mypg AS user = 'pguser', password = 'jw8s0F4', host = '127.0.0.1', port = 5432, database = 'test', schema = 'test_schema', ``` Example of configuration: ```xml <clickhouse> <named_collections> <mypg> <user>pguser</user> <password>jw8s0F4</password> <host>127.0.0.1</host> <port>5432</port> <database>test</database> <schema>test_schema</schema> </mypg> </named_collections> </clickhouse> ``` ```sql SELECT * FROM postgresql(mypg, table = 'test'); ab 2 two 1 one SELECT * FROM postgresql(mypg, table = 'test', schema = 'public'); a 1 2 3 ``` ```sql CREATE TABLE mypgtable (a Int64) ENGINE = PostgreSQL(mypg, table = 'test', schema = 'public'); SELECT * FROM mypgtable; a 1 2 3 ``` ```sql CREATE DATABASE mydatabase ENGINE = PostgreSQL(mypg); SHOW TABLES FROM mydatabase name test ``` ```sql CREATE DICTIONARY dict (a Int64, b String) PRIMARY KEY a SOURCE(POSTGRESQL(NAME mypg TABLE test)) LIFETIME(MIN 1 MAX 2) LAYOUT(HASHED()); SELECT dictGet('dict', 'b', 2); dictGet('dict', 'b', 2) two ``` The description of parameters see . Example of configuration: ```sql CREATE NAMED COLLECTION remote1 AS host = 'remote_host', port = 9000, database = 'system', user = 'foo', password = 'secret', secure = 1 ``` ```xml <clickhouse> <named_collections> <remote1> <host>remote_host</host> <port>9000</port> <database>system</database> <user>foo</user> <password>secret</password> <secure>1</secure> </remote1> </named_collections> </clickhouse> ``` `secure` is not needed for connection because of `remoteSecure`, but it can be used for dictionaries. ```sql SELECT * FROM remote(remote1, table = one); dummy 0 SELECT * FROM remote(remote1, database = merge(system, '^one')); dummy 0 INSERT INTO FUNCTION remote(remote1, database = default, table = test) VALUES (1,'a'); SELECT * FROM remote(remote1, database = default, table = test); ab 1 a ``` ```sql CREATE DICTIONARY dict(a Int64, b String) PRIMARY KEY a SOURCE(CLICKHOUSE(NAME remote1 TABLE test DB default)) LIFETIME(MIN 1 MAX 2) LAYOUT(HASHED()); SELECT dictGet('dict', 'b', 1); dictGet('dict', 'b', 1) a ```