Project: ClickHouse (App Definition and Development / Database)
File: rabbitmq.md

slug: /en/engines/table-engines/integrations/rabbitmq sidebar_position: 170 sidebar_label: RabbitMQ This engine allows integrating ClickHouse with . `RabbitMQ` lets you: Publish or subscribe to data flows. Process streams as they become available. ``` sql CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster] ( name1 [type1], name2 [type2], ... ) ENGINE = RabbitMQ SETTINGS rabbitmqhostport = 'host:port' [or rabbitmq_address = 'amqp(s)://guest:guest@localhost/vhost'], rabbitmqexchangename = 'exchange_name', rabbitmqformat = 'dataformat'[,] [rabbitmqexchangetype = 'exchange_type',] [rabbitmqroutingkey_list = 'key1,key2,...',] [rabbitmq_secure = 0,] [rabbitmq_schema = '',] [rabbitmqnumconsumers = N,] [rabbitmqnumqueues = N,] [rabbitmqqueuebase = 'queue',] [rabbitmqdeadletterexchange = 'dl-exchange',] [rabbitmq_persistent = 0,] [rabbitmqskipbroken_messages = N,] [rabbitmqmaxblock_size = N,] [rabbitmqflushinterval_ms = N,] [rabbitmqqueuesettings_list = 'x-dead-letter-exchange=my-dlx,x-max-length=10,x-overflow=reject-publish',] [rabbitmqqueueconsume = false,] [rabbitmq_address = '',] [rabbitmq_vhost = '/',] [rabbitmq_username = '',] [rabbitmq_password = '',] [rabbitmqcommiton_select = false,] [rabbitmqmaxrowspermessage = 1,] [rabbitmqhandleerror_mode = 'default'] ``` Required parameters: `rabbitmqhostport` host:port (for example, `localhost:5672`). `rabbitmqexchangename` RabbitMQ exchange name. `rabbitmq_format` Message format. Uses the same notation as the SQL `FORMAT` function, such as `JSONEachRow`. For more information, see the section. Optional parameters: `rabbitmqexchangetype` The type of RabbitMQ exchange: `direct`, `fanout`, `topic`, `headers`, `consistent_hash`. Default: `fanout`. `rabbitmqroutingkey_list` A comma-separated list of routing keys. `rabbitmq_schema` Parameter that must be used if the format requires a schema definition. For example, requires the path to the schema file and the name of the root `schema.capnp:Message` object. `rabbitmqnumconsumers` The number of consumers per table. Specify more consumers if the throughput of one consumer is insufficient. Default: `1` `rabbitmqnumqueues` Total number of queues. Increasing this number can significantly improve performance. Default: `1`. `rabbitmqqueuebase` - Specify a hint for queue names. Use cases of this setting are described below. `rabbitmqdeadletterexchange` - Specify name for a . You can create another table with this exchange name and collect messages in cases when they are republished to dead letter exchange. By default dead letter exchange is not specified. `rabbitmq_persistent` - If set to 1 (true), in insert query delivery mode will be set to 2 (marks messages as 'persistent'). Default: `0`. `rabbitmqskipbrokenmessages` RabbitMQ message parser tolerance to schema-incompatible messages per block. If `rabbitmqskipbrokenmessages = N` then the engine skips N RabbitMQ messages that cannot be parsed (a message equals a row of data). Default: `0`. `rabbitmqmaxblock_size` - Number of row collected before flushing data from RabbitMQ. Default: . `rabbitmqflushinterval_ms` - Timeout for flushing data from RabbitMQ. Default: . `rabbitmqqueuesettings_list` - allows to set RabbitMQ settings when creating a queue. Available settings: `x-max-length`, `x-max-length-bytes`, `x-message-ttl`, `x-expires`, `x-priority`, `x-max-priority`, `x-overflow`, `x-dead-letter-exchange`, `x-queue-type`. The `durable` setting is enabled automatically for the queue. `rabbitmqaddress` - Address for connection. Use ether this setting or `rabbitmqhost_port`. `rabbitmq_vhost` - RabbitMQ vhost. Default: `'\'`. `rabbitmqqueueconsume` - Use user-defined queues and do not make any RabbitMQ setup: declaring exchanges, queues, bindings. Default: `false`. `rabbitmq_username` - RabbitMQ username. `rabbitmq_password` - RabbitMQ password. `rejectunhandledmessages` - Reject messages (send RabbitMQ negative acknowledgement) in case of errors. This setting is automatically enabled if there is a `x-dead-letter-exchange` defined in `rabbitmqqueuesettings_list`. `rabbitmqcommiton_select` - Commit messages when select query is made. Default: `false`. `rabbitmqmaxrowspermessage` The maximum number of rows written in one RabbitMQ message for row-based formats. Default :

`rabbitmqemptyqueuebackoffstart` A start backoff point to reschedule read if the rabbitmq queue is empty. `rabbitmqemptyqueuebackoffend` An end backoff point to reschedule read if the rabbitmq queue is empty. `rabbitmqhandleerrormode` How to handle errors for RabbitMQ engine. Possible values: default (the exception will be thrown if we fail to parse a message), stream (the exception message and raw message will be saved in virtual columns `error` and `rawmessage`). [ ] SSL connection: Use either `rabbitmqsecure = 1` or `amqps` in connection address: `rabbitmqaddress = 'amqps://guest:guest@localhost/vhost'`. The default behaviour of the used library is not to check if the created TLS connection is sufficiently secure. Whether the certificate is expired, self-signed, missing or invalid: the connection is simply permitted. More strict checking of certificates can possibly be implemented in the future. Also format settings can be added along with rabbitmq-related settings. Example: ``` sql CREATE TABLE queue ( key UInt64, value UInt64, date DateTime ) ENGINE = RabbitMQ SETTINGS rabbitmqhostport = 'localhost:5672', rabbitmqexchangename = 'exchange1', rabbitmq_format = 'JSONEachRow', rabbitmqnumconsumers = 5, datetimeinputformat = 'besteffort'; ``` The RabbitMQ server configuration should be added using the ClickHouse config file. Required configuration: ``` xml <rabbitmq> <username>root</username> <password>clickhouse</password> </rabbitmq> ``` Additional configuration: ``` xml <rabbitmq> <vhost>clickhouse</vhost> </rabbitmq> ``` `SELECT` is not particularly useful for reading messages (except for debugging), because each message can be read only once. It is more practical to create real-time threads using . To do this: Use the engine to create a RabbitMQ consumer and consider it a data stream. Create a table with the desired structure. Create a materialized view that converts data from the engine and puts it into a previously created table. When the `MATERIALIZED VIEW` joins the engine, it starts collecting data in the background. This allows you to continually receive messages from RabbitMQ and convert them to the required format using `SELECT`. One RabbitMQ table can have as many materialized views as you like. Data can be channeled based on `rabbitmqexchangetype` and the specified `rabbitmqroutingkey_list`. There can be no more than one exchange per table. One exchange can be shared between multiple tables - it enables routing into multiple tables at the same time. Exchange type options: `direct` - Routing is based on the exact matching of keys. Example table key list: `key1,key2,key3,key4,key5`, message key can equal any of them. `fanout` - Routing to all tables (where exchange name is the same) regardless of the keys. `topic` - Routing is based on patterns with dot-separated keys. Examples: `.logs`, `records...2020`, `.2018,.2019,.2020`. `headers` - Routing is based on `key=value` matches with a setting `x-match=all` or `x-match=any`. Example table key list: `x-match=all,format=logs,type=report,year=2020`. `consistenthash` - Data is evenly distributed between all bound tables (where the exchange name is the same). Note that this exchange type must be enabled with RabbitMQ plugin: `rabbitmq-plugins enable rabbitmqconsistenthashexchange`. Setting `rabbitmqqueuebase` may be used for the following cases: to let different tables share queues, so that multiple consumers could be registered for the same queues, which makes a better

If using `rabbitmqnumconsumers` and/or `rabbitmqnumqueues` settings, the exact match of queues is achieved in case these parameters are the same. to be able to restore reading from certain durable queues when not all messages were successfully consumed. To resume consumption from one specific queue - set its name in `rabbitmqqueuebase` setting and do not specify `rabbitmqnumconsumers` and `rabbitmqnumqueues` (defaults to 1). To resume consumption from all queues, which were declared for a specific table - just specify the same settings: `rabbitmqqueuebase`, `rabbitmqnumconsumers`, `rabbitmqnumqueues`. By default, queue names will be unique to tables. to reuse queues as they are declared durable and not auto-deleted. (Can be deleted via any of RabbitMQ CLI tools.) To improve performance, received messages are grouped into blocks the size of . If the block wasnt formed within milliseconds, the data will be flushed to the table regardless of the completeness of the block. If `rabbitmqnumconsumers` and/or `rabbitmqnumqueues` settings are specified along with `rabbitmqexchangetype`, then: `rabbitmq-consistent-hash-exchange` plugin must be enabled. `message_id` property of the published messages must be specified (unique for each message/batch). For insert query there is message metadata, which is added for each published message: `messageID` and `republished` flag (true, if published more than once) - can be accessed via message headers. Do not use the same table for inserts and materialized views. Example: ``` sql CREATE TABLE queue ( key UInt64, value UInt64 ) ENGINE = RabbitMQ SETTINGS rabbitmqhostport = 'localhost:5672', rabbitmqexchangename = 'exchange1', rabbitmqexchangetype = 'headers', rabbitmqroutingkey_list = 'format=logs,type=report,year=2020', rabbitmq_format = 'JSONEachRow', rabbitmqnumconsumers = 5; CREATE TABLE daily (key UInt64, value UInt64) ENGINE = MergeTree() ORDER BY key; CREATE MATERIALIZED VIEW consumer TO daily AS SELECT key, value FROM queue; SELECT key, value FROM daily ORDER BY key; ``` `exchangename` - RabbitMQ exchange name. Data type: `String`. `channelid` - ChannelID, on which consumer, who received the message, was declared. Data type: `String`. `deliverytag` - DeliveryTag of the received message. Scoped per channel. Data type: `UInt64`. `_redelivered` - `redelivered` flag of the message. Data type: `UInt8`. `messageid` - messageID of the received message; non-empty if was set, when message was published. Data type: `String`. `_timestamp` - timestamp of the received message; non-empty if was set, when message was published. Data type: `UInt64`. Additional virtual columns when `kafkahandleerror_mode='stream'`: `rawmessage` - Raw message that couldn't be parsed successfully. Data type: `Nullable(String)`. `_error` - Exception message happened during failed parsing. Data type: `Nullable(String)`. Note: `rawmessage` and `_error` virtual columns are filled only in case of exception during parsing, they are always `NULL` when message was parsed successfully. Even though you may specify (such as `DEFAULT`, `MATERIALIZED`, `ALIAS`) in the table definition, these will be ignored. Instead, the columns will be filled with their respective default values for their types. RabbitMQ engine supports all supported in ClickHouse. The number of rows in one RabbitMQ message depends on whether the format is row-based or block-based: For row-based formats the number of rows in one RabbitMQ message can be controlled by setting `rabbitmqmaxrowspermessage`. For block-based formats we cannot divide block into smaller parts, but the number of rows in one block can be controlled by general setting .