Chapter 7. Configuring memory usage for addresses AMQ Broker transparently supports huge queues containing millions of messages, even if the machine that is hosting the broker is running with limited memory. In these situations, it might be not possible to store all of the queues in memory at any one time. To protect against excess memory consumption, you can configure the maximum memory usage that is allowed for each address on the broker. In addition, you can configure the broker to take one of the following actions when memory usage for an address reaches the configured limit: Page messages Silently drop messages Drop messages and notify the sending clients Block clients from sending messages If you configure the broker to page messages when the maximum memory usage for an address is reached, you can configure limits for specific addresses to: Limit the disk space used to page incoming messages Limit the memory used for paged messages that the broker transfers from disk back to memory when clients are ready to consume messages. You can also set a disk usage threshold, which overrides all the configured paging limits. If the disk usage threshold is reached, the broker stops paging and blocks all incoming messages. Important When you use transactions, the broker might allocate extra memory to ensure transactional consistency. In this case, the memory usage reported by the broker might not reflect the total number of bytes being used in memory. Therefore, if you configure the broker to page, drop, or block messages based on a specified maximum memory usage, you should not also use transactions. 7.1. Configuring message paging For any address that has a maximum memory usage limit specified, you can also specify what action the broker takes when that usage limit is reached. One of the options that you can configure is paging . If you configure the paging option, when the maximum size of an address is reached, the broker starts to store messages for that address on disk, in files known as page files . Each page file has a maximum size that you can configure. Each address that you configure in this way has a dedicated folder in your file system to store paged messages. Both queue browsers and consumers can navigate through page files when inspecting messages in a queue. However, a consumer that is using a very specific filter might not be able to consume a message that is stored in a page file until existing messages in the queue have been consumed first. For example, suppose that a consumer filter includes a string expression such as "color='red'" . If a message that meets this condition follows one million messages with the property "color='blue'" , the consumer cannot consume the message until those with "color='blue'" have been consumed first. The broker transfers (that is, depages ) messages from disk into memory when clients are ready to consume them. The broker removes a page file from disk when all messages in that file have been acknowledged. Important AMQ Broker orders pending messages in a queue by JMS message priority. However, messages that are paged are not ordered by priority. If you want to preserve the ordering, do not configure paging and size the broker sufficiently to keep all messages in memory. 7.1.1. Specifying a paging directory The following procedure shows how to specify the location of the paging directory. Procedure Open the <broker_instance_dir> /etc/broker.xml configuration file. Within the core element, add the paging-directory element. Specify a location for the paging directory in your file system. <configuration ...> <core ...> ... <paging-directory> /path/to/paging-directory </paging-directory> ... </core> </configuration> For each address that you subsequently configure for paging, the broker adds a dedicated directory within the paging directory that you have specified. 7.1.2. Configuring an address for paging The following procedure shows how to configure an address for paging. Prerequisites You should be familiar with how to configure addresses and address settings. For more information, see Chapter 4, Configuring addresses and queues . Procedure Open the <broker_instance_dir> /etc/broker.xml configuration file. For an address-setting element that you have configured for a matching address or set of addresses, add configuration elements to specify maximum memory usage and define paging behavior. For example: <address-settings> <address-setting match="my.paged.address"> ... <max-size-bytes>104857600</max-size-bytes> <max-size-messages>20000</max-size-messages> <page-size-bytes>10485760</page-size-bytes> <address-full-policy>PAGE</address-full-policy> ... </address-setting> </address-settings> max-size-bytes Maximum size, in bytes, of the memory allowed for the address before the broker executes the action specified for the address-full-policy attribute. The default value is -1 , which means that there is no limit. The value that you specify also supports byte notation such as "K", "MB", and "GB". max-size-messages Maximum number of messages allowed for the address before the broker executes the action specified for the address-full-policy attribute. The default value is -1, which means that there is no message limit. page-size-bytes Size, in bytes, of each page file used on the paging system. The default value is 10485760 (that is, 10 MiB). The value that you specify also supports byte notation such as "K", "MB", and "GB". address-full-policy Action that the broker takes when the maximum size for an address has been reached. The default value is PAGE . Valid values are: PAGE The broker pages any further messages to disk. DROP The broker silently drops any further messages. FAIL The broker drops any further messages and issues exceptions to client message producers. BLOCK Client message producers block when they try to send further messages. If you set limits for the max-size-bytes and max-size-message attributes, the broker executes the action specified for the address-full-policy attribute when either limit is reached. With the configuration in the example, the broker starts paging messages for the my.paged.address address when the total messages for the address in memory exceeds 20,000 or uses 104857600 bytes of available memory. Additional paging configuration elements that are not shown in the preceding example are described below. page-sync-timeout Time, in nanoseconds, between periodic page synchronizations. If you are using an asynchronous IO journal (that is, journal-type is set to ASYNCIO in the broker.xml configuration file), the default value is 3333333 . If you are using a standard Java NIO journal (that is, journal-type is set to NIO ), the default value is the configured value of the journal-buffer-timeout parameter. In the preceding example , when messages sent to the address my.paged.address exceed 104857600 bytes in memory, the broker begins paging. Note If you specify max-size-bytes in an address-setting element, the value applies to each matching address. Specifying this value does not mean that the total size of all matching addresses is limited to the value of max-size-bytes . 7.1.3. Configuring a global paging size Sometimes, configuring a memory limit per address is not practical, for example, when a broker manages many addresses that have different usage patterns. In these situations, you can specify a global memory limit. The global limit is the total amount of memory that the broker can use for all addresses. When this memory limit is reached, the broker executes the action specified for the address-full-policy attribute for the address associated with each new incoming message. The following procedure shows how to configure a global paging size. Prerequisites You should be familiar with how to configure an address for paging. For more information, see Section 7.1.2, "Configuring an address for paging" . Procedure Stop the broker. On Linux: On Windows: Open the <broker_instance_dir> /etc/broker.xml configuration file. Within the core element, add the global-max-size element and specify a value. For example: <configuration> <core> ... <global-max-size>1GB</global-max-size> <global-max-messages>900000</global-max-messages> ... </core> </configuration> global-max-size Total amount of memory, in bytes, that the broker can use for all addresses. When this limit is reached, the broker executes the action specified for the address-full-policy attribute for the address associated with each incoming message. The default value of global-max-size is half of the maximum memory available to the Java virtual machine (JVM) that is hosting the broker. The value for global-max-size is in bytes, but also supports byte notation (for example, "K", "Mb", "GB"). In the preceding example, the broker is configured to use a maximum of one gigabyte of available memory when processing messages. global-max-messages The total number of messages allowed for all addresses. When this limit is reached, the broker executes the action specified for the address-full-policy attribute for the address associated with each incoming message. The default value is -1, which means that there is no message limit. If you set limits for the global-max-size and global-max-messages attributes, the broker executes the action specified for the address-full-policy attribute when either limit is reached. With the configuration in the example, the broker starts paging messages for all addresses when the number of messages in memory exceeds 900,000 or uses 1 GB of available memory. Note If limits that are set for an individual address, by using the max-size-bytes or max-size-message attributes, are reached before the limits set for the global-max-size or global-max-messages attributes, the broker executes the action specified for the address-full-policy attribute for that address. Start the broker. On Linux: On Windows: 7.1.4. Limiting disk usage during paging for specific addresses You can limit the amount of disk space that the broker can use before it stops paging incoming messages for an individual address or set of addresses. Procedure Stop the broker. On Linux: On Windows: Open the <broker_instance_dir> /etc/broker.xml configuration file. Within the core element, add attributes to specify paging limits based on disk usage or number of messages, or both, and specify the action to take if either limit is reached. For example: <address-settings> <address-setting match="match="my.paged.address""> ... <page-limit-bytes>10G</page-limit-bytes> <page-limit-messages>1000000</page-limit-messages> <page-full-policy>FAIL</page-full-policy> ... </address-setting> </address-settings> page-limit-bytes Maximum size, in bytes, of the disk space allowed for paging incoming messages for the address before the broker executes the action specified for the page-full-policy attribute. The value that you specify supports byte notation such as "K", "MB", and "GB". The default value is -1, which means that there is no limit. page-limit-messages Maximum number of incoming message that can be paged for the address before the broker executes the action specified for the page-full-policy attribute. The default value is -1, which means that there is no message limit. page-full-policy Action that the broker takes when a limit set in the page-limit-bytes or page-limit-messages attributes is reached for an address. Valid values are: DROP The broker silently drops any further messages. FAIL The broker drops any further messages and notifies the sending clients In the preceding example, the broker pages message for the my.paged.address address until paging uses 10GB of disk space or until a total of one million messages are paged. Start the broker. On Linux: On Windows: 7.1.5. Controlling the flow of paged messages into memory If AMQ Broker is configured to page messages to disk, the broker reads paged messages and transfers the messages into memory when clients are ready to consume messages. To prevent messages from consuming excess memory, you can limit the memory used by each address for messages that the broker transfers from disk to memory. Important If client applications leave too many messages pending acknowledgment, the broker does not read paged messages until the pending messages are acknowledged, which can cause message starvation on the broker. For example, if the limit for the transfer of paged messages into memory, which is 20 MB by default, is reached, the broker waits for an acknowledgment from a client before it reads any more messages. If, at the same time, clients are waiting to receive sufficient messages before they send an acknowledgment to the broker, which is determined by the batch size used by clients, the broker is starved of messages. To avoid starvation, either increase the broker limits that control the transfer of paged message into memory or reduce the number of delivering messages. You can reduce the number of delivering messages by ensuring that clients either commit message acknowledgments sooner or use a timeout and commit acknowledgments when no more messages are received from the broker. You can see the number and size of delivering messages in a queue's Delivering Count and Delivering Bytes metrics in AMQ Management Console. Procedure Open the <broker_instance_dir> /etc/broker.xml configuration file. For an address-settings element that you have configured for a matching address or set of addresses, specify limits on the transfer of paged messages into memory. For example: address-settings> <address-setting match="my.paged.address"> ... <max-read-page-messages>104857600</max-read-page-messages> <max-read-page-bytes>20MB</max-read-page-bytes> ... </address-setting> </address-settings> Max-read-page-messages Maximum number of paged messages that the broker can read from disk into memory per-address. The default value is -1, which means that no limit applies. Max-read-page-bytes Maximum size in bytes of paged messages that the broker can read from disk into memory per-address. The default value is 20 MB. When it applies these limits, the broker counts both messages in memory that are ready for delivery to consumers and messages that are currently delivering. If consumers are slow to acknowledge messages, delivering messages can cause the memory or message limit to be reached and prevent the broker from reading new messages into memory. As a result, the broker can be starved of messages. If consumers are slow to acknowledge messages and the configured max-read-page-messages or max-read-page-bytes limits are reached by messages that are currently delivering, specify separate limits on the transfer of paged messages into memory for messages that are currently delivering. For example address-settings> <address-setting match="my.paged.address"> ... <prefetch-page-bytes>20MB</prefetch-page-bytes> <prefetch-page-messages>104857600</prefetch-page-messages> ... </address-setting> </address-settings> prefetch-page-bytes Memory, in bytes, that is available to read paged messages into memory per-queue. The default value is 20 MB. prefetch-page-messages Number of paged messages that the broker can read from disk into memory per-queue. The default value is -1, which means that no limit applies. If you specify limits for the prefetch-page-bytes or prefetch-page-messages parameters to limit the amount of memory or number of messages used by messages currently delivering, set higher limits for the max-read-page-bytes or max-read-page-message parameter to provide capacity to read new messages into memory. Note If the value of the max-read-page-bytes parameter is reached before the value of the prefetch-page-bytes parameter, the broker stops reading further paged messages into memory. 7.1.6. Setting a disk usage threshold You can set a disk usage threshold which, if reached, causes the broker to stop paging and block all incoming messages. Procedure Stop the broker. On Linux: On Windows: Open the <broker_instance_dir> /etc/broker.xml configuration file. Within the core element add the max-disk-usage configuration element and specify a value. For example: <configuration> <core> ... <max-disk-usage>80</max-disk-usage> ... </core> </configuration> max-disk-usage Maximum percentage of the available disk space that the broker can use. When this limit is reached, the broker blocks incoming messages. The default value is 90 . In the preceding example, the broker is limited to using eighty percent of available disk space. Start the broker. On Linux: On Windows: 7.2. Configuring message dropping Section 7.1.2, "Configuring an address for paging" shows how to configure an address for paging. As part of that procedure, you set the value of address-full-policy to PAGE . To drop messages (rather than paging them) when an address reaches its specified maximum size, set the value of the address-full-policy to one of the following: DROP When the maximum size of a given address has been reached, the broker silently drops any further messages. FAIL When the maximum size of a given address has been reached, the broker drops any further messages and issues exceptions to producers. 7.3. Configuring message blocking The following procedures show how to configure message blocking when a given address reaches the maximum size limit that you have specified. Note You can configure message blocking only for the Core, OpenWire, and AMQP protocols. 7.3.1. Blocking Core and OpenWire producers The following procedure shows how to configure message blocking for Core and OpenWire message producers when a given address reaches the maximum size limit that you have specified. Prerequisites You should be familiar with how to configure addresses and address settings. For more information, see Chapter 4, Configuring addresses and queues . Procedure Open the <broker_instance_dir> /etc/broker.xml configuration file. For an address-setting element that you have configured for a matching address or set of addresses, add configuration elements to define message blocking behavior. For example: <address-settings> <address-setting match="my.blocking.address"> ... <max-size-bytes>300000</max-size-bytes> <address-full-policy>BLOCK</address-full-policy> ... </address-setting> </address-settings> max-size-bytes Maximum size, in bytes, of the memory allowed for the address before the broker executes the policy specified for address-full-policy . The value that you specify also supports byte notation such as "K", "MB", and "GB". Note If you specify max-size-bytes in an address-setting element, the value applies to each matching address. Specifying this value does not mean that the total size of all matching addresses is limited to the value of max-size-bytes . address-full-policy Action that the broker takes when then the maximum size for an address has been reached. In the preceding example, when messages sent to the address my.blocking.address exceed 300000 bytes in memory, the broker begins blocking further messages from Core or OpenWire message producers. 7.3.2. Blocking AMQP producers Protocols such as Core and OpenWire use a window-size flow control system. In this system, credits represent bytes and are allocated to producers. If a producer wants to send a message, the producer must wait until it has sufficient credits for the size of the message. By contrast, AMQP flow control credits do not represent bytes. Instead, AMQP credits represent the number of messages a producer is permitted to send, regardless of message size. Therefore, it is possible, in some situations, for AMQP producers to significantly exceed the max-size-bytes value of an address. Therefore, to block AMQP producers, you must use a different configuration element, max-size-bytes-reject-threshold . For a matching address or set of addresses, this element specifies the maximum size, in bytes, of all AMQP messages in memory. When the total size of all messages in memory reaches the specified limit, the broker blocks AMQP producers from sending further messages. The following procedure shows how to configure message blocking for AMQP message producers. Prerequisites You should be familiar with how to configure addresses and address settings. For more information, see Chapter 4, Configuring addresses and queues . Procedure Open the <broker_instance_dir> /etc/broker.xml configuration file. For an address-setting element that you have configured for a matching address or set of addresses, specify the maximum size of all AMQP messages in memory. For example: <address-settings> <address-setting match="my.amqp.blocking.address"> ... <max-size-bytes-reject-threshold>300000</max-size-bytes-reject-threshold> ... </address-setting> </address-settings> max-size-bytes-reject-threshold Maximum size, in bytes, of the memory allowed for the address before the broker blocks further AMQP messages. The value that you specify also supports byte notation such as "K", "MB", and "GB". By default, max-size-bytes-reject-threshold is set to -1 , which means that there is no maximum size. Note If you specify max-size-bytes-reject-threshold in an address-setting element, the value applies to each matching address. Specifying this value does not mean that the total size of all matching addresses is limited to the value of max-size-bytes-reject-threshold . In the preceding example, when messages sent to the address my.amqp.blocking.address exceed 300000 bytes in memory, the broker begins blocking further messages from AMQP producers. 7.4. Understanding memory usage on multicast addresses When a message is routed to an address that has multicast queues bound to it, there is only one copy of the message in memory. Each queue has only a reference to the message. Because of this, the associated memory is released only after all queues referencing the message have delivered it. In this type of situation, if you have a slow consumer, the entire address might experience a negative performance impact. For example, consider this scenario: An address has ten queues that use the multicast routing type. Due to a slow consumer, one of the queues does not deliver its messages. The other nine queues continue to deliver messages and are empty. Messages continue to arrive to the address. The queue with the slow consumer continues to accumulate references to the messages, causing the broker to keep the messages in memory. When the maximum size of the address is reached, the broker starts to page messages. In this scenario because of a single slow consumer, consumers on all queues are forced to consume messages from the page system, requiring additional IO. Additional resources To learn how to configure flow control to regulate the flow of data between the broker and producers and consumers, see Flow control in the AMQ Core Protocol JMS documentation.

Commands:
<configuration ...> <core ...> <paging-directory> /path/to/paging-directory </paging-directory> </core> </configuration>
<address-settings> <address-setting match="my.paged.address"> <max-size-bytes>104857600</max-size-bytes> <max-size-messages>20000</max-size-messages> <page-size-bytes>10485760</page-size-bytes> <address-full-policy>PAGE</address-full-policy> </address-setting> </address-settings>
<broker_instance_dir> /bin/artemis stop
<broker_instance_dir> \bin\artemis-service.exe stop
<configuration> <core> <global-max-size>1GB</global-max-size> <global-max-messages>900000</global-max-messages> </core> </configuration>
<broker_instance_dir> /bin/artemis run
<broker_instance_dir> \bin\artemis-service.exe start
<broker_instance_dir> /bin/artemis stop
<broker_instance_dir> \bin\artemis-service.exe stop
<address-settings> <address-setting match="match="my.paged.address""> <page-limit-bytes>10G</page-limit-bytes> <page-limit-messages>1000000</page-limit-messages> <page-full-policy>FAIL</page-full-policy> </address-setting> </address-settings>
<broker_instance_dir> /bin/artemis run
<broker_instance_dir> \bin\artemis-service.exe start
address-settings> <address-setting match="my.paged.address"> <max-read-page-messages>104857600</max-read-page-messages> <max-read-page-bytes>20MB</max-read-page-bytes> </address-setting> </address-settings>
address-settings> <address-setting match="my.paged.address"> <prefetch-page-bytes>20MB</prefetch-page-bytes> <prefetch-page-messages>104857600</prefetch-page-messages> </address-setting> </address-settings>
<broker_instance_dir> /bin/artemis stop
<broker_instance_dir> \bin\artemis-service.exe stop
<configuration> <core> <max-disk-usage>80</max-disk-usage> </core> </configuration>
<broker_instance_dir> /bin/artemis run
<broker_instance_dir> \bin\artemis-service.exe start
<address-settings> <address-setting match="my.blocking.address"> <max-size-bytes>300000</max-size-bytes> <address-full-policy>BLOCK</address-full-policy> </address-setting> </address-settings>
<address-settings> <address-setting match="my.amqp.blocking.address"> <max-size-bytes-reject-threshold>300000</max-size-bytes-reject-threshold> </address-setting> </address-settings>