Project: StarRocks (App Definition and Development / Database)
File: Memory_management.md

displayed_sidebar: "English" This section briefly introduces memory classification and StarRocks methods of managing memory. Explanation: | Metric | Name | Description | | | | | | process | Total memory used of BE | | | query\_pool | Memory used by data querying | Consists of two parts: memory used by the execution layer and memory used by the storage layer.| | load | Memory used by data loading | Generally MemTable| | table_meta | Metadata memory | S Schema, Tablet metadata, RowSet metadata, Column metadata, ColumnReader, IndexReader | | compaction | Multi-version memory compaction | compaction that happens after data import is complete | | snapshot | Snapshot memory | Generally used for clone, little memory usage | | column_pool | Column pool memory | Request to release column cache for accelerated column | | page_cache | BE's own PageCache | The default is off, the user can turn it on by modifying the BE file | BE Configuration* | Name | Default| Description| | | | | | vectorchunksize | 4096 | Number of chunk rows | | mem_limit | 90% | BE process memory upper limit. You can set it as a percentage ("80%") or a physical limit ("100G"). The default hard limit is 90% of the server's memory size, and the soft limit is 80%. You need to configure this parameter if you want to deploy StarRocks with other memory-intensive services on a same server. | | disablestoragepagecache | false | The boolean value to control whether to disable PageCache. When PageCache is enabled, StarRocks caches the recently scanned data. PageCache can significantly improve the query performance when similar queries are repeated frequently. `true` indicates to disable PageCache. Use this item together with `storagepagecachelimit`, you can accelerate query performance in scenarios with sufficient memory resources and much data scan. The default value of this item has been changed from `true` to `false` since StarRocks v2.4. | | writebuffersize | 104857600 | The capacity limit of a single MemTable, exceeding which a disk swipe will be performed. | | loadprocessmaxmemorylimitbytes | 107374182400 | The upper limit of memory resources that can be taken up by all load processes on a BE node. Its value is the smaller one between `memlimit * loadprocessmaxmemorylimitpercent / 100` and `loadprocessmaxmemorylimitbytes`. If this threshold is exceeded, a flush and backpressure will be

| | loadprocessmaxmemorylimitpercent | 30 | The maximum percentage of memory resources that can be taken up by all load processes on a BE node. Its value is the smaller one between `memlimit * loadprocessmaxmemorylimitpercent / 100` and `loadprocessmaxmemorylimitbytes`. If this threshold is exceeded, a flush and backpressure will be triggered. | | defaultloadmemlimit | 2147483648 | If the memory limit on the receiving side is reached for a single import instance, a disk swipe will be triggered. This needs to be modified with the Session variable `loadmem_limit` to take effect. | | maxcompactionconcurrency | -1 | The maximum concurrency of compactions (both Base Compaction and Cumulative Compaction). The value -1 indicates that no limit is imposed on the concurrency. | | cumulativecompactioncheckintervalseconds | 1 | Interval of compaction check| Session variables* | Name| Default| Description| | | | | | querymemlimit| 0| Memory limit of a query on each BE node | | loadmemlimit | 0| Memory limit of a single import task. If the value is 0, `execmemlimit` will be taken| mem\_tracker* ~~~ bash //View the overall memory statistics <http://beip:behttpport/memtracker> // View fine-grained memory statistics <http://beip:behttpport/memtracker?type=querypool&upperlevel=3> ~~~ tcmalloc* ~~~ bash <http://beip:behttp_port/memz> ~~~ ~~~plain text MALLOC: 777276768 ( 741.3 MiB) Bytes in use by application MALLOC: + 8851890176 ( 8441.8 MiB) Bytes in page heap freelist MALLOC: + 143722232 ( 137.1 MiB) Bytes in central cache freelist MALLOC: + 21869824 ( 20.9 MiB) Bytes in transfer cache freelist MALLOC: + 832509608 ( 793.9 MiB) Bytes in thread cache freelists MALLOC: + 58195968 ( 55.5 MiB) Bytes in malloc metadata MALLOC: MALLOC: = 10685464576 (10190.5 MiB) Actual memory used (physical + swap) MALLOC: + 25231564800 (24062.7 MiB) Bytes released to OS (aka unmapped) MALLOC: MALLOC: = 35917029376 (34253.1 MiB) Virtual address space used MALLOC: MALLOC: 112388 Spans in use MALLOC: 335 Thread heaps in use MALLOC: 8192 Tcmalloc page size Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. ~~~ The memory queried by this method is accurate. However, some memory in StarRocks is reserved but not in use. TcMalloc counts the memory that is reserved, not the memory used. Here `Bytes in use by application` refers to the memory currently in use. metrics* ~~~bash curl -XGET http://beip:behttp_port/metrics | grep 'mem' curl -XGET http://beip:behttpport/metrics | grep 'columnpool' ~~~ The value of metrics is updated every 10 seconds. It is possible to monitor some of the memory statistics with older versions.