Project: StarRocks (App Definition and Development / Database)
File: query_profile_details.md

displayed_sidebar: "English" The structure of a Query Profile is closely related to the design of StarRocks' execution engine and consists of the following five parts: Fragment: Execution tree. A query is composed of one or more fragments. FragmentInstance: Each fragment can have multiple instances, each instance is called a FragmentInstance, and is executed by different computing nodes. Pipeline: A FragmentInstance is split into multiple pipelines. A pipeline is an execution chain consisting of a group of connected Operator instances. PipelineDriver: A Pipeline can have multiple instances, each instance is called a PipelineDriver, to fully utilize multiple computing cores. Operator: A PipelineDriver consists of multiple Operator instances. By analyzing the structure of Query Profile, you can easily observe that multiple FragmentInstances associated with the same Fragment have a high degree of similarity in structure. Similarly, multiple PipelineDrivers belonging to the same Pipeline also exhibit similar structural features. To reduce the volume of the Query Profile, StarRocks by default merges the FragmentInstance layer and the PipelineDriver layer. As a result, the original five-layer structure is simplified to three layers: Fragment Pipeline Operator You can control this merging behavior through a session variable `pipelineprofilelevel`, which has two valid values: `1` (Default): StarRocks merges the metrics into a three-layer structure. `2`: StarRocks does not merge the metrics. The original five-layer structure is retained. Any other value will be treated as the default value `1`. Generally, we do not recommend setting this parameter to `2` because the Query Profile with the five-layer structure has many limitations. For example, you cannot perform visualized analysis on the profile using any tools. Therefore, unless the merging process leads to the loss of crucial information, you do not need to adjust this parameter. When merging FragmentInstance and PipelineDriver, all metrics with the same name are merged. Only the minimum and maximum values of each metric in all concurrent instances are recorded. Different types of metrics use different merging strategies: Time-related metrics take the average. For example: `OperatorTotalTime` is the average time consumption of all concurrent instances. `MAXOFOperatorTotalTime` is the maximum time consumption among all concurrent instances. `MINOFOperatorTotalTime` is the minimum time consumption among all concurrent instances. ```SQL OperatorTotalTime: 2.192us MAXOFOperatorTotalTime: 2.502us MINOFOperatorTotalTime: 1.882us ``` Non-time-related metrics are summed. For example: `PullChunkNum` is the sum of this metric in all concurrent instances. `MAXOFPullChunkNum` is the maximum value of this metric among all concurrent instances. `MINOFPullChunkNum` is the minimum value of this metric among all concurrent instances. ```SQL PullChunkNum: 146.66K (146660) MAXOFPullChunkNum: 24.45K (24450) MINOFPullChunkNum: 24.435K (24435) ``` Some metrics without extreme values have the same value in all concurrent instances, for example: `DegreeOfParallelism`. Usually, if there is a significant difference between MIN and MAX values, it indicates a high probability of data skew. Possible scenarios include aggregation and join operations. ```SQL OperatorTotalTime: 2m48s MAXOFOperatorTotalTime: 10m30s MINOFOperatorTotalTime: 279.170us ``` The Query Profile includes a multitude of metrics providing detailed information about query execution. In most cases, you only need to focus on the execution time of operators and the size of processed data. Once you identify bottlenecks, you can address them specifically. Description: The total time consumed by the query, including Planning, Executing, and Profiling phase

Description: Query state, possible states include Finished, Error, and Running. Description: Query Profile processing time on the Frontend (FE) side. Description: Cumulative allocated memory across all compute nodes. Description: Cumulative deallocated memory across all compute nodes. Description: Maximum peak memory across all compute nodes. Description: Wall time of the execution. Description: Cumulative CPU time across all compute nodes. Description: Cumulative time across all nodes. This is a simple linear accumulation, but in reality, execution times of different operators may overlap. This parameter serves as the denominator for calculating the percentage of time spent on each operator. Description: Cumulative network time of all Exchange nodes. Similar to cumulative operator time, actual execution times of different Exchanges may overlap. Description: Cumulative IO time of all Scan nodes. Similar to cumulative operator time, actual execution times of different Scan operations may overlap. Description: Maximum ScheduleTime metric across all Pipelines. Description: Size of data spilled to local disks. Description: Additional time to transfer results. For query statements, this parameter refers to the time it takes to send data back to the client; for insert statements, it refers to the time it takes to write data to the storage layer. Description: Number of all FragmentInstances for this Fragment. Description: IDs of all FragmentInstances for this Fragment. Description: Number of BEs participating in the execution of this Fragment. Description: Addresses of all BEs participating in the execution of this Fragment. Description: Time spent in the Fragment Prepare phase. Description: Cumulative allocated memory for all FragmentInstances under this Fragment. Description: Cumulative deallocated memory for all FragmentInstances under this Fragment. Description: The peak memory usage across all FragmentInstances under this Fragment. The relationship between core metrics is illustrated in the following diagram: DriverTotalTime = ActiveTime + PendingTime + ScheduleTime ActiveTime = OperatorTotalTime + OverheadTime PendingTime = InputEmptyTime + OutputFullTime + PreconditionBlockTime + PendingFinishTime InputEmptyTime = FirstInputEmptyTime + FollowupInputEmptyTime Description: Degree of pipeline execution parallelism. Description: Sum of degrees of parallelism. Since the same Pipeline may execute on multiple machines, this item aggregates all values. Description: Time taken by the Prepare phase. This metric is not included in DriverTotalTime. Description: Total execution time of the Pipeline, excluding the time spent in the Prepare phase. Description: Execution time of the Pipeline, including the execution time of each operator and the overall framework overhead, such as time spent in invoking methods like hasoutput, needinput, etc. Description: Time the Pipeline is blocked from being scheduled for various reasons. Description: Time the Pipeline is blocked due to an empty input queue. Description: Time the Pipeline is first blocked due to an empty input queue. The first blocking time is separately calculated because the first blocking is mainly caused by Pipeline dependencies. Description: Time the Pipeline is subsequently blocked due to an empty input queue. Description: Time the Pipeline is blocked due to a full output queue. Description: Time the Pipeline is blocked due to unmet dependencies. Description: Time the Pipeline is blocked waiting for asynchronous tasks to finish. Description: Scheduling time of the Pipeline, from entering the ready queue to being scheduled for execution. Description: Number of times the pipeline is blocked due to

Description: Number of times the pipeline is blocked due to OutputFull. Description: Number of times the pipeline is blocked due to unmet preconditions. Description: Cumulative memory allocated by the Operator. Description: Cumulative memory deallocated by the Operator. Description: Peak memory usage by the Operator across all compute nodes. This metric is meaningful for certain materialization operators, such as aggregation, sorting, Join, etc. It is not relevant for operators like Project because memory is allocated by the current operator and released by subsequent operators, making peak memory equivalent to cumulative allocated memory for the current operator. In versions earlier than v3.1.8 and v3.2.3, this metric represents the peak memory usage by the Operator across all PipelineDrivers. Description: Time spent on preparation. Description: Total time consumed by the Operator. It satisfies the equation: OperatorTotalTime = PullTotalTime + PushTotalTime + SetFinishingTime + SetFinishedTime + CloseTime. It excludes time spent on preparation. Description: Total time the Operator spends executing push_chunk. Description: Total time the Operator spends executing pull_chunk. Description: Total time the Operator spends executing set_finishing. Description: Total time the Operator spends executing set_finished. Description: Cumulative number of input rows for the Operator. Description: Cumulative number of output rows for the Operator. Description: Number of times Join Runtime Filter is evaluated. Description: Time spent computing hash for Join Runtime Filter. Description: Number of input rows for Join Runtime Filter. Description: Number of output rows for Join Runtime Filter. Description: Time spent on Join Runtime Filter. The Scan Operator utilizes an additional thread pool for executing IO tasks. Therefore, the relationship between time metrics for this node is illustrated below: To facilitate a better understanding of the various metrics within the Scan Operator, the following diagram demonstrates the associations between these metrics and storage structures. Description: Table name. Level: Primary metric Description: Materialized view name. If no materialized view is hit, it is equivalent to the table name. Level: Primary metric Description: Whether the enablesharedscan session variable is enabled. Level: Primary metric Description: Number of tablets. Level: Primary metric Description: Number of morsels. Level: Primary metric Description: Number of pushdown predicates. Level: Primary metric Description: Predicate expressions. Level: Primary metric Description: Size of data read. Level: Primary metric Description: Size of compressed data read from disk. Level: Primary metric Description: Size of uncompressed data read from disk. Level: Primary metric Description: Number of rows read (after predicate filtering). Level: Primary metric Description: Number of raw rows read (before predicate filtering). Level: Primary metric Description: Number of pages read. Level: Primary metric Description: Number of cached pages. Level: Primary metric Description: Capacity of the Chunk Buffer. Level: Primary metric Description: Default capacity of the Chunk Buffer. Level: Primary metric Description: Peak memory usage of the Chunk Buffer. Level: Primary metric Description: Peak size of the Chunk Buffer. Level: Primary metric Description: Time spent preparing the Chunk Source. Level: Primary metric Description: Cumulative scan time. Scan operations are completed in an asynchronous I/O thread pool. Level: Primary metric Description: Execution time of IO tasks. Level: Primary metric Sub-metrics: CreateSegmentIter, DictDecode, GetDelVec, GetDeltaColumnGroup, GetRowsets, IOTime, LateMaterialize, ReadPKIndex, SegmentInit, SegmentRead Description: Time spent creating the Segment

Level: Secondary metric Description: Time spent on decoding dictionary for low cardinality optimization. Level: Secondary metric Description: Time spent loading DelVec (delete vector). Level: Secondary metric Description: Time spent loading DelVecColumnGroup. Level: Secondary metric Description: Time spent loading RowSet. Level: Secondary metric Description: Time spent on file I/O. Level: Secondary metric Description: Time spent on late materialization. Level: Secondary metric Description: Time spent reading Primary Key index. Level: Secondary metric Description: Time spent initializing the Segment. Level: Secondary metric Sub-metrics: BitmapIndexFilter, BitmapIndexFilterRows, BloomFilterFilter, BloomFilterFilterRows, ColumnIteratorInit, ShortKeyFilter, ShortKeyFilterRows, ShortKeyRangeNumber, RemainingRowsAfterShortKeyFilter, ZoneMapIndexFilter, ZoneMapIndexFilterRows, SegmentZoneMapFilterRows, SegmentRuntimeZoneMapFilterRows Description: Time spent on Bitmap index filtering. Level: Tertiary metric Description: Number of rows filtered by Bitmap index. Level: Tertiary metric Description: Time spent on Bloom filter indexing. Level: Tertiary metric Description: Number of rows filtered by Bloom filter. Level: Tertiary metric Description: Time spent initializing the Column Iterator. Level: Tertiary metric Description: Time spent on ShortKey index filtering. Level: Tertiary metric Description: Number of rows filtered by ShortKey index. Level: Tertiary metric Description: Number of ShortKey ranges. Level: Tertiary metric Description: Number of rows remaining after ShortKey index filtering. Level: Tertiary metric Description: Time spent on ZoneMap index filtering. Level: Tertiary metric Description: Number of rows filtered by ZoneMap index. Level: Tertiary metric Description: Number of rows filtered by Segment ZoneMap index. Level: Tertiary metric Description: Number of rows filtered by Segment Runtime ZoneMap index. Level: Tertiary metric Description: Time spent reading the Segment. Level: Secondary metric Sub-metrics: BlockFetch, BlockFetchCount, BlockSeek, BlockSeekCount, ChunkCopy, DecompressT, DelVecFilterRows, PredFilter, PredFilterRows, RowsetsReadCount, SegmentsReadCount, TotalColumnsDataPageCount Description: Time spent on Block fetch. Level: Tertiary metric Description: Number of Block fetches. Level: Tertiary metric Description: Time spent on Block seek. Level: Tertiary metric Description: Number of Block seeks. Level: Tertiary metric Description: Time spent on Chunk copy. Level: Tertiary metric Description: Time spent on decompression. Level: Tertiary metric Description: Number of rows filtered by DELETE vector. Level: Tertiary metric Description: Time spent on predicate filtering. Level: Tertiary metric Description: Number of rows filtered by predicate. Level: Tertiary metric Description: Number of Rowsets reads. Level: Tertiary metric Description: Number of Segments reads. Level: Tertiary metric Description: Number of Column Data Pages. Level: Tertiary metric Description: Waiting time from successful submission to scheduled execution of IO tasks. Level: Primary metric Description: Number of times IO tasks are submitted. Level: Primary metric Description: Time spent on task submission. Level: Primary metric Description: Peak number of IO tasks. Level: Primary metric Description: Peak size of the IO task queue. Level: Primary metric Description: Data source type, can be HiveDataSource, ESDataSource, and so on. Level: Primary Metric Description: Table name. Level: Primary Metric Description: Number of tablets. Level: Primary Metric Description: Number of morsels. Level: Primary Metric Description: Predicate expression. Level: Primary Metric Description: Predicate expression applied to partitions. Level: Primary Metric Description: Whether the `enablesharedscan` Session variable is enabled. Level: Primary Metric Description: Capacity of the Chunk Buffer. Level: Primary Metric Description: Default capacity of the Chunk Buffer. Level: Primary Metric Description: Peak memory usage of the Chunk Buffer. Level: Primary Metric Description: Peak size of the Chunk Buffer. Level: Primary Metric Description: Time taken to prepare the Chunk Source. Level: Primary Metric Description: Cumulative time for

Scan operation is completed in the asynchronous I/O thread pool. Level: Primary Metric Description: Execution time of I/O tasks. Level: Primary Metric Sub-metrics: ColumnConvertTime, ColumnReadTime, ExprFilterTime, InputStream, ORC, OpenFile, ReaderInit, RowsRead, RowsSkip, ScanRanges, SharedBuffered Description: Time taken for column conversion. Level: Secondary Metric Description: Time taken for reading and parsing data by the reader. Level: Secondary Metric Description: Time taken for expression filtering. Level: Secondary Metric Description: Used for classification purposes. It has no specific meaning. Level: Secondary Metric Sub-metrics: AppIOBytesRead, AppIOCounter, AppIOTime, FSIOBytesRead, FSIOCounter, FSIOTime Description: Amount of data read at the application layer. Level: Tertiary Metric Description: Number of I/O operations read at the application layer. Level: Tertiary Metric Description: Cumulative reading time at the application layer. Level: Tertiary Metric Description: Amount of data read by the storage system. Level: Tertiary Metric Description: Number of I/O operations read by the storage layer. Level: Tertiary Metric Description: Cumulative reading time by the storage layer. Level: Tertiary Metric Description: Used for classification purposes. It has no specific meaning. Level: Secondary Metric Sub-metrics: IcebergV2FormatTimer, StripeNumber, StripeSizes Description: Time taken for format conversion. Level: Tertiary Metric Description: Number of ORC files. Level: Tertiary Metric Description: Average size of each stripe in ORC files. Level: Tertiary Metric Description: Time taken to open files. Level: Secondary Metric Description: Time taken to initialize the reader. Level: Secondary Metric Description: Number of rows read. Level: Secondary Metric Description: Number of rows skipped. Level: Secondary Metric Description: Total number of data ranges scanned. Level: Secondary Metric Description: Used for classification purposes. It has no specific meaning. Level: Secondary Metric Sub-metrics: DirectIOBytes, DirectIOCount, DirectIOTime, SharedIOBytes, SharedIOCount, SharedIOTime Description: Amount of data directly read by IO. Level: Tertiary Metric Description: Number of times direct IO is performed. Level: Tertiary Metric Description: Time taken for direct IO. Level: Tertiary Metric Description: Amount of data read by shared IO. Level: Tertiary Metric Description: Number of times shared IO is performed. Level: Tertiary Metric Description: Time taken for shared IO. Level: Tertiary Metric Description: Waiting time from successful submission to scheduled execution of IO tasks. Level: Primary Metric Description: Number of times IO tasks are submitted. Level: Primary Metric Description: Time taken to submit tasks. Level: Primary Metric Description: Peak number of IO tasks. Level: Primary Metric Description: Peak size of the IO task queue. Level: Primary Metric Description: Number of channels. Generally, the number of channels is equal to the number of receivers. Description: List of destination FragmentInstance IDs. Description: Destination node ID. Description: Data distribution mode, including: UNPARTITIONED, RANDOM, HASHPARTITIONED, and BUCKETSHUFFLEHASHPARTITIONED. Description: Time taken to serialize chunks. Description: Size of serialized data. Description: Number of Chunk Append operations when PartType is HASHPARTITIONED or BUCKETSHUFFLEHASHPARTITIONED. Description: Time taken for Chunk Append operations when PartType is HASHPARTITIONED or BUCKETSHUFFLEHASHPARTITIONED. Description: Time taken to calculate hash when PartType is HASHPARTITIONED or BUCKETSHUFFLEHASHPARTITIONED. Description: Number of data packets sent. Description: Number of unsent data packets. This metric is non-zero when there is a short-circuit logic; otherwise, it is zero. Description: Size of sent data. Description: Size of unsent data. This metric is non-zero when there is a short-circuit logic; otherwise, it is

Description: If the destination node is the current node, data will not be transmitted over the network, which is called passthrough data. This metric indicates the size of such passthrough data. Passthrough is controlled by `enableexchangepass_through`. Description: Peak memory usage of the PassThrough Buffer. Description: Compression time. Description: Size of compressed data. Description: Throughput rate. Description: Time taken for data packet transmission (excluding post-reception processing time). Description: Estimated network bandwidth. Description: Waiting time due to a full sender queue. Description: Total time for the entire transmission process, i.e., from sending the first data packet to confirming the correct reception of the last data packet. Description: Average time for RPC. Description: Total number of RPCs. Description: Size of received data packets. Description: Size of received data. Description: Time taken to decompress chunks. Description: Time taken to deserialize chunks. Description: Number of blocked Closures. Description: Blocked time for Closures. Description: Total time taken for receiver-side processing. Description: Lock waiting time. Description: GROUP BY columns. Description: Time taken for aggregate function calculations. Description: Time for AggregateFunctions + Group By. Description: Peak memory usage of the Chunk Buffer. Description: Peak size of the Chunk Buffer. Description: Time for expression computation. Description: Time for expression release. Description: Time to extract aggregate results. Description: Size of the Hash Table. Description: Memory size of the Hash Table. Description: Number of input rows. Description: In Auto mode, the number of data rows processed in streaming mode after low aggregation leads to degradation to streaming mode. Description: Time taken to append aggregate result columns. Description: Time taken to append Group By columns. Description: Time to iterate over the Hash Table. Description: Processing time in streaming mode. Description: Distribution type, including: BROADCAST, PARTITIONED, COLOCATE, etc. Description: Join predicates. Description: Join type. Description: Number of buckets in the Hash Table. Description: Number of keys per bucket in the Hash Table. Description: Time taken for conjunct evaluation during build phase. Description: Time taken to build the Hash Table. Description: Time taken for conjunct evaluation during probe phase. Description: Time taken to search the Hash Table. Description: Time taken to copy chunks from the right table. Description: Memory usage of the Hash Table. Description: Time taken to build runtime filters. Description: Number of runtime filters. Description: Execution mode, including two parts: the first part includes Materializing and Streaming; the second part includes Cumulative, RemovableCumulative, ByDefinition. Description: Time taken for window function calculations. Description: Partition columns. Description: Aggregate functions. Description: Time taken for column resizing. Description: Time taken to search partition boundaries. Description: Time taken to search Peer Group boundaries. Meaningful only when the window type is RANGE. Description: Peak number of rows in the buffer. Description: Number of times unused buffers are removed. Description: Total number of rows removed from unused buffers. Description: Sorting keys. Description: Query result sorting method: full sorting or sorting the top N results. Description: Peak size of buffered data. Description: Peak number of buffered rows. Description: Number of sorted runs. Description: Time taken to maintain internal data structures during sorting. Description: Time taken to merge sorted runs during sorting. Description: Time taken for sorting. Description: Time taken to build the output sorted

For ease of understanding various metrics, Merge can be represented as the following state mechanism: ```plaintext PENDING INIT PREPARE SPLITCHUNK FETCHCHUNK FINISHED | | one traverse from leaf to root | PROCESS ``` Description: Limit. Level: Primary Metric Description: Offset. Level: Primary Metric Description: Size of data processed per Merge operation when Merge is performed in Streaming mode Level: Primary Metric Description: Maximum number of chunks in the buffer when late materialization is enabled. Level: Primary Metric Description: Total execution count of all stages. Level: Primary Metric Sub-metrics: 1-InitStageCount, 2-PrepareStageCount, 3-ProcessStageCount, 4-SplitChunkStageCount, 5-FetchChunkStageCount, 6-PendingStageCount, 7-FinishedStageCount Description: Execution count of the Init stage. Level: Secondary Metric Description: Execution count of the Prepare stage. Level: Secondary Metric Description: Execution count of the Process stage. Level: Secondary Metric Description: Execution count of the SplitChunk stage. Level: Secondary Metric Description: Execution count of the FetchChunk stage. Level: Secondary Metric Description: Execution count of the Pending stage. Level: Secondary Metric Description: Execution count of the Finished stage. Level: Secondary Metric Description: Total execution time for each stage. Level: Primary metric Sub-metrics: 1-InitStageTime, 2-PrepareStageTime, 3-ProcessStageTime, 4-SplitChunkStageTime, 5-FetchChunkStageTime, 6-PendingStageTime, 7-FinishedStageTime Description: Execution time for the Init stage. Level: Secondary metric Description: Execution time for the Prepare stage. Level: Secondary metric Description: Execution time for the Process stage. Level: Secondary metric Sub-metrics: LateMaterializationGenerateOrdinalTime, SortedRunProviderTime Description: Time taken for generating ordinal columns during late materialization. Level: Tertiary Metric Description: Time taken to retrieve data from the provider during the Process stage. Level: Tertiary Metric Description: Time taken for the Split stage. Level: Secondary metric Description: Time taken for the Fetch stage. Level: Secondary metric Description: Time taken for the Pending stage. Level: Secondary metric Description: Time taken for the Finished stage. Level: Secondary metric Description: Computation time for the Table Function. Description: Number of executions for the Table Function. Description: Computation time for expressions. Description: Computation time for common sub-expressions. Description: Type of Local Exchange, including: `Passthrough`, `Partition`, and `Broadcast`. Description: Number of shuffles. This metric is only valid when `Type` is `Partition`. Description: Peak memory usage. OlapTableSink is the operator that profiles data loading with INSERT INTO FILES() and Broker Load. This feature is supported from v3.3.0 onwards. :::tip An excessive difference between the Max and Min values of the PushChunkNum metric of OlapTableSink indicates data skew in the upstream operators, which may lead to a bottleneck in loading performance. RpcClientSideTime equals RpcServerSideTime plus network transmission time plus RPC framework processing time. If there is a significant difference between RpcClientSideTime and RpcServerSideTime, consider enabling compression to reduce transmission time. ::: Description: Number of the synchronous materialized views created for the destination table. Description: Whether Single Leader Replication is enabled. Description: ID of the loading transaction. Description: Number of rows read from upstream operators. Description: Number of rows filtered out due to inadequate data quality. Description: Number of rows written to the destination table. Description: Total RPC time consumption for loading recorded by the client side. Description: Total RPC time consumption for loading recorded by the server side. Description: Total time consumption for the data preparation phase, including data format conversion and data quality check. Description: Local time consumption for sending the data, including time for serializing and compressing data, and for submitting tasks to the sender queue.