Project: StarRocks (App Definition and Development / Database)
File: datalake_faq.md

displayed_sidebar: "English" This topic describes some commonly asked questions (FAQ) about data lake and provides solutions to these issues. Some metrics mentioned in this topic can be obtained only from the profiles of the SQL queries. To obtain the profiles of SQL queries, you must specify `set enable_profile=true`. When you access the data files stored in your HDFS cluster, you may find a huge difference between the values of the `MAX_OF_FSIOTime` and `MINOFFSIOTime` metrics from the profiles of the SQL queries you run. This indicates that some DataNodes in the HDFS cluster are slow. The following example is a typical profile that indicates a slow HDFS DataNode issue: ```plaintext InputStream: 0 AppIOBytesRead: 22.72 GB MAXOFAppIOBytesRead: 187.99 MB MINOFAppIOBytesRead: 64.00 KB AppIOCounter: 964.862K (964862) MAXOFAppIOCounter: 7.795K (7795) MINOFAppIOCounter: 1 AppIOTime: 1s372ms MAXOFAppIOTime: 4s358ms MINOFAppIOTime: 1.539ms FSBytesRead: 15.40 GB MAXOFFSBytesRead: 127.41 MB MINOFFSBytesRead: 64.00 KB FSIOCounter: 1.637K (1637) MAXOFFSIOCounter: 12 MINOFFSIOCounter: 1 FSIOTime: 9s357ms MAXOFFSIOTime: 60s335ms MINOFFSIOTime: 1.536ms ``` You can use one of the following solutions to resolve this issue: feature, which eliminates the impact of slow HDFS DataNodes on queries by automatically caching the data from external storage systems to the BEs or CNs of your StarRocks cluster. [Recommended] Shorten the timeout duration between the HDFS client and DataNode. This solution is suitable when Data Cache cannot help resolve the slow HDFS DataNode issue. Enable the feature. With this feature enabled, if a read from a block is slow, StarRocks starts up a new read, which runs in parallel to the original read, to read against a different block replica. Whenever one of the two reads returns, the other read is cancelled. The Hedged Read feature can help accelerate reads, but it also significantly increases heap memory consumption on Java virtual machines (JVMs). Therefore, if your physical machines provide a small memory capacity, we recommend that you do not enable the Hedged Read feature. See . Configure the `dfs.client.socket-timeout` property in the `hdfs-site.xml` file to shorten the timeout duration between the HDFS client and

(The default timeout duration is 60s, which is a bit long.) As such, when StarRocks encounters a slow DataNode, the connection request from it can time out within a very short period of time and then be forwarded to another DataNode. The following example sets a 5-second timeout duration: ```xml <configuration> <property> <name>dfs.client.socket-timeout</name> <value>5000</value> </property> </configuration> ``` Use the following parameters (supported from v3.0 onwards) in the BE or CN configuration file `be.conf` to enable and configure the Hedged Read feature in your HDFS cluster. | Parameter | Default value | Description | | - | - | - | | hdfsclientenablehedgedread | false | Specifies whether to enable the hedged read feature. | | hdfsclienthedgedreadthreadpool_size | 128 | Specifies the size of the Hedged Read thread pool on your HDFS client. The thread pool size limits the number of threads to dedicate to the running of hedged reads in your HDFS client. This parameter is equivalent to the `dfs.client.hedged.read.threadpool.size` parameter in the `hdfs-site.xml` file of your HDFS cluster. | | hdfsclienthedgedreadthreshold_millis | 2500 | Specifies the number of milliseconds to wait before starting up a hedged read. For example, you have set this parameter to `30`. In this situation, if a read from a block has not returned within 30 milliseconds, your HDFS client immediately starts up a hedged read against a different block replica. This parameter is equivalent to the `dfs.client.hedged.read.threshold.millis` parameter in the `hdfs-site.xml` file of your HDFS cluster. | If the value of any of the following metrics in your query profiles exceeds `0`, the Hedged Read feature is enabled. | Metric | Description | | | | | TotalHedgedReadOps | The number of hedged reads that are started up. | | TotalHedgedReadOpsInCurThread | The number of times that StarRocks has to start up a hedged read in the current thread instead of in a new thread because the Hedged Read thread pool has reached its maximum size specified by the `hdfsclienthedgedreadthreadpool_size` parameter. | | TotalHedgedReadOpsWin | The number of times that a hedged read beats its original read. |