Repository: apache/doris
Pull request #48757: [improvement](statistics)Improve analyze partition column and key column corner case.
Author: login: Jibing-Li
Labels: name: approved
description: Indicates a PR has been approved by one committer.
name: reviewed
name: usercase
description: Important user case type label
name: p0_b
name: dev/2.1.9-merged
name: dev/3.0.5-merged
Created: 2025-03-06T08:59:38Z | Merged: 2025-03-11T02:52:38Z

### What problem does this PR solve?

Improve 2 corner cases:
1. When we pick the tablets for sample analyze, skip the very large tablets.
2. When sample analyze for partition column that choose more than 1,000,000,000 rows, we switch to randomly choose some partitions to analyze. In this case, we don't want to choose the oldest partition with all the history data. 

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
        - [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
    - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [x] Confirm the release note
- [x] Confirm test cases
- [x] Confirm document
- [x] Add branch pick label <!-- Add branch pick label that this PR should merge into -->




Diff:
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
index ba2b2770a5d109..5cee3db36a8e9c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
@@ -25,6 +25,10 @@
 import org.apache.doris.catalog.MaterializedIndexMeta;
 import org.apache.doris.catalog.OlapTable;
 import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.PartitionInfo;
+import org.apache.doris.catalog.PartitionKey;
+import org.apache.doris.catalog.PartitionType;
+import org.apache.doris.catalog.RangePartitionItem;
 import org.apache.doris.common.Config;
 import org.apache.doris.common.DdlException;
 import org.apache.doris.common.FeConstants;
@@ -36,6 +40,8 @@
 
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Range;
 import com.google.common.collect.Sets;
 import org.apache.commons.text.StringSubstitutor;
 
@@ -63,6 +69,7 @@ public class OlapAnalysisTask extends BaseAnalysisTask {
     private boolean partitionColumnSampleTooManyRows = false;
     private boolean scanFullTable = false;
     private static final long MAXIMUM_SAMPLE_ROWS = 1_000_000_000;
+    public static final long NO_SKIP_TABLET_ID = -1;
 
     @VisibleForTesting
     public OlapAnalysisTask() {
@@ -172,6 +179,8 @@ protected Pair<List<Long>, Long> getSampleTablets() {
         // Sort the partitions to get stable result.
         List<Partition> sortedPartitions = olapTable.getPartitions().stream().sorted(
                 Comparator.comparing(Partition::getName)).collect(Collectors.toList());
+        long largeTabletId = 0;
+        long largeTabletRows = Long.MAX_VALUE;
         for (Partition p : sortedPartitions) {
             MaterializedIndex materializedIndex = info.indexId == -1 ? p.getBaseIndex() : p.getIndex(info.indexId);
             if (materializedIndex == null) {
@@ -190,8 +199,20 @@ protected Pair<List<Long>, Long> getSampleTablets() {
             for (int i = 0; i < tabletCounts; i++) {
                 int seekTid = (int) ((i + seek) % ids.size());
                 long tabletId = ids.get(seekTid);
-                sampleTabletIds.add(tabletId);
                 long tabletRows = materializedIndex.getTablet(tabletId).getMinReplicaRowCount(p.getVisibleVersion());
+                if (tabletRows > MAXIMUM_SAMPLE_ROWS) {
+                    LOG.debug("Found one large tablet id {} in table {}, rows {}",
+                            largeTabletId, tbl.getName(), largeTabletRows);
+                    // Skip very large tablet and record the smallest large tablet id and row count.
+                    if (tabletRows < largeTabletRows) {
+                        LOG.debug("Current smallest large tablet id {} in table {}, rows {}",
+                                largeTabletId, tbl.getName(), largeTabletRows);
+                        largeTabletId = tabletId;
+                        largeTabletRows = tabletRows;
+                    }
+                    continue;
+                }
+                sampleTabletIds.add(tabletId);
                 if (tabletRows > 0) {
                     selectedRows += tabletRows;
                     // For regular column, will stop adding more tablets when selected tablets'
@@ -208,6 +229,13 @@ protected Pair<List<Long>, Long> getSampleTablets() {
                 break;
             }
         }
+        // If we skipped some large tablets and this cause the sampled rows is not enough, we add the large tablet back.
+        if (!enough && largeTabletId != 0) {
+            sampleTabletIds.add(largeTabletId);
+            selectedRows += largeTabletRows;
+            LOG.info("Add large tablet {} in table {} back, with rows {}",
+                    largeTabletId, tbl.getName(), largeTabletRows);
+        }
         if (selectedRows < targetSampleRows) {
             scanFullTable = true;
         } else if (forPartitionColumn && selectedRows > MAXIMUM_SAMPLE_ROWS) {
@@ -215,7 +243,7 @@ protected Pair<List<Long>, Long> getSampleTablets() {
             partitionColumnSampleTooManyRows = true;
             sampleTabletIds.clear();
             Collections.shuffle(sortedPartitions);
-            selectedRows = pickSamplePartition(sortedPartitions, sampleTabletIds);
+            selectedRows = pickSamplePartition(sortedPartitions, sampleTabletIds, getSkipPartitionId(sortedPartitions));
         } else if (col.isKey() && selectedRows > MAXIMUM_SAMPLE_ROWS) {
             // For key column, if a single tablet contains too many rows, need to use limit to control rows to read.
             // In most cases, a single tablet shouldn't contain more than MAXIMUM_SAMPLE_ROWS, in this case, we
@@ -372,12 +400,65 @@ protected String getIndex() {
         }
     }
 
-    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds) {
-        long averageRowsPerPartition = tbl.getRowCount() / partitions.size();
+    // For partition tables with single time type partition column, we'd better to skip sampling the partition
+    // that contains all the history data. Because this partition may contain many old data which is not
+    // visited by most queries. To sample this partition may cause the statistics not accurate.
+    // For example, one table has 366 partitions, partition 1 ~ 365 store date for each day of the year from now.
+    // Partition 0 stores all the history data earlier than 1 year. We want to skip sampling partition 0.
+    protected long getSkipPartitionId(List<Partition> partitions) {
+        if (partitions == null || partitions.size() < StatisticsUtil.getPartitionSampleCount()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionInfo partitionInfo = ((OlapTable) tbl).getPartitionInfo();
+        if (!PartitionType.RANGE.equals(partitionInfo.getType())) {
+            return NO_SKIP_TABLET_ID;
+        }
+        if (partitionInfo.getPartitionColumns().size() != 1) {
+            return NO_SKIP_TABLET_ID;
+        }
+        Column column = partitionInfo.getPartitionColumns().get(0);
+        if (!column.getType().isDateType()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionKey lowestKey = PartitionKey.createMaxPartitionKey();
+        long lowestPartitionId = -1;
+        for (Partition p : partitions) {
+            RangePartitionItem item = (RangePartitionItem) partitionInfo.getItem(p.getId());
+            Range<PartitionKey> items = item.getItems();
+            if (!items.hasLowerBound()) {
+                lowestPartitionId = p.getId();
+                break;
+            }
+            if (items.lowerEndpoint().compareTo(lowestKey) < 0) {
+                lowestKey = items.lowerEndpoint();
+                lowestPartitionId = p.getId();
+            }
+        }
+        return lowestPartitionId;
+    }
+
+    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds, long skipPartitionId) {
+        Partition partition = ((OlapTable) tbl).getPartition(skipPartitionId);
+        long averageRowsPerPartition;
+        if (partition != null) {
+            LOG.debug("Going to skip partition {} in table {}", skipPartitionId, tbl.getName());
+            // If we want to skip the oldest partition, calculate the average rows per partition value without
+            // the oldest partition, otherwise if the oldest partition is very large, we may skip all partitions.
+            // Because we only pick partitions which meet partitionRowCount >= averageRowsPerPartition.
+            Preconditions.checkNotNull(partitions, "Partition list of table " + tbl.getName() + " is null");
+            Preconditions.checkState(partitions.size() > 1, "Too few partitions in " + tbl.getName());
+            averageRowsPerPartition = (tbl.getRowCount() - partition.getRowCount()) / (partitions.size() - 1);
+        } else {
+            averageRowsPerPartition = tbl.getRowCount() / partitions.size();
+        }
         long indexId = info.indexId == -1 ? ((OlapTable) tbl).getBaseIndexId() : info.indexId;
         long pickedRows = 0;
         int pickedPartitionCount = 0;
         for (Partition p : partitions) {
+            if (skipPartitionId == p.getId()) {
+                LOG.info("Partition {} in table {} skipped", skipPartitionId, tbl.getName());
+                continue;
+            }
             long partitionRowCount = p.getRowCount();
             if (partitionRowCount >= averageRowsPerPartition) {
                 pickedRows += partitionRowCount;
diff --git a/fe/fe-core/src/test/java/org/apache/doris/statistics/OlapAnalysisTaskTest.java b/fe/fe-core/src/test/java/org/apache/doris/statistics/OlapAnalysisTaskTest.java
index 9bda917b34ea1a..1404855f0928f9 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/statistics/OlapAnalysisTaskTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/statistics/OlapAnalysisTaskTest.java
@@ -17,26 +17,34 @@
 
 package org.apache.doris.statistics;
 
+import org.apache.doris.analysis.PartitionValue;
 import org.apache.doris.analysis.TableSample;
 import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.DataProperty;
 import org.apache.doris.catalog.DatabaseIf;
 import org.apache.doris.catalog.MaterializedIndex;
 import org.apache.doris.catalog.OlapTable;
 import org.apache.doris.catalog.Partition;
 import org.apache.doris.catalog.PartitionInfo;
+import org.apache.doris.catalog.PartitionKey;
 import org.apache.doris.catalog.PartitionType;
 import org.apache.doris.catalog.PrimitiveType;
 import org.apache.doris.catalog.RandomDistributionInfo;
+import org.apache.doris.catalog.RangePartitionItem;
 import org.apache.doris.catalog.TableIf;
+import org.apache.doris.catalog.Tablet;
 import org.apache.doris.catalog.Type;
+import org.apache.doris.common.AnalysisException;
 import org.apache.doris.common.Pair;
 import org.apache.doris.datasource.CatalogIf;
 import org.apache.doris.statistics.AnalysisInfo.AnalysisMethod;
 import org.apache.doris.statistics.AnalysisInfo.JobType;
 import org.apache.doris.statistics.util.StatisticsUtil;
+import org.apache.doris.thrift.TStorageMedium;
 
 import com.google.common.collect.Lists;
 import com.google.common.collect.Maps;
+import com.google.common.collect.Range;
 import mockit.Expectations;
 import mockit.Mock;
 import mockit.MockUp;
@@ -45,6 +53,7 @@
 import org.junit.jupiter.api.Test;
 
 import java.util.ArrayList;
+import java.util.Collection;
 import java.util.List;
 import java.util.Map;
 
@@ -279,7 +288,7 @@ public List<Long> getTabletIdsInOrder() {
                 return ret;
             }
         };
-        long rows = task.pickSamplePartition(partitions, ids);
+        long rows = task.pickSamplePartition(partitions, ids, 0);
         Assertions.assertEquals(900000000, rows);
         Assertions.assertEquals(4, ids.size());
         Assertions.assertEquals(0, ids.get(0));
@@ -419,4 +428,228 @@ protected long getSampleRows() {
         Assertions.assertEquals("2000000000", params.get("ndvFunction"));
         Assertions.assertEquals("limit 1000000000", params.get("limit"));
     }
+
+    @Test
+    public void testGetSkipPartitionId(@Mocked OlapTable tableIf) throws AnalysisException {
+        // test null partition list
+        OlapAnalysisTask task = new OlapAnalysisTask();
+        long skipPartitionId = task.getSkipPartitionId(null);
+        Assertions.assertEquals(OlapAnalysisTask.NO_SKIP_TABLET_ID, skipPartitionId);
+
+        // test empty partition list
+        List<Partition> partitions = Lists.newArrayList();
+        skipPartitionId = task.getSkipPartitionId(partitions);
+        Assertions.assertEquals(OlapAnalysisTask.NO_SKIP_TABLET_ID, skipPartitionId);
+
+        // test partition list item less than session variable partition_sample_count
+        Partition p1 = new Partition(1, "p1", new MaterializedIndex(), new RandomDistributionInfo());
+        partitions.add(p1);
+        skipPartitionId = task.getSkipPartitionId(partitions);
+        Assertions.assertEquals(OlapAnalysisTask.NO_SKIP_TABLET_ID, skipPartitionId);
+
+        partitions.clear();
+        int partitionSampleCount = StatisticsUtil.getPartitionSampleCount();
+        for (int i = 1; i <= partitionSampleCount; i++) {
+            Partition p = new Partition(i, "p" + i, new MaterializedIndex(), new RandomDistributionInfo());
+            partitions.add(p);
+        }
+
+        // Test List partition return NO_SKIP_TABLET_ID
+        new MockUp<OlapTable>() {
+            @Mock
+            public PartitionInfo getPartitionInfo() {
+                return new PartitionInfo(PartitionType.LIST);
+            }
+        };
+        task.tbl = tableIf;
+        skipPartitionId = task.getSkipPartitionId(partitions);
+        Assertions.assertEquals(OlapAnalysisTask.NO_SKIP_TABLET_ID, skipPartitionId);
+
+        // Test Unpartition return NO_SKIP_TABLET_ID
+        new MockUp<OlapTable>() {
+            @Mock
+            public PartitionInfo getPartitionInfo() {
+                return new PartitionInfo(PartitionType.UNPARTITIONED);
+            }
+        };
+        skipPartitionId = task.getSkipPartitionId(partitions);
+        Assertions.assertEquals(OlapAnalysisTask.NO_SKIP_TABLET_ID, skipPartitionId);
+
+        // Test more than 1 partition column return NO_SKIP_TABLET_ID
+        new MockUp<OlapTable>() {
+            @Mock
+            public PartitionInfo getPartitionInfo() {
+                ArrayList<Column> columns = Lists.newArrayList();
+                columns.add(new Column("col1", PrimitiveType.DATEV2));
+                columns.add(new Column("col2", PrimitiveType.DATEV2));
+                return new PartitionInfo(PartitionType.RANGE, columns);
+            }
+        };
+        skipPartitionId = task.getSkipPartitionId(partitions);
+        Assertions.assertEquals(OlapAnalysisTask.NO_SKIP_TABLET_ID, skipPartitionId);
+
+        // Test not Date type return NO_SKIP_TABLET_ID
+        new MockUp<OlapTable>() {
+            @Mock
+            public PartitionInfo getPartitionInfo() {
+                ArrayList<Column> columns = Lists.newArrayList();
+                columns.add(new Column("col1", PrimitiveType.STRING));
+                return new PartitionInfo(PartitionType.RANGE, columns);
+            }
+        };
+        skipPartitionId = task.getSkipPartitionId(partitions);
+        Assertions.assertEquals(OlapAnalysisTask.NO_SKIP_TABLET_ID, skipPartitionId);
+
+        // Test return the partition id with the oldest date range.
+        ArrayList<Column> columns = Lists.newArrayList();
+        Column col1 = new Column("col1", PrimitiveType.DATEV2);
+        columns.add(col1);
+        PartitionInfo partitionInfo = new PartitionInfo(PartitionType.RANGE, columns);
+
+        List<PartitionValue> lowKey = Lists.newArrayList();
+        lowKey.add(new PartitionValue("2025-01-01"));
+        List<PartitionValue> highKey = Lists.newArrayList();
+        highKey.add(new PartitionValue("2025-01-02"));
+        Range<PartitionKey> range1 = Range.closedOpen(PartitionKey.createPartitionKey(lowKey, columns),
+                    PartitionKey.createPartitionKey(highKey, columns));
+        RangePartitionItem item1 = new RangePartitionItem(range1);
+
+        lowKey.clear();
+        lowKey.add(new PartitionValue("2024-11-01"));
+        highKey.clear();
+        highKey.add(new PartitionValue("2024-11-02"));
+        Range<PartitionKey> range2 = Range.closedOpen(PartitionKey.createPartitionKey(lowKey, columns),
+                PartitionKey.createPartitionKey(highKey, columns));
+        RangePartitionItem item2 = new RangePartitionItem(range2);
+
+        lowKey.clear();
+        lowKey.add(new PartitionValue("2025-02-13"));
+        highKey.clear();
+        highKey.add(new PartitionValue("2025-02-14"));
+        Range<PartitionKey> range3 = Range.closedOpen(PartitionKey.createPartitionKey(lowKey, columns),
+                PartitionKey.createPartitionKey(highKey, columns));
+        RangePartitionItem item3 = new RangePartitionItem(range3);
+
+        partitionInfo.addPartition(1, false, item1, new DataProperty(TStorageMedium.HDD), null, false, false);
+        partitionInfo.addPartition(2, false, item2, new DataProperty(TStorageMedium.HDD), null, false, false);
+        partitionInfo.addPartition(3, false, item3, new DataProperty(TStorageMedium.HDD), null, false, false);
+
+        new MockUp<OlapTable>() {
+            @Mock
+            public PartitionInfo getPartitionInfo() {
+                return partitionInfo;
+            }
+        };
+        new MockUp<StatisticsUtil>() {
+            @Mock
+            public int getPartitionSampleCount() {
+                return 3;
+            }
+        };
+        partitions.clear();
+        for (int i = 1; i <= 3; i++) {
+            Partition p = new Partition(i, "p" + i, new MaterializedIndex(), new RandomDistributionInfo());
+            partitions.add(p);
+        }
+        skipPartitionId = task.getSkipPartitionId(partitions);
+        Assertions.assertEquals(2, skipPartitionId);
+
+        // Test less than partition
+        partitions.add(new Partition(4, "p4", new MaterializedIndex(), new RandomDistributionInfo()));
+        partitions.add(new Partition(5, "p5", new MaterializedIndex(), new RandomDistributionInfo()));
+        new MockUp<StatisticsUtil>() {
+            @Mock
+            public int getPartitionSampleCount() {
+                return 5;
+            }
+        };
+        highKey.clear();
+        highKey.add(new PartitionValue("2024-01-01"));
+        Range<PartitionKey> range4 = Range.lessThan(PartitionKey.createPartitionKey(highKey, columns));
+        RangePartitionItem item4 = new RangePartitionItem(range4);
+        partitionInfo.addPartition(4, false, item4, new DataProperty(TStorageMedium.HDD), null, false, false);
+        lowKey.clear();
+        lowKey.add(new PartitionValue("2024-03-13"));
+        highKey.clear();
+        highKey.add(new PartitionValue("2024-03-14"));
+        Range<PartitionKey> range5 = Range.closedOpen(PartitionKey.createPartitionKey(lowKey, columns),
+                PartitionKey.createPartitionKey(highKey, columns));
+        RangePartitionItem item5 = new RangePartitionItem(range5);
+        partitionInfo.addPartition(5, false, item5, new DataProperty(TStorageMedium.HDD), null, false, false);
+        skipPartitionId = task.getSkipPartitionId(partitions);
+        Assertions.assertEquals(4, skipPartitionId);
+    }
+
+    @Test
+    public void testGetSampleTablets(@Mocked MaterializedIndex index, @Mocked Tablet t) {
+        OlapAnalysisTask task = new OlapAnalysisTask();
+        task.tbl = new OlapTable();
+        task.col = new Column("col1", PrimitiveType.STRING);
+        task.info = new AnalysisInfoBuilder().setIndexId(-1L).build();
+        task.tableSample = new TableSample(false, 4000000L, 0L);
+        List<Partition> partitions = Lists.newArrayList();
+        partitions.add(new Partition(1, "p1", new MaterializedIndex(), new RandomDistributionInfo()));
+        final int[] i = {0};
+        long[] tabletsRowCount = {1100000000, 100000000};
+        List<Long> ret = Lists.newArrayList();
+        ret.add(10001L);
+        ret.add(10002L);
+        new MockUp<OlapAnalysisTask>() {
+            @Mock
+            protected long getSampleRows() {
+                return 4000000;
+            }
+        };
+        new MockUp<OlapTable>() {
+            @Mock
+            boolean isPartitionColumn(String columnName) {
+                return false;
+            }
+
+            @Mock
+            public Collection<Partition> getPartitions() {
+                return partitions;
+            }
+        };
+        new MockUp<Partition>() {
+            @Mock
+            public MaterializedIndex getBaseIndex() {
+                return index;
+            }
+        };
+        new MockUp<MaterializedIndex>() {
+            @Mock
+            public List<Long> getTabletIdsInOrder() {
+                return ret;
+            }
+
+            @Mock
+            public long getRowCount() {
+                return 1_200_000_000L;
+            }
+
+            @Mock
+            public Tablet getTablet(long tabletId) {
+                return t;
+            }
+        };
+        new MockUp<Tablet>() {
+            @Mock
+            public long getMinReplicaRowCount(long version) {
+                return tabletsRowCount[i[0]++];
+            }
+        };
+        // Test set large tablet id back if it doesn't pick enough sample rows.
+        Pair<List<Long>, Long> sampleTablets = task.getSampleTablets();
+        Assertions.assertEquals(1, sampleTablets.first.size());
+        Assertions.assertEquals(10001, sampleTablets.first.get(0));
+        Assertions.assertEquals(1100000000L, sampleTablets.second);
+
+        // Test normal pick
+        task.tableSample = new TableSample(false, 4000000L, 1L);
+        sampleTablets = task.getSampleTablets();
+        Assertions.assertEquals(1, sampleTablets.first.size());
+        Assertions.assertEquals(10002, sampleTablets.first.get(0));
+        Assertions.assertEquals(100000000L, sampleTablets.second);
+    }
 }


Reviews:
user: login: morrySnow
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T07:14:21Z
user: login: Jibing-Li
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:25:53Z
user: login: Jibing-Li
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:26:24Z
user: login: Jibing-Li
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:29:11Z
user: login: Jibing-Li
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:31:28Z
user: login: Jibing-Li
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:38:04Z
user: login: Jibing-Li
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:42:13Z
user: login: morrySnow
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T11:38:04Z
user: login: zfr9527
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T02:39:10Z

Review comments:
diff_hunk: @@ -190,8 +199,16 @@ protected Pair<List<Long>, Long> getSampleTablets() {
             for (int i = 0; i < tabletCounts; i++) {
                 int seekTid = (int) ((i + seek) % ids.size());
                 long tabletId = ids.get(seekTid);
-                sampleTabletIds.add(tabletId);
                 long tabletRows = materializedIndex.getTablet(tabletId).getMinReplicaRowCount(p.getVisibleVersion());
+                if (tabletRows > MAXIMUM_SAMPLE_ROWS) {
+                    // Skip very large tablet.
+                    largeTabletId = tabletId;
path: fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
user: login: morrySnow
body: we should record the smallest one in the large tablet list
created_at: 2025-03-07T07:10:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -372,12 +396,64 @@ protected String getIndex() {
         }
     }
 
-    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds) {
-        long averageRowsPerPartition = tbl.getRowCount() / partitions.size();
+    // For partition tables with single time type partition column, we'd better to skip sampling the partition
+    // that contains all the history data. Because this partition may contain many old data which is not
+    // visited by most queries. To sample this partition may cause the statistics not accurate.
+    // For example, one table has 366 partitions, partition 1 ~ 365 store date for each day of the year from now.
+    // Partition 0 stores all the history data earlier than 1 year. We want to skip sampling partition 0.
+    protected long getSkipPartitionId(List<Partition> partitions) {
+        if (partitions == null || partitions.size() < StatisticsUtil.getPartitionSampleCount()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionInfo partitionInfo = ((OlapTable) tbl).getPartitionInfo();
+        if (!PartitionType.RANGE.equals(partitionInfo.getType())) {
+            return NO_SKIP_TABLET_ID;
+        }
+        if (partitionInfo.getPartitionColumns().size() != 1) {
+            return NO_SKIP_TABLET_ID;
+        }
+        Column column = partitionInfo.getPartitionColumns().get(0);
+        if (!column.getType().isDateType()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionKey lowestKey = PartitionKey.createMaxPartitionKey();
+        long lowestPartitionId = -1;
+        for (Partition p : partitions) {
+            RangePartitionItem item = (RangePartitionItem) partitionInfo.getItem(p.getId());
+            Range<PartitionKey> items = item.getItems();
+            if (!items.hasLowerBound()) {
+                lowestPartitionId = p.getId();
+                break;
+            }
+            if (items.lowerEndpoint().compareTo(lowestKey) < 0) {
+                lowestKey = items.lowerEndpoint();
+                lowestPartitionId = p.getId();
+            }
+        }
+        return lowestPartitionId;
+    }
+
+    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds, long skipPartition) {
path: fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
user: login: morrySnow
body: ```suggestion
    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds, long skipPartitionId) {
```
created_at: 2025-03-07T07:12:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -372,12 +396,64 @@ protected String getIndex() {
         }
     }
 
-    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds) {
-        long averageRowsPerPartition = tbl.getRowCount() / partitions.size();
+    // For partition tables with single time type partition column, we'd better to skip sampling the partition
+    // that contains all the history data. Because this partition may contain many old data which is not
+    // visited by most queries. To sample this partition may cause the statistics not accurate.
+    // For example, one table has 366 partitions, partition 1 ~ 365 store date for each day of the year from now.
+    // Partition 0 stores all the history data earlier than 1 year. We want to skip sampling partition 0.
+    protected long getSkipPartitionId(List<Partition> partitions) {
+        if (partitions == null || partitions.size() < StatisticsUtil.getPartitionSampleCount()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionInfo partitionInfo = ((OlapTable) tbl).getPartitionInfo();
+        if (!PartitionType.RANGE.equals(partitionInfo.getType())) {
+            return NO_SKIP_TABLET_ID;
+        }
+        if (partitionInfo.getPartitionColumns().size() != 1) {
+            return NO_SKIP_TABLET_ID;
+        }
+        Column column = partitionInfo.getPartitionColumns().get(0);
+        if (!column.getType().isDateType()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionKey lowestKey = PartitionKey.createMaxPartitionKey();
+        long lowestPartitionId = -1;
+        for (Partition p : partitions) {
+            RangePartitionItem item = (RangePartitionItem) partitionInfo.getItem(p.getId());
+            Range<PartitionKey> items = item.getItems();
+            if (!items.hasLowerBound()) {
+                lowestPartitionId = p.getId();
+                break;
+            }
+            if (items.lowerEndpoint().compareTo(lowestKey) < 0) {
+                lowestKey = items.lowerEndpoint();
+                lowestPartitionId = p.getId();
+            }
+        }
+        return lowestPartitionId;
+    }
+
+    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds, long skipPartition) {
+        Partition partition = ((OlapTable) tbl).getPartition(skipPartition);
+        long averageRowsPerPartition;
+        if (partition != null) {
+            LOG.info("Going to skip partition {} in table {}", skipPartition, tbl.getName());
path: fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
user: login: morrySnow
body: log debug is ok?
created_at: 2025-03-07T07:12:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -372,12 +396,64 @@ protected String getIndex() {
         }
     }
 
-    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds) {
-        long averageRowsPerPartition = tbl.getRowCount() / partitions.size();
+    // For partition tables with single time type partition column, we'd better to skip sampling the partition
+    // that contains all the history data. Because this partition may contain many old data which is not
+    // visited by most queries. To sample this partition may cause the statistics not accurate.
+    // For example, one table has 366 partitions, partition 1 ~ 365 store date for each day of the year from now.
+    // Partition 0 stores all the history data earlier than 1 year. We want to skip sampling partition 0.
+    protected long getSkipPartitionId(List<Partition> partitions) {
+        if (partitions == null || partitions.size() < StatisticsUtil.getPartitionSampleCount()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionInfo partitionInfo = ((OlapTable) tbl).getPartitionInfo();
+        if (!PartitionType.RANGE.equals(partitionInfo.getType())) {
+            return NO_SKIP_TABLET_ID;
+        }
+        if (partitionInfo.getPartitionColumns().size() != 1) {
+            return NO_SKIP_TABLET_ID;
+        }
+        Column column = partitionInfo.getPartitionColumns().get(0);
+        if (!column.getType().isDateType()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionKey lowestKey = PartitionKey.createMaxPartitionKey();
+        long lowestPartitionId = -1;
+        for (Partition p : partitions) {
+            RangePartitionItem item = (RangePartitionItem) partitionInfo.getItem(p.getId());
+            Range<PartitionKey> items = item.getItems();
+            if (!items.hasLowerBound()) {
+                lowestPartitionId = p.getId();
+                break;
+            }
+            if (items.lowerEndpoint().compareTo(lowestKey) < 0) {
+                lowestKey = items.lowerEndpoint();
+                lowestPartitionId = p.getId();
+            }
+        }
+        return lowestPartitionId;
+    }
+
+    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds, long skipPartition) {
+        Partition partition = ((OlapTable) tbl).getPartition(skipPartition);
+        long averageRowsPerPartition;
+        if (partition != null) {
+            LOG.info("Going to skip partition {} in table {}", skipPartition, tbl.getName());
+            // If we want to skip the oldest partition, calculate the average rows per partition value without
+            // the oldest partition, otherwise if the oldest partition is very large, we may skip all partitions.
+            // Because we only pick partitions partitionRowCount >= averageRowsPerPartition.
+            Preconditions.checkState(partitions.size() > 1);
path: fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
user: login: morrySnow
body: add check message to avoid get a NPE exception
created_at: 2025-03-07T07:12:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -372,12 +396,64 @@ protected String getIndex() {
         }
     }
 
-    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds) {
-        long averageRowsPerPartition = tbl.getRowCount() / partitions.size();
+    // For partition tables with single time type partition column, we'd better to skip sampling the partition
+    // that contains all the history data. Because this partition may contain many old data which is not
+    // visited by most queries. To sample this partition may cause the statistics not accurate.
+    // For example, one table has 366 partitions, partition 1 ~ 365 store date for each day of the year from now.
+    // Partition 0 stores all the history data earlier than 1 year. We want to skip sampling partition 0.
+    protected long getSkipPartitionId(List<Partition> partitions) {
+        if (partitions == null || partitions.size() < StatisticsUtil.getPartitionSampleCount()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionInfo partitionInfo = ((OlapTable) tbl).getPartitionInfo();
+        if (!PartitionType.RANGE.equals(partitionInfo.getType())) {
+            return NO_SKIP_TABLET_ID;
+        }
+        if (partitionInfo.getPartitionColumns().size() != 1) {
+            return NO_SKIP_TABLET_ID;
+        }
+        Column column = partitionInfo.getPartitionColumns().get(0);
+        if (!column.getType().isDateType()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionKey lowestKey = PartitionKey.createMaxPartitionKey();
+        long lowestPartitionId = -1;
+        for (Partition p : partitions) {
+            RangePartitionItem item = (RangePartitionItem) partitionInfo.getItem(p.getId());
+            Range<PartitionKey> items = item.getItems();
+            if (!items.hasLowerBound()) {
+                lowestPartitionId = p.getId();
+                break;
+            }
+            if (items.lowerEndpoint().compareTo(lowestKey) < 0) {
+                lowestKey = items.lowerEndpoint();
+                lowestPartitionId = p.getId();
+            }
+        }
+        return lowestPartitionId;
+    }
+
+    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds, long skipPartition) {
+        Partition partition = ((OlapTable) tbl).getPartition(skipPartition);
+        long averageRowsPerPartition;
+        if (partition != null) {
+            LOG.info("Going to skip partition {} in table {}", skipPartition, tbl.getName());
+            // If we want to skip the oldest partition, calculate the average rows per partition value without
+            // the oldest partition, otherwise if the oldest partition is very large, we may skip all partitions.
+            // Because we only pick partitions partitionRowCount >= averageRowsPerPartition.
+            Preconditions.checkState(partitions.size() > 1);
+            averageRowsPerPartition = (tbl.getRowCount() - partition.getRowCount()) / (partitions.size() - 1);
+        } else {
+            averageRowsPerPartition = tbl.getRowCount() / partitions.size();
+        }
         long indexId = info.indexId == -1 ? ((OlapTable) tbl).getBaseIndexId() : info.indexId;
         long pickedRows = 0;
         int pickedPartitionCount = 0;
         for (Partition p : partitions) {
+            if (skipPartition == p.getId()) {
+                LOG.info("Partition {} in table {} skipped", skipPartition, tbl.getName());
path: fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
user: login: morrySnow
body: debug?
created_at: 2025-03-07T07:12:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -372,12 +396,64 @@ protected String getIndex() {
         }
     }
 
-    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds) {
-        long averageRowsPerPartition = tbl.getRowCount() / partitions.size();
+    // For partition tables with single time type partition column, we'd better to skip sampling the partition
+    // that contains all the history data. Because this partition may contain many old data which is not
+    // visited by most queries. To sample this partition may cause the statistics not accurate.
+    // For example, one table has 366 partitions, partition 1 ~ 365 store date for each day of the year from now.
+    // Partition 0 stores all the history data earlier than 1 year. We want to skip sampling partition 0.
+    protected long getSkipPartitionId(List<Partition> partitions) {
+        if (partitions == null || partitions.size() < StatisticsUtil.getPartitionSampleCount()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionInfo partitionInfo = ((OlapTable) tbl).getPartitionInfo();
+        if (!PartitionType.RANGE.equals(partitionInfo.getType())) {
+            return NO_SKIP_TABLET_ID;
+        }
+        if (partitionInfo.getPartitionColumns().size() != 1) {
+            return NO_SKIP_TABLET_ID;
+        }
+        Column column = partitionInfo.getPartitionColumns().get(0);
+        if (!column.getType().isDateType()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionKey lowestKey = PartitionKey.createMaxPartitionKey();
+        long lowestPartitionId = -1;
+        for (Partition p : partitions) {
+            RangePartitionItem item = (RangePartitionItem) partitionInfo.getItem(p.getId());
+            Range<PartitionKey> items = item.getItems();
+            if (!items.hasLowerBound()) {
+                lowestPartitionId = p.getId();
+                break;
+            }
+            if (items.lowerEndpoint().compareTo(lowestKey) < 0) {
+                lowestKey = items.lowerEndpoint();
+                lowestPartitionId = p.getId();
+            }
+        }
+        return lowestPartitionId;
+    }
+
+    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds, long skipPartition) {
+        Partition partition = ((OlapTable) tbl).getPartition(skipPartition);
+        long averageRowsPerPartition;
+        if (partition != null) {
+            LOG.info("Going to skip partition {} in table {}", skipPartition, tbl.getName());
+            // If we want to skip the oldest partition, calculate the average rows per partition value without
+            // the oldest partition, otherwise if the oldest partition is very large, we may skip all partitions.
+            // Because we only pick partitions partitionRowCount >= averageRowsPerPartition.
+            Preconditions.checkState(partitions.size() > 1);
path: fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
user: login: morrySnow
body: how to ensure partition.size() always greater than 1?
created_at: 2025-03-07T07:13:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -190,8 +199,16 @@ protected Pair<List<Long>, Long> getSampleTablets() {
             for (int i = 0; i < tabletCounts; i++) {
                 int seekTid = (int) ((i + seek) % ids.size());
                 long tabletId = ids.get(seekTid);
-                sampleTabletIds.add(tabletId);
                 long tabletRows = materializedIndex.getTablet(tabletId).getMinReplicaRowCount(p.getVisibleVersion());
+                if (tabletRows > MAXIMUM_SAMPLE_ROWS) {
+                    // Skip very large tablet.
+                    largeTabletId = tabletId;
path: fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
user: login: Jibing-Li
body: Fixed
created_at: 2025-03-07T09:25:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -372,12 +396,64 @@ protected String getIndex() {
         }
     }
 
-    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds) {
-        long averageRowsPerPartition = tbl.getRowCount() / partitions.size();
+    // For partition tables with single time type partition column, we'd better to skip sampling the partition
+    // that contains all the history data. Because this partition may contain many old data which is not
+    // visited by most queries. To sample this partition may cause the statistics not accurate.
+    // For example, one table has 366 partitions, partition 1 ~ 365 store date for each day of the year from now.
+    // Partition 0 stores all the history data earlier than 1 year. We want to skip sampling partition 0.
+    protected long getSkipPartitionId(List<Partition> partitions) {
+        if (partitions == null || partitions.size() < StatisticsUtil.getPartitionSampleCount()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionInfo partitionInfo = ((OlapTable) tbl).getPartitionInfo();
+        if (!PartitionType.RANGE.equals(partitionInfo.getType())) {
+            return NO_SKIP_TABLET_ID;
+        }
+        if (partitionInfo.getPartitionColumns().size() != 1) {
+            return NO_SKIP_TABLET_ID;
+        }
+        Column column = partitionInfo.getPartitionColumns().get(0);
+        if (!column.getType().isDateType()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionKey lowestKey = PartitionKey.createMaxPartitionKey();
+        long lowestPartitionId = -1;
+        for (Partition p : partitions) {
+            RangePartitionItem item = (RangePartitionItem) partitionInfo.getItem(p.getId());
+            Range<PartitionKey> items = item.getItems();
+            if (!items.hasLowerBound()) {
+                lowestPartitionId = p.getId();
+                break;
+            }
+            if (items.lowerEndpoint().compareTo(lowestKey) < 0) {
+                lowestKey = items.lowerEndpoint();
+                lowestPartitionId = p.getId();
+            }
+        }
+        return lowestPartitionId;
+    }
+
+    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds, long skipPartition) {
path: fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
user: login: Jibing-Li
body: fixed
created_at: 2025-03-07T09:26:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -372,12 +396,64 @@ protected String getIndex() {
         }
     }
 
-    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds) {
-        long averageRowsPerPartition = tbl.getRowCount() / partitions.size();
+    // For partition tables with single time type partition column, we'd better to skip sampling the partition
+    // that contains all the history data. Because this partition may contain many old data which is not
+    // visited by most queries. To sample this partition may cause the statistics not accurate.
+    // For example, one table has 366 partitions, partition 1 ~ 365 store date for each day of the year from now.
+    // Partition 0 stores all the history data earlier than 1 year. We want to skip sampling partition 0.
+    protected long getSkipPartitionId(List<Partition> partitions) {
+        if (partitions == null || partitions.size() < StatisticsUtil.getPartitionSampleCount()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionInfo partitionInfo = ((OlapTable) tbl).getPartitionInfo();
+        if (!PartitionType.RANGE.equals(partitionInfo.getType())) {
+            return NO_SKIP_TABLET_ID;
+        }
+        if (partitionInfo.getPartitionColumns().size() != 1) {
+            return NO_SKIP_TABLET_ID;
+        }
+        Column column = partitionInfo.getPartitionColumns().get(0);
+        if (!column.getType().isDateType()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionKey lowestKey = PartitionKey.createMaxPartitionKey();
+        long lowestPartitionId = -1;
+        for (Partition p : partitions) {
+            RangePartitionItem item = (RangePartitionItem) partitionInfo.getItem(p.getId());
+            Range<PartitionKey> items = item.getItems();
+            if (!items.hasLowerBound()) {
+                lowestPartitionId = p.getId();
+                break;
+            }
+            if (items.lowerEndpoint().compareTo(lowestKey) < 0) {
+                lowestKey = items.lowerEndpoint();
+                lowestPartitionId = p.getId();
+            }
+        }
+        return lowestPartitionId;
+    }
+
+    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds, long skipPartition) {
+        Partition partition = ((OlapTable) tbl).getPartition(skipPartition);
+        long averageRowsPerPartition;
+        if (partition != null) {
+            LOG.info("Going to skip partition {} in table {}", skipPartition, tbl.getName());
path: fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
user: login: Jibing-Li
body: fixed
created_at: 2025-03-07T09:29:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -372,12 +396,64 @@ protected String getIndex() {
         }
     }
 
-    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds) {
-        long averageRowsPerPartition = tbl.getRowCount() / partitions.size();
+    // For partition tables with single time type partition column, we'd better to skip sampling the partition
+    // that contains all the history data. Because this partition may contain many old data which is not
+    // visited by most queries. To sample this partition may cause the statistics not accurate.
+    // For example, one table has 366 partitions, partition 1 ~ 365 store date for each day of the year from now.
+    // Partition 0 stores all the history data earlier than 1 year. We want to skip sampling partition 0.
+    protected long getSkipPartitionId(List<Partition> partitions) {
+        if (partitions == null || partitions.size() < StatisticsUtil.getPartitionSampleCount()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionInfo partitionInfo = ((OlapTable) tbl).getPartitionInfo();
+        if (!PartitionType.RANGE.equals(partitionInfo.getType())) {
+            return NO_SKIP_TABLET_ID;
+        }
+        if (partitionInfo.getPartitionColumns().size() != 1) {
+            return NO_SKIP_TABLET_ID;
+        }
+        Column column = partitionInfo.getPartitionColumns().get(0);
+        if (!column.getType().isDateType()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionKey lowestKey = PartitionKey.createMaxPartitionKey();
+        long lowestPartitionId = -1;
+        for (Partition p : partitions) {
+            RangePartitionItem item = (RangePartitionItem) partitionInfo.getItem(p.getId());
+            Range<PartitionKey> items = item.getItems();
+            if (!items.hasLowerBound()) {
+                lowestPartitionId = p.getId();
+                break;
+            }
+            if (items.lowerEndpoint().compareTo(lowestKey) < 0) {
+                lowestKey = items.lowerEndpoint();
+                lowestPartitionId = p.getId();
+            }
+        }
+        return lowestPartitionId;
+    }
+
+    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds, long skipPartition) {
+        Partition partition = ((OlapTable) tbl).getPartition(skipPartition);
+        long averageRowsPerPartition;
+        if (partition != null) {
+            LOG.info("Going to skip partition {} in table {}", skipPartition, tbl.getName());
+            // If we want to skip the oldest partition, calculate the average rows per partition value without
+            // the oldest partition, otherwise if the oldest partition is very large, we may skip all partitions.
+            // Because we only pick partitions partitionRowCount >= averageRowsPerPartition.
+            Preconditions.checkState(partitions.size() > 1);
+            averageRowsPerPartition = (tbl.getRowCount() - partition.getRowCount()) / (partitions.size() - 1);
+        } else {
+            averageRowsPerPartition = tbl.getRowCount() / partitions.size();
+        }
         long indexId = info.indexId == -1 ? ((OlapTable) tbl).getBaseIndexId() : info.indexId;
         long pickedRows = 0;
         int pickedPartitionCount = 0;
         for (Partition p : partitions) {
+            if (skipPartition == p.getId()) {
+                LOG.info("Partition {} in table {} skipped", skipPartition, tbl.getName());
path: fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
user: login: Jibing-Li
body: I think we can keep it as info. Because there couldn't be too much of this log, at most one log per table, and it's probably useful when debuging.
created_at: 2025-03-07T09:31:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -372,12 +396,64 @@ protected String getIndex() {
         }
     }
 
-    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds) {
-        long averageRowsPerPartition = tbl.getRowCount() / partitions.size();
+    // For partition tables with single time type partition column, we'd better to skip sampling the partition
+    // that contains all the history data. Because this partition may contain many old data which is not
+    // visited by most queries. To sample this partition may cause the statistics not accurate.
+    // For example, one table has 366 partitions, partition 1 ~ 365 store date for each day of the year from now.
+    // Partition 0 stores all the history data earlier than 1 year. We want to skip sampling partition 0.
+    protected long getSkipPartitionId(List<Partition> partitions) {
+        if (partitions == null || partitions.size() < StatisticsUtil.getPartitionSampleCount()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionInfo partitionInfo = ((OlapTable) tbl).getPartitionInfo();
+        if (!PartitionType.RANGE.equals(partitionInfo.getType())) {
+            return NO_SKIP_TABLET_ID;
+        }
+        if (partitionInfo.getPartitionColumns().size() != 1) {
+            return NO_SKIP_TABLET_ID;
+        }
+        Column column = partitionInfo.getPartitionColumns().get(0);
+        if (!column.getType().isDateType()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionKey lowestKey = PartitionKey.createMaxPartitionKey();
+        long lowestPartitionId = -1;
+        for (Partition p : partitions) {
+            RangePartitionItem item = (RangePartitionItem) partitionInfo.getItem(p.getId());
+            Range<PartitionKey> items = item.getItems();
+            if (!items.hasLowerBound()) {
+                lowestPartitionId = p.getId();
+                break;
+            }
+            if (items.lowerEndpoint().compareTo(lowestKey) < 0) {
+                lowestKey = items.lowerEndpoint();
+                lowestPartitionId = p.getId();
+            }
+        }
+        return lowestPartitionId;
+    }
+
+    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds, long skipPartition) {
+        Partition partition = ((OlapTable) tbl).getPartition(skipPartition);
+        long averageRowsPerPartition;
+        if (partition != null) {
+            LOG.info("Going to skip partition {} in table {}", skipPartition, tbl.getName());
+            // If we want to skip the oldest partition, calculate the average rows per partition value without
+            // the oldest partition, otherwise if the oldest partition is very large, we may skip all partitions.
+            // Because we only pick partitions partitionRowCount >= averageRowsPerPartition.
+            Preconditions.checkState(partitions.size() > 1);
path: fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
user: login: Jibing-Li
body: added
created_at: 2025-03-07T09:38:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -372,12 +396,64 @@ protected String getIndex() {
         }
     }
 
-    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds) {
-        long averageRowsPerPartition = tbl.getRowCount() / partitions.size();
+    // For partition tables with single time type partition column, we'd better to skip sampling the partition
+    // that contains all the history data. Because this partition may contain many old data which is not
+    // visited by most queries. To sample this partition may cause the statistics not accurate.
+    // For example, one table has 366 partitions, partition 1 ~ 365 store date for each day of the year from now.
+    // Partition 0 stores all the history data earlier than 1 year. We want to skip sampling partition 0.
+    protected long getSkipPartitionId(List<Partition> partitions) {
+        if (partitions == null || partitions.size() < StatisticsUtil.getPartitionSampleCount()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionInfo partitionInfo = ((OlapTable) tbl).getPartitionInfo();
+        if (!PartitionType.RANGE.equals(partitionInfo.getType())) {
+            return NO_SKIP_TABLET_ID;
+        }
+        if (partitionInfo.getPartitionColumns().size() != 1) {
+            return NO_SKIP_TABLET_ID;
+        }
+        Column column = partitionInfo.getPartitionColumns().get(0);
+        if (!column.getType().isDateType()) {
+            return NO_SKIP_TABLET_ID;
+        }
+        PartitionKey lowestKey = PartitionKey.createMaxPartitionKey();
+        long lowestPartitionId = -1;
+        for (Partition p : partitions) {
+            RangePartitionItem item = (RangePartitionItem) partitionInfo.getItem(p.getId());
+            Range<PartitionKey> items = item.getItems();
+            if (!items.hasLowerBound()) {
+                lowestPartitionId = p.getId();
+                break;
+            }
+            if (items.lowerEndpoint().compareTo(lowestKey) < 0) {
+                lowestKey = items.lowerEndpoint();
+                lowestPartitionId = p.getId();
+            }
+        }
+        return lowestPartitionId;
+    }
+
+    protected long pickSamplePartition(List<Partition> partitions, List<Long> pickedTabletIds, long skipPartition) {
+        Partition partition = ((OlapTable) tbl).getPartition(skipPartition);
+        long averageRowsPerPartition;
+        if (partition != null) {
+            LOG.info("Going to skip partition {} in table {}", skipPartition, tbl.getName());
+            // If we want to skip the oldest partition, calculate the average rows per partition value without
+            // the oldest partition, otherwise if the oldest partition is very large, we may skip all partitions.
+            // Because we only pick partitions partitionRowCount >= averageRowsPerPartition.
+            Preconditions.checkState(partitions.size() > 1);
path: fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
user: login: Jibing-Li
body: In getSkipPartitionId function. We have minimum number of partitions threshold to skip the oldest partition. Or the result may not good enough. For example, if there are only 2 partitions and we skip one of them.
created_at: 2025-03-07T09:42:13Z
author_association: CONTRIBUTOR

PR comments:
user: login: hello-stephen
created_at: 2025-03-06T08:59:43Z
body: 
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).

Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
    
author_association: CONTRIBUTOR
user: login: Jibing-Li
created_at: 2025-03-06T09:02:59Z
body: run buildall
author_association: CONTRIBUTOR
user: login: doris-robot
created_at: 2025-03-06T09:41:39Z
body: 
<details>
<summary>TPC-H: <b>Total hot run time: 32492 ms</b></summary>

```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f33e898f8658b2e6202e9570bf737ac3f987e472, data reload: false

------ Round 1 ----------------------------------
q1	17624	5198	5184	5184
q2	2053	317	179	179
q3	10592	1286	778	778
q4	10219	1041	543	543
q5	7532	2429	2310	2310
q6	190	175	135	135
q7	895	745	600	600
q8	9904	1281	1087	1087
q9	4888	4578	4697	4578
q10	6822	2296	1894	1894
q11	493	277	258	258
q12	350	351	224	224
q13	17801	3722	3079	3079
q14	227	220	206	206
q15	546	516	476	476
q16	628	614	596	596
q17	596	846	337	337
q18	6847	6328	6289	6289
q19	1817	944	547	547
q20	325	320	200	200
q21	2813	2246	2004	2004
q22	1053	995	988	988
Total cold run time: 104215 ms
Total hot run time: 32492 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5336	5124	5117	5117
q2	238	333	239	239
q3	2151	2646	2247	2247
q4	1470	1829	1387	1387
q5	4269	4112	4133	4112
q6	207	165	124	124
q7	1931	1952	1777	1777
q8	2600	2625	2540	2540
q9	7262	7334	7160	7160
q10	3030	3229	2798	2798
q11	600	508	515	508
q12	706	738	623	623
q13	3413	3966	3271	3271
q14	278	304	269	269
q15	536	488	491	488
q16	628	677	634	634
q17	1144	1653	1326	1326
q18	7684	7714	7536	7536
q19	848	846	921	846
q20	1967	2027	1887	1887
q21	5433	4928	4833	4833
q22	1115	1063	1071	1063
Total cold run time: 52846 ms
Total hot run time: 50785 ms
```
</details>

author_association: NONE
user: login: doris-robot
created_at: 2025-03-06T09:53:50Z
body: 
<details>
<summary>TPC-DS: <b>Total hot run time: 191217 ms</b></summary>

```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f33e898f8658b2e6202e9570bf737ac3f987e472, data reload: false

query1	1429	1052	993	993
query2	6221	1895	1855	1855
query3	11146	4491	4564	4491
query4	54186	24464	22998	22998
query5	5329	530	489	489
query6	375	203	198	198
query7	5167	499	288	288
query8	326	244	237	237
query9	6610	2573	2550	2550
query10	447	309	258	258
query11	15191	15080	14813	14813
query12	157	108	108	108
query13	1177	523	400	400
query14	10514	6465	6576	6465
query15	202	198	191	191
query16	7081	694	490	490
query17	1104	716	586	586
query18	1530	420	330	330
query19	206	202	173	173
query20	135	130	125	125
query21	215	168	115	115
query22	4449	4552	4257	4257
query23	34010	33477	33398	33398
query24	5964	2432	2411	2411
query25	443	462	411	411
query26	739	279	154	154
query27	2104	478	341	341
query28	2816	2442	2428	2428
query29	591	563	434	434
query30	277	232	193	193
query31	874	866	822	822
query32	73	60	65	60
query33	468	378	309	309
query34	770	954	511	511
query35	826	864	774	774
query36	925	1004	897	897
query37	123	105	74	74
query38	4354	4365	4175	4175
query39	1499	1445	1437	1437
query40	213	113	100	100
query41	63	50	50	50
query42	135	110	108	108
query43	524	514	459	459
query44	1321	801	791	791
query45	185	175	164	164
query46	872	1065	659	659
query47	1826	1843	1772	1772
query48	383	427	325	325
query49	729	528	456	456
query50	712	742	428	428
query51	4241	4259	4292	4259
query52	114	111	99	99
query53	241	278	196	196
query54	494	492	414	414
query55	87	85	82	82
query56	301	272	272	272
query57	1156	1190	1147	1147
query58	280	236	248	236
query59	2794	2825	2670	2670
query60	283	273	272	272
query61	136	120	116	116
query62	731	739	690	690
query63	234	186	192	186
query64	2268	1054	681	681
query65	4524	4449	4373	4373
query66	750	385	294	294
query67	15797	15431	15327	15327
query68	7385	880	503	503
query69	532	285	274	274
query70	1164	1069	1105	1069
query71	481	298	283	283
query72	5971	3584	3729	3584
query73	1202	748	337	337
query74	9032	9173	8665	8665
query75	3812	3165	2672	2672
query76	4167	1190	743	743
query77	578	371	283	283
query78	10139	10043	9315	9315
query79	2519	826	600	600
query80	740	511	454	454
query81	475	255	216	216
query82	480	127	95	95
query83	280	219	159	159
query84	287	96	73	73
query85	798	357	319	319
query86	399	262	264	262
query87	4391	4551	4398	4398
query88	3451	2220	2207	2207
query89	406	307	291	291
query90	1906	206	208	206
query91	149	136	109	109
query92	76	58	60	58
query93	1740	1039	573	573
query94	660	424	295	295
query95	342	266	257	257
query96	481	569	271	271
query97	3323	3402	3273	3273
query98	221	205	203	203
query99	1433	1390	1305	1305
Total cold run time: 300467 ms
Total hot run time: 191217 ms
```
</details>

author_association: NONE
user: login: doris-robot
created_at: 2025-03-06T09:59:10Z
body: 
<details>
<summary>ClickBench: <b>Total hot run time: 31.53 s</b></summary>

```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f33e898f8658b2e6202e9570bf737ac3f987e472, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.11	0.10
query5	0.55	0.56	0.55
query6	1.20	0.72	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.51	0.54
query10	0.57	0.58	0.57
query11	0.15	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	2.80	2.69	2.80
query15	0.94	0.84	0.85
query16	0.39	0.37	0.37
query17	1.04	1.02	1.01
query18	0.21	0.20	0.20
query19	1.86	1.82	1.97
query20	0.01	0.01	0.01
query21	15.35	0.87	0.54
query22	0.77	1.08	0.61
query23	15.05	1.40	0.61
query24	6.99	1.72	1.42
query25	0.54	0.32	0.13
query26	0.53	0.16	0.15
query27	0.06	0.05	0.05
query28	10.16	0.84	0.43
query29	12.61	4.04	3.40
query30	0.25	0.10	0.07
query31	2.81	0.58	0.38
query32	3.23	0.55	0.46
query33	2.97	3.04	3.03
query34	15.76	5.14	4.54
query35	4.58	4.55	4.58
query36	0.68	0.49	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.03
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.91 s
Total hot run time: 31.53 s
```
</details>

author_association: NONE
user: login: github-actions[bot]
created_at: 2025-03-10T11:38:54Z
body: PR approved by at least one committer and no changes requested.
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-10T11:38:56Z
body: PR approved by anyone and no changes requested.
author_association: CONTRIBUTOR
user: login: Jibing-Li
created_at: 2025-03-10T14:06:37Z
body: run buildall
author_association: CONTRIBUTOR
user: login: doris-robot
created_at: 2025-03-10T14:26:10Z
body: 
<details>
<summary>TPC-H: <b>Total hot run time: 32896 ms</b></summary>

```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eff237ccbaf17d47ac800a2921d0daa8dd36e769, data reload: false

------ Round 1 ----------------------------------
q1	17656	5280	5119	5119
q2	2045	297	172	172
q3	10406	1405	696	696
q4	10250	1015	538	538
q5	8273	2967	2402	2402
q6	201	166	135	135
q7	901	759	605	605
q8	9300	1320	1151	1151
q9	5062	4689	4629	4629
q10	6840	2328	1922	1922
q11	467	274	254	254
q12	345	359	220	220
q13	17770	3691	3159	3159
q14	235	236	212	212
q15	538	472	496	472
q16	652	613	581	581
q17	583	874	346	346
q18	6846	6520	6465	6465
q19	1630	969	599	599
q20	324	330	196	196
q21	2791	2295	2006	2006
q22	1055	1033	1017	1017
Total cold run time: 104170 ms
Total hot run time: 32896 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5246	5191	5458	5191
q2	245	340	229	229
q3	2172	2709	2269	2269
q4	1478	1807	1370	1370
q5	4261	4155	4183	4155
q6	209	173	133	133
q7	2069	1913	1798	1798
q8	2647	2498	2560	2498
q9	7183	7140	7294	7140
q10	2998	3210	2794	2794
q11	595	511	505	505
q12	707	815	633	633
q13	3545	3848	3285	3285
q14	284	299	272	272
q15	518	476	490	476
q16	664	700	626	626
q17	1136	1652	1305	1305
q18	7725	7636	7571	7571
q19	837	896	1096	896
q20	1940	2073	1872	1872
q21	5544	5138	4718	4718
q22	1064	1051	1016	1016
Total cold run time: 53067 ms
Total hot run time: 50752 ms
```
</details>

author_association: NONE
user: login: doris-robot
created_at: 2025-03-10T14:37:46Z
body: 
<details>
<summary>TPC-DS: <b>Total hot run time: 186559 ms</b></summary>

```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eff237ccbaf17d47ac800a2921d0daa8dd36e769, data reload: false

query1	1005	386	388	386
query2	6541	1926	1887	1887
query3	6805	219	220	219
query4	26290	23840	23581	23581
query5	4335	675	523	523
query6	317	207	188	188
query7	4618	502	305	305
query8	293	250	242	242
query9	8622	2663	2652	2652
query10	457	310	252	252
query11	15440	15149	15095	15095
query12	162	111	107	107
query13	1665	538	407	407
query14	9210	6887	6437	6437
query15	227	191	179	179
query16	7795	674	505	505
query17	1222	745	606	606
query18	1994	417	318	318
query19	193	192	161	161
query20	127	120	120	120
query21	228	132	107	107
query22	4263	4246	4230	4230
query23	34076	32967	33100	32967
query24	7675	2408	2423	2408
query25	531	456	397	397
query26	1219	273	156	156
query27	2110	500	342	342
query28	3873	2452	2419	2419
query29	723	552	444	444
query30	287	219	194	194
query31	919	844	771	771
query32	77	66	67	66
query33	557	358	300	300
query34	800	847	499	499
query35	795	851	766	766
query36	954	984	872	872
query37	120	99	80	80
query38	4281	4183	4060	4060
query39	1456	1404	1387	1387
query40	219	125	105	105
query41	56	52	63	52
query42	131	104	106	104
query43	488	506	470	470
query44	1336	814	810	810
query45	178	171	166	166
query46	847	1031	629	629
query47	1743	1787	1693	1693
query48	373	422	308	308
query49	788	502	446	446
query50	709	733	412	412
query51	4161	4217	4191	4191
query52	112	110	99	99
query53	236	263	192	192
query54	497	503	427	427
query55	84	87	88	87
query56	305	274	251	251
query57	1127	1148	1083	1083
query58	246	238	241	238
query59	2613	2789	2789	2789
query60	292	291	258	258
query61	129	123	127	123
query62	816	824	641	641
query63	225	190	229	190
query64	4314	1023	688	688
query65	4447	4340	4352	4340
query66	1089	427	312	312
query67	15718	15358	15066	15066
query68	8707	896	515	515
query69	469	307	266	266
query70	1213	1146	1127	1127
query71	467	303	288	288
query72	5304	3480	3703	3480
query73	775	732	355	355
query74	9174	9135	8943	8943
query75	3969	3174	2714	2714
query76	3717	1275	756	756
query77	791	389	295	295
query78	10117	10313	9295	9295
query79	2584	853	585	585
query80	628	527	445	445
query81	504	261	224	224
query82	721	128	98	98
query83	180	169	151	151
query84	237	96	74	74
query85	804	362	319	319
query86	392	292	285	285
query87	4568	4467	4341	4341
query88	3749	2250	2270	2250
query89	401	329	293	293
query90	1820	225	218	218
query91	147	142	110	110
query92	83	64	60	60
query93	1738	1077	585	585
query94	702	433	309	309
query95	371	275	272	272
query96	484	566	273	273
query97	3360	3427	3324	3324
query98	220	208	204	204
query99	1641	1402	1300	1300
Total cold run time: 275578 ms
Total hot run time: 186559 ms
```
</details>

author_association: NONE
user: login: doris-robot
created_at: 2025-03-10T14:43:05Z
body: 
<details>
<summary>ClickBench: <b>Total hot run time: 31.21 s</b></summary>

```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eff237ccbaf17d47ac800a2921d0daa8dd36e769, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.03
query3	0.23	0.06	0.06
query4	1.62	0.10	0.11
query5	0.55	0.56	0.54
query6	1.19	0.72	0.72
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.51	0.53
query10	0.58	0.60	0.58
query11	0.16	0.10	0.11
query12	0.14	0.12	0.12
query13	0.61	0.61	0.60
query14	2.70	2.82	2.69
query15	0.91	0.85	0.84
query16	0.40	0.39	0.38
query17	1.04	1.03	1.01
query18	0.22	0.20	0.20
query19	1.93	1.84	1.98
query20	0.01	0.01	0.01
query21	15.35	0.92	0.55
query22	0.76	1.23	0.74
query23	14.82	1.39	0.64
query24	6.81	2.27	0.89
query25	0.54	0.31	0.07
query26	0.55	0.16	0.13
query27	0.04	0.05	0.04
query28	10.02	0.85	0.45
query29	12.52	4.02	3.38
query30	0.25	0.09	0.07
query31	2.82	0.59	0.38
query32	3.24	0.56	0.47
query33	3.08	3.09	3.06
query34	15.81	5.15	4.58
query35	4.56	4.58	4.59
query36	0.67	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.4 s
Total hot run time: 31.21 s
```
</details>

author_association: NONE