Repository: apache/lucene
Pull request #14443: Support adaptive refresh in Searcher Managers.
Author: login: vigyasharma
Labels: name: module:core/search
name: module:facet
Created: 2025-04-05T22:39:32Z | Merged: 2025-05-18T17:43:57Z

In segment based replication systems, a large replication payload (checkpoint) can induce heavy page faults, cause thrashing for in-flight search requests, and affect overall search performance. 

A potential way to handle these bursts, is to leverage multiple commit points in the Lucene index. Instead of refreshing to the latest commit for a large replication payload, searchers can intelligently select the commit point that they can safely absorb. By processing through multiple such points, searchers can eventually get to the latest commit, without incurring too many page faults.

This change lets users define a commit selection strategy, controlling which commit the searcher manager refreshes on. Addresses #14219 


**Usage:**
To incrementally refresh through multiple commit points until searcher is current with its directory:
- Define a commit selection strategy using the `RefreshCommitSupplier` interface.
- Update searcher managers with this strategy via `setRefreshCommitSupplier()`
- Invoke `maybeRefresh()` or `maybeRefreshBlocking` in a loop until `isSearcherCurrent()` returns true.




Diff:
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index f9f036eb1040..5b5e061a30e6 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -90,6 +90,8 @@ Improvements
 * GITHUB#14602: Refactor the expressions compiler to use official ClassData BSM with indexed lookup
   (Uwe Schindler)
 
+* GITHUB#14443: Support adaptive refresh in Searcher Managers (Vigya Sharma)
+
 Optimizations
 ---------------------
 * GITHUB#14418: Quick exit on filter query matching no docs when rewriting knn query. (Pan Guixin)
diff --git a/lucene/core/src/java/org/apache/lucene/search/RefreshCommitSupplier.java b/lucene/core/src/java/org/apache/lucene/search/RefreshCommitSupplier.java
new file mode 100644
index 000000000000..9c95a620d528
--- /dev/null
+++ b/lucene/core/src/java/org/apache/lucene/search/RefreshCommitSupplier.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.search;
+
+import java.io.IOException;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexCommit;
+
+/**
+ * Expert: Interface to supply commit for searcher refresh.
+ *
+ * @lucene.experimental
+ */
+public interface RefreshCommitSupplier {
+
+  /**
+   * Expert: Returns the index commit that searcher should refresh on. A null return value (default)
+   * indicates reader should refresh on the latest commit.
+   *
+   * @param reader DirectoryReader to refresh
+   */
+  default IndexCommit getSearcherRefreshCommit(DirectoryReader reader) throws IOException {
+    return null;
+  }
+}
diff --git a/lucene/core/src/java/org/apache/lucene/search/SearcherManager.java b/lucene/core/src/java/org/apache/lucene/search/SearcherManager.java
index 072c13d92746..2d3c9f84d2db 100644
--- a/lucene/core/src/java/org/apache/lucene/search/SearcherManager.java
+++ b/lucene/core/src/java/org/apache/lucene/search/SearcherManager.java
@@ -18,6 +18,7 @@
 
 import java.io.IOException;
 import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexCommit;
 import org.apache.lucene.index.IndexReader;
 import org.apache.lucene.index.IndexWriter;
 import org.apache.lucene.store.Directory;
@@ -52,6 +53,7 @@
 public final class SearcherManager extends ReferenceManager<IndexSearcher> {
 
   private final SearcherFactory searcherFactory;
+  private RefreshCommitSupplier refreshCommitSupplier = new RefreshCommitSupplier() {};
 
   /**
    * Creates and returns a new SearcherManager from the given {@link IndexWriter}.
@@ -88,6 +90,34 @@ public SearcherManager(
       boolean writeAllDeletes,
       SearcherFactory searcherFactory)
       throws IOException {
+    this(writer, applyAllDeletes, writeAllDeletes, searcherFactory, null);
+  }
+
+  /**
+   * Expert: creates and returns a new SearcherManager from the given {@link IndexWriter},
+   * controlling whether past deletions should be applied.
+   *
+   * @param writer the IndexWriter to open the IndexReader from.
+   * @param applyAllDeletes If <code>true</code>, all buffered deletes will be applied (made
+   *     visible) in the {@link IndexSearcher} / {@link DirectoryReader}. If <code>false</code>, the
+   *     deletes may or may not be applied, but remain buffered (in IndexWriter) so that they will
+   *     be applied in the future. Applying deletes can be costly, so if your app can tolerate
+   *     deleted documents being returned you might gain some performance by passing <code>false
+   *     </code>. See {@link DirectoryReader#openIfChanged(DirectoryReader, IndexWriter, boolean)}.
+   * @param writeAllDeletes If <code>true</code>, new deletes will be forcefully written to index
+   *     files.
+   * @param searcherFactory An optional {@link SearcherFactory}. Pass <code>null</code> if you don't
+   *     require the searcher to be warmed before going live or other custom behavior.
+   * @param refreshCommitSupplier supplier for providing the commit to refresh on
+   * @throws IOException if there is a low-level I/O error
+   */
+  public SearcherManager(
+      IndexWriter writer,
+      boolean applyAllDeletes,
+      boolean writeAllDeletes,
+      SearcherFactory searcherFactory,
+      RefreshCommitSupplier refreshCommitSupplier)
+      throws IOException {
     if (searcherFactory == null) {
       searcherFactory = new SearcherFactory();
     }
@@ -95,6 +125,9 @@ public SearcherManager(
     current =
         getSearcher(
             searcherFactory, DirectoryReader.open(writer, applyAllDeletes, writeAllDeletes), null);
+    if (refreshCommitSupplier != null) {
+      this.refreshCommitSupplier = refreshCommitSupplier;
+    }
   }
 
   /**
@@ -124,11 +157,32 @@ public SearcherManager(Directory dir, SearcherFactory searcherFactory) throws IO
    */
   public SearcherManager(DirectoryReader reader, SearcherFactory searcherFactory)
       throws IOException {
+    this(reader, searcherFactory, null);
+  }
+
+  /**
+   * Creates and returns a new SearcherManager from an existing {@link DirectoryReader}. Note that
+   * this steals the incoming reference.
+   *
+   * @param reader the DirectoryReader.
+   * @param searcherFactory An optional {@link SearcherFactory}. Pass <code>null</code> if you don't
+   *     require the searcher to be warmed before going live or other custom behavior.
+   * @param refreshCommitSupplier supplier for providing the commit to refresh on
+   * @throws IOException if there is a low-level I/O error
+   */
+  public SearcherManager(
+      DirectoryReader reader,
+      SearcherFactory searcherFactory,
+      RefreshCommitSupplier refreshCommitSupplier)
+      throws IOException {
     if (searcherFactory == null) {
       searcherFactory = new SearcherFactory();
     }
     this.searcherFactory = searcherFactory;
     this.current = getSearcher(searcherFactory, reader, null);
+    if (refreshCommitSupplier != null) {
+      this.refreshCommitSupplier = refreshCommitSupplier;
+    }
   }
 
   @Override
@@ -141,7 +195,9 @@ protected IndexSearcher refreshIfNeeded(IndexSearcher referenceToRefresh) throws
     final IndexReader r = referenceToRefresh.getIndexReader();
     assert r instanceof DirectoryReader
         : "searcher's IndexReader should be a DirectoryReader, but got " + r;
-    final IndexReader newReader = DirectoryReader.openIfChanged((DirectoryReader) r);
+    DirectoryReader dr = (DirectoryReader) r;
+    IndexCommit refreshCommit = refreshCommitSupplier.getSearcherRefreshCommit(dr);
+    final IndexReader newReader = DirectoryReader.openIfChanged(dr, refreshCommit);
     if (newReader == null) {
       return null;
     } else {
@@ -159,13 +215,21 @@ protected int getRefCount(IndexSearcher reference) {
     return reference.getIndexReader().getRefCount();
   }
 
+  /** Return index commit generation for current searcher. pkg-private for testing */
+  long getSearcherCommitGeneration() throws IOException {
+    IndexSearcher s = acquire();
+    long gen = ((DirectoryReader) s.getIndexReader()).getIndexCommit().getGeneration();
+    release(s);
+    return gen;
+  }
+
   /**
    * Returns <code>true</code> if no changes have occurred since this searcher ie. reader was
-   * opened, otherwise <code>false</code>.
+   * opened, otherwise <code>false</code>. pkg-private for testing
    *
    * @see DirectoryReader#isCurrent()
    */
-  public boolean isSearcherCurrent() throws IOException {
+  boolean isSearcherCurrent() throws IOException {
     final IndexSearcher searcher = acquire();
     try {
       final IndexReader r = searcher.getIndexReader();
diff --git a/lucene/core/src/test/org/apache/lucene/search/TestSearcherManager.java b/lucene/core/src/test/org/apache/lucene/search/TestSearcherManager.java
index e44855afe870..29f49bd239f6 100644
--- a/lucene/core/src/test/org/apache/lucene/search/TestSearcherManager.java
+++ b/lucene/core/src/test/org/apache/lucene/search/TestSearcherManager.java
@@ -32,11 +32,13 @@
 import org.apache.lucene.index.DirectoryReader;
 import org.apache.lucene.index.FilterDirectoryReader;
 import org.apache.lucene.index.FilterLeafReader;
+import org.apache.lucene.index.IndexCommit;
 import org.apache.lucene.index.IndexReader;
 import org.apache.lucene.index.IndexWriter;
 import org.apache.lucene.index.IndexWriterConfig;
 import org.apache.lucene.index.LeafReader;
 import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.NoDeletionPolicy;
 import org.apache.lucene.index.Term;
 import org.apache.lucene.store.AlreadyClosedException;
 import org.apache.lucene.store.Directory;
@@ -747,4 +749,64 @@ public void run() {
     writerRef.get().close();
     dir.close();
   }
+
+  /** Returns the first commit with generation higher than current reader commit */
+  public static class NextCommitSelector implements RefreshCommitSupplier {
+    @Override
+    public IndexCommit getSearcherRefreshCommit(DirectoryReader reader) throws IOException {
+      List<IndexCommit> commits = DirectoryReader.listCommits(reader.directory());
+      IndexCommit current = reader.getIndexCommit();
+      for (int i = 0; i < commits.size(); i++) {
+        IndexCommit commit = commits.get(i);
+        if (commit.getGeneration() > current.getGeneration()) {
+          return commit;
+        }
+      }
+      // we're already on latest commit
+      return null;
+    }
+  }
+
+  public void testStepWiseCommitRefresh() throws Exception {
+    Directory dir = newDirectory();
+    IndexWriter w =
+        new IndexWriter(
+            dir,
+            newIndexWriterConfig(new MockAnalyzer(random()))
+                .setIndexDeletionPolicy(NoDeletionPolicy.INSTANCE));
+    int docId = 0;
+    // create initial commit
+    for (int i = 0; i < 20; i++) {
+      Document doc = new Document();
+      doc.add(newStringField("docId", "doc-" + docId++, Field.Store.YES));
+      w.addDocument(doc);
+    }
+    w.commit();
+    SearcherManager sm =
+        new SearcherManager(DirectoryReader.open(dir), null, new NextCommitSelector());
+    final int numCommits = 5;
+    for (int i = 0; i < numCommits; i++) {
+      for (int j = 0; j < 20; j++) {
+        Document doc = new Document();
+        doc.add(newStringField("docId", "doc-" + docId++, Field.Store.YES));
+        w.addDocument(doc);
+      }
+      w.commit();
+    }
+
+    // maybeRefresh only refreshes on the next incremental commit
+    // so it takes us numCommits to get to latest
+    int stepsToCurrent = 0;
+    while (sm.isSearcherCurrent() == false) {
+      long oldGen = sm.getSearcherCommitGeneration();
+      sm.maybeRefreshBlocking();
+      long newGen = sm.getSearcherCommitGeneration();
+      assertTrue(newGen == oldGen + 1);
+      stepsToCurrent++;
+    }
+    assertEquals(numCommits, stepsToCurrent);
+    sm.close();
+    w.close();
+    dir.close();
+  }
 }
diff --git a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/SearcherTaxonomyManager.java b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/SearcherTaxonomyManager.java
index 5b768f48a053..64f906a8f031 100644
--- a/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/SearcherTaxonomyManager.java
+++ b/lucene/facet/src/java/org/apache/lucene/facet/taxonomy/SearcherTaxonomyManager.java
@@ -20,10 +20,12 @@
 import org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyReader;
 import org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter;
 import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexCommit;
 import org.apache.lucene.index.IndexReader;
 import org.apache.lucene.index.IndexWriter;
 import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.ReferenceManager;
+import org.apache.lucene.search.RefreshCommitSupplier;
 import org.apache.lucene.search.SearcherFactory;
 import org.apache.lucene.search.SearcherManager;
 import org.apache.lucene.store.Directory;
@@ -50,6 +52,7 @@ public record SearcherAndTaxonomy(
   private final SearcherFactory searcherFactory;
   private final long taxoEpoch;
   private final DirectoryTaxonomyWriter taxoWriter;
+  private RefreshCommitSupplier refreshCommitSupplier = new RefreshCommitSupplier() {};
 
   /** Creates near-real-time searcher and taxonomy reader from the corresponding writers. */
   public SearcherTaxonomyManager(
@@ -68,6 +71,20 @@ public SearcherTaxonomyManager(
       SearcherFactory searcherFactory,
       DirectoryTaxonomyWriter taxoWriter)
       throws IOException {
+    this(writer, applyAllDeletes, searcherFactory, taxoWriter, null);
+  }
+
+  /**
+   * Expert: creates near-real-time searcher and taxonomy reader from the corresponding writers,
+   * controlling whether deletes should be applied.
+   */
+  public SearcherTaxonomyManager(
+      IndexWriter writer,
+      boolean applyAllDeletes,
+      SearcherFactory searcherFactory,
+      DirectoryTaxonomyWriter taxoWriter,
+      RefreshCommitSupplier refreshCommitSupplier)
+      throws IOException {
     if (searcherFactory == null) {
       searcherFactory = new SearcherFactory();
     }
@@ -80,6 +97,9 @@ public SearcherTaxonomyManager(
                 searcherFactory, DirectoryReader.open(writer, applyAllDeletes, false), null),
             taxoReader);
     this.taxoEpoch = taxoWriter.getTaxonomyEpoch();
+    if (refreshCommitSupplier != null) {
+      this.refreshCommitSupplier = refreshCommitSupplier;
+    }
   }
 
   /**
@@ -111,6 +131,19 @@ public SearcherTaxonomyManager(
   public SearcherTaxonomyManager(
       IndexReader reader, DirectoryTaxonomyReader taxoReader, SearcherFactory searcherFactory)
       throws IOException {
+    this(reader, taxoReader, searcherFactory, null);
+  }
+
+  /**
+   * Creates this from already opened {@link IndexReader} and {@link DirectoryTaxonomyReader}
+   * instances. Note that the incoming readers will be closed when you call {@link #close}.
+   */
+  public SearcherTaxonomyManager(
+      IndexReader reader,
+      DirectoryTaxonomyReader taxoReader,
+      SearcherFactory searcherFactory,
+      RefreshCommitSupplier refreshCommitSupplier)
+      throws IOException {
     if (searcherFactory == null) {
       searcherFactory = new SearcherFactory();
     }
@@ -120,6 +153,9 @@ public SearcherTaxonomyManager(
             SearcherManager.getSearcher(searcherFactory, reader, null), taxoReader);
     this.taxoWriter = null;
     taxoEpoch = -1;
+    if (refreshCommitSupplier != null) {
+      this.refreshCommitSupplier = refreshCommitSupplier;
+    }
   }
 
   @Override
@@ -154,11 +190,16 @@ protected SearcherAndTaxonomy refreshIfNeeded(SearcherAndTaxonomy ref) throws IO
     // new reader that references ords not yet known to the
     // taxonomy reader:
     final IndexReader r = ref.searcher.getIndexReader();
-    final IndexReader newReader = DirectoryReader.openIfChanged((DirectoryReader) r);
+    DirectoryReader dr = (DirectoryReader) r;
+    IndexCommit refreshCommit = refreshCommitSupplier.getSearcherRefreshCommit(dr);
+    IndexReader newReader = DirectoryReader.openIfChanged(dr, refreshCommit);
     if (newReader == null) {
       return null;
     } else {
-      DirectoryTaxonomyReader tr;
+      DirectoryTaxonomyReader tr = null;
+      // taxonomy should always be ahead of searchers. Otherwise, searchers
+      // might reference ordinals that taxonomy doesn't know of.
+      // To ensure this, we always refresh taxonomy on the latest commit.
       try {
         tr = TaxonomyReader.openIfChanged(ref.taxonomyReader);
       } catch (Throwable t1) {
@@ -183,6 +224,47 @@ protected SearcherAndTaxonomy refreshIfNeeded(SearcherAndTaxonomy ref) throws IO
     }
   }
 
+  /** Return index commit generation for current searcher. pkg-private for testing */
+  long getSearcherCommitGeneration() throws IOException {
+    SearcherAndTaxonomy sat = acquire();
+    long gen = ((DirectoryReader) sat.searcher.getIndexReader()).getIndexCommit().getGeneration();
+    release(sat);
+    return gen;
+  }
+
+  /**
+   * Returns <code>true</code> if no new changes have occurred since the current searcher (i.e.
+   * reader) was opened, <code>false</code> otherwise. pkg-private for testing
+   *
+   * @see DirectoryReader#isCurrent()
+   */
+  boolean isSearcherCurrent() throws IOException {
+    final SearcherAndTaxonomy sat = acquire();
+    try {
+      final IndexReader r = sat.searcher.getIndexReader();
+      assert r instanceof DirectoryReader
+          : "searcher's IndexReader should be a DirectoryReader, but got " + r;
+      return ((DirectoryReader) r).isCurrent();
+    } finally {
+      release(sat);
+    }
+  }
+
+  /**
+   * Returns <code>true</code> if no new changes have occurred since the current taxonomy reader was
+   * opened, <code>false</code> otherwise. pkg-private for testing
+   *
+   * @see DirectoryReader#isCurrent()
+   */
+  boolean isTaxonomyCurrent() throws IOException {
+    final SearcherAndTaxonomy sat = acquire();
+    try {
+      return sat.taxonomyReader.getInternalIndexReader().isCurrent();
+    } finally {
+      release(sat);
+    }
+  }
+
   @Override
   protected int getRefCount(SearcherAndTaxonomy reference) {
     return reference.searcher.getIndexReader().getRefCount();
diff --git a/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/TestSearcherTaxonomyManager.java b/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/TestSearcherTaxonomyManager.java
index 3f319acdb780..327ed8d727e0 100644
--- a/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/TestSearcherTaxonomyManager.java
+++ b/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/TestSearcherTaxonomyManager.java
@@ -23,6 +23,7 @@
 import java.util.Set;
 import java.util.concurrent.atomic.AtomicBoolean;
 import org.apache.lucene.document.Document;
+import org.apache.lucene.document.Field;
 import org.apache.lucene.facet.FacetField;
 import org.apache.lucene.facet.FacetResult;
 import org.apache.lucene.facet.FacetTestCase;
@@ -31,14 +32,19 @@
 import org.apache.lucene.facet.FacetsCollectorManager;
 import org.apache.lucene.facet.FacetsConfig;
 import org.apache.lucene.facet.taxonomy.SearcherTaxonomyManager.SearcherAndTaxonomy;
+import org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyReader;
 import org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexCommit;
 import org.apache.lucene.index.IndexNotFoundException;
 import org.apache.lucene.index.IndexWriter;
 import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.NoDeletionPolicy;
 import org.apache.lucene.index.SegmentInfos;
 import org.apache.lucene.index.TieredMergePolicy;
 import org.apache.lucene.search.MatchAllDocsQuery;
 import org.apache.lucene.search.ReferenceManager;
+import org.apache.lucene.search.RefreshCommitSupplier;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.tests.analysis.MockAnalyzer;
 import org.apache.lucene.tests.util.LuceneTestCase;
@@ -356,4 +362,80 @@ public void testExceptionDuringRefresh() throws Exception {
     expectThrows(IndexNotFoundException.class, mgr::maybeRefreshBlocking);
     IOUtils.close(w, tw, mgr, indexDir, taxoDir);
   }
+
+  /** Returns the first commit with generation higher than current reader commit */
+  public static class NextCommitSelector implements RefreshCommitSupplier {
+    @Override
+    public IndexCommit getSearcherRefreshCommit(DirectoryReader reader) throws IOException {
+      List<IndexCommit> commits = DirectoryReader.listCommits(reader.directory());
+      IndexCommit current = reader.getIndexCommit();
+      for (int i = 0; i < commits.size(); i++) {
+        IndexCommit commit = commits.get(i);
+        if (commit.getGeneration() > current.getGeneration()) {
+          return commit;
+        }
+      }
+      // we're already on latest commit
+      return null;
+    }
+  }
+
+  public void testStepWiseCommitRefresh() throws Exception {
+    Directory dir = newDirectory();
+    Directory taxoDir = newDirectory();
+    IndexWriter w =
+        new IndexWriter(
+            dir,
+            newIndexWriterConfig(new MockAnalyzer(random()))
+                .setIndexDeletionPolicy(NoDeletionPolicy.INSTANCE));
+    DirectoryTaxonomyWriter tw = new DirectoryTaxonomyWriter(taxoDir);
+    int docId = 0;
+    // create initial commit
+    for (int i = 0; i < 20; i++) {
+      Document doc = new Document();
+      doc.add(newStringField("docId", "doc-" + docId++, Field.Store.YES));
+      w.addDocument(doc);
+    }
+    w.commit();
+    tw.commit();
+
+    int colorIndex = 0;
+    final String[] colors = new String[] {"red", "green", "blue", "yellow"};
+    FacetsConfig config = new FacetsConfig();
+    DirectoryTaxonomyReader taxoReader = new DirectoryTaxonomyReader(taxoDir);
+    SearcherTaxonomyManager sat =
+        new SearcherTaxonomyManager(
+            DirectoryReader.open(dir), taxoReader, null, new NextCommitSelector());
+
+    final int numCommits = 5;
+    for (int i = 0; i < numCommits; i++) {
+      for (int j = 0; j < 20; j++) {
+        Document doc = new Document();
+        doc.add(newStringField("docId", "doc-" + docId++, Field.Store.YES));
+        doc.add(new FacetField("Color", colors[colorIndex++ % colors.length]));
+        doc = config.build(tw, doc);
+        w.addDocument(doc);
+      }
+      w.commit();
+      tw.commit();
+    }
+
+    // maybeRefresh only refreshes on the next incremental commit
+    // so it takes us numCommits to get to latest
+    int stepsToCurrent = 0;
+    while (sat.isSearcherCurrent() == false) {
+      long oldGen = sat.getSearcherCommitGeneration();
+      sat.maybeRefreshBlocking();
+      long newGen = sat.getSearcherCommitGeneration();
+      assertEquals(newGen, oldGen + 1);
+      stepsToCurrent++;
+      assertTrue(sat.isTaxonomyCurrent());
+    }
+    assertEquals(numCommits, stepsToCurrent);
+    sat.close();
+    w.close();
+    tw.close();
+    dir.close();
+    taxoDir.close();
+  }
 }


Reviews:
user: login: mikemccand
body: Looks great, thanks @vigyasharma -- I left some small comments.  It would be nice to give this capability (carefully pick which commit point to refresh to, when there are multiple choices, for resilience in distributed NRT segment replication case) to `SearcherManager`.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-01T14:39:49Z
user: login: vigyasharma
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T00:00:13Z
user: login: vigyasharma
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-10T22:29:46Z
user: login: vigyasharma
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-10T22:41:43Z

Review comments:
diff_hunk: @@ -131,17 +133,32 @@ public SearcherManager(DirectoryReader reader, SearcherFactory searcherFactory)
     this.current = getSearcher(searcherFactory, reader, null);
   }
 
+  /** Set supplier for selecting commits to refresh on */
+  public void setRefreshCommitSupplier(RefreshCommitSupplier refreshCommitSupplier) {
path: lucene/core/src/java/org/apache/lucene/search/SearcherManager.java
user: login: mikemccand
body: Should this maybe be passed to ctor instead?  (And we'd add another telescoping ctor).  Typically one would not need to change this after creating the `SearcherManager`?
created_at: 2025-05-01T14:34:48Z
author_association: MEMBER
diff_hunk: @@ -131,17 +133,32 @@ public SearcherManager(DirectoryReader reader, SearcherFactory searcherFactory)
     this.current = getSearcher(searcherFactory, reader, null);
   }
 
+  /** Set supplier for selecting commits to refresh on */
+  public void setRefreshCommitSupplier(RefreshCommitSupplier refreshCommitSupplier) {
+    this.refreshCommitSupplier = refreshCommitSupplier;
+  }
+
   @Override
   protected void decRef(IndexSearcher reference) throws IOException {
     reference.getIndexReader().decRef();
   }
 
+  /** Return index commit generation for current searcher */
+  public long getSearcherCommitGeneration() throws IOException {
path: lucene/core/src/java/org/apache/lucene/search/SearcherManager.java
user: login: mikemccand
body: I wonder if we should leave this to caller?

It's a bit dangerous to provide such a method because if a refresh happens concurrently, the returned `gen` might be stale by the time caller does something with it?  Though of course the app would presumably "know" if it has a background refresh thread ...
created_at: 2025-05-01T14:36:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.search;
+
+import java.io.IOException;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexCommit;
+
+/**
+ * Expert: Interface to supply commit for searcher refresh.
+ *
+ * @lucene.experimental
+ */
+public interface RefreshCommitSupplier {
path: lucene/core/src/java/org/apache/lucene/search/RefreshCommitSupplier.java
user: login: mikemccand
body: If we don't like the added API surface area of a new `interface`, we could maybe un-final `SearcherManager` and allow subclass to override the default of picking the latest `IndexCommit`?  I don't really have a preference... the new interface is `@lucene.experimental` so we are free to evolve it, even within stable release branch.
created_at: 2025-05-01T14:37:37Z
author_association: MEMBER
diff_hunk: @@ -131,17 +133,32 @@ public SearcherManager(DirectoryReader reader, SearcherFactory searcherFactory)
     this.current = getSearcher(searcherFactory, reader, null);
   }
 
+  /** Set supplier for selecting commits to refresh on */
+  public void setRefreshCommitSupplier(RefreshCommitSupplier refreshCommitSupplier) {
path: lucene/core/src/java/org/apache/lucene/search/SearcherManager.java
user: login: vigyasharma
body: Makes sense, I'll add it to the ctor.
created_at: 2025-05-05T00:00:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.search;
+
+import java.io.IOException;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexCommit;
+
+/**
+ * Expert: Interface to supply commit for searcher refresh.
+ *
+ * @lucene.experimental
+ */
+public interface RefreshCommitSupplier {
path: lucene/core/src/java/org/apache/lucene/search/RefreshCommitSupplier.java
user: login: vigyasharma
body: I went through original discussions in [LUCENE-3761](https://issues.apache.org/jira/browse/LUCENE-3761) where SearcherManager was added.. The choice to make it final seems to have been from the beginning, I didn't find any discussion explicitly about this.

Overriding this class safely does require a sound understanding of the `ReferenceManager` base class which has threading/synchronization code. I feel it's safer to leave this class as final, in line with the original intent.
created_at: 2025-05-10T22:29:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -131,17 +133,32 @@ public SearcherManager(DirectoryReader reader, SearcherFactory searcherFactory)
     this.current = getSearcher(searcherFactory, reader, null);
   }
 
+  /** Set supplier for selecting commits to refresh on */
+  public void setRefreshCommitSupplier(RefreshCommitSupplier refreshCommitSupplier) {
+    this.refreshCommitSupplier = refreshCommitSupplier;
+  }
+
   @Override
   protected void decRef(IndexSearcher reference) throws IOException {
     reference.getIndexReader().decRef();
   }
 
+  /** Return index commit generation for current searcher */
+  public long getSearcherCommitGeneration() throws IOException {
path: lucene/core/src/java/org/apache/lucene/search/SearcherManager.java
user: login: vigyasharma
body: Good point. I was exposing this as a hook for callers to monitor the searcher commit generation at a point in time, but it could be confusing. It's also just a utility functions that consumers can define outside of this class. Will remove it.
created_at: 2025-05-10T22:41:43Z
author_association: CONTRIBUTOR

PR comments:
user: login: jpountz
created_at: 2025-04-07T12:17:37Z
body: Thanks for tackling this!

> To incrementally refresh through multiple commit points until searcher is current with its directory:
> 
> [...]
> Invoke maybeRefresh() or maybeRefreshBlocking in a loop until isSearcherCurrent() returns true.

Is this the way we anticipate this to be used? I had imagined that the application would not change the way it refreshes and still call it on a schedule, but commit more frequently and retain multiple commits. E.g. commit every 30 seconds, retain commits for 300 seconds and refresh every 120 seconds (these numbers are just for the sake of the example). So every 120 seconds, SearcherManager would pick the most recent commit that differs by less than X GB (configurable based on the amount of trashing that the app can sustain between consecutive point-in-time views of the index) from the current point-in-time reader, or the commit that differs by the least amount of data if there is no such commit (typically the oldest commit). Most of the time, `SearcherManager` would pick the newest commit point, but under heavy merging it may decide to lag behind the latest commit point a bit for the sake of smoothing out page cache trashing.
author_association: CONTRIBUTOR
user: login: vigyasharma
created_at: 2025-04-07T23:22:46Z
body: > every 120 seconds, SearcherManager would pick the most recent commit that differs by less than X GB

This is indeed how we anticipate it being used. In NRT style segment replicated setups, if searchers refresh more often than replication frequency, they will eventually catch up to the latest commit. I mentioned the while loop for cases where users want to wait and verify that their searchers are current. 

The PR of course supports both patterns, I'll update the description to reflect it as well. 
author_association: CONTRIBUTOR
user: login: jpountz
created_at: 2025-04-08T06:21:42Z
body: > if searchers refresh more often than replication frequency

OK I think I misunderstood how it would be used. I had assumed that commits would always get replicated immediately, but you are suggesting that replications are infrequent and bring several commits at once to leave time to replica nodes to smoothly absorb the delta.
author_association: CONTRIBUTOR
user: login: jpountz
created_at: 2025-04-10T13:39:56Z
body: Sorry I'm still a bit confused: how is this approach better than just committing more frequently, replicating commits as soon as they are created, and refreshing searchers as soon as commits are replicated?
author_association: CONTRIBUTOR
user: login: msokolov
created_at: 2025-04-11T23:41:07Z
body: > Sorry I'm still a bit confused: how is this approach better than just committing more frequently, replicating commits as soon as they are created, and refreshing searchers as soon as commits are replicated?

One scenario of interest is when replication becomes delayed, for example when working with cross-datacenter replication this is expected. In that case commit points may pile up, even to the extent of completely replacing the entire index.  In such a case we'd like to be able to recover without undue impact to searchers.
author_association: CONTRIBUTOR
user: login: vigyasharma
created_at: 2025-04-12T03:53:48Z
body: > just committing more frequently, replicating commits as soon as they are created, and refreshing searchers as soon as commits are replicated?

This is more or less the setup we have today at Amazon Product Search. We have separate indexing and search fleets that use s3 as a sink. Some fleets replicate across aws data centers. I believe this is a common architecture, for e.g. DoorDash seems to have a [similar search architecture](https://careersatdoordash.com/blog/introducing-doordashs-in-house-search-engine/?ref=dailydev).

However, as Mike mentioned, these commits go over network hops and are vulnerable to networking lags. Our searchers periodically pull the latest commit from s3 and refresh. If replication is delayed, searchers can skip a few commits to pull the latest one available. This latest commit can have a very high delta to what searchers are currently on.

With adaptive refresh, we are experimenting with making searchers pull the last N commits and refresh on the newest commit that they can safely absorb. At the extreme, if the entire index has changed, it will be no different than refreshing on the latest commit. But for moderate delay windows, we could find "bite sized" hops for searchers to catch up safely.
author_association: CONTRIBUTOR
user: login: vigyasharma
created_at: 2025-04-12T03:54:32Z
body: Another scenario where adaptive refresh might be useful is with heterogenous search fleets. Searchers with less memory would benefit from stepping through smaller commit deltas, while high memory searchers can jump ahead.
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-27T00:27:25Z
body: This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!
author_association: CONTRIBUTOR
user: login: mikemccand
created_at: 2025-05-01T14:30:16Z
body: > Sorry I'm still a bit confused: how is this approach better than just committing more frequently, replicating commits as soon as they are created, and refreshing searchers as soon as commits are replicated?

In Amazon's usage, and I would expect other high-rate NRT segment replication systems, it's helpful to strongly decouple the production of the new commit points (triggered by time, or by X GB new segments), from the replication of these commits.

During peace time (happy path), what you're suggesting works well -- commit and replication can always match each other 1 for 1.  But under duress (unhappy path), commit point production can be faster (usually, hopefully, temporarily) than the replication can keep up, maybe because crazy high rate of document updates, or slow pipe for replication, or bit errors needing lots of retries, etc. (distributed systems seem to have all sorts of fun ways to become problematic!) ... and for those situations, it's really nice to have this possible decoupling easily accessible.

It's also delightful because Lucene makes it quite simple to keep more than one commit point alive at once (it's just a custom `IndexDeletionPolicy`) ... building this decoupling on top of that is "relatively" easy heh.
author_association: MEMBER
user: login: vigyasharma
created_at: 2025-05-10T23:44:57Z
body: Thanks for the review, @mikemccand . I've updated the PR with changes. 

I did a rebase on `main` since it had moved quite a bit. There was no conflict for the files here, but since rebase rewrote commit history, you may not be able to see just the changes since your last review. Apologies for the inconvenience, hopefully the PR is small enough to manage.

author_association: CONTRIBUTOR
user: login: vigyasharma
created_at: 2025-05-15T05:24:50Z
body: Thanks everyone for the discussion and feedback. Unless there are concerns, I'll go with lazy consensus and merge this on Friday.
author_association: CONTRIBUTOR