Repository: opensearch-project/cross-cluster-replication
Pull request #1520: feat: change IOContext strategy to DEFAULT 
Author: login: 10000-ki
Created: 2025-04-07T09:10:44Z | Merged: 2025-04-08T13:59:58Z

### Description
When using the `READONCE` option in Apache Lucene to read index files via `IndexInput`, it is not suitable for use cases where multiple threads access the file concurrently. This is because `READONCE` is optimized for single-use I/O and assumes the input will be read only once.

Therefore, I have changed it to use the `READ` (or `DEFAULT`) option, which is more appropriate for multi-threaded access scenarios.

### Related Issues
Resolves #1482
Resolves #1465


### Check List
- [ ] New functionality includes testing.
- [ ] New functionality has been documented.
- [ ] API changes companion pull request [created](https://github.com/opensearch-project/opensearch-api-specification/blob/main/DEVELOPER_GUIDE.md).
- [ ] Commits are signed per the DCO using `--signoff`.
- [ ] Public documentation issue/PR [created](https://github.com/opensearch-project/documentation-website/issues/new/choose).

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/cross-cluster-replication/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).


Linked issue #1482: [BUG] CCR fails when there is existing data in the index.
**What is the bug?**

If the index is created and CCR is set up immediately, there are no issues. However, when CCR is configured on an index that already contains a certain amount of data, an error occurs.

| index | pri | rep | pri size | ccr result |
| ---- | ---- | ---- | ---- | ---- |
| ccr-100m | 1 | 1 | 127m | ✔️ |
| ccr-150m | 1 | 1 | 152m | ✔️ |
| ccr-180m | 1 | 1 | 177m | ✔️ |
| ccr-200m-v1 | 1 | 1 | 212m | ✔️ |
| ccr-220m-v1 | 1 | 1 | 226m | ❌ |
| ccr-220m-v2 | 1 | 1 | 226m | ❌ |


- error msg : `
Failed to fetch file chunk for _w.cfs with offset 20971520: java.lang.IllegalStateException: confined`

**How can one reproduce the bug?**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**What is the expected behavior?**
A clear and concise description of what you expected to happen.

**What is your host/environment?**
 - OpenSearch : v2.18.0

**Do you have any screenshots?**
If applicable, add screenshots to help explain your problem.

**Do you have any additional context?**
Add any other context about the problem.

Linked issue comments: user: login: 10000-ki
created_at: 2025-02-03T05:50:58Z
body: Are there any configuration errors or unknown constraints?
author_association: CONTRIBUTOR
user: login: krisfreedain
created_at: 2025-02-24T17:07:58Z
body: [Catch All Triage](https://github.com/opensearch-project/.github/blob/main/TRIAGING.md) - [1](https://github.com/krisfreedain) [2](https://github.com/finnegancarroll)
author_association: MEMBER
user: login: ankitkala
created_at: 2025-04-02T10:05:52Z
body: Not sure if its related to this change: https://github.com/apache/lucene/commit/c8e05c8cd6e018133d3d739d159d5b3b6b29c179#diff-be8459728adce212557bc19afbe0a8b91c788a49b1379323af40e3ba3ec4e479R154

Can you retry after disabling this feature by adding this JVM arg: `"-Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false"`
author_association: MEMBER

Diff:
diff --git a/src/main/kotlin/org/opensearch/replication/repository/RestoreContext.kt b/src/main/kotlin/org/opensearch/replication/repository/RestoreContext.kt
index 4658eca99..742b63894 100644
--- a/src/main/kotlin/org/opensearch/replication/repository/RestoreContext.kt
+++ b/src/main/kotlin/org/opensearch/replication/repository/RestoreContext.kt
@@ -12,47 +12,47 @@
 package org.opensearch.replication.repository
 
 import org.apache.lucene.index.IndexCommit
-import org.opensearch.replication.util.performOp
 import org.apache.lucene.store.IOContext
 import org.apache.lucene.store.IndexInput
-import org.opensearch.OpenSearchException
 import org.opensearch.common.concurrent.GatedCloseable
-import org.opensearch.index.engine.Engine
 import org.opensearch.index.shard.IndexShard
 import org.opensearch.index.store.Store
 import java.io.Closeable
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.locks.ReentrantLock
+import kotlin.concurrent.withLock
 
-class RestoreContext(val restoreUUID: String,
-                     val shard: IndexShard,
-                     val indexCommitRef: GatedCloseable<IndexCommit>,
-                     val metadataSnapshot: Store.MetadataSnapshot,
-                     val replayOperationsFrom: Long): Closeable {
+class RestoreContext(
+    val restoreUUID: String,
+    val shard: IndexShard,
+    val indexCommitRef: GatedCloseable<IndexCommit>,
+    val metadataSnapshot: Store.MetadataSnapshot,
+    val replayOperationsFrom: Long,
+) : Closeable {
 
-    companion object {
-        private const val INITIAL_FILE_CACHE_CAPACITY = 20
-    }
-    private val currentFiles = LinkedHashMap<String, IndexInput>(INITIAL_FILE_CACHE_CAPACITY)
+    private val currentFiles = ConcurrentHashMap<String, IndexInput>(INITIAL_FILE_CACHE_CAPACITY)
+
+    private val fileLocks = ConcurrentHashMap<String, ReentrantLock>()
 
     fun openInput(store: Store, fileName: String): IndexInput {
-        var currentIndexInput = currentFiles.getOrDefault(fileName, null)
-        if(currentIndexInput != null) {
-            return currentIndexInput.clone()
-        }
-        store.performOp({
-            currentIndexInput = store.directory().openInput(fileName, IOContext.READONCE)
-        })
+        val lock = fileLocks.computeIfAbsent(fileName) { ReentrantLock() }
 
-        currentFiles[fileName] = currentIndexInput!!
-        return currentIndexInput!!.clone()
+        lock.withLock {
+            val baseInput = currentFiles.computeIfAbsent(fileName) {
+                store.directory().openInput(fileName, IOContext.DEFAULT)
+            }
+            return baseInput.clone()
+        }
     }
 
     override fun close() {
-        // Close all the open index input obj
-        currentFiles.entries.forEach {
-            it.value.close()
-        }
+        currentFiles.values.forEach { it.close() }
         currentFiles.clear()
+        fileLocks.clear()
         indexCommitRef.close()
     }
 
+    companion object {
+        private const val INITIAL_FILE_CACHE_CAPACITY = 20
+    }
 }
diff --git a/src/test/kotlin/org/opensearch/replication/repository/RestoreContextTests.kt b/src/test/kotlin/org/opensearch/replication/repository/RestoreContextTests.kt
new file mode 100644
index 000000000..7c45f7b38
--- /dev/null
+++ b/src/test/kotlin/org/opensearch/replication/repository/RestoreContextTests.kt
@@ -0,0 +1,49 @@
+package org.opensearch.replication.repository
+
+import org.apache.lucene.index.IndexCommit
+import org.apache.lucene.store.IOContext
+import org.apache.lucene.store.IndexInput
+import org.mockito.ArgumentMatchers.any
+import org.mockito.ArgumentMatchers.eq
+import org.mockito.Mockito.*
+import org.opensearch.common.concurrent.GatedCloseable
+import org.opensearch.index.shard.IndexShard
+import org.opensearch.index.store.Store
+import org.opensearch.test.OpenSearchTestCase
+import java.util.UUID
+
+class RestoreContextTests : OpenSearchTestCase() {
+
+    fun `test openInput returns cloned IndexInput from cache`() {
+        // given
+        val fileName = "test_file"
+
+        val mockShard = mock(IndexShard::class.java)
+        val mockIndexCommit = mock(GatedCloseable::class.java) as GatedCloseable<IndexCommit>
+        val mockStore = mock(Store::class.java)
+        val mockDirectory = mock(org.apache.lucene.store.Directory::class.java)
+        val mockBaseInput = mock(IndexInput::class.java)
+        val mockClonedInput = mock(IndexInput::class.java)
+
+        // when
+        `when`(mockStore.directory()).thenReturn(mockDirectory)
+        `when`(mockDirectory.openInput(eq(fileName), any(IOContext::class.java))).thenReturn(mockBaseInput)
+        `when`(mockBaseInput.clone()).thenReturn(mockClonedInput)
+        val sut = RestoreContext(
+            restoreUUID = UUID.randomUUID().toString(),
+            shard = mockShard,
+            indexCommitRef = mockIndexCommit,
+            metadataSnapshot = Store.MetadataSnapshot.EMPTY,
+            replayOperationsFrom = 0L,
+        )
+
+        val result1 = sut.openInput(mockStore, fileName)
+        val result2 = sut.openInput(mockStore, fileName)
+
+        // then
+        assertSame(mockClonedInput, result1)
+        assertSame(mockClonedInput, result2)
+        verify(mockDirectory, times(1)).openInput(eq(fileName), any(IOContext::class.java))
+        verify(mockBaseInput, times(2)).clone()
+    }
+}


Reviews:
user: login: 10000-ki
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T09:11:54Z
user: login: 10000-ki
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T09:13:23Z
user: login: 10000-ki
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T09:13:33Z
user: login: 10000-ki
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T10:07:05Z
user: login: 10000-ki
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T01:52:33Z
user: login: ankitkala
body: LGTM! Can you confirm that you were able to test bootstrapping on bigger indices? Please add details on any manual test you might've run to verify.  
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T03:47:26Z
user: login: ankitkala
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-08T10:30:51Z
user: login: gbbafna
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-08T13:59:57Z
user: login: mishail
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-08T15:24:18Z
user: login: 10000-ki
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T01:41:09Z

Review comments:
diff_hunk: @@ -12,47 +12,47 @@
 package org.opensearch.replication.repository
 
 import org.apache.lucene.index.IndexCommit
-import org.opensearch.replication.util.performOp
 import org.apache.lucene.store.IOContext
 import org.apache.lucene.store.IndexInput
-import org.opensearch.OpenSearchException
 import org.opensearch.common.concurrent.GatedCloseable
-import org.opensearch.index.engine.Engine
 import org.opensearch.index.shard.IndexShard
 import org.opensearch.index.store.Store
 import java.io.Closeable
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.locks.ReentrantLock
+import kotlin.concurrent.withLock
 
-class RestoreContext(val restoreUUID: String,
-                     val shard: IndexShard,
-                     val indexCommitRef: GatedCloseable<IndexCommit>,
-                     val metadataSnapshot: Store.MetadataSnapshot,
-                     val replayOperationsFrom: Long): Closeable {
+class RestoreContext(
+    val restoreUUID: String,
+    val shard: IndexShard,
+    val indexCommitRef: GatedCloseable<IndexCommit>,
+    val metadataSnapshot: Store.MetadataSnapshot,
+    val replayOperationsFrom: Long,
+) : Closeable {
 
-    companion object {
-        private const val INITIAL_FILE_CACHE_CAPACITY = 20
-    }
-    private val currentFiles = LinkedHashMap<String, IndexInput>(INITIAL_FILE_CACHE_CAPACITY)
+    private val currentFiles = ConcurrentHashMap<String, IndexInput>(INITIAL_FILE_CACHE_CAPACITY)
+
+    private val fileLocks = ConcurrentHashMap<String, ReentrantLock>()
 
     fun openInput(store: Store, fileName: String): IndexInput {
-        var currentIndexInput = currentFiles.getOrDefault(fileName, null)
-        if(currentIndexInput != null) {
-            return currentIndexInput.clone()
-        }
-        store.performOp({
-            currentIndexInput = store.directory().openInput(fileName, IOContext.READONCE)
-        })
+        val lock = fileLocks.computeIfAbsent(fileName) { ReentrantLock() }
 
-        currentFiles[fileName] = currentIndexInput!!
-        return currentIndexInput!!.clone()
+        lock.withLock {
+            val baseInput = currentFiles.computeIfAbsent(fileName) {
+                store.directory().openInput(fileName, IOContext.DEFAULT)
path: src/main/kotlin/org/opensearch/replication/repository/RestoreContext.kt
user: login: 10000-ki
body: https://github.com/apache/lucene/blob/219cd5eaebca686c1df93226f47924337ee1a783/lucene/MIGRATE.md#iocontextload-and-iocontextread-removed

`IOContext.READ` has been removed, it should be replaced with `IOContext.DEFAULT`.
created_at: 2025-04-07T09:11:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -12,47 +12,47 @@
 package org.opensearch.replication.repository
 
 import org.apache.lucene.index.IndexCommit
-import org.opensearch.replication.util.performOp
 import org.apache.lucene.store.IOContext
 import org.apache.lucene.store.IndexInput
-import org.opensearch.OpenSearchException
 import org.opensearch.common.concurrent.GatedCloseable
-import org.opensearch.index.engine.Engine
 import org.opensearch.index.shard.IndexShard
 import org.opensearch.index.store.Store
 import java.io.Closeable
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.locks.ReentrantLock
+import kotlin.concurrent.withLock
 
-class RestoreContext(val restoreUUID: String,
-                     val shard: IndexShard,
-                     val indexCommitRef: GatedCloseable<IndexCommit>,
-                     val metadataSnapshot: Store.MetadataSnapshot,
-                     val replayOperationsFrom: Long): Closeable {
+class RestoreContext(
+    val restoreUUID: String,
+    val shard: IndexShard,
+    val indexCommitRef: GatedCloseable<IndexCommit>,
+    val metadataSnapshot: Store.MetadataSnapshot,
+    val replayOperationsFrom: Long,
+) : Closeable {
 
-    companion object {
-        private const val INITIAL_FILE_CACHE_CAPACITY = 20
-    }
-    private val currentFiles = LinkedHashMap<String, IndexInput>(INITIAL_FILE_CACHE_CAPACITY)
+    private val currentFiles = ConcurrentHashMap<String, IndexInput>(INITIAL_FILE_CACHE_CAPACITY)
+
+    private val fileLocks = ConcurrentHashMap<String, ReentrantLock>()
 
     fun openInput(store: Store, fileName: String): IndexInput {
-        var currentIndexInput = currentFiles.getOrDefault(fileName, null)
-        if(currentIndexInput != null) {
-            return currentIndexInput.clone()
-        }
-        store.performOp({
-            currentIndexInput = store.directory().openInput(fileName, IOContext.READONCE)
-        })
+        val lock = fileLocks.computeIfAbsent(fileName) { ReentrantLock() }
 
-        currentFiles[fileName] = currentIndexInput!!
-        return currentIndexInput!!.clone()
+        lock.withLock {
+            val baseInput = currentFiles.computeIfAbsent(fileName) {
+                store.directory().openInput(fileName, IOContext.DEFAULT)
+            }
+            return baseInput.clone()
path: src/main/kotlin/org/opensearch/replication/repository/RestoreContext.kt
user: login: 10000-ki
body: The code has been updated to be thread-safe, allowing safe access to index files even when multiple threads attempt to open the same file simultaneously.
created_at: 2025-04-07T09:13:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,49 @@
+package org.opensearch.replication.repository
+
+import org.apache.lucene.index.IndexCommit
+import org.apache.lucene.store.IOContext
+import org.apache.lucene.store.IndexInput
+import org.mockito.ArgumentMatchers.any
+import org.mockito.ArgumentMatchers.eq
+import org.mockito.Mockito.*
+import org.opensearch.common.concurrent.GatedCloseable
+import org.opensearch.index.shard.IndexShard
+import org.opensearch.index.store.Store
+import org.opensearch.test.OpenSearchTestCase
+import java.util.UUID
+
+class RestoreContextTests : OpenSearchTestCase() {
+
+    fun `test openInput returns cloned IndexInput from cache`() {
+        // given
+        val fileName = "test_file"
+
+        val mockShard = mock(IndexShard::class.java)
+        val mockIndexCommit = mock(GatedCloseable::class.java) as GatedCloseable<IndexCommit>
+        val mockStore = mock(Store::class.java)
+        val mockDirectory = mock(org.apache.lucene.store.Directory::class.java)
+        val mockBaseInput = mock(IndexInput::class.java)
+        val mockClonedInput = mock(IndexInput::class.java)
+
+        // when
+        `when`(mockStore.directory()).thenReturn(mockDirectory)
+        `when`(mockDirectory.openInput(eq(fileName), any(IOContext::class.java))).thenReturn(mockBaseInput)
+        `when`(mockBaseInput.clone()).thenReturn(mockClonedInput)
+        val sut = RestoreContext(
+            restoreUUID = UUID.randomUUID().toString(),
+            shard = mockShard,
+            indexCommitRef = mockIndexCommit,
+            metadataSnapshot = Store.MetadataSnapshot.EMPTY,
+            replayOperationsFrom = 0L,
+        )
+
+        val result1 = sut.openInput(mockStore, fileName)
+        val result2 = sut.openInput(mockStore, fileName)
+
+        // then
+        assertSame(mockClonedInput, result1)
+        assertSame(mockClonedInput, result2)
+        verify(mockDirectory, times(1)).openInput(eq(fileName), any(IOContext::class.java))
+        verify(mockBaseInput, times(2)).clone()
path: src/test/kotlin/org/opensearch/replication/repository/RestoreContextTests.kt
user: login: 10000-ki
body: add test case
created_at: 2025-04-07T09:13:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -12,47 +12,47 @@
 package org.opensearch.replication.repository
 
 import org.apache.lucene.index.IndexCommit
-import org.opensearch.replication.util.performOp
 import org.apache.lucene.store.IOContext
 import org.apache.lucene.store.IndexInput
-import org.opensearch.OpenSearchException
 import org.opensearch.common.concurrent.GatedCloseable
-import org.opensearch.index.engine.Engine
 import org.opensearch.index.shard.IndexShard
 import org.opensearch.index.store.Store
 import java.io.Closeable
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.locks.ReentrantLock
+import kotlin.concurrent.withLock
 
-class RestoreContext(val restoreUUID: String,
-                     val shard: IndexShard,
-                     val indexCommitRef: GatedCloseable<IndexCommit>,
-                     val metadataSnapshot: Store.MetadataSnapshot,
-                     val replayOperationsFrom: Long): Closeable {
+class RestoreContext(
+    val restoreUUID: String,
+    val shard: IndexShard,
+    val indexCommitRef: GatedCloseable<IndexCommit>,
+    val metadataSnapshot: Store.MetadataSnapshot,
+    val replayOperationsFrom: Long,
+) : Closeable {
 
-    companion object {
-        private const val INITIAL_FILE_CACHE_CAPACITY = 20
-    }
-    private val currentFiles = LinkedHashMap<String, IndexInput>(INITIAL_FILE_CACHE_CAPACITY)
+    private val currentFiles = ConcurrentHashMap<String, IndexInput>(INITIAL_FILE_CACHE_CAPACITY)
+
+    private val fileLocks = ConcurrentHashMap<String, ReentrantLock>()
 
     fun openInput(store: Store, fileName: String): IndexInput {
-        var currentIndexInput = currentFiles.getOrDefault(fileName, null)
-        if(currentIndexInput != null) {
-            return currentIndexInput.clone()
-        }
-        store.performOp({
-            currentIndexInput = store.directory().openInput(fileName, IOContext.READONCE)
-        })
+        val lock = fileLocks.computeIfAbsent(fileName) { ReentrantLock() }
 
-        currentFiles[fileName] = currentIndexInput!!
-        return currentIndexInput!!.clone()
+        lock.withLock {
+            val baseInput = currentFiles.computeIfAbsent(fileName) {
+                store.directory().openInput(fileName, IOContext.DEFAULT)
path: src/main/kotlin/org/opensearch/replication/repository/RestoreContext.kt
user: login: 10000-ki
body: By changing this option, I expect that `MemorySegmentIndexInput` will no longer be used, and instead, a combination of `BufferedIndexInput` and `MMapIndexInput` will be used.
Although this may result in slightly reduced performance for random access reads, it seems to be an unavoidable trade-off given the current multi-threaded architecture of CCR.

For now, we should prioritize fixing the bug and then explore ways to further improve overall index read performance by optimizing the CCR architecture.
created_at: 2025-04-07T10:07:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -12,47 +12,47 @@
 package org.opensearch.replication.repository
 
 import org.apache.lucene.index.IndexCommit
-import org.opensearch.replication.util.performOp
 import org.apache.lucene.store.IOContext
 import org.apache.lucene.store.IndexInput
-import org.opensearch.OpenSearchException
 import org.opensearch.common.concurrent.GatedCloseable
-import org.opensearch.index.engine.Engine
 import org.opensearch.index.shard.IndexShard
 import org.opensearch.index.store.Store
 import java.io.Closeable
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.locks.ReentrantLock
+import kotlin.concurrent.withLock
 
-class RestoreContext(val restoreUUID: String,
-                     val shard: IndexShard,
-                     val indexCommitRef: GatedCloseable<IndexCommit>,
-                     val metadataSnapshot: Store.MetadataSnapshot,
-                     val replayOperationsFrom: Long): Closeable {
+class RestoreContext(
+    val restoreUUID: String,
+    val shard: IndexShard,
+    val indexCommitRef: GatedCloseable<IndexCommit>,
+    val metadataSnapshot: Store.MetadataSnapshot,
+    val replayOperationsFrom: Long,
+) : Closeable {
 
-    companion object {
-        private const val INITIAL_FILE_CACHE_CAPACITY = 20
-    }
-    private val currentFiles = LinkedHashMap<String, IndexInput>(INITIAL_FILE_CACHE_CAPACITY)
+    private val currentFiles = ConcurrentHashMap<String, IndexInput>(INITIAL_FILE_CACHE_CAPACITY)
+
+    private val fileLocks = ConcurrentHashMap<String, ReentrantLock>()
 
     fun openInput(store: Store, fileName: String): IndexInput {
-        var currentIndexInput = currentFiles.getOrDefault(fileName, null)
-        if(currentIndexInput != null) {
-            return currentIndexInput.clone()
-        }
-        store.performOp({
-            currentIndexInput = store.directory().openInput(fileName, IOContext.READONCE)
-        })
+        val lock = fileLocks.computeIfAbsent(fileName) { ReentrantLock() }
 
-        currentFiles[fileName] = currentIndexInput!!
-        return currentIndexInput!!.clone()
+        lock.withLock {
+            val baseInput = currentFiles.computeIfAbsent(fileName) {
+                store.directory().openInput(fileName, IOContext.DEFAULT)
+            }
+            return baseInput.clone()
+        }
     }
 
     override fun close() {
-        // Close all the open index input obj
-        currentFiles.entries.forEach {
-            it.value.close()
-        }
+        currentFiles.values.forEach { it.close() }
         currentFiles.clear()
+        fileLocks.clear()
         indexCommitRef.close()
     }
 
+    companion object {
+        private const val INITIAL_FILE_CACHE_CAPACITY = 20
+    }
path: src/main/kotlin/org/opensearch/replication/repository/RestoreContext.kt
user: login: 10000-ki
body: According to [Kotlin’s coding conventions](https://kotlinlang.org/docs/coding-conventions.html#class-layout), the companion object should be placed at the bottom of the class.
created_at: 2025-04-08T01:52:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -12,47 +12,47 @@
 package org.opensearch.replication.repository
 
 import org.apache.lucene.index.IndexCommit
-import org.opensearch.replication.util.performOp
 import org.apache.lucene.store.IOContext
 import org.apache.lucene.store.IndexInput
-import org.opensearch.OpenSearchException
 import org.opensearch.common.concurrent.GatedCloseable
-import org.opensearch.index.engine.Engine
 import org.opensearch.index.shard.IndexShard
 import org.opensearch.index.store.Store
 import java.io.Closeable
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.locks.ReentrantLock
+import kotlin.concurrent.withLock
 
-class RestoreContext(val restoreUUID: String,
-                     val shard: IndexShard,
-                     val indexCommitRef: GatedCloseable<IndexCommit>,
-                     val metadataSnapshot: Store.MetadataSnapshot,
-                     val replayOperationsFrom: Long): Closeable {
+class RestoreContext(
+    val restoreUUID: String,
+    val shard: IndexShard,
+    val indexCommitRef: GatedCloseable<IndexCommit>,
+    val metadataSnapshot: Store.MetadataSnapshot,
+    val replayOperationsFrom: Long,
+) : Closeable {
 
-    companion object {
-        private const val INITIAL_FILE_CACHE_CAPACITY = 20
-    }
-    private val currentFiles = LinkedHashMap<String, IndexInput>(INITIAL_FILE_CACHE_CAPACITY)
+    private val currentFiles = ConcurrentHashMap<String, IndexInput>(INITIAL_FILE_CACHE_CAPACITY)
+
+    private val fileLocks = ConcurrentHashMap<String, ReentrantLock>()
 
     fun openInput(store: Store, fileName: String): IndexInput {
-        var currentIndexInput = currentFiles.getOrDefault(fileName, null)
-        if(currentIndexInput != null) {
-            return currentIndexInput.clone()
-        }
-        store.performOp({
-            currentIndexInput = store.directory().openInput(fileName, IOContext.READONCE)
-        })
+        val lock = fileLocks.computeIfAbsent(fileName) { ReentrantLock() }
 
-        currentFiles[fileName] = currentIndexInput!!
-        return currentIndexInput!!.clone()
+        lock.withLock {
+            val baseInput = currentFiles.computeIfAbsent(fileName) {
+                store.directory().openInput(fileName, IOContext.DEFAULT)
path: src/main/kotlin/org/opensearch/replication/repository/RestoreContext.kt
user: login: mishail
body: Just an observation - `store.directory().openInput()` used to be within the`store.performOp` call (i.e. within `Store.incRef`/`Store.decRef`) and now it's not.
created_at: 2025-04-08T15:24:18Z
author_association: NONE
diff_hunk: @@ -12,47 +12,47 @@
 package org.opensearch.replication.repository
 
 import org.apache.lucene.index.IndexCommit
-import org.opensearch.replication.util.performOp
 import org.apache.lucene.store.IOContext
 import org.apache.lucene.store.IndexInput
-import org.opensearch.OpenSearchException
 import org.opensearch.common.concurrent.GatedCloseable
-import org.opensearch.index.engine.Engine
 import org.opensearch.index.shard.IndexShard
 import org.opensearch.index.store.Store
 import java.io.Closeable
+import java.util.concurrent.ConcurrentHashMap
+import java.util.concurrent.locks.ReentrantLock
+import kotlin.concurrent.withLock
 
-class RestoreContext(val restoreUUID: String,
-                     val shard: IndexShard,
-                     val indexCommitRef: GatedCloseable<IndexCommit>,
-                     val metadataSnapshot: Store.MetadataSnapshot,
-                     val replayOperationsFrom: Long): Closeable {
+class RestoreContext(
+    val restoreUUID: String,
+    val shard: IndexShard,
+    val indexCommitRef: GatedCloseable<IndexCommit>,
+    val metadataSnapshot: Store.MetadataSnapshot,
+    val replayOperationsFrom: Long,
+) : Closeable {
 
-    companion object {
-        private const val INITIAL_FILE_CACHE_CAPACITY = 20
-    }
-    private val currentFiles = LinkedHashMap<String, IndexInput>(INITIAL_FILE_CACHE_CAPACITY)
+    private val currentFiles = ConcurrentHashMap<String, IndexInput>(INITIAL_FILE_CACHE_CAPACITY)
+
+    private val fileLocks = ConcurrentHashMap<String, ReentrantLock>()
 
     fun openInput(store: Store, fileName: String): IndexInput {
-        var currentIndexInput = currentFiles.getOrDefault(fileName, null)
-        if(currentIndexInput != null) {
-            return currentIndexInput.clone()
-        }
-        store.performOp({
-            currentIndexInput = store.directory().openInput(fileName, IOContext.READONCE)
-        })
+        val lock = fileLocks.computeIfAbsent(fileName) { ReentrantLock() }
 
-        currentFiles[fileName] = currentIndexInput!!
-        return currentIndexInput!!.clone()
+        lock.withLock {
+            val baseInput = currentFiles.computeIfAbsent(fileName) {
+                store.directory().openInput(fileName, IOContext.DEFAULT)
path: src/main/kotlin/org/opensearch/replication/repository/RestoreContext.kt
user: login: 10000-ki
body: @Mishail 

> Just an observation - store.directory().openInput() used to be within thestore.performOp call (i.e. within Store.incRef/Store.decRef) and now it's not.

Thank you! I completely missed that, I’ll make the fix soon.
created_at: 2025-04-09T01:41:08Z
author_association: CONTRIBUTOR

PR comments:
user: login: 10000-ki
created_at: 2025-04-07T09:14:47Z
body: @ankitkala hi!

I made code changes as a follow-up to [this issue](https://github.com/opensearch-project/cross-cluster-replication/issues/1465).
Could you please take a look and review the changes?
author_association: CONTRIBUTOR
user: login: 10000-ki
created_at: 2025-04-08T10:22:08Z
body: I set up two OpenSearch clusters and configured Cross-Cluster Replication (CCR). Then, I tested whether CCR works correctly for indexes of different sizes.

## Spec

- leader:
   - hot node : 5
      - cpu : 8core
      - mem : 32gi
      - storage type : remote-ssd

- follower:
   - hot node : 5
      - cpu : 8core
      - mem : 32gi
      - storage type : remote-ssd

## Result

| Index Size  | Result  | Execute Time | Comment                  | config |
|------------|---------|--------------|--------------------------| ------|
| 10GB      | SUCCESS | 5 minutes    | No confined thread error | "plugins.replication.follower.index.recovery.chunk_size": 10mb
| 50GB      | SUCCESS | 14 minutes    | No confined thread error | "plugins.replication.follower.index.recovery.chunk_size": 1gb
| 150GB (primary 75GB + replica 75GB) | SUCCESS | 32 minutes  | No confined thread error | "plugins.replication.follower.index.recovery.chunk_size": 1gb

## More Detail 

### Sync Status (50GB)

```json
{
    "status": "SYNCING",
    "reason": "User initiated",
    "leader_alias": "leader",
    "leader_index": "ccr150gb-test",
    "follower_index": "ccr150gb-test",
    "syncing_details": {
        "leader_checkpoint": 565939,
        "follower_checkpoint": 565939,
        "seq_no": 565939
    },
    "bootstrap_details": {
        "bytes_total": 52972615872,
        "bytes_recovered": 42651962848,
        "bytes_percent": 80.52625,
        "files_total": 357,
        "files_recovered": 230,
        "files_percent": 66.10331,
        "start_time": 1744106558304,
        "running_time": 858621
    }
}
```

![스크린샷 2025-04-08 오후 7 10 38](https://github.com/user-attachments/assets/48f550af-6bdb-4813-b11b-9025fbc65dd9)
![스크린샷 2025-04-08 오후 7 10 57](https://github.com/user-attachments/assets/1bfc7ac7-2d32-4890-91c1-90e704867612)



## Conclusion

The confined error no longer occurs, so it seems that the issue we intended to resolve has been mostly addressed.
However, when the existing index contains a significant amount of data (e.g., over 50GB), it is necessary to increase the value of `plugins.replication.follower.index.recovery.chunk_size`.
(i recommend `1gb`)
Using the default value of `10MB` can result in excessive replication time during initial data recovery.
author_association: CONTRIBUTOR
user: login: 10000-ki
created_at: 2025-04-08T10:23:53Z
body: @ankitkala 

https://github.com/opensearch-project/cross-cluster-replication/pull/1520#issuecomment-2785960264

I’d like to share the test results.

> The confined error no longer occurs, so it seems that the issue we intended to resolve has been mostly addressed.
However, when the existing index contains a significant amount of data (e.g., over 50GB), it is necessary to increase the value of plugins.replication.follower.index.recovery.chunk_size.

I think it would be good to include this information in the CCR guide documentation.
author_association: CONTRIBUTOR
user: login: 10000-ki
created_at: 2025-04-08T10:25:43Z
body: Anyway, the following error no longer seems to occur

```
failed due to java.lang.IllegalStateException: confined
at org.apache.lucene.store.MemorySegmentIndexInput.ensureAccessible(MemorySegmentIndexInput.java:103)
at org.apache.lucene.store.MemorySegmentIndexInput.buildSlice(MemorySegmentIndexInput.java:461)
at org.apache.lucene.store.MemorySegmentIndexInput.clone(MemorySegmentIndexInput.java:425)
at org.apache.lucene.store.MemorySegmentIndexInput$SingleSegmentImpl.clone(MemorySegmentIndexInput.java:530)
at org.opensearch.replication.repository.RestoreContext.openInput(RestoreContext.kt:39)
at org.opensearch.replication.repository.RemoteClusterRestoreLeaderService.openInputStream(RemoteClusterRestoreLeaderService.kt:76)
...
```
author_association: CONTRIBUTOR