Repository: weaviate/weaviate
Pull request #8079: Shard replication additional host writes
Author: login: nathanwilk7
Created: 2025-05-09T06:07:33Z | Merged: 2025-05-14T17:59:43Z

### What's being changed:

- The target node receives "best effort writes" (eg fire and forget) during the finalizing phase. These best effort writes are done using the "additional host
- Adds the target node to the "write quorum" during the dehydrating phase, this is necessary since the source node no longer receives reads/writes during this phase and we have added the target node to the schema.
- Renamed config "minimum finalizing wait" to "minimum async wait"
- The finalizing async replication using time.now instead of the configured minimum async wait.
- The minimum async wait was moved to the dehydrating step. This is beneficial because it allows to ensure that ALL writes from the source node are on the target node after we've paused writes to the source node and before we delete the source node's shard replica. However, this means that for copies (eg not moves), we'll have to think about whether we want to add more guarantees around ensuring data consistency between shard replicas. For now, we want to make sure copy is working well.

Future
- Adds a test which inserts as many objects as possible during the entire replica movement operation. verifies that all the objects were inserted successfully, and that each node has returns all the objects. NOTE this test is currently flaky and so was commented out, will investigate and address the flakiness/root cause in a follow up PR.
- Decide if/how to handle replication factor = 1 (currently if replication factor is 1 then we don't hit any of the routing code which means we don't get best effort writes.
- More tests 

### Review checklist

- [ ] Documentation has been updated, if necessary. Link to changed documentation:
- [ ] Chaos pipeline run or not necessary. Link to pipeline:
- [ ] All new code is covered by tests where it is reasonable.
- [ ] Performance tests have been run or not necessary.

<!-- Uncomment the following section if this PR requires changes in related projects (e.g., documentation, client libraries).

GitHub actions will automatically create an issue in the corresponding repository for each checked box below. (See `.github/workflows/create-cross-functional-issues.yml`)

### Cross-functional impact

- [ ] This change requires public documentation (weaviate-io) to be updated. Check the box to automatically create a corresponding issue.
- Does it require a change in the client libraries? If yes, please check the boxes for the affected client libraries.
    - [ ] Python (weaviate-python-client)
    - [ ] JavaScript/TypeScript (typescript-client)
    - [ ] Go (weaviate-go-client)
    - [ ] Java (java-client)

-->



Diff:
diff --git a/adapters/handlers/rest/configure_api.go b/adapters/handlers/rest/configure_api.go
index ace6ca56599..111994b4e48 100644
--- a/adapters/handlers/rest/configure_api.go
+++ b/adapters/handlers/rest/configure_api.go
@@ -486,39 +486,39 @@ func MakeAppState(ctx context.Context, options *swag.CommandLineOptionsGroup) *s
 	schemaParser := schema.NewParser(appState.Cluster, vectorIndex.ParseAndValidateConfig, migrator, appState.Modules)
 	replicaCopier := copier.New(remoteIndexClient, appState.Cluster, dataPath, appState.DB, appState.Logger)
 	rConfig := rCluster.Config{
-		WorkDir:                              filepath.Join(dataPath, config.DefaultRaftDir),
-		NodeID:                               nodeName,
-		Host:                                 addrs[0],
-		RaftPort:                             appState.ServerConfig.Config.Raft.Port,
-		RPCPort:                              appState.ServerConfig.Config.Raft.InternalRPCPort,
-		RaftRPCMessageMaxSize:                appState.ServerConfig.Config.Raft.RPCMessageMaxSize,
-		BootstrapTimeout:                     appState.ServerConfig.Config.Raft.BootstrapTimeout,
-		BootstrapExpect:                      appState.ServerConfig.Config.Raft.BootstrapExpect,
-		HeartbeatTimeout:                     appState.ServerConfig.Config.Raft.HeartbeatTimeout,
-		ElectionTimeout:                      appState.ServerConfig.Config.Raft.ElectionTimeout,
-		SnapshotInterval:                     appState.ServerConfig.Config.Raft.SnapshotInterval,
-		SnapshotThreshold:                    appState.ServerConfig.Config.Raft.SnapshotThreshold,
-		TrailingLogs:                         appState.ServerConfig.Config.Raft.TrailingLogs,
-		ConsistencyWaitTimeout:               appState.ServerConfig.Config.Raft.ConsistencyWaitTimeout,
-		MetadataOnlyVoters:                   appState.ServerConfig.Config.Raft.MetadataOnlyVoters,
-		EnableOneNodeRecovery:                appState.ServerConfig.Config.Raft.EnableOneNodeRecovery,
-		ForceOneNodeRecovery:                 appState.ServerConfig.Config.Raft.ForceOneNodeRecovery,
-		DB:                                   nil,
-		Parser:                               schemaParser,
-		NodeNameToPortMap:                    server2port,
-		NodeSelector:                         appState.Cluster,
-		Logger:                               appState.Logger,
-		IsLocalHost:                          appState.ServerConfig.Config.Cluster.Localhost,
-		LoadLegacySchema:                     schemaRepo.LoadLegacySchema,
-		SaveLegacySchema:                     schemaRepo.SaveLegacySchema,
-		SentryEnabled:                        appState.ServerConfig.Config.Sentry.Enabled,
-		AuthzController:                      appState.AuthzController,
-		DynamicUserController:                appState.APIKey.Dynamic,
-		ReplicaCopier:                        replicaCopier,
-		AuthNConfig:                          appState.ServerConfig.Config.Authentication,
-		ReplicationEngineMaxWorkers:          appState.ServerConfig.Config.ReplicationEngineMaxWorkers,
-		DistributedTasks:                     appState.ServerConfig.Config.DistributedTasks,
-		ReplicaMovementMinimumFinalizingWait: appState.ServerConfig.Config.ReplicaMovementMinimumFinalizingWait,
+		WorkDir:                         filepath.Join(dataPath, config.DefaultRaftDir),
+		NodeID:                          nodeName,
+		Host:                            addrs[0],
+		RaftPort:                        appState.ServerConfig.Config.Raft.Port,
+		RPCPort:                         appState.ServerConfig.Config.Raft.InternalRPCPort,
+		RaftRPCMessageMaxSize:           appState.ServerConfig.Config.Raft.RPCMessageMaxSize,
+		BootstrapTimeout:                appState.ServerConfig.Config.Raft.BootstrapTimeout,
+		BootstrapExpect:                 appState.ServerConfig.Config.Raft.BootstrapExpect,
+		HeartbeatTimeout:                appState.ServerConfig.Config.Raft.HeartbeatTimeout,
+		ElectionTimeout:                 appState.ServerConfig.Config.Raft.ElectionTimeout,
+		SnapshotInterval:                appState.ServerConfig.Config.Raft.SnapshotInterval,
+		SnapshotThreshold:               appState.ServerConfig.Config.Raft.SnapshotThreshold,
+		TrailingLogs:                    appState.ServerConfig.Config.Raft.TrailingLogs,
+		ConsistencyWaitTimeout:          appState.ServerConfig.Config.Raft.ConsistencyWaitTimeout,
+		MetadataOnlyVoters:              appState.ServerConfig.Config.Raft.MetadataOnlyVoters,
+		EnableOneNodeRecovery:           appState.ServerConfig.Config.Raft.EnableOneNodeRecovery,
+		ForceOneNodeRecovery:            appState.ServerConfig.Config.Raft.ForceOneNodeRecovery,
+		DB:                              nil,
+		Parser:                          schemaParser,
+		NodeNameToPortMap:               server2port,
+		NodeSelector:                    appState.Cluster,
+		Logger:                          appState.Logger,
+		IsLocalHost:                     appState.ServerConfig.Config.Cluster.Localhost,
+		LoadLegacySchema:                schemaRepo.LoadLegacySchema,
+		SaveLegacySchema:                schemaRepo.SaveLegacySchema,
+		SentryEnabled:                   appState.ServerConfig.Config.Sentry.Enabled,
+		AuthzController:                 appState.AuthzController,
+		DynamicUserController:           appState.APIKey.Dynamic,
+		ReplicaCopier:                   replicaCopier,
+		AuthNConfig:                     appState.ServerConfig.Config.Authentication,
+		ReplicationEngineMaxWorkers:     appState.ServerConfig.Config.ReplicationEngineMaxWorkers,
+		DistributedTasks:                appState.ServerConfig.Config.DistributedTasks,
+		ReplicaMovementMinimumAsyncWait: appState.ServerConfig.Config.ReplicaMovementMinimumAsyncWait,
 	}
 	for _, name := range appState.ServerConfig.Config.Raft.Join[:rConfig.BootstrapExpect] {
 		if strings.Contains(name, rConfig.NodeID) {
@@ -1754,7 +1754,7 @@ func initRuntimeOverrides(appState *state.State) {
 		registered.MaximumAllowedCollectionsCount = appState.ServerConfig.Config.SchemaHandlerConfig.MaximumAllowedCollectionsCount
 		registered.AsyncReplicationDisabled = appState.ServerConfig.Config.Replication.AsyncReplicationDisabled
 		registered.AutoschemaEnabled = appState.ServerConfig.Config.AutoSchema.Enabled
-		registered.ReplicaMovementMinimumFinalizingWait = appState.ServerConfig.Config.ReplicaMovementMinimumFinalizingWait
+		registered.ReplicaMovementMinimumAsyncWait = appState.ServerConfig.Config.ReplicaMovementMinimumAsyncWait
 
 		cm, err := configRuntime.NewConfigManager(
 			appState.ServerConfig.Config.RuntimeOverrides.Path,
diff --git a/adapters/repos/db/index.go b/adapters/repos/db/index.go
index 48fa60b54a7..0fe7fe02729 100644
--- a/adapters/repos/db/index.go
+++ b/adapters/repos/db/index.go
@@ -755,6 +755,8 @@ func (i *Index) putObject(ctx context.Context, object *storobj.Object,
 		return nil
 	}
 
+	// TODO how to support "additional host writes" with RF 1 during shard replica movement?
+
 	// no replication, remote shard (or local not yet inited)
 	shard, release, err := i.GetShard(ctx, shardName)
 	if err != nil {
diff --git a/cluster/replication/consumer.go b/cluster/replication/consumer.go
index 75ca9f8c937..dafc1265100 100644
--- a/cluster/replication/consumer.go
+++ b/cluster/replication/consumer.go
@@ -434,12 +434,10 @@ func (c *CopyOpConsumer) processFinalizingOp(ctx context.Context, op ShardReplic
 		logger.WithError(err).Error("failure while initializing async replication on local node")
 		return api.ShardReplicationState(""), err
 	}
-	defer c.replicaCopier.RevertAsyncReplicationLocally(ctx, op.Op.TargetShard.CollectionId, op.Op.SourceShard.ShardId)
 
-	// TODO start best effort writes before upper time bound is hit
-	// TODO make sure/test reads sent to target node do not use target node until op is ready/done and that writes
-	// received during movement work as expected
-	asyncReplicationUpperTimeBoundUnixMillis := time.Now().Add(c.asyncReplicationMinimumWait.Get()).UnixMilli()
+	// this time will be used to make sure async replication has propagated any writes which
+	// were received during the hydrating phase
+	asyncReplicationUpperTimeBoundUnixMillis := time.Now().UnixMilli()
 
 	// start async replication from source node to target node
 	targetNodeOverride := additional.AsyncReplicationTargetNodeOverride{
@@ -452,7 +450,6 @@ func (c *CopyOpConsumer) processFinalizingOp(ctx context.Context, op ShardReplic
 	if err := c.replicaCopier.AddAsyncReplicationTargetNode(ctx, targetNodeOverride); err != nil {
 		return api.ShardReplicationState(""), err
 	}
-	defer c.replicaCopier.RemoveAsyncReplicationTargetNode(ctx, targetNodeOverride)
 
 	do := func() bool {
 		asyncReplicationStatus, err := c.replicaCopier.AsyncReplicationStatus(ctx, op.Op.SourceShard.NodeId, op.Op.TargetShard.NodeId, op.Op.SourceShard.CollectionId, op.Op.SourceShard.ShardId)
@@ -493,6 +490,8 @@ func (c *CopyOpConsumer) processFinalizingOp(ctx context.Context, op ShardReplic
 
 	switch op.Op.TransferType {
 	case api.COPY:
+		c.replicaCopier.RevertAsyncReplicationLocally(ctx, op.Op.TargetShard.CollectionId, op.Op.SourceShard.ShardId)
+		c.replicaCopier.RemoveAsyncReplicationTargetNode(ctx, targetNodeOverride)
 		return api.READY, nil
 	case api.MOVE:
 		return api.DEHYDRATING, nil
@@ -506,6 +505,55 @@ func (c *CopyOpConsumer) processDehydratingOp(ctx context.Context, op ShardRepli
 	logger := getLoggerForOp(c.logger.Logger, op.Op)
 	logger.Info("processing dehydrating replication operation")
 
+	asyncReplicationUpperTimeBoundUnixMillis := time.Now().Add(c.asyncReplicationMinimumWait.Get()).UnixMilli()
+
+	// Async replication was started in processFinalizingOp, but here we want to "increase" the upper time bound
+	// to make sure any writes received by the source node before the op entered the DEHYDRATING state are
+	// propagated to the target node. We assume writes will complete or time out (default 90s) within the
+	// asyncReplicationMinimumWait time (default 100s). The source node should not receive any writes after the op
+	// enters the DEHYDRATING state.
+	targetNodeOverride := additional.AsyncReplicationTargetNodeOverride{
+		CollectionID:   op.Op.SourceShard.CollectionId,
+		ShardID:        op.Op.TargetShard.ShardId,
+		TargetNode:     op.Op.TargetShard.NodeId,
+		SourceNode:     op.Op.SourceShard.NodeId,
+		UpperTimeBound: asyncReplicationUpperTimeBoundUnixMillis,
+	}
+	defer c.replicaCopier.RemoveAsyncReplicationTargetNode(ctx, targetNodeOverride)
+	defer c.replicaCopier.RevertAsyncReplicationLocally(ctx, op.Op.TargetShard.CollectionId, op.Op.SourceShard.ShardId)
+
+	if ctx.Err() != nil {
+		logger.WithError(ctx.Err()).Debug("context cancelled, stopping replication operation")
+		return api.ShardReplicationState(""), ctx.Err()
+	}
+
+	if err := c.replicaCopier.AddAsyncReplicationTargetNode(ctx, targetNodeOverride); err != nil {
+		return api.ShardReplicationState(""), err
+	}
+
+	do := func() bool {
+		asyncReplicationStatus, err := c.replicaCopier.AsyncReplicationStatus(ctx, op.Op.SourceShard.NodeId, op.Op.TargetShard.NodeId, op.Op.SourceShard.CollectionId, op.Op.SourceShard.ShardId)
+		c.logger.WithFields(logrus.Fields{"err": err, "objects_propagated": asyncReplicationStatus.ObjectsPropagated, "start_diff_time_unix_millis": asyncReplicationStatus.StartDiffTimeUnixMillis, "upper_time_bound_unix_millis": asyncReplicationUpperTimeBoundUnixMillis}).Info("async replication status")
+		return err == nil && asyncReplicationStatus.ObjectsPropagated == 0 && asyncReplicationStatus.StartDiffTimeUnixMillis >= asyncReplicationUpperTimeBoundUnixMillis
+	}
+
+	if ctx.Err() != nil {
+		logger.WithError(ctx.Err()).Debug("error while processing replication operation, shutting down")
+		return api.ShardReplicationState(""), ctx.Err()
+	}
+	// we only check the status of the async replication every 5 seconds to avoid
+	// spamming with too many requests too quickly
+	err := backoff.Retry(func() error {
+		if do() {
+			return nil
+		}
+		return errors.New("async replication not done")
+	}, backoff.WithContext(backoff.NewConstantBackOff(asyncStatusInterval), ctx))
+	if err != nil {
+		logger.WithError(err).Error("failure while waiting for async replication to complete")
+		return api.ShardReplicationState(""), err
+	}
+
 	if ctx.Err() != nil {
 		logger.WithError(ctx.Err()).Debug("context cancelled, stopping replication operation")
 		return api.ShardReplicationState(""), ctx.Err()
diff --git a/cluster/replication/shard_replication_fsm.go b/cluster/replication/shard_replication_fsm.go
index 874e9d14d1e..bc0267c62ca 100644
--- a/cluster/replication/shard_replication_fsm.go
+++ b/cluster/replication/shard_replication_fsm.go
@@ -233,17 +233,55 @@ func (s *ShardReplicationFSM) GetOpState(op ShardReplicationOp) (ShardReplicatio
 	return v, ok
 }
 
-func (s *ShardReplicationFSM) FilterOneShardReplicasReadWrite(collection string, shard string, shardReplicasLocation []string) ([]string, []string) {
+func (s *ShardReplicationFSM) FilterOneShardReplicasRead(collection string, shard string, shardReplicasLocation []string) []string {
 	s.opsLock.RLock()
 	defer s.opsLock.RUnlock()
 
-	_, ok := s.opsByCollectionAndShard[collection][shard]
 	// Check if the specified shard is current undergoing replication at all.
-	// If not we can return early as all replicas can be used for read/writes
+	// If not we can return early as all replicas can be used for reads
+	byCollection, ok := s.opsByCollectionAndShard[collection]
 	if !ok {
-		return shardReplicasLocation, shardReplicasLocation
+		return shardReplicasLocation
+	}
+	_, ok = byCollection[shard]
+	if !ok {
+		return shardReplicasLocation
+	}
+	readReplicas, _ := s.readWriteReplicas(collection, shard, shardReplicasLocation)
+	return readReplicas
+}
+
+func (s *ShardReplicationFSM) FilterOneShardReplicasWrite(collection string, shard string, shardReplicasLocation []string) ([]string, []string) {
+	s.opsLock.RLock()
+	defer s.opsLock.RUnlock()
+
+	// Check if the specified shard is current undergoing replication at all.
+	// If not we can return early as all replicas can be used for writes
+	byCollection, ok := s.opsByCollectionAndShard[collection]
+	if !ok {
+		return shardReplicasLocation, []string{}
+	}
+	ops, ok := byCollection[shard]
+	if !ok {
+		return shardReplicasLocation, []string{}
 	}
 
+	_, writeReplicas := s.readWriteReplicas(collection, shard, shardReplicasLocation)
+
+	additionalWriteReplicas := []string{}
+	for _, op := range ops {
+		opState, ok := s.GetOpState(op)
+		if !ok {
+			continue
+		}
+		if opState.GetCurrentState() == api.FINALIZING {
+			additionalWriteReplicas = append(additionalWriteReplicas, op.TargetShard.NodeId)
+		}
+	}
+	return writeReplicas, additionalWriteReplicas
+}
+
+func (s *ShardReplicationFSM) readWriteReplicas(collection, shard string, shardReplicasLocation []string) ([]string, []string) {
 	readReplicas := make([]string, 0, len(shardReplicasLocation))
 	writeReplicas := make([]string, 0, len(shardReplicasLocation))
 	for _, shardReplicaLocation := range shardReplicasLocation {
@@ -255,7 +293,6 @@ func (s *ShardReplicationFSM) FilterOneShardReplicasReadWrite(collection string,
 			writeReplicas = append(writeReplicas, shardReplicaLocation)
 		}
 	}
-
 	return readReplicas, writeReplicas
 }
 
@@ -281,6 +318,9 @@ func (s *ShardReplicationFSM) filterOneReplicaReadWrite(node string, collection
 	switch opState.GetCurrentState() {
 	case api.FINALIZING:
 		writeOk = true
+	case api.DEHYDRATING:
+		writeOk = true
+		readOk = true
 	case api.READY:
 		readOk = true
 		writeOk = true
diff --git a/cluster/replication/types/mock_replication_fsm_reader.go b/cluster/replication/types/mock_replication_fsm_reader.go
index 957135d1d00..9a9553237d1 100644
--- a/cluster/replication/types/mock_replication_fsm_reader.go
+++ b/cluster/replication/types/mock_replication_fsm_reader.go
@@ -28,12 +28,62 @@ func (_m *MockReplicationFSMReader) EXPECT() *MockReplicationFSMReader_Expecter
 	return &MockReplicationFSMReader_Expecter{mock: &_m.Mock}
 }
 
-// FilterOneShardReplicasReadWrite provides a mock function with given fields: collection, shard, shardReplicasLocation
-func (_m *MockReplicationFSMReader) FilterOneShardReplicasReadWrite(collection string, shard string, shardReplicasLocation []string) ([]string, []string) {
+// FilterOneShardReplicasRead provides a mock function with given fields: collection, shard, shardReplicasLocation
+func (_m *MockReplicationFSMReader) FilterOneShardReplicasRead(collection string, shard string, shardReplicasLocation []string) []string {
 	ret := _m.Called(collection, shard, shardReplicasLocation)
 
 	if len(ret) == 0 {
-		panic("no return value specified for FilterOneShardReplicasReadWrite")
+		panic("no return value specified for FilterOneShardReplicasRead")
+	}
+
+	var r0 []string
+	if rf, ok := ret.Get(0).(func(string, string, []string) []string); ok {
+		r0 = rf(collection, shard, shardReplicasLocation)
+	} else {
+		if ret.Get(0) != nil {
+			r0 = ret.Get(0).([]string)
+		}
+	}
+
+	return r0
+}
+
+// MockReplicationFSMReader_FilterOneShardReplicasRead_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'FilterOneShardReplicasRead'
+type MockReplicationFSMReader_FilterOneShardReplicasRead_Call struct {
+	*mock.Call
+}
+
+// FilterOneShardReplicasRead is a helper method to define mock.On call
+//   - collection string
+//   - shard string
+//   - shardReplicasLocation []string
+func (_e *MockReplicationFSMReader_Expecter) FilterOneShardReplicasRead(collection interface{}, shard interface{}, shardReplicasLocation interface{}) *MockReplicationFSMReader_FilterOneShardReplicasRead_Call {
+	return &MockReplicationFSMReader_FilterOneShardReplicasRead_Call{Call: _e.mock.On("FilterOneShardReplicasRead", collection, shard, shardReplicasLocation)}
+}
+
+func (_c *MockReplicationFSMReader_FilterOneShardReplicasRead_Call) Run(run func(collection string, shard string, shardReplicasLocation []string)) *MockReplicationFSMReader_FilterOneShardReplicasRead_Call {
+	_c.Call.Run(func(args mock.Arguments) {
+		run(args[0].(string), args[1].(string), args[2].([]string))
+	})
+	return _c
+}
+
+func (_c *MockReplicationFSMReader_FilterOneShardReplicasRead_Call) Return(_a0 []string) *MockReplicationFSMReader_FilterOneShardReplicasRead_Call {
+	_c.Call.Return(_a0)
+	return _c
+}
+
+func (_c *MockReplicationFSMReader_FilterOneShardReplicasRead_Call) RunAndReturn(run func(string, string, []string) []string) *MockReplicationFSMReader_FilterOneShardReplicasRead_Call {
+	_c.Call.Return(run)
+	return _c
+}
+
+// FilterOneShardReplicasWrite provides a mock function with given fields: collection, shard, shardReplicasLocation
+func (_m *MockReplicationFSMReader) FilterOneShardReplicasWrite(collection string, shard string, shardReplicasLocation []string) ([]string, []string) {
+	ret := _m.Called(collection, shard, shardReplicasLocation)
+
+	if len(ret) == 0 {
+		panic("no return value specified for FilterOneShardReplicasWrite")
 	}
 
 	var r0 []string
@@ -60,32 +110,32 @@ func (_m *MockReplicationFSMReader) FilterOneShardReplicasReadWrite(collection s
 	return r0, r1
 }
 
-// MockReplicationFSMReader_FilterOneShardReplicasReadWrite_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'FilterOneShardReplicasReadWrite'
-type MockReplicationFSMReader_FilterOneShardReplicasReadWrite_Call struct {
+// MockReplicationFSMReader_FilterOneShardReplicasWrite_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'FilterOneShardReplicasWrite'
+type MockReplicationFSMReader_FilterOneShardReplicasWrite_Call struct {
 	*mock.Call
 }
 
-// FilterOneShardReplicasReadWrite is a helper method to define mock.On call
+// FilterOneShardReplicasWrite is a helper method to define mock.On call
 //   - collection string
 //   - shard string
 //   - shardReplicasLocation []string
-func (_e *MockReplicationFSMReader_Expecter) FilterOneShardReplicasReadWrite(collection interface{}, shard interface{}, shardReplicasLocation interface{}) *MockReplicationFSMReader_FilterOneShardReplicasReadWrite_Call {
-	return &MockReplicationFSMReader_FilterOneShardReplicasReadWrite_Call{Call: _e.mock.On("FilterOneShardReplicasReadWrite", collection, shard, shardReplicasLocation)}
+func (_e *MockReplicationFSMReader_Expecter) FilterOneShardReplicasWrite(collection interface{}, shard interface{}, shardReplicasLocation interface{}) *MockReplicationFSMReader_FilterOneShardReplicasWrite_Call {
+	return &MockReplicationFSMReader_FilterOneShardReplicasWrite_Call{Call: _e.mock.On("FilterOneShardReplicasWrite", collection, shard, shardReplicasLocation)}
 }
 
-func (_c *MockReplicationFSMReader_FilterOneShardReplicasReadWrite_Call) Run(run func(collection string, shard string, shardReplicasLocation []string)) *MockReplicationFSMReader_FilterOneShardReplicasReadWrite_Call {
+func (_c *MockReplicationFSMReader_FilterOneShardReplicasWrite_Call) Run(run func(collection string, shard string, shardReplicasLocation []string)) *MockReplicationFSMReader_FilterOneShardReplicasWrite_Call {
 	_c.Call.Run(func(args mock.Arguments) {
 		run(args[0].(string), args[1].(string), args[2].([]string))
 	})
 	return _c
 }
 
-func (_c *MockReplicationFSMReader_FilterOneShardReplicasReadWrite_Call) Return(_a0 []string, _a1 []string) *MockReplicationFSMReader_FilterOneShardReplicasReadWrite_Call {
+func (_c *MockReplicationFSMReader_FilterOneShardReplicasWrite_Call) Return(_a0 []string, _a1 []string) *MockReplicationFSMReader_FilterOneShardReplicasWrite_Call {
 	_c.Call.Return(_a0, _a1)
 	return _c
 }
 
-func (_c *MockReplicationFSMReader_FilterOneShardReplicasReadWrite_Call) RunAndReturn(run func(string, string, []string) ([]string, []string)) *MockReplicationFSMReader_FilterOneShardReplicasReadWrite_Call {
+func (_c *MockReplicationFSMReader_FilterOneShardReplicasWrite_Call) RunAndReturn(run func(string, string, []string) ([]string, []string)) *MockReplicationFSMReader_FilterOneShardReplicasWrite_Call {
 	_c.Call.Return(run)
 	return _c
 }
diff --git a/cluster/replication/types/replication_fsm_reader.go b/cluster/replication/types/replication_fsm_reader.go
index 7e7fe4af20a..f5badb0fb47 100644
--- a/cluster/replication/types/replication_fsm_reader.go
+++ b/cluster/replication/types/replication_fsm_reader.go
@@ -12,5 +12,10 @@
 package types
 
 type ReplicationFSMReader interface {
-	FilterOneShardReplicasReadWrite(collection string, shard string, shardReplicasLocation []string) ([]string, []string)
+	// FilterOneShardReplicasRead returns the read and write replicas for a given shard
+	// It returns a tuple of readReplicas
+	FilterOneShardReplicasRead(collection string, shard string, shardReplicasLocation []string) []string
+	// FilterOneShardReplicasWrite returns the write replicas for a given shard
+	// It returns a tuple of (writeReplicas, additionalWriteReplicas)
+	FilterOneShardReplicasWrite(collection string, shard string, shardReplicasLocation []string) ([]string, []string)
 }
diff --git a/cluster/router/router.go b/cluster/router/router.go
index d228bdde42a..7bb70347c8d 100644
--- a/cluster/router/router.go
+++ b/cluster/router/router.go
@@ -38,25 +38,26 @@ func New(logger *logrus.Logger, clusterStateReader cluster.NodeSelector, metadat
 	}
 }
 
-func (r *Router) GetReadWriteReplicasLocation(collection string, shard string) ([]string, []string, error) {
+func (r *Router) GetReadWriteReplicasLocation(collection string, shard string) ([]string, []string, []string, error) {
 	replicas, err := r.metadataReader.ShardReplicas(collection, shard)
 	if err != nil {
-		return nil, nil, err
+		return nil, nil, nil, err
 	}
-	readReplicas, writeReplicas := r.replicationFSMReader.FilterOneShardReplicasReadWrite(collection, shard, replicas)
-	return readReplicas, writeReplicas, nil
+	readReplicas := r.replicationFSMReader.FilterOneShardReplicasRead(collection, shard, replicas)
+	writeReplicas, additionalWriteReplicas := r.replicationFSMReader.FilterOneShardReplicasWrite(collection, shard, replicas)
+	return readReplicas, writeReplicas, additionalWriteReplicas, nil
 }
 
-func (r *Router) GetWriteReplicasLocation(collection string, shard string) ([]string, error) {
-	_, writeReplicasLocation, err := r.GetReadWriteReplicasLocation(collection, shard)
+func (r *Router) GetWriteReplicasLocation(collection string, shard string) ([]string, []string, error) {
+	_, writeReplicasLocation, additionalWriteReplicas, err := r.GetReadWriteReplicasLocation(collection, shard)
 	if err != nil {
-		return nil, err
+		return nil, nil, err
 	}
-	return writeReplicasLocation, nil
+	return writeReplicasLocation, additionalWriteReplicas, nil
 }
 
 func (r *Router) GetReadReplicasLocation(collection string, shard string) ([]string, error) {
-	readReplicasLocation, _, err := r.GetReadWriteReplicasLocation(collection, shard)
+	readReplicasLocation, _, _, err := r.GetReadWriteReplicasLocation(collection, shard)
 	if err != nil {
 		return nil, err
 	}
@@ -73,6 +74,37 @@ func (r *Router) BuildReadRoutingPlan(params types.RoutingPlanBuildOptions) (typ
 		return types.RoutingPlan{}, fmt.Errorf("could not get read replicas location from sharding state: %w", err)
 	}
 
+	return r.routingPlanFromReplicas(params, replicas)
+}
+
+func (r *Router) BuildWriteRoutingPlan(params types.RoutingPlanBuildOptions) (types.RoutingPlan, error) {
+	if err := params.Validate(); err != nil {
+		return types.RoutingPlan{}, err
+	}
+
+	writeReplicas, additionalWriteReplicas, err := r.GetWriteReplicasLocation(params.Collection, params.Shard)
+	if err != nil {
+		return types.RoutingPlan{}, fmt.Errorf("could not get read replicas location from sharding state: %w", err)
+	}
+
+	routingPlan, err := r.routingPlanFromReplicas(params, writeReplicas)
+	if err != nil {
+		return types.RoutingPlan{}, err
+	}
+
+	for _, replica := range additionalWriteReplicas {
+		if replicaAddr, ok := r.clusterStateReader.NodeHostname(replica); ok {
+			routingPlan.AdditionalHostAddrs = append(routingPlan.AdditionalHostAddrs, replicaAddr)
+		}
+	}
+
+	return routingPlan, nil
+}
+
+func (r *Router) routingPlanFromReplicas(
+	params types.RoutingPlanBuildOptions,
+	replicas []string,
+) (types.RoutingPlan, error) {
 	routingPlan := types.RoutingPlan{
 		Collection:        params.Collection,
 		Shard:             params.Shard,
@@ -104,17 +136,14 @@ func (r *Router) BuildReadRoutingPlan(params types.RoutingPlanBuildOptions) (typ
 	if len(routingPlan.Replicas) == 0 {
 		return routingPlan, fmt.Errorf("no replicas found for class %s shard %s", routingPlan.Collection, routingPlan.Shard)
 	}
-
-	routingPlan.IntConsistencyLevel, err = routingPlan.ValidateConsistencyLevel()
+	cl, err := routingPlan.ValidateConsistencyLevel()
+	if err != nil {
+		return types.RoutingPlan{}, err
+	}
+	routingPlan.IntConsistencyLevel = cl
 	return routingPlan, err
 }
 
-func (r *Router) BuildWriteRoutingPlan(params types.RoutingPlanBuildOptions) (types.RoutingPlan, error) {
-	// TODO: See if there is any sense in having writes be propagated to the "new" shard currently.
-	// For now discarding that idea because we need doc id synced to avoid colisions
-	return r.BuildReadRoutingPlan(params)
-}
-
 func (r *Router) NodeHostname(nodeName string) (string, bool) {
 	return r.clusterStateReader.NodeHostname(nodeName)
 }
diff --git a/cluster/service.go b/cluster/service.go
index 1fe0daaf079..2f61b9e12de 100644
--- a/cluster/service.go
+++ b/cluster/service.go
@@ -88,7 +88,7 @@ func New(cfg Config, authZController authorization.Controller, snapshotter fsm.S
 		replication.NewOpsCache(),
 		replicationOperationTimeout,
 		cfg.ReplicationEngineMaxWorkers,
-		cfg.ReplicaMovementMinimumFinalizingWait,
+		cfg.ReplicaMovementMinimumAsyncWait,
 		metrics.NewReplicationEngineOpsCallbacks(prometheus.DefaultRegisterer),
 	)
 	replicationEngine := replication.NewShardReplicationEngine(
diff --git a/cluster/store.go b/cluster/store.go
index 0bc852236c0..44016dc64b7 100644
--- a/cluster/store.go
+++ b/cluster/store.go
@@ -168,8 +168,9 @@ type Config struct {
 	// DistributedTasks is the configuration for the distributed task manager.
 	DistributedTasks config.DistributedTasksConfig
 
-	// ReplicaMovementMinimumFinalizingWait is the upper time bound duration for replica movement operations.
-	ReplicaMovementMinimumFinalizingWait *runtime.DynamicValue[time.Duration]
+	// ReplicaMovementMinimumAsyncWait is the minimum time bound that replica movement operations will wait before
+	// async replication can complete.
+	ReplicaMovementMinimumAsyncWait *runtime.DynamicValue[time.Duration]
 }
 
 // Store is the implementation of RAFT on this local node. It will handle the local schema and RAFT operations (startup,
diff --git a/test/acceptance/replication/replica_replication/replica_replication_test.go b/test/acceptance/replication/replica_replication/replica_replication_test.go
index 6767a9d0d05..90901bedb30 100644
--- a/test/acceptance/replication/replica_replication/replica_replication_test.go
+++ b/test/acceptance/replication/replica_replication/replica_replication_test.go
@@ -532,8 +532,6 @@ func (suite *ReplicationTestSuite) TestReplicaMovementOneWriteExtraSlowFileCopy(
 				wg.Add(1)
 				enterrors.GoWrapper(func() {
 					defer wg.Done()
-					// TODO replace/remove this sleep once we have a test that constantly inserts in parallel
-					// during shard replica movement
 					// sleep 20s so that the source node has paused compaction but not resumed yet
 					time.Sleep(20 * time.Second)
 					for i := 0; i < numParagraphsInsertedWhileStarting; i++ {
@@ -544,6 +542,7 @@ func (suite *ReplicationTestSuite) TestReplicaMovementOneWriteExtraSlowFileCopy(
 								"contents": fmt.Sprintf("paragraph#%d", numParagraphsInsertedBeforeStart+i),
 							},
 							uuid.New().String(),
+							"",
 						)
 						require.NoError(t, err)
 						time.Sleep(time.Millisecond)
@@ -606,12 +605,13 @@ func (suite *ReplicationTestSuite) TestReplicaMovementOneWriteExtraSlowFileCopy(
 	})
 }
 
-func createObjectThreadSafe(uri string, class string, properties map[string]interface{}, id string) error {
+func createObjectThreadSafe(uri string, class string, properties map[string]interface{}, id string, tenant string) error {
 	// Define the data structure for the request body
 	type Object struct {
 		Class      string                 `json:"class"`
 		Properties map[string]interface{} `json:"properties"`
 		ID         string                 `json:"id"`
+		Tenant     string                 `json:"tenant,omitempty"`
 	}
 
 	// Create an instance of the object with sample data
@@ -619,6 +619,7 @@ func createObjectThreadSafe(uri string, class string, properties map[string]inte
 		Class:      class,
 		Properties: properties,
 		ID:         id,
+		Tenant:     tenant,
 	}
 
 	// Marshal the object to JSON
@@ -651,3 +652,266 @@ func createObjectThreadSafe(uri string, class string, properties map[string]inte
 
 	return nil
 }
+
+// TODO fix flake and uncomment
+// func (suite *ReplicationTestSuite) TestReplicaMovementTenantParallelWrites() {
+// 	t := suite.T()
+// 	mainCtx := context.Background()
+// 	logger, _ := logrustest.NewNullLogger()
+
+// 	clusterSize := 3
+// 	compose, err := docker.New().
+// 		WithWeaviateCluster(clusterSize).
+// 		WithText2VecContextionary().
+// 		WithWeaviateEnv("REPLICA_MOVEMENT_MINIMUM_ASYNC_WAIT", "10s").
+// 		Start(mainCtx)
+// 	require.Nil(t, err)
+// 	defer func() {
+// 		if err := compose.Terminate(mainCtx); err != nil {
+// 			t.Fatalf("failed to terminate test containers: %s", err.Error())
+// 		}
+// 	}()
+
+// 	_, cancel := context.WithTimeout(mainCtx, 5*time.Minute)
+// 	defer cancel()
+
+// 	helper.SetupClient(compose.GetWeaviate().URI())
+// 	paragraphClass := articles.ParagraphsClass()
+// 	articleClass := articles.ArticlesClass()
+
+// 	t.Run("create schema", func(t *testing.T) {
+// 		paragraphClass.ReplicationConfig = &models.ReplicationConfig{
+// 			Factor:       2,
+// 			AsyncEnabled: false,
+// 		}
+// 		paragraphClass.MultiTenancyConfig = &models.MultiTenancyConfig{
+// 			Enabled:              true,
+// 			AutoTenantActivation: true,
+// 			AutoTenantCreation:   true,
+// 		}
+// 		paragraphClass.Vectorizer = "text2vec-contextionary"
+// 		helper.CreateClass(t, paragraphClass)
+// 		articleClass.ReplicationConfig = &models.ReplicationConfig{
+// 			Factor:       2,
+// 			AsyncEnabled: false,
+// 		}
+// 		articleClass.MultiTenancyConfig = &models.MultiTenancyConfig{
+// 			Enabled:              true,
+// 			AutoTenantActivation: true,
+// 			AutoTenantCreation:   true,
+// 		}
+// 		helper.CreateClass(t, articleClass)
+// 	})
+
+// 	t.Run("insert initial paragraphs", func(t *testing.T) {
+// 		batch := make([]*models.Object, len(paragraphIDs))
+// 		for i, id := range paragraphIDs {
+// 			batch[i] = articles.NewParagraph().
+// 				WithID(id).
+// 				WithContents(fmt.Sprintf("paragraph#%d", i)).
+// 				WithTenant("tenant0").
+// 				Object()
+// 		}
+// 		common.CreateObjects(t, compose.GetWeaviate().URI(), batch)
+// 	})
+
+// 	parallelWriteWg := sync.WaitGroup{}
+// 	parallelWriteIDs := []string{}
+// 	replicationDone := make(chan struct{})
+// 	t.Run("start parallel writes", func(t *testing.T) {
+// 		parallelWriteWg.Add(1)
+// 		enterrors.GoWrapper(func() {
+// 			defer parallelWriteWg.Done()
+// 			containerId := 1
+// 			for {
+// 				select {
+// 				case <-replicationDone:
+// 					return
+// 				default:
+// 					newWriteId := uuid.New().String()
+// 					err = createObjectThreadSafe(
+// 						compose.ContainerURI(containerId),
+// 						paragraphClass.Class,
+// 						map[string]interface{}{
+// 							"contents": fmt.Sprintf("paragraph#%d", len(paragraphIDs)+len(parallelWriteIDs)),
+// 						},
+// 						newWriteId,
+// 						"tenant0",
+// 					)
+// 					assert.NoError(t, err, "error creating object on node with id %d", containerId)
+// 					parallelWriteIDs = append(parallelWriteIDs, newWriteId)
+// 					containerId++
+// 					if containerId >= clusterSize+1 {
+// 						containerId = 1
+// 					}
+// 				}
+// 			}
+// 		}, logger)
+// 	})
+
+// 	// any node can be chosen as the tenant source node (even if that node is down at the time of creation)
+// 	// so we dynamically find the source and target nodes
+// 	var opUuid strfmt.UUID
+// 	var shardName string
+// 	type nodeInfo struct {
+// 		nodeContainerIndex int
+// 		nodeURI            string
+// 		nodeName           string
+// 	}
+// 	sourceNode := nodeInfo{}
+// 	targetNode := nodeInfo{}
+// 	replicaNode := nodeInfo{}
+// 	allNodeInfos := []nodeInfo{}
+// 	// TODO test copy as well
+// 	transferType := api.MOVE.String()
+// 	t.Run("start replica replication to node3 for paragraph", func(t *testing.T) {
+// 		verbose := verbosity.OutputVerbose
+// 		params := nodes.NewNodesGetClassParams().WithOutput(&verbose).WithClassName(paragraphClass.Class)
+// 		body, clientErr := helper.Client(t).Nodes.NodesGetClass(params, nil)
+// 		require.NoError(t, clientErr)
+// 		require.NotNil(t, body.Payload)
+
+// 		hasFoundNode := false
+// 		hasFoundShard := false
+
+// 		// Find two source nodes that have shards
+// 		for i, node := range body.Payload.Nodes {
+// 			containerIndex := i + 1
+// 			allNodeInfos = append(allNodeInfos, nodeInfo{nodeURI: compose.ContainerURI(containerIndex), nodeName: node.Name, nodeContainerIndex: containerIndex})
+// 			if len(node.Shards) >= 1 {
+// 				hasFoundNode = true
+// 				for _, shard := range node.Shards {
+// 					if shard.Class != paragraphClass.Class {
+// 						continue
+// 					}
+// 					hasFoundShard = true
+// 					shardName = shard.Name
+// 					// i + 1 because stop/start routine are 1 based not 0
+// 					if sourceNode.nodeName == "" {
+// 						sourceNode = nodeInfo{nodeURI: compose.ContainerURI(containerIndex), nodeName: node.Name, nodeContainerIndex: containerIndex}
+// 					} else {
+// 						replicaNode = nodeInfo{nodeURI: compose.ContainerURI(containerIndex), nodeName: node.Name, nodeContainerIndex: containerIndex}
+// 					}
+// 				}
+// 			}
+// 		}
+
+// 		require.True(t, hasFoundShard, "could not find shard for class %s", paragraphClass.Class)
+// 		require.True(t, hasFoundNode, "could not find node with shards for paragraph")
+// 		require.NotEmpty(t, sourceNode.nodeName, "could not find two source nodes with shards for paragraph")
+// 		require.NotEmpty(t, sourceNode.nodeURI, "could not find two source nodes with shards for paragraph")
+// 		require.NotEmpty(t, replicaNode.nodeName, "could not find two source nodes with shards for paragraph")
+// 		require.NotEmpty(t, replicaNode.nodeURI, "could not find two source nodes with shards for paragraph")
+// 		require.NotEqual(t, sourceNode.nodeName, replicaNode.nodeName, "source and replica nodes are the same")
+// 		require.NotEqual(t, sourceNode.nodeURI, replicaNode.nodeURI, "source and replica nodes are the same")
+
+// 		// Choose a target node that is not one of the source nodes
+// 		for i, node := range body.Payload.Nodes {
+// 			if node.Name == sourceNode.nodeName || node.Name == replicaNode.nodeName {
+// 				continue
+// 			}
+// 			targetNode = nodeInfo{nodeURI: compose.ContainerURI(i + 1), nodeName: node.Name}
+// 			break
+// 		}
+
+// 		require.NotEmpty(t, targetNode, "could not find a target node different from the source nodes")
+
+// 		t.Logf("Starting replica replication from %s to %s for shard %s", sourceNode.nodeName, targetNode.nodeName, shardName)
+// 		resp, err := helper.Client(t).Replication.Replicate(
+// 			replication.NewReplicateParams().WithBody(
+// 				&models.ReplicationReplicateReplicaRequest{
+// 					CollectionID:        &paragraphClass.Class,
+// 					SourceNodeName:      &sourceNode.nodeName,
+// 					DestinationNodeName: &targetNode.nodeName,
+// 					ShardID:             &shardName,
+// 					TransferType:        &transferType,
+// 				},
+// 			),
+// 			nil,
+// 		)
+// 		require.NoError(t, err)
+// 		require.Equal(t, http.StatusOK, resp.Code(), "replication replicate operation didn't return 200 OK")
+// 		require.NotNil(t, resp.Payload)
+// 		require.NotNil(t, resp.Payload.ID)
+// 		require.NotEmpty(t, *resp.Payload.ID)
+// 		opUuid = *resp.Payload.ID
+// 	})
+
+// 	require.NotEmpty(t, opUuid, "opUuid is empty")
+// 	require.NotEmpty(t, sourceNode.nodeName, "sourceNode is empty")
+// 	require.NotEmpty(t, sourceNode.nodeURI, "sourceNode is empty")
+// 	require.NotEmpty(t, targetNode.nodeName, "targetNode is empty")
+// 	require.NotEmpty(t, targetNode.nodeURI, "targetNode is empty")
+// 	require.NotEmpty(t, replicaNode.nodeName, "replicaNode is empty")
+// 	require.NotEmpty(t, replicaNode.nodeURI, "replicaNode is empty")
+// 	require.NotEqual(t, sourceNode.nodeName, replicaNode.nodeName, "source and replica nodes are the same")
+// 	require.NotEqual(t, sourceNode.nodeURI, replicaNode.nodeURI, "source and replica nodes are the same")
+// 	require.NotEqual(t, targetNode.nodeName, sourceNode.nodeName, "target and source nodes are the same")
+// 	require.NotEqual(t, targetNode.nodeURI, sourceNode.nodeURI, "target and source nodes are the same")
+// 	require.NotEqual(t, targetNode.nodeName, replicaNode.nodeName, "target and replica nodes are the same")
+// 	require.NotEqual(t, targetNode.nodeURI, replicaNode.nodeURI, "target and replica nodes are the same")
+
+// 	// Wait for the replication to finish
+// 	t.Run("waiting for replication to finish", func(t *testing.T) {
+// 		assert.EventuallyWithT(t, func(ct *assert.CollectT) {
+// 			details, err := helper.Client(t).Replication.ReplicationDetails(
+// 				replication.NewReplicationDetailsParams().WithID(opUuid), nil,
+// 			)
+// 			assert.Nil(t, err, "failed to get replication details %s", err)
+// 			assert.NotNil(t, details, "expected replication details to be not nil")
+// 			assert.NotNil(t, details.Payload, "expected replication details payload to be not nil")
+// 			assert.NotNil(t, details.Payload.Status, "expected replication status to be not nil")
+// 			assert.Equal(ct, "READY", details.Payload.Status.State, "expected replication status to be READY")
+// 		}, 240*time.Second, 1*time.Second, "replication operation %s not finished in time", opUuid)
+// 		// let some writes keep going for a few seconds after the op is ready
+// 		time.Sleep(5 * time.Second)
+// 		// now stop the writes
+// 		close(replicationDone)
+// 		parallelWriteWg.Wait()
+// 	})
+
+// 	t.Run("all parallel writes are available", func(t *testing.T) {
+// 		numParallelWrites := len(parallelWriteIDs)
+// 		assert.True(t, numParallelWrites > 1000, "expected at least 1000 parallel writes")
+// 		for _, nodeInfo := range allNodeInfos {
+// 			// in a move, sourceNode no longer has the shard replica, so we skip it
+// 			if transferType == api.MOVE.String() && nodeInfo.nodeName == sourceNode.nodeName {
+// 				continue
+// 			}
+
+// 			assert.Equal(t, int64(numParallelWrites+len(paragraphIDs)), common.CountTenantObjects(t, nodeInfo.nodeURI, paragraphClass.Class, "tenant0"), fmt.Sprintf("expected %d objects on node %s", numParallelWrites+len(paragraphIDs), nodeInfo.nodeName))
+// 		}
+// 		for _, nodeInfo := range allNodeInfos {
+// 			firstMissingObjectForNode := ""
+// 			numMissingObjectsForNode := 0
+// 			for _, id := range parallelWriteIDs {
+// 				// in a move, sourceNode no longer has the shard replica, so we skip it
+// 				if transferType == api.MOVE.String() && nodeInfo.nodeName == sourceNode.nodeName {
+// 					continue
+// 				}
+
+// 				obj, err := common.GetTenantObjectFromNode(t, nodeInfo.nodeURI, paragraphClass.Class, strfmt.UUID(id), nodeInfo.nodeName, "tenant0")
+// 				if err != nil || obj == nil {
+// 					numMissingObjectsForNode++
+// 					if firstMissingObjectForNode == "" {
+// 						assert.Nil(t, err, "error getting object from node %s", nodeInfo.nodeName, id, err)
+// 						assert.NotNil(t, obj, "object not found on node", nodeInfo.nodeName, id)
+// 						firstMissingObjectForNode = id
+// 					}
+// 				}
+// 			}
+// 			assert.Equal(t, 0, numMissingObjectsForNode, "expected no missing objects on node %s", nodeInfo.nodeName)
+// 			assert.Empty(t, firstMissingObjectForNode, "expected no missing objects on node %s", nodeInfo.nodeName)
+// 		}
+// 	})
+
+// 	t.Run("assert data is available for paragraph on node3 with consistency level one", func(t *testing.T) {
+// 		assert.EventuallyWithT(t, func(ct *assert.CollectT) {
+// 			for _, objId := range paragraphIDs {
+// 				obj, err := common.GetTenantObjectFromNode(t, targetNode.nodeURI, paragraphClass.Class, objId, targetNode.nodeName, "tenant0")
+// 				assert.Nil(ct, err)
+// 				assert.NotNil(ct, obj)
+// 			}
+// 		}, 10*time.Second, 1*time.Second, "node3 doesn't have paragraph data")
+// 	})
+// }
diff --git a/usecases/config/config_handler.go b/usecases/config/config_handler.go
index 333634746d3..4349066c11d 100644
--- a/usecases/config/config_handler.go
+++ b/usecases/config/config_handler.go
@@ -163,7 +163,7 @@ type Config struct {
 
 	RuntimeOverrides RuntimeOverrides `json:"runtime_overrides" yaml:"runtime_overrides"`
 
-	ReplicaMovementMinimumFinalizingWait *runtime.DynamicValue[time.Duration] `json:"replica_movement_minimum_finalizing_wait" yaml:"replica_movement_minimum_finalizing_wait"`
+	ReplicaMovementMinimumAsyncWait *runtime.DynamicValue[time.Duration] `json:"REPLICA_MOVEMENT_MINIMUM_ASYNC_WAIT" yaml:"REPLICA_MOVEMENT_MINIMUM_ASYNC_WAIT"`
 }
 
 type MapToBlockamaxConfig struct {
diff --git a/usecases/config/environment.go b/usecases/config/environment.go
index e86fed53840..2ac33444d0d 100644
--- a/usecases/config/environment.go
+++ b/usecases/config/environment.go
@@ -46,8 +46,8 @@ const (
 	DefaultDistributedTasksSchedulerTickInterval = time.Minute
 	DefaultDistributedTasksCompletedTaskTTL      = 5 * 24 * time.Hour
 
-	DefaultReplicationEngineMaxWorkers          = 5
-	DefaultReplicaMovementMinimumFinalizingWait = 100 * time.Second
+	DefaultReplicationEngineMaxWorkers     = 5
+	DefaultReplicaMovementMinimumAsyncWait = 100 * time.Second
 
 	DefaultTransferInactivityTimeout = 5 * time.Minute
 )
@@ -686,17 +686,17 @@ func FromEnv(config *Config) error {
 		return err
 	}
 
-	if v := os.Getenv("REPLICA_MOVEMENT_MINIMUM_FINALIZING_WAIT"); v != "" {
+	if v := os.Getenv("REPLICA_MOVEMENT_MINIMUM_ASYNC_WAIT"); v != "" {
 		duration, err := time.ParseDuration(v)
 		if err != nil {
-			return fmt.Errorf("parse REPLICA_MOVEMENT_MINIMUM_FINALIZING_WAIT as time.Duration: %w", err)
+			return fmt.Errorf("parse REPLICA_MOVEMENT_MINIMUM_ASYNC_WAIT as time.Duration: %w", err)
 		}
 		if duration < 0 {
-			return fmt.Errorf("REPLICA_MOVEMENT_MINIMUM_FINALIZING_WAIT must be a positive duration")
+			return fmt.Errorf("REPLICA_MOVEMENT_MINIMUM_ASYNC_WAIT must be a positive duration")
 		}
-		config.ReplicaMovementMinimumFinalizingWait = runtime.NewDynamicValue(duration)
+		config.ReplicaMovementMinimumAsyncWait = runtime.NewDynamicValue(duration)
 	} else {
-		config.ReplicaMovementMinimumFinalizingWait = runtime.NewDynamicValue(DefaultReplicaMovementMinimumFinalizingWait)
+		config.ReplicaMovementMinimumAsyncWait = runtime.NewDynamicValue(DefaultReplicaMovementMinimumAsyncWait)
 	}
 
 	return nil
diff --git a/usecases/config/runtimeconfig.go b/usecases/config/runtimeconfig.go
index eda0c68af62..92a3a268b34 100644
--- a/usecases/config/runtimeconfig.go
+++ b/usecases/config/runtimeconfig.go
@@ -26,10 +26,10 @@ import (
 // WeaviateRuntimeConfig is the collection all the supported configs that is
 // managed dynamically and can be overridden during runtime.
 type WeaviateRuntimeConfig struct {
-	MaximumAllowedCollectionsCount       *runtime.DynamicValue[int]           `json:"maximum_allowed_collections_count" yaml:"maximum_allowed_collections_count"`
-	AutoschemaEnabled                    *runtime.DynamicValue[bool]          `json:"autoschema_enabled" yaml:"autoschema_enabled"`
-	AsyncReplicationDisabled             *runtime.DynamicValue[bool]          `json:"async_replication_disabled" yaml:"async_replication_disabled"`
-	ReplicaMovementMinimumFinalizingWait *runtime.DynamicValue[time.Duration] `json:"replica_movement_minimum_finalizing_wait" yaml:"replica_movement_minimum_finalizing_wait"`
+	MaximumAllowedCollectionsCount  *runtime.DynamicValue[int]           `json:"maximum_allowed_collections_count" yaml:"maximum_allowed_collections_count"`
+	AutoschemaEnabled               *runtime.DynamicValue[bool]          `json:"autoschema_enabled" yaml:"autoschema_enabled"`
+	AsyncReplicationDisabled        *runtime.DynamicValue[bool]          `json:"async_replication_disabled" yaml:"async_replication_disabled"`
+	ReplicaMovementMinimumAsyncWait *runtime.DynamicValue[time.Duration] `json:"REPLICA_MOVEMENT_MINIMUM_ASYNC_WAIT" yaml:"REPLICA_MOVEMENT_MINIMUM_ASYNC_WAIT"`
 }
 
 // ParseRuntimeConfig decode WeaviateRuntimeConfig from raw bytes of YAML.
diff --git a/usecases/config/runtimeconfig_test.go b/usecases/config/runtimeconfig_test.go
index 770e41f9186..8d9ff878ef5 100644
--- a/usecases/config/runtimeconfig_test.go
+++ b/usecases/config/runtimeconfig_test.go
@@ -47,16 +47,16 @@ func TestUpdateRuntimeConfig(t *testing.T) {
 		)
 
 		reg := &WeaviateRuntimeConfig{
-			MaximumAllowedCollectionsCount:       &colCount,
-			AutoschemaEnabled:                    &autoSchema,
-			AsyncReplicationDisabled:             &asyncRep,
-			ReplicaMovementMinimumFinalizingWait: &minFinWait,
+			MaximumAllowedCollectionsCount:  &colCount,
+			AutoschemaEnabled:               &autoSchema,
+			AsyncReplicationDisabled:        &asyncRep,
+			ReplicaMovementMinimumAsyncWait: &minFinWait,
 		}
 
 		// parsed from yaml configs for example
 		buf := []byte(`autoschema_enabled: true
 maximum_allowed_collections_count: 13
-replica_movement_minimum_finalizing_wait: 10s`)
+REPLICA_MOVEMENT_MINIMUM_ASYNC_WAIT: 10s`)
 		parsed, err := ParseRuntimeConfig(buf)
 		require.NoError(t, err)
 
@@ -86,16 +86,16 @@ replica_movement_minimum_finalizing_wait: 10s`)
 		)
 
 		reg := &WeaviateRuntimeConfig{
-			MaximumAllowedCollectionsCount:       &colCount,
-			AutoschemaEnabled:                    &autoSchema,
-			AsyncReplicationDisabled:             &asyncRep,
-			ReplicaMovementMinimumFinalizingWait: &minFinWait,
+			MaximumAllowedCollectionsCount:  &colCount,
+			AutoschemaEnabled:               &autoSchema,
+			AsyncReplicationDisabled:        &asyncRep,
+			ReplicaMovementMinimumAsyncWait: &minFinWait,
 		}
 
 		// parsed from yaml configs for example
 		buf := []byte(`autoschema_enabled: true
 maximum_allowed_collections_count: 13
-replica_movement_minimum_finalizing_wait: 10s`)
+REPLICA_MOVEMENT_MINIMUM_ASYNC_WAIT: 10s`)
 		parsed, err := ParseRuntimeConfig(buf)
 		require.NoError(t, err)
 
diff --git a/usecases/replica/coordinator.go b/usecases/replica/coordinator.go
index ec07a25afc9..3b81a021828 100644
--- a/usecases/replica/coordinator.go
+++ b/usecases/replica/coordinator.go
@@ -198,7 +198,16 @@ func (c *coordinator[T]) Push(ctx context.Context,
 		"level":    level,
 	}).Debug("context.WithTimeout")
 	nodeCh := c.broadcast(ctxWithTimeout, routingPlan.ReplicasHostAddrs, ask, level)
-	return c.commitAll(context.Background(), nodeCh, com), level, nil
+	commitCh := c.commitAll(context.Background(), nodeCh, com)
+
+	// if there are additional hosts, we do a "best effort" write to them
+	// where we don't wait for a response because they are not part of the
+	// replicas used to reach level consistency
+	if len(routingPlan.AdditionalHostAddrs) > 0 {
+		additionalHostsBroadcast := c.broadcast(ctxWithTimeout, routingPlan.AdditionalHostAddrs, ask, len(routingPlan.AdditionalHostAddrs))
+		c.commitAll(context.Background(), additionalHostsBroadcast, com)
+	}
+	return commitCh, level, nil
 }
 
 // Pull data from replica depending on consistency level, trying to reach level successful calls
diff --git a/usecases/replica/replicator_test.go b/usecases/replica/replicator_test.go
index 81b0bcf8e03..5767f8619bd 100644
--- a/usecases/replica/replicator_test.go
+++ b/usecases/replica/replicator_test.go
@@ -753,8 +753,11 @@ func (f *fakeFactory) newRouter(thisNode string) *clusterRouter.Router {
 		return v, nil
 	}).Maybe()
 	replicationFsmMock := replicationTypes.NewMockReplicationFSMReader(f.t)
-	replicationFsmMock.On("FilterOneShardReplicasReadWrite", mock.Anything, mock.Anything, mock.Anything).Return(func(collection string, shard string, shardReplicasLocation []string) ([]string, []string) {
-		return shardReplicasLocation, shardReplicasLocation
+	replicationFsmMock.On("FilterOneShardReplicasRead", mock.Anything, mock.Anything, mock.Anything).Return(func(collection string, shard string, shardReplicasLocation []string) []string {
+		return shardReplicasLocation
+	}).Maybe()
+	replicationFsmMock.On("FilterOneShardReplicasWrite", mock.Anything, mock.Anything, mock.Anything).Return(func(collection string, shard string, shardReplicasLocation []string) ([]string, []string) {
+		return shardReplicasLocation, []string{}
 	}).Maybe()
 	router := clusterRouter.New(f.log, clusterState, schemaReaderMock, replicationFsmMock)
 	return router


Reviews:
user: login: orca-security-eu[bot]
body: ### Orca Security Scan Summary
| Status  | Check | Issues by priority |   |
| ------- | ----- | ------------------ | - |
| <img width="16" alt="Passed" src="https://raw.githubusercontent.com/orcasecurity/orca-cli/main/resources/images/prcomment/status_v1/passed.png" title="Passed"> Passed | Secrets | <img width="12" alt="high" src="https://raw.githubusercontent.com/orcasecurity/orca-cli/main/resources/images/prcomment/priority/high.png" title="High"> 0 &emsp; <img width="12" alt="medium" src="https://raw.githubusercontent.com/orcasecurity/orca-cli/main/resources/images/prcomment/priority/medium.png" title="Medium"> 0 &emsp; <img width="12" alt="low" src="https://raw.githubusercontent.com/orcasecurity/orca-cli/main/resources/images/prcomment/priority/low.png" title="Low"> 0 &emsp; <img width="12" alt="info" src="https://raw.githubusercontent.com/orcasecurity/orca-cli/main/resources/images/prcomment/priority/info.png" title="Info"> 0 | <a href="https://eu.app.orcasecurity.io/shift-left/file_system_secret_detection/scan-log/be2266d0-1ff3-403f-a2ed-bfe9b92f1b95" target="_blank">View in Orca</a> |
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-09T06:08:39Z
user: login: nathanwilk7
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T06:08:54Z
user: login: reyreaud-l
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T14:09:46Z
user: login: reyreaud-l
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T14:15:53Z
user: login: reyreaud-l
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T14:15:54Z
user: login: reyreaud-l
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T14:17:38Z
user: login: reyreaud-l
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T14:34:24Z
user: login: jeroiraz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T15:34:39Z
user: login: nathanwilk7
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T07:07:57Z
user: login: nathanwilk7
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T07:16:33Z
user: login: nathanwilk7
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T07:34:10Z
user: login: reyreaud-l
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T08:10:38Z
user: login: reyreaud-l
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T08:12:12Z
user: login: reyreaud-l
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T08:22:57Z
user: login: reyreaud-l
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T14:49:24Z
user: login: reyreaud-l
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T15:26:08Z

Review comments:
diff_hunk: @@ -112,7 +112,56 @@ func (r *Router) BuildReadRoutingPlan(params types.RoutingPlanBuildOptions) (typ
 func (r *Router) BuildWriteRoutingPlan(params types.RoutingPlanBuildOptions) (types.RoutingPlan, error) {
 	// TODO: See if there is any sense in having writes be propagated to the "new" shard currently.
 	// For now discarding that idea because we need doc id synced to avoid colisions
-	return r.BuildReadRoutingPlan(params)
+	// return r.BuildReadRoutingPlan(params)
+	if err := params.Validate(); err != nil {
path: cluster/router/router.go
user: login: nathanwilk7
body: TODO dry this method with the read version
created_at: 2025-05-09T06:08:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -73,26 +73,59 @@ func (r *Router) BuildReadRoutingPlan(params types.RoutingPlanBuildOptions) (typ
 		return types.RoutingPlan{}, fmt.Errorf("could not get read replicas location from sharding state: %w", err)
 	}
 
+	return r.routingPlanFromReplicas(params.Collection, params.Shard, replicas, params.ConsistencyLevel, params.DirectCandidateReplica)
+}
+
+func (r *Router) BuildWriteRoutingPlan(params types.RoutingPlanBuildOptions) (types.RoutingPlan, error) {
+	if err := params.Validate(); err != nil {
+		return types.RoutingPlan{}, err
+	}
+
+	writeReplicas, additionalWriteReplicas, err := r.GetWriteReplicasLocation(params.Collection, params.Shard)
+	if err != nil {
+		return types.RoutingPlan{}, fmt.Errorf("could not get read replicas location from sharding state: %w", err)
+	}
+
+	routingPlan, err := r.routingPlanFromReplicas(params.Collection, params.Shard, writeReplicas, params.ConsistencyLevel, params.DirectCandidateReplica)
+	if err != nil {
+		return types.RoutingPlan{}, err
+	}
+
+	for _, replica := range additionalWriteReplicas {
+		if replicaAddr, ok := r.clusterStateReader.NodeHostname(replica); ok {
+			routingPlan.AdditionalHostAddrs = append(routingPlan.AdditionalHostAddrs, replicaAddr)
+		}
+	}
+
+	return routingPlan, nil
+}
+
+func (r *Router) routingPlanFromReplicas(
+	collection, shard string,
+	replicas []string,
+	consistencyLevel types.ConsistencyLevel,
+	directCandidateReplica string,
+) (types.RoutingPlan, error) {
path: cluster/router/router.go
user: login: reyreaud-l
body: We can drastically reduce the amount of parameters here by re-taking as parameter `params types.RoutingPlanBuildOptions`
created_at: 2025-05-13T14:09:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -211,15 +211,15 @@ func (s *ShardReplicationFSM) GetOpState(op ShardReplicationOp) (ShardReplicatio
 	return v, ok
 }
 
-func (s *ShardReplicationFSM) FilterOneShardReplicasReadWrite(collection string, shard string, shardReplicasLocation []string) ([]string, []string) {
+func (s *ShardReplicationFSM) FilterOneShardReplicasReadWrite(collection string, shard string, shardReplicasLocation []string) ([]string, []string, []string) {
 	s.opsLock.RLock()
 	defer s.opsLock.RUnlock()
 
-	_, ok := s.opsByCollectionAndShard[collection][shard]
+	ops, ok := s.opsByCollectionAndShard[collection][shard]
path: cluster/replication/shard_replication_fsm.go
user: login: reyreaud-l
body: Just noticed here first that we do a double map lookup without checking if the first map lookup succeed. To avoid a panic if you can change it to be more defensive it would be nice.

```
colOps, ok := s.opsByCollectionAndShard[collection]
if !ok {...}
shardOps, ok = s.opsByCollectionAndShard[collection][shard]
if !ok {...}
```
created_at: 2025-05-13T14:15:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -211,15 +211,15 @@ func (s *ShardReplicationFSM) GetOpState(op ShardReplicationOp) (ShardReplicatio
 	return v, ok
 }
 
-func (s *ShardReplicationFSM) FilterOneShardReplicasReadWrite(collection string, shard string, shardReplicasLocation []string) ([]string, []string) {
+func (s *ShardReplicationFSM) FilterOneShardReplicasReadWrite(collection string, shard string, shardReplicasLocation []string) ([]string, []string, []string) {
path: cluster/replication/shard_replication_fsm.go
user: login: reyreaud-l
body: Instead of adding a third function, you can split this one into
`FilterOneShardReplicasRead` and `FilterOneShardReplicasWrite` where the read one is unchanged and the write one returns the additional replicas.
created_at: 2025-05-13T14:15:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -233,8 +233,17 @@ func (s *ShardReplicationFSM) FilterOneShardReplicasReadWrite(collection string,
 			writeReplicas = append(writeReplicas, shardReplicaLocation)
 		}
 	}
-
-	return readReplicas, writeReplicas
+	additionalWriteReplicas := []string{}
path: cluster/replication/shard_replication_fsm.go
user: login: reyreaud-l
body: Follow-up to "splitting it into two function" you can put the above for loop returning read/write replicas in it's own function and call it where needed.

Note: We will have to have either an additional check or a different function to also check if the replica is the source of an operation `filterOneReplicaReadWrite` only check if it's a target of an op. This is important in the case of DEHYDRATING to ensure we start not writing to the source. But not critical :)
created_at: 2025-05-13T14:17:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -197,8 +203,33 @@ func (c *coordinator[T]) Push(ctx context.Context,
 		"duration": 20 * time.Second,
 		"level":    level,
 	}).Debug("context.WithTimeout")
+
 	nodeCh := c.broadcast(ctxWithTimeout, routingPlan.ReplicasHostAddrs, ask, level)
-	return c.commitAll(context.Background(), nodeCh, com), level, nil
+	commitCh := c.commitAll(context.Background(), nodeCh, com)
+	pushResp := pushResponse[T]{
path: usecases/replica/coordinator.go
user: login: reyreaud-l
body: I think you're overly complicating the change here. It's not necessary to change the return value here and considering the coordinator code, I would avoid touching it as much as possible.

The only requirement we have to satisfy is "writes should be sent to the additional replicas, and if *one* fails we need to hookback to the FSM to "increase" the current upper bound of the async replication check".
Note: I'm not thinking about sync follower here, purely best effort writes.

We can't easily hook back to the FSM because the coordinator can be on any node, and we can't really make it pass via raft because timing issues could lead to writes being lost. So we're out of luck.

I think this leaves us with this as the only option. Let's reduce the async replication "upper bound" to `time.Now()` when we enter the FINALIZING state and update the `DEHYDRATING` state to be delayed/wait for X amount of seconds with *no* writes being sent to the source, but having the target be part of sharding state (and therefore part of the quorum).

Async replication can "finish" (and it will if we disable writes from landing on the source in that state) while we're in the DEHYDRATING state.

So we would go from

```
1. FINALIZING state
1.1 start async repl
1.2 wait now + 100s for async repl
1.3 (async repl complete) add target to sharding state
2. DEHYDRATING state
2.1 immediately remove source from sharding state
2.2 done
```

to

```
1. FINALIZING state
1.1 start async repl
1.2 wait async repl to report to change up to time = now
1.3 (async repl complete) add target to sharding state
2. DEHYDRATING state
2.1 kickstart another round of async repl between source and destination
2.2 wait for async repl to report no change to propagate from now + 100s
2.3 stop async repl
2.4 remove source from sharding state
2.5 done
```

By shifting async replication "long wait" to dehydrating (once the target is also part of the sharding state) we can ensure we lose no writes at all, because no more writes will land on the source. 

How does this simplify our "best effort writes" here ? Not really directly, but it allows us to "not care" about the return value of the best writes, and instead just "send them" and we don't have to increase the async replication "timeout" for the check. Instead whatever writes doesn't succeed will be caught in the DEHYDRATING step.

This also helps making "COPY" a lot faster as we don't have to wait ages for async replication. And we can just recommend/enforce using async replication all the time when doing replica copies.

For "MOVE" this ensures that even in RF=1 case we will not drop any write as the two round of async replication should allow us to catch them.
created_at: 2025-05-13T14:34:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -197,8 +203,33 @@ func (c *coordinator[T]) Push(ctx context.Context,
 		"duration": 20 * time.Second,
 		"level":    level,
 	}).Debug("context.WithTimeout")
+
 	nodeCh := c.broadcast(ctxWithTimeout, routingPlan.ReplicasHostAddrs, ask, level)
-	return c.commitAll(context.Background(), nodeCh, com), level, nil
+	commitCh := c.commitAll(context.Background(), nodeCh, com)
+	pushResp := pushResponse[T]{
path: usecases/replica/coordinator.go
user: login: jeroiraz
body: wondering if it wouldn't be enough to enable async replication during dehydrating state with now() as upper bound
created_at: 2025-05-13T15:34:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -233,8 +233,17 @@ func (s *ShardReplicationFSM) FilterOneShardReplicasReadWrite(collection string,
 			writeReplicas = append(writeReplicas, shardReplicaLocation)
 		}
 	}
-
-	return readReplicas, writeReplicas
+	additionalWriteReplicas := []string{}
path: cluster/replication/shard_replication_fsm.go
user: login: nathanwilk7
body: I think the DEHYDRATING source node exclusion is already handled here: https://github.com/weaviate/weaviate/blob/d98588e5104551faa99086111fb8cc0b8f102367/cluster/replication/shard_replication_fsm.go#L290-L292, which gets called from `filterOneReplicaReadWrite` if there are no ops for the target node of the movement
created_at: 2025-05-14T07:07:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -197,8 +203,33 @@ func (c *coordinator[T]) Push(ctx context.Context,
 		"duration": 20 * time.Second,
 		"level":    level,
 	}).Debug("context.WithTimeout")
+
 	nodeCh := c.broadcast(ctxWithTimeout, routingPlan.ReplicasHostAddrs, ask, level)
-	return c.commitAll(context.Background(), nodeCh, com), level, nil
+	commitCh := c.commitAll(context.Background(), nodeCh, com)
+	pushResp := pushResponse[T]{
path: usecases/replica/coordinator.go
user: login: nathanwilk7
body: Thanks for these thoughts.

I have removed the return value and now just "fire and forget" any additional host replica writes. I've moved the "long" async replication step to the dehydrating phase.

The reason I've kept the minimum async wait time in the dehydrating step is to catch any in-progress writes which were sent to the source node right before the op started dehydrating. For example, if a write arrives on the source node which takes an unexpectedly long time to complete (eg 80s), then to ensure that write gets propagated to the target node via async replication, we would need the minimum async wait time to be greater than the time the write took. If you believe we can use a shorter minimum wait time and still avoid losing writes let's discuss more. Note that any writes sent during the finalizing phase are "best effort" on the target node and will not return an error to the user if they fail.
created_at: 2025-05-14T07:16:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -197,8 +203,33 @@ func (c *coordinator[T]) Push(ctx context.Context,
 		"duration": 20 * time.Second,
 		"level":    level,
 	}).Debug("context.WithTimeout")
+
 	nodeCh := c.broadcast(ctxWithTimeout, routingPlan.ReplicasHostAddrs, ask, level)
-	return c.commitAll(context.Background(), nodeCh, com), level, nil
+	commitCh := c.commitAll(context.Background(), nodeCh, com)
+	pushResp := pushResponse[T]{
path: usecases/replica/coordinator.go
user: login: nathanwilk7
body: if we decide we want "success required writes" for additional hosts in the future, saved that work on this branch: https://github.com/weaviate/weaviate/compare/shard-replication-can-require-success-on-additional-host-writes?expand=1
created_at: 2025-05-14T07:34:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -506,6 +505,51 @@ func (c *CopyOpConsumer) processDehydratingOp(ctx context.Context, op ShardRepli
 	logger := getLoggerForOp(c.logger.Logger, op.Op)
 	logger.Info("processing dehydrating replication operation")
 
+	asyncReplicationUpperTimeBoundUnixMillis := time.Now().Add(c.asyncReplicationMinimumWait.Get()).UnixMilli()
+
+	// start async replication from source node to target node
path: cluster/replication/consumer.go
user: login: reyreaud-l
body: nitpick: I would explain in a comment here why we're kicking off yet-another round of async replication in the dehydrating step.

For example something like "Kicking another round of async replication to ensure that all writes that could have landed on the source shard will be replicated to the target shard before it being removed from the sharding state. We have to wait by default X seconds because our internal write timeout is of X seconds, so we want to ensure we're able to catch any ongoing long writes". Feel free to add more details :)
created_at: 2025-05-14T08:10:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -233,8 +233,17 @@ func (s *ShardReplicationFSM) FilterOneShardReplicasReadWrite(collection string,
 			writeReplicas = append(writeReplicas, shardReplicaLocation)
 		}
 	}
-
-	return readReplicas, writeReplicas
+	additionalWriteReplicas := []string{}
path: cluster/replication/shard_replication_fsm.go
user: login: reyreaud-l
body: Hoooo I didn't remember that code, I thought I did *not* ended up adding it and that we needed to. Then we indeed already have the different lookup between "is the replica a source, or is the replica a target of a repl op currently". Sorry then yes my comment isn't useful :)
created_at: 2025-05-14T08:12:11Z
author_association: CONTRIBUTOR

PR comments:
user: login: weaviate-git-bot
created_at: 2025-05-09T10:00:09Z
body: Great to see you again! Thanks for the contribution.<br><br>beep boop - the Weaviate bot 👋🤖<br><br>PS:<br>Are you already a member of the <a target="_blank" href="https://weaviate.io/slack">Weaviate Slack channel?</a>
author_association: NONE
user: login: sonarqubecloud[bot]
created_at: 2025-05-14T14:44:39Z
body: ## [![Quality Gate Failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-failed-20px.png 'Quality Gate Failed')](https://sonarcloud.io/dashboard?id=weaviate_weaviate&pullRequest=8079) **Quality Gate failed**  
Failed conditions  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png '') [8.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=weaviate_weaviate&pullRequest=8079&metric=new_duplicated_lines_density&view=list) (required ≤ 3%)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=weaviate_weaviate&pullRequest=8079)


author_association: NONE
user: login: nathanwilk7
created_at: 2025-05-14T15:52:29Z
body: flaky test being addressed in follow up work
author_association: CONTRIBUTOR