Repository: cockroachdb/cockroach
Pull request #146220: revert "kv: use origin timestamp to validate lww semantics"
Author: login: jeffswenson
Labels: name: backport-failed
name: backport-25.2.x
description: Flags PRs that need to be backported to 25.2
name: v25.3.0-prerelease
Created: 2025-05-06T17:58:47Z | Merged: 2025-05-06T22:23:10Z

## revert "kv: use origin timestamp to validate lww semantics"
This reverts commit https://github.com/cockroachdb/cockroach/commit/9336d4faff4a56d90d272557a305e7adf2893db1.

## logical: switch back to kv writer
25.2 was going to use the sql writer by default. But https://github.com/cockroachdb/cockroach/issues/146117 requires us
to roll back https://github.com/cockroachdb/cockroach/pull/143100, which means the sql writer no longer implements
LWW correctly in the presence of tombstones.

This also includes a change to fix the cput origin time validation. If
the cput is allowed to include origin time in the batch header and the
cput request iff the times match. This behavior will be required by the
sql writer to fix tombstone handling.

This removes the version check in `getWriterType` since the SQL writer
no longer changes it behavior based on the version of the KV server.

Release note: none
Part of: https://github.com/cockroachdb/cockroach/issues/146117
Informs: https://github.com/cockroachdb/cockroach/issues/146217


Diff:
diff --git a/pkg/crosscluster/logical/logical_replication_job_test.go b/pkg/crosscluster/logical/logical_replication_job_test.go
index 192f0e9ec7ba..194b9173ff7f 100644
--- a/pkg/crosscluster/logical/logical_replication_job_test.go
+++ b/pkg/crosscluster/logical/logical_replication_job_test.go
@@ -2798,7 +2798,7 @@ func TestGetWriterType(t *testing.T) {
 		)
 		wt, err := getWriterType(ctx, jobspb.LogicalReplicationDetails_Immediate, st)
 		require.NoError(t, err)
-		require.Equal(t, sqlclustersettings.LDRWriterTypeSQL, wt)
+		require.Equal(t, sqlclustersettings.LDRWriterTypeLegacyKV, wt)
 	})
 
 	t.Run("immediate-mode-post-25.2", func(t *testing.T) {
diff --git a/pkg/crosscluster/logical/logical_replication_writer_processor.go b/pkg/crosscluster/logical/logical_replication_writer_processor.go
index 4bc0d67668ab..5cb34c06cda5 100644
--- a/pkg/crosscluster/logical/logical_replication_writer_processor.go
+++ b/pkg/crosscluster/logical/logical_replication_writer_processor.go
@@ -754,12 +754,7 @@ func getWriterType(
 ) (sqlclustersettings.LDRWriterType, error) {
 	switch mode {
 	case jobspb.LogicalReplicationDetails_Immediate:
-		// Require v25.2 to use the sql writer by default to ensure that the
-		// KV origin timestamp validation is available on all nodes.
-		if settings.Version.IsActive(ctx, clusterversion.V25_2) {
-			return sqlclustersettings.LDRWriterType(sqlclustersettings.LDRImmediateModeWriter.Get(&settings.SV)), nil
-		}
-		return sqlclustersettings.LDRWriterTypeSQL, nil
+		return sqlclustersettings.LDRWriterType(sqlclustersettings.LDRImmediateModeWriter.Get(&settings.SV)), nil
 	case jobspb.LogicalReplicationDetails_Validated:
 		return sqlclustersettings.LDRWriterTypeSQL, nil
 	default:
diff --git a/pkg/crosscluster/logical/lww_row_processor_test.go b/pkg/crosscluster/logical/lww_row_processor_test.go
index 87265b7aef5d..2ef4adbae43e 100644
--- a/pkg/crosscluster/logical/lww_row_processor_test.go
+++ b/pkg/crosscluster/logical/lww_row_processor_test.go
@@ -467,6 +467,11 @@ func TestLWWConflictResolution(t *testing.T) {
 				runner.CheckQueryResults(t, fmt.Sprintf("SELECT * from %s", tableNameDst), expectedRows)
 			})
 			t.Run("cross-cluster-local-delete", func(t *testing.T) {
+				if !useKVProc {
+					// TODO(jeffswenson): re-enable this test
+					skip.WithIssue(t, 146217, "crud writer does not correctly implement lww")
+				}
+
 				tableNameDst, rp, encoder := setup(t, useKVProc)
 
 				runner.Exec(t, fmt.Sprintf("INSERT INTO %s VALUES ($1, $2)", tableNameDst), row1...)
diff --git a/pkg/crosscluster/logical/table_batch_handler_test.go b/pkg/crosscluster/logical/table_batch_handler_test.go
index 90f21209c286..4c541c72beae 100644
--- a/pkg/crosscluster/logical/table_batch_handler_test.go
+++ b/pkg/crosscluster/logical/table_batch_handler_test.go
@@ -19,6 +19,7 @@ import (
 	"github.com/cockroachdb/cockroach/pkg/sql/catalog/lease"
 	"github.com/cockroachdb/cockroach/pkg/sql/sem/tree"
 	"github.com/cockroachdb/cockroach/pkg/testutils/serverutils"
+	"github.com/cockroachdb/cockroach/pkg/testutils/skip"
 	"github.com/cockroachdb/cockroach/pkg/testutils/sqlutils"
 	"github.com/cockroachdb/cockroach/pkg/util/hlc"
 	"github.com/cockroachdb/cockroach/pkg/util/leaktest"
@@ -199,6 +200,9 @@ func TestTableHandlerExhaustive(t *testing.T) {
 	defer leaktest.AfterTest(t)()
 	defer log.Scope(t).Close(t)
 
+	// TODO(jeffswenson): re-enable this test
+	skip.WithIssue(t, 146217, "crud writer does not correctly implement lww")
+
 	// This test is an "exhaustive" test of the table handler. It tries to test
 	// cross product of every possible (replication event type, local value,
 	// previous value, lww win).
diff --git a/pkg/kv/BUILD.bazel b/pkg/kv/BUILD.bazel
index 2f549eada3b4..890a85648059 100644
--- a/pkg/kv/BUILD.bazel
+++ b/pkg/kv/BUILD.bazel
@@ -52,7 +52,6 @@ go_test(
         "admission_test.go",
         "client_test.go",
         "db_test.go",
-        "kv_client_lww_test.go",
         "main_test.go",
         "range_lookup_test.go",
         "txn_external_test.go",
diff --git a/pkg/kv/kv_client_lww_test.go b/pkg/kv/kv_client_lww_test.go
deleted file mode 100644
index bfb109ec6ee2..000000000000
--- a/pkg/kv/kv_client_lww_test.go
+++ /dev/null
@@ -1,355 +0,0 @@
-// Copyright 2025 The Cockroach Authors.
-//
-// Use of this software is governed by the CockroachDB Software License
-// included in the /LICENSE file.
-
-package kv_test
-
-import (
-	"context"
-	"testing"
-
-	"github.com/cockroachdb/cockroach/pkg/base"
-	"github.com/cockroachdb/cockroach/pkg/kv"
-	"github.com/cockroachdb/cockroach/pkg/kv/kvpb"
-	"github.com/cockroachdb/cockroach/pkg/roachpb"
-	"github.com/cockroachdb/cockroach/pkg/testutils/serverutils"
-	"github.com/cockroachdb/cockroach/pkg/util/hlc"
-	"github.com/cockroachdb/cockroach/pkg/util/leaktest"
-	"github.com/cockroachdb/cockroach/pkg/util/log"
-	"github.com/cockroachdb/errors"
-	"github.com/stretchr/testify/require"
-)
-
-// unwrapConditionFailedError unwraps an error into a kvpb.ConditionFailedError.
-// It fails the test if there is no error or if the error cannot be unwrapped
-// into the expected type.
-func unwrapConditionFailedError(t *testing.T, err error) *kvpb.ConditionFailedError {
-	t.Helper()
-	require.Error(t, err, "expected an error")
-
-	var conditionFailedErr *kvpb.ConditionFailedError
-	require.True(t, errors.As(err, &conditionFailedErr),
-		"expected error to unwrap to *kvpb.ConditionFailedError, got: %v", err)
-
-	return conditionFailedErr
-}
-
-func TestDeleteWithOriginTimestamp(t *testing.T) {
-	defer leaktest.AfterTest(t)()
-	defer log.Scope(t).Close(t)
-
-	s, _, db := serverutils.StartServer(t, base.TestServerArgs{})
-	defer s.Stopper().Stop(context.Background())
-
-	ctx := context.Background()
-	key := roachpb.Key("lww-test-key")
-	value := []byte("value")
-
-	// Put a value with the current timestamp.
-	require.NoError(t, db.Put(ctx, key, value))
-
-	// Get the current timestamp to use as a reference.
-	gr, err := db.Get(ctx, key)
-	require.NoError(t, err)
-	currentTS := gr.Value.Timestamp
-
-	// Helper function to attempt a delete with a specific origin timestamp
-	attemptDelete := func(ts hlc.Timestamp) error {
-		b := &kv.Batch{}
-		b.Del(key)
-		b.Header.WriteOptions = &kvpb.WriteOptions{
-			OriginTimestamp: ts,
-		}
-		return db.Run(ctx, b)
-	}
-
-	t.Run("OlderTimestamp", func(t *testing.T) {
-		// Delete with an origin timestamp older than the mvcc timestamp should fail
-		err := attemptDelete(currentTS.Prev())
-		condErr := unwrapConditionFailedError(t, err)
-		require.False(t, condErr.OriginTimestampOlderThan.IsEmpty(), "expected OriginTimestampOlderThan to be set")
-		require.Equal(t, currentTS, condErr.OriginTimestampOlderThan, "expected OriginTimestampOlderThan to match current timestamp")
-
-		// Verify the original value still exists
-		gr, err := db.Get(ctx, key)
-		require.NoError(t, err)
-		require.NotNil(t, gr.Value, "expected value to exist")
-		require.Equal(t, value, gr.ValueBytes(), "value should match what was put")
-	})
-
-	t.Run("NewerTimestamp", func(t *testing.T) {
-		// Delete with an origin timestamp newer than the mvcc timestamp should succeed
-		err := attemptDelete(currentTS.Next())
-		require.NoError(t, err)
-
-		// Verify the value was deleted
-		gr, err := db.Get(ctx, key)
-		require.NoError(t, err)
-		require.Nil(t, gr.Value, "expected value to be deleted")
-	})
-}
-
-func TestDelRangeWithOriginTimestamp(t *testing.T) {
-	defer leaktest.AfterTest(t)()
-	defer log.Scope(t).Close(t)
-
-	s, _, db := serverutils.StartServer(t, base.TestServerArgs{})
-	defer s.Stopper().Stop(context.Background())
-
-	ctx := context.Background()
-
-	// Setup test data once for all subtests
-	keys := []roachpb.Key{
-		roachpb.Key("lww-delrange-a"),
-		roachpb.Key("lww-delrange-b"),
-		roachpb.Key("lww-delrange-c"),
-	}
-	startKey := keys[0]
-	endKey := keys[len(keys)-1].Next()
-	value := []byte("value")
-
-	// Put values for all keys
-	for _, key := range keys {
-		require.NoError(t, db.Put(ctx, key, value))
-	}
-
-	// Get the current timestamp to use as a reference
-	gr, err := db.Get(ctx, keys[2])
-	require.NoError(t, err)
-	currentTS := gr.Value.Timestamp
-
-	t.Run("OlderTimestamp", func(t *testing.T) {
-		// DelRange with an origin timestamp older than the mvcc timestamp should fail
-		b := &kv.Batch{}
-		b.DelRange(startKey, endKey, false)
-		b.Header.WriteOptions = &kvpb.WriteOptions{
-			OriginTimestamp: currentTS.Prev(),
-		}
-		err := db.Run(ctx, b)
-		condErr := unwrapConditionFailedError(t, err)
-		require.False(t, condErr.OriginTimestampOlderThan.IsEmpty(), "expected OriginTimestampOlderThan to be set")
-		require.Equal(t, currentTS, condErr.OriginTimestampOlderThan, "expected OriginTimestampOlderThan to match current timestamp")
-
-		// Verify the original values still exist
-		for _, key := range keys {
-			gr, err := db.Get(ctx, key)
-			require.NoError(t, err)
-			require.NotNil(t, gr.Value, "expected value to exist at %s", key)
-			require.Equal(t, value, gr.ValueBytes(), "value should match what was put")
-		}
-	})
-
-	t.Run("NewerTimestamp", func(t *testing.T) {
-		// Delete the range with the newer timestamp
-		b := &kv.Batch{}
-		b.DelRange(startKey, endKey, false)
-		b.Header.WriteOptions = &kvpb.WriteOptions{
-			OriginTimestamp: currentTS.Next(),
-		}
-		err := db.Run(ctx, b)
-		require.NoError(t, err, "unexpected error when deleting range with newer origin timestamp")
-
-		// Verify all values were deleted
-		for _, key := range keys {
-			gr, err := db.Get(ctx, key)
-			require.NoError(t, err)
-			require.Nil(t, gr.Value, "expected value to be deleted at %s", key)
-		}
-	})
-}
-
-func TestPutWithOriginTimestamp(t *testing.T) {
-	defer leaktest.AfterTest(t)()
-	defer log.Scope(t).Close(t)
-
-	s, _, db := serverutils.StartServer(t, base.TestServerArgs{})
-	defer s.Stopper().Stop(context.Background())
-
-	ctx := context.Background()
-	key := roachpb.Key("lww-put-test-key")
-	value1 := []byte("value1")
-	value2 := []byte("value2")
-
-	// Put a value with the current timestamp.
-	require.NoError(t, db.Put(ctx, key, value1))
-
-	// Get the current timestamp to use as a reference.
-	gr, err := db.Get(ctx, key)
-	require.NoError(t, err)
-	currentTS := gr.Value.Timestamp
-
-	// Helper function to attempt a put with a specific origin timestamp
-	attemptPut := func(ts hlc.Timestamp, value []byte) error {
-		b := &kv.Batch{}
-		b.Put(key, value)
-		b.Header.WriteOptions = &kvpb.WriteOptions{
-			OriginTimestamp: ts,
-		}
-		return db.Run(ctx, b)
-	}
-
-	t.Run("OlderTimestamp", func(t *testing.T) {
-		// Put with an origin timestamp older than the mvcc timestamp should fail
-		olderTS := currentTS.Prev()
-		err := attemptPut(olderTS, value2)
-		condErr := unwrapConditionFailedError(t, err)
-		require.False(t, condErr.OriginTimestampOlderThan.IsEmpty(), "expected OriginTimestampOlderThan to be set")
-		require.Equal(t, currentTS, condErr.OriginTimestampOlderThan, "expected OriginTimestampOlderThan to match current timestamp")
-
-		// Verify the original value still exists
-		gr, err := db.Get(ctx, key)
-		require.NoError(t, err)
-		require.NotNil(t, gr.Value, "expected value to exist")
-		require.Equal(t, value1, gr.ValueBytes(), "value should match expected")
-	})
-
-	t.Run("NewerTimestamp", func(t *testing.T) {
-		// Put with an origin timestamp newer than the mvcc timestamp should succeed
-		newerTS := currentTS.Next()
-		err := attemptPut(newerTS, value2)
-		require.NoError(t, err, "unexpected error when putting with newer origin timestamp")
-
-		// Verify the value was updated
-		gr, err := db.Get(ctx, key)
-		require.NoError(t, err)
-		require.NotNil(t, gr.Value, "expected value to exist")
-		require.Equal(t, value2, gr.ValueBytes(), "value should match expected")
-
-		// Get the new timestamp
-		gr, err = db.Get(ctx, key)
-		require.NoError(t, err)
-		updatedTS := gr.Value.Timestamp
-
-		// Try to put with an even newer timestamp
-		newestValue := []byte("newest-value")
-		newestTS := updatedTS.Next()
-		err = attemptPut(newestTS, newestValue)
-		require.NoError(t, err, "unexpected error when putting with newest origin timestamp")
-
-		// Verify the newest value was written
-		gr, err = db.Get(ctx, key)
-		require.NoError(t, err)
-		require.NotNil(t, gr.Value, "expected value to exist")
-		require.Equal(t, newestValue, gr.ValueBytes(), "value should match new value")
-	})
-}
-
-func TestCPutWithOriginTimestamp(t *testing.T) {
-	defer leaktest.AfterTest(t)()
-	defer log.Scope(t).Close(t)
-
-	s, _, db := serverutils.StartServer(t, base.TestServerArgs{})
-	defer s.Stopper().Stop(context.Background())
-
-	ctx := context.Background()
-	key := roachpb.Key("lww-cput-test-key")
-	initialValue := []byte("initial")
-	newValue := []byte("new-value")
-	wrongExpectedValue := []byte("wrong-expected")
-
-	// Put a value with the current timestamp.
-	require.NoError(t, db.Put(ctx, key, initialValue))
-
-	// Get the current timestamp to use as a reference.
-	gr, err := db.Get(ctx, key)
-	require.NoError(t, err)
-	currentTS := gr.Value.Timestamp
-
-	// Helper function to attempt a conditional put with a specific origin timestamp
-	attemptCPut := func(ts hlc.Timestamp, expectedValue, newValue []byte) error {
-		b := &kv.Batch{}
-		b.CPut(key, newValue, expectedValue)
-		b.Header.WriteOptions = &kvpb.WriteOptions{
-			OriginTimestamp: ts,
-		}
-		return db.Run(ctx, b)
-	}
-
-	t.Run("ConditionFailsAndLWWFails", func(t *testing.T) {
-		// Both the condition fails (wrong expected value) and the origin timestamp is older
-		olderTS := currentTS.Prev()
-		err := attemptCPut(olderTS, wrongExpectedValue, newValue)
-		condErr := unwrapConditionFailedError(t, err)
-
-		// The error should indicate the origin timestamp is older
-		require.False(t, condErr.OriginTimestampOlderThan.IsEmpty(), "expected OriginTimestampOlderThan to be set")
-		require.Equal(t, currentTS, condErr.OriginTimestampOlderThan, "expected OriginTimestampOlderThan to match current timestamp")
-
-		// Verify the original value still exists
-		gr, err := db.Get(ctx, key)
-		require.NoError(t, err)
-		require.NotNil(t, gr.Value, "expected value to exist")
-		require.Equal(t, initialValue, gr.ValueBytes(), "value should match initial value")
-	})
-
-	t.Run("ConditionPassesButLWWFails", func(t *testing.T) {
-		// The condition passes (correct expected value) but the origin timestamp is older
-		olderTS := currentTS.Prev()
-		err := attemptCPut(olderTS, initialValue, newValue)
-		condErr := unwrapConditionFailedError(t, err)
-
-		// The error should indicate the origin timestamp is older
-		require.False(t, condErr.OriginTimestampOlderThan.IsEmpty(), "expected OriginTimestampOlderThan to be set")
-		require.Equal(t, currentTS, condErr.OriginTimestampOlderThan, "expected OriginTimestampOlderThan to match current timestamp")
-
-		// Verify the original value still exists
-		gr, err := db.Get(ctx, key)
-		require.NoError(t, err)
-		require.NotNil(t, gr.Value, "expected value to exist")
-		require.Equal(t, initialValue, gr.ValueBytes(), "value should match initial value")
-	})
-
-	t.Run("ConditionFailsButLWWPasses", func(t *testing.T) {
-		// The condition fails (wrong expected value) but the origin timestamp is newer
-		newerTS := currentTS.Next()
-		err := attemptCPut(newerTS, wrongExpectedValue, newValue)
-		condErr := unwrapConditionFailedError(t, err)
-
-		// The error should indicate the condition failed but not that the timestamp is older
-		require.True(t, condErr.OriginTimestampOlderThan.IsEmpty(), "expected OriginTimestampOlderThan to be empty")
-		require.NotNil(t, condErr.ActualValue, "expected ActualValue to be set")
-
-		// Verify the original value still exists
-		gr, err := db.Get(ctx, key)
-		require.NoError(t, err)
-		require.NotNil(t, gr.Value, "expected value to exist")
-		require.Equal(t, initialValue, gr.ValueBytes(), "value should match initial value")
-	})
-
-	t.Run("BothConditionAndLWWPass", func(t *testing.T) {
-		// Create a new key for this test to avoid interference from previous test cases
-		successKey := roachpb.Key("lww-cput-success-key")
-
-		// Put an initial value
-		require.NoError(t, db.Put(ctx, successKey, initialValue))
-
-		// Get the current value and timestamp
-		gr, err := db.Get(ctx, successKey)
-		require.NoError(t, err)
-		require.NotNil(t, gr.Value)
-		currentValue := gr.Value.TagAndDataBytes()
-		currentTS := gr.Value.Timestamp
-
-		// Both the condition passes and the origin timestamp is newer
-		newerTS := currentTS.Next()
-
-		// Use a helper function to perform the CPut with the current value
-		cputErr := func() error {
-			b := &kv.Batch{}
-			b.CPut(successKey, newValue, currentValue)
-			b.Header.WriteOptions = &kvpb.WriteOptions{
-				OriginTimestamp: newerTS,
-			}
-			return db.Run(ctx, b)
-		}()
-
-		require.NoError(t, cputErr, "unexpected error when both condition and timestamp are valid")
-
-		// Verify the value was updated
-		gr, err = db.Get(ctx, successKey)
-		require.NoError(t, err)
-		require.NotNil(t, gr.Value, "expected value to exist")
-		require.Equal(t, newValue, gr.ValueBytes(), "value should match new value")
-	})
-}
diff --git a/pkg/kv/kvpb/api.proto b/pkg/kv/kvpb/api.proto
index 8ebae1a2d2ba..6799b25a4432 100644
--- a/pkg/kv/kvpb/api.proto
+++ b/pkg/kv/kvpb/api.proto
@@ -347,8 +347,29 @@ message ConditionalPutRequest {
   // the use of this value with transactions.
   bool inline = 7;
 
-  // OriginTimestamp is deprecated. If set, it must be equal to the
-  // OriginTimestamp in the batch's WriteOptions.
+  // OriginTimestamp, if set, indicates that the ConditionalPut should
+  // only succeed if the provided timestamp is greater than the
+  // OriginTimestamp field in the MVCCValueHeader of the existing
+  // value (if the value has a non-zero OriginTimestamp) or the MVCC
+  // timestamp of the existing value. If there is no existing value,
+  // the timestamp check succeeds.
+  //
+  // The conditional put request will only succeed if this timestamp
+  // comparison is successful and the expected bytes matches the
+  // actual bytes.
+  //
+  // On success, this timestamp will also be stored in the
+  // OriginTimestamp field of the the MVCCValueHeader for the newly
+  // written value.
+  //
+  // In the proposed use case, the OriginTimestamp is the MVCC version
+  // timestamp of the row on the source cluster and the semantics lead
+  // to idempotent last-write-wins semantics against the "original
+  // MVCC timestamp" of the value on the destination, which is its
+  // MVCC timestamp or, if set, the OriginTimestamp (reflecting that
+  // this value originated from the source cluster).
+  //
+  // Used by logical data replication.
   util.hlc.Timestamp origin_timestamp = 8 [(gogoproto.nullable) = false];
 
   reserved 9;
@@ -3007,26 +3028,9 @@ message Header {
 
 message WriteOptions {
   uint32 origin_id = 1[(gogoproto.customname) = "OriginID"];
-  // OriginTimestamp is used by logical data replication. It is the MVCC
-  // timestamp of the row in the source cluster.
-  //
-  // If set the write will only succeed if the provided timestamp is greater than
-  // the OriginTimestamp field in the MVCCValueHeader of the existing value (if
-  // the value has a non-zero OriginTimestamp) or the MVCC timestamp of the
-  // existing value. If there is no existing value, the timestamp check succeeds.
-	//
-	// The origin_timestamp header is only handled by the put, cput, delete, and
-	// range delete commands.
-  //
-  // On success, this timestamp will also be stored in the
-  // OriginTimestamp field of the the MVCCValueHeader for the newly
-  // written value.
-  //
-  // In LDR, the OriginTimestamp is the MVCC version timestamp of the row on the
-  // source cluster and the semantics leads to idempotent last-write-wins
-  // semantics against the "original" MVCC timestamp of the value on the
-  // destination, which is its MVCC timestamp or, if set, the OriginTimestamp
-  // (reflecting that this value originated from the source cluster).
+  // OriginTimestamp is bound to the MVCCValueHeader of written key in the
+  // batch. Note that a kv client cannot set this if they use CPut's origin
+  // timestamp arg.
   util.hlc.Timestamp origin_timestamp = 2 [(gogoproto.nullable) = false];
 }
 
diff --git a/pkg/kv/kvserver/batcheval/cmd_conditional_put.go b/pkg/kv/kvserver/batcheval/cmd_conditional_put.go
index e5e26f23f89f..18551ce035a5 100644
--- a/pkg/kv/kvserver/batcheval/cmd_conditional_put.go
+++ b/pkg/kv/kvserver/batcheval/cmd_conditional_put.go
@@ -58,14 +58,12 @@ func ConditionalPut(
 		return result.Result{}, err
 	}
 
-	originTimestamp := h.WriteOptions.GetOriginTimestamp()
+	originTimestampForValueHeader := h.WriteOptions.GetOriginTimestamp()
 	if args.OriginTimestamp.IsSet() {
-		// NOTE: this is a work around for old clients. As of 25.2, the
-		// OriginTimestamp is deprecated.
-		if originTimestamp.IsSet() && args.OriginTimestamp != originTimestamp {
-			return result.Result{}, errors.AssertionFailedf("origin timestamp mismatch")
+		originTimestampForValueHeader = args.OriginTimestamp
+		if h.WriteOptions.GetOriginTimestamp().IsSet() && args.OriginTimestamp != h.WriteOptions.GetOriginTimestamp() {
+			return result.Result{}, errors.AssertionFailedf("OriginTimestamp passed via CPut arg must match the origin timestamp in the request header")
 		}
-		originTimestamp = args.OriginTimestamp
 	}
 
 	opts := storage.ConditionalPutWriteOptions{
@@ -76,12 +74,13 @@ func ConditionalPut(
 			ReplayWriteTimestampProtection: h.AmbiguousReplayProtection,
 			OmitInRangefeeds:               cArgs.OmitInRangefeeds,
 			OriginID:                       h.WriteOptions.GetOriginID(),
-			OriginTimestamp:                originTimestamp,
+			OriginTimestamp:                originTimestampForValueHeader,
 			MaxLockConflicts:               storage.MaxConflictsPerLockConflictError.Get(&cArgs.EvalCtx.ClusterSettings().SV),
 			TargetLockConflictBytes:        storage.TargetBytesPerLockConflictError.Get(&cArgs.EvalCtx.ClusterSettings().SV),
 			Category:                       fs.BatchEvalReadCategory,
 		},
 		AllowIfDoesNotExist: storage.CPutMissingBehavior(args.AllowIfDoesNotExist),
+		OriginTimestamp:     args.OriginTimestamp,
 	}
 
 	var err error
diff --git a/pkg/sql/sqlclustersettings/clustersettings.go b/pkg/sql/sqlclustersettings/clustersettings.go
index 89c4c4c9a9cb..e49d29033d16 100644
--- a/pkg/sql/sqlclustersettings/clustersettings.go
+++ b/pkg/sql/sqlclustersettings/clustersettings.go
@@ -129,7 +129,9 @@ var LDRImmediateModeWriter = settings.RegisterStringSetting(
 	settings.ApplicationLevel,
 	"logical_replication.consumer.immediate_mode_writer",
 	"the writer to use when in immediate mode",
-	metamorphic.ConstantWithTestChoice("logical_replication.consumer.immediate_mode_writer", string(LDRWriterTypeSQL), string(LDRWriterTypeLegacyKV), string(LDRWriterTypeCRUD)),
+	// TODO(jeffswenson): make the sql writer the default.
+	//metamorphic.ConstantWithTestChoice("logical_replication.consumer.immediate_mode_writer", string(LDRWriterTypeSQL), string(LDRWriterTypeLegacyKV), string(LDRWriterTypeCRUD)),
+	metamorphic.ConstantWithTestChoice("logical_replication.consumer.immediate_mode_writer", string(LDRWriterTypeLegacyKV)),
 	settings.WithValidateString(func(sv *settings.Values, val string) error {
 		if val != string(LDRWriterTypeSQL) && val != string(LDRWriterTypeLegacyKV) && val != string(LDRWriterTypeCRUD) {
 			return errors.Newf("immediate mode writer must be either 'sql', 'legacy-kv', or 'crud', got '%s'", val)
diff --git a/pkg/storage/mvcc.go b/pkg/storage/mvcc.go
index a8728eb9cc4b..7eb242b20752 100644
--- a/pkg/storage/mvcc.go
+++ b/pkg/storage/mvcc.go
@@ -423,13 +423,11 @@ func (v *optionalValue) ToPointer() *roachpb.Value {
 	return &cpy
 }
 
-func (v *optionalValue) checkOriginTimestamp(originTimestamp hlc.Timestamp) error {
+func (v *optionalValue) isOriginTimestampWinner(
+	proposedTS hlc.Timestamp, inclusive bool,
+) (bool, hlc.Timestamp) {
 	if !v.exists {
-		return nil
-	}
-
-	if !originTimestamp.IsSet() {
-		return nil
+		return true, hlc.Timestamp{}
 	}
 
 	existTS := v.Value.Timestamp
@@ -437,33 +435,7 @@ func (v *optionalValue) checkOriginTimestamp(originTimestamp hlc.Timestamp) erro
 		existTS = v.MVCCValueHeader.OriginTimestamp
 	}
 
-	// TODO(jeffswenson): add support for handling ties. The KV LDR writer
-	// originally had a ShouldWinOriginTimestampTie field, but it was removed
-	// due to an issue with replays. ShouldWinOriginTimestampTie was set
-	// unconditionally if the replicating cluster had a larger origin cluster
-	// id.
-	//
-	// There are two problems with this:
-	// 1. When replaying replicated writes, the replicated writes will win LWW
-	// vs themselves, which is undesirable because it writes duplicate KVs which
-	// is slow and wastes storage until the duplicate KVs are cleaned up by GC.
-	// 2. It doesn't implement semantics correctly in the case of three way
-	// replication.
-	//
-	// For now, LDR is going to ignore the possibility of ties. A tie will only
-	// occur if there are racing updates to the same row in two different
-	// clusters at the same nanosecond.
-	//
-	// The proper fix for this is to pass the source cluster and local cluster
-	// id to checkOriginTimestamp. That lets us correctly implement cross
-	// cluster ties while allowing a replicated write to lose ties against
-	// itself.
-	if originTimestamp.LessEq(existTS) {
-		return &kvpb.ConditionFailedError{
-			OriginTimestampOlderThan: existTS,
-		}
-	}
-	return nil
+	return existTS.Less(proposedTS) || (inclusive && existTS.Equal(proposedTS)), existTS
 }
 
 // isSysLocal returns whether the key is system-local.
@@ -1968,7 +1940,6 @@ func MVCCPut(
 	// key we can utilize a blind put to avoid reading any existing value.
 	var iter MVCCIterator
 	var ltScanner *lockTableKeyScanner
-	var valueFn func(existVal optionalValue) (roachpb.Value, error)
 	blind := opts.Stats == nil && timestamp.IsEmpty()
 	if !blind {
 		var err error
@@ -1994,16 +1965,8 @@ func MVCCPut(
 			}
 			defer ltScanner.close()
 		}
-		if !opts.OriginTimestamp.IsEmpty() {
-			valueFn = func(existVal optionalValue) (roachpb.Value, error) {
-				if err := existVal.checkOriginTimestamp(opts.OriginTimestamp); err != nil {
-					return roachpb.Value{}, err
-				}
-				return value, nil
-			}
-		}
 	}
-	return mvccPutUsingIter(ctx, rw, iter, ltScanner, key, timestamp, value, valueFn, opts)
+	return mvccPutUsingIter(ctx, rw, iter, ltScanner, key, timestamp, value, nil, opts)
 }
 
 // MVCCBlindPut is a fast-path of MVCCPut. See the MVCCPut comments for details
@@ -2069,18 +2032,9 @@ func MVCCDelete(
 	buf := newPutBuffer()
 	defer buf.release()
 
-	var valueFn func(existVal optionalValue) (roachpb.Value, error)
-	if !opts.OriginTimestamp.IsEmpty() {
-		valueFn = func(existVal optionalValue) (roachpb.Value, error) {
-			if err := existVal.checkOriginTimestamp(opts.OriginTimestamp); err != nil {
-				return roachpb.Value{}, err
-			}
-			return noValue, nil
-		}
-	}
 	// TODO(yuzefovich): can we avoid the put if the key does not exist?
 	return mvccPutInternal(
-		ctx, rw, iter, ltScanner, key, timestamp, noValue, buf, valueFn, opts)
+		ctx, rw, iter, ltScanner, key, timestamp, noValue, buf, nil, opts)
 }
 
 var noValue = roachpb.Value{}
@@ -2952,6 +2906,12 @@ type ConditionalPutWriteOptions struct {
 	MVCCWriteOptions
 
 	AllowIfDoesNotExist CPutMissingBehavior
+	// OriginTimestamp, if set, indicates that the caller wants to put the
+	// value only if any existing key is older than this timestamp.
+	//
+	// See the comment on the OriginTimestamp field of
+	// kvpb.ConditionalPutRequest for more details.
+	OriginTimestamp hlc.Timestamp
 }
 
 // MVCCConditionalPut sets the value for a specified key only if the expected
@@ -3058,17 +3018,40 @@ func mvccConditionalPutUsingIter(
 		}
 	}
 
-	valueFn := func(actualValue optionalValue) (roachpb.Value, error) {
-		if err := actualValue.checkOriginTimestamp(opts.OriginTimestamp); err != nil {
-			return roachpb.Value{}, err
+	var valueFn func(existVal optionalValue) (roachpb.Value, error)
+	if opts.OriginTimestamp.IsEmpty() {
+		valueFn = func(actualValue optionalValue) (roachpb.Value, error) {
+			if err := maybeConditionFailedError(expBytes, actualValue, bool(opts.AllowIfDoesNotExist)); err != nil {
+				return roachpb.Value{}, err
+			}
+			return value, nil
 		}
-		if err := maybeConditionFailedError(expBytes, actualValue, bool(opts.AllowIfDoesNotExist)); err != nil {
-			if !opts.OriginTimestamp.IsEmpty() {
+	} else {
+		valueFn = func(existVal optionalValue) (roachpb.Value, error) {
+			originTSWinner, existTS := existVal.isOriginTimestampWinner(opts.OriginTimestamp, false)
+			if !originTSWinner {
+				return roachpb.Value{}, &kvpb.ConditionFailedError{
+					OriginTimestampOlderThan: existTS,
+				}
+			}
+
+			// We are the OriginTimestamp comparison winner. We
+			// check the expected bytes because a mismatch implies
+			// that the caller may have produced other commands with
+			// outdated data.
+			if err := maybeConditionFailedError(expBytes, existVal, false); err != nil {
 				err.HadNewerOriginTimestamp = true
+				return roachpb.Value{}, err
 			}
-			return roachpb.Value{}, err
+			return value, nil
 		}
-		return value, nil
+
+		// TODO(ssd): We set the OriginTimestamp on our write
+		// options to the originTimestamp passed to us. We
+		// don't assert they are the same yet because it is
+		// still unclear how exactly we want to manage this in
+		// the long run.
+		opts.MVCCWriteOptions.OriginTimestamp = opts.OriginTimestamp
 	}
 
 	return mvccPutUsingIter(ctx, writer, iter, ltScanner, key, timestamp, noValue, valueFn, opts.MVCCWriteOptions)
@@ -3790,21 +3773,11 @@ func MVCCDeleteRange(
 	buf := newPutBuffer()
 	defer buf.release()
 
-	var valueFn func(existVal optionalValue) (roachpb.Value, error)
-	if !opts.OriginTimestamp.IsEmpty() {
-		valueFn = func(existVal optionalValue) (roachpb.Value, error) {
-			if err := existVal.checkOriginTimestamp(opts.OriginTimestamp); err != nil {
-				return roachpb.Value{}, err
-			}
-			return noValue, nil
-		}
-	}
-
 	var keys []roachpb.Key
 	var acqs []roachpb.LockAcquisition
 	for i, kv := range res.KVs {
 		_, acq, err := mvccPutInternal(
-			ctx, rw, iter, ltScanner, kv.Key, timestamp, noValue, buf, valueFn, opts,
+			ctx, rw, iter, ltScanner, kv.Key, timestamp, noValue, buf, nil, opts,
 		)
 		if err != nil {
 			return nil, nil, 0, nil, err
diff --git a/pkg/storage/mvcc_history_test.go b/pkg/storage/mvcc_history_test.go
index bf05559d0f3f..5d8f0d411435 100644
--- a/pkg/storage/mvcc_history_test.go
+++ b/pkg/storage/mvcc_history_test.go
@@ -1215,9 +1215,9 @@ func cmdCPut(e *evalCtx) error {
 				Stats:                          e.ms,
 				ReplayWriteTimestampProtection: e.getAmbiguousReplay(),
 				MaxLockConflicts:               e.getMaxLockConflicts(),
-				OriginTimestamp:                originTimestamp,
 			},
 			AllowIfDoesNotExist: behavior,
+			OriginTimestamp:     originTimestamp,
 		}
 		acq, err := storage.MVCCConditionalPut(e.ctx, rw, key, ts, val, expVal, opts)
 		if err != nil {
@@ -1241,12 +1241,6 @@ func cmdInitPut(e *evalCtx) error {
 	key := e.getKey()
 	val := e.getVal()
 	failOnTombstones := e.hasArg("failOnTombstones")
-
-	originTimestamp := hlc.Timestamp{}
-	if e.hasArg("origin_ts") {
-		originTimestamp = e.getTsWithName("origin_ts")
-	}
-
 	resolve, resolveStatus := e.getResolve()
 
 	return e.withWriter("initput", func(rw storage.ReadWriter) error {
@@ -1256,7 +1250,6 @@ func cmdInitPut(e *evalCtx) error {
 			Stats:                          e.ms,
 			ReplayWriteTimestampProtection: e.getAmbiguousReplay(),
 			MaxLockConflicts:               e.getMaxLockConflicts(),
-			OriginTimestamp:                originTimestamp,
 		}
 		acq, err := storage.MVCCInitPut(e.ctx, rw, key, ts, val, failOnTombstones, opts)
 		if err != nil {
@@ -1277,14 +1270,7 @@ func cmdDelete(e *evalCtx) error {
 	key := e.getKey()
 	ts := e.getTs(txn)
 	localTs := hlc.ClockTimestamp(e.getTsWithName("localTs"))
-
-	originTimestamp := hlc.Timestamp{}
-	if e.hasArg("origin_ts") {
-		originTimestamp = e.getTsWithName("origin_ts")
-	}
-
 	resolve, resolveStatus := e.getResolve()
-
 	return e.withWriter("del", func(rw storage.ReadWriter) error {
 		opts := storage.MVCCWriteOptions{
 			Txn:                            txn,
@@ -1292,7 +1278,6 @@ func cmdDelete(e *evalCtx) error {
 			Stats:                          e.ms,
 			ReplayWriteTimestampProtection: e.getAmbiguousReplay(),
 			MaxLockConflicts:               e.getMaxLockConflicts(),
-			OriginTimestamp:                originTimestamp,
 		}
 		foundKey, acq, err := storage.MVCCDelete(e.ctx, rw, key, ts, opts)
 		if err == nil || errors.HasType(err, &kvpb.WriteTooOldError{}) {
@@ -1324,11 +1309,6 @@ func cmdDeleteRange(e *evalCtx) error {
 		e.scanArg("max", &max)
 	}
 
-	originTimestamp := hlc.Timestamp{}
-	if e.hasArg("origin_ts") {
-		originTimestamp = e.getTsWithName("origin_ts")
-	}
-
 	resolve, resolveStatus := e.getResolve()
 	return e.withWriter("del_range", func(rw storage.ReadWriter) error {
 		opts := storage.MVCCWriteOptions{
@@ -1337,7 +1317,6 @@ func cmdDeleteRange(e *evalCtx) error {
 			Stats:                          e.ms,
 			ReplayWriteTimestampProtection: e.getAmbiguousReplay(),
 			MaxLockConflicts:               e.getMaxLockConflicts(),
-			OriginTimestamp:                originTimestamp,
 		}
 		deleted, resumeSpan, num, acqs, err := storage.MVCCDeleteRange(
 			e.ctx, rw, key, endKey, int64(max), ts, opts, returnKeys)
@@ -1502,11 +1481,6 @@ func cmdIncrement(e *evalCtx) error {
 		inc = int64(incI)
 	}
 
-	originTimestamp := hlc.Timestamp{}
-	if e.hasArg("origin_ts") {
-		originTimestamp = e.getTsWithName("origin_ts")
-	}
-
 	resolve, resolveStatus := e.getResolve()
 
 	return e.withWriter("increment", func(rw storage.ReadWriter) error {
@@ -1516,7 +1490,6 @@ func cmdIncrement(e *evalCtx) error {
 			Stats:                          e.ms,
 			ReplayWriteTimestampProtection: e.getAmbiguousReplay(),
 			MaxLockConflicts:               e.getMaxLockConflicts(),
-			OriginTimestamp:                originTimestamp,
 		}
 		curVal, acq, err := storage.MVCCIncrement(e.ctx, rw, key, ts, opts, inc)
 		if err != nil {
@@ -1559,11 +1532,6 @@ func cmdPut(e *evalCtx) error {
 		e.scanArg("import_epoch", &importEpoch)
 	}
 
-	originTimestamp := hlc.Timestamp{}
-	if e.hasArg("origin_ts") {
-		originTimestamp = e.getTsWithName("origin_ts")
-	}
-
 	resolve, resolveStatus := e.getResolve()
 
 	return e.withWriter("put", func(rw storage.ReadWriter) error {
@@ -1574,7 +1542,6 @@ func cmdPut(e *evalCtx) error {
 			Stats:                          e.ms,
 			ReplayWriteTimestampProtection: e.getAmbiguousReplay(),
 			MaxLockConflicts:               e.getMaxLockConflicts(),
-			OriginTimestamp:                originTimestamp,
 		}
 		acq, err := storage.MVCCPut(e.ctx, rw, key, ts, val, opts)
 		if err != nil {
diff --git a/pkg/storage/mvcc_test.go b/pkg/storage/mvcc_test.go
index 8bc1ec87ec33..593d92a032a3 100644
--- a/pkg/storage/mvcc_test.go
+++ b/pkg/storage/mvcc_test.go
@@ -234,82 +234,6 @@ func TestMVCCStatsAddSubForward(t *testing.T) {
 	require.Equal(t, exp, neg)
 }
 
-func TestCheckOriginTimestamp(t *testing.T) {
-	defer leaktest.AfterTest(t)()
-
-	testCases := []struct {
-		name     string
-		value    optionalValue
-		ts       hlc.Timestamp
-		wantErr  bool
-		expError *kvpb.ConditionFailedError
-	}{
-		{
-			name: "no-existing-value",
-			value: optionalValue{
-				exists: false,
-			},
-			ts:      hlc.Timestamp{WallTime: 100},
-			wantErr: false,
-		},
-		{
-			name: "existing-value-older-origin-timestamp",
-			value: optionalValue{
-				exists: true,
-				MVCCValue: MVCCValue{
-					Value: roachpb.Value{},
-					MVCCValueHeader: enginepb.MVCCValueHeader{
-						OriginTimestamp: hlc.Timestamp{WallTime: 100},
-					},
-				},
-			},
-			ts:      hlc.Timestamp{WallTime: 200},
-			wantErr: false,
-		},
-		{
-			name: "existing-value-newer-origin-timestamp",
-			value: optionalValue{
-				exists: true,
-				MVCCValue: MVCCValue{
-					Value: roachpb.Value{},
-					MVCCValueHeader: enginepb.MVCCValueHeader{
-						OriginTimestamp: hlc.Timestamp{WallTime: 200},
-					},
-				},
-			},
-			ts:      hlc.Timestamp{WallTime: 100},
-			wantErr: true,
-			expError: &kvpb.ConditionFailedError{
-				OriginTimestampOlderThan: hlc.Timestamp{WallTime: 200},
-			},
-		},
-		{
-			name: "no-origin-timestamp",
-			value: optionalValue{
-				exists: true,
-				MVCCValue: MVCCValue{
-					Value:           roachpb.Value{},
-					MVCCValueHeader: enginepb.MVCCValueHeader{},
-				},
-			},
-			ts:      hlc.Timestamp{WallTime: 200},
-			wantErr: false,
-		},
-	}
-
-	for _, tc := range testCases {
-		t.Run(tc.name, func(t *testing.T) {
-			err := tc.value.checkOriginTimestamp(tc.ts)
-			if tc.wantErr {
-				require.Error(t, err)
-				require.True(t, errors.Is(err, tc.expError))
-			} else {
-				require.NoError(t, err)
-			}
-		})
-	}
-}
-
 func TestMVCCStatsHasUserDataCloseTo(t *testing.T) {
 	defer leaktest.AfterTest(t)()
 	defer log.Scope(t).Close(t)
@@ -1940,6 +1864,7 @@ func TestMVCCDeleteRangeOldTimestamp(t *testing.T) {
 	require.Equal(t, int64(1), keyCount)
 	require.NoError(t, err)
 }
+
 func TestMVCCDeleteRangeInline(t *testing.T) {
 	defer leaktest.AfterTest(t)()
 	defer log.Scope(t).Close(t)
diff --git a/pkg/storage/testdata/mvcc_histories/delete_range_with_origin_timestamps b/pkg/storage/testdata/mvcc_histories/delete_range_with_origin_timestamps
deleted file mode 100644
index cf4054a576f7..000000000000
--- a/pkg/storage/testdata/mvcc_histories/delete_range_with_origin_timestamps
+++ /dev/null
@@ -1,46 +0,0 @@
-run ok
-with v=abc
-  put k=a ts=1
-  put k=b ts=1
-  put k=c ts=3
-----
->> at end:
-data: "a"/1.000000000,0 -> /BYTES/abc
-data: "b"/1.000000000,0 -> /BYTES/abc
-data: "c"/3.000000000,0 -> /BYTES/abc
-
-run error
-del_range k=c end=d ts=4 origin_ts=2
-----
->> at end:
-data: "a"/1.000000000,0 -> /BYTES/abc
-data: "b"/1.000000000,0 -> /BYTES/abc
-data: "c"/3.000000000,0 -> /BYTES/abc
-error: (*kvpb.ConditionFailedError:) OriginTimestamp older than 3.000000000,0
-
-run ok
-del_range k=a end=c ts=5 origin_ts=2
-----
-del_range: "a"-"c" -> deleted 2 key(s)
->> at end:
-data: "a"/5.000000000,0 -> {originTs=2.000000000,0}/<empty>
-data: "a"/1.000000000,0 -> /BYTES/abc
-data: "b"/5.000000000,0 -> {originTs=2.000000000,0}/<empty>
-data: "b"/1.000000000,0 -> /BYTES/abc
-data: "c"/3.000000000,0 -> /BYTES/abc
-
-# NOTE: there is a subtle behavior demonstrated by this test. Delete range does
-# not advance the origin ts if the key was already deleted. This is okay
-# because the SQL writer will issue a CPUT to update the origin timestamp
-# assigned to the primary key.
-run ok
-del_range k=a end=d ts=6 origin_ts=4
-----
-del_range: "a"-"d" -> deleted 1 key(s)
->> at end:
-data: "a"/5.000000000,0 -> {originTs=2.000000000,0}/<empty>
-data: "a"/1.000000000,0 -> /BYTES/abc
-data: "b"/5.000000000,0 -> {originTs=2.000000000,0}/<empty>
-data: "b"/1.000000000,0 -> /BYTES/abc
-data: "c"/6.000000000,0 -> {originTs=4.000000000,0}/<empty>
-data: "c"/3.000000000,0 -> /BYTES/abc
diff --git a/pkg/storage/testdata/mvcc_histories/delete_with_origin_timestamp b/pkg/storage/testdata/mvcc_histories/delete_with_origin_timestamp
deleted file mode 100644
index 3c165ce4ad2c..000000000000
--- a/pkg/storage/testdata/mvcc_histories/delete_with_origin_timestamp
+++ /dev/null
@@ -1,43 +0,0 @@
-run ok
-put k=a v=wow ts=3
-----
->> at end:
-data: "a"/3.000000000,0 -> /BYTES/wow
-
-# deleting with a younger origin time should lose LWW
-run error
-del k=a ts=4 origin_ts=2
-----
->> at end:
-data: "a"/3.000000000,0 -> /BYTES/wow
-error: (*kvpb.ConditionFailedError:) OriginTimestamp older than 3.000000000,0
-
-run ok
-del k=a ts=5 origin_ts=4
-----
-del: "a": found key true
->> at end:
-data: "a"/5.000000000,0 -> {originTs=4.000000000,0}/<empty>
-data: "a"/3.000000000,0 -> /BYTES/wow
-
-# it should be possible to write over an existing tombstone with a newer origin
-# timestamp
-run ok
-del k=a ts=6 origin_ts=5
-----
-del: "a": found key false
->> at end:
-data: "a"/6.000000000,0 -> {originTs=5.000000000,0}/<empty>
-data: "a"/5.000000000,0 -> {originTs=4.000000000,0}/<empty>
-data: "a"/3.000000000,0 -> /BYTES/wow
-
-# it is possible to write a tombstone for a value that does not exist locally
-run ok
-del k=notfound ts=7 origin_ts=5
-----
-del: "notfound": found key false
->> at end:
-data: "a"/6.000000000,0 -> {originTs=5.000000000,0}/<empty>
-data: "a"/5.000000000,0 -> {originTs=4.000000000,0}/<empty>
-data: "a"/3.000000000,0 -> /BYTES/wow
-data: "notfound"/7.000000000,0 -> {originTs=5.000000000,0}/<empty>
diff --git a/pkg/storage/testdata/mvcc_histories/put_with_origin_timestamp b/pkg/storage/testdata/mvcc_histories/put_with_origin_timestamp
deleted file mode 100644
index 73e229b5d5b9..000000000000
--- a/pkg/storage/testdata/mvcc_histories/put_with_origin_timestamp
+++ /dev/null
@@ -1,20 +0,0 @@
-run ok
-put k=a v=wow ts=3
-----
->> at end:
-data: "a"/3.000000000,0 -> /BYTES/wow
-
-# put with a younger origin time should lose LWW
-run error
-put k=a v=wow ts=4 origin_ts=2
-----
->> at end:
-data: "a"/3.000000000,0 -> /BYTES/wow
-error: (*kvpb.ConditionFailedError:) OriginTimestamp older than 3.000000000,0
-
-run ok
-put k=a v=wow ts=5 origin_ts=4
-----
->> at end:
-data: "a"/5.000000000,0 -> {originTs=4.000000000,0}/BYTES/wow
-data: "a"/3.000000000,0 -> /BYTES/wow


Reviews:
user: login: msbutler
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-06T19:37:56Z
user: login: msbutler
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-06T20:07:38Z
user: login: jeffswenson
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-06T20:10:04Z
user: login: msbutler
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-06T20:24:18Z

Review comments:
diff_hunk: @@ -129,7 +129,9 @@ var LDRImmediateModeWriter = settings.RegisterStringSetting(
 	settings.ApplicationLevel,
 	"logical_replication.consumer.immediate_mode_writer",
 	"the writer to use when in immediate mode",
-	metamorphic.ConstantWithTestChoice("logical_replication.consumer.immediate_mode_writer", string(LDRWriterTypeSQL), string(LDRWriterTypeLegacyKV), string(LDRWriterTypeCRUD)),
+	// TODO(jeffswenson): make the sql writer the default.
+	//metamorphic.ConstantWithTestChoice("logical_replication.consumer.immediate_mode_writer", string(LDRWriterTypeSQL), string(LDRWriterTypeLegacyKV), string(LDRWriterTypeCRUD)),
path: pkg/sql/sqlclustersettings/clustersettings.go
user: login: msbutler
body: why remove the metamorphic wrapper with the other modes? Perhaps this comment above was temporary.
created_at: 2025-05-06T20:07:34Z
author_association: COLLABORATOR
diff_hunk: @@ -129,7 +129,9 @@ var LDRImmediateModeWriter = settings.RegisterStringSetting(
 	settings.ApplicationLevel,
 	"logical_replication.consumer.immediate_mode_writer",
 	"the writer to use when in immediate mode",
-	metamorphic.ConstantWithTestChoice("logical_replication.consumer.immediate_mode_writer", string(LDRWriterTypeSQL), string(LDRWriterTypeLegacyKV), string(LDRWriterTypeCRUD)),
+	// TODO(jeffswenson): make the sql writer the default.
+	//metamorphic.ConstantWithTestChoice("logical_replication.consumer.immediate_mode_writer", string(LDRWriterTypeSQL), string(LDRWriterTypeLegacyKV), string(LDRWriterTypeCRUD)),
path: pkg/sql/sqlclustersettings/clustersettings.go
user: login: jeffswenson
body: This is temporary until we re-fix the sql writer tombstone handling.
created_at: 2025-05-06T20:10:04Z
author_association: COLLABORATOR
diff_hunk: @@ -129,7 +129,9 @@ var LDRImmediateModeWriter = settings.RegisterStringSetting(
 	settings.ApplicationLevel,
 	"logical_replication.consumer.immediate_mode_writer",
 	"the writer to use when in immediate mode",
-	metamorphic.ConstantWithTestChoice("logical_replication.consumer.immediate_mode_writer", string(LDRWriterTypeSQL), string(LDRWriterTypeLegacyKV), string(LDRWriterTypeCRUD)),
+	// TODO(jeffswenson): make the sql writer the default.
+	//metamorphic.ConstantWithTestChoice("logical_replication.consumer.immediate_mode_writer", string(LDRWriterTypeSQL), string(LDRWriterTypeLegacyKV), string(LDRWriterTypeCRUD)),
path: pkg/sql/sqlclustersettings/clustersettings.go
user: login: msbutler
body: sg
created_at: 2025-05-06T20:24:18Z
author_association: COLLABORATOR

PR comments:
user: login: cockroach-teamcity
created_at: 2025-05-06T17:59:01Z
body: 

<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/cockroachdb/cockroach/146220)
<!-- Reviewable:end -->

author_association: MEMBER
user: login: jeffswenson
created_at: 2025-05-06T19:48:26Z
body: Thanks for the review!

bors r+
author_association: COLLABORATOR
user: login: jeffswenson
created_at: 2025-05-06T19:54:54Z
body: bors r-
author_association: COLLABORATOR
user: login: craig[bot]
created_at: 2025-05-06T19:54:58Z
body: Canceled.
author_association: CONTRIBUTOR
user: login: jeffswenson
created_at: 2025-05-06T20:46:26Z
body: Thanks for the review!

bors r+
author_association: COLLABORATOR
user: login: craig[bot]
created_at: 2025-05-06T22:23:08Z
body: Build succeeded:
  * [linux_amd64_build](https://github.com/cockroachdb/cockroach/actions/runs/14869924352/job/41755801462)
  * [local_roachtest](https://github.com/cockroachdb/cockroach/actions/runs/14869924352/job/41755801821)
  * [check_generated_code](https://github.com/cockroachdb/cockroach/actions/runs/14869924352/job/41755800240)
  * [unit_tests](https://github.com/cockroachdb/cockroach/actions/runs/14869924352/job/41755803049)
  * [local_roachtest_fips](https://github.com/cockroachdb/cockroach/actions/runs/14869924352/job/41755802214)
  * [linux_amd64_fips_build](https://github.com/cockroachdb/cockroach/actions/runs/14869924352/job/41755802717)
  * [docker_image_amd64](https://github.com/cockroachdb/cockroach/actions/runs/14869924352/job/41755800613)
  * [examples_orms](https://github.com/cockroachdb/cockroach/actions/runs/14869924352/job/41755799486)
  * [lint](https://github.com/cockroachdb/cockroach/actions/runs/14869924352/job/41755799878)
  * [acceptance](https://github.com/cockroachdb/cockroach/actions/runs/14869924352/job/41755801069)
author_association: CONTRIBUTOR
user: login: blathers-crl[bot]
created_at: 2025-05-06T22:23:17Z
body: Encountered an error creating backports. Some common things that can go wrong:
1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the [backport](https://github.com/benesch/backport) tool.

----

:bulb: Consider backporting to the fork repo instead of the main repo. See [instructions](https://github.com/cockroachlabs/blathers-bot#backporting) for more details.

error creating merge commit from 8d4a5e6422fb34c8fb0ea0d12825a39ca1ca2fb7 to blathers/backport-release-25.2-146220: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 25.2.x failed. See errors above.

----

<sub>:owl: Hoot! I am a [Blathers](https://github.com/apps/blathers-crl), a bot for [CockroachDB](https://github.com/cockroachdb). My owner is [dev-inf](https://github.com/orgs/cockroachdb/teams/dev-inf).</sub>
author_association: NONE