Project: YugabyteDB (App Definition and Development / Database)
File: ta-21297.md

title: TA-21297 headerTitle: Missing writes during Batch Execution in a transaction headcontent: 26 March 2024 type: docs showRightNav: true cascade: unversioned: true menu: preview_releases: identifier: ta-21297 weight: 1 rightNav: hideH2: true type: docs | Product | Affected Versions | Related Issues | Fixed In | | :- | : | :- | :- | | {{<product "ysql">}} | {{<release "All">}} | {{<issue 21297>}} | {{<release "2.14.16.0, 2.18.7.0, 2.21.1.0">}} | Due to a day-0 issue with transaction retries it is possible that, in specific circumstances, a transaction is partially executed and no error is returned. This is an atomicity violation and may result in data loss. This can happen in the following circumstance: You use some form of write "batch statement" (for example, multiple write statements into the same message) specifically as the first message/statement in the transaction; and a transaction conflict occurs for any but the first internal statement in that batch statement. The workaround is to add a read statement (for example, `SELECT 1`) as the first statement in each transaction. You can also disable the transaction retries according to the installed release as follows: Versions prior to {{<release "2.18.5.1, 2.18.6.0, 2.20.1.1, 2.20.2.0">}}, and {{<release "2.21.0.0">}}. Set the following flags: ```sql ysqlmaxreadrestartattempts = 0 ysqlmaxwriterestartattempts = 0 ``` Versions starting from {{<release "2.18.5.1, 2.18.6.0, 2.20.1.1, 2.20.2.0">}}, and {{<release "2.21.0.0">}}. Set the following YSQL configuration parameter: ```sql ybmaxquerylayerretries = 0 ``` In YSQL, if YugabyteDB receives certain errors from the storage layer (for example, transaction conflict and read-restart errors), YugabyteDB tries to internally retry the transaction if possible. To ensure correctness, YugabyteDB checks that it has not sent any response to the client/user before retrying. Normally, this check implies only retrying in case the error is hit during the first statement in a transaction. This is because after the first statement is executed, YugabyteDB responds to the client and disables retries for this transaction until it is done. Therefore, the database doesn't need to replay the entire transaction, just the current statement. However, if the first statement uses some form of batching (one message including multiple statements), YugabyteDB may not respond to the client until the full batch is executed. If YugabyteDB hits an error for the Nth statement in the batch, it may only retry that statement. This results in all statements before the conflicting one being missed in the retried transaction. There are two currently known cases where this can occur: A wire-protocol Batch Statement. A message with multiple semicolon-separated individual statements. The Java driver will split them into individual messages and avoid the issue, but other drivers may behave differently. ```java connection.setAutoCommit(false); statement.addBatch("INSERT INTO t2 VALUES (1,2,3)"); statement.addBatch("INSERT INTO t2 VALUES (2,3,4)"); statement.addBatch("INSERT INTO t2 VALUES (3,4,5)"); statement.addBatch("UPDATE t1 SET v2 = src + 1 WHERE k = 1"); statement.executeBatch(); connection.commit(); ``` ```sql BEGIN Batch Statement Write statement 1 Write statement 2 Write statement N-1 Write statement N -- if there is a conflict here, we only retry current statement Statement N+1 Statement N+2 Statement M BatchExecute -- we don't respond to the client until here COMMIT ```