Repository: h2database/h2database

Issue #3484: LOB issue
Created: 2022-04-17T09:14:30Z
I'm sorry to inform you that 2.1.212 has an issue related to LOB removal, probably a race condition.
After upgrading from 2.1.210 I'm experiencing a freeze during DB close right after adding more than 1000 LOBs (approx. few hundred bytes to few KB each) in a single transaction (embedded mode, single connection). It's hard to tell where the freeze happens, it seems asynchronous wrt currently running code in SessionLocal.close(): if I put less LOBs then it happens AFTER returning from Database.removeSession(); if I put more LOBs then it happens in a well-above 1000th call to LobStorageMap.doRemoveLob(), in line 'streamStoreId = tempLobMap.remove(lobId)' - the control enters MVMap.remove() but never returns.
I'm able to reproduce the issue on a large DB consisting of millions of LOBs. On an empty DB there is no repro, so I don't think I'll be able to provide you with a standalone testcase.
Regards,
wburzyns

Duplicate issue #3536: Deadlocks when Querying Small Blobs/Clobs
Created: 2022-06-09T22:58:44Z
I've been tracking down an issue in a long-running endurance test of our system where the H2 embedded database appears to lock up and can no longer process certain kinds of queries. The database didn't seem to be corrupted and restarting the process fixes the issue. It happens very irregularly, but I've been able to isolate it.
After pulling down our database and running queries using the H2 JDBC connector, I noticed that this issue only occurs when issuing lots and lots of queries against our clob columns. If I create select statements that query the non-blob columns, the issue does not occur.
When the issue occurs, a debugger shows a deadlock between several threads waiting for the other to complete. My main thread that runs the query in my test harness code is blocked by the MVStore Background Writer, which is blocked waiting for the H2-Serialization thread, which is blocked waiting for the H2-Save thread, which I think is waiting for the MVStore Background Writer to unlock the store. I'm not familiar with the code, so perhaps most of the parking that happens is probably okay, but it seems like a red flag.
I've also observed that this issue appears to be related to the LOB cache cleanup code in some fashion. I found that turning down the LOB_TIMEOUT property to a shorter period would cause the issue to appear faster. Turning it down to a millisecond makes the issue happen very quickly.
Lastly, I noticed that the issue only seems to happen with Blobs of a certain size. It was easiest to replicate with about 200 characters. With 1000 characters, the issue doesn't occur. I've also noticed that the number of rows in the system isn't critical. I can replicate it with just a handful of rows (20 in the example below).
Below is the test harness I was using to play with the issue. The first half of the harness builds a test database, the second half keeps submitting queries forever to try and force the issue to occur. I've tested on my machine and another developers in the office and found that after at most a minute the queries halt and the deadlock occurs. The only dependency is h2 version 2.1.212. For reference, here's some other information on my machine:
Java version information:
$  java -version
openjdk version "17.0.2" 2022-01-18 LTS
OpenJDK Runtime Environment Corretto-17.0.2.8.1 (build 17.0.2+8-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.2.8.1 (build 17.0.2+8-LTS, mixed mode, sharing)

Machine information:
MacBook Pro 16-inch 2019
macOS Big Sir Version 11.6.6
Test Harness
import java.io.IOException;
import java.io.StringReader;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.security.SecureRandom;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;

public class TestH2 {

  public static void main(String[] args) throws SQLException, ClassNotFoundException, IOException {

    if (Files.exists(Paths.get("./test.mv.db"))) {
      Files.delete(Paths.get("./test.mv.db"));
    }
    Class.forName("org.h2.Driver");


    // Create a test database with a couple records with some CLOBs in it.
    System.out.println("Creating test database...");

    // Why LOB_TIMEOUT=1? If I don't set this, it takes 5 minutes to replicate the issue. If I set it to a
    // millisecond, it triggers the lockup within a minute.
    try (Connection connection = DriverManager.getConnection("jdbc:h2:./test;LOB_TIMEOUT=1")) {

      // Make the mylobs table to query later.
      connection.createStatement().
          execute("create table mylobs(id int, thelob CLOB)");

      // Make some random blobs using this random generator.
      SecureRandom random = new SecureRandom();
      byte[] bytes = new byte[100];

      PreparedStatement preparedStatement =
          connection.prepareStatement("insert into mylobs (id, thelob) VALUES (?, ?)");

      // Now make 20 records with blobs and insert them.
      for (int i = 0; i < 20; i++) {
        random.nextBytes(bytes);
        preparedStatement.setInt(1, i);
        preparedStatement.setClob(2, new StringReader(bytesToHex(bytes)));
        preparedStatement.execute();
      }
    }
    System.out.println("Test database created.");

    // Run a series of queries
    System.out.println("Querying the test database");
    try (Connection connection = DriverManager.getConnection("jdbc:h2:./test;LOB_TIMEOUT=1")) {
      while (true) {
        // Just make sure we're controlling the transactions.
        connection.setAutoCommit(false);

        Statement statement = connection.createStatement();

        // Run a query for blobs.
        ResultSet resultSet = statement.executeQuery("select id, thelob from mylobs");

        // Flip through the result set to make sure they're read and there's no lazy evaluation happening.
        int count = 0;
        while (resultSet.next()) {
          count++;
        }
        // Make sure we close our the result set to try to avoid any possible memory issues.
        resultSet.close();

        System.out.println("Query ran and fetched " + count + " records");
        connection.commit();
      }
    }
  }

  private static final byte[] HEX_ARRAY = "0123456789ABCDEF".getBytes(StandardCharsets.US_ASCII);
  // Just a simple byte array to hex function taken from the internet.
  public static String bytesToHex(byte[] bytes) {
    byte[] hexChars = new byte[bytes.length * 2];
    for (int j = 0; j < bytes.length; j++) {
      int v = bytes[j] & 0xFF;
      hexChars[j * 2] = HEX_ARRAY[v >>> 4];
      hexChars[j * 2 + 1] = HEX_ARRAY[v & 0x0F];
    }
    return new String(hexChars, StandardCharsets.UTF_8);
  }
}

Closing comment (2022-06-10T14:34:38Z): This is probably the same as #3484 which is fixed in HEAD