Repository: wiredtiger/wiredtiger
Pull request #11771: WT-14278 Clean up live restore bitmap metadata on backup open, not on backup close
Author: login: ajmorton
Created: 2025-03-23T23:05:56Z | Merged: 2025-03-24T03:37:39Z

Re-delivery of #11757. That PR failed to update our performance testing to use a proper backup as the source directory.

This change simplifies the lifecycle of the live_restore=(nbits) metadata item.
When a file finished migration we set nbits=-1, but when the entire live restore process completed we would then transition to nbits=0 in preparation for the current database to be used as a source in future live restore.

However we could only set nbits=0 if the file was opened and then dirtied so it wrote metadata back to disk. This lead to a lot of scenarios where a fully migrated file sometimes has nbits=0, and sometimes had nbits=-1 even though they both meant the same thing.

To simplify this logic we no longer set nbits=0 during normal operations. nbits=-1 indicates that the migration has completed, and we only clean the metadata to say nbits=0 during the process of taking a backup.

This means that only a valid backup can be used as a source directory, so this change adds that condition and updates all of our tests to use valid backups in their source directories.



Diff:
diff --git a/src/live_restore/live_restore_fs.c b/src/live_restore/live_restore_fs.c
index 007a1b304c0..5240534d10f 100644
--- a/src/live_restore/live_restore_fs.c
+++ b/src/live_restore/live_restore_fs.c
@@ -1251,25 +1251,10 @@ __wt_live_restore_metadata_to_fh(
       "Reading in bitmap information from metadata but bitmap information already exists in "
       "memory");
 
-    /*
-     * Once we're in the clean up stage or later all data has been migrated across to the
-     * destination. There's no need for hole tracking and therefore nothing to reconstruct. The
-     * migration state must be checked before we check the number of bits in the bitmap as we clear
-     * that once it finishes.
-     */
-    if (__wti_live_restore_migration_complete(session)) {
-        /*
-         * Even though the migration has completed we may still have a source file, this indicates
-         * that the live restore finished while this file was being opened.
-         */
-        WT_ERR(__live_restore_fh_close_source(session, lr_fh, true));
-        return (0);
-    }
-
     __wt_writelock(session, &lr_fh->lock);
     lr_fh->allocsize = lr_fh_meta->allocsize;
 
-    /* If there is no source file the migration has completed. */
+    /* If there is no source file there is nothing to migrate and therefore no metadata to parse. */
     if (WTI_DEST_COMPLETE(lr_fh)) {
         __wt_writeunlock(session, &lr_fh->lock);
         return (0);
@@ -1279,12 +1264,10 @@ __wt_live_restore_metadata_to_fh(
      * !!!
      * While the live restore is in progress, the bit count reported by in the live restore metadata
      * can hold three states:
-     *  (0)         : This means file has not yet had a bitmap representation written to the
-     *                metadata file and therefore no application writes have gone to the
-     *                destination. In theory background thread writes may have happened but unless
-     *                the tree was dirtied the metadata update was not written out.
+     *  (0)         : This means the file has not started migration.
      *  (-1)        : This indicates the file has finished migration and the bitmap is empty.
-     *  (nbits > 0) : The number of bits in the bitmap.
+     *  (nbits > 0) : The number of bits in the bitmap. This is set when the file is in the
+     *                process of being migrated.
      */
     if (lr_fh_meta->nbits == 0) {
         WT_ASSERT(session, !WTI_DEST_COMPLETE(lr_fh));
@@ -1305,6 +1288,10 @@ __wt_live_restore_metadata_to_fh(
           session, lr_fh_meta->bitmap_str, (uint64_t)lr_fh_meta->nbits, lr_fh));
     } else {
         WT_ASSERT(session, lr_fh_meta->nbits == -1);
+        /*
+         * Our file open logic always opens the backing source file when it exists, but since we've
+         * completed migration we don't need it.
+         */
         WT_ERR(__live_restore_fh_close_source(session, lr_fh, false));
     }
 
@@ -1328,10 +1315,6 @@ __wt_live_restore_fh_to_metadata(WT_SESSION_IMPL *session, WT_FILE_HANDLE *fh, W
     if (!F_ISSET(S2C(session), WT_CONN_LIVE_RESTORE_FS))
         return (WT_NOTFOUND);
 
-    /* Once we're past the background migration stage there's no need to track hole information. */
-    if (__wti_live_restore_migration_complete(session))
-        return (WT_NOTFOUND);
-
     WTI_LIVE_RESTORE_FILE_HANDLE *lr_fh = (WTI_LIVE_RESTORE_FILE_HANDLE *)fh;
 
     WT_ITEM buf;
@@ -1387,20 +1370,16 @@ __wt_live_restore_clean_metadata_string(WT_SESSION_IMPL *session, char *value)
         WT_ASSERT_ALWAYS(
           session, cval.len == 0, "Found non-empty bitmap when cleaning config string");
 
+        WT_RET(__wt_config_subgets(session, &v, "nbits", &cval));
+
         /*
          * Live restore uses -1 in the nbits field to indicate the file has been fully migrated.
-         * This value should be updated to 0 when live restore moves past the background migration
-         * phase, but we can only do so if the file is open when we force a checkpoint during clean
-         * up, or if its btree is dirtied after live restore completes but before restarting in
-         * non-live restore mode. If neither of these events take place the metadata won't be
-         * updated on disk and the -1 value persists. A future live restore using this file as a
-         * source will see this value and incorrectly assume the file has already been migrated. To
-         * prevent this manually overwrite the -1 with the correct value 0.
+         * However, if this value is copied into a backup, future live restores using this backup as
+         * a source will see the nbits=-1 value and assume a file that still needs migrating has
+         * already been migrated. Set it to 0 now to indicate it is yet to be migrated.
          */
-        WT_RET(__wt_config_subgets(session, &v, "nbits", &cval));
-
-        wt_off_t nbits_val_str_offset = cval.str - value;
         if (WT_STRING_LIT_MATCH("-1", cval.str, 2)) {
+            wt_off_t nbits_val_str_offset = cval.str - value;
             /*
              * We need to overwrite two characters, but only need to write one. Add a redundant
              * comma so we don't need to resize the string. The config parser will ignore it.
@@ -1408,8 +1387,14 @@ __wt_live_restore_clean_metadata_string(WT_SESSION_IMPL *session, char *value)
             value[nbits_val_str_offset] = '0';
             value[nbits_val_str_offset + 1] = ',';
         } else
-            WT_ASSERT_ALWAYS(session, cval.len == 1 && WT_STRING_LIT_MATCH("0", cval.str, 1),
-              "nbits value other than -1 or 0 found when cleaning metadata string: %s\n", value);
+            /*
+             * There are only two possible values for nbits here. Either nbits=-1 because the file
+             * underwent a complete live restore, or nbits=0 because we're backing up a database
+             * that didn't undergo live restore. Any other value indicates a file has been partially
+             * live restored and is missing data.
+             */
+            WT_ASSERT_ALWAYS(session, WT_STRING_LIT_MATCH("0", cval.str, 1),
+              "Invalid live restore metadata detected while cleaning string");
     }
 
     return (0);
diff --git a/src/live_restore/live_restore_state.c b/src/live_restore/live_restore_state.c
index 937682fbc0c..1158fe1d91a 100644
--- a/src/live_restore/live_restore_state.c
+++ b/src/live_restore/live_restore_state.c
@@ -369,21 +369,37 @@ __wti_live_restore_validate_directories(WT_SESSION_IMPL *session, WTI_LIVE_RESTO
     char **dirlist_source = NULL, **dirlist_dest = NULL;
     uint32_t num_source_files = 0, num_dest_files = 0;
     WTI_LIVE_RESTORE_STATE state_from_file;
+    bool contain_backup_file = false;
 
-    /* First check that the source doesn't contain any live restore metadata files. */
+    /*
+     * First check that the source doesn't contain any live restore stop files, but does contain a
+     * backup file.
+     */
     WT_ERR(lr_fs->os_file_system->fs_directory_list(lr_fs->os_file_system, (WT_SESSION *)session,
       lr_fs->source.home, "", &dirlist_source, &num_source_files));
 
     if (num_source_files == 0)
         WT_ERR_MSG(session, EINVAL, "Source directory is empty. Nothing to restore!");
 
-    for (uint32_t i = 0; i < num_source_files; ++i)
+    for (uint32_t i = 0; i < num_source_files; ++i) {
         if (WT_SUFFIX_MATCH(dirlist_source[i], WTI_LIVE_RESTORE_STOP_FILE_SUFFIX))
             WT_ERR_MSG(session, EINVAL,
               "Source directory contains live restore stop file: %s. This implies it is a "
               "destination directory that hasn't finished restoration",
               dirlist_source[i]);
 
+        if (WT_SUFFIX_MATCH(dirlist_source[i], WT_METADATA_BACKUP))
+            contain_backup_file = true;
+    }
+
+    /*
+     * We rely on the backup process to clean the metadata file in the source and remove instances
+     * of nbits=-1. If we don't live restore could see this nbits=-1, think it applies to the file
+     * in the destination, and never copy across the file causing data loss.
+     */
+    if (!contain_backup_file)
+        WT_ERR_MSG(session, EINVAL, "Source directory is not a valid backup directory");
+
     /* Now check the destination folder */
 
     /* Read state directly from the turtle file in the destination. */
diff --git a/test/catch2/live_restore/live_restore_test_env.cpp b/test/catch2/live_restore/live_restore_test_env.cpp
index 6a19579d173..283f0f4d545 100644
--- a/test/catch2/live_restore/live_restore_test_env.cpp
+++ b/test/catch2/live_restore/live_restore_test_env.cpp
@@ -18,26 +18,35 @@ namespace utils {
  */
 live_restore_test_env::live_restore_test_env()
 {
-    // Clean up any pre-existing folders. Make sure an empty DB_SOURCE exists
-    // as it need to exist to open the connection in live restore mode.
-    testutil_recreate_dir(DB_DEST.c_str());
-    testutil_recreate_dir(DB_SOURCE.c_str());
+    // Clean up any pre-existing folders.
+    testutil_remove(DB_DEST.c_str());
+    testutil_remove(DB_SOURCE.c_str());
 
-    // WiredTiger stores state in the turtle file so we always need to have a valid database in
-    // the source folder. Open and close a connection to initialize the source folder.
+    // Live restore requires the source directory to be a valid backup. Create one now.
     {
         static std::string non_lr_config = "create=true";
-        conn = std::make_unique<connection_wrapper>(DB_SOURCE.c_str(), non_lr_config.c_str());
-        conn->clear_do_cleanup();
+        auto backup_conn =
+          std::make_unique<connection_wrapper>(DB_DEST.c_str(), non_lr_config.c_str());
+
+        backup_conn->get_wt_connection_impl();
+        WT_SESSION *session = (WT_SESSION *)backup_conn->create_session();
+        WT_CURSOR *backup_cursor = nullptr;
+        REQUIRE(session->open_cursor(session, "backup:", nullptr, nullptr, &backup_cursor) == 0);
+
+        testutil_mkdir(DB_SOURCE.c_str());
+        while (backup_cursor->next(backup_cursor) == 0) {
+            const char *uri = nullptr;
+            REQUIRE(backup_cursor->get_key(backup_cursor, &uri) == 0);
+            std::string dest_file = DB_DEST + "/" + uri;
+            std::string source_file = DB_SOURCE + "/" + uri;
+            testutil_copy(dest_file.c_str(), source_file.c_str());
+        }
+
+        backup_cursor->close(backup_cursor);
+        session->close(session, nullptr);
     }
 
-    /*
-     * We're using a connection to set up the file system and let us print WT traces, but all of our
-     * tests will use empty folders where we create files manually. The issue here is
-     * wiredtiger_open will create metadata and turtle files on open and think it needs to remove
-     * them on close. Move these files to a temp location. We'll restore them in destructor before
-     * _conn->close() is called.
-     */
+    testutil_remove(DB_DEST.c_str());
     static std::string cfg_string =
       "create=true,live_restore=(enabled=true, path=" + DB_SOURCE + ",threads_max=0)";
     conn = std::make_unique<connection_wrapper>(DB_DEST.c_str(), cfg_string.c_str());
diff --git a/test/catch2/sub_level_error/unit/test_sub_level_error_live_restore_conflict.cpp b/test/catch2/sub_level_error/unit/test_sub_level_error_live_restore_conflict.cpp
index 1b14a64d24e..bc26971536f 100644
--- a/test/catch2/sub_level_error/unit/test_sub_level_error_live_restore_conflict.cpp
+++ b/test/catch2/sub_level_error/unit/test_sub_level_error_live_restore_conflict.cpp
@@ -13,6 +13,7 @@
 #include "../../wrappers/connection_wrapper.h"
 #include "../utils_sub_level_error.h"
 #include "../../../utility/test_util.h"
+#include "../../live_restore/live_restore_test_env.h"
 
 /*
  * [sub_level_error_live_restore_conflict]: test_sub_level_error_live_restore_conflict.cpp
@@ -33,17 +34,10 @@ TEST_CASE("Test WT_CONFLICT_LIVE_RESTORE",
 
     SECTION("Test WT_CONFLICT_LIVE_RESTORE while opening backup cursor")
     {
-        // We need to have a real WiredTiger database in the source directory.
-        // Wrap it in its own scope to close the connection.
-        {
-            connection_wrapper source_conn_wrapper =
-              connection_wrapper("WT_LR_SOURCE", "create=true");
-            source_conn_wrapper.clear_do_cleanup();
-        }
 
-        connection_wrapper conn_wrapper =
-          connection_wrapper(".", "create=true,live_restore=(enabled=true, path=WT_LR_SOURCE)");
-        utils::prepare_session_and_error(&conn_wrapper, &session, &err_info);
+        utils::live_restore_test_env env;
+        connection_wrapper *conn_wrapper = env.conn.get();
+        utils::prepare_session_and_error(conn_wrapper, &session, &err_info);
 
         REQUIRE(session->open_cursor(session, "backup:", NULL, NULL, &cursor) == EINVAL);
         REQUIRE(cursor == NULL);
diff --git a/test/cppsuite/tests/test_live_restore.cpp b/test/cppsuite/tests/test_live_restore.cpp
index 74db0d58684..ef94ca6f721 100644
--- a/test/cppsuite/tests/test_live_restore.cpp
+++ b/test/cppsuite/tests/test_live_restore.cpp
@@ -339,6 +339,42 @@ configure_database(scoped_session &session)
     }
 }
 
+// Take a backup of the provided database and then delete the original db. This backup will be used
+// in the next loop as a source directory.
+static void
+take_backup_and_delete_original(const std::string &home, const std::string &backup_dir)
+{
+    testutil_recreate_dir(backup_dir.c_str());
+    const std::string conn_config = "log=(enabled=true,path=journal)";
+    connection_manager::instance().reopen(conn_config, home);
+
+    {
+        scoped_session backup_session = connection_manager::instance().create_session();
+        scoped_cursor backup_cursor = backup_session.open_scoped_cursor("backup:", "");
+
+        while (backup_cursor->next(backup_cursor.get()) == 0) {
+            char *file_c_str = nullptr;
+            testutil_check(backup_cursor->get_key(backup_cursor.get(), &file_c_str));
+            std::string file = std::string(file_c_str);
+
+            // If the file is a log file prepend the hard-coded journal folder path
+            if (strncmp(file.c_str(), "WiredTigerLog", 13) == 0) {
+                if (!testutil_exists(backup_dir.c_str(), "journal")) {
+                    testutil_mkdir((backup_dir + "/" + std::string("journal")).c_str());
+                }
+                file = "journal/" + file;
+            }
+
+            std::string dest_file = home + "/" + file;
+            std::string source_file = backup_dir + "/" + file;
+            testutil_copy(dest_file.c_str(), source_file.c_str());
+        }
+    }
+
+    connection_manager::instance().close();
+    testutil_remove(home.c_str());
+}
+
 static void
 run_restore(const std::string &home, const std::string &source, const int64_t thread_count,
   const int64_t collection_count, const int64_t op_count, const bool background_thread_mode,
@@ -529,7 +565,7 @@ main(int argc, char *argv[])
 
         // We need to create a database to restore from initially.
         create_db(home_path, thread_count, coll_count, op_count, verbose_level);
-        testutil_move(home_path.c_str(), SOURCE_PATH);
+        take_backup_and_delete_original(home_path, std::string(SOURCE_PATH));
     }
 
     /* When setting up the database we don't want to wait for the background threads to complete. */
@@ -542,8 +578,7 @@ main(int argc, char *argv[])
         logger::log_msg(LOG_INFO, "!!!! Beginning iteration: " + std::to_string(i) + " !!!!");
         run_restore(home_path, SOURCE_PATH, thread_count, coll_count, op_count,
           background_thread_debug_mode, verbose_level, i == death_it, recovery);
-        testutil_remove(SOURCE_PATH);
-        testutil_move(home_path.c_str(), SOURCE_PATH);
+        take_backup_and_delete_original(home_path, std::string(SOURCE_PATH));
     }
 
     return (0);
diff --git a/test/evergreen.yml b/test/evergreen.yml
index 4bf73b568a5..f4a691f690b 100644
--- a/test/evergreen.yml
+++ b/test/evergreen.yml
@@ -154,7 +154,7 @@ functions:
     params:
       aws_key: ${aws_key}
       aws_secret: ${aws_secret}
-      remote_file: wiredtiger/${build_variant}/${revision}/artifacts/${dependent_task|compile}_${build_id}.tgz
+      remote_file: wiredtiger/${build_variant}/${revision}/artifacts/${dependent_task|compile}_${build_id}${postfix|}.tgz
       bucket: build_external
       extract_to: ${destination|wiredtiger}
   "fetch endian format artifacts" :
@@ -4948,6 +4948,21 @@ tasks:
         vars:
           upload_filename: WT_TEST.tgz
           upload_source_dir: wiredtiger/cmake_build/bench/wtperf/WT_TEST_0_0/
+      # Now take and upload a backup. We use this as the source in our live restore perf tests.
+      - command: shell.exec
+        params:
+          working_dir: "wiredtiger/cmake_build/bench/wtperf"
+          shell: bash
+          script: |
+            set -o errexit
+            set -o verbose
+            ${PREPARE_TEST_ENV}
+            ${python_binary|python3} ../../../test/live_restore/take_backup.py
+      - func: "upload artifact"
+        vars:
+          upload_filename: WT_TEST_backup.tgz
+          upload_source_dir: wiredtiger/cmake_build/bench/wtperf/WT_TEST_0_0_backup/
+          postfix: "_backup"
       # Call cleanup function to avoid duplicated artifact upload in the post-task stage.
       - func: "cleanup"
 
@@ -5106,6 +5121,7 @@ tasks:
         vars:
           dependent_task: perf-test-long-500m-btree-populate
           destination: "wiredtiger/cmake_build/bench/wtperf/SOURCE"
+          postfix: "_backup"
       # Create an empty home directory to be used as the destination for live restore.
       - command: shell.exec
         params:
@@ -5146,6 +5162,7 @@ tasks:
         vars:
           dependent_task: perf-test-long-500m-btree-populate
           destination: "wiredtiger/cmake_build/bench/wtperf/SOURCE"
+          postfix: "_backup"
       # Create an empty home directory to be used as the destination for live restore.
       - command: shell.exec
         params:
diff --git a/test/live_restore/take_backup.py b/test/live_restore/take_backup.py
new file mode 100755
index 00000000000..4130ef93831
--- /dev/null
+++ b/test/live_restore/take_backup.py
@@ -0,0 +1,69 @@
+#!/usr/bin/env python
+#
+# Public Domain 2014-present MongoDB, Inc.
+# Public Domain 2008-2014 WiredTiger, Inc.
+#
+# This is free and unencumbered software released into the public domain.
+#
+# Anyone is free to copy, modify, publish, use, compile, sell, or
+# distribute this software, either in source code form or as a compiled
+# binary, for any purpose, commercial or non-commercial, and by any
+# means.
+#
+# In jurisdictions that recognize copyright laws, the author or authors
+# of this software dedicate any and all copyright interest in the
+# software to the public domain. We make this dedication for the benefit
+# of the public at large and to the detriment of our heirs and
+# successors. We intend this dedication to be an overt act of
+# relinquishment in perpetuity of all present and future rights to this
+# software under copyright law.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+# OTHER DEALINGS IN THE SOFTWARE.
+#
+# take_backup.py
+#   Take a backup of the database created by btree-500m-populate.wtperf to be used
+#   as the source directory in live restore perf tests.
+#   This script needs to be run from build/bench/wtperf and the btree-500m-populate
+#   task must have run already to generate the WT_TEST_0_0 folder.
+
+import os
+import shutil
+import sys
+
+sys.path.append(os.path.abspath('../../lang/python'))
+
+from  wiredtiger import wiredtiger_open
+
+def main():
+
+    home = "WT_TEST_0_0"
+    backup_dir = "WT_TEST_0_0_backup"
+
+    # Create a clean backup directory
+    if os.path.exists(backup_dir):
+        shutil.rmtree(backup_dir)
+    os.makedirs(backup_dir)
+
+    # Connect to the database and open a session
+    conn = wiredtiger_open('WT_TEST_0_0', None)
+    session = conn.open_session()
+
+    backup_cursor = session.open_cursor('backup:', None)
+
+    while backup_cursor.next() == 0:
+        orig_file = f'{home}/{backup_cursor.get_key()}'
+        backup_file = f'{backup_dir}/{backup_cursor.get_key()}'
+        shutil.copyfile(orig_file, backup_file)
+
+    backup_cursor.close()
+    session.close()
+    conn.close()
+
+if __name__ == "__main__":
+    main()
diff --git a/test/suite/test_live_restore01.py b/test/suite/test_live_restore01.py
index 43f84b8bb06..647e7a2cbf7 100644
--- a/test/suite/test_live_restore01.py
+++ b/test/suite/test_live_restore01.py
@@ -28,14 +28,14 @@
 
 import os
 import wiredtiger, wttest
-from helper import copy_wiredtiger_home
 import glob
 import shutil
+from wtbackup import backup_base
 
 # test_live_restore01.py
 # Test live restore compatibility with various other connection options.
 @wttest.skip_for_hook("tiered", "using multiple WT homes")
-class test_live_restore01(wttest.WiredTigerTestCase):
+class test_live_restore01(backup_base):
 
     def expect_success(self, config_str):
         self.open_conn("DEST", config=config_str)
@@ -52,9 +52,11 @@ def expect_failure(self, config_str, expected_error):
 
     def test_live_restore01(self):
         # Close the default connection.
+
+        os.mkdir("SOURCE")
+        self.take_full_backup("SOURCE")
         self.close_conn()
 
-        copy_wiredtiger_home(self, '.', "SOURCE")
         # Remove everything but SOURCE / stderr / stdout.
         for f in glob.glob("*"):
             if not f == "SOURCE" and not f == "stderr.txt" and not f == "stdout.txt":
diff --git a/test/suite/test_live_restore02.py b/test/suite/test_live_restore02.py
index 492d3c1a7e2..89f9f2c74af 100644
--- a/test/suite/test_live_restore02.py
+++ b/test/suite/test_live_restore02.py
@@ -30,12 +30,12 @@
 from wiredtiger import stat
 from wtdataset import SimpleDataSet
 from wtscenario import make_scenarios
-from helper import copy_wiredtiger_home
+from wtbackup import backup_base
 
 # test_live_restore02.py
 # Enable background thread migration and loop until it completes
 @wttest.skip_for_hook("tiered", "using multiple WT homes")
-class test_live_restore02(wttest.WiredTigerTestCase):
+class test_live_restore02(backup_base):
     format_values = [
         ('column', dict(key_format='r', value_format='S')),
         ('row_integer', dict(key_format='i', value_format='S')),
@@ -75,10 +75,10 @@ def test_live_restore02(self):
         ds2.populate()
 
         # Close the default connection.
+        os.mkdir("SOURCE")
+        self.take_full_backup("SOURCE")
         self.close_conn()
 
-        copy_wiredtiger_home(self, '.', "SOURCE")
-
         # Remove everything but SOURCE / stderr / stdout.
         for f in glob.glob("*"):
             if not f == "SOURCE" and not f == "stderr.txt" and not f == "stdout.txt":
diff --git a/test/suite/test_live_restore03.py b/test/suite/test_live_restore03.py
index b5ecfdee129..6914820c9bd 100644
--- a/test/suite/test_live_restore03.py
+++ b/test/suite/test_live_restore03.py
@@ -29,14 +29,14 @@
 import os, glob, wttest
 from wiredtiger import stat
 from wtdataset import SimpleDataSet
-from helper import copy_wiredtiger_home
+from wtbackup import backup_base
 
 # test_live_restore03.py
 # Test that live_restore->fs_size can returns a valid size when the file only exists in the source
 # directory.
 # Note: The block_size statistic corresponds to underlying file size.
 @wttest.skip_for_hook("tiered", "using multiple WT homes")
-class test_live_restore03(wttest.WiredTigerTestCase):
+class test_live_restore03(backup_base):
     nrows = 100
 
     def test_live_restore03(self):
@@ -52,10 +52,10 @@ def test_live_restore03(self):
             ds.populate()
 
         # Close the default connection.
+        os.mkdir("SOURCE")
+        self.take_full_backup("SOURCE")
         self.close_conn()
 
-        copy_wiredtiger_home(self, '.', "SOURCE")
-
         # Remove everything but SOURCE / stderr / stdout.
         for f in glob.glob("*"):
             if not f == "SOURCE" and not f == "stderr.txt" and not f == "stdout.txt":
diff --git a/test/suite/test_live_restore04.py b/test/suite/test_live_restore04.py
index 8b834b91832..cfeb8788b18 100644
--- a/test/suite/test_live_restore04.py
+++ b/test/suite/test_live_restore04.py
@@ -29,14 +29,13 @@
 import filecmp, os, glob, wiredtiger, wttest
 from wtdataset import SimpleDataSet
 from wtscenario import make_scenarios
-from helper import copy_wiredtiger_home
-from suite_subprocess import suite_subprocess
+from wtbackup import backup_base
 
 
 # test_live_restore04.py
 # Test using the wt utility with live restore.
 @wttest.skip_for_hook("tiered", "using multiple WT homes")
-class test_live_restore04(wttest.WiredTigerTestCase, suite_subprocess):
+class test_live_restore04(backup_base):
     format_values = [
         ('column', dict(key_format='r', value_format='S')),
         ('row_integer', dict(key_format='i', value_format='S')),
@@ -75,10 +74,10 @@ def test_live_restore04(self):
             self.runWt(['dump', '-x', uris[i]], outfilename=dump_out)
 
         # Close the default connection.
+        os.mkdir("SOURCE")
+        self.take_full_backup("SOURCE")
         self.close_conn()
 
-        copy_wiredtiger_home(self, '.', "SOURCE")
-
         # Remove everything but SOURCE / stderr / stdout / util output folder.
         for f in glob.glob("*"):
             if not f == "SOURCE" and not f == "UTIL" and not f == "stderr.txt" and not f == "stdout.txt":
diff --git a/test/suite/test_live_restore05.py b/test/suite/test_live_restore05.py
index daaf5d017a7..82bb3d01984 100644
--- a/test/suite/test_live_restore05.py
+++ b/test/suite/test_live_restore05.py
@@ -29,14 +29,13 @@
 import os, glob, wttest
 from wtdataset import SimpleDataSet
 from wtscenario import make_scenarios
-from helper import copy_wiredtiger_home
-from suite_subprocess import suite_subprocess
+from wtbackup import backup_base
 
 
 # test_live_restore05.py
 # Reproduce a live restore edge case that resulted in duplicate metadata entries.
 @wttest.skip_for_hook("tiered", "using multiple WT homes")
-class test_live_restore05(wttest.WiredTigerTestCase, suite_subprocess):
+class test_live_restore05(backup_base):
     format_values = [
         ('row_integer', dict(key_format='i', value_format='S')),
         ('column_store', dict(key_format='r', value_format='S'))
@@ -63,10 +62,10 @@ def test_live_restore05(self):
             ds.populate()
 
         # Close the default connection.
+        os.mkdir("SOURCE")
+        self.take_full_backup("SOURCE")
         self.close_conn()
 
-        copy_wiredtiger_home(self, '.', "SOURCE")
-
         # Remove everything but SOURCE / stderr / stdout / util output folder.
         for f in glob.glob("*"):
             if not f == "SOURCE" and not f == "UTIL" and not f == "stderr.txt" and not f == "stdout.txt":
diff --git a/test/suite/test_live_restore06.py b/test/suite/test_live_restore06.py
index 8c2fade30cf..e281df145fc 100644
--- a/test/suite/test_live_restore06.py
+++ b/test/suite/test_live_restore06.py
@@ -29,12 +29,11 @@
 import os, glob, time, wiredtiger, wttest
 from wiredtiger import stat
 from wtdataset import SimpleDataSet
-from helper import copy_wiredtiger_home
 from wtbackup import backup_base
 
 
 # test_live_restore06.py
-# Ensure WiredTiger cleans any remaining 'nbits=-1' strings from file metadata during backups.
+# Ensure WiredTiger cleans all 'nbits=-1' strings from file metadata during backups.
 @wttest.skip_for_hook("tiered", "using multiple WT homes")
 class test_live_restore06(backup_base):
 
@@ -49,14 +48,15 @@ def test_live_restore06(self):
         if os.name == 'nt':
             return
 
-        # Create a DB in SOURCE to be restored
+        # Create an initial DB in SOURCE to be restored
         for i in range(0, 3):
             ds = SimpleDataSet(self, f'file:collection_{i}', 10_000,
             key_format='S', value_format='S')
             ds.populate()
 
+        os.mkdir("SOURCE")
+        self.take_full_backup("SOURCE")
         self.close_conn()
-        copy_wiredtiger_home(self, '.', "SOURCE")
 
         # Remove everything but SOURCE / stderr / stdout.
         for f in glob.glob("*"):
@@ -88,21 +88,13 @@ def test_live_restore06(self):
             iteration_count += 1
         self.assertEqual(state, wiredtiger.WT_LIVE_RESTORE_COMPLETE)
 
-        # Walk through all the metadata configs to find a file with nbits == -1.
-        # This is the value we need to clean up.
+        # Once live restore has completed the metadata of all files will have nbits=-1
         meta_cursor = self.session.open_cursor('metadata:', None, None)
-        neg_one_metadata_file = None
-        while True:
-            ret = meta_cursor.next()
-            if ret != 0:
-                break
+        while meta_cursor.next() != 0:
             uri = meta_cursor.get_key()
-            if uri.find("file:") != -1 and "nbits=-1" in meta_cursor[uri]:
-                neg_one_metadata_file = uri
-                break
-
+            if uri.find("file:") != -1:
+                self.assertTrue("nbits=-1" in meta_cursor[uri])
         meta_cursor.close()
-        self.assertTrue(neg_one_metadata_file is not None)
 
         # Now take a backup of the destination.
         # This requires opening a new connection in non-live restore mode
@@ -117,11 +109,12 @@ def test_live_restore06(self):
         with open("backup/WiredTiger.backup", "r") as f:
             self.assertTrue("nbits=-1" not in f.read())
 
-        # Now open the backup and check the metadata for the nbits=-1 string is not present.
+        # Now open the backup and check all files have been cleaned and contain nbits=0.
         self.reopen_conn(directory="backup", config="statistics=(all),live_restore=(enabled=false)")
         meta_cursor = self.session.open_cursor('metadata:', None, None)
-        meta_cursor.set_key(neg_one_metadata_file)
-        config = meta_cursor[neg_one_metadata_file]
-        self.assertTrue("nbits=-1" not in config)
+        while meta_cursor.next() != 0:
+            uri = meta_cursor.get_key()
+            if uri.find("file:") != -1:
+                self.assertTrue("nbits=0," in meta_cursor[uri])
         meta_cursor.close()
 
diff --git a/test/suite/test_live_restore07.py b/test/suite/test_live_restore07.py
index 9f53c63c274..a6905c6bdd0 100644
--- a/test/suite/test_live_restore07.py
+++ b/test/suite/test_live_restore07.py
@@ -29,10 +29,10 @@
 import os, wiredtiger, wttest
 from wtscenario import make_scenarios
 
-# test_live_restore06.py
+# test_live_restore07.py
 # Test that restoring from an empty database fails.
 @wttest.skip_for_hook("tiered", "using multiple WT homes")
-class test_live_restore02(wttest.WiredTigerTestCase):
+class test_live_restore07(wttest.WiredTigerTestCase):
     format_values = [
         ('column', dict(key_format='r', value_format='S')),
         ('row_integer', dict(key_format='i', value_format='S')),
@@ -41,7 +41,7 @@ class test_live_restore02(wttest.WiredTigerTestCase):
 
     scenarios = make_scenarios(format_values)
 
-    def test_live_restore02(self):
+    def test_live_restore07(self):
         # Live restore is not supported on Windows.
         if os.name == 'nt':
             return


Reviews:
user: login: Sean04
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T01:41:17Z
user: login: ajmorton
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T02:30:10Z
user: login: ajmorton
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T02:31:38Z
user: login: Sean04
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T02:32:27Z
user: login: Sean04
body: lgtm
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T02:32:39Z

Review comments:
diff_hunk: @@ -0,0 +1,73 @@
+#!/usr/bin/env python
+#
+# Public Domain 2014-present MongoDB, Inc.
+# Public Domain 2008-2014 WiredTiger, Inc.
+#
+# This is free and unencumbered software released into the public domain.
+#
+# Anyone is free to copy, modify, publish, use, compile, sell, or
+# distribute this software, either in source code form or as a compiled
+# binary, for any purpose, commercial or non-commercial, and by any
+# means.
+#
+# In jurisdictions that recognize copyright laws, the author or authors
+# of this software dedicate any and all copyright interest in the
+# software to the public domain. We make this dedication for the benefit
+# of the public at large and to the detriment of our heirs and
+# successors. We intend this dedication to be an overt act of
+# relinquishment in perpetuity of all present and future rights to this
+# software under copyright law.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+# OTHER DEALINGS IN THE SOFTWARE.
+#
+# take_backup.py
+#   Take a backup of the database created by btree-500m-populate.wtperf to be used
+#   as the source directory in live restore perf tests.
+#   This script needs to be run from build/bench/wtperf and the btree-500m-populate
+#   task must have run already to generate the WT_TEST_0_0 folder.
+
+import os
+import shutil
+import sys
+
+sys.path.append(os.path.abspath('../../lang/python'))
+
+from  wiredtiger import wiredtiger_open
+
+def main():
+
+    home = "WT_TEST_0_0"
+    backup_dir = "WT_TEST_0_0_backup"
+
+    # Create a clean backup directory
+    if os.path.exists(backup_dir):
+        shutil.rmtree(backup_dir)
+    os.makedirs(backup_dir)
+
+    # Connect to the database and open a session
+    conn = wiredtiger_open('WT_TEST_0_0', None)
+    session = conn.open_session()
+
+    backup_cursor = session.open_cursor('backup:', None)
+
+    while backup_cursor.next() == 0:
+        orig_file = f'{home}/{backup_cursor.get_key()}'
+        backup_file = f'{backup_dir}/{backup_cursor.get_key()}'
+        shutil.copyfile(orig_file, backup_file)
+
+    # # I don't know what this is
+    # shutil.copyfile(f"{home}/CONFIG.wtperf", f"{backup_dir}/CONFIG.wtperf")
path: test/live_restore/take_backup.py
user: login: Sean04
body: remove?
created_at: 2025-03-24T01:37:02Z
author_association: CONTRIBUTOR
path: test/live_restore/take_backup.py
user: login: Sean04
body: It would be better if we could do this in wtperf but `backup_interval` doesn't quite match what we need here. Did you consider adding a new config to wtperf for it to create a backup at the end of a run?
created_at: 2025-03-24T01:41:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,73 @@
+#!/usr/bin/env python
+#
+# Public Domain 2014-present MongoDB, Inc.
+# Public Domain 2008-2014 WiredTiger, Inc.
+#
+# This is free and unencumbered software released into the public domain.
+#
+# Anyone is free to copy, modify, publish, use, compile, sell, or
+# distribute this software, either in source code form or as a compiled
+# binary, for any purpose, commercial or non-commercial, and by any
+# means.
+#
+# In jurisdictions that recognize copyright laws, the author or authors
+# of this software dedicate any and all copyright interest in the
+# software to the public domain. We make this dedication for the benefit
+# of the public at large and to the detriment of our heirs and
+# successors. We intend this dedication to be an overt act of
+# relinquishment in perpetuity of all present and future rights to this
+# software under copyright law.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+# IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+# OTHER DEALINGS IN THE SOFTWARE.
+#
+# take_backup.py
+#   Take a backup of the database created by btree-500m-populate.wtperf to be used
+#   as the source directory in live restore perf tests.
+#   This script needs to be run from build/bench/wtperf and the btree-500m-populate
+#   task must have run already to generate the WT_TEST_0_0 folder.
+
+import os
+import shutil
+import sys
+
+sys.path.append(os.path.abspath('../../lang/python'))
+
+from  wiredtiger import wiredtiger_open
+
+def main():
+
+    home = "WT_TEST_0_0"
+    backup_dir = "WT_TEST_0_0_backup"
+
+    # Create a clean backup directory
+    if os.path.exists(backup_dir):
+        shutil.rmtree(backup_dir)
+    os.makedirs(backup_dir)
+
+    # Connect to the database and open a session
+    conn = wiredtiger_open('WT_TEST_0_0', None)
+    session = conn.open_session()
+
+    backup_cursor = session.open_cursor('backup:', None)
+
+    while backup_cursor.next() == 0:
+        orig_file = f'{home}/{backup_cursor.get_key()}'
+        backup_file = f'{backup_dir}/{backup_cursor.get_key()}'
+        shutil.copyfile(orig_file, backup_file)
+
+    # # I don't know what this is
+    # shutil.copyfile(f"{home}/CONFIG.wtperf", f"{backup_dir}/CONFIG.wtperf")
path: test/live_restore/take_backup.py
user: login: ajmorton
body: Fixed in [fba653d](https://github.com/wiredtiger/wiredtiger/pull/11771/commits/fba653d79ad2e4329cafa11d887b0c16aed5e41f)
created_at: 2025-03-24T02:30:10Z
author_association: CONTRIBUTOR
path: test/live_restore/take_backup.py
user: login: ajmorton
body: We could, but I think this is the only use case we'll have for taking backups in this way. I also don't want to spend too much time learning wtperf when the script works.
Would you be ok with merging but I create a tech debt ticket, or leave a comment to generalise it if we find a new use case for taking backups?
created_at: 2025-03-24T02:31:38Z
author_association: CONTRIBUTOR
path: test/live_restore/take_backup.py
user: login: Sean04
body: Yeah sounds good
created_at: 2025-03-24T02:32:26Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-23T23:06:07Z
body: ## Type of change made in this PR
- [x] Functional change
- [ ] Test-only change
- [ ] Refactor-only change
- [ ] Other non-functional change

## What makes this change safe?
Low risk. This simplifies the code to make it more understandable. It adds asserts to prevent the newly invalid states that could arise.

## Checklist before requesting a review
- [x] I have performed a self-review of my code.
- [x] I have made corresponding changes to the documentation, or no documentation change is needed.
- [x] I have tests that show my change is correct, or have described above why functional test changes are not required.

author_association: NONE
user: login: ajmorton
created_at: 2025-03-23T23:06:50Z
body: @Sean04 this is the re-delivery of #11757. The only changes are fixes to our performance testing to create proper backup folders for the source directory. These changes are in [6c35339](https://github.com/wiredtiger/wiredtiger/pull/11771/commits/6c35339613e0c12792a399f3e26d4b8ccbe32a89).

I've smoke tested the new perf testing logic in a patch with shorted runtimes, but I'll also add the proper perf tests to this PR for safety.
author_association: CONTRIBUTOR
user: login: wiredtiger-prbot[bot]
created_at: 2025-03-23T23:15:51Z
body: 
Test coverage is very low, please refer to the Code change/coverage report links below and try to improve it if feasible.

| Metric (for added/changed code) | Coverage |
|------------------------------------------|-------|
| Line coverage                            | 78% (7/9) |
| Branch coverage                          | 64% (9/14) |

- [Code change report](https://build_external.s3.amazonaws.com/wiredtiger/code-statistics/5c854595e3da1f0107d7db390172988e3b5b2bb9/code-change-report_wiredtiger_code_statistics_patch_5c854595e3da1f0107d7db390172988e3b5b2bb9_67e0c3631b24d4000714a671_25_03_24_02_28_52-0/code_change_report.html)
- [Code coverage report (catch2)](https://build_external.s3.amazonaws.com/wiredtiger/code-statistics/5c854595e3da1f0107d7db390172988e3b5b2bb9/coverage-report-catch2_wiredtiger_code_statistics_patch_5c854595e3da1f0107d7db390172988e3b5b2bb9_67e0c3631b24d4000714a671_25_03_24_02_28_52-0/1_coverage_report_main.html)
- [Code coverage report (full)](https://build_external.s3.amazonaws.com/wiredtiger/code-statistics/5c854595e3da1f0107d7db390172988e3b5b2bb9/generate-coverage-report_wiredtiger_code_statistics_patch_5c854595e3da1f0107d7db390172988e3b5b2bb9_67e0c3631b24d4000714a671_25_03_24_02_28_52-0/1_coverage_report_main.html)




<!-- STICKY_COMMENT:CODE_QUALITY -->
author_association: NONE