Repository: apple/foundationdb
Pull request #12137: BulkLoad Load Balance Based on DataMove Busy Map 
Author: login: kakaiu
Created: 2025-05-07T03:21:44Z | Merged: 2025-05-08T18:28:27Z

When load balancing for bulkload tasks, we consider to assign a task to SSes with minimal ongoing tasks. To achieve this, previously, DD periodically (for every 2 seconds) contacts SSes to collect their latest number of ongoing tasks. However, this number may not be accurate and causing load imbalance in the beginning phase of a job, because the right ongoing tasks number experiences some delay before it propagates to the DD. To solve this issue, DDQueue tracks the number of ongoing bulkload data moves for each storage server, which represents the number of ongoing tasks. Since this number is updated at any time, we do not have the above problem.

With this PR, we achieved significant performance boost: to load 12TB logical data, we previous takes 3.5 hours. With the fix, the time span reduces to 2.5 hours. 

500K bulkdump:
  20250507-171421-zhewang-fb9e5940a5d1f51a           compressed=True data_size=41210462 duration=16266167 ended=500000 fail=3 fail_fast=10 max_runs=500000 pass=499997 priority=100 remaining=0 runtime=2:39:41 sanity=False started=500000 stopped=20250507-195402 submitted=20250507-171421 timeout=5400 username=zhewang
  
500K bulkload:
  20250507-202607-zhewang-7122b4d2edda6103           compressed=True data_size=41210767 duration=7431916 ended=500000 fail=2 fail_fast=10 max_runs=500000 pass=499998 priority=100 remaining=0 runtime=1:54:55 sanity=False started=500000 stopped=20250507-222102 submitted=20250507-202607 timeout=5400 username=zhewang

100K correctness with irrelevant unit test failures:
  20250508-052442-zhewang-78fdb7adb1023299           compressed=True data_size=41168284 duration=5487418 ended=100000 fail=3 fail_fast=10 max_runs=100000 pass=99997 priority=100 remaining=0 runtime=0:52:44 sanity=False started=100000 stopped=20250508-061726 submitted=20250508-052442 timeout=5400 username=zhewang
      
# Code-Reviewer Section

The general pull request guidelines can be found [here](https://github.com/apple/foundationdb/wiki/FoundationDB-Commit-Process).

Please check each of the following things and check *all* boxes before accepting a PR.

- [ ] The PR has a description, explaining both the problem and the solution.
- [ ] The description mentions which forms of testing were done and the testing seems reasonable.
- [ ] Every function/class/actor that was touched is reasonably well documented.

## For Release-Branches

If this PR is made against a release-branch, please also check the following:

- [ ] This change/bugfix is a cherry-pick from the next younger branch (younger `release-branch` or `main` if this is the youngest branch)
- [ ] There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)



Diff:
diff --git a/fdbclient/ServerKnobs.cpp b/fdbclient/ServerKnobs.cpp
index 5551ee9d236..51bc502639f 100644
--- a/fdbclient/ServerKnobs.cpp
+++ b/fdbclient/ServerKnobs.cpp
@@ -390,6 +390,7 @@ void ServerKnobs::initialize(Randomize randomize, ClientKnobs* clientKnobs, IsSi
 	init( BULKLOAD_ASYNC_READ_WRITE_BLOCK_SIZE,            1024*1024 ); if (isSimulated) BULKLOAD_ASYNC_READ_WRITE_BLOCK_SIZE = deterministicRandom()->randomInt(1024, 10240);
 	init( MANIFEST_COUNT_MAX_PER_BULKLOAD_TASK,                   10 ); if (isSimulated) MANIFEST_COUNT_MAX_PER_BULKLOAD_TASK = deterministicRandom()->randomInt(1, 11);
 	init( BULKLOAD_SIM_FAILURE_INJECTION,                      false ); if (isSimulated) BULKLOAD_SIM_FAILURE_INJECTION = true;
+	init( DD_BULKLOAD_POWER_OF_D_RATIO,                          2.0 ); if (isSimulated) DD_BULKLOAD_POWER_OF_D_RATIO = deterministicRandom()->randomInt(1, 11);
 
 	// BulkDumping
 	init( DD_BULKDUMP_SCHEDULE_MIN_INTERVAL_SEC,                 5.0 ); if( randomize && BUGGIFY ) DD_BULKDUMP_SCHEDULE_MIN_INTERVAL_SEC = deterministicRandom()->random01() * 10 + 1;
diff --git a/fdbclient/include/fdbclient/ServerKnobs.h b/fdbclient/include/fdbclient/ServerKnobs.h
index 012332ab0d5..bc2942c1d28 100644
--- a/fdbclient/include/fdbclient/ServerKnobs.h
+++ b/fdbclient/include/fdbclient/ServerKnobs.h
@@ -415,6 +415,9 @@ class SWIFT_CXX_IMMORTAL_SINGLETON_TYPE ServerKnobs : public KnobsImpl<ServerKno
 	int BULKLOAD_ASYNC_READ_WRITE_BLOCK_SIZE; // the block size when performing async read/write for bulkload
 	int MANIFEST_COUNT_MAX_PER_BULKLOAD_TASK; // the max number of manifest that a bulkload task can process
 	bool BULKLOAD_SIM_FAILURE_INJECTION; // Set to true to inject failure in bulkload simulation
+	double DD_BULKLOAD_POWER_OF_D_RATIO; // When selecting the dest team, DD randomly chooses 1/D portion of all valid
+	                                     // teams as the candidates and the DD selects the team with the minimal number
+	                                     // of ongoing tasks from the candidates as the dest team.
 	bool CC_ENFORCE_USE_UNFIT_DD_IN_SIM; // Set for CC to enforce to use an unfit DD in the simulation. This knob takes
 	                                     // effect only in the simulation.
 	bool DISABLE_AUDIT_STORAGE_FINAL_REPLICA_CHECK_IN_SIM; // Set to disable audit storage replica check in the
diff --git a/fdbserver/DDRelocationQueue.actor.cpp b/fdbserver/DDRelocationQueue.actor.cpp
index d0028100685..a860b6d5958 100644
--- a/fdbserver/DDRelocationQueue.actor.cpp
+++ b/fdbserver/DDRelocationQueue.actor.cpp
@@ -2087,7 +2087,16 @@ ACTOR Future<Void> dataDistributionRelocator(DDQueue* self,
 			healthyDestinations.addDataInFlightToTeam(+metrics.bytes);
 			healthyDestinations.addReadInFlightToTeam(+metrics.readLoadKSecond());
 
+			// At this point, we are about to launch the data move, so we should update the busy map counter
+			// for destination servers.
 			launchDest(rd, bestTeams, self->destBusymap);
+			if (doBulkLoading) {
+				for (const auto& [team, _] : bestTeams) {
+					for (const UID& ssid : team->getServerIDs()) {
+						self->bulkLoadTaskCollection->busyMap.addTask(ssid);
+					}
+				}
+			}
 
 			TraceEvent ev(relocateShardInterval.severity, "RelocateShardHasDestination", distributorId);
 			RelocateDecision decision{ rd, destIds, extraIds, metrics, parentMetrics };
@@ -2296,6 +2305,11 @@ ACTOR Future<Void> dataDistributionRelocator(DDQueue* self,
 					}
 
 					if (doBulkLoading) {
+						for (const auto& [team, _] : bestTeams) {
+							for (const UID& ssid : team->getServerIDs()) {
+								self->bulkLoadTaskCollection->busyMap.removeTask(ssid);
+							}
+						}
 						try {
 							self->bulkLoadTaskCollection->terminateTask(rd.bulkLoadTask.get().coreState);
 							TraceEvent(
@@ -2323,6 +2337,11 @@ ACTOR Future<Void> dataDistributionRelocator(DDQueue* self,
 						    .errorUnsuppressed(error)
 						    .detail("JobID", rd.bulkLoadTask.get().coreState.getJobId())
 						    .detail("TaskID", rd.bulkLoadTask.get().coreState.getTaskId());
+						for (const auto& [team, _] : bestTeams) {
+							for (const UID& ssid : team->getServerIDs()) {
+								self->bulkLoadTaskCollection->busyMap.removeTask(ssid);
+							}
+						}
 					}
 					throw error;
 				}
@@ -2342,6 +2361,18 @@ ACTOR Future<Void> dataDistributionRelocator(DDQueue* self,
 				}
 				rd.completeDests.clear();
 
+				if (doBulkLoading) {
+					TraceEvent(bulkLoadVerboseEventSev(), "DDBulkLoadTaskRelocatorError")
+					    .errorUnsuppressed(error)
+					    .detail("JobID", rd.bulkLoadTask.get().coreState.getJobId())
+					    .detail("TaskID", rd.bulkLoadTask.get().coreState.getTaskId());
+					for (const auto& [team, _] : bestTeams) {
+						for (const UID& ssid : team->getServerIDs()) {
+							self->bulkLoadTaskCollection->busyMap.removeTask(ssid);
+						}
+					}
+				}
+
 				wait(delay(SERVER_KNOBS->RETRY_RELOCATESHARD_DELAY, TaskPriority::DataDistributionLaunch));
 			}
 		}
diff --git a/fdbserver/DDTeamCollection.actor.cpp b/fdbserver/DDTeamCollection.actor.cpp
index 1ed07a8b3e2..3fc3ae9f230 100644
--- a/fdbserver/DDTeamCollection.actor.cpp
+++ b/fdbserver/DDTeamCollection.actor.cpp
@@ -236,7 +236,6 @@ class DDTeamCollectionImpl {
 			int unhealthyTeamCount = 0;
 			int notEligibileTeamCount = 0;
 			int duplicatedCount = 0;
-			int failToGetMetricsCount = 0;
 			for (const auto& dest : self->teams) {
 				if (!dest->isHealthy()) {
 					unhealthyTeamCount++;
@@ -255,12 +254,6 @@ class DDTeamCollectionImpl {
 						continue;
 					}
 				}
-				Optional<int> ongoingBulkLoadTaskCount = dest->getMaxOngoingBulkLoadTaskCount();
-				if (!ongoingBulkLoadTaskCount.present()) {
-					// This team may have an unhealthy SS, so avoid selecting it
-					failToGetMetricsCount++;
-					continue;
-				}
 				bool ok = true;
 				for (const auto& srcId : req.src) {
 					std::vector<UID> serverIds = dest->getServerIDs();
@@ -283,23 +276,25 @@ class DDTeamCollectionImpl {
 
 			// Step 2: Conduct Power-of-D-Choice to select a team
 			std::vector<Reference<TCTeamInfo>> candidateTeams;
-			if (validTeams.size() <= 4) {
+			if (validTeams.size() <= SERVER_KNOBS->DD_BULKLOAD_POWER_OF_D_RATIO) {
 				candidateTeams = validTeams;
 			} else {
 				deterministicRandom()->randomShuffle(validTeams);
-				candidateTeams =
-				    std::vector<Reference<TCTeamInfo>>(validTeams.begin(), validTeams.begin() + validTeams.size() / 4);
+				candidateTeams = std::vector<Reference<TCTeamInfo>>(
+				    validTeams.begin(),
+				    validTeams.begin() + std::floor(validTeams.size() / SERVER_KNOBS->DD_BULKLOAD_POWER_OF_D_RATIO));
 			}
 
 			Optional<Reference<IDataDistributionTeam>> res;
-			int minOngoingBulkLoadTaskCount = 0;
+			int minOngoingBulkLoadTaskCount = std::numeric_limits<int>::max();
 			for (int i = 0; i < candidateTeams.size(); i++) {
-				Optional<int> ongoingBulkLoadTaskCount = candidateTeams[i]->getMaxOngoingBulkLoadTaskCount();
-				if (!ongoingBulkLoadTaskCount.present()) {
-					continue;
+				int ongoingBulkLoadTaskCount = -1;
+				for (const auto& ssid : candidateTeams[i]->getServerIDs()) {
+					ongoingBulkLoadTaskCount =
+					    std::max(ongoingBulkLoadTaskCount, self->bulkLoadTaskCollection->busyMap.getTaskCount(ssid));
 				}
-				if (!res.present() || ongoingBulkLoadTaskCount.get() < minOngoingBulkLoadTaskCount) {
-					minOngoingBulkLoadTaskCount = ongoingBulkLoadTaskCount.get();
+				if (!res.present() || ongoingBulkLoadTaskCount < minOngoingBulkLoadTaskCount) {
+					minOngoingBulkLoadTaskCount = ongoingBulkLoadTaskCount;
 					res = candidateTeams[i];
 				}
 			}
@@ -314,7 +309,6 @@ class DDTeamCollectionImpl {
 				    .detail("UnhealthyTeamCount", unhealthyTeamCount)
 				    .detail("DuplicatedCount", duplicatedCount)
 				    .detail("NotEligibileTeamCount", notEligibileTeamCount)
-				    .detail("FailToGetMetricsCount", failToGetMetricsCount)
 				    .detail("DestIds", describe(res.get()->getServerIDs()))
 				    .detail("DestTeam", res.get()->getTeamID());
 			} else {
@@ -326,7 +320,6 @@ class DDTeamCollectionImpl {
 				    .detail("ValidTeamSize", validTeams.size())
 				    .detail("UnhealthyTeamCount", unhealthyTeamCount)
 				    .detail("DuplicatedCount", duplicatedCount)
-				    .detail("FailToGetMetricsCount", failToGetMetricsCount)
 				    .detail("NotEligibileTeamCount", notEligibileTeamCount);
 			}
 			req.reply.send(std::make_pair(res, false));
@@ -4446,7 +4439,8 @@ DDTeamCollection::DDTeamCollection(DDTeamCollectionInitParams const& params)
     storageServerRecruitmentEventHolder(
         makeReference<EventCacheHolder>("StorageServerRecruitment_" + params.distributorId.toString())),
     primary(params.primary), distributorId(params.distributorId), underReplication(false),
-    configuration(params.configuration), storageServerSet(new LocalityMap<UID>()) {
+    configuration(params.configuration), storageServerSet(new LocalityMap<UID>()),
+    bulkLoadTaskCollection(params.bulkLoadTaskCollection) {
 
 	if (!primary || configuration.usableRegions == 1) {
 		TraceEvent("DDTrackerStarting", distributorId)
@@ -6230,7 +6224,8 @@ class DDTeamCollectionUnitTest {
 		                                                     Promise<UID>(),
 		                                                     PromiseStream<Promise<int>>(),
 		                                                     PromiseStream<Promise<int64_t>>(),
-		                                                     PromiseStream<RebalanceStorageQueueRequest>() }));
+		                                                     PromiseStream<RebalanceStorageQueueRequest>(),
+		                                                     makeReference<BulkLoadTaskCollection>(UID(0, 0)) }));
 
 		for (int id = 1; id <= processCount; ++id) {
 			UID uid(id, 0);
@@ -6284,7 +6279,8 @@ class DDTeamCollectionUnitTest {
 		                                                     Promise<UID>(),
 		                                                     PromiseStream<Promise<int>>(),
 		                                                     PromiseStream<Promise<int64_t>>(),
-		                                                     PromiseStream<RebalanceStorageQueueRequest>() }));
+		                                                     PromiseStream<RebalanceStorageQueueRequest>(),
+		                                                     makeReference<BulkLoadTaskCollection>(UID(0, 0)) }));
 
 		for (int id = 1; id <= processCount; id++) {
 			UID uid(id, 0);
diff --git a/fdbserver/DataDistribution.actor.cpp b/fdbserver/DataDistribution.actor.cpp
index 728b07b7a18..392e7fdeee5 100644
--- a/fdbserver/DataDistribution.actor.cpp
+++ b/fdbserver/DataDistribution.actor.cpp
@@ -50,6 +50,7 @@
 #include "flow/Platform.h"
 #include "flow/Trace.h"
 #include "flow/UnitTest.h"
+#include "flow/flow.h"
 #include "flow/genericactors.actor.h"
 #include "flow/serialize.h"
 #include "flow/actorcompiler.h" // This must be the last #include.
@@ -1939,6 +1940,7 @@ ACTOR Future<Void> scheduleBulkLoadJob(Reference<DataDistributor> self, Promise<
 					                                    self->bulkLoadJobManager.get().manifestLocalTempFolder,
 					                                    manifestEntries,
 					                                    errorOut));
+					wait(yield()); // Avoid busy loop
 				}
 				ASSERT(beginKey == res[i + 1].key);
 			}
@@ -2748,7 +2750,8 @@ ACTOR Future<Void> dataDistribution(Reference<DataDistributor> self,
 			    removeFailedServer,
 			    getUnhealthyRelocationCount,
 			    getAverageShardBytes,
-			    triggerStorageQueueRebalance });
+			    triggerStorageQueueRebalance,
+			    self->bulkLoadTaskCollection });
 			teamCollectionsPtrs.push_back(self->context->primaryTeamCollection.getPtr());
 			Reference<IAsyncListener<RequestStream<RecruitStorageRequest>>> recruitStorage;
 			if (!isMocked) {
@@ -2774,7 +2777,8 @@ ACTOR Future<Void> dataDistribution(Reference<DataDistributor> self,
 				                                removeFailedServer,
 				                                getUnhealthyRelocationCount,
 				                                getAverageShardBytes,
-				                                triggerStorageQueueRebalance });
+				                                triggerStorageQueueRebalance,
+				                                self->bulkLoadTaskCollection });
 				teamCollectionsPtrs.push_back(self->context->remoteTeamCollection.getPtr());
 				self->context->remoteTeamCollection->teamCollections = teamCollectionsPtrs;
 				actors.push_back(reportErrorsExcept(DDTeamCollection::run(self->context->remoteTeamCollection,
diff --git a/fdbserver/include/fdbserver/DDTeamCollection.h b/fdbserver/include/fdbserver/DDTeamCollection.h
index cee0479dadb..46817c8b7b5 100644
--- a/fdbserver/include/fdbserver/DDTeamCollection.h
+++ b/fdbserver/include/fdbserver/DDTeamCollection.h
@@ -200,6 +200,7 @@ struct DDTeamCollectionInitParams {
 	PromiseStream<Promise<int>> getUnhealthyRelocationCount;
 	PromiseStream<Promise<int64_t>> getAverageShardBytes;
 	PromiseStream<RebalanceStorageQueueRequest> triggerStorageQueueRebalance;
+	Reference<BulkLoadTaskCollection> bulkLoadTaskCollection;
 };
 
 class DDTeamCollection : public ReferenceCounted<DDTeamCollection> {
@@ -283,6 +284,8 @@ class DDTeamCollection : public ReferenceCounted<DDTeamCollection> {
 	AsyncVar<Optional<Key>> healthyZone;
 	Future<bool> clearHealthyZoneFuture;
 
+	Reference<BulkLoadTaskCollection> bulkLoadTaskCollection;
+
 	// team pivot values
 	struct {
 		double lastPivotValuesUpdate = 0.0;
diff --git a/fdbserver/include/fdbserver/DataDistribution.actor.h b/fdbserver/include/fdbserver/DataDistribution.actor.h
index 1ed9b7f4dc7..d410e697cb1 100644
--- a/fdbserver/include/fdbserver/DataDistribution.actor.h
+++ b/fdbserver/include/fdbserver/DataDistribution.actor.h
@@ -539,6 +539,33 @@ struct TeamCollectionInterface {
 	PromiseStream<GetTeamRequest> getTeam;
 };
 
+// Used to track the number of ongoing bulkload tasks for each storage server
+struct DDBulkLoadTaskBusyMap {
+public:
+	void addTask(const UID& ssid) { busyMap[ssid]++; }
+
+	void removeTask(const UID& ssid) {
+		auto it = busyMap.find(ssid);
+		ASSERT(it != busyMap.end());
+		it->second--;
+		if (it->second == 0) {
+			busyMap.erase(it);
+		}
+	}
+
+	int getTaskCount(const UID& ssid) {
+		auto it = busyMap.find(ssid);
+		if (it == busyMap.end()) {
+			return 0;
+		} else {
+			return it->second;
+		}
+	}
+
+private:
+	std::unordered_map<UID, int> busyMap; // <Storage Server ID, Task Count>
+};
+
 // Used to piggyback the data move priority when an unretrievable error happens to the task datamove.
 // If the priority indicates the data move is a team unhealthy related data move, the bulkload engine
 // system trigger a new data move when terminate the error task.
@@ -749,6 +776,8 @@ class BulkLoadTaskCollection : public ReferenceCounted<BulkLoadTaskCollection> {
 		return res;
 	}
 
+	DDBulkLoadTaskBusyMap busyMap; // <SSID, taskCount>
+
 private:
 	KeyRangeMap<Optional<DDBulkLoadEngineTask>> bulkLoadTaskMap;
 	UID ddId;
diff --git a/fdbserver/workloads/PerpetualWiggleStatsWorkload.actor.cpp b/fdbserver/workloads/PerpetualWiggleStatsWorkload.actor.cpp
index fa9ef7c8d2c..6bc50442f52 100644
--- a/fdbserver/workloads/PerpetualWiggleStatsWorkload.actor.cpp
+++ b/fdbserver/workloads/PerpetualWiggleStatsWorkload.actor.cpp
@@ -214,7 +214,8 @@ struct PerpetualWiggleStatsWorkload : public TestWorkload {
 		                                      Promise<UID>(),
 		                                      PromiseStream<Promise<int>>(),
 		                                      PromiseStream<Promise<int64_t>>(),
-		                                      PromiseStream<RebalanceStorageQueueRequest>() });
+		                                      PromiseStream<RebalanceStorageQueueRequest>(),
+		                                      makeReference<BulkLoadTaskCollection>(UID(8, 6)) });
 		tester.configuration.storageTeamSize = 3;
 		tester.configuration.perpetualStorageWiggleSpeed = 1;
 


Reviews:
user: login: saintstack
body: LG. A few comments. I think you meant 12TB in your general comment on this PR (you had 1.2TB). In your test run, could you see the difference this change made? Any other way of testing/seeing the improvement this change adds? Thanks.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T04:10:37Z
user: login: kakaiu
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T04:46:05Z
user: login: kakaiu
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T04:47:00Z
user: login: kakaiu
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T04:47:22Z
user: login: saintstack
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T04:48:19Z
user: login: saintstack
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T04:49:39Z
user: login: saintstack
body: nits
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T10:19:55Z
user: login: kakaiu
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T16:29:37Z
user: login: kakaiu
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T16:29:59Z
user: login: saintstack
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T18:32:29Z
user: login: jzhou77
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T03:52:31Z
user: login: kakaiu
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-08T05:20:18Z
user: login: jzhou77
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T15:37:13Z

Review comments:
diff_hunk: @@ -539,6 +541,37 @@ struct TeamCollectionInterface {
 	PromiseStream<GetTeamRequest> getTeam;
 };
 
+struct DDBulkLoadTaskBusyMap {
path: fdbserver/include/fdbserver/DataDistribution.actor.h
user: login: saintstack
body: A bit of doc. on head of this struct would be welcome; mostly how it is used.
created_at: 2025-05-07T03:55:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -579,7 +612,7 @@ inline bool bulkDumpIsEnabled(int bulkDumpModeValue) {
 	return bulkDumpModeValue == 1;
 }
 
-class BulkLoadTaskCollection : public ReferenceCounted<BulkLoadTaskCollection> {
path: fdbserver/include/fdbserver/DataDistribution.actor.h
user: login: saintstack
body: Any reason for moving to shard_ptr? Something we should be doing generally?
created_at: 2025-05-07T03:59:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2088,6 +2088,13 @@ ACTOR Future<Void> dataDistributionRelocator(DDQueue* self,
 			healthyDestinations.addReadInFlightToTeam(+metrics.readLoadKSecond());
 
 			launchDest(rd, bestTeams, self->destBusymap);
+			if (doBulkLoading) {
path: fdbserver/DDRelocationQueue.actor.cpp
user: login: saintstack
body: dataDistributionRelocator is a massive method. A comment on what the (expected) context is when we are at this point would help (I scrolled back a few pages and couldn't figure if this a bulk loading context)

There is an if (doBulkLoading) on line 2060. Could its validateBulkLoadRelocateData be moved here?
created_at: 2025-05-07T04:05:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -539,6 +541,37 @@ struct TeamCollectionInterface {
 	PromiseStream<GetTeamRequest> getTeam;
 };
 
+struct DDBulkLoadTaskBusyMap {
path: fdbserver/include/fdbserver/DataDistribution.actor.h
user: login: kakaiu
body: Added. Thanks!
created_at: 2025-05-07T04:46:05Z
author_association: MEMBER
diff_hunk: @@ -579,7 +612,7 @@ inline bool bulkDumpIsEnabled(int bulkDumpModeValue) {
 	return bulkDumpModeValue == 1;
 }
 
-class BulkLoadTaskCollection : public ReferenceCounted<BulkLoadTaskCollection> {
path: fdbserver/include/fdbserver/DataDistribution.actor.h
user: login: kakaiu
body: I feel I should follow the DD convention. So, I revert it back.
created_at: 2025-05-07T04:47:00Z
author_association: MEMBER
diff_hunk: @@ -2088,6 +2088,13 @@ ACTOR Future<Void> dataDistributionRelocator(DDQueue* self,
 			healthyDestinations.addReadInFlightToTeam(+metrics.readLoadKSecond());
 
 			launchDest(rd, bestTeams, self->destBusymap);
+			if (doBulkLoading) {
path: fdbserver/DDRelocationQueue.actor.cpp
user: login: kakaiu
body: Added.
created_at: 2025-05-07T04:47:22Z
author_association: MEMBER
diff_hunk: @@ -579,7 +612,7 @@ inline bool bulkDumpIsEnabled(int bulkDumpModeValue) {
 	return bulkDumpModeValue == 1;
 }
 
-class BulkLoadTaskCollection : public ReferenceCounted<BulkLoadTaskCollection> {
path: fdbserver/include/fdbserver/DataDistribution.actor.h
user: login: saintstack
body: ok. Makes the PR smaller.
created_at: 2025-05-07T04:48:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -390,6 +390,7 @@ void ServerKnobs::initialize(Randomize randomize, ClientKnobs* clientKnobs, IsSi
 	init( BULKLOAD_ASYNC_READ_WRITE_BLOCK_SIZE,            1024*1024 ); if (isSimulated) BULKLOAD_ASYNC_READ_WRITE_BLOCK_SIZE = deterministicRandom()->randomInt(1024, 10240);
 	init( MANIFEST_COUNT_MAX_PER_BULKLOAD_TASK,                   10 ); if (isSimulated) MANIFEST_COUNT_MAX_PER_BULKLOAD_TASK = deterministicRandom()->randomInt(1, 11);
 	init( BULKLOAD_SIM_FAILURE_INJECTION,                      false ); if (isSimulated) BULKLOAD_SIM_FAILURE_INJECTION = true;
+	init( DD_BULKLOAD_POWER_OF_D_RATIO,                          2.0 );
path: fdbclient/ServerKnobs.cpp
user: login: saintstack
body: Is this a low number?  We select half of all valid teams?
created_at: 2025-05-07T10:17:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -20,6 +20,7 @@
 #ifndef FOUNDATIONDB_DDRELOCATIONQUEUE_H
 #define FOUNDATIONDB_DDRELOCATIONQUEUE_H
 
+#include <memory>
path: fdbserver/include/fdbserver/DDRelocationQueue.h
user: login: saintstack
body: This include needed?
created_at: 2025-05-07T10:19:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -390,6 +390,7 @@ void ServerKnobs::initialize(Randomize randomize, ClientKnobs* clientKnobs, IsSi
 	init( BULKLOAD_ASYNC_READ_WRITE_BLOCK_SIZE,            1024*1024 ); if (isSimulated) BULKLOAD_ASYNC_READ_WRITE_BLOCK_SIZE = deterministicRandom()->randomInt(1024, 10240);
 	init( MANIFEST_COUNT_MAX_PER_BULKLOAD_TASK,                   10 ); if (isSimulated) MANIFEST_COUNT_MAX_PER_BULKLOAD_TASK = deterministicRandom()->randomInt(1, 11);
 	init( BULKLOAD_SIM_FAILURE_INJECTION,                      false ); if (isSimulated) BULKLOAD_SIM_FAILURE_INJECTION = true;
+	init( DD_BULKLOAD_POWER_OF_D_RATIO,                          2.0 );
path: fdbclient/ServerKnobs.cpp
user: login: kakaiu
body: This should not be low. We want to minimize the queuing time of the task by each decision.
created_at: 2025-05-07T16:29:37Z
author_association: MEMBER
diff_hunk: @@ -20,6 +20,7 @@
 #ifndef FOUNDATIONDB_DDRELOCATIONQUEUE_H
 #define FOUNDATIONDB_DDRELOCATIONQUEUE_H
 
+#include <memory>
path: fdbserver/include/fdbserver/DDRelocationQueue.h
user: login: kakaiu
body: Removed.
created_at: 2025-05-07T16:29:59Z
author_association: MEMBER
diff_hunk: @@ -18,6 +18,8 @@
  * limitations under the License.
  */
 
+#include "flow/IRandom.h"
+#include <unordered_map>
path: fdbserver/include/fdbserver/DataDistribution.actor.h
user: login: jzhou77
body: Move to the include section.
created_at: 2025-05-08T03:44:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -539,6 +541,40 @@ struct TeamCollectionInterface {
 	PromiseStream<GetTeamRequest> getTeam;
 };
 
+// Used to track the number of ongoing bulkload tasks for each storage server
+struct DDBulkLoadTaskBusyMap {
+public:
+	void addTask(const UID& ssid) {
+		auto it = busyMap.find(ssid);
+		if (it == busyMap.end()) {
+			busyMap[ssid] = 1;
+		} else {
+			it->second++;
+		}
path: fdbserver/include/fdbserver/DataDistribution.actor.h
user: login: jzhou77
body: Nit: can be simplified as `busyMap[ssid]++;`, avoiding an extra lookup if not present. If not present, the entry will be created with `0`.
created_at: 2025-05-08T03:47:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2315,6 +2324,11 @@ ACTOR Future<Void> dataDistributionRelocator(DDQueue* self,
 							    .detail("JobID", rd.bulkLoadTask.get().coreState.getJobId())
 							    .detail("TaskID", rd.bulkLoadTask.get().coreState.getTaskId());
 						}
+						for (const auto& [team, _] : bestTeams) {
+							for (const UID& ssid : team->getServerIDs()) {
+								self->bulkLoadTaskCollection->busyMap.removeTask(ssid);
path: fdbserver/DDRelocationQueue.actor.cpp
user: login: jzhou77
body: do we need to do the same in the above `throw bulkLoadError;`?
created_at: 2025-05-08T03:51:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2315,6 +2324,11 @@ ACTOR Future<Void> dataDistributionRelocator(DDQueue* self,
 							    .detail("JobID", rd.bulkLoadTask.get().coreState.getJobId())
 							    .detail("TaskID", rd.bulkLoadTask.get().coreState.getTaskId());
 						}
+						for (const auto& [team, _] : bestTeams) {
+							for (const UID& ssid : team->getServerIDs()) {
+								self->bulkLoadTaskCollection->busyMap.removeTask(ssid);
path: fdbserver/DDRelocationQueue.actor.cpp
user: login: kakaiu
body: Nice catch!
created_at: 2025-05-08T05:20:18Z
author_association: MEMBER

PR comments:
user: login: foundationdb-ci
created_at: 2025-05-07T03:46:52Z
body: 
### Result of foundationdb-pr-clang-ide on Linux RHEL 9

* Commit ID: eb08917c9157163210f0271219339d202249998c
* Duration 0:24:59
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-ide/c5f8f6ad-863a-4405-bc58-67528ffed0a2.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-ide/c5f8f6ad-863a-4405-bc58-67528ffed0a2/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T04:02:28Z
body: 
### Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

* Commit ID: eb08917c9157163210f0271219339d202249998c
* Duration 0:40:33
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-macos-m1/6b72adeb-b363-46f2-bce2-2386d2711ff3.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-macos-m1/6b72adeb-b363-46f2-bce2-2386d2711ff3/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T04:12:04Z
body: 
### Result of foundationdb-pr-clang-arm on Linux CentOS 7

* Commit ID: eb08917c9157163210f0271219339d202249998c
* Duration 0:50:09
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-arm/e4f3de22-5414-45a3-bb0b-5c170ded8068.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-arm/e4f3de22-5414-45a3-bb0b-5c170ded8068/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T04:20:24Z
body: 
### Result of foundationdb-pr-macos on macOS Ventura 13.x

* Commit ID: eb08917c9157163210f0271219339d202249998c
* Duration 0:58:27
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-macos/df0bfa07-c5c2-4272-83e2-b7bd1e966027.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-macos/df0bfa07-c5c2-4272-83e2-b7bd1e966027/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T04:20:42Z
body: 
### Result of foundationdb-pr-cluster-tests on Linux RHEL 9

* Commit ID: eb08917c9157163210f0271219339d202249998c
* Duration 0:58:48
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-cluster-tests/28661c52-a77f-4167-ac01-d545012f4d21.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/28661c52-a77f-4167-ac01-d545012f4d21/workspace.zip) zip file of the working directory (available for 30 days)
* [Cluster Test Logs](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/28661c52-a77f-4167-ac01-d545012f4d21/cluster_test_logs.zip) zip file of the test logs (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T04:27:52Z
body: 
### Result of foundationdb-pr-clang on Linux RHEL 9

* Commit ID: eb08917c9157163210f0271219339d202249998c
* Duration 1:05:58
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang/4c64696d-6c31-4966-a665-58934141f08d.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang/4c64696d-6c31-4966-a665-58934141f08d/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T04:31:47Z
body: 
### Result of foundationdb-pr on Linux RHEL 9

* Commit ID: eb08917c9157163210f0271219339d202249998c
* Duration 1:09:47
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr/c265cf6c-fac6-4931-809e-252de59a8f19.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr/c265cf6c-fac6-4931-809e-252de59a8f19/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T04:50:24Z
body: 
### Result of foundationdb-pr-clang-ide on Linux RHEL 9

* Commit ID: bd3a72581e14989a8d7750021c05f348fb0bd448
* Duration 0:24:41
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-ide/2fd7cc2d-cbb4-44d3-b692-bc1da28dfdb6.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-ide/2fd7cc2d-cbb4-44d3-b692-bc1da28dfdb6/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:02:54Z
body: 
### Result of foundationdb-pr-clang on Linux RHEL 9

* Commit ID: bd3a72581e14989a8d7750021c05f348fb0bd448
* Duration 0:37:11
* Result: :x: **FAILED**
* Error: `Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang/1a8e9277-a691-4f83-9738-70594ace0eba.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang/1a8e9277-a691-4f83-9738-70594ace0eba/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:03:50Z
body: 
### Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

* Commit ID: bd3a72581e14989a8d7750021c05f348fb0bd448
* Duration 0:38:09
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-macos-m1/4ecf3af1-dd97-46ee-a915-5170346e7eed.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-macos-m1/4ecf3af1-dd97-46ee-a915-5170346e7eed/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:11:33Z
body: 
### Result of foundationdb-pr-clang-ide on Linux RHEL 9

* Commit ID: f1972dc7c47f323fa8fbb9dc8b84228bc153235c
* Duration 0:25:24
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-ide/5c5467ee-be8f-4b44-8f32-cdc0e1e6a9d0.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-ide/5c5467ee-be8f-4b44-8f32-cdc0e1e6a9d0/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:15:04Z
body: 
### Result of foundationdb-pr-clang-arm on Linux CentOS 7

* Commit ID: bd3a72581e14989a8d7750021c05f348fb0bd448
* Duration 0:49:21
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-arm/ee526a0a-e368-4829-9bca-848bc87e5a47.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-arm/ee526a0a-e368-4829-9bca-848bc87e5a47/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:24:23Z
body: 
### Result of foundationdb-pr-cluster-tests on Linux RHEL 9

* Commit ID: bd3a72581e14989a8d7750021c05f348fb0bd448
* Duration 0:58:41
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-cluster-tests/f4e2bbc3-077c-4207-b9a3-900c151cb74e.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/f4e2bbc3-077c-4207-b9a3-900c151cb74e/workspace.zip) zip file of the working directory (available for 30 days)
* [Cluster Test Logs](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/f4e2bbc3-077c-4207-b9a3-900c151cb74e/cluster_test_logs.zip) zip file of the test logs (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:24:44Z
body: 
### Result of foundationdb-pr-macos on macOS Ventura 13.x

* Commit ID: bd3a72581e14989a8d7750021c05f348fb0bd448
* Duration 0:59:03
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-macos/b6430fee-4d7d-4db1-9881-bd315fe7f3b0.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-macos/b6430fee-4d7d-4db1-9881-bd315fe7f3b0/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:26:44Z
body: 
### Result of foundationdb-pr-clang-ide on Linux RHEL 9

* Commit ID: 6103b726e57a6762f225a6767a0b145a1339cde1
* Duration 0:27:05
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-ide/891aefef-4a67-4574-9370-0d559f29859e.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-ide/891aefef-4a67-4574-9370-0d559f29859e/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:35:03Z
body: 
### Result of foundationdb-pr-clang-arm on Linux CentOS 7

* Commit ID: f1972dc7c47f323fa8fbb9dc8b84228bc153235c
* Duration 0:48:56
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-arm/46d7b531-efea-4136-99e1-a942c090ee02.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-arm/46d7b531-efea-4136-99e1-a942c090ee02/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:35:59Z
body: 
### Result of foundationdb-pr on Linux RHEL 9

* Commit ID: bd3a72581e14989a8d7750021c05f348fb0bd448
* Duration 1:10:15
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr/14c129f4-c2aa-48ce-aec7-63d2c1ebf9e0.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr/14c129f4-c2aa-48ce-aec7-63d2c1ebf9e0/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:44:44Z
body: 
### Result of foundationdb-pr-clang on Linux RHEL 9

* Commit ID: f1972dc7c47f323fa8fbb9dc8b84228bc153235c
* Duration 0:58:31
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang/70c3170d-c697-43c8-909f-09281637f541.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang/70c3170d-c697-43c8-909f-09281637f541/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:47:00Z
body: 
### Result of foundationdb-pr on Linux RHEL 9

* Commit ID: f1972dc7c47f323fa8fbb9dc8b84228bc153235c
* Duration 1:00:48
* Result: :x: **FAILED**
* Error: `Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr/0d0c7f6e-1b35-4495-a9dd-a0b5e2824e18.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr/0d0c7f6e-1b35-4495-a9dd-a0b5e2824e18/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:47:41Z
body: 
### Result of foundationdb-pr-clang-arm on Linux CentOS 7

* Commit ID: 6103b726e57a6762f225a6767a0b145a1339cde1
* Duration 0:48:01
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-arm/57466e3e-1b6f-4e62-9d7a-38bd1914bed9.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-arm/57466e3e-1b6f-4e62-9d7a-38bd1914bed9/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:48:09Z
body: 
### Result of foundationdb-pr-cluster-tests on Linux RHEL 9

* Commit ID: f1972dc7c47f323fa8fbb9dc8b84228bc153235c
* Duration 1:01:58
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-cluster-tests/7780a359-7d97-42c0-bedf-2475aaa0e9ec.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/7780a359-7d97-42c0-bedf-2475aaa0e9ec/workspace.zip) zip file of the working directory (available for 30 days)
* [Cluster Test Logs](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/7780a359-7d97-42c0-bedf-2475aaa0e9ec/cluster_test_logs.zip) zip file of the test logs (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:50:52Z
body: 
### Result of foundationdb-pr-clang on Linux RHEL 9

* Commit ID: 6103b726e57a6762f225a6767a0b145a1339cde1
* Duration 0:51:12
* Result: :x: **FAILED**
* Error: `Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang/89a4cf8c-6417-4728-8a3a-06ab0bb58dfe.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang/89a4cf8c-6417-4728-8a3a-06ab0bb58dfe/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:55:45Z
body: 
### Result of foundationdb-pr-clang-ide on Linux RHEL 9

* Commit ID: db9bcceef224fffa9c8770f767851ddd1ffbaddb
* Duration 0:26:07
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-ide/71c61f0c-1ad4-44d1-bd74-7b0437a3fa7a.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-ide/71c61f0c-1ad4-44d1-bd74-7b0437a3fa7a/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:56:30Z
body: 
### Result of foundationdb-pr on Linux RHEL 9

* Commit ID: 6103b726e57a6762f225a6767a0b145a1339cde1
* Duration 0:56:51
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr/3191d3a3-8aee-4305-b845-0134aaf9ffb8.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr/3191d3a3-8aee-4305-b845-0134aaf9ffb8/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T05:59:05Z
body: 
### Result of foundationdb-pr-cluster-tests on Linux RHEL 9

* Commit ID: 6103b726e57a6762f225a6767a0b145a1339cde1
* Duration 0:59:22
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-cluster-tests/73c0fbdc-6732-46aa-8de8-2c5e42fbebd5.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/73c0fbdc-6732-46aa-8de8-2c5e42fbebd5/workspace.zip) zip file of the working directory (available for 30 days)
* [Cluster Test Logs](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/73c0fbdc-6732-46aa-8de8-2c5e42fbebd5/cluster_test_logs.zip) zip file of the test logs (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T06:07:29Z
body: 
### Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

* Commit ID: db9bcceef224fffa9c8770f767851ddd1ffbaddb
* Duration 0:37:48
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-macos-m1/dc462c5f-7da6-48e4-8d36-a2ec58a791c9.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-macos-m1/dc462c5f-7da6-48e4-8d36-a2ec58a791c9/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T06:13:43Z
body: 
### Result of foundationdb-pr-clang-ide on Linux RHEL 9

* Commit ID: 36843709aee85f0d596cfac4505b1e53b81f7f57
* Duration 0:26:01
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-ide/1c80b90d-401c-4293-baa6-7252e43ec4c7.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-ide/1c80b90d-401c-4293-baa6-7252e43ec4c7/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T06:18:20Z
body: 
### Result of foundationdb-pr-clang-arm on Linux CentOS 7

* Commit ID: db9bcceef224fffa9c8770f767851ddd1ffbaddb
* Duration 0:48:44
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-arm/e8fd7ed5-5d91-42f4-b83b-3b2d21a66c03.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-arm/e8fd7ed5-5d91-42f4-b83b-3b2d21a66c03/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T06:30:26Z
body: 
### Result of foundationdb-pr-macos on macOS Ventura 13.x

* Commit ID: db9bcceef224fffa9c8770f767851ddd1ffbaddb
* Duration 1:00:48
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-macos/a32e0a55-2cfb-43d9-a3d4-78ccee30c82a.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-macos/a32e0a55-2cfb-43d9-a3d4-78ccee30c82a/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T06:32:09Z
body: 
### Result of foundationdb-pr-clang on Linux RHEL 9

* Commit ID: db9bcceef224fffa9c8770f767851ddd1ffbaddb
* Duration 1:02:30
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang/692da3d1-5507-418c-9971-c540a7b621aa.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang/692da3d1-5507-418c-9971-c540a7b621aa/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T06:32:19Z
body: 
### Result of foundationdb-pr-cluster-tests on Linux RHEL 9

* Commit ID: db9bcceef224fffa9c8770f767851ddd1ffbaddb
* Duration 1:02:41
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-cluster-tests/3f690bcd-a802-479d-acc3-7701a55ea59b.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/3f690bcd-a802-479d-acc3-7701a55ea59b/workspace.zip) zip file of the working directory (available for 30 days)
* [Cluster Test Logs](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/3f690bcd-a802-479d-acc3-7701a55ea59b/cluster_test_logs.zip) zip file of the test logs (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T06:36:15Z
body: 
### Result of foundationdb-pr-clang-arm on Linux CentOS 7

* Commit ID: 36843709aee85f0d596cfac4505b1e53b81f7f57
* Duration 0:48:35
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-arm/ca1870a4-9237-4107-9e8b-1d175392d1db.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-arm/ca1870a4-9237-4107-9e8b-1d175392d1db/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T06:36:59Z
body: 
### Result of foundationdb-pr on Linux RHEL 9

* Commit ID: db9bcceef224fffa9c8770f767851ddd1ffbaddb
* Duration 1:07:19
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr/93f9ca22-c642-493d-b121-d50a0c56ab44.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr/93f9ca22-c642-493d-b121-d50a0c56ab44/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T06:40:33Z
body: 
### Result of foundationdb-pr-clang on Linux RHEL 9

* Commit ID: 36843709aee85f0d596cfac4505b1e53b81f7f57
* Duration 0:52:52
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang/e0ad93b3-5e67-4eb2-b5e3-537b88d08091.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang/e0ad93b3-5e67-4eb2-b5e3-537b88d08091/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T06:44:35Z
body: 
### Result of foundationdb-pr on Linux RHEL 9

* Commit ID: 36843709aee85f0d596cfac4505b1e53b81f7f57
* Duration 0:56:54
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr/96ee2047-ce05-483e-8253-c49a06a87d84.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr/96ee2047-ce05-483e-8253-c49a06a87d84/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T06:44:59Z
body: 
### Result of foundationdb-pr-cluster-tests on Linux RHEL 9

* Commit ID: 36843709aee85f0d596cfac4505b1e53b81f7f57
* Duration 0:57:19
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-cluster-tests/826d511e-a911-4176-bd7b-fb8252a3dde6.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/826d511e-a911-4176-bd7b-fb8252a3dde6/workspace.zip) zip file of the working directory (available for 30 days)
* [Cluster Test Logs](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/826d511e-a911-4176-bd7b-fb8252a3dde6/cluster_test_logs.zip) zip file of the test logs (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T16:55:46Z
body: 
### Result of foundationdb-pr-clang-ide on Linux RHEL 9

* Commit ID: 6092a9fbae2a0235d97bcb81fa9f8420ad80e6d2
* Duration 0:25:08
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-ide/8bbef712-b432-4deb-8d81-7783a231fa05.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-ide/8bbef712-b432-4deb-8d81-7783a231fa05/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T17:08:52Z
body: 
### Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

* Commit ID: 6092a9fbae2a0235d97bcb81fa9f8420ad80e6d2
* Duration 0:38:15
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-macos-m1/d83b5964-b5a1-45fb-aaaa-1ecafd3c4933.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-macos-m1/d83b5964-b5a1-45fb-aaaa-1ecafd3c4933/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T17:19:50Z
body: 
### Result of foundationdb-pr-clang-arm on Linux CentOS 7

* Commit ID: 6092a9fbae2a0235d97bcb81fa9f8420ad80e6d2
* Duration 0:49:12
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-arm/2cfc38e3-d7e9-4f01-af54-24cbd5aa6f19.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-arm/2cfc38e3-d7e9-4f01-af54-24cbd5aa6f19/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T17:29:37Z
body: 
### Result of foundationdb-pr-macos on macOS Ventura 13.x

* Commit ID: 6092a9fbae2a0235d97bcb81fa9f8420ad80e6d2
* Duration 0:58:59
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-macos/a5384f01-5b7a-4ac0-8e89-8b8c28e73f22.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-macos/a5384f01-5b7a-4ac0-8e89-8b8c28e73f22/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T17:32:14Z
body: 
### Result of foundationdb-pr-cluster-tests on Linux RHEL 9

* Commit ID: 6092a9fbae2a0235d97bcb81fa9f8420ad80e6d2
* Duration 1:01:34
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-cluster-tests/8671b0a1-daf3-43e1-a7c4-e406c5aaf5be.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/8671b0a1-daf3-43e1-a7c4-e406c5aaf5be/workspace.zip) zip file of the working directory (available for 30 days)
* [Cluster Test Logs](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/8671b0a1-daf3-43e1-a7c4-e406c5aaf5be/cluster_test_logs.zip) zip file of the test logs (available for 30 days)

author_association: CONTRIBUTOR
user: login: kakaiu
created_at: 2025-05-07T17:34:32Z
body: > LG. A few comments. I think you meant 12TB in your general comment on this PR (you had 1.2TB). In your test run, could you see the difference this change made? Any other way of testing/seeing the improvement this change adds? Thanks.

I runs K8s test using 145 SSes with single DC and EBS and SQLite engine to load 12TB data (10B records).

With the change: 2.5 hours
Job information: [BulkLoadJobState]: [JobId]: 996978645a55f766aab287c9fb7670a9, [JobRoot]: xxx, [JobRange]: { begin=  end=\xff }, [Phase]: Complete, [TransportMethod]: BlobStore, [SubmitTime]: 1746639844.775547, [SinceSubmitMins]: 147.972741, [TaskCount]: 287955

Without the change: 3.5 hours

author_association: MEMBER
user: login: foundationdb-ci
created_at: 2025-05-07T18:04:57Z
body: 
### Result of foundationdb-pr-clang on Linux RHEL 9

* Commit ID: 6092a9fbae2a0235d97bcb81fa9f8420ad80e6d2
* Duration 1:34:21
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang/78fb47d7-5eb6-4179-be79-bf2529541a98.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang/78fb47d7-5eb6-4179-be79-bf2529541a98/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T18:07:12Z
body: 
### Result of foundationdb-pr on Linux RHEL 9

* Commit ID: 6092a9fbae2a0235d97bcb81fa9f8420ad80e6d2
* Duration 1:36:32
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr/91fe3b2b-7932-4373-8fa6-ccef28b88758.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr/91fe3b2b-7932-4373-8fa6-ccef28b88758/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: saintstack
created_at: 2025-05-07T18:33:27Z
body: > I runs K8s test using 145 SSes with single DC and EBS and SQLite engine to load 12TB data (10B records).
> 
> With the change: 2.5 hours.
> 
> Without the change: 3.5 hours.
> 
> 

Whatever you pasted didn't come across... Thanks. +1'd the PR.
author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T19:16:12Z
body: 
### Result of foundationdb-pr-clang-ide on Linux RHEL 9

* Commit ID: 9928c335644cf11275721329ace4a41ac87b9691
* Duration 0:25:06
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-ide/89c9bfb6-1b56-4077-b849-dbba89e4e8a4.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-ide/89c9bfb6-1b56-4077-b849-dbba89e4e8a4/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T19:29:26Z
body: 
### Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

* Commit ID: 9928c335644cf11275721329ace4a41ac87b9691
* Duration 0:38:20
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-macos-m1/057755d0-751d-4fe5-a4a6-59689b81a6bf.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-macos-m1/057755d0-751d-4fe5-a4a6-59689b81a6bf/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T19:38:45Z
body: 
### Result of foundationdb-pr-clang-arm on Linux CentOS 7

* Commit ID: 9928c335644cf11275721329ace4a41ac87b9691
* Duration 0:47:40
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-arm/55e2ffd4-3597-4b18-9eb0-7113cafcb592.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-arm/55e2ffd4-3597-4b18-9eb0-7113cafcb592/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T19:45:37Z
body: 
### Result of foundationdb-pr-clang on Linux RHEL 9

* Commit ID: 9928c335644cf11275721329ace4a41ac87b9691
* Duration 0:54:31
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang/c705ed0f-d057-4ccd-b16c-be17681a5eff.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang/c705ed0f-d057-4ccd-b16c-be17681a5eff/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T19:49:24Z
body: 
### Result of foundationdb-pr-macos on macOS Ventura 13.x

* Commit ID: 9928c335644cf11275721329ace4a41ac87b9691
* Duration 0:58:19
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-macos/3a65f51f-34a6-46c8-8afb-184197ad256a.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-macos/3a65f51f-34a6-46c8-8afb-184197ad256a/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T19:50:56Z
body: 
### Result of foundationdb-pr-cluster-tests on Linux RHEL 9

* Commit ID: 9928c335644cf11275721329ace4a41ac87b9691
* Duration 0:59:50
* Result: :x: **FAILED**
* Error: `Error while executing command: TEST_USERNAME=fdb-pr-${CODEBUILD_BUILD_NUMBER} make -kj -C tests foundationdb-pr-tests. Reason: exit status 2`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-cluster-tests/bd1e7891-7c27-4377-8633-6201e5dd6147.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/bd1e7891-7c27-4377-8633-6201e5dd6147/workspace.zip) zip file of the working directory (available for 30 days)
* [Cluster Test Logs](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/bd1e7891-7c27-4377-8633-6201e5dd6147/cluster_test_logs.zip) zip file of the test logs (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-07T19:58:04Z
body: 
### Result of foundationdb-pr on Linux RHEL 9

* Commit ID: 9928c335644cf11275721329ace4a41ac87b9691
* Duration 1:06:56
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr/56fbd88c-171e-44fa-ab4d-2a1e2f59a731.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr/56fbd88c-171e-44fa-ab4d-2a1e2f59a731/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T05:42:47Z
body: 
### Result of foundationdb-pr-clang-ide on Linux RHEL 9

* Commit ID: 8810418413c6144969e86cf4ddd0296cf55d5b87
* Duration 0:33:27
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-ide/e3656bbe-d24c-4ecd-910a-5f04103cd1ba.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-ide/e3656bbe-d24c-4ecd-910a-5f04103cd1ba/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T05:47:15Z
body: 
### Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

* Commit ID: 8810418413c6144969e86cf4ddd0296cf55d5b87
* Duration 0:37:58
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-macos-m1/ffafbe60-a7cc-43a5-ad91-0c927e59f323.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-macos-m1/ffafbe60-a7cc-43a5-ad91-0c927e59f323/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T05:58:50Z
body: 
### Result of foundationdb-pr-clang-ide on Linux RHEL 9

* Commit ID: c976f0c28ebe54aee12ef6d00cc456ff062f6bc4
* Duration 0:37:40
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-ide/74b1f254-9b4e-4d1b-be2f-b1175da594b0.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-ide/74b1f254-9b4e-4d1b-be2f-b1175da594b0/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T05:58:56Z
body: 
### Result of foundationdb-pr-clang-arm on Linux CentOS 7

* Commit ID: 8810418413c6144969e86cf4ddd0296cf55d5b87
* Duration 0:49:37
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-arm/fee14a12-c021-46d0-a7ce-07618a07d405.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-arm/fee14a12-c021-46d0-a7ce-07618a07d405/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T06:08:11Z
body: 
### Result of foundationdb-pr-macos on macOS Ventura 13.x

* Commit ID: 8810418413c6144969e86cf4ddd0296cf55d5b87
* Duration 0:58:52
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-macos/f3852c46-e9d7-4f5c-9469-f2169f169047.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-macos/f3852c46-e9d7-4f5c-9469-f2169f169047/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T06:10:11Z
body: 
### Result of foundationdb-pr-clang-arm on Linux CentOS 7

* Commit ID: c976f0c28ebe54aee12ef6d00cc456ff062f6bc4
* Duration 0:48:59
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-arm/62a54385-88c7-4f26-8fab-44d8aba62b79.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-arm/62a54385-88c7-4f26-8fab-44d8aba62b79/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T06:20:40Z
body: 
### Result of foundationdb-pr-cluster-tests on Linux RHEL 9

* Commit ID: 8810418413c6144969e86cf4ddd0296cf55d5b87
* Duration 1:11:25
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-cluster-tests/3f139d1f-ad09-4e15-8f8d-49470e11a214.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/3f139d1f-ad09-4e15-8f8d-49470e11a214/workspace.zip) zip file of the working directory (available for 30 days)
* [Cluster Test Logs](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/3f139d1f-ad09-4e15-8f8d-49470e11a214/cluster_test_logs.zip) zip file of the test logs (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T06:27:52Z
body: 
### Result of foundationdb-pr-cluster-tests on Linux RHEL 9

* Commit ID: c976f0c28ebe54aee12ef6d00cc456ff062f6bc4
* Duration 1:06:42
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-cluster-tests/1d5877b7-39f9-4257-ae34-033c8f70345b.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/1d5877b7-39f9-4257-ae34-033c8f70345b/workspace.zip) zip file of the working directory (available for 30 days)
* [Cluster Test Logs](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/1d5877b7-39f9-4257-ae34-033c8f70345b/cluster_test_logs.zip) zip file of the test logs (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T06:40:40Z
body: 
### Result of foundationdb-pr on Linux RHEL 9

* Commit ID: 8810418413c6144969e86cf4ddd0296cf55d5b87
* Duration 1:31:20
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr/80075ead-aa72-4988-8e0e-1a0ca75e3c40.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr/80075ead-aa72-4988-8e0e-1a0ca75e3c40/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T06:41:16Z
body: 
### Result of foundationdb-pr-clang on Linux RHEL 9

* Commit ID: c976f0c28ebe54aee12ef6d00cc456ff062f6bc4
* Duration 1:20:06
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang/e6898d77-9101-4937-a5fb-253476d541a7.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang/e6898d77-9101-4937-a5fb-253476d541a7/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T06:43:01Z
body: 
### Result of foundationdb-pr-clang on Linux RHEL 9

* Commit ID: 8810418413c6144969e86cf4ddd0296cf55d5b87
* Duration 1:33:43
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang/6914e3e5-b393-4faa-bfe4-ddac5ed02930.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang/6914e3e5-b393-4faa-bfe4-ddac5ed02930/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T06:44:34Z
body: 
### Result of foundationdb-pr on Linux RHEL 9

* Commit ID: c976f0c28ebe54aee12ef6d00cc456ff062f6bc4
* Duration 1:22:41
* Result: :x: **FAILED**
* Error: `Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr/c3de6d2e-8831-42b0-8ae8-7ebebb853da8.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr/c3de6d2e-8831-42b0-8ae8-7ebebb853da8/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T17:29:28Z
body: 
### Result of foundationdb-pr-clang-ide on Linux RHEL 9

* Commit ID: c976f0c28ebe54aee12ef6d00cc456ff062f6bc4
* Duration 0:26:20
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-ide/deef91e3-b3b7-4a5b-b33c-695272b1e689.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-ide/deef91e3-b3b7-4a5b-b33c-695272b1e689/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T17:41:53Z
body: 
### Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

* Commit ID: c976f0c28ebe54aee12ef6d00cc456ff062f6bc4
* Duration 0:38:48
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-macos-m1/ef256340-53d1-47fd-8c0a-65900e9f9acd.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-macos-m1/ef256340-53d1-47fd-8c0a-65900e9f9acd/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T17:52:29Z
body: 
### Result of foundationdb-pr-clang-arm on Linux CentOS 7

* Commit ID: c976f0c28ebe54aee12ef6d00cc456ff062f6bc4
* Duration 0:49:21
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang-arm/495067ca-4066-439a-9c1c-158940951c96.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang-arm/495067ca-4066-439a-9c1c-158940951c96/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T18:02:31Z
body: 
### Result of foundationdb-pr-macos on macOS Ventura 13.x

* Commit ID: c976f0c28ebe54aee12ef6d00cc456ff062f6bc4
* Duration 0:59:25
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-macos/1e742431-0efe-45e1-b4bb-914ebf3b13b7.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-macos/1e742431-0efe-45e1-b4bb-914ebf3b13b7/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T18:05:38Z
body: 
### Result of foundationdb-pr-cluster-tests on Linux RHEL 9

* Commit ID: c976f0c28ebe54aee12ef6d00cc456ff062f6bc4
* Duration 1:02:27
* Result: :x: **FAILED**
* Error: `Error while executing command: TEST_USERNAME=fdb-pr-${CODEBUILD_BUILD_NUMBER} make -kj -C tests foundationdb-pr-tests. Reason: exit status 2`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-cluster-tests/de27149f-6ff8-40d6-940d-92cafc1528ba.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/de27149f-6ff8-40d6-940d-92cafc1528ba/workspace.zip) zip file of the working directory (available for 30 days)
* [Cluster Test Logs](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-cluster-tests/de27149f-6ff8-40d6-940d-92cafc1528ba/cluster_test_logs.zip) zip file of the test logs (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T18:25:16Z
body: 
### Result of foundationdb-pr on Linux RHEL 9

* Commit ID: c976f0c28ebe54aee12ef6d00cc456ff062f6bc4
* Duration 1:22:09
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr/2d59fd8f-73f6-4b7c-8066-74d39558a016.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr/2d59fd8f-73f6-4b7c-8066-74d39558a016/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-05-08T18:26:57Z
body: 
### Result of foundationdb-pr-clang on Linux RHEL 9

* Commit ID: c976f0c28ebe54aee12ef6d00cc456ff062f6bc4
* Duration 1:23:46
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb/foundationdb-pr-clang/682d3954-24bd-4266-ab5c-8ab98372da3c.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/foundationdb-pr-clang/682d3954-24bd-4266-ab5c-8ab98372da3c/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR