Repository: hiero-ledger/hiero-json-rpc-relay
Pull request #3777: fix: fixed flaky/failing acceptance tests
Author: login: quiet-node
Labels: name: internal
description: For changes that affect the project's internal workings but not its outward-facing functionality.
Created: 2025-05-20T23:42:39Z | Merged: 2025-05-23T18:06:48Z

**Description**:
 - fixed batch request WS release test due to mismatch request ID
 - fixed flaky `"eth_getBalance" with block number in the last 15 minutes for account that has performed contract deploys/calls` test in batch2 due to race condition during initialization

**Related issue(s)**:

Fixes #3665 
Fixes #3666 

**Notes for reviewer**:
<!-- Provide logs, performance numbers or screenshots of the new functionality -->

**Checklist**

- [ ] Documented (Code comments, README, etc.)
- [ ] Tested (unit, integration, etc.)


Linked issue #3665: Fix web server release test due to mismatch in `Request ID`
The following release test for Web Server consistently fails (you can run it using `acceptancetest:release_ws`)

```console
$ npm run acceptancetest:release_ws
[...]
 1) RPC Server Acceptance Tests
       Acceptance tests
         @web-socket-batch-request Batch Requests
           given WS_BATCH_REQUESTS_ENABLED=true are set
             @release Should submit batch requests to WS server using Standard Web Socket and retrieve batch responses:

      AssertionError: expected [ { result: '0x129', …(2) }, …(4) ] to deeply equal [ { result: '0x129', …(2) }, …(4) ]
      + expected - actual

         }
         {
           "error": {
             "code": -32602
      -      "message": "[Request ID: d79be947-5fcb-44e8-8ee9-5c04847cb26c] Invalid parameter 0: Expected 0x prefixed string representing the address (20 bytes), value: 0xhedera"
      +      "message": "[Request ID: 7c3b0dc6-f7b7-4219-9f64-672488ec10e8] Invalid parameter 0: Expected 0x prefixed string representing the address (20 bytes), value: 0xhedera"
           }
           "id": 1
           "jsonrpc": "2.0"
         }
      
      at Suite.<anonymous> (packages/ws-server/tests/acceptance/batchRequest.spec.ts:80:38)
      at Generator.next (<anonymous>)
      at fulfilled (packages/ws-server/tests/acceptance/batchRequest.spec.ts:6:58)
[...]
```

The problem seems to be just the comparison. Note that the messages only differ in the `Request ID` (which makes sense) given how the test work

https://github.com/hiero-ledger/hiero-json-rpc-relay/blob/69bf5102b3ebbbc841f4d417664a58938b3bc272/packages/ws-server/tests/acceptance/batchRequest.spec.ts#L69-L81

A possible solution would just to mask the `Request ID` when a given response is an error.

> [!NOTE]
> I believe this test is not being run on CI, that's why we didn't detect it earlier.

Diff:
diff --git a/packages/server/tests/acceptance/rpc_batch2.spec.ts b/packages/server/tests/acceptance/rpc_batch2.spec.ts
index 60c1f2f73b..c8b064cc18 100644
--- a/packages/server/tests/acceptance/rpc_batch2.spec.ts
+++ b/packages/server/tests/acceptance/rpc_batch2.spec.ts
@@ -72,7 +72,7 @@ describe('@api-batch-2 RPC Server Acceptance Tests', function () {
   const FEE_SCHEDULE_FILE_CONTENT_UPDATED =
     '0a280a0a08541a061a0440a8953a0a0a08061a061a0440889d2d0a0a08071a061a0440b0b63c120208011200'; // Eth gas = 953000
 
-  let blockNumberAtStartOfTests = 0;
+  let blockNumAfterCreateChildTx = 0;
 
   const signSendAndConfirmTransaction = async (transaction, accounts, requestId) => {
     const signedTx = await accounts.wallet.signTransaction(transaction);
@@ -118,17 +118,28 @@ describe('@api-batch-2 RPC Server Acceptance Tests', function () {
     });
     await relay.pollForValidTransactionReceipt(initialFundsTx.hash);
 
-    const [childTx, blockNumber, balance] = await Promise.all([
-      parentContract.createChild(1),
-      accounts[0].wallet.provider?.getBlockNumber(),
-      accounts[0].wallet.provider?.getBalance(accounts[0].address),
-    ]);
+    createChildTx = await parentContract.createChild(1);
+    const createChildTxReceipt = await relay.pollForValidTransactionReceipt(createChildTx.hash);
+
+    const blockNumBeforeCreateChildTx = parseInt(createChildTxReceipt.blockNumber, 16);
+    blockNumAfterCreateChildTx = await relay.call(RelayCalls.ETH_ENDPOINTS.ETH_BLOCK_NUMBER, [], requestId);
 
-    createChildTx = childTx;
-    await relay.pollForValidTransactionReceipt(createChildTx.hash);
+    // Note: There is currently a caching solution for eth_blockNumber that stores the block number.
+    // This loop is designed to poll for the latest block number until it is correctly updated.
+    for (let i = 0; i < 5; i++) {
+      blockNumAfterCreateChildTx = await relay.call(RelayCalls.ETH_ENDPOINTS.ETH_BLOCK_NUMBER, [], requestId);    
+      if (blockNumAfterCreateChildTx > blockNumBeforeCreateChildTx) {
+        console.log("Block number updated succesfully")
+        break;
+      }
+      await Utils.wait(1500);
+    }
 
-    blockNumberAtStartOfTests = blockNumber as number;
-    accounts0StartBalance = balance as bigint;
+    accounts0StartBalance = await relay.call(
+      RelayCalls.ETH_ENDPOINTS.ETH_GET_BALANCE,
+      [accounts[0].address, 'latest'],
+      requestId,
+    );
 
     htsAddress = Utils.idToEvmAddress(tokenId.toString());
     await Promise.all([
@@ -544,11 +555,10 @@ describe('@api-batch-2 RPC Server Acceptance Tests', function () {
     it('@release should execute "eth_getBalance" with block number in the last 15 minutes for account that has performed contract deploys/calls', async function () {
       const res = await relay.call(
         RelayCalls.ETH_ENDPOINTS.ETH_GET_BALANCE,
-        [accounts[0].address, numberTo0x(blockNumberAtStartOfTests)],
+        [accounts[0].address, blockNumAfterCreateChildTx],
         requestId,
       );
-      const balanceAtBlock = BigInt(accounts0StartBalance);
-      expect(res).to.eq(`0x${balanceAtBlock.toString(16)}`);
+      expect(res).to.eq(accounts0StartBalance);
     });
 
     it('@release should correctly execute "eth_getBalance" with block number in the last 15 minutes with several txs around that time', async function () {
diff --git a/packages/server/tests/localAcceptance.env b/packages/server/tests/localAcceptance.env
index c8569ca7c6..e8b7c27401 100644
--- a/packages/server/tests/localAcceptance.env
+++ b/packages/server/tests/localAcceptance.env
@@ -33,4 +33,5 @@ HBAR_RATE_LIMIT_BASIC=6000000000# 60 HBARs
 HBAR_SPENDING_PLANS_CONFIG_FILE=./packages/server/tests/testSpendingPlansConfig.json
 USE_ASYNC_TX_PROCESSING=false
 JUMBO_TX_ENABLED=false
+WS_BATCH_REQUESTS_ENABLED=true
 
diff --git a/packages/ws-server/tests/acceptance/batchRequest.spec.ts b/packages/ws-server/tests/acceptance/batchRequest.spec.ts
index 242ca2481c..b372397e92 100644
--- a/packages/ws-server/tests/acceptance/batchRequest.spec.ts
+++ b/packages/ws-server/tests/acceptance/batchRequest.spec.ts
@@ -77,7 +77,20 @@ describe('@web-socket-batch-request Batch Requests', async function () {
       });
       const individualResponses = await Promise.all(promises);
 
-      expect(batchResponses).to.deep.eq(individualResponses);
+      // Compare responses
+      expect(batchResponses.length).to.equal(individualResponses.length);
+
+      batchResponses.forEach((batch, i) => {
+        const ind = individualResponses[i];
+
+        // For error responses, ignore the request ID
+        if (batch.error?.message?.includes('Request ID:')) {
+          expect(batch.error.code).to.equal(ind.error.code);
+          expect(batch.error.message).to.include(batch.error.message.split('] ')[1]);
+        } else {
+          expect(batch).to.deep.equal(ind);
+        }
+      });
     });
 
     it('@release Should return errors for blacklisted methods', async function () {


Reviews:
user: login: simzzz
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T10:02:54Z
user: login: acuarica
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T16:36:58Z
user: login: quiet-node
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T18:06:24Z
user: login: acuarica
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T18:58:39Z
user: login: acuarica
body: In any case, going to approve it.
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T19:00:07Z
user: login: quiet-node
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T19:42:04Z
user: login: quiet-node
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T16:48:04Z
user: login: acuarica
body: nice stuff!
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T18:12:36Z
user: login: acuarica
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T18:13:09Z
user: login: quiet-node
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-23T15:00:15Z
user: login: quiet-node
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-23T15:05:58Z
user: login: acuarica
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-23T15:09:47Z
user: login: acuarica
body: Cool stuff!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-23T15:10:16Z
user: login: natanasow
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-23T15:41:20Z

Review comments:
diff_hunk: @@ -118,15 +118,17 @@ describe('@api-batch-2 RPC Server Acceptance Tests', function () {
     });
     await relay.pollForValidTransactionReceipt(initialFundsTx.hash);
 
-    const [childTx, blockNumber, balance] = await Promise.all([
-      parentContract.createChild(1),
+    createChildTx = await parentContract.createChild(1);
+    await relay.pollForValidTransactionReceipt(createChildTx.hash);
+
+    // added delay to wait for balances properly settled on MN
+    await Utils.wait(1500);
path: packages/server/tests/acceptance/rpc_batch2.spec.ts
user: login: acuarica
body: Isn't this a double wait? If `createChildTx` is finished, doesn't it mean that balances are already settled on the MN?
created_at: 2025-05-21T16:36:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -118,15 +118,17 @@ describe('@api-batch-2 RPC Server Acceptance Tests', function () {
     });
     await relay.pollForValidTransactionReceipt(initialFundsTx.hash);
 
-    const [childTx, blockNumber, balance] = await Promise.all([
-      parentContract.createChild(1),
+    createChildTx = await parentContract.createChild(1);
+    await relay.pollForValidTransactionReceipt(createChildTx.hash);
+
+    // added delay to wait for balances properly settled on MN
+    await Utils.wait(1500);
path: packages/server/tests/acceptance/rpc_batch2.spec.ts
user: login: quiet-node
body: Hey Luis, thanks for the question. Without the extra wait there still occasional balance mismatches on production networks. I believe that once a transaction reaches consensus, the balances are updated in the Merkle tree almost instantly however for the Mirror Node to reflect those updates, there’s still a slight delay. Without the wait it should work fine, but given the busy traffic on production networks, it seems the Mirror Node sometimes takes longer to catch up, leading the flakyness we're dealing with. That's why I added the extra wait to stablize the results. Not ideal but flexible for different production environemnt. 
created_at: 2025-05-21T18:06:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -118,15 +118,17 @@ describe('@api-batch-2 RPC Server Acceptance Tests', function () {
     });
     await relay.pollForValidTransactionReceipt(initialFundsTx.hash);
 
-    const [childTx, blockNumber, balance] = await Promise.all([
-      parentContract.createChild(1),
+    createChildTx = await parentContract.createChild(1);
+    await relay.pollForValidTransactionReceipt(createChildTx.hash);
+
+    // added delay to wait for balances properly settled on MN
+    await Utils.wait(1500);
path: packages/server/tests/acceptance/rpc_batch2.spec.ts
user: login: acuarica
body: Thanks for the clarification, really helpful. Maybe we should expand the comment to include a bit of this explanation?

Also, we don't remove the flakiness entirely, this is just a mitigation. What if we poll the MN to ensure the mentioned transaction is visible from it? This way, we are 100% sure to have solved the problem. I remember seeing this mechanism in tests in the Smart Contract repo.
created_at: 2025-05-21T18:58:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -118,15 +118,17 @@ describe('@api-batch-2 RPC Server Acceptance Tests', function () {
     });
     await relay.pollForValidTransactionReceipt(initialFundsTx.hash);
 
-    const [childTx, blockNumber, balance] = await Promise.all([
-      parentContract.createChild(1),
+    createChildTx = await parentContract.createChild(1);
+    await relay.pollForValidTransactionReceipt(createChildTx.hash);
+
+    // added delay to wait for balances properly settled on MN
+    await Utils.wait(1500);
path: packages/server/tests/acceptance/rpc_batch2.spec.ts
user: login: quiet-node
body: Yeah, you’re right—a comment would definitely help.

As for the extra wait doesn’t eliminate the flakiness entirely, You’re correct that it’s more of a mitigation to improve stability. If the production environment needs more than 1500 ms, the issue can still occur, though that’s rare—but still possible.

Regarding polling the Mirror Node to confirm the transaction is visible, that’s actually what’s happening under the hood with `await relay.pollForValidTransactionReceipt(initialFundsTx.hash)`, which essentially calls `eth_getTransactionReceipt` to fetch the receipt from the Mirror Node.

Maybe we should consider polling the Mirror Node for updated balances instead? Kinda feel like I have done this before but it was quite cumbersome somehow but let me see what I can do.

You're right we should aim for a more robust fix rather than just a mitigation. Will push in an update for this.
created_at: 2025-05-21T19:42:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -118,15 +118,17 @@ describe('@api-batch-2 RPC Server Acceptance Tests', function () {
     });
     await relay.pollForValidTransactionReceipt(initialFundsTx.hash);
 
-    const [childTx, blockNumber, balance] = await Promise.all([
-      parentContract.createChild(1),
+    createChildTx = await parentContract.createChild(1);
+    await relay.pollForValidTransactionReceipt(createChildTx.hash);
+
+    // added delay to wait for balances properly settled on MN
+    await Utils.wait(1500);
path: packages/server/tests/acceptance/rpc_batch2.spec.ts
user: login: quiet-node
body: Synced offline with @acuarica and discovered that the flakiness was caused by a cached value of the latest block number. The new solution implements a polling mechanism to ensure the block is correctly updated after contract execution, resulting in stable and consistent outcomes.
created_at: 2025-05-22T16:48:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -118,17 +118,27 @@ describe('@api-batch-2 RPC Server Acceptance Tests', function () {
     });
     await relay.pollForValidTransactionReceipt(initialFundsTx.hash);
 
-    const [childTx, blockNumber, balance] = await Promise.all([
-      parentContract.createChild(1),
-      accounts[0].wallet.provider?.getBlockNumber(),
-      accounts[0].wallet.provider?.getBalance(accounts[0].address),
-    ]);
+    createChildTx = await parentContract.createChild(1);
+    const createChildTxReceipt = await relay.pollForValidTransactionReceipt(createChildTx.hash);
+
+    const blockNumBeforeCreateChildTx = parseInt(createChildTxReceipt.blockNumber, 16);
+    blockNumAfterCreateChildTx = await relay.call(RelayCalls.ETH_ENDPOINTS.ETH_BLOCK_NUMBER, [], requestId);
 
-    createChildTx = childTx;
-    await relay.pollForValidTransactionReceipt(createChildTx.hash);
+    // Note: There is currently a caching solution for eth_blockNumber that stores the block number.
+    // This loop is designed to poll for the latest block number until it is correctly updated.
+    for (let i = 0; i < 5; i++) {
+      if (blockNumAfterCreateChildTx > blockNumBeforeCreateChildTx) {
+        break;
+      }
+      await Utils.wait(1500);
+      blockNumAfterCreateChildTx = await await relay.call(RelayCalls.ETH_ENDPOINTS.ETH_BLOCK_NUMBER, [], requestId);
+    }
path: packages/server/tests/acceptance/rpc_batch2.spec.ts
user: login: acuarica
body: couple of things: remove double `await`, include `console.log` have feedback when block number is properly updated, move the `blockNumAfterCreateChildTx` assignment to have it only once.

```suggestion
    for (let i = 0; i < 5; i++) {
      blockNumAfterCreateChildTx = await relay.call(RelayCalls.ETH_ENDPOINTS.ETH_BLOCK_NUMBER, [], requestId);    
      if (blockNumAfterCreateChildTx > blockNumBeforeCreateChildTx) {
        console.log("Block number updated succesfully")
        break;
      }
      await Utils.wait(1500);
    }
```
created_at: 2025-05-22T18:11:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -72,7 +72,7 @@ describe('@api-batch-2 RPC Server Acceptance Tests', function () {
   const FEE_SCHEDULE_FILE_CONTENT_UPDATED =
     '0a280a0a08541a061a0440a8953a0a0a08061a061a0440889d2d0a0a08071a061a0440b0b63c120208011200'; // Eth gas = 953000
 
-  let blockNumberAtStartOfTests = 0;
+  let blockNumAfterCreateChildTx = 0;
path: packages/server/tests/acceptance/rpc_batch2.spec.ts
user: login: acuarica
body: nit: move this declaration closer to the loop
created_at: 2025-05-22T18:12:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -118,15 +118,17 @@ describe('@api-batch-2 RPC Server Acceptance Tests', function () {
     });
     await relay.pollForValidTransactionReceipt(initialFundsTx.hash);
 
-    const [childTx, blockNumber, balance] = await Promise.all([
-      parentContract.createChild(1),
+    createChildTx = await parentContract.createChild(1);
+    await relay.pollForValidTransactionReceipt(createChildTx.hash);
+
+    // added delay to wait for balances properly settled on MN
+    await Utils.wait(1500);
path: packages/server/tests/acceptance/rpc_batch2.spec.ts
user: login: acuarica
body: Awesome!
created_at: 2025-05-22T18:13:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -118,17 +118,27 @@ describe('@api-batch-2 RPC Server Acceptance Tests', function () {
     });
     await relay.pollForValidTransactionReceipt(initialFundsTx.hash);
 
-    const [childTx, blockNumber, balance] = await Promise.all([
-      parentContract.createChild(1),
-      accounts[0].wallet.provider?.getBlockNumber(),
-      accounts[0].wallet.provider?.getBalance(accounts[0].address),
-    ]);
+    createChildTx = await parentContract.createChild(1);
+    const createChildTxReceipt = await relay.pollForValidTransactionReceipt(createChildTx.hash);
+
+    const blockNumBeforeCreateChildTx = parseInt(createChildTxReceipt.blockNumber, 16);
+    blockNumAfterCreateChildTx = await relay.call(RelayCalls.ETH_ENDPOINTS.ETH_BLOCK_NUMBER, [], requestId);
 
-    createChildTx = childTx;
-    await relay.pollForValidTransactionReceipt(createChildTx.hash);
+    // Note: There is currently a caching solution for eth_blockNumber that stores the block number.
+    // This loop is designed to poll for the latest block number until it is correctly updated.
+    for (let i = 0; i < 5; i++) {
+      if (blockNumAfterCreateChildTx > blockNumBeforeCreateChildTx) {
+        break;
+      }
+      await Utils.wait(1500);
+      blockNumAfterCreateChildTx = await await relay.call(RelayCalls.ETH_ENDPOINTS.ETH_BLOCK_NUMBER, [], requestId);
+    }
path: packages/server/tests/acceptance/rpc_batch2.spec.ts
user: login: quiet-node
body: ha funny - `await await`! Good catch thanks for pointing it out! Implemented your suggestion!
created_at: 2025-05-23T15:00:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -72,7 +72,7 @@ describe('@api-batch-2 RPC Server Acceptance Tests', function () {
   const FEE_SCHEDULE_FILE_CONTENT_UPDATED =
     '0a280a0a08541a061a0440a8953a0a0a08061a061a0440889d2d0a0a08071a061a0440b0b63c120208011200'; // Eth gas = 953000
 
-  let blockNumberAtStartOfTests = 0;
+  let blockNumAfterCreateChildTx = 0;
path: packages/server/tests/acceptance/rpc_batch2.spec.ts
user: login: quiet-node
body: This is actually the closest I could get lol it’s a global variable, and the polling happens within the beforeAll block, so I couldn’t really place it near the polling.
created_at: 2025-05-23T15:05:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -72,7 +72,7 @@ describe('@api-batch-2 RPC Server Acceptance Tests', function () {
   const FEE_SCHEDULE_FILE_CONTENT_UPDATED =
     '0a280a0a08541a061a0440a8953a0a0a08061a061a0440889d2d0a0a08071a061a0440b0b63c120208011200'; // Eth gas = 953000
 
-  let blockNumberAtStartOfTests = 0;
+  let blockNumAfterCreateChildTx = 0;
path: packages/server/tests/acceptance/rpc_batch2.spec.ts
user: login: acuarica
body: Ah, I see it now.
created_at: 2025-05-23T15:09:47Z
author_association: CONTRIBUTOR

PR comments:
user: login: lfdt-bot
created_at: 2025-05-20T23:42:51Z
body: ![](https://res.cloudinary.com/snyk/image/upload/r-d/scm-platform/snyk-pull-requests/pr-banner-default.svg)

### :tada: **Snyk checks have passed. No issues have been found so far.**

:white_check_mark: **security/snyk** check is complete. No issues have been found. [(View Details)](https://app.snyk.io/org/hiero-json-rpc-relay/pr-checks/9128f971-7c36-4241-9230-0fa7856cdbb4?refs=5680e9a28df8649d6c2eae5a50d55793c4f49545&source=prChecksComment)

:white_check_mark: **license/snyk** check is complete. No issues have been found. [(View Details)](https://app.snyk.io/org/hiero-json-rpc-relay/pr-checks/9128f971-7c36-4241-9230-0fa7856cdbb4/license?refs=5680e9a28df8649d6c2eae5a50d55793c4f49545&source=prChecksComment)



author_association: NONE
user: login: github-actions[bot]
created_at: 2025-05-20T23:48:22Z
body: ## Test Results
 22 files  +1  295 suites  +35   37m 36s :stopwatch: + 2m 55s
649 tests ±0  643 :white_check_mark: ± 0  5 :zzz: ±0  1 :x: ±0 
745 runs   - 5  738 :white_check_mark:  -  6  5 :zzz: ±0  2 :x: +1 

For more details on these failures, see [this check](https://github.com/hiero-ledger/hiero-json-rpc-relay/runs/42798091345).

Results for commit 5680e9a2. ± Comparison against base commit bcd473bb.

[test-results]:data:application/gzip;base64,H4sIAHSiMGgC/03Oyw6DIBCF4VcxrLuwyHDpyzSES0Kq0nBZmb57RyvY5f9NcjIb8WF2mTwGSm8DyTWUXynAtDXpEuJ6nIGj4LXsd85Uq2euxhw0XfQKbyTo4HWYEe4dXEoxoYwoqa77pmBwRpsUk+xyLR59DtLW/3smLksoGAS4HJ3SVFov8WXLDXXagYbRAgg1GeaZAgbk8wU/YaQ8CQEAAA==

:recycle: This comment has been updated with latest results.
author_association: NONE