Repository: NillionNetwork/nilvm
Pull request #36: feat(tests): limit number of nilchain payers in tests
Author: login: andreasbros
Created: 2025-02-20T18:58:16Z | Merged: 2025-03-04T15:16:57Z

## Motivation

There is a need to limit the number of payers (blockchain payment accounts)  to be created when running functional tests. Right now fixture creates 100 of them.

## Solution

This change limits how many payers can be created and allows overwritting with env vars in remote func tests, where we can limit to 1 payer and update balance top up amount if needed.

## Total time to runs tests

- locally with 20 payers: 9 seconds
- remote with testnet nilchain with 1 payer: 229 sec

Fixes #
Design discussion issue (if applicable) #

## Merge requirement checklist

* [ ] [CONTRIBUTING](https://github.com/NillionNetwork/nillion/blob/main/CONTRIBUTING.md) guidelines followed
* [ ] Unit tests added/updated (if applicable)
* [ ] Breaking change analysis completed (if applicable). "Will this change require all network cluster operators to update? Does it break public APIs?"
* [ ] For new features or breaking changes, created a documentation issue in [nillion-docs](https://github.com/NillionNetwork/nillion-docs/issues/new/choose)



Diff:
diff --git a/tests/fixtures/nodes/src/nodes.rs b/tests/fixtures/nodes/src/nodes.rs
index c03a3a4..c46cfc3 100644
--- a/tests/fixtures/nodes/src/nodes.rs
+++ b/tests/fixtures/nodes/src/nodes.rs
@@ -30,7 +30,7 @@ use std::{
     net::TcpStream,
     path::{Path, PathBuf},
     process::{Child, Command},
-    sync::atomic::{AtomicU64, Ordering},
+    sync::atomic::{AtomicU64, AtomicUsize, Ordering},
     thread::{self, sleep},
     time::{Duration, Instant},
 };
@@ -39,10 +39,15 @@ use tokio::runtime::Runtime;
 use tracing_fixture::{tracing, Tracing};
 use xshell::{cmd, Shell};
 
-const PAYER_FUND_CHUNK: usize = 20;
+// These can be overwritten by environment variables
+const MAX_PAYERS_NUM: usize = 20;
+// this is how much it costs to run this test suite with prices set in `tests/resources/network/default/1.network.yaml` when running with 1 payer
+const BALANCE_TOP_UP_AMOUNT_NIL: TokenAmount = TokenAmount::Nil(45);
+const PAYMENTS_SEED: &str = "payment-seed";
+
 const SIGNING_KEY_PREFIX: &str = "signinig-key-";
-const TOTAL_PREFUNDED_KEYS: u64 = 1024;
-const PREFUND_AMOUNT: TokenAmount = TokenAmount::Nil(100_000);
+const TOTAL_PREFUNDED_KEYS: u64 = 1024; // user balances in leader DB
+const PREFUND_AMOUNT: TokenAmount = TokenAmount::Nil(100_000); // user balance amount in leader DB
 
 static PAYMENTS_RUNTIME: Lazy<Runtime> = Lazy::new(|| Runtime::new().expect("failed to create runtime"));
 
@@ -208,8 +213,11 @@ pub struct Nodes {
     next_payment_key_id: AtomicU64,
     next_signing_key_id: AtomicU64,
     funded_payers: tokio::sync::Mutex<Vec<NillionChainClientPayer>>,
+    funded_payers_cursor: AtomicUsize,
     bootnode_party_id: Option<PartyId>,
     payments_seed: String,
+    max_payers_num: usize,
+    balance_top_up_amount: TokenAmount,
 }
 
 impl Nodes {
@@ -284,9 +292,7 @@ impl Nodes {
             MembershipClient::new(self.bootnode_channel_config().build().expect("failed to create config"));
         let cluster = loop {
             match membership_client.cluster().await {
-                Ok(cluster) => {
-                    break cluster;
-                }
+                Ok(cluster) => break cluster,
                 Err(e) => {
                     warn!("Bootnode is not up yet, retrying: {e}")
                 }
@@ -321,37 +327,42 @@ impl Nodes {
     }
 
     pub async fn allocate_payer(&self) -> NillionChainClientPayer {
-        // enter the payments runtime so all payers share the same reqwest client pool
-        let _guard = PAYMENTS_RUNTIME.enter();
-        let mut funded_payers = self.funded_payers.lock().await;
-
-        if funded_payers.is_empty() {
-            // Otherwise fund a chunk of payers at once if we don't have any
-            let mut addresses = Vec::new();
-            let mut keys = Vec::new();
-            for _ in 0..PAYER_FUND_CHUNK {
-                let payment_key_id = self.next_payment_key_id.fetch_add(1, Ordering::AcqRel);
-                let payments_seed = format!("{}-{payment_key_id}", self.payments_seed);
-                let key = NillionChainPrivateKey::from_seed(&payments_seed).expect("private key creation failed");
-                let address = key.address.clone();
-                keys.push(key);
-                addresses.push(address);
-            }
-            self.top_up_balances(addresses, TokenAmount::Nil(1)).await;
+        let mut pool_guard = self.funded_payers.lock().await;
+        if pool_guard.is_empty() {
+            let payers = self.fund_payers(self.max_payers_num).await;
+            pool_guard.extend(payers);
+        }
+        let pool_size = pool_guard.len();
+        let index = self.funded_payers_cursor.fetch_add(1, Ordering::Relaxed) % pool_size;
 
-            // Create all the clients in bulk as this performs a lookup on the chain.
-            let mut futs = Vec::new();
-            for key in keys {
-                let payments_rpc_endpoint = self.nillion_chain_rpc_endpoint();
-                futs.push(NillionChainClient::new(payments_rpc_endpoint, key));
-            }
-            for result in future::join_all(futs).await {
-                let client = result.expect("failed to look up client");
-                let payer = NillionChainClientPayer::new(client);
-                funded_payers.push(payer);
-            }
+        pool_guard[index].clone()
+    }
+
+    pub(crate) async fn fund_payers(&self, count: usize) -> Vec<NillionChainClientPayer> {
+        let _guard = PAYMENTS_RUNTIME.enter();
+        let mut addresses = Vec::with_capacity(count);
+        let mut keys = Vec::with_capacity(count);
+        for _ in 0..count {
+            let payment_key_id = self.next_payment_key_id.fetch_add(1, Ordering::AcqRel);
+            let payments_seed = format!("{}-{payment_key_id}", self.payments_seed);
+
+            let key = NillionChainPrivateKey::from_seed(&payments_seed).expect("private key creation failed");
+            let address = key.address.clone();
+            keys.push(key);
+            addresses.push(address);
+        }
+        self.top_up_balances(addresses, self.balance_top_up_amount).await;
+        let mut futs = Vec::with_capacity(count);
+        for key in keys {
+            let payments_rpc_endpoint = self.nillion_chain_rpc_endpoint();
+            futs.push(NillionChainClient::new(payments_rpc_endpoint, key));
+        }
+        let mut payers = Vec::with_capacity(count);
+        for result in future::join_all(futs).await {
+            let client = result.expect("failed to look up client");
+            payers.push(NillionChainClientPayer::new(client));
         }
-        funded_payers.pop().expect("should not be empty")
+        payers
     }
 }
 
@@ -469,7 +480,12 @@ pub fn nodes(_tracing: &Tracing) -> Nodes {
         cleanup::register_child_process(child_process);
     }
 
-    let payments_seed = env::var("TEST_PAYMENTS_SEED").unwrap_or_else(|_| "payment-seed".to_string());
+    let payments_seed = env::var("PAYMENTS_SEED").unwrap_or_else(|_| PAYMENTS_SEED.to_string());
+    let max_payers_num = env::var("MAX_PAYERS_NUM").ok().and_then(|val| val.parse().ok()).unwrap_or(MAX_PAYERS_NUM);
+    let balance_top_up_amount = env::var("BALANCE_TOP_UP_AMOUNT_NIL")
+        .ok()
+        .map(|val| TokenAmount::Nil(val.parse().unwrap()))
+        .unwrap_or(BALANCE_TOP_UP_AMOUNT_NIL);
 
     let mut nodes = Nodes {
         context,
@@ -479,8 +495,11 @@ pub fn nodes(_tracing: &Tracing) -> Nodes {
         next_payment_key_id: Default::default(),
         next_signing_key_id: Default::default(),
         funded_payers: Default::default(),
+        funded_payers_cursor: AtomicUsize::new(0),
         bootnode_party_id: None,
         payments_seed,
+        max_payers_num,
+        balance_top_up_amount,
     };
 
     // This is because this is a non async rstest fixture but it gets run within an async context
diff --git a/tests/functional/src/tests.rs b/tests/functional/src/tests.rs
index 7bcba9c..bfd8766 100644
--- a/tests/functional/src/tests.rs
+++ b/tests/functional/src/tests.rs
@@ -1216,9 +1216,12 @@ async fn pay_with_funds(nodes: &Nodes) {
     let balance = client.account_balance().await.expect("failed to look up").balance;
     assert_eq!(balance, 0);
 
+    let payments_config = client.payments_config().await.expect("failed to look up payments config");
+    let credits_per_nil = payments_config.credits_per_nil;
+
     // 1 nil = 100 credits at a $1 rate
     let added_nils = 1;
-    let added_credits = 100;
+    let added_credits = added_nils * credits_per_nil;
     client
         .add_funds()
         .amount(TokenAmount::Nil(added_nils))
diff --git a/tests/resources/network/default/1.network.yaml b/tests/resources/network/default/1.network.yaml
index 6c63380..1938728 100644
--- a/tests/resources/network/default/1.network.yaml
+++ b/tests/resources/network/default/1.network.yaml
@@ -103,16 +103,20 @@ program_auditor:
       Compare: 1000
       EqualsIntegerSecret: 1000
 
+dollar_token_conversion_enabled: false
+# 1 NIL token price in dollars
+dollar_token_conversion_fixed: 1.0
+
 payments:
   rpc_endpoint: http://localhost:26648
   pricing:
-    retrieve_permissions_price: 100
-    pool_status_price: 100
-    overwrite_permissions_price: 100
-    update_permissions_price: 100
-    retrieve_values_price: 100
-    store_program_price: 100
-    store_values_price: 100
+    retrieve_permissions_price: 10
+    pool_status_price: 10
+    overwrite_permissions_price: 10
+    update_permissions_price: 10
+    retrieve_values_price: 10
+    store_program_price: 10
+    store_values_price: 50
     invoke_compute_price: 100
   # 1 cent/credit
   minimum_add_funds_payment: 1


Reviews:
user: login: lumasepa
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-21T11:24:02Z
user: login: lumasepa
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-02-21T11:24:26Z
user: login: mfontanini
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-02-21T15:32:50Z
user: login: andreasbros
state: COMMENTED
author_association: NONE
submitted_at: 2025-02-21T19:45:33Z
user: login: andreasbros
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-03T14:20:47Z
user: login: mfontanini
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T17:08:40Z
user: login: andreasbros
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-03T17:31:02Z
user: login: mfontanini
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T17:36:31Z
user: login: lumasepa
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T18:16:03Z

Review comments:
diff_hunk: @@ -321,37 +319,63 @@ impl Nodes {
     }
 
     pub async fn allocate_payer(&self) -> NillionChainClientPayer {
-        // enter the payments runtime so all payers share the same reqwest client pool
+        self.funded_payers.allocate_payer(self).await
+    }
+
+    pub(crate) async fn fund_payers(&self, count: usize) -> Vec<NillionChainClientPayer> {
         let _guard = PAYMENTS_RUNTIME.enter();
-        let mut funded_payers = self.funded_payers.lock().await;
-
-        if funded_payers.is_empty() {
-            // Otherwise fund a chunk of payers at once if we don't have any
-            let mut addresses = Vec::new();
-            let mut keys = Vec::new();
-            for _ in 0..PAYER_FUND_CHUNK {
-                let payment_key_id = self.next_payment_key_id.fetch_add(1, Ordering::AcqRel);
-                let payments_seed = format!("{}-{payment_key_id}", self.payments_seed);
-                let key = NillionChainPrivateKey::from_seed(&payments_seed).expect("private key creation failed");
-                let address = key.address.clone();
-                keys.push(key);
-                addresses.push(address);
-            }
-            self.top_up_balances(addresses, TokenAmount::Nil(1)).await;
+        let mut addresses = Vec::with_capacity(count);
+        let mut keys = Vec::with_capacity(count);
+        for _ in 0..count {
+            let payment_key_id = self.next_payment_key_id.fetch_add(1, Ordering::AcqRel);
+            let payments_seed = format!("{}-{payment_key_id}", self.payments_seed);
+
+            let key = NillionChainPrivateKey::from_seed(&payments_seed).expect("private key creation failed");
+            let address = key.address.clone();
+            keys.push(key);
+            addresses.push(address);
+        }
+        self.top_up_balances(addresses, TokenAmount::Unil(1_500_000)).await;
+        let mut futs = Vec::with_capacity(count);
+        for key in keys {
+            let payments_rpc_endpoint = self.nillion_chain_rpc_endpoint();
+            futs.push(NillionChainClient::new(payments_rpc_endpoint, key));
+        }
+        let mut payers = Vec::with_capacity(count);
+        for result in future::join_all(futs).await {
+            let client = result.expect("failed to look up client");
+            payers.push(NillionChainClientPayer::new(client));
+        }
+        payers
+    }
+}
 
-            // Create all the clients in bulk as this performs a lookup on the chain.
-            let mut futs = Vec::new();
-            for key in keys {
-                let payments_rpc_endpoint = self.nillion_chain_rpc_endpoint();
-                futs.push(NillionChainClient::new(payments_rpc_endpoint, key));
-            }
-            for result in future::join_all(futs).await {
-                let client = result.expect("failed to look up client");
-                let payer = NillionChainClientPayer::new(client);
-                funded_payers.push(payer);
-            }
+pub struct ManagedNillionChainClientPayer {
+    pool: Mutex<Option<Vec<NillionChainClientPayer>>>,
path: tests/fixtures/nodes/src/nodes.rs
user: login: lumasepa
body: You can avoid the Option using an empty Vec
created_at: 2025-02-21T09:07:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -207,7 +207,7 @@ pub struct Nodes {
     stash_client: tokio::sync::Mutex<NillionChainClient>,
     next_payment_key_id: AtomicU64,
     next_signing_key_id: AtomicU64,
-    funded_payers: tokio::sync::Mutex<Vec<NillionChainClientPayer>>,
+    funded_payers: ManagedNillionChainClientPayer,
path: tests/fixtures/nodes/src/nodes.rs
user: login: mfontanini
body: What was the reasoning behind moving this logic to a new type? This pooling logic could have been here right? You essentially moved the existing `funded_payers` to a new type.
created_at: 2025-02-21T15:10:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -30,16 +30,16 @@ use std::{
     net::TcpStream,
     path::{Path, PathBuf},
     process::{Child, Command},
-    sync::atomic::{AtomicU64, Ordering},
+    sync::atomic::{AtomicU64, AtomicUsize, Ordering},
     thread::{self, sleep},
     time::{Duration, Instant},
 };
 use tempfile::TempDir;
-use tokio::runtime::Runtime;
+use tokio::{runtime::Runtime, sync::Mutex};
 use tracing_fixture::{tracing, Tracing};
 use xshell::{cmd, Shell};
 
-const PAYER_FUND_CHUNK: usize = 20;
+pub const MAX_PAYERS_NUM: usize = 2;
path: tests/fixtures/nodes/src/nodes.rs
user: login: mfontanini
body: The idea here was to fund lots of payers in a single TX to speed tests up. If this doesn't slow anything down we might as well have a single payer and get rid of all this pooling logic? e.g. if going from 40+ payers to 2 doesn't slow it down then making it go down to 1 won't either. 

I think this was added because of bors r+ tests though, so you likely won't see the slowdown locally. Did you see how much slower it is there?
created_at: 2025-02-21T15:29:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -30,16 +30,16 @@ use std::{
     net::TcpStream,
     path::{Path, PathBuf},
     process::{Child, Command},
-    sync::atomic::{AtomicU64, Ordering},
+    sync::atomic::{AtomicU64, AtomicUsize, Ordering},
     thread::{self, sleep},
     time::{Duration, Instant},
 };
 use tempfile::TempDir;
-use tokio::runtime::Runtime;
+use tokio::{runtime::Runtime, sync::Mutex};
 use tracing_fixture::{tracing, Tracing};
 use xshell::{cmd, Shell};
 
-const PAYER_FUND_CHUNK: usize = 20;
+pub const MAX_PAYERS_NUM: usize = 2;
path: tests/fixtures/nodes/src/nodes.rs
user: login: andreasbros
body: some test(s) failed when running all of them with 1 payer, i think some tests require two payers?, but let me double check that
created_at: 2025-02-21T19:45:33Z
author_association: NONE
diff_hunk: @@ -30,16 +30,16 @@ use std::{
     net::TcpStream,
     path::{Path, PathBuf},
     process::{Child, Command},
-    sync::atomic::{AtomicU64, Ordering},
+    sync::atomic::{AtomicU64, AtomicUsize, Ordering},
     thread::{self, sleep},
     time::{Duration, Instant},
 };
 use tempfile::TempDir;
-use tokio::runtime::Runtime;
+use tokio::{runtime::Runtime, sync::Mutex};
 use tracing_fixture::{tracing, Tracing};
 use xshell::{cmd, Shell};
 
-const PAYER_FUND_CHUNK: usize = 20;
+pub const MAX_PAYERS_NUM: usize = 2;
path: tests/fixtures/nodes/src/nodes.rs
user: login: andreasbros
body: I forgot that we have `PrefundedAccount'`s which are used in tests. When prefunding is disabled, having more payers make tests run faster, 30 payers vs 1 => 9 sec vs 18 sec.
Also, I think for remote functional tests having more payers will also help, as tx commit times are much slower in testnet.
created_at: 2025-03-03T14:20:47Z
author_association: NONE
diff_hunk: @@ -39,10 +39,14 @@ use tokio::runtime::Runtime;
 use tracing_fixture::{tracing, Tracing};
 use xshell::{cmd, Shell};
 
-const PAYER_FUND_CHUNK: usize = 20;
+// These can be overwritten by environment variables
+const TEST_MAX_PAYERS_NUM: usize = 20;
+const TEST_BALANCE_TOP_UP_AMOUNT_NIL: TokenAmount = TokenAmount::Nil(45); // this is how much it costs to run this test suite with prices set in `tests/resources/network/default/1.network.yaml` when running with 1 payer
path: tests/fixtures/nodes/src/nodes.rs
user: login: mfontanini
body: This used to be 1nil and now it's 45nil right? Why is this 45x what is was before? Can you also put the comment above this line rather than after it? This line is unnecessarily long
created_at: 2025-03-03T17:02:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -39,10 +39,14 @@ use tokio::runtime::Runtime;
 use tracing_fixture::{tracing, Tracing};
 use xshell::{cmd, Shell};
 
-const PAYER_FUND_CHUNK: usize = 20;
+// These can be overwritten by environment variables
+const TEST_MAX_PAYERS_NUM: usize = 20;
path: tests/fixtures/nodes/src/nodes.rs
user: login: mfontanini
body: These `TEST_` prefixes on these constants shouldn't be here I think
created_at: 2025-03-03T17:02:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -469,7 +479,13 @@ pub fn nodes(_tracing: &Tracing) -> Nodes {
         cleanup::register_child_process(child_process);
     }
 
-    let payments_seed = env::var("TEST_PAYMENTS_SEED").unwrap_or_else(|_| "payment-seed".to_string());
+    let payments_seed = env::var("TEST_PAYMENTS_SEED").unwrap_or_else(|_| TEST_PAYMENT_SEED.to_string());
path: tests/fixtures/nodes/src/nodes.rs
user: login: mfontanini
body: `TEST_PAYMENTS_SEED` vs `TEST_PAYMENT_SEED`
created_at: 2025-03-03T17:08:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -39,10 +39,14 @@ use tokio::runtime::Runtime;
 use tracing_fixture::{tracing, Tracing};
 use xshell::{cmd, Shell};
 
-const PAYER_FUND_CHUNK: usize = 20;
+// These can be overwritten by environment variables
+const TEST_MAX_PAYERS_NUM: usize = 20;
+const TEST_BALANCE_TOP_UP_AMOUNT_NIL: TokenAmount = TokenAmount::Nil(45); // this is how much it costs to run this test suite with prices set in `tests/resources/network/default/1.network.yaml` when running with 1 payer
path: tests/fixtures/nodes/src/nodes.rs
user: login: andreasbros
body: This is the cost of running all func tests by 1 payer with current prices. Before we had 100 payers.
Also, this balance (in nilchain) is used only when where is no pre-funded balance in the leader DB (in remote func tests).
created_at: 2025-03-03T17:31:02Z
author_association: NONE
diff_hunk: @@ -39,10 +39,14 @@ use tokio::runtime::Runtime;
 use tracing_fixture::{tracing, Tracing};
 use xshell::{cmd, Shell};
 
-const PAYER_FUND_CHUNK: usize = 20;
+// These can be overwritten by environment variables
+const TEST_MAX_PAYERS_NUM: usize = 20;
+const TEST_BALANCE_TOP_UP_AMOUNT_NIL: TokenAmount = TokenAmount::Nil(45); // this is how much it costs to run this test suite with prices set in `tests/resources/network/default/1.network.yaml` when running with 1 payer
path: tests/fixtures/nodes/src/nodes.rs
user: login: mfontanini
body: But with default parameters the test uses max payers (20) * top up amount (45) = 900 nils right? Should this default be kept lower and let any environment that wants to use 1 payer adjust the amount accordingly?
created_at: 2025-03-03T17:36:31Z
author_association: CONTRIBUTOR

PR comments:
user: login: andreasbros
created_at: 2025-02-27T18:34:49Z
body: bors try
author_association: NONE
user: login: nilogy-bors[bot]
created_at: 2025-02-27T18:45:26Z
body: ## try

Build failed:
  * [CI Status](https://github.com/NillionNetwork/nilvm/actions/runs/13573438877/job/37944450135)
author_association: CONTRIBUTOR
user: login: andreasbros
created_at: 2025-03-04T12:27:23Z
body: bors merge
author_association: NONE
user: login: nilogy-bors[bot]
created_at: 2025-03-04T15:16:57Z
body: Build succeeded:
  * [CI Status](https://github.com/NillionNetwork/nilvm/actions/runs/13654856721/job/38178229876)
author_association: CONTRIBUTOR