Repository: hyperlane-xyz/hyperlane-monorepo
Pull request #6114: feat: chunk message db insertions
Author: login: kamiyaa
Created: 2025-05-02T14:27:30Z | Merged: 2025-05-09T09:58:30Z

### Description

Instead of inserting all the messages we receive in a single db transaction, we chunk it into smaller sizes.
This prevents us from running into `Too many arguments` errors from Postgres for exceeding their `u16::MAX` parameter count.

### Related issues
- fixes https://linear.app/hyperlane-xyz/issue/ENG-1390/scraper-cannot-insert-too-many-records

### Backward compatibility

Yes

### Testing

Manual. From Jeff:

Tried it locally with a postgres instance with 100k messages and it failed without chunking 
```
running 1 test
RES: Err(Query Error: encountered unexpected or invalid data: PgConnection::run(): too many arguments for query: 1000000 (sqlx_postgres::connection::executor:215)

Caused by:
   0: encountered unexpected or invalid data: PgConnection::run(): too many arguments for query: 1000000 (sqlx_postgres::connection::executor:215)
   1: encountered unexpected or invalid data: PgConnection::run(): too many arguments for query: 1000000 (sqlx_postgres::connection::executor:215)

Location:
    agents/scraper/src/db/message.rs:320:9)
test db::message::tests::test_store_dispatched_messages_real_postgres ... ok
```

Tried it again with chunking + transaction and it didn't have any errors :+1:
So we can use the transaction if we want the action to be atomic, but the important part is the chunking.



Diff:
diff --git a/rust/main/agents/scraper/src/db/message.rs b/rust/main/agents/scraper/src/db/message.rs
index fc39049049c..714cdd32f35 100644
--- a/rust/main/agents/scraper/src/db/message.rs
+++ b/rust/main/agents/scraper/src/db/message.rs
@@ -2,7 +2,9 @@
 
 use eyre::Result;
 use itertools::Itertools;
-use sea_orm::{prelude::*, ActiveValue::*, DeriveColumn, EnumIter, Insert, QuerySelect};
+use sea_orm::{
+    prelude::*, ActiveValue::*, DeriveColumn, EnumIter, Insert, QuerySelect, TransactionTrait,
+};
 use tracing::{debug, instrument, trace};
 
 use hyperlane_core::{
@@ -32,6 +34,14 @@ pub struct StorableMessage<'a> {
 }
 
 impl ScraperDb {
+    /// Used for store_dispatched_messages()
+    /// message::ActiveModel has 11 fields, on conflict has 3,
+    /// update has 6. So there should be about a maximum of 20 sql
+    /// parameters per ActiveModel
+    /// u16::MAX (65_535u16) is the maximum amount of parameters we can
+    /// have for Postgres. So 65000 / 20 = 3250
+    const STORE_MESSAGE_CHUNK_SIZE: usize = 3250;
+
     /// Get the delivered message associated with a sequence.
     #[instrument(skip(self))]
     pub async fn retrieve_delivery_by_sequence(
@@ -274,9 +284,11 @@ impl ScraperDb {
         messages: impl Iterator<Item = StorableMessage<'_>>,
     ) -> Result<u64> {
         let origin_mailbox = address_to_bytes(origin_mailbox);
+
         let latest_id_before = self
             .latest_dispatched_id(domain, origin_mailbox.clone())
             .await?;
+
         // we have a race condition where a message may not have been scraped yet even
         let models = messages
             .map(|storable| message::ActiveModel {
@@ -305,24 +317,36 @@ impl ScraperDb {
             return Ok(0);
         }
 
-        Insert::many(models)
-            .on_conflict(
-                OnConflict::columns([
-                    message::Column::OriginMailbox,
-                    message::Column::Origin,
-                    message::Column::Nonce,
-                ])
-                .update_columns([
-                    message::Column::TimeCreated,
-                    message::Column::Destination,
-                    message::Column::Sender,
-                    message::Column::Recipient,
-                    message::Column::MsgBody,
-                    message::Column::OriginTxId,
-                ])
-                .to_owned(),
-            )
-            .exec(&self.0)
+        // ensure all chunks are inserted or none at all
+        self.0
+            .transaction::<_, (), DbErr>(|txn| {
+                Box::pin(async move {
+                    // insert messages in chunks, to not run into
+                    // "Too many arguments" error
+                    for chunk in models.chunks(Self::STORE_MESSAGE_CHUNK_SIZE) {
+                        Insert::many(chunk.to_vec())
+                            .on_conflict(
+                                OnConflict::columns([
+                                    message::Column::OriginMailbox,
+                                    message::Column::Origin,
+                                    message::Column::Nonce,
+                                ])
+                                .update_columns([
+                                    message::Column::TimeCreated,
+                                    message::Column::Destination,
+                                    message::Column::Sender,
+                                    message::Column::Recipient,
+                                    message::Column::MsgBody,
+                                    message::Column::OriginTxId,
+                                ])
+                                .to_owned(),
+                            )
+                            .exec(txn)
+                            .await?;
+                    }
+                    Ok(())
+                })
+            })
             .await?;
 
         let new_dispatch_count = self
@@ -336,3 +360,170 @@ impl ScraperDb {
         Ok(new_dispatch_count)
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use std::collections::BTreeMap;
+
+    use time::macros::*;
+
+    use hyperlane_core::{HyperlaneMessage, LogMeta, H256};
+    use sea_orm::{Database, DatabaseBackend, DbErr, MockDatabase, RuntimeErr, Value};
+    use time::PrimitiveDateTime;
+
+    use crate::db::{generated::message, ScraperDb, StorableMessage};
+
+    /// Tests store_dispatched_messages() a transaction works
+    #[tokio::test]
+    async fn test_store_dispatched_messages_transaction() {
+        const MESSAGE_AMOUNT: usize = 10000;
+
+        let query_results: Vec<Vec<_>> = (0..10000)
+            .map(|i| message::Model {
+                id: i as i64,
+                time_created: PrimitiveDateTime::new(date!(2019 - 01 - 01), time!(0:00)),
+                msg_id: vec![],
+                origin: 0,
+                destination: 0,
+                nonce: 0,
+                sender: vec![],
+                recipient: vec![],
+                msg_body: None,
+                origin_mailbox: vec![],
+                origin_tx_id: 0,
+            })
+            .collect::<Vec<_>>()
+            .chunks(ScraperDb::STORE_MESSAGE_CHUNK_SIZE)
+            .map(|v| v.to_vec())
+            .collect();
+
+        let mock_result: BTreeMap<&str, _> = [
+            ("num_items", Into::<Value>::into(10000i64)),
+            ("last_insert_id", Into::<Value>::into(10000i64)),
+        ]
+        .into_iter()
+        .collect();
+        let mock_db = MockDatabase::new(DatabaseBackend::Postgres)
+            .append_query_results([[message::Model {
+                id: 0,
+                time_created: PrimitiveDateTime::new(date!(2019 - 01 - 01), time!(0:00)),
+                msg_id: vec![],
+                origin: 0,
+                destination: 0,
+                nonce: 0,
+                sender: vec![],
+                recipient: vec![],
+                msg_body: None,
+                origin_mailbox: vec![],
+                origin_tx_id: 0,
+            }]])
+            .append_query_results(query_results)
+            .append_query_results([[mock_result.clone()]])
+            .into_connection();
+        let scraper_db = ScraperDb::with_connection(mock_db);
+
+        let logs_meta: Vec<_> = (0..MESSAGE_AMOUNT).map(|_| LogMeta::default()).collect();
+        let messages: Vec<_> = (0..MESSAGE_AMOUNT)
+            .map(|i| StorableMessage {
+                msg: HyperlaneMessage::default(),
+                meta: &logs_meta[i],
+                txn_id: i as i64,
+            })
+            .collect();
+        let res = scraper_db
+            .store_dispatched_messages(0, &H256::zero(), messages.into_iter())
+            .await;
+        assert!(res.is_ok());
+    }
+
+    /// Tests store_dispatched_messages() fails if one of the queries
+    /// within the transaction fails
+    #[tokio::test]
+    async fn test_store_dispatched_messages_fail() {
+        const MESSAGE_AMOUNT: usize = 5000;
+
+        let query_results: Vec<Vec<_>> = (0..MESSAGE_AMOUNT)
+            .map(|i| message::Model {
+                id: i as i64,
+                time_created: PrimitiveDateTime::new(date!(2019 - 01 - 01), time!(0:00)),
+                msg_id: vec![],
+                origin: 0,
+                destination: 0,
+                nonce: 0,
+                sender: vec![],
+                recipient: vec![],
+                msg_body: None,
+                origin_mailbox: vec![],
+                origin_tx_id: 0,
+            })
+            .collect::<Vec<_>>()
+            .chunks(ScraperDb::STORE_MESSAGE_CHUNK_SIZE)
+            .map(|v| v.to_vec())
+            .collect();
+
+        let mock_db = MockDatabase::new(DatabaseBackend::Postgres)
+            .append_query_results([[message::Model {
+                id: 0,
+                time_created: PrimitiveDateTime::new(date!(2019 - 01 - 01), time!(0:00)),
+                msg_id: vec![],
+                origin: 0,
+                destination: 0,
+                nonce: 0,
+                sender: vec![],
+                recipient: vec![],
+                msg_body: None,
+                origin_mailbox: vec![],
+                origin_tx_id: 0,
+            }]])
+            .append_query_results(query_results)
+            // fail halfway through the transaction
+            .append_query_errors([DbErr::Exec(RuntimeErr::Internal(
+                "Unknown error".to_string(),
+            ))])
+            .into_connection();
+        let scraper_db = ScraperDb::with_connection(mock_db);
+
+        let logs_meta: Vec<_> = (0..MESSAGE_AMOUNT).map(|_| LogMeta::default()).collect();
+        let messages: Vec<_> = (0..MESSAGE_AMOUNT)
+            .map(|i| StorableMessage {
+                msg: HyperlaneMessage::default(),
+                meta: &logs_meta[i],
+                txn_id: i as i64,
+            })
+            .collect();
+        let res = scraper_db
+            .store_dispatched_messages(0, &H256::zero(), messages.into_iter())
+            .await;
+        assert!(res.is_err());
+    }
+
+    /// Tests store_dispatched_messages() with a real postgres instance
+    #[ignore]
+    #[tokio::test]
+    async fn test_store_dispatched_messages_real_postgres() {
+        const POSTGRES_URL: &str = "postgresql://postgres:password@localhost:5432";
+        const MESSAGE_AMOUNT: usize = 100000;
+
+        let db = Database::connect(POSTGRES_URL).await.unwrap();
+        let scraper_db = ScraperDb::with_connection(db);
+
+        let logs_meta: Vec<_> = (0..MESSAGE_AMOUNT).map(|_| LogMeta::default()).collect();
+        let messages: Vec<_> = (0..MESSAGE_AMOUNT)
+            .map(|i| {
+                let mut msg = HyperlaneMessage::default();
+                msg.nonce = i as u32;
+                msg.origin = 1;
+                msg.destination = 1;
+                StorableMessage {
+                    msg,
+                    meta: &logs_meta[i],
+                    txn_id: 0 as i64,
+                }
+            })
+            .collect();
+        let res = scraper_db
+            .store_dispatched_messages(0, &H256::zero(), messages.into_iter())
+            .await;
+        assert!(res.is_ok());
+    }
+}
diff --git a/rust/main/helm/hyperlane-agent/templates/relayer-statefulset.yaml b/rust/main/helm/hyperlane-agent/templates/relayer-statefulset.yaml
index 4dcf1a258c8..59448b681b1 100644
--- a/rust/main/helm/hyperlane-agent/templates/relayer-statefulset.yaml
+++ b/rust/main/helm/hyperlane-agent/templates/relayer-statefulset.yaml
@@ -11,7 +11,7 @@ spec:
     matchLabels:
       {{- include "agent-common.selectorLabels" . | nindent 6 }}
       app.kubernetes.io/component: relayer
-  replicas: 1 
+  replicas: 1
   serviceName: {{ include "agent-common.fullname" . }}-relayer
   template:
     metadata:
@@ -119,7 +119,7 @@ spec:
           mountPath: {{ .Values.hyperlane.dbPath }}
         - name: relayer-configmap
           mountPath: /relayer-configmap
-        ports: 
+        ports:
         - name: metrics
           containerPort: {{ .Values.hyperlane.metrics.port }}
       volumes:
diff --git a/rust/main/helm/hyperlane-agent/templates/scraper-statefulset.yaml b/rust/main/helm/hyperlane-agent/templates/scraper-statefulset.yaml
index 0ca8c184b7a..9afe1e01428 100644
--- a/rust/main/helm/hyperlane-agent/templates/scraper-statefulset.yaml
+++ b/rust/main/helm/hyperlane-agent/templates/scraper-statefulset.yaml
@@ -11,7 +11,7 @@ spec:
     matchLabels:
       {{- include "agent-common.selectorLabels" . | nindent 6 }}
       app.kubernetes.io/component: scraper3
-  replicas: 1 
+  replicas: 1
   serviceName: {{ include "agent-common.fullname" . }}-scraper3
   template:
     metadata:
@@ -60,7 +60,7 @@ spec:
         {{- include "agent-common.config-env-vars" (dict "config" .Values.hyperlane.scraper.config) | nindent 8 }}
         resources:
           {{- toYaml .Values.hyperlane.scraper.resources | nindent 10 }}
-        ports: 
+        ports:
         - name: metrics
           containerPort: {{ .Values.hyperlane.metrics.port }}
       {{- with .Values.nodeSelector }}
diff --git a/rust/main/helm/hyperlane-agent/templates/validator-statefulset.yaml b/rust/main/helm/hyperlane-agent/templates/validator-statefulset.yaml
index b5929bfd99f..f4df1ce54a3 100644
--- a/rust/main/helm/hyperlane-agent/templates/validator-statefulset.yaml
+++ b/rust/main/helm/hyperlane-agent/templates/validator-statefulset.yaml
@@ -75,7 +75,7 @@ spec:
           mountPath: /config-env-vars
         - name: secret-env-vars
           mountPath: /secret-env-vars
-        ports: 
+        ports:
         - name: metrics
           containerPort: {{ .Values.hyperlane.metrics.port }}
       volumes:


Reviews:
user: login: kamiyaa
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T14:29:57Z
user: login: kamiyaa
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T14:30:29Z
user: login: ameten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T09:23:45Z
user: login: ameten
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T09:23:48Z
user: login: ameten
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T13:52:23Z
user: login: daniel-savu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T14:49:30Z
user: login: daniel-savu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T14:52:23Z
user: login: kamiyaa
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T17:18:43Z
user: login: kamiyaa
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T19:43:16Z
user: login: daniel-savu
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T09:47:28Z
user: login: daniel-savu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T09:47:56Z
user: login: kamiyaa
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T12:53:56Z

Review comments:
diff_hunk: @@ -32,6 +32,14 @@ pub struct StorableMessage<'a> {
 }
 
 impl ScraperDb {
+    /// Used for store_dispatched_messages()
+    /// message::ActiveModel has 11 fields, on conflict has 3,
+    /// update has 6. So there should be about a maximum of 20 sql
+    /// parameters per ActiveModel
+    /// u16::MAX (65_535u16) is the maximum amount of parameters we can
+    /// have for Postgres. So 65000 / 20 = 3250
+    const STORE_MESSAGE_CHUNK_SIZE: usize = 3250;
path: rust/main/agents/scraper/src/db/message.rs
user: login: kamiyaa
body: kind of a magic number I came up with, so open to changing this.
created_at: 2025-05-02T14:29:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -305,25 +313,27 @@ impl ScraperDb {
             return Ok(0);
         }
 
-        Insert::many(models)
-            .on_conflict(
-                OnConflict::columns([
-                    message::Column::OriginMailbox,
-                    message::Column::Origin,
-                    message::Column::Nonce,
-                ])
-                .update_columns([
-                    message::Column::TimeCreated,
-                    message::Column::Destination,
-                    message::Column::Sender,
-                    message::Column::Recipient,
-                    message::Column::MsgBody,
-                    message::Column::OriginTxId,
-                ])
-                .to_owned(),
-            )
-            .exec(&self.0)
-            .await?;
+        for chunk in models.chunks(Self::STORE_MESSAGE_CHUNK_SIZE) {
+            Insert::many(chunk.to_vec())
path: rust/main/agents/scraper/src/db/message.rs
user: login: kamiyaa
body: Not sure if theres a way to prevent cloning the chunk here.
created_at: 2025-05-02T14:30:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -32,6 +32,14 @@ pub struct StorableMessage<'a> {
 }
 
 impl ScraperDb {
+    /// Used for store_dispatched_messages()
+    /// message::ActiveModel has 11 fields, on conflict has 3,
+    /// update has 6. So there should be about a maximum of 20 sql
+    /// parameters per ActiveModel
+    /// u16::MAX (65_535u16) is the maximum amount of parameters we can
+    /// have for Postgres. So 65000 / 20 = 3250
+    const STORE_MESSAGE_CHUNK_SIZE: usize = 3250;
path: rust/main/agents/scraper/src/db/message.rs
user: login: ameten
body: It makes sense.
created_at: 2025-05-05T09:23:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -305,25 +313,27 @@ impl ScraperDb {
             return Ok(0);
         }
 
-        Insert::many(models)
-            .on_conflict(
-                OnConflict::columns([
-                    message::Column::OriginMailbox,
-                    message::Column::Origin,
-                    message::Column::Nonce,
-                ])
-                .update_columns([
-                    message::Column::TimeCreated,
-                    message::Column::Destination,
-                    message::Column::Sender,
-                    message::Column::Recipient,
-                    message::Column::MsgBody,
-                    message::Column::OriginTxId,
-                ])
-                .to_owned(),
-            )
-            .exec(&self.0)
-            .await?;
+        for chunk in models.chunks(Self::STORE_MESSAGE_CHUNK_SIZE) {
+            Insert::many(chunk.to_vec())
path: rust/main/agents/scraper/src/db/message.rs
user: login: ameten
body: Ok
created_at: 2025-05-05T09:23:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -336,3 +360,136 @@ impl ScraperDb {
         Ok(new_dispatch_count)
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use std::collections::BTreeMap;
+
+    use time::macros::*;
+
+    use hyperlane_core::{HyperlaneMessage, LogMeta, H256};
+    use sea_orm::{DatabaseBackend, DbErr, MockDatabase, RuntimeErr, Value};
+    use time::PrimitiveDateTime;
+
+    use crate::db::{generated::message, ScraperDb, StorableMessage};
+
+    /// Tests store_dispatched_messages() a transaction works
+    #[tokio::test]
+    async fn test_store_dispatched_messages_transaction() {
+        let query_results: Vec<Vec<_>> = (0..10000)
+            .map(|i| message::Model {
+                id: i,
+                time_created: PrimitiveDateTime::new(date!(2019 - 01 - 01), time!(0:00)),
+                msg_id: vec![],
+                origin: 0,
+                destination: 0,
+                nonce: 0,
+                sender: vec![],
+                recipient: vec![],
+                msg_body: None,
+                origin_mailbox: vec![],
+                origin_tx_id: 0,
+            })
+            .collect::<Vec<_>>()
+            .chunks(ScraperDb::STORE_MESSAGE_CHUNK_SIZE)
+            .map(|v| v.to_vec())
+            .collect();
+
+        let mock_result: BTreeMap<&str, _> = [
+            ("num_items", Into::<Value>::into(10000i64)),
+            ("last_insert_id", Into::<Value>::into(10000i64)),
+        ]
+        .into_iter()
+        .collect();
+        let mock_db = MockDatabase::new(DatabaseBackend::Postgres)
path: rust/main/agents/scraper/src/db/message.rs
user: login: daniel-savu
body: it seems like this doesn't actually create a postgres instance. Were you able to reproduce the `too many arguments` error with this setup?
created_at: 2025-05-07T14:49:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -336,3 +360,136 @@ impl ScraperDb {
         Ok(new_dispatch_count)
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use std::collections::BTreeMap;
+
+    use time::macros::*;
+
+    use hyperlane_core::{HyperlaneMessage, LogMeta, H256};
+    use sea_orm::{DatabaseBackend, DbErr, MockDatabase, RuntimeErr, Value};
+    use time::PrimitiveDateTime;
+
+    use crate::db::{generated::message, ScraperDb, StorableMessage};
+
+    /// Tests store_dispatched_messages() a transaction works
+    #[tokio::test]
+    async fn test_store_dispatched_messages_transaction() {
+        let query_results: Vec<Vec<_>> = (0..10000)
+            .map(|i| message::Model {
+                id: i,
+                time_created: PrimitiveDateTime::new(date!(2019 - 01 - 01), time!(0:00)),
+                msg_id: vec![],
+                origin: 0,
+                destination: 0,
+                nonce: 0,
+                sender: vec![],
+                recipient: vec![],
+                msg_body: None,
+                origin_mailbox: vec![],
+                origin_tx_id: 0,
+            })
+            .collect::<Vec<_>>()
+            .chunks(ScraperDb::STORE_MESSAGE_CHUNK_SIZE)
+            .map(|v| v.to_vec())
+            .collect();
+
+        let mock_result: BTreeMap<&str, _> = [
+            ("num_items", Into::<Value>::into(10000i64)),
+            ("last_insert_id", Into::<Value>::into(10000i64)),
+        ]
+        .into_iter()
+        .collect();
+        let mock_db = MockDatabase::new(DatabaseBackend::Postgres)
path: rust/main/agents/scraper/src/db/message.rs
user: login: daniel-savu
body: asking because i'm not convinced that creating a `transaction` of insertions will fix this. Would like to see a doc / reference that confirms that a "transaction" of smaller insertion won't run into the same error. Since atomicity for these insertions is not a requirement (because we don't expect write race conditions on these items), it may be safer to just not use a "transaction"
created_at: 2025-05-07T14:52:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -336,3 +360,136 @@ impl ScraperDb {
         Ok(new_dispatch_count)
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use std::collections::BTreeMap;
+
+    use time::macros::*;
+
+    use hyperlane_core::{HyperlaneMessage, LogMeta, H256};
+    use sea_orm::{DatabaseBackend, DbErr, MockDatabase, RuntimeErr, Value};
+    use time::PrimitiveDateTime;
+
+    use crate::db::{generated::message, ScraperDb, StorableMessage};
+
+    /// Tests store_dispatched_messages() a transaction works
+    #[tokio::test]
+    async fn test_store_dispatched_messages_transaction() {
+        let query_results: Vec<Vec<_>> = (0..10000)
+            .map(|i| message::Model {
+                id: i,
+                time_created: PrimitiveDateTime::new(date!(2019 - 01 - 01), time!(0:00)),
+                msg_id: vec![],
+                origin: 0,
+                destination: 0,
+                nonce: 0,
+                sender: vec![],
+                recipient: vec![],
+                msg_body: None,
+                origin_mailbox: vec![],
+                origin_tx_id: 0,
+            })
+            .collect::<Vec<_>>()
+            .chunks(ScraperDb::STORE_MESSAGE_CHUNK_SIZE)
+            .map(|v| v.to_vec())
+            .collect();
+
+        let mock_result: BTreeMap<&str, _> = [
+            ("num_items", Into::<Value>::into(10000i64)),
+            ("last_insert_id", Into::<Value>::into(10000i64)),
+        ]
+        .into_iter()
+        .collect();
+        let mock_db = MockDatabase::new(DatabaseBackend::Postgres)
path: rust/main/agents/scraper/src/db/message.rs
user: login: kamiyaa
body: Yeah, it doesn't create a postgres setup. Let me test it with a postgres setup
created_at: 2025-05-07T17:18:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -336,3 +360,136 @@ impl ScraperDb {
         Ok(new_dispatch_count)
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use std::collections::BTreeMap;
+
+    use time::macros::*;
+
+    use hyperlane_core::{HyperlaneMessage, LogMeta, H256};
+    use sea_orm::{DatabaseBackend, DbErr, MockDatabase, RuntimeErr, Value};
+    use time::PrimitiveDateTime;
+
+    use crate::db::{generated::message, ScraperDb, StorableMessage};
+
+    /// Tests store_dispatched_messages() a transaction works
+    #[tokio::test]
+    async fn test_store_dispatched_messages_transaction() {
+        let query_results: Vec<Vec<_>> = (0..10000)
+            .map(|i| message::Model {
+                id: i,
+                time_created: PrimitiveDateTime::new(date!(2019 - 01 - 01), time!(0:00)),
+                msg_id: vec![],
+                origin: 0,
+                destination: 0,
+                nonce: 0,
+                sender: vec![],
+                recipient: vec![],
+                msg_body: None,
+                origin_mailbox: vec![],
+                origin_tx_id: 0,
+            })
+            .collect::<Vec<_>>()
+            .chunks(ScraperDb::STORE_MESSAGE_CHUNK_SIZE)
+            .map(|v| v.to_vec())
+            .collect();
+
+        let mock_result: BTreeMap<&str, _> = [
+            ("num_items", Into::<Value>::into(10000i64)),
+            ("last_insert_id", Into::<Value>::into(10000i64)),
+        ]
+        .into_iter()
+        .collect();
+        let mock_db = MockDatabase::new(DatabaseBackend::Postgres)
path: rust/main/agents/scraper/src/db/message.rs
user: login: kamiyaa
body: Tried it locally with a postgres instance with 100k messages and it failed without chunking 
```
running 1 test
RES: Err(Query Error: encountered unexpected or invalid data: PgConnection::run(): too many arguments for query: 1000000 (sqlx_postgres::connection::executor:215)

Caused by:
   0: encountered unexpected or invalid data: PgConnection::run(): too many arguments for query: 1000000 (sqlx_postgres::connection::executor:215)
   1: encountered unexpected or invalid data: PgConnection::run(): too many arguments for query: 1000000 (sqlx_postgres::connection::executor:215)

Location:
    agents/scraper/src/db/message.rs:320:9)
test db::message::tests::test_store_dispatched_messages_real_postgres ... ok
```

Tried it again with chunking + transaction and it didn't have any errors :+1:
So we can use the transaction if we want the action to be atomic, but the important part is the chunking.

created_at: 2025-05-07T19:43:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -336,3 +360,136 @@ impl ScraperDb {
         Ok(new_dispatch_count)
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use std::collections::BTreeMap;
+
+    use time::macros::*;
+
+    use hyperlane_core::{HyperlaneMessage, LogMeta, H256};
+    use sea_orm::{DatabaseBackend, DbErr, MockDatabase, RuntimeErr, Value};
+    use time::PrimitiveDateTime;
+
+    use crate::db::{generated::message, ScraperDb, StorableMessage};
+
+    /// Tests store_dispatched_messages() a transaction works
+    #[tokio::test]
+    async fn test_store_dispatched_messages_transaction() {
+        let query_results: Vec<Vec<_>> = (0..10000)
+            .map(|i| message::Model {
+                id: i,
+                time_created: PrimitiveDateTime::new(date!(2019 - 01 - 01), time!(0:00)),
+                msg_id: vec![],
+                origin: 0,
+                destination: 0,
+                nonce: 0,
+                sender: vec![],
+                recipient: vec![],
+                msg_body: None,
+                origin_mailbox: vec![],
+                origin_tx_id: 0,
+            })
+            .collect::<Vec<_>>()
+            .chunks(ScraperDb::STORE_MESSAGE_CHUNK_SIZE)
+            .map(|v| v.to_vec())
+            .collect();
+
+        let mock_result: BTreeMap<&str, _> = [
+            ("num_items", Into::<Value>::into(10000i64)),
+            ("last_insert_id", Into::<Value>::into(10000i64)),
+        ]
+        .into_iter()
+        .collect();
+        let mock_db = MockDatabase::new(DatabaseBackend::Postgres)
path: rust/main/agents/scraper/src/db/message.rs
user: login: daniel-savu
body: nice! I updated the testing section of the PR description with this
created_at: 2025-05-09T09:47:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -336,3 +360,136 @@ impl ScraperDb {
         Ok(new_dispatch_count)
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use std::collections::BTreeMap;
+
+    use time::macros::*;
+
+    use hyperlane_core::{HyperlaneMessage, LogMeta, H256};
+    use sea_orm::{DatabaseBackend, DbErr, MockDatabase, RuntimeErr, Value};
+    use time::PrimitiveDateTime;
+
+    use crate::db::{generated::message, ScraperDb, StorableMessage};
+
+    /// Tests store_dispatched_messages() a transaction works
+    #[tokio::test]
+    async fn test_store_dispatched_messages_transaction() {
+        let query_results: Vec<Vec<_>> = (0..10000)
+            .map(|i| message::Model {
+                id: i,
+                time_created: PrimitiveDateTime::new(date!(2019 - 01 - 01), time!(0:00)),
+                msg_id: vec![],
+                origin: 0,
+                destination: 0,
+                nonce: 0,
+                sender: vec![],
+                recipient: vec![],
+                msg_body: None,
+                origin_mailbox: vec![],
+                origin_tx_id: 0,
+            })
+            .collect::<Vec<_>>()
+            .chunks(ScraperDb::STORE_MESSAGE_CHUNK_SIZE)
+            .map(|v| v.to_vec())
+            .collect();
+
+        let mock_result: BTreeMap<&str, _> = [
+            ("num_items", Into::<Value>::into(10000i64)),
+            ("last_insert_id", Into::<Value>::into(10000i64)),
+        ]
+        .into_iter()
+        .collect();
+        let mock_db = MockDatabase::new(DatabaseBackend::Postgres)
path: rust/main/agents/scraper/src/db/message.rs
user: login: kamiyaa
body: Thanks!
created_at: 2025-05-09T12:53:56Z
author_association: CONTRIBUTOR

PR comments:
user: login: changeset-bot[bot]
created_at: 2025-05-02T14:27:34Z
body: ###  ⚠️  No Changeset found

Latest commit: 75d6c9b8f0268f933d0967975c0538df604f9903

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. **If these changes should result in a version bump, you need to add a changeset.**

<details><summary>This PR includes no changesets</summary>

  When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

</details>

[Click here to learn what changesets are, and how to add one](https://github.com/changesets/changesets/blob/main/docs/adding-a-changeset.md).

[Click here if you're a maintainer who wants to add a changeset to this PR](https://github.com/hyperlane-xyz/hyperlane-monorepo/new/jeff/chunk?filename=.changeset/sour-wasps-knock.md&value=---%0A%0A---%0A%0Afeat%3A%20chunk%20message%20db%20insertions%0A)


author_association: NONE
user: login: ameten
created_at: 2025-05-05T09:24:13Z
body: I wonder how we can test that the query is executed inside a transaction?
author_association: CONTRIBUTOR
user: login: kamiyaa
created_at: 2025-05-05T14:18:08Z
body: > I wonder how we can test that the query is executed inside a transaction?

let me see if I can write a unittest for it :+1: 
author_association: CONTRIBUTOR
user: login: codecov[bot]
created_at: 2025-05-09T09:58:39Z
body: ## [Codecov](https://app.codecov.io/gh/hyperlane-xyz/hyperlane-monorepo/pull/6114?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperlane-xyz) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 0.00%. Comparing base [(`8fd2f9d`)](https://app.codecov.io/gh/hyperlane-xyz/hyperlane-monorepo/commit/8fd2f9d56b17e55c90bff5fd302afac3456d04cd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperlane-xyz) to head [(`75d6c9b`)](https://app.codecov.io/gh/hyperlane-xyz/hyperlane-monorepo/commit/75d6c9b8f0268f933d0967975c0538df604f9903?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperlane-xyz).
> Report is 35 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main   #6114       +/-   ##
==========================================
- Coverage   76.92%       0   -76.93%     
==========================================
  Files         109       0      -109     
  Lines        2158       0     -2158     
  Branches      195       0      -195     
==========================================
- Hits         1660       0     -1660     
+ Misses        477       0      -477     
+ Partials       21       0       -21     
```

| [Components](https://app.codecov.io/gh/hyperlane-xyz/hyperlane-monorepo/pull/6114/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperlane-xyz) | Coverage Δ | |
|---|---|---|
| [core](https://app.codecov.io/gh/hyperlane-xyz/hyperlane-monorepo/pull/6114/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperlane-xyz) | `∅ <ø> (∅)` | |
| [hooks](https://app.codecov.io/gh/hyperlane-xyz/hyperlane-monorepo/pull/6114/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperlane-xyz) | `∅ <ø> (∅)` | |
| [isms](https://app.codecov.io/gh/hyperlane-xyz/hyperlane-monorepo/pull/6114/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperlane-xyz) | `∅ <ø> (∅)` | |
| [token](https://app.codecov.io/gh/hyperlane-xyz/hyperlane-monorepo/pull/6114/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperlane-xyz) | `∅ <ø> (∅)` | |
| [middlewares](https://app.codecov.io/gh/hyperlane-xyz/hyperlane-monorepo/pull/6114/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperlane-xyz) | `∅ <ø> (∅)` | |

</details>
<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE