Repository: xmtp/libxmtp
Pull request #1816: fix: mark published and processed intents and processed instead of deleting them
Author: login: mchenani
Created: 2025-04-02T21:04:46Z | Merged: 2025-04-16T19:50:29Z

### Modify group intent handling to mark intents as processed instead of deleting them to improve debugging capabilities
* Adds new `Processed` state (value=5) to `IntentState` enum in [group_intent.rs](https://github.com/xmtp/libxmtp/pull/1816/files#diff-17a09cbc9886e242f0ff0189d4750dbfd8226be9344e7b41814967745cfe0a0c) and implements `set_group_intent_processed()` method
* Updates message processing logic in [mls_sync.rs](https://github.com/xmtp/libxmtp/pull/1816/files#diff-10f818f7918c8c0b2ed32c4f63e5060050527ed58df00ec081bbecb66065144e) to mark intents as processed rather than deleting them
* Adds comprehensive error handling tests in [mls.rs](https://github.com/xmtp/libxmtp/pull/1816/files#diff-3a24c3e76565487a710ac9863ac05160128f4f90892e07849b555a6de43a6e8f) and [stream_all.rs](https://github.com/xmtp/libxmtp/pull/1816/files#diff-4a3d32ed6005867d963b0eae023dc9c1fd8ffd9531413ba572f4287210a18cc1) for message streaming
* Improves error logging in [stream_messages.rs](https://github.com/xmtp/libxmtp/pull/1816/files#diff-ee8c2d1c90650351a2245875b62af09f2c152c29643200276da5743cd59424e7) with more detailed cursor and group information

#### 📍Where to Start
Start with the `IntentState` enum and `set_group_intent_processed()` method in [group_intent.rs](https://github.com/xmtp/libxmtp/pull/1816/files#diff-17a09cbc9886e242f0ff0189d4750dbfd8226be9344e7b41814967745cfe0a0c), which defines the new processed state, then move to [mls_sync.rs](https://github.com/xmtp/libxmtp/pull/1816/files#diff-10f818f7918c8c0b2ed32c4f63e5060050527ed58df00ec081bbecb66065144e) to see how this state is used in message processing.

----

_[Macroscope](https://app.macroscope.com) summarized 165f7f4._


Diff:
diff --git a/bindings_ffi/src/mls.rs b/bindings_ffi/src/mls.rs
index 7af3dc2621..02606627c6 100644
--- a/bindings_ffi/src/mls.rs
+++ b/bindings_ffi/src/mls.rs
@@ -4210,7 +4210,7 @@ mod tests {
 
         // alix published + processed group creation and name update
         assert_eq!(alix_provider.conn_ref().intents_published(), 2);
-        assert_eq!(alix_provider.conn_ref().intents_deleted(), 2);
+        assert_eq!(alix_provider.conn_ref().intents_processed(), 2);
 
         bo_group
             .conversation
diff --git a/xmtp_db/src/encrypted_store/group_intent.rs b/xmtp_db/src/encrypted_store/group_intent.rs
index 842ce5a194..7709a50daa 100644
--- a/xmtp_db/src/encrypted_store/group_intent.rs
+++ b/xmtp_db/src/encrypted_store/group_intent.rs
@@ -54,6 +54,7 @@ pub enum IntentState {
     Published = 2,
     Committed = 3,
     Error = 4,
+    Processed = 5,
 }
 
 #[derive(Queryable, Identifiable, PartialEq, Clone)]
@@ -257,6 +258,25 @@ impl DbConnection {
         Ok(())
     }
 
+    // Set the intent with the given ID to `Committed`
+    pub fn set_group_intent_processed(&self, intent_id: ID) -> Result<(), StorageError> {
+        let rows_changed = self.raw_query_write(|conn| {
+            diesel::update(dsl::group_intents)
+                .filter(dsl::id.eq(intent_id))
+                // State machine requires that the only valid state transition to Committed is from
+                // Published
+                .set(dsl::state.eq(IntentState::Processed))
+                .execute(conn)
+        })?;
+
+        // If nothing matched the query, return an error. Either ID or state was wrong
+        if rows_changed == 0 {
+            return Err(NotFound::IntentById(intent_id).into());
+        }
+
+        Ok(())
+    }
+
     // Set the intent with the given ID to `ToPublish`. Wipe any values for `payload_hash` and
     // `post_commit_data`
     pub fn set_group_intent_to_publish(&self, intent_id: ID) -> Result<(), StorageError> {
@@ -390,6 +410,7 @@ where
             2 => Ok(IntentState::Published),
             3 => Ok(IntentState::Committed),
             4 => Ok(IntentState::Error),
+            5 => Ok(IntentState::Processed),
             x => Err(format!("Unrecognized variant {}", x).into()),
         }
     }
diff --git a/xmtp_db/src/lib.rs b/xmtp_db/src/lib.rs
index 97b308e87c..e10cc3523a 100644
--- a/xmtp_db/src/lib.rs
+++ b/xmtp_db/src/lib.rs
@@ -49,6 +49,7 @@ pub mod test_util {
                     intents_created INT DEFAULT 0,
                     intents_published INT DEFAULT 0,
                     intents_deleted INT DEFAULT 0,
+                    intents_processed INT DEFAULT 0,
                     rowid integer PRIMARY KEY CHECK (rowid = 1) -- There can only be one meta
                 );
                 "#,
@@ -62,6 +63,12 @@ pub mod test_util {
                 BEGIN
                     UPDATE test_metadata SET intents_published = intents_published + 1;
                 END;"#,
+                r#"CREATE TRIGGER intents_processed_tracking AFTER UPDATE OF state ON group_intents
+                FOR EACH ROW
+                WHEN NEW.state = 5
+                BEGIN
+                    UPDATE test_metadata SET intents_processed = intents_processed + 1;
+                END;"#,
                 r#"CREATE TRIGGER intents_deleted_tracking AFTER DELETE ON group_intents
                 FOR EACH ROW
                 BEGIN
@@ -70,8 +77,9 @@ pub mod test_util {
                 r#"INSERT INTO test_metadata (
                     intents_created,
                     intents_deleted,
-                    intents_published
-                ) VALUES (0, 0,0);"#,
+                    intents_published,
+                    intents_processed
+                ) VALUES (0, 0, 0, 0);"#,
             ];
 
             for query in queries {
@@ -104,6 +112,22 @@ pub mod test_util {
             .unwrap()
         }
 
+        pub fn intents_processed(&self) -> i32 {
+            self.raw_query_read(|conn| {
+                let mut row = conn
+                    .load(sql_query(
+                        "SELECT intents_processed FROM test_metadata WHERE rowid = 1",
+                    ))
+                    .unwrap();
+                let row = row.next().unwrap().unwrap();
+                Ok::<_, StorageError>(
+                    <i32 as FromSqlRow<diesel::sql_types::Integer, _>>::build_from_row(&row)
+                        .unwrap(),
+                )
+            })
+            .unwrap()
+        }
+
         pub fn intents_deleted(&self) -> i32 {
             self.raw_query_read(|conn| {
                 let mut row = conn
diff --git a/xmtp_mls/src/groups/mls_sync.rs b/xmtp_mls/src/groups/mls_sync.rs
index 8c6844eea0..6ea0cb7b94 100644
--- a/xmtp_mls/src/groups/mls_sync.rs
+++ b/xmtp_mls/src/groups/mls_sync.rs
@@ -35,7 +35,7 @@ use xmtp_db::{
     refresh_state::EntityKind,
     sql_key_store,
     user_preferences::StoredUserPreferences,
-    Delete, Fetch, ProviderTransactions, StorageError, StoreOrIgnore,
+    Fetch, ProviderTransactions, StorageError, StoreOrIgnore,
 };
 use xmtp_db::{group::ConversationType, xmtp_openmls_provider::XmtpOpenMlsProvider};
 
@@ -342,6 +342,17 @@ where
                     );
                     return Err(last_err.unwrap_or(GroupError::IntentNotCommitted));
                 }
+                Ok(Some(StoredGroupIntent {
+                    id,
+                    state: IntentState::Processed,
+                    ..
+                })) => {
+                    tracing::warn!(
+                        "not retrying intent ID {id}. since it is in state processed. {:?}",
+                        last_err
+                    );
+                    return Ok(());
+                }
                 Ok(Some(StoredGroupIntent { id, state, .. })) => {
                     tracing::warn!("retrying intent ID {id}. intent currently in state {state:?}");
                 }
@@ -1014,6 +1025,10 @@ where
                                 tracing::warn!("Intent [{}] moved to error status", intent_id);
                                 Ok(provider.conn_ref().set_group_intent_error(intent_id)?)
                             }
+                            IntentState::Processed => {
+                                tracing::warn!("Intent [{}] moved to Processed status", intent_id);
+                                Ok(provider.conn_ref().set_group_intent_processed(intent_id)?)
+                            }
                         }
                     })
                 }).await?;
@@ -1208,7 +1223,7 @@ where
                     // If the error is retryable we cannot move on to the next message
                     // otherwise you can get into a forked group state.
                     if is_retryable {
-                        tracing::error!(
+                        tracing::info!(
                             error = %error_message,
                             "Aborting message processing for retryable error: {}",
                             error_message
@@ -1405,8 +1420,7 @@ where
                             installation_id = %self.client.installation_id(),
                             "Skipping intent because no publish data returned"
                         );
-                        let deleter: &dyn Delete<StoredGroupIntent, Key = i32> = provider.conn_ref();
-                        deleter.delete(intent.id)?;
+                        provider.conn_ref().set_group_intent_processed(intent.id)?
                     }
                 }
             }
@@ -1562,8 +1576,7 @@ where
                     }
                 }
             }
-            let deleter: &dyn Delete<StoredGroupIntent, Key = i32> = conn;
-            deleter.delete(intent.id)?;
+            conn.set_group_intent_processed(intent.id)?
         }
 
         Ok(())
@@ -2129,7 +2142,7 @@ pub(crate) mod tests {
 
         // create group intent
         amal_group_a.sync().await.unwrap();
-        assert_eq!(provider.conn_ref().intents_deleted(), 1);
+        assert_eq!(provider.conn_ref().intents_processed(), 1);
 
         for _ in 0..100 {
             let s = xmtp_common::rand_string::<100>();
diff --git a/xmtp_mls/src/groups/mod.rs b/xmtp_mls/src/groups/mod.rs
index 0adaa8b789..96e4ff1f43 100644
--- a/xmtp_mls/src/groups/mod.rs
+++ b/xmtp_mls/src/groups/mod.rs
@@ -2108,8 +2108,8 @@ pub(crate) mod tests {
     use prost::Message;
     use std::sync::Arc;
     use wasm_bindgen_test::wasm_bindgen_test;
-    use xmtp_common::assert_err;
     use xmtp_common::time::now_ns;
+    use xmtp_common::{assert_err, assert_ok};
     use xmtp_content_types::{group_updated::GroupUpdatedCodec, ContentCodec};
     use xmtp_cryptography::utils::generate_local_wallet;
     use xmtp_id::associations::test_utils::WalletTestExt;
@@ -4742,6 +4742,46 @@ pub(crate) mod tests {
             .any(|err| err.to_string().contains("already processed")));
     }
 
+    #[xmtp_common::test]
+    async fn skip_already_processed_intents() {
+        let alix = ClientBuilder::new_test_client(&generate_local_wallet()).await;
+
+        let bo_wallet = generate_local_wallet();
+        let bo_client = ClientBuilder::new_test_client(&bo_wallet).await;
+
+        let alix_group = alix
+            .create_group(None, GroupMetadataOptions::default())
+            .unwrap();
+
+        alix_group
+            .add_members_by_inbox_id(&[bo_client.inbox_id()])
+            .await
+            .unwrap();
+
+        bo_client
+            .sync_welcomes(&bo_client.mls_provider().unwrap())
+            .await
+            .unwrap();
+        let bo_groups = bo_client.find_groups(GroupQueryArgs::default()).unwrap();
+        let bo_group = bo_groups.first().unwrap();
+        bo_group.send_message(&[2]).await.unwrap();
+        let bo_provider = bo_client.mls_provider().unwrap();
+        let intent = bo_provider
+            .conn_ref()
+            .find_group_intents(
+                bo_group.clone().group_id,
+                Some(vec![IntentState::Processed]),
+                None,
+            )
+            .unwrap();
+        assert_eq!(intent.len(), 2); //key_update and send_message
+
+        let process_result = bo_group
+            .sync_until_intent_resolved(&bo_provider, intent[1].id)
+            .await;
+        assert_ok!(process_result);
+    }
+
     #[xmtp_common::test(flavor = "multi_thread")]
     async fn test_parallel_syncs() {
         let wallet = generate_local_wallet();
diff --git a/xmtp_mls/src/subscriptions/stream_messages.rs b/xmtp_mls/src/subscriptions/stream_messages.rs
index 68fa30bbd2..d353f7689e 100644
--- a/xmtp_mls/src/subscriptions/stream_messages.rs
+++ b/xmtp_mls/src/subscriptions/stream_messages.rs
@@ -403,7 +403,9 @@ where
                 cursor_id,
                 inbox_id = self.inbox_id(),
                 group_id = hex::encode(&self.msg.group_id),
-                "group message not found"
+                "no further processing for streamed message [{}] in group [{}]",
+                &cursor_id,
+                hex::encode(&self.msg.group_id),
             );
 
             Ok(None)


Reviews:
user: login: insipx
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T15:46:49Z
user: login: neekolas
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T15:29:53Z
user: login: mchenani
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T15:37:32Z
user: login: neekolas
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T19:35:56Z
user: login: cameronvoell
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T19:46:06Z

Review comments:
diff_hunk: @@ -5391,6 +5391,40 @@ mod tests {
         assert!(stream.is_closed());
     }
 
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 5)]
+    async fn test_stream_all_messages_error() {
path: bindings_ffi/src/mls.rs
user: login: insipx
body: if you write the tests in the style of tests in https://github.com/xmtp/libxmtp/blob/main/xmtp_mls/src/subscriptions/stream_all.rs, does it still reproduce?
created_at: 2025-04-03T15:46:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1381,8 +1396,7 @@ where
                             installation_id = %self.client.installation_id(),
                             "Skipping intent because no publish data returned"
                         );
-                        let deleter: &dyn Delete<StoredGroupIntent, Key = i32> = provider.conn_ref();
-                        deleter.delete(intent.id)?;
+                        provider.conn_ref().set_group_intent_processed(intent.id)?
path: xmtp_mls/src/groups/mls_sync.rs
user: login: neekolas
body: The one thing we should keep in mind is that the old workflow was idempotent, and the new one will return an error if the intent was already deleted.

How confident are we that this won't cause issues with concurrency/retries?
created_at: 2025-04-15T15:29:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1381,8 +1396,7 @@ where
                             installation_id = %self.client.installation_id(),
                             "Skipping intent because no publish data returned"
                         );
-                        let deleter: &dyn Delete<StoredGroupIntent, Key = i32> = provider.conn_ref();
-                        deleter.delete(intent.id)?;
+                        provider.conn_ref().set_group_intent_processed(intent.id)?
path: xmtp_mls/src/groups/mls_sync.rs
user: login: mchenani
body: Great question, as far as I checked the code we have two places that we get the intents to process them and we already handle the processed case there.
created_at: 2025-04-15T15:37:32Z
author_association: CONTRIBUTOR

PR comments:
user: login: macroscopeapp[bot]
created_at: 2025-04-14T17:19:19Z
body: ### Changes in 3dcf033

- Redesigned message processing in `xmtp_mls` by introducing a `Processed` state for group intents, implementing cursor-based duplicate message detection, and adding thread-safe intent processing with improved locking mechanisms
- Extended `StorageError` retryability in `xmtp_db` to include I/O errors, `OpenMlsStorage` errors, and platform-specific errors for both WASM and native environments
- Added streaming error test suite in `xmtp_mls` with multiple test groups, super admin operations, and complex message delivery scenarios
- Updated Rust dependencies including `anyhow`, `auto_impl`, `clap`, `diesel`, `h2`, `indexmap`, `openssl`, `rustls`, and added new packages `accessory`, `delegate-display`, `derive_more`, `fancy_constructor`


----

_[Macroscope](https://app.macroscope.com) summarized 3dcf033._
author_association: CONTRIBUTOR
user: login: macroscopeapp[bot]
created_at: 2025-04-15T13:00:19Z
body: ### Changes in 35269f2

- Refactored `xmtp_mls` message processing logic by splitting `MlsSync::process_message` into separate paths for client-originated and external messages, moving cursor validation to `MlsSync::consume_message`
- Modified intent handling in `xmtp_mls` to use `Fetch::<StoredGroupIntent>::fetch()` instead of `find_group_intent_by_intent_id` for retrieving group intents
- Removed unused imports from `xmtp_mls` including `Delete`, `LocalEventError`, `InvalidEpoch`, and added detailed logging for intent and message processing


----

_[Macroscope](https://app.macroscope.com) summarized 35269f2._
author_association: CONTRIBUTOR
user: login: macroscopeapp[bot]
created_at: 2025-04-15T13:02:11Z
body: ### Changes in 33f8b3e

- Removed `test_stream_all_messages_errors()` test function from `xmtp_mls` module that verified message streaming functionality through client creation, group management, and message reception scenarios


----

_[Macroscope](https://app.macroscope.com) summarized 33f8b3e._
author_association: CONTRIBUTOR
user: login: macroscopeapp[bot]
created_at: 2025-04-15T13:25:23Z
body: ### Changes in c8fabdb

- Refactored `test_stream_all_messages_error` in `MLS` test module by reducing test client instances from 6 to 1, decreasing group creation operations from 6 to 1, and removing multiple group operations including name updates, member additions, and message sending
- Modified logging level in `MLS` sync module from `tracing::error!` to `tracing::info!` for retryable message processing errors


----

_[Macroscope](https://app.macroscope.com) summarized c8fabdb._
author_association: CONTRIBUTOR
user: login: macroscopeapp[bot]
created_at: 2025-04-15T14:34:50Z
body: ### Changes in 82a2436

- Added test coverage in `xmtp_mls` to verify successful processing of already processed intents, including key updates and message sending scenarios
- Removed message streaming error handling test from `bindings_ffi`


----

_[Macroscope](https://app.macroscope.com) summarized 82a2436._
author_association: CONTRIBUTOR
user: login: macroscopeapp[bot]
created_at: 2025-04-15T14:36:50Z
body: ### Changes in bbbb055

- Reformatted test method calls `find_group_intents` and `sync_until_intent_resolved` in `xmtp_mls` groups test module to use multi-line formatting with proper indentation
- Reordered imports for `xmtp_common` time and assertion utilities in `xmtp_mls` groups test module


----

_[Macroscope](https://app.macroscope.com) summarized bbbb055._
author_association: CONTRIBUTOR
user: login: macroscopeapp[bot]
created_at: 2025-04-15T15:27:35Z
body: ### Changes in 076470c

- Removed packages `accessory`, `delegate-display`, `indexed_db_futures`, `macroific`, `sealed`, `derive_more v2.0.1`, `fancy_constructor`, and `impartial-ord` along with their dependencies from the project
- Downgraded multiple core dependencies including `anyhow`, `auto_impl`, `clap`, `data-encoding`, `h2`, `indexmap`, `openssl`, `rustls`, `smallvec`, `vergen`, and `winnow` to previous stable versions
- Updated `diesel` and related packages git reference from commit `90a2a1ac` to `1712c604`


----

_[Macroscope](https://app.macroscope.com) summarized 076470c._
author_association: CONTRIBUTOR
user: login: macroscopeapp[bot]
created_at: 2025-04-15T15:47:41Z
body: ### Changes in 7b4c142

- Replaced vector creation with slice literal in `xmtp_mls` module's `BoGroup.send_message()` test call, changing `&vec![2]` to `&[2]`


----

_[Macroscope](https://app.macroscope.com) summarized 7b4c142._
author_association: CONTRIBUTOR
user: login: macroscopeapp[bot]
created_at: 2025-04-15T16:36:27Z
body: ### Changes in f4d6254

- Modified intent tracking in `xmtp_db` to mark intents as published and processed instead of deleting them, adding new `intents_processed` column to `test_metadata` table and implementing corresponding tracking mechanisms
- Updated `publish_intents_worst_case_scenario` test in `xmtp_mls` groups module to verify processed intents count using `provider.conn_ref().intents_processed()` instead of checking deleted intents


----

_[Macroscope](https://app.macroscope.com) summarized f4d6254._
author_association: CONTRIBUTOR
user: login: macroscopeapp[bot]
created_at: 2025-04-15T16:42:18Z
body: ### Changes in c55d2a5

- Modified `bindings_ffi` test assertions to verify intents are marked as processed using `intents_processed()` instead of being deleted via `intents_deleted()`


----

_[Macroscope](https://app.macroscope.com) summarized c55d2a5._
author_association: CONTRIBUTOR
user: login: macroscopeapp[bot]
created_at: 2025-04-15T16:49:05Z
body: ### Changes in 176548d

- Added `intents_created_tracking` SQL trigger to `xmtp_db` crate that increments `intents_created` counter in `test_metadata` table after INSERT operations on `group_intents` table


----

_[Macroscope](https://app.macroscope.com) summarized 176548d._
author_association: CONTRIBUTOR