Repository: xmtp/libxmtp
Pull request #1802: Unable to remove member with invalid keyPackages.
Author: login: nplasterer
Created: 2025-03-27T23:26:59Z | Merged: 2025-03-28T20:23:17Z

### Fix member removal handling to properly manage invalid keyPackages in MLS group membership updates
The changes improve handling of failed installations in MLS group membership management across several files:

* Implement deduplication of failed installations using `HashSet` in [intents.rs](https://github.com/xmtp/libxmtp/pull/1802/files#diff-2288952982788551d8fa6f279ebf922ee9fb1b15222414137277d097ead95e35) and [mls_sync.rs](https://github.com/xmtp/libxmtp/pull/1802/files#diff-10f818f7918c8c0b2ed32c4f63e5060050527ed58df00ec081bbecb66065144e)
* Add proper filtering of failed installations during membership validation in [validated_commit.rs](https://github.com/xmtp/libxmtp/pull/1802/files#diff-779227110741fa6ce78048adfaac1895d306d8fd502dab800774b62aada72365)
* Introduce test cases for handling installations with bad/malformed key packages in [mod.rs](https://github.com/xmtp/libxmtp/pull/1802/files#diff-c29f56a38916c7410eff8091df1a2e43487ffe20646d96827e846e475f4608d3)

#### 📍Where to Start
Start with the `calculate_membership_changes_with_keypackages` function in [mls_sync.rs](https://github.com/xmtp/libxmtp/pull/1802/files#diff-10f818f7918c8c0b2ed32c4f63e5060050527ed58df00ec081bbecb66065144e) which contains the core logic for handling failed installations in group membership updates.

----

_[Macroscope](https://app.macroscope.com) summarized 71945d3._


Diff:
diff --git a/xmtp_mls/src/groups/intents.rs b/xmtp_mls/src/groups/intents.rs
index 8585bae0f8..92f01972b0 100644
--- a/xmtp_mls/src/groups/intents.rs
+++ b/xmtp_mls/src/groups/intents.rs
@@ -3,7 +3,7 @@ use openmls::prelude::{
     MlsMessageOut,
 };
 use prost::{bytes::Bytes, DecodeError, Message};
-use std::collections::HashMap;
+use std::collections::{HashMap, HashSet};
 use thiserror::Error;
 
 use xmtp_proto::xmtp::mls::database::{
@@ -363,9 +363,13 @@ impl UpdateGroupMembershipIntentData {
             new_membership.remove(inbox_id)
         }
 
-        new_membership
+        new_membership.failed_installations = new_membership
             .failed_installations
-            .extend(self.failed_installations.iter().cloned());
+            .into_iter()
+            .chain(self.failed_installations.iter().cloned())
+            .collect::<HashSet<_>>()
+            .into_iter()
+            .collect();
 
         tracing::info!("updated group membership: {:?}", new_membership.members);
         new_membership
diff --git a/xmtp_mls/src/groups/mls_sync.rs b/xmtp_mls/src/groups/mls_sync.rs
index f65a4838c6..8a5aea8945 100644
--- a/xmtp_mls/src/groups/mls_sync.rs
+++ b/xmtp_mls/src/groups/mls_sync.rs
@@ -1644,6 +1644,9 @@ where
             for (inbox_id, sequence_id) in changed_inbox_ids.iter() {
                 new_membership.add(inbox_id.clone(), *sequence_id);
             }
+            for inbox_id in inbox_ids_to_remove {
+                new_membership.remove(inbox_id);
+            }
 
             let changes_with_kps = calculate_membership_changes_with_keypackages(
                 &self.client,
@@ -1664,19 +1667,13 @@ where
                 ));
             }
 
-            let failed_installations = [
-                old_group_membership.failed_installations,
-                changes_with_kps.failed_installations,
-            ]
-            .concat();
-
             Ok(UpdateGroupMembershipIntentData::new(
                 changed_inbox_ids,
                 inbox_ids_to_remove
                     .iter()
                     .map(|s| s.to_string())
                     .collect::<Vec<String>>(),
-                failed_installations,
+                changes_with_kps.failed_installations,
             ))
         })
         .await
@@ -1836,7 +1833,7 @@ async fn calculate_membership_changes_with_keypackages<'a>(
 ) -> Result<MembershipDiffWithKeyPackages, GroupError> {
     let membership_diff = old_group_membership.diff(new_group_membership);
 
-    let installation_diff = client
+    let mut installation_diff = client
         .get_installation_diff(
             provider.conn_ref(),
             old_group_membership,
@@ -1847,13 +1844,13 @@ async fn calculate_membership_changes_with_keypackages<'a>(
 
     let mut new_installations = Vec::new();
     let mut new_key_packages = Vec::new();
-    let mut failed_installations = Vec::new();
+    let mut new_failed_installations = Vec::new();
 
     if !installation_diff.added_installations.is_empty() {
         let key_packages = get_keypackages_for_installation_ids(
             client,
             installation_diff.added_installations,
-            &mut failed_installations,
+            &mut new_failed_installations,
         )
         .await?;
         for (installation_id, result) in key_packages {
@@ -1864,11 +1861,32 @@ async fn calculate_membership_changes_with_keypackages<'a>(
                     ));
                     new_key_packages.push(verified_key_package.inner.clone());
                 }
-                Err(_) => failed_installations.push(installation_id.clone()),
+                Err(_) => new_failed_installations.push(installation_id.clone()),
             }
         }
     }
 
+    let mut failed_installations: Vec<Vec<u8>> = old_group_membership
+        .failed_installations
+        .clone()
+        .into_iter()
+        .chain(new_failed_installations)
+        .collect::<HashSet<_>>()
+        .into_iter()
+        .collect();
+
+    let common: HashSet<_> = failed_installations
+        .iter()
+        .filter(|item| installation_diff.removed_installations.contains(*item))
+        .cloned()
+        .collect();
+
+    failed_installations.retain(|item| !common.contains(item));
+
+    installation_diff
+        .removed_installations
+        .retain(|item| !common.contains(item));
+
     Ok(MembershipDiffWithKeyPackages::new(
         new_installations,
         new_key_packages,
@@ -1914,7 +1932,7 @@ async fn get_keypackages_for_installation_ids(
 async fn get_keypackages_for_installation_ids(
     client: impl ScopedGroupClient,
     added_installations: HashSet<Vec<u8>>,
-    failed_installations: &mut Vec<Vec<u8>>,
+    failed_installations: &mut [Vec<u8>],
 ) -> Result<HashMap<Vec<u8>, Result<VerifiedKeyPackageV2, KeyPackageVerificationError>>, ClientError>
 {
     let my_installation_id = client.context().installation_public_key().to_vec();
@@ -1941,7 +1959,7 @@ async fn apply_update_group_membership_intent(
 ) -> Result<Option<PublishIntentData>, GroupError> {
     let extensions: Extensions = openmls_group.extensions().clone();
     let old_group_membership = extract_group_membership(&extensions)?;
-    let mut new_group_membership = intent_data.apply_to_group_membership(&old_group_membership);
+    let new_group_membership = intent_data.apply_to_group_membership(&old_group_membership);
     let membership_diff = old_group_membership.diff(&new_group_membership);
 
     let changes_with_kps = calculate_membership_changes_with_keypackages(
@@ -1963,11 +1981,7 @@ async fn apply_update_group_membership_intent(
 
     // Update the extensions to have the new GroupMembership
     let mut new_extensions = extensions.clone();
-    new_group_membership.failed_installations = [
-        old_group_membership.failed_installations,
-        changes_with_kps.failed_installations,
-    ]
-    .concat();
+
     new_extensions.add_or_replace(build_group_membership_extension(&new_group_membership));
 
     // Create the commit
diff --git a/xmtp_mls/src/groups/mod.rs b/xmtp_mls/src/groups/mod.rs
index e29026c6cf..b50c6b5e2c 100644
--- a/xmtp_mls/src/groups/mod.rs
+++ b/xmtp_mls/src/groups/mod.rs
@@ -2893,6 +2893,257 @@ pub(crate) mod tests {
         );
     }
 
+    #[cfg(not(target_arch = "wasm32"))]
+    #[tokio::test(flavor = "current_thread")]
+    async fn test_add_inbox_with_bad_installation_to_group() {
+        use crate::utils::set_test_mode_upload_malformed_keypackage;
+        use xmtp_id::associations::test_utils::WalletTestExt;
+
+        let alix = ClientBuilder::new_test_client(&generate_local_wallet()).await;
+        let bo_wallet = generate_local_wallet();
+        let caro_wallet = generate_local_wallet();
+        let bo_1 = ClientBuilder::new_test_client(&bo_wallet).await;
+        let bo_2 = ClientBuilder::new_test_client(&bo_wallet).await;
+        let caro = ClientBuilder::new_test_client(&caro_wallet).await;
+
+        set_test_mode_upload_malformed_keypackage(
+            true,
+            Some(vec![bo_1.installation_id().to_vec()]),
+        );
+
+        let group = alix
+            .create_group_with_members(
+                &[caro_wallet.identifier()],
+                None,
+                GroupMetadataOptions::default(),
+            )
+            .await
+            .unwrap();
+
+        let _ = group.add_members(&[bo_wallet.identifier()]).await;
+
+        bo_2.sync_welcomes(&bo_2.mls_provider().unwrap())
+            .await
+            .unwrap();
+        caro.sync_welcomes(&caro.mls_provider().unwrap())
+            .await
+            .unwrap();
+
+        let bo_2_groups = bo_2.find_groups(GroupQueryArgs::default()).unwrap();
+        assert_eq!(bo_2_groups.len(), 1);
+        let caro_groups = caro.find_groups(GroupQueryArgs::default()).unwrap();
+        assert_eq!(caro_groups.len(), 1);
+        let alix_groups = alix.find_groups(GroupQueryArgs::default()).unwrap();
+        assert_eq!(alix_groups.len(), 1);
+    }
+
+    #[cfg(not(target_arch = "wasm32"))]
+    #[tokio::test(flavor = "current_thread")]
+    async fn test_add_inbox_with_good_installation_to_group_with_bad_installation() {
+        use crate::utils::set_test_mode_upload_malformed_keypackage;
+        use xmtp_id::associations::test_utils::WalletTestExt;
+
+        let bo_wallet = generate_local_wallet();
+        let caro_wallet = generate_local_wallet();
+        let alix = ClientBuilder::new_test_client(&generate_local_wallet()).await;
+        let bo_1 = ClientBuilder::new_test_client(&bo_wallet).await;
+        let bo_2 = ClientBuilder::new_test_client(&bo_wallet).await;
+        let caro = ClientBuilder::new_test_client(&caro_wallet).await;
+
+        set_test_mode_upload_malformed_keypackage(
+            true,
+            Some(vec![bo_1.installation_id().to_vec()]),
+        );
+
+        let group = alix
+            .create_group_with_members(
+                &[bo_wallet.identifier()],
+                None,
+                GroupMetadataOptions::default(),
+            )
+            .await
+            .unwrap();
+
+        let _ = group.add_members(&[caro_wallet.identifier()]).await;
+
+        caro.sync_welcomes(&caro.mls_provider().unwrap())
+            .await
+            .unwrap();
+        bo_2.sync_welcomes(&bo_2.mls_provider().unwrap())
+            .await
+            .unwrap();
+        let caro_groups = caro.find_groups(GroupQueryArgs::default()).unwrap();
+        assert_eq!(caro_groups.len(), 1);
+        let bo_groups = bo_2.find_groups(GroupQueryArgs::default()).unwrap();
+        assert_eq!(bo_groups.len(), 1);
+        let alix_groups = alix.find_groups(GroupQueryArgs::default()).unwrap();
+        assert_eq!(alix_groups.len(), 1);
+    }
+
+    #[cfg(not(target_arch = "wasm32"))]
+    #[tokio::test(flavor = "current_thread")]
+    async fn test_remove_inbox_with_good_installation_from_group_with_bad_installation() {
+        use crate::utils::set_test_mode_upload_malformed_keypackage;
+        use xmtp_id::associations::test_utils::WalletTestExt;
+
+        let alix_wallet = generate_local_wallet();
+        let bo_wallet = generate_local_wallet();
+        let caro_wallet = generate_local_wallet();
+        let alix_1 = ClientBuilder::new_test_client(&alix_wallet).await;
+        let alix_2 = ClientBuilder::new_test_client(&alix_wallet).await;
+        let bo = ClientBuilder::new_test_client(&bo_wallet).await;
+        let caro = ClientBuilder::new_test_client(&caro_wallet).await;
+
+        set_test_mode_upload_malformed_keypackage(
+            true,
+            Some(vec![alix_2.installation_id().to_vec()]),
+        );
+
+        let group = alix_1
+            .create_group_with_members(
+                &[bo_wallet.identifier(), caro_wallet.identifier()],
+                None,
+                GroupMetadataOptions::default(),
+            )
+            .await
+            .unwrap();
+
+        assert_eq!(group.members().await.unwrap().len(), 3);
+        let _ = group.remove_members(&[caro_wallet.identifier()]).await;
+
+        caro.sync_welcomes(&caro.mls_provider().unwrap())
+            .await
+            .unwrap();
+        bo.sync_welcomes(&bo.mls_provider().unwrap()).await.unwrap();
+        group.sync().await.unwrap();
+
+        let caro_groups = caro.find_groups(GroupQueryArgs::default()).unwrap();
+        let caro_group = caro_groups.first().unwrap();
+        caro_group.sync().await.unwrap();
+        assert!(!caro_group.is_active(&caro.mls_provider().unwrap()).unwrap());
+        let bo_groups = bo.find_groups(GroupQueryArgs::default()).unwrap();
+        let bo_group = bo_groups.first().unwrap();
+        bo_group.sync().await.unwrap();
+        assert_eq!(bo_group.members().await.unwrap().len(), 2);
+        assert_eq!(group.members().await.unwrap().len(), 2);
+    }
+
+    #[cfg(not(target_arch = "wasm32"))]
+    #[tokio::test(flavor = "current_thread")]
+    async fn test_remove_inbox_with_bad_installation_from_group() {
+        use crate::utils::set_test_mode_upload_malformed_keypackage;
+        use xmtp_id::associations::test_utils::WalletTestExt;
+
+        let alix_wallet = generate_local_wallet();
+        let bo_wallet = generate_local_wallet();
+        let caro_wallet = generate_local_wallet();
+        let alix = ClientBuilder::new_test_client(&alix_wallet).await;
+        let bo_1 = ClientBuilder::new_test_client(&bo_wallet).await;
+        let bo_2 = ClientBuilder::new_test_client(&bo_wallet).await;
+        let caro = ClientBuilder::new_test_client(&caro_wallet).await;
+
+        set_test_mode_upload_malformed_keypackage(
+            true,
+            Some(vec![bo_1.installation_id().to_vec()]),
+        );
+
+        let group = alix
+            .create_group_with_members(
+                &[bo_wallet.identifier(), caro_wallet.identifier()],
+                None,
+                GroupMetadataOptions::default(),
+            )
+            .await
+            .unwrap();
+
+        group.sync().await.unwrap();
+
+        let message_from_alix = b"Hello from Alix";
+        group.send_message(message_from_alix).await.unwrap();
+
+        bo_2.sync_welcomes(&bo_2.mls_provider().unwrap())
+            .await
+            .unwrap();
+        caro.sync_welcomes(&caro.mls_provider().unwrap())
+            .await
+            .unwrap();
+        group.sync().await.unwrap();
+
+        let bo_groups = bo_2.find_groups(GroupQueryArgs::default()).unwrap();
+        let bo_group = bo_groups.first().unwrap();
+        bo_group.sync().await.unwrap();
+        let bo_msgs = bo_group.find_messages(&MsgQueryArgs::default()).unwrap();
+        assert_eq!(bo_msgs.len(), 1);
+        assert_eq!(bo_msgs[0].decrypted_message_bytes, message_from_alix);
+
+        let caro_groups = caro.find_groups(GroupQueryArgs::default()).unwrap();
+        let caro_group = caro_groups.first().unwrap();
+        caro_group.sync().await.unwrap();
+        let caro_msgs = caro_group.find_messages(&MsgQueryArgs::default()).unwrap();
+        assert_eq!(caro_msgs.len(), 1);
+        assert_eq!(caro_msgs[0].decrypted_message_bytes, message_from_alix);
+
+        // Bo replies before removal
+        let bo_reply = b"Hey Alix!";
+        bo_group.send_message(bo_reply).await.unwrap();
+
+        group.sync().await.unwrap();
+        let group_msgs = group.find_messages(&MsgQueryArgs::default()).unwrap();
+        assert_eq!(group_msgs.len(), 3);
+        assert_eq!(group_msgs.last().unwrap().decrypted_message_bytes, bo_reply);
+
+        // Remove Bo
+        group
+            .remove_members(&[bo_wallet.identifier()])
+            .await
+            .unwrap();
+
+        bo_2.sync_welcomes(&bo_2.mls_provider().unwrap())
+            .await
+            .unwrap();
+        caro.sync_welcomes(&caro.mls_provider().unwrap())
+            .await
+            .unwrap();
+        group.sync().await.unwrap();
+
+        // Bo should no longer be active
+        bo_group.sync().await.unwrap();
+        assert!(!bo_group.is_active(&bo_2.mls_provider().unwrap()).unwrap());
+
+        let post_removal_msg = b"Caro, just us now!";
+        group.send_message(post_removal_msg).await.unwrap();
+        let caro_post_removal_msg = b"Nice!";
+        caro_group
+            .send_message(caro_post_removal_msg)
+            .await
+            .unwrap();
+
+        caro_group.sync().await.unwrap();
+        let caro_msgs = caro_group.find_messages(&MsgQueryArgs::default()).unwrap();
+        assert_eq!(caro_msgs.len(), 5);
+        assert_eq!(
+            caro_msgs.last().unwrap().decrypted_message_bytes,
+            caro_post_removal_msg
+        );
+        group.sync().await.unwrap();
+        let alix_msgs = group.find_messages(&MsgQueryArgs::default()).unwrap();
+        assert_eq!(alix_msgs.len(), 6);
+        assert_eq!(
+            alix_msgs.last().unwrap().decrypted_message_bytes,
+            caro_post_removal_msg
+        );
+
+        let bo_msgs = bo_group.find_messages(&MsgQueryArgs::default()).unwrap();
+        assert_eq!(
+            bo_msgs.len(),
+            3,
+            "Bo should not receive messages after being removed"
+        );
+
+        assert_eq!(caro_group.members().await.unwrap().len(), 2);
+        assert_eq!(group.members().await.unwrap().len(), 2);
+    }
+
     #[xmtp_common::test]
     async fn test_add_invalid_member() {
         let client = ClientBuilder::new_test_client(&generate_local_wallet()).await;
@@ -5434,4 +5685,92 @@ pub(crate) mod tests {
             .await;
         assert!(result.is_ok());
     }
+
+    #[cfg(not(target_arch = "wasm32"))]
+    #[tokio::test(flavor = "multi_thread")]
+    async fn test_can_make_inbox_with_a_bad_key_package_an_admin() {
+        use crate::utils::set_test_mode_upload_malformed_keypackage;
+
+        // 1) Prepare clients
+        let amal = ClientBuilder::new_test_client(&generate_local_wallet()).await;
+        let charlie = ClientBuilder::new_test_client(&generate_local_wallet()).await;
+
+        // Create a wallet for the user with a bad key package
+        let bola_wallet = generate_local_wallet();
+        let bola = ClientBuilder::new_test_client(&bola_wallet).await;
+        // Mark bola's installation as having a malformed key package
+        set_test_mode_upload_malformed_keypackage(
+            true,
+            Some(vec![bola.installation_id().to_vec()]),
+        );
+
+        // 2) Create a group with amal as the only member
+        let amal_group = amal
+            .create_group(
+                Some(PreconfiguredPolicies::AdminsOnly.to_policy_set()),
+                GroupMetadataOptions::default(),
+            )
+            .unwrap();
+        amal_group.sync().await.unwrap();
+
+        // 3) Add charlie to the group (normal member)
+        let result = amal_group
+            .add_members_by_inbox_id(&[charlie.inbox_id()])
+            .await;
+        assert!(result.is_ok());
+
+        // 4) Initially fail to add bola since they only have one bad key package
+        let result = amal_group.add_members_by_inbox_id(&[bola.inbox_id()]).await;
+        assert!(result.is_err());
+
+        // 5) Add a second installation for bola and try and re-add them
+        let bola_2 = ClientBuilder::new_test_client(&bola_wallet).await;
+        let result = amal_group.add_members_by_inbox_id(&[bola.inbox_id()]).await;
+        assert!(result.is_ok());
+
+        // 6) Test that bola can not perform an admin only action
+        bola_2
+            .sync_welcomes(&bola_2.mls_provider().unwrap())
+            .await
+            .unwrap();
+        let binding = bola_2.find_groups(GroupQueryArgs::default()).unwrap();
+        let bola_group = binding.first().unwrap();
+        bola_group.sync().await.unwrap();
+        let result = bola_group
+            .update_group_name("Bola's Group".to_string())
+            .await;
+        assert!(result.is_err());
+
+        // 7) Test adding bola as an admin
+        let result = amal_group
+            .update_admin_list(UpdateAdminListType::Add, bola.inbox_id().to_string())
+            .await;
+        assert!(result.is_ok());
+
+        // 8) Verify bola can perform an admin only action
+        bola_group.sync().await.unwrap();
+        let result = bola_group
+            .update_group_name("Bola's Group".to_string())
+            .await;
+        assert!(result.is_ok());
+
+        // 9) Verify we can remove bola as an admin
+        let result = amal_group
+            .update_admin_list(UpdateAdminListType::Remove, bola.inbox_id().to_string())
+            .await;
+        assert!(result.is_ok());
+
+        // 10) Verify bola is not an admin
+        let admins = amal_group
+            .admin_list(&amal_group.mls_provider().unwrap())
+            .unwrap();
+        assert!(!admins.contains(&bola.inbox_id().to_string()));
+
+        // 11) verify bola can't perform an admin only action
+        bola_group.sync().await.unwrap();
+        let result = bola_group
+            .update_group_name("Bola's Group Forever".to_string())
+            .await;
+        assert!(result.is_err());
+    }
 }
diff --git a/xmtp_mls/src/groups/validated_commit.rs b/xmtp_mls/src/groups/validated_commit.rs
index 2efa1ba622..e751015b9e 100644
--- a/xmtp_mls/src/groups/validated_commit.rs
+++ b/xmtp_mls/src/groups/validated_commit.rs
@@ -396,6 +396,7 @@ impl ValidatedCommit {
             added_installations,
             removed_installations,
             current_group_members,
+            &new_group_membership.failed_installations,
         )?;
         credentials_to_verify.push(actor.clone());
 
@@ -669,6 +670,7 @@ fn expected_diff_matches_commit(
     added_installations: HashSet<Vec<u8>>,
     removed_installations: HashSet<Vec<u8>>,
     existing_installation_ids: HashSet<Vec<u8>>,
+    failed_installation_ids: &[Vec<u8>],
 ) -> Result<(), CommitValidationError> {
     // Check and make sure that any added installations are either:
     // 1. In the expected diff
@@ -687,12 +689,21 @@ fn expected_diff_matches_commit(
         ));
     }
 
-    if removed_installations.ne(&expected_diff.removed_installations) {
+    let filtered_expected: HashSet<_> = expected_diff
+        .removed_installations
+        .iter()
+        .filter(|id| !failed_installation_ids.contains(*id))
+        .cloned()
+        .collect();
+
+    if removed_installations != filtered_expected {
+        let unexpected: Vec<_> = removed_installations
+            .difference(&expected_diff.removed_installations)
+            .cloned()
+            .collect();
+
         return Err(CommitValidationError::UnexpectedInstallationsRemoved(
-            removed_installations
-                .difference(&expected_diff.removed_installations)
-                .cloned()
-                .collect::<Vec<Vec<u8>>>(),
+            unexpected,
         ));
     }
 
diff --git a/xmtp_mls/src/identity_updates.rs b/xmtp_mls/src/identity_updates.rs
index 8dd682c674..c0880797bb 100644
--- a/xmtp_mls/src/identity_updates.rs
+++ b/xmtp_mls/src/identity_updates.rs
@@ -623,7 +623,7 @@ pub(crate) mod tests {
         builder::ClientBuilder,
         groups::group_membership::GroupMembership,
         storage::{db_connection::DbConnection, identity_update::StoredIdentityUpdate},
-        utils::test::FullXmtpClient,
+        utils::{set_test_mode_upload_malformed_keypackage, test::FullXmtpClient},
         Client, XmtpApi,
     };
     use xmtp_common::rand_vec;
@@ -1022,6 +1022,71 @@ pub(crate) mod tests {
         assert_eq!(association_state.installation_ids().len(), 1);
     }
 
+    #[cfg(not(target_arch = "wasm32"))]
+    #[tokio::test(flavor = "multi_thread")]
+    pub async fn revoke_installation_with_malformed_keypackage() {
+        let wallet = generate_local_wallet();
+        let client1: FullXmtpClient = ClientBuilder::new_test_client(&wallet).await;
+        let client2: FullXmtpClient = ClientBuilder::new_test_client(&wallet).await;
+
+        let association_state = get_association_state(&client1, client1.inbox_id()).await;
+        // Ensure there are two installations on the inbox
+        assert_eq!(association_state.installation_ids().len(), 2);
+
+        set_test_mode_upload_malformed_keypackage(
+            true,
+            Some(vec![client2.installation_public_key().to_vec()]),
+        );
+
+        // Now revoke the second client
+        let mut revoke_installation_request = client1
+            .revoke_installations(vec![client2.installation_public_key().to_vec()])
+            .await
+            .unwrap();
+        add_wallet_signature(&mut revoke_installation_request, &wallet).await;
+        client1
+            .apply_signature_request(revoke_installation_request)
+            .await
+            .unwrap();
+
+        // Make sure there is only one installation on the inbox
+        let association_state = get_association_state(&client1, client1.inbox_id()).await;
+        assert_eq!(association_state.installation_ids().len(), 1);
+    }
+
+    #[cfg(not(target_arch = "wasm32"))]
+    #[tokio::test(flavor = "multi_thread")]
+    pub async fn revoke_good_installation_with_other_malformed_keypackage() {
+        let wallet = generate_local_wallet();
+        let client1: FullXmtpClient = ClientBuilder::new_test_client(&wallet).await;
+        let client2: FullXmtpClient = ClientBuilder::new_test_client(&wallet).await;
+        let client3: FullXmtpClient = ClientBuilder::new_test_client(&wallet).await;
+
+        let association_state = get_association_state(&client1, client1.inbox_id()).await;
+        // Ensure there are two installations on the inbox
+        assert_eq!(association_state.installation_ids().len(), 3);
+
+        set_test_mode_upload_malformed_keypackage(
+            true,
+            Some(vec![client2.installation_public_key().to_vec()]),
+        );
+
+        // Now revoke the second client
+        let mut revoke_installation_request = client1
+            .revoke_installations(vec![client3.installation_public_key().to_vec()])
+            .await
+            .unwrap();
+        add_wallet_signature(&mut revoke_installation_request, &wallet).await;
+        client1
+            .apply_signature_request(revoke_installation_request)
+            .await
+            .unwrap();
+
+        // Make sure there is only one installation on the inbox
+        let association_state = get_association_state(&client1, client1.inbox_id()).await;
+        assert_eq!(association_state.installation_ids().len(), 2);
+    }
+
     #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
     #[cfg_attr(not(target_arch = "wasm32"), tokio::test)]
     pub async fn change_recovery_address() {


Reviews:
user: login: mchenani
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T18:50:45Z
user: login: mchenani
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T18:52:40Z
user: login: codabrink
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T19:32:54Z
user: login: mchenani
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T19:35:33Z
user: login: mchenani
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T19:36:56Z
user: login: codabrink
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T19:37:17Z
user: login: codabrink
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T19:49:22Z
user: login: codabrink
body: Other than suggestions, looks good. We don't have to do the hashset conversion now, but it does look like we're doing some gymnastics to keep the vec.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T19:50:15Z

Review comments:
diff_hunk: @@ -1644,6 +1644,9 @@ where
             for (inbox_id, sequence_id) in changed_inbox_ids.iter() {
                 new_membership.add(inbox_id.clone(), *sequence_id);
             }
+            for inbox_id in inbox_ids_to_remove {
path: xmtp_mls/src/groups/mls_sync.rs
user: login: mchenani
body: fyi: put the removed members here to include them in the calculation always.
created_at: 2025-03-28T18:50:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1864,11 +1861,32 @@ async fn calculate_membership_changes_with_keypackages<'a>(
                     ));
                     new_key_packages.push(verified_key_package.inner.clone());
                 }
-                Err(_) => failed_installations.push(installation_id.clone()),
+                Err(_) => new_failed_installations.push(installation_id.clone()),
             }
         }
     }
 
+    let mut failed_installations: Vec<Vec<u8>> = old_group_membership
+        .failed_installations
+        .clone()
+        .into_iter()
+        .chain(new_failed_installations)
+        .collect::<HashSet<_>>()
+        .into_iter()
+        .collect();
+
+    let common: HashSet<_> = failed_installations
path: xmtp_mls/src/groups/mls_sync.rs
user: login: mchenani
body: Failed installations and removed installations may have some overlap. We can safely remove these shared entries from both sets—since they should no longer be in the failed installations, and they were never actually added, it’s fine to remove them here.

created_at: 2025-03-28T18:52:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -363,9 +363,13 @@ impl UpdateGroupMembershipIntentData {
             new_membership.remove(inbox_id)
         }
 
-        new_membership
+        new_membership.failed_installations = new_membership
path: xmtp_mls/src/groups/intents.rs
user: login: codabrink
body: Can we just convert this field to a HashSet?
created_at: 2025-03-28T19:32:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -363,9 +363,13 @@ impl UpdateGroupMembershipIntentData {
             new_membership.remove(inbox_id)
         }
 
-        new_membership
+        new_membership.failed_installations = new_membership
path: xmtp_mls/src/groups/intents.rs
user: login: mchenani
body: let me try, it's everywhere!
created_at: 2025-03-28T19:35:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -363,9 +363,13 @@ impl UpdateGroupMembershipIntentData {
             new_membership.remove(inbox_id)
         }
 
-        new_membership
+        new_membership.failed_installations = new_membership
path: xmtp_mls/src/groups/intents.rs
user: login: mchenani
body: I have to update protos as well! I would say let's have it for later. wdyt?
created_at: 2025-03-28T19:36:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1864,11 +1861,32 @@ async fn calculate_membership_changes_with_keypackages<'a>(
                     ));
                     new_key_packages.push(verified_key_package.inner.clone());
                 }
-                Err(_) => failed_installations.push(installation_id.clone()),
+                Err(_) => new_failed_installations.push(installation_id.clone()),
             }
         }
     }
 
+    let mut failed_installations: Vec<Vec<u8>> = old_group_membership
+        .failed_installations
+        .clone()
+        .into_iter()
+        .chain(new_failed_installations)
+        .collect::<HashSet<_>>()
+        .into_iter()
+        .collect();
+
+    let common: HashSet<_> = failed_installations
path: xmtp_mls/src/groups/mls_sync.rs
user: login: codabrink
body: Can we add something to this tune as an inline comment? :point_right::point_left: :smile: 
created_at: 2025-03-28T19:37:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1864,11 +1861,32 @@ async fn calculate_membership_changes_with_keypackages<'a>(
                     ));
                     new_key_packages.push(verified_key_package.inner.clone());
                 }
-                Err(_) => failed_installations.push(installation_id.clone()),
+                Err(_) => new_failed_installations.push(installation_id.clone()),
             }
         }
     }
 
+    let mut failed_installations: Vec<Vec<u8>> = old_group_membership
+        .failed_installations
+        .clone()
+        .into_iter()
+        .chain(new_failed_installations)
+        .collect::<HashSet<_>>()
+        .into_iter()
+        .collect();
+
+    let common: HashSet<_> = failed_installations
path: xmtp_mls/src/groups/mls_sync.rs
user: login: codabrink
body: And would you be okay with renaming this to something more descriptive like `failed_and_removed_installations` or `failed_removed_overlap`?
created_at: 2025-03-28T19:49:22Z
author_association: CONTRIBUTOR

PR comments:
user: login: cameronvoell
created_at: 2025-03-28T04:57:29Z
body: @nplasterer Just noting on here that I verified that adding the user with the bad key package as an admin works fine. See branch: https://github.com/xmtp/libxmtp/compare/np/unable-to-remove-member...cv/bad-key-package-tests?expand=1

@mchenani Also here is a log of the error we get when we try and remove the user with the bad key package (`Error validating commit for own message. Intent ID [5]: UnexpectedInstallationsRemoved([])`):

```
2025-03-28T04:48:28.257515Z  INFO sync_until_intent_resolved:sync_with_conn: xmtp_mls::groups::validated_commit: Group context extensions proposal found: GroupMembership { members: {"1c8f907e4dcb709382178f8b0d00f8ddf4624f893b20fd99fa51be1cb359815f": 143376}, failed_installations: [[132, 236, 125, 170, 154, 155, 106, 170, 47, 207, 157, 107, 36, 112, 42, 169, 103, 113, 99, 7, 162, 187, 166, 202, 221, 11, 30, 165, 224, 12, 198, 119], [243, 139, 15, 82, 195, 179, 91, 103, 219, 93, 8, 170, 142, 174, 0, 39, 139, 171, 76, 105, 13, 232, 250, 209, 235, 118, 9, 111, 116, 173, 89, 221], [65, 174, 205, 155, 107, 181, 125, 236, 138, 246, 98, 194, 168, 235, 127, 219, 64, 199, 219, 61, 221, 66, 197, 143, 146, 166, 85, 104, 211, 235, 64, 119]] }
2025-03-28T04:48:28.257570Z  INFO sync_until_intent_resolved:sync_with_conn: xmtp_mls::identity_updates: Getting installation diff. Old: GroupMembership { members: {"1c8f907e4dcb709382178f8b0d00f8ddf4624f893b20fd99fa51be1cb359815f": 143376, "f87420435131ea1b911ad66fbe4b626b107f81955da023d049f8aef6636b8e1b": 118148}, failed_installations: [[132, 236, 125, 170, 154, 155, 106, 170, 47, 207, 157, 107, 36, 112, 42, 169, 103, 113, 99, 7, 162, 187, 166, 202, 221, 11, 30, 165, 224, 12, 198, 119], [243, 139, 15, 82, 195, 179, 91, 103, 219, 93, 8, 170, 142, 174, 0, 39, 139, 171, 76, 105, 13, 232, 250, 209, 235, 118, 9, 111, 116, 173, 89, 221], [65, 174, 205, 155, 107, 181, 125, 236, 138, 246, 98, 194, 168, 235, 127, 219, 64, 199, 219, 61, 221, 66, 197, 143, 146, 166, 85, 104, 211, 235, 64, 119]] }. New GroupMembership { members: {"1c8f907e4dcb709382178f8b0d00f8ddf4624f893b20fd99fa51be1cb359815f": 143376}, failed_installations: [[132, 236, 125, 170, 154, 155, 106, 170, 47, 207, 157, 107, 36, 112, 42, 169, 103, 113, 99, 7, 162, 187, 166, 202, 221, 11, 30, 165, 224, 12, 198, 119], [243, 139, 15, 82, 195, 179, 91, 103, 219, 93, 8, 170, 142, 174, 0, 39, 139, 171, 76, 105, 13, 232, 250, 209, 235, 118, 9, 111, 116, 173, 89, 221], [65, 174, 205, 155, 107, 181, 125, 236, 138, 246, 98, 194, 168, 235, 127, 219, 64, 199, 219, 61, 221, 66, 197, 143, 146, 166, 85, 104, 211, 235, 64, 119]] }
2025-03-28T04:48:28.257994Z ERROR sync_until_intent_resolved:sync_with_conn: xmtp_mls::groups::mls_sync: Error validating commit for own message. Intent ID [5]: UnexpectedInstallationsRemoved([])
2025-03-28T04:48:28.258143Z  WARN sync_until_intent_resolved:sync_with_conn: xmtp_mls::groups::mls_sync: Intent [5] moved to error status
2025-03-28T04:48:28.258984Z  INFO sync_until_intent_resolved:sync_with_conn: xmtp_mls::groups::mls_sync: Transaction completed successfully: process for group [4b42a03e313c76794bee6f8944d8c1c2] envelope cursor[1051380]
2025-03-28T04:48:28.259250Z  WARN sync_until_intent_resolved: xmtp_mls::groups::mls_sync: not retrying intent ID 5. since it is in state Error. None
```
author_association: CONTRIBUTOR
user: login: macroscopeapp[bot]
created_at: 2025-03-28T18:55:29Z
body: ### Changes in 37e61ac

- Adjusted indentation in documentation comment for `envelope` parameter within `xmtp_mls::groups::prepare_message` method


----

_[Macroscope](https://app.macroscope.com) summarized 37e61ac._
author_association: CONTRIBUTOR
user: login: macroscopeapp[bot]
created_at: 2025-03-28T19:15:43Z
body: ### Changes in 8851824

- Refactored `xmtp_mls` group synchronization installation difference handling to replace manual iteration with `retain` method for filtering common elements from `installation_diff.removed_installations`


----

_[Macroscope](https://app.macroscope.com) summarized 8851824._
author_association: CONTRIBUTOR
user: login: macroscopeapp[bot]
created_at: 2025-03-28T19:25:36Z
body: ### Changes in 287d921

- Added test suite in `xmtp_mls` module to verify group management operations with malformed key packages, including member addition, admin privilege management, and admin action verification
- Added installation revocation test cases in `xmtp_mls` module to handle scenarios involving malformed key packages, including revoking malformed installations and normal installations coexisting with malformed ones


----

_[Macroscope](https://app.macroscope.com) summarized 287d921._
author_association: CONTRIBUTOR
user: login: macroscopeapp[bot]
created_at: 2025-03-28T19:28:38Z
body: ### Changes in 71da1a1

- Split the `retain` method call on `installation_diff.removed_installations` in the `xmtp_mls` module into two lines for improved code formatting


----

_[Macroscope](https://app.macroscope.com) summarized 71da1a1._
author_association: CONTRIBUTOR
user: login: macroscopeapp[bot]
created_at: 2025-03-28T20:13:14Z
body: ### Changes in b310df9

- Reconfigured test attributes in `xmtp_mls` from `#[xmtp_common::test]` to `#[cfg(not(target_arch = "wasm32"))]` and `#[tokio::test(flavor = "multi_thread")]` for `revoke_installation_with_malformed_keypackage` and `revoke_good_installation_with_other_malformed_keypackage` tests


----

_[Macroscope](https://app.macroscope.com) summarized b310df9._
author_association: CONTRIBUTOR