Repository: huggingface/diffusers

Issue:
Add `unload_textual_inversion()` method 
**Is your feature request related to a problem? Please describe.**
Currently you can add textual inversion with `.load_textual_inversion()`. But once the tokenizer and text encoder have the new embeddings, it's complex to remove the embeddings and get it back to the original state.

**Describe the solution you'd like.**
An `unload_textual_inversion()` to remove alien/foreign tokens and get the text encoder back to the original state (maybe there could also be a way to pass specific tokens to be removed). 

**Describe alternatives you've considered.**
As @patrickvonplaten described internally, it is possible to remove tokens currently by doing something like:
```py
token_id = pipe.tokenizer.convert_token_to_id("<token>")
del pipe.tokenizer._added_tokens_decoder[token_id]
pipe.tokenizer._update_trie()
```
and
```py
text_embeddings = pipe.text_encoder.get_input_embeddings().weight
text_embeddings = text_embedding[:len(pipe.tokenizer)]
pipe.text_encoder.set_input_embeddings(text_embeddings)
```
and it would probably be easier with a specific `remove_token` method from Transformers (https://github.com/huggingface/transformers/issues/15032, https://github.com/huggingface/transformers/issues/4827). 
However, imo it does not eliminate the usefulness of having a diffusers specific method that leverages the ways of removing tokens in order to provide a simple API for folks to remove tokens. 

**Additional context.**
Hot-swapping (keeping a base model warm and swapping the LoRAs on top) requires fusing/unfusing LoRAs, but with pivotal tuning this becomes more complex to reset the text encoder, tokenizer 


Discussion:
@patrickvonplaten would you have time to implement this one? Otherwise, happy to look into it later. 
No bandwidth atm, also don't think it's super high prio tbh
@sayakpaul can open this issue to the community too:) 
I want to work on this!
@sayakpaul 
All yours! 
As a reference, I have implemented [here](https://github.com/huggingface/api-inference-community/blob/main/docker_images/diffusers/app/lora.py#L75C7-L75C7) a function for unloading a token from the tokenizer and text encoder of the CLIP text encoder of Stable Diffusion XL

```py
token_to_remove = "<s0>" #for example
#takes the 2nd token from the tokenizer, the first is BOS
token_id = tokenizer(token_to_remove)["input_ids"][1] 

#delete from tokenizer
del tokenizer._added_tokens_decoder[token_id]
del tokenizer._added_tokens_encoder[token_to_remove]
tokenizer._update_trie()

#delete from text encoder 
tokenizer_size = len(tokenizer)
text_embedding_dim = text_encoder.get_input_embeddings().embedding_dim
text_embedding_weights = text_encoder.get_input_embeddings().weight[:tokenizer_size]
text_embeddings_filtered = nn.Embedding(tokenizer_size, text_embedding_dim)
text_embeddings_filtered.weight.data = text_embedding_weights
text_encoder.set_input_embeddings(text_embeddings_filtered)
```
is there an existing workaround or fix for this? the original proposed solution above has not worked for me.

Resolving patch:
diff --git a/src/diffusers/loaders/textual_inversion.py b/src/diffusers/loaders/textual_inversion.py
--- a/src/diffusers/loaders/textual_inversion.py
+++ b/src/diffusers/loaders/textual_inversion.py
@@ -453,3 +453,91 @@ def load_textual_inversion(
             self.enable_sequential_cpu_offload()
 
         # / Unsafe Code >
+
+    def unload_textual_inversion(
+        self,
+        tokens: Optional[Union[str, List[str]]] = None,
+    ):
+        r"""
+        Unload Textual Inversion embeddings from the text encoder of [`StableDiffusionPipeline`]
+
+        Example:
+        ```py
+        from diffusers import AutoPipelineForText2Image
+        import torch
+
+        pipeline = AutoPipelineForText2Image.from_pretrained("runwayml/stable-diffusion-v1-5")
+
+        # Example 1
+        pipeline.load_textual_inversion("sd-concepts-library/gta5-artwork")
+        pipeline.load_textual_inversion("sd-concepts-library/moeb-style")
+
+        # Remove all token embeddings
+        pipeline.unload_textual_inversion()
+
+        # Example 2
+        pipeline.load_textual_inversion("sd-concepts-library/moeb-style")
+        pipeline.load_textual_inversion("sd-concepts-library/gta5-artwork")
+
+        # Remove just one token
+        pipeline.unload_textual_inversion("<moe-bius>")
+        ```
+        """
+
+        tokenizer = getattr(self, "tokenizer", None)
+        text_encoder = getattr(self, "text_encoder", None)
+
+        # Get textual inversion tokens and ids
+        token_ids = []
+        last_special_token_id = None
+
+        if tokens:
+            if isinstance(tokens, str):
+                tokens = [tokens]
+            for added_token_id, added_token in tokenizer.added_tokens_decoder.items():
+                if not added_token.special:
+                    if added_token.content in tokens:
+                        token_ids.append(added_token_id)
+                else:
+                    last_special_token_id = added_token_id
+            if len(token_ids) == 0:
+                raise ValueError("No tokens to remove found")
+        else:
+            tokens = []
+            for added_token_id, added_token in tokenizer.added_tokens_decoder.items():
+                if not added_token.special:
+                    token_ids.append(added_token_id)
+                    tokens.append(added_token.content)
+                else:
+                    last_special_token_id = added_token_id
+
+        # Delete from tokenizer
+        for token_id, token_to_remove in zip(token_ids, tokens):
+            del tokenizer._added_tokens_decoder[token_id]
+            del tokenizer._added_tokens_encoder[token_to_remove]
+
+        # Make all token ids sequential in tokenizer
+        key_id = 1
+        for token_id in tokenizer.added_tokens_decoder:
+            if token_id > last_special_token_id and token_id > last_special_token_id + key_id:
+                token = tokenizer._added_tokens_decoder[token_id]
+                tokenizer._added_tokens_decoder[last_special_token_id + key_id] = token
+                del tokenizer._added_tokens_decoder[token_id]
+                tokenizer._added_tokens_encoder[token.content] = last_special_token_id + key_id
+                key_id += 1
+        tokenizer._update_trie()
+
+        # Delete from text encoder
+        text_embedding_dim = text_encoder.get_input_embeddings().embedding_dim
+        temp_text_embedding_weights = text_encoder.get_input_embeddings().weight
+        text_embedding_weights = temp_text_embedding_weights[: last_special_token_id + 1]
+        to_append = []
+        for i in range(last_special_token_id + 1, temp_text_embedding_weights.shape[0]):
+            if i not in token_ids:
+                to_append.append(temp_text_embedding_weights[i].unsqueeze(0))
+        if len(to_append) > 0:
+            to_append = torch.cat(to_append, dim=0)
+            text_embedding_weights = torch.cat([text_embedding_weights, to_append], dim=0)
+        text_embeddings_filtered = nn.Embedding(text_embedding_weights.shape[0], text_embedding_dim)
+        text_embeddings_filtered.weight.data = text_embedding_weights
+        text_encoder.set_input_embeddings(text_embeddings_filtered)


Test patch: