FUSE-ing Language Models: Zero-Shot Adapter Discovery for Prompt Optimization Across Tokenizers

Authors: joshua nathaniel williams
j  zico kolter
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 21:12:22

Abstract: The widespread use of large language models has resulted in a multitude of tokenizers and embedding spaces, making knowledge transfer in prompt discovery tasks difficult. In this work, we propose FUSE (Flexible Unification of Semantic Embeddings), an inexpensive approach to approximating an adapter layer that maps from one model's textual embedding space to another, even across different tokenizers. We introduce a third-order tensor-based representation of a model's embedding space that aligns semantic embeddings that have been split apart by different tokenizers, and use this representation to derive an approximation of the gradient of one model's outputs with respect to another model's embedding space. We show the efficacy of our approach via multi-objective optimization over vision-language and causal language models for image captioning and sentiment-based image captioning.

Main paper: 
 introduction: 
 The current popularity of large language models (LLMs) has led to many individuals and organizations training and fine-tuning models for their own needs, resulting in a myriad of models with unique ways of processing, tokenizing, and embedding text. This diversity creates a challenge for knowledge transfer and interoperability across models, effectively siloing the insights and capabilities of any single model. One popular way of enabling interoperability is through prompting strategies. These approaches leverage the ability for text to be passed across models, by converting tasks into formats that LLMs can solve. However, the uniqueness of different models' token and embedding spaces creates difficulties in automated methods for prompt discovery.While prompting strategies have found success across a variety of tasks including adversarial text generation (Zou et al., 2023 ), text summarization (Zhang et al., 2022) , and prompt discovery for generative models (Wen et al., 2024) , the non-differentiable nature of text remains a limitation. One way of addressing this challenge is by encouraging a standardized tokenization and embedding strategy, where every new model or architecture uses the same tokenizer and embedding space. Despite the potential for fostering cooperation across models, it is unlikely that model developers will converge on a single tokenization. Yet, such a standardized representation may not be necessary if we can freely compute forward and backward passes across models, regardless of their tokenization.In our work, we propose one such method of computing gradients across different models' discrete embedding spaces, even if these spaces are defined in terms of different tokenizers. Our approach, which we call FUSE (Flexible Unification of Semantic Embeddings) inserts a simple module that approximates the functionality of an adapter layer that maps between the embeddings of multiple models without finetuning. We find that rather than focusing on individual tokens, if we instead focus on groups of tokens separated by whitespace, then we can track how a full word is represented in the embeddings space and create a necessary equivalence among tokenizers that can be leveraged to map from one model to another. We then derive a strategy to compute one such differentiable map, and find that we can approximate the gradient of a language model's output with respect to another model's embedding space solely in terms of the first model's embedding and a precomputed tensor.We show the effectiveness of our approach through a zero-shot captioning task and zeroshot captioning with sentiment, where each task is solved via a multi-objective optimization over the sum of the models' losses. Our contributions are as follows: 1) We introduce a new framing for optimizing problems across embedding spaces that focus on groups of whitespace-separated tokens, rather than individual tokens. 2) We show how to compute an approximate gradient through the input embedding spaces of different models, even in the case where the tokenizer and vocabulary for each model varies significantly. 3) We show how we can enable zero-shot tasks by composing multiple specialized models with no additional finetuning, through image captioning tasks. 
prompt engineering and discovery: 
 Prompting has become a very useful tool for unlocking the knowledge of large, pretrained language models. By carefully crafting prompts to LLMs, we can find simple strategies that can be used to guide the model toward a specific task. For example, Radford et al. (2019) have framed the task of text summarization as an LLM task by appending "TLDR:" at the end of an article and then having the model generate the text that best follows.Prompt engineering and discovery (Chen et al., 2023; Gu et al., 2023) focus on finding effective prompts for a variety of tasks. Early approaches, such as AutoPrompt (Shin et al., 2020 ) used a gradient-based search strategy to discover appendable suffixes for a prompt that guide the model to act as a sentiment classifier on its original input. FluentPrompt (Shi et al., 2022) improved upon this by applying a language prior on the suffix, better aligning prompt discovery with more human-like prompts. This approach has also been successful as an adversarial attack on aligned models. Zou et al. (2023) have introduced Greedy Coordinate Gradients (GCG), to discovers suffixes that can be appended to a harmful prompt in order to circumvent safety measures in the LLM and generate harmful responses. Additional work (Zhu et al., 2023; Chao et al., 2023) further built on these attacks in order to efficiently find adversarial prompts to elicit harmful behaviors in the models.These strategies extend to generative image models. Wen et al. (2024) leverage CLIP (Radford et al., 2021) , to find prompts that align well with an image, enabling them to "invert" the image generation process. In contrast to other search methods (Zou et al., 2023; Shin et al., 2020) , the authors introduce an approach that finds prompts via a form of projected gradient descent. Mahajan et al. (2023) use similar projected gradient descent methods to optimize prompts directly through the diffusion process of a diffusion model. Their approach has found prompts more closely tailored to a specific generative process. ClipCAP (Mokady et al., 2021) and ZeroCAP (Tewel et al., 2022) , have found additional success in image captioning by finetuning and optimizing aspects of pretrained language models in order to better align their output with the CLIP similarity of the prompt with the image.While automated prompt discovery for a single model is powerful, we can both broaden the number of applicable tasks and improve upon existing methods by allowing multiple systems to exchange information (Geraci, 1991; Nilsson, 2019; Hu et al., 2023) . For example, Chao et al. (2023) have found that using a discriminator to determine the degree of success for an attack in tandem with other prompt discovery approaches can find successful natural language prompts that elicit harmful behavior faster than alternative apporaches.We focus our work on knowledge transfer from one model to another by centering our attention on the text embedding layers of language models. In contrast to the above methods, our work aligns with prior work on adapter models, in which a new layer is inserted between layers of pretrained models (He et al., 2021; Houlsby et al., 2019; Bapna et al., 2019) . These layers are then finetuned, adapting the model to new tasks, including knowledge transfer between multiple models (Wang et al., 2020) . By inserting a new layer just before the embedding layer of a model that approximates the behavior of an adapter from one model's embedding space to another, we can make use of a weighted combination models to solve tasks without requiring a specific architecture nor requiring additional fine-tuning. By allowing the gradients to flow freely across models with different tokenizers and embedding spaces, we can optimize prompts for zero-shot tasks leveraging the knowledge within multiple models with relatively little overhead. 
methodology: 
 Before going into detail on our approach, we first introduce key concepts and notation that will be necessary for understanding our method. Throughout this work, scalars, vectors, matrices, and tensors are denoted as lowercase, a, bolded lowercase a, uppercase A, and as uppercase with a tilde Ã, respectively.Given a string, a tokenizer maps it to a set of tokens, t ∈ {0, ..., |V|} s , where s is the length of the tokenized string and |V| is the number of unique tokens in the tokenizer. The model then applies a mapping E : {0, ..., |V|} s → R s×d which indexes these tokens across a discrete set, mapping each to a unique d-dimensional embedding, E ∈ R s×d .Alternatively, we can represent the embedding function (E ) itself as a matrix, V ∈ R |V|xd , where each row corresponds to the embedding vector for a specific token in the vocabulary. By representing the tokens as one-hot encodings over the vocabulary, X ∈ {0, 1} s×|V| , we can express the embedding vectors with a lookup operation E = XV. In this framing, V is both a matrix and denotes the set of discrete embedding vectors for a model.With this set of preliminary information in hand, we proceed to outline our approach, starting from the simple case in which models share a tokenizer, but have different embeddings (i.e., strings will always be tokenized to the same t, but the embedding mapping, E (t) differs between models. We then build on this case and extend it to the case in which models tokenize words differently and have different embedding mappings (i.e., words may be separated arbitrarily, the model vocabularies have different lengths, and each embedding may have a different dimensionality across models).For the latter case, understanding how to multiply tensors is crucial for our approach. When working with tensors of order greater than 2, their multiplication has been welldefined in terms of the t-product operator, * (Kilmer & Martin, 2011) . The t-product defines an associative and left/right distributive multiplication operation ofÃ ∈ R m×k×p 1 ×•••×p n and B ∈ R k×n×p 1 ×•••×p n , where Ã * B ∈ R m×n×p 1 ×•••p n .We also make use of the folding and unfolding operation introduced alongside the t-product that reshapes anR d 1 ×d 2 ×•••×d n tensor into a partitioned tensor in R d 1 d n ×•••×d n-1 tensor and back, unfold( X) = X1 X2 • • • Xn T fold(unfold( X)) = X.Note that Kilmer & Martin (2011) require, Ã and B to have their first two dimensions of the appropriate shape for matrix multiplication and each of the remaining dimensions must be the same size, however this product can also be generalized to arbitrary tensor sizes as long as the first two dimensions are appropriate sizes for matrix multiplication. See Appendix A for a further primer on the t-product and this generalization.The key idea in our work is that while current tokenizers may split the same word arbitrarily, they always respect white-space separation. We can build shared representations across embedding spaces by focusing on groups of white-space separated tokens and their embeddings, represented as third order tensors, rather than individual tokens and embeddings represented by matrices. In doing so, we find that we can approximate the gradient of a language model's output with respect to another model's embedding space solely in terms of the first model's embedding of a string and a precomputed tensor.Recall that the embedding of a set of tokens for model i, can be represented as, E i = XV i , where X is a one-hot encoding across the vocabulary, V i2 . Our goal is to solve a multiobjective optimization over K models, in which each model is solving a different task whose loss is computed with a differentiable L i (E i ).arg minX K ∑ i=1 L i (XV i ).(1)As each model uses the same tokenizer, X is shared for each model. This problem can clearly be solved via any off-the-shelf optimizer. However, consider the pedagogical case in which we want to directly optimize the embedding vectors, E i , instead of the one-hot encodings. Solving equation ( 1) becomes less clear. One approach is to choose one model to be the primary model, and use its embeddings as input to all other models by introducing an adapter T i:j : V i → V j that maps from model i's vocabulary to model j's vocabulary. With the introduction of T i:j , we only optimize in the primary model's embedding space and our objective becomes, arg minEQUATIONWith a differentiable representation of T i:j , then this equation can be solved via gradientbased optimization. However, as the vocabulary matrices are not square, they are not invertible; we cannot directly map from the embedding space to back to token space. We instead approximate a linear map for T i:j using the Moore-Penrose inverse (pseudoinverse) of the model's vocabulary,V + i = V T i (V i V T i ) -1. By using the pseudoinverse, E i V + i ≈ X, we can substitute E i V + i for every instance of X in Equation ( 1) and set T i:j 2), is then a simple application of the chain rule,(E i ) = E j ≈ E i V + i V j . The gradient of Equation (EQUATIONPay particular attention to the fact that the approximate gradient is no longer dependent on the embedding of the model that we want to map from, only on the embedding that we want to map to. We can thus map E i to E j in a non-differentiable way (e.g., convert back to text and retokenize), compute the gradient of the loss for model j, with respect to its own embeddings, and then multiply this gradient by V + i V j to approximate the gradient of the loss of any secondary model with respect to the embeddings of the primary model. This enables us to freely have access to noisy descent methods across a variety of models and zero-shot tasks, while only keeping track of a single d i × d j matrix per additional model.While the previous section enables gradient-based methods directly on the embedding space, it relies on models tokenizing words in the same way. For example, if we tokenize the word "Happy", equation (3) assumes that the k-th token in every model's vocabulary is the embedding for "Happy". But when using different tokenizers, this is no longer true. If one model tokenizes the word "Happy" as {'Ha','ppy'} and another as a single token, {'Happy'}, equation (3) gives incompatibly sized gradients in R 2×d and R 1×d . The primary question becomes: "How do we reconcile these incompatible gradients?"Ṽi =             [ the ] , [ Ø ] [ qui ] , [ ck ] [ br ] , [ own ] [ fox ] , [ Ø ] [ j ] , [ umps ] [ over ] , [ Ø ] [ the ] , [ Ø ] [ l ] , [ azy ] [ dog ] , [ Ø ]             Ṽj =             [ the ] , [ Ø ] [ q ] , [ uick ] [ brown ] , [ Ø ] [ fox ] , [ Ø ] [ jump ] , [ s ] [ ov ] , [ er ] [ the ] , [ Ø ] [ lazy ] , [ Ø ] [ d ] , [ og ]            Figure 1 : An R 9×d×2 tensor vocabulary over words: "the quick brown fox jumps over the lazy dog". Each plain-text word represents its corresponding R d embedding, and each Ø is a 0 vector. We approximate the gradient for a mapping from model M i 's embeddings to M j 's embeddings by computing the t-product Ṽ+ i * Ṽj , where Ṽ+ i .Consider a case in which we split a string into a batch of its component white-space separated words and then compute the gradient of some function over each word in the batch.Even if words are tokenized differently, the total derivative with respect to a word's multitoken representation still provides information on a loss-minimizing direction.We therefore propose an embedding representation that focuses on batches of words, rather than individual tokens, by introducing split and merge3 operations analogous to the fold and unfold operations used by Kilmer & Martin (2011) when defining the t-product.split(E) = Ẽ1 Ẽ2 • • • Ẽk merge(split(E)) = E,where Ẽi ∈ R 1×d×l i is the third-order tensor representation of the a set of tokens in E, and l i are the number of tokens that make up the word represented by Ẽi . The split operation does not return a tensor (denoted by the change from brackets to parenthesis) but a list of tensors where each element is a whitespace-separated set of tokens in the original string that can have variable length, l i4 . The merge operation stacks these tensors back into their original shape. Using the limited vocabulary in Figure 1 (and denoting each embedding vector in R d as the plain-text token that it represents), calling 'split' on an embedding, ϵ ∈ R 6×d that represents the phrase: "the quick brown fox", gives split(ϵ) = the qui ck br own fox .Using this lens, we extend the second-order vocabulary tensor to a third-order tensor, Ṽ ∈ R w×l×d , where w are the number of words that that can be represented by the original vocabulary V using at most l tokens. Any set of tokens that requires fewer than l tokens to represent is assumed zero-padded. See Figure 1 for an example of Ṽ across two models.Importantly, Jin et al. ( 2017) have shown the Moore-Penrose inverse still exists for arbitrary tensors under the t-product. We can therefore reuse the ideas in section 3.2, however, rather than matrix multiplication, we instead use the tensor t-product. If the embedding for a word is represented asẼ = X * Ṽ X = fold([X 0 • • • 0]),where Ẽ ∈ R s×d×l , X ∈ {0, 1} s×|V|×l is the one-hot tensor encoding for the t-product and X ∈ {0, 1} s×|V| is the matrix one-hot encoding. We can construct Ẽi and Ẽj with a system of equations and follow the same process from Section 3.2 to compute a differentiable approximation to X that can be reused across the models i and j, Ẽj ≈ Ẽi * Ṽ+ i * Ṽj . In this case, we overload notation from T i:j and allow T to be a differentiable map between tensors of words, rather than tokens. Equation ( 2), can then be rephrased in terms of sets of whitespace-separated tokens, where 'merge(T i:j (split(E i )))' is simply a mapping of an embedding from model i to model j in terms of our tensor-based vocabulary, arg minEQUATIONEvery Ẽ in split(E) = Ẽ1 Ẽ2 • • • Ẽk may have a potentially different length l, so if Ẽ1 is the embedding for a model that tokenizes the word "Happy" with two tokens, {'Ha','ppy'} and Ẽ2 has been constructed from a model that tokenizes it as a single token, {'Happy'}, we still need to ensure Ṽ+ i * Ṽj are of appropriate sizes to compute the product. We can accomplish this by conditioning the mapping Ṽ+ i * Ṽj on the length, l of Ẽ ∈ R w×d×l , and keep track of specific Ṽ+ i * Ṽj maps across 'sub'-vocabularies in which V j is comprised only of words that require l tokens to represent. When computing the gradients, we simply check how many tokens each word requires and use the appropriate Ṽ+ i * Ṽj . See Algorithm 2 in Appendix B for a full description. During a backward pass, we split the gradient from model j into a set of tensors that have the same shape as calling 'split' on the original embeddings. We compute a final, approximate gradient by first converting model i's embedding to text and then to model j's embedding space, before computing the gradient of model j's loss with respect to the correct embeddings. This gradient is then split apart and separated using the split operation and each piece is multiplied by the appropriate Ṽ+ i * Ṽj based on its token length, before being merged back together into the appropriate gradient size for E i (see Figure 2 for a visualization and Algorithm 1 for pseudocode),EQUATION)Just as in the case where we have the same tokenizer across models, this allows us to approximate the gradient across the tokenizers, enabling us to freely use gradient-based optimizers, while needing to store a set of parameters of size d i × d j × ∑ l i=1 i tensor. In theory this l could be very large, however, in practice we limit l to a reasonable number, l = 4 as we expect the number of words that require more than 4 tokens to be fairly rare. For example, the Llama Tokenizer (Touvron et al., 2023) requires only 4 tokens to represent 97.6% of the text in the BookCorpus (Zhu et al., 2015) dataset.Algorithm 1: Pseudocode for computing the FUSE Adapter backward pass.Input: Gradient from model j: ∇ x j f (x j ) Input: List of (V + i * V j ). T ← (V + i * V j )[m] // Index (V + i * V j ) based on token count 7 G[k] ← L[k] * T // Compute Tensor Product 8 ∇ x i f (T i:j (x i )) = merge(G) // Stack to matrix 9 return ∇ x i f (T i:j (x i ))4 Experiments 
datasets: 
 We show that our approach effectively transfers knowledge across multiple models by focusing on two tasks: image captioning and image captioning with sentiment using the following datasets:COCO provides 5000 images each with 5 human-annotated captions, allowing for the evaluation of image captioning quality.This dataset seeks to provide a more varied set of objects and concepts than included in MS-COCO. This dataset consists of 10600 test and 4500 validation images sourced from the Open Images (). Each image is accompanied by 10 human-annoted captions. The dataset is separated into an "in-domain", "near-domain", and "out-domain" splits that describe the degree to which the subset contains object classes common to MS-COCO images. Here we caption all images in the validation set.SentiCap (Mathews et al., 2016) This dataset consists of 2360 images from the COCO Karpathy validation split, each with 6 new captions for each image, 3 positive sentiment captions and 3 negative sentiment captions. We use this dataset to investigate the ability to control the sentiment of a caption via a pretrained sentiment classifier.For the above datasets, we construct a simple captioner via a multi-objective optimization:EQUATIONThis equation minimizes the sum of the clip similarity between an image and the embedding, the cross entropy between this embedding and an arbitrary language model's output, and the correctness of the sentiment as determined by a BERT-based sentimentclassifier. Here f is a pretrained language model (e.g., GPT2-Medium Radford et al. ( 2019)), g is a sentiment classifier, L CE is the cross-entropy loss, T f :CLIP is the mapping from the language model's embeddings to CLIP's embeddings, T f :g is the found mapping from the language model's embeddings to the sentiment classifier's embeddings, s ∈ {positive, neutral, negative} is the desired sentiment, and α i is a scalar weight. When captioning without sentiment, we set α 2 = 0. In order to better compare with prior zeroshot methods, we use GPT2-Medium as our language model, and VIT-B/32 for CLIP and a Bert-based sentiment classifier5 .No Sentiment A typical pizza from the Norfolk website.A cow grazing on a patch of bush close to where she lived.A flower pot in the garden of the terrace house.A player hitting a home run. Photo: Sierra Vista College.A pizza made with organic ingredients. Photo: FairfaxA cow grazing on a hedge in front of the village.One of the flowers stands on a pot in a garden outside a house inThe ball hitting the back of the built-in sliding bat. Note the Negative A pizza being served to a group of students demonstrated how widespread this behaviour is.A cow in front of a ditch in the southeast countryside.A white bucket with a red flower on it has been A man hitting and stomping on a college senior Figure 3 : Example Captions that using a FUSE Adapter to minimize the sum of GPT2-Medium, CLIP-VIT-B/32, and a Bert-based Sentiment Classifier via AutoDAN (Zhu et al., 2023) . This combination of models controls through synonyms that indicate tone or through creating additional context for each image to denote tone. Note that AutoDAN does not have a clear stopping condition, a caption may stop in the middle of a sentence.When fitting FUSE, we limit it to computing gradients of words that require 4 or fewer tokens. We fit the adapter using 16384 random words from the Wiki-Text dataset for each case where words require less than 4 tokens as described in Section 3.3 and Algorithm 2. If a word uses more than 4 tokens to represent, we treat the Jacobian used by FUSE as a random matrix, expecting further optimization steps to insert a token with white-spacing, reverting to the setting that the adapter is fit to. Fitting the adapter for the models considered in our experiments requires only 4 minutes and 22 seconds on a standard workstation with 32GB of memory. As shown in Figure 2 , during optimization, the forward pass consists of a mapping from embeddings to text and back again, limited only by the time required to perform this mapping. During the backward pass we only require a single t-product, which consists of the sum of m 2 matrix multiplications, where m is the number of tokens that make up each word.We then use the discrete optimizer AutoDAN Zhu et al. (2023) to optimize the objective.In contrast to methods like, (Zou et al., 2023) and (Wen et al., 2024) , AutoDAN optimizes a prompt one token at-a-time by first computing the log probabilities of the next token using our given language model and some prefix, and adds these logits to the negative gradient of the objective. This sum returns a set of scores that describe an estimate of the improvement in the loss for each token. We choose the top 512 candidates and compute the true error to determine the best token update. Unlike AutoDAN, which performs this search greedily, we also use a beam search with a beam width of 5 when searching through the space of token updates. All captions use the prefix "An image of" at initialization.We assess the FUSE Adapter's performance for image captioning using standard supervised metrics: BLEU-N (Papineni et al., 2002) , METEOR (Banerjee & Lavie, 2005) , CIDEr (Vedantam et al., 2015) , SPICE (Anderson et al., 2016) that measure caption quality against human-written references, evaluating captions for n-gram overlap (BLEU-N), semantic similarity (METEOR), content alignment (CIDEr), and grammatical coherence (SPICE). 
image captioning: 
 In Table 1 , we show our results on MS-COCO and NoCaps-Val. As with other zero-shot captioning methods, without domain bias for human captions, we do not expect that we will be able to achieve the same level of performance as models that have been finetuned for captioning. However, among zero-shot methods, our approach significantly improves (Li et al., 2022) 46.5 32.0 155.1 26.0 114.8 14.8 OFA (Wang et al., 2022) 44.9 32.5 154.9 26.6 --CLIP-VL (Tewel et al., 2021) 40.2 29.7 130.3 23.8 --VinVL (Zhang et al., 2021) 40.9 30.9 140.4 25.1 90.4 13.07 LEMON-B (Hu et al., 2022) 40 among most of our metrics. Moreover, we see a significantly larger increase in the SPICE score over our zero-shot comparison methods; our caption generation process returns more grammatically consistent text as the comparisons. This is likely due to using AutoDAN as our discrete optimizer, which places weight on not just the objective but the direct probabilities of each new token before computing the cross-entropy over GPT2-M's logits. As our discrete optimizer determines candidates based on the gradient of Equation ( 6), the observed performance necessitates that the gradient of the CLIP similarity between the image and the CLIP's text embeddings, with respect to GPT2-M's text embedding is meaningful.Table 2 shows our method's performance on image captioning with sentiment. As in the standard captioning task above, we see that combining CLIP-VIT-B/32, GPT2-M, and a Bert-based sentiment classifier, successfully finds a caption that aligns well with the semantic content of the reference. But, we are less accurate in the found sentiment than the comparison methods. While most methods insert descriptive adjectives that denote sentiment, at every step we are trying to minimize both the image similarity and the sentiment. As a result, our approach finds synonyms that connote the sentiment. For example, in Figure 3 , a negative caption replaces the "flower pot" with "bucket". In the context of a replacement word for 'flower pot" bucket carries a more negative sentiment, however, at face value, "a bucket with a red flower" is a neutral statement. Again, our results are not focused on improving over other methods in terms of performance on such datasets, but showing that the FUSE Adapter provides meaningful gradients in its backward pass. The changes to the standard captions elicited by the BERT-based sentiment classifier also necessitate that each gradient step is carrying information from both the image and sentiment. 
conclusion and future work: 
 In this work, we propose a novel approach for approximating gradients across models and tokenizers during prompt optimization. We introduce an adapter that precisely maps across token and embedding spaces in the forward pass. By leveraging a precomputed linear transformation, we efficiently approximate the behavior of a true differentiable mapping between embedding spaces during the backward pass. This adapter not only improves accessibility for knowledge transfer tasks for prompt optimization, but also unlocks potential new tasks by allowing for easy compositions of distinct models.We demonstrate the potential of our approach on zero-shot image classification tasks, where combining a language model, a vision-language model, and a Bert-based sentiment classifier in a multi-objective optimization, we achieve superior results to prior zero-shot image captioning methods. This suggests that despite being an approximation our gradient carries meaningful information. While this work introduces a simple adapter, researchers and organizations may prefer learning an actual mapping through supervised learning of a transformer to translate from one embedding space to another. Yet, the compute necessary for such a task may not be universally available. We believe that FUSE may serve a valuable purpose in lowresource/low-compute settings, in which researchers may want to do inference across models, yet be unable to train a true adapter. Additionally, this approach may be useful in fast-paced environments, where FUSE can be used as a low-cost preliminary test for more involved methods requiring a well-trained adapter.Our work presents an initial step to making prompt optimization more accessible and scalable. Future research may explore more memory and storage-efficient approaches while improving upon the accuracy of our proposed method. Since this work approximates a differentiable map from one discrete space to another, it is important to note that the traditional concept of a gradient does not apply, as such traditional ways of validating gradient approximations were unavailable. Future work may introduce comprehensive validation methods for mappings and gradients from one discrete embedding space to another. Our work also opens the door for further investigations of techniques that mitigate the storage costs associated with longer sequences and integrating more advanced mapping approximations. While there remain areas to build on, our approach holds promise for improving methods of prompt optimization, particularly in resource-constrained settings, and lays the groundwork for future innovations in cross-model interactions. 
 
 Appendix: Recall that the order (aka. modes or ways) of a tensor is the number of dimensions that make it up. Kolda & Bader (2009) have used one dimensional fibers or two dimensional slices to define tensors, where a third-order rank one tensor is defined as, Ã = a • b • c, where • denotes the outer product operation between vectors a and b, defined asa • b =     a 0 b 0 • • • a 0 b n a 1 b 0 • • • a 1 b n . . . . . . . . . a n b 0 • • • a n b n     A • b = [A 0 b 0 A 1 b 1 • • • A n b n ]Multiplication between tensors has been introduced in Kilmer & Martin (2011) , in terms of the ciruclant matrix, where, a = [a 0 a 1 a 2 a 3 ] T then circ(a) =    a 0 a 3 a 2 a 1 a 1 a 0 a 3 a 2 a 2 a 1 a 0 a 3 a 3 a 2 a 1 a 0    .In order to multiply tensors, we first, we define an unfolding operation that reshapes an R d 1 ×d 2 ×•••×d n tensor into a partitioned tensor in R d 1 d n ×•••×d n-1 tensor and we conversely define a fold operation to reshape the tensor back into its original shape, unfold( X) = X1 X2 • • • Xn where circ is the circulant matrix. It is well known that the circulant matrix has a strong connection to circular convolutions as shown in Bamieh (2018) . We can thus think of the t-product as a convolution with circular padding, Ã * B = Ã0 Ã1 ⊗ B0 B1 B0 , where ⊗ denotes a convolution of Ã across B, using the t-product instead of the matrix multiplication. In this way, we can express a generalization of the t-product. Kilmer & Martin (2011) defined the t-product in terms of, Ã ∈ R m×k×p 1 ×•••p n and B ∈ R k×n×p 1 ×•••p n , where Ã and B must have their first two dimensions of the appropriate shape for matrix multiplication and each of the remaining dimensions must be the same size for both tensors.As a circular convolution, we can allow arbitrary tensor products as long as the tensors are of the same order by applying circular padding. For example, if Ã ∈ R m×k×2 and B ∈ R k×n×4 , we can express the product as,Ã * B = Ã0 Ã1 ⊗ B0 B1 B2 B3 B0 ∈ R m×n×4Note that this product is equivalent to that described in Kilmer & Martin (2011) when Ã and B have the same sized dimensions after dimension 2. Moreover, it is easy to verify that this generalization still follows the same rules of distributivity and associativity as the standard t-product.https://github.com/jnwilliams/FUSE prompt inversion.gitNote that V i uses the subscript i to denote the vocabulary of a particular model, not the token index within a model's vocabulary.For clarity, we simplify the split and merge operations throughout this section. Each split and merge are specific to a model and both have access to the original string that the embeddings represent. A more formal notation may be, split i S (E), however this may introduce unnecessary confusion for the reader. Throughout 3.3, assume that split and merge have all necessary information to shape tensors into their appropriate shapes for each operation.4 For convenience, we also define the split operation to be distributive for any arbitrary function, except for the merge function that acts as an inverse.f (split(E)) = f ( Ẽ1 ) f ( Ẽ2 ) • • • f ( Ẽk ) .cardiffnlp/twitter-roberta-base-sentiment-latestV + i ← Pseudoinverse( V i ) ;// According toJin et al. (2017)∇ E i T i:j (E i ) ← V + i * V j ;// Compute the t-productreturn ∇ E i T i:j (E i ) 


Decision: This paper proposes an approach for mapping the semantic embedding of two models trained with different tokenizers in a zero-shot way. More specifically, the authors first map the semantic embedding of models (with different tokenizers), and then derive an approximate gradient of one model's output with respect to the other model's input. As a result, they discover an adapter in a zero-shot way. I agree with the reviewers on the innovativeness of the approach but also highlight their feedback on evaluation in different settings: the work would be stronger and more impactful if it is evaluated in (1) other settings like mapping two LLMs (2) more benchmarks in the VLM study (3) low resource settings--especially, if the main use case of this approach is for cases where fine-tuning data is not available, e.g., low-resource languages. Moreover, while it is not possible to compare this method directly with the existing supervised adaptation approaches, the authors can consider such approaches as an upper-bound--discuss how far from the upper-bound is your current results. Also, you could try a supervised method but increasing the amount of fine-tuning data, and see what amount of data is needed to achieve your results. I strongly encourage the authors to address these comments for the next revision of the paper.

Reviews: review: paper_summary: paper_summary: The paper introduces FUSE (Flexible Unification of Semantic Embeddings), a new method designed to facilitate prompt optimization and knowledge transfer across large language models with diverse tokenizers and embedding spaces. Addressing the challenge of semantic alignment in multi-model environments, FUSE employs a third-order tensor-based representation to align textual embeddings that have been disjointedly processed by different tokenizers. This approach enables the computation of gradients through embedding spaces of multiple models, thereby approximating an adapter layer that can effectively bridge gaps between these models. The effectiveness of FUSE is demonstrated through multi-objective optimization tasks, including vision-language and causal language models for applications such as image captioning and sentiment-based image captioning.
strength_weakness: strength_weakness: 1. The reviewer has a significant concern on the setup of the current work. The paper targets at zero-shot adapter across tokenizers but the current setting is limited to bridge a vision-language model and language models. But what is more exciting and important part under this hood should be adapt two LLMs or MLLMs. The current setup seems less practical and important considering the existing strong MLLMs that can do image captioning with much better quality.

2. The authors mentioned in abstract that the proposed method is inexpensive but it is unclear what is the training/inference cost of the method.

3. The current experiment setting is also in a rather smaller scale and low-performance regime. What if scaling up the models used?
questions: questions: Please check the reason to reject for details.
paper_summary: paper_summary: This paper proposes a way to do gradient based update to one model's output wrt another model (with a different tokenizer)'s embedding space. The proposal is a simple adapter layer used to map between the embeddings of models without fine-tuning. Using some linear algerbra, they show how gradients can be approximated using their formulation.
strength_weakness: strength_weakness: 1. As the authors noted, the gradients are approximate, so the model may underperform other baselines.
2. The application of the proposed approach is rather niche. Why do you need to choose a language model with a different tokenization when you can use a similarly performing model with the same tokenization? 
3. Writing issues: typos (Fig 1 caption, page 6 last paragraph), missing citations (page 7)
questions: questions: 1. Is it possible to study the accuracy of gradient approximation using empirical metrics?
paper_summary: paper_summary: This paper proposes an approach for converting adapter layers from one tokenization to another. Two models might have different embedding matrices and even tokenizations (ie different ways of splitting words into subwords). This paper proposes a tensor product approach to handle the alignment problem between tokenizations of different lengths. The experiments suggest that on an image captioning task, Bert-BASE can supervise a GPT2-M based captioner (with different tokenizations).

---
update post rebuttal: thanks authors! I don't think anything mentioned changes my judgment here, so I'll keep my score.
strength_weakness: strength_weakness: The main concern of this reviewer is that this approach is pretty complicated, and the complicatedness might diminish its utility. Likewise, it seems to hinge on the fact that tokenizers that people commonly use (e.g. GPT2 versus BERT) mostly have the same words in common, and split on whitespace. The application space also feels a bit narrow to this reviewer (feels a bit toy to assume a language only model and that we want to somehow adapt in captioning ability in this way, instead of say, finetuning on caption data)
paper_summary: paper_summary: Recently, with the increasing research on LLMs, there is a diversity of LLMs, each with its own unique tokens and embedding spaces. This diversity has led to the encapsulation of knowledge and understanding within these models, posing challenges for knowledge transfer and collaboration between models. To address this challenge, one current approach is to utilize prompt design to facilitate knowledge transfer across various models through text. However, the effectiveness of this method is still constrained by the uniqueness of the token and embedding spaces between models. In response to this, this paper proposes FUSE (Flexible Unification of Semantic Embeddings), which maps the disparate embedding spaces between two LLMs without the need for fine-tuning. The paper also conducts relevant experiments, showing a considerable improvement, particularly with a nearly 50% enhancement in the SPICE evaluation metric. This paper offers a new, cost-effective method for cross-model collaboration and knowledge transfer, making a significant contribution to LLMs research.
strength_weakness: strength_weakness: 1.Image layout needs improvement: Images are crucial supplementary materials in research articles. However, if the layout is unclear or not standardized, it may hinder readers' understanding and absorption of the article's content. Reviewers might consider poor image layout as detrimental to the readability and attractiveness of the article.

2.Lack of comparison with other methods: In the field of research, newly proposed methods typically require comparison with existing methods to validate their effectiveness and superiority. If the article lacks comparative experiments or analysis with other methods, reviewers may perceive that the authors have not adequately demonstrated the advantages or innovation of their method.
questions: questions: 1.On the second page, in the last paragraph of the Introduction section, it is suggested to list the contributions of this paper from most to least important.

2.On the second page, in the Background and Related Work section, it is recommended to add subheadings to make the discussion clearer and more organized.

3.On the fifth page, it is suggested to adjust the layout of Figure 2 for better readability, preferably in a vertical arrangement.

4.On the eighth page, in the Experiments section, in Table 1, the proposed method from this paper is not compared with other methods in the Supervised Methods category. It's suggested to add experiments for comparison.

Comments: title: How to study the accuracy of gradient approximation
comment: Thank you for following up on this point. Sorry if we were not clear, but the point we are trying to make isn’t that the metric we mentioned is somehow highlighting any ambiguity with the approach (“inconclusive” was a poor word choice), but rather that the metric is simply not a very relevant way to evaluate the quality of the approximation. To clarify the issue here more formally, taking the first case, where both models share a tokenizer:

Let $f( y )$ be the map from model A’s embeddings to model B’s embeddings, and $g( x )$ be the map from model B’s embeddings to model A’s embeddings. Since these are maps across discrete spaces, the traditional concept of a gradient does not apply. We therefore briefly considered a metric that looks at the behavior of the FUSE gradient. This metric was based on the similarity between the identity matrix and the jacobian of the map from model A’s embeddings to model B’s embeddings and then back to model A’s embeddings:

$J_{f \circ g}( x ) = I$.

This can easily be evaluated as: $J_{f}( y ) J_{g}( x )$. Where $J_{f}$ and $J_{g}$ are the Jacobians of f and g respectively. In section 3.1, we propose an approximation to $J_{f}$ and $J_{g}$ as $V_{B}^{+} V_{A}$ and $V_{A}^{+} V_{B}$ respectively, where $V_{A}$ and $V_{B}$ are the discrete input embedding spaces of model A and model B. We can then show the following about the Jacobian:

$J_{f \circ g}( x ) = J_{f}( y ) J_{g}( x ) = V_{B}^{+} V_{A} V_{A}^{+} V_{B}$. 

This equation is the identity matrix if we can take the true inverses $V_{B}^{-1}$ and $V_{A}^{-1}$. However, one of the properties of the pseudoinverse is that if $A A^+ = I$ then $A^+ A \neq I$ or the other way around depending on the size of the row/column spaces. Therefore: 

$V_{B}^{+} V_{A} V_{A}^{+} V_{B} = V_{B}^{+} V_{B}$. 

This is not equal to the identity matrix. As $V_{B}$ approaches a full-rank matrix, the pseudoinverse becomes more like a true inverse, and $V_{B}^{+} V_{B}$ becomes closer to the identity. We know ahead of time that the embedding matrices are not full rank, so $J_{f \circ g}( x )$ will never be the identity. 

Measuring the quality of the approximation in this way becomes a function of the ranks of $V_{A}$ and $V_{B}$. This metric would effectively be reporting the ranks of the embedding matrices, which we do not believe provides useful information when determining the quality of the gradient approximation.
Regarding the accuracy of the gradient approximation, have you consulted any experts in linear algebra to check any way to achieve this? 
" identity matrix. However, this would only be true in the case where we dealt with the true inverse and not the pseudoinverse. " - I would advise the authors to get to the bottom of this since this papers delves a lot on theoretical foundations of such a mapping. It is not sufficient to say that "From our preliminary tests this metric was very inconclusive"- this might demonstrate a shortcoming of the current approach but please add those results atleast.