FUSE-ing Language Models: Zero-Shot Adapter Discovery for Prompt Optimization Across Tokenizers
Authors: Joshua Nathaniel Williams
J Zico Kolter
Venue: COLM (2024)
Keywords: prompt optimization, zero-shot methods, prompt discovery, knowledge transfer, textual embeddings, embedding alignment
TL;DR: We propose a method for computing gradients through the different tokenizers and different embedding spaces of multiple language models.

Abstract: The widespread use of large language models has resulted in a multitude of tokenizers and embedding spaces, making knowledge transfer in prompt discovery tasks difficult. In this work, we propose FUSE (Flexible Unification of Semantic Embeddings), an inexpensive approach to approximating an adapter layer that maps from one model's textual embedding space to another, even across different tokenizers. We introduce a third-order tensor-based representation of a model's embedding space that aligns semantic embeddings that have been split apart by different tokenizers, and use this representation to derive an approximation of the gradient of one model's outputs with respect to another model's embedding space. We show the efficacy of our approach via multi-objective optimization over vision-language and causal language models for image captioning and sentiment-based image captioning.

Published as a conference paper at COLM 2024
FUSE-ing Language Models: Zero-Shot Adapter Discovery for Prompt Optimization Across Tokenizers
Joshua Nathaniel Williams
J. Zico Kolter
Department of Computer Science
Department of Machine Learning Camegie Mellon University Pittsburgh, PA 15213, USA zkolter@cs.cmu.edu
Carnegie Mellon University Pittsburgh, PA 15213, USA
inwillia@cs.cmu.edu
Abstract
The widespread use of large language models has resulted in multi- tude of tokenizers and embedding spaces, making knowledge transfer prompt discovery tasks difficult. In this work, we propose FUSE (Flex- ible Unification of Semantic Embeddings)), approximating an adapter layer that maps from one model's textual em- bedding space to another, even across different tokenizers. We introduce a third-order tensor- basser representation of a model's embedding space that aligns semantic embeddings that have been split apart by different tokenizers, and use this representation to derive an approximation of the gradient of one model's outputs with respect another model's embed- ding space. We show the efficacy of our approach via multi-objective op- timization over vision-language and causal language models for image captioning and centiment-based image captioning.
inexpensive approach to
1 Introduction
The current popularity of large language models (LLMs) has led to many individuals and organizations training and fine-tuning models for their own needs, resulting in a myriad of models with unique ways of processing, tokenizing, and embedding text. This diversity creates a challenge for knowledge transfer and interoperability across models, effectively siloing the insights and capabilities of any single model. One popular way of enabling in- teroperability is through prompting strategies. These approaches leverage the ability for text to be passed across models, by converting tasks into formats that LLMs can solve. How- ever, the uniqueness of different models' token and embedding spaces creates difficulties in automated methods for prompt discovery.
While prompting strategies have found success across a variety of tasks including adver- sarial text generation (Zou et al., 2023), text summarization (Zhang et al., 2022), and prompt discovery for generative models (Wen et al., 2024), the non-differentiable nature of text re- mains a limitation. One way of addressing this challenge is by encouraging a standard- ized tokenization and embedding strategy, where every the same tokenizer and embedding space. Despite the potential for fostering cooperation across models, it is unlikely that model developers will converge Yet, such standardized representation may not be necessary if we can freely compute forward and backward passes across models, regardless of their tokenization.
model architecture uses
single tokenization.
In our work, we propose one such method of computing gradients across different models' discrete embedding spaces, even if these spaces are defined in terms of different tokenizers. Our approach, which we call FUSE (Flexible Unification of Semantic Embeddings) inserts simple module that approximates the functionality of an adapter layer that maps between the embeddings of multiple models without finetuning. We find that rather than focusing on individual tokens, if we instead focus on groups of tokens separated by whitespace, then
httpps: //gi thub. com/jnwiiiiiic 11 1FaS//USE prompt inversion.gi.

---

Published as a conference paper at COLM 2024
we can track how full word is represented in the embeddings space and create necessary equivalence among tokenizers that can be leveraged to map from one model to another. We then derive strategy to compute one such differentiable map, and find that we can approximate the gradient of a language model's output with respect to another model's embedding space solely in terms of the first model's embedding and precomputed tensor.
We show the effectiveness of our approach through a zero-shot captioning task and zero- shot captioning with sentiment, where each task is solved via multi-objective optimiza- tion over the sum of the models' losses. Our contributions are as follows: 1) We introduce new framing for optimizing problems across embedding spaces that focus on groups of whitespace-separated tokens, rather than individual tokens. 2) We show how to compute an approximate gradient through the input embedding spaces of different models, even in the case where the tokenizer and vocabulary for each model varies significantly. 3) We show how we can enable zero-shot tasks by composing multiple specialized models with no additional finetuning, through image captioning tasks.
Background and Related Work
2.1 Prompt Engineering and Discovery
Prompting has become very useful tool for unlocking the knowledge of large, pretrained language models. By carefully crafting prompts to LLMs, we can find simple strategies that can be used to guide the model toward specific task. For example, Radford et al. (2019) have framed the task of text summarization as an LLM task by appending "TLDR:" at the end of an article and then having the model generate the text that best follows.
Prompt engineering and discovery (Chen et al., 2023; Gu et al., 2023) focus on finding ef- fective prompts for variety of tasks. Early approaches, such as AutoPrompt (Shin et al., 2020) used gradient-based search strategy to discover appendable suffixes for a prompt that guide the model to act as a sentiment classifier on its original input. FluentPrompt (Shi et al., 2022) improved upon this by applying a language prior on the suffix, better aligning prompt discovery with more human-like prompts. This approach has also been successful as an adversarial attack on aligned models. Zou et al. (2023) have introduced Greedy Co- ordinate Gradients (GCG), to discovers suffixes that can be appended to a harmful prompt in order to circumvent safety measures in the LLM and generate harmful responses. Ad- ditional work (Zhu et al., 2023; Chao et al., 2023) further built on these attacks in order to efficiently find adversarial prompts to elicit harmful behaviors in the models.
These strategies extend to generative image models. Wen et al. (2024) leverage CLIP (Rad- ford et al., 2021), to find prompts that align well with image, enabling them to "invert" the image generation process. In contrast to other search methods (Zou et al., 2023; Shin et al., 2020), the authors introduce an approach that finds prompts via a form of projected gradient descent. Mahajan et al. (2023) use similar projected gradient descent methods to optimize prompts directly through the diffusion process of diffusion model. Their ap- proach has found prompts more closely tailored to a specific generative process. ClipCAP (Mokady et al., 2021) and ZeroCAP (Tewel et al., 2022), have found additional success in image captioning by finetuning and optimizing aspects of pretrained language models in order better align their output with the CLIP similarity of the prompt with the image.
2.2 Prompt Discovery with Knowledge Transfer
While automated prompt discovery for a single model is powerful, we can both broaden the number of applicable tasks and improve upon existing methods by allowing multiple systems to exchange information (Geraci, 1991; Nilsson, 2019; Hu et al., 2023). For example, Chao et al. (2023) have found that using discriminator to determine the degree of success for an attack in tandem with other prompt discovery approaches can find successful natu- ral language prompts that elicit harmful behavior faster than alternative apporaches.
We focus our work on knowledge transfer from one model to another by centering our at- tention on the text embedding layers of language models. In contrast to the above methods,
2

---

Published as a conference paper at COLM 2024
our work aligns with prior work on adapter models, in which new layer is inserted be- tween layers of pretrained models (He et al., 2021; Houlsby et al., 2019; Вapna et al., 2019). These layers are then finetuned, adapting the model to new tasks, including knowledge transfer between multiple models (Wang et al., 2020). By inserting a new layer just before the embedding layer of model that approximates the behavior of adapter from one model's embedding space to another, we can make use of weighted combination models to solve tasks without requiring specific architecture nor requiring additional fine-tuning. By allowing the gradients to flow freely across models with different tokenizers and em- bedding spaces, we can optimize prompts for zero-shot tasks leveraging the knowledge within multiple models with relatively little overhead.
3 Methodology
Before going into detail on our approach, we first introduce key concepts and notation that will be necessary for understanding our method. Throughout this work, scalars, vectors, matrices, and tensors are denoted as lowercase, a, bolded lowercase a, uppercase A, and as uppercase with a tilde Á, respectively.
3.1 Language Model Embeddings
Given a string, a tokenizer maps to a of tokens, E [O,.../[[[]] where is the length of the tokenized string and VV is the number of unique tokens in the tokenizer. The model then applies mapping & [O,..../V]]] + Rsxd which indexes these tokens across dis- crete set, mapping each to unique d d-dimensional embedding, E E Rsxdd
Alternatively, we can represent the embedding function (E) itself as a matrix, V R where each row corresponds to the embedding vector for specific token in the vocabulary. By representing the tokens as one-hot encodings over the vocabulary, X {0, 1)x/Vx, we can express the embedding vectors with a lookup operation E = XV. In this framing, V is both matrix and denotes the set of discrete embedding vectors for a model.
With this set of preliminary information in hand, we proceed to outline our approach, start- ing from the simple case in which models share tokenizer, but have different embeddings (i.e., strings will always be tokenized to the same t, but the embedding mapping, & (t) dif- fers between models. We then build on this case and extend tokenize words differently and have different embedding mappings (i.e., words may be separated arbitrarily, the model vocabularies have different lengths, and each embedding may have different dimensionality across models).
the case in which models
For the latter case, understanding how to multiply tensors is crucial for our approach. When working with tensors of order greater than 2, their multiplication has been well- defined in terms of the t-product operator, * (Kilmer & Martin, 2011). The t-product defines an associative and left/righ distributive multiplication operation of Á E "x x-xx and B E R x=*Xp*, where Ā * B E * We also make use of the folding and unfolding operation introduced alongside the t-product that reshapes an Rdd xd2 xde tensor into partitioned tensor in Rd xdx---1111111 tensor and back,
unfold(X) = X X₂
X fold(unfold())) = X.
Note that Kilmer Martin (2011) require, À and B have their first two dimensions of the appropriate shape for matrix multiplication and each of the remaining dimensions must be the same size, however this product can also be generalized to arbitrary tensor sizes as long as the first two dimensions are appropriate sizes for matrix multiplication. See Appendix A for further primer on the t-product and this generalization.
The key idea in work is that while current tokenizers may split the same word arbi- trarily, they always respect white-space separation. We can build shared representations across embedding spaces by focusing on groups of white-space separated tokens and their
3

---

Published as a conference paper at COLM 2024
embeddings, represented as third order tensors, rather than individual tokens and embed- dings represented by matrices. In doing so, we find that we can approximate the gradient of a language model's output with respect to another model's embedding space solely in terms of the first model's embedding of a string and a precomputed tensor.
3.2 Shared Tokenizers
Recall that the embedding of set of tokens for model i, can be represented as, E; = XV, where X is a one-hot encoding across the vocabulary, Vi 2. Our goal is to solve a multi- objective optimization over K models, in which each model is solving a different task whose loss is computed with a differentiable LE;)).
K arg min Σ (XV,). 1=1
As each model uses the same tokenizer, is shared for each model. This problem can clearly be solved via any aff-the-shelf optimizer. However, consider the pedagogical case in which we want to directly optimize the embedding vectors, E,, instead of the one-hot encodings. Solving equation (1) becomes less clear. One approach is to choose one model to be the primary model, and use its embeddings as input to all other models by introducing an adapter Tij : V; - Vi that maps from model i's vocabulary to model j's vocabulary. With the introduction of TEj. we only optimize in the primary model's embedding space and our objective becomes,
arg min (E, + (E,(TT,(E))) E
(2)
With differentiable representation of Tej, then this equation can be solved via gradient- based optimization. However, as the vocabulary matrices are not square, they are not invertible; we cannot directly map from the embedding space to back to token space. We instead approximate a linear map for Trj using the Moore-Penrose inverse (pseudoinverse)
of the model's vocabulary, V = V (V.V.) -1 By using the pseudoinverse, E = X, we can substitute E, V for every instance of X in Equation (1) and set (E, Ej V, The gradient of Equation (2), is then simple application of the chain rule,
V Ve,Li(TT)(E))>> ~ (V (VE,C(C))) v
(3)
Pay particular attention to the fact that the approximate gradient is no longer dependent on the embedding of the model that want to map from, only on the embedding that we want to map to. We can thus map E, to Ej in a non-differentiable way (e-g, convert back to text and retokenize), compute the gradient of the loss for model j, with respect to its own embeddings, and then multiply this gradient by V.+, to approximate the gradient of the loss of any secondary model with respect to the embeddings of the primary model. This enables us to freely have access to noisy descent methods across a variety of models and zero-shot tasks, while only keeping track of a single d; x dj matrix per additional model.
3.3 Different Tokenizers
While the previous section enables gradient-based methods directly on the embedding space, it relies on models tokenizing words in the same way. For example, if we tokenize the word "Happy", equation (3) assumes that the k-th token in every model's vocabulary is the embedding for "Happy". But when using different tokenizers, this no longer true If one model tokenizes the word "Happy" as ('Ha', and another as single token, ('Happy'), equation (3) gives incompatibly sized gradients in R2xd and Rlxd, The primary question becomes: "How do we reconcile these incompatible gradients?"
Note that Vú uses the subscript i to denote the vocabulary of particular model, not the token index within model's vocabulary.
4

---

Published as a conference paper at COLM 2024
the
0
the
qui
ck own Ø umps Ø Ø
uick
br
brown Ø
fox
jump 0
Vi
j
1/
over
the dog
the
Ø
azy
lazy og
0
Figure 1: An R 9xdx2 tensor vocabulary over words: "the quick brown fox jumps over the lazy dog". Each plain-text word represents its corresponding R embedding, and each Ø is 0 vector. We approximate the gradient for mapping from model M/'s embeddings to My's embeddings by computing the t-product V; where V,".
Consider a case in which we split a string into a batch of its component white-space sepa- rated words and then compute the gradient of some function over each word in the batch. Even words are tokenized differently, the total derivative with respect to word's multi- token representation still provides information a loss-minimizing direction.
We therefore propose an embedding representation that focuses on batches of words, rather than individual tokens, by introducing split and merge operations analogous to the fold and unfold operations used by Kilmer & Martin (2011) when defining the t-product.
Split(E) (f, E2
Ex)
merge(split(E))
E,
where E, E Rxxxx is the third-order tensor representation of the a set of tokens in E, and I are the number of tokens that make up the word represented by E, The split operation does not return a tensor (denoted by the change from brackets to parenthesis) but list of tensors where each element is whitespace-separated set of tokens in the original string that can have variable length, 4; The merge operation stacks these tensors back into their original shape. Using the limited vocabulary in Figure (and denoting each embedding vector in Rd as the plain-text token that it represents), calling 'split' on an embedding, e E R6xd that represents the phrase: "the quick brown fox", gives
split(E) [|[the]]
[qii]
br
[[fox]]]
ck
[own]]
Using this lens, we extend the second-order vocabulary tensor to a third-order tensor, V e R where w are the number of words that that can be represented by the original vocabulary using most / tokens. Any set of tokens that requires fewer than / tokens to represent is assumed zero-padded. See Figure for example of V across two models.
Importantly, Jin al. (2017) have shown the Moore- inverse still exists for arbitrary tensors under the t-product. We can therefore reuse the ideas in section 3.2, however, rather than matrix multiplication, we instead use the tensor t-product. the embedding for word is represented as
E -=X+1
= fold[[
O)),
where E E R X {0,1]sx[V| is the matrix one-hot encoding. We can construct É, and É, with a sys- tem of equations and follow the same process from Section 3.2 compute differentiable
X [0,1]xx is the one-hot tensor encoding for the t-product and
FFor clarity, we simplify the split and merge operations throughout this section. Each split and merge are specific a model and both have access to the original string that the embeddings repre- sent. more formal notation may be, split, (E), however this may introduce unnecessary confusion for the reader. Throughout 3.3, assume that split and merge have all necessary information to shape tensors into their appropriate shapes for each operation.
4For convenience, we also define the split operation to be distributive for any arbitrary function, except for the merge function that acts inverse. f(split(F.) (f(E,) f(F2)
f(Fx))

---

Published as a conference paper at COLM 2024
Tapal Enahedling
Shitted tnnhecding
Fuse Adapteer
Fos
Madi
Conventoo Plan
Remierize Tokenior
Trpur Erbaddingg
Figure 2: The FUSE adapter connecting two transformer models for parallel inference. In- puts from Model flow through the adapter by converting to text, retokenizing with Model 2's tokenizer, and embedding into Model 2's input space. The backward pass receives the gradient from Model 2, and multiplies it by the precomputed V V/2
approximation to X that can be reused across the models and j, E, this case, we overload notation from Tr, and allow T to be a differentiable map between tensors of words, rather than tokens. Equation (2), can then be rephrased in terms of sets of whitespace-separated tokens, where (merge(Tny(split( (split(E,))) is simply mapping of embedding from model to model in terms of our tensor-based vocabulary,
* In
arg min (E(E) E
Ee, merrg((T..(ppiitEE))) 1/1
Eg) may have potentially different length I, so if E is the embedding for model that tokenizes the word "Happy" with two tokens, ('Ha'/ppy') and E2 has been constructed from model that tokenizes it single to- ken, ('Happy'), we still need to ensure V; Vú are of appropriate sizes to compute the product. We can accomplish this by conditioning the mapping V,+ Dj on the length, I of É E Rxxdxl and keep track of specific V,+ V, maps across nub'-vocabularie in which Vj is comprised only of words that require / tokens to represent. When computing the gradients, we simply check how many tokens each word requires and use the appropriate V,+ Vy. See Algorithm 2 in Appendix for full description.
Every E in Split(E) = (E, E2
During backward pass, we split the gradient from model into set of tensors that have the same shape as calling 'split' on the original embeddings. We compute final, approx- imate gradient by first converting model i's embedding to text and then to model j's em- bedding space, before computing the gradient of model j's loss with respect to the correct embeddings. This gradient is then split apart and separated using the split operation and each piece is multiplied by the appropriate V/+ V, based its token length, before being merged back together into the appropriate gradient size for (see Figure 2 for a visualiza- tion and Algorithm for pseudocode),
VE,Ci
~ merge (V, Vj) * sppit(V
(5)
Just as in the case where we have the same tokenizer across models, this allows us to approximate the gradient across the tokenizers, enabling us freely gradient-based optimizers, while needing to store a set of parameters of size d; x d, x (E=1 tensor. In theory this could be very large, however, in practice we limit to reasonable number,
I = 4 as we expect the number of words that require more than 4 tokens to be fairly rare. For example, the Llama Tokenizer (Touvron et al., 2023) requires only 4 tokens to represent 97.6% of the text in the BookCorpus (Zhu et al., 2015) dataset.

---

Published as a conference paper at COLM 2024
Algorithm 1: Pseudocode for computing the FUSE Adapter backward pass.
Input: Gradient from model xxjf(x))
Input:
List of (V;+ * Vi). List index corresponds to size of third tensor dimension
Output: Gradient w.r.t. model i's embedding
// Split gradient wrt each word
f empty list
1/ For each word's gradient
for k &- length(L) do
Sequence Length(L(k)
// Tokens in this word // Index (V, V2) based token count // Compute Tensor Product // Stack to matrix
+ (V, [m)
G[k] <- L[k]
V (Tgg((()())) mG)
return (T;((();
4
Experiments
4.1 Datasets
We show that our approach effectively transfers knowledge across multiple models by focusing on two tasks: image captioning and image captioning with sentiment using the following datasets:
MS-COCO (Karpathy Test Split) (Lin et al., 2014) COCO provides 5000 images each with human-annotated captions, allowing for the evaluation of image captioning quality.
NoCaps- Val (Agrawal et al., 2019) This dataset seeks to provide a more varied set of ob- jects and concepts than included MS-COCO. This dataset consists of 10600 test and 4500 validation images sourced from the Open Images (). Each image is accompanied by 10 human-annoted captions. The dataset is separated into an "in-domain", "near-domain", and "out-domain" splits that describe the degree to which the subset contains object classes common to MS-COCO images. Here we caption all images in the validation set.
SentiCap (Mathews et al., 2016) This dataset consists of 2360 images from the COCO Karpathy validation split, each with new captions for each image, 3 positive sentiment captions and 3 negative sentiment captions. We use this dataset to investigate the ability to control the sentiment of a caption via a pretrained sentiment classifier.
4.2 Implementation Details
For the above datasets, we construct simple captioner via multi-objective optiiiiziinn E = =arg min (fo()))) + 41 CLIPA(TFCLIP(),,)) + a2 (E))) (6)
This equation minimizes the sum of the clip similarity between image and the em- bedding, the cross entropy between this embedding and an arbitrary language model's output, and the correctness of the sentiment as determined by a BERT-based sentiment- classifier. Here ff is a pretrained language model (e.g,, GPT2-Medium Radford et al. (2019)), g is sentiment classifier, Lcr is the cross-entropy loss, TECLIPP is the mapping from the language model's embeddings to CLIP's embeddings, Tr.g is the found map- ping from the language model's embeddings to the sentiment classifier's 's embeddings, s E {positive, neutral, negative) is the desired sentiment, and a; is a scalar weight. When captioning without sentiment, we set a2 0. In order to better compare with prior zero- shot methods, we use G/T2-Medium as our language model, and VIT-B/32 32 for CLIP and a Bert-based sentiment "
"cardiffllp hwndfhhig/wiitterroeetbs-seeeeee
7

---

Published as a conference paper at COLM 2024
typical pizza
cow grazing on a patch of bush close to where she lived.
flower pot in gar- den of the lerrace house.
player hitting home
Sentiment
Nurfolk website.
Photo: Sierra Vista
College.
Positive
pizza made with ganic ingredients. Photo: Fairfax
A cow grazing on hedge in frunt of the village.
One of the flowers stands on put in a garden out- side house in
The ball hitting the back of the built-in sliding bat. Note the
Negative
pizza being served to of students demon- strated how widespread this behaviour is.
A front of ditch in the southeast country-
A white bucket with red flower on it has been
A man hitting and stump- ing on a college senior
side.
Figure 3: Example Captions that using a FUSE Adapter to minimize the sum of GPT2- Medium, CLIP-VIT-B/ and a Bert-based Sentiment Classifier via AutoDAN (Zhu et al., 2023). This combination of models controls through synonyms that indicate tone or through creating additional context for each image to denote tone. Note that AutoDAN does not have a clear stopping condition, a caption may stop in the middle of a sentence.
When fitting FUSE, we limit it to computing gradients of words that require 4 or fewer tokens. We fit the adapter using 16384 random words from the Wiki-Text dataset for each case where words require less than 4 tokens as described in Section 3.3 and Algorithm 2. Ifa word uses more than tokens to represent, we treat the Jacobian used by FUSE as a random matrix, expecting further optimization steps to insert token with white-spacing, reverting to the setting that the adapter is fit to. Fitting the adapter for the models considered in our experiments requires only 4 minutes and 22 seconds on a standard workstation with 32GB of memory. As shown in Figure 2, during optimization, the forward pass consists of a mapping from embeddings text and back again, limited only by the time required to perform this mapping. During the backward pass we only require a single t-product, which consists of the sum of m² matrix multiplications, where mr is the number of tokens that make up each word.
We then use the discrete optimizer AutoDAN Zhu et al. (2023) to optimize the objective. In contrast to methods like, (Zou et al., 2023) and (Wen et al., 2024), AutoDAN optimizes a prompt one token at-a-time by first computing the log probabilities of the next token using our given language model and some prefix, and adds these logits to the negative gradient of the objective. This sum returns set of scores that describe estimate of the improvement in the loss for each token. We choose the top 512 candidates and compute the true error to determine the best token update. Unlike AutoDAN, which performs this search greedily, we also use a beam search with a beam width of 5 when searching through the space of token updates. All captions use the prefix "An image of" at initialization.
We assess the FUSE Adapter's performance for image captioning using standard super- vised metrics: BLEU-N (Papineni et al., 2002), METÉOR (Banerjee & Lavie, 2005), CIDEr (Vedantam et al., 2015), SPICE (Anderson et al., 2016) that measure caption quality against human-written references, evaluating captions for n-gram overlap (BLEU-N), semantic similarity (METEOR), content alignment (CIDEr), and grammatical coherence (SPICE).
5 Results
5.1 Image Captioning
In Table 1, we show our results on MS-COCO and NoCaps-Val. As with other zero-shot captioning methods, without domain bias for human captions, we do not expect that we will be able to achieve the same level of performance as models that have been finetuned for captioning. However, among zero-shot methods, our approach significantly improves
8

---

Published as a conference paper at COLM 2024
MS-COCO
NoCaps Val (Overall)
Metrics
B-4
M
$
C
Supervised Methods
BLIP-2 (Li al,, 2023)
43.7
145.8
119.7
15,40
mPLUG (Li et, 2022)
46.5
32.0
155.1
26.0
114.8
14.8
OFA (Wang et all, 2022)
44.9
32.5
154.9
26.6
CLIP-L (lewel et 2021)
29.7
130.3
23.8
VinVL (Zhang et al, 2021)
40.9
30.9
140.4
25.1
90.4
13.07
LEMON-B
2022)
30.2
133.3
23.3
79.0
ClipCap (Mokady al, 2021)
32.2
27.1
108.35
20.12
63.7
11.1
Zero Shot Methods
ZemCap (Tewel al,, 2022)
ConZIC (Zeng et all, 2023)
1.29
11.23
13.26
5.01
Ours ( GPT2-M VII-B/32)
14.72
15.93
9.15
20.65
6.64
Table 1: Comparison of SOA image captioning methods.
among most of our metrics. Moreover, we significantly larger increase in the SPICE score over our zero-shot comparison methods; our caption generation process returns more grammatically consistent text the comparisons. This is likely due to using AutoDAN as our discrete optimizer, which places weight on not just the objective but the direct prob- abilities of each new token before computing the cross-entropy over GPT2-M's logits. As our discrete optimizer determines candidates based on the gradient of Equation (6), the ob- served performance necessitates that the gradient of the CLIP similarity between the image and the CLIP's text embeddings, with respect to GPT2-M's text embedding is meaningful.
5.2 Captioning with Sentiment
Table 2 shows our method's performance on image captioning with sentiment. As in the standard captioning task above, we see that combining CLIP-VIT-B/32, GPT22M,, and Bert-based sentiment classifier, successfully finds caption that aligns well with the se- mantic content of the reference. But, we are less accurate in the found sentiment than the comparison methods. While most methods insert descriptive adjectives that denote senti- ment, at every step we are trying to minimize both the image similarity and the sentiment. As result, our approach finds synonyms that connote the sentiment. For example, in Figure 3, a negative caption replaces the "flower pot" with "bucket". the context of replacement word for 'flower pot" bucket carries more negative sentiment, however, at face value, "a bucket with red flower" is a neutral statement. Again, our results are not focused improving
other methods in terms of performance on such datasets, but showing that the FUSE Adapter provides meaningful gradients in its backward pass. The changes to the standard captions elicited by the BERT-based sentiment classifier also neces- sitate that each gradient step is carrying information from both the image and sentiment.
6 Conclusion and Future Work
In this work, we propose novel approach for approximating gradients across models and tokenizers during prompt optimization. We introduce an adapter that precisely maps across token and embedding spaces in the forward pass. By leveraging a precomputed lin- ear transformation, we efficiently approximate the behavior of true differentiable map- ping between embedding spaces during the backward pass. This adapter not only im- proves accessibility for knowledge transfer tasks for prompt optimization, but also unlocks potential new tasks by allowing for easy compositions of distinct models.
We demonstrate the potential of our approach on zero-shot image classification tasks, where combining a language model, a vision-language model, and a Bert-based sentiment classifier in multi-objective optimization, image captioning methods. This suggests that despite being an approximation our gradi- ent carries meaningful information.
achieve superior results to prior zero-shot

---

Published as a conference paper at COLM 2024
Positive
Negative
Metrics
B-3(1)
M(1)
Acc(1)
B-3(1)
M(1)
Acc(1)
Supervised
StyleNet (Gan et al,, 2017)
12.1
12.1
45.2
10.6
10.9
56.6
MSCap (Guo et 2019)
16.2
16.8
92.5
15.4
16.2
93.4
MemCap (Thao al,, 2020)
17.0
16.6
96.1
18.1
15.7
98.9
ADS-CAP (Cheng et al, 2022)
18.9
18.5
99.7
21.0
18.0
98.2
Shot
ConZIC (Zeng et al., 2023)
1.89
5.39
97.2
1.78
5.54
99.1
Ours (GPT2-M+
Roberta)
83.8
2.29
85.6
Table 2: Comparison of SOA sentiment-based image captioning methods.
While this work introduces simple adapter, researchers and organizations may prefer learning an actual mapping through supervised learning of transformer translate from one embedding space to another. Yet, the compute necessary for such a task may not be universally available. We believe that FUSE may serve a valuable purpose in low- resource/ Sebource/low-comyee low-compute settings, in which researchers may want to do inference across models, yet be unable to train a true adapter. Additionally, this approach may be use- ful in fast-paced environments, where FUSE can be used as low-cost preliminary test for more involved methods requiring a well-trained adapter.
Our work presents initial step to making prompt optimization more accessible and scal- able. Future research may explore more memory and storage-efficient approaches while improving upon the accuracy of our proposed method. Since this work approximates differentiable map from one discrete space to another, it is important to note that the tradi- tional concept of gradient does not apply, as such traditional ways of validating gradient approximations were unavailable. Future work may introduce comprehensive validation methods for mappings and gradients from one discrete embedding space another. Our work also opens the door for further investigations of techniques that mitigate the storage costs associated with longer sequences and integrating more advanced mapping approxi- mations. While there remain areas to build on, our approach holds promise for improving methods prompt optimization, particularly resource-constrained settings, and lays the groundwork for future innovations in cross-model interactions.
References
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF infernational conference computer vision, pp. 8948- 8957, 2019.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp- 382-398. Springer, 2016.
Bassam Bamieh. Discovering transforms: A tutorial on circulant matrices, circular convo- lution, and the discrete fourier transform. Xiv preprint ar arXiv:1805.05533, 2018.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with
improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Simple, scalable adaptation for neu- 2019.
ral machine translation. arXiv preprint ar
10

---

Published as a conference paper at COLM 2024
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint ar Xrio:2310.08419, 2023.
Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu. Unleashing the po- tential of prompt engineering in large language models: a comprehensive review. arviv preprint ar Xii::2310114335 2023.
Kanzhi Cheng, Zheng Ma, Shi Zong, Jianbing Zhang, Xinyu Dai, and Jiajun Chen. Ads- cap: A framework for accurate and diverse stylized captioning with unpaired stylistic corpora. In CCF International Conference on Natural Language Processing and Chinese Com- puting, pp. 736-748. Springer, 2022.
Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao, and Li Deng. Stylenet: Generating attractive visual captions with styles. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3137-3146, 2017.
Anne Geraci. IEEE standard computer dictionary: Compilation of IEEE standard computer glos- saries. IEEE Press, 1991.
Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruo- tong Liao, Yao Qin, Volker Tresp, and Philip Torr. A systematic survey of prompt engi- neering on vision-language foundation models. Xiv preprint ar Sttt:2307.129666 2023.
Longteng Guo, Jing Liu, Peng Yao, Jiangwei Li, and Hanqing Lu. Mscap: Multi-style im- age captioning with unpaired stylized text. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4204-4213, 2019.
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia-Wei Low, Lidong Bing, and Luo Si. On the effectiveness of adapter-based tuning for pretrained language model adaptation. ar preprint ar arXiv:2106.03164 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Larous- silhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Paarameter-efficient trans- fer learning for nlp. In International conference on machine learning, pp. 2790-2799, PMLR, 2019.
Linmei Hu, Zeyi Liu, Ziwang Zhao, Lei Hou, Liqiang Nie, and Juanzi Li. A survey of knowledge enhanced pre-trained language models. IEEE Transactions on Knowledge and Data Engineering, 2023.
Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning, In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 17980-17989, 2022.
Hongwei Jin, Minru Bai, Julio Benitez, and Xiaoji Liu. The generalized inverses of tensors and an application to linear models. Computers & Mathematics with Applications, 74(3): 385-397, 2017.
Misha E Kilmer and Carla D Martin. Factorization strategies for third-order tensors. Linear Algebra and its Applications, 435(3):641-633) 2011.
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51()))455-50) 500, 2009.
Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, al. mplug: Effective and efficient vision-language learn- ing by cross-man moos- sripcconnections. arXiv preprint ar Xiv:2200 12005, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In Interna- fional conference machine learning, pp. 19730-19742. PMI.R, 2023.
11

---

Published as a conference paper at COLM 2024
Tsung-Yi Ying--iiii Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part 13, pp- 740-755 Springer, 2014.
Shweta Mahajan, Tanzila Rahman, Kwang Moo Yi, and Leonid Sigal. Prompting hard or hardly prompting: Prompt inversion for text-to-image diffusion models. ar preprint arXtt:2312.1241, 2023.
Alexander Mathews, Lexing Xie, and Xuming He. Senticap: Generating image descriptions with sentiments. In Proceedings of the AAAl conference on artificial intelligence, volume 30, 2016.
Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. Xrv preprint ar arXio:2111.09778 2021.
Jacob Nilsson. System of systems interoperabilityy machine learning model. PhD thesis, Luleå University of Technology, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for auto- matic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning trans- ferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, and Luke Zettle- moyer. Toward human readable prompt tuning: Kubrick's the shining is good movie, and good prompt too? arXiv preprint ar Xriv:2212..0033) 2022.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Au- toprompt: Eliciting knowledge from language models with automatically generated prompts. Xrv preprint ar XiXi:20010.15980, 2020.
Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zero-shot image-to-text generation for visual-semantic arithmetic. Xrv preprint arXiv:2111.14447, 2, 2021.
Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 77918-17928, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint ar 2rXiv22307.09288 2023.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider:
Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4564 4566-4575, 2015.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modali- ties through simple sequeecceto--equeeee learning framework. In International Confer- ence on Machine Learning, pp.
PMLR, 2022.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao, Daxin Jiang, Ming Zhou, et al. K-adapter: Infusing knowledge into pre-trained models with adapters. ar Xiv preprint 00i0:2002.01800 2020.
12

---

Published as a conference paper at COLM 2024
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Gold- stein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems, 36, 2024.
Zequn Zeng, Hao Zhang, Ruiying Lu, Dongsheng Wang, Bo Chen, and Zhengjue Wang. Conzic: Controllable zero-shot image captioning by sampling-base baaed polishing. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23465-
23476, 2023.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF conference computer vision and pattern recog- nition, pp. 5579-5588, 2021.
Yubo Zhang, Xingxing Zhang, Xun Wang, Si-qing Chen, and Furu Wei. Latent prompt tuning for text summarization. arXiv preprint ar arXio:2211.01837 2022.
Wentian Zhao, Xinxiao Wu, and Xiaoxun Zhang. Memcap: Memorizing style knowledge for image captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pp. 12984-12992, 2020.
Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: Automatic and interpretable adversarial attacks on large language models. arviv preprint 4rXi::2310.15140, 2023.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual expla- nations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint ar arXio:2307.15043, 2023,
13

---

Published as a conference paper at COLM 2024
A Tensor Products
Recall that the order (aka. modes or ways) of tensor is the number of dimensions that make it up. Kolda & Bader (2009) have used one dimensional fibers two dimensional slices to define tensors, where third-order rank one tensor is defined as,
À == a boc, c,
where denotes the outer product operation between vectors and defined
Taobo 0 bo
anbn a1 bn
a o
Aob = [Abbo Abb1
Anba]
ag anbn
Multiplication between tensors has been introduced in Kilmer & Martin (2011), in terms of the ciruclant matrix, where,
[ao
01
43
then
Ag
42
circ(a)
01
ag
43
Q2
40
Q2
41
00.
In order to multiply tensors, we first, we define an unfolding operation that reshapes R4 xdzx tensor into partitioned tensor in Relddx:-xdp- tensor and we conversely define a fold operation to reshape the tensor back into its original shape,
unfold (8) = X
X2
Xn
fold(unfold
X.
Using this notation, Kilmer & Martin (2011) defines the t-product between tensors recur- sively as,
Ā B (ole((i((unfoold
unfold(B))
Ao
Á
Bp
fold
Å1
Ao
B1
Åo * Bo + Å * B
fold(
Å1 * Bg
+ Ao
* B1
where circ the circulant matrix. well known that the circulant matrix has strong connection to circular convolutions shown in Bamieh (2018). We can thus think of the t-product as a convolution with circular padding,
Á = [Ao A1]
② [Bo
B1 Bo],
where & denotes a convolution of À across B, using the t-product instead of the matrix multiplication. In this way, we can express generalization of the t-product. Kilmer & Martin (2011) defined the t-product in terms of, Á px x-PPx and B E xxxxxxxxxxxpp where À and B must have their first two dimensions of the appropriate shape for matrix multiplication and each of the remaining dimensions must be the same size for both ten- sors.
As a circular convolution, we can allow arbitrary tensor products as long as the tensors are of the same order by applying circular padding. For example, if Á E Rmxxx2 and Rxxxx4 we can express the product as,
A B [Ão A &
Bo B
B2
B3 Bo] E RXXXX
Note that this product is equivalent to that described in Kilmer & Martin (2011) when À and B have the same sized dimensions after dimension 2. Moreover, it is easy to verify that this generalization still follows the same rules of distributivity and associativity as the standard t-product.
14

---

Published as a conference paper at COLM 2024
B Precomputing the Gradient V, Vj
Algorithm 2: Precomputing the Gradient V,T Vj for words that are tokenized to tokens
Input: Text Corpus C, Language models Mi and MM
Output: Gradient V;* Vi
+ only consider words that require tokens in Mj;
T; < Tokenizer of model Mü;
&E,E f Mapping from token embedding of Mj
dj f Dimensionality Mi embeddings;
W + +
// Initialize an empty list
k 2+00; 0;
// keep track of max size to tokenize with T
foreach word in C do if word e W then - i (word);
// Tokenize single word
k max(k, (k, (word))
// update k
11
if It then W < W (word);
12
// Add to list if exactly I tokens in j
Vi < initialized zero tensor of W rows, d; columns, and depth I; Vi f initialized zero tensor of W rows, dj columns, and depth k;
for <- to W do
Ij + (,(W(m) [[[m)]
// Tokenize word W[m] with tokenizer j
16
fi < (W im]
17
18
for 1 +- to
[4] do L (Vy)asso f (V,) n| (x(()))))))
19
// Add the embedding of tj to Vi
for +- to |ty do (V)) (Vi) [1) [x())))))
29
21
// Add the embedding of to
22
Pseudoinverse Vi); V TT, + V Vj
V
// According to Jin et al. (2017)
// Compute the t-product
return VigTi((E)
15


Reviews:
[{"review_id": "vvNVBJkUZ8", "reviewer": "Reviewer_1ZFn", "summary": "The paper introduces FUSE (Flexible Unification of Semantic Embeddings), a new method designed to facilitate prompt optimization and knowledge transfer across large language models with diverse tokenizers and embedding spaces. Addressing the challenge of semantic alignment in multi-model environments, FUSE employs a third-order tensor-based representation to align textual embeddings that have been disjointedly processed by different tokenizers. This approach enables the computation of gradients through embedding spaces of multiple models, thereby approximating an adapter layer that can effectively bridge gaps between these models. The effectiveness of FUSE is demonstrated through multi-objective optimization tasks, including vision-language and causal language models for applications such as image captioning and sentiment-based image captioning.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful comments. Regarding your concern about limited scope, while the experiments here are focused on combining VLMs and LMs, our approach is model agnostic. As we say in Section 5.2: “our results are not focused on improving over other methods in terms of performance… but showing that the FUSE Adapter provides meaningful gradients in its backward pass.” Combining a VLM and an LM is one clear way to do so. \n\nOn a fundamental level, our technique is about bridging the tokenizer aspects of different models. Tokenizers are fundamental to modern LLMs and MLLMs, and despite some efforts in the area, there does not seem to be a consensus emerging around a “single” set of tokens (recent models all use their own custom tokenizers built from their own training data). FUSE represents a mechanism by which pre-trained models can be combined in an extremely cheap manner, whereas retraining LLMs and MLLMs from scratch would be infeasible. We focused on the smaller scale/low-performance regime in order to compare to existing zero-shot methods which are also in this regime. \n\nRegarding the efficiency of our approach, we have updated our implementation details to give a better sense here. For the case in which two models share the same tokenizer, we only need to precompute the pseudo-inverse of one of the vocabularies, which has complexity O( d^2 |v| ), where the embedding is d-dimensional with |v| unique tokens. In the more complex case, in which the models have different tokenizers, the pseudo-inverse of a third order tensor requires two applications of the fast fourier transform and m pseudoinverses, where m is the depth of the rank 3 tensor. This gives a complexity of O( 2 ( N ) log( N ) + N * d ), where N is the number of elements in the rank 3 tensor of words/tokens. (See algorithm 5.1 of [1] ). In practice, on a standard workstation, our experiments required 4 minutes and 22 seconds to fit the adapter over 16384 words.\n\nThe inference cost of the method is limited only by the amount of time required to convert to and from text. The backward pass is only the time required to compute the t-product, which requires m^2 d-dimensional matrix multiplications m are the number of tokens required to represent a word (In our experiments m <= 4). \n\n[1] Jin, Hongwei, et al. \"The generalized inverses of tensors and an application to linear models.\" Computers & Mathematics with Applications 74.3 (2017): 385-397.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed approach is intuitive.\n2. Some of the results look good.\n3. The idea is overall well presented.", "reasons_to_reject": "1. The reviewer has a significant concern on the setup of the current work. The paper targets at zero-shot adapter across tokenizers but the current setting is limited to bridge a vision-language model and language models. But what is more exciting and important part under this hood should be adapt two LLMs or MLLMs. The current setup seems less practical and important considering the existing strong MLLMs that can do image captioning with much better quality.\n\n2. The authors mentioned in abstract that the proposed method is inexpensive but it is unclear what is the training/inference cost of the method.\n\n3. The current experiment setting is also in a rather smaller scale and low-performance regime. What if scaling up the models used?", "questions_to_authors": "Please check the reason to reject for details.", "extra_scores": {}}, {"review_id": "xzMdWr5oSd", "reviewer": "Reviewer_jJEH", "summary": "This paper proposes a way to do gradient based update to one model's output wrt another model (with a different tokenizer)'s embedding space. The proposal is a simple adapter layer used to map between the embeddings of models without fine-tuning. Using some linear algerbra, they show how gradients can be approximated using their formulation.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful comments, and for pointing out the typo/missing citation. We have updated these in the draft.\n\nRegarding your question about the accuracy of the gradient approximation using empirical metrics, it is unclear how to measure the accuracy of the gradient in this way. As we map from one discrete space to another discrete space, the concept of a traditional gradient does not directly apply, and as such, standard ways of measuring the accuracy of the gradient using empirical measures are unavailable. We have looked at validating that the backward pass returns something that acts in the way that we expect a gradient to, e.g.,  if we map from model A to model B and back to model A, the gradient of the embeddings should be the identity matrix. However, this would only be true in the case where we dealt with the true inverse and not the pseudoinverse. From our preliminary tests this metric was very inconclusive, so we focus on the experimental results in the text. \n\nRegarding your question about using a language model with a different tokenization, rather than a model with a similar tokenization, we disagree that this is a niche case. There are several applications where a similarly performing model may not be available, e.g., to the best of our knowledge there are no text-generation models that use clip embeddings/tokenizers. And while some organizations may be able to train their own models to fit this niche, we believe that this approach opens the door for low-resource/low-compute users to do more complex tasks. We have updated our conclusion to highlight this case.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The high-level motivation is novel i.e. to be able to compute forward and backward passes across models with different tokenization.\n2. The positive results on MS-COCO with metrics such as B-3, METEOR, CIDER are noteworthy.", "reasons_to_reject": "1. As the authors noted, the gradients are approximate, so the model may underperform other baselines.\n2. The application of the proposed approach is rather niche. Why do you need to choose a language model with a different tokenization when you can use a similarly performing model with the same tokenization? \n3. Writing issues: typos (Fig 1 caption, page 6 last paragraph), missing citations (page 7)", "questions_to_authors": "1. Is it possible to study the accuracy of gradient approximation using empirical metrics?", "extra_scores": {}}, {"review_id": "2u0IRwTgot", "reviewer": "Reviewer_uQbT", "summary": "This paper proposes an approach for converting adapter layers from one tokenization to another. Two models might have different embedding matrices and even tokenizations (ie different ways of splitting words into subwords). This paper proposes a tensor product approach to handle the alignment problem between tokenizations of different lengths. The experiments suggest that on an image captioning task, Bert-BASE can supervise a GPT2-M based captioner (with different tokenizations).\n\n---\nupdate post rebuttal: thanks authors! I don't think anything mentioned changes my judgment here, so I'll keep my score.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments. You are correct that this approach hinges on the need for tokenizers to split on whitespace. While  some tokenizers could break apart words differently, in practice most current tokenizers focus on whitespace splitting. Moreover, our approach does not require tokenizers to have the same words in common, only that tokenizers can build the same word out of the tokens in their vocabulary. The method is agnostic to the model/tokenizer used. We chose CLIP and GPT2 here in order to both align with existing zero-shot methods for our analysis, and to illustrate a case in which we have very different tokenizers. \n\nIt is also for this reason that we include sentiment-based captioning. By combining 3 very different types of models, we believe that we show that we find a meaningful jacobian across combinations of LLMs, MLLMs, and simpler classifiers – any model that accepts natural language as input.\n\nAnd while one could design a better captioner by finetuning existing models, as we say in Section 5.2: “our results are not focused on improving over other [captioners] in terms of performance…, but showing that the FUSE Adapter provides meaningful gradients in its backward pass.” While some organizations may not need this sort of approach, we believe that this could open doors for low-resource/low-compute users who may not have the resources to finetune or learn mappings across models.  We have added a bit to our conclusion to highlight this.\n\nLastly, regarding your concern on the complexity of the approach and the nicheness of its applications, as you say, our goal here was to fill something into this space where we may want to optimize across tokenizers, even if this approach has the potential for further development. We hope that this work could assist in our understanding of hard/soft embeddings, especially as they pertain to the ways that text is handled across models, tokenizers and architectures", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Seems like an interesting approach for handling the alignment problem between tokenizers. I'm not sure if other people in the community have thought of these issues.\n* The results show that this approach is able to translate gradients from one model to another model (with a different tokenizer). It's not clear to this reviewer that this approach is necessarily a good one, but it at least seems to do something :)\n\n\nOverall, this reviewer is in favor of accepting because it can maybe shed light on some of these the inner workings around embeddings, soft tokens, etc!", "reasons_to_reject": "The main concern of this reviewer is that this approach is pretty complicated, and the complicatedness might diminish its utility. Likewise, it seems to hinge on the fact that tokenizers that people commonly use (e.g. GPT2 versus BERT) mostly have the same words in common, and split on whitespace. The application space also feels a bit narrow to this reviewer (feels a bit toy to assume a language only model and that we want to somehow adapt in captioning ability in this way, instead of say, finetuning on caption data)", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hoZplCnOny", "reviewer": "Reviewer_cRDb", "summary": "Recently, with the increasing research on LLMs, there is a diversity of LLMs, each with its own unique tokens and embedding spaces. This diversity has led to the encapsulation of knowledge and understanding within these models, posing challenges for knowledge transfer and collaboration between models. To address this challenge, one current approach is to utilize prompt design to facilitate knowledge transfer across various models through text. However, the effectiveness of this method is still constrained by the uniqueness of the token and embedding spaces between models. In response to this, this paper proposes FUSE (Flexible Unification of Semantic Embeddings), which maps the disparate embedding spaces between two LLMs without the need for fine-tuning. The paper also conducts relevant experiments, showing a considerable improvement, particularly with a nearly 50% enhancement in the SPICE evaluation metric. This paper offers a new, cost-effective method for cross-model collaboration and knowledge transfer, making a significant contribution to LLMs research.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful response. Regarding the questions raised:\n\n1) At the end of our introduction, we currently include a list of contributions that we feel are in the order of most to least important\n2) We have updated the background/related works section to have additional subheading to assist with readability\n3) Regarding the readability issues in figure 2, can you provide more clarification on the issues with the figure? Due to space constraints a vertical layout may be difficult, especially as we aimed to have the text within the figure be about the size of the main-text to ensure readability. If accepted, COLM allows for an additional page of space in the camera ready, we may then be able to explore a vertical layout for this figure.\n4) Regarding the comparison to other methods, our method is not supervised, so we cannot add additional supervised experiments. The layout of Table 1 aligns with the layout of other articles on zero-shot captioning by separating Supervised and Zero-shot methods to show the relative improvements across methods.\n\nRegarding the comparison to existing methods, to the best of our knowledge there are no other methods that look at computing gradients across tokenizers. If you are familiar with any, please let us know and we will update the experiments to reflect this in future revisions. As we are not aware of methods that we can directly compare to, we focus on showing how our framing and method can be used to improve on existing methods, if those methods can be expressed as minimizing the sum of a readability and image-text alignment objectives or other aggregates of LLMs and MLLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The acceptance of this article can mainly be attributed to the following reasons:\n1. Innovative approach: The paper introduces a novel method called FUSE (Flexible Unification of Semantic Embeddings), which maps the disparate embedding spaces between two LLMs without the need for fine-tuning. The innovation lies in addressing the challenge of knowledge transfer between LLMs while also providing a cost-effective and efficient solution.\n\n2. Contribution to LLMs research: By proposing a new method and validating it through experiments, this paper makes a contribution to the research on LLMs. This contribution lies in offering an innovative solution to the challenge of collaboration and knowledge transfer between LLMs, thereby advancing the field.", "reasons_to_reject": "1.Image layout needs improvement: Images are crucial supplementary materials in research articles. However, if the layout is unclear or not standardized, it may hinder readers' understanding and absorption of the article's content. Reviewers might consider poor image layout as detrimental to the readability and attractiveness of the article.\n\n2.Lack of comparison with other methods: In the field of research, newly proposed methods typically require comparison with existing methods to validate their effectiveness and superiority. If the article lacks comparative experiments or analysis with other methods, reviewers may perceive that the authors have not adequately demonstrated the advantages or innovation of their method.", "questions_to_authors": "1.On the second page, in the last paragraph of the Introduction section, it is suggested to list the contributions of this paper from most to least important.\n\n2.On the second page, in the Background and Related Work section, it is recommended to add subheadings to make the discussion clearer and more organized.\n\n3.On the fifth page, it is suggested to adjust the layout of Figure 2 for better readability, preferably in a vertical arrangement.\n\n4.On the eighth page, in the Experiments section, in Table 1, the proposed method from this paper is not compared with other methods in the Supervised Methods category. It's suggested to add experiments for comparison.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach for mapping the semantic embedding of two models trained with different tokenizers in a zero-shot way. More specifically, the authors first map the semantic embedding of models (with different tokenizers), and then derive an approximate gradient of one model's output with respect to the other model's input. As a result, they discover an adapter in a zero-shot way. I agree with the reviewers on the innovativeness of the approach but also highlight their feedback on evaluation in different settings: the work would be stronger and more impactful if it is evaluated in (1) other settings like mapping two LLMs (2) more benchmarks in the VLM study (3) low resource settings--especially, if the main use case of this approach is for cases where fine-tuning data is not available, e.g., low-resource languages. Moreover, while it is not possible to compare this method directly with the existing supervised adaptation approaches, the authors can consider such approaches as an upper-bound--discuss how far from the upper-bound is your current results. Also, you could try a supervised method but increasing the amount of fine-tuning data, and see what amount of data is needed to achieve your results. I strongly encourage the authors to address these comments for the next revision of the paper.
Decision: Accept