Venue: NAACL 2024

# David helps Goliath: Inference-Time Collaboration Between

Small Specialized and Large General Diffusion LMs

 Xiaochuang Han\({}^{\spadesuit\diamondsuit}\)   Sachin Kumar\({}^{\clubsuit}\)

**Yulia Tsvetkov\({}^{\clubsuit}\)   Marjan Ghazvininejad\({}^{\diamondsuit}\)  University of Washington   \({}^{\diamondsuit}\)Meta AI   \({}^{\clubsuit}\)Carnegie Mellon University {xhan77, yuliats}@cs.washington.edu asachink@cs.cmu.edu ghazvini@meta.com**

###### Abstract

Diffusion-based language models are emerging as a promising alternative to autoregressive LMs: they approach the competence of autoregressive LMs while offering nuanced controllability at inference time. While autoregressive LMs have benefited immensely from scaling and instruction-based learning, existing studies of diffusion LMs have been conducted on a smaller scale. Starting with a recently proposed diffusion model Ssd-LM, in this work we first explore methods to scale it from 0.4B to 13B parameters, proposing techniques to improve its training and inference efficiency, and to finetune the model to follow instructions. Armed with a more powerful, general purpose diffusion LM, we introduce the primary contribution of this work - Ssd-2 - an approach to easily ensemble at inference time a large general-purpose diffusion LM with smaller, but specialized and contextualized diffusion LMs. We show that Ssd-2 facilitates novel ensembles with 100x smaller models that can be customized and deployed by individual users. We find that compared to autoregressive models, the collaboration between diffusion LMs is more effective, leading to higher-quality model responses due to their ability to dynamically incorporate bi-directional contexts.1

Footnote 1: [https://github.com/xhan77/ssd-2](https://github.com/xhan77/ssd-2).

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 8385–8400 June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Following the footsteps of diffusion-based generative models for continuously valued data such as images, audio, and video Ho et al. (2020); Kong et al. (2021); Ho et al. (2022), recent works have attempted to replicate these successes on discrete text data Austin et al. (2021); Li et al. (2022); Han et al. (2022); Strudel et al. (2022); Dieleman et al. (2022). Several studies have shown that diffusion-based language models (LMs) perform competitively to their autoregressive counterparts, and even surpass them at post-hoc controllable text generation Li et al. (2022); Han et al. (2022).

Meanwhile, autoregressive language models Brown et al. (2020); Touvron et al. (2023) have emerged as general-purpose solutions capable of holding conversations with humans and solving tasks by following instructions Ouyang et al. (2022); Wang et al. (2022); Longpre et al. (2023); Taori et al. (2023); Chiang et al. (2023); Xu et al. (2023). Their abilities are primarily due to two factors: scaling the model parameters as well as pretraining datasets, and instruction finetuning with carefully curated datasets Ouyang et al. (2022).

However, as the models become increasingly immense and proprietary, it is difficult for individual users to customize the system with their own data (e.g., specialized knowledge) due to cost or privacy reasons (SS3). A primary contribution of this work is to illustrate a novel setup of _inference-time collabora

Figure 1: _Inference-time collaboration_ between a large general model and a small user model that incorporates user-specified knowledge. The collaboration between autoregressive models performs decoding token-by-token, while the collaboration between diffusion models refines a block of generated tokens iteratively with bi-directional contexts (§3).



ration_ between LMs and show a unique advantage of diffusion LMs compared to autoregressive models in this scenario. With diffusion models' iterative generation design over a span of bi-directional contexts, multiple diffusion LMs with different capabilities can be easily ensembled at the sequence level at test time, leveraging advantages of each LM in the ensemble.

As a preliminary to our experiments, we first present an exploratory study to scale and incorporate instruction-following and conversational capabilities in diffusion-based LMs. We introduce Ssd-2, an improved version of recently introduced simplex-based diffusion LM SSD-LM [14] proposing several modifications to its training and inference procedures. We incorporate these improvements in scaling SSD-2 to 13B parameters, up from 0.4B in SSD-LM. We show that similarly to autoregressive LMs, by finetuning with curated instruction datasets, SSD-2 is well-suited to follow chat-style instructions.

We then present our main case study highlighting the setup of inference-time collaboration: we augment a general-purpose large SSD-2 model with 13B parameters with a 100x smaller, user-accessible model. This setup allows incorporating user-provided knowledge into the generation process without directly inputting it into the large model (which can be undesirable due to cost or privacy reasons, more details in SS3). We show that SSD-2's instruction finetuned model is substantially more effective at this collaboration than the autoregressive baselines, leveraging bi-directional contexts in the ensemble.

## 2 Background

Semi-autoregressive simplex-based diffusion LM (SSd-LM) is trained to generate text in blocks of tokens by performing diffusion in the simplex space of the model vocabulary [15]. For text continuation tasks, it has shown competitive performance against autoregressive models (e.g., GPT-2; Radford et al., 2019) when trained with a similar number of model parameters and pretraining data. Furthermore, it naturally enables post-hoc control in generated text using off-the-shelf classifiers, outperforming prior approaches to controlling autoregressive models. Below we briefly overview the training and decoding algorithm of SSD-LM.

TrainingThe core idea behind the training of diffusion models [8] is to add a series of progressive noise to the input data representations and learn a model to reverse this process, reconstructing the original data at different noise levels. Assume we have a sequence of tokens \(\{w^{0},\dots,w^{c-1},w^{c},\dots,w^{c+B-1}\}\), where we condition on a context of length c, \(\{w^{0},\dots,w^{c-1}\}\) (or \(\mathbf{w}^{<c}\)), and learn to generate the subsequent block of text \(\{w^{c},\dots,w^{c+B-1}\}\) (or \(\mathbf{w}^{cc+B}\) using a Python-style notation) containing \(B\) tokens. In SSD-LM, a progressive Gaussian noise is added to the block of text \(\mathbf{w}^{cc+B}\).

\[\tilde{\mathbf{w}}^{cc+B}_{t}=\mathrm{logits\text{-initialization}}(\mathbf{ w}^{cc+B})\] \[\tilde{\mathbf{w}}^{cc+B}_{t}=\sqrt{\bar{\alpha}_{t}}\tilde{\mathbf{w}}^ {cc+B}_{0}+\sqrt{1-\bar{\alpha}_{t}}\mathbf{\epsilon}\]

where \(\mathrm{logits\text{-initialization}}(\cdot)\) maps each discrete token to a base, almost-one-hot logits representation in the model's vocabulary space \(V\), \(\{-K,+K\}^{|V|}\). A noise schedule \(\bar{\alpha}_{t}\) controls the level of noise \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},K^{2}\mathbf{I})\) added to the base representation, where timestep \(t\!\sim\!\mathcal{U}(1,T)\) and larger \(t\) means a noisier representation.

SSd-LM's training loss on \(\mathbf{w}^{cc+B}\) is conditioned both on the noisy representation \(\tilde{\mathbf{w}}^{cc+B}_{t}\) and the prior context \(\mathbf{w}^{<c}\) to the block.

\[\mathbb{E}_{t}[-\log p_{\theta}(\mathbf{w}^{cc+B}\mid\tilde{\mathbf{w}}^{ cc+B}_{t},\mathbf{w}^{<c})]\\ =\mathbb{E}_{t}\left[\sum_{j=c}^{j<c+B}-\log p_{\theta}(w^{j}\mid \tilde{\mathbf{w}}^{cc+B}_{t},\mathbf{w}^{<c})\right]\]

The model has access to a locally bi-directional context through the noisy representation. In contrast, the canonical autoregressive training loss for \(\mathbf{w}^{cc+B}\) would be \(\sum_{j=c}^{j<c+B}-\log p_{\theta}(w^{j}\mid\mathbf{w}^{<j})\), conditioned on the uni-directional left context only.

DecodingAt inference time, given a context \(\mathbf{w}^{<c}\), SSD-LM generates a block \(\mathbf{w}^{cc+B}\) through an iterative denoising algorithm, backtracking the noise timesteps from \(t=T\) to 1. Each iteration \(t\) consists of three main steps: (1) predict logits representation \(\mathbf{w}^{cc+B}_{\mathrm{logits},t}\) for the decoding text block using the learned model, (2) project the logits to an almost-one-hot representation \(\hat{\mathbf{w}}^{cc+B}_{t}\) in the base space \(\{-K,+K\}^{|V|}\) (with optional modifications), (3) add a Gaussian noise corresponding to timestep \(t-1\) to the projected representation and proceed to the next iteration.

\[\mathbf{w}^{cc+B}_{\mathrm{logits},t}=\mathrm{logits\text{${}_{\theta}$}}(\mathbf{w}^{cc+B}\mid\tilde{\mathbf{w}}^{cc+B}_{t},\mathbf{w}^{<c})\] \[\hat{\mathbf{w}}^{cc+B}_{t}=\mathrm{logits\text{-projection}}(\mathbf{w}^{cc+B}_{\mathrm{logits},t})\] \[\tilde{\mathbf{w}}^{cc+B}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\hat{\mathbf{w}} ^{cc+B}_{t}+\sqrt{1-\bar{\alpha}_{t-1}}\mathbf{z}\] 

where \(\theta\) is the trained diffusion model and \(\tilde{\mathbf{w}}_{T}^{:c+B}\) is initialized with a Gaussian noise.

## 3 Ssd-2

In the age of LLMs, individual users of NLP models may often face a dilemma when they wish to customize the system with their own data. On one hand, it is difficult for user-owned devices to fit very large models and smaller models are usually not powerful enough. On the other hand, uploading their data to a commercial host of large models for finetuning or long in-context learning is expensive and also may not be desirable due to privacy risks. We aim to address this dilemma in this work by proposing a collaborative inference-time algorithm between two diffusion models: a large general-purpose model (such as ones only accessible through an API) and a small model which a user can customize (SS3.2).

We first present Ssd-2 building on top of Ssd-LM with several modifications to improve its training and decoding efficiency (SS3.1). We train Ssd-2 with a larger pretraining corpus and more parameters (ranging from 0.1B to 13B) than Ssd-LM and fine-tune it to follow instructions (SS4). Next, we present how different versions of Ssd-2 (general-purpose large models and user-enhanced small models) can be effectively interpolated at inference time, outperforming their autoregressive counterparts (SS5).

### Algorithmic improvements over Ssd-Lm

Figure 2 describes the training and decoding algorithms of Ssd-2. We highlight the changes in Ssd-2 over Ssd-LM below.

Self-conditioningThe core idea behind self-conditioning Chen et al. (2022) is that at iteration \(t\), the model takes as input not just the noised sample \(\tilde{\mathbf{w}}_{t}^{cc+B}\), but also a clean output from the previous timestep \(t+1\), \(\mathbf{w}_{\text{logits},t+1}^{cc+B}\). This allows the model to reuse useful information in the previous prediction and focus on refining it in the current timestep, allowing convergence in fewer iterations. That is, for \(T>t\geq 1\):

\[\mathbf{w}_{\text{begin},t}^{cc+B}=\mathrm{logits}_{9}(\mathbf{w}^{c<b}~{}|~{}\tilde {\mathbf{w}}_{t}^{cc+B},\mathbf{w}_{\text{begin},t+1}^{cc+B},\mathbf{w}_{\text{begin}, t+1}^{cc+B},\mathbf{w}^{cc+B}_{\text{end},t+1},\mathbf{w}^{cc})\]

More specifically, the noisy representation \(\tilde{\mathbf{w}}_{t}^{cc+B}\) and the previous timestep prediction \(\mathbf{w}_{\text{logits},t+1}^{cc+B}\) are combined before the transformer blocks of \(\theta\), along with the positional embeddings and timestep embeddings as follows:2

Footnote 2: As a shorthand, we dropped the superscript for token positions \(c\) to \(c+B\), and use \(\mathrm{sm}\) for \(\mathrm{softmax}\), \(\mathrm{Emb}\) for the embedding layer, and \(W_{\text{fair}}\) and \(W_{\text{freq}}\) for the embedding matrix for the noisy representation and self-conditioning prediction.

\[\tilde{\mathbf{h}} = W_{\text{diff}}[\mathrm{sm}(\tilde{\mathbf{w}}_{t})]+W_{\text{pred}} [\mathrm{sm}(\mathbf{w}_{\text{logits},t+1})]\] \[+\mathrm{Emb}_{\text{pos}}(c:c+B)+\mathrm{Emb}_{\text{diff-time}}( t/T)\] \[\mathbf{h}^{<c} =\mathrm{Emb}_{\text{ctx}}(\mathbf{w}^{<c})+\mathrm{Emb}_{\text{ pos}}(<c)\] \[+\mathrm{Emb}_{\text{ctx-time}}(t/T)\] \[\mathbf{w}_{\text{logits},t}^{ccc+B}=\mathrm{Transformer}[\mathrm{ concat}(\mathbf{h}^{<c},\tilde{\mathbf{h}})]^{cc+B}\]

To train the model to learn to reuse the predicted logits, we add an additional forward pass during the training phase, activated with a probability \(p=0.5\). We predict \(\mathbf{w}_{\text{logits},t}^{cc+B}\) disabling gradient backpropagation, and use it in the new cross entropy loss \(-\log p_{\theta}(w^{j}~{}|~{}\tilde{\mathbf{w}}_{t}^{cc+B},\mathbf{w}_{\text{logits},t}^{ cc+B},\mathbf{w}^{<c})\).

Removing context length sampling for efficiencyThe original training algorithm of Ssd-LM first samples a context length \(c\sim\mathcal{U}(1,|\mathbf{w}|-B)\) for each example, encodes the context bi-directionally and computes the diffusion loss for a block of \(B\) tokens following that context. The bi-directional encoding of the context \(\mathbf{w}^{<c}\) cannot be shared across different context sizes \(c\) for the same example. Moreover, when the sequence length \(|\mathbf{w}|\) is large, a high variance in the sampled \(c\) across devices in distributed training reduces the effective batch size, slowing down the training considerably. Therefore, in the pretraining and finetuning of Ssd-2, we eliminate sampling different \(c\)'s while equivalently modeling the same training loss as shown in Figure 2 for all \(\frac{|\mathbf{w}|}{B}\) blocks in one data, by using a special attention mask. The transformer modules of Ssd-2 encode the context \(\mathbf{w}^{<c}\) uni-directionally while preserving the bi-directional attention for the diffusion generation block \(\mathbf{w}^{c:c+B}\). This leads to a 2x speedup in our pilot pretraining. More details can be found in SSA.

Sharded models across time-ranges and early stopping in decodingWe observe that at test time Ssd-2 often shows distinct behaviors at different timestep ranges. We empirically divide the number of iterations into five ranges of equal sizes. In the beginning of decoding (\(t\in(0.8T,T]\)), when the noise level is very high, there is no discernable pattern in which the model's intermediate predictions (\(\operatorname*{argmax}\mathbf{w}_{\text{logits},t}^{cc+B}\)) in different iterations differ 

from each other. Larger changes often happen at \(t\in(0.6T,0.8T]\) after which the majority of the content is in place, and for \(t\in(0.4T,0.6T]\) only minor changes happen sparsely to make a grammatical correction or settle down on an uncertain word choice. Finally, for \(t\in(0,0.4T]\), the sequence does not update at all in most cases. We hence hypothesize that the first three timestep ranges require different capabilities from the model. In Ssd-2, we propose to optionally train three separate models \(\theta_{(0.4,0.6)}\), \(\theta_{(0.6,0.8)}\), and \(\theta_{(0.8,1.0)}\) for the three ranges.3 We still train a single model at pretraining to save resources and only perform this step during a final finetuning as described in SS4.4 We start the decoding at \(t=T\) and stop at \(t=0.4T\), saving 40% of the inference computation.5

Footnote 3: A similar setup has also been explored in image diffusion as expert denoisers Feng et al. (2022); Balaji et al. (2022).

Footnote 4: This setup could further be improved by considering models of different sizes for the three ranges where \(\theta_{(0.4,0.6)}\) and \(\theta_{(0.8,1.0)}\) could contain fewer parameters as they arguably perform simpler tasks to reduce the effective inference time. We leave it as future work.

```
1:\(\hat{\mathbf{w}}^{cic+B}_{t}=\text{logits-initialization}(\mathbf{w}^{ec+B})\)
2:\(t\sim\text{Uniform}(\{1,\dots,T\})\)
3:\(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},K^{2}\mathbf{I})\)
4:\(\tilde{\mathbf{w}}^{cc+B}_{t}=\sqrt{\alpha_{t}}\tilde{\mathbf{w}}^{ec+B}_{0}+\sqrt{1- \bar{\alpha}_{t}}\mathbf{\epsilon}\)
5:\(r\sim\text{Bernoulli}(0.5)\)
6:if\(\tau=0\)then
7: Take \(\mathbf{\epsilon}\) gradient descent step on \(\nabla_{b}[\sum_{j=c}^{j<e+B}-\log p_{b}(w^{i}~{}|~{}\tilde{\mathbf{w}}^{ec+B}_{ t},\mathbf{w}^{ec})]\)
8:else
9: With gradient calculation disabled, calculate \(\mathbf{w}^{e+B}_{\text{input},t}=\mathrm{logits}_{b}(\mathbf{w}^{ec+B}\mid\tilde{\mathbf{ w}}^{ec+B}_{t},\mathbf{w}^{ec})\)
10: Take a gradient descent step on \(\nabla_{a}[\sum_{j=c}^{j<e+B}-\log p_{b}(w^{i}~{}|~{}\tilde{\mathbf{w}}^{ec+B}_{t} ,\mathbf{w}^{ec+B}_{\text{input},t},\mathbf{w}^{ec})]\)
11:endif
```

**Algorithm 1** Training (at a given \(c\))

### Inference-time collaboration

As shown in Ssd-LM Han et al. (2022) and prior work in other domains Dhariwal and Nichol (2021), diffusion models are naturally suited to allow for controlling the properties of the model outputs by interpolating the model outputs with gradients from a control function such as a classifier. Follow-up studies have extended this idea to classifier-free guidance where diffusion models with and without controlling attributes can be interpolated contrastively using a weighted sum of their outputs Ho and Salimans (2021). We explore a new setup of the latter idea for enabling collaboration between two versions of Ssd-2 where we interpolate the output logits of the models. Intrinsic to the diffusion paradigm, this interpolation is sequence-level and through many iterations it leverages benefits of the bi-directional context.

SetupWe first define a _core_ model \(\theta_{\text{core}}\) which is computationally expensive to train or deploy (e.g., a large model which can only be loaded on mutiple GPUs). We assume the model is good at general-domain instruction following. We then define a _user_ model \(\theta_{\text{user}}\) which is computationally friendly for a typical user to run on their personal device or a cloud device to their control. It allows incorporating data of their specific interest which they may not prefer to input to the large model. For both the core and user models, we also assume they do not have access to each others' model parameters.

We also assume a prompting instruction \(\mathbf{w}_{\text{inst}}\) which both the models have access to, and expert data \(D_{\text{user}}\) that only the user model and not the core model has access to (see Figure 1). During inference,

* \(\theta_{\text{core}}\) only takes in the prompt \(\mathbf{w}_{\text{inst}}\), \(f_{\theta_{\text{core}}}(\mathbf{w}_{\text{inst}})\).
* \(\theta_{\text{user}}\) can be finetuned with \(D_{\text{user}}\), or use \(D_{\text{user}}\) in in-context learning. In this work, we experiment with the latter setup, where the user

Figure 2: Training and decoding algorithms for Ssd-2. The training algorithm describes the training objective at an arbitrary context length \(c\). The decoding algorithm can be applied multiple rounds by appending the generation from one round to the context for the next. The decoding may stop after a fixed number of rounds or until a special end-of-sequence token is encountered.

 

model takes in both the user expert data and the instruction as input, \(f_{\theta_{\text{user}}}(D_{\text{user}},\mathbf{w}_{\text{inst}})\).
* Additionally, we assume the model size \(|\theta_{\text{core}}|\gg|\theta_{\text{user}}|\) (the size difference is 100x in our experiments).

We will discuss the specific instantiation of the setup in SS5. In the section below, we first introduce a prominent collaboration algorithm when \(\theta_{\text{core}}\) and \(\theta_{\text{user}}\) are autoregressive, and then propose a novel algorithm when the models are diffusion-based Ssd-2.

MethodThe collaboration between \(\theta_{\text{core}}\) and \(\theta_{\text{user}}\) is essentially an ensemble of the model outputs. One prominent way of approaching it is through a weighted average of the models' logits at inference time.6 For autoregressive LMs, this averaging can be performed at the token level where the logits are first combined and then transformed into probability distribution like a product-of-experts ensemble (e.g., Liu et al. (2021)).

Footnote 6: Training-time ensemble can be achieved through methods like parameter-averaging Li et al. (2022). However, it is not the focus of this work since our models have drastically different shapes and we do not assume the models have access to the parameters of other models.

\[w^{c} \sim p_{\text{collab}}(w^{c}\mid D_{\text{user}},\mathbf{w}_{\text{ inst}},\mathbf{w}^{<c})\] \[=\operatorname{softmax}[(1-\lambda_{\text{user}})\operatorname{ logits}_{\theta_{\text{core}}}(w^{c}\mid\mathbf{w}_{\text{inst}},\mathbf{w}^{<c})\] \[+\lambda_{\text{user}}\operatorname{logits}_{\theta_{\text{ coet}}}(w^{c}\mid D_{\text{user}},\mathbf{w}_{\text{inst}},\mathbf{w}^{<c})]\]

We also consider an extension of this setup where we add a contrastive term to \(\theta_{\text{user}}\) without the input \(D_{\text{user}}\), to promote the pointwise mutual information between the expert data and the generation conditioned on the instruction Malkin et al. (2021).7

Footnote 7: We set the contrastive hyperparameter \(\alpha=1.0\) throughout the evalution, though the results with \(\alpha=0.0\) follow a similar trend.

\[w^{c} \sim\operatorname{softmax}[(1-\lambda_{\text{user}}) \operatorname{logits}_{\theta_{\text{out}}}(w^{c}\mid\mathbf{w}_{\text{inst}},\mathbf{w}^{< c})\] \[+\lambda_{\text{user}}(1+\alpha)\operatorname{logits}_{\theta_{\text{new}}}(w^{c}\mid D_{\text{user}},\mathbf{w}_{\text{inst}},\mathbf{ w}^{<c})\] \[-\lambda_{\text{user}}\alpha\operatorname{logits}_{\theta_{\text{new}}}(w^{c}\mid\mathbf{w}_{\text{inst}},\mathbf{w}^{<c})]\]

For Ssd-2, the process of generating tokens is intrinsically different from autoregressive models. However, since it preserves the notion of logits in its iterative decoding procedure (\(\mathbf{w}^{cc+B}_{\text{logits},t}\)), we propose a similar logits-averaging method for a diffusion \(\theta_{\text{core}}\) and \(\theta_{\text{user}}\), performing an ensemble for a block of tokens at each diffusion timestep.

\[\mathbf{w}^{cc+B}_{\text{one-begin}}=\operatorname{logits}_{\theta_{\text{ core}}}(\mathbf{w}^{cc+B}\mid\mathbf{w}_{\text{inst}},\mathbf{w}^{<c},\tilde{\mathbf{w}}^{ cc+B}_{i})\] \[\mathbf{w}^{cc+B}_{\text{one-begin}}=\operatorname{logits}_{\theta_{\text{core}}}(\mathbf{w}^{cc+B}\mid\mathbf{w}_{\text{inst}},\mathbf{w}^{<c}, \tilde{\mathbf{w}}^{cc+B}_{i})\] \[\mathbf{w}^{cc+B}_{\text{one-begin}}=\operatorname{logits}_{\theta_{\text{core}}}(\mathbf{w}^{cc+B}\mid\mathbf{w}_{\text{inst}},\mathbf{w}^{<c}, \tilde{\mathbf{w}}^{cc+B}_{i})\] \[\quad\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;  

still considerably undertrained. Due to our computing budget, we leave to future work a potential continued pretraining over current Ssd-2 models on larger and better curated data.

Instruction finetuningWhile Han et al. (2022) show the effectiveness of pretrained Ssd-LM in general-domain text continuation, in this work, we primarily investigate the use of Ssd-2 in downstream fine-tuning tasks, particularly on chat-style instruction following.11 We finetune the models with the dolly dataset12 containing 15K human-collected instructions and responses Databricks (2023). dolly covers categories like open/closed-QA, brainstorming, and creative writing, though it may still be less powerful than the distillation-based data in terms of size and quality.13 We finetune on 95% of the dolly data and use the rest for held-out evaluation. We finetune with a batch size of 384 and for 500 or 1000 steps for the 0.1B/2.7B/13B models. As a baseline, we finetune the autoregressive model OPT (0.1B/2.7B/13B) on dollly with the same setup.

Footnote 11: We make an additional change while finetuning Ssd-2 to address end of sequence (EOS) issues in variable length sequences in the downstream datasets. Since a sequence could terminate in the middle of a diffusion block, while training, we pad the sequence with the EOS token to the nearest boundary of a diffusion block of size \(B\). We do not mask this padding while computing the loss. We use the standard padding token after the last diffusion block boundary. At inference, if the generated text block \({\rm argmax}\,\hat{\mathbf{w}}_{\text{o}}^{c+cl}\) in the final iteration contains an EOS token, we prune the trailing tokens after the first EOS token in the block.

Footnote 12: [https://huggingface.co/datasets/databases/databricks-doally-15k](https://huggingface.co/datasets/databases/databricks-doally-15k). We deliberately choose to finetune with dolly because as opposed to other similar datasets (e.g. the ones used to train models like Alpaca Taori et al. (2023) and Vicuna Chiang et al. (2023)), dolly has an open-source license and does not depend on distillations from OpenAI models.

## 5 Experiments

### Inference-time collaboration

As introduced in SS3.2, a main focus of this work is to explore the advantages of a diffusion-based LM Ssd-2 in a collaboration setup: interpolating the outputs of a large, general model \(\theta_{\text{core}}\) and a small model \(\theta_{\text{user}}\) enhanced by user expert data \(D_{\text{user}}\).

We use the 13B-parameter Ssd-2 finetuned with dolly as \(\theta_{\text{core}}\) and the 0.1B finetuned Ssd-2 as \(\theta_{\text{user}}\). We use OPT 13B and 0.1B finetuned with dolly under the same collaboration setup as the autoregressive baseline. dolly's held-out test prompts are used as \(\mathbf{w}_{\text{inst}}\). A subset of dolly test examples is annotated with loosely related Wikipedia passages to support the output answers; we use these passages as a proxy for \(D_{\text{user}}\). To avoid prompts with trivial answers, we additionally constrain the test instructions to those with an original annotated response of at least 50 tokens.

Inference-time collaboration is effective if the core model \(\theta_{\text{core}}\) generates better responses after collaborating with the 100x smaller but user-enhanced \(\theta_{\text{user}}\). We investigate a range of weights \(\lambda_{\text{user}}\), starting from 0 where the output of the collaboration solely depends on the large \(\theta_{\text{core}}\), and gradually increasing \(\lambda_{\text{user}}\) to incorporate more \(\theta_{\text{user}}\).

Automatic evaluationWe first conduct an automatic evaluation, using state-of-the-art, production-level LMs to evaluate the quality of our models' generations, which have been shown to correlate highly with human judgments and are easier to scale Liu et al. (2023). We use GPT-3.5-turbo to rate our models' responses to the test instructions on a scale of 10, towards the aspects of relevance, factuality, informativeness, coherence, and understandability. The specific prompting template we used is detailed in SSG.

Table 1 summarizes the automatic evaluation results. We observe that when \(\lambda_{\text{user}}=0\) (\(\theta_{\text{core}}\) only, no \(D_{\text{user}}\) incorporated), the OPT model finetuned with dolly consistently outperforms our finetuned Ssd-2.14 However, for OPT, collaborating with the small user model does not improve the core model's performance any further across all considered weights. Within the experimented weighting factors, \(\lambda_{\text{user}}\) of 0.1 to 0.3 is relatively optimal, though still leading to lower scores than without collaboration.

Footnote 14: We conjecture the reason is that Ssd-2 is undertrained as discussed in §4 and can have a larger domain gap w.r.t. the dolly data. In §5.2, we compare the intrinsic instruction following ability of Ssd-2 and OPT using one-shot in-context learning instead of finetuning.

In contrast, the small user model \(\theta_{\text{user}}\) improves the core model's performance in all tested attributes in Ssd-2. With appropriate weight factors (\(\lambda_{\text{user}}=0.2,0.3\)), the collaborated Ssd-2 system surpasses the best OPT performance in four of the five metrics and matches the fifth. We highlight in Table 1 the best absolute 

performance gain due to the collaboration. We additionally show that when \(\lambda_{\text{user}}=1\), the small user model \(\theta_{\text{user}}\) alone performs worse in Ssd-2 than in OPT. This further indicates that the observed performance gain comes from an effective collaboration rather than a significantly better \(\theta_{\text{user}}\).

Human evaluationTo corroborate our findings, we further perform a human evaluation comparing the outputs from SSD-2 and OPT under a collaborative setup. For each test prompt, we show SSD-2 and OPT responses with \(\lambda_{\text{user}}\) of 0.2 to the human annotators as a randomized pair. We asked the annotators to choose the preferred response while allowing for annotating equally good responses or equally bad responses. A total of 9 annotators (graduate and undergraduate researchers in NLP, not authoring this work) made 259 human preference annotations over 94 test prompts, with each response pair receiving 1-4 annotations. We show in Table 2 that the collaboration between SSD-2 \(\theta_{\text{core}}\) and \(\theta_{\text{user}}\) is overall more preferred by humans to the OPT models under the same setup. SSD-2 wins in 43 cases (45.7%) while loses only in 25 cases (26.6%). We additionally measure an average Cohen's kappa coefficient between all pairs of annotators who annotated the same subset of instances. We observe \(\kappa\)=0.31 indicating a fair agreement, especially that the task is highly subjective by nature.

Overall, through automatic and human evaluations, we show that SSD-2 offers unique benefits in an interesting case of inference-time collaboration, effectively fusing a general-purpose large model and a small model enhanced by some expert data.

### Ablation study: Ssd-2 as a standalone diffusion chat model

In this section, we divert from our main inference-time collaboration setup and investigate the capabil

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & \(\lambda_{\text{user}}=0\) & \(\lambda_{\text{user}}=0.1\) & \(\lambda_{\text{user}}=0.2\) & \(\lambda_{\text{user}}=0.3\) & \(\lambda_{\text{user}}=0.4\) & \(\lambda_{\text{user}}=0.5\) & \(\lambda_{\text{user}}=1.0\) \\ \hline Relevance & & & & & & \\ OPT\{core,user\} & 9.76 & 9.59 & 9.61 & 9.65 & 9.65 & 9.39 & 8.23 \\ SSD-2\{core,user\} & 9.72 & 9.65 & **9.91** & 9.85 & 9.64 & 9.52 & 7.16 \\ \(\Delta_{\text{collab}}\) OPT & & -0.17 & -0.15 & -0.11 & -0.11 & -0.37 & -1.53 \\ \(\Delta_{\text{collab}}\) SSD-2 & & -0.07 & **+0.19** & +0.13 & -0.08 & -0.20 & -2.56 \\ Factuality & & & & & & & \\ OPT\{core,user\} & **9.64** & 9.57 & 9.51 & 9.55 & 9.57 & 9.27 & 8.15 \\ SSD-2\{core,user\} & 9.34 & 9.49 & 9.63 & **9.64** & 9.56 & 9.48 & 7.26 \\ \(\Delta_{\text{collab}}\) OPT & & -0.06 & -0.12 & -0.08 & -0.05 & -0.29 & -1.44 \\ \(\Delta_{\text{Collab}}\) SSD-2 & & +0.14 & +0.30 & **+0.31** & +0.26 & +0.15 & -2.03 \\ Informativeness & & & & & & & \\ OPT\{core,user\} & 9.30 & 9.20 & 9.12 & 9.27 & 9.06 & 8.95 & 7.41 \\ SSD-2\{core,user\} & 8.97 & 9.02 & 9.33 & **9.36** & 9.06 & 8.97 & 6.38 \\ \(\Delta_{\text{collab}}\) OPT & & -0.10 & -0.18 & -0.03 & -0.24 & -0.35 & -1.89 \\ \(\Delta_{\text{collab}}\) SSD-2 & & +0.05 & +0.36 & **+0.39** & +0.09 & 0.00 & -2.59 \\ Coherence & & & & & & & \\ OPT\{core,user\} & 9.61 & 9.47 & 9.37 & 9.44 & 9.41 & 9.13 & 7.70 \\ SSD-2\{core,user\} & 9.41 & 9.35 & **9.65** & 9.59 & 9.25 & 9.17 & 5.84 \\ \(\Delta_{\text{collab}}\) OPT & & -0.14 & -0.24 & -0.17 & -0.20 & -0.48 & -1.91 \\ \(\Delta_{\text{collab}}\) SSD-2 & & -0.06 & **+0.24** & +0.18 & -0.16 & -0.24 & -3.57 \\ Understandability & & & & & & & \\ OPT\{core,user\} & 9.66 & 9.54 & 9.53 & 9.54 & 9.51 & 9.30 & 8.10 \\ SSD-2\{core,user\} & 9.53 & 9.56 & **9.72** & 9.67 & 9.42 & 9.34 & 6.21 \\ \(\Delta_{\text{collab}}\) OPT & & -0.12 & -0.13 & -0.12 & -0.15 & -0.36 & -1.56 \\ \(\Delta_{\text{collab}}\) SSD-2 & & +0.03 & **+0.19** & +0.14 & -0.11 & -0.19 & -3.32 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Evaluation of the inference-time collaboration between the large core model \(\theta_{\text{core}}\) and the small user model \(\theta_{\text{user}}\). A negative impact led by \(\theta_{\text{user}}\) to \(\theta_{\text{core}}\) is marked in red, and a positive impact in blue. SSD–2 is substantially more collaborative than the autoregressive OPT baseline.

 

ities of Ssd-2 as a standalone language model. We are interested in the instruction following ability intrinsic to the vanilla Ssd-2 _without_ inference-time collaboration or any finetuning (like with dolly). We compare original Ssd-2 and OPT 13B in responding to the prompts from the Vicuna test set Chiang et al. (2023), which include problems of open-ended question answering, creative writing, etc.15 We formulate the setup as a one-shot inncontext learning problem. Before each Vicuna test prompt, we add one fixed, handcrafted in-context example from Zhou et al. (2023a) to help the models capture the format of the answers without changing the model parameters.

Footnote 15: Out of the 80 Vicuna test prompts, we empirically find both models constantly fail on prompts from the math and coding categories. We therefore filter them out and keep the rest 70 test cases for our experiments.

The main metric we report is the win rate from an automatic evaluation based on GPT-4 OpenAI, 2023). We follow the original evaluation template as introduced in Chiang et al. (2023), prompting GPT-4 to rate SSD-2 and OPT responses along with explanations. As additional metrics, we also compute the conditional perplexity of the responses using external language models GPT-Neo-1.3B Black et al. (2021) and GPT-2-large Radford et al. (2019). While there are no gold answers to the Vicuna test prompts, we use GPT-3.5's responses as reference answers and subsequently compute a BERTScore w.r.t. them for the responses from SSD-2 and OPT. As shown in Table 3, we overall observe a higher win rate, lower perplexity, and higher BERTScore for our diffusion language model SSD-2 compared to the autoregressive OPT. We additionally evaluate SSD-2 finetuned with dolly and report results in SSE. We show some qualitative examples of SSD-2's generations in SSF.

## 6 Related work

Diffusion-based language models have been receiving increasing attention as a potential alternative to autoregressive language models. We identify three main categories of diffusion language models based on how they represent discrete data like text. Discrete diffusion language models represent language naturally as categorical data, while the diffusion or noising steps are often formulated as transition matrices Hoogeboom et al. (2021); Austin et al. (2021); He et al. (2022); Reid et al. (2022); Zheng et al. (2023); Zhou et al. (2023b). Embedding-based diffusion language models often learn a mapping between the discrete language tokens and an embedding latent space, and the diffusion process is on the embedding space via a series of Gaussian noise Li et al. (2022c); Gong et al. (2022); Dieleman et al. (2022); Gao et al. (2022); Lovelace et al. (2022); Yuan et al. (2022); Lin et al. (2022); Ye et al. (2023); Chen et al. (2023); Tang et al. (2023); Balagansky and Gavrilov (2023). In this work, we focus on simplex-based diffusion language models that project discrete tokens to a simplex space and perform the diffusion process with a simple Gaussian noise Han et al. (2022); Mahabadi et al. (2023). Our proposed inference-time collaboration setup is most straightforward to apply to simplex-based diffusion language models, since models with different sizes share the same simplex (vocabulary) space. Embedding-based models over different latent representation spaces are not suitable for a direct representation interpolation. Furthermore, to the best of our knowledge, SSD-2 is the first of this line of literature to pretrain and finetune a diffusion language model as a chat model, encouraging future work to compare and improve over our work.

With autoregressive language models, various efforts have been made towards building chat-style instruction following models based on open source language models Touvron et al. (2023); Biderman et al. (2023) to replicate strong production-level closed source counterparts Ouyang et al. (2022);

\begin{table}
\begin{tabular}{l l l} \hline \hline Ssd-2collab win & Draw & OPTcollab win \\ \hline
43 (45.7\%) & 26 (27.7\%) & 25 (26.6\%) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Human preference of the outputs from the inference-time collaboration experiments, comparing the diffusion-based SSD-2 and the autoregressive OPT.

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & Win rate & PPL & BERTScore \\  & & (GPT-Neo/GPT2) & (Precision/F1) \\ \hline Ssd-213B & 52.3\% & 7.58 / 9.62 & 85.9 / 85.2 \\ OPT13B & 47.7\% & 8.44 / 10.08 & 85.3 / 84.9 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Original SSD-2 responding to Vicuna test instructions in an one-shot in-context learning setup. The win rate is computed between SSD-2 and OPT models using the original GPT-4 evaluation introduced in Chiang et al. (2023). BERTScore is computed for the model responses w.r.t. the generations from GPT-3.5.

 

OpenAI, 2023). Many of such work are concurrent to ours and collect high-quality finetuning datasets by distilling prompts and responses from OpenAI models Taori et al. (2023); Chiang et al. (2023); Xu et al. (2023). In this work, we deliberately seek fully open source data not depending on OpenAI models and adopt the dally data Databricks (2023). We expect our models can be further improved with future releases of more curated chat-style instruction tuning datasets Zhou et al. (2023).

One novel setup we explored in this work is the inference-time collaboration between a large, general-purpose diffusion chat model and small, user-specific models. Inference-time collaboration has been generally explored in autoregressive models via ensembles of logits, either in an interpolation or contrastive manner Liu et al. (2021); Malkin et al. (2021); Li et al. (2022); Peng et al. (2022); Li et al. (2022). In diffusion models, classifier-free guidance in image generation Ho and Salimans (2021) contrastively reconstruct representations with and without a controlling attribute using a single model, whereas our work collaboratively decode with models with different sizes and inputs. We show an unique advantage of simplex-based diffusion language models in such inference-time collaboration compared to autoregressive language models.

## 7 Conclusion

We present an exploratory step towards pretraining a large simplex-based diffusion language model Ssd-2 and finetuning it with an open-source chat-style instruction dataset. In a motivated setup where large general models and small user models are to collaborate with each other at inference time, we find Ssd-2 substantially more collaborative than its autoregressive counterparts. These findings show the promise of diffusion language models as an instruction-following chat model and a worthy alternative to autoregressive language models.

### Limitations

In this work, we explore a novel setup of fusing large general diffusion language models and small customizable models enhanced with user expert data. One limitation of the proposed fusion algorithm is that it requires a search through a range of candidate balancing factors \(\lambda_{\text{user}}\). Furthermore, a selected balancing factor remains the same across different diffusion timesteps, which is not necessarily optimal. Future work can explore and learn an optimal, dynamic schedule of the balancing factors. Another limitation of diffusion language models in general is a slow decoding speed compared to autoregressive models. Though our proposed Ssd-2 model already includes improvements over the original Ssd-LM leading to faster decoding speed (more details in SSD), future work may further adapt methods from image diffusion models targeting specifically for efficient decoding Song et al. (2021); Nichol and Dhariwal (2021); Rombach et al. (2022); Meng et al. (2022).

## Acknowledgements

The authors would like to thank Alisa Liu, Jiacheng Liu, Weijia Shi, Zihao Ye, members of TsvetShop, and the anonymous reviewers for their insightful discussions and feedback. We additionally thank Shangbin Feng, Tianxing He, Abe Hou, Yuhan Liu, Heng Wang, Jack Zhang, and Michael Zhang for their helpful evaluation. X.H. gratefully acknowledges funding from the UW-Meta AI Mentorship program. This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-2207220004. This material is also funded in part by the DARPA Grant under Contract No. HR001120C0124. We also gratefully acknowledge support from NSF CAREER Grant No. IIS2142739, NSF Grants No. IIS2125201, IIS2203097, and the Alfred P. Sloan Foundation Fellowship. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

## References

* Austin et al. (2021) Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. 2021. Structured denoising diffusion models in discrete state-spaces. In _Proc. NeurIPS_.
* Balagansky and Gavrilov (2023) Nikita Balagansky and Daniil Gavrilov. 2023. Democratized diffusion language model. _ArXiv_, abs/2305.10818 

Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jianming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Alia, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu. 2022. eConfir-i: Text-to-image diffusion models with an ensemble of expert denoisers. _ArXiv_, abs/2211.01324.
* Biderman et al. (2022) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Afhah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raffy, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2022. Pythia: A suite for analyzing large language models across training and scaling.
* Black et al. (2021) Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with MeshTensorflow.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlisi, Alex Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. _ArXiv_, abs/2005.14165.
* Chen et al. (2023) Jiaao Chen, Aston Zhang, Mu Li, Alexander J. Smola, and Diyi Yang. 2023. A cheaper and better diffusion language model with soft-masked noise. _ArXiv_, abs/2304.04746.
* Chen et al. (2022) Ting Chen, Ruixiang Zhang, and Geo rey E. Hinton. 2022. Analog bits: Generating discrete data using diffusion models with self-conditioning. _ArXiv_, abs/2208.04202.
* Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
* Databricks (2023) Databricks. 2023. Databricks dolly 15k. [https://huggingface.co/datasets/databricks/databricks-doolly-15k](https://huggingface.co/datasets/databricks/databricks-doolly-15k).
* Dhariwal and Nichol (2021) Prafulla Dhariwal and Alex Nichol. 2021. Diffusion models beat gans on image synthesis. _ArXiv_, abs/2105.05233.
* Dieleman et al. (2022)Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, A. Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Remi Leblond, Will Grathwohl, and Jonas Adler. 2022. Continuous diffusion for categorical data. _ArXiv_, abs/2211.15089.
* Feng et al. (2022) Zhidan Feng, Zhenyu Zhang, Xinton Yu, Yewei Fang, Lanxin Li, Xuvi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shi Feng, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2022. Ernie-vilg 2.0: Improving text-to-image diffusion model with knowledge-enhanced mixture-of-denoising-experts. _ArXiv_, abs/2210.15257.
* Gao et al. (2022)Zhujin Gao, Junliang Guo, Xuejiao Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, and Linli Xu. 2022. Difformer: Empowering diffusion model on embedding space for text generation. _ArXiv_, abs/2212.09412.
* Gong et al. (2022) Shansen Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2022. Diffuseq: Sequence to sequence text generation with diffusion models. _ArXiv_, abs/2210.08933.
* Han et al. (2022) Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. 2022. ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. _ArXiv_, abs/2210.17432.
* He et al. (2022) Zhengfu He, Tianxiang Sun, Kuan Wang, Yuanjing Huang, and Xipeng Qiu. 2022. Diffusionbert: Improving generative masked language models with diffusion models. _ArXiv_, abs/2211.15029.
* Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In _Proc. NeurIPS_.
* Ho and Salimans (2021) Jonathan Ho and Tim Salimans. 2021. Classifier-free diffusion guidance. In _NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications_.
* Ho et al. (2022) Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. 2022. Video diffusion models. _ArXiv_, abs/2204.03458.
* Loogeboom et al. (2021) Emiel Loogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forre, and Max Welling. 2021. Argmax flows and multinomial diffusion: Learning categorical distributions. In _Proc. NeurIPS_.
* Kong et al. (2021) Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. 2021. Diffwave: A versatile diffusion model for audio synthesis. In _Proc. ICLR_.
* Li et al. (2022a) Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. 2022a. Branch-train-merge: Embarrassingly parallel training of expert language models. _ArXiv_, abs/2208.03306.
* Li et al. (2022b) Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2022b. Contrastive decoding: Open-ended text generation as optimization. _ArXiv_, abs/2210.15097.
* Li et al. (2022c) Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. 2022c. Diffusion-lm improves controllable text generation. _ArXiv_, abs/2205.14217 

Zheng-Wen Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Weizhu Chen, and Nan Duan. 2022. Genie: Large scale pre-training for text generation with diffusion model. _ArXiv_, abs/2212.11685.
* Liu et al. (2021) Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. 2021. Dexperts: Decoding-time controlled text generation with experts and anti-experts. In _Proc. ACL_.
* Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuo Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. _ArXiv_, abs/2303.16634.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _ArXiv_, abs/1907.11692.
* Longpre et al. (2023) S. Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The final collection: Designing data and methods for effective instruction tuning. _ArXiv_, abs/2301.13688.
* Lovelace et al. (2022) Justin Lovelace, Varsha Kishore, Chao gang Wan, Eliot Shekhtman, and Kilian Q. Weinberger. 2022. Latent diffusion for language generation. _ArXiv_, abs/2212.09462.
* Mahabadi et al. (2023) Rabeeh Karimi Mahabadi, Jaesung Tae, Hamish Ivison, James Henderson, Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2023. Tess: Text-to-text self-conditioned simplex diffusion. _ArXiv_, abs/2305.08379.
* Maklin et al. (2021) Nikolay Maklin, Zhen Wang, and Nebojsa Joic. 2021. Coherence boosting: When your pretrained language model is not paying enough attention. In _Annual Meeting of the Association for Computational Linguistics_.
* Meng et al. (2022) Chenlin Meng, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. 2022. On distillation of guided diffusion models. _ArXiv_, abs/2210.03142.
* Nichol and Dhariwal (2021) Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. In _Proc. ICML_.
* OpenAI (2023) OpenAI. 2023. GPT-4 technical report. _ArXiv_, abs/2303.08774.
* Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. _ArXiv_, abs/2203.02155.
* Peng et al. (2022) Xiangyu Peng, Chen Xing, Prafulla Kumar Choubey, Chien-Sheng Wu, and Caiming Xiong. 2022. Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning. _ArXiv_, abs/2210.12587.
* Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
* Raffel et al. (2020) Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _JMLR_.
* Reid et al. (2022) Machel Reid, Vincent J. Hellendoorn, and Graham Neubig. 2022. Diffuser: Discrete diffusion via edit-based reconstruction. _ArXiv_, abs/2210.16886.
* Rombach et al. (2022) Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. 2022. High-resolution image synthesis with latent diffusion models. _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 10674-10685.
* Song et al. (2021) Jiaming Song, Chenlin Meng, and Stefano Ermon. 2021. Denoising diffusion implicit models. In _Proc. ICLR_.
* Strudel et al. (2022) Robin Strudel, Corentin Tallec, Florent Altch'e, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, L. Sifre, and Remi Leblond. 2022. Self-conditioned embedding diffusion for text generation. _ArXiv_, abs/2211.04236.
* Tang et al. (2023) Zecheng Tang, Pinzheng Wang, Keyan Zhou, Juntao Li, Ziqiang Cao, and M. Zhang. 2023. Can diffusion model achieve better performance in text generation? bridging the gap between training and inference! _ArXiv_, abs/2305.04465.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunri B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izcard, Xavier Martinet, Marie-Anne Lacthaux, Timothee Lacroix, Baptiste Rozziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aur'elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLama: Open and efficient foundation language models. _ArXiv_, abs/2302.13971.
* Wang et al. (2023) Yizhong Wang, Swaroop Mishra, Pegah Alipomermolabshi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Artul Selvan Dhanasekaran, Athara Naik, David Step, Esbaan Pathak, Giannius Karamanolakis, Hazii Gary Lai, Ishnan Purohit, Ishnait Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maiteya Patel, Kuntal Kumar Pal, M. Moradshahi, Mihir Parmar, Marial Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehabj Singh Puri, Rushang Karia, Shahilaja Keyr 

Sampat, Savan Doshi, Siddharth Deepak Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hanna Hajishirzi, and Daniel Khashabi. 2022. Supernatural instructions: Generalization via declarative instructions on 1600+ nlp tasks. In _Conference on Empirical Methods in Natural Language Processing_.
* Xu et al. (2023) Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. _ArXiv_, abs/2304.01196.
* Ye et al. (2023) Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. 2023. Dinoiser: Diffused conditional sequence learning by manipulating noises. _ArXiv_, abs/2302.10025.
* Yuan et al. (2022) Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. 2022. Seqdiffuseq: Text diffusion with encoder-decoder transformers. _ArXiv_, abs/2212.10325.
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xien Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Kotra, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models. _ArXiv_, abs/2205.01068.
* Zheng et al. (2023) Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. 2023. A reparameterized discrete diffusion model for text generation. _ArXiv_, abs/2302.05737.
* Zhou et al. (2023a) Chunting Zhou, Pengfei Liu, Pavin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xueze Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023a. Lima: Less is more for alignment. _ArXiv_, abs/2305.11206.
* Zhou et al. (2023b) Kun Zhou, Yifan Li, Wayne Xin Zhao, and Ji rong Wen. 2023b. Diffusion-nat: Self-prompting discrete diffusion for non-autoregressive text generation. _ArXiv_, abs/2305.04044.

## Appendix A Eliminating the sampling of context size \(c\) in training

In the original training algorithm of Ssd-LM Han et al. (2022), they first sample a context length \(c\sim\mathcal{U}(1,|\mathbf{w}|-B)\), and then compute the diffusion loss for reconstructing a block of length \(B\) following that context. When the sequence length \(|\mathbf{w}|\) is large, this can lead to a drastic variance in the values of \(c\). Implemented naively in a distribution training setup, this setup wastes computations, and reduces the effective batch size considerably slowing down training. We eliminate the sampling of the context length \(c\) in Ssd-2 by processing multiple \(c\)'s in parallel. To facilitate this, we encode the context \(\mathbf{w}^{<c}\) uni-directionally while preserving the locally bi-directional attention for the diffusion generation block \(\mathbf{w}^{cc+B}\).

More specifically, assume we have a prompt \(\mathbf{w}^{<c_{0}}\) and want to form the same training objective as in Figure 2 on all of the following \(n\) text blocks \(\mathbf{w}^{c_{0}:c_{0}+nB}\). We prepare a context sequence \(\mathbf{w}^{<c_{0}+(n-1)B}\) and obtain \(\mathbf{h}^{<c_{0}+(n-1)B}\) as described previously in SS3.1. We prepare a diffusion sequence \(\tilde{\mathbf{w}}^{c_{0}:c_{0}+nB}\) and obtain \(\tilde{\mathbf{h}}^{c_{0}:c_{0}+nB}\) as described previously. Then a forward pass of \(\theta\) works as below.

\[\begin{split}&\mathbf{o}^{<c_{0}+(2n-1)B}=\mathrm{Transformer}[\\ &\quad\mathrm{concat}(\mathbf{h}^{<c_{0}+(n-1)B},\tilde{\mathbf{h}}^{ \alpha_{0}:c_{0}+nB});\mathbf{\delta}(c_{0},n,B)]\\ &\mathbf{w}^{c_{0}:c_{0}+nB}_{\text{logits},t}=\mathbf{o}^{c_{0}+(n-1)B: c_{0}+(2n-1)B}\end{split}\]

where \(\mathbf{\delta}(c_{0},n,B)\) is a special attention mask for the transformer model, allowing a reuse of the encoded contexts while preserving the original training loss:

\[\delta_{i,j}=\left\{\begin{array}{ll}\mathds{1}_{j\leq i}&\textrm{ if $i<c_{0}+(n-1)B$}.\\ \mathds{1}_{j\leq c_{0}+kB\,\sigma\,c_{0}+(n-1+k)B<j<c_{0}+(n+k)B}\\ \textrm{if $c_{0}+(n-1+k)B<i<c_{0}+(n+k)B$},\\ \textrm{for $0\leq k<n$}.\end{array}\right.\]

Row \(i\) of \(\mathbf{\delta}\) indicates the attention-accessible positions for the \(i\)-th input token of the transformer. For example, assume the original context is \([a]\) and the target generation is in two blocks \([b,c]\) and \([d,e]\). The input sequence to the Ssd-2 transformer model is \([a,b,c,\tilde{b},\tilde{c},\tilde{d},\tilde{e}]\), and the attention mask is:

\[\mathbf{\delta}(1,2,2)=\begin{bmatrix}1&0&0&0&0&0&0\\ 1&1&0&0&0&0&0\\ 1&1&1&0&0&0&0\\ 1&0&0&1&1&0&0\\ 1&0&0&1&1&0&0\\ 1&1&1&0&0&1&1\\ \end{bmatrix}\]

Comparing to regular language models, Ssd-2 has a uni-directional encoder and locally bi-directional decoder. In a pilot pretraining session, we observe this change leads to a twice as fast training speed compared to the original Ssd-LM on a same amount of training tokens.

## Appendix B Pretraining hyperparameters

For the Ssd-2 model of each size (13B/2.7B/0.1B), we conduct two phases of training, a warmup phase 

without self-conditioning and a formal phase with self-conditioning using the complete algorithm shown in Figure 2. Throughout all pretraining setups, we use a max sequence length of 500, a learning rate of 1e-4, and a weight decay of 0.01. For the 13B Ssd-2, we train with a warmup batch size of 768 for 50,000 steps (19B tokens) and a formal batch size of 384 for 100,000 steps (19B tokens). For the 2.7B Ssd-2, we train with a warmup batch size of 256 for 100,000 steps (13B tokens) and a formal batch size of 1024 for 100,000 steps (51B tokens). For the 0.1B Ssd-2, we train with a warmup batch size of 2,048 for 200,000 steps (205B tokens) and a formal batch size of 2,048 for 100,000 steps (102B tokens). We use Nvidia V100 GPUs in distributed training, and the different batch size and number of warmup steps across different models are due to the models' memory footprint and the relative cluster traffic during our pilot pretraining. Future work with a dedicated group of computing resources can explore pretraining for longer to mitigate the undertraining issue mentioned in SS4.

## Appendix C Pretraining losses

Figure 3 shows the pretraining losses of Ssd-2 over time. We report the losses after the warmup stage and average them across batches with a self-conditioning \(p=0.5\) as described in Figure 2. We see a sign of undertraining from the loss curves. Due to our computing budget, we leave to future work a potential continued pretraining over current Ssd-2 models on larger and better curated data.

## Appendix D Decoding speed

Though the decoding of Ssd-2 is still significantly slower than an autoregressive language model, it achieves a great speedup compared to the original Ssd-LM. We use a same setup as the reported Ssd-LM decoding in Han et al. (2022). Conditioning on 50 prompting tokens, we record the speed of generating the next 25 tokens with timestep \(T=1000\) on a Nvidia V100 GPU.

The 0.4B Ssd-LM takes 25 seconds. By contrast, though our 2.7B Ssd-2 and 13B Ssd-2 are 7x and 33x larger than Ssd-LM, they only take 22 seconds and 48 seconds respectively, indicating a significant speedup.16

Footnote 16: It is worth noting that we add a timestep embedding when encoding the context in Ssd-2 (§3.1). During inference across timesteps, to reuse the encoded states of the context, we quantize the input timestep \(t\) to the context timestep embedding layer, \(t_{\text{output}}=t-(t\mod Q)\). We use \(Q=50\) throughout the work and do not find the quantization to negatively impact the generation quality empirically.

## Appendix E Standalone Ssd-2 finetuned with dolly

Following SS5.2, we evaluate the outputs from the finetuned models, Ssd-2-dolly and OPT-dolly, on both dolly's held-out test set and Vicuna's test set. As shown in Table 4, we find that against very strong baselines pretrained on much larger datasets, our model still wins on a moderate percentage of test examples. Compared to LLaMA (which is trained on 1T tokens for much longer but not fine-tuned for chat), Ssd-2 performs marginally better. It is overall mildly less preferred than the OPT-dolly model on both dolly's and Vicuna's test sets, and significantly less than the Alpaca model. We emphasize that compared to OPT and LLaMAN-based models, Ssd-2 is currently pretrained with a relatively small, single-corpus dataset,17 and finetuned on an open-source dataset much smaller compared to its non-open-source licensed counterparts that Alpaca relies on. We believe if trained on similar datasets, Ssd-2 can fill the current performance gap considerably.

Footnote 17: Despite that we initialize Ssd-2 with the OPT weights at the beginning of pretraining, the diffusion objective is fundamentally different from a causal language modeling objective, and the encoded knowledge of OPT may not transfer well.

## Appendix F Qualitative examples

In Table 5, we show some qualitative examples of our finetuned diffusion language model, Ssd-2-dolly's outputs. The outputs are generated by

Figure 3: Pretraining losses across training steps (with self-conditioning, after the warmup stage). We conjecture that the models can benefit from more training given more resources.

 

the 13B model to the queries from the Vicuna test set Chiang et al. (2023), on open-ended question answering, creative writing, etc. We empirically find that the generations have an overall reasonable format and content.

## Appendix G Evaluation templates

We show in Table 6 the templates we used to evaluate model responses via GPT-4 or GPT-3.5-turbo. For the inference-time collaboration results, apart from the five evaluated attributes described in SS5, in the template we additionally queried for a rating of conciseness and human preference. However, we consider conciseness a less relevant attribute compared to the other attributes and GPT-3.5-turbo less qualified to give ratings of human preference. We filtered them out in our main results but they hold a similar trend as the results of the other five attributes.

\begin{table}
\begin{tabular}{l l c c} \hline Comparing against & Win & Draw & Lose \\ _On Dolly’s test set_ & & & \\ OPT-Dolly-13B & 41.4\% & 0\% & 58.6\% \\ _On Vicuna’s test set_ & & & \\ LLaMa-13B & 51.4\% & 0\% & 48.6\% \\ OPT-Dolly-13B & 37.1\% & 0\% & 62.9\% \\ Alpaca-13B & 20.0\% & 1.4\% & 78.6\% \\ \hline \end{tabular}
\end{table}
Table 4: Quantitative evaluation of the finetuned Ssd-2’s outputs against those of autogressive baselines based on ratings of GPT-4. We report the percentage our Ssd-2 wins / draws / loses to the baseline models.

 

Prompt: How can I improve my time management skills?

Time management is in short, an important skill to develop. A person needs to learn how to manage their time more effectively. Here are a few techniques that can help you do that: 1. Prioritizing: One of the first things that you should try in order to improve your time management skills is to identify which tasks have high priority and need to be given immediate attention, and those that have low priority but can be postponed until a later date. 2. Multi-tasking: Although it might feel productive to handle more than one task at the same time , it's actually less efficient at managing our time. It's better to focus on one thing at a time, and fully dedicate 100% of our efforts to it. 3. Delegating: It's perfectly fine to say no to some tasks or even ask for help when needed. It's sometimes considered a sign of weakness, but delegation will allow us to free up our times so that we can work on the things that really matter. 4. Being organized: Making a list will not only help us keep track of tasks that need to get done each day, but it will also help us avoid feeling overwhelmed by all the ideas and thoughts running through our heads at any given moment. By following these tips, you'll be able to significantly improve your time management skills! Prompt: Write a symphony concert review, discussing the orchestra's performance and overall audience experience. It was a magical night at Symphony Hall on Saturday, as the Boston Symphony Orchestra, under the baton of Andin Mehta, performed Tchaikovsky's Symphonies No. 5 and No. 6. Tchaikovsky has always been one of my favorite composers, and these two symphonies are among his most beautiful. They are full of sweeping melodies, lyrical string writing, and lush orchestral textures that transport you to a different world each time you listen. And Mehta did an incredible job in leading the BS0 through these masterpieces. He brought out every nuance in the music - from the quiet tenderness of the first movement to the pulsing energy of the Allegroagio to the breathtaking crescendo of the Finale. The entire experience was absolutely magical: I couldn't have asked for a better way to spend a Saturday night! And I would highly recommend this performance for anyone who is a fan of Tchaikovsky or classical music in general.

\begin{table} \end{table}
Table 5: Qualitative examples of Ssd-2-dolly’s responses to instructions. We show the first and last prompt from the Vicuna test set. The outputs of our diffusion chat model have an overall reasonable format and content, though being inaccurate in details like the conductor’s name and the tempo terminology.

 

Evaluation template used to compare Ssd-2's responses with baseline models' responses (SS5.2), following Chiang et al. (2023).

[Question] {test instruction}

[The Start of Assistant 1's Answer] {baseline model's response}

[The End of Assistant 1's Answer]

[The Start of Assistant 2's Answer] {SSD-2's response}

[The End of Assistant 2's Answer]

[System] We would like to request your feedback on the performance of two AI assistants in response to the user question displayed above. Please rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance. Please first output a single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores are separated by a space. In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.

Evaluation template used to rate responses from either the baseline models or Ssd-2 w.r.t. different attributes (SS5).

Rate the response below to an instruction, from the aspects of relevance, factuality, informativeness, conciseness, coherence, understandability, and overall human preference, each on a scale of 10 (format: x/10).

=======

 Instruction: (test instruction) Response: (model response) ===========

Please give the ratings now.

\begin{table} \end{table}
Table 6: Evaluation templates used in §5.2 and §5. The first template was used with GPT-4 (temperature=0.2), whereas the second was used with GPT-3.5-turbo (greedy) since we need significantly more queries across different \(\lambda_{\text{user}}\)’s. In the comparative evaluation using the first template, flipping the order of the baseline model’s response and Ssd-2’s response leads to a similar result.