Sen2Pro: A Probabilistic Perspective to Sentence Embedding from Pre-trained Language Model
Year: 2023

Abstract:
Sentence embedding is one of the most fundamental tasks in Natural Language Processing and plays an important role in various tasks. The recent breakthrough in sentence embedding is achieved by pre-trained language models (PLMs). Despite its success, an embedded vector (Sen2Vec) representing a point estimate does not naturally express uncertainty in a taskagnostic way. This paper thereby proposes an efficient framework on probabilistic sentence embedding (Sen2Pro) from PLMs, and it represents a sentence as a probability density distribution in an embedding space to reflect both model uncertainty and data uncertainty (i.e., many-to-one nature) in the sentence representation. The proposed framework performs in a plug-and-play way without retraining PLMs anymore, and it is easy to implement and generally applied on top of any PLM. The superiority of Sen2Pro over Sen2Vec has been theoretically verified and practically illustrated on different NLP tasks.

heading: 1 Introduction
text: Sentence embedding, which maps an input sentence to a point (i.e., a vector) in an embedded space, is one of the most fundamental tasks in Natural Language Processing (NLP), and it plays an important role in various downstream tasks such as sentiment analysis, text classification, and natural language inference (Howard and Ruder, 2018; Reimers and Gurevych, 2019; Gao et al., 2021). There is a surge of interest in learning sentence embedding. The early work resorts to word embedding (Bengio et al., 2003; Mikolov et al., 2013; Pennington et al., 2014) and represents an input sentence by a pooling vector (mean or weighted mean) over all embeddings of its words (Kiros et al., 2015; Wieting et al., 2015a). More recently, sentence embedding obtained from pre-trained language models (PLMs) made a breakthrough thanks to PLM’s powerful ability in modeling global context (Peters
et al., 2018; Devlin et al., 2019; Liu et al., 2019), and it quickly became the standard practice for sentence embedding.
Despite the success of sentence embedding from PLMs, an embedded vector (Sen2Vec) representing a point estimate does not naturally express uncertainty about the target concepts associated with the input sentence (Vilnis and McCallum, 2015). In essence, this uncertainty originates from many-toone nature in language representation: (1) model uncertainty: model uncertainty refers to the notion of randomness caused by inherently random effects within the model. (i.e. one sentence may have many representations according to different representations within the same model (e.g., dropout)) Considering that these representation are from the same sentence, they should remain close to each other; (2) data uncertainty: many sentences with different linguistic structure (e.g., paraphrase) may have the same meaning in semantics. Considering the identical semantics, their corresponding representation should get close with each other.
When quantifying uncertainty, we assume that close-semantic sentences’ representation follows the same probability distribution. Given a sentence, since the model only observes one sample, it is natural to ask how much a language model can capture such a rich distribution.
A natural solution to this issue is to merge such a probabilistic perspective into sentence embedding, which represents a sentence as a distribution P (µ,Σ), where µ is the mean and covariance Σ intuitively portrays the uncertainty of the distribution P . Unfortunately, there is a critical challenge to putting this idea into practice: previous works (Bamler and Mandt, 2017; Camacho-Collados and Pilehvar, 2018; Zhou et al., 2019) are only plausible on word embedding and require retraining word embedding with probabilistic embedding on large-scale data to advance SOTA. It is costly even for training a PLM without probabilistic embed-
315
ding (Devlin et al., 2019; Radford et al., 2019; He et al., 2020; Clark et al., 2019; Raffel et al., 2020), which typically consumes considerable GPU computations for weeks.
In this paper, we propose an efficient framework for probabilistic sentence embedding (Sen2Pro) from PLMs that represents a sentence as a probability density in an embedding space to reflect the uncertainty in the sentence representation.
Concerning model uncertainty and data uncertainty, we propose two simple methods to quantify both on top of a PLM. Specifically, to measure model uncertainty, we assume a sentence vector is drawn from a distribution P (µm,Σm), which can be estimated by many representations of the targeted sentence using a set of stochastic PLMs obtained from Monte Carlo dropout (Gal and Ghahramani, 2016).
Similarly, we apply the data augmentation technique to generate many semantically equivalent sentences to the targeted sentence for measuring data uncertainty. Then we assume a sentence vector is drawn from a distribution P (µd,Σd), which the representations of the augmented sentences from the PLM can estimate. In addition, we also introduce some ways to utilize both µm (µd) and Σm (Σd) as the final sentence representation for different downstream tasks.
Moreover, drawing from previous works (Chen et al., 2016; Li and Chen, 2019; Gao et al., 2019; Tschannen et al., 2019; Grohs et al., 2021) that explored the relationships between deep learning representation and relative entropy, we present theoretical explanations of why our probabilistic sentence embedding (Sen2Pro) is superior to point vectorbased sentence embedding (Sen2Vec). Meanwhile, extensive experiments demonstrate the practical effectiveness of Sen2Pro on text classification, semantic similarity match, dialogue generation evaluation, and machine translation evaluation. Besides, Sen2Pro demonstrates its superiority in capturing sentence-level linguistic analogy over Sen2Vec.

heading: 2 Related Work

heading: 2.1 Sentence Embedding
text: Methods for sentence embedding learning have been extensively explored, and all these methods represent a sentence as a point embedding. Early works use the weighted sum of word embedding to represent a sentence. Then some methods based on the distributional hypothesis have been done.
Hill (Hill et al., 2016) learned sentence representations with the internal structure of each sentence, and Kiros (Kiros et al., 2015) followed the idea of Word2Vec (Mikolov et al., 2013) to represent a sentence by predicting its surrounding sentences. In recent years, the pre-trained language model (Devlin et al., 2019) has become the standard sentence paradigm because of its strong ability to capture syntactic and semantic features of sentences by learning from the large-scale corpus. Furthermore, several researchers used contrastive learning to augment sentence representation (Zhang et al., 2020; Yan et al., 2021; Meng et al., 2021; Gao et al., 2021; Wang et al., 2021), based on the assumption that a high-quality representation method should bring similar sentences closer while pushing away dissimilar ones. They belong to Sen2Vec and thus fail to model uncertainty. This paper goes beyond point sentence embedding and explores probabilistic sentence embedding.

heading: 2.2 Probabilistic Word Embedding
text: In NLP, probabilistic embedding originated from word embedding (Bengio et al., 2003; Mikolov et al., 2013), and existing probabilistic embedding methods work only on words, where a word from the vocabulary is represented as a density distribution in an embedding space. Although variants of probabilistic word embedding have been developed (Vilnis and McCallum, 2015; Bamler and Mandt, 2017; Camacho-Collados and Pilehvar, 2018; Zhou et al., 2019), they used a similar paradigm, which adapts the Skip-gram model (Mikolov et al., 2013) with a non-parametric approach. Specifically, a Skip-gram model is retrained as the density distribution with a specific word sampling (e.g., synonym) method and a specific loss function (e.g., a margin loss). Therefore, existing probabilistic embedding needs an extremely time-consuming retraining stage and can not be applied to PLMs (e.g., BERT). Different from them, this paper contributes to the literature by developing probabilistic embedding for sentences that serves as a plug-and-play method on pre-trained language models (Devlin et al., 2019) without any time-consuming retraining stage.

heading: 3 Methodology: Sen2Pro
text: Because of the many-to-one nature of sentence embedding, we model the uncertainty from two perspectives, i.e., model uncertainty and data uncer-
tainty. Accordingly, we assume that the representation of one sentence follows a distribution P (µ,Σ), which measures either model uncertainty or data uncertainty. The goal of our Sen2Pro framework is to estimate the two distributions P (µ,Σ) for a sentence embedding based on a pre-trained language model fθ (e.g., BERT), where θ denotes its parameters. In general, there are two steps in Sen2Pro: the sampling stage and the estimation stage. The sampling stage generates embedding instances to capture two kinds of uncertainties: model uncertainty and data uncertainty (§3.1). The estimation stage aims to estimate the parameters in the density distributions (i.e., mean vector and covariance matrix) based on the embedding instances (§3.2). After both distributions are estimated, the general idea to apply them to specific tasks is presented (§3.3).

heading: 3.1 Sampling Stage
text: Model Uncertainty Model uncertainty originates from the fact that one sentence may have different representations due to inherent randomness within models. In this paper, we use a pre-trained language model fθ and try to quantify model uncertainty. Considering that the key ingredient of model uncertainty is to vary the model while keeping the sentence s unchanged, we utilize MC dropout to create embedding instances for quantifying model uncertainty. Specifically, for each sentence s, we utilize MC Dropout (Gal and Ghahramani, 2016; Lakshminarayanan et al., 2017) over the parameters θ as sampling, and repeat sampling N times to obtain different subsets of the parameters θ: {θ̂i | i = 1, . . . , N}. In this way, we generate a set of embeddings as follows:
Sm = { xi = fθ̂i (s) | i = 1, . . . , N } (1)
As shown in Eq. 1, each subset of model’s parameters represent a sub-structure of the model, which naturally matches with the definition of model uncertainty.
Data Uncertainty Data uncertainty corresponds to the many-to-one nature of sentence embedding. In other words, sentences that are semantically similar but slightly different should have close representations. Data uncertainty exists in universal real-world scenarios: since the model requires lots of training data to perform well, it is common to augment high-quality labeled sentences with lowerquality web-crawled data to save time and effort.
To naturally imitate such uncertainty, in this paper, a simple data augmentation method, word-level operation, is applied to the input sentence, which adds proper noise to an input sentence. After repeating data augmentation N times (i.e., randomly dropping a word in s, swapping two words, replacing or inserting a word in s with any word from vocabulary) for the input sentence s, a set of new sentences, s1, s2, . . . , sN are obtained. Then, the sentences are fed to the pre-trained model fθ to get N embeddings. In this way, we can obtain a set of embeddings as follows:
Sd = { xi = fθ (si) | i = 1, . . . , N } (2)

heading: 3.2 Estimation Stage
text: After obtaining the required embedding instances for model and data uncertainty, we can estimate the probability distributions on them, respectively. Similar to cases of (Kendall and Gal, 2017; Maddox et al., 2019; Ovadia et al., 2019; Abdar et al., 2021; Hüllermeier and Waegeman, 2021), we do estimation towards two uncertainty individually rather than unifying, which is also empirically verified in Appendix D. Suppose S denotes either Sm or Sd, it is natural to estimate its mean and covariance as follows:
µ = 1 |S| ∑
x∈S x (3)
Σ = 1 |S| ∑
x∈S (x− µ)(x− µ)⊤ (4)
where |S| means the size of the set S and (.)⊤ is an transpose operation. We use µm and Σm respectively to denote the statistics estimated from model uncertainty (i.e., S = Sm), and µd and Σd denote those estimated through data uncertainty (i.e., S = Sd).
However, such a simple covariance matrix estimator (SCE) owns severe problems on both theoretical (Xiao and Wu, 2012) and practical sides: it is known to degrade rapidly as an estimator as the number of variables increases, thus performing badly in a high-dimensional case (e.g., 768 dimensions in BERT). To address this issue, inspired by Bien et al. (2016), we instead employ the banding estimator, which uses an off-diagonal entry removal operation on the covariance matrix.
Specifically, for a covariance matrix Σ = (Σij)k×k where k is the dimension of Σ, we use
B(Σ) as the estimation of Σ as follows:
Σ̂ = B(Σ) = Diag(Σ) (5)
Besides, Theorem 1 provides an estimation error bound for our banding estimator, whose proof is presented in Appendix A.
Theorem 1. Suppose Σ is the covariance matrix of the ground truth distribution P (µ,Σ), and Σ̂ denote B(Σ), then we have
∥∥∥Σ̂− Σ ∥∥∥ 2 = Op
(( log k
n
)M) (6)
where k is the dimension of Σ, and M is a positive constant satisfying M < 12 . Op and n means log k n )
M is stochastically bounded as n → ∞. Besides, SCE also in practicability problems compared to the banding estimator. SCE owns a significantly worse performance-efficiency tradeoff than our banding estimator, which will be empirically verified in Sec 5. Moreover, theoretical analyses for comparison between Sen2Vec and Sen2Pro are deferred to Appendix B.

heading: 3.3 Usage of Sen2Pro
text: Unlike previous works on probabilistic embedding that drop Σ̂ in tasks, in Sen2Pro, both mean vector µ (i.e., µm and µd) and covariance vector Σ̂ (i.e., Σ̂m and Σ̂d) are used for sentence embedding. In the next section, we illustrate our strategies to use µ and Σ̂, more details are presented in Sec 4.1.

heading: 4 Experiment
text: This section comprehensively evaluates the effectiveness of our Sen2Pro framework on the following various tasks. The results consistently show that Sen2Pro outperforms Sen2Vec. Specifically, we choose two sets of tasks for evaluation: Finetune-need Task and Fine-tune-free Task.

heading: 4.1 Basic Setting
text: Recall that µm and Σ̂m (µd and Σ̂d) denote the estimated mean and covariance from model (data) uncertainty. Then we present how to use them for different tasks.
Fine-tune-need Task: Text Classification After estimation stage in data and model uncertainty, each sentence is represented as the concatenation of µ m+µd
2 and (diagonal entries of) Σ̂m+Σ̂d
2 . Specifically, we use reparameterization to handle the nondifferentiable issue of the sampling process.
Fine-tune-free Task: Sentence Similarity, Dialogue Evaluation, Neural Machine Translation Evaluation For such NLP tasks, the distance between two representations is needed. Although KL divergence is natural to measure the distance between two distributions, it has practical drawbacks. For example, in BERT-base case where µ ∈ R768×1, Σ̂ ∈ R768×768, KL divergence not only is time-consuming but also results in numerical errors, considering that KL divergence includes operations det(Σ̂) and Σ̂−1. In practice, most entries of Σ̂ are between 0 and 0.5, so the determinant becomes extremely small, which leads to numerical errors. Moreover, the computation of Σ̂−1 will be unstable when the dimension is high.
Therefore, a simple function is taken to measure the distance between two probabilistic sentence embeddings (µa,Σa) and (µb,Σb) as follows:
d(N (µa,Σa),N (µb,Σb)) (7) = (1− α)l1(µa − µb) + αl1(Σa − Σb)
where l1 represents the l1-norm and α is to balance the two terms. For α, we consider it as a balance factor for different magnitudes of µ and Σ, defined as follows:
α = l1(µa − µb) l1(Σa − Σb)
(8)
In most cases, α ranges from 0.01 to 0.05. Besides, we will try to apply Sen2Pro on other evaluation tasks (Shen et al., 2022a,b) like paraphrase, datato-text, and summarization.

heading: 4.2 Text Classification
text: Benchmarks and Setting We choose four widely-used text classification benchmarks: AG News (Zhang et al., 2015), DBpedia (Maas et al., 2011), Yahoo! Answers (Chang et al., 2008) and IMDB (Maas et al., 2011). The evaluation metric is the test accuracy, and the best performance is selected based on validation set accuracy. The baseline is Sen2Vec, with 15 data augmentation samples per sentence. In Sen2Pro, we choose the BERTbase model as the PLM and use the ‘first-last-avg’(a pooling way that average first and last layer representation). The sampling number is 15 for the model uncertainty and data uncertainty. Moreover, we use two settings for model evaluation: few-shot and full-dataset. In the few-shot setting, models are trained with randomly selected 10 and 200 labeled sentences per class. In the full-dataset setting, models are trained with the whole training set.
Performances The results are listed in Table 1. Our Sen2Pro consistently performs better than Sen2Vec under few-shot settings because Sen2Pro contains more semantic information about a sentence due to its probabilistic characteristic. Moreover, Sen2Pro achieves comparable or better performances in the full training setting than Sen2Vec.

heading: 4.3 Sentence Similarity
text: Benchmarks and Setting We use seven commonly-used STS datasets (Agirre et al., 2012, 2013, 2014, 2015, 2016) for evaluation. Besides, considering the limitation of traditional intrinsic evaluation (Wang et al., 2022), we choose EvalRank (Wang et al., 2022) for linguistic analogy evaluation, which overcomes the previous limita-
tion. In Sen2Pro, we use several state-of-the-art pre-trained language models, including BERT-base (Devlin et al., 2019), BERT-whitening (Su et al., 2021; Huang et al., 2021), Sentence-BERT (Reimers and Gurevych, 2019), and SimCSE (Gao et al., 2021). Specifically, EvalRank uses the mean reciprocal rank (MRR) and Hits@k scores for evaluation, and a higher score indicates a better embedding model. The sampling number for each uncertainty is set as 15.
Performances The results of Sen2Pro are reported in Table 2 and 3, which illustrate that Sen2Pro outperforms Sen2Vec with a substantial gap under all settings. Moreover, Sen2Pro using the‘base’ PLMs can achieve better or comparable performance than Sen2Vec using ‘large’ PLMs.

heading: 4.4 Dialogue Evaluation
text: Benchmark and Setting We choose three widely-used dialogue benchmarks: Daily(H) (Lowe et al., 2017), Convai2 (Dinan et al., 2020), and Empathetic (Rashkin et al., 2019). Each benchmark consists of dialogue queries, the corresponding responses, and human-annotated responses. For baseline metrics, we choose BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Denkowski and Lavie, 2014), Greedy Matching (Rus and Lintean, 2012), Embedding Average (Wieting et al., 2015b), Vector Extrema (Forgues et al., 2014) and BERTScore (Zhang et al., 2019). For
Sen2Pro, we use the BERT-base model and the ‘first-last-avg’ representation and set the sampling number of uncertainty to 15.
Performances The performances of various evaluation metrics are reported in Table 4. Sen2Pro performs best in most cases, demonstrating its good generalization ability and robustness for the dialogue evaluation task.

heading: 4.5 NMT Evaluation
text: Benchmark and Setting We work on the WMT17 machine translation benchmark (Bojar et al., 2017). Moreover, BLEU (Papineni et al., 2002), CDER (Leusch et al., 2006), BLEND (Ma et al., 2017), Sen2Vec, and BERTScore (Zhang et al., 2019) are chosen as baseline metrics. The setting of Sen2Pro is the same as the one of dialogue evaluation.
Performances The results are shown in Table 5. Our Sen2Pro achieves comparable performance towards a top metric (i.e., BERTScore) and significantly outperforms Sen2Vec, which demonstrates the effectiveness of Sen2Pro in the NMT evaluation task. Segment-level results are shown in Appendix H. The results indicate that our Sen2Pro can yield competitive performance to SOTA as an automatic metric to evaluate dialogue generation.

heading: 5 Analysis and Discussion
text: This section presents analyses of Sen2Pro, and the BERT model is used as the PLM in the analyses. Specifically, the representation uncertainty analysis is made on the STS task, and detailed results
are deferred to Appendix E and F. Moreover, we choose the BERT model as our pre-trained sentence encoder in the analysis.
Feature with higher model uncertainty is more important We investigate the relation between the model uncertainty and the feature importance on the STS task. In Sen2Pro, a sentence is represented as µm ∈ R768×1 and diagonal entries of Σ̂m ∈ R768×768. From another perspective, a sentence is represented by 768 features in µm, and Σ̂m reflects the corresponding model uncertainties of such features. Let T be the feature set, and t ∈ T is a feature subset. We define the feature importance as follows:
score(t) = |ρ(T )− ρ(T/t)| (9)
where ρ represents Spearman’s correlation in STS evaluation, and score(t) describes the performance change after removing the feature subset t from T . Then we separate the 768 features into five groups according to their uncertainty Σ̂m, and name them as ‘I’ to ‘V’, as shown in Figure 1. When a group of features is removed from T , these features are set to 0 in µm and Σ̂m, respectively. Figure 1 lists the results on STS-12. In Figure 1, as the feature’s uncertainty decreases, the importance score drops, indicating that features with higher uncertainty are more important in the STS task.
Sen2Pro brings more benefits when model uncertainty are higher We also investigate the relation between the model uncertainty and the performance improvement of Sen2Pro over Sen2Vec on the STS task. Firstly, we define a metric called
Fluctuation Rate Q as follows:
Q(f,D) = Σ |D| i=1(Σ k j=1σij)
k × |D| (10)
where1 f and D represent the PLM and benchmark, and σij is j-th element of the covariance matrix for i-th sentence in D, and k denotes the dimension of model f . Such a metric can generally reflect the uncertainty of a specific model towards a specific benchmark. Based on Q(f,D), we define the improvement score I reflecting the improvement of Sen2Pro over Sen2Vec as follows:
I = PSen2Pro(f,D)− PSen2V ec(f,D) (11)
where PSen2Pro(f,D) and PSen2V ec(f,D) represent the performance of model f on D under Sen2Pro and Sen2Vec, respectively. Besides, we add the result of [CLS] representation into experiments since the [CLS] representation owns a significantly higher fluctuation rate than the ‘first-lastavg’ representation. The results on STS-12 are
1Note that σ corresponds to Σ̂m, we change the notation here to avoid repeating with the sum operation Σ.
illustrated in Figure 2. As the fluctuation rate increases, the improvement score becomes more significant. Such empirical results demonstrate that the effectiveness of Sen2Pro is highly correlated to the model uncertainty and show Sen2Pro’s superiority over Sen2Vec owns a positive correlation to model uncertainty.
Effectiveness of Two Uncertainties We verify the effect of model and data uncertainty on
Sen2Pro’s performances. Specifically, Sen2Pro is evaluated on one intrinsic evaluation (STS) and one downstream task (text classification), and the results are demonstrated in Table 6 and Table 7, respectively. As shown in Table 6, the performance decreases when the data uncertainty is applied alone for the STS task since sentences’ semantics may be changed due to the data augmentation. In contrast, the model uncertainty consistently brings benefits to the sentence representation. For text classification, both uncertainties improve the representation’s generalization. Specifically, the contribution of the data uncertainty is higher than the model uncertainty. These empirical results also illustrate the usage of Sen2Pro (§3.3).
Effect of the Banding Estimator As mentioned in Sec 3.2, we use the banding estimator for covariance estimation. This part compares SCE (the usual estimator) and the banding estimator in two aspects: performance and efficiency. We choose BERT-base as the PLM, and the results are presented in Table 8, and the efficiency is shown in Figure 3. As shown in Figure 3, our estimator achieves a significantly better performance-efficiency trade-off than SCE, demonstrating the banding estimator’s effectiveness.

heading: 6 Linguistic Case Study
text: We conduct case studies following the famous analogy from Word2Vec (Mikolov et al., 2013). The widely used analogy takes the following form: Knowing that A is to B as C is to D, given l2 normalized embeddings v⃗A, v⃗B, v⃗C , v⃗D for sentences A,B,C,D for an analogy of the above form, the task compare the embedding distance dis(A,B) and dis(C,D), which is defined as follows:
x = |v⃗A − v⃗B|2 − |v⃗C − v⃗D|2 (12)
Then we use the sentence analogy set created from (Zhu and de Melo, 2020), which is specifically for this test. Here is a quadruple example:
A: A man is not singing. B: A man is singing. C: A girl is not playing the piano. D: A girl is playing the piano. We can see that the relation between A(C) and B(D) is negation; ideally, x in this quadruple should be small. We list the performance of the sentence embedding method w/ and w/o Sen2Pro in Table 9.

heading: 7 Conclusion and Future Work
text: This paper investigates the probabilistic representation for sentences and proposes Sen2Pro, which portrays the representation uncertainty from model and data uncertainty. The effectiveness of Sen2Pro is theoretically explained and empirically verified through extensive experiments, which show the great potential of probabilistic sentence embedding. In the future, we will investigate several aspects of Sen2Pro, like how to pre-train language models from an uncertainty perspective since existing pre-training models are based on Sen2Vec. Also, we expect to design more natural schemes that utilize Sen2Pro instead of concatenating the mean and variance vectors. Such directions can further enhance Sen2Pro’s performance and efficiency.
Limitation
This major limitation of Sen2Pro lies in the computational cost due to the generation of several samples. Thus, improving the efficiency of Sen2Pro is a future direction. Besides, since we choose to concat the representation of different samples, there may be more natural ways to merge information from samples.

heading: A Proof of Theorem 1
text: To accomplish further derivation on theorem 1. we first present a lemma from (Bickel and Levina, 2008; Bien et al., 2016).
Lemma 1. Let Xi be i.i.d. N (0,Σ) and λmax (Σ) ≤ ϵ, where ϵ is a positive constant. Also, let σ denote Σ, then we have
P (∣∣∣∣∣ n∑
i=1
(XijXik − σjk) ∣∣∣∣∣ ≥ nh ) ≤ C1e−C2nv 2
for |h| ≤ δ, where C1, C2 and threshold δ depend on ϵ only, and n is the sample number for Xi. Proof 1. We firstly denote the maximum element of a matrix A as |A|max, we have
∥∥∥B ( Σ̂ ) −B (Σ) ∥∥∥ 2
= Op (∥∥∥B ( Σ̂ ) −B (Σ) ∥∥∥ ∞ )
= Op (∣∣∣B ( Σ̂ ) −B (Σ) ∣∣∣ max )
Naturally, based on Lemma 2, we can know that
P (∣∣∣B ( Σ̂ ) −B (Σ) ∣∣∣ max ≥ t ) ≤ 3ke−nt2γ(ϵ,λ)
for |t| ≤ λ(ϵ). By choosing t = M(log(k)/n)1/2, we conclude that ∣∣∣B ( Σ̂ ) − Σ
∣∣∣ max = Op (( n−1 log k )1/2)
Putting them together, we have the result that completes the proof for Theorem 1.

heading: B Theoretical Analysis
text: This section presents theoretical explanations why Sen2Pro is better than Sen2Vec. Our framework is built on comparisons through distribution distances, which is inspired from representation learning based on KL divergence (relative entropy) (Chen et al., 2016; Li and Chen, 2019; Gao et al., 2019; Tschannen et al., 2019).
B.1 Problem Formulation
We present the formulation of sentence representation: for a representation X (with dimension k) of a sentence s, we assume that X ∼ P (µ,Σ), where P (µ,Σ) is the ground-truth but unknown probability distribution.
Recall that both Sen2Pro and Sen2Vec methods adopt the following representations:
• Sen2Pro: sentence s is represented as (µ̂, Σ̂), where µ̂ ∈ Rk×1 and Σ̂ ∈ Rk×1 are estimated through i.i.d. samples X1, X2, · · · , XN . Therefore, Sen2Pro corresponds to a random variable X̂ ∼ P (µ̂, Σ̂).
• Sen2Vec: sentence s is represented as v ∈ Rk×1, where v may be a sample Xi from Sen2Pro.
Probabilistic Viewpoint Accordingly, from the viewpoint of probability, both Sen2Pro and Sen2Vec can be considered as the following random variables whose density functions are unknown but mean and covariance are known:
• Sen2Pro: XP ∼ P (µ̂, Σ̂).
• Sen2Vec: XV ∼ P (v, ϵI).
where ϵ is a constant as a smoothing operator which makes a deterministic quantity v like a random variable Xv. Note that since ϵI is constant for any sentence s, (v, ϵI) as sentence embedding is equivalent to v as sentence embedding for downstream tasks.
Then our goal is to show whether the P (µ̂, Σ̂) is closer to P (µ,Σ) than P (v, ϵI), which is defined as follows:
D(XP, X) < D(XV, X) (13)
Specifically, we use the KL divergence D as the distance measurement.
B.2 Superiority of Sen2Pro
In this part, we conduct derivations based on distribution distance to answer the above question. Unfortunately, since the density distributions of all three random variables X , XP, XV are unknown, it is intractable to derive the equations on KL divergence. To this end, we make the following assumption about their density functions.
Gaussian Distribution Assumption Intuitively, an exponential family is suitable for the density function of Sen2Pro, such as Gaussian, Poisson, and Bernoulli. In this paper, we assume all three random variables X , XP, XV are from Gaussian distribution, because non-Gaussian random variables possess smaller entropy (i.e., less uncertainty) than Gaussian random variables (Cover, 1999). In this way, the ground-truth random variable X contains the maximal uncertainty.
Based on the above assumption, we have: X ∼ N (µ,Σ), XP ∼ N (µ̂, Σ̂), and XV ∼ N (v̂, ϵI). Then, we provide the following theoretical guarantee to show the superiority of Sen2Pro, which indicates that N (µ̂, Σ̂) is closer to N (µ,Σ) than N (v, ϵI): Theorem 2. If ϵ < k √
|Σ̂| e with e as the natural number, then the following inequality holds:
D(XP, X) < D(XV, X) (14)

heading: C Proof of Theorem 2
text: Before we start to prove the theorem, we provide some definitions that will be useful for further derivation:
Definition 1. For a random variable X with mean µ, the moment generating function is
MX−µ(λ) = E exp(λ(X − EX)) (15)
the cumulant-generating function is
ΓX−µ(λ) = logMX−µ(λ) (16)
We denote D(XP, X) and D(XV, X) as Da and Db, and rewrite D(XP, X) and D(XV, X) as follows:
D(N (µ,Σ)∥N (µ̂, Σ̂)) = 1 2 [log( |Σ| |ϵI|) + tr ( ϵIΣ−1 ) − k] (17)
D(N (µ,Σ)∥N (v̂, ϵI)) = 1 2 [log( |Σ| |Σ̂|
) + tr ( ΣΣ̂−1 ) − k + (µ− µ̂)⊤Σ̂−1(µ− µ̂)] (18)
Proof 2. As defined in the main paper, we have Da = D(N (µ,Σ)∥N (µ̂, Σ̂)) = 12 [log( |Σ| |ϵI|) + tr ( ϵIΣ−1 ) −p] and Db = D(N (µ,Σ)∥N (v̂, ϵI)) = 12 [log( |Σ| |Σ̂|)+tr ( ΣΣ̂−1 ) −p+(µ−µ̂)⊤Σ̂−1(µ−µ̂)]. As illustrated in the methodology of Sen2Pro, we have the following derivation:
|Σ−1| = n∏
i=1
1
Σii (19)
|Σ| = n∏
i=1
Σii (20)
Then we focus on the term:
Da = D(N (µ,Σ)∥N (µ̂, Σ̂))
= 1
2 [log( |Σ| |ϵI|) + tr ( ϵIΣ−1 ) − p]
≈ 1 2 [log( |Σ| |ϵI|)− p]
(21)
since ϵ is an extremely small number. Besides, we also make derivation on Db:
Da = D(N (µ,Σ)∥N (µ̂, Σ̂))
= 1
2 [log( |Σ| |Σ̂|
) + tr ( ΣΣ̂−1 ) − p
+ (µ− µ̂)⊤Σ̂−1(µ− µ̂)]
= 1
2 [log( |Σ| |Σ̂| ) + (µ− µ̂)⊤Σ̂−1(µ− µ̂)]
(22)
To better analyze the (µ− µ̂)⊤Σ̂−1(µ− µ̂) term of Da, we present the following lemma: Lemma 2. For X ∼ N ( µ, σ2 ) , where X is a single Gaussian variable,
ΓX−µ(λ) = λ2σ2
2 , Γ∗X−µ(ϵ) =
ϵ2
2σ2 .
For X1, . . . , Xn ∼ N ( µ, σ2 ) , it’s easy to check that the bound is tight:
lim n→∞
1 n lnP
( X̄n − µ ≥ ϵ ) = − ϵ 2
2σ2 .
Therefore, we can represent |µ− µ̂|2 as O(ϵ) and we have the following:
(µ− µ̂)⊤Σ̂−1(µ− µ̂) = D(N (µ,Σ)∥N (µ̂, Σ̂)) = O(ϵ2)
(23)
Naturally, we drop all O(ϵ2) term and formulate Da as follows:
Da = 1
2 [log( |Σ| |Σ̂|
) + tr ( ΣΣ̂−1 ) − p] (24)
= 1
2 [log( |Σ| |Σ̂| )] (25)
Then we insert the condition to the equality Da < Db, and we can complete the proof.

heading: D Comparisons between individual estimation and unified estimation
text: This section conducts comparisons between individual estimation and unified estimation.
• Concretely, individual estimation refers to estimate µm, Σ̂m and µd, Σ̂d based on Sm and Sd, respectively. Then we use µ m+µd
2 and Σ̂m+Σ̂d 2 for sentence representation.
• Correspondingly, unified estimation refers to estimate µu, Σ̂u based on Sm ∩ Sd. Then use µu, Σ̂u for sentence representation.
The experimental settings follow the Sec 4, and the results are shown in Table 10. We can observe that unified estimation under-performs individual estimation. Thus, we choose to estimate uncertainty in an individual way.

heading: E Relation between model uncertainty and feature importance
text: This section serves as a complementary material for demonstrating the relation between the model uncertainty and feature importance, and the results on the STS task are illustrated in the Figure ?? ?? 4.

heading: F Relation between model uncertainty and performance improvements
text: This section serves as a complementary material for demonstrating the relation between the model uncertainty and performance improvements, and the results on the STS task are illustrated in the Figures 5.
G Influence of the Sampling Number
This section examines the effect of the sampling number n1 in the model uncertainty and the sampling number n2. To examine the effect of n1, an experiment is run, where n1 varies and n2 is set 30. The results of the STS and text classification task are illustrated in Figure 6. From Figure 6, we select n1 = 50 and n1 = 15 for the STS and text classification task, respectively, since smaller n1 result in imprecise sentence representation and larger n1 brings significant computational burden with little improvements. To examine the effect of the sampling number n2, we change n2 and keep the optimal n1, and show the results in Figure 7. Overall, when n2 is greater than 30, Sen2Pro achieves a stable performance.

heading: H Results of Segment-level Evaluation of Neural Machine Translation
text: In the main paper, we make the evaluations on the system-level NMT. Here, we present the results of segment-level Evaluation of Neural Machine Translation. The experimental settings are kept the same as the ones in our main paper. The results are listed in Table 11. As the results demonstrate, Sen2Pro outperforms Sen2Vec and achieves comparable performances to BERTScore.

heading: Future Work
text: In the future, we will investigate several aspects of Sen2Pro, like how to pre-train language models from an uncertainty perspective since existing pre-training models are based on Sen2Vec. Also, we expect to design more natural schemes that utilize Sen2Pro instead of concatenating the mean and variance vectors. Such directions can further enhance Sen2Pro’s performance and efficiency. Limitation
This major limitation of Sen2Pro lies in the computational cost due to the generation of several samples. Thus, improving the efficiency of Sen2Pro is a future direction. Besides, since we choose to concat the representation of different samples, there may be more natural ways to merge information from samples