PAC-Bayes Analysis of Transferred Sentence Vectors

Learning sentence vectors from an unlabeled corpus have attracted attention because such vectors are able to represent sentences flexibility. Simple heuristic methods using pre-trained word vectors are a strong baseline for machine learning tasks. However, they are not well understood from a theoretical perspective. We analyze sentence vector algorithms from a transfer learning perspective by using a PAC-Bayes bound, which enables us to understand existing heuristic methods and provides novel sentence vector algorithms. 1 Learning Word Vectors and Sentence Vectors Word Vectors: Word vector models aim to learn a map from each word w to d-dimensional vector representation hw given a sequence of words [w1, . , wT ]. This word vector is also called word embedding. In this paper, we mainly treat the continuous skip-gram model (skip-gram) [1], which is a widely used word vector model.1 Skip-gram learns two types of word vectors h and h′. We call h as an input word vector and h′ as an output word vector. The terms ĥ and ĥ′ define fixed input word vector and fixed output word vector trained on a corpus, respectively. Intuitively, skip-gram estimates word vectors by predicting word wc from its neighbor wt. A fast skip-gram model training method is negative sampling proposed by Mikolov et al. [2], whose loss function is defined by LSG = − T ∑ t=1 [ ∑ wc∈Ct lnσ ( hwth ′ wc ) + ∑ wn∈NS lnσ ( −hwth ′ wn )] , (1) where Ct represents a bag-of-words surrounding wt in a sequence, σ is the logistic sigmoid function, and NS is a bag-of-words including negative words sampled from a noise distribution [2]. Sentence Vectors from Pre-trained Word Vectors: In the same way as word vector models, sentence vector models aim to learn a map from a sentence2 from d-dimensional vector representation hS given sentences {S1, . ,SN}, where S is a sequence of words. Since sentences consist of several words, sentence representation is affected by word information, such as sentiment polarity. Given pretrained word vectors, the simplest way to obtain sentence vector hS is to average of pre-trained word vectors of words appearing in the sentence S , e.g., hS = 1 |S| ∑ w∈S ĥw, or hS = 1 |S| ∑ w∈S ĥw+ĥ ′ w 2 . Empirically, these simple methods can beat more complex methods in machine learning tasks for sentences [3, 4]. 2 PAC-Bayesian Analysis of Transferred Sentence Vectors We typically use pre-trained word vectors to learn sentence vectors because of the usefulness of their representations, which capture syntactic and semantics [2]. We formulate learning word vectors and Another popular model, Continuous Bag-of-Words [1], is described in Appendix A.1. We name a sequence of words as a sentence including phrase, paragraph, and document. Third workshop on Bayesian Deep Learning (NeurIPS 2018), Montréal, Canada. sentence vectors in terms of transfer learning [5]. In our transfer learning perspective, a source task minimizes the loss function of skip-gram with negative sampling on word sequences by updating input and output word vectors h and h′. Then, a target task minimizes the loss function by updating sentence vector hS with the fixed pre-trained word vectors ĥ and ĥ′ given a sentence. This formulation enables us to analyze generalization errors in learning sentence vector with PACBayesian theory, which can consider transferability to a target hypothesis from a learned source hypothesis through prior knowledge [6] (See general PAC-Bayes bound in Appendix A.2, and related work of PAC-Bayesian transfer learning in Appendix B.2). PAC-Bayes Bound: We define a PAC-Bayes bound to analyze sentence vector algorithms based on averaging of pre-trained word vectors. Given a sentence S and word vectors trained on the source task, let D be a data distribution over X × Y , where X := {xw|w ∈ N≤|V|} is input sample representing one-hot vector of word w, and Y is output space, e.g., {−1, 1} for binary classification. DS is a training dataset sampled i.i. d. from D, that means sentence S assumes bag-of-words. Let |DS | be the number of training samples, and |S| be the number of words in S. Our goal is to minimize generalization error RD(QS) = Eh∼QS E(x,y)∼D l(x, y, h), where l is a bounded loss function, and h is hypothesis parameterized by sentence vector hS . We give a concrete form of y, l, and h in a later analysis. We define prior distribution over hypothesis class F as PS ∝ ∏ w∈S N (; ĥw, σI), where σ is variance parameter, and I is identity matrix. This prior distribution is known as the product of experts [7] such that each expert is a Gaussian distribution parameterized by pre-trained ĥ′ of word appearing in the given sentence S. We also define posterior distribution over F as QS = N (;hS , σI). Theorem 1 shows the upper bound of generalization error of target task by empirical error of target task with the regulalizer by pre-trained output word vectors. Theorem 1 (PAC-Bayes Bound of Learning Sentence Vector with Pre-trained Word Vectors). Given a sentence S, ∀λ > 0, with probability at least 1− δ over training samples DS , ∀h ∈ QS , RD(QS) ≤ E h∼QS 1 |DS | |DS | ∑ i=1 l(xi, yi, h) + λ 2σ2 ∣∣∣∣ ∣∣∣∣hS − 1 |S|∑ w∈S ĥw ∣∣∣∣ ∣∣∣∣ 2 2 + C, (2) where C is a constant term that does not depends sentence vector hS . We prove Theorem 1 by using the PAC-Bayes bound [8]. The proof is shown in Appendix C.1. Sentence Vectors via Regression: We use Theorem 1 to analyze a heuristic sentence vector models. Let Y := {0}, hypothesis h be h(xw) = 1 2 ||Ĥxw − hS || 2 2, where the w-th column of Ĥ ∈ Rd×|V| corresponds to the w-th pre-trained input word vector ĥw. We introduce the loss function l(x, y, h) = h(x)− y = h(x), because y = 0.3 We can obtain the closed form of hS from Eq. (2), hS = 1 (1 + α)|S| ∑ w∈S ( ĥw + αĥ ′ w ) , (3) where α = λ σ2 . The details of this solution are shown in Appendix C.2. Corollary 1.1 (Sentence Vector by Averaging of Pre-trained Input Word Vectors). The sentence vector hS , estimated by minimizing Eq. (2) with α = 0, is equivalent to 1 |S| ∑ w∈S ĥw. Corollary 1.2 (Sentence Vector by Averaging of Pre-trained Input and Output Word Vectors). Empirically, the averaged vector of the input word vector and output word vector, ĥ+ĥ ′ 2 , can improve the performance of downstream tasks (e.g., Levy et al. [9]). This operation corresponds to the solution of Eq. (2) with the prior defined by the pre-trained output vectors and α = 1. We also introduce another heuristic method with inverse document frequency (IDF) weighting (The details are in Appendix C.3). We change loss function to sample-dependent weighted loss function l(x, y, h) = β(x)h(x) − y = β(x)h(x), where β is an weighing function from one-hot vector to R+. We also change the prior to PS ∝ ∏ w∈S N (; ĥw, σ wI). When we set β(x) = |S|IDF(x) and When we want to evaluate the upper bound, bounded loss l is in [0, 2] if we normalize each pre-trained word vectors and sentence vector by L2-norm.