Title: SEMI-SUPERVISED NAMED ENTITY RECOGNITION WITH CRF-VAES

Abstract: We investigate methods for semi-supervised learning (SSL) of a neural linearchain conditional random field (CRF) for Named Entity Recognition (NER) by treating the tagger as the amortized variational posterior in a generative model of text given tags. We first illustrate how to incorporate a CRF in a VAE, enabling end-to-end training on semi-supervised data. We then investigate a series of increasingly complex deep generative models of tokens given tags enabled by end-to-end optimization, comparing the proposed models against supervised and strong CRF SSL baselines on the Ontonotes5 NER dataset. We find that our best proposed model consistently improves performance by ≈ 1% F1 in low-and moderate-resource regimes and easily addresses degenerate model behavior in a more difficult, partially supervised setting.

1. INTRODUCTION:

Named entity recognition (NER) is a critical subtask of many domain-specific natural language understanding tasks in NLP, such as information extraction, entity linking, semantic parsing, and question answering. State-of-the-art models treat NER as a tagging problem (Lample et al., 2016; Ma & Hovy, 2016; Strubell et al., 2017; Akbik et al., 2018) , and while they have become quite accurate on benchmark datasets in recent years (Lample et al., 2016; Ma & Hovy, 2016; Strubell et al., 2017; Akbik et al., 2018; Peters et al., 2018; Devlin et al., 2018) , utilizing them for new tasks is still expensive, requiring a large corpus of exhaustively annotated sentences (Snow et al., 2008) . This problem has been largely addressed by extensive pretraining of high-capacity sentence encoders on massive-scale language modeling tasks (Peters et al., 2018; Devlin et al., 2018; Howard & Ruder, 2018; Radford et al., 2019; Liu et al., 2019b) , but it is natural to ask if we can squeeze more signal from our unlabeled data.

Latent-variable generative models of sentences are a natural approach to this problem: by treating the tags for unlabeled data as latent variables, we can appeal to the principle of maximum marginal likelihood (Berger, 1985; Bishop, 2006) and learn a generative model on both labeled and unlabeled data. For models of practical interest, however, this presents multiple challenges: learning and prediction both require an intractable marginalization over the latent variables and the specification of the generative model can imply a posterior family that may not be as performant as the current state-of-the-art discriminative models.

We address these challenges using a semi-supervised Variational Autoencoder (VAE) (Kingma et al., 2014) , treating a neural tagging CRF as the approximate posterior. We address the issue of optimization through discrete latent tag sequences by utilizing a differentiable relaxation of the Perturb-and-MAP algorithm (Papandreou & Yuille, 2011; Mensch & Blondel, 2018; Corro & Titov, 2018) , allowing for end-to-end optimization via backpropagation (Rumelhart et al., 1988) and SGD (Robbins & Monro, 1951) . Armed with this learning approach, we no longer need to restrict the generative model family (as in Ammar et al. (2014) ; Zhang et al. (2017) ), and explore the use of rich deep generative models of text given tag sequences for improving NER performance. We also demonstrate how to use the VAE framework to learn in a realistic annotation scenario where we only observe a biased subset of the named entity tags.

Our contributions can be summarized as follows:

1. We address the problem of semi-supervised learning (SSL) for NER by treating a neural CRF as the amortized approximate posterior in a discrete structured VAE. To the best of our knowledge, we are the first to utilize VAEs for NER.

2. We explore several variants of increasingly complex deep generative models of text given tags with the goal of improving tagging performance. We find that a joint tag-encoding Transformer (Vaswani et al., 2017) architecture leads to an ≈ 1% improvement in F1 score over supervised and strong CRF SSL baselines.

3. We demonstrate that the proposed approach elegantly corrects for degenerate model performance in a more difficult partially supervised regime where sentences are not exhaustively annotated and again find improved performance.

4. Finally, we show the utility of our method in realistic low-and high-resource scenarios, varying the amount of unlabeled data. The resulting high-resource model is competitive with state-of-the-art results and, to the best of our knowledge, achieves the highest reported F1 score (88.4%) for models that do not use additional labeled data or gazetteers.

2. METHODS:

We first introduce the tagging problem and tagging model. We then detail our proposed modeling framework and architectures.

2.1. PROBLEM STATEMENT:

NER is the task of assigning coarsely-typed categories to contiguous spans of text. State-of-the-art approaches (Lample et al., 2016; Ma & Hovy, 2016; Strubell et al., 2017; Akbik et al., 2018; Liu et al., 2019a) do so by treating span extraction as a tagging problem, which we now formally define.

We are given a tokenized text sequence x 1:N ∈ X N and would like to predict the corresponding tag sequence y 1:N ∈ Y N which correctly encodes the observed token spans. 1 In this work, we use the BILOU (Ratinov & Roth, 2009 ) tag-span encoding, which assigns four tags for each of the C span categories (e.g., B-PER, I-PER, L-PER, U-PER for the PERSON category.) The tag types B, I, L, U respectively encode beginning, inside, last, and unary tag positions in the original span. Additionally we have one O tag for tokens that are not in any named entity span. Thus our tag space has size |Y| = 4C + 1.

2.2. TAGGING CRF:

We call the NER task of predicting tags for tokens inference, and model it with a discriminative distribution q φ (y 1:N |x 1:N ) having parameters φ. Following state-of-the-art NER approaches (Lample et al., 2016; Ma & Hovy, 2016; Strubell et al., 2017; Akbik et al., 2018) , we use a neural encoding of the input followed by a linear-chain CRF (Lafferty et al., 2001) decoding layer on top.

We use the same architecture for q φ throughout this work, as follows:

1. Encode the token sequence, represented as byte-pairs, with a fixed pretrained language model.2 That is, we first calculate:

h 0 1:N = Pretrained-LM(x 1:N ), h 0 1:N ∈ R N ×dLM

In our first experiments exploring the use of pretrained autoregressive information for generation ( §3.1), we use the GPT2-SM model (Radford et al., 2019; Hugging Face, 2019) .

In the experiments after ( §3.2) we use the RoBERTa-LG model (Liu et al., 2019b; Hugging Face, 2019) .

2. Down-project the states:

h 1 1:N = h 0 1:N W 1 + b 1 , W 1 ∈ R dLM×dy q , b 1 ∈ R dy q 3. Compute local tag scores: s yi = v y h 1 i + b 2,y , v y ∈ R dy q , b 2 ∈ R |Y| 4.

Combine local and transition potentials: ψ yi,yi+1 = s yi + T yi,yi+1 , T yi,yi+1 ∈ R 5. Using special start and end states y 0 = * , y N +1 = with binary potentials ψ * ,y = T * ,y , ψ y, = T y, and the forward algorithm (Lafferty et al., 2001) to compute the the partition function Z, we can compute the joint distribution:

EQUATION

Our tagging CRF has trainable parameters φ = {W 1 , b 1 , V, b 2 , T }3 and we learn them on a dataset of fully annotated sentences D S = {(x i 1:N i , y i 1:N i )} using stochastic gradient descent (SGD) and maximum likelihood estimation.

EQUATION

2.3. SEMI-SUPERVISED CRF-VAE:

We now present the CRF-VAE, which treats the tagging CRF as the amortized approximate posterior in a Variational Autoencoder. We first describe our loss formulations for semi-supervised and partially supervised data. We then address optimizing these objectives end-to-end using backpropagation and the Relaxed Perturb-and-MAP algorithm. Finally, we propose a series of increasingly complex generative models to explore the potential of our modeling framework for improving tagging performance.

2.3.1. SEMI-SUPERVISED VAE:

The purpose of this work is to consider methods for estimation of q φ in semi-supervised data regimes, as in Kingma et al. (2014) ; Miao & Blunsom (2016) ; Yang et al. (2017) , where there is additional unlabeled data D U = {(x i 1:N i )}. To learn in this setting, we consider generative models of tags and tokens p θ (x 1:N |y 1:N )p(y 1:N ) and, for unobserved tags, aim to optimize the marginal likelihood of the observed tokens under the generative model.

log p θ (x 1:N ) = log y 1:N p θ (x 1:N |y 1:N )p(y 1:N )

This marginalization is intractable for models that are not factored among y i , so we resort to optimizing the familiar evidence lower bound (ELBO) (Jordan et al., 1999; Blei et al., 2017) with an approximate variational posterior distribution, which we set to our tagging model q φ . We maximize the ELBO on unlabeled data in addition to maximum likelihood losses for both the inference and generative models on labeled data, yielding the following objectives:

EQUATION

EQUATION

)

where α is scalar hyper-parameter used to balance the supervised loss L S and the unsupervised loss L U (Kingma et al., 2014) . β is a scalar hyper-parameter used to balance the reconstruction and KL terms for the unsupervised loss (Bowman et al., 2015; Higgins et al., 2017) . We note that, unlike a traditional VAE, this model contains no continuous latent variables.

2.3.2. PARTIALLY SUPERVISED LEARNING (PSL):

Assuming that supervised sentences are completely labeled is a restrictive setup for semi-supervised learning of a named entity tagger. It would be useful to be able to learn the tagger on sentences which are only partially labeled, where we only observe some named entity spans, but are not guaranteed all entity spans in the sentence are annotated and no O tags are manually annotated. 4 This presents a challenge in that we are no longer able to assume the usual implicit presence of O tags, since unannotated tokens are ambiguous. While it is possible to optimize the marginal likelihood of the CRF on only the observed tags y O , O ⊂ {1, . . . , N } in the sentence (Tsuboi et al., 2008) , doing so naively will result in a degenerate model that never predicts O, by far the most common tag (Jie et al., 2019) . Interestingly, this scenario is easily addressed by the variational framework via the KL term. We do this by reformulating the objective in Equation 5to account for partially observed tag sequences:

Let D P = {(x i 1:Ni , y i O )

} be the partially observed dataset where, for some sentence i, O ⊂ {1, . . . , N i } is the set of observed positions and U = {1, . . . , N i } \ O is the set of unobserved positions. Our partially supervised objective is then

L P = (x,y O )∈D P log q φ (y O |x) + αE q φ [log p θ (x|y O ∪ y U )] -αβKL(q φ (y U |x, y O )||p(y U )) (6)

which can be optimized as before using the constrained forward-backward and KL algorithms detailed in Appendix B.

We also explore using this approach simply for regularization of the CRF posterior by omitting the token model p θ (x|y). Since we do not have trainable parameters for the generative model in this case, the reconstruction likelihood drops out of the objective and we have, for a single datum (x i , y i O ) ∈ D P , the following loss:

L i P = log q φ (y i O |x i ) -αβKL(q φ (y i U |x i , y i O )||p(y i U ))

2.3.3. DIFFERENTIABLE PERTURB-AND-MAP:

Optimizing Equations 5 and 6 with respect to θ and φ using backpropagation and SGD is straightforward for every term except for the expectation terms E q φ (y|x) [log p θ (x|y)]. To optimize these expectations, we first make an Monte Carlo approximation using a single sample drawn from q φ . This discrete sample, however, is not differentiable with respect to φ and blocks gradient computation. While we may appeal to score function estimation (Miller, 1967; Williams, 1992; Paisley et al., 2012; Ranganath et al., 2014; Miao & Blunsom, 2016; Mohamed et al., 2019) to work around this, its high-variance gradients make successful optimization difficult.

Following Papandreou & Yuille (2011) ; Mensch & Blondel (2018) ; Corro & Titov (2018) ; Kim et al. (2019) , we can compute approximate samples from q φ that are differentiable with respect to φ using the Relaxed Perturb-and-MAP algorithm (Corro & Titov, 2018; Kim et al., 2019) . Due to space limitations, we leave the derivation of Relaxed Perturb-and-MAP for linear-chain CRFs to Appendix A and detail the resulting CRF algorithms in Appendix B.

2.4. PROPOSED GENERATIVE MODELS:

We model the prior distribution of tag sequences y 1:N as the per-tag product of a fixed categorical distribution p(y 1:N ) = i p(y i ). The KL between q φ and this distribution can be computed in polynomial time using a modification of the forward recursion derived in Mann & McCallum (2007) , detailed in Appendix B.

We experiment with several variations of architectures for p θ (x 1:N |y 1:N ), presented in order of increasing complexity.

Baseline -CRF-Autoencoder (AE): The CRF Autoencoder (Ammar et al., 2014; Zhang et al., 2017) is the previous state-of-the-art semi-supervised linear-chain CRF, which we consider a strong baseline. This model uses a tractable, fully factored generative model of tokens given tags and does not require approximate inference. Due to space limitations, we have detailed our implementation in Appendix C.

MF: This is our simplest proposed generative model. We first embed the relaxed tag samples, represented as simplex vectors y i ∈ ∆ |Y| , into R dy p as the weighted combination of the input vector representations for each possible tag:

u i = U y i , U ∈ R dy p ×|Y| (7)

We then compute factored token probabilities with an inner product

p θ (x i |y i ) = σ X (w xi u i )

where σ X is the softmax function normalized over X . This model is generalization of the CRF Autoencoder architecture in Appendix C where the tag-token parameters θ x,y are computed with a low-rank factorization W U .

MT::

The restrictive factorization of MF is undesirable, since we expect that information about nearby tags may be discriminative of individual tokens. To test this, we extend MF to use the full tag context by encoding the embedded tag sequence jointly using a two-layer transformer (Vaswani et al., 2017) with four attention heads per layer before predicting the tokens independently. That is,

p θ (x i |y 1:N ) = σ X (w xi v i ), v 1:N = Transformer θ (u 1:N )

MF-GPT2: Next, we see if we can leverage information from a pretrained language model to provide additional training signal to p θ . We extend MF by adding the fixed pretrained language modeling parameters from GPT2 to the token scores:

p θ (x i |y i , x <i ) = σ X w xi u i d yp + z xi h 0 i √ d GPT2

where z xi and h 0 i are the input token embeddings and hidden states from GPT2, respectively. We additionally normalize the scales of the factors by the square root of the vector dimensionalities to prevent the GPT2 scores from washing out the tag-encoding scores (d yp = 300 and d GP T 2 = 768).

MT-GPT2::

We add the same autoregressive extention to MT, using the tag encodings v instead of embeddings u.

p θ (x i |y 1:N , x <i ) = σ X w xi v i d yp + z xi h 0 i √ d GPT2

MT-GPT2-PoE: We also consider an autoregressive extension of MT, similar to MT-GPT2, that uses a product of experts (PoE) (Hinton, 2002) factorization instead

p θ (x i |y, x <i ) = σ X p θ (x i |y 1:N )p GPT2 (x i |x <i ) p θ (x i |y 1:N ) = σ X (w xi v i ), p GPT2 (x i |x <i ) = σ X (z xi h 0 i )

MT-GPT2-Residual: Our last variation directly couples GPT2 with p θ by predicting a residual via a two-layer MLP based on the tag encoding and GPT2 state:

p θ (x i |y 1:N , x <i ) = σ X (z xi h0 i ), h0 i = h 0 i + f MLP ( h 0 i , v i )

For the MF-GPT2, MT-GPT2, and MT-GPT2-PoE models, we choose these factorizations specifically to prevent the trainable parameters from conditioning on previous word information, removing the possibility of the model learning to ignore the noisy latent tags in favor of the strong signal provided by pretrained encodings of the sentence histories (Bowman et al., 2015; Yang et al., 2017; Kim et al., 2018) . We further freeze the GPT2 parameters for all models, forcing the only path for improving the generative likelihood to be through the improved estimation and encoding of the tags y 1:N .

3. EXPERIMENTS AND RESULTS:

We experiment first with the proposed models generative models for SSL and PSL in a moderately resourced regime (keeping 10% labeled data) to explore their relative merits. We then evaluate our best generative model from these experiments, (MT), with an improved bidirectional encoder language model in a low-and high-resource settings, varying the amount of unlabeled data.5 

For data, we use the OntoNotes 5 (Hovy et al., 2006) NER corpus, which consists of 18 entity types annotated in 82,120 train, 12,678 validation, and 8,968 test sentences.

3.1. EXPLORATION OF GENERATIVE ARCHITECTURES:

We begin by comparing the proposed generative models, M* along with the following baselines:

1. Supervised (S): The supervised tagger trained only on the 10% labeled data. 2. Supervised 100% (S*): The supervised tagger trained on the 100% labeled data, used for quantifying the performance loss from using less data. 3. AE-Exact: The CRF Autoencoder using exact inference (detailed in Appendix C.) 4. AE-Approx: The same tag-token pair parameterization used by the CRF Autoencoder, but trained with the approximate ELBO objective as in Equation 11instead of the exact objective in Equation 12. The purpose here is to see if we lose anything by resorting to the approximate ELBO objective.

To simulate moderate-resource SSL, we keep annotations for only 10% of the sentences, yielding 8, 212 labeled sentences with 13, 025 annotated spans and 73, 908 unlabeled sentences. Results are shown in Table 1 . All models except S* use this 10% labeled data.

We first evaluate the proposed models and baselines without the use of a prior, since the use of a locally normalized factored prior can encourage overly uncertain joint distributions and degrade performance (Jiao et al., 2006; Mann & McCallum, 2007; Corro & Titov, 2018) . We then explore the inclusion of the priors for the supervised and MT models with β = 0.01.6 

We explore two varieties of prior tag distributions: (1) the "gold" empirical tag distribution (Emp) from the full training dataset and (2) a simple, but informative, hand-crafted prior (Sim) that places 50% mass on the O tag and distributes the rest of its mass evenly among the remaining tags. We view (2) as a practical approach, since it does not require knowledge of the gold tag distribution, and use (1) to quantify any relative disadvantage from not using the gold prior. We find that including the prior with a small weight, β = 0.01, marginally improved performance and interestingly, the simple prior outperforms the empirical prior, most likely because it is slightly smoother and does not emphasize the O tag as heavily. 7Curiously, we found that the approximate training of the CRF Autoencoder AE-Approx outperformed the exact approach AE-Exact by nearly 2% F1.

We also note that our attempts to leverage signal from the pretrained autoregressive GPT2 states had negligible or negative effects on performance, thus we conclude that it is the addition of the joint encoding transformer architecture MT that provides the most gains (+0.8% F1).

PSL::

We also evaluate the supervised and transformer-based generative models, S and MT, on the more difficult PSL setup, where naively training the supervised model on the marginal likelihood of observed tags produces a degenerate model, due to the observation bias of never having O tags. In this setting we drop 90% of the annotations from sentences randomly, resulting in 82,120 incompletely annotated sentences with 12,883 annotations total. We compare the gold and simple priors for each model. From the bottom of Table 1 , we see that again our proposed transformer model MT outperforms the supervised-only model, this time by +1.3% F1. We also find that in this case, the MT models need to be trained with higher prior weights β = 0.1, otherwise they diverge towards using the O tag more uniformly with the other tags to achieve better generative likelihoods. 

3.2. VARYING RESOURCES:

Next we explore our best proposed architecture MT and the supervised baseline in low-and highresource settings (1% and 100% training data, respectively) and study the effects of training with an additional 100K unlabeled sentences sampled from Wikipedia (detailed in Appendix E).

Since we found no advantage from using pretrained GPT2 information in the previous experiment, we evaluate the use of the bidirectional pretrained language model, RoBERTa (Liu et al., 2019b) , since we expect bidirectional information to highly benefit performance (Strubell et al. (2017) ; Akbik et al. (2018) , among others). We also experiment with a higher-capacity tagging model, S-LG, by adding more trainable Transformers (L = 4, A = 8, H = 1024) between the RoBERTa encodings and down-projection layers.

From Table 2 we see that, like in the 10% labeled data setting, the CRF-VAE improves upon the supervised model by 0.9% F1 in this 1% setting, but we find that including additional data from Wikipedia has a negative impact. A likely reason for this is the domain mismatch between Ontonotes5 and Wikipedia (news and encyclopedia, respectively).

In the high-resource setting, we find that using RoBERTa significantly improves upon GPT2 (+5.7% F1) and the additional capacity of S-LG further improves performance by +2.2% F1. Although we do not see a significant improvement from semi-supervised training with Wikipedia sentences, our model is competitive with previous state-of-the-art NER approaches and outperforms all previous approaches that do not use additional labeled data or gazetteers.

4. RELATED WORK:

Utilizing unlabeled data for semi-supervised learning in NER has been studied considerably in the literature. A common approach is a two-stage process where useful features are learned from unsupervised data, then incorporated into models which are then trained only on the supervised data (Fernandes & Brefeld, 2011; Kim et al., 2015) . With the rise of neural approaches, large-scale word vector (Mikolov et al., 2013; Pennington et al., 2014) and language model pretraining methods (Peters et al., 2018; Akbik et al., 2018; Devlin et al., 2018) can be regarded in the same vein. Another approach is to automatically create silver-labeled data using outside resources, whose low recall induces a partially supervised learning problem. Bellare & McCallum (2007) approach the problem by distantly supervising (Mintz et al., 2009) spans using a database. Carlson et al. (2009) similarly use a gazetteer and adapt the structured perceptron (Collins, 2002) to handle partially labeled sequences, while Yang et al. (2018) optimize the marginal likelihood (Tsuboi et al., 2008) of the distantly annotated tags. Yang et al. (2018) 's method, however, still requires some fully labeled data to handle proper prediction of the O tag. The problem setup from Jie et al. (2019) is the same as our PSL regime, but they use a cross-validated self-training approach. Greenberg et al. (2018) use a marginal likelihood objective to pool overlapping NER tasks and datasets, but must exploit datasetspecific constraints, limiting the allowable latent tags to debias the model from never predicting O tags.

Model |D S | |D U | α β P R F1 S 13K 0 - - 0.

Generative latent-variable approaches also provide an attractive approach to learning on unsupervised data. Ammar et al. (2014) present an approach that uses the CRF for autoencoding and Zhang et al. (2017) extend it to neural CRFs, but both require the use of a restricted factored generative model to make learning tractable. Deep generative models of text have shown promise in recent years, with demonstrated applications to document representation learning (Miao et al., 2016) , sentence generation (Bowman et al., 2015; Yang et al., 2017; Kim et al., 2018) , compression (Miao & Blunsom, 2016) , translation (Deng et al., 2018) , and parsing (Corro & Titov, 2018) . However, to the best of our knowledge, this framework has yet to be utilized for NER and tagging CRFs. A key challenge for learning VAEs with discrete latent variables is optimization with respect to the inference model parameters φ. While we may appeal to score function estimation (Williams, 1992; Paisley et al., 2012; Ranganath et al., 2014; Miao & Blunsom, 2016) , its empirical high-variance gradients make successful optimization difficult. Alternatively, obtaining gradients with respect to φ can be achieved using the relaxed Gumbel-max trick (Jang et al., 2016; Maddison et al., 2016) and has been recently extended to latent tree-CRFs by (Corro & Titov, 2018) , which we make use of here for sequence CRFs.

5. CONCLUSIONS:

We proposed a novel generative model for semi-supervised learning in NER. By treating a neural CRF as the amortized variational posterior in the generative model and taking relaxed differentiable samples, we were able to utilize a transformer architecture in the generative model to condition on more context and provide appreciable performance gains over supervised and strong baselines on both semi-supervised and partially-supervised datasets. We also found that inclusion of powerful pretrained autoregressive language modeling states had neglible or negative effects while using a pretrained bidirectional encoder offers significant performance gains. Future work includes the use of larger in-domain unlabeled corpora and the inclusion of latent-variable CRFs in more interesting joint semi-supervised models of annotations, such as relation extraction and entity linking.

Algorithm 1 Relaxed, Constrained, Perturbed Forward-Backward Notation: LSE x := log x exp Input: Local potentials Ψ y 1:N , transition potentials Ψ y,y , perturb boolean, temperature τ , the special start symbol and end symbols * , , and the set of allowable tags for each position Y i ⊆ Y (allows for partially observed/constrained sequences.) Procedure:

1: log α[0, y] ← ψ * ,y /τ, log β[N + 1, y] ← ψ y, /τ Initialize recursions bases 2: if perturb then 3: 

ψ yi ← ψ yi + γ yi , γ yi iid ∼ G(0,

log α[i, y i ] ← LSE yi-1∈Yi-1 (ψ yi-1,yi + ψ yi ) + log α[i -1, y i-1 ] 5:

end for 6:

for y i+1 ∈ Y i+1 do Compute KL lattice 7: where log Z(ψ + log p θ ) is a slight abuse of notation intended to illustrate that the first term in Equation 12is the same computation as the partition function, but with the generative log-likelihoods added to the CRF potentials.

q(y i |y i+1 ) ← σ Yi (log α[i, y i ] + ψ yi,yi+1 + ψ yi+1 ) 8: KL α [i, y i+1 ] ← yi∈Yi q(y i |y i+1 ) log q(y i |y i+1 ) -log p(y i ) + KL α [i -1, y i ]

We note that instead of using the Mixed-EM procedure from Zhang et al. (2017) , we model p θ (x i |y i ) using free logit parameters θ x,y for each token-tag pair and normalize using a softmax, which allows for end-to-end optimization via backpropagation and SGD.

D EXPERIMENT HYPERPARAMETER AND OPTIMIZATION SETTINGS:

We train each model to convergence using early-stopping on the F1 score of the validation data, with a patience of 10 epochs. For all models that do not have trainable transformers, we train using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0.001, and a batch size of 128. For those with transformers (MT*), we train using Adam, a batch size of 32, and the Noam learning rate schedule from Vaswani et al. (2017) with a model size of d yp = 300 and 16, 000 warm-up steps (Popel & Bojar, 2018) .

Additionally, we use gradient clipping of 5 for all models and a temperature of τ = .66 for all relaxed sampling models. We implemented our models in PyTorch (Paszke et al., 2017) using the AllenNLP framework (Gardner et al., 2018) and the Hugging Face (2019) implementation of the pretrained GPT2 and RoBERTa.

We have made all code, data, and experiments available online at github.com/ <anonymizedforsubmission> for reproducibility and reuse. All experimental settings can be reproduced using the configuration files in the repo.

E GATHERING ADDITIONAL UNLABELED DATA:

For the experiments in §3.2, we gather an additional training corpus of out-of-domain encyclopedic sentences from Wikipedia. To try to get a sample that better aligns with the Ontonotes5 data, these sentences were gathered with an informed process, which was performed as follows:

1. Using the repository <anonymized for submission>, we extract English Wikipedia and align it with Wikidata. 2. We then look up the entity classes from the Ontonotes5 specification (Hovy et al., 2006) in Wikidata and, for each NER class, find all Wikidata classes that are below this class in ontology (all subclasses). 3. We then find all items which are instances of these classes and also have Wikipedia pages.

These are the Wikipedia entities which are likely to be instances of the NER classes.

:

We will often omit sequence boundaries (x ← x1:N ) to save space, but will always index individual elements xi.

We represent the tags at the byte-pair level to ensure alignment between the number of tokens and tags for the generative models in §2.3

We omit the Pretrained-LM parameters since they are not updated during training.

This regime applies to situations such as weak supervision (i.e. a low-recall database or gazatteer used for distant supervision), incidental supervision (i.e., a random Wikipedia sentence), or online and active learning.

Code and experiments are available online at github.com/<anonymizedforsubmission>

In preliminary SSL experiments we found β > 0.01 to have a negative impact on performance, likely due to global/local normalization mismatch of the CRF and the prior.

The empirical prior puts

85% mass on the O tag

Figures and Tables:

10: end for Output: KL(q||p) ← KL α [N, ] here for clarity. Termed the CRF Autoencoder, the model treats the the tags as intermediate latent variables in a conditional model and optimizes the marginal conditional likelihood of reconstructing the input. log p(x|x) = log y 1:N p θ (x|y 1:N )q φ (y 1:N |x) By judiciously choosing p θ (x|y) to be factored among positions i, we can compute the marginal reconstruction likelihood exactly: yi+1 + log p θ (x i |y i )} -log Z(ψ) = log Z(ψ + log p θ ) -log Z(ψ) (12)

Semi-supervised and partially-supervised models on 10% supervised training data: best in bold, second best underlined. The proposed MT* improves performance in SSL and PSL by +1.1% F1 and +1.3% F1, respectively.

Table content: <table><tr><td>Model</td><td>α</td><td>β</td><td>P</td><td>R</td><td>F1</td></tr><tr><td>Supervised 100% (S*)</td><td>-</td><td colspan="4">0.0 0.808 0.798 0.803</td></tr><tr><td>Supervised (S)</td><td>-</td><td colspan="4">0.0 0.761 0.738 0.749</td></tr><tr><td>AE-Exact</td><td>-</td><td colspan="4">0.0 0.736 0.721 0.728</td></tr><tr><td>AE-Approx</td><td colspan="5">0.1 0.0 0.767 0.728 0.747</td></tr><tr><td>MF (Factored)</td><td colspan="5">0.1 0.0 0.761 0.719 0.739</td></tr><tr><td>MT (Transformer)</td><td colspan="5">0.1 0.0 0.758 0.756 0.757</td></tr><tr><td>MF-GPT2</td><td colspan="5">0.1 0.0 0.754 0.710 0.731</td></tr><tr><td>MT-GPT2</td><td colspan="5">0.1 0.0 0.762 0.755 0.759</td></tr><tr><td>MT-GPT2 (no-scale)</td><td colspan="5">0.1 0.0 0.766 0.734 0.751</td></tr><tr><td>MT-GPT2-PoE</td><td colspan="5">0.1 0.0 0.766 0.742 0.753</td></tr><tr><td>MT-GPT2-Residual</td><td colspan="5">0.1 0.0 0.766 0.740 0.753</td></tr><tr><td>S (Emp)</td><td colspan="5">0.1 0.01 0.754 0.733 0.743</td></tr><tr><td>S (Sim)</td><td colspan="5">0.1 0.01 0.754 0.734 0.743</td></tr><tr><td>MT (Emp)</td><td colspan="5">0.1 0.01 0.760 0.756 0.758</td></tr><tr><td>MT (Sim)</td><td colspan="5">0.1 0.01 0.762 0.757 0.760</td></tr><tr><td>S (Emp) -PSL</td><td colspan="5">0.1 0.01 0.741 0.725 0.733</td></tr><tr><td>S (Sim) -PSL</td><td colspan="5">0.1 0.01 0.730 0.740 0.735</td></tr><tr><td>MT (Emp) -PSL</td><td colspan="5">0.1 0.1 0.731 0.761 0.746</td></tr><tr><td>MT (Sim) -PSL</td><td colspan="5">0.1 0.1 0.724 0.774 0.748</td></tr></table>

Low-and high-resource results with RoBERTa, varying available unlabeled data. Best scores not using additional labeled data in bold. † Uses additional labeled data or gazetteers.

Table content: <table><tr><td>744 0.712 0.728</td></tr></table>

Local potentials Ψ y 1:N , transition potentials Ψ y,y , prior distribution p(y 1:N ) = p(y i ), the special start symbol and end symbols * , , and the set of allowable tags for each position Y i ⊆ Y (allows for partially observed/constrained sequences.) Procedure:1: log α[0, y] ← ψ * ,y , KL α [0, y] ← 0 ∀y ∈ Y 1 Initialize recursions bases 2: for i = 1, . . . , N do

Table content: <table><tr><td/><td/><td/><td>1)</td><td>Perturb local potentials</td></tr><tr><td colspan="2">4: end if</td><td/></tr><tr><td colspan="3">5: for i = 1, . . . , N do</td><td>Compute forward lattice</td></tr><tr><td>6:</td><td colspan="2">for y ∈ Y i do</td></tr><tr><td>7:</td><td colspan="2">log α[i, y] ← LSE y ∈Yi-1</td><td>(ψ y ,y + ψ y )/τ + log α[i -1, y ]</td></tr><tr><td>8:</td><td>end for</td><td/></tr><tr><td colspan="2">9: end for</td><td/></tr><tr><td colspan="3">10: for i = N, . . . , 1 do</td><td>Compute backward lattice</td></tr><tr><td>11:</td><td colspan="2">for y ∈ Y i do</td></tr><tr><td>12:</td><td colspan="2">log β[i, y] ← LSE</td></tr><tr><td>13:</td><td>end for</td><td/></tr><tr><td colspan="2">14: end for</td><td/></tr><tr><td colspan="2">15: µ yi ← σ Yi</td><td>LSE</td></tr><tr><td>3:</td><td colspan="2">for y i ∈ Y i do</td><td>Same as forward algorithm</td></tr><tr><td>4:</td><td/><td/></tr></table>

Peer review (iclr2020):
The paper considers the task of Named Entity Recognition and formulates it as a the task of segmenting a sequence of text tokens by a corresponding sequence of tags. This requires to use a special tag symbol which is labelling text parts that are not in any entity span. Consequently this special tag will never occur in partially annotated training examples. Moreover, the authors consider and explore a multitude of generative models whose decoders (conditional probability of the text sequence given the tag sequence) are beyond simple conditional independent models (as in standard HMMs). To learn such models from semi-supervised and partially labelled training data, the authors propose to use the VAE approach, assuming the encoder to be a linear CRF model (i.e. a conditional HMM). Implementing this program requires (i) to formulate the tasks for unlabelled and partially labelled training data, (2) to use an approximate but differentiable sampler for the encoder model and (3) to compute the KL-divergence for Markov chain models. The authors consider known options for each of these problems and then analyse the resulting approach for a multitude of generative models experimentally. In my opinion the task formulation, the related models and considered learning tasks are highly interesting and conceptually relevant. Nevertheless, I would not recommend to publish the paper in its present state for the following reasons. The paper is in my view to much application oriented and cluttered with various application/implementation details, which are rather obscuring several interesting and relevant conceptual questions. The same holds in my view for the unnecessary large number of considered model variants. The paper can be improved by moving the conceptual and technical explanations from appendices to the main body and delegating application details, details of some model variants etc. to the appendices. Further questions/comments: - Why are you using a fully factorising prior for the tag sequences? Would it be possible to use a Markov chain model here? - Why and where from comes the log probability of the encoder (q) in the supervised loss? - The score function estimation and other similar approaches indeed suffer from high variance gradients. On the other hand, the proposed relaxed *perturb and MAP* approach is clearly an approximation. It is therefore not clear to me why the latter is to be preferred to the former.