Nonsymbolic Text Representation

Abstract: We introduce the first generic text representation model that is completely nonsymbolic, i.e., it does not require the availability of a segmentation or tokenization method that attempts to identify words or other symbolic units in text. This applies to training the parameters of the model on a training corpus as well as to applying it when computing the representation of a new text. We show that our model performs better than prior work on an information extraction and a text denoising task.

Character-level models can be grouped into three classes. (i) End-to-end models learn a separate model on the raw character (or byte) input for each task; these models estimate task-specific parameters, but no representation of text that would be usable across tasks is computed. Throughout this paper, we refer to INLINEFORM0 as the “representation” of INLINEFORM1 only if INLINEFORM2 is a generic rendering of INLINEFORM3 that can be used in a general way, e.g., across tasks and domains. The activation pattern of a hidden layer for a given input sentence in a multilayer perceptron (MLP) is not a representation according to this definition if it is not used outside of the MLP. (ii) Character-level models of words derive a representation of a word INLINEFORM4 from the character string of INLINEFORM5 , but they are symbolic in that they need text segmented into tokens as input. (iii) Bag-of-character-ngram models, bag-of-ngram models for short, use character ngrams to encode sequence-of-character information, but sequence-of-ngram information is lost in the representations they produce.[0]A short version of this paper appears as BIBREF0 .
Our premise is that text representations are needed in NLP. A large body of work on word embeddings demonstrates that a generic text representation, trained in an unsupervised fashion on large corpora, is useful. Thus, we take the view that group (i) models, end-to-end learning without any representation learning, is not a good general approach for NLP.
We distinguish training and utilization of the text representation model. We use “training” to refer to the method by which the model is learned and “utilization” to refer to the application of the model to a piece of text to compute a representation of the text. In many text representation models, utilization is trivial. For example, for word embedding models, utilization amounts to a simple lookup of a word to get its precomputed embedding. However, for the models we consider, utilization is not trivial and we will discuss different approaches.
Both training and utilization can be either symbolic or nonsymbolic. We define a symbolic approach as one that is based on tokenization, i.e., a segmentation of the text into tokens. Symbol identifiers (i.e., tokens) can have internal structure – a tokenizer may recognize tokens like “to and fro” and “London-based” that contain delimiters – and may be morphologically analyzed downstream.
We define a nonsymbolic approach as one that is tokenization-free, i.e., no assumption is made that there are segmentation boundaries and that each segment (e.g., a word) should be represented (e.g., by a word embedding) in a way that is independent of the representations (e.g., word embeddings) of neighboring segments. Methods for training text representation models that require tokenized text include word embedding models like word2vec BIBREF1 and most group (ii) methods, i.e., character-level models like fastText skipgram BIBREF2 .
Bag-of-ngram models, group (iii) models, are text representation utilization models that typically compute the representation of a text as the sum of the embeddings of all character ngrams occurring in it, e.g., WordSpace BIBREF3 and CHARAGRAM BIBREF4 . WordSpace and CHARAGRAM are examples of mixed training-utilization models: training is performed on tokenized text (words and phrases), utilization is nonsymbolic.
We make two contributions in this paper. (i) We propose the first generic method for training text representation models without the need for tokenization and address the challenging sparseness issues that make this difficult. (ii) We propose the first nonsymbolic utilization method that fully represents sequence information – in contrast to utilization methods like bag-of-ngrams that discard sequence information that is not directly encoded in the character ngrams themselves.
chung16characternmt give two motivations for their work on character-level models. First, tokenization (or, equivalently, segmentation) algorithms make many mistakes and are brittle: “we do not have a perfect word segmentation algorithm for any one language”. Tokenization errors then propagate throughout the NLP pipeline.
Second, there is currently no general solution for morphology in statistical NLP. For many languages, high-coverage and high-quality morphological resources are not available. Even for well resourced languages, problems like ambiguity make morphological processing difficult; e.g., “rung” is either the singular of a noun meaning “part of a ladder” or the past participle of “to ring”. In many languages, e.g., in German, syncretism, a particular type of systematic morphological ambiguity, is pervasive. Thus, there is no simple morphological processing method that would produce a representation in which all inflected forms of “to ring” are marked as having a common lemma; and no such method in which an unseen form like “aromatizing” is reliably analyzed as a form of “aromatize” whereas an unseen form like “antitrafficking” is reliably analyzed as the compound “anti+trafficking”.
Of course, it is an open question whether nonsymbolic methods can perform better than morphological analysis, but the foregoing discussion motivates us to investigate them.
chung16characternmt focus on problems with the tokens produced by segmentation algorithms. Equally important is the problem that tokenization fails to capture structure across multiple tokens. The job of dealing with cross-token structure is often given to downstream components of the pipeline, e.g., components that recognize multiwords and named entitites in English or in fact any word in a language like Chinese that uses no overt delimiters. However, there is no linguistic or computational reason in principle why we should treat the recognition of a unit like “electromechanical” (containing no space) as fundamentally different from the recognition of a unit like “electrical engineering” (containing a space). Character-level models offer the potential of uniform treatment of such linguistic units.
Many text representation learning algorithms can be understood as estimating the parameters of the model from a unit-context matrix INLINEFORM0 where each row corresponds to a unit INLINEFORM1 , each column to a context INLINEFORM2 and each cell INLINEFORM3 measures the degree of association between INLINEFORM4 and INLINEFORM5 . For example, the skipgram model is closely related to an SVD factorization of a pointwise mutual information matrix BIBREF5 ; in this case, both units and contexts are words. Many text representation learning algorithms are formalized as matrix factorization (e.g., BIBREF6 , BIBREF7 , BIBREF8 ), but there may be no big difference between implicit (e.g., BIBREF9 ) and explicit factorization methods; see also BIBREF10 , BIBREF11 .
Our goal in this paper is not to develop new matrix factorization methods. Instead, we will focus on defining the unit-context matrix in such a way that no symbolic assumption has to be made. This unit-context matrix can then be processed by any existing or still to be invented algorithm.
Definition of units and contexts. How to define units and contexts without relying on segmentation boundaries? In initial experiments, we simply generated all character ngrams of length up to INLINEFORM0 (where INLINEFORM1 is a parameter), including character ngrams that cross token boundaries; i.e., no segmentation is needed. We then used a skipgram-type objective for learning embeddings that attempts to predict, from ngram INLINEFORM2 , an ngram INLINEFORM3 in INLINEFORM4 's context. Results were poor because many training instances consist of pairs INLINEFORM5 in which INLINEFORM6 and INLINEFORM7 overlap, e.g., one is a subsequence of the other. So the objective encourages trivial predictions of ngrams that have high string similarity with the input and nothing interesting is learned.
In this paper, we propose an alternative way of defining units and contexts that supports well-performing nonsymbolic text representation learning: multiple random segmentation. A pointer moves through the training corpus. The current position INLINEFORM0 of the pointer defines the left boundary of the next segment. The length INLINEFORM1 of the next move is uniformly sampled from INLINEFORM2 where INLINEFORM3 and INLINEFORM4 are the minimum and maximum segment lengths. The right boundary of the segment is then INLINEFORM5 . Thus, the segment just generated is INLINEFORM6 , the subsequence of the corpus between (and including) positions INLINEFORM7 and INLINEFORM8 . The pointer is positioned at INLINEFORM9 , the next segment is sampled and so on. An example of a random segmentation from our experiments is “@he@had@b egu n@to@show @his@cap acity@f” where space was replaced with “@” and the next segment starts with “or@”.
The corpus is segmented this way INLINEFORM0 times (where INLINEFORM1 is a parameter) and the INLINEFORM2 random segmentations are concatenated. The unit-context matrix is derived from this concatenated corpus.
Multiple random segmentation has two advantages. First, there is no redundancy since, in any given random segmentation, two ngrams do not overlap and are not subsequences of each other. Second, a single random segmentation would only cover a small part of the space of possible ngrams. For example, a random segmentation of “a rose is a rose is a rose” might be “[a ros][e is a ros][e is][a rose]”. This segmentation does not contain the segment “rose” and this part of the corpus can then not be exploited to learn a good embedding for the fourgram “rose”. However, with multiple random segmentation, it is likely that this part of the corpus does give rise to the segment “rose” in one of the segmentations and can contribute information to learning a good embedding for “rose”.
We took the idea of random segmentation from work on biological sequences BIBREF12 , BIBREF13 . Such sequences have no delimiters, so they are a good model if one believes that delimiter-based segmentation is problematic for text.
The main text representation model that is based on ngram embeddings similar to ours is the bag-of-ngram model. A sequence of characters is represented by a single vector that is computed as the sum of the embeddings of all ngrams that occur in the sequence. In fact, this is what we did in the entity typing experiment. In most work on bag-of-ngram models, the sequences considered are words or phrases. In a few cases, the model is applied to longer sequences, including sentences and documents; e.g., BIBREF3 , BIBREF4 .
The basic assumption of the bag-of-ngram model is that sequence information is encoded in the character ngrams and therefore a “bag-of” approach (which usually throws away all sequence information) is sufficient. The assumption is not implausible: for most bags of character sequences, there is only a single way of stitching them together to one coherent sequence, so in that case information is not necessarily lost (although this is likely when embeddings are added). But the assumption has not been tested experimentally.
Here, we propose position embeddings, character-ngram-based embeddings that more fully preserve sequence information. The simple idea is to represent each position as the sum of all ngrams that contain that position. When we set INLINEFORM0 , INLINEFORM1 , this means that the position is the sum of INLINEFORM2 ngram embeddings (if all of these ngrams have embeddings, which generally will be true for some, but not for most positions). A sequence of INLINEFORM3 characters is then represented as a sequence of INLINEFORM4 such position embeddings.
Form-meaning homomorphism premise. Nonsymbolic representation learning does not preprocess the training corpus by means of tokenization and considers many ngrams that would be ignored in tokenized approaches because they span token boundaries. As a result, the number of ngrams that occur in a corpus is an order of magnitude larger for tokenization-free approaches than for tokenization-based approaches. See supplementary for details.
We will see below that this sparseness impacts performance of nonsymbolic text representation negatively. We address sparseness by defining ngram equivalence classes. All ngrams in an equivalence class receive the same embedding.
The relationship between form and meaning is mostly arbitrary, but there are substructures of the ngram space and the embedding space that are systematically related by homomorphism. In this paper, we will assume the following homomorphism: INLINEFORM0 
where INLINEFORM0 iff INLINEFORM1 for string transduction INLINEFORM2 and INLINEFORM3 iff INLINEFORM4 .
As a simple example consider a transduction INLINEFORM0 that deletes spaces at the beginning of ngrams, e.g., INLINEFORM1 . This is an example of a meaning-preserving INLINEFORM2 since for, say, English, INLINEFORM3 will not change meaning. We will propose a procedure for learning INLINEFORM4 below.
We define INLINEFORM0 as “closeness” – not as identity – because of estimation noise when embeddings are learned. We assume that there are no true synonyms and therefore the direction INLINEFORM1 also holds. For example, “car” and “automobile” are considered synonyms, but we assume that their embeddings are different because only “car” has the literary sense “chariot”. If they were identical, then the homomorphism would not hold since “car” and “automobile” cannot be converted into each other by any plausible meaning-preserving INLINEFORM2 .
Learning procedure. To learn INLINEFORM0 , we define three templates that transform one ngram into another: (i) replace character INLINEFORM1 with character INLINEFORM2 , (ii) delete character INLINEFORM3 if its immediate predecessor is character INLINEFORM4 , (iii) delete character INLINEFORM5 if its immediate successor is character INLINEFORM6 . The learning procedure takes a set of ngrams and their embeddings as input. It then exhaustively searches for all pairs of ngrams, for all pairs of characters INLINEFORM7 / INLINEFORM8 , for each of the three templates. (This takes about 10 hours on a multicore server.) When two matching embeddings exist, we compute their cosine. For example, for the operation “delete space before M”, an ngram pair from our embeddings that matches is “@Mercedes” / “Mercedes” and we compute its cosine. As the characteristic statistic of an operation we take the average of all cosines; e.g., for “delete space before M” the average cosine is .7435. We then rank operations according to average cosine and take the first INLINEFORM9 as the definition of INLINEFORM10 where INLINEFORM11 is a parameter. For characters that are replaced by each other (e.g., 1, 2, 3 in Table TABREF7 ), we compute the equivalence class and then replace the learned operations with ones that replace a character by the canonical member of its equivalence class (e.g., 2 INLINEFORM12 1, 3 INLINEFORM13 1).
Permutation premise. Tokenization algorithms can be thought of as assigning a particular function or semantics to each character and making tokenization decisions accordingly; e.g., they may disallow that a semicolon, the character “;”, occurs inside a token. If we want to learn representations from the data without imposing such hard constraints, then characters should not have any particular function or semantics. A consequence of this desideratum is that if any two characters are exchanged for each other, this should not affect the representations that are learned. For example, if we interchange space and “A” throughout a corpus, then this should have no effect on learning: what was the representation of “NATO” before, should now be the representation of “N TO”. We can also think of this type of permutation as a sanity check: it ensures we do not inadvertantly make use of text preprocessing heuristics that are pervasive in NLP.
Let INLINEFORM0 be the alphabet of a language, i.e., its set of characters, INLINEFORM1 a permutation on INLINEFORM2 , INLINEFORM3 a corpus and INLINEFORM4 the corpus permuted by INLINEFORM5 . For example, if INLINEFORM6 , then all “a” in INLINEFORM7 are replaced with “e” in INLINEFORM8 . The learning procedure should learn identical equivalence classes on INLINEFORM9 and INLINEFORM10 . So, if INLINEFORM11 after running the learning procedure on INLINEFORM12 , then INLINEFORM13 after running the learning procedure on INLINEFORM14 .
This premise is motivated by our desire to come up with a general method that does not rely on specific properties of a language or genre; e.g., the premise rules out exploiting the fact through feature engineering that in many languages and genres, “c” and “C” are related. Such a relationship has to be learned from the data.
We run experiments on INLINEFORM0 , a 3 gigabyte English Wikipedia corpus, and train word2vec skipgram (W2V, BIBREF1 ) and fastText skipgram (FTX, BIBREF2 ) models on INLINEFORM1 and its derivatives. We randomly generate a permutation INLINEFORM2 on the alphabet and learn a transduction INLINEFORM3 (details below). In Table TABREF8 (left), the columns “method”, INLINEFORM4 and INLINEFORM5 indicate the method used (W2V or FTX) and whether experiments in a row were run on INLINEFORM6 , INLINEFORM7 or INLINEFORM8 . The values of “whitespace” are: (i) ORIGINAL (whitespace as in the original), (ii) SUBSTITUTE (what INLINEFORM9 outputs as whitespace is used as whitespace, i.e., INLINEFORM10 becomes the new whitespace) and (iii) RANDOM (random segmentation with parameters INLINEFORM11 , INLINEFORM12 , INLINEFORM13 ). Before random segmentation, whitespace is replaced with “@” – this character occurs rarely in INLINEFORM14 , so that the effect of conflating two characters (original “@” and whitespace) can be neglected. The random segmenter then indicates boundaries by whitespace – unambiguously since it is applied to text that contains no whitespace.
We learn INLINEFORM0 on the embeddings learned by W2V on the random segmentation version of INLINEFORM1 (C-RANDOM in the table) as described in § SECREF4 for INLINEFORM2 . Since the number of equivalence classes is much smaller than the number of ngrams, INLINEFORM3 reduces the number of distinct character ngrams from 758M in the random segmentation version of INLINEFORM4 (C/D-RANDOM) to 96M in the random segmentation version of INLINEFORM5 (E/F-RANDOM).
Table TABREF7 shows a selection of the INLINEFORM0 operations. Throughout the paper, if we give examples from INLINEFORM1 or INLINEFORM2 as we do here, we convert characters back to the original for better readability. The two uppercase/lowercase conversions shown in the table (E INLINEFORM3 e, C INLINEFORM4 c) were the only ones that were learned (we had hoped for more). The postdeletion rule ml INLINEFORM5 m usefully rewrites “html” as “htm”, but is likely to do more harm than good. We inspected all 200 rules and, with a few exceptions like ml INLINEFORM6 m, they looked good to us.
Evaluation. We evaluate the three models on an entity typing task, similar to BIBREF14 , but based on an entity dataset released by xie16entitydesc2 in which each entity has been assigned one or more types from a set of 50 types. For example, the entity “Harrison Ford” has the types “actor”, “celebrity” and “award winner” among others. We extract mentions from FACC (http://lemurproject.org/clueweb12/FACC1) if an entity has a mention there or we use the Freebase name as the mention otherwise. This gives us a data set of 54,334, 6085 and 6747 mentions in train, dev and test, respectively. Each mention is annotated with the types that its entity has been assigned by xie16entitydesc2. The evaluation has a strong cross-domain aspect because of differences between FACC and Wikipedia, the training corpus for our representations. For example, of the 525 mentions in dev that have a length of at least 5 and do not contain lowercase characters, more than half have 0 or 1 occurrences in the Wikipedia corpus, including many like “JOHNNY CARSON” that are frequent in other case variants.
Since our goal in this experiment is to evaluate tokenization-free learning, not tokenization-free utilization, we use a simple utilization baseline, the bag-of-ngram model (see § SECREF1 ). A mention is represented as the sum of all character ngrams that embeddings were learned for. Linear SVMs BIBREF15 are then trained, one for each of the 50 types, on train and applied to dev and test. Our evaluation measure is micro INLINEFORM0 on all typing decisions; e.g., one typing decision is: “Harrison Ford” is a mention of type “actor”. We tune thresholds on dev to optimize INLINEFORM1 and then use these thresholds on test.
We again use the embeddings corresponding to A-RANDOM in Table TABREF8 . We randomly selected 2,000,000 contexts of size 40 characters from Wikipedia. We then created a noise context for each of the 2,000,000 contexts by replacing one character at position i ( INLINEFORM0 , uniformly sampled) with space (probability INLINEFORM1 ) or a random character otherwise. Finally, we selected 1000 noise contexts randomly and computed their nearest neighbors among the 4,000,000 contexts (excluding the noise query). We did this in two different conditions: for a bag-of-ngram representation of the context (sum of all character ngrams) and for the concatenation of 11 position embeddings, those between 15 and 25. Our evaluation measure is mean reciprocal rank of the clean context corresponding to the noise context. This simulates a text denoising experiment: if the clean context has rank 1, then the noisy context can be corrected.
Table TABREF15 shows that sequence-preserving position embeddings perform better than bag-of-ngram representations.
Table TABREF16 shows an example of a context in which position embeddings did better than bag-of-ngrams, demonstrating that sequence information is lost by bag-of-ngram representations, in this case the exact position of “Seahawks”.
Table TABREF12 gives further intuition about the type of information position embeddings contain, showing the ngram embeddings closest to selected position embeddings; e.g., “estseller” (the first 9-gram on the line numbered 3 in the table) is closest to the embedding of position 3 (corresponding to the first “s” of “best-selling”). The kNN search space is restricted to alphanumeric ngrams.
Results are presented in Table TABREF8 (left). Overall performance of FTX is higher than W2V in all cases. For ORIGINAL, FTX's recall is a lot higher than W2V's whereas precision decreases slightly. This indicates that FTX is stronger in both learning and application: in learning it can generalize better from sparse training data and in application it can produce representations for OOVs and better representations for rare words. For English, prefixes, suffixes and stems are of particular importance, but there often is not a neat correspondence between these traditional linguistic concepts and internal FTX representations; e.g., bojanowski17enriching show that “asphal”, “sphalt” and “phalt” are informative character ngrams of “asphaltic”.
Running W2V on random segmentations can be viewed as an alternative to the learning mechanism of FTX, which is based on character ngram cooccurrence; so it is not surprising that for RANDOM, FTX has only a small advantage over W2V.
For C/D-SUBSTITUTE, we see a dramatic loss in performance if tokenization heuristics are not used. This is not surprising, but shows how powerful tokenization can be.
C/D-ORIGINAL is like C/D-SUBSTITUTE except that we artificially restored the space – so the permutation INLINEFORM0 is applied to all characters except for space. By comparing C/D-ORIGINAL and C/D-SUBSTITUTE, we see that the space is the most important text preprocessing feature employed by W2V and FTX. If space is restored, there is only a small loss of performance compared to A/B-ORIGINAL. So text preprocessing heuristics other than whitespace tokenization in a narrow definition of the term (e.g., downcasing) do not seem to play a big role, at least not for our entity typing task.
For tokenization-free embedding learning on random segmentation, there is almost no difference between original data (A/B-RANDOM) and permuted data (C/D-RANDOM). This confirms that our proposed learning method is insensitive to permutations and makes no use of text preprocessing heuristics.
We achieve an additional improvement by applying the transduction INLINEFORM0 . In fact, FTX performance for F-RANDOM ( INLINEFORM1 of .582) is better than tokenization-based W2V and FTX performance. Thus, our proposed method seems to be an effective tokenization-free alternative to tokenization-based embedding learning.
Table TABREF8 (right) shows nearest neighbors of ten character ngrams, for the A-RANDOM space. Queries were chosen to contain only alphanumeric characters. To highlight the difference to symbol-based representation models, we restricted the search to 9-grams that contained a delimiter at positions 3, 4, 5, 6 or 7.
Lines 1–4 show that “delimiter variation”, i.e., cases where a word has two forms, one with a delimiter, one without a delimiter, is handled well: “Abdulaziz” / “Abdul Azi”, “codenamed” / “code name”, “Quarterfinal” / “Quarter-Final”, “worldrecord” / “world-record”.
Lines 5–9 are cases of ambiguous or polysemous words that are disambiguated through “character context”. “stem”, “cell”, “rear”, “wheel”, “crash”, “land”, “scripts”, “through”, “downtown” all have several meanings. In contrast, the meanings of “stem cell”, “rear wheel”, “crash land”, “(write) scripts for” and “through downtown” are less ambiguous. A multiword recognizer may find the phrases “stem cell” and “crash land” automatically. But the examples of “scripts for” and “through downtown” show that what is accomplished here is not multiword detection, but a more general use of character context for disambiguation.
Line 10 shows that a 9-gram of “face-to-face” is the closest neighbor to a 9-gram of “facilitating”. This demonstrates that form and meaning sometimes interact in surprising ways. Facilitating a meeting is most commonly done face-to-face. It is not inconceivable that form – the shared trigram “fac” or the shared fourgram “faci” in “facilitate” / “facing” – is influencing meaning here in a way that also occurs historically in cases like “ear” `organ of hearing' / “ear” `head of cereal plant', originally unrelated words that many English speakers today intuit as one word.
Single vs. multiple segmentation. The motivation for multiple segmentation is exhaustive coverage of the space of possible segmentations. An alternative approach would be to attempt to find a single optimal segmentation.
Our intuition is that in many cases overlapping segments contain complementary information. Table TABREF17 gives an example. Historic exchange rates are different from floating exchange rates and this is captured by the low similarity of the ngrams ic@exchang and ing@exchan. Also, the meaning of “historic” and “floating” is noncompositional: these two words take on a specialized meaning in the context of exchange rates. The same is true for “rates”: its meaning is not its general meaning in the compound “exchange rates”. Thus, we need a representation that contains overlapping segments, so that “historic” / “floating” and “exchange” can disambiguate each other in the first part of the compound and “exchange” and “rates” can disambiguate each other in the second part of the compound. A single segmentation cannot capture these overlapping ngrams.
What text-type are tokenization-free approaches most promising for? The reviewers thought that language and text-type were badly chosen for this paper. Indeed, a morphologically complex language like Turkish and a noisy text-type like Twitter would seem to be better choices for a paper on robust text representation.
However, robust word representation methods like FTX are effective for within-token generalization, in particular, effective for both complex morphology and OOVs. If linguistic variability and noise only occur on the token level, then a tokenization-free approach has fewer advantages.
On the other hand, the foregoing discussion of cross-token regularities and disambiguation applies to well-edited English text as much as it does to other languages and other text-types as the example of “exchange” shows (which is disambiguated by prior context and provides disambiguating context to following words) and as is also exemplified by lines 5–9 in Table TABREF8 (right).
Still, this paper does not directly evaluate the different contributions that within-token character ngram embeddings vs. cross-token character ngram embeddings make, so this is an open question. One difficulty is that few corpora are available that allow the separate evaluation of whitespace tokenization errors; e.g., OCR corpora generally do not distinguish a separate class of whitespace tokenization errors.
Position embeddings vs. phrase/sentence embeddings. Position embeddings may seem to stand in opposition to phrase/sentence embeddings. For many tasks, we need a fixed length representation of a longer sequence; e.g., sentiment analysis models compute a fixed-length representation to classify a sentence as positive / negative.
To see that position embeddings are compatible with fixed-length embeddings, observe first that, in principle, there is no difference between word embeddings and position embeddings in this respect. Take a sequence that consists of, say, 6 words and 29 characters. The initial representation of the sentence has length 6 for word embeddings and length 29 for position embeddings. In both cases, we need a model that reduces the variable length sequence into a fixed length vector at some intermediate stage and then classifies this vector as positive or negative. For example, both word and position embeddings can be used as the input to an LSTM whose final hidden unit activations are a fixed length vector of this type.
So assessing position embeddings is not a question of variable-length vs. fixed-length representations. Word embeddings give rise to variable-length representations too. The question is solely whether the position-embedding representation is a more effective representation.
A more specific form of this argument concerns architectures that compute fixed-length representations of subsequences on intermediate levels, e.g., CNNs. The difference between position-embedding-based CNNs and word-embedding-based CNNs is that the former have access to a vastly increased range of subsequences, including substrings of words (making it easier to learn that “exchange” and “exchanges” are related) and cross-token character strings (making it easier to learn that “exchange rate” is noncompositional). Here, the questions are: (i) how useful are subsequences made available by position embeddings and (ii) is the increased level of noise and decreased efficiency caused by many useless subsequences worth the information gained by adding useful subsequences.
Independence of training and utilization. We note that our proposed training and utilization methods are completely independent. Position embeddings can be computed from any set of character-ngram-embeddings (including FTX) and our character ngram learning algorithm could be used for applications other than position embeddings, e.g., for computing word embeddings.
Context-free vs. context-sensitive embeddings. Word embeddings are context-free: a given word INLINEFORM0 like “king” is represented by the same embedding independent of the context in which INLINEFORM1 occurs. Position embeddings are context-free as well: if the maximum size of a character ngram is INLINEFORM2 , then the position embedding of the center of a string INLINEFORM3 of length INLINEFORM4 is the same independent of the context in which INLINEFORM5 occurs.
It is conceivable that text representations could be context-sensitive. For example, the hidden states of a character language model have been used as a kind of nonsymbolic text representation BIBREF16 , BIBREF17 , BIBREF18 and these states are context-sensitive. However, such models will in general be a second level of representation; e.g., the hidden states of a character language model generally use character embeddings as the first level of representation. Conversely, position embeddings can also be the basis for a context-sensitive second-level text representation. We have to start somewhere when we represent text. Position embeddings are motivated by the desire to provide a representation that can be computed easily and quickly (i.e., without taking context into account), but that on the other hand is much richer than the symbolic alphabet.
Processing text vs. speech vs. images. gillick16 write: “It is worth noting that noise is often added ... to images ... and speech where the added noise does not fundamentally alter the input, but rather blurs it. [bytes allow us to achieve] something like blurring with text.” It is not clear to what extent blurring on the byte level is useful; e.g., if we blur the bytes of the word “university” individually, then it is unlikely that the noise generated is helpful in, say, providing good training examples in parts of the space that would otherwise be unexplored. In contrast, the text representation we have introduced in this paper can be blurred in a way that is analogous to images and speech. Each embedding of a position is a vector that can be smoothly changed in every direction. We have showed that the similarity in this space gives rise to natural variation.
Prospects for completely tokenization-free processing. We have focused on whitespace tokenization and proposed a whitespace-tokenization-free method that computes embeddings of higher quality than tokenization-based methods. However, there are many properties of edited text beyond whitespace tokenization that a complex rule-based tokenizer exploits. In a small explorative experiment, we replaced all non-alphanumeric characters with whitespace and repeated experiment A-ORIGINAL for this setting. This results in an INLINEFORM0 of .593, better by .01 than the best tokenization-free method. This illustrates that there is still a lot of work to be done before we can obviate the need for tokenization.
In the following, we will present an overview of work on character-based models for a variety of tasks from different NLP areas.
The history of character-based research in NLP is long and spans a broad array of tasks. Here we make an attempt to categorize the literature of character-level work into three classes based on the way they incorporate character-level information into their computational models. The three classes we identified are: tokenization-based models, bag-of-n-gram models and end-to-end models. However, there are also mixtures possible, such as tokenization-based bag-of-n-gram models or bag-of-n-gram models trained end-to-end.
On top of the categorization based on the underlying representation model, we sub-categorize the work within each group into six abstract types of NLP tasks (if possible) to be able to compare them more directly. These task types are the following:
We group character-level models that are based on tokenization as a necessary preprocessing step in the category of tokenization-based approaches. Those can be either models with tokenized text as input or models that operate only on individual tokens (such as studies on morphological inflection of words).
In the following paragraphs, we cover a subset of tokenization-based models that are used for representation learning, sequence-to-sequence generation, sequence labeling, language modeling, and sequence classification tasks.
Representation learning for character sequences. Creating word representations based on characters has attracted much attention recently. Such representations can model rare words, complex words, out-of-vocabulary words and noisy texts. In comparison to traditional word representation models that learn separate vectors for word types, character-level models are more compact as they only need vector representations for characters as well as a compositional model.
Various neural network architectures have been proposed for learning token representations based on characters. Examples of such architectures are averaging character embeddings, (bidirectional) recurrent neural networks (RNNs) (with or without gates) over character embeddings and convolutional neural networks (CNNs) over character embeddings. Studies on the general task of learning word representations from characters include BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 . These character-based word representations are often combined with word embeddings and integrated into a hierarchical system, such as hierarchical RNNs or CNNs or combinations of both to solve other task types. We will provide more concrete examples in the following paragraphs.
Sequence-to-sequence generation (machine translation). Character-based machine translation is no new topic. Using character-based methods has been a natural way to overcome challenges like rare words or out-of-vocabulary words in machine translation. Traditional machine translation models based on characters or character n-grams have been investigated by BIBREF34 , BIBREF35 , BIBREF36 . Neural machine translation with character-level and subword units has become popular recently BIBREF37 , BIBREF38 , BIBREF39 , BIBREF33 . In such neural models, using a joint attention/translation model makes joint learning of alignment and translation possible BIBREF31 .
Both hierarchical RNNs BIBREF31 , BIBREF38 and combinations of CNNs and RNNs have been proposed for neural machine translation BIBREF37 , BIBREF33 .
Sequence labeling. Examples of early efforts on sequence labeling using tokenization-based models include: bilingual character-level alignment extraction BIBREF40 ; unsupervised multilingual part-of-speech induction based on characters BIBREF41 ; part-of-speech tagging with subword/character-level information BIBREF42 , BIBREF43 , BIBREF44 ; morphological segmentation and tagging BIBREF45 , BIBREF46 ; and identification of language inclusion with character-based features BIBREF47 .
Recently, various hierarchical character-level neural networks have been applied to a variety of sequence labeling tasks.
Recurrent neural networks are used for part-of-speech tagging BIBREF48 , BIBREF49 , BIBREF50 , named entity recognition BIBREF51 , BIBREF50 , chunking BIBREF50 and morphological segmentation/inflection generation BIBREF52 , BIBREF53 , BIBREF54 , BIBREF55 , BIBREF56 , BIBREF57 , BIBREF58 , BIBREF59 . Such hierarchical RNNs are also used for dependency parsing BIBREF60 . This work has shown that morphologically rich languages benefit from character-level models in dependency parsing.
Convolutional neural networks are used for part-of-speech tagging BIBREF61 and named entity recognition BIBREF62 .
The combination of RNNs and CNNs is used, for instance, for named entity recognition.
Language modeling. Earlier work on sub-word language modeling has used morpheme-level features for language models BIBREF63 , BIBREF64 , BIBREF65 , BIBREF66 , BIBREF67 . In addition, hybrid word/n-gram language models for out-of-vocabulary words have been applied to speech recognition BIBREF68 , BIBREF69 , BIBREF70 , BIBREF71 . Furthermore, characters and character n-grams have been used as input to restricted boltzmann machine-based language models for machine translation BIBREF72 .
More recently, character-level neural language modeling has been proposed by a large body of work BIBREF73 , BIBREF74 , BIBREF75 , BIBREF48 , BIBREF76 , BIBREF66 , BIBREF72 . Although most of this work is using RNNs, there exist architectures that combine CNNs and RNNs BIBREF75 . While most of these studies combine the output of the character model with word embeddings, the authors of BIBREF75 report that this does not help them for their character-aware neural language model. They use convolution over character embeddings followed by a highway network BIBREF77 and feed its output into a long short-term memory network that predicts the next word using a softmax function.
Sequence classification. Examples of tokenization-based models that perform sequence classification are CNNs used for sentiment classification BIBREF78 and combinations of RNNs and CNNs used for language identification BIBREF79 .
Character n-grams have a long history as features for specific NLP applications, such as information retrieval. However, there is also work on representing words or larger input units, such as phrases, with character n-gram embeddings. Those embeddings can be within-token or cross-token, i.e., there is no tokenization necessary.
Although such models learn/use character n-gram embeddings from tokenized text or short text segments, to represent a piece of text, the occurring character n-grams are usually summed without the need for tokenization. For example, the phrase “Berlin is located in Germany” is represented with character 4-grams as follows: “Berl erli rlin lin_ in_i n_is _is_ is_l s_lo _loc loca ocat cate ated ted_ ed_i d_in _in_ in_G n_Ge _Ger Germ erma rman many any.” Note that the input has not been tokenized and there are n-grams spanning token boundaries. We also include non-embedding approaches using bag-of-n-grams within this group as they go beyond word and token representations.
In the following, we explore a subset of bag-of-ngram models that are used for representation learning, information retrieval, and sequence classification tasks.
Representation learning for character sequences. An early study in this category of character-based models is BIBREF3 . Its goal is to create corpus-based fixed-length distributed semantic representations for text. To train k-gram embeddings, the top character k-grams are extracted from a corpus along with their cooccurrence counts. Then, singular value decomposition (SVD) is used to create low dimensional k-gram embeddings given their cooccurrence matrix. To apply them to a piece of text, the k-grams of the text are extracted and their corresponding embeddings are summed. The study evaluates the k-gram embeddings in the context of word sense disambiguation.
A more recent study BIBREF4 trains character n-gram embeddings in an end-to-end fashion with a neural network. They are evaluated on word similarity, sentence similarity and part-of-speech tagging.
Training character n-gram embeddings has also been proposed for biological sequences BIBREF12 , BIBREF13 for a variety of bioinformatics tasks.
Information retrieval. As mentioned before, character n-gram features are widely used in the area of information retrieval BIBREF80 , BIBREF81 , BIBREF82 , BIBREF83 , BIBREF84 , BIBREF85 .
Sequence classification. Bag-of-n-gram models are used for language identification BIBREF86 , BIBREF87 , topic labeling BIBREF88 , authorship attribution BIBREF89 , word/text similarity BIBREF2 , BIBREF90 , BIBREF4 and word sense disambiguation BIBREF3 .
Similar to bag-of-n-gram models, end-to-end models are tokenization-free. Their input is a sequence of characters or bytes and they are directly optimized on a (task-specific) objective. Thus, they learn their own, task-specific representation of the input sequences. Recently, character-based end-to-end models have gained a lot of popularity due to the success of neural networks.
We explore the subset of these models that are used for sequence generation, sequence labeling, language modeling and sequence classification tasks.
Sequence-to-sequence generation. In 2011, the authors of BIBREF91 already proposed an end-to-end model for generating text. They train RNNs with multiplicative connections on the task of character-level language modeling. Afterwards, they use the model to generate text and find that the model captures linguistic structure and a large vocabulary. It produces only a few uncapitalized non-words and is able to balance parantheses and quotes even over long distances (e.g., 30 characters). A similar study by BIBREF92 uses a long short-term memory network to create character sequences.
Recently, character-based neural network sequence-to-sequence models have been applied to instances of generation tasks like machine translation BIBREF93 , BIBREF94 , BIBREF95 , BIBREF96 , BIBREF97 (which was previously proposed on the token-level BIBREF98 ), question answering BIBREF99 and speech recognition BIBREF100 , BIBREF101 , BIBREF102 , BIBREF103 .
Sequence labeling. Character and character n-gram-based features were already proposed in 2003 for named entity recognition in an end-to-end manner using a hidden markov model BIBREF104 . More recently, the authors of BIBREF105 have proposed an end-to-end neural network based model for named entity recognition and part-of-speech tagging. An end-to-end model is also suggested for unsupervised, language-independent identification of phrases or words BIBREF106 .
A prominent recent example of neural end-to-end sequence labeling is the paper by BIBREF107 about multilingual language processing from bytes. A window is slid over the input sequence, which is represented by its byte string. Thus, the segments in the window can begin and end mid-word or even mid-character. The authors apply the same model for different languages and evaluate it on part-of-speech tagging and named entity recognition.
Language modeling. The authors of BIBREF108 propose a hierarchical multiscale recurrent neural network for language modeling. The model uses different timescales to encode temporal dependencies and is able to discover hierarchical structures in a character sequence without explicit tokenization. Other studies on end-to-end language models include BIBREF94 , BIBREF109 .
Sequence classification. Another recent end-to-end model uses character-level inputs for document classification BIBREF110 , BIBREF111 , BIBREF112 . To capture long-term dependencies of the input, the authors combine convolutional layers with recurrent layers. The model is evaluated on sentiment analysis, ontology classification, question type classification and news categorization.
End-to-end models are also used for entity typing based on the character sequence of the entity's name BIBREF113 .
We introduced the first generic text representation model that is completely nonsymbolic, i.e., it does not require the availability of a segmentation or tokenization method that identifies words or other symbolic units in text. This is true for the training of the model as well as for applying it when computing the representation of a new text. In contrast to prior work that has assumed that the sequence-of-character information captured by character ngrams is sufficient, position embeddings also capture sequence-of-ngram information. We showed that our model performs better than prior work on entity typing and text denoising.
Future work.
The most important challenge that we need to address is how to use nonsymbolic text representation for tasks that are word-based like part-of-speech tagging. This may seem like a contradiction at first, but gillick16 have shown how character-based methods can be used for “symbolic” tasks. We are currently working on creating an analogous evaluation for our nonsymbolic text representation.
This work was supported by DFG (SCHUE 2246/10-1) and Volkswagenstiftung. We are grateful for their comments to: the anonymous reviewers, Ehsan Asgari, Annemarie Friedrich, Helmut Schmid, Martin Schmitt and Yadollah Yaghoobzadeh.
Nonsymbolic representation learning does not preprocess the training corpus by means of tokenization and considers many ngrams that would be ignored in tokenized approaches because they span token boundaries. As a result, the number of ngrams that occur in a corpus is an order of magnitude larger for tokenization-free approaches than for tokenization-based approaches. See Figure FIGREF33 .
W2V hyperparameter settings. size of word vectors: 200, max skip length between words: 5, threshold for occurrence of words: 0, hierarchical softmax: 0, number of negative examples: 5, threads: 50, training iterations: 1, min-count: 5, starting learning rate: .025, classes: 0
FTX hyperparameter settings. learning rate: .05, lrUpdateRate: 100, size of word vectors: 200, size of context window: 5, number of epochs: 1, minimal number of word occurrences: 5, number of negatives sampled: 5, max length of word ngram: 1, loss function: ns, number of buckets: 2,000,000, min length of char ngram: 3, max length of char ngram: 6, number of threads: 50, sampling threshold: .0001
We ran some experiments with more epochs, but this did not improve the results.
We did not tune INLINEFORM0 , but results are highly sensitive to the value of this parameter. If INLINEFORM1 is too small, then beneficial conflations (collapse punctuation marks, replace all digits with one symbol) are not found. If INLINEFORM2 is too large, then precision suffers – in the extreme case all characters are collapsed into one.
We also did not tune INLINEFORM0 , but we do not consider results to be very sensitive to the value of INLINEFORM1 if it is reasonably large. Of course, if a larger range of character ngram lengths is chosen, i.e., a larger interval INLINEFORM2 , then at some point INLINEFORM3 will not be sufficient and possible segmentations would not be covered well enough in sampling.
The type of segmentation used in multiple segmentation can also be viewed as a hyperparameter. An alternative to random segmentation would be exhaustive segementation, but a naive implementation of that strategy would increase the size of the training corpus by several orders of magnitude. Another alternative is to choose one fixed size, e.g., 4 or 5 (similar to BIBREF3 ). Many of the nice disambiguation effects we see in Table TABREF8 (right) and in Table TABREF17 would not be possible with short ngrams. On the other hand, a fixed ngram size that is larger, e.g., 10, would make it difficult to get 100% coverage: there would be positions for which no position embedding can be computed.

Figure and table captions:
Table 1: String operations that on average do not change meaning. “@” stands for space. ‡ is the left or right boundary of the ngram.
Table 2: Left: Evaluation results for named entity typing. Right: Neighbors of character ngrams. Rank r = 1/r = 2: nearest / second-nearest neighbor.
Table 3: Nearest ngram embeddings (rank r ∈ [1, 5]) of the position embeddings for “POS”, the positions 2/3 (best), 15/16 (monthly), 23/24 (comic), 29/30 (book) and 34/35 (publications) in the Wikipedia excerpt “best-selling monthly comic book publications sold in North America”
Table 5: Illustration of the result in Table 4. “rep. space” = “representation space”. We want to correct the error in the corrupted “noise” context (line 2) and produce “correct” (line 1). The nearest neighbor to line 2 in position-embedding space is the correct context (line 3, r = 1). The nearest neighbor to line 2 in bag-of-ngram space is incorrect (line 4, r = 1) because the precise position of “Seahawks” in the query is not encoded. The correct context in bag-of-ngram space is instead at rank r = 6 (line 5). “similarity” is average cosine (over eleven position embeddings) for position embeddings.
Table 6: Cosine similarity of ngrams that cross word boundaries and disambiguate polysemous words. The tables show three disambiguating ngrams for “exchange” and “rates” that have different meanings as indicated by low cosine similarity. In phrases like “floating exchange rates” and “historic exchange rates”, disambiguating ngrams overlap. Parts of the word “exchange” are disambiguated by preceding context (ic@exchang, ing@exchan) and parts of “exchange” provide context for disambiguating “rates” (xchange@ra).
Figure 1: The graph shows how many different character ngrams (kmin = 3, kmax = 10) occur in the first n bytes of the English Wikipedia for symbolic (tokenization-based) vs. nonsymbolic (tokenization-free) processing. The number of ngrams is an order of magnitude larger in the nonsymbolic approach. We counted all segments, corresponding to m = ∞. For the experiments in the paper (m = 50), the number of nonsymbolic character ngrams is smaller.

Questions about this paper:
By how much do they outpeform existing text denoising models?

Answers:
Answer with content missing: (Table 4) Mean reciprocal rank of proposed model is 0.76 compared to 0.64 of bag-of-ngrams.