Besides Word Embedding, why you need to know Character Embedding? | by Edward Ma | Towards Data Science

In 2013, Tomas Mikolov [1] introduced word embedding to learn a better quality of word. At that time word embedding is state of the art on dealing with text. Later on, doc2vec is introduced as well. What if we think in another angel? Instead of aggregate from word to document, is it possible to aggregate from character to word.
In this article, you will go through what, why, when and how on Character Embedding.
Xiang and Yann [2] introduced character CNN. They found that character includes key signal to improve model performance. In the paper, a list of character are defined 70 characters which including 26 English letters, 10 digits, 33 special characters and new line character.
# Copy from Char CNN paperabcdefghijklmnopqrstuvwxyz0123456789 -,;.!?:’’’/\|_@#$%ˆ&* ̃‘+-=()[]{}
On the other hand, Google Brain team introduced Exploring the Limits of Language Modeling and released the lm_1b model which includes 256 vectors (including 52 characters, special characters) and the dimension is just 16. By comparing to word embedding, the dimension can increase up to 300 while the number of vectors is huge.
In english, all words are formed by 26 (or 52 if including both upper and lower case character, or even more if including special characters). Having the character embedding, every single word’s vector can be formed even it is out-of-vocabulary words (optional). On the other hand, word embedding can only handle those seen words.
Another benefit is that it good fits for misspelling words, emoticons, new words (e.g. In 2018, Oxford English Dictionary introduced new word which is boba tea 波霸奶茶. Before that we do not have any pre-trained word embedding for that).
It handles infrequent words better than word2vec embedding as later one suffers from lack of enough training opportunity for those rare words.
Third reason is that as there are only small amount of vector, it reduces model complexity and improving the performance (in terms of speed)
In NLP, we can apply character embedding on:
Text Classification
Language Model
Named Entity Recognition
Character Embedding
Define a list a characters (i.e. m). For example, can use alphanumeric and some special characters. For my example, English characters (52), number (10), special characters (20) and one unknown character, UNK. Total 83 characters.Transfer characters as 1-hot encoding and got a sequence for vectors. For unknown characters and blank characters, use all-zero vector to replace it. If exceeding pre-defined maximum length of characters (i.e. l), ignoring it. The output is 16 dimensions vector per every single character.Using 3 1D CNN layers (configurable) to learn the sequence
Define a list a characters (i.e. m). For example, can use alphanumeric and some special characters. For my example, English characters (52), number (10), special characters (20) and one unknown character, UNK. Total 83 characters.
Transfer characters as 1-hot encoding and got a sequence for vectors. For unknown characters and blank characters, use all-zero vector to replace it. If exceeding pre-defined maximum length of characters (i.e. l), ignoring it. The output is 16 dimensions vector per every single character.
Using 3 1D CNN layers (configurable) to learn the sequence
Sentence Embedding
Bi-directional LSTM followed CNN layersSome dropout layers are added after LSTM.
Bi-directional LSTM followed CNN layers
Some dropout layers are added after LSTM.
This sample code will demonstrate how to use Character Embedding to do classification. Testing dataset can be found in the UCI ML Repository.
Preprocess
First of all, we need to prepare meta information including character dictionary and converting label from text to numeric (as keras support numeric input only).
preporcess(labels=df['category'].unique())
Output
-----> Stage: preprocessTotoal number of chars: 83First 3 char_indices sample: {';': 0, '"': 1, 'A': 2}First 3 indices_char sample: {0: ';', 1: '"', 2: 'A'}Label to Index:  {'b': 0, 'e': 2, 'm': 3, 't': 1}Index to Label:  {0: 'b', 1: 't', 2: 'e', 3: 'm'}
Process
We have to transform raw input training data and testing to numpy format for keras input
x_train, y_train = char_cnn.process(    df=train_df, x_col='headline', y_col='category')x_test, y_test = char_cnn.process(    df=test_df, x_col='headline', y_col='category')
Output
-----> Stage: processNumber of news: 100Actual max sentence: 3Train Shape:  (100, 5, 256) (100,)-----> Stage: processNumber of news: 10Actual max sentence: 1Train Shape:  (10, 5, 256) (10,)
Build Model
Leveraging CNN layer and bidirectional LSTM to build a Character CNN text classifier. First of all, using 1 dimension CNN to build character block
def _build_character_block(    self, block, dropout=0.3,     filters=[64, 100], kernel_size=[3, 3],     pool_size=[2, 2], padding='valid', activation='relu',     kernel_initializer='glorot_normal'):            for i in range(len(filters)):        block = Conv1D(            filters=filters[i], kernel_size=kernel_size[i],            padding=padding, activation=activation,            kernel_initializer=kernel_initializer)(block)        block = Dropout(dropout)(block)        block = MaxPooling1D(pool_size=pool_size[i])(block)    block = GlobalMaxPool1D()(block)    block = Dense(128, activation='relu')(block)    return block
Sentence level block will be built by using 3 character blocks
def _build_sentence_block(    self, max_len_of_sentence, max_num_of_setnence,     filters=[[100, 200, 200], [200, 300, 300], [300, 400, 400]],     kernel_sizes=[[4, 3, 3], [5, 3, 3], [6, 3, 3]],     pool_sizes=[[2, 2, 2], [2, 2, 2], [2, 2, 2]],    dropout=0.4):            sent_input = Input(shape=(max_len_of_sentence, ), dtype='int64')    embedded = Embedding(self.num_of_char, char_dimension,        input_length=max_len_of_sentence)(sent_input)            blocks = []    for i, filter_layers in enumerate(filters):        blocks.append(            self._build_character_block(                    block=embedded, filters=filter_layers,                    kernel_size=kernel_sizes[i],                    pool_size=pool_sizes[i])            )    sent_output = concatenate(blocks, axis=-1)    sent_output = Dropout(dropout)(sent_output)    sent_encoder = Model(inputs=sent_input, outputs=sent_output)    return sent_encoder
Finally, we will build document block according to sentences
def _build_document_block(    self, sent_encoder, max_len_of_sentence, max_num_of_setnence,     num_of_label, dropout=0.3,     loss='sparse_categorical_crossentropy', optimizer='rmsprop',    metrics=['accuracy']):        doc_input = Input(shape=(        max_num_of_setnence, max_len_of_sentence), dtype='int64')    doc_output = TimeDistributed(sent_encoder)(doc_input)    doc_output = Bidirectional(LSTM(        128, return_sequences=False,         dropout=dropout))(doc_output)    doc_output = Dropout(dropout)(doc_output)    doc_output = Dense(128, activation='relu')(doc_output)    doc_output = Dropout(dropout)(doc_output)    doc_output = Dense(        num_of_label, activation='sigmoid')(doc_output)    doc_encoder = Model(inputs=doc_input, outputs=doc_output)    doc_encoder.compile(        loss=loss, optimizer=optimizer, metrics=metrics)    return doc_encoder
Output
Training
train(x_train, y_train, x_test, y_test, epochs=10)
Output
You can found out all code from github.
Character Embedding is a brilliant design for solving lots of text classification. It resolved some word embedding. Difference between Character Embedding and Word Embedding is that Character Embedding can build any word as long as those character are included.
Facebook AI Research (FAIR) did a further step. Team use subword to train the model. Taking “Facebook” as an example, they use “F”, “Fa”, “Fac” etc to train the word. For detail of fasttext, you may visit their website for better understanding.
This is some comment on Character Embedding as it does not include any word meaning but just using characters. We may include both Character Embedding and Word Embedding together to solve our NLP problem.
A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval
Learning Character-level Representations for Part-of-Speech Tagging
Character-level Convolutional Networks for Text Classification
Character Level Deep Learning in Sentiment Analysis
I am Data Scientist in Bay Area. Focusing on state-of-the-art in Data Science, Artificial Intelligence , especially in NLP and platform related. You can reach me from Medium Blog, LinkedIn or Github.
[1] Tomas Mikolov, Greg Corrado, Kai Chen & Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. September 2013. https://arxiv.org/pdf/1301.3781.pdf
[2] Xiang Zhang, Yann LeCun. Text Understanding from Scratch. 2016. https://arxiv.org/pdf/1502.01710v5.pdf
[3] Kim Y., Jernite Y., Sontag D., Rush A. M.. Character-Aware Neural Language Models. 2015. https://arxiv.org/pdf/1508.06615.pdf