Learning Character-level Compositionality with Visual Features
Frederick Liu1 , Han Lu1 , Chieh Lo2 , Graham Neubig1
1

Language Technology InstituteElectrical and Computer Engineering
Carnegie Mellon University, Pittsburgh, PA 15213{fliu1,hlu2,gneubig}@cs.cmu.edu
chiehl@andrew.cmu.edu
2

Abstract

(a)

jing

(c)

 

Previous work has modeled the compositionality of words by creating characterlevel models of meaning, reducing problems of sparsity for rare words. However,in many writing systems compositionality
has an effect even on the character-level:
the meaning of a character is derived by
the sum of its parts. In this paper, we
model this effect by creating embeddings
for characters based on their visual characteristics, creating an image for the character and running it through a convolutional
neural network to produce a visual character embedding. Experiments on a textclassification task demonstrate that suchmodel allows for better processing of instances with rare characters in languages
such as Chinese, Japanese, and Korean.
Additionally, qualitative analyses demonstrate that our proposed model learns tofocus on the parts of characters that carrysemantic content, resulting in embeddingsthat are coherent in visual space.

1

gui

(b)

Salmon Whale
gui

gui

han'da ham''ni'''da 

(d)


 
Salmon LaurelDo Do'(polite)

a
Kalb Kälber
Calf

CalvesFigure 1: Examples of character-level compositionality in (a, b) Chinese, (c) Korean, and (d) German. The red part of the characters are shared, andaffects the pronunciation (top) or meaning (bottom).

Introduction

Compositionality—the fact that the meaning of acomplex expression is determined by its structure
and the meanings of its constituents—is a hallmark of every natural language (Frege and Austin,
1980; Szabó, 2010). Recently, neural models haveprovided a powerful tool for learning how to compose words together into a meaning representationof whole sentences for many downstream tasks.
This is done using models of various levels of
sophistication, from simpler bag-of-words (Iyyer
et al., 2015) and linear recurrent neural network(RNN) models (Sutskever et al., 2014; Kiros et al.,2015), to more sophisticated models using tree-

structured (Socher et al., 2013) or convolutional
networks (Kalchbrenner et al., 2014).
In fact, a growing body of evidence shows that it
is essential to look below the word-level and consider compositionality within words themselves.
For example, several works have proposed models that represent words by composing together
the characters into a representation of the word itself (Ling et al., 2015; Zhang et al., 2015; Dhingra
et al., 2016). Additionally, for languages with productive word formation (such as agglutination andcompounding), models calculating morphologysensitive word representations have been found effective (Luong et al., 2013; Botha and Blunsom,2014). These models help to learn more robust
representations for rare words by exploiting morphological patterns, as opposed to models that operate purely on the lexical level as the atomic units.
For many languages, compositionality stops at
the character-level: characters are atomic units ofmeaning or pronunciation in the language, and nofurther decomposition can be done.1 However, forother languages, character-level compositionality,
where a character’s meaning or pronunciation can
1

In English, for example, this is largely the case.

2059
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 2059–2068
Vancouver, Canada, July 30 - August 4, 2017. c 2017 Association for Computational Linguistics
https://doi.org/10.18653/v1/P17-1188

Lang
ChineseJapanese
KoreanLang
ChineseJapanese
KoreanGeography
32.4k
18.6k
6k
Medical
30.3k
66.5k
16.1k

Sports
49.8k
82.7k
580
Education
66.2k
86.7k
4.71k

Arts
50.4k
84.1k
5.74k
Food
554
20.2k
33

Military
3.6k
81.6k
840
Religion
66.9k
98.1k
2.60k

Economics
82.5k
80.9k
5.78k
Agriculture
89.5k
97.4k
1.51k

Transportation
40.4k
91.8k
1.68k
Electronics
80.5k
1.08k
1.03k

Table 1: By-category statistics for the Wikipedia dataset. Note that Food is the abbreviation for “Foodand Culture” and Religion is the abbreviation for “Religion and Belief”.
be derived from the sum of its parts, is very mucha reality. Perhaps the most compelling example
of compositionality of sub-character units can befound in logographic writing systems such as the
Han and Kanji characters used in Chinese andJapanese, respectively.2 As shown on the left sideof Fig. 1, each part of a Chinese character (called
a “radical”) potentially contributes to the meaning(i.e., Fig. 1(a)) or pronunciation (i.e., Fig. 1(b))
of the overall character. This is similar to how
English characters combine into the meaning or
pronunciation of an English word. Even in languages with phonemic orthographies, where eachcharacter corresponds to a pronunciation insteadof a meaning, there are cases where composition
occurs. Fig. 1(c) and (d) show the examples of Korean and German, respectively, where morphological inflection can cause single characters to makechanges where some but not all of the component
parts are shared.
In this paper, we investigate the feasibility ofmodeling the compositionality of characters in away similar to how humans do: by visually observing the character and using the features of its
shape to learn a representation encoding its meaning. Our method is relatively simple, and generalizable to a wide variety of languages: we firsttransform each character from its Unicode representation to a rendering of its shape as an image,then calculate a representation of the image using Convolutional Neural Networks (CNNs) (Cunet al., 1990). These features then serve as inputs
to a down-stream processing task and trained in
an end-to-end manner, which first calculates a loss
function, then back-propagates the loss back to the
CNN.
2

Other prominent examples are largely for extinct languages: Egyptian hieroglyphics, Mayan glyphs, and Sumerian cuneiform scripts (Daniels and Bright, 1996).

As demonstrated by our motivating examples
in Fig. 1, in logographic languages character-level
semantic or phonetic similarity is often indicatedby visual cues; we conjecture that CNNs canappropriately model these visual patterns. Consequently, characters with similar visual appearances will be biased to have similar embeddings,
allowing our model to handle rare characters effectively, just as character-level models have beeneffective for rare words.
To evaluate our model’s ability to learn representations, particularly for rare characters, we perform experiments on a downstream task of classifying Wikipedia titles for three Asian languages:
Chinese, Japanese, and Korean. We show thatour proposed framework outperforms a baselinemodel that uses standard character embeddings for
instances containing rare characters. A qualitative analysis of the characteristics of the learned
embeddings of our model demonstrates that visually similar characters share similar embeddings.
We also show that the learned representations are
particularly effective under low-resource scenarios and complementary with standard character
embeddings; combining the two representations
through three different fusion methods (Snoek
et al., 2005; Karpathy et al., 2014) leads to consistent improvements over the strongest baseline
without visual features.

2

DatasetBefore delving into the details of our model, we
first describe a dataset we constructed to examine the ability of our model to capture the compositional characteristics of characters. Specifically, the dataset must satisfy two desiderata: (1)it must be necessary to fully utilize each character in the input in order to achieve high accuracy,and (2) there must be enough regularity and com-

2060

106

⎯⎯ Chinese⎯⎯"Japanese⎯⎯ KoreanFrequency

105
104

36
36 

Visual model(Image as input)

CNN CNN CNN
GRUSoftmax103102 Rank < 20%101 Freq. > 80%

Lookup model(Symbol as input)

Geography
Sports
Arts
Military
Economics
Transportation
Health Science
Education
Food Culture
Religion and Belief
Agriculture
Electronics

100 0
10

101

102
Rank

103

104

Figure 2: The character rank-frequency distribution of the corpora we considered in this paper. All
three languages have a long-tail distribution.
positionality in the characters of the language. To
satisfy these desiderata, we create a text classification dataset where the input is a Wikipedia article title in Chinese, Japanese, or Korean, andthe output is the category to which the article belongs.3 This satisfies (1), because Wikipedia titles
are short and thus each character in the title will
be important to our decision about its category. It
also satisfies (2), because Chinese, Japanese, andKorean have writing systems with large numbersof characters that decompose regularly as shownin Fig. 1. While this task in itself is novel, it issimilar to previous work in named entity type inference using Wikipedia (Toral and Munoz, 2006;
Kazama and Torisawa, 2007; Ratinov and Roth,2009), which has proven useful for downstream
named entity recognition systems.
2.1

Dataset Collection

As the labels we would like to predict, we use
12 different main categories from the Wikipedia
web page: Geography, Sports, Arts, Military, Economics, Transportation, Health Science, Education, Food Culture, Religion and Belief, Agriculture and Electronics. Wikipedia has a hierarchicalstructure, where each of these main categories has
a number of subcategories, and each subcategory
has its own subcategories, etc. We traverse thishierarchical structure, adding each main category
tag to all of its descendants in this subcategory tree
structure. In the case that a particular article is thedescendant of multiple main categories, we favor
the main category that minimizes the depth of the3 

The link to the dataset and the crawling scripts–
https://github.com/frederick0329/
Wikipedia_title_dataset



Figure 3: An illustration of two models, our proposed V ISUAL model at the top and the baseline L OOKUP model at the bottom using the same
RNN architecture. A string of characters (e.g. “温病学”), each converted into a 36x36 image, serves
as input of our V ISUAL model. dc is the dimension of the character embedding for the L OOKUP
model.
article in the tree (e.g., if an article is two steps
away from Sports and three steps away from Arts,
it will receive the “Sports” label). We also perform some rudimentary filtering, removing pages
that match the regular expression “.*:.*”, whichcatches special pages such as “title:agriculture”.
2.2

Statistics

For Chinese, Japanese, and Korean, respectively,
the number of articles is 593k/810k/46.6k, and theaverage length and standard deviation of the title is 6.25±3.96/8.60±5.58/6.10±3.71. As shownin Fig. 2, the character rank-frequency distributions of all three languages follows the 80/20
rule (Newman, 2005) (i.e., top 20% ranked characters that appear more than 80% of total frequencies), demonstrating that the characters in theselanguages belong to a long tail distribution.
We further split the dataset into training, validation, and testing sets with a 6:2:2 ratio. The category distribution for each language can be seenin Tab. 1. Chinese has two varieties of characters,
traditional and simplified, and the dataset is a mixof the two. Hence, we transform this dataset into
two separate sets, one completely simplified andthe other completely traditional using the Chinese
text converter provided with Mac OS.

3

Model

Our overall model for the classification task follows the encoder model by Sutskever et al. (2014).

2061

Layer#
1
2
3
4
5
6
7
8
9
10
11
12

3-layer CNN Configuration
Spatial Convolution (3, 3) → 32
ReLu
MaxPool (2, 2)
Spatial Convolution (3, 3) → 32
ReLu
MaxPool (2, 2)
Spatial Convolution (3, 3) → 32
ReLu
Linear (800, 128)
ReLu
Linear (128, 128)
ReLu

Table 2: Architecture of the CNN used in the experiments. All the convolutional layers have 32
3×3 filters.
We calculate character representations, use a RNN
to combine the character representations into a
sentence representation, and then add a softmax
layer after that to predict the probability for eachclass. As shown in Fig. 2.1, the baseline model,
which we call it the L OOKUP model, calculates
the representation for each character by looking it
up in a character embedding matrix. Our proposedmodel, the V ISUAL model instead learns the representation of each character from its visual appearance via CNN.

loss. Because we are training embeddings basedon this classification loss, we expect that the CNN
will focus on parts of the image that contain semantic information useful for category classification, a hypothesis that we examine in the experiments (see Section 5.5).
In more detail, the specific structure of the CNN
that we utilize consists of three convolution layers
where each convolution layer is followed by themax pooling and ReLU nonlinear activation layers. The configurations of each layer are listed in
Tab. 2. The output vector for the image embeddings also has size dc which is the same as the
L OOKUP model.
Encoder and Classifier For both the
L OOKUP and the V ISUAL models, we adopt
an RNN encoder using Gated Recurrent Units(GRUs) (Chung et al., 2014). Each of the GRU
units processes the character embeddings sequentially. At the end of the sequence, the incremental
GRU computation results in a hidden state eembedding the sentence. The encoded sentence
embedding is passed through a linear layer whose
output is the same size as the number of classes.
We use a softmax layer to compute the posterior
class probabilities:
exp(wjT e + bj )P (y = j|e) = PLTi=1 exp(wi e + bi )(1)

L OOKUP model Given a character vocabulary
C, for the L OOKUP model as in the bottom part of
Fig. 2.1, the input to the network is a stream ofcharacters c1 , c2 , ...cN , where cn ∈ C. Each character is represented by a 1-of-|C| (one-hot) encoding. This one-hot vector is then multiplied by
the lookup matrix TC ∈ R|C|×dc , where dc is thedimension of the character embedding. The randomly initialized character embeddings were optimized with classification loss.

To train the model, we use cross-entropy loss
between predicted and true targets:

V ISUAL model The proposed method aims tolearn a representation that includes image information, allowing for better parameter sharing
among characters, particularly characters that areless common. Different from the L OOKUP model,
each character is first transformed into a 36-by-36
image based on its Unicode encoding as shown in
the upper part of Fig 2.1. We then pass the image through a CNN to get the embedding for theimage. The parameters for the CNN are learned
through backpropagation from the classification

4

B

L

1 XX
J=
−ti,j log(pi,j )B(2)

i=1 j=1

where ti,j ∈ {0, 1} represents the ground truth label of the j-th class in the i-th Wikipedia page title. B is the batch size and L is the number ofcategories.

Fusion-based Models

One thing to note is that the L OOKUP and the
V ISUAL models have their own advantages. The
L OOKUP model learns embedding that captures
the semantics of each character symbol without
sharing information with each other. In contrast, the proposed V ISUAL model directly learns
embedding from visual information, which naturally shares information between visually similar
characters. This characteristic gives the V ISUAL

2062

Lookup/Visualzh tradzh simpjako100%
0.55/0.54
0.55/0.54
0.42/0.39
0.47/0.4250%
0.53/0.50
0.53/0.52
0.47/0.45
0.44/0.39

12.5%
0.48/0.47
0.48/0.46
0.44/0.41
0.37/0.36

Table 3: The classification results of the L OOKUP/ V ISUAL models for different percentages of fulltraining size.
model the ability to generalize better to rare characters, but also has the potential disadvantage ofintroducing noise for characters with similar appearances but different meanings.
With the complementary nature of these twomodels in mind, we further combine the two embeddings to achieve better performances. We
adopt three fusion schemes, early fusion, late fusion (described by Snoek et al. (2005) and Karpathy et al. (2014)), and fallback fusion, a method
specific to this paper.
Early Fusion Early fusion works by concatenating the two varieties of embeddings before feedingthem into the RNN. In order to ensure that the dimensions of the RNN are the same after concatenation, the concatenated vector is fed through ahidden layer to reduce the size from 2 × dc to dc .
The whole model is then fine-tuned with training
data.
Late Fusion Instead of learning a joint representation like early fusion, late fusion averages themodel predictions. Specifically, it takes the outputof the softmax layers from both models and averages the probabilities to create a final distribution
used to make the prediction.
Fallback Fusion Our final fallback fusion
method hypothesizes that our V ISUAL model does
better with instances which contain more rare
characters. First, in order to quantify the overall rareness of an instance consisting of multiple
characters, we calculate the average training set
frequency of the characters therein. The fallback
fusion method uses the V ISUAL model to predict
testing instances with average character frequency
below or equal to a threshold (here we use 0.0 frequency as cutoff, which means all characters in the
instance do not appear in the training set), and uses
the L OOKUP model to predict the rest of the instances.

5

Experiments and Results

In this section, we compare our proposed V ISUAL
model with the baseline L OOKUP model throughthree different sets of experiments. First, we examine whether our model is capable of classifying text and achieving similar performance as thebaseline model. Next, we examine the hypothesisthat our model will outperform the baseline model
when dealing with low frequency characters. Finally, we examine the fusion methods described in
Section 4.
5.1

Experimental Configurations

The dimension of the embeddings and batch sizefor both models are set to dc = 128 and B =400, respectively. We build our proposed modelusing Torch (Collobert et al., 2002), and use Adam
(Kingma and Ba, 2014) with a learning rate η =0.001 for stochastic optimization. The length ofeach instance is cut off or padded to 10 charactersfor batch training.
5.2

Comparison with the Baseline ModelIn this experiment, we examine whether our V I SUAL model achieves similar performance withthe baseline L OOKUP model in classification accuracy.
The results in Tab. 3 show that the baselinemodel performs 1-2% better across four datasets;
this is due to the fact that the L OOKUP model candirectly learn character embeddings that capture
the semantics of each character symbol for frequent characters. In contrast, the V ISUAL model
learns embeddings from visual information, whichconstraints characters that has similar appearance
to have similar embeddings. This is an advantage
for rare characters, but a disadvantage for high frequency characters because being similar in appearance does not always lead to similar semantics.
To demonstrate that this is in fact the case, besides looking at the overall classification accuracy,
we also examine the performance on classifying
low frequency instances which are sorted according to the average training set frequency of thecharacters therein. Tab. 4 and Fig. 4 both show that
our model performs better in the 100 lowest frequency instances (the intersection point of the twomodels). More specifically, take Fig. 4(a)’ as example, the solid (proposed) line is higher than thedashed (baseline) line up to 102 , indicating that the
proposed model outperforms the baseline for the

2063

(a)

(b)

3

10

2

2

10

Accumulated Number ofCorrectly Predicted Instances

10⎯⎯!Visual,(TP(=(100%⎯⎯!Visual,(TP(=(50%⎯⎯!Visual,(TP(=(12.5%⎯!⎯!Lookup,(TP(=(100%⎯!⎯!Lookup,(TP(=(50%⎯!⎯!Lookup,(TP(=(12.5%

1

10

0

10

(c)

2

0

10

3

10

⎯⎯!Visual,(TP(=(100%⎯⎯!Visual,(TP(=(50%⎯⎯!Visual,(TP(=(12.5%⎯!⎯!Lookup,(TP(=(100%⎯!⎯!Lookup,(TP(=(50%⎯!⎯!Lookup,(TP(=(12.5%

1

10

10

(d)

3

10

2

3

10

10

3

10

2

2

10

10

⎯⎯!Visual,(TP(=(100%⎯⎯!Visual,(TP(=(50%⎯⎯!Visual,(TP(=(12.5%⎯!⎯!Lookup,(TP(=(100%⎯!⎯!Lookup,(TP(=(50%⎯!⎯!Lookup,(TP(=(12.5%

1

10

0

10

3

10

2

1

10

0

3

10

10

10 1
10

⎯⎯!Visual,(TP(=(100%⎯⎯!Visual,(TP(=(50%⎯⎯!Visual,(TP(=(12.5%⎯!⎯!Lookup,(TP(=(100%⎯!⎯!Lookup,(TP(=(50%⎯!⎯!Lookup,(TP(=(12.5%
2

10

3

10

Rank

Figure 4: Experiments on different training sizes for four different datasets. More specifically, we consider three different training data size percentages (TPs) (100%, 50%, and 12.5%) and four datasets: (a)traditional Chinese, (b) simplified Chinese, (c) Japanese, and (d) Korean. We calculate the accumulated
number of correctly predicted instances for the V ISUAL model (solid lines) and the L OOKUP model(dashed lines). This figure is a log-log plot, where x-axis shows rarity (rarest to the left), y-axis shows
cumulative correctly classified instances up to this rank; a perfect classifier will result in a diagonal line.
first 100 instances. Lines depart the x-axis whenthe model classifies its first instance correctly, andthe L OOKUP model did not correctly classify anyof the first 80 rarest instances, resulting in it crossing later than the proposed model. This confirmsthat the V ISUAL model can share visual information among characters and help to classify low frequency instances.
For training time, visual features take significantly more time, as expected. VISUAL is 30x
slower than LOOKUP, although they are equivalent at test time. For space, images of Chinesecharacters took 36MB to store for 8985 characters.
5.3

Lookup/Visualzh tradzh simpjako100
0.22/0.49
0.25/0.53
0.30/0.35
0.44/0.33

1000
0.35/0.35
0.39/0.37
0.45/0.41
0.44/0.33

10000
0.40/0.39
0.41/0.40
0.44/0.41
0.48/0.42

Table 4: Classification results for the L OOKUP/ V ISUAL of the k lowest frequency instancesacross four datasets. The 100 lowest frequency instances for traditional and simplified Chinese andKorean were both significant (p-value < 0.05).
Those for Japanese were not (p-value = 0.13);
likely because there was less variety than Chineseand more data than Korean.

Experiments on Different Training Sizes

In our second experiment, we consider two smaller
training sizes (i.e., 50% and 12.5% of the fulltraining size) indicated by green and red lines inFig. 4. We performed this experiment under the
hypothesis that because the proposed method wasmore robust to infrequent characters, the proposedmodel may perform better in low-resourced scenarios. If this is the case, the intersection point of
the two models will shift right because of the increase of the number of instances with low averagecharacter frequency.

As we can see in Fig. 4, the intersection point
for 100% training data lies between the intersection point for 50% training data and 12.5%. Thisdisagrees with our hypothesis; this is likely because while the number of low-frequency characters increases, smaller amounts of data also adversely impact the ability of CNN to learn usefulvisual features, and thus there is not a clear gain
nor loss when using the proposed method.
As a more extreme test of the ability of our proposed framework to deal with the unseen char-

2064

Lookup
Visual
early
late
fall

zh trad0.55030.54340.55200.56580.5507zh simp0.55430.54030.5546
0.56850.5547

ja
0.49140.47750.4896
0.50290.4914ko
0.47650.42070.4796
0.48690.4766

Table 5: Experiment results for three different fusion methods across 4 datasets. The late fusion
model was better (p-value < 0.001) across four
datasets.
acters in the test set, we use traditional Chinese
as our training data and simplified Chinese as
our testing data. The model was able to achieve
around 40% classification accuracy when we use
the full training set, compared to 55%, which isachieved by the model trained on simplified Chinese. This result demonstrates that the model isable to transfer between similar scripts, similarly
to how most Chinese speakers can guess the meaning of the text, even if it is written in the otherscript.
5.4

Experiment on Different Fusion Methods

Results of different fusion methods can be foundin Tab. 5. The results show that late fusion
gives the best performance among all the fusion schemes combining the L OOKUP modeland the proposed V ISUAL model. Early fusion
achieves small improvements for all languages except Japanese, where it displays a slight drop.
Unsurprisingly, fallback fusion performs better
than the L OOKUP model and the V ISUAL model
alone, since it directly targets the weakness of the
L OOKUP model (e.g., rare characters) and replaces
the results with the V ISUAL model. These results show that simple integration, no matter whichschemes we use, is beneficial, demonstrating thatboth methods are capturing complementary information.
5.5

Visualization of Character Embeddings

Finally, we qualitatively examine what is learned
by our proposed model in two ways. First, we
visualize which parts of the image are most important to the V ISUAL model’s embedding calculation. Second, we show the 6-nearest neighbor
results for characters using both the L OOKUP and
the V ISUAL embeddings.

Iron

BronzeSalmon Serranidae

Silk

Coil

Rhyme

PleasedWave

Put onCypress

PillarCuckooEagle

Mosquito

Ant

Figure 5: Examples of how much each part of thecharacter contributes to its embedding (the darker
the more). Two characters are shown per radical toemphasize that characters with same radical have
similar patterns.
Emphasis of the V ISUAL Model In order todelve deeper into what the V ISUAL model has
learned, we measure a modified version of the occlusion sensitivity proposed by Zeiler and Fergus(2014) by masking the original character image infour ways, and examine the importance of eachpart of the character to the model’s calculated representations. Specifically, we leave only the upper half, bottom half, left half, or right half of theimage, and mask the remainder with white pixels since Chinese characters are usually formedby combining two radicals vertically or horizontally. We run these four images forward throughthe CNN part of the model and calculate the L2
distance between the masked image embeddings
with the full image embedding. The larger the distance, the more the masked part of the character
contributes to the original embedding. The contribution of each part (e.g. the L2 distance) is represented as a heat map, and then it is normalized toadjust the opacity of the character strokes for better visualization. The value of each corner of theheatmap is calculated by adding the two L2 distances that contribute to this corner.
The visualization is shown in Fig. 5. The meaning of each Chinese character in English is shown
below the Chinese character. The opacity of thecharacter strokes represent how much the corresponding parts contribute to the original embedding (the darker the more). In general, the darker
part of the character is related to its semantics. For
example, “金” means gold in Chinese, which is2065

Visual'model

Lookup'model Visual'model

Lookup'model

6 Related Work
!
 


Methods that utilize neural networks to learn

! 
! 




distributed representations of words or charac


! !




! 
ters have been widely developed. However,word2vec (Mikolov et al., 2013), for example, re 






quires storing an extremely large table of vectors





for all word types. For example, due to the size
 






of word types in twitter tweets, work has been

Figure 6: Visualization of the Chinese traditional
characters by finding the 6-nearest neighbors of
the query (i.e., center) characters. The highlighted
red indicates the radical along with the meaning of
the characters.

highlighted in both “鐵” (Iron) and “銅” (Bronze).
We can also find similar results for other examples shown in Fig. 5. Fig. 5 also demonstrated
that our model captures the compositionality ofChinese characters, both meaning of sub-character
units and their structure (e.g. the semantic content
tends to be structurally localized on one side of a
Chinese character).
K-nearest neighbors Finally, to illustrate thedifference of the learned embeddings between thetwo models, we display 6-nearest neighbors (L2distance) for selected characters in Fig. 6. As canbe seen, the V ISUAL embedding for characterswith similar appearances are close to each other.
In addition, similarity in the radical part indicatessemantic similarity between the characters. For
example, the characters with radical “鳥” all referto different type of birds.
The L OOKUP embedding do not show such feature, as it learns the embedding individually foreach symbol and relies heavily on the training set
and the task. In fact, the characters shown in Fig. 6
for the L OOKUP model do not exhibit semantic
similarity either. There are two potential explanations for this: First, the category classification
task that we utilized do not rely heavily on the finegrained semantics of each character, and thus the
L OOKUP model was able to perform well without
exactly capturing the semantics of each character
precisely. Second, the Wikipedia dataset contains
a large number of names and location and the characters therein might not have the same semantic
meaning used in daily vocabulary.

done to generate vector representations of tweets
at character-level (Dhingra et al., 2016).
There is also work done in understanding mathematical expressions with a convolutional network for text and layout recognition by using
an attention-based neural machine translation system (Deng et al., 2016). They tested on realworld rendered mathematical expressions pairedwith LaTeX markup and show the system is effective at generating accurate markup. Other thanthat, there are several works that combine visual
information with text in improving machine translation (Sutskever et al., 2014), visual question answering, caption generation (Xu et al., 2015), etc.
These works extract image representations from a
pre-trained CNN (Zhu et al., 2016; Wang et al.,2016).
Unrelated to images, CNNs have also been used
for text classification (Kim, 2014; Zhang et al.,2015). These models look at the sequential dependencies at the word or character-level and achieve
the state-of-the-art results. These works inspire
us to use CNN to extract features from image andserve as the input to the RNN. Our model is able
to directly back-propagate the gradient all the way
through the CNN, which generates visual embeddings, in a way such that the embedding can contain both semantic and visual information.
Several techniques for reducing the rare words
effects have been introduced in the literature, including spelling expansion (Habash, 2008), dictionary term expansion (Habash, 2008), proper nametransliteration (Daumé and Jagarlamudi, 2011),
treating words as a sequence of characters (Luong and Manning, 2016), subword units (Sennrichet al., 2015), and reading text as bytes (Gillick
et al., 2015). However, most of these techniques
still have no mechanism for handling low frequency characters, which are the target of thiswork.
Finally, there are works on improving embeddings with radicals, which explicitly splits Chinese characters into radicals based on a dictionary2066

of what radicals are included in which characters(Li et al., 2015; Shi et al., 2015; Yin et al., 2016).
The motivation of this method is similar to ours,
but is only applicable to Chinese, in contrast to
the method in this paper, which works on any language for which we can render text.

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho,and Yoshua Bengio. 2014. Empirical evaluation of
gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 .

7

Y. Le Cun, B. Boser, J. S. Denker, R. E. Howard,
W. Habbard, L. D. Jackel, and D. Henderson. 1990.
Advances in neural information processing systems
2. pages 396–404.

Conclusion and Future WorkIn this paper, we proposed a new frameworkthat utilizes appearance of characters, convolutional neural networks, recurrent neural networks
to learn embeddings that are compositional in thecomponent parts of the characters. More specifically, we collected a Wikipedia dataset, which
consists of short titles of three different languages
and satisfies the compositionality in the charactersof the language. Next, we proposed an end-to-end
model that learns visual embeddings for charactersusing CNN and showed that the features extracted
from the CNN include both visual and semantic
information. Furthermore, we showed that our
V ISUAL model outperforms the L OOKUP baselinemodel in low frequency instances. Additionally,
by examining the character embeddings visually,
we found that our V ISUAL model is able to learn
visually related embeddings.
In summary, we tackled the problem of rare
characters by using embeddings learned from images. In the future, we hope to further generalize this method to other tasks such as pronunciation estimation, which can take advantage of thefact that pronunciation information is encoded inparts of the characters as demonstrated in Fig. 1,
or machine translation, which could benefit froma wholistic view that considers both semantics andpronunciation. We also hope to apply the model toother languages with complicated compositional
writing systems, potentially including historicaltexts such as hieroglyphics or cuneiform.

Acknowledgments
We thank Taylor Berg-Kirkpatrick, Adhiguna
Kuncoro, Chen-Hsuan Lin, Wei-Cheng Chang,
Wei-Ning Hsu and the anonymous reviewers fortheir enlightening comments and feedbacks.

Ronan Collobert, Samy Bengio, and Johnny Marithoz.
2002. Torch: A modular machine learning software
library.

Peter T Daniels and William Bright. 1996. The world’s
writing systems. Oxford University Press.
Hal Daumé and Jagadeesh Jagarlamudi. 2011. Domain
adaptation for machine translation by mining unseen
words. In ACL-HLT. pages 407–412.
Yuntian Deng, Anssi Kanervisto, and Alexander M.
Rush. 2016. What you get is what you see:A visual markup decompiler.
arXiv preprint
arXiv:1609.04938 .
Bhuwan Dhingra, Zhong Zhou, Dylan Fitzpatrick,Michael Muehl, and William W Cohen. 2016.
Tweet2vec: Character-based distributed representations for social media. ACL .
Gottlob Frege and John Langshaw Austin. 1980. The
foundations of arithmetic: A logico-mathematicalenquiry into the concept of number. Northwestern
University Press.
Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag
Subramanya. 2015. Multilingual language processing from bytes. arXiv preprint arXiv:1512.00103 .
Nizar Habash. 2008. Four techniques for online handling of out-of-vocabulary words in Arabic-English
statistical machine translation. In HLT-Short. pages
57–60.
Mohit Iyyer, Varun Manjunatha, and Jordan L BoydGraber. 2015. Deep unordered composition rivals
syntactic methods for text classification. In ACL.
Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network formodelling sentences. ACL pages 655–665.
Andrej Karpathy, George Toderici, Sanketh Shetty,
Thomas Leung, Rahul Sukthankar, and Li Fei-Fei.
2014. Large-scale video classification with convolutional neural networks. In CVPR. pages 1725–1732.

References

Jun’ichi Kazama and Kentaro Torisawa. 2007. Exploiting Wikipedia as external knowledge for namedentity recognition. In EMNLP-CoNLL. pages 698–
707.

Jan A Botha and Phil Blunsom. 2014. Compositional
morphology for word representations and language
modelling. In ICML. pages 1899–1907.

Yoon Kim. 2014. Convolutional neural networks forsentence classification. In EMNLP. pages 1746–
1751.

2067

Diederik Kingma and Jimmy Ba. 2014. Adam: A
method for stochastic optimization. arXiv preprint
arXiv:1412.6980 .
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov,
Richard Zemel, Raquel Urtasun, Antonio Torralba,
and Sanja Fidler. 2015. Skip-thought vectors. In
NIPS. pages 3294–3302.
Yanran Li, Wenjie Li, Fei Sun, and Sujian Li.
2015. Component-enhanced chinese character embeddings. EMNLP pages 829–834.
Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo,
and Tiago Luis. 2015. Finding function in form:
Compositional character models for open vocabulary word representation. In EMNLP. pages 1520–
1530.
Minh-Thang Luong and Christopher D Manning. 2016.
Achieving open vocabulary neural machine translation with hybrid word-character models. ACL pages
1054–1063.
Thang Luong, Richard Socher, and Christopher Manning. 2013. Better word representations with recursive neural networks for morphology. In CoNLL.
pages 104–113.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS. pages 3111–3119.

Antonio Toral and Rafael Munoz. 2006. A proposal
to automatically build and maintain gazetteers for
named entity recognition by using wikipedia. In
EACL. pages 56–61.
Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang,
Chang Huang, and Wei Xu. 2016. Cnn-rnn: A unified framework for multi-label image classification.
In CVPR. pages 2285–2294.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho,
Aaron C Courville, Ruslan Salakhutdinov, Richard S
Zemel, and Yoshua Bengio. 2015. Show, attend andtell: Neural image caption generation with visual attention. In ICML.
Rongchao Yin, Quan Wang, Rui Li, Peng Li, and Bin
Wang. 2016. Multi-granularity chinese word embedding. EMNLP pages 981–986.
Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In
ECCV. Springer, pages 818–833.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.
Character-level convolutional networks for text classification. In NIPS. pages 649–657.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li FeiFei. 2016. Visual7w: Grounded question answering
in images. In CVPR. pages 4995–5004.

Mej Newman. 2005. Power laws, Pareto distributions
and Zipf’s law. CONTEMP PHYS pages 323–351.
Lev Ratinov and Dan Roth. 2009. Design challenges
and misconceptions in named entity recognition. In
CoNLL. pages 147–155.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2015. Neural machine translation of rare words withsubword units. ACL pages 1715–1725.
Xinlei Shi, Junjie Zhai, Xudong Yang, Zehua Xie,and Chao Liu. 2015. Radical embedding: Delving
deeper to chinese radicals. In ACL. pages 594–598.
Cees GM Snoek, Marcel Worring, and Arnold WM
Smeulders. 2005. Early versus late fusion in semantic video analysis. In ACM MM. pages 399–402.
Richard Socher, Alex Perelygin, Jean Y Wu, Jason
Chuang, Christopher D Manning, Andrew Y Ng,
and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentimenttreebank. In EMNLP. pages 1631–1642.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.
Sequence to sequence learning with neural networks. In NIPS. pages 3104–3112.
Zoltán Gendler Szabó. 2010. Compositionality. Stanford encyclopedia of philosophy .

2068