CharManteau: Character Embedding Models For Portmanteau Creation
Varun Gangal ∗, Harsh Jhamtani ∗, Graham Neubig, Eduard Hovy, Eric Nyberg
Language Technologies Institute,Carnegie Mellon University{vgangal,jharsh,gneubig,hovy,ehn}@cs.cmu.edu

Abstract
Portmanteaus are a word formation phenomenon where two words are combined to form a new word. We propose character-level neural sequence-tosequence (S2S) methods for the task ofportmanteau generation that are end-toend-trainable, language independent, anddo not explicitly use additional phonetic
information. We propose a noisy-channelstyle model, which allows for the incorporation of unsupervised word lists, improving performance over a standard sourceto-target model. This model is made possible by an exhaustive candidate generation strategy specifically enabled by thefeatures of the portmanteau task. Experiments find our approach superior toa state-of-the-art FST-based baseline withrespect to ground truth accuracy and human evaluation.

1

Figure 1: A sketch of our BACKWARD, noisychannel model. The attentional S2S model withbidirectional encoder gives P (x|y) and nextcharacter model gives P (y), where y (spime) isthe portmanteau and x = concat(x(1) , “;”, x(2) )are the concatenated root words (space and time).

Introduction

Portmanteaus (or lexical blends Algeo (1977)) arenovel words formed from parts of multiple rootwords in order to refer to a new concept which
can’t otherwise be expressed concisely. Portmanteaus have become frequent in modern-day social
media, news reports and advertising, one popular example being Brexit (Britain + Exit). Petri(2012). These are found not only in English but
many other languages such as Bahasa Indonesia Dardjowidjojo (1979), Modern Hebrew BatEl (1996); Berman (1989) and Spanish Piñeros(2004). Their short length makes them ideal forheadlines and brandnames (Gabler, 2015). Unlike
better-defined morphological phenomenon such as
inflection and derivation, portmanteau generation∗* denotes equal contribution

is difficult to capture using a set of rules. For
instance, Shaw et al. (2014) state that the composition of the portmanteau from its root words
depends on several factors, two important ones
being maintaining prosody and retaining character segments from the root words, especially thehead. An existing work by Deri and Knight (2015)
aims to solve the problem of predicting portmanteau using a multi-tape FST model, which is datadriven, unlike prior approaches. Their methods
rely on a grapheme to phoneme converter, whichtakes into account the phonetic features of the language, but may not be available or accurate fornon-dictionary words, or low resource languages.
Prior works, such as Faruqui et al. (2016), havedemonstrated the efficacy of neural approaches formorphological tasks such as inflection. We hypothesize that such neural methods can (1) provide a simpler and more integrated end-to-end
framework than multiple FSTs used in the previous work, and (2) automatically capture features
such as phonetic similarity through the use of character embeddings, removing the need for explicit

2907

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2907–2912
Copenhagen, Denmark, September 7–11, 2017. c 2017 Association for Computational Linguistics

grapheme-to-phoneme prediction. To test thesehypotheses, in this paper, we propose a neural S2Smodel to predict portmanteaus given the two rootwords, specifically making 3 major contributions:• We propose an S2S model that attends to thetwo input words to generate portmanteaus,
and an additional improvement that leverages noisy-channel-style modelling to incorporate a language model over the vocabularyof words (§2).
• Instead of using the model to directly predict output character-by-character, we use thefeatures of portmanteaus to exhaustively generate candidates, making scoring using thenoisy channel model possible (§3).
• We curate and share a new and larger datasetof 1624 portmanteaus (§4).
In experiments (§5), our model performs better
than the baseline Deri and Knight (2015) on bothobjective and subjective measures, demonstratingthat such methods can be used effectively in a morphological task.

2

Proposed Models

This section describes our neural models.
2.1

Forward Architecture

Under our first proposed architecture, the input sequence x = concat(x(1) , “;”, x(2) ), while the output sequence is the portmanteau y. The model
learns the distribution P (y|x).
The network architecture we use is an attentional S2S model (Bahdanau et al., 2014). We
use a bidirectional encoder, which is known to
work well for S2S problems with similar token−−−−→
order, which is true in our case. Let LST M←−−−−
and LST M represent the forward and reverse encoder; eenc () and edec () represent the character
embedding functions used by encoder and decoder
The following equations describe the model:−→→− ←− →−henc= 0 , henc0|x| = 0
−→
−−−−→
henc
= LST M (henctt−1 , eenc (xt ))←−←−−−− enc
enc
ht = LST M (ht+1 , eenc (xt ))−→The context vector ct is computed using dotproduct attention over encoder states. We choose
dot-product attention because it doesn’t add extra
parameters, which is important in a low-data scenario such as portmanteau generation.
enc
ati = dot(hdec), αt = softmax(at )t , hii=|x|ct =In addition to capturing the fact that portmanteaus of two English words typically soundEnglish-like, and to compensate for the fact thatavailable portmanteau data will be small, we pretrain the character embeddings on English language words. We use character embeddings learnt
using an LSTM language model over words in
an English dictionary,1 where each word is a sequence of characters, and the model will predict
next character in sequence conditioned on previous characters in the sequence.
2.2

Backward Architecture

The second proposed model uses Bayes’s rule to(y)reverse the probabilities P (y|x) = P (x|y)PtoP (x)get argmaxy P (y|x) = argmaxy P (x|y)P (y).
Thus, we have a reverse model of the probability P (x|y) that the given root words were generated from the portmanteau and a character language model model P (y). This is a probability
distribution over all character sequences y ∈ A∗ ,where A is the alphabet of the language. This way
of factorizing the probability is also known as anoisy channel model, which has recently also beenshown to be effective for neural MT (Hoang et al.
(2017), Yu et al. (2016)). Such a model offers two
advantages
1. The reverse direction model (or alignment
model) gives higher probability to those portmanteaus from which one can discern theroot words easily, which is one feature ofgood portmanteaus.
2. The character language model P (y) can betrained on a large vocabulary of words in thelanguage. The likelihood of a word y is faci=|y|torized as P (y) = Πi=1 P (yi |y1i−1 ), whereiyj = yi , yi+1 . . . yj , and we train a LSTM tomaximize this likelihood.

←−

enc
hdec
0 = h|x|pt = softmax(Whs [concat(hdect , ct )] + bs )αit hencii=1+ henchenc= hencttthdec= LST M (hdectt−1 , [concat(edec (yt−1 ), ct−1 )])X

1Specifically in our experiments, 134K words from the
CMU dictionary (Weide, 1998).

2908

3

Making Predictions

Given these models, we must make predictions,which we do by two methods
Greedy Decoding: In most neural sequenceto-sequence models, we perform autoregressive greedy decoding, selecting thenext character greedily based on the probability distribution for the next character atcurrent time step. We refer to this decoding
strategy as G REEDY.
Exhaustive Generation: Many
portmanteaus
were observed to be concatenation of a prefixof the first word and a suffix of the second.
We therefore generate all candidate outputs
which follow this rule. Thereafter we scorethese candidates with the decoder and output
the one with the maximum score. We refer tothis decoding strategy as S CORE.
Given that our training data is small in size, we
expect ensembling (Breiman, 1996) to help reduce
model variance and improve performance. In thispaper, we ensemble our models wherever mentioned by training multiple models on 80% subsamples of the training data, and averaging log
probability scores across the ensemble at test-time.

4

Dataset

The existing dataset by Deri and Knight (2015)
contains 401 portmanteau examples from
Wikipedia. We refer to this dataset as DWiki .
Besides being small for detailed evaluation, DWiki
is biased by being from just one source. We
manually collect DLarge , a dataset of 1624 distinct
English portmanteaus from following sources:• Urban Dictionary2• Wikipedia• Wiktionary• BCU’s Neologism Lists from ’94 to ’12.
Naturally, DWiki ⊂ DLarge . We define DBlind =DLarge − DWiki as the dataset of 1223 examples notfrom Wikipedia. We observed that 84.7% of thewords in DLarge can be generated by concatenating
prefix of first word with a suffix of the second.
2

Not all neologisms are portmanteaus, so we manuallychoose those which are for our dataset.

Model

Attn

Ens

Init

SearchMatches

Distance

BASELINE

X
X
X
X
X
X
X
X
×
×
X
X
X
X
×
×

×
×
×
×
×
×
X
X
×
×
×
×
X
X
×
×

×
X
×
X
×
X
X
×
X
×
×
X
X
×
X
×

G REEDY
G REEDY
B EAM
B EAM
S CORES CORES CORES CORES CORES CORES CORES CORES CORES CORES CORES CORE

45.39%
22.00%
28.00%
13.25%
15.25%
30.25%
32.88%
42.25%
41.25%
6.75%
6.50%
37.00%
42.25%
48.75%
46.50%
5.00%
4.75%

1.59
1.98
1.90
2.47
2.37
1.64
1.53
1.33
1.34
3.78
3.76
1.53
1.35
1.12
1.24
3.95
3.98

F ORWARD

BACKWARD

Table 1: 10-Fold Cross-Validation results, DWiki .
Attn, Ens, Init denote attention, ensembling, andinitializing character embeddings respectively.

5

Experiments

In this section, we show results comparing various configurations of our model to the baseline FST model of Deri and Knight (2015)(BASELINE). Models are evaluated using exactmatches (Matches) and average Levenshtein editdistance (Distance) w.r.t ground truth.
5.1

Objective Evaluation Results

In Experiment 1, we follow the same setup as Deriand Knight (2015). DWiki is split into 10 folds.
Each fold model uses 8 folds for training, 1 for validation, and 1 for test. The average (10 fold crossvalidation style approach) performance metrics onthe test fold are then evaluated. Table 1 shows theresults of Experiment 1 for various model configurations. We get the BASELINE numbers from
Deri and Knight (2015). Our best model obtains
48.75% Matches and 1.12 Distance, compared to45.39% Matches and 1.59 Distance using BASELINE.
For Experiment 2, we seek to compare our best
approaches from Experiment 1 to the BASELINE
on a large, held-out dataset. Each model is trainedon DWiki and tested on DBlind . BASELINE wassimilarly trained only on DWiki , making it a faircomparison. Table 2 shows the results3 . Our bestmodel gets Distance of 1.96 as compared to 2.32
from BASELINE.
We observe that the Backward architecture performs better than Forward architecture, confirming our hypothesis in §2.2. In addition, ablation
results confirm the importance of attention, and3

For BASELINE (Deri and Knight, 2015), we use
their trained model from http://leps.isi.edu/fst/
step-all.php

2909Model

Attn

Ens

Init

SearchMatches

Distance

BASELINE

X
X
X
X
X
X
X
X

×
×
X
X
×
×
X
X

X
×
×
X
X
×
×
X

SCORE
SCORE
SCORE
SCORE
SCORE
SCORE
SCORE
SCORE

31.56%
25.26%
24.93%
31.23%
28.94%
25.75%
25.26%
31.72%
32.78%

2.32
2.13
2.32
1.98
2.04
2.14
2.17
1.96
1.96

F ORWARD

BACKWARD

Input
shopping;marathon
fashion;fascism
wiki;etiquetteclown;president

FORWARD

BACKWARD

shopparathon
fashism
wikiquetteclowident

shoathonfashismwiquette
clownsident

G ROUND T RUTH
shopathonfashism
wikiquetteclownsident

Table 3: Example outputs from different models(Refer to appendix for more examples)

Table 2: Results on DBlind (1223 Examples). In
general, BACKWARD architecture performs better
than F ORWARD architecture.

Judgement
Much Better (1)Better (2)Worse (3)Much Worse (4)Percentage of total
29.06
29.06
25.11
16.74

Table 4: AMT annotator judgements on whether
our system’s proposed portmanteau is better orworse compared to the baseline

Figure 2: Attention matrices while generating slurve from slider;curve, and bennifer fromben;jennifer respectively, using Forward model. ;and . are separator and stop characters. Darker
cells are higher-valued

In order to do this, we use a paired bootstrap4
comparison (Koehn, 2004) between BACKWARD
and BASELINE in terms of Matches. BACKWARD
is found to be better (gets more Matches) than
BASELINE in 99.9% (p = 0.999) of the subsets.
Similarly, BACKWARD has a lower Distancethan BASELINE by a margin of 0.2 in 99.5% (p =0.995) of the subsets.
5.3

initializing the word embeddings. We believe thisis because portmanteaus have high fidelity towards
their root word characters and its critical that themodel can observe all root sequence characters,
which attention manages to do as shown in Fig. 2.
5.1.1 Performance on Uncovered Examples
The set of candidates generated before scoringin the approximate SCORE decoding approachsometimes do not cover the ground truth. This
holds true for 229 out of 1223 examples in DBlind .
We compare the F ORWARD approach along with a
G REEDY decoding strategy to the BASELINE approach for these examples.
Both F ORWARD +G REEDY and the BASELINE
get 0 Matches on these examples. The Distancefor these examples is 4.52 for BASELINE and 4.09
for F ORWARD +G REEDY. Hence, we see that oneof our approaches (F ORWARD +G REEDY) outperforms BASELINE even for these examples.
5.2

Significance Tests

Since our dataset is still small relatively small(1223 examples), it is essential to verify whether
BACKWARD is indeed statistically significantly
better than BASELINE in terms of Matches.

Subjective Evaluation and Analysis

On inspecting outputs, we observed that often output from our system seemed good in spite of highedit distance from ground truth. Such aspect of an
output seeming good is not captured satisfactorily
by measures like edit distance. To compare theerrors made by our model to the baseline, we designed and conducted a human evaluation task on
AMT.5 In the survey, we show human annotators
outputs from our system and that of the baseline.
We ask them to judge which alternative is betteroverall based on following criteria: 1. It is a goodshorthand for two original words 2. It sounds better. We requested annotation on a scale of 1-4.
To avoid ordering bias, we shuffled the order of
two portmanteau between our system and that ofbaseline. We restrict annotators to be from Anglophone countries, have HIT Approval Rate > 80%
and pay 0.40$ per HIT (5 Questions per HIT).
As seen in Table 4, output from our system waslabelled better by humans as compared to the baseline 58.12% of the time. Table 3 shows outputs
from different models for a few examples.
4
We average across M = 1000 randomly chosen subsets
of DBlind , each of size N = 611 (≈ 1223/2)
5
We avoid ground truth comparison because annotators
can be biased to ground truth due to its existing popularity.

2910

6

Related Work

ReferencesÖzbal and Strapparava (2012) generate new words
to describe a product given its category and properties. However, their method is limited to handcrafted rules as compared to our data driven approach. Also, their focus is on brand names.
Hiranandani et al. (2017) have proposed an approach to recommend brand names based onbrand/product description. However, they consider only a limited number of features like memorability and readability. Smith et al. (2014) devise an approach to generate portmanteaus, which
requires user-defined weights for attributes likesounding good. Generating a portmanteau fromtwo root words can be viewed as a S2S problem.
Recently, neural approaches have been used for
S2S problems (Sutskever et al., 2014) such as MT.
Ling et al. (2015) and Chung et al. (2016) haveshown that character-level neural sequence models work as well as word-level ones for language
modelling and MT. Zoph and Knight (2016) propose S2S models for multi-source MT, which havemulti-sequence inputs, similar to our case.

7

Conclusion

We have proposed an end-to-end neural system tomodel portmanteau generation. Our experiments
show the efficacy of proposed system in predicting portmanteaus given the root words. We conclude that pre-training character embeddings on
the English vocabulary helps the model. Throughhuman evaluation we show that our model’s predictions are superior to the baseline. We havealso released our dataset and code6 to encourage
further research on the phenomenon of portmanteaus. We also release an online demo 7 where ourtrained model can be queried for portmanteau suggestions. An obvious extension to our work is totry similar models on multiple languages.

Acknowledgements
We thank Dongyeop Kang, David Mortensen,
Qinlan Shen and anonymous reviewers for theirvaluable comments. This research was supported in part by DARPA grant FA8750-12-20342 funded under the DEFT program.
6

https://github.com/vgtomahawk/
Charmanteau-CamReady7
http://tinyurl.com/y9x6mvy

John Algeo. 1977. Blends, a structural and systemic
view. American speech, 52(1/2):47–64.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly
learning to align and translate. arXiv:1409.0473.
Outi Bat-El. 1996. Selecting the best of the worst:
the grammar of Hebrew blends.
Phonology,
13(03):283–328.
Ruth Berman. 1989. The role of blends in Modern
Hebrew word-formation. Studia linguistica et orientalia memoriae Haim Blanc dedicata. Wiesbaden:
Harrassowitz, pages 45–61.
Leo Breiman. 1996. Bagging predictors. Machinelearning, 24(2):123–140.
Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation.
arXiv:1603.06147.
Soenjono Dardjowidjojo. 1979. Acronymic Patterns inIndonesian. Pacific Linguistics Series C, 45:143–
160.
Aliya Deri and Kevin Knight. 2015. How to make afrenemy: Multitape FSTs for portmanteau generation. In Proceedings of NAACL-HLT, pages 206–
210.
Manaal Faruqui, Yulia Tsvetkov, Graham Neubig, andChris Dyer. 2016. Morphological Inflection Generation using Character Sequence to Sequence Learning. In Proceedings of NAACL-HLT, pages 634–
643.
Neal Gabler. 2015. The Weird Science of Naming New
Products. New York Times - http://tinyurl.
com/lmlq7ex.
Gaurush Hiranandani, Pranav Maneriker, and Harsh
Jhamtani. 2017. Generating appealing brand names.
arXiv preprint arXiv:1706.09335.
Cong Duy Vu Hoang, Gholamreza Haffari, and Trevor
Cohn. 2017. Decoding as Continuous Optimizationin Neural Machine Translation. arXiv:1701.02854.
Philipp Koehn. 2004. Statistical significance tests formachine translation evaluation. In EMNLP, pages
388–395.
Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W
Black. 2015. Character-based neural machine translation. arXiv:1511.04586.
Gözde Özbal and Carlo Strapparava. 2012. A computational approach to the automation of creative naming. In Proceedings of ACL, pages 703–711. Association for Computational Linguistics.

2911

Alexandra Petri. 2012.
Say No to Portmanteaus. Washington Post - http://tinyurl.
com/kvmep2t.
Carlos-Eduardo Piñeros. 2004. The creation of portmanteaus in the extragrammatical morphology of
Spanish. Probus, 16(2):203–240.
Katherine E Shaw, Andrew M White, Elliott Moreton, and Fabian Monrose. 2014. Emergent faithfulness to morphological and semantic heads in lexical
blends. In Proceedings of the Annual Meetings on
Phonology, volume 1.
Michael R Smith, Ryan S Hintze, and Dan Ventura.
2014. Nehovah: A neologism creator nomen ipsum.
In Proceedings of the International Conference on
Computational Creativity, pages 173–181.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.
Sequence to sequence learning with neural networks. In Neural information processing systems,
pages 3104–3112.
R Weide. 1998. The CMU pronunciation dictionary,
release 0.6. Carnegie Mellon University.
Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Tomas Kocisky. 2016. The Neural Noisy
Channel. arXiv:1611.02554.
Barret Zoph and Kevin Knight. 2016. Multi-source
neural translation. arXiv:1601.00710.

2912