Consonant is all you need: a compact representation of English text for efficient NLP
Authors: Maged S. Al-shaibani
Irfan Ahmad
Venue: EMNLP 2023 Findings (2023)
Keywords: Text Representation
Tokenization
Language Modeling
Text Classification
POS Tagging
NER
NMT
TL;DR: A novel approach to represent English text in NLP using only consonants, which achieves comparable performance to standard representation while significantly reducing computational resources.

Abstract: In natural language processing (NLP), the representation of text plays a crucial role in various tasks such as language modeling, sentiment analysis, and machine translation. The standard approach is to represent text in the same way as we, as humans, read and write. In this paper, we propose a novel approach to represent text with only consonants which presents a compact representation of English text that offers improved efficiency without sacrificing performance. We exploit the fact that consonants are more discriminative than vowels and by representing text using consonants, we can significantly reduce the overall memory and compute footprint required for storing and processing textual data.

We present two alternative representations: 'consonants-only', where we completely remove the vowels from the text, and 'masked-vowels', where we mask all the vowels into one special symbol. To evaluate our approaches, we conducted experiments on various NLP tasks, including text classification, part-of-speech (POS) tagging, named-entity recognition (NER), and neural machine translation (NMT), in addition to language modeling. Our results demonstrate that the proposed consonant-based representation achieves comparable performance compared to the standard text representation while requiring significantly fewer computational resources. Furthermore, we show that our representation can be seamlessly integrated with existing NLP models and frameworks, providing a practical solution for efficient text processing.

Last but not the least, we present a technique to retrieve the vowel information from our processed text representation keeping in mind the need to reproduce text in human readable form in some NLP applications.

Consonant is all you need: a compact representation of English text for efficient NLP
Maged S. Al-Shaibani and Irfan Ahmad ICS Department, KFUPM, Dhahran 31261, Saudi Arabia SDAIA- -FUPM Joint Research Center for AI, Dhahran 31261, Saudi Arabia g201 381338777 @0kfupm. edu. irfan. aaaannahmaakuumm edu. sa
Abstract
Introduction
natural language processing (NLP), the rep- resentation of text plays crucial role in vari- ous tasks such as language modeling, sentiment analysis, and machine translation. The standard approach is to represent text in the same way as we, as humans, read and write. In this paper, we propose novel approach represent text with only consonants which presents compact rep- resentation of English text that offers improved efficiency without sacrificing performance. We exploit the fact that consonants are more dis- criminative than vowels and by representing text using consonants, we can significantly re- duce the overall memory and compute footprint required for storing and processing textual data.
In the field of Natural Language Processing (NLP), text representation and tokenization are closely in- tertwined and have significant implications for var- ious aspects of an NLP task, such as the vocabu- lary size, embedding size, and out-of-vocabulary (OOV) rates. The standard approach to text repre- sentation in English involves including both conso- nants and vowels, mirroring the way humans read and write. However, this representation may not be an optimal representation for English NLP. In this paper, we propose novel approach to English text representation in NLP, inspired by the Abjad writing system, where only consonants are con- sidered as characters and vowels can be inferred (cf. (Daniels, 2013)). Our objective is to develop compact and efficient representation of English text that not only reduces computational demands but also impacts key aspects of NLP tasks. By focusing solely on consonants, we aim achieve a significant reduction in vocabulary size, result- ing in smaller embedding layers and lower OOV rates. We present two alternative representations: the "consonants-only" approach, where vowels are entirely removed from the text, and the "masked- vowels" approach, where all the vowels are masked to special symbol. By leveraging the inherent discriminative nature of consonants, which carry more linguistic information compared to vowels, we can substantially decrease the memory and com- pute footprint associated with storing and process- ing textual data. To evaluate the effectiveness of our proposed consonant-basee representations, we conducted experiments on various NL.P tasks, in- cluding text classification, POS tagging, named- entity recognition (NER), neural machine transla- tion (NMT), and language modeling. Our results demonstrate that our approach not only achieves comparable performance to the standard text repre- sentation but also leads to significant reductions
We present two alternative representations: 'consonantsoolly. where we completely re- move the vowels from the text, and 'masked- vowels", where mask all the vowels into one special symbol. To evaluate our ap- proaches, we conducted experiments on var- ious NLP tasks, including text classification, part-of-speech (POS) tagging, named-entity recognition (NER), and neural machine trans- lation (NMT), in addition to language model- ing. Our results demonstrate that the proposed consonant-based representation achieves com- parable performance compared to the standard text representation while requiring significantly fewer computational resources. Furthermore, we show that our representation can be seam- lessly integrated with existing NLP models and frameworks, providing practical solution for efficient text processing.
Last but not the least, we present a technique retrieve the vowel information from our pro- cessed text representation keeping in mind the need to reproduce text in human readable form in some NL.P applications. The experiments implementation for this study is made publicly available!.

---

in vocabulary size, resulting in smaller embed- ding layers and lower OOV rates. These findings highlight the practical benefits of our approach in terms of memory efficiency and improved handling of unseen words. Furthermore, we address the need for human readability in certain NLP appli- cations, such as machine translation and question answering, by introducing technique to retrieve the vowel information from our consonants-only text representations. This ensures that our approach remains practical and applicable in scenarios where reproducing text in its original form is necessary. The contributions of the paper can be summarized as below:
2 Related Work
To the best of our knowledge, we did not find any work in the literature that introduces consonant- based text representations for NLP tasks. How- ever, one of the prominent advantages of using these representations is the reduction in vocabu- lary size. Word-based embedding layer is usually large layer due to the size of the vocabulary. For that reason, it has high potential to suffer from overfitting. To address this issue, regularization techniques like embedding dropout can be applied (Gal and Ghahramani. 2016). On the other side, techniques such as dimensionality reduction (Rau- nak et al., 2019) quantization (Gholami et al., 2021) and distillation networks (Hinton et al., 2015) can be also be used. Although these methods are effec- tive in addressing the issue of embedding overfit- ting, they do not address the issue by manipulating the vocabulary before embedding them. would be an interesting to explore the model performance where these methods are combined with vocabulary reduction techniques.
We present novel representations of English text for NLP where the vowel characters are either completely removed or masked to one special symbol. This leads to significant re- duction in vocabulary size, embedding layers, OOV rates, and training times.
A statistical n-gram language model, well as two neural language models, RNN-based and transformer-based were built and ana- lyzed to study and compare the new represen- tations.
Switching focus back to the issue of vocabulary size and reduction techniques, methods such bag- of-words and TF-IDF were employed, traditionally. However, these approaches suffered from the issue of vut-of-vocabulary (OOV) words and represent- ing the words with huge sparse embeddings. While these methods excelled in modeling word relations, the OOV problem limited their use, particularly in natural language generation tasks. Additionally, they also faced the challenge of vocabulary explo- sion, where large number of vocabulary items needed be embedded, most of which had very few occurrences in the text (cf. Zipf's law (Zipf, 2016)). To overcome the limitations of sparse em- beddings, dense-embedding representations, such as Word2 (Mikolov et al., 2013), were proposed that were more efficient and effective as compared to spare representations.
We performed several NLP tasks such as sen- timent analysis. text classification, POS tag- ging, NER and NMT using the standard text representation and our two proposed represen- tations and compared their performances.
We present and evaluate technique to retrieve the vowel characters from our consonants- only or masked-vowel. representations. This can be useful in NLP tasks where the output from an NLP task is English text to be used by humans.
On the other hand, using characters as tokens do not suffer from the aforementioned issue. More- over, tokenizing text as characters leads to very small vocabulary size and no OOV issues. How- ever, they do result in longer sequences, which can introduce bottlenecks such vanishing and exploding gradients in sequence modeling models (Bengio et al., 1994). Character-based approaches are known to generate meaningless new words. Nonetheless, a promising approach is to first in- put characters into CNNs to extract features be-
The rest of the paper is organized as follows: in Section 2 we present related works. In Section 3 we present our text representations and the different tasks we conducted to compare the effectiveness of approach. In Section 4, present the experi- ments conducted, the results and the discussions on the results. In Section 5, we present the conclusions and discuss some possible future works. Finally, in Section 6, we present some of the limitations of our work.

---

fore feeding them into an RNN layer, this has been shown to yield good results (Bradbury al.,
Standing the shoulders of giants
2017; Zhang et al., 2015; Conneau et al., 2017). This character-level approach has also been incor- porated into transformer-based architectures (Ma et al., 2020; El Boukkouri et al., 2020).
Stndng n th shidrs f gnts
St#nd#ng #n th# sh##ld#rs #f g##nts
Subword tokenization techniques were intro- duced as a trade-off between characters and words as tokens. These methods split given word into chunks of characters called subwords. Numerous techniques have been proposed to achieve optimal subword splits for a given text. The main motiva- tion behind this approach is to represent the text with smaller vocabulary size of subwords without compromising performance on given task. Some of these methods are language-specifte while oth- ers are data-driven (Alyafeai et al., 2022; Mielke et al., 2021). Data-driven methods are prevalent in recent language models such as the BERT family (Devlin et al., 2019), GPT family (Brown et al., 2020), and other large models. Examples of these methods include SentencePiece (Kudo and Richard- son, 2018), WordPiece (Song al., 2021), and UnigramLM (Kudo, 2018). Most of these meth- ods are based on Byte-Pair- Laite-Parr-Eneee Pair-Encoding (BPE) data compression methods (Gage, 1994).
Figure I: An example of the proposed representations compared to the standard English text. The first line is the standard English text. The second line represents the consonants-only representation. The third line rep- resents the masked- vowels representation.
the symbol '#' Figure illustrates an example of an English statement written in standard English as well using our introduced representations.
3.1 Text analysis
We investigated the vocabulary reduction achieved by both of these representations compared to stan- dard English text in a number of text corpora. In deep learning systems, the vocabulary size plays a crucial role in determining the embedding size and, in certain tasks involving text generation, the output layer size. This factor significantly impacts various aspects of the resulting model, including its overall size, the number of parameters involved. and training and inference times.
Another direction to mitigate the issues associ- ated with tokenization is to utilize visual represen- tation (Mansimov et al., 2020; Salesky et al., 2021). This approach has demonstrated robustness against noise and spelling mistakes. However, is not widely adopted and may not be sufficiently compet- itive at its current stage of development (Mansimov et al., 2020).
To assess the information loss resulting from the removal of vowels and evaluate the efficiency of the proposed representations, we examined their entropy (Shannon, 1951). Deep learning models learn their parameters by minimizing losses, such as cross-entropy, between their predictions and the expected output. If the entropy of these representa- tions is similar the entropy of standard English text, then, theoretically, they should be approxi- mately as learnable as the standard English text. We computed entropy using the formula provided in Equation 1, where represents a token from the set of T vocabulary in given text.
3 Methodology
We introduce two variants of consonant-based rep- resentations for English text. The first representa- tion involves removing all vowel letters (A, E, I, O, and U) regardless of their case, from the text, resulting consonants-only representation. This approach may exclude complete words that consist solely of vowels, such as the determiner 'A' and pronoun 'T'. However, such vowel-only words are rare in English. It is worth noting that certain NL.P tasks require preserving the total number of words in the input sequence in the output sequence, as is the case with sequence labeling tasks such as POS tagging. For these tasks, this representation might be less suitable. This leads us to propose our sec- ond representation, where we mask all vowels with
T H(I) = Σ P(t) P(()
(1)
3.2 Language modeling
To investigate language models, we trained three types of causal language models: statistical, RNN- based, and transformer-based models. The train- ing corpus used for these models is the Wikitext-2 benchmark (Stephen et al., 2017).

---

For the statistical models, we trained (2 6)- gram models using the KenLM toolkit (Heafield, 2011). KenLM implements a modified Kneser- Ney smoothing technique (Heafield et al., 2013), which has been demonstrated to produce sequences with low perplexity. In all of our experiments, we maintained the default hyperparameters.
lation, and optical character recognition (OCR). Therefore, it is erucial to evaluate the performance of the proposed representations on this task. Ad- ditionally, POS tagging and NER are examples of sequence labeling tasks, where the output has the same length as the imput.
We emphasize that, in order to ensure a fair com- parison, the proposed models were tuned to achieve the best results for standard text. Tuning the models to optimize the performance for the proposed repre- sentations might potentially yield superior results compared to standard text. However, this aspect was not explored in the scope of this study.
For the RNN-based language models, we con- structed a network comprising an embedding layer, followed by a dropout layer (set to 0.333) (Gal and Ghahramani, 2016). This was then followed by 4 LSTM layers, each with 512 hidden units. ac-
and another dropout layer. The outputs tivated by a ReLU activation function and passed dense layer for prediction. We utilized cross- entropy with softmax activation as the loss func- tion and employed the Adam optimizer for network optimization. We set the initial learning rate to 0.001 and decayed it by half when there was no improvement for an entire epoch. Training was performed for 100 epochs, and carly stopping was implemented when there was improvement in the validation loss for consecutive epochs. Addi- tionally, we employed output and embedding tying techniques similar to those described in (Press and Wolf, 7; Inan et al.).
3.3.1 Text classification
For text classification, we explored two types of tasks: binary sentiment analysis and multiclass text classification. For binary text classification, we utilized the IMDB reviews dataset (Maas et al., 2011). This dataset comprises 25,000 samples for training and 25,000 samples for testing. From the training set, we randomly allocated 10% of the samples for the validation set. For multiclass clas- sification, we employed the AGNews benchmark (Zhang et al., 2015). This benchmark consists of four news classes, with 120,000 samples for train- ing and 7,600 samples for testing. Similar to the IMDB dataset, reserved 10% of the training set for validation.
For the transformer-bant based language model, implemented two decoder layers with two attention heads with dropout of 0.2. The feed-forward size is 200. The optimizer used is SGD with a decay fac- tor of 0.25 if the validation loss does not improve for 1 epoch. The training stops if there is no im- provement for 5 consecutive epochs. To overcome potential gradient explosion, we clipped gradient norms 0.25.
We employed a similar architecture for both the datasets, consisting of a 4-layer bidirectional LSTM with 256 hidden units each and embedding layers of size 512. We applied dropout rate of 0.4 to the embeddings layer and 0.333 to the LSTM lay- ers. For the sentiment analysis task, used binary cross entropy loss with sigmoid activation. For the multiclass text classification task, we utilized cross entropy loss with softmax activation. Similar steps were applicd for clcaning and preprocessing in both experiments. In the cleaning phase, HTTP links, HTML tags, numbers, punctuation marks, emojis, and non-ASCII characters were removed from the text. The text was further processed by removing stopwords and lemmatizing the remaining words using the NLTK toolkit (Loper and Bird, 2002).
For the RNN-based and the transformer-basee models, we experimented with two tokenization schemes, word-based tokenization, as well as sub- words tokenization using BPE implementation from sentencepiece (Kudo and Richardson, 2018).
3.3 NLP tasks
We expanded our investigation on these represen- tations by comparing their performance with stan- dard English text across various NLP tasks, employ- ing different settings and configurations. These tasks include binary and multiclass text classifi- cation, POS tagging, NER and NMT. Language modeling serves as an upstream task for many downstream tasks such as spell correction, auto- matic speech recognition (ASR), machine trans-
3.3.2 Sequence labeling
Sequence labeling category of tasks where the input is a sequence of tokens, and the output is another sequence of tokens with the same length. These tasks, also known as token classification tasks, involve assigning a label to each token

---

the input sequence, hence the name. In our study, we focused on two sequence labeling tasks: POS tagging and NER.
3.4 Retrieving standard English representation from the proposed representations
For POS tagging, we utilized the Universal De- pendencies project (Nivre et al., 2020) tags, These tags consist of total of 17 labels that can be used in multilingual settings. We trained our model on the English Web Treebank (EWT) dataset. The dataset comprises 254,825 words from 16,621 sen- tences. has already been split into training, val- idation, and testing sets. For NER, we utilized the amended version of the CONIL22003 dataset (Tjong Kim Sang and De Meulder, 2003), which was corrected by (Wang et al., 2019) and referred to as CoNLLpp. The CoNLLpp dataset consists of total of 20,744 samples, with 14.041 samples allocated for training, 3,250 samples for validation, and 3,453 samples for testing. The dataset contains a total of 301,418 words. In this dataset, there are different named entities. The model configuration used for training on the CoNLLpp dataset is similar to the one implemented for the text classification task, with the exception of the embedding dropout, which is set to 0.5. Additionally, the output size of the model is equal to the number of classes specific each task.
We propose sequence-labeling-ban baase approach to retrieve the standard text from the proposed repre- sentations. This retrieval is crucial for tasks where human-readable text is required. We trained our system on the AGnews dataset, where the input texts are in our representations and the output text is in standard English. For the word-based sequence- labeling approach, we used models similar to those used in the text classification task. The models con- sist of 2 layers of bidirectional LSTMs with 512 hidden units each, and an embedding layer with dropout of 0.25. The output layer is softmax with the size equal to the vocabulary of the standard English text computed from the training set of the dataset.
For evaluating the performance of the models, we used the word error rate (WER) and character error rate (CER) as metries. Additionally, specif- ically analyze the performance on vowels, we in- troduced a new metric called the vowels error rate (VER). In this metric, we exclude the consonant characters while calculating the character error rate as mistakes can only be made on the vowel charac- ters.
3.3.3 Translation
Translation is an interesting task to study for this problem. We implemented a mansformer-based translation model translate from English (en) German (de), and vere-versa. The tokenization method used for this experiment is word-based to- kenization. To model the unknown tokens, we re- moved tokens that occur only once from the tar- get sequence. The model is trained on Multi30K dataset (Elliott et al., 2016) which is an English- German image description parallel dataset. The dataset consists of 31.1K samples. IK of these samples are reserved for testing, and another 1K samples are reserved for validation. We used the split provided by (Bentrevett). The employed ar- chitecture is a transformer model consisting of one encoder layer as well as one decoder layer with 8 multi-head attention, 2048 latent dimension, and 256 embedding size. The batch size used is 64. The optimizer used is RMSProb with sparse cat- egorical cross-entropy loss. The model is trained for 10 epochs saving the checkpoint that achieved the lowest validation loss during training. The met- ric used to analyze the model performance is the 4 4-gram BLEU score.
Results and Discussion
This section presents the results of the experiments conducted in this study. We begin by presenting the text analysis on the text corpora, followed by discussion of the performance of the proposed representations compared to standard text the context of language modeling. Next, we compare the performance of the proposed representations with standard English text across various NLP tasks. Additionally, we report the results on retricval of the standard text representation from the proposed vowels-free representations. Finally, we conclude this section by providing further comparisons be- tween these representations.
4.1 Text analysis
In Table 1, we present summary of the vocabu- lary and token statistics at the word level for all the datasets (training sets) used in our experiments for the three text representations. The table reveals that dropping vowels results in a reduction of vo- cabulary size by at least 15% on the training sets, with an average reduction of 18%. This reduction

---

ratio tends to increase as the dataset size increases. as evidenced on the Wikitext dataset, where the reduction reaches approximately 23%. A similar pattern is observed for masked vowels, although the reduction is less significant, with a maximum of around 10% and an average of around 7%.
is significantly reduced as vowels comprise more than a third of the text (approximately 38% in the datasets we are using).
4.2 Language Modeling
Table presents the perplexity results for the lan- guage modeling (LM) tasks. From the table, we observe that, for n-gram LMs, the standard text has the highest perplexity, followed by the masked- vowels representation, and then the consonants- only representation. This pattern holds true across all the ngrams. The results are inline with the vo- cabulary sizes for the three representations. In contrast to the n-gram models, the RNN-based language models exhibit significantly lower per- plexities, highlighting their power and capacity in language modeling tasks. Interestingly, there is minimal difference in perplexity among the three representations in this experiment. In fact, the consonants-only representation shows slightly higher perplexity compared to the standard repre- sentation. This can be attributed to the absence of words like 'a" and 'I' in the consonants-only representation, which may pose a slight challenge in predicting the next words in their absence. The perplexity result for the standard text is the low- est for the transformer-based model showcasing the unique capabilities of transformers this task. Following similar pattern RNN-based models, yet more notable, the difference between the three representations almost negligible.
Regarding token size, the consonants-only repre- sentation experiences a decrease of approximately 4% caused by the omission of words consisting of vowels-only characters, except for the Multi30K dataset where this reduction is relatively large com- pared to other datasets. This is due to the nature of this dataset, as describing images extensively uses the "a" determiner. In fact, the determiner "a" and the pronoun "T" are the most prominent examples of such tokens. The smallest decrease in token size is observed in the Wikitext dataset, with only a 2% reduction. This discrepancy may be attributed to the fact that the dataset contains various charac- ters besides English letters, including punctuation marks and characters from other languages. We intentionally did not preprocess this dataset by re- moving these characters to maintain its originality and reflect worldw language usage, stated by the dataset authors (Stephen al., 2017).
In Table 2, we present the entropy of the text at the word and character levels. In terms of word entropy, we observe that the consonants represen- tation has the lowest entropy except for Multi30K due to the removal of the determiner "a" as previ- ously discussed, followed by the masked-vowel. representation, and then the standard English text. This result is expected considering the vocabulary size of each representation.
For the subwords as tokens, Table 4 presents the results for RNN-based and Transformer-basee models. The vocabulary sizes in these experiments are 20,1 10, 13,041, and 17,247 for standard-text, consonants, and masked- vowels representations, spectively. The results of this experiment are inline with the results from the word-based tokenization. RNN- based and transformer-basee models show- case even a smaller difference between the three text representations regardless of the notable differ- ence in the vocabulary size.
In terms of entropy at the character level, we observe different pattern compared to the word level. The masked-vowels representation has the lowest entropy of all the three representations. This may seem counter-intuitive but it can be explained by the fact that, on average, one or two consonant letters are followed by one or two vowels within a single word in the masked-vower vowels representation. By masking all vowels with single character, the task of predicting the next character becomes eas- ier, resulting in reduction in the entropy of the entire text. The consonants-only representation ex- hibits a character entropy that is close to that of the standard English text, despite the fact that almost a fifth of the character set has been dropped (i.e., the 5 vowels out of the 26 characters). However, it should also be noted that the length of the text
4.3 NLP tasks
Table 5 provides an overview of the results obtained from sentiment analysis on the IMDB dataset and multiclass text classification on the AGNews dataset. The table indicates that the proposed rep- resentations exhibit competitive performance com- pared to the standard English text on both datasets. Notably, the consonants representation achieves slightly higher accuracy than the standard text in

---

Vocabulary size IV
Tokens (N)
Text Corpora
Vs
Vc
VM
Vc/Vs
VM/ VM/VS Vs
Ns
Nc
Nc/Ns
Wikitext
33,277
25,538
30,090
0.77
0.90
2.265,796
2,221,913
0.98
IMDB
280,617
231,543
260,846
0.83
0.93
5,844.680
5,613,725
0.96
AGNews
188.110
154,700
173,749
0.82
0.92
4,541.694
4,430,395
0.98
EWT
32,273
27,305
30,376
0.85
0.94
199,040
191,482
0.96
CoNLLpP 26,883
22,420
25,250
0.83
0.94
254,983
250,387
0.98
Multi30k 15,456
12,844
14,394
0.83
0.93
345,020
295,833
0.86
Table I: Summary of the vocabulary and token statistics (at word level) comparing proposed representations with the standard text. For Multi30K, we only considered the English portion of the dataset. (Please note: from here on, S represents the standard text, C represents consonants-only. and M represents masked-vowel representations)
Dataset
Accuracy
Corpora
Hwd
Hoh
S
C
M
S
C
M
C
M
IMDB 85.17
83.68
84.68
Wikitext
10.2
9.8
10
4.8
4.7
4
AGNews
91.70
91.80
91.67
IMDB
11.2
10.8
11
4.7
4.5
3.8
AGNews
12.1
11.6
11.8
4.9
4.7
4
Table A summary of the text classification results using the three representations.
EWT
11.1
10.6
10.8
4.8
4.6
3.9
CoNLLpP
10.8
10.4
10.7
4.9
4.2
Accuracy
Multi30k
8.9
8.5
4.4
4.1
3.5
Task (Dataset)
S
C
M
Table 2: Text entropy at word and character levels on the training split of each dataset. Hud and Hehh the
POS Tagging (EWT)
91.44
87.26
91.16
NER (CoNLLpp)
93.97
93.88
94.54
entropy at word and character levels, respectively. For Multi30K, we only considered the English portion of the dataset.
Table 6: Summary of the results on the sequence label- ing tasks.
the AGNews dataset. These results suggest that vowels contribute minimal information in text clas- sification tasks, and comparable performance be achieved using the proposed representations, of- fering the advantages of smaller model size and faster training. Further discussions on this topic are presented Section 4.5.
Perplexity (PPL)
Model
C
M
2-gram
515.89
451.74
481.18
3-gram
444.02
388.08
411.84
4-gram
434.92
379.16
402.76
5-gram
433.22
377.49
401.01
6-gram
432.90
377.21
400.73
In Table 6, we present the results of the POS tagging and NER using the EWT and CoNLLpp datasets, respectively. The reported results in terms of accuracy. In the POS tagging exper- iment, the consonants-only representation exhibits a significant decrease performance by more than 4% compared to the standard text representation. However, the masked-vowels representation main- tains a comparable performance to the standard text. To investigate the reasons behind the performance drop, manual analysis conducted using the confusion matrix. The analysis revealed that the consonants-only representation lacks words like 'a" and 'I', which constitutes significant number of tokens. These words are easy to predict for POS tagging as determiner and pronoun, respectively.
RNN LM
102.47
106.83
100.29
Transformer LM
94.90
94.60
95.79
Table 3: Perplexity results from the statistical n-grams and neural language models on the test set of Wikitext dataset.
Perplexity (PPL)
Model
S
C
M
RNN LM
8.33
8.20
8.17
Transformer LM
7.52
7.50
7.52
Table 4:
Language models perplexity results on sub- words tokenization

---

BLEU
WER
CER
VER
Task
S
C
M
Consonants
8.99
4.06
5.60
English-to-German
30.56
29.70
28.19
Vowels
2.87
1.78
1.87
German-to-English
29.70
25.19
29.80
Table Standard text retrieval results. WER is the word-error-rate, CER is the character-error-ratee, and VER is the vowel-error-rate.
Table 7: Translation results in BLEU score
This constitutes the main reason for the drop in per- formance for the consonants-ond representation. NER, on the other hand, shows competitive per- formance compared to the standard English text. In fact, the masked-vowel representation outper- forms the standard text representation by reducing the absolute error by over 0.5%, while consonants- only representation provides a comparable perfor- mance.
plementing a post-processing step, such as spelling and grammar correction, to refine the output.
To evaluate the model performance while dis- regarding words that consist solely of vowels, we caleulate the aforementioned metrics after remov- ing these tokens from both the predicted output and the original text. The resulting metries are as follows: 6.4% for word-error-rate, 3.15% for character-crror-ruu and 3.85% for vowels-error- rate, The approximately 3% difference in word- error-rate aligns with the reported ratio of conso- nant tokens to standard text tokens presented in Table for the AGNews dataset.
Table 7 presents the translation results from En- glish German (en-de) and vise-versa. Based on the results from the table, we can see that all the representations are giving us comparable results on the (en-de) translation task. For the (de-en) translation task, we can notice that the consonant- only approach witnesses a noticeable drop. This can be attributed mainly to the missing determi- nant "a" as discussed in Section 4.1. Interestingly, masked-vowels representation yields better results than the standard English representation. It should be noted that the current model is optimally calibrated on the standard English representation. Tuning it on the proposed representations might lead to even better performances.
To further investigate the errors in vowel retrieval from both the representations, we manually ana- lyzed a subset of the first 100 samples from the test set. Our observations revealed the following insights:
Masked-vowell representation: Some errors occurred due to the presence of unknown words-words in the test set that were not present the training set. Since the model was not exposed these words during train- ing, it struggled to accurately retrieve the vow- els. Additionally, we noticed that a signifi- cant portion of the errors stemmed from nouns. These nouns might have been relatively rare in the training set, resulting in lower model fa- miliarity and increased difficulty in predicting the correct vowels for these words.
4.4 Retrieval of standard English representation from the proposed representations
this section, we focus on the task of retrieving vowel information from the two vowel-free rep- resentations introduced in this study. We utilize the AGnews dataset for training and evaluating the models. Table 6 presents the results of standard text retrieval from the proposed representations, mea- sured in terms word-error-rate (WER), character- error-rate (CER), and vowels-error-rate (VER).
Consonants-only representation: In the consonants representation, observed additional sources of errors in vowel retrieval. These include errors relate abbreviations and cases of letters, as well as challenges in capturing vowel positions within words.
The table indicates that retrieving standard text from the masked-vowels representation yields su- perior results compared to the consonants-only rep- resentation. It is worth noting that part of the errors encountered during the retrieval from consonants- only representation can be attributed to the omis- sion of words containing only vowels. However, we believe that such errors can be mitigated by im-
4.5 Further comparisons
This section presents further comparisons of the proposed representations compared to the standard English text. Table shows the model sizes of each dataset represented with our proposed repre- sentations compared to the standard English text.

---

Number of parameters (M)
comprehensive analysis of these representations by comparing their corpus statistics, including vo- cabulary and token sizes, text entropy, as well as the language model perplexity. Additionally, we evaluated the performance of these representations on various NLP tasks and compared the results with those obtained using the standard English text representation.
Dataset
S
C
M
Wikitext
25.5
21.5
23.8
IMDB
32.5
25.2
29.3
AGNews
32.8
25.4
29.5
EWT
16.4
14.3
15.6
ConLLpp
18.4
16.4
17.7
Multi30K
16.2
15.0
15.7
The experimental results demonstrated that the proposed representations achieved competitive per- formance without significant sacrifices. Notably, the models based on these representations exhib- ited smaller model sizes and a reduced number of parameters. This indicates the potential for more efficient training and deployment of models using these representations.
Table Models sizes differed text representations in terms in the number of parameters (in Millions). For Multi3KK, the reported results are for de-en experiment.
The table illustrates that the model size is closely related to the vocabulary size, as the embedding layers have a dominant effect on the model size. For example, the language models have sizes of 101 MB, 86 MB, and 95 MB respectively for the standard English, consonants-only, and masked- vowels representations. Additionally, the model size on storage devices also influenced by its total number of parameters.
In future work, focus will be on optimiz- ing the architectures specifically for the proposed representations to achieve superior performances compared to the standard English text. We aim to evaluate the representations on a broader range of NLP tasks, such as text summarization and ques- tion answering, to assess their generalization abil- ities. Further investigation will be conducted improve vowel retrieval.
Morcover, a lower model size indicates lower training time per epoch. In the language model ex- periment, for example, the training time per epoch is 82.14 seconds for the standard text, 67.55 sec- onds for the consonants-only representation, and 77.07 seconds for the maokes-vowels representa- tion. Thus, the reduced model size contributes to faster training.
Limitations
is crucial conduct additional experiments evaluate the generalization capabilities of the pro- posed representations, particularly in important NLP tasks such as question answering and text sum- marization. Moreover, the present work would ben- efit from exploring larger and more diverse datasets to validate the effectiveness of the representations across different domains. Additionally, to improve the accuracy of retrieving the standard text from the consonants-only representation, to investigate the use of an encoder-decode ar- chitecture that can deal with missing vowels-only words, as the presented architecture was unable to deal with these missing words. This will con- tribute to enhancing the overall performance and applicability of the proposed representations.
Another aspect of comparison is the Out-Of- Vocabulary (OOV) rate and unknown tokens. In some tasks, it is common to remove words with single occurrence to reduce model size and avoid overfitting. In the AGNews dataset, the number single-occurrence vocabularies 27,348 out of 79,037 for the standard text representation. For the consonants-only representation, the number of these vocabularies is 17,283 out of 54,596, and for the masked-vowels representation, it is 22,660 out of 67,982. Accordingly, the ratio of these unknown vocabularies 34.60%, 31.66% and 33.33% for the standard text, consonants- representation, and masked-vowels representation, respectively.
recommended
Acknowledgment
5 Conclusion and future work
The authors would like to thank Saudi Data and AI Authority (SDAIA) and King Fahd University of Petroleum and Minerals (KFUPM) for supporting this work through SDATA-KFUPM Joint Research Center for Artificial Intelligence grant number JRC- AI-RFP-06.
In this paper, we introduced novel text repre- sentations based on English consonants. The first representation completely removes vowels from the input sequence, while the second representation masks vowels to a unique symbol. We conducted

---

References
Yarin Gal and Zoubin Ghahramani. 2016. A theoret- ically grounded application of dropout in recurrent neural networks. Advances in neural information processing systems, 29.
Zaid Alyafeai, Maged S Al-shaibani, Mustafa Ghaleb, and Irfan Ahmad. 2022. Evaluating various tokeniz- ers for arabic text classification. Neural Processing Letters, pages 1-23.
Amir Gholami. Schoon Kim, Zhen Dong, Zhewei Yao, Michael Mahoney, and Kurt Keutzer. 2021. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2 13630.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long- regg-temm dependencies with gradi- ent descent is difficult. IEEE transactions on neural networks, 562):157-166.
Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pages 187-197.
Bentrevett.
bentreveet/multt30kk datasets at hug- ging face. https://hugging:: nggace co/dataseta bentrevett/multikkk (Accessed on 10/16/2023).
Kenneth Heafield, Ivan Pouzynevsky, Jonathan H Clark, and Philipp Kochn. 2013. Scalable modified kneser- ney language model estimation. In Proceedings of the 51st Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 690-696.
James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2017. Quasi-recurrent neural net- works. International Conference on Learning Rep- rexentations.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:7797-1
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in neural network. arXiv preprint
Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In International Conference on Learning Representations.
Alexis Conneau, Holger Schwenk, Yann Le Cun, and
Löe Barrault. 2017. Very deep convolutional net-
works for text classification. In 15th Conference of
Taku Kudo. 2018 Subword regularization: Improv- ing neural network translation models with multiple subword candidates. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
the European Chapter of the Association for Compu-
tational Linguisticx, FACI. 2017, pages 100--1116. Association for Computational Linguistics (ACL).
Peter T Daniels. 2013. The arabic writing system. In
Jonathan Owens, editor, The Oxford handbook of Ara- bic linguisties, chapter 18, pages 422-431. 431. Oxford University Press, New York.
Taku Kudo and John Richardson. 2018 SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedingy of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguisties.
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Edward Loper and Steven Bird. 2002. Nltk: the natu- ral language toolkit. In Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computa- tional linguistics- Volume 1, pages 63-70.
Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun iicii Tsu- jii 2020. Characterbert: Reconciling elmo and bert for word-level open- vocabulary representations from characters. In International Conference on Computa- tional Linguistics, pages 6903-6915.
Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. 2020. CharBERT: Character- aware pre- praitrdined language model. In Proceedings of the 28th International Conference on Computa- rional Linguistics, pages 39-50, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Desmond Elliott Stella Frank, Khalil Sima'an an, and Lu- cia Specia. 2016. Multi30k: Multilingual english- german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70-74. Association for Computational Linguistics.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human
Philip Gage. 1994, A new algorithm for data compres- sion. C Users Journal, 12(2):3338.

---

Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin-
Erik F. Tjong Kim Sang and Fien De Meulder, 2003. Introduction to the CoNLL-2003 shared task: imguage-independerr named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142- 147.
guistics.
Elman Mansimov, Mitchell Stern, Mia Chen, Orhan Fi- rat, Jakob Uszkoreit, and Puneet Jain. 2020. Towards end-to-end in- -n-i neural machine translation. In Proceedings of the First International Workshop on Natural Language Processing Beyond Text, pages 70-74, Online. Association for Computational Lin- guistics.
Zihan Wang. Jingho Shang, Liyuan Liu, Lihao Lu, Ji- acheng Liu, and Jiawei Han. 2019. Crossweigh: Training named entity tagger from imperfect anno- tations. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNI.--JJCNLP) pages 5157-5166.
Sabrina J Mielke, Zaid Alyafeai, Elizabeth Salesky. Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y Lee, Benoît Sagot, et al. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in nip.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in Neural Information Pro- cessing Systems, volume 28. Curran Associates, Inc.
arXiv preprint arXiv:21 12. 10508.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean, 2013. Efficient estimation of word representa- tions in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scoussdule Arizona, USA, May 2-4, 2013, Workshop Track Pro- ceedings.
George Kingsley Zipf. 2016. Human behavior and the principle of least effort: An introduction to human ecology. Ravenio Books.
Joakim Nivre, Marie- Catherine de Marneffe, Filip Gin- ter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Langage Resources and Evaluation Conference, pages 4034-4043, Marseille, France. European Language Resources Association.
Ofr Press and Lior Wolf. 2017. Using the output em- bedding to improve language models. In Proceedings of the 15tth Conference of the European Chapter of the Association for Computational Linguistics: Vol- ume 2, Short Pupers, pages 157-163, Valencia, Spain. Association for Computational Linguistics.
Vikas Raunak, Vivek Gupta, and Florian Metze. 2019. Effective dimensionality reduction for word embed- dings. In Proceedings of the 4th Workshop on Rep- resentation Leurning for NLP (RepLANLP-2015) pages 235-243.
Elizaheth Salesky, David Euteer and Matt Post. 2021. I. Robust open-vocabulary translation from visual text representations. In Proceedings of the 2021 Confer- ence on Empirical Methods Natural Language Pro- cessing, pages 7235-7252, Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Claude E Shannon. 1951.
Prediction and entropy of printed english. Bell system technical journal, 30(1):50-64.
Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast wordpiece tokenization.
Merity Stephen, Xiong Caiming, Bradbury James, and Richard Socher. 2017. Pointer sentinel mixture mod- els. Proceedings of ICLR.


Reviews:
[{"review_id": "b7rT2zlrml", "reviewer": "Reviewer_yLpJ", "summary": "", "questions": "see above pls", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his/her efforts in reviewing the paper and for providing constructive comments and feedback.\nBelow we provide a point-by-point rebuttal to the concerns raised:\n\n***Comment #1:***\n\nReducing the vocabulary size is one way of reducing the size of embedding, however, there are other alternatives such as dimensionality reduction (Raunak et al. 2019), quantization (see some works here (Gholami et al. 2021)), bloom embedding (Serra & Karatzoglou 2017), distillation networks (Hinton et al. 2015), etc. This work should be compared against some of these related baselines to show its true potential as an innovative approach for embedding compactness.\n\n*Raunak, V., Gupta, V., & Metze, F. (2019, August). Effective dimensionality reduction for word embeddings. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019) (pp. 235-243).\n*Serrà, J., & Karatzoglou, A. (2017, August). Getting deep recommenders fit: Bloom embeddings for sparse binary input/output networks. In Proceedings of the Eleventh ACM Conference on Recommender Systems (pp. 279-287).\n*Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2021). A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630.\n*Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.\n\n***Reply to Comment #1***\n\nWe thank the reviewer for pointing us to important works done in this area. We believe that these techniques are complementary to what we are proposing, i.e., they can be applied after our text representation scheme which can potentially lead to further embedding size reduction. Furthermore, in order to apply this, we will first have to reproduce the original work which was not feasible in this short period of time. We will update our related work and other relevant sections to address this. As a future work, we should point to applying these techniques over our proposed text representations. Moreover, we would like to point out that reducing the embedding size is one of major goals but there are other aspects to our work such as reduction of OOV rate and training time.\n\n***Comment #2:***\n\nThe perplexity experiments are carried out on obsolete language models (n-gram HMM, RNN) that are rarely used nowadays. To better align the paper with current NLP trends, I believe the authors should showcase their approach using transformer-based (masked) language models.\n\n***Reply to Comment #2:***\n\nWe thank the reviewer for this valuable comment. We implemented a transformer-based language model on wikitext-2 dataset using both word and subword tokenization schemes. The updated results (PPL) are shown in the table below (part of Table 3 in the original manuscript):\n\n|     Tokenization    |     Model           |     Normal English    |     Consonants    |     Masked Vowels    |\n|---------------------|---------------------|-----------------------|-------------------|----------------------|\n|     Words           |     Transformers    |     94.9              |     94.59         |     95.79            |\n|                     |     LSTMs           |     105.61            |     108.44        |     103.56           |\n|     Subwords        |     Transformers    |     7.52              |     7.50          |     7.52             |\n|                     |     LSTMs           |     8.33              |     8.20          |     8.17             |\n\nSubword vocabulary size in:\nStandard English: 20,110, \nConsonants-only representation: 13,041, and \nMasked-vowels representation: 17,247.\n\nWe can observe from that table that transformer-based language models show better perplexity as compared to other LMs. Moreover, both our text representations (with reduced character set) show similar perplexity as compared to the original representation thereby implying that little information is lost by using our modified representations. We plan to update our manuscript with these new experiments and results.\n\n***Comment #3:***\n\nThe reliance of this approach on a secondary step (vowel-retrieval) to make the text human-readable again could limit its applicability. It would be interesting to see how this representation would perform on generation tasks such as translation or summarization. Since the vowel-retrieval process is not loss-less (word-error-rate 9 for consonant-only and ~3 for masked-vowel representations), it may cause a drastic drop in the performance of the models on such tasks.\n\n***Reply to Comment #3***\n\nAs pointed out by the reviewer, vowel retrieval is a secondary step which needs to be applied in some applications but not all. Based on the reviewer’s feedback, we did perform machine translation tasks using Multi30k benchmark dataset [1]. Below are the results (4-gram BLUE scores) for two separate transformer-based translation models, one from German to English based on the reviewer feedback and the other one is from English to German to further explore the potential of the proposed representations.\n\n|  Translation Task | Normal English | Consonants | Masked Vowels |\n|:-----------------:|:--------------:|:----------:|:-------------:|\n| English-to-German |      30.56     |    29.70   |     28.19     |\n| German-to-English |      29.70     |    25.19   |     29.80     |\n\nBased on the results from the table, we can see that all the different representations are giving us comparable results on the English-to-German (en-de) translation task. For the German-to-English (de-en) translation task, we can notice that the consonants approach witnesses a noticeable drop. This might be the result of this representation lacking important words like the particle ‘a’ and the pronoun ‘I’. Interestingly, Masked-vowels representation on German-to-English (de-en) translation task yields even better results than the standard English representation. It should be noted that the current model is optimally calibrated on the standard English representation. Tuning it on the proposed representations might lead to a better performance. The details of experiments and the results will be added in the updated manuscript. Moreover, higher WER in our vowels-retrieval process can be reduced by using dictionary and spell-checking. \n\n[1]: Elliott, Desmond, et al. \"Multi30K: Multilingual English-German Image Descriptions.\" Proceedings of the 5th Workshop on Vision and Language. 2016.\n\n***Comment #4***\n\nIn addition, this extra vowel-retrieval step would add to the required computational steps and may actually increase the computational requirements (as opposed to the paper’s claim on saving on computational resources).\n\n***Reply to Comment #4***\n\nWe thank the reviewer for pointing this out. Please note that this retrieval step should be applied to a selective set of tasks and not to every NLP tasks. Moreover, the retrieval system is a common system across all the tasks which should be trained only once.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed idea is interesting and straight forward to implement and can be easily integrated with existing NLP models and frameworks.\nThe paper is well-written and easy to read.", "reasons_to_reject": "1) Reducing the vocabulary size is one way of reducing the size of embedding, however, there are other alternatives such as dimensionality reduction (Raunak et al. 2019), quantization (see some works here (Gholami et al. 2021)), bloom embedding (Serra & Karatzoglou 2017), distillation networks (Hinton et al. 2015), etc.\nThis work should be compared against some of these related baselines to show its true potential as an innovative approach for embedding compactness. \n\n*Raunak, V., Gupta, V., & Metze, F. (2019, August). Effective dimensionality reduction for word embeddings. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019) (pp. 235-243).\n\n*Serrà, J., & Karatzoglou, A. (2017, August). Getting deep recommenders fit: Bloom embeddings for sparse binary input/output networks. In Proceedings of the Eleventh ACM Conference on Recommender Systems (pp. 279-287).\n\n*Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2021). A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630.\n\n*Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.\n\n2) The perplexity experiments are carried out on obsolete language models (n-gram HMM, RNN) that are rarely used nowadays. To better align the paper with current NLP trends, I believe the authors should showcase their approach using transformer-based (masked) language models.\n \n3) The reliance of this approach on a secondary step (vowel-retrieval) to make the text human-readable again could limit its applicability. It would be interesting to see how this representation would perform on generation tasks such as translation or summarization. Since the vowel-retrieval process is not loss-less (word-error-rate 9 for consonant-only and ~3 for masked-vowel representations), it may cause a drastic drop in the performance of the models on such tasks.\n\n4) In addition, this extra vowel-retrieval step would add to the required computational steps and may actually increase the computational requirements (as opposed to the paper’s claim on saving on computational resources).", "questions_to_authors": "see above pls", "paper_topic_and_main_contributions": "This work presents an approach for reducing the size of embedding by representing text only with consonants, or by masking vowels.\nThey show that such representation would reduce the size of vocabulary and thus the size of embeddings more that the drop in prediction accuracy on text and token classification tasks.\nThe paper also presents a technique to retrieve vowel information from the processed representation to create human-readable text output in NLP applications.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UpbckJ5Zlo", "reviewer": "Reviewer_6XDg", "summary": "", "questions": "* A. What do you think about the words that have opposite meaning with and without vowel, such as asynchronous and asymmetry?\n* B. Why the results of text retrieval task using the standard model are not shown in Table 6?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his/her efforts in reviewing the paper and for providing constructive comments and feedback. Below we provide a point-by-point rebuttal to the concerns raised:\n\n***Comment #1:***\n\nA. They tested the traditional n-gram models and neural models, but no results were shown on the recent methods of generative decoder models.\n\n***Reply to Comment #1***\n\nWe thank the reviewer for the comment. Based on the feedback, we implemented a transformer-based language model on wikitext-2 dataset using both word and subword tokenization schemes. The updated results are shown in the table below (part of Table 3 in the original manuscript):\n\n|     Tokenization    |     Model           |     Normal English    |     Consonants    |     Masked Vowels    |\n|---------------------|---------------------|-----------------------|-------------------|----------------------|\n|     Words           |     Transformers    |     94.9              |     94.59         |     95.79            |\n|                     |     LSTMs           |     105.61            |     108.44        |     103.56           |\n|     Subwords        |     Transformers    |     7.52              |     7.50          |     7.52             |\n|                     |     LSTMs           |     8.33              |     8.20          |     8.17             |\n\nSubword vocabulary size in:\nStandard English: 20,110, \nConsonants-only representation: 13,041, and \nMasked-vowels representation: 17,247.\n\nWe can observe from that table that transformer-based language models show better perplexity as compared to other LMs. Moreover, both our text representations (with reduced character set) show similar perplexity as compared to the original representation thereby implying that little information is lost by using our modified representations. We plan to update our manuscript with these new experiments and results.\n\n***Comment #2:***\n\nB. The overall benefit of this method is not shown. It is good if the trade-off between the accuracy and model size is compared with other distillation methods like corpus shrinking and layer/dimension reduction.\n\n***Reply to Comment #2:***\n\nThe overall benefits of the proposed representations come in many aspects like reductions in vocabulary size, OOV-rates, and training time. We will update our manuscript to further highlight these points. \n\nRegarding comparison with other distillation methods like corpus shrinking and layer/dimension reduction, we plan to add them in the related work section of the updated manuscript. For an empirical comparison, we will need to first reproduce the original results and then compare with our approach which was not possible in this short time. Moreover, it is worth noting that these techniques can also apply to our representations. It would be interesting to apply these methods on the proposed approach and investigate the results. We will update the manuscript to reflect this literature on the related work and future work.\n\n***Comment #3:***\n\nA. What do you think about the words that have opposite meanings with and without vowels, such as asynchronous and asymmetry?\n\n***Reply to Comment #3***\n\nThis is an interesting observation. However, these aspects are not new to natural language processing. As an example, polysemous words like ‘mouse’ [computer device vs animal] and ‘bank’ [financial institute vs river bank vs use of bank as a verb] are common in English but are effectively captured from the context. Our results related to language modeling and the downstream tasks gives us an indication that this approach seems to work. Contextual embeddings similar to BERT shall be able to capture many of these situations.\n\n***Comment #4***\n\nB. Why are the results of text retrieval tasks using the standard model not shown in Table 6?\n\n***Reply to Comment #4***\n\nIt seems there was some confusion related to the phrase “text retrieval”. By “text retrieval”, we meant retrieving the standard English representation from our proposed representations (vowel retrieval). We will rephrase the section title and captions in our updated manuscript to make it clearer.\n\n***Comment related to Typos Grammar Style And Presentation Improvements***\n\nWe thank the reviewer for pointing this out. We will correct the Table number in our updated manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* A. An interesting findings in the vowel-less representation: in some situation it works comparably with the original English representation.\n* B. Exhaustive experiments with multiple language modelings and multiple tasks.", "reasons_to_reject": "* A. They tested the traditional n-gram models and neural models, but no results were shown on the recent methods of generative decoder models.\n* B. The overall benefit of this method is not shown.  It is good if the trade-off between the accuracy and model size is compared with other distillation method like corpus shrinking and layer/dimension reduction.\n", "questions_to_authors": "* A. What do you think about the words that have opposite meaning with and without vowel, such as asynchronous and asymmetry?\n* B. Why the results of text retrieval task using the standard model are not shown in Table 6?", "paper_topic_and_main_contributions": "This paper presents English representation formats without vowels, and generated language models with the representations.  The experimental results show that the vowel-less representations have almost the same representation power and they work comparably with the model on the standard English in some application tasks.  The drop of vowels reduced the vocabulary size and it contributes to make a smaller models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "A. LL532: Table 5 -> Table 6\n"}, "extra_text": {}}, {"review_id": "1pYH2cVRQW", "reviewer": "Reviewer_bMmj", "summary": "", "questions": "- A: It seems complicated to apply the proposed method to generate text for humans because the output tokens need vowels. Is this correct?\n\n- B: Have the authors evaluated performances when using subwords?\n\n- C: Is the same trend actual when using Transformer, which would be the current standard instead of LSTM?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his/her efforts in reviewing the paper and for providing constructive comments and feedback. Below we provide a point-by-point rebuttal to the concerns raised:\n\n***Comment #1:***\n\nThe degree of effectiveness for reducing vocabulary size and number of model parameters is not so much.\n\n***Reply to Comment #1***\n\nWe thank the reviewer for the comment. According to table 1, the reduction of vocabulary size ranges from 15-23% for consonants-only representation (35% and 14% for consonant-only and masked-vowel subword representations, respectively). Furthermore, from Table 7, the model size reduction ranges from 11-22% for the consonants-only representation. These reductions might have significant implications in many tasks.\n\n***Comment #2:***\n\nSince only one architecture was experimented with, the generality for other model architectures needs to be clarified.\n\n***Reply to Comment #2:***\n\nWe thank the reviewer for the comment. We have performed additional set of experiments using transformer models on the language modeling as well as machine translation tasks.\n\nWe implemented a transformer-based language model on wikitext-2 dataset using both word and subword tokenization schemes. The updated results (PPL) are shown in the table below (part of Table 3 in the original manuscript):\n\n|     Tokenization    |     Model           |     Normal English    |     Consonants    |     Masked Vowels    |\n|---------------------|---------------------|-----------------------|-------------------|----------------------|\n|     Words           |     Transformers    |     94.9              |     94.59         |     95.79            |\n|                     |     LSTMs           |     105.61            |     108.44        |     103.56           |\n|     Subwords        |     Transformers    |     7.52              |     7.50          |     7.52             |\n|                     |     LSTMs           |     8.33              |     8.20          |     8.17             |\n\nSubword vocabulary size in:\nStandard English: 20,110, \nConsonants-only representation: 13,041, and \nMasked-vowels representation: 17,247.\n\nWe can observe from that table that transformer-based language models show better perplexity as compared to other LMs. Moreover, both our text representations (with reduced character set) show similar perplexity as compared to the original representation thereby implying that little information is lost by using our modified representations. \n\nWe performed machine translation tasks using Multi30k benchmark dataset [1]. Below are the results (4-gram BLUE scores) for two separate transformer-based translation models, one from German to English and the other one is from English to German to further explore the potential of the proposed representations.\n\n|  Translation Task | Normal English | Consonants | Masked Vowels |\n|:-----------------:|:--------------:|:----------:|:-------------:|\n| English-to-German |      30.56     |    29.70   |     28.19     |\n| German-to-English |      29.70     |    25.19   |     29.80     |\n\nBased on the results from the table, we can see that all the different representations are giving us comparable results on the English-to-German (en-de) translation task. For the German-to-English (de-en) translation task, we can notice that the consonants approach witnesses a noticeable drop. This might be the result of this representation lacking important words like the particle ‘a’ and the pronoun ‘I’. Interestingly, Masked-vowels representation on German-to-English (de-en) translation task yields even better results than the standard English representation. It should be noted that the current model is optimally calibrated on the standard English representation. Tuning it on the proposed representations might lead to a better performance. \n\nThe details of experiments and the results will be added in the updated manuscript.\n\n***Comment #3:***\n\nThere are no evaluation results for using commonly used subwords in the neural language model. Subwords also are compact text representations that can reduce the vocabulary size and OOV rate like the proposed method.\n\n***Reply to Comment #3***\n\nWe added subwords tokenization for the language modeling task. As presented in a previous comment (Comment #3).\n\n***Comment #4***\n\nA: It seems complicated to apply the proposed method to generate text for humans because the output tokens need vowels. Is this correct?\n\n***Reply to Comment #4***\n\nThanks for that comment. In fact, Section 4.4 of the manuscript presents a technique to retrieve vowels from our representation. Based on the conducted experiments with results shown in Table 6, we can notice that retrieving vowels gives promising results (CER ranging from 1.78-4.06). We will change the title of the section to better reflect the task.\n\n***Comment #5***\n\nB: Have the authors evaluated performances when using subwords?\n\n***Reply to Comment #5***\n\nWe performed an additional set of experiments using subwords as tokenization as detailed in reply to Comment#2.\n\n***Comment #6***\n\nC: Is the same trend actual when using Transformer, which would be the current standard instead of LSTM?\n\n***Reply to Comment #6***\n\nWe thank the reviewer for pointing this out. We performed additional sets of experiments using transformer models for language modeling and machine translation tasks as discussed above. Similar trends were observed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper shows a simple method that could be easily applied.\n- The effectiveness has been validated on a variety of NLP tasks.", "reasons_to_reject": "- The degree of effectiveness for reducing vocabulary size and number of model parameters is not so much.\n- Since only one architecture was experimented with, the generality for other model architectures needs to be clarified.\n- There are no evaluation results for using commonly used subwords in the neural language model. Subwords also are compact text representations that can reduce the vocabulary size and OOV rate like the proposed method.", "questions_to_authors": "- A: It seems complicated to apply the proposed method to generate text for humans because the output tokens need vowels. Is this correct?\n\n- B: Have the authors evaluated performances when using subwords?\n\n- C: Is the same trend actual when using Transformer, which would be the current standard instead of LSTM?", "paper_topic_and_main_contributions": "This paper proposes a method of representing English text with consonants for enabling to reduce vocabulary size, out-of-vocabulary rate, and training time. Specifically, they show two methods to remove vowels and mask vowels with a specific symbol.\n\nThe experiment results on several NLP tasks showed that the method using consonants gives comparable performances to using both consonants and vowels.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to represent English text with only consonants, in order to increase model efficiency, based on the assumption that consonants are more discriminative than vowels. The proposed idea is simple yet effective. However, Reviewers pointed out several problems in the experimental setup: 1) the paper currently lacks a comparison with alternative strategies to reduce the embedding matrix size, such as dimensionality reduction and quantisation (among others); 2) the neural architectures chosen for the language modelling experiment is obsolete and should include subword-based Transformers. (The Authors provided some first results in the rebuttal to this effect but these should be expanded to be significant). Moreover, the approach suffers from a major limitation, namely relying on retrieving back vowels to make generated text human-readable, which is error-prone and partly reduces efficiency gains.
Decision: Accept-Findings