Amharic Text Normalization with Sequence-to-Sequence Models
Authors: Seifedin Shifaw Mohamed
Solomon Teferra Abate (PhD)
Venue:  (2020)
Keywords: Text Normalization
Sequence-to-Sequence Model
Encoder-Decoder

Abstract: All areas of language and speech technology, directly or indirectly, require handling of real text. In addition to ordinary words and names, the real text contains non-standard words (NSWs), including numbers, abbreviations, dates, currency, amounts, and acronyms. Typically, one cannot find NSWs in a dictionary, nor can one find their pronunciation by an application of ordinary letter-to-sound rules. It is desirable to normalize text by replacing such non-standard words with a consistently formatted and contextually appropriate variant in several NLP applications. To address this challenge, in this paper, we model the problem as character-level sequence-to-sequence learning where we map a sequence of input characters to a sequence of output words. It consists of two neural networks, the encoder network, and the decoder network. The encoder maps the input characters to a fixed dimensional vector and the decoder generates the output words. We have achieved an accuracy of 94.8 % which is promising given the resource we use.

A mmaricc Text Normalization with Sequence-to-Sequence Models
1. Introduction
Text normalization or the transformation of words from the written to the spoken form is important problem. It is also quite a complex problem, due to the range of different kinds of non-standard words (NSW's) s), the special processing required for each case, and the propensity for ambiguity among NSW 's as class. Unfortunately, text normalization is not a problem that has received a great deal of attention.
2. Method
We have collected data that includes many types of non- noa-dtaddard words from different A mharic news Media and websites, FBC (Fana B Braaccatting Corporate) more than eighty percent, VOA (V oice of A merica) and BBC (British Broadcasting corporate) with four types of news domain, including political, economic, sport and health, which contributes to the inclusion of most of Amharic NSWs.
A After data gathering and preprocessing, the processed text is an input for our general architecture first component, tokenization, a process of segmentation of collected document into sentence-level and then to word level. Later every token has pass through token identification process that identifies its token class type, from which we are going to determine its expansion form for training data preparation.
Ambaric Text
Tokenization
Classification
Sequence to Sequence Modeling
Training Data
A/P
b>
<Eos>
Training With Class Label
Abbreviation
Telephone
Accronym
Money
Ordinal
Range
Time
Panctuation
Plam
Prediction
Normalized Words
Figure 1: A Framework for Amharic Text Normalization
Detecting the class of the token or the taxonomy is the key part of this task. Once we have determined the class of a token correctly, we expand it based on our rule- based normalizer accordingly. The usage of token in a sentence determines its class. To determine the class of the token in focus, the surrounding tokens play important role.
Finally, the training data is the input for architecture last component, the sequence- no-sequence architecture, which model the whole task as one where we map a sequence of input characters to sequence of output words.

---

Mah
Mn
Rinca
Softmax
Decoder
Encoder
M&h
Ann R.NCA
Embeding
ACCRONYM
Label
Tokens
Figure 2: The trrueene-to-Sqquenee A rchitecture
Our architecture performs at a character-level input sequence and word-level output sequence. We used an embedding layer, Bidirectional 3-layer Gated Recurrent Units (GRU) encoder that reads input tokens, and one-llaerr GRU decoder that produces word sequences. These are the translation for the architecture inputs and outputs, "X" for "A", "p" for "U", "Man" for "Addis", "XAa" for "A babab" and "P2NChh" for "University".
3. Experiment and Results
We use bidirectional GRU with the size of 250 hidden units both for encoding and decoding layers. The training takes 7 days on the whole 200 thousand sentences training data on 8GB RAM with 2.5 GHz intel core i5 Laptop. A cross all experiments, to reduce the input dimension, we choose character- based solution over word-based.
Class
Word Count
Accuracy
PLAIN
96,101
0.996
PUNC
7,629
1.000
ACRYM
1,320
0.998
ABR
1,402
0.986
CARD
3,430
0.926
ORD
1,275
0.969
MONEY
1,104
0.872
DATE
4,509
0,899
YEAR
2,880
0,859
TIME
4.690
0,861
TELE
1,606
0.895
RANGE
1.208
0.891
Table 1: Accuracy of Experiment 4
4. Conclusion
In this work, based on our final experiment results, we found 94.8 percent accuracy which is promising, considering that we trained on a small annotated dataset. Besides, on top of the corpus we prepared, we believe the performance of the model can be further improved by designing more balanced training dataset.

---

References
E. Flint, E. Ford, O. Thomas, and A. Caines, "A Text Normalisation System for Non-Standard English Words," Proc. 3rd Work. Noisy User-generated Text, vol. no 2, pp. 107-115, 2017.
I. Saito, J. Suzuki, K. Nishida, and K. Sadamitsu, "Improving Neural Text Normalization with Data A ugmentation at Character-and Morphological Levels," Proc. 8th Int. Jt. Conf. Nat. Lang. Process., pp. 257-262. 2017.
M. Zare and S. Rohatgi, "DeepNorm-A Deep Learning A Approach to Text Normalization, CoRR, vol, no.1712, 2017
R. Sproat and N. Jaitly, "RNN Approaches to Text Normalization: A Challenge," " Proc. 7th ACM Conf. Recomm, Syst. RecSys '13, 2016.
S. Pramanik and A. Hussain, "Text Normalization using Memory Augmented Neural Networks," no. 2017, 2018.
S. Demir, "Context Tailoring for Text Normalization," " pp. 6-14, 2016.


Reviews:
[{"review_id": "Skg1Jt4msB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a method for word normalization of Amharic text using a word classification system followed by a character-based GRU attentive encoder-decoder model.\n\nThe paper is very short and lacks many important details, such as where the data is collected from, how it is processed and split into training and evaluation sets, and how the initial token classification is performed. The paper also doesn't adhere to the conference paper template, which is grounds for desk rejection.\n\nThe authors should revise the paper with this information and consider submitting to a different venue, as the task considered, while interesting, seems far from the core focus of ICLR.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkgF8KJT9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Text normalization or the transformation of words from the written to the spoken form is an important and realistic question in natural language processing. This paper aims to use sequence-to-sequence models to perform text normalization.\n\nHowever, this paper does not use the official template and the content is too short to be a conference paper.\nI suggested resubmitting to another (NLP) conference after extending the content with detailed description for the model and the method, and conducting more experiments on public acceptable benchmarks.  ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "BkxPOKMe9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the text normalization problem, where  the special processing is required for different kinds of non-standard words (NSW’s).\n\nDataset:  a new dataset is collected, including many types of non-standard words from different Amharic news Media\nand websites, FBC more than eighty percent, VOA and BBC. \n\nModel: Bidirectional GRU with the size of 250 hidden units both are used for encoding and decoding layers.\n\nThis paper is not ready to publish. Please consider to complete the project, polish the writing, and submit to a different venue.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a text normalisation model for Amharic text. The model uses word classification, followed by a character-based GRU attentive encoder-decoder model. The paper is very short and does not present reproducible experiments. It also does not conform to the style guidelines of the conference. There has been no discussion of this paper beyond the initial reviews, all of which reject it with a score of 1. It is not ready to publish and the authors should consider a more NLP focussed venue for future research of this kind. 

Decision: Reject