SemSyn: Semantic-Syntactic Similarity Based Automatic Machine Translation Evaluation Metric

AbstractMachine translation evaluation is difficult and challenging for natural languages because different languages behave differently for the same dataset. Lexical-based metrics have been poorly represented semantic relationships and impose strict identity matching. However, translation and assessment become difficult for target morphologically rich languages with relatively free word order. Most of the standard evaluation metrics consider word order but do not effectively consider sentence structure. In this paper, we propose a novel machine translation evaluation metric SemSyn which incorporates both semantic and syntactic similarity. We incorporate the term frequency-inverse document frequency with the earth mover's distance and word embedding to cover the semantic similarity. The part of speech and dependency parsing tags assist in covering syntactic similarity in the sentence structure. Part of speech and dependency parsing tags are extracted from universal dependencies and trained on the SpaCy library. Experimental results show that SemSyn has a higher correlation with human judgment than other evaluation metrics for morphologically rich language and other languages.Keywords: Evaluation metricMachine translation evaluationSemanticSyntacticWord embeddings Disclosure statementNo potential conflict of interest was reported by the author(s).Additional informationNotes on contributorsShweta ChauhanShweta Chauhan pursued her BTech (H'ons) in ECE from HP University Shimla. master of technology in VLSI Design automation and Technique from the National Institute of Technology Hamirpur in the ECE Department. She is pursuing her PhD in natural language processing and low-resource languages from the National Institute of Technology Hamirpur (Himachal Pradesh). She has two years' experience in the software industry and seven years of teaching experience.Rahul KumarRahul Kumar received his B Tech in electronics and communication engineering from Gautam Buddha Technical University, Uttar Pradesh, India in 2011 and M Tech in microwave and optical communication from Delhi Technological University, Delhi, India in 2014. He has more than two years of teaching experience. Currently He is pursuing his PhD in wireless communication from NIT Hamirpur, Himachal Pradesh, India. His research interests include cognitive radio and wireless communication. Email: rahulmishra.dtu@gmail.comShefali SaxenaShefali Saxena is pursuing her PhD in natural language processing, statistical machine translation, low resource languages from the National Institute of Technology Hamirpur (Himachal Pradesh). Email: shefali@nith.ac.inAmandeep KaurDr Amandeep Kaur received the PhD degree from National Institute of Technology Hamirpur. Presently, she is working as an assistant professor, ABV-Indian Institute of Information Technology and Management Gwalior, India. Earlier, Dr Amandeep served Dr B R Ambedkar National Institute of Technology Jalandhar and National Institute of Technology Hamirpur as an assistant professor. Her areas of expertise are in the intersecting domains of artificial intelligence, machine learning and business intelligence. She has published in various IEEE Transactions and other SCI listed journals. Email: adeep5524@gmail.comPhilemon DanielPhilemon Daniel, PhD in electronics and communication engineering (NIT Hamirpur), MTech in VLSI Design (VIT Vellore), BE in electronics and communication engineering (Bharathidasan University). Daniel has over 13 years of teaching experience at NIT Hamirpur. Prior to joining NIT Hamirpur, he worked as design engineer at Sasken Communication Technologies Limited, Bangalore.His research interests include VLSI Testing, embedded systems, IoT, image and speech processing, natural language processing, self-driving cars, and deep learning architectures. He has mentored many students to win several national competitions. To name some of the few: 3rd in Mentor Graphics Design Contest (2011), 1st & 2nd in ARM Design Contest (2017), and 3rd in Anveshan by Analog Devices (2016). He gives regular talks on hardware and embedded system design, deep learning architectures, ARM processors and applications, and similar areas. He was awarded ARM Accredited Microcontroller Engineer (AAME) in 2015. He is the recipient of the NVIDIA GPU Grant in 2018. He is responsible for many industry institute partnerships with mentor graphics, analog devices, seimens, GE, and Xilinx to name a few. Email: phildani7@nith.ac.in