# UPC: An Open Word-Sense Annotated Parallel Corpora for Machine Translation Study
## 2 Parallel Corpus Acquisition  
This paper introduces the Korean-English and Korean-Vietnamese parallel corpora UPC for use in training MT models. Korean-English and Korean-Vietnamese parallel corpora are built independently, but the building processes for both are similar, as shown in Figure 1.  
The initial step of the parallel corpora collection is to select resources that contain Korean, English, and Vietnamese sentences. Since the resources determine the quality of the parallel corpora, we had to select resources that have good alignment and literal translations. After carefully verifying the contents, the following resources were selected to build the parallel corpora:  
* \"National Institute of Korean Language's Learner Dictionary ([[https://krdict.korean.go.kr/eng/mainAction](https://krdict.korean.go.kr/eng/mainAction)]([https://krdict.korean.go.kr/eng/mainAction](https://krdict.korean.go.kr/eng/mainAction))) (NIKLLD)\" is an online dictionary containing definition statements of words in Korean and ten other languages, including English and Vietnamese. * \"Watchtowers and Awake! ([[https://www.jw.org/en/publications/magazines](https://www.jw.org/en/publications/magazines)]([https://www.jw.org/en/publications/magazines](https://www.jw.org/en/publications/magazines)))\", where \"Watchtowers\" is a monthly online magazine and \"Awake!\" is a bi-weekly online magazine. * \"Books and Brochures for Bible Study ([[https://www.jw.org/en/publications/books](https://www.jw.org/en/publications/books)]([https://www.jw.org/en/publications/books](https://www.jw.org/en/publications/books)))\" contains multilingual books in PDF format. * \"Danuri portal of information about Korean life\" ([[https://www.liveinkorea.kr](https://www.liveinkorea.kr)]([https://www.liveinkorea.kr](https://www.liveinkorea.kr))) contains a guide for living in Korea (HTML format) and a serial of multicultural magazines--Rainbow ([[https://www.liveinkorea.kr/app/rainbow/webzineList.do](https://www.liveinkorea.kr/app/rainbow/webzineList.do)]([https://www.liveinkorea.kr/app/rainbow/webzineList.do](https://www.liveinkorea.kr/app/rainbow/webzineList.do))) (PDF format). * Online News and Korean-learning websites that contain Korean, English, and Vietnamese texts available in HTML format. In the next step, we downloaded the PDF files and other contents from the websites. Except for NIKLLD, where we directly received word definition statements from the organization in charge. For the rest of the selected resources, we generated URL for each website and made a web spider to  
Figure 1: The acquisition process of UPC. crawl the contents and PDF files of these websites. The \"National Institute of Korean Language's Learner Dictionary\" contains the definition statements of Korean, English, and Vietnamese words aligned at the sentence level, which we directly received. Hence, we did not need to crawl texts nor align at the document or sentence level. The \"Watchtower and Awakel\" magazines are available on websites (i.e., HTML format). We first generated URLs based on the structure of the magazine over time. Then, we used these URLs to feed an automatic crawler that was developed based on the \"requests\" library of Python to download each URL and we used the \"BeautifulSoup\" library to parse the HTML format and extract the content in texts. Likewise, the automatic crawler was used to extract the content of \"Books and Brochures for Bible Study\" and the \"Danuri portal of information about Korean life\". However, their URLs were generated according to their URL structures. In the document-alignment stage, each URL of the magazines is an issue that contains several articles or topics. In this step, we separated topics by topic and matched them in each language pairs: Korean-English and Korean-Vietnamese. Then, we removed topics that only exist in one language. After which, the document pairs were correctly aligned. Data crawling from the website is in the form \"_<tag> content <flag>\"_.