Question: What corpus is chosen for the task？
Evidence: The corpus selected for this experiment was a combination of different corpora from various domains. The major part of the corpus was made up by the EnTam v2 corpus BIBREF2 . This corpus contained sentences taken from parallel news articles, English and Tamil bible corpus and movie subtitles. It also comprised of a tourism corpus that was obtained from TDIL (Technology Development for Indian Languages) and a corpus created from Tamil novels and short stories from AU-KBC, Anna university. The complete corpus consisted of 197,792 sentences. Fig. FIGREF20 shows the skinny shift and heatmap representations of the relativity between the sentences in terms of their sentence lengths.
An extra monolingual Tamil corpus, collated from various online sources was used for the word2vec embedding of the Tamil target language to enhance the richness of context of the word vectors. It was also used to create the language model for the phrase-based SMT model. This corpus contained 567,772 sentences and was self-collected by combining hundreds of ancient Tamil scriptures, novels and poems by accessing the websites of popular online ebook libraries in Python using the urllib package. Since the sources had Tamil text in different encodings, the encoding scheme was standardized to be UTF-8 for the entirety of the monolingual and parallel corpora using the chardet package. The corpora were cleaned for any stray special characters, unnecessary html tags and website URLs.
The corpus selected for this experiment was a combination of different corpora from various domains. The major part of the corpus was made up by the EnTam v2 corpus BIBREF2 . This corpus contained sentences taken from parallel news articles, English and Tamil bible corpus and movie subtitles. It also comprised of a tourism corpus that was obtained from TDIL (Technology Development for Indian Languages) and a corpus created from Tamil novels and short stories from AU-KBC, Anna university. The complete corpus consisted of 197,792 sentences. Fig. FIGREF20 shows the skinny shift and heatmap representations of the relativity between the sentences in terms of their sentence lengths.
An extra monolingual Tamil corpus, collated from various online sources was used for the word2vec embedding of the Tamil target language to enhance the richness of context of the word vectors. It was also used to create the language model for the phrase-based SMT model. This corpus contained 567,772 sentences and was self-collected by combining hundreds of ancient Tamil scriptures, novels and poems by accessing the websites of popular online ebook libraries in Python using the urllib package. Since the sources had Tamil text in different encodings, the encoding scheme was standardized to be UTF-8 for the entirety of the monolingual and parallel corpora using the chardet package. The corpora were cleaned for any stray special characters, unnecessary html tags and website URLs.
The corpus selected for this experiment was a combination of different corpora from various domains. The major part of the corpus was made up by the EnTam v2 corpus (Ramasamy, Bojar, and Zabokrtsky 2014). ****An extra monolingual Tamil corpus, collated from various online sources was used for the word2vec embedding of the Tamil target language to enhance the richness of context of the word vectors.
Answer: The corpus selected for this experiment was a combination of different corpora from various domains. The major part of the corpus was made up by the EnTam v2 corpus (Ramasamy, Bojar, and Zabokrtsky 2014), and an extra monolingual Tamil corpus, collated from various online sources was used for the word2vec embedding of the Tamil target language to enhance the richness of context of the word vectors.