Omkar2118/datasets huggingface dataset:
WMT 2016

**WMT 2016** is a collection of datasets used in shared tasks of the First Conference on Machine Translation. The conference builds on ten previous Workshops on statistical Machine Translation.

The conference featured ten shared tasks:

- a news translation task,
- an IT domain translation task,
- a biomedical translation task,
- an automatic post-editing task,
- a metrics task (assess MT quality given reference translation).
- a quality estimation task (assess MT quality without access to any reference),
- a tuning task (optimize a given MT system),
- a pronoun translation task,
- a bilingual document alignment task,
- a multimodal translation task.

Source: [http://www.statmt.org/wmt16/index.html](http://www.statmt.org/wmt16/index.html)

Introducing paper: Findings of the 2016 Conference on Machine Translation
WMT 2016 News
Full name: WMT 2016 News Translation Task

News translation is a recurring WMT task. The test set is a collection of parallel corpora consisting of about 1500 English sentences translated into 5 languages (Czech, German, Finnish, Romanian, Russian, Turkish) and additional 1500 sentences from each of the 5 languages translated to English. For Romanian a third of the test set were released as a development set instead. For Turkish additional 500 sentence development set was released. The sentences were selected from dozens of news websites and translated by professional translators.
The training data consists of parallel corpora to train translation models, monolingual corpora to train language models and development sets for tuning.
Some training corpora were identical from WMT 2015 (Europarl, United Nations, French-English 10⁹ corpus, Common Crawl, Russian-English parallel data provided by Yandex, Wikipedia Headlines provided by CMU) and some were update (CzEng v1.6pre, News Commentary v11, monolingual news data). Additionally, the following new corpora were added: Romanian Europarl, SETIMES2 from OPUS for Romanian-English and Turkish-English, Monolingual data sets from CommonCrawl.

Source: [https://paperswithcode.com/paper/findings-of-the-2016-conference-on-machine/](https://paperswithcode.com/paper/findings-of-the-2016-conference-on-machine/)
Image Source: [https://www.aclweb.org/anthology/W16-2301.pdf](https://www.aclweb.org/anthology/W16-2301.pdf)

Introducing paper: Findings of the 2016 Conference on Machine Translation
WMT 2016 IT
Full name: WMT 2016 IT Translation Task

The IT Translation Task is a shared task introduced in the First Conference on Machine Translation. Compared to WMT 2016 News, this task brought several novelties to WMT:

*  4 out of the 7 langauges of the IT task are new in WMT,
* adaptation to the IT domain with its specifics such as frequent named entities (mostly menu items, names of products and companies) and technical jargon,
* adaptation to translation of answers in helpdesk service setting (many of the sentences are instructions with imperative verbs, which is very rare in the News translation task).

The test set consisted of 1000 answers from the Batch 3 of the QTLeap Corpus. The in-domain training data contained 2000 answers from the Batches 1 and 2 and also localization files from several open-source projects (LibreOffice, KDE, VLC) and bilingual dictionaries of IT-related terms extracted from Wikipedia. The out-of-domain training data contained all the corpora from the WMT 2016 News, plus PaCo2-EuEn Basque-English corpus and SETimes with Bulgarian-English parallel sentences. “Constrained” systems were restricted to use only these training data provided by the organizers.

The task was evaluated on the following language pairs:

* English → Bulgarian
* English → Czech
* English → German
* English → Spanish
* English → Basque
* English → Dutch
* English → Portuguese

Source: [http://www.statmt.org/wmt16/index.html](http://www.statmt.org/wmt16/index.html)
Image Source: [Bojar et al](https://www.aclweb.org/anthology/W16-2301.pdf)

Introducing paper: Findings of the 2016 Conference on Machine Translation
WMT 2016 Biomedical
Full name: WMT 2016 Biomedical Translation Task

The Biomedical Translation Shared Task was first introduced at the First Conference of Machine Translation. The task aims to evaluate systems for the translation of biomedical titles and abstracts from scientific publications. The data includes three language pairs (English ↔ Portuguese, English  ↔ Spanish, English  ↔ French) and two sub-domains of biological sciences and health sciences.

The training data consists mainly of the Scielo corpus, a parallel collection of scientific publications composed of either titles, abstracts or title and abstracts which were retrieved from the Scielo database. For the Scielo corpus, a parallel documents are provided for all language pairs in the two sub-domains, except for the English  ↔ French, where only health was considered, as there were inadequate parallel documents available for biology in that pair. The training data was aligned using the GMA alignment tool. Additionally, a corpus of parallel titles from MEDLINEⓇ for all three language pairs were provided as well as monolingual documents for the four languages, retrieved from the Scielo database. These consist of documents in the Scielo database which have no corresponding document in another language.

The test set consisted of 500 documents (title and abstract) for each of the two directions of each language pair. None of the test documents was included in the training data and there is no overlap of documents between the test sets for any language pair, translation direction and sub-domain.

Source: [http://www.statmt.org/wmt16/index.html](http://www.statmt.org/wmt16/index.html)
Image Source: [https://www.aclweb.org/anthology/W16-2301.pdf](https://www.aclweb.org/anthology/W16-2301.pdf)

Introducing paper: Findings of the 2016 Conference on Machine Translation
XSum

The Extreme Summarization (**XSum**) dataset is a dataset for evaluation of abstractive single-document summarization systems. The goal is to create a short, one-sentence new summary answering the question “What is the article about?”. The dataset consists of 226,711 news articles accompanied with a one-sentence summary. The articles are collected from BBC articles (2010 to 2017) and cover a wide variety of domains (e.g., News, Politics, Sports, Weather, Business, Technology, Science, Health, Family, Education, Entertainment and Arts). The official random split contains 204,045 (90%), 11,332 (5%) and 11,334 (5) documents in training, validation and test sets, respectively.

Source: [https://arxiv.org/pdf/1808.08745.pdf](https://arxiv.org/pdf/1808.08745.pdf)
Image Source: [https://arxiv.org/pdf/1808.08745.pdf](https://arxiv.org/pdf/1808.08745.pdf)

Introducing paper: Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
WMT 2014 Medical
Full name: WMT 2014 Medical Translation Task

The Medical Translation Task of WMT 2014 addresses the problem of domain-specific and genre-specific machine translation. The task is split into two subtasks: summary translation, focused on translation of sentences from summaries of medical articles, and query translation, focused on translation of queries entered by users into medical information search engines. Both subtasks included translation between English and Czech, German, and French, in both directions.

Source: [https://www.aclweb.org/anthology/W14-3302.pdf](https://www.aclweb.org/anthology/W14-3302.pdf)

Introducing paper: Findings of the 2014 Workshop on Statistical Machine Translation
WMT 2015 News
Full name: WMT 2015 News Translation Task

News translation is a recurring WMT task. The test set is a collection of parallel corpora consisting of about 1500 English sentences translated into 5 languages (Czech, German, Finnish, French, Russian) and additional 1500 sentences from each of the 5 languages translated to English. The sentences are taken from newspaper articles for each language pair, except for French, where the test set was drawn from user-generated comments on the news articles (from Guardian and Le Monde). The translation was done by professional translators.

The training data consists of parallel corpora to train translation models, monolingual corpora to train language models and development sets for tuning.
Some training corpora were identical from WMT 2014 (Europarl, United Nations, French-English 10⁹ corpus, CzEng, Common Crawl, Russian-English parallel data provided by Yandex, Wikipedia Headlines provided by CMU) and some were update (News Commentary, monolingual news data). Additionally, the Finnish Europarl and Finnish-English Wikipedia Headline corpus were added.

Source: [https://paperswithcode.com/paper/findings-of-the-2016-conference-on-machine/](https://paperswithcode.com/paper/findings-of-the-2016-conference-on-machine/)
Image Source: [httpshttps://www.aclweb.org/anthology/W15-3001.pdf](httpshttps://www.aclweb.org/anthology/W15-3001.pdf)

Introducing paper: Findings of the 2015 Workshop on Statistical Machine Translation
QUASAR-S
Full name: QUestion Answering by Search And Reading – Stack Overflow

**QUASAR-S** is a large-scale dataset aimed at evaluating systems designed to comprehend a natural language query and extract its answer from a large corpus of text. It consists of 37,362 cloze-style (fill-in-the-gap) queries constructed from definitions of software entity tags on the popular website Stack Overflow. The posts and comments on the website serve as the background corpus for answering the cloze questions. The answer to each question is restricted to be another software entity, from an output vocabulary of 4874 entities.

Source: [Quasar: Datasets for Question Answering by Search and Reading](https://paperswithcode.com/paper/quasar-datasets-for-question-answering-by/)
Image Source: [Quasar: Datasets for Question Answering by Search and Reading](https://paperswithcode.com/paper/quasar-datasets-for-question-answering-by/)

Introducing paper: Quasar: Datasets for Question Answering by Search and Reading
SegmentedTables

The **SegmentedTables** dataset is a collection of almost 2,000 tables extracted from 352 machine learning papers. Each table consists of rich text content, layout and caption. Tables are annotated with types (leaderboard, ablation, irrelevant) and cells of relevant tables are annotated with semantic roles (such as “paper model”, “competing model”, “dataset”, “metric”).

Due to the license of source papers the dataset is published as a metadata, all annotations and open-source pipeline that can be used to extract the tables.

Source: [AxCell: Automatic Extraction of Results from Machine Learning Papers](https://paperswithcode.com/paper/axcell-automatic-extraction-of-results-from)
Image Source: [AxCell: Automatic Extraction of Results from Machine Learning Papers](https://paperswithcode.com/paper/axcell-automatic-extraction-of-results-from)

Introducing paper: AxCell: Automatic Extraction of Results from Machine Learning Papers
LinkedResults

The **LinkedResults** dataset contains around 1,600 results capturing performance of machine learning models from tables of 239 papers. All tables come from a subset of SegmentedTables dataset. Each result is a tuple of form (task, dataset, metric name, metric value) and is linked to a particular table, row and cell it originates from.

Source: [AxCell: Automatic Extraction of Results from Machine Learning Papers](https://paperswithcode.com/paper/axcell-automatic-extraction-of-results-from)

Introducing paper: AxCell: Automatic Extraction of Results from Machine Learning Papers
PWC Leaderboards
Full name: Papers with Code Leaderboards

The **Papers with Code Leaderboards** dataset is a collection of over 5,000 results capturing performance of machine learning models. Each result is a tuple of form (task, dataset, metric name, metric value). The data was collected using the Papers with Code review interface.

Source: [AxCell: Automatic Extraction of Results from Machine Learning Papers](https://paperswithcode.com/paper/axcell-automatic-extraction-of-results-from)
Image Source: [AxCell: Automatic Extraction of Results from Machine Learning Papers](https://paperswithcode.com/paper/axcell-automatic-extraction-of-results-from)

Introducing paper: AxCell: Automatic Extraction of Results from Machine Learning Papers
SKU110K
Full name: SKU110K

The Sku110k dataset provides 11,762 images with more than 1.7 million annotated bounding boxes captured in densely packed scenarios, including 8,233 images for training, 588 images for validation, and 2,941 images for testing. There are around 1,733,678 instances in total. The images are collected from thousands of supermarket stores and are of various scales, viewing angles, lighting conditions, and noise levels. All the images are resized into a resolution of one megapixel. Most of the instances in the dataset are tightly packed and typically of a certain orientation in the rage of [−15∘, 15∘].

Source: [Rethinking Object Detection in Retail Stores](https://arxiv.org/abs/2003.08230)
Image Source: [https://github.com/eg4000/SKU110K_CVPR19](https://github.com/eg4000/SKU110K_CVPR19)

Introducing paper: Precise Detection in Densely Packed Scenes
UBIRIS.v2
Full name: UBIRIS.v2

The **UBIRIS.v2** iris dataset contains 11,102 iris images from 261 subjects with 10 images each subject. The images were captured under unconstrained conditions (at-a-distance, on-the-move and on the visible wavelength), with realistic noise factors.

Source: [Constrained Design of Deep Iris Networks](https://arxiv.org/abs/1905.09481)
Image Source: [https://arxiv.org/pdf/1905.09481.pdf](https://arxiv.org/pdf/1905.09481.pdf)

Introducing paper: The UBIRIS.v2: A database of visible wavelength images captured on-the-move and at-a-distance
VIVA
Full name: Vision for Intelligent Vehicles and Applications

The **VIVA** challenge’s dataset is a multimodal dynamic hand gesture dataset specifically designed with difficult settings of cluttered background, volatile illumination, and frequent occlusion for studying natural human activities in real-world driving settings. This dataset was captured using a Microsoft Kinect device, and contains 885 intensity and depth video sequences of 19 different dynamic hand gestures performed by 8 subjects inside a vehicle.

Source: [Short-Term Temporal Convolutional Networks for Dynamic Hand Gesture Recognition](https://arxiv.org/abs/2001.05833)
Image Source: [http://www.site.uottawa.ca/research/viva/projects/hand_detection/index.html](http://www.site.uottawa.ca/research/viva/projects/hand_detection/index.html)