Omkar2118/datasets huggingface dataset:
NorQuAD

**NorQuAD** is the **first Norwegian question answering dataset** specifically designed for **machine reading comprehension**. This dataset comprises **4,752 manually created question-answer pairs**. The data collection procedure is detailed in the paper, along with relevant statistics. Additionally, several multilingual and Norwegian monolingual language models were **benchmarked** against human performance using this dataset¹²³.

(1) [2305.01957] NorQuAD: Norwegian Question Answering Dataset - arXiv.org. https://arxiv.org/abs/2305.01957.
(2) NorQuAD: Norwegian Question Answering Dataset - ACL Anthology. https://aclanthology.org/2023.nodalida-1.17/.
(3) NorQuAD: Norwegian Question Answering Dataset. https://researchportal.helsinki.fi/en/publications/norquad-norwegian-question-answering-dataset.
(4) undefined. https://doi.org/10.48550/arXiv.2305.01957.

Introducing paper: NorQuAD: Norwegian Question Answering Dataset
NQiI
Full name: Natural Questions In Icelandic

**Natural Questions in Icelandic (NQiI)** is a valuable dataset designed for **extractive question answering (QA)** in the Icelandic language. Let me provide you with some details about this dataset:

1. **Purpose and Importance**:
   - The NQiI dataset was created to facilitate the development and evaluation of Icelandic QA systems.
   - It also supports the development of QA methods that need to work across a wide range of morphologically and grammatically diverse languages in a multilingual context¹.

2. **Dataset Creation**:
   - Contributors were asked to come up with questions they were genuinely interested in knowing the answers to.
   - Later, they were tasked with finding answers to each other's questions using a previously published methodology.
   - The questions in NQiI are "natural" in the sense that they arise from genuine curiosity¹.

3. **Dataset Details**:
   - The complete NQiI dataset contains **18,000 labeled entries**.
   - Among these, **5,568 entries** are directly suitable for training an extractive QA system specifically for Icelandic¹.

4. **Resource and Evaluation**:
   - NQiI serves as a valuable resource for Icelandic language research.
   - Researchers have used it to create and evaluate systems capable of extractive QA in Icelandic¹.

(1) Natural Questions in Icelandic - ACL Anthology. https://aclanthology.org/2022.lrec-1.477/.
(2) NQiI - Natural Questions In Icelandic - v1.0 - CLARIN. https://repository.clarin.is/repository/xmlui/handle/20.500.12537/143.
(3) Natural Questions in Icelandic - ACL Anthology. https://aclanthology.org/2022.lrec-1.477/.
(4) Natural Questions in Icelandic - ACL Anthology. https://aclanthology.org/2022.lrec-1.477.pdf.
(5) undefined. https://aclanthology.org/2022.lrec-1.477.

Introducing paper: Natural Questions in Icelandic
Nordjylland News

The **Nordjylland News dataset** is a collection of news articles from **Northern Jutland in Denmark**. It provides valuable information about events, developments, and stories specific to that region. Here are some key details about this dataset:

- **Developers**: The dataset was created by **Oliver Kinch** (oliver.kinch@alexandra.dk) and **Dan Saattrup Nielsen** (dan.nielsen@alexandra.dk).

- **Content**: It contains news articles covering a wide range of topics relevant to Northern Jutland, including local news, culture, politics, and more.

- **Setup**: To work with this dataset, you can set up the environment by running `make install`. This command installs **Poetry** (if not already installed), sets up a virtual environment, and installs all Python dependencies.

- **Tools Used**:
    - **Poetry**: Used for dependency management.
    - **Hydra**: Manages configuration files.
    - **pre-commit plugins**: Automates code reviewing and formatting.
    - **pdoc**: Automatically generates API documentation for the project.

- **Project Structure**:
    - The dataset is organized into various directories, including `.github`, `config`, `data`, `models`, `notebooks`, and more.
    - The actual news articles are stored within the `nordjylland_news` directory.

For more detailed information, you can explore the [GitHub repository](https://github.com/alexandrainst/NordjyllandNews) containing this dataset¹. If you're interested in specific tasks related to this dataset, there are also related datasets for **news summarization** and **image captioning** available on platforms like [Hugging Face](https://huggingface.co/datasets/alexandrainst/nordjylland-news-summarization) and [Hugging Face](https://huggingface.co/datasets/alexandrainst/nordjylland-news-image-captioning)²³.

(1) GitHub - alexandrainst/NordjyllandNews: Dataset containing news from .... https://github.com/alexandrainst/NordjyllandNews.
(2) alexandrainst/nordjylland-news-summarization · Datasets at Hugging Face. https://huggingface.co/datasets/alexandrainst/nordjylland-news-summarization.
(3) Dataset Card for "nordjylland-news-image-captioning" - Hugging Face. https://huggingface.co/datasets/alexandrainst/nordjylland-news-image-captioning.
no sammendrag

The **"no-sammendrag" dataset** is a collection of summarized texts in Norwegian. It covers various topics and is useful for tasks related to **summarization**. The dataset contains documents along with their corresponding summaries. Let me provide more details about it:

- **Repository**: [norkart/no-sammendrag](https://huggingface.co/datasets/norkart/no-sammendrag/viewer/default)
- **Languages**: Norwegian, Norwegian Bokmål, Norwegian Nynorsk
- **Size**: The dataset falls within the range of **100K to 1M** examples.
- **Task**: Summarization

One of the documents in this dataset discusses changes to maternity services at the ABC department of Ullevål Hospital in Oslo. The decision to limit the ABC department's services has raised concerns among expectant mothers and healthcare professionals. The article emphasizes the importance of feeling safe and supported during childbirth, and how this impacts both physical and psychological well-being. While the dataset provides a wealth of information, it's essential to recognize that the value of a safe and supportive birthing environment cannot always be quantified in hospital budgets¹.

(1) norkart/no-sammendrag · Datasets at Hugging Face. https://huggingface.co/datasets/norkart/no-sammendrag/viewer/default.
(2) ScandEval/no-sammendrag-mini · Datasets at Hugging Face. https://huggingface.co/datasets/ScandEval/no-sammendrag-mini.
(3) 800 Free Data Sets - Database Star. https://www.databasestar.com/free-data-sets/.
Dutch Social

The **Dutch Social Dataset** is a collection of tweets primarily written in **Dutch**. Let me provide you with some details about this dataset:

- **Size**: It contains approximately **271,342 tweets**.
- **Language**: The tweets are filtered to include only those in the **Dutch language** or by users who have specified their location information within the **Netherlands geographical boundaries**.
- **Creators**: The annotations were generated by **machine learning models**, and the source datasets are original.
- **License**: The dataset is available under the **cc-by-nc-4.0** license.
- **Content**: Each tweet includes information such as the **full text**, **translation**, **screen name**, **description**, **location**, and other relevant details.
- **Geolocation**: Some tweets also provide **geolocation information** (latitude, longitude, and altitude) for specific points.
- **Sentiment**: Sentiment patterns are included, ranging from **negative** to **positive**.
- **Labels**: The dataset includes class labels, such as "Maar, er iets nuttigs mee doen? Zie jij 'm vertrekken?" (However, there is something useful to do with it? Do you see him leave?) ¹.
SB10k

The **SB10k dataset** is a valuable resource for sentiment analysis in German. Here are the key details:

- **Corpus Size**: It contains approximately **10,000 German tweets**¹.
- **Language**: German.
- **Task**: Text classification, specifically sentiment analysis.
- **Multilinguality**: Monolingual (German only).
- **Size Category**: Falls within the range of **1K to 10K** examples.
- **Tags**: Sentiment analysis.
- **License**: **CC-BY-4.0**.

The dataset was created by annotating German tweets, with each tweet labeled by **three annotators**. Researchers have used SB10k to benchmark various machine learning classifiers, including convolutional neural networks (CNNs) and feature-based support vector machines (SVMs) for sentiment analysis²³.

(1) Alienmaster/SB10k · Datasets at Hugging Face. https://huggingface.co/datasets/Alienmaster/SB10k.
(2) A Twitter Corpus and Benchmark Resources for German Sentiment Analysis. https://aclanthology.org/W17-1106/.
(3) A Twitter Corpus and Benchmark Resources for German Sentiment Analysis. https://aclanthology.org/W17-1106.pdf.
(4) undefined. http://t.co/9rhta65MSx.
(5) undefined. http://t.co/G84qcIGk7k.
(6) undefined. http://t.co/LvwyZgew4Q.

Introducing paper: A Twitter Corpus and Benchmark Resources for German Sentiment Analysis
Nordjylland News Summarization

The **Nordjylland News summarization dataset** is a collection of news articles from the Nordjylland region in Denmark, along with their corresponding summaries. This dataset is valuable for training and evaluating **text summarization** models. Let's delve into the details:

- **Dataset Information**:
    - **Repository**: The dataset is available on Hugging Face Datasets under the repository **alexandrainst/nordjylland-news-summarization** ¹.
    - **Contents**: Each data entry consists of the following fields:
        - `text`: The full news article.
        - `summary`: A concise summary of the article.
        - `text_len`: Length of the article (measured in characters).
        - `summary_len`: Length of the summary (measured in characters).

- **Sample Articles**:
    1. *Missing Man Search*:
        - Summary: A man reported missing was last seen near his parents' residence in Gistrup. His car, a metal blue Skoda Fabia, was found parked on Gammel Strandvej in Aalborg. Authorities are urging residents in the area to check sheds, basements, and similar places for any signs of him ¹.
    2. *Funding for Regan Vest Museum*:
        - Summary: The government's proposal for the upcoming budget negotiations indicates that the fixed allocation for the Cold War museum Regan Vest will not continue in 2020. However, despite concerns, the future of Regan Vest remains secure, as it will now be part of a funding pool that requires applications ¹.

(1) alexandrainst/nordjylland-news-summarization · Datasets at Hugging Face. https://huggingface.co/datasets/alexandrainst/nordjylland-news-summarization.
(2) +64 Summarization Datasets - NLP Database - Metatext. https://metatext.io/datasets-list/summarization-task.
(3) CNN-DailyMail News Text Summarization | Kaggle. https://www.kaggle.com/datasets/gowrishankarp/newspaper-text-summarization-cnn-dailymail.
(4) ScandEval/nordjylland-news-summarization-mini - Hugging Face. https://huggingface.co/datasets/ScandEval/nordjylland-news-summarization-mini/viewer.
(5) alexandrainst/nordjylland-news-summarization at main - Hugging Face. https://huggingface.co/datasets/alexandrainst/nordjylland-news-summarization/tree/main.
Absabank-Imm

The **Swedish ABSAbank-Imm 1.1** is an annotated corpus designed for **aspect-based sentiment analysis** related to immigration in Sweden. Let's delve into the details:

1. **Title and Subtitle**:
   - **Title**: Swedish ABSAbank-Imm v1.1
   - **Subtitle**: An annotated Swedish corpus for aspect-based sentiment analysis (a version of Absabank)

2. **Creation and Purpose**:
   - **Created by**: Aleksandrs Berdicevskis
   - **Purpose**: To analyze sentiments expressed toward immigration in Sweden.
   - **Description**: The dataset is a subset of the original **Swedish ABSAbank**, specifically tailored for aspect-based sentiment analysis related to immigration. In this subset, paragraphs are manually labeled with sentiment scores on a scale from 1 (very negative) to 5 (very positive).

3. **Dataset Details**:
   - **Size**: It contains **4872 short texts (paragraphs)**.
   - **Language**: Swedish
   - **Tokens**: Approximately **199,000 tokens**.
   - **Content**: The original ABSAbank had two layers of annotation: token-level and text-level. In ABSAbank-Imm, only the paragraph-level annotation is preserved. Annotators labeled paragraphs related to immigration with sentiment values.

4. **Usage**:
   - **Applications**: Machine Learning, Aspect-based Sentiment Analysis, Stance classification, Evaluation of language models.
   - **Intended Task**: Given a text or paragraph, label the sentiment expressed toward immigration in Sweden.
   - **Recommended Evaluation Measures**: Krippendorff's alpha (official SuperLim measure), Spearman's rho, or other correlation coefficients.
   - **Recommended Split**: Consecutive split into train, dev, and test sets.

5. **Citation and Related Datasets**:
   - **Cite as**: Consider citing ¹.
   - **Related Datasets**: Part of the SuperLim collection (derived from ABSAbank) ¹.

(1) Svensk ABSAbank-Imm 1.1 | Språkbanken Text - Göteborgs universitet. https://spraakbanken.gu.se/resurser/absabank-imm.
(2) ScandEval/absabank-imm-mini · Datasets at Hugging Face. https://huggingface.co/datasets/ScandEval/absabank-imm-mini.
(3) ScandEval/absabank-imm-mini at main - Hugging Face. https://huggingface.co/datasets/ScandEval/absabank-imm-mini/tree/main.
(4) undefined. https://spraakbanken.gu.se/en/resources/superlim.
(5) undefined. https://spraakbanken.gu.se/en/resources/swe-absa-bank.
MIM-GOLD-NER

The **MIM-GOLD-NER** dataset is an Icelandic named entity (NE) corpus. It is a version of the **MIM-GOLD** corpus that has been specifically tagged for named entities. In this dataset, over **48,000 NEs** (named entities) are labeled within a corpus of **one million tokens**. Researchers and developers can use this dataset to train named entity recognizers for Icelandic¹²³.

(1) vesteinn/icelandic-ner-MIM-GOLD-NER · Datasets at Hugging Face. https://huggingface.co/datasets/vesteinn/icelandic-ner-MIM-GOLD-NER.
(2) svanhvit/icelandic-ner-MIM-GOLD-NER · Datasets at Hugging Face. https://huggingface.co/datasets/svanhvit/icelandic-ner-MIM-GOLD-NER.
(3) vesteinn/icelandic-ner-MIM-GOLD-NER at main - Hugging Face. https://huggingface.co/datasets/vesteinn/icelandic-ner-MIM-GOLD-NER/blob/main/icelandic-ner-MIM-GOLD-NER.py.
(4) undefined. http://hdl.handle.net/20.500.12537/42.
SweDN

The **SweDN 1.0 dataset** is a valuable resource for natural language processing (NLP) tasks, specifically **text summarization**. Let's delve into the details:

1. **Title and Subtitle**:
   - **Title**: SweDN 1.0
   - **Subtitle**: A Swedish text summarization corpus

2. **Description**:
   - The SweDN 1.0 corpus is based on **1,963,576 news articles** from the Swedish newspaper **Dagens Nyheter (DN)** spanning the years **2000 to 2020**.
   - These articles have been filtered to resemble the **CNN/DailyMail dataset** in terms of their textual structure.

3. **Purpose and Usage**:
   - **Model Development**: SweDN 1.0 serves as a training resource for both **extractive** and **abstractive** text summarizers.
   - **Intended Task**: Given a text (article), the goal is to provide its summary.
   - **Evaluation Measures**: The recommended evaluation metrics include the **harmonic mean of Bleu and Rouge**, along with **Rouge**, **BERTScore**, and **Coh-Metrix**.

4. **Data Details**:
   - **Language**: Swedish
   - **Number of Articles**: The dataset comprises **38,121 news articles** along with their corresponding preambles.
   - **Format**: The data is available in **JSONL** and **TSV** files, containing fields such as **ID**, **headline**, **summary**, **article**, and **article category**.
   - An additional file provides various statistics for each entry, including **length measures**, **embedding similarity**, and **article category**.

5. **Ethical Considerations**:
   - The dataset does not involve any specific data labeling or annotator characteristics.
   - As with any NLP dataset, it's essential to consider ethical aspects and potential biases.

6. **References**:
   - Monsen, J., & Jönsson, A. (2021). *A method for building non-English corpora for abstractive text summarization*. Proceedings of the CLARIN Annual Conference ¹.

(1) SweDN 1.0 | Språkbanken Text - Göteborgs universitet. https://spraakbanken.gu.se/en/resources/swedn.
(2) Language resources | Språkbanken Text - Göteborgs universitet. https://spraakbanken.gu.se/en/resources/train.
(3) Kylberg Texture Dataset v. 1.0 – Kylberg.org. https://kylberg.org/kylberg-texture-dataset-v-1-0/.
(4) undefined. https://spraakbanken.gu.se/resurser/superlim.
DOTA 2 toxic chat data

The dataset was collected from DOTA 2 using OpenDota API via Python. The collection consists of DOTA 2 in-game chat data that were manually categorized into 3 classifications: non-toxic, mild (toxicity), and toxic chats.

Target value categorization:
0 = non-toxic 
1 = mild (toxicity)  
2 = toxic

Introducing paper: Fine-Tuning Pre-trained Language Models to Detect In-Game Trash Talks
CC3M-TagMask

The dataset offers tag and mask annotations for image-text pairs from the CC3M validation set. Tag annotations denote words that aptly describe the relationship between the image and the corresponding text. These annotations provide valuable insights into the semantic connection between each pair's visual and textual elements.

Introducing paper: TTD: Text-Tag Self-Distillation Enhancing Image-Text Alignment in CLIP to Alleviate Single Tag Bias
DMS
Full name: Dense Material Segmentation Dataset

The Dense Material Segmentation Dataset (DMS) consists of 3 million polygon labels of material categories (metal, wood, glass, etc) for 44 thousand RGB images. The dataset is described in the research paper, A Dense Material Segmentation Dataset for Indoor and Outdoor Scene Parsing.
PortraitMode-400

The **PortraitMode-400** dataset is a significant contribution to the field of **video recognition**, specifically focusing on **portrait mode videos**. Let me provide you with more details:

1. **Dataset Overview**:
   - The **PortraitMode-400 (PM-400)** dataset is the **first of its kind** and is dedicated to **portrait mode video recognition**.
   - It was created to address the unique challenges associated with recognizing videos captured in portrait mode.
   - Portrait mode videos are increasingly important due to the growing popularity of smartphones and social media applications.

2. **Data Collection and Annotation**:
   - The dataset consists of **76,000 videos** collected from **Douyin**, a popular short-video application.
   - These videos were meticulously annotated with **400 fine-grained categories**.
   - Rigorous quality assurance measures were implemented to ensure the accuracy of human annotations.

3. **Research Insights and Impact**:
   - The creators of the dataset conducted a comprehensive analysis to understand the impact of video format (portrait mode vs. landscape mode) on recognition accuracy.
   - They also explored spatial bias arising from different video formats.
   - Key aspects of portrait mode video recognition were investigated, including data augmentation, evaluation procedures, the importance of temporal information, and the role of audio modality.

(1) [2312.13746] Video Recognition in Portrait Mode - arXiv.org. https://arxiv.org/abs/2312.13746.
(2) Video Recognition in Portrait Mode | Papers With Code. https://paperswithcode.com/paper/video-recognition-in-portrait-mode.
(3) Video Recognition in Portrait Mode - arXiv.org. https://arxiv.org/pdf/2312.13746.pdf.
(4) undefined. https://doi.org/10.48550/arXiv.2312.13746.

Introducing paper: Video Recognition in Portrait Mode
MaSaC_ERC

The E-MASAC Dataset is a collection of code-mixed conversations sourced from an Indian TV series, focusing on Hindi-English interactions. It was derived from the MASAC dataset and specifically annotated for Emotion Recognition in Conversations (ERC) tasks. The dataset comprises 8,607 dialogues with 11,440 utterances, containing instances of sarcasm and humor. Emotions such as anger, fear, joy, sadness, surprise, contempt, and neutral are annotated for each utterance by three fluent English and Hindi-speaking linguists, ensuring a high inter-annotator agreement of 0.85.

Introducing paper: From Multilingual Complexity to Emotional Clarity: Leveraging Commonsense to Unveil Emotions in Code-Mixed Dialogues