3497hyy/evaluation-tables huggingface dataset:
Task: Anomaly Detection
Categories: Methodology
Graphs
Miscellaneous
Computer Vision

**Anomaly Detection** is a binary classification identifying unusual or unexpected patterns in a dataset, which deviate significantly from the majority of the data. The goal of anomaly detection is to identify such anomalies, which could represent errors, fraud, or other types of unusual events, and flag them for further investigation.

[Image source]: [GAN-based Anomaly Detection in Imbalance Problems](https://paperswithcode.com/paper/gan-based-anomaly-detection-in-imbalance)
Task: Reading Comprehension
Categories: Natural Language Processing

Most current question answering datasets frame the task as reading comprehension where the question is about a paragraph or document and the answer often is a span in the document.

Some specific tasks of reading comprehension include multi-modal machine reading comprehension and textual machine reading comprehension, among others. In the literature, machine reading comprehension can be divide into four categories: **cloze style**, **multiple choice**, **span prediction**, and **free-form answer**. Read more about each category [here](https://paperswithcode.com/paper/a-survey-on-machine-reading-comprehension-1).

Benchmark datasets used for testing a model's reading comprehension abilities include [MovieQA](/dataset/movieqa), [ReCoRD](dataset/record), and [RACE](/dataset/race), among others.

The Machine Reading group at UCL also provides an [overview of reading comprehension tasks](https://uclnlp.github.io/ai4exams/data.html).

Figure source: [A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets](https://arxiv.org/pdf/2006.11880.pdf)
Task: Few-Shot Learning
Categories: Methodology
Audio
Natural Language Processing
Computer Vision

**Few-Shot Learning** is an example of meta-learning, where a learner is trained on several related tasks, during the meta-training phase, so that it can generalize well to unseen (but related) tasks with just few examples, during the meta-testing phase. An effective approach to the Few-Shot Learning problem is to learn a common representation for various tasks and train task specific classifiers on top of this representation.

Source: [Penalty Method for Inversion-Free Deep Bilevel Optimization ](https://arxiv.org/abs/1911.03432)
Task: Color Constancy
Categories: Computer Vision

**Color Constancy** is the ability of the human vision system to perceive the colors of the objects in the scene largely invariant to the color of the light source. The task of computational Color Constancy is to estimate the scene illumination and then perform the chromatic adaptation in order to remove the influence of the illumination color on the colors of the objects in the scene.

Source: [CroP: Color Constancy Benchmark Dataset Generator ](https://arxiv.org/abs/1903.12581)
Task: Relation Classification
Categories: Natural Language Processing

**Relation Classification** is the task of identifying the semantic relation holding between two nominal entities in text.

Source: [Structure Regularized Neural Network for Entity Relation Classification for Chinese Literature Text ](https://arxiv.org/abs/1803.05662)
Task: Zero-Shot Learning
Categories: Methodology
Computer Vision

**Zero-shot learning (ZSL)** is a model's ability to detect classes never seen during training. The condition is that the classes are not known during supervised learning.

Earlier work in zero-shot learning use attributes in a two-step approach to infer unknown classes. In the computer vision context, more recent advances learn mappings from image feature space to semantic space. Other approaches learn non-linear multimodal embeddings. In the modern NLP context, language models can be evaluated on downstream tasks without fine tuning.

Benchmark datasets for zero-shot learning include [aPY](/dataset/apy), [AwA](/dataset/awa2-1), and [CUB](/dataset/cub-200-2011), among others.

( Image credit: [Prototypical Networks for Few shot Learning in PyTorch
](https://github.com/orobix/Prototypical-Networks-for-Few-shot-Learning-PyTorch) )

Further readings:

- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://paperswithcode.com/paper/zero-shot-learning-a-comprehensive-evaluation)
- [Zero-Shot Learning in Modern NLP](https://joeddav.github.io/blog/2020/05/29/ZSL.html)
- [Zero-Shot Learning for Text Classification](https://amitness.com/2020/05/zero-shot-text-classification/)
Task: Transfer Learning
Categories: Methodology
Miscellaneous

**Transfer Learning** is a machine learning technique where a model trained on one task is re-purposed and fine-tuned for a related, but different task. The idea behind transfer learning is to leverage the knowledge learned from a pre-trained model to solve a new, but related problem. This can be useful in situations where there is limited data available to train a new model from scratch, or when the new task is similar enough to the original task that the pre-trained model can be adapted to the new problem with only minor modifications.

( Image credit: [Subodh Malgonde](https://medium.com/@subodh.malgonde/transfer-learning-using-tensorflow-52a4f6bcde3e) )
Task: Question Answering
Categories: Natural Language Processing
Miscellaneous
Reasoning

Question answering can be segmented into domain-specific tasks like community question answering and knowledge-base question answering. Popular benchmark datasets for evaluation question answering systems include [SQuAD](/dataset/squad), [HotPotQA](/dataset/hotpotqa), [bAbI](/dataset/babi-1), [TriviaQA](/dataset/triviaqa), [WikiQA](/dataset/wikiqa), and many others. Models for question answering are typically evaluated on metrics like EM and F1. Some recent top performing models are T5 and XLNet.

( Image credit: [SQuAD](https://rajpurkar.github.io/mlx/qa-and-squad/) )
Task: Summarization
Categories: Natural Language Processing

Summarization is the task of producing a shorter version of one or several documents that preserves most of the
input's meaning.
Task: Visual Question Answering (VQA)
Categories: Natural Language Processing
Computer Vision

**Visual Question Answering (VQA)** is a task in computer vision that involves answering questions about an image. The goal of VQA is to teach machines to understand the content of an image and answer questions about it in natural language.

Image Source: [visualqa.org](https://visualqa.org/)
Task: Multi-step retrosynthesis

These leaderboards are used to track progress in Multi-step retrosynthesis.
Task: Multi-Object Tracking
Categories: Computer Vision

**Multi-Object Tracking** is a task in computer vision that involves detecting and tracking multiple objects within a video sequence. The goal is to identify and locate objects of interest in each frame and then associate them across frames to keep track of their movements over time. This task is challenging due to factors such as occlusion, motion blur, and changes in object appearance, and is typically solved using algorithms that integrate object detection and data association techniques.
Task: Time Series Classification
Categories: Time Series

**Time Series Classification** is a general task that can be useful across many subject-matter domains and applications. The overall goal is to identify a time series as coming from one of possibly many sources or predefined groups, using labeled training data. That is, in this setting we conduct supervised learning, where the different time series sources are considered known.

Source: [Nonlinear Time Series Classification Using Bispectrum-based Deep Convolutional Neural Networks ](https://arxiv.org/abs/2003.02353)
Task: Time Series Prediction
Categories: Time Series

The goal of **Time Series Prediction** is to infer the future values of a time series from the past.

Source: [Orthogonal Echo State Networks and stochastic evaluations of likelihoods ](https://arxiv.org/abs/1601.05911)
Task: Time Series Forecasting
Categories: Time Series

**Time Series Forecasting** is the task of fitting a model to historical, time-stamped data in order to predict future values. Traditional approaches include moving average, exponential smoothing, and ARIMA, though models as various as RNNs, Transformers, or XGBoost can also be applied. The most popular benchmark is the ETTh1 dataset. Models are typically evaluated using the Mean Square Error (MSE) or Root Mean Square Error (RMSE).

( Image credit: [ThaiBinh Nguyen](https://github.com/tn16jv/Stock-Price-Prediction) )
Task: Scene Parsing
Categories: Computer Vision

Scene parsing is to segment and parse an image into different image regions associated with semantic categories, such as sky, road, person, and bed. [MIT Description](http://sceneparsing.csail.mit.edu/#:~:text=Scene%20parsing%20is%20to%20segment,the%20algorithms%20of%20scene%20parsing.)
Task: 3D Multi-Person Pose Estimation (absolute)
Categories: Computer Vision

This task aims to solve absolute 3D multi-person pose Estimation (camera-centric coordinates). No ground truth human bounding box and human root joint coordinates are used during testing stage.

( Image credit: [RootNet](https://github.com/mks0601/3DMPPE_ROOTNET_RELEASE) )
Task: Text Generation
Categories: Adversarial
Speech
Natural Language Processing
Computer Code

**Text Generation** is the task of generating text with the goal of appearing indistinguishable to human-written text. This task is more formally known as "natural language generation" in the literature.

Text generation can be addressed with Markov processes or deep generative models like LSTMs. Recently, some of the most advanced methods for text generation include [BART](/method/bart), [GPT](/method/gpt) and other [GAN-based approaches](/method/gan). Text generation systems are evaluated either through human ratings or automatic evaluation metrics like METEOR, ROUGE, and BLEU.

Further readings:

- [The survey: Text generation models in deep learning](https://www.sciencedirect.com/science/article/pii/S1319157820303360)
- [Modern Methods for Text Generation](https://arxiv.org/abs/2009.04968)

( Image credit: [Adversarial Ranking for Language Generation](https://arxiv.org/abs/1705.11001) )
Task: Code Generation
Categories: Natural Language Processing
Computer Code
Reasoning

**Code Generation** is an important field to predict explicit code or program structure from multimodal data sources such as incomplete code, programs in another programming language, natural language descriptions or execution examples. Code Generation tools can assist the development of automatic programming tools to improve programming productivity.

Source: [Deep Learning for Source Code Modeling and Generation ](https://arxiv.org/abs/2002.05442)

Image source: [Measuring Coding Challenge Competence With APPS](https://paperswithcode.com/paper/measuring-coding-challenge-competence-with)
Task: Activity Recognition
Categories: Robots
Time Series
Computer Vision

Human **Activity Recognition** is the problem of identifying events performed by humans given a video input. It is formulated as a binary (or multiclass) classification problem of outputting activity class labels. Activity Recognition is an important problem with many societal applications including smart surveillance, video search/retrieval, intelligent robots, and other monitoring systems.

Source: [Learning Latent Sub-events in Activity Videos Using Temporal Attention Filters ](https://arxiv.org/abs/1605.08140)
Task: Relational Reasoning
Categories: Natural Language Processing

The goal of **Relational Reasoning** is to figure out the relationships among different entities, such as image pixels, words or sentences, human skeletons or interactive moving agents.

Source: [Social-WaGDAT: Interaction-aware Trajectory Prediction via Wasserstein Graph Double-Attention Network ](https://arxiv.org/abs/2002.06241)
Task: Dictionary Learning
Categories: Methodology

**Dictionary Learning** is an important problem in multiple areas, ranging from computational neuroscience, machine learning, to computer vision and image processing. The general goal is to find a good basis for given data. More formally, in the Dictionary Learning problem, also known as sparse coding, we are given samples of a random vector $y\in\mathbb{R}^n$, of the form $y=Ax$ where $A$ is some unknown matrix in $\mathbb{R}^{n×m}$, called dictionary, and $x$ is sampled from an unknown distribution over sparse vectors. The goal is to approximately recover the dictionary $A$.

Source: [Polynomial-time tensor decompositions with sum-of-squares ](https://arxiv.org/abs/1610.01980)
Task: Zero-Shot Machine Translation
Categories: Natural Language Processing

Translate text or speech from one language to another without supervision.
Task: Knowledge Tracing
Categories: Miscellaneous

**Knowledge Tracing** is the task of modelling student knowledge over time so that we can accurately predict how students will perform on future interactions. Improvement on this task means that resources can be suggested to students based on their individual needs, and content which is predicted to be too easy or too hard can be skipped or delayed.

Source: [Deep Knowledge Tracing ](https://arxiv.org/abs/1506.05908)
Task: Knowledge Graphs
Categories: Knowledge Base
Graphs

A knowledge graph is a structured representation of information that organizes data into nodes (entities) and edges (relationships) to show how different pieces of knowledge are interconnected. It enables enhanced data integration, search, and inference by modeling the relationships between concepts and entities in a graph format.
Task: Visual Reasoning
Categories: Reasoning
Computer Vision

Ability to understand actions and reasoning associated with any visual images
Task: Visual Odometry
Categories: Robots
Computer Vision

**Visual Odometry** is an important area of information fusion in which the central aim is to estimate the pose of a robot using data collected by visual sensors.

Source: [Bi-objective Optimization for Robust RGB-D Visual Odometry ](https://arxiv.org/abs/1411.7445)
Task: Speech Separation
Categories: Speech

The task of extracting all overlapping speech sources in a given mixed speech signal refers to the **Speech Separation**. Speech Separation is a special scenario of source separation problem, where the focus is only on the overlapping speech signal sources and other interferences such as music or noise signals are not the main concern of the study. A recent representative Github project can be referred to [ClearerVoice-Studio](https://github.com/modelscope/ClearerVoice-Studio).

Source: [A Unified Framework for Speech Separation ](https://arxiv.org/abs/1912.07814)

Image credit: [Speech Separation of A Target Speaker Based on Deep Neural Networks](http://staff.ustc.edu.cn/~jundu/Publications/publications/ICSP2014_Du.pdf)
Task: Chinese
Categories: Natural Language Processing

Chinese language processing is the task of applying natural language processing to the Chinese language.
Task: Common Sense Reasoning
Categories: Natural Language Processing
Reasoning

Common sense reasoning tasks are intended to require the model to go beyond pattern
recognition. Instead, the model should use "common sense" or world knowledge
to make inferences.
Task: Word Sense Disambiguation
Categories: Natural Language Processing

The task of Word Sense Disambiguation (WSD) consists of associating words in context with their most suitable entry in a pre-defined sense inventory. The de-facto sense inventory for English in WSD is [WordNet](https://wordnet.princeton.edu)..
For example, given the word “mouse” and the following sentence:

“A mouse consists of an object held in one's hand, with one or more buttons.”

we would assign “mouse” with its electronic device sense ([the 4th sense in the WordNet sense inventory](http://wordnetweb.princeton.edu/perl/webwn?c=8&sub=Change&o2=&o0=1&o8=1&o1=1&o7=&o5=&o9=&o6=&o3=&o4=&i=-1&h=000000&s=mouse)).
Task: Word Sense Induction
Categories: Natural Language Processing

Word sense induction (WSI) is widely known as the “unsupervised version” of WSD. The problem states as: Given a target word (e.g., “cold”) and a collection of sentences (e.g., “I caught a cold”, “The weather is cold”) that use the word, cluster the sentences according to their different senses/meanings. We do not need to know the sense/meaning of each cluster, but sentences inside a cluster should have used the target words with the same sense.

Description from [NLP Progress](http://nlpprogress.com/english/word_sense_disambiguation.html)
Task: Sleep Quality
Categories: Medical

( Image credit: [DeepSleep](https://github.com/GuanLab/DeepSleep) )
Task: Natural Language Inference
Categories: Natural Language Processing
Reasoning

**Natural language inference (NLI)** is the task of determining whether a "hypothesis" is
true (entailment), false (contradiction), or undetermined (neutral) given a "premise".

Example:

| Premise | Label | Hypothesis |
| --- | ---| --- |
| A man inspects the uniform of a figure in some East Asian country. | contradiction | The man is sleeping. |
| An older and younger man smiling. | neutral | Two men are smiling and laughing at the cats playing on the floor. |
| A soccer game with multiple males playing. | entailment | Some men are playing a sport. |

Approaches used for NLI include earlier symbolic and statistical approaches to more recent deep learning approaches. Benchmark datasets used for NLI include [SNLI](/dataset/snli), [MultiNLI](/dataset/multinli), [SciTail](/dataset/scitail), among others. You can get hands-on practice on the SNLI task by following this [d2l.ai chapter](https://d2l.ai/chapter_natural-language-processing-applications/natural-language-inference-and-dataset.html).

Further readings:

- [Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches](https://arxiv.org/abs/1904.01172)
Task: Language Modelling
Categories: Medical
Natural Language Processing
Miscellaneous

A language model is a model of natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, and information retrieval.

Large language models (LLMs), currently their most advanced form, are predominantly based on transformers trained on larger datasets (frequently using words scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as word n-gram language model.

Source: [Wikipedia](https://en.wikipedia.org/wiki/Language_model)
Task: Speaker Diarization
Categories: Speech

**Speaker Diarization** is the task of segmenting and co-indexing audio recordings by speaker. The way the task is commonly defined, the goal is not to identify known speakers, but to co-index segments that are attributed to the same speaker; in other words, diarization implies finding speaker boundaries and grouping segments that belong to the same speaker, and, as a by-product, determining the number of distinct speakers. In combination with speech recognition, diarization enables speaker-attributed speech-to-text transcription.

Source: [Improving Diarization Robustness using Diversification, Randomization and the DOVER Algorithm ](https://arxiv.org/abs/1910.11691)
Task: Speaker Recognition
Categories: Speech

**Speaker Recognition** is the process of identifying or confirming the identity of a person given his speech segments.

Source: [Margin Matters: Towards More Discriminative Deep Neural Network Embeddings for Speaker Recognition ](https://arxiv.org/abs/1906.07317)
Task: Speaker Verification
Categories: Speech

Speaker verification is the verifying the identity of a person from characteristics of the voice.

( Image credit: [Contrastive-Predictive-Coding-PyTorch
](https://github.com/jefflai108/Contrastive-Predictive-Coding-PyTorch) )
Task: Drug Discovery
Categories: Medical

Drug discovery is the task of applying machine learning to discover new candidate drugs.

( Image credit: [A Turing Test for Molecular Generators](https://pubs.acs.org/doi/10.1021/acs.jmedchem.0c01148) )
Task: Seizure Detection
Categories: Medical

**Seizure Detection** is a binary supervised classification problem with the aim of classifying between seizure and non-seizure states of a patient.

Source: [ResOT: Resource-Efficient Oblique Trees for Neural Signal Classification ](https://arxiv.org/abs/2006.07900)
Task: Medical Diagnosis
Categories: Medical
Computer Vision

**Medical Diagnosis** is the process of identifying the disease a patient is affected by, based on the assessment of specific risk factors, signs, symptoms and results of exams.

Source: [A probabilistic network for the diagnosis of acute cardiopulmonary diseases ](https://arxiv.org/abs/1609.06864)
Task: Malware Classification
Categories: Miscellaneous

**Malware Classification** is the process of assigning a malware sample to a specific malware family. Malware within a family shares similar properties that can be used to create signatures for detection and classification. Signatures can be categorized as static or dynamic based on how they are extracted. A static signature can be based on a byte-code sequence, binary assembly instruction, or an imported Dynamic Link Library (DLL). Dynamic signatures can be based on file system activities, terminal commands, network communications, or function and system call sequences.

Source: [Behavioral Malware Classification using Convolutional Recurrent Neural Networks ](https://arxiv.org/abs/1811.07842)
Task: Saliency Detection
Categories: Computer Vision

**Saliency Detection** is a preprocessing step in computer vision which aims at finding salient objects in an image.

Source: [An Unsupervised Game-Theoretic Approach to Saliency Detection ](https://arxiv.org/abs/1708.02476)
Task: Matrix Completion
Categories: Methodology

**Matrix Completion** is a method for recovering lost information. It originates from machine learning and usually deals with highly sparse matrices. Missing or unknown data is estimated using the low-rank matrix of the known data.

Source: [A Fast Matrix-Completion-Based Approach for Recommendation Systems ](https://arxiv.org/abs/1912.00600)
Task: Fraud Detection
Categories: Miscellaneous

**Fraud Detection** is a vital topic that applies to many industries including the financial sectors, banking, government agencies, insurance, and law enforcement, and more. Fraud endeavors have detected a radical rise in current years, creating this topic more critical than ever. Despite struggles on the part of the troubled organizations, hundreds of millions of dollars are wasted to fraud each year. Because nearly a few samples confirm fraud in a vast community, locating these can be complex. Data mining and statistics help to predict and immediately distinguish fraud and take immediate action to minimize costs.

Source: [Applying support vector data description for fraud detection ](https://arxiv.org/abs/2006.00618)
Task: Abuse Detection
Categories: Natural Language Processing

Abuse detection is the task of identifying abusive behaviors, such as hate speech, offensive language, sexism and racism, in utterances from social media platforms (Source: https://arxiv.org/abs/1802.00385).
Task: Intrusion Detection
Categories: Natural Language Processing
Miscellaneous

**Intrusion Detection** is the process of dynamically monitoring events occurring in a computer system or network, analyzing them for signs of possible incidents and often interdicting the unauthorized access. This is typically accomplished by automatically collecting information from a variety of systems and network sources, and then analyzing the information for possible security problems.

Source: [Machine Learning Techniques for Intrusion Detection ](https://arxiv.org/abs/1312.2177)