3497hyy/evaluation-tables huggingface dataset:
Task: Speech Enhancement
Categories: Audio
Speech

**Speech Enhancement** is a signal processing task that involves improving the quality of speech signals captured under noisy or degraded conditions. The goal of speech enhancement is to make speech signals clearer, more intelligible, and more pleasant to listen to, which can be used for various applications such as voice recognition, teleconferencing, and hearing aids. A representative Github project with online demo : [ClearerVoice-Studio](https://github.com/modelscope/ClearerVoice-Studio).

( Image credit: [A Fully Convolutional Neural Network For Speech Enhancement](https://arxiv.org/pdf/1609.07132v1.pdf) )
Task: Time Series Analysis
Categories: Time Series

**Time Series Analysis** is a statistical technique used to analyze and model time-based data. It is used in various fields such as finance, economics, and engineering to analyze patterns and trends in data over time. The goal of time series analysis is to identify the underlying patterns, trends, and seasonality in the data, and to use this information to make informed predictions about future values.

( Image credit: [Autoregressive CNNs for Asynchronous Time Series](https://arxiv.org/pdf/1703.04122v4.pdf) )
Task: One-Shot Learning
Categories: Methodology
Computer Vision

One-shot learning is the task of learning information about object categories from a single training example.

( Image credit: [Siamese Neural Networks for One-shot Image Recognition](https://www.cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf) )
Task: Speech Synthesis
Categories: Audio
Speech

Speech synthesis is the task of generating speech from some other modality like text, lip movements etc.

Please note that the leaderboards here are not really comparable between studies - as they use mean opinion score as a metric and collect different samples from Amazon Mechnical Turk.

( Image credit: [WaveNet: A generative model for raw audio](https://deepmind.com/blog/article/wavenet-generative-model-raw-audio) )
Task: Math Word Problem Solving
Categories: Reasoning

A math word problem is a mathematical exercise (such as in a textbook, worksheet, or exam) where significant background information on the problem is presented in ordinary language rather than in mathematical notation. As most word problems involve a narrative of some sort, they are sometimes referred to as story problems and may vary in the amount of technical language used.
Task: Relationship Extraction (Distant Supervised)
Categories: Natural Language Processing

Relationship extraction is the task of extracting semantic relationships from a text. Extracted relationships usually
occur between two or more entities of a certain type (e.g. Person, Organisation, Location) and fall into a number of
semantic categories (e.g. married to, employed by, lives in).
Task: Information Retrieval
Categories: Natural Language Processing

Information retrieval is the task of ranking a list of documents or search results in response to a query

( Image credit: [sudhanshumittal](https://github.com/sudhanshumittal/Information-retrieval-system) )
Task: X-Ray Diffraction (XRD)
Categories: Miscellaneous

Diffraction of X-ray patterns and images, with common applications for materials and images.
Task: Scene Understanding
Categories: Computer Vision

Scene understanding involves interpreting the visual information of a scene, including objects, their spatial relationships, and the overall layout. It goes beyond simple object recognition by considering the context and how objects relate to each other and the environment.
Task: CCG Supertagging
Categories: Natural Language Processing

Combinatory Categorical Grammar (CCG; [Steedman, 2000](http://www.citeulike.org/group/14833/article/8971002)) is a
highly lexicalized formalism. The standard parsing model of [Clark and Curran (2007)](https://www.mitpressjournals.org/doi/abs/10.1162/coli.2007.33.4.493)
uses over 400 lexical categories (or _supertags_), compared to about 50 part-of-speech tags for typical parsers.

Example:

| Vinken | , | 61 | years | old |
| --- | ---| --- | --- | --- |
| N| , | N/N | N | (S[adj]\ NP)\ NP |
Task: Graph Classification
Categories: Graphs

**Graph Classification** is a task that involves classifying a graph-structured data into different classes or categories. Graphs are a powerful way to represent relationships and interactions between different entities, and graph classification can be applied to a wide range of applications, such as social network analysis, bioinformatics, and recommendation systems. In graph classification, the input is a graph, and the goal is to learn a classifier that can accurately predict the class of the graph.

( Image credit: [Hierarchical Graph Pooling with Structure Learning](https://github.com/cszhangzhen/HGP-SL) )
Task: Object Recognition
Categories: Computer Vision

Object recognition is a computer vision technique for detecting + classifying objects in images or videos. Since this is a combined task of object detection plus image classification, the state-of-the-art tables are recorded for each component task [here](https://www.paperswithcode.com/task/object-detection) and [here](https://www.paperswithcode.com/task/image-classification2).

( Image credit: [Tensorflow Object Detection API
](https://github.com/tensorflow/models/tree/master/research/object_detection) )
Task: Multi-Armed Bandits
Categories: Methodology
Miscellaneous

Multi-armed bandits refer to a task where a fixed amount of resources must be allocated between competing resources that maximizes expected gain. Typically these problems involve an exploration/exploitation trade-off.

( Image credit: [Microsoft Research](http://research.microsoft.com/en-us/projects/bandits/) )
Task: Vision-Language Navigation
Categories: Robots
Computer Vision

Vision-language navigation (VLN) is the task of navigating an embodied agent to carry out natural language instructions inside real 3D environments.

( Image credit: [Learning to Navigate Unseen Environments:
Back Translation with Environmental Dropout](https://arxiv.org/pdf/1904.04195v1.pdf) )
Task: SQL-to-Text
Categories: Computer Code

( Image credit: [SQL-to-Text Generation with Graph-to-Sequence Model](https://arxiv.org/pdf/1809.05255v2.pdf) )
Task: Q-Learning
Categories: Methodology

The goal of Q-learning is to learn a policy, which tells an agent what action to take under what circumstances.

( Image credit: [Playing Atari with Deep Reinforcement Learning](https://arxiv.org/pdf/1312.5602v1.pdf) )
Task: Human Parsing
Categories: Computer Vision

Human parsing is the task of segmenting a human image into different fine-grained semantic parts such as head, torso, arms and legs.

( Image credit: [Multi-Human-Parsing (MHP)
](https://github.com/ZhaoJ9014/Multi-Human-Parsing) )
Task: Node Classification
Categories: Graphs

**Node Classification** is a machine learning task in graph-based data analysis, where the goal is to assign labels to nodes in a graph based on the properties of nodes and the relationships between them.

**Node Classification** models aim to predict non-existing node properties (known as the target property) based on other node properties. Typical models used for node classification consists of a large family of graph neural networks. Model performance can be measured using benchmark datasets like [Cora](/dataset/cora), [Citeseer](/dataset/citeseer), and [Pubmed](/dataset/pubmed), among others, typically using Accuracy and F1.

( Image credit: [Fast Graph Representation Learning With PyTorch Geometric](https://arxiv.org/pdf/1903.02428v3.pdf) )
Task: Road Damage Detection
Categories: Computer Vision

Road damage detection is the task of detecting damage in roads.

( Image credit: [Road Damage Detection And Classification In Smartphone Captured Images Using Mask R-CNN](https://arxiv.org/pdf/1811.04535v1.pdf) )
Task: Multi-Person Pose Estimation and Tracking
Categories: Computer Vision

Joint multi-person pose estimation and tracking following the PoseTrack benchmark.
https://posetrack.net/

( Image credit: [PoseTrack](https://github.com/iqbalu/PoseTrack-CVPR2017) )
Task: Medical Image Generation
Categories: Medical

Medical image generation is the task of synthesising new medical images.

( Image credit: [Towards Adversarial Retinal Image Synthesis](https://arxiv.org/pdf/1701.08974v1.pdf) )
Task: Meta-Learning
Categories: Methodology
Computer Vision

Meta-learning is a methodology considered with "learning to learn" machine learning algorithms.

( Image credit: [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://arxiv.org/pdf/1703.03400v3.pdf) )
Task: Surgical Skills Evaluation
Categories: Medical

The task is to classify surgical skills using data that is recorded during the surgical intervention.
Task: Motion Planning
Categories: Robots

( Image credit: [Motion Planning Among Dynamic, Decision-Making Agents with Deep Reinforcement Learning](https://arxiv.org/pdf/1805.01956v1.pdf) )
Task: Music Modeling
Categories: Music

( Image credit: [R-Transformer](https://arxiv.org/pdf/1907.05572v1.pdf) )
Task: Mortality Prediction
Categories: Medical

( Image credit: [Early hospital mortality prediction using vital signals](https://arxiv.org/pdf/1803.06589v2.pdf) )
Task: Click-Through Rate Prediction
Categories: Miscellaneous

Click-through rate prediction is the task of predicting the likelihood that something on a website (such as an advertisement) will be clicked.

( Image credit: [Deep Spatio-Temporal Neural Networks for Click-Through Rate Prediction](https://arxiv.org/pdf/1906.03776v2.pdf) )
Task: 3D Shape Generation
Categories: Robots

Image: [Mo et al](https://arxiv.org/pdf/1908.00575v1.pdf)
Task: Novel View Synthesis
Categories: Computer Vision

Synthesize a target image with an arbitrary target camera pose from given source images and their camera poses.

See [Wiki](https://en.wikipedia.org/wiki/View_synthesis) for more introductions.

The Synthesis method include: NeRF, MPI and so on.

( Image credit: [Multi-view to Novel view: Synthesizing novel views with Self-Learned Confidence](https://github.com/shaohua0116/Multiview2Novelview) )
Task: Sentence Compression
Categories: Natural Language Processing

**Sentence Compression** is the task of reducing the length of text by removing non-essential content while preserving important facts and grammaticality.
Task: Sign Language Recognition
Categories: Computer Vision

**Sign Language Recognition** is a computer vision and natural language processing task that involves automatically recognizing and translating sign language gestures into written or spoken language. The goal of sign language recognition is to develop algorithms that can understand and interpret sign language, enabling people who use sign language as their primary mode of communication to communicate more easily with non-signers.

( Image credit: [Word-level Deep Sign Language Recognition from Video:
A New Large-scale Dataset and Methods Comparison](https://arxiv.org/pdf/1910.11006v1.pdf) )
Task: Entity Typing
Categories: Natural Language Processing

**Entity Typing** is an important task in text analysis. Assigning types (e.g., person, location, organization) to mentions of entities in documents enables effective structured analysis of unstructured text corpora. The extracted type information can be used in a wide range of ways (e.g., serving as primitives for information extraction and knowledge base (KB) completion, and assisting question answering). Traditional Entity Typing systems focus on a small set of coarse types (typically fewer than 10). Recent studies work on a much larger set of fine-grained types which form a tree-structured hierarchy (e.g., actor as a subtype of artist, and artist is a subtype of person).

Source: [Label Noise Reduction in Entity Typing by Heterogeneous Partial-Label Embedding ](https://arxiv.org/abs/1602.05307)

Image Credit: [Label Noise Reduction in Entity Typing by Heterogeneous Partial-Label Embedding ](https://arxiv.org/abs/1602.05307)
Task: Causal Discovery
Categories: Knowledge Base

( Image credit: [TCDF](https://github.com/M-Nauta/TCDF) )
Task: Shooter Localization
Categories: Audio

Shooter localization based on videos.
Task: Generalized Zero-Shot Learning - Unseen
Categories: Computer Vision

The average of the normalized top-1 prediction scores of unseen classes in the generalized zero-shot learning setting, where the label of a test sample is predicted among all (seen + unseen) classes.
Task: Entity Disambiguation
Categories: Natural Language Processing

**Entity Disambiguation** is the task of linking mentions of ambiguous entities to their referent entities in a knowledge base such as Wikipedia.

Source: [Leveraging Deep Neural Networks and Knowledge Graphs for Entity Disambiguation ](https://arxiv.org/abs/1504.07678)
Task: Sentence Pair Modeling
Categories: Natural Language Processing

Comparing two sentences and their relationship based on their internal representation.
Task: RGB Salient Object Detection
Categories: Computer Vision

RGB Salient object detection is a task-based on a visual attention mechanism, in which algorithms aim to explore objects or regions more attentive than the surrounding areas on the scene or RGB images.

( Image credit: [Attentive Feedback Network for Boundary-Aware Salient Object Detection](http://openaccess.thecvf.com/content_CVPR_2019/papers/Feng_Attentive_Feedback_Network_for_Boundary-Aware_Salient_Object_Detection_CVPR_2019_paper.pdf) )
Task: multi-word expression embedding
Categories: Natural Language Processing

Learn embeddings for multi-word expressions
Task: multi-word expression sememe prediction
Categories: Natural Language Processing

Predict sememes for unannotated multi-word expressions.
Task: Activity Prediction
Categories: Methodology
Time Series
Computer Vision

Predict human activities in videos
Task: Multimodal Machine Translation
Categories: Natural Language Processing
Computer Vision

Multimodal machine translation is the task of doing machine translation with multiple data sources - for example, translating "a bird is flying over water" + an image of a bird over water to German text.

( Image credit: [Findings of the Third Shared Task on Multimodal Machine Translation](https://www.aclweb.org/anthology/W18-6402.pdf) )
Task: Speculation Detection
Categories: Natural Language Processing

Identifying information in text that is speculative as opposed to factual information.
Task: Data Poisoning
Categories: Adversarial

**Data Poisoning** is an adversarial attack that tries to manipulate the training dataset in order to control the prediction behavior of a trained model such that the model will label malicious examples into a desired classes (e.g., labeling spam e-mails as safe).

Source: [Explaining Vulnerabilities to Adversarial Machine Learning through Visual Analytics ](https://arxiv.org/abs/1907.07296)
Task: Sentence Ordering
Categories: Natural Language Processing

Sentence ordering task deals with finding the correct order of sentences given a randomly ordered paragraph.
Task: Continual Learning
Categories: Methodology
Natural Language Processing
Computer Vision

**Continual Learning** (also known as **Incremental Learning**, **Life-long Learning**) is a concept to learn a model for a large number of tasks sequentially without forgetting knowledge obtained from the preceding tasks, where the data in the old tasks are not available anymore during training new ones.
If not mentioned, the benchmarks here are **Task-CL**, where task-id is provided on validation.

Source:
[Continual Learning by Asymmetric Loss Approximation with Single-Side Overestimation](https://arxiv.org/abs/1908.02984)
[Three scenarios for continual learning](https://arxiv.org/abs/1904.07734)
[Lifelong Machine Learning](https://books.google.ca/books/about/Lifelong_Machine_Learning.html?id=JQ5pDwAAQBAJ&redir_esc=y)
[Continual lifelong learning with neural networks: A review](https://www.sciencedirect.com/science/article/pii/S0893608019300231)
Task: Cyber Attack Detection
Categories: Methodology
Miscellaneous

Cybersecurity attacks prediction using deep learning
Task: Twitter Bot Detection
Categories: Miscellaneous

Academic studies estimate that up to 15% of Twitter users are automated bot accounts [1]. The prevalence of Twitter bots coupled with the ability of some bots to give seemingly human responses has enabled these non-human accounts to garner widespread influence. Hence, detecting non-human Twitter users or automated bot accounts using machine learning techniques has become an area of interest to researchers in the last few years.

[1] https://aaai.org/ocs/index.php/ICWSM/ICWSM17/paper/view/15587
Task: Acoustic Novelty Detection
Categories: Audio

Detect novel events given acoustic signals, either in domestic or outdoor environments.
Task: RDF Dataset Discovery
Categories: Knowledge Base

Given a URI find the RDF datasets containing this URI.
Task: 3D Shape Modeling
Categories: Computer Vision

Image: [Gkioxari et al](https://arxiv.org/pdf/1906.02739v2.pdf)
Task: Video Reconstruction
Categories: Computer Vision

Source: [Deep-SloMo](https://github.com/avinashpaliwal/Deep-SloMo)
Task: Weather Forecasting
Categories: Miscellaneous
Time Series

**Weather Forecasting** is the prediction of future weather conditions such as precipitation, temperature, pressure and wind.

Source: [MetNet: A Neural Weather Model for Precipitation Forecasting ](https://arxiv.org/abs/2003.12140)
Task: Vulnerability Detection
Categories: Miscellaneous

Vulnerability detection plays a crucial role in safeguarding against these threats by identifying weaknesses and potential entry points that malicious actors could exploit. Through advanced scanning techniques and penetration testing, vulnerability detection tools meticulously analyze web applications and websites for vulnerabilities such as SQL injection, cross-site scripting (XSS), and insecure authentication mechanisms.

By proactively identifying and addressing vulnerabilities, organizations can strengthen their online security posture and mitigate the risk of data breaches, financial loss, and reputational damage. Additionally, vulnerability detection empowers businesses to stay compliant with industry regulations and standards, demonstrating their commitment to safeguarding sensitive information and maintaining the trust of their customers. With the evolving threat landscape and increasingly sophisticated attack vectors, investing in robust vulnerability detection measures is paramount for staying one step ahead of cyber threats and ensuring the resilience of web-based platforms and services.
Task: Molecular Property Prediction
Categories: Methodology
Medical
Graphs
Miscellaneous

Molecular property prediction is the task of predicting the properties of a molecule from its structure.
Task: Few-Shot Image Classification
Categories: Computer Vision

**Few-Shot Image Classification** is a computer vision task that involves training machine learning models to classify images into predefined categories using only a few labeled examples of each category (typically < 6 examples). The goal is to enable models to recognize and classify new images with minimal supervision and limited data, without having to train on large datasets. (typically < 6 examples)

( Image credit: [Learning Embedding Adaptation for Few-Shot Learning](https://github.com/Sha-Lab/FEAT) )
Task: Content-Based Image Retrieval
Categories: Computer Vision

**Content-Based Image Retrieval** is a well studied problem in computer vision, with retrieval problems generally divided into two groups: category-level retrieval and instance-level retrieval. Given a query image of the Sydney Harbour bridge, for instance, category-level retrieval aims to find any bridge in a given dataset of images, whilst instance-level retrieval must find the Sydney Harbour bridge to be considered a match.

Source: [Camera Obscurer: Generative Art for Design Inspiration ](https://arxiv.org/abs/1903.02165)
Task: Text Style Transfer
Categories: Natural Language Processing

Text Style Transfer is the task of controlling certain attributes of generated text. The state-of-the-art methods can be categorized into two main types which are used on parallel and non-parallel data. Methods on parallel data are typically supervised methods that use a neural sequence-to-sequence model with the encoder-decoder architecture. Methods on non-parallel data are usually unsupervised approaches using Disentanglement, Prototype Editing and Pseudo-Parallel Corpus Construction.

The popular benchmark for this task is the Yelp Review Dataset. Models are typically evaluated with the metrics of Sentiment Accuracy, BLEU, and PPL.
Task: Compressive Sensing
Categories: Computer Vision

**Compressive Sensing** is a new signal processing framework for efficiently acquiring and reconstructing a signal that have a sparse representation in a fixed linear basis.

Source: [Sparse Estimation with Generalized Beta Mixture and the Horseshoe Prior ](https://arxiv.org/abs/1411.2405)
Task: Activity Detection
Categories: Computer Vision

Detecting activities in extended videos.
Task: Lightfield
Categories: Computer Vision

Tasks related to the light-field imagery
Task: Synthetic Data Generation
Categories: Medical
Miscellaneous
Time Series

The generation of tabular data by any means possible.
Task: COVID-19 Diagnosis
Categories: Medical

Covid-19 Diagnosis is the task of diagnosing the presence of COVID-19 in an individual with machine learning.
Task: L2 Regularization
Categories: Methodology

See [Weight Decay](https://paperswithcode.com/method/weight-decay).

**$L_{2}$ Regularization** or **Weight Decay**, is a regularization technique applied to the weights of a neural network. We minimize a loss function compromising both the primary loss function and a penalty on the $L\_{2}$ Norm of the weights:

$$L\_{new}\left(w\right) = L\_{original}\left(w\right) + \lambda{w^{T}w}$$

where $\lambda$ is a value determining the strength of the penalty (encouraging smaller weights).

Weight decay can be incorporated directly into the weight update rule, rather than just implicitly by defining it through to objective function. Often weight decay refers to the implementation where we specify it directly in the weight update rule (whereas L2 regularization is usually the implementation which is specified in the objective function).
Task: Arabic Text Diacritization
Categories: Speech
Natural Language Processing

Addition of diacritics for undiacritized arabic texts for words disambiguation.
Task: Traffic Classification
Categories: Miscellaneous

**Traffic Classification** is a task of categorizing traffic flows into application-aware classes such as chats, streaming, VoIP, etc. Classification can be used for several purposes including policy enforcement and control or QoS management.

Source: [Classification of Traffic Using Neural Networks by Rejecting: a Novel Approach in Classifying VPN Traffic ](https://arxiv.org/abs/2001.03665)
Task: Virtual Try-on
Categories: Computer Vision

Virtual try-on of clothing or other items such as glasses and makeup. Most recent techniques use Generative Adversarial Networks.
Task: Automated Essay Scoring
Categories: Natural Language Processing

Essay scoring: **Automated Essay Scoring** is the task of assigning a score to an essay, usually in the context of assessing the language ability of a language learner. The quality of an essay is affected by the following four primary dimensions: topic relevance, organization and coherence, word usage and sentence complexity, and grammar and mechanics.

Source: [A Joint Model for Multimodal Document Quality Assessment ](https://arxiv.org/abs/1901.01010)
Task: Automated Writing Evaluation
Categories: Natural Language Processing

Automated writing evaluation refers to the task of analysing and measuring written text based on features, such as syntax, text complexity and vocabulary range.
Task: Author Attribution
Categories: Natural Language Processing

Authorship attribution is the task of determining the author of a text.
Task: Bias Detection
Categories: Natural Language Processing

Bias detection is the task of detecting and measuring racism, sexism and otherwise discriminatory behavior in a model (Source: https://stereoset.mit.edu/)