3497hyy/evaluation-tables huggingface dataset:
Task: SMAC
Categories: Playing Games

The StarCraft Multi-Agent Challenge (SMAC) is a benchmark that provides elements of partial observability, challenging dynamics, and high-dimensional observation spaces. SMAC is built using the StarCraft II game engine, creating a testbed for research in cooperative MARL where each game unit is an independent RL agent.
Task: Gaze Target Estimation
Categories: Computer Vision

Gaze Target Estimation refers to predicting the image 2D gaze location of a person in the image.
Task: Morpheme Segmentaiton
Categories: Natural Language Processing

Succesful systems segment a given word or sentence into a sequence of morphemes.
Task: Chart Question Answering
Categories: Computer Code

Question Answering task on charts images
Task: Human Judgment Correlation
Categories: Reasoning

A task where an algorithm should generate the judgment scores correlating with human judgments.
Task: Human Judgment Classification
Categories: Reasoning

A task where an algorithm judges which sample is better in accordance with human judgment.
Task: Chemical Indexing
Categories: Natural Language Processing

Predict which chemicals should be indexed.
Task: Procedure Learning
Categories: Computer Vision

Given a set of videos of the same task, the goal is to identify the key-steps required to perform the task.
Task: Dataset Condensation

Condense the full dataset into a tiny set of synthetic data.
Task: Image Fusion
Categories: Computer Vision

Image fusion is a process in computational imaging where information from multiple images, often from different imaging modalities or spectral bands, is combined into a single image. The aim is to enhance the quality or utility of the information. This technique is widely used in various fields such as medical imaging, remote sensing, and surveillance. The fused image retains important features from each of the original images, thereby providing a more comprehensive representation of the scene or object being imaged.
Task: Novel Class Discovery
Categories: Methodology
Computer Vision

The goal of Novel Class Discovery (NCD) is to identify new classes in unlabeled data, by exploiting prior knowledge from known classes. In this specific setup, the data is split in two sets. The first is a labeled set containing known classes and the second is an unlabeled set containing unknown classes that must be discovered.
Task: Semi-Supervised Image Classification
Categories: Computer Vision

Semi-supervised image classification leverages unlabelled data as well as labelled data to increase classification performance.

You may want to read some blog posts to get an overview before reading the papers and checking the leaderboards:

- [An overview of proxy-label approaches for semi-supervised learning](https://ruder.io/semi-supervised/) - Sebastian Ruder
- [Semi-Supervised Learning in Computer Vision](https://amitness.com/2020/07/semi-supervised-learning/) - Amit Chaudhary

( Image credit: [Self-Supervised Semi-Supervised Learning](https://arxiv.org/pdf/1905.03670v2.pdf) )
Task: Emotional Intelligence
Categories: Natural Language Processing

Emotional Intelligence (EI) is a measure of "The ability to monitor one’s own and others’ feelings, to discriminate among them, and to use this information to guide one’s thinking and action." (Salovey and Mayer, 1990). EI is further broken down into four branches: perceiving, using, understanding and managing emotions (Mayer & Salovey, 1997). Of particular relevance to language models that operate exclusively in the text modality is _emotional understanding_ (EU). This is defined as the ability to interpret and analyse the language of emotions, to comprehend complex emotional states, and understand how these emotions can influence behaviour and decision-making.
Task: General Knowledge
Categories: Miscellaneous

This task aims to evaluate the ability of a model to answer general-knowledge questions.

Source: [BIG-bench](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/general_knowledge)
Task: Situation Recognition
Categories: Computer Vision

Situation Recognition aims to produce the structured image summary which describes the primary activity (verb), and its relevant entities (nouns).
Task: Grounded Situation Recognition
Categories: Computer Vision

Grounded Situation Recognition aims to produce the structured image summary which describes the primary activity (verb), its relevant entities (nouns), and their bounding-box groundings.
Task: Weight Space Learning

Learning from populations of neural network models (model zoo), where each model is given by a set of model parameters.
Task: Video Grounding
Categories: Computer Vision

**Video grounding** is the task of linking spoken language descriptions to specific video segments. In video grounding, the model is given a video and a natural language description, such as a sentence or a caption, and its goal is to identify the specific segment of the video that corresponds to the description. This can involve tasks such as localizing the objects or actions mentioned in the description within the video, or associating a specific time interval with the description.
Task: Multimodal Text and Image Classification
Categories: Natural Language Processing

Classification with both source Image and Text
Task: File difference
Categories: Computer Code

Generate edit script comparing 2 strings or files, which contains instruction of insert, delete and substitute to convert first string to the second.
Task: Edit script generation
Categories: Computer Code

Generating edit scripts by comparing 2 different files or strings to convert one to another. this script will contain instruction like insert, delete and substitute.
Task: Temporal Relation Extraction
Categories: Natural Language Processing

Temporal relation extraction systems aim to identify and classify the temporal relation between a pair of entities provided in a text. For instance, in the sentence "Bob sent a message to Alice while she was leaving her birthday party." one can infer that the actions "sent" and "leaving" entails a temporal relation that can be described as "simultaneous".
Task: Fine-Grained Urban Flow Inference
Categories: Miscellaneous

Fine-grained urban flow inference (FUFI) aims to infer the fine-grained urban flow map from the coarse-grained one.
Task: Multi-modal Knowledge Graph
Categories: Knowledge Base

[Link to A Survey for Multi-modal Knowledge Graphs.](https://github.com/zjukg/KG-MM-Survey)
Papers integrating Knowledge Graphs (KGs) and Multi-Modal Learning, focusing on research in two principal aspects: KG-driven Multi-Modal (KG4MM) learning, where KGs support multi-modal tasks, and Multi-Modal Knowledge Graph (MM4KG), which extends KG studies into the MMKG realm.
Task: Persuasion Strategies
Categories: Computer Vision

Prediction of Persuasion Strategy in Advertisements
Task: Text to Video Retrieval
Categories: Computer Vision

She's gone
I can't find her anywhere
I'm looking everywhere for her
Everywhere is dark
Task: Intelligent Communication
Categories: Miscellaneous
Time Series

Intelligently decide (i) the content of data
to be shared/communicated and (ii) the direction in which the chosen
data is transmitted.
Task: Joint Radar-Communication
Categories: Robots

Intelligently decide how to simultaneously conduct radar and communication over a shared radio channel.
Task: Story Continuation
Categories: Computer Vision

The task involves providing an initial scene that can be obtained in real world use cases. By including this scene, a model can then copy and adapt elements from it as it generates subsequent images.

Source: [StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation](https://paperswithcode.com/paper/storydall-e-adapting-pretrained-text-to-image)
Task: energy management
Categories: Time Series

energy management is to schedule energy units inside the systems, enabling an reliable, safe and cost-effective operation
Task: Semantic Similarity
Categories: Methodology
Natural Language Processing

The main objective **Semantic Similarity** is to measure the distance between the semantic meanings of a pair of words, phrases, sentences, or documents. For example, the word “car” is more similar to “bus” than it is to “cat”. The two main approaches to measuring Semantic Similarity are knowledge-based approaches and corpus-based, distributional methods.

Source: [Visual and Semantic Knowledge Transfer for Large Scale Semi-supervised Object Detection ](https://arxiv.org/abs/1801.03145)
Task: Wireframe Parsing
Categories: Computer Vision

Detect Line Segments and their connecting Junctions in a single perspective image.
Task: Cross-Lingual Entity Linking
Categories: Natural Language Processing

Cross-lingual entity linking is the task of using data and models available for one language for which ample such resources are available (e.g., English) to solve entity linking tasks (i.e., assigning a unique identity to entities in a text) in another, commonly low-resource, language.

Image Source: [Towards Zero-resource Cross-lingual Entity Linking](https://www.aclweb.org/anthology/D19-6127.pdf)
Task: Cultural Vocal Bursts Intensity Prediction
Categories: Speech

to predict the intensity of 40 culture-specific emotions (10 emotions from each culture)
Task: Image Outpainting
Categories: Computer Vision

Predicting the visual context of an image beyond its boundary.

Image credit: [NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis](https://paperswithcode.com/paper/nuwa-infinity-autoregressive-over?from=n35)
Task: 3D Anomaly Detection
Categories: Computer Vision

3D-only Anomaly Detection. Structures out of normal distribution are detected from the 3D-only point cloud.
Task: Molecular Docking
Categories: Medical

Predicting the binding structure of a small molecule ligand to a protein, which is critical to drug design.

Description from: [DiffDock: Diffusion Steps, Twists, and Turns for Molecular Docking](https://paperswithcode.com/paper/diffdock-diffusion-steps-twists-and-turns-for)
Task: Continual Named Entity Recognition
Categories: Natural Language Processing

Continual learning for named entity recogntion
Task: Penn Machine Learning Benchmark (Real-World)
Categories: Miscellaneous

Real-World Datasets in Penn Machine Learning Benchmark
Task: Computer Vision Transduction
Categories: Computer Vision

Transductive learning in computer vision tasks
Task: Transductive Learning

In this setting, both a labeled training sample and an (unlabeled) test sample are provided at training time. The goal is to predict only the labels of the given test instances as accurately as possible.
Task: Laminar-Turbulent Flow Localisation
Categories: Computer Vision

It is a segmentation task on thermographic measurement images in order to separate laminar and turbulent flow regions on flight body parts.
Task: Brain Morphometry
Categories: Medical

Measurement of brain structures from neuroimaging (MRI).
Task: HYPERVIEW Challenge
Categories: Computer Vision

The objective of this challenge is to advance the state of the art for soil parameter retrieval from hyperspectral data in view of the upcoming Intuition-1 mission. A campaign took place in March 2021 over agricultural areas in Poland with extensive ground samplings collocated with airborne hyperspectral measurements from imagers mounted onboard an airplane. The hyperspectral data contains 150 contiguous hyperspectral bands (462-942 nm, with a spectral resolution of 3.2 nm), which reflects the spectral range of the hyperspectral imaging sensor deployed on-board Intuition-1.
Task: Seeing Beyond the Visible
Categories: Computer Vision

The objective of this challenge is to automate the process of estimating the soil parameters, specifically, potassium (KKK), phosphorus pentoxide (P2O5P_2O_5P2​O5​), magnesium (MgMgMg) and pHpHpH, through extracting them from the airborne hyperspectral images captured over agricultural areas in Poland (the exact locations are not revealed). To make the solution applicable in real-life use cases, all the parameters should be estimated as precisely as possible.
Task: Raw reconstruction
Categories: Computer Vision

Reconstruct RAW camera sensor readings from the corresponding sRGB images
Task: Handwritten Text Recognition
Categories: Adversarial
Computer Vision

Handwritten Text Recognition (HTR) is the task of automatically identifying and transcribing handwritten text from images or scanned documents into machine-readable text. The goal is to develop a system capable of accurately interpreting diverse handwriting styles, accounting for variations in alignment, stroke, spacing, and noise. This task involves detecting handwritten regions within an image, extracting the text content, and converting it into a structured digital format, enabling further search, indexing, or data analysis.
Task: Rubik's Cube
Categories: Graphs

Solving the Rubik's Cube is a pathfinding task on a massive implicit graph.
Task: Thompson Sampling
Categories: Methodology

Thompson sampling, named after William R. Thompson, is a heuristic for choosing actions that addresses the exploration-exploitation dilemma in the multi-armed bandit problem. It consists of choosing the action that maximizes the expected reward with respect to a randomly drawn belief.
Task: Instruction Following
Categories: Natural Language Processing
Computer Vision

Instruction following is the basic task of the model. This task is dedicated to evaluating the ability of the large model to follow human instructions. It is hoped that the model can generate controllable and safe answers.
Task: Reinforcement Learning (RL)
Categories: Methodology
Knowledge Base
Computer Code
Computer Vision

**Reinforcement Learning (RL)** involves training an agent to take actions in an environment to maximize a cumulative reward signal. The agent interacts with the environment and learns by receiving feedback in the form of rewards or punishments for its actions. The goal of reinforcement learning is to find the optimal policy or decision-making strategy that maximizes the long-term reward.
Task: Semantic correspondence
Categories: Computer Vision

The task of semantic correspondence aims to establish reliable visual correspondence between different instances of the same object category.
Task: Irregular Time Series
Categories: Time Series

Irregular Time Series
Task: Federated Unsupervised Learning
Categories: Methodology

Federated unsupervised learning trains models from decentralized data that have no labels.
Task: Phonocardiogram Classification
Categories: Time Series

Classify labels/murmur/clinical outcome based on Phonocardiograms (PCGs)
Task: Predict clinical outcome
Categories: Time Series

A cost-based metric that considers the costs of algorithmic prescreening, expert screening, treatment, and diagnostic errors that result in late or missed treatments. This metric is further described here: https://moody-challenge.physionet.org/2022/
Task: Aggregate xView3 Metric
Categories: Computer Vision

The aggregate xView3 metric is the combination of five metrics: object detection F1 score, close-to-shore object detection F1 score, vessel/not vessel classification F1 score, fishing/not fishing classification F1 score, and vessel length estimation percent error regression.
Task: Humanoid Control
Categories: Robots

Control of a high-dimensional humanoid. This can include skill learning by tracking motion capture clips, learning goal-directed tasks like going towards a moving target, and generating motion within a physics simulator.
Task: Target Sound Extraction
Categories: Audio

Target Sound Extraction is the task of extracting a sound corresponding to a given class from an audio mixture. The audio mixture may contain background noise with a relatively low amplitude compared to the foreground mixture components. The choice of the sound class is provided as input to the model in form of a string, integer, or a one-hot encoding of the sound class.
Task: Action Recognition In Videos
Categories: Computer Vision

**Action Recognition in Videos** is a task in computer vision and pattern recognition where the goal is to identify and categorize human actions performed in a video sequence. The task involves analyzing the spatiotemporal dynamics of the actions and mapping them to a predefined set of action classes, such as running, jumping, or swimming.
Task: Spatial Token Mixer
Categories: Computer Vision

Spatial Token Mixer (STM) is a module for vision transformers that aims to improve the efficiency of token mixing. STM is a type of depthwise convolution that operates on the spatial dimension of the tokens. STM is a drop-in replacement for the token mixing layers in vision transformers.
Task: Steganalysis
Categories: Computer Vision

Detect the usage of Steganography
Task: Semantic entity labeling
Categories: Natural Language Processing

- One of Form Understanding task (Word grouping, Semantic entity labeling, Entity linking)
- Classifying entities into one of four pre-defined categories: question, answer, header and, other.

cited from

G. Jaume, H. K. Ekenel, J. Thiran "FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents," 2019
Task: Image-Variation
Categories: Computer Vision

Given an image, generate variations of the image
Task: Solar Flare Prediction
Categories: Time Series

Solar flare prediction in heliophysics
Task: Text-Variation
Categories: Natural Language Processing

Generate variations of the input text
Task: Detect Ground Reflections
Categories: Miscellaneous

This task helps in detecting the significant ground reflections at mm-wave bands. The harvested ground reflections can help in overcoming transient blockages at mm-wave bands
Task: Video Super-Resolution
Categories: Computer Vision

**Video Super-Resolution** is a computer vision task that aims to increase the resolution of a video sequence, typically from lower to higher resolutions. The goal is to generate high-resolution video frames from low-resolution input, improving the overall quality of the video.

( Image credit: [Detail-revealing Deep Video Super-Resolution](https://github.com/jiangsutx/SPMC_VideoSR) )
Task: Perpetual View Generation
Categories: Computer Vision

**Perpetual View Generation** is the task of generating long-range novel views by flying into a given image.
Task: 3D Inpainting
Categories: Computer Vision

**3D Inpainting** is the removal of unwanted objects
from a 3D scene, such that the replaced region is visually
plausible and consistent with its context.
Task: Conformal Prediction
Categories: Computer Vision

Conformal Prediction is a machine learning framework that provides valid measures of confidence for individual predictions. It offers a principled approach to quantify uncertainty in predictions without assuming any specific distribution for the data. This section features papers that explore various aspects of conformal prediction, including theoretical advancements, algorithmic developments, and applications across different domains.
Task: Motion Disentanglement
Categories: Computer Vision

Disentangling irregular (anomalous) motion from regular motion.
Task: HD semantic map learning
Categories: Computer Vision

The goal of task is to generate map elements in a vectorized form using data from onboard sensors, e.g., RGB cameras and/or LiDARs. These map elements include but are not limited to : Road boundaries, boundaries of roads that split roads and sidewalks.
Task: Directional Hearing
Categories: Audio

Extremely low-latency audio source separation from a known direction of arrival.
Task: Spectral Efficiency Analysis of Uplink-Downlink Decoupled Access in C-V2X Networks
Categories: Computer Code

Code for Spectral Efficiency Analysis of Uplink-Downlink Decoupled Access in C-V2X Networks
Task: Partial Video Copy Detection
Categories: Computer Vision

The PVCD goal is identifying and locating if one or more segments of a long testing video have been copied (transformed) from the reference videos dataset.
Task: Prompt-driven Zero-shot Domain Adaptation
Categories: Computer Vision

Domain adaptation using only a single source domain and a description of the target domain in natural language (No images from target domain are available)
Task: Single-shot HDR Reconstruction
Categories: Computer Vision

SVE-based HDR imaging, also known as single-shot HDR imaging, algorithms capture a scene with pixel-wise varying exposures in a single image and then computationally synthesize an HDR image, which benefits from the multiple exposures of the single image.
Task: Surgical phase recognition
Categories: Computer Vision

The first 40 videos are used for training, the last 40 videos are used for testing.
Task: highlight removal
Categories: Computer Vision

Highlight removal refers to the process of eliminating or reducing the presence of specular highlights in an image. Specular highlights are bright spots or reflections that occur when light reflects off a shiny or reflective surface, such as glass, metal, or oily skin. These highlights can often obscure or distort the underlying details of the image, making it difficult to analyze or process.
Task: Metadata quality
Categories: Miscellaneous

This task consists of the automatic assessment of the quality of the metadata of an information system.
Task: Job Shop Scheduling

Scheduling Task
Task: Scheduling

Project or Job Scheduling
Task: Readability optimization
Categories: Natural Language Processing

It consists of improving the readability of a text automatically and without significantly altering the form or meaning.
Task: human-scene contact detection
Categories: Computer Vision

detecting contact between human bodies and scenes
Task: Text-to-Code Generation
Categories: Computer Code

**Text-to-Code Generation** is a task where we can generate code based on the natural language description.

Source: [Text-to-code Generation with TensorFlow, 🤗 & MBPP](https://www.kaggle.com/code/rhtsingh/text-to-code-generation-with-tensorflow-mbpp)
Task: ValNov
Categories: Natural Language Processing

Given a textual premise and conclusion candidate, the Argument-Validity-and-Novelty-Prediction-Shared-Task ValNov consists in predicting two aspects of a conclusion: its validity and novelty.

Validity is defined as the degree to which the conclusion is justified with respect to the given premise. A conclusion is considered to be valid if it is supported by inferences that link the premise to the conclusion, based on logical principles or commonsense or world knowledge, which may be defeasible. A conclusion will be trivially considered valid if it repeats or summarizes the premise – in which case it can hardly be considered as novel.

Novelty defines the degree to which the conclusion contains content that is new in relation to the premise. As extreme cases, a conclusion candidate that repeats or summarizes the premise or is unrelated to the premise will not be considered novel.
Task: NMT
Categories: Natural Language Processing
Computer Code

Neural machine translation is an approach to machine translation that uses an artificial neural network to predict the likelihood of a sequence of words, typically modeling entire sentences in a single integrated model.
Task: Document-level Relation Extraction
Categories: Natural Language Processing

Document-level RE aim to identify the relations of various entity pairs expressed across multiple sentences.
Task: Human motion prediction
Categories: Computer Vision

Action prediction is a pre-fact video understanding task, which focuses on future states, in other words, it needs to reason about future states or infer action labels before the end of action execution.
Task: Penn Machine Learning Benchmark
Categories: Miscellaneous

Penn Machine Learning Benchmarks (PMLB) is a large collection of curated benchmark datasets for evaluating and comparing supervised machine learning algorithms.