J0nasW/paperswithcode huggingface dataset:
Task: Conversational Response Generation
Area: natural-language-processing

Given an input conversation, generate a natural-looking text reply to the last conversation element.

Image credit: [DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation](https://www.aclweb.org/anthology/2020.acl-demos.30.pdf)
Task: Text Infilling
Area: natural-language-processing

**Text Infilling** is the task of predicting missing spans of text which are consistent with the preceding and subsequent text. Text Infilling is a generalization of the cloze task—cloze historically refers to infilling individual words.


<span class="description-source">Source: [Enabling Language Models to Fill in the Blanks ](https://arxiv.org/abs/2005.05339)</span>
Task: Table-based Fact Verification
Area: natural-language-processing

Verifying facts given semi-structured data.
Task: Unsupervised Sentence Summarization
Area: natural-language-processing

Generating a summary of a given sentence without supervision.
Task: Complex Word Identification
Area: natural-language-processing

Identifying difficult words or expressions in a text.
Task: FG-1-PG-1
Area: natural-language-processing

The model learns one entity in the first step (no continuous learning is required), and learns one new entity in each subsequent continuous learning step.
Task: Cross-lingual zero-shot dependency parsing
Area: natural-language-processing

Cross-lingual zero-shot parsing is the task of inferring the dependency parse of sentences from one language without any labeled training trees for that language.

Description from [NLP Progress](http://nlpprogress.com/english/dependency_parsing.html)
Task: Argument Mining
Area: natural-language-processing

**Argument Mining** is a field of corpus-based discourse analysis that involves the automatic identification of argumentative structures in text.


<span class="description-source">Source: [AMPERSAND: Argument Mining for PERSuAsive oNline Discussions ](https://arxiv.org/abs/2004.14677)</span>
Task: Aspect Category Detection
Area: natural-language-processing

Aspect category detection (ACD) in sentiment analysis aims to identify the aspect categories mentioned in a sentence.
Task: Chunking
Area: natural-language-processing

Chunking, also known as shallow parsing, identifies continuous spans of tokens that form syntactic units such as noun phrases or verb phrases.

Example:

| Vinken | , | 61 | years | old |
| --- | ---| --- | --- | --- |
| B-NLP| I-NP | I-NP | I-NP | I-NP |
Task: Vietnamese Text Diacritization
Area: natural-language-processing

Addition of diacritics for undiacritized Vietnamese Wikipedia texts.
Task: Key Point Matching
Area: natural-language-processing

Given a debatable topic, a set of key points per stance, and a set of crowd arguments supporting or contesting the topic, report for each argument its match score for each of the key points under the same stance towards the topic.
Task: multi-word expression sememe prediction
Area: natural-language-processing

Predict sememes for unannotated multi-word expressions.
Task: Text Effects Transfer
Area: natural-language-processing

Text effects transfer refers to the task of transferring typography styles (e.g., color, texture) to an input image of a text element.
Task: Emotion Classification
Area: natural-language-processing

Emotion classification, or emotion categorization, is the task of recognising emotions to classify them into the corresponding category. Given an input, classify it as 'neutral or no emotion' or as one, or more, of several given emotions that best represent the mental state of the subject's facial expression, words, and so on. Some example benchmarks include ROCStories, Many Faces of Anger (MFA), and GoEmotions. Models can be evaluated using metrics such as the Concordance Correlation Coefficient (CCC) and the Mean Squared Error (MSE).
Task: OpenAI Gym
Area: playing-games

An open-source toolkit from OpenAI that implements several Reinforcement Learning benchmarks including: classic control, Atari, Robotics and MuJoCo tasks.

(Description by [Evolutionary learning of interpretable decision trees](https://paperswithcode.com/paper/evolutionary-learning-of-interpretable))

(Image Credit: [OpenAI Gym](https://gym.openai.com/))
Task: NetHack Score
Area: playing-games

Mean in-game score over 1000 episodes with random seeds not seen during training. See https://arxiv.org/abs/2006.13760 (Section 2.4 Evaluation Protocol) for details.
Task: Game of Go
Area: playing-games

Go is an abstract strategy board game for two players, in which the aim is to surround more territory than the opponent. The task is to train an agent to play the game and be superior to other players.
Task: Deep-Sea Treasure, Image version
Area: playing-games

Image state version of the multi-objective reinforcement learning toy environment originally introduced in "Empirical evaluation methods for multiobjective reinforcement learning algorithms" by P. Vamplew et al.
Task: text-based games
Area: playing-games

Text-based games to evaluate the Reinforcement Learning Agents
Task: Montezuma's Revenge
Area: playing-games

Montezuma's Revenge is an ATARI 2600 Benchmark game that is known to be difficult to perform on for reinforcement learning algorithms. Solutions typically employ algorithms that incentivise environment exploration in different ways.

For the state-of-the art tables, please consult the parent Atari Games task.

<span style="color:grey; opacity: 0.6">( Image credit: [Q-map](https://github.com/fabiopardo/qmap) )</span>
Task: Control with Prametrised Actions
Area: playing-games

Most reinforcement learning research papers focus on environments where the agent’s actions are either discrete or continuous. However, when training an agent to play a video game, it is common to encounter situations where actions have both discrete and continuous components. For example,  a set of high-level discrete actions (ex: move, jump, fire), each of them being associated with continuous parameters (ex: target coordinates for the move action, direction for the jump action, aiming angle for the fire action). These kinds of tasks are included in Control with Parameterised Actions.
Task: SNES Games
Area: playing-games

The task is to train an agent to play SNES games such as Super Mario.

<span style="color:grey; opacity: 0.6">( Image credit: [Large-Scale Study of Curiosity-Driven Learning](https://github.com/openai/large-scale-curiosity) )</span>
Task: Starcraft II
Area: playing-games

Starcraft II is a RTS game; the task is to train an agent to play the game.

<span style="color:grey; opacity: 0.6">( Image credit: [The StarCraft Multi-Agent Challenge](https://arxiv.org/pdf/1902.04043v2.pdf) )</span>
Task: SMAC+
Area: playing-games

Bechmarks for Efficient Exploration of Completion of Multi-stage Tasks and Usage of Environmental Factors
Task: Real-Time Strategy Games
Area: playing-games

Real-Time Strategy (RTS) tasks involve training an agent to play video games with continuous gameplay and high-level macro-strategic goals such as map control, economic superiority and more.

<span style="color:grey; opacity: 0.6">( Image credit: [Multi-platform Version of StarCraft: Brood War in a Docker Container](https://github.com/Games-and-Simulations/sc-docker) )</span>
Task: Game of Chess
Area: playing-games

Chess is a two-player strategy board game played on a chessboard, a checkered gameboard with 64 squares arranged in an 8×8 grid. The idea of making a machine that could beat a Grandmaster human player was a fascination in the artificial community for decades. Famously IBM's DeepBlue beat Kasparov in the 1990s. More recently more human-like approaches such as AlphaZero have appeared.
Task: Klondike
Area: playing-games

The most commonly played game in the family of Solitaire card games.
Task: Car Racing
Area: playing-games

https://gym.openai.com/envs/CarRacing-v0/
Task: SMAC Plus
Area: playing-games

Multi-agent reinforcement learning for completion of multi-stage tasks
Task: Dota 2
Area: playing-games

Dota 2 is a multiplayer online battle arena (MOBA). The task is to train one-or-more agents to play and win the game.

<span style="color:grey; opacity: 0.6">( Image credit: [OpenAI Five](https://openai.com/five/) )</span>
Task: Starcraft
Area: playing-games

Starcraft I is a RTS game; the task is to train an agent to play the game.

<span style="color:grey; opacity: 0.6">( Image credit: [Macro Action Selection with Deep Reinforcement Learning in StarCraft](https://arxiv.org/pdf/1812.00336v3.pdf) )</span>
Task: Atari Games
Area: playing-games

The Atari 2600 Games task (and dataset) involves training an agent to achieve high game scores.

<span style="color:grey; opacity: 0.6">( Image credit: [Playing Atari with Deep Reinforcement Learning](https://arxiv.org/pdf/1312.5602v1.pdf) )</span>
Task: NetHack
Area: playing-games

Mean in-game score over 1000 episodes with random seeds not seen during training. See https://arxiv.org/abs/2006.13760 (Section 2.4 Evaluation Protocol) for details.
Task: Solitaire
Area: playing-games

A family of single-player games using one or more standard decks of playing cards.
Task: Acrobot
Area: playing-games

The acrobot system includes two joints and two links, where the joint between the two links is actuated. Initially, the links are hanging downwards, and the goal is to swing the end of the lower link up to a given height.
Task: Card Games
Area: playing-games

Card games involve playing cards: the task is to train an agent to play the game with specified rules and beat other players.
Task: Game of Doom
Area: playing-games

Doom is an FPS game : the task is typically to train an agent to navigate the game environment, and additionally, acquire points by eliminating enemies.

<span style="color:grey; opacity: 0.6">( Image credit: [Playing FPS Games with Deep Reinforcement Learning](https://arxiv.org/pdf/1609.05521v2.pdf) )</span>
Task: FPS Games
Area: playing-games

First-person shooter (FPS) games Involve like call of duty so enjoy

<span style="color:grey; opacity: 0.6">( Image credit: [Procedural Urban Environments for FPS Games](https://arxiv.org/pdf/1604.05791v1.pdf) )</span>
Task: SMAC
Area: playing-games

The StarCraft Multi-Agent Challenge (SMAC) is a benchmark that provides elements of partial observability, challenging dynamics, and high-dimensional observation spaces. SMAC is built using the StarCraft II game engine, creating a testbed for research in cooperative MARL where each game unit is an independent RL agent.
Task: Temporal Sequences
Area: reasoning

This task asks models to answer questions about which times certain events could have occurred.

Source: [BIG-bench](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/temporal_sequences)

Image source:  [BIG-bench](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/temporal_sequences)
Task: Human Judgment Correlation
Area: reasoning

A task where an algorithm should generate the judgment scores correlating with human judgments.
Task: Abstract Argumentation
Area: reasoning

Identifying argumentative statements from natural language dialogs.
Task: Commonsense Reasoning for RL
Area: reasoning

Commonsense reasoning for Reinforcement Learning agents
Task: Odd One Out
Area: reasoning

This task tests to what extent a language model is able to identify the odd word.

Source: [BIG-bench](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/odd_one_out)
Task: Automated Theorem Proving
Area: reasoning

The goal of **Automated Theorem Proving** is to automatically generate a proof, given a conjecture (the target theorem) and a knowledge base of known facts, all expressed in a formal language. Automated Theorem Proving is useful in a wide range of applications, including the verification and synthesis of software and hardware systems.


<span class="description-source">Source: [Learning to Prove Theorems by Learning to Generate Theorems ](https://arxiv.org/abs/2002.07019)</span>
Task: Mathematical Induction
Area: reasoning

Tests the language model's capability to understand induction by asking the model to verify the correctness of an induction argument.

Source: [BIG-bench](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/mathematical_induction)
Task: Program Repair
Area: reasoning

Task of teaching ML models to modify an existing program to fix a bug in a given code.
Task: Common Sense Reasoning
Area: reasoning

Common sense reasoning tasks are intended to require the model to go beyond pattern 
recognition. Instead, the model should use "common sense" or world knowledge
to make inferences.
Task: Visual Commonsense Reasoning
Area: reasoning

Image source: [Visual Commonsense Reasoning](https://paperswithcode.com/dataset/vcr)
Task: Human Judgment Classification
Area: reasoning

A task where an algorithm judges which sample is better in accordance with human judgment.
Task: Visual Reasoning
Area: reasoning

Ability to understand  actions and reasoning  associated with any  visual images
Task: Visual Entailment
Area: reasoning

Visual Entailment (VE) - is a task consisting of image-sentence pairs
whereby a premise is defined by an image, rather than a
natural language sentence as in traditional Textual Entailment tasks. The goal is to predict
whether the image semantically entails the text.
Task: Decision Making
Area: reasoning

**Decision Making** is a complex task that involves analyzing data (of different level of abstraction) from disparate sources and with different levels of certainty, merging the information by weighing in on some data source more than other, and arriving at a conclusion by exploring all possible alternatives.


<span class="description-source">Source: [Complex Events Recognition under Uncertainty in a Sensor Network ](https://arxiv.org/abs/1411.0085)</span>
Task: StrategyQA
Area: reasoning

StrategyQA aims to measure the ability of models to answer questions that require multi-step implicit reasoning.

Source: [BIG-bench](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/strategyqa)
Task: Novel Concepts
Area: reasoning

Measures the ability of models to uncover an underlying concept that unites several ostensibly disparate entities, which hopefully would not co-occur frequently. This provides a limited test of a model's ability to creatively construct the necessary abstraction to make sense of a situation that it cannot have memorized in training.

Source: [BIG-bench](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/novel_concepts)
Task: Humanoid Control
Area: robots

Control of a high-dimensional humanoid. This can include skill learning by tracking motion capture clips, learning goal-directed tasks like going towards a moving target, and generating motion within a physics simulator.
Task: Muscle-Computer Interfaces (MCIs)
Area: robots

An interaction methodology that directly senses and decodes human muscular activity rather than relying on physical device actuation or user actions that are externally visible or audible.
Task: Sequential Place Learning
Area: robots

State-of-the-art algorithms for route-based place recognition under changing conditions.
Task: Industrial Robots
Area: robots

An industrial robot is a robot system used for manufacturing. Industrial robots are automated, programmable and capable of movement on three or more axes.
Task: Social Navigation
Area: robots

This task studies how to navigate robot(s) among humans in a safe and socially acceptable way.
Task: CARLA MAP Leaderboard
Area: robots

https://leaderboard.carla.org/leaderboard/
Task: Robot Navigation
Area: robots

The fundamental objective of mobile **Robot Navigation** is to arrive at a goal position without collision. The mobile robot is supposed to be aware of obstacles and move freely in different working scenarios.


<span class="description-source">Source: [Learning to Navigate from Simulation via Spatial and Semantic Information Synthesis with Noise Model Embedding ](https://arxiv.org/abs/1910.05758)</span>
Task: Omniverse Isaac Gym
Area: robots

The Omniverse Isaac Gym extension provides an interface for performing reinforcement learning training and inferencing in Isaac Sim. This framework simplifies the process of connecting reinforcement learning libraries and algorithms with other components in Isaac Sim. Similar to existing frameworks and environment wrapper classes that inherit from gym.Env, the Omniverse Isaac Gym extension also provides an interface inheriting from gym.Env and implements a simple set of APIs required by most common RL libraries. This interface can be used as a bridge connecting RL libraries with physics simulation and tasks running in the Isaac Sim framework.
Task: Robotic Grasping
Area: robots

This task is composed of using Deep Learning to identify how best to grasp objects using robotic arms in different scenarios. This is a very complex task as it might involve dynamic environments and objects unknown to the network.
Task: Skill Generalization
Area: robots

Image credit: [A Generalist Agent](https://storage.googleapis.com/deepmind-media/A%20Generalist%20Agent/Generalist%20Agent.pdf)
Task: Radar odometry
Area: robots

Radar odometry is the task of estimating the trajectory of the radar sensor, e.g. as presented in https://arxiv.org/abs/2105.01457.
A well established performance metric was presented by Geiger (2012)  - "Are we ready for autonomous driving? the KITTI vision benchmark suite"
Task: Radar Object Detection
Area: robots

The radar object detection (ROD) task aims to classify and localize the objects in 3D purely from radar's radio frequency (RF) images.
Task: Joint Radar-Communication
Area: robots

Intelligently decide how to simultaneously conduct radar and communication over a shared radio channel.
Task: Visual Navigation
Area: robots

**Visual Navigation** is the problem of navigating an agent, e.g. a mobile robot, in an environment using camera input only. The agent is given a target image (an image it will see from the target position), and its goal is to move from its current position to the target by applying a sequence of actions, based on the camera observations only.


<span class="description-source">Source: [Vision-based Navigation Using Deep Reinforcement Learning ](https://arxiv.org/abs/1908.03627)</span>
Task: Sequential Place Recognition
Area: robots

State-of-the-art algorithms for route-based place recognition under changing conditions.
Task: Visual Odometry
Area: robots

**Visual Odometry** is an important area of information fusion in which the central aim is to estimate the pose of a robot using data collected by visual sensors.


<span class="description-source">Source: [Bi-objective Optimization for Robust RGB-D Visual Odometry ](https://arxiv.org/abs/1411.7445)</span>
Task: Safe Exploration
Area: robots

**Safe Exploration** is an approach to collect ground truth data by safely interacting with the environment.


<span class="description-source">Source: [Chance-Constrained Trajectory Optimization for Safe Exploration and Learning of Nonlinear Systems ](https://arxiv.org/abs/2005.04374)</span>
Task: Trajectory Planning
Area: robots

Trajectory planning for industrial robots consists of moving the tool center point from point A to point B while avoiding body collisions over time.
Trajectory planning is sometimes referred to as motion planning and erroneously as path planning. Trajectory planning is distinct from path planning in that it is parametrized by time. Essentially trajectory planning encompasses path planning in addition to planning how to move based on velocity, time, and kinematics.
Task: Grasp rectangle generation
Area: robots

Grasp rectangles are a popular way to represent a two-finger grasp. Grasp rectangle generation is a task to (automatically) label datasets for parallel-jaw grap learning.
Task: Gesture Generation
Area: robots

Generation of gestures, as a sequence of 3d poses
Task: Vision-based navigation with language-based assistance
Area: robots

A grounded vision-language task where an agent with visual perception is guided via language to find objects in photorealistic indoor environments. The task emulates a real-world scenario in that (a) the requester may not know how to navigate to the target objects and thus makes requests by only specifying high-level endgoals, and (b) the agent is capable of sensing when it is lost and querying an advisor, who is more qualified at the task, to obtain language subgoals to make progress.
Task: Isaac Gym Preview
Area: robots

Isaac Gym offers a high performance learning platform to train policies for wide variety of robotics tasks directly on GPU. Both physics simulation and the neural network policy training reside on GPU and communicate by directly passing data from physics buffers to PyTorch tensors without ever going through any CPU bottlenecks. This leads to blazing fast training times for complex robotics tasks on a single GPU with 2-3 orders of magnitude improvements compared to conventional RL training that uses a CPU based simulator and GPU for neural networks.
Task: Motion Planning
Area: robots

<span style="color:grey; opacity: 0.6">( Image credit: [Motion Planning Among Dynamic, Decision-Making Agents with Deep Reinforcement Learning](https://arxiv.org/pdf/1805.01956v1.pdf) )</span>
Task: Voice Conversion
Area: speech

**Voice Conversion** is a technology that modifies the speech of a source speaker and makes their speech sound like that of another target speaker without changing the linguistic information.


<span class="description-source">Source: [Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNet ](https://arxiv.org/abs/1903.12389)</span>
Task: Speech Recognition
Area: speech

**Speech Recognition** is the task of converting spoken language into text. It involves recognizing the words spoken in an audio recording and transcribing them into a written format. The goal is to accurately transcribe the speech in real-time or from recorded audio, taking into account factors such as accents, speaking speed, and background noise.

<span style="color:grey; opacity: 0.6">( Image credit: [SpecAugment](https://arxiv.org/pdf/1904.08779v2.pdf) )</span>
Task: Bandwidth Extension
Area: speech

Bandwidth extension is the task of expanding the bandwidth of a signal in a way that approximates the original or desired higher bandwidth signal.
Task: Keyword Spotting
Area: speech

In speech processing, keyword spotting deals with the identification of keywords in utterances.

<span style="color:grey; opacity: 0.6">( Image credit: [Simon Grest](https://github.com/simongrest/kaggle-freesound-audio-tagging-2019) )</span>
Task: Dialogue Generation
Area: speech

Dialogue generation is the task of "understanding" natural language inputs - within natural language processing in order to produce output. The systems are usually intended for conversing with humans, for instance back and forth dialogue with a conversation agent like a chatbot. Some example benchmarks for this task (see others such as Natural Language Understanding) include FusedChat and Ubuntu DIalogue Corpus (UDC). Models can be evaluated via metrics such as BLEU, ROUGE, and METEOR albeit with challenges in terms of weak correlation with human judgement, that may be addressed by new ones like UnSupervised and Reference-free (USR) and Metric for automatic Unreferenced dialog evaluation (MaUde).
Task: A-VB Culture
Area: speech

In the A-VB Culture track, participants will be challenged with predicting the intensity of 10 emotions associated with each vocal burst as a multi-output regression task, using a model or multiple models that generate predictions specific to each of the four cultures  (the U.S., China, Venezuela, or South Africa). Specifically, annotations of each vocal burst will consist of culture-specific ground truth, meaning that the ground truth for each sample will be the average of annotations solely from the country of origin of the sample. Participants will report the average Concordance Correlation Coefficient (CCC), as well as the Pearson correlation coefficient, across all 10 emotions. The baseline for this challenge will be based on CCC.