J0nasW/paperswithcode huggingface dataset:
Task: Self-Supervised Learning
Area: computer-vision

**Self-Supervised Learning** is proposed for utilizing unlabeled data with the success of supervised learning. Producing a dataset with good labels is expensive, while unlabeled data is being generated all the time. The motivation of Self-Supervised Learning is to make use of the large amount of unlabeled data. The main idea of Self-Supervised Learning is to generate the labels from unlabeled data, according to the structure or characteristics of the data itself, and then train on this unsupervised data in a supervised manner. Self-Supervised Learning is wildly used in representation learning to make a model learn the latent features of the data. This technique is often employed in computer vision, video processing and robot control.


<span class="description-source">Source: [Self-supervised Point Set Local Descriptors for Point Cloud Registration ](https://arxiv.org/abs/2003.05199)</span>

Image source: [LeCun](https://www.youtube.com/watch?v=7I0Qt7GALVk)
Task: Anomaly Detection at 30% anomaly
Area: computer-vision

Performance of unsupervised anomaly detection at specific anomaly percentage.
Task: Activity Recognition
Area: computer-vision

Human **Activity Recognition** is the problem of identifying events performed by humans given a video input. It is formulated as a binary (or multiclass) classification problem of outputting activity class labels. Activity Recognition is an important problem with many societal applications including smart surveillance, video search/retrieval, intelligent robots, and other monitoring systems.


<span class="description-source">Source: [Learning Latent Sub-events in Activity Videos Using Temporal Attention Filters ](https://arxiv.org/abs/1605.08140)</span>
Task: Action Triplet Recognition
Area: computer-vision

Recognising action as a triplet of subject verb and object. Example HOI = Human Object Interaction, Surgical IVT = Instrument Verb Target, etc.
Task: Image-Based Localization
Area: computer-vision

Determining the location of an image without GPS based on cross-view matching. In most of the cases a database of satellite images is used to match the ground images to them.
Task: Robust Object Detection
Area: computer-vision

A Benchmark for the: 
Robustness of Object Detection Models to Image Corruptions and Distortions

To allow fair comparison of robustness enhancing methods all models have to use a standard ResNet50 backbone because performance strongly scales with backbone capacity. If requested an unrestricted category can be added later.

Benchmark Homepage: https://github.com/bethgelab/robust-detection-benchmark


Metrics:

mPC [AP]: Mean Performance under Corruption [measured in AP]

rPC [%]: Relative Performance under Corruption [measured in %]

Test sets:
Coco: val 2017; Pascal VOC: test 2007; Cityscapes: val;

<span style="color:grey; opacity: 0.6">( Image credit: [Benchmarking Robustness in Object Detection](https://arxiv.org/pdf/1907.07484v1.pdf) )</span>
Task: Video Quality Assessment
Area: computer-vision

Video Quality Assessment is a computer vision task aiming to mimic video-based human subjective perception. The goal is to produce a mos score, where higher score indicates better perceptual quality. Some well-known benchmarks for this task are KoNViD-1k, LIVE-VQC, YouTube-UGC and LSVQ.  SROCC/PLCC/RMSE are usually used to evaluate the performance of different models.
Task: Semi-Supervised Image Classification
Area: computer-vision

Semi-supervised image classification leverages unlabelled data as well as labelled data to increase classification performance.

You may want to read some blog posts to get an overview before reading the papers and checking the leaderboards:

- [An overview of proxy-label approaches for semi-supervised learning](https://ruder.io/semi-supervised/) - Sebastian Ruder
- [Semi-Supervised Learning in Computer Vision](https://amitness.com/2020/07/semi-supervised-learning/) - Amit Chaudhary

<span style="color:grey; opacity: 0.6">( Image credit: [Self-Supervised Semi-Supervised Learning](https://arxiv.org/pdf/1905.03670v2.pdf) )</span>
Task: Gait Recognition in the Wild
Area: computer-vision

Gait Recognition in the Wild refers to methods under real-world senses, i.e., unconstrained environment.
Task: 3D Object Reconstruction
Area: computer-vision

Image: [Choy et al](https://arxiv.org/pdf/1604.00449v1.pdf)
Task: Video Matting
Area: computer-vision

Image credit: [https://arxiv.org/pdf/2012.07810v1.pdf](https://arxiv.org/pdf/2012.07810v1.pdf)
Task: Gait Recognition
Area: computer-vision

<span style="color:grey; opacity: 0.6">( Image credit: [GaitSet: Regarding Gait as a Set for Cross-View Gait Recognition](https://github.com/AbnerHqC/GaitSet) )</span>
Task: Birds Eye View Object Detection
Area: computer-vision

KITTI birds eye view detection task
Task: Medical Image Enhancement
Area: computer-vision

Aims to improve the perceptual quality of low-quality medical images
Task: Visual Tracking
Area: computer-vision

**Visual Tracking** is an essential and actively researched problem in the field of computer vision with various real-world applications such as robotic services, smart surveillance systems, autonomous driving, and human-computer interaction. It refers to the automatic estimation of the trajectory of an arbitrary target object, usually specified by a bounding box in the first frame, as it moves around in subsequent video frames.


<span class="description-source">Source: [Learning Reinforced Attentional Representation for End-to-End Visual Tracking ](https://arxiv.org/abs/1908.10009)</span>
Task: 3D Lane Detection
Area: computer-vision

The goal of **3D Lane Detection** is to perceive lanes that provide guidance for autonomous vehicles. A lane can be represented as a visible laneline or a conceptual centerline. Furthermore, a lane obtains extra attributes from the understanding of the surrounding environment.

<span style="color:grey; opacity: 0.6">( Image credit: [OpenLane-V2](https://github.com/OpenDriveLab/OpenLane-V2 ) )</span>
Task: Face Detection
Area: computer-vision

**Face Detection** is a computer vision task that involves automatically identifying and locating human faces within digital images or videos. It is a fundamental technology that underpins many applications such as face recognition, face tracking, and facial analysis.

<span style="color:grey; opacity: 0.6">( Image credit: [insightface](https://github.com/deepinsight/insightface) )</span>
Task: Infrared And Visible Image Fusion
Area: computer-vision

Image fusion with paired infrared and visible images
Task: 3D Point Cloud Linear Classification
Area: computer-vision

Training a linear classifier(e.g. SVM) on the embeddings/representations of 3D point clouds. The embeddings/representations are usually trained in an unsupervised manner.
Task: 3D Object Detection
Area: computer-vision

**3D Object Detection** is a task in computer vision where the goal is to identify and locate objects in a 3D environment based on their shape, location, and orientation. It involves detecting the presence of objects and determining their location in the 3D space in real-time. This task is crucial for applications such as autonomous vehicles, robotics, and augmented reality.

<span style="color:grey; opacity: 0.6">( Image credit: [AVOD](https://github.com/kujason/avod) )</span>
Task: Semantic Segmentation
Area: computer-vision

**Semantic Segmentation** is a computer vision task in which the goal is to categorize each pixel in an image into a class or object. The goal is to produce a dense pixel-wise segmentation map of an image, where each pixel is assigned to a specific class or object. Some example benchmarks for this task are Cityscapes, PASCAL VOC and ADE20K. Models are usually evaluated with the Mean Intersection-Over-Union (Mean IoU) and Pixel Accuracy metrics. 

<span style="color:grey; opacity: 0.6">( Image credit: [CSAILVision](https://github.com/CSAILVision/semantic-segmentation-pytorch) )</span>
Task: Plan2Scene
Area: computer-vision

Converting floorplans + RGB photos to textured 3D mesh models of houses.
Task: Offline Handwritten Chinese Character Recognition
Area: computer-vision

Handwritten Chinese characters recognition is the task of detecting and interpreting the components of Chinese characters (i.e. radicals and two-dimensional structures).
Task: De-aliasing
Area: computer-vision

De-aliasing is the problem of recovering the original high-frequency information that has been aliased during the acquisition of an image.
Task: Video Recognition
Area: computer-vision

**Video Recognition** is a process of obtaining, processing, and analysing data that it receives from a visual source, specifically video.
Task: 3D Plane Detection
Area: computer-vision

Image: [Liu et al](https://arxiv.org/pdf/1812.04072v2.pdf)
Task: Supervised Video Summarization
Area: computer-vision

**Supervised video summarization** rely on datasets with human-labeled ground-truth annotations (either in the form of video summaries, as in the case of the [SumMe](https://paperswithcode.com/dataset/summe) dataset, or in the form of frame-level importance scores, as in the case of the [TVSum](https://paperswithcode.com/dataset/tvsum-1) dataset), based on which they try to discover the underlying criterion for video frame/fragment selection and video summarization.

Source: [Video Summarization Using Deep Neural Networks: A Survey](https://arxiv.org/abs/2101.06072)
Task: Self-Supervised Anomaly Detection
Area: computer-vision

Self-Supervision towards anomaly detection
Task: RGB-D Salient Object Detection
Area: computer-vision

RGB-D Salient object detection (SOD) aims at distinguishing the most visually distinctive objects or regions in a scene from the given RGB and Depth data. It has a wide range of applications, including video/image segmentation, object recognition, visual tracking, foreground maps evaluation, image retrieval, content-aware image editing, information discovery, photosynthesis, and weakly
supervised semantic segmentation. Here, depth information plays an important complementary role in finding salient objects. Online benchmark: http://dpfan.net/d3netbenchmark.


<span style="color:grey; opacity: 0.6">( Image credit: [Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks, TNNLS20](https://ieeexplore.ieee.org/abstract/document/9107477) )</span>
Task: Synthetic Image Detection
Area: computer-vision

Identify if the image is real or generated/manipulated by any generative models (GAN or Diffusion).
Task: Vocabulary-free Image Classification
Area: computer-vision

Recent advances in large vision-language models have revolutionized the image classification paradigm. Despite showing impressive zero-shot capabilities, a pre-defined set of categories, a.k.a. the vocabulary, is assumed at test time for composing the textual prompts. However, such assumption can be impractical when the semantic context is unknown and evolving. Vocabulary-free Image Classification (VIC) aims to assign to an input image a class that resides in an unconstrained language-induced semantic space, without the prerequisite of a known vocabulary.
Task: Video Generation
Area: computer-vision

<span style="color:grey; opacity: 0.6">( Various Video Generation Tasks.
Gif credit: [MaGViT](https://paperswithcode.com/paper/magvit-masked-generative-video-transformer) )</span>
Task: One-Shot Instance Segmentation
Area: computer-vision

<span style="color:grey; opacity: 0.6">( Image credit: [Siamese Mask R-CNN
](https://github.com/bethgelab/siamese-mask-rcnn) )</span>
Task: Explainable artificial intelligence
Area: computer-vision

XAI refers to methods and techniques in the application of artificial intelligence (AI) such that the results of the solution can be understood by humans. It contrasts with the concept of the "black box" in machine learning where even its designers cannot explain why an AI arrived at a specific decision. XAI may be an implementation of the social right to explanation. XAI is relevant even if there is no legal right or regulatory requirement—for example, XAI can improve the user experience of a product or service by helping end users trust that the AI is making good decisions. This way the aim of XAI is to explain what has been done, what is done right now, what will be done next and unveil the information the actions are based on. These characteristics make it possible (i) to confirm existing knowledge (ii) to challenge existing knowledge and (iii) to generate new assumptions.
Task: Key-Frame-based Video Super-Resolution (K = 15)
Area: computer-vision

Key-Frame-based Video Super-Resolution is a sub-task of [Video Super-Resolution](https://paperswithcode.com/task/video-super-resolution), where, in addition to the low-resolution frames, high-resolution ground-truth frames for every Kth input frame are also provided as inputs to the model. For example, if `[LR-frame-1, LR-frame-2, LR-frame-3, ..., LR-frame-100]` is the sequence of low-resolution frames to be upscaled, the Key-Frame-based Video Super-Resolution (K = 15) model is also provided with the high-resolution frames `[HR-frame-1, HR-frame-16, ..., HR-frame-91]` . Key-frames are excluded when measuring the evaluation metrics.
Task: Amodal Instance Segmentation
Area: computer-vision

Different from traditional segmentation which only focuses on visible regions, amodal instance segmentation also predicts the occluded parts of object instances.  

Description Credit: [Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers, CVPR'21](https://openaccess.thecvf.com/content/CVPR2021/papers/Ke_Deep_Occlusion-Aware_Instance_Segmentation_With_Overlapping_BiLayers_CVPR_2021_paper.pdf)
Task: DeepFake Detection
Area: computer-vision

**DeepFake Detection** is the task of detecting fake videos or images that have been generated using deep learning techniques. Deepfakes are created by using machine learning algorithms to manipulate or replace parts of an original video or image, such as the face of a person. The goal of deepfake detection is to identify such manipulations and distinguish them from real videos or images.

Description source: [DeepFakes: a New Threat to Face Recognition? Assessment and Detection](https://arxiv.org/pdf/1812.08685.pdf)

Image source: [DeepFakes: a New Threat to Face Recognition? Assessment and Detection](https://paperswithcode.com/paper/deepfakes-a-new-threat-to-face-recognition)
Task: Action Detection
Area: computer-vision

Action Detection aims to find both where and when an action occurs within a video clip and classify what the action is taking place. Typically results are given in the form of action tublets, which are action bounding boxes linked across time in the video. This is related to temporal localization, which seeks to identify the start and end frame of an action, and action recognition,  which seeks only to classify which action is taking place and typically assumes a trimmed video.
Task: Human-Object Interaction Detection
Area: computer-vision

Human-Object Interaction (HOI) detection is a task of identifying "a set of interactions" in an image, which involves the i) localization of the subject (i.e., humans) and target (i.e., objects) of interaction, and ii) the classification of the interaction labels.
Task: Lighting Estimation
Area: computer-vision

Lighting Estimation analyzes given images to provide detailed information about the lighting in a scene.
Task: Road Damage Detection
Area: computer-vision

Road damage detection is the task of detecting damage in roads.

<span style="color:grey; opacity: 0.6">( Image credit: [Road Damage Detection And Classification In Smartphone Captured Images Using Mask R-CNN](https://arxiv.org/pdf/1811.04535v1.pdf) )</span>
Task: Multiple Object Tracking
Area: computer-vision

**Multiple Object Tracking** is the problem of automatically identifying multiple objects in a video and representing them as a set of trajectories with high accuracy.


<span class="description-source">Source: [SOT for MOT ](https://arxiv.org/abs/1712.01059)</span>
Task: Medical Image Denoising
Area: computer-vision

Image credit: [Learning Medical Image Denoising with Deep Dynamic Residual Attention Network](https://paperswithcode.com/paper/learning-medical-image-denoising-with-deep)
Task: 3D Semantic Scene Completion from a single RGB image
Area: computer-vision

This task relies on a single RGB image to infer the dense 3D voxelized semantic scene.
Task: Rolling Shutter Correction
Area: computer-vision

Rolling Shutter Correction
Task: Learning with coarse labels
Area: computer-vision

Learning fine-grained representation with coarsely-labelled dataset, which can significantly reduce the labelling cost. As a simple example, for the task of differentiation between different pets, we need a knowledgeable cat lover to distinguish between ‘British short’ and ‘Siamese’, but even a child annotator may help to discriminate between ‘cat’ and ‘non-cat’.
Task: Open Vocabulary Object Detection
Area: computer-vision

Open-vocabulary detection (OVD) aims to generalize beyond the limited number of base classes labeled during the training phase. The goal is to detect novel classes defined by an unbounded
(open) vocabulary at inference.
Task: Surface Normals Estimation from Point Clouds
Area: computer-vision

Parent task: 3d Point Clouds Analysis
Task: Video Captioning
Area: computer-vision

**Video Captioning** is a task of automatic captioning a video by understanding the action and event in the video which can help in the retrieval of the video efficiently through text.


<span class="description-source">Source: [NITS-VC System for VATEX Video Captioning Challenge 2020 ](https://arxiv.org/abs/2006.04058)</span>
Task: Video Compression
Area: computer-vision

**Video Compression** is a process of reducing the size of an image or video file by exploiting spatial and temporal redundancies within an image or video frame and across multiple video frames. The ultimate goal of a successful Video Compression system is to reduce data volume while retaining the perceptual quality of the decompressed data.


<span class="description-source">Source: [Adversarial Video Compression Guided by Soft Edge Detection ](https://arxiv.org/abs/1811.10673)</span>
Task: Real-time Instance Segmentation
Area: computer-vision

Similar to its parent task, instance segmentation, but with the goal of achieving real-time capabilities under a defined setting.

Image Credit: [SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation](https://arxiv.org/pdf/2007.14772v1.pdf)
Task: Few-Shot Video Object Detection
Area: computer-vision

Few-Shot Video Object Detection
(FSVOD): given only a few support images of the target
object in an unseen class, detect all the objects belonging to
the same class in a given query video.
Task: Small Data Image Classification
Area: computer-vision

Supervised image classification with tens to hundreds of labeled training examples.
Task: 3D Shape Recognition
Area: computer-vision

Image: [Wei et al](https://arxiv.org/pdf/1908.10098v1.pdf)
Task: Video Narrative Grounding
Area: computer-vision

**Video Narrative Grounding** is the task of linking video narratives to specific video segments. The input is a video with a text description (the narrative) and the positions of certain nouns marked. For each marked
noun, the method must output a segmentation mask for the
object it refers to, in each video frame.

Source: [Connecting Vision and Language with Video Localized Narratives](/paper/connecting-vision-and-language-with-video)
Task: Action Anticipation
Area: computer-vision

Next action anticipation is defined as observing 1, ... , T frames and predicting the action that happens after a gap of T_a seconds. It is important to note that a new action starts after T_a seconds that is not seen in the observed frames. Here T_a=1 second.
Task: Depth Estimation
Area: computer-vision

**Depth Estimation** is the task of measuring the distance of each pixel relative to the camera. Depth is extracted from either monocular (single) or stereo (multiple views of a scene) images. Traditional methods use multi-view geometry to find the relationship between the images. Newer methods can directly estimate depth by minimizing the regression loss, or by learning to generate a novel view from a sequence. The most popular benchmarks are KITTI and NYUv2. Models are typically evaluated according to a RMS metric.

<span class="description-source">Source: [DIODE: A Dense Indoor and Outdoor DEpth Dataset ](https://arxiv.org/abs/1908.00463)</span>
Task: Motion Synthesis
Area: computer-vision

Image source: [Multi-View Motion Synthesis via Applying Rotated Dual-Pixel Blur Kernels
](https://paperswithcode.com/paper/multi-view-motion-synthesis-via-applying)
Task: Intrinsic Image Decomposition
Area: computer-vision

**Intrinsic Image Decomposition** is the process of separating an image into its formation components such as reflectance (albedo) and shading (illumination). Reflectance is the color of the object, invariant to camera viewpoint and illumination conditions, whereas shading, dependent on camera viewpoint and object geometry, consists of different illumination effects, such as shadows, shading and inter-reflections. Using intrinsic images, instead of the original images, can be beneficial for many computer vision algorithms. For instance, for shape-from-shading algorithms, the shading images contain important visual cues to recover geometry, while for segmentation and detection algorithms, reflectance images can be beneficial as they are independent of confounding illumination effects. Furthermore, intrinsic images are used in a wide range of computational photography applications, such as material recoloring, relighting, retexturing and stylization.


<span class="description-source">Source: [CNN based Learning using Reflection and Retinex Models for Intrinsic Image Decomposition ](https://arxiv.org/abs/1712.01056)</span>
Task: Neural Rendering
Area: computer-vision

Given a representation of a 3D scene of some kind (point cloud, mesh, voxels, etc.), the task is to create an algorithm that can produce photorealistic renderings of this scene from an arbitrary viewpoint. Sometimes, the task is accompanied by image/scene appearance manipulation.
Task: 3D Human Dynamics
Area: computer-vision

Image: [Zhang et al](https://openaccess.thecvf.com/content_ICCV_2019/papers/Zhang_Predicting_3D_Human_Dynamics_From_Video_ICCV_2019_paper.pdf)
Task: 3D Object Recognition
Area: computer-vision

3D object recognition is the task of recognising objects from 3D data.

Note that there are related tasks you can look at, such as [3D Object Detection](https://paperswithcode.com/task/3d-object-detection) which have more leaderboards.

<span style="color:grey; opacity: 0.6">(Image credit: [Look Further to Recognize Better](https://arxiv.org/pdf/1907.12924v1.pdf))</span>
Task: Object Proposal Generation
Area: computer-vision

Object proposal generation is a preprocessing technique that has been widely used in current object detection pipelines to guide the search of objects and avoid exhaustive sliding window search across images.

<span style="color:grey; opacity: 0.6">( Image credit: [Multiscale Combinatorial Grouping
for Image Segmentation and Object Proposal Generation](https://arxiv.org/pdf/1503.00848v4.pdf) )</span>
Task: 3D Shape Generation
Area: computer-vision

Image: [Mo et al](https://arxiv.org/pdf/1908.00575v1.pdf)
Task: Multi-Animal Tracking with identification
Area: computer-vision

Tracking all animals in a video maintaining their identities after touches or occlusions.
Task: Real-Time Semantic Segmentation
Area: computer-vision

Semantic Segmentation is a computer vision task that involves assigning a semantic label to each pixel in an image. In **Real-Time Semantic Segmentation**, the goal is to perform this labeling quickly and accurately in real-time, allowing for the segmentation results to be used for tasks such as object recognition, scene understanding, and autonomous navigation.

<span style="color:grey; opacity: 0.6">( Image credit: [TorchSeg](https://github.com/ycszen/TorchSeg) )</span>