nils-herrmann/clean_scirepeval huggingface dataset:
Compressive Quantization for Fast Object Instance Search in Videos

Most of current visual search systems focus on image-to-image (point-to-point) search such as image and object retrieval. Nevertheless, fast image-to-video (point-to-set) search is much less exploited. This paper tackles object instance search in videos, where efficient point-to-set matching is essential. Through jointly optimizing vector quantization and hashing, we propose compressive quantization method to compressM object proposals extracted from each video into only k binary codes, where k ≪ M. Then the similarity between the query object and the whole video can be determined by the Hamming distance between the queryfs binary code and the videofs best-matched binary code. Our compressive quantization not only enables fast search but also significantly reduces the memory cost of storing the video features. Despite the high compression ratio, our proposed compressive quantization still can effec- tively retrieve small objects in large video datasets. System- atic experiments on three benchmark datasets verify the ef- fectiveness and efficiency of our compressive quantization.
Adobe Boxes: Locating Object Proposals Using Object Adobes

Despite the previous efforts of object proposals, the detection rates of the existing approaches are still not satisfactory enough. To address this, we propose Adobe Boxes to efficiently locate the potential objects with fewer proposals, in terms of searching the object adobes that are the salient object parts easy to be perceived. Because of the visual difference between the object and its surroundings, an object adobe obtained from the local region has a high probability to be a part of an object, which is capable of depicting the locative information of the proto-object. Our approach comprises of three main procedures. First, the coarse object proposals are acquired by employing randomly sampled windows. Then, based on local-contrast analysis, the object adobes are identified within the enlarged bounding boxes that correspond to the coarse proposals. The final object proposals are obtained by converging the bounding boxes to tightly surround the object adobes. Meanwhile, our object adobes can also refine the detection rate of most state-of-the-art methods as a refinement approach. The extensive experiments on four challenging datasets (PASCAL VOC2007, VOC2010, VOC2012, and ILSVRC2014) demonstrate that the detection rate of our approach generally outperforms the state-of-the-art methods, especially with relatively small number of proposals. The average time consumed on one image is about 48 ms, which nearly meets the real-time requirement.
Fast Action Detection via Discriminative Random Forest Voting and Top-K Subvolume Search

Multiclass action detection in complex scenes is a challenging problem because of cluttered backgrounds and the large intra-class variations in each type of actions. To achieve efficient and robust action detection, we characterize a video as a collection of spatio-temporal interest points, and locate actions via finding spatio-temporal video subvolumes of the highest mutual information score towards each action class. A random forest is constructed to efficiently generate discriminative votes from individual interest points, and a fast top-K subvolume search algorithm is developed to find all action instances in a single round of search. Without significantly degrading the performance, such a top-K search can be performed on down-sampled score volumes for more efficient localization. Experiments on a challenging MSR Action Dataset II validate the effectiveness of our proposed multiclass action detection method. The detection speed is several orders of magnitude faster than existing methods.
Real time hand gesture recognition via finger-emphasized multi-scale description

The development of depth cameras, e.g., the Kinect sensor, provides new opportunities for human computer interaction (HCI). Although the Kinect sensor has been extensively applied for human tracking, human action recognition and hand gesture recognition, real time hand gesture recognition is still a challenging problem. In this paper, we propose a new real time hand gesture recognition method. To represent the noisy and articulated hand shape segmented from the Kinect images, a finger emphasized multi-scale descriptor is proposed. To fully utilize hand shape features, this descriptor incorporates three types of parameters of multiple scales, which emphasize the finger features. Hand gesture recognition is then achieved with both DTW algorithm and BP neural network. Extensive experimental results and the comparison with state-of-the-art methods demonstrate that our method is accurate (a 100% accuracy on a challenging hand gesture dataset), efficient (average 0.941ms per frame), and robust to noise, articulations and rigid transformations.
Discovering Primary Objects in Videos by Saliency Fusion and Iterative Appearance Estimation

In this paper, we propose a new method for detecting primary objects in unconstrained videos in a completely automatic setting. Here, we define the primary object in a video as the object that presents saliently in most of the frames. Unlike previous works considering only local saliency detection or common pattern discovery, the proposed method integrates the local visual/motion saliency extracted from each frame, global appearance consistency throughout the video, and spatiotemporal smoothness constraint on object trajectories. We first identify a temporal coherent salient region throughout the whole video, and then explicitly learn a global appearance model to distinguish the primary object against the background. In order to obtain high-quality saliency estimations from both appearance and motion cues, we propose a novel self-adaptive saliency map fusion method by learning the reliability of saliency maps from labeled data. As a whole, our method can robustly localize and track primary objects in diverse video content, and handle the challenges such as fast object and camera motion, large scale and appearance variation, background clutter, and pose deformation. Moreover, compared with some existing approaches that assume the object is present in all the frames, our approach can naturally handle the case where the object is present only in part of the frames, e.g., the object enters the scene in the middle of the video or leaves the scene before the video ends. We also propose a new video data set containing 51 videos for primary object detection with per-frame ground-truth labeling. Quantitative experiments on several challenging video data sets demonstrate the superiority of our method compared with the recent state of the arts.
Parsing the Hand in Depth Images

Hand pose tracking and gesture recognition are useful for human-computer interaction, while a major problem is the lack of discriminative features for compact hand representation. We present a robust hand parsing scheme to extract a high-level description of the hand from the depth image. A novel distance-adaptive selection method is proposed to get more discriminative depth-context features. Besides, we propose a Superpixel-Markov Random Field (SMRF) parsing scheme to enforce the spatial smoothness and the label co-occurrence prior to remove the misclassified regions. Compared to pixel-level filtering, the SMRF scheme is more suitable to model the misclassified regions. By fusing the temporal constraints, its performance can be further improved. Overall, the proposed hand parsing scheme is accurate and efficient. The tests on synthesized dataset show it gives much higher accuracy for single-frame parsing and enhanced robustness for continuous sequence parsing compared to benchmarks. The tests on real-world depth images of the hand and human body show the robustness to complex hand configurations of our method and its generalization power to different kinds of articulated objects.
Curb detection and tracking using 3D-LIDAR scanner

This paper presents a novel road curb detection method using 3D-LIDAR scanner. To detect the curbs, the ground points are separated from the pointcloud first. Then the candidate curb points are selected using three spatial cues: the elevation difference, gradient value and normal orientation. Afterwards the false curb points caused by obstacles are removed using the short-term memory technique. Next the curbs are fitted using the parabola model. Finally, the particle filter is used to smooth the curb detection result. The proposed approach was evaluated on a dataset collected by an autonomous ground vehicle driving around the Ford Research campus and downtown Dearborn. Our curb detection results are accurate and robust despite variations introduced by moving vehicles and pedestrians, static obstacles, road curvature changes, etc.
Real-time hierarchical fusion system for semantic segmentation in offroad scenes

Semantic segmentation is an important task for autonomous vehicle navigation in off road environments. However, several natural factors make this problem uniquely challenging. For example, road segmentation is often difficult under heavy shadow or steel terrain, and dangerous muddy water puddles may have the similar visual appearance to dirt road surfaces (and thus are hard to identify). To tacule these challenges, we present a semantic segmentation system based on a two-stage hierarchical fusion pipeline. The first stage improves the road segmentation by effectively fusing information from camera and 3D Lidar point cloud. The second stage is dedicated to detecting water puddles, based on the results from the first stage. Due to the parallelized architecture, our system can be deployed for real-time applications. We achieved an F1 score of around 93% for road segmentation and 80% for water puddle segmentation at more than 10 Hz.
Randomized visual phrases for object search

Accurate matching of local features plays an essential role in visual object search. Instead of matching individual features separately, using the spatial context, e.g., bundling a group of co-located features into a visual phrase, has shown to enable more discriminative matching. Despite previous work, it remains a challenging problem to extract appropriate spatial context for matching. We propose a randomized approach to deriving visual phrase, in the form of spatial random partition. By averaging the matching scores over multiple randomized visual phrases, our approach offers three benefits: 1) the aggregation of the matching scores over a collection of visual phrases of varying sizes and shapes provides robust local matching; 2) object localization is achieved by simple thresholding on the voting map, which is more efficient than subimage search; 3) our algorithm lends itself to easy parallelization and also allows a flexible trade-off between accuracy and speed by adjusting the number of partition times. Both theoretical studies and experimental comparisons with the state-of-the-art methods validate the advantages of our approach.
Self-Supervised Online Metric Learning With Low Rank Constraint for Scene Categorization

Conventional visual recognition systems usually train an image classifier in a bath mode with all training data provided in advance. However, in many practical applications, only a small amount of training samples are available in the beginning and many more would come sequentially during online recognition. Because the image data characteristics could change over time, it is important for the classifier to adapt to the new data incrementally. In this paper, we present an online metric learning method to address the online scene recognition problem via adaptive similarity measurement. Given a number of labeled data followed by a sequential input of unseen testing samples, the similarity metric is learned to maximize the margin of the distance among different classes of samples. By considering the low rank constraint, our online metric learning model not only can provide competitive performance compared with the state-of-the-art methods, but also guarantees convergence. A bi-linear graph is also defined to model the pair-wise similarity, and an unseen sample is labeled depending on the graph-based label propagation, while the model can also self-update using the more confident new samples. With the ability of online learning, our methodology can well handle the large-scale streaming video data with the ability of incremental self-updating. We evaluate our model to online scene categorization and experiments on various benchmark datasets and comparisons with state-of-the-art methods demonstrate the effectiveness and efficiency of our algorithm.
Discovering Thematic Patterns in Videos via Cohesive Sub-graph Mining

One category of videos usually contains the same thematic pattern, e.g., the spin action in skating videos. The discovery of the thematic pattern is essential to understand and summarize the video contents. This paper addresses two critical issues in mining thematic video patterns: (1) automatic discovery of thematic patterns without any training or supervision information, and (2) accurate localization of the occurrences of all thematic patterns in videos. The major contributions are two-fold. First, we formulate the thematic video pattern discovery as a cohesive sub-graph selection problem by finding a sub-set of visual words that are spatio-temporally collocated. Then spatio-temporal branch-and-bound search can locate all instances accurately. Second, a novel method is proposed to efficiently find the cohesive sub-graph of maximum overall mutual information scores. Our experimental results on challenging commercial and action videos show that our approach can discover different types of thematic patterns despite variations in scale, view-point, color and lighting conditions, or partial occlusions. Our approach is also robust to the videos with cluttered and dynamic backgrounds.
Robust 3D Hand Pose Estimation From Single Depth Images Using Multi-View CNNs

Articulated hand pose estimation is one of core technologies in human–computer interaction. Despite the recent progress, most existing methods still cannot achieve satisfactory performance, partly due to the difficulty of the embedded high-dimensional nonlinear regression problem. Most existing data-driven methods directly regress 3D hand pose from 2D depth image, which cannot fully utilize the depth information. In this paper, we propose a novel multi-view convolutional neural network (CNN)-based approach for 3D hand pose estimation. To better exploit 3D information in the depth image, we project the point cloud generated from the query depth image onto multiple views of two projection settings and integrate them for more robust estimation. Multi-view CNNs are trained to learn the mapping from projected images to heat-maps, which reflect probability distributions of joints on each view. These multi-view heat-maps are then fused to estimate the optimal 3D hand pose with learned pose priors, and the unreliable information in multi-view heat-maps is suppressed using a view selection method. Experimental results show that the proposed method is superior to the state-of-the-art methods on two challenging data sets. Furthermore, a cross-data set experiment also validates that our proposed approach has good generalization ability.
Robust Part-Based Hand Gesture Recognition Using Kinect Sensor

The recently developed depth sensors, e.g., the Kinect sensor, have provided new opportunities for human-computer interaction (HCI). Although great progress has been made by leveraging the Kinect sensor, e.g., in human body tracking, face recognition and human action recognition, robust hand gesture recognition remains an open problem. Compared to the entire human body, the hand is a smaller object with more complex articulations and more easily affected by segmentation errors. It is thus a very challenging problem to recognize hand gestures. This paper focuses on building a robust part-based hand gesture recognition system using Kinect sensor. To handle the noisy hand shapes obtained from the Kinect sensor, we propose a novel distance metric, Finger-Earth Mover's Distance (FEMD), to measure the dissimilarity between hand shapes. As it only matches the finger parts while not the whole hand, it can better distinguish the hand gestures of slight differences. The extensive experiments demonstrate that our hand gesture recognition system is accurate (a 93.2% mean accuracy on a challenging 10-gesture dataset), efficient (average 0.0750 s per frame), robust to hand articulations, distortions and orientation or scale changes, and can work in uncontrolled environments (cluttered backgrounds and lighting conditions). The superiority of our system is further demonstrated in two real-life HCI applications.
mCENTRIST: A Multi-Channel Feature Generation Mechanism for Scene Categorization

mCENTRIST, a new multichannel feature generation mechanism for recognizing scene categories, is proposed in this paper. mCENTRIST explicitly captures the image properties that are encoded jointly by two image channels, which is different from popular multichannel descriptors. In order to avoid the curse of dimensionality, tradeoffs at both feature and channel levels have been executed to make mCENTRIST computationally practical. As a result, mCENTRIST is both efficient and easy to implement. In addition, a hyperopponent color space is proposed by embedding Sobel information into the opponent color space for further performance improvements. Experiments show that mCENTRIST outperforms established multichannel descriptors on four RGB and RGB-near infrared data sets, including aerial orthoimagery, indoor, and outdoor scene category recognition tasks. Experiments also verify that the hyper opponent color space enhances descriptors' performance effectively.
Hierarchical multi-feature fusion for multimodal data analysis

Multimedia data is usually represented with different low-level features, and different types of multimedia data, namely multimodal data, often coexist in many data sources. It is interesting and challenging to learn comprehensive semantics from multiple low-level features for multimodal data analysis. In this paper, we propose a new algorithm, namely hierarchical multi-feature fusion for multimodal data semantics understanding. Our approach explores intra-modality structural information derived from each type of feature, and further proposes transductive inter-modality fusion strategy, which analyzes canonical correlation between different modalities. Extensive experiments are conducted on collected multimodal database for data classification application. The experiment results show that the performance of our algorithm is remarkable and demonstrate its superiority over several existing algorithms.
Fried Binary Embedding: From High-Dimensional Visual Features to High-Dimensional Binary Codes

Most existing binary embedding methods prefer compact binary codes (<inline-formula> <tex-math notation="LaTeX">$b$ </tex-math></inline-formula>-dimensional) to avoid high computational and memory cost of projecting high-dimensional visual features (<inline-formula> <tex-math notation="LaTeX">$d$ </tex-math></inline-formula>-dimensional, <inline-formula> <tex-math notation="LaTeX">$b < d$ </tex-math></inline-formula>). We argue that long binary codes (<inline-formula> <tex-math notation="LaTeX">$b \sim \mathcal {O}(d)$ </tex-math></inline-formula>) are critical to fully utilize the discriminative power of high-dimensional visual features, and can achieve better results in various tasks such as approximate nearest neighbor search. Generating long binary codes involves large projection matrix and high-dimensional matrix-vector multiplication, thus is memory and compute intensive. We propose Fried binary embedding (FBE) and Supervised Fried Binary Embedding (SuFBE), to tackle these problems. FBE is suitable for most of the practical applications in which the labels of training data are not given, while SuFBE can significantly boost the accuracy in the cases that the training labels are available. The core idea is to decompose the projection matrix using adaptive Fastfood transform, which is the multiplication of several structured matrices. As a result, FBE and SuFBE can reduce the computational complexity from <inline-formula> <tex-math notation="LaTeX">$\mathcal {O}(d^{2})$ </tex-math></inline-formula> to <inline-formula> <tex-math notation="LaTeX">$\mathcal {O}(d\log d)$ </tex-math></inline-formula>, and memory cost from <inline-formula> <tex-math notation="LaTeX">$\mathcal {O}(d^{2})$ </tex-math></inline-formula> to <inline-formula> <tex-math notation="LaTeX">$\mathcal {O}(d)$ </tex-math></inline-formula>, respectively. More importantly, by using the structured matrices, FBE and SuFBE can well regulate projection matrix by reducing its tunable parameters and lead to even better accuracy than using either unconstrained projection matrix (like ITQ) or sparse matrix such as SP and SSP with the same long code length. Experimental comparisons with state-of-the-art methods over various visual applications demonstrate both the efficiency and performance advantages of FBE and SuFBE.
Voxel labelling in CT images with data-driven contextual features

Spatial contextual information is useful for voxel labelling and especially suitable for the images with relatively fixed scene structure such as CT images. For each voxel, the intensity values of nearby and far away positions are sampled as its contextual features and such contextual features have shown promising performance. However how to determine sampling position to construct good contextual features remains a critical problem since a good sampling could significantly improve the classification performance. In this paper we proposed a novel approach by discovering discriminative sampling pattern. We emphasize that the sampling pattern is not hand craft but data driven and can cater to a particular type of problem, such as kidneys labelling in contrast-enhanced CT images. After discriminative pattern is discovered it can be adapted for use in other datasets of the same problem. Experiments on kidney dataset showed considerable improvements over competing methods.
Two-layer optimized light field display using depth initialization

In this paper, we propose a method to optimize two-layer light field display using depth initialization. In contrast to existing trade-off work between performance and processing time, this paper firstly models the display principle of layered light field display, and then performs layered initialization with the prior known depth of 3D objects, and finally optimizes the layered images for light field display. Experiments demonstrate that the proposed initialization method can obviously save the iterations and related processing time for the existing online or offline algorithms to achieve the same reconstructed peak signal to noise ratio (PSNR) and present a better subjective reconstructed performance using the same computation resource.
Optimizing Inter-server Communication for Online Social Networks

Distributed storage systems are the key infrastructures for hosting the user data of large-scale Online Social Networks (OSNs). The amount of inter-server communication is an important scalability indicator for these systems. Data partitioning and replication are two inter-related issues affecting the inter-server traffic caused by user-initiated read and write operations. This paper investigates the problem of minimizing the total inter-server traffic among a cluster of OSN servers through joint partitioning and replication optimization. We propose a Traffic-Optimized Partitioning and Replication (TOPR) method based on an analysis of how replica allocation affects the inter-server communication. Lightweight algorithms are developed to adjust partitioning and replication dynamically according to data read and write rates. Evaluations with real Facebook and Twitter social graphs show that TOPR significantly reduces the inter-server communication compared with state-of-the-art methods.