nils-herrmann/clean_scirepeval huggingface dataset:
Multi-label Object Categorization Using Histograms of Global Relations

In this paper, we present an object categorization system capable of assigning multiple and related categories for novel objects using multi-label learning. In this system, objects are described using global geometric relations of 3D features. We propose using the Joint SVM method for learning and we investigate the extraction of hierarchical clusters as a higher-level description of objects to assist the learning. We make comparisons with other multi-label learning approaches as well as single-label approaches (including a state-of-the-art methods using different object descriptors). The experiments are carried out on a dataset of 100 objects belonging to 13 visual and action-related categories. The results indicate that multi-label methods are able to identify the relation between the dependent categories and hence perform categorization accordingly. It is also found that extracting hierarchical clusters does not lead to gain in the system's performance. The results also show that using histograms of global relations to describe objects leads to fast learning in terms of the number of samples required for training.
Building pixel classifiers using the interactive teacher/learner (ITL) system

We present an interactive paradigm for the construction of pixel classifiers. The user selects training pixels incrementally, based on real-time feedback from the classifier running in the background. Experiments show that this facilitates the construction of very small yet accurate decision tree classifiers. The framework is extensible in many ways. For example, we show how classifiers can be composed hierarchically and trained to find seagulls in aerial images.
Task-Driven Learning of Spatial Combinations of Visual Features

Solving a visual, interactive task can often be thought of as building a mapping from visual stimuli to appropriate actions. Clearly, the extracted visual characteristics that index into the repertoire of actions must be sufficiently rich to distinguish situations that demand distinct actions. Spatial combinations of local features permit, in principle, the construction of features at various levels of discriminative power. We present an algorithm for selecting relevant spatial combinations of visual features by exercising a given task in a closed-loop learning process based on Reinforcement Learning. The algorithm operates by progressively splitting the perceptual space into distinct regions. Whenever the agent detects perceptual aliasing of distinct world states, it constructs a spatial combination of visual features that disambiguates the aliased states. We demonstrate the efficacy of our algorithm on a version of the classical "Car on the Hill" control problem where position and velocity are presented to the agent visually, in a way that the task is unsolvable using individual point features.
Statistical Learning of Visual Feature Hierarchies

We propose an unsupervised, probabilistic method for learning visual feature hierarchies. Starting from local, low-level features computed at interest point locations, the method combines these primitives into high-level abstractions. Our appearance-based learning method uses local statistical analysis between features and Expectation- Maximization (EM) to identify and code spatial correlations. Spatial correlation is asserted when two features tend to occur at the same relative position of each other. This learning scheme results in a graphical model that allows a probabilistic representation of a flexible visual feature hierarchy. For feature detection, evidence is propagated using Nonparametric Belief Propagation (NBP), a recent generalization of particle filtering. In experiments, the proposed approach demonstrates efficient learning and robust detection of object models in the presence of clutter and occlusion and under view point changes.
Learning object-specific grasp affordance densities

This paper addresses the issue of learning and representing object grasp affordances, i.e. object-gripper relative configurations that lead to successful grasps. The purpose of grasp affordances is to organize and store the whole knowledge that an agent has about the grasping of an object, in order to facilitate reasoning on grasping solutions and their achievability. The affordance representation consists in a continuous probability density function defined on the 6D gripper pose space - 3D position and orientation -, within an object-relative reference frame. Grasp affordances are initially learned from various sources, e.g. from imitation or from visual cues, leading to grasp hypothesis densities. Grasp densities are attached to a learned 3D visual object model, and pose estimation of the visual model allows a robotic agent to execute samples from a grasp hypothesis density under various object poses. Grasp outcomes are used to learn grasp empirical densities, i.e. grasps that have been confirmed through experience. We show the result of learning grasp hypothesis densities from both imitation and visual cues, and present grasp empirical densities learned from physical experience by a robot.
Feature learning for recognition with Bayesian networks

Many realistic visual recognition tasks are "open" in the sense that the number and nature of the categories to be learned are not initially known, and there is no closed set of training images available to the system. We argue that open recognition tasks require incremental learning methods, and feature sets that are capable of expressing distinctions at any level of specificity or generality. We describe progress toward such a system that is based on an infinite combinatorial feature space. Feature primitives can be composed into increasing complex and specific compound features. Distinctive features are learned incrementally, and are incorporated into dynamically updated Bayesian network classifiers. Experimental results illustrate the applicability and potential of our approach.
Probabilistic Detection of Pointing Directions for Human-Robot Interaction

Deictic gestures - pointing at things in human-human collaborative tasks - constitute a pervasive, non-verbal way of communication, used e.g. to direct attention towards objects of interest. In a human-robot interactive scenario, in order to delegate tasks from a human to a robot, one of the key requirements is to recognize and estimate the pose of the pointing gesture. Standard approaches rely on full-body or partial-body postures to detect the pointing direction. We present a probabilistic, appearance-based object detection framework to detect pointing gestures and robustly estimate the pointing direction. Our method estimates the pointing direction without assuming any human kinematic model. We propose a functional model for pointing which incorporates two types of pointing, finger pointing and tool pointing using an object in hand. We evaluate our method on a new dataset with 9 participants pointing at 10 objects.
Generalizing grasps across partly similar objects

The paper starts by reviewing the challenges associated to grasp planning, and previous work on robot grasping. Our review emphasizes the importance of agents that generalize grasping strategies across objects, and that are able to transfer these strategies to novel objects. In the rest of the paper, we then devise a novel approach to the grasp transfer problem, where generalization is achieved by learning, from a set of grasp examples, a dictionary of object parts by which objects are often grasped. We detail the application of dimensionality reduction and unsupervised clustering algorithms to the end of identifying the size and shape of parts that often predict the application of a grasp. The learned dictionary allows our agent to grasp novel objects which share a part with previously seen objects, by matching the learned parts to the current view of the new object, and selecting the grasp associated to the best-fitting part. We present and discuss a proof-of-concept experiment in which a dictionary is learned from a set of synthetic grasp examples. While prior work in this area focused primarily on shape analysis (parts identified, e.g., through visual clustering, or salient structure analysis), the key aspect of this work is the emergence of parts from both object shape and grasp examples. As a result, parts intrinsically encode the intention of executing a grasp.
Learning probabilistic discriminative models of grasp affordances under limited supervision

This paper addresses the problem of learning and efficiently representing discriminative probabilistic models of object-specific grasp affordances particularly when the number of labeled grasps is extremely limited. The proposed method does not require an explicit 3D model but rather learns an implicit manifold on which it defines a probability distribution over grasp affordances. We obtain hypothetical grasp configurations from visual descriptors that are associated with the contours of an object. While these hypothetical configurations are abundant, labeled configurations are very scarce as these are acquired via time-costly experiments carried out by the robot. Kernel logistic regression (KLR) via joint kernel maps is trained to map the hypothesis space of grasps into continuous class-conditional probability values indicating their achievability. We propose a soft-supervised extension of KLR and a framework to combine the merits of semi-supervised and active learning approaches to tackle the scarcity of labeled grasps. Experimental evaluation shows that combining active and semi-supervised learning is favorable in the existence of an oracle. Furthermore, semi-supervised learning outperforms supervised learning, particularly when the labeled data is very limited.
Unsupervised Learning of Dense Hierarchical Appearance Representations

We describe an unsupervised, probabilistic method for learning visual feature hierarchies. Starting from local, low-level features computed at random locations, the method combines features hierarchically. At each level of the hierarchy, pairs of features are identified that tend to occur at stable positions relative to each other, by clustering the configurational distributions of observed feature cooccurrences using expectation-maximization. Stable pairs of features thus identified are combined into higher-level features. This learning scheme results in a graphical model that constitutes a probabilistic representation of a flexible visual feature hierarchy. For detection, evidence is propagated using nonparametric belief propagation, a recent generalization of particle filtering. In experiments, the proposed approach demonstrates effective learning and robust detection of objects in the presence of clutter and occlusion
Teaching a Robot the Semantics of Assembly Tasks

We present a three-level cognitive system in a learning by demonstration context. The system allows for learning and transfer on the sensorimotor level as well as the planning level. The fundamentally different data structures associated with these two levels are connected by an efficient mid-level representation based on so-called “semantic event chains.” We describe details of the representations and quantify the effect of the associated learning procedures for each level under different amounts of noise. Moreover, we demonstrate the performance of the overall system by three demonstrations that have been performed at a project review. The described system has a technical readiness level (TRL) of 4, which in an ongoing follow-up project will be raised to TRL 6.
Learning real-time stereo vergence control

Online learning robotic systems have many desirable properties. This work contributes a reinforcement learning framework for learning a time-constrained closed-loop control policy. The task is to verge the two cameras of a stereo vision system to foveate on the same world feature, within a limited number of perception-action cycles. Online learning is beneficial in at least the following ways: 1) the control parameters are optimized with respect to the characteristics of the environment actually encountered during operation; 2) visual feedback contributes to the choice of the best control action at every step in a multi-step control policy; 3) no initial calibration or explicit modeling of system parameters is required; and 4) the system can be made to adapt to non-stationary environments. Our vergence system provides a running estimate of the resulting verge quality that can be exploited by a real-time scheduler. It is shown to perform superior to two hand-calibrated vergence policies.
Active learning of manipulation sequences

We describe a system allowing a robot to learn goal-directed manipulation sequences such as steps of an assembly task. Learning is based on a free mix of exploration and instruction by an external teacher, and may be active in the sense that the system tests actions to maximize learning progress and asks the teacher if needed. The main component is a symbolic planning engine that operates on learned rules, defined by actions and their pre- and postconditions. Learned by model-based reinforcement learning, rules are immediately available for planning. Thus, there are no distinct learning and application phases. We show how dynamic plans, replanned after every action if necessary, can be used for automatic execution of manipulation sequences, for monitoring of observed manipulation sequences, or a mix of the two, all while extending and refining the rule base on the fly. Quantitative results indicate fast convergence using few training examples, and highly effective teacher intervention at early stages of learning.
Complex affordance learning based on basic affordances

In this paper, we study how complex object affordances can be efficiently learned and how previously learned structures can be used for this purpose. We discuss that besides standard visual features, using previously learned basic affordances in predicting complex affordances would speed up this complex learning task. In order to prove our hypothesis, we compared two different types of complex affordance predictors: The predictors that are based on shape features and the ones that use basic affordances. The results obtained from a synthetic (object, action) interaction database showed that basic-affordance based predictors can generalize over novel objects even with small training sets. This result shows that although the basic affordances are related to basic simpler actions, as they encode object-robot-environment dynamics, they can speed up learning of complex actions.
SCurV: A 3D descriptor for object classification

3D Object recognition is one of the big problems in Computer Vision which has a direct impact in Robotics. There have been great advances in the last decade thanks to point cloud descriptors. These descriptors do very well at recognizing object instances in a wide variety of situations. Of great interest is also to know how descriptors perform in object classification tasks. With that idea in mind, we introduce a descriptor designed for the representation of object classes. Our descriptor, named SCurV, exploits 3D shape information and is inspired by recent findings from neurophysiology. We compute and incorporate surface curvatures and distributions of local surface point projections that represent flatness, concavity and convexity in a 3D object-centered and view-dependent descriptor. These different sources of information are combined in a novel and simple, yet effective, way of combining different features to improve classification results which can be extended to the combination of any type of descriptor. Our experimental setup compares SCurV with other recent descriptors on a large classification task. Using a large and heterogeneous database of 3D objects, we perform our experiments both on a classical, flat classification task and within a novel framework for hierarchical classification. On both tasks, the SCurV descriptor outperformed all other 3D descriptors tested.
Generalized Exemplar-Based Full Pose Estimation from 2D Images without Correspondences

This paper addresses the problem of full pose estimation of objects in 2D images, using registered 2D examples as training data. We present a general formulation of the problem, which departs from traditional approaches by not focusing on one specific type of image features. The proposed algorithm avoids relying on specific model-to-scene correspondences, allowing using similar-looking and generally unmatchable features. We effectively demonstrate this capability by applying the method to edge segments. Our algorithm uses successive histogram-based and probabilistic evaluations, which ultimately recover a complete description of the probability distribution of the pose of the object, in the 6 degree-of-freedom 3D pose space, thereby accounting for the inherent ambiguities in the 2D input data. Furthermore, we propose, in a rigorous framework, an efficient procedure for fusing multiple sources of evidence, such as multiple registered 2D views of the same scene. The proposed method is evaluated qualitatively and quantitatively on synthetic and real test images. It shows promising results under challenging conditions, including occlusions and heavy clutter, while being capable of handling objects with little texture and detail.
A Study of Point Cloud Registration with Probability Product Kernel Functions

3D point cloud registration is an essential problem in 3D object and scene understanding. In many realistic circumstances, however, because of noise during data acquisition and large motion between two point clouds, most existing approaches can hardly work satisfactorily without good initial alignment or manually marked correspondences. Inspired by the popular kernel methods in machine learning community, this paper puts forward a general point cloud registration framework by constructing kernel functions over 3D point clouds. More specifically, Gaussian mixtures Based on the point clouds are established and probability product kernel functions are exploited for the registration. To enhance the generality of the framework, SE(3) on-manifold optimization scheme is employed to compute the optimal motion. Experimental results show that our registration framework works robustly when many outliers are presented and motion between point clouds is relatively large, and compares favorably to related methods.
A robust pushing skill for object delivery between obstacles

Nonprehensile manipulation can play a significant role in complex robotic scenarios, especially for maneuvering non-graspable objects. A big challenge is to construct a robust skill for pushing highly-diverse objects. We present a strategy for pushing unknown objects that differ widely in their properties. For this purpose we introduce the concept of a pushing corridor for cluttered environments that leaves the robot sufficient space for corrective motions. We propose a reactive manipulation skill for pushing objects along this collision-free corridor. The motion of the robot is generated and adapted on the fly based on the observed reactions of the object in response to pushing actions. Our results show that the robot is able to successfully push objects in various complex scenarios.
Adaptive Patch Features for Object Class Recognition with Learned Hierarchical Models

We present a hierarchical generative model for object recognition that is constructed by weakly-supervised learning. A key component is a novel, adaptive patch feature whose width and height are automatically determined. The optimality criterion is based on minimum-variance analysis, which first computes the variance of the appearance model for various patch deformations, and then selects the patch dimensions that yield the minimum variance over the training data. They are integrated into each level of our hierarchical representation that is learned in an iterative, bottom-up fashion. At each level of the hierarchy, pairs of features are identified that tend to occur at stable positions relative to each other, by clustering the configurational distributions of observed feature co-occurrences using expectation-maximization. For recognition, evidence is propagated using nonparametric belief propagation. Discriminative models are learned on the basis of our feature hierarchy by combining a SVM classifier with feature selection based on the Fisher score. Experiments on two very different, challenging image databases demonstrate the effectiveness of this framework for object class recognition, as well as the contribution of the adaptive patch features towards attaining highly competitive results.
Exercising Affordances of Objects: A Part-Based Approach

This study shows how learning relations between affordances facilitates performing robotic tasks. Tasks usually involve multiple affordances. For example, for pounding a nail with a hammer, grasp-ability and pound-ability of the hammer are important for performing the pounding task successfully. Furthermore, these affordances are associated with parts of the hammer. In the pounding task, the head of the hammer affords pounding and the handle of the hammer affords grasping. We propose an Red Green Blue-Depth (RGB-D) part-based approach for performing tasks. In this paper, affordances are linked to object parts. We learn affordances associated with manipulation and execution of the tasks, i.e., grasping for manipulation and pounding for execution in the task of pounding a nail. Since affordances are associated with parts, tasks can be executed directly on the objects. Our approach is evaluated in six different robotic tasks on a real robot. We obtained an average of 65% task detection rate superior to the baseline methods and an average of 77% task success rate.
Supervised Learning of Gesture-Action Associations for Human-Robot Collaboration

As human-robot collaboration methodologies develop robots need to adapt fast learning methods in domestic scenarios. The paper presents a novel approach to learn associations between the human hand gestures and the robot’s manipulation actions. The role of the robot is to operate as an assistant to the user. In this context we propose a supervised learning framework to explore the gesture-action space for human-robot collaboration scenario. The framework enables the robot to learn the gesture-action associations on the fly while performing the task with the user; an example of zero-shot learning. We discuss the effect of an accurate gesture detection in performing the task. The accuracy of the gesture detection system directly accounts for the amount of effort put by the user and the number of actions performed by the robot.
On-line Simultaneous Learning and Tracking of Visual Feature Graphs

Model learning and tracking are two important topics in computer vision. While there are many applications where one of them is used to support the other, there are currently only few where both aid each other simultaneously. In this work, we seek to incrementally learn a graphical model from tracking and to simultaneously use whatever has been learned to improve the tracking in the next frames. The main problem encountered in this situation is that the current intermediate model may be inconsistent with future observations, creating a bias in the tracking results. We propose an uncertain model that explicitly accounts for such uncertainties by representing relations by an appropriately weighted sum of informative (parametric) and uninformative (uniform) components. The method is completely unsupervised and operates in real time.