nils-herrmann/clean_scirepeval huggingface dataset:
Creating audio keywords for event detection in soccer video

This paper presents a novel framework called audio keywords to assist event detection in soccer video. Audio keyword is a middle-level representation that can bridge the gap between low-level features and high-level semantics. Audio keywords are created from low-level audio features by using support vector machine learning. The created audio keywords can be used to detect semantic events in soccer video by applying a heuristic mapping. Experiments of audio keywords creation and event detection based on audio keywords have illustrated promising results. According to the experimental results, we believe that audio keyword is an effective representation that is able to achieve more intuitionistic result for event detection in sports video compared with the method of event detection directly based on low-level features.
A multimodal approach for image de-fencing and depth inpainting

Low cost RGB-D sensors such as the Microsoft Kinect have enabled the use of depth data along with color images. In this work, we propose a multi-modal approach to address the problem of removal of fences/occlusions from images captured using a Kinect camera. We also perform depth completion by fusing data from multiple recorded depth maps affected by occlusions. The availability of aligned image and depth data from Kinect aids us in the detection of the fence locations. However, accurate estimation of the relative shifts between the captured color frames is necessary. Initially, for the case of static scene elements with simple relative motion between the camera and the objects, we propose the use of affine scale-invariant feature transform descriptor (ASIFT) to compute the relative global displacements. We also address the scenario wherein the relative motion between the frames may not be global using the depth map obtained by Kinect. For such a scenario involving complex motion of scene pixels, we use a recently proposed robust optical flow technique. We show results for challenging real-world data wherein the scene is dynamic. The inverse ill-posed problems of estimation of the de-fenced image and the inpainted depth map are solved using an optimization-based framework. Specifically, we model the unoccluded image and the completed depth map as two distinct Markov random fields, respectively, and obtain their maximum a-posteriori estimates using loopy belief propagation.
A Synaesthetic Approach for Image Slideshow Generation

In this paper, we present a novel automatic image slideshow system that explores a new medium between images and music. It can be regarded as a new image selection and slideshow composition criterion. Based on the idea of ``hearing colors, seeing sounds" from the art of music visualization, equal importance is assigned to image features and audio properties for better synchronization. We minimize the aesthetic energy distance between visual and audio features. Given a set of images, a subset is selected by correlating image features with the input audio properties. The selected images are then synchronized with the music subclips by their audio-visual distance. The inductive image displaying approach has been introduced for common displaying devices.
Precise pitch profile feature extraction from musical audio for key detection

The majority of pieces of music, including classical and popular music,are composed using music scales, such as keys. The key or the scale information of a piece provides important clues on its high level musical content, like harmonic and melodic context. Automatic key detection from music data can be useful for music classification, retrieval or further content analysis. Many researchers have addressed key finding from symbolically encoded music(MIDI); however, works for key detection in musical audio is still limited. Techniques for key detection from musical audio mainly consist of two steps:pitch extraction and key detection. The pitch feature typically characterizes the weights of presence of particular pitch classes in the music audio. In the existing approaches to pitch extraction, little consideration has been taken on pitch mistuning and interference of noisy percussion sounds in the audio signals, which inevitably affects the accuracy of key detection. In this paper, we present a novel technique of precise pitch profile feature extraction, which deals with pitch mistuning and noisy percussive sounds. The extracted pitch profile feature can characterize the pitch content in the signal more accurately than the previous techniques, thus lead to a higher key detection accuracy. Experiments based on classical and popular music data were conducted. The results showed that the proposed method has higher key detection accuracy than previous methods, especially for popular music with a lot of noisy drum sounds.
Experiential Sampling based Foreground/Background Segmentation for Video Surveillance

Segmentation of foreground and background has been an important research problem arising out of many applications including video surveillance. A method commonly used for segmentation is "background subtraction" or thresholding the difference between the estimated background image and current image. Adaptive Gaussian mixture based background modelling has been proposed by many researchers for increasing the robustness against environmental changes. However, all these methods, being computationally intensive, need to be optimized for efficient and real-time performance especially at a higher image resolution. In this paper, we propose an improved foreground/background segmentation method which uses experiential sampling technique to restrict the computational efforts in the region of interest. We exploit the fact that the region of interest in general is present only in a small part of the image, therefore, the attention should only be focused in those regions. The proposed method shows a significant gain in processing speed at the expense of minor loss in accuracy. We provide experimental results and detailed analysis to show the utility of our method
Adversary Aware Surveillance Systems

We consider surveillance problems to be a set of system-adversary interaction problems in which an adversary can be modeled as a rational (selfish) agent trying to maximize his utility. We feel that appropriate adversary modeling can provide deep insights into the system performance and also clues for optimizing the system's performance against the adversary. Further, we propose that system designers should exploit the fact that they can impose certain restrictions on the intruders and the way they interact with the system. The system designers can analyze the scenario to determine conditions under which system outperforms the adversaries, and then suitably reengineer the environment under a "scenario engineering" approach to help the system outperform the adversary. We study the proposed enhancements using a game theoretic framework and present results of their adaptation to two significantly different surveillance scenarios. While the precise enforcements for the studied zero-sum ATM lobby monitoring scenario and the nonzero-sum traffic monitoring scenario were different, they lead to some useful generic guidelines for surveillance system designers.
Multimedia Surveillance and Monitoring

In spite of the development of various media sensors, multimedia (and computer vision) researchers have mostly adopted a video-centric approach to solve the automated surveillance and monitoring related problems. We look at the monitoring/surveillance problem from an information-centric perspective and advocate the use of diverse sources of information which enables the use of multiple correlated media. We advocate a design methodology for building systems which can explicitly take performance into account. We then propose that the surveillance problem can be better posed as an "information-search" problem in which the user can query for the information of his/her interest. We will present a framework for multimedia monitoring that uses a domain-data transformation model based approach to map domain-events to their equivalent data-events. We will motivate the new approach, present the architecture and highlight the information assimilation aspects. We will also present some open problems and issues arising from the novel way of looking at monitoring.
Privacy Preserving Multiparty Multilevel DRM Architecture

Traditional digital rights management (DRM) systems are only two party systems, involving the owner and consumers. However, for scalability of business it is often necessary to involve additional levels of distributors and sub-distributors, who can promote and distribute the content in regions unknown to the owner. Thus, we propose an architecture for multiparty multilevel DRM system. The term 'multiparty' refers to involvement of many parties such as the owner, distributors, sub-distributors and consumers and the term 'multilevel' refers to multiple levels of distributors/sub-distributors. The architecture also supports the log files based violation detection, in case of violation of DRM system by any party. However, violation detection imposes a problem of preserving privacy of consumers. So, in the architecture, a provision is made to preserve their privacy.
A robust music retrieval method for query-by-humming

The increasing availability of digital music has created a need for effective music retrieval methods. We present a novel content-based music retrieval method that is robust against pitch errors and tempo variations in the queries, which is especially advantageous for query-by-humming. The melody of a music item and that of a hummed query are represented by point sequences, and the skeletons of the melody are used to do melody searching and alignment. A novel point skipping dynamic programming algorithm is proposed for robust and efficient melody skeleton matching. The melody similarity measure is then computed based on the alignment of the point sequences. Our experiment, including a comparison to our previous methods, has demonstrated the performance of the method.
Key-based melody segmentation for popular songs

In music theory, a key specifies the tonal structure of a music piece. Popular songs are mostly composed using particular music keys. It is common that the key changes at some point of the piece for an emotional uplifting effect. The change is usually in pitch of the tonic rather than in key style. Automatically segmenting melody based on the key changes can facilitate content-based music retrieval. Little work has been done for this problem. In this paper, we present a novel approach for detecting multiple keys and locating the key boundaries in the melody of popular songs in MIDI format. A tonality analysis of the melody using diatonic scale model first extracts overlapping segments of the melody that each conforms to the tonal structure of a single key. A modality (key style) analysis then determines the center mode of the melody based on the modes of the extracted melody segments. The segments of unrelated modes are then eliminated, and finally the keys and the key boundaries are determined by grouping remaining segments according to the pitch of the tonic. The proposed method is not confined to the two key styles, major and minor, as the previous techniques. Thus, this method is effective for detecting key changes and insensitive to the presence of accidental notes. Experimental results on 50 randomly selected MIDI songs have demonstrated the performance of the proposed method.
Harmonicity and dynamics-based features for audio

Features are very important for audio processing. Tasks like speech recognition and instrument identification are based on features. Most low-level features currently used are based on LPC and cepstral analysis. We propose a class of features based on dynamics and harmonicity. In particular, we define the notion of harmonic derivative. The efficacy of the features is demonstrated for music genre classification and instrument family classification. In particular, the features are shown to be cepstrum-equivalent.
A Spring-Electric Graph Model for Socialized Group Photography

Visual balance is considered as one of the important factors in defining the aesthetic quality of visual arts. In this paper, we propose a novel method to obtain visual balance in a layout with dynamic visual elements. We use the idea of a spring-electric graph model and augment it with the concept of color energy from the literature of visual arts. We also present an interesting application of the proposed model in photography assistance. We focus on group photography and utilize social media images along with the proposed spring-electric model for providing a recommendation to the user. The proposed method can provide real-time feedback to the user regarding the arrangement of people, their position, and relative size on the image frame. We conducted qualitative experiments along with user studies to evaluate the proposed method. Experimental results and user studies show the effectiveness of the proposed model in obtaining visual balance and group photography recommendation.
A Hierarchical Approach for Music Chord Modeling Based on the Analysis of Tonal Characteristics

This paper first discusses how the signal segmentation and tonal characteristics of music notes effect in music chord detection. Two approaches, pitch class profile approach and psycho-acoustical approach, which differently represent these tonal characteristics, are examined for chord detection. The analysis of the tonal characteristics reveals that not only the fundamental frequency of music note but also its harmonics and sub-harmonies in different octaves contribute for detecting related music chord. A hierarchical approach, which transforms the music chord tonal characteristics in each octave onto probabilistic space, is then proposed for modeling the music chord. Our experimental results show that detection of chord type, major, minor, diminish, and augmented, and individual chords, 12 chords per chord type, are improved with the proposed hierarchical chord modeling approach. Experimental results also reveal that the tempo proportional signal segmentation is more effective extracting tonal characteristics than using fixed length segmentation
Tracking by cluster analysis of feature points using a mixture particle filter

A moving target produces a coherent cluster of feature points in the image plane. This motivates our novel method of tracking multiple targets via feature points. First, the Harris corner detector and the Lucas-Kanade tracker are applied in each frame to detect feature points and their associated velocities. Points that are both spatially co-located and exhibit similar motion are grouped into clusters. Due to the non-Gaussian distribution of the points in a cluster and the multi-modality resulting from multiple targets, a special particle filter, the mixture particle filter, is adopted to model the mixture point distribution over time. Each cluster is treated as a mixture component and is modeled by an individual particle filter. The filters in the mixture are instantiated and initialized by applying the EM algorithm, are reclustered by merging overlapping clusters and splitting spatially disjoint clusters, and are terminated when their component weights drop below a threshold. The advantage of using mixture particle filtering is that it is capable of tracking multiple targets simultaneously and also of handling appearing and disappearing targets. We demonstrate the effectiveness of our method on different PETS datasets.
A Simple Ontology of Manipulation Actions Based on Hand-Object Relations

Humans can perform a multitude of different actions with their hands (manipulations). In spite of this, so far there have been only a few attempts to represent manipulation types trying to understand the underlying principles. Here we first discuss how manipulation actions are structured in space and time. For this we use as temporal anchor points those moments where two objects (or hand and object) touch or un-touch each other during a manipulation. We show that by this one can define a relatively small tree-like manipulation ontology. We find less than 30 fundamental manipulations. The temporal anchors also provide us with information about when to pay attention to additional important information, for example when to consider trajectory shapes and relative poses between objects. As a consequence a highly condensed representation emerges by which different manipulations can be recognized and encoded. Examples of manipulations recognition and execution by a robot based on this representation are given at the end of this study.
Refining discovered symbols with multi-step interaction experience

In our previous work, we showed how symbolic planning operators can be formed in the continuous perceptual space of a manipulator robot that explored the world with its single-step actions. In this paper, we extend our previous framework by enabling the robot to progressively update the previously learned concepts and rules in order to better deal with novel situations that appear during multi-step action executions. Our proposed system can infer categories of the novel objects based on previously learned rules, and form new object categories for these novel objects if their interaction characteristics and appearance do not match with the existing categories. Our system further learns probabilistic rules that predict the action effects and the next object states. There rules are automatically encoded in Planning Domain and Definition Language (PDDL), enabling use of powerful symbolic AI planners. Using this framework, our manipulator robot updated its reasoning skills from multi-step stack action executions. After learning, the robot was able to build stable towers in real world, exhibiting some interesting reasoning capabilities such as stacking larger objects before smaller ones, and predicting that cups remain insertable even with other objects inside.
Active learning using mean shift optimization for robot grasping

When children learn to grasp a new object, they often know several possible grasping points from observing a parent's demonstration and subsequently learn better grasps by trial and error. From a machine learning point of view, this process is an active learning approach. In this paper, we present a new robot learning framework for reproducing this ability in robot grasping. For doing so, we chose a straightforward approach: first, the robot observes a few good grasps by demonstration and learns a value function for these grasps using Gaussian process regression. Subsequently, it chooses grasps which are optimal with respect to this value function using a mean-shift optimization approach, and tries them out on the real system. Upon every completed trial, the value function is updated, and in the following trials it is more likely to choose even better grasping points. This method exhibits fast learning due to the data-efficiency of the Gaussian process regression framework and the fact that the mean-shift method provides maxima of this cost function. Experiments were repeatedly carried out successfully on a real robot system. After less than sixty trials, our system has adapted its grasping policy to consistently exhibit successful grasps.
Using structural bootstrapping for object substitution in robotic executions of human-like manipulation tasks

In this work we address the problem of finding replacements of missing objects that are needed for the execution of human-like manipulation tasks. This is a usual problem that is easily solved by humans provided their natural knowledge to find object substitutions: using a knife as a screwdriver or a book as a cutting board. On the other hand, in robotic applications, objects required in the task should be included in advance in the problem definition. If any of these objects is missing from the scenario, the conventional approach is to manually redefine the problem according to the available objects in the scene. In this work we propose an automatic way of finding object substitutions for the execution of manipulation tasks. The approach uses a logic-based planner to generate a plan from a prototypical problem definition and searches for replacements in the scene when some of the objects involved in the plan are missing. This is done by means of a repository of objects and attributes with roles, which is used to identify the affordances of the unknown objects in the scene. Planning actions are grounded using a novel approach that encodes the semantic structure of manipulation actions. The system was evaluated in a KUKA arm platform for the task of preparing a salad with successful results.
Affordances in Psychology, Neuroscience, and Robotics: A Survey

The concept of affordances appeared in psychology during the late 60s as an alternative perspective on the visual perception of the environment. It was revolutionary in the intuition that the way living beings perceive the world is deeply influenced by the actions they are able to perform. Then, across the last 40 years, it has influenced many applied fields, e.g., design, human-computer interaction, computer vision, and robotics. In this paper, we offer a multidisciplinary perspective on the notion of affordances. We first discuss the main definitions and formalizations of the affordance theory, then we report the most significant evidence in psychology and neuroscience that support it, and finally we review the most relevant applications of this concept in robotics.
Bottom-up learning of object categories, action effects and logical rules: From continuous manipulative exploration to symbolic planning

This work aims for bottom-up and autonomous development of symbolic planning operators from continuous interaction experience of a manipulator robot that explores the environment using its action repertoire. Development of the symbolic knowledge is achieved in two stages. In the first stage, the robot explores the environment by executing actions on single objects, forms effect and object categories, and gains the ability to predict the object/effect categories from the visual properties of the objects by learning the nonlinear and complex relations among them. In the next stage, with further interactions that involve stacking actions on pairs of objects, the system learns logical high-level rules that return a stacking-effect category given the categories of the involved objects and the discrete relations between them. Finally, these categories and rules are encoded in Planning Domain Definition Language (PDDL), enabling symbolic planning. We realized our method by learning the categories and rules in a physics-based simulator. The learned symbols and operators are verified by generating and executing non-trivial symbolic plans on the real robot in a tower building task.
Can Affordances Guide Object Decomposition into Semantically Meaningful Parts?

Most objects are designed for certain functionalities. For example, a knife is designed for cutting, and a hammer for pounding. Indeed, functionalities are not related to the objects themselves but to certain object parts, e.g., the blade of a knife affords cutting. A part can have different shapes and can exist in different objects such as a scraper or a peeler, but it carries the same functional meaning. There is a strong correlation between object parts and affordances. In this paper, we exploit this correlation to decompose objects into semantically meaningful parts. The semantics are limited here to object affordances. We evaluate our method on a part decomposition task, and obtained 77% weighted overlap with ground-truth object parts.