nils-herrmann/clean_scirepeval huggingface dataset:
Benchmarking a Multimodal and Multiview and Interactive Dataset for Human Action Recognition

Human action recognition is an active research area in both computer vision and machine learning communities. In the past decades, the machine learning problem has evolved from conventional single-view learning problem, to cross-view learning, cross-domain learning and multitask learning, where a large number of algorithms have been proposed in the literature. Despite having large number of action recognition datasets, most of them are designed for a subset of the four learning problems, where the comparisons between algorithms can further limited by variances within datasets, experimental configurations, and other factors. To the best of our knowledge, there exists no dataset that allows concurrent analysis on the four learning problems. In this paper, we introduce a novel multimodal and multiview and interactive (<inline-formula> <tex-math notation="LaTeX">$\text{M}^{{2}}\text{I}$ </tex-math></inline-formula>) dataset, which is designed for the evaluation of human action recognition methods under all four scenarios. This dataset consists of 1760 action samples from 22 action categories, including nine person-person interactive actions and 13 person-object interactive actions. We systematically benchmark state-of-the-art approaches on <inline-formula> <tex-math notation="LaTeX">$\text{M}^{{2}}\text{I}$ </tex-math></inline-formula> dataset on all four learning problems. Overall, we evaluated 13 approaches with nine popular feature and descriptor combinations. Our comprehensive analysis demonstrates that <inline-formula> <tex-math notation="LaTeX">$\text{M}^{{2}}\text{I}$ </tex-math></inline-formula> dataset is challenging due to significant intraclass and view variations, and multiple similar action categories, as well as provides solid foundation for the evaluation of existing state-of-the-art algorithms.
Demo Paper: PreSense — An Assistive Presentation Self-Quantification System

This technical demo presents PreSense, an AssistivePresentation Self-Quantification System. Oral presentation istraditionally evaluated by a human instructor, which is cost-ineffective and time-consuming. PreSense allows individuals toself-evaluate their presentation skills. The system is designedto receive multimodal inputs from multiple sources, includingwebcam, Kinect sensor and Google Glass. The multimodal datais processed by a deep assessment framework, which outputsthe evaluation results based on a carefully designed assessmentrubric. In order to systematically visualize the individual as-sessment results, we develop an interactive graphical interfaceand demonstrate its efficacy using presentation data from theNUS Multi-Sensor Presentation (NUSMAP) dataset.
Compressed domain object tracking for automatic indexing of objects in MPEG home video

Object tracking is of utmost importance for automatic indexing of video content. This work presents an object tracker that operates directly on MPEG compressed data. Motion vectors and discrete cosine transform (DCT) coefficients directly available from the compressed video stream are exploited for the purpose of tracking. Tracking proceeds in two steps: motion vector based tracking in P and B frames within the groups of pictures (GOPs), and object identification in I frames. Colour, which is one of the strongest cues for tracking is used for the identification step. Such a system offers speed, simplicity and robustness against occlusion and camera motion, with good intra-shot tracking for shots in excess of 500 frames, as shown in the experimental results.
Multi-stream Deep Learning Framework for Automated Presentation Assessment

Presentation is one of the most effective methods to disseminate information. Traditional methods to evaluate the quality of a presentation generally involves a human instructor, which is infeasible in many scenarios. Recent studies have focused on the automated assessment of presentations. A variety of systems have been developed that focus on analyzing various aspects of presentations. However, those systems are mainly limited by their performance, as they mostly adopt hand-crafted features and ad-hoc algorithms. In this work, we propose a multi-stream deep learning framework customized for presentation assessment. The framework uses Bidirectional Long Short-Term Memory with attention mechanism for temporal modeling, and fuses information from multiple modalities for the final decision. We also design a novel assessment rubric based on input from a domain expert. Experimental results on the NUS Multi-Sensor Presentation (NUSMAP) dataset show that the proposed framework is computationally efficient and achieves significant improvement in classification accuracy.
Towards Adversary Aware Surveillance Systems

We consider surveillance problems to be a set of system-adversary interaction problems in which an adversary can be modeled as a rational (selfish) agent trying to maximize his utility. We feel that appropriate adversary modeling can provide deep insights into the system performance and also clues for optimizing the system's performance against the adversary. Further, we propose that system designers should exploit the fact that they can impose certain restrictions on the intruders and the way they interact with the system. The system designers can find the assumptions under which the surveillance system shall out-perform the intruder and then enforce those assumptions over the system-intruder interaction as part of a 'scenario engineering' approach. We study both these aspects using a game theoretic framework and undertake practical experiments to verify the proposed enhancements.
Experiential Sampling in Multimedia Systems

Multimedia systems must deal with multiple data streams. Each data stream usually contains significant volume of redundant noisy data. In many real-time applications, it is essential to focus the computing resources on a relevant subset of data streams at any given time instant and use it to build the model of the environment. We formulate this problem as an experiential sampling problem and propose an approach to utilize computing resources efficiently on the most informative subset of data streams. First, in this paper, we focus on theoretical background and develop a theoretical framework for a single data stream. We generalize the notion of static visual attention in a dynamical systems setting and propose a dynamical attention-orientated analysis method. This is achieved by a sampling representation that utilizes the current context and past experience for attention evolution. Hence, the multimedia analysis task at hand can select its data of interest while immediately discarding the irrelevant data to achieve efficiency and adaptability
Motion trajectory based video authentication

With the relentless digitization of commercial media, it is increasingly susceptible to the violation of its intellectual property rights. We address the issue of authenticating digital video in this paper by presenting a novel technique based on motion trajectory and cryptographic secret sharing. Given a video, we segment it into shots and select the key frames in a shot based on the motion trajectory. We then consider these key frames as shares of a secret-sharing scheme for color images. We then reconstruct the secret corresponding to these shares. This secret frame is deemed as the secret key of this shot. Utilizing the secret keys of all the shots, we hierarchically construct the master key for the entire video by recursively employing the same method of color image sharing. Any modifications in a shot or in the important content of a shot will be reflected as changes in the computed master key. Thus, any video can be authenticated by comparing its computed master key with the original master key. The advantages of the proposed technique are that we take the correlation among the video frames into account, all authentication data is encapsulated within the master key, and this master key authentication information need not be embedded into the video. Thus the video quality is not affected by the auxiliary authentication information.
Quality-aware GSM speech watermarking

Use of watermarking techniques to provide authentication and tamper proofing of speech in mobile environment is becoming important. However, the current efforts do not allow for user-specifiable quality for the watermarked speech. This paper proposes a watermarking algorithm that allows user-customizable quality for watermarked GSM (Global System for Mobile) speech. Sensitivity (in terms of quality degradation) of each GSM coefficient bits against bit watermark embedding was investigated first, which is then used to select the coefficient bits in a secure manner for watermarking. The proposed algorithm's execution time requirement was studied to draw conclusions on the real-time usability of the algorithm. The embedding capacity and the quality awareness of the algorithm were also investigated.
Automatic music summarization in compressed domain

A novel compressed domain automatic music summarization approach is presented in this paper. The proposed method works directly in the compressed domain. Only the encoded subband samples are extracted and processed for characterizing music content and discovering the music structure. The experimental results and the evaluation by a subjective study have shown that the summarization based on MPEG-1 Layer 3 (MP3) music is comparable to the summarization based on uncompressed PCM music samples.
Automatically summarize musical audio using adaptive clustering

Automatic music summarization is very useful for music indexing, content-based music retrieval and on-line music distribution, but it is a challenge to extract automatically the most common and salient themes from unstructured raw music data. We propose an effective approach to summarize music content automatically. First, a number of features are extracted to characterize the music content. Based on the extracted features, an adaptive clustering algorithm is then applied to structure the music content. Finally, the music summary is created in terms of the clustering results and domain-related music knowledge. A user study is conducted to evaluate the quality of summarization. The experiments on different genres of music illustrate the results of summarization are significant and effective to actual expectation.
Non Identical Duplicate Video Detection using SIFT Method

Non-Identical Duplicate video detection is a challenging 
research problem. Non-Identical Duplicate video are a pair 
of videos that are not exactly identical but are almost similar.In this paper, we evaluate two methods - Keyframe -based and Tomography-based methods to determine the Non-Identical Duplicate videos. These two methods make use of the existing scale based shift invariant (SIFT) method to find the match between the key frames in first method, and the cross-sections through the temporal axis of the videos in second method.We provide extensive experimental results and the analysis of accuracy and efficiency of the above two methods on a data set of Non- Identical Duplicate video-pair.
Melody curve processing for music retrieval

There have been several query-by-humming techniques developed for music retrieval. The techniques either are errorprone due to the inaccuracy of the hummed query or force the users to hum according to a metronome. This paper presents a new slope-based query-by-humming technique, in which the retrieval is robust to the inaccuracy in query and the use of metronome is eliminated. We use melody curve to represent the melodies of the original songs and the hummed query. And curve features: slope ranges and spans and note changes are extracted from the melody curves. Music retrieval is done by matching the curve features of query with those of the original musical songs. Results have shown the features and the algorithms are robust to humming inaccuracy.
Multimedia Fusion With Mean-Covariance Analysis

The number of multimedia applications has been increasing over the past two decades. Multimedia information fusion has therefore attracted significant attention with many techniques having been proposed. However, the uncertainty and correlation among different information sources have not been fully considered in the existing fusion methods. In general, the predictions of individual information source have uncertainty. Furthermore, many information sources in the multimedia systems are correlated with each other. In this paper, we propose a novel multimedia fusion method based on the portfolio theory. Portfolio theory is a widely used financial investment theory dealing with how to allocate funds across securities. The key idea is to maximize the performance of the allocated portfolio while minimize the risk in returns. We adapt this approach to multimedia fusion to derive optimal weights that can achieve good fusion results. The optimization is formulated as a quadratic programming problem. Experimental results with both simulation and real data confirm the theoretical insights and show promising results.
A hierarchical framework for face tracking using state vector fusion for compressed video

Faces usually are the most interesting objects in certain categories of video, like home videos and news clips. A novel sensor fusion based face tracking system is presented that tracks faces in compressed video, and aids automatic video indexing. Tracking is done by fusing the measurements from three independent sensors - motion and colour based trackers (Achanta, R. et al.,  2002) and a face detector (Wang, J. et al.,  2002) using a novel hierarchical framework based on Kalman filter state vector fusion. The tracking results show that the fused results are better than those of any individual sensors or their mean.
Pre-attentive discrimination of interestingness in images

Interestingness is an important aesthetic property, which literally means something that arouses curiosity and is a precursor to attention. Aesthetics is becoming more important as multimedia systems become more human and content centric as opposed to technology centric. In this paper, we use insights from cognitive science, neurophysiology of the early visual system and a mix of human experiments and computational modeling for the purpose of investigating interestingness. Categories in image interestingness and their computational realization are explored through a nontrivial dataset and a real-world problem.
Predominant Vocal Pitch Detection in Polyphonic Music

We present a novel method for predominant vocal pitch detection in two-channel polyphonic music. The proposed method contains two stages. In the first stage, we apply the frequency domain independent component Analysis (FD-ICA) for the two-channel polyphonic music to separate the vocal content from the background music. Considering the vocal singing voice and background music are two heterogeneous signals, we employ a statistical learning based method to solve the permutation inconsistency problem in FD-ICA. In the second stage, a noise insensitive vocal pitch detection method is proposed, which is robust to noise and errors introduced by the separation process in the first stage. The proposed method has been tested on the two-channel polyphonic music signals, and experimental results show promising performance
Adaptive Workload Equalization in Multi-Camera Surveillance Systems

Surveillance and monitoring systems generally employ a large number of cameras to capture people's activities in the environment. These activities are analyzed by hosts (human operators and/or computers) for threat detection. Threat detection is a target centric task in which the behavior of each target is analyzed separately, which requires a significant amount of human attention and is a computationally intensive task for automatic analysis. In order to meet the real-time requirements of surveillance, it is necessary to distribute the video processing load over multiple hosts. In general, cameras are statically assigned to the hosts; we show that this is not a desirable solution as the workload for a particular camera may vary over time depending on the number of targets in its view. In the future, this uneven distribution of workload will become more critical as the sensing infrastructures are being deployed on the cloud. In this paper, we model the camera workload as a function of the number of targets, and use that to dynamically assign video feeds to the hosts. Experimental results show that the proposed model successfully captures the variability of the workload, and that the dynamic workload assignment provides better results than a static assignment.
Dynamic watermarking of images

With the rapid growth of networked multimedia data systems, copyright protection of proprietary digitized media has gained importance. Inserting a robust and invisible signal that clearly identifies the owner or the recipient is beginning to emerge as the solution. Previous research in the field of watermarking has been successful in inserting a 'static' watermark, which endures ownership rights but is not as robust and tamper-proof as the 'dynamic' watermarks. This paper presents a novel invisible and robust watermarking technique that aims at dynamically inserting the watermark in digital images. Dynamic generation of watermark ensures that whenever the image is viewed, the watermark inserted in the image is different from the previous one providing more security against copyright attacks. This is accomplished by bundling the viewer and the image together, in which the viewer is responsible for embedding the new watermark using the spread spectrum watermarking algorithm every time. We have implemented the proposed scheme and present experimental results.
Audio Based Event Detection for Multimedia Surveillance

With the increasing use of audio sensors in surveillance and monitoring applications, event detection using audio streams has emerged as an important research problem. This paper presents a hierarchical approach for audio based event detection for surveillance. The proposed approach first classifies a given audio frame into vocal and nonvocal events, and then performs further classification into normal and excited events. We model the events using a Gaussian mixture model and optimize the parameters for four different audio features ZCR, LPC, LPCC and LFCC. Experiments have been performed to evaluate the effectiveness of the features for detecting various normal and the excited state human activities. The results show that the proposed top-down event detection approach works significantly better than the single level approach
An Authentication Mechanism Using Chinese Remainder Theorem for Efficient Surveillance Video Transmission

Now-a-days, surveillance cameras have been widely deployedin various security applications. In many surveillanceapplications, the background changes very slowly andthe foreground objects occupy only a relatively small portionof a video frame. In these type of applications, an efficientsolution for transmissions over bandwidth-limited networksis to send only the foreground objects for every framein real time while the background is sent occasionally. Atthe receiving end of the transmission, the objects and themost recent background can be fused together and the originalframe can be reconstructed. However, protecting theauthenticity of the video becomes more challenging in thiscase as a malicious entity can modify/replace/remove the individualforeground objects and background in the video. Inthis paper, we propose a Chinese remainder theorem basedwatermarking mechanism for protecting the authenticity ofvideos transmitted or stored as objects and background.Our mechanism ensures the authenticity between video objectsand their associated background.
A method for solmization of melody

This work presents a novel method for the automatic solmization of a melody, by which a melody (a sequence of MIDI notes) can be transcribed to sol-fa syllables (i.e., do, re, me, fa, sol, la, ti). Automatic solmization can assist in music skill training, music notation and content-based music retrieval. The proposed method is based on an approach for estimating the music scale of a melody. The key of the major scale ("do") is estimated using music scale models. Due to the diversity of melody types, models for both diatonic and pentatonic scales are employed to avoid the possible key ambiguity for folk songs. The decision of the key of a melody is based on the scale estimation results for aggregating music notes, so that the method can work for both short and long melodies. Experiments have shown that the technique can achieve 95% correct solmization of the melodies of pop songs.
Competence-Based Song Recommendation: Matching Songs to One’s Singing Skill

Singing is a popular social activity and a pleasant way of expressing one's feelings. One important reason for unsuccessful singing performance is because the singer fails to choose a suitable song. In this paper, we propose a novel competence-based song recommendation framework for the purpose of singing. It is distinguished from most existing music recommendation systems which rely on the computation of listeners' interests or similarity. We model a singer's vocal competence as a singer profile, which takes voice pitch, intensity, and quality into consideration. Then we propose techniques to acquire singer profiles. We also present a song profile model which is used to construct a human annotated song database. Then we propose a learning-to-rank scheme for recommending songs by a singer profile. Finally, we introduce a reduced singer profile which can greatly simplify the vocal competence modelling process. The experimental study on real singers demonstrates the effectiveness of our approach and its advantages over two baseline methods.
Compressed-encrypted domain JPEG2000 image watermarking

In digital rights management (DRM) systems, digital media is often distributed by multiple levels of distributors in a compressed and encrypted format. The distributors in the chain face the problem of embedding their watermark in compressed, encrypted domain for copyright violation detection purpose. In this paper, we propose a robust watermark embedding technique for JPEG2000 compressed and encrypted images. While the proposed technique embeds watermark in the compressed-encrypted domain, the extraction of watermark can be done either in decrypted domain or in encrypted domain.
Similarity matching of continuous melody contours for humming querying of melody databases

Music query-by-humming is a challenging problem since the humming query inevitably contains much variation and inaccuracy. In this paper, we present a novel melody similarity matching technique, which is based on continuous melody contour. We introduce a contour alignment technique, which addresses the robustness and efficiency issues. We also present a new melody similarity metric, which is performed directly on continuous melody contours of the query data. This approach cleanly separates the alignment and similarity measurement in the retrieval process. Our melody alignment method can reduce the matching candidate to 1.7% with 90% correct alignment rate. The overall retrieval system achieved 88% correct retrieval in the top 20 rank lists.