nils-herrmann/clean_scirepeval huggingface dataset:
An MPEG-7 Extension for Describing Visual Impairments

Analysing the condition of audiovisual essence is an important step in audiovisual production and preservation. Standardised impairment description of audiovisual media is a pre-requisite for system interoperability between content digitisation, documentation, management, restoration, production and delivery systems.We analyse existing capabilities for describing impairment in audiovisual metadata standards. Because of its unique detailed spatiotemporally structured description capabilities we have selected MPEG-7 as the basis for visual impairment description. Following the approach for audio quality description, we define a general description scheme for visual impairments, which allows representing defect events and statistical quality measures. For certain defects, more specialised descriptors are proposed. In addition, we define a comprehensive classification scheme for visual impairments.
Multi-cue learning and visualization of unusual events

Unusual event detection, i.e., identifying unspecified rare/critical events, has become one of the major challenges in visual surveillance. The main solution for this problem is to describe local or global normalness and to report events that do not fit to the estimated models. The majority of existing approaches, however, is limited to a single description (e.g., either appearance or motion) and/or builds on inflexible (unsupervised) learning techniques, both clearly degrading the practical applicability. To overcome these limitations, we demonstrate a system that is capable of extracting and modeling several representations in parallel, while in addition allows for user interaction within a continuous learning setup. Novel yet intuitive concepts of result visualization and user interaction will be presented that allow for exploiting the underlying data.
Multimodal highway monitoring for robust incident detection

We present detection and tracking methods for highway monitoring based on video and audio sensors, and the combination of these two modalities. We evaluate the performance of the different systems on realistic data sets that have been recorded on Austrian highways. It is shown that we can achieve a very good performance for video-based incident detection of wrong-way drivers, still standing vehicles, and traffic jams. Algorithms for simultaneous vehicle and driving direction detection using microphone arrays were evaluated and also showed a good performance on these tasks. Robust tracking in case of difficult weather conditions is achieved through multimodal sensor fusion of video and audio sensors.
A Perceptual Image Sharpness Metric Based on Local Edge Gradient Analysis

In this letter, a no-reference perceptual sharpness metric based on a statistical analysis of local edge gradients is presented. The method takes properties of the human visual system into account. Based on perceptual properties, a relationship between the extracted statistical features and the metric score is established to form a Perceptual Sharpness Index (PSI). A comparison with state-of-the-art metrics shows that the proposed method correlates highly with human perception and exhibits low computational complexity. In contrast to existing metrics, the PSI performs well for a wide range of blurriness and shows a high degree of invariance for different image contents.
Creating Rich Metadata in the TV Broadcast Archives Environment: The PrestoSpace Project

This paper describes the part of the European PrestoSpace project dedicated to the study and development of a metadata access and delivery (MAD) system for television broadcast archives. The mission of the MAD system, inside the wider perspective of the PrestoSpace factory, is to generate, validate and deliver to the archive users metadata created through the employment of both automatic and manual information extraction tools. Automatic tools include audiovisual content analysis and semantic analysis of text extracted by automatic speech recognition (ASR). The MAD publication platform provides access and search facilities to the imported and newly produced metadata in a synergic and easy-to-use interface
Semantically Guided Visual Question Answering

We present a novel approach to enhance the challenging task of Visual Question Answering (VQA) by incorporating and enriching semantic knowledge in a VQA model. We first apply Multiple Instance Learning (MIL) to extract a richer visual representation addressing concepts beyond objects such as actions and colors. Motivated by the observation that semantically related answers often appear together in prediction, we further develop a new semantically-guided loss function for model learning which has the potential to drive weakly-scored but correct answers to the top while suppressing wrong answers. We show that these two ideas contribute to performance improvement in a complementary way. We demonstrate competitive results comparable to the state of the art on two VQA benchmark datasets.
Robust Foreground and Abandonment Analysis for Large-Scale Abandoned Object Detection in Complex Surveillance Videos

We present a robust system for large-scale abandoned object detection (AOD) with low false positive rates and good detection accuracy under complex realistic scenarios. The robustness of our system is largely attributed to an approach we develop for foreground analysis, which can effectively differentiate foreground objects from background under challenging conditions such as lighting changes, low textureness and low contrast as well as cluttered background. This significantly eliminates false positives caused by lighting changes while retaining true drops better. We further perform abandonment analysis to reduce more false positives including those related to people, at a small cost of accuracy (≤ 2%). We demonstrate the effectiveness of our approach on two large data sets collected in various challenging scenes, providing detailed analysis of experiments.
Graph based event detection from realistic videos using weak feature correspondence

We study the problem of event detection from realistic videos with repetitive sequential human activities. Despite the large body of work on event detection and recognition, very few have addressed low-quality videos captured from realistic environments. Our framework is based on solving the shortest path on a temporal-event graph constructed from the video content. Graph vertices correspond to detected event primitives, and edge weights are set according to generic knowledge of the event patterns and the discrepancy between event primitives based on a greedy matching of their visual features. Experimental results on videos collected from a retail environment validate the usefulness of the proposed approach.
Robust Spatiotemporal Matching of Electronic Slides to Presentation Videos

We describe a robust and efficient method for automatically matching and time-aligning electronic slides to videos of corresponding presentations. Matching electronic slides to videos provides new methods for indexing, searching, and browsing videos in distance-learning applications. However, robust automatic matching is challenging due to varied frame composition, slide distortion, camera movement, low-quality video capture, and arbitrary slides sequence. Our fully automatic approach combines image-based matching of slide to video frames with a temporal model for slide changes and camera events. To address these challenges, we begin by extracting scale-invariant feature-transformation (SIFT) keypoints from both slides and video frames, and matching them subject to a consistent projective transformation (homography) by using random sample consensus (RANSAC). We use the initial set of matches to construct a background model and a binary classifier for separating video frames showing slides from those without. We then introduce a new matching scheme for exploiting less distinctive SIFT keypoints that enables us to tackle more difficult images. Finally, we improve upon the matching based on visual information by using estimated matching probabilities as part of a hidden Markov model (HMM) that integrates temporal information and detected camera operations. Detailed quantitative experiments characterize each part of our approach and demonstrate an average accuracy of over 95% in 13 presentation videos.
Recognition of repetitive sequential human activity

We present a novel framework for recognizing repetitive sequential events performed by human actors with strong temporal dependencies and potential parallel overlap. Our solution incorporates sub-event (or primitive) detectors and a spatiotemporal model for sequential event changes. We develop an effective and efficient method to integrate primitives into a set of sequential events where strong temporal constraints are imposed on the ordering of the primitives. In particular, the combination process is approached as an optimization problem. A specialized Viterbi algorithm is designed to learn and infer the target sequential events and handle the event overlap simultaneously. To demonstrate the effectiveness of the proposed framework, we report detailed quantitative analysis on a large set of cashier checkout activities in a retail store.
Accurate alignment of presentation slides with educational video

Spatio-temporal alignment of electronic slides with corresponding presentation video opens up a number of possibilities for making the instructional content more accessible and understandable, such as video quality improvement, better content analysis and novel compression approaches for low bandwidth access. However, these applications need finding accurate transformations between slides and video frames, which is quite challenging in capture settings using pan-tiltzoom (PTZ) cameras. In this paper we present a nonlinear optimization approach for accurate registration of slide images to video frames. Instead of estimating the projective transformation (i.e., homography) between a single pair of slide and frame images, we solve a set of homographies jointly in a frame sequence that is associated with a given slide. Quantitative evaluation confirms that this substantively improves alignment accuracy.
Reducing correspondence ambiguity in loosely labeled training data

We develop an approach to reduce correspondence ambiguity in training data where data items are associated with sets of plausible labels. Our domain is images annotated with keywords where it is not known which part of the image a keyword refers to. In contrast to earlier approaches that build predictive models or classifiers despite the ambiguity, we argue that that it is better to first address the correspondence ambiguity, and then build more complex models from the improved training data. This addresses difficulties of fitting complex models in the face of ambiguity while exploiting all the constraints available from the training data. We contribute a simple and flexible formulation of the problem, and show results validated by a recently developed comprehensive evaluation data set and corresponding evaluation methodology.
Enhanced face detection using body part detections for wearable cameras

With the recent broad acceptance of body worn cameras (BWC) for police departments, there is an increased need to perform video analytics for this domain. However, body worn cameras pose several challenges including severe motion blur, barrel camera distortion from wide angle lenses, close proximity and odd viewing angles, and poor lighting conditions. In this paper, we evaluate the performance of several state of the art face detection approaches including Aggregate Channel Features  and Faster R-CNN [2] and show their limitations in this domain. We then describe how face detection can be improved for BWC by corroborating information from body parts detection. We design a system using 0–1 linear integer programming to optimize the matching of body parts detections for each person in the scene and to maximize the hit rate of faces with supporting evidence. By leveraging information from body parts detection, we are able to improve the average precision by nearly two percent.
Long-term object tracking for parked vehicle detection

We develop a robust approach to detect parked vehicles in real time. Our approach particularly focuses on tracking vehicles in long term under challenging conditions such as lighting changes and occlusions. Vehicle tracking is performed by template matching based on fast-computed corner points. The template model is made self-adaptive over time to accommodate lighting changes. We also present an effective way to manage and track multiple vehicles when they are parked close together and occlude one another. We demonstrate the effectiveness of our approach on the challenging i-LIDs data set and another large one collected from real-world scenarios.
People detection in crowded scenes by context-driven label propagation

Exploiting contextual cues has been a key idea to improve people detection in crowded scenes. Along this line we present a novel context-driven approach to detect people in crowded scenes. Based on a context graph that incorporates both geometric and social contextual patterns in crowds, we apply label propagation to discover weak detections contextually compatible with true detections while suppressing irrelevant false alarms. Compared to previous approaches for context modeling limited to only pairwise spatial interactions between local object neighbors, our approach provides a more effective way to model people interactions in a global context. Our approach achieves performance comparable to state of the art on two challenging datasets for people and pedestrian detection.
Studying on the Move - Enriched Presentation Video for Mobile Devices

The rapid adoption of distance learning means that significant lecture video content is available on-line. However, access from mobile devices is hampered by low bandwidth and small screen size. In this paper, we address these issues by manipulating two key elements of lecture videodisplayed slides and laser pointer gestures. Displayed slides need to be very crisp compared to background content. Fortunately, the needed data is available from the presentation slides, and we describe a method for splicing them into the video on the client side, increasing fidelity, and reducing bandwidth needs. This operation removes laser pointer gestures, which are often lost due to compression, and are hard to see on the small screens of mobile regardless. But these gestures are part of what makes watching lecture video different than simply looking at the slides. Hence we interpret the laser pointer gestures as we analyze the videos, creating representations that can be transmitted at a low cost. These representations can then be iconified on the client side and displayed clearly.
Robust abandoned object detection using region-level analysis

We propose a robust abandoned object detection algorithm for real-time video surveillance. Different from conventional approaches that mostly rely on pixel-level processing, we perform region-level analysis in both background maintenance and static foreground object detection. In background maintenance, region-level information is fed back to adaptively control the learning rate. In static foreground object detection, region-level analysis double-checks the validity of candidate abandoned blobs. Attributed to such analysis, our algorithm is robust against illumination change, “ghosts” left by removed objects, distractions from partially static objects, and occlusions. Experiments on nearly 130,000 frames of i-LIDS dataset show the superior performance of our approach.
Multi-media compliance: A practical paradigm for managing business integrity

In virtually every business context there is a need to establish some form of monitoring system to ensure that employees comply with business processes and policies. Compliance failures range from organized theft to gaps in procedure that can be easily remedied through retraining. It is clearly important for businesses to capture and record these deviations to minimize loss prevention and maximize workplace safety and efficiency. In this workshop, we discuss the growing problem of compliance failure and how our system addresses this problem in a retail context to detect checkout-related fraud through the integration of visual and non-visual data.
Soft margin keyframe comparison: Enhancing precision of fraud detection in retail surveillance

We propose a novel approach for enhancing precision in a leading video analytics system that detects cashier fraud in grocery stores for loss prevention. While intelligent video analytics has recently become a promising means of loss prevention for retailers, most of the real-world systems suffer from a large number of false alarms, resulting in a significant waste of human labor during manual verification. Our proposed approach starts with the candidate fraudulent events detected by a state-of-the-art system. Such fraudulent events are a set of visually recognized checkout-related activities of the cashier without barcode associations. Instead of conducting costly video analysis, we extract a few keyframes to represent the essence of each candidate fraudulent event, and compare those keyframes to identify whether or not the event is a valid check-out process that involves consistent appearance changes on the lead-in belt, the scan area and the take-away belt. Our approach also performs a margin-based soft classification so that the user could trade off between saving human labor and preserving high recall. Experiments on days of surveillance videos collected from real grocery stores show that our algorithm can save about 50% of human labor while preserving over 90% of true alarms with small computational overhead.
Relative Attributes for Large-Scale Abandoned Object Detection

Effective reduction of false alarms in large-scale video surveillance is rather challenging, especially for applications where abnormal events of interest rarely occur, such as abandoned object detection. We develop an approach to prioritize alerts by ranking them, and demonstrate its great effectiveness in reducing false positives while keeping good detection accuracy. Our approach benefits from a novel representation of abandoned object alerts by relative attributes, namely static ness, foreground ness and abandonment. The relative strengths of these attributes are quantified using a ranking function[19] learnt on suitably designed low-level spatial and temporal features. These attributes of varying strengths are not only powerful in distinguishing abandoned objects from false alarms such as people and light artifacts, but also computationally efficient for large-scale deployment. With these features, we apply a linear ranking algorithm to sort alerts according to their relevance to the end-user. We test the effectiveness of our approach on both public data sets and large ones collected from the real world.
Hand tracking by binary quadratic programming and its application to retail activity recognition

Substantial ambiguities arise in hand tracking due to issues such as small hand size, deformable hand shapes and similar hand appearances. These issues have greatly limited the capability of current multi-target tracking techniques in hand tracking. As an example, state-of-the-art approaches for people tracking handle indentity switching by exploiting the appearance cues using advanced object detectors. For hand tracking, such approaches will fail due to similar, or even identical hand appearances. The main contribution of our work is a global optimization framework based on binary quadratic programming (BQP) that seamlessly integrates appearance, motion and complex interactions between hands. Our approach effectively handles key challenges such as occlusion, detection failure, identity switching, and robustly tracks both hands in two challenging real-life scenarios: retail surveillance and sign languages. In addition, we demonstrate that an automatic method based on hand trajectory analysis outperforms state-of-the-art on checkout-related activity recognition in grocery stores.
Detecting human activities in retail surveillance using hierarchical finite state machine

Cashiers in retail stores usually exhibit certain repetitive and periodic activities when processing items. Detecting such activities plays a key role in most retail fraud detection systems. In this paper, we propose a highly efficient, effective and robust vision technique to detect checkout-related primitive activities, based on a hierarchical finite state machine (FSM). Our deterministic approach uses visual features and prior spatial constraints on the hand motion to capture particular motion patterns performed in primitive activities. We also apply our approach to the problem of retail fraud detection. Experimental results on a large set of video data captured from retail stores show that our approach, while much simpler and faster, achieves significantly better results than state-of-the-art machine learning-based techniques both in detecting checkout-related activities and in detecting checkout-related fraudulent incidents.
Detecting sweethearting in retail surveillance videos

A significant portion of retail shrink is attributed to employees and occurs around the point of sale (POS). In this paper, we target a major type of retail fraud in surveillance videos, known as sweethearting (or fake scan), where a cashier intentionally fails to enter one or more items into the transaction in an attempt to get free merchandise for the customer. We first develop a motion-based algorithm to identify video segments as candidates for primitive events at the POS. We then apply spatio-temporal features to recognize true primitive events from the candidates and prune those falsely alarmed. In particular, we learn location-aware event models by Multiple-Instance Learning to address the location-sensitive issues that appear in our problem. Finally, we validate the entire transaction by combining primitive events according to temporal ordering constraints. We demonstrate the effectiveness of our approach on data captured from a real grocery store.
A closer look at Faster R-CNN for vehicle detection

Faster R-CNN achieves state-of-the-art performance on generic object detection. However, a simple application of this method to a large vehicle dataset performs unimpressively. In this paper, we take a closer look at this approach as it applies to vehicle detection. We conduct a wide range of experiments and provide a comprehensive analysis of the underlying structure of this model. We show that through suitable parameter tuning and algorithmic modification, we can significantly improve the performance of Faster R-CNN on vehicle detection and achieve competitive results on the KITTI vehicle dataset. We believe our studies are instructive for other researchers investigating the application of Faster R-CNN to their problems and datasets.
Fast detection of retail fraud using polar touch buttons

Video analytics have recently emerged as a promising technique of retail fraud detection for loss prevention. Efficient video analytic algorithms are highly desired for a practical fraud detection system. In this paper, we present a real-time algorithm for recognizing a cashier's actions at the Point of Sale (POS), which can be further used to analyze cashier behaviors for identifying fraudulent incidents. The algorithm uses a set of simple but effective features derived from a global representation of motion energy called Polar Motion Map (PMM). These features capture the motion patterns exhibited in a cashier's actions as a focused beam of motion energy, characterizing the actions as the extension and retraction movement of the cashier's arm with respect to a prespecified region. Our algorithm demonstrates comparable accuracy against one of the state-of-the-art event recognition techniques  while running significantly faster.