nils-herrmann/clean_scirepeval huggingface dataset:
Depth-aware indoor staircase detection and recognition for the visually impaired

A mobile vision-based navigation aid is capable of assisting the visually impaired to travel independently, especially in unfamiliar environments. Despite many effective navigation algorithms having been developed in recent decades, accurate, efficient, and reliable staircase detection in indoor navigation still remains to be a challenging problem. In this paper, we propose an effective indoor staircase detection algorithm based on an RGB-D camera. The candidates of staircases are first detected from RGB frames by extracting a set of concurrent parallel lines based on Hough transform. The complement depth frames are further employed to recognize the staircase candidates as upstairs, downstairs, and negatives (i.e., corridors). A support vector machine (SVM) based multi-classifier is trained and tested for the staircase recognition with our newly collected staircase dataset. The detection and recognition results demonstrate the effectiveness and efficiency of the proposed algorithm.
Super-Resolved Fine-Scale Sea Ice Motion Tracking

Monitoring sea ice activities is particularly critical to safe naval operations in the Arctic Ocean. Accurately tracking sea ice motions is essential to validate or even improve sea ice models for ice hazard forecasts at a fine scale. Fine-scale motions can be tracked from high-resolution radar or optical satellite imagery but with limited coverage. Daily motions over the entire Arctic are retrievable from passive microwave data, but at a much lower spatial resolution. Thus, providing motions at the passive microwave spatial and temporal coverage, but at an enhanced spatial resolution, will be a significant benefit. To break the resolution limitation and to boost tracking accuracy, a sequential super-resolved fine-scale sea ice motion tracking framework is proposed in which a hybrid example-based single image super-resolution algorithm is employed before the tracking procedure. Experiments demonstrate that the proposed framework significantly improves the tracking performance in both accuracy and robustness for a benchmark algorithm and a recently proposed state-of-the-art tracking algorithm.
S3: The IBM Smart Surveillance System: From Transactional Systems to Observational Systems

Pervasive sensor based systems are transforming Information Technology systems from being transactional in nature to being observational in nature. Observational systems are inherently distributed and capture information at a much finer grain of space and time. Enabling and building such systems also poses many technology challenges, extracting information from sensor signals, indexing and searching sensor meta-data, data mining and scalability. In this paper we use S3: the IBM smart surveillance system as an example of an observational system to explore several of these issues through real world deployment examples.
Robust Detection of Abandoned and Removed Objects in Complex Surveillance Videos

Tracking-based approaches for abandoned object detection often become unreliable in complex surveillance videos due to occlusions, lighting changes, and other factors. We present a new framework to robustly and efficiently detect abandoned and removed objects based on background subtraction (BGS) and foreground analysis with complement of tracking to reduce false positives. In our system, the background is modeled by three Gaussian mixtures. In order to handle complex situations, several improvements are implemented for shadow removal, quick-lighting change adaptation, fragment reduction, and keeping a stable update rate for video streams with different frame rates. Then, the same Gaussian mixture models used for BGS are employed to detect static foreground regions without extra computation cost. Furthermore, the types of the static regions (abandoned or removed) are determined by using a method that exploits context information about the foreground masks, which significantly outperforms previous edge-based techniques. Based on the type of the static regions and user-defined parameters (e.g., object size and abandoned time), a matching method is proposed to detect abandoned and removed objects. A person-detection process is also integrated to distinguish static objects from stationary people. The robustness and efficiency of the proposed method is tested on IBM Smart Surveillance Solutions for public safety applications in big cities and evaluated by several public databases, such as The Image library for intelligent detection systems (i-LIDS) and IEEE Performance Evaluation of Tracking and Surveillance Workshop (PETS) 2006 datasets. The test and evaluation demonstrate our method is efficient to run in real-time, while being robust to quick-lighting changes and occlusions in complex environments.
Recognizing lower face action units for facial expression analysis

Most automatic expression analysis systems attempt to recognize a small set of prototypic expressions (e.g., happiness and anger). Such prototypic expressions, however, occur infrequently. Human emotions and intentions are communicated more often by changes in one or two discrete facial features. We develop an automatic system to analyze subtle changes in facial expressions based on both permanent (e.g., mouth, eye, and brow) and transient (e.g., furrows and wrinkles) facial features in a nearly frontal image sequence. Multi-state facial component models are proposed for tracking and modeling different facial features. Based on these multi-state models, and without artificial enhancement, we detect and track the facial features, including mouth, eyes, brow, cheeks, and their related wrinkles and facial furrows. Moreover we recover detailed parametric descriptions of the facial features. With these features as the inputs, 11 individual action units or action unit combinations are recognized by a neural network algorithm. A recognition rate of 96.7% is obtained. The recognition results indicate that our system can identify action units regardless of whether they occur singly or in combinations.
Robust internal exemplar-based image enhancement

Image enhancement aims to modify images to achieve a better perception for human visual system or a more suitable representation for further analysis. Based on different attributes of given input images, tasks vary, e.g., noise removal, deblur-ring, resolution enhancement, prediction of missing pixels, etc. The latter two are usually referred to as image super-resolution and image inpainting. There exist complicated circumstances where low-quality input images suffer from insufficient resolution with missing regions. In this paper, we propose a novel uniform framework to accomplish both image super-resolution and inpainting simultaneously. The proposed approach adopts internal exemplar similarities in image level and gradient level where later enhancement results from both levels are fed into a pre-defined cost function to restore the final output. Experimental results demonstrate that our method is capable of generating visually plausible, natural-looking results with clear edges and realistic textures.
Semantic Indoor Navigation with a Blind-User Oriented Augmented Reality

The aim of this paper is to design an inexpensive conceivable wearable navigation system that can aid in the navigation of a visually impaired user. A novel approach of utilizing the floor plan map posted on the buildings is used to acquire a semantic plan. The extracted landmarks such as room numbers, doors, etc act as a parameter to infer the way points to each room. This provides a mental mapping of the environment to design a navigation framework for future use. A human motion model is used to predict a path based on how real humans ambulate towards a goal by avoiding obstacles. We demonstrate the possibilities of augmented reality (AR) as a blind user interface to perceive the physical constraints of the real world using haptic and voice augmentation. The haptic belt vibrates to direct the user towards the travel destination based on the metric localization at each step. Moreover, travel route is presented using voice guidance, which is achieved by accurate estimation of the user's location and confirmed by extracting the landmarks, based on landmark localization. The results show that it is feasible to assist a blind user to travel independently by providing the constraints required for safe navigation with user oriented augmented reality.
High-Resolution Sea Ice Motion Estimation With Optical Flow Using Satellite Spectroradiometer Data

The purpose of this paper is twofold: 1) to propose an approach based on optical flow for the estimation of sea ice motion as an accurate, dense, and computationally efficient alternative to state-of-the-art pattern matching approaches and 2) to investigate the potential of moderate resolution imaging spectroradiometer (MODIS) optical satellite data for combined daily and high-resolution motion estimation. A series of MODIS image pairs for a selected region in Arctic Ocean is employed, and sparse pairwise correspondences between nonrigid patches are calculated. An edge-preserving sparse-to-dense interpolation is applied followed by variational energy minimization to compute the final optical flow. A state-of-the-art multiresolution pattern matching method based on phase correlation and normalized cross correlation is also implemented and evaluated for comparison. Thorough experimentation with different settings of input data preprocessing as well as varying scales of motion is performed. The derived motion vectors are compared with coarser resolution operational sea ice motion vector products from combined buoy and microwave satellite data. The proposed optical flow approach clearly outperforms the pattern matching method in most cases, both in terms of accuracy and motion vector consistency. The estimated motion vectors from the MODIS images highly correlate with the operational vector products. In addition, MODIS provides a vector field with spatial resolution two orders of magnitude higher than the operational products that are able to detect significantly smaller sea ice drifts.
S3-R1: the IBM smart surveillance system release 1

One of the key components of tele-presence systems is automatic awareness of the remote environment. This very same capability of automatic situation awareness is currently being developed and deployed in the context of the next generation smart surveillance systems. Smart surveillance systems use a number of automatic video analysis techniques like object detection, tracking and classification in conjunction with database and Web application servers to provide users with the capability of distributed smart surveillance. The IBM smart surveillance system is one of the few advanced surveillance systems which provides not only the capability to automatically monitor a scene but also the capability to manage the surveillance data, perform event based retrieval, receive real time event alerts thru standard web infrastructure and extract long term statistical patterns of activity.
Capturing People in Surveillance Video

This paper presents reliable techniques for detecting, tracking, and storing keyframes of people in surveillance video. The first component of our system is a novel face detector algorithm, which is based on first learning local adaptive features for each training image, and then using Adaboost learning to select the most general features for detection. This method provides a powerful mechanism for combining multiple features, allowing faster training time and better detection rates. The second component is a face tracking algorithm that interleaves multiple view-based classifiers along the temporal domain in a video sequence. This interleaving technique, combined with a correlation-based tracker, enables fast and robust face tracking over time. Finally, the third component of our system is a keyframe selection method that combines a person classifier with a face classifier. The basic idea is to generate a person keyframe in case the face is not visible, in order to reduce the number of false negatives. We performed quantitatively evaluation of our techniques on standard datasets and on surveillance videos captured by a camera over several days.
Identifying medicine bottles by incorporating RFID and video analysis

In this paper, we present a new framework of identifying medicine bottles using a combination of a video camera and Radio Frequency Identification (RFID) sensors for applications of monitoring the elderly's activities of daily living (ADLs) at home. RFID tags are attached to medicine bottles and first detected by RFID readers from the antenna. However, the RFID detection can only detect RFID tags within a certain range of the antenna. Once a medicine bottle is moved out of the range of the RFID antenna, a camera will be activated to continue detecting and tracking the medicine bottle for further action analysis based on moving object detection and color model of the medicine bottle. The experimental results demonstrate 100% detection accuracy for identifying medicine bottles.
Discriminative Hierarchical K-Means Tree for Large-Scale Image Classification

A key challenge in large-scale image classification is how to achieve efficiency in terms of both computation and memory without compromising classification accuracy. The learning-based classifiers achieve the state-of-the-art accuracies, but have been criticized for the computational complexity that grows linearly with the number of classes. The nonparametric nearest neighbor (NN)-based classifiers naturally handle large numbers of categories, but incur prohibitively expensive computation and memory costs. In this brief, we present a novel classification scheme, i.e., discriminative hierarchical K-means tree (D-HKTree), which combines the advantages of both learning-based and NN-based classifiers. The complexity of the D-HKTree only grows sublinearly with the number of categories, which is much better than the recent hierarchical support vector machines-based methods. The memory requirement is the order of magnitude less than the recent Naïve Bayesian NN-based approaches. The proposed D-HKTree classification scheme is evaluated on several challenging benchmark databases and achieves the state-of-the-art accuracies, while with significantly lower computation cost and memory requirement.
Dual-state parametric eye tracking

Most eye trackers work well for open eyes. However blinking is a physiological necessity for humans. More over, for applications such as facial expression analysis and driver awareness systems, we need to do more than tracking of the locations of the person's eyes but obtain their detailed description. We need to recover the state of the eyes (i.e., whether they are open or closed), and the parameters of an eye model (e.g., the location and radius of the iris, and the corners and height of the eye opening). We develop a dual-state model-based system for tracking eye features that uses convergent tracking techniques and show how it can be used to detect whether the eyes are open or closed, and to recover the parameters of the eye model. Processing speed on a Pentium II 400 MHz PC is approximately 3 frames/second. In experimental tests on 500 image sequences from child and adult subjects with varying colors of skin and eye, accurate tracking results are obtained in 98% of image sequences.
Door detection via signage context-based Hierarchical Compositional Model

Door detection by using wearable cameras helps people with severe vision impairment to independently access unknown environments. The goal of this paper is to robustly detect different doors and classify them as office doors, elevators, exits, etc. These tasks are challenging due to the factors: 1) small inter-class variations of different objects such as office doors and elevators, 2) only part of an object is captured due to occlusions or continuous camera moving of a mobile system. To overcome the above challenges, we propose a Hierarchical Compositional Model (HCM) approach which incorporates context information into the model decomposition process of a part-based HCM to handle partially captured objects as well as large intra-class variations in different environments. Our preliminary experimental results demonstrate promising performance on doors detection over a wide range of scales, view points, and occlusions.
Resolution enhancement in single depth map and aligned image

Depth resolution enhancement aims to recover a high quality depth map from one or multiple low-resolution depth input(s) with missing pixels. While a registered high-resolution intensity image is often utilized to assist, little attention has been paid to the circumstances when there is only one pair of low-resolution depth map and aligned intensity image available. In this paper, we propose a novel resolution enhancement approach that targets at improving the quality of both the input depth map and the low-resolution RGB image. By exploiting the statistical dependency between the input pairs, a label matrix is generated utilizing the support vector machine classifier. Guided by the constructed label matrix and the aligned intensity image, the missing values in the depth map are well predicted in a manner consistent with the embedded structure. After that, the completed depth map and the intensity image are super-resolved through a set of regression models trained via external exemplars. Extensive experiments demonstrate that our framework is effective with satisfying performance.
Evaluation of Low-Level Features for Real-World Surveillance Event Detection

Event detection targets at recognizing and localizing specified spatio-temporal patterns in videos. Most research of human activity recognition in the past decades experimented on relatively clean scenes with limited actors performing explicit actions. Recently, more efforts have been paid to the real-world surveillance videos in which the human activity recognition is more challenging due to large variations caused by factors, such as scaling, resolution, viewpoint, cluttered background, and crowdedness. In this paper, we systematically evaluate seven different types of low-level spatio-temporal features in the context of surveillance event detection (SED) using a uniform experimental setup. Fisher vector is employed to aggregate low-level features as the representation of each video clip. A set of random forests is then learned as the classification models. To bridge the research efforts and real-world applications, we utilize the NIST TRECVID SED as our testbed in which seven events are predefined involving different levels of human activity analysis. Strengths and limitations for each low-level feature type are analyzed and discussed.
A SLAM Based Semantic Indoor Navigation System for Visually Impaired Users

This paper proposes a novel assistive navigation system based on simultaneous localization and mapping (SLAM) and semantic path planning to help visually impaired users navigate in indoor environments. The system integrates multiple wearable sensors and feedback devices including a RGB-D sensor and an inertial measurement unit (IMU) on the waist, a head mounted camera, a microphone and an earplug/speaker. We develop a visual odometry algorithm based on RGB-D data to estimate the user's position and orientation, and refine the orientation error using the IMU. We employ the head mounted camera to recognize the door numbers and the RGB-D sensor to detect major landmarks such as corridor corners. By matching the detected landmarks against the corresponding features on the digitalized floor map, the system localizes the user, and provides verbal instruction to guide the user to the desired destination. The software modules of our system are implemented in Robotics Operating System (ROS). The prototype of the proposed assistive navigation system is evaluated by blindfolded sight persons. The field tests confirm the feasibility of the proposed algorithms and the system prototype.
Unambiguous Text Localization and Retrieval for Cluttered Scenes

Text instance as one category of self-described objects provides valuable information for understanding and describing cluttered scenes. In this paper, we explore the task of unambiguous text localization and retrieval, to accurately localize a specific targeted text instance in a cluttered image given a natural language description that refers to it. To address this issue, first a novel recurrent Dense Text Localization Network (DTLN) is proposed to sequentially decode the intermediate convolutional representations of a cluttered scene image into a set of distinct text instance detections. Our approach avoids repeated detections at multiple scales of the same text instance by recurrently memorizing previous detections, and effectively tackles crowded text instances in close proximity. Second, we propose a Context Reasoning Text Retrieval (CRTR) model, which jointly encodes text instances and their context information through a recurrent network, and ranks localized text bounding boxes by a scoring function of context compatibility. Quantitative evaluations on standard scene text localization benchmarks and a newly collected scene text retrieval dataset demonstrate the effectiveness and advantages of our models for both scene text localization and retrieval.
Portable Camera-Based Assistive Text and Product Label Reading From Hand-Held Objects for Blind Persons

We propose a camera-based assistive text reading framework to help blind persons read text labels and product packaging from hand-held objects in their daily lives. To isolate the object from cluttered backgrounds or other surrounding objects in the camera view, we first propose an efficient and effective motion-based method to define a region of interest (ROI) in the video by asking the user to shake the object. This method extracts moving object region by a mixture-of-Gaussians-based background subtraction method. In the extracted ROI, text localization and recognition are conducted to acquire text information. To automatically localize the text regions from the object ROI, we propose a novel text localization algorithm by learning gradient features of stroke orientations and distributions of edge pixels in an Adaboost model. Text characters in the localized text regions are then binarized and recognized by off-the-shelf optical character recognition software. The recognized text codes are output to blind users in speech. Performance of the proposed text localization algorithm is quantitatively evaluated on ICDAR-2003 and ICDAR-2011 Robust Reading Datasets. Experimental results demonstrate that our algorithm achieves the state of the arts. The proof-of-concept prototype is also evaluated on a dataset collected using ten blind persons to evaluate the effectiveness of the system's hardware. We explore user interface issues and assess robustness of the algorithm in extracting and reading text from different objects with complex backgrounds.