nils-herrmann/clean_scirepeval huggingface dataset:
On the biased estimation of nonlocal means filter

The recent nonlocal means filter (NLM) exhibits exciting performance for image or video denoising. However, the estimation of NLM is biased and tends to keep the value of noise. In the process of NLM, the target pixel (the pixel being processed) is involved in patch distance measuring, which, in turn, controls the restoration of the target pixel. Such design makes the estimation of the target pixel to be biased to the noise. The biased estimation may degrade the reconstruction especially when the noise energy is large. In this paper, the cause and influence of such biased estimation phenomenon is analyzed in detail. We will show that the biased estimation can be extremely harmful or helpful, depending on the degree of noise. Taking advantage of this, a new method is proposed to adaptively control the contribution of target pixel in patch distance measuring. This method achieves much better denoising performance compared with NLM.
EFRI-REM at CCNY: Research experience and mentoring for underrepresented groups in cross-disciplinary research on assistive technology

We have successfully run the NSF EFRI-REM program at The City College of The City University of New York (CUNY) for four years (2012 to 2015) with over 50 research participants (RPs), most of whom are from underrepresented groups, particularly individuals who are disabled, women, veterans and minorities. Each year we started with an 8 to 10-week summer research program and then mentored the participants' education and career paths in the following academic year. We have not only provided exciting cross-disciplinary, student initiated research opportunities in human and machine vision research and applications in assisting visually impaired people, but also shaped an innovative model to support career development of high school students, community and senior college students, graduate students (mentors), high school teachers, as well as community college professors. The EFRI-REM program provides a novel and effective platform to allow more underrepresented students in the greater NYC area to participate in our multidisciplinary research.
Multi-modality American Sign Language recognition

American Sign Language (ASL) is a visual gestural language which is used by many people who are deaf or hard-of-hearing. In this paper, we design a visual recognition system based on action recognition techniques to recognize individual ASL signs. Specifically, we focus on recognition of words in videos of continuous ASL signing. The proposed framework combines multiple signal modalities because ASL includes gestures of both hands, body movements, and facial expressions. We have collected a corpus of RBG + depth videos of multi-sentence ASL performances, from both fluent signers and ASL students; this corpus has served as a source for training and testing sets for multiple evaluation experiments reported in this paper. Experimental results demonstrate that the proposed framework can automatically recognize ASL.
Assistive Clothing Pattern Recognition for Visually Impaired People

Choosing clothes with complex patterns and colors is a challenging task for visually impaired people. Automatic clothing pattern recognition is also a challenging research problem due to rotation, scaling, illumination, and especially large intraclass pattern variations. We have developed a camera-based prototype system that recognizes clothing patterns in four categories (plaid, striped, patternless, and irregular) and identifies 11 clothing colors. The system integrates a camera, a microphone, a computer, and a Bluetooth earpiece for audio description of clothing patterns and colors. A camera mounted upon a pair of sunglasses is used to capture clothing images. The clothing patterns and colors are described to blind users verbally. This system can be controlled by speech input through microphone. To recognize clothing patterns, we propose a novel Radon Signature descriptor and a schema to extract statistical properties from wavelet subbands to capture global features of clothing patterns. They are combined with local features to recognize complex clothing patterns. To evaluate the effectiveness of the proposed approach, we used the CCNY Clothing Pattern dataset. Our approach achieves 92.55% recognition accuracy which significantly outperforms the state-of-the-art texture analysis methods on clothing pattern recognition. The prototype was also used by ten visually impaired participants. Most thought such a system would support more independence in their daily life but they also made suggestions for improvements.
Visual semantic parameterization - To enhance blind user perception for indoor navigation

This paper presents a novel approach of utilizing the floor plan maps posted on the buildings to infer a semantic plan that aids in the navigation of a visually impaired person. The extracted landmarks such as room numbers, doors, etc act as a parameter to infer the way points to each room. This provides a mental mapping of the environment to design a navigation framework for future use. A human motion model is used to predict a path based on how real humans ambulate towards a goal by avoiding obstacles. Travel route is presented in terms of blind understandable units, which is achieved by accurate estimation of the user's location and confirmed by extracting the landmarks posted on the doors. The results show that it is feasible to make a blind user to travel independently by providing the constraints required for safe navigation.
Vision-Based Mobile Indoor Assistive Navigation Aid for Blind People

This paper presents a new holistic vision-based mobile assistive navigation system to help blind and visually impaired people with indoor independent travel. The system detects dynamic obstacles and adjusts path planning in real-time to improve navigation safety. First, we develop an indoor map editor to parse geometric information from architectural models and generate a semantic map consisting of a global 2D traversable grid map layer and context-aware layers. By leveraging the visual positioning service (VPS) within the Google Tango device, we design a map alignment algorithm to bridge the visual area description file (ADF) and semantic map to achieve semantic localization. Using the on-board RGB-D camera, we develop an efficient obstacle detection and avoidance approach based on a time-stamped map Kalman filter (TSM-KF) algorithm. A multi-modal human-machine interface (HMI) is designed with speech-audio interaction and robust haptic interaction through an electronic SmartCane. Finally, field experiments by blindfolded and blind subjects demonstrate that the proposed system provides an effective tool to help blind individuals with indoor navigation and wayfinding.
Robust and efficient foreground analysis for real-time video surveillance

We present a new method to robustly and efficiently analyze foreground when we detect background for a fixed camera view by using mixture of Gaussians models and multiple cues. The background is modeled by three Gaussian mixtures as in the work of Stauffer and Grimson (1999). Then the intensity and texture information are integrated to remove shadows and to enable the algorithm working for quick lighting changes. For foreground analysis, the same Gaussian mixture model is employed to detect the static foreground regions without using any tracking or motion information. Then the whole static regions are pushed back to the background model to avoid a common problem in background subtraction /spl times/ fragmentation (one object becomes multiple parts). The method was tested on our real time video surveillance system. It is robust and run about 130 fpsfor color images and 150 fps for grayscale images at size 160/spl times/120 on a 2GB Pentium IV machine with MMX optimization.
Adaptive shrinkage cascades for blind image deconvolution

Recently emerged discriminative non-blind deconvolution methods achieve excellent performance with only a fraction of computation cost w.r.t. generative competitors, but their extension to blind deconvolution field was seldom addressed in a practical manner, albeit equally vital in image restoration area. We propose a novel framework for effective blind image deblurring by patch-wise prior based adaptive shrinkage cascades, which introduces the powerful internal patch-based image statistics to the non-blind shrinkage field formulations. The rich expressiveness of internal patch prior brings instance-specific adaptivity to alternating kernel refinement between neighboring shrinkage cascades, while shrinkage model trained from varieties of natural image collections benefits internal patch-wise prior inference with external information and superior efficiency.
Recognizing Action Units for Facial Expression Analysis

Most automatic expression analysis systems attempt to recognize a small set of prototypic expressions, such as happiness, anger, surprise, and fear. Such prototypic expressions, however, occur rather infrequently. Human emotions and intentions are more often communicated by changes in one or a few discrete facial features. In this paper, we develop an Automatic Face Analysis (AFA) system to analyze facial expressions based on both permanent facial features (brows, eyes, mouth) and transient facial features (deepening of facial furrows) in a nearly frontal-view face image sequence. The AFA system recognizes fine-grained changes in facial expression into action units (AUs) of the Facial Action Coding System (FACS), instead of a few prototypic expressions. Multistate face and facial component models are proposed for tracking and modeling the various facial features, including lips, eyes, brows, cheeks, and furrows. During tracking, detailed parametric descriptions of the facial features are extracted. With these parameters as the inputs, a group of action units (neutral expression, six upper face AUs and 10 lower face AUs) are recognized whether they occur alone or in combinations. The system has achieved average recognition rates of 96.4 percent (95.4 percent if neutral expressions are excluded) for upper face AUs and 96.7 percent (95.6 percent with neutral expressions excluded) for lower face AUs. The generalizability of the system has been tested by using independent image databases collected and FACS-coded for ground-truth by different research teams.
Detection and tracking in the IBM PeopleVision system

The detection and tracking of people lie at the heart of many current and near-future applications of computer vision. We describe a background subtraction system designed to detect moving objects in a wide variety of conditions, and a second system to detect objects moving in front of moving backgrounds. Detected foreground regions are tracked with a tracking system which can initiate real-time alarms and generate a smart surveillance index which can be searched to find interesting events in stored video
Detecting stairs and pedestrian crosswalks for the blind by RGBD camera

A computer vision-based wayfinding and navigation aid can improve the mobility of blind and visually impaired people to travel independently. In this paper, we develop a new framework to detect and recognize stairs and pedestrian crosswalks using a RGBD camera. Since both stairs and pedestrian crosswalks are featured by a group of parallel lines, we first apply Hough transform to extract the concurrent parallel lines based on the RGB channels. Then, the Depth channel is employed to further recognize pedestrian crosswalks, upstairs, and downstairs using support vector machine (SVM) classifiers. Furthermore, we estimate the distance between the camera and stairs for the blind users. The detection and recognition results on our collected dataset demonstrate that the effectiveness and efficiency of our proposed framework.
Margin-constrained multiple kernel learning based multi-modal fusion for affect recognition

Recent advances in multiple-kernel learning (MKL) show the effectiveness to fuse multiple base features in object detection and recognition. However, MKL tends to select only the most discriminative base features but ignore other less discriminative base features which may provide complementary information. Moreover, MKL usually employ Gaussian RBF kernels to transform each base feature to its high dimensional space. Generally, base features from different modalities require different kernel parameters for obtaining the optimal performance. Therefore, MKL may fail to utilize the maximum discriminative power of all base features from multiple modalities at the same time. In order to address these issues, we propose a margin-constrained multiple-kernel learning (MCMKL) method by extending MKL with margin constraints and applying dimensionally normalized RBF (DNRBF) kernels for application of multi-modal feature fusion. The proposed MCMKL method learns weights of different base features according to their discriminative power. Unlike the conventional MKL, MCMKL incorporates less discriminative base features by assigning smaller weights when constructing the optimal combined kernel, so that we can fully take the advantages of the complementary features from different modalities. We validate the proposed MCMKL method for affect recognition from face and body gesture modalities on the FABO dataset. Our extensive experiments demonstrate favorable results as compared to the existing work, and MKL-based approach.
Super Normal Vector for Human Activity Recognition with Depth Cameras

The advent of cost-effectiveness and easy-operation depth cameras has facilitated a variety of visual recognition tasks including human activity recognition. This paper presents a novel framework for recognizing human activities from video sequences captured by depth cameras. We extend the surface normal to polynormal by assembling local neighboring hypersurface normals from a depth sequence to jointly characterize local motion and shape information. We then propose a general scheme of super normal vector (SNV) to aggregate the low-level polynormals into a discriminative representation, which can be viewed as a simplified version of the Fisher kernel representation. In order to globally capture the spatial layout and temporal order, an adaptive spatio-temporal pyramid is introduced to subdivide a depth video into a set of space-time cells. In the extensive experiments, the proposed approach achieves superior performance to the state-of-the-art methods on the four public benchmark datasets, i.e., MSRAction3D, MSRDailyActivity3D, MSRGesture3D, and
Detecting good quality frames in videos captured by a wearable camera for blind navigation

Recent technology developments in computer vision, digital cameras, and portable computers make it possible to assist blind individuals by developing camera-based object recognition products. However, motion blur caused by a moving camera limits the real-world application of wayfinding for blind users. In this paper, we propose a new method to detect good quality frames from videos captured by cameras, which are taken by blind users. In our proposed method, both gradient and intensity statistics are extracted from video frames. Then a support vector machine (SVM) based classifier is applied to identify the frames with good quality (Unblurred) from those blurred frames. The Unblurred frames will be further processed to extract essential information for blind wayfinding and navigation such as signage recognition and text extraction. Experimental results demonstrate that our proposed method is able to robustly handle video motions in both indoor and outdoor environments.
Improving Performance via Post Track Analysis

In this paper, we improve the effective performance of a surveillance system via post track analysis. Our system performs object detection via background subtraction followed by appearance based tracking. The primary outputs of the system however, are customized alarms which depend on the user's domain and needs. The ultimate performance therefore depends most critically on the Receiver Operating Characteristic curve of these alarms. We show that by strategically designing post tracking and alarm conditions, the effective performance of the system can be improved dramatically. This addresses the most significant error sources, namely, errors due to shadows, ghosting, temporally or spatially missing fragments and many of the false positives due to extreme lighting variations, specular reflections or irrelevant motion.
Prediction of sea ice motion with recurrent neural networks

Prediction of sea ice motion is important for ocean-atmosphere interaction modeling and safe naval operations in polar regions. In this study, we investigate the potential of Recurrent Neural Networks (RNNs) in predictions of motion for several days in the future based only on previously observed satellite image data. We collect a large dataset of daily Advanced Microwave Scanning Radiometer — Earth Observing System (AMSR-E) images that cover the entire Arctic. Optical flow is employed to calculate dense sea ice motion between images of each consecutive-day pair. The optical flow images are then used to train an encoder-decoder Long Short-Term Memory (LSTM) RNN and estimate motion for several days in the future. Experiments demonstrate that the proposed method is successful in predicting short-term sea ice motion with accuracy close to motion calculated from the original images and buoys, and proves promising for further applications and research.
Segment and recognize expression phase by fusion of motion area and neutral divergence features

An expression can be approximated by a sequence of temporal segments called neutral, onset, offset and apex. However, it is not easy to accurately detect such temporal segments only based on facial features. Some researchers try to temporally segment expression phases with the help of body gesture analysis. The problem of this approach is that the expression temporal phases from face and gesture channels are not synchronized. Additionally, most previous work adopted facial key points tracking or body tracking to extract motion information, which is unreliable in practice due to illumination variations and occlusions. In this paper, we present a novel algorithm to overcome the above issues, in which two simple and robust features are designed to describe face and gesture information, i.e., motion area and neutral divergence features. Both features do not depend on motion tracking, and they can be easily calculated too. Moreover, it is different from previous work in that we integrate face and body gesture together in modeling the temporal dynamics through a single channel of sensorial source, so it avoids the unsynchronized issue between face and gesture channels. Extensive experimental results demonstrate the effectiveness of the proposed algorithm.
Assisting blind people to avoid obstacles: An wearable obstacle stereo feedback system based on 3D detection

A wearable Obstacle Stereo Feedback (OSF) System for the Blind people based on 3D space obstacle detection is presented to assist the navigation. The OSF system embedded with a depth sensor to perceive the in-front 3D spatial information in the form of point clouds. We implemented the downsampling Random Sample Consensus (RANSAC) algorithm to process the perceived point cloud, and detect the obstacles in front of the user. Finally, Head-Related Transfer Functions (HRTF) are applied to create the virtual stereo sound which represents the obstacles according to its coordinate in the 3D space. The experiment shows that OSF system can detect the obstacle in the indoor environment effectively and provides a feasible auditory perception to indicate the in-front safety zone for the blind user.
Feature Representations for Scene Text Character Recognition: A Comparative Study

Recognizing text character from natural scene images is a challenging problem due to background interferences and multiple character patterns. Scene Text Character (STC) recognition, which generally includes feature representation to model character structure and multi-class classification to predict label and score of character class, mostly plays a significant role in word-level text recognition. The contribution of this paper is a complete performance evaluation of image-based STC recognition, by comparing different sampling methods, feature descriptors, dictionary sizes, coding and pooling schemes, and SVM kernels. We systematically analyze the impact of each option in the feature representation and classification. The evaluation results on two datasets CHARS74K and ICDAR2003 demonstrate that Histogram of Oriented Gradient (HOG) descriptor, soft-assignment coding, max pooling, and Chi-Square Support Vector Machines (SVM) obtain the best performance among local sampling based feature representations. To improve STC recognition, we apply global sampling feature representation. We generate Global HOG (GHOG) by computing HOG descriptor from global sampling. GHOG enables better character structure modeling and obtains better performance than local sampling based feature representations. The GHOG also outperforms existing methods in the two benchmark datasets.
Robust Salient Motion Detection with Complex Background for Real-Time Video Surveillance

Moving object detection is very important for video surveillance. In many environments, motion maybe either interesting (salient) motion (e.g., a person) or uninteresting motion (e.g., swaying branches.) In this paper, we propose a new real-time algorithm to detect salient motion in complex environments by combining temporal difference imaging and a temporal filtered motion field. We assume that the object with salient motion moves in a consistent direction for a period of time. No prior knowledge about object size and shape is necessary. Compared to background subtraction methods, our method does NOT need to learn the background model from hundreds of images and can handle quick image variations; e.g., a light being turned on or off. The average speed of our method is about 50fps on images at size 160x120 in 1GB Pentium III machines. The effectiveness of the proposed algorithm to robust detect salient motion is demonstrated for a variety of real environments with distracting motions such as lighting changes, swaying branches, rippling water, waterfall, and fountains.
Histogram of 3D Facets: A characteristic descriptor for hand gesture recognition

The availability of 3D sensors has recently made it possible to capture depth maps in real time, which facilitates a variety of visual recognition tasks including hand gesture recognition. However, most existing methods simply treat depth information as intensities of gray images and ignore the strong 3D shape information. In this paper, we propose a novel characteristic descriptor, i.e., Histogram of 3D Facets (H3DF), to explicitly encode the 3D shape information from depth maps. We define a 3D facet as a 3D local support surface associated with each 3D cloud point. By robust coding and pooling 3D facets from a depth map, the proposed H3DF descriptor can effectively represent the 3D shapes and structures of various hand gestures. We evaluate the proposed descriptor on two challenging 3D datasets of hand gesture recognition. The recognition results in the context of both decimal digits and letters in American Sign Language (ASL) demonstrate that our approach significantly outperforms the state-of-the-art methods.