nils-herrmann/clean_scirepeval huggingface dataset:
Increasing spatial resolution of sea ice motion estimation

Estimation of sea ice motion at a fine scale is essential for climate modeling and naval operations in polar regions. This study proposes an approach for increasing the spatial resolution of the motion estimated from passive microwave satellite images. A hierarchical pattern matching approach, based on normalized cross-correlation and phase correlation, is applied to calculate sea ice drifts between Advanced Microwave Scanning Radiometer 2 (AMSR2) pairs of images captured at different time. Contrary to the widely used approach of embedding oversampling within the pattern matching framework that increases the resolution of the minimum detectable drift, in this study a nearest-neighbor interpolation upscaling is applied directly to the images. This additionally increases the density of the estimated motion vectors. Experiments demonstrate that the proposed method can lead to motion vector field with increased resolution by up to eight times, while outperforming the original images. Image upscaling by four times also outperforms corresponding previous examples with embedded oversampling.
Searching surveillance video

Surveillance video is used in two key modes, watching for known threats in real-time and searching for events of interest after the fact. Typically, real-time alerting is a localized function, e.g. airport security center receives and reacts to a "perimeter breach alert", while investigations often tend to encompass a large number of geographically distributed cameras like the London bombing, or Washington sniper incidents. Enabling effective search of surveillance video for investigation & preemption, involves indexing the video along multiple dimensions. This paper presents a framework for surveillance search which includes, video parsing, indexing and query mechanisms. It explores video parsing techniques which automatically extract index data from video, indexing which stores data in relational tables, retrieval which uses SQL queries to retrieve events of interest and the software architecture that integrates these technologies.
BCA: Bi-symmetric component analysis for temporal symmetry in human actions

In the past, many research efforts are invested into discriminative action recognition task but the general temporal structure of human actions is overlooked. In this paper, we focus on a specific yet common structure of human actions: temporal symmetry. The key contribution is that we model the temporal symmetry property of human action and separate this signal out of original action sequences without specifying which action category. Based on this modeling, a novel and effective method is proposed to detect the temporal symmetric part of any given human action sequence. Experimental results on two popular human action datasets verify that the temporal symmetry benefits both action detection and action recognition.
Indoor signage detection based on saliency map and bipartite graph matching

Object detection plays a very important role in many applications such as image retrieval, surveillance, robot navigation, wayfinding, etc. In this paper, we propose a novel approach to detect indoor signage to help blind people find their destinations in unfamiliar environments. Our method first extracts the attended areas by using a saliency map. Then the signage is detected in the attended areas by using bipartite graph matching. The proposed method can handle multiple signage detection. Experimental results on our collected indoor signage dataset demonstrate the effectiveness and efficiency of our proposed method. Furthermore, saliency maps could eliminate the interference information and improve the accuracy of the detection results.
Robust and Effective Component-Based Banknote Recognition for the Blind

We develop a novel camera-based computer vision technology to automatically recognize banknotes to assist visually impaired people. Our banknote recognition system is robust and effective with the following features: 1) high accuracy: high true recognition rate and low false recognition rate; 2) robustness: handles a variety of currency designs and bills in various conditions; 3) high efficiency: recognizes banknotes quickly; and 4) ease of use: helps blind users to aim the target for image capture. To make the system robust to a variety of conditions including occlusion, rotation, scaling, cluttered background, illumination change, viewpoint variation, and worn or wrinkled bills, we propose a component-based framework by using speeded up robust features (SURF). Furthermore, we employ the spatial relationship of matched SURF features to detect if there is a bill in the camera view. This process largely alleviates false recognition and can guide the user to correctly aim at the bill to be recognized. The robustness and generalizability of the proposed system are evaluated on a dataset including both positive images (with U.S. banknotes) and negative images (no U.S. banknotes) collected under a variety of conditions. The proposed algorithm achieves 100% true recognition rate and 0% false recognition rate. Our banknote recognition system is also tested by blind users.
Rotation and illumination invariant texture analysis: Matching clothes with complex patterns for blind people

Matching clothes is a challenging task for blind people. In this paper, we describe a new texture analysis approach which is robust to variations of rotation and illumination for matching two images of clothes with complex patterns. The approach integrates Radon transform, wavelet transform, and co-occurrence matrix for texture analysis of clothes images. We first calculate directional properties of texture pattern in each image by applying Radon transform. The directional properties are used to rotate the clothes image with the dominant orientation of the texture patterns as horizontal. To handle illumination changes, we perform Haar wavelet transform to extract texture features on three directions (horizontal, vertical and diagonal). For each wavelet sub image, grey level cooccurrence matrix for texture analysis is calculated. Finally, texture matching is performed based on six statistical features (i.e. mean, variance, smoothness, energy, homogeneity, and entropy). The robustness and effectiveness of the proposed method are evaluated on our database which contains 128 images of complex pattern clothes. The matching results are presented for blind users as speech outputs.
EigenJoints-based action recognition using Naïve-Bayes-Nearest-Neighbor

In this paper, we propose an effective method to recognize human actions from 3D positions of body joints. With the release of RGBD sensors and associated SDK, human body joints can be extracted in real time with reasonable accuracy. In our method, we propose a new type of features based on position differences of joints, EigenJoints, which combine action information including static posture, motion, and offset. We further employ the Naïve-Bayes-Nearest-Neighbor (NBNN) classifier for multi-class action classification. The recognition results on the Microsoft Research (MSR) Action3D dataset demonstrate that our approach significantly outperforms the state-of-the-art methods. In addition, we investigate how many frames are necessary for our method to recognize actions on the MSR Action3D dataset. We observe 15-20 frames are sufficient to achieve comparable results to that using the entire video sequences.
Reading labels of cylinder objects for blind persons

We propose a camera-based assistive framework to help blind persons to read text labels from cylinder objects in their daily life. First, the object is detected from the background or other surrounding objects in the camera view by shaking the object. Then we propose a mosaic model to unwarp the text label on the cylinder object surface and reconstruct the whole label for recognizing text information. This model can handle cylinder objects in any orientations and scales. The text information is then extracted from the unwarped and flatted labels. The recognized text codes are then output to blind users in speech. Experimental results demonstrate the efficiency and effectiveness of the proposed framework from different cylinder objects with complex backgrounds.
Localizing Text in Scene Images by Boundary Clustering, Stroke Segmentation, and String Fragment Classification

In this paper, we propose a novel framework to extract text regions from scene images with complex backgrounds and multiple text appearances. This framework consists of three main steps: boundary clustering (BC), stroke segmentation, and string fragment classification. In BC, we propose a new bigram-color-uniformity-based method to model both text and attachment surface, and cluster edge pixels based on color pairs and spatial positions into boundary layers. Then, stroke segmentation is performed at each boundary layer by color assignment to extract character candidates. We propose two algorithms to combine the structural analysis of text stroke with color assignment and filter out background interferences. Further, we design a robust string fragment classification based on Gabor-based text features. The features are obtained from feature maps of gradient, stroke distribution, and stroke width. The proposed framework of text localization is evaluated on scene images, born-digital images, broadcast video images, and images of handheld objects captured by blind persons. Experimental results on respective datasets demonstrate that the framework outperforms state-of-the-art localization algorithms.
Super Normal Vector for Activity Recognition Using Depth Sequences

This paper presents a new framework for human activity recognition from video sequences captured by a depth camera. We cluster hypersurface normals in a depth sequence to form the polynormal which is used to jointly characterize the local motion and shape information. In order to globally capture the spatial and temporal orders, an adaptive spatio-temporal pyramid is introduced to subdivide a depth video into a set of space-time grids. We then propose a novel scheme of aggregating the low-level polynormals into the super normal vector (SNV) which can be seen as a simplified version of the Fisher kernel representation. In the extensive experiments, we achieve classification results superior to all previous published results on the four public benchmark datasets, i.e., MSRAction3D, MSRDailyActivity3D, MSRGesture3D, and
Comprehensive database for facial expression analysis

Within the past decade, significant effort has occurred in developing methods of facial expression analysis. Because most investigators have used relatively limited data sets, the generalizability of these various methods remains unknown. We describe the problem space for facial expression analysis, which includes level of description, transitions among expressions, eliciting conditions, reliability and validity of training and test data, individual differences in subjects, head orientation and scene complexity image characteristics, and relation to non-verbal behavior. We then present the CMU-Pittsburgh AU-Coded Face Expression Image Database, which currently includes 2105 digitized image sequences from 182 adult subjects of varying ethnicity, performing multiple tokens of most primary FACS action units. This database is the most comprehensive testbed to date for comparative studies of facial expression analysis.
A primary travelling assistant system of bus detection and recognition for visually impaired people

It is a challenging task for many blind and visually impaired passengers travelling by bus. To assist visually impaired passengers to travel more independently, we design a computer vision-based system to detect and recognize bus information from images captured by a camera at a bus stop. Our system is able to notify the visually impaired people in speech the information of the coming bus, and detect the route number and other related information which is depicted in the form of text. In bus detection, histogram of the oriented gradient (HOG) descriptor is employed to extract the image-based features of bus facade. Cascade SVM model is applied to train a bus classifier to detect the existence of bus facade in sliding windows. In bus route number recognition, we design a text detection algorithm on the basis of layout analysis and text feature learning, and then recognize the text codes from detected text regions for audio notification. This algorithm is able to compute the image regions containing text information. Experimental results demonstrate the effectiveness of our proposed algorithm for bus detection and route number recognition.
Text Detection in Natural Scene Images by Stroke Gabor Words

In this paper, we propose a novel algorithm, based on stroke components and descriptive Gabor filters, to detect text regions in natural scene images. Text characters and strings are constructed by stroke components as basic units. Gabor filters are used to describe and analyze the stroke components in text characters or strings. We define a suitability measurement to analyze the confidence of Gabor filters in describing stroke component and the suitability of Gabor filters on an image window. From the training set, we compute a set of Gabor filters that can describe principle stroke components of text by their parameters. Then a K-means algorithm is applied to cluster the descriptive Gabor filters. The clustering centers are defined as Stroke Gabor Words (SGWs) to provide a universal description of stroke components. By suitability evaluation on positive and negative training samples respectively, each SGW generates a pair of characteristic distributions of suitability measurements. On a testing natural scene image, heuristic layout analysis is applied first to extract candidate image windows. Then we compute the principle SGWs for each image window to describe its principle stroke components. Characteristic distributions generated by principle SGWs are used to classify text or non-text windows. Experimental results on benchmark datasets demonstrate that our algorithm can handle complex backgrounds and variant text patterns (font, color, scale, etc.).
IBM smart surveillance system (S3): a open and extensible framework for event based surveillance

As smart surveillance technology becomes a critical component in security infrastructures, the system architecture assumes a critical importance. This paper considers the example of smart surveillance in an airport environment. We start with a threat model for airports and use this to derive the security requirements. These requirements are used to motivate an open-standards based architecture for surveillance. We discuss the critical aspects of this architecture and its implementation in the IBM S3 smart surveillance system. Demo results from a pilot deployment in Hawthorne, NY are presented.
Scene text recognition in multiple frames based on text tracking

Text signage as visual indicators in natural scene plays an important role in navigation and notification in our daily life. Most previous methods of scene text extraction are developed from a single scene image. In this paper, we propose a multi-frame based scene text recognition method by tracking text regions in a video captured by a moving camera. The main contributions of this paper are as follows. First, we present a framework of scene text recognition in multiple frames based on feature representation of scene text character (STC) for character prediction and conditional random field (CRF) model for word configuration. Second, a feature representation of STC is employed from dense sampled SIFT descriptors and Fisher Vector. Third, we collect a dataset for text information extraction from natural scene videos. Our proposed multi-frame scene text recognition is more compatible with image/video-based mobile applications. The experimental results demonstrate that STC prediction and word configuration in multiple frames based on text tracking significantly improves the performance of scene text recognition.
Edge Enhanced Depth Motion Map for Dynamic Hand Gesture Recognition

In this paper, we propose a novel approach to recognize dynamic hand gestures from depth video by integrating Edge Enhanced Depth Motion Map together with Histogram of Gradient descriptor. The novelty of this paper has two aspects: first, we propose a novel feature representation, Edge Enhanced Depth Motion Map (E2DMM), balancing the information weighing between shape and motion, which is more suitable for hand gesture recognition, second, we further employ a dynamic temporal pyramid to segment the depth video sequence to address temporal structure information of dynamic hand gestures. Histogram of Gradient is applied on E 2 DMM to generate vectored representation. Comparison study has been conducted with the state-of-the-art approaches and demonstrates that our approach can achieve better and more stable performance while keeping a relative simple model with lower complexity as well as higher generality.
Embedding Sequential Information into Spatiotemporal Features for Action Recognition

In this paper, we introduce a novel framework for videobased action recognition, which incorporates the sequential information with the spatiotemporal features. Specifically, the spatiotemporal features are extracted from the sliced clips of videos, and then a recurrent neural network is applied to embed the sequential information into the final feature representation of the video. In contrast to most current deep learning methods for the video-based tasks, our framework incorporates both long-term dependencies and spatiotemporal information of the clips in the video. To extract the spatiotemporal features from the clips, both dense trajectories (DT) and a newly proposed 3D neural network, C3D, are applied in our experiments. Our proposed framework is evaluated on the benchmark datasets of UCF101 and HMDB51, and achieves comparable performance compared with the state-of-the-art results.
Text String Detection From Natural Scenes by Structure-Based Partition and Grouping

Text information in natural scene images serves as important clues for many image-based applications such as scene understanding, content-based image retrieval, assistive navigation, and automatic geocoding. However, locating text from a complex background with multiple colors is a challenging task. In this paper, we explore a new framework to detect text strings with arbitrary orientations in complex natural scene images. Our proposed framework of text string detection consists of two steps: 1) image partition to find text character candidates based on local gradient features and color uniformity of character components and 2) character candidate grouping to detect text strings based on joint structural features of text characters in each text string such as character size differences, distances between neighboring characters, and character alignment. By assuming that a text string has at least three characters, we propose two algorithms of text string detection: 1) adjacent character grouping method and 2) text line grouping method. The adjacent character grouping method calculates the sibling groups of each character candidate as string segments and then merges the intersecting sibling groups into text string. The text line grouping method performs Hough transform to fit text line among the centroids of text candidates. Each fitted text line describes the orientation of a potential text string. The detected text string is presented by a rectangle region covering all characters whose centroids are cascaded in its text line. To improve efficiency and accuracy, our algorithms are carried out in multi-scales. The proposed methods outperform the state-of-the-art results on the public Robust Reading Dataset, which contains text only in horizontal orientation. Furthermore, the effectiveness of our methods to detect text strings with arbitrary orientations is evaluated on the Oriented Scene Text Dataset collected by ourselves containing text strings in nonhorizontal orientations.
Finding objects for blind people based on SURF features

Nowadays computer vision technology is helping the visually impaired by recognizing objects in their surroundings. Unlike research of navigation and wayfinding, there are no camera-based systems available in the market to find personal items for the blind. This paper proposes an object recognition method to help blind people find missing items using Speeded-Up Robust Features (SURF). SURF features can extract distinctive invariant features that can be utilized to perform reliable matching between different images in multiple scenarios. These features are invariant to image scale, translation, rotation, illumination, and partial occlusion. The proposed recognition process begins by matching individual features of the user queried object to a database of features with different personal items which are saved in advance. Experiment results demonstrate the effectiveness and efficiency of the proposed method.
3D convolutional neural network with multi-model framework for action recognition

In this paper, we propose an efficient and effective action recognition framework by combining multiple feature models from dynamic image, optical flow and raw frame, with 3D convolutional neural network (CNN). Dynamic image preserves the long-term temporal information, while optical flow captures short-term temporal information, and raw frame represents the appearance information. Experiments demonstrate that dynamic image provides complementary information to raw frame feature and optical flow feature. Furthermore, with the approximate rank pooling, the computation of dynamic images is about 360 times faster than optical flow, and the dynamic image requires far less memory than optical flow and raw frame.
An effective view and time-invariant action recognition method based on depth videos

Little progress has been achieved in hand-crafted feature based human action recognition (HAR) for RGB videos in recent years. The emergence of low price depth camera presents more information for action recognition. Compared to RGB videos, depth video sequences are more insensitive to light changes and more discriminative in many vision tasks such as segmentation and activity recognition. In this paper, we propose an effective and straightforward HAR method by using skeleton joints information of the depth sequence. First, we calculate three feature vectors which capture angle and position information between joints. Then, the obtained vectors are used as the inputs of three separate support vector machine (SVM) classifiers. Finally, the action recognition is conducted by fusing the SVM classification results. Our features are viewinvariant because the extracted vectors contain only angle and normalized position information based on joint coordinates. By normalizing action videos with different temporal lengths to a fixed size using interpolation, the extracted features have the same dimension for different videos and can still keep the principal movement patterns which make the proposed method timeinvariant. Experimental results demonstrate that our method performs comparable results on the UTKinect-Action3D dataset, and is more efficient and simpler than state-of-the-art methods.