nils-herrmann/clean_scirepeval huggingface dataset:
Understanding the Nature of First-Person Videos: Characterization and Classification Using Low-Level Features

First-person view (FPV) video data is set to proliferate rapidly, due to many consumer wearable-camera devices coming onto the market. Research into FPV (or "egocentric") vision is also becoming more common in the computer vision community. However, it is still unclear what the fundamental characteristics of such data are. How is it really different from third-person view (TPV) data? Can all FPV data be treated the same? In this first attempt to approach these questions in a quantitative and empirical manner, we analyzed a meta-collection of 21 FPV and TPV datasets totaling more than 165 hours of video. We performed the first quantitative characterization of FPV videos over multiple datasets, encompassing virtually all available FPV datasets. Validating this characterization, linear classifiers trained on low-level features to perform FPV-versus-TPV classification achieved good baseline performance. Accuracy peaked at 81% for 2-minute clips, but 67% accuracy was achieved even with 1-second clips. Our low-level features are fast to compute and do not require annotation. Overall, our work uncovered insights regarding the basic nature and characteristics of FPV data.
Automatic cell classification and population estimation in blastocystis autophagy images

Blastocystis is a unicellular but polymorphic protozoan parasite causing digestive diseases in humans. Autophagy, a self-degradation process, is only recently found in Blastocystis. Identifying and enumerating autophagic Blastocystis cells using fluorescent microscopy are important in biology. Doing this manually is laborious and error-prone. This paper proposes image analysis techniques to automate the process. The difficulties are poor image quality and large variations in illumination and cell morphology. We divide the cells into several sub-classes of different morphology. Support vector machines are used to learn domain knowledge and classify the cells. Validation experiments on separate data sets show reliable performance for manually segmented cells with sensitivity 82.2% and specificity 86.7%. For automatically segmented cells, the sensitivity is the same. However, the specificity drops down to 68.4%. To our knowledge, this is the first attempt in automatic processing these images.
Unifying local and global content-based similarities for home photo retrieval

Unlike professional or domain-specific images, home photos vary significantly. They pose great challenge for content-based image retrieval. In this paper, we propose a Bayesian formulation to unify both local and global content-based similarities for image matching and demonstrate its superior retrieval performance on 2400 genuine home photos. Our proposed framework uses support vector machines to extract and combine intraimage and interclass semantics. Support vector detectors are first trained on semantically meaningful regions and used to form detection image indexes. The indexes then serve as input for support vector learning of image classifiers to generate class-relative indexes. During image retrieval, similarities based on both detection-based and class-relative indexes are combined to rank images. Query-by-example experiments on 2400 home photos with 16 semantic queries show that the combined matching approach is better than matching with single index. It also outperformed the fusion of color and texture features by 55% in average precision.
Learning from an ensemble of Receptive Fields

In this paper, we construct a neural-inspired computational model based on the representational capabilities of receptive fields. The proposed model, known as Shape Encoding Receptive Fields (SERF), is able to perform fast and accurate data classification and regression of multi-dimensional data. A SERF is a histogram structure that encodes the shape of multi-dimensional data relative to its center, in a manner similar to the neural coding of sensory stimulus by the receptive fields. The bins of this histogram represent a local region in an n-dimensional space. During the training phase, an ensemble of K SERF structures are initialized and data is summarized into the corresponding bins of each SERF structure. The collection of local data summaries makes each SERF a coarse nonlinear data predictor over the entire feature space. The output prediction of an unknown query is computed by the weighted aggregation of the hypotheses of the ensemble of K SERFs. In our series of experiments, we demonstrate the model's superiority to perform fast and accurate data prediction.
Fuzzy constraint logic programming: a proposal

In this paper, we propose a new framework to integrate fuzzy logic programming and fuzzy mathematical programming. While fuzzy relation unifies fuzzy predicate and fuzzy constraint, the compositional rule of inference, when viewed in the right way, bridges the gap between a fuzzy logic clause and a fuzzy optimization formulation. This framework can also be seen as a fuzzification of the constraint logic programming paradigm. We describe a calculus for it and finally discuss the potential approaches for deriving a computational procedure.<<ETX>>
Support regions and images for photo event retrieval

The notion of event conveys rich semantics to consumers in their collection of photos. Our user studies confirm that consumers would like to organize and access their digital photo memories by events, complemented by other semantic axes such as people, time, and place. In this paper, we address photo event retrieval by means of semantic support regions and images. Semantic support regions are salient image regions that exhibit semantic meanings and that can be learned from sample images to span a new indexing space. Semantic support images are key photos derived statistically to model photo events. Retrieval of a photo event is performed using a winner-take-all approach to compute the relevance measure of a given photo to the event models. This novel approach for photo event retrieval is experimented on 2400 heterogeneous consumer photos with very promising results.
Summarization of Egocentric Videos: A Comprehensive Survey

The introduction of wearable video cameras (e.g., GoPro) in the consumer market has promoted video life-logging, motivating users to generate large amounts of video data. This increasing flow of first-person video has led to a growing need for automatic video summarization adapted to the characteristics and applications of egocentric video. With this paper, we provide the first comprehensive survey of the techniques used specifically to summarize egocentric videos. We present a framework for first-person view summarization and compare the segmentation methods and selection algorithms used by the related work in the literature. Next, we describe the existing egocentric video datasets suitable for summarization and, then, the various evaluation methods. Finally, we analyze the challenges and opportunities in the field and propose new lines of research.
Cell Clumping Quantification and Automatic Area Classification in Peripheral Blood Smear Images

Cell enumeration in peripheral blood smears and cell are widely applied in biological and pathological practice. Not every area in the smear is appropriate for enumeration due to severe cell clumping or sparseness arising from smear preparation. The automatic selection of good areas for cell enumeration can reduce manual labor and provide objective and consistent results. However, this has been infrequently studied and it is often difficult to count the exact number of cells in the clumps. To select good areas, we do not have to do this. Instead, we measure the goodness of such areas in terms of the degree of cell spread and the degree of clumping. The later is defined based on the distances and linking strengths of local voting peaks generated in the accumulator space after multi-scale circular Hough transforms. Support vector machines are then applied to classify the image areas into good or non-good classes. We have validated our method over 4500 testing cell images and achieved 89% sensitivity and 87% specificity.
Towards Automatic Mobile Blogging

Weblog (usually shortened as blog) has gained its popularity lately. There are about 70,000 new blogs a day and about 29,100 blog updates an hour. As an emerging blogging phenomenon, with the proliferation of camera phones, mobile bloggers can write their blogs almost instantaneously. But how much further can current mobile blogging tools enhance the experience? In this paper, we propose a Mobilog framework to automate context-relevant annotation and synthesise personalised content for mobile blogging. In particular, we describe a system implementation of the framework, Travelog, adapted for tourism applications. Finally, we discuss the challenges and future possibilities for mobile blogging
Adaptive picture-in-picture technology based on visual saliency

Picture-in-picture (PiP) is a feature of some television receivers and video devices, which allows one main program to be displayed on the full screen while one or more subprogram displayed in inset windows. Currently most TV/video devices require users to specify where and how large to place the sub-program over the main program display. This process is however not user-friendly as it involves a manual process and once specified, the size and the location of the sub-program will be fixed even when they block some key visual information from the main program. We propose an automatic and adaptive PiP technology that makes use of computational modeling of visual saliency. For each frame of the main program, a saliency map is computed efficiently which quantifies how probable a display region of the main program contains useful information and will attract humans' attention/eyes. The sub-program can thus be adaptively resized and placed to the display region that contains the least useful information. Preliminary experiments show the effectiveness of the proposed technology.
Automatic detection of cortical and PSC cataracts using texture and intensity analysis on retro-illumination lens images

Cataract remains a leading cause for blindness worldwide. Cataract diagnosis via human grading is subjective and time-consuming. Several methods of automatic grading are currently available, but each of them suffers from some drawbacks. In this paper, a new approach for automatic detection based on texture and intensity analysis is proposed to address the problems of existing methods and improve the performance from three aspects, namely ROI detection, lens mask generation and opacity detection. In the detection method, image clipping and texture analysis are applied to overcome the over-detection problem for clear lens images and global thresholding is exploited to solve the under-detection problem for severe cataract images. The proposed method is tested on 725 retro-illumination lens images randomly selected from a database of a community study. Experiments show improved performance compared with the state-of-the-art method.
Learning visual keywords for content-based retrieval

Today keyword-based teat retrieval systems have enjoyed reasonable success in real usage. Despite the simplicity of the keyword metaphor, practical test search engines are able to handle huge volumes of free text documents. In image and video retrieval, search mainly relies on pre-annotated words or/and primitive visual features. We propose the notion of visual keywords for content-based retrieval. The visual keywords of a given visual content domain are typical visual entities that are extracted from statistical learning. A visual content is spatially described in terms of the extracted visual keywords and coded via singular value decomposition for similarity matching. We demonstrate our framework in retrieval of natural scene images.
Scene Recognition with Camera Phones for Tourist Information Access

Camera phones present new opportunities and challenges for mobile information association and retrieval. The visual input in the real environment is a new and rich interaction modality between a mobile user and vast information base connected to a user's device via rapidly advancing communication infrastructure. We have developed a system for tourist information access to provide scene description based on an image taken of the scene. In this paper, we describe the working system, the STOIC 101 database, and a new pattern discovery algorithm to learn image patches that are recurrent within a scene class and discriminative across others. We report preliminary scene recognition results on 90 scenes, trained on 5 images per scene, with an accuracy of 92% and 88% on a test set of 110 images, with and without location priming.
Foveated neural network: Gaze prediction on egocentric videos

A novel deep convolution neural network is proposed to predict gaze on current frames in egocentric videos. Inspired by human visual system, we introduce a fovea module responsible for sharp central vision and name our model as Foveated Neural Network (FNN). The retina-like visual inputs from the region of interest on the previous frame are analysed and encoded. The fusion of the hidden representations of the previous frame and the feature maps of the current frame guides the gaze prediction on the current frame. In order to simulate motion, we also include the dense optical flow between these adjacent frames as additional input. Experimental results show that FNN outperforms the state-of-the-art algorithms in the publicly available egocentric dataset. The analysis of FNN demonstrates that the hidden representations of the foveated visual input from the previous frame as well as the motion information between adjacent frames are efficient in improving gaze prediction performance in egocentric videos.
Automated layer segmentation of optical coherence tomography images

By measuring the thickness of the retinal nerve fiber layer, retinal optical coherence tomography (OCT) images are now increasingly used for the diagnosis of glaucoma. This paper reports an automatic OCT layer segmentation technique that can be used for computer-aided glaucoma diagnosis. In the proposed technique, blood vessels are first detected through an iterative polynomial smoothing procedure. OCT images are then filtered by a bilateral filter and a median filter sequentially. In particular, both filters suppress the local image noise but the bilateral filter has a special characteristic that keeps the global trend of the image value variation. After the image filtering, edges are detected and the edge segments corresponding to the layer boundary are further identified and clustered to form the layer boundary. Experiments over OCT images of four subjects show that the proposed technique segments layers of OCT images efficiently.
Automatic working area classification in peripheral blood smears using spatial distribution features across scales

Automatic classification of working areas in peripheral blood smears can provide objective and reproducible quality control for the evaluation of smears and smear maker devices. However, it has drawn little research attention. In this paper we study this topic using image analysis and statistical pattern recognition methods. We employ generic features without requiring the extraction of individual cells. Two new spatial distribution features across scales are defined and utilized to classify working areas. We demonstrate that the only feature and method proposed in a similar work by others is insufficient to characterize the goodness of working areas, particularly the cell distribution. However, by utilizing it together with the features developed in this paper, we can achieve much better results. Our method has been tested on about 150 labeled images acquired from three malaria-infected Giemsa-stained blood smears using an oil immersion 100x objective lens.
Multi-layer linear model for top-down modulation of visual attention in natural egocentric vision

Top-down attention plays an important role in guidance of human attention in real-world scenarios, but less efforts in computational modeing of visual attention has been put on it. Inspired by the mechanisms of top-down attention in human visual perception, we propose a multi-layer linear model of top-down attention to modulate bottom-up saliency maps actively. The first layer is a linear regression model which combines the bottom-up saliency maps on various visual features and objects. A contextual dependent upper layer is introduced to tune the parameters of the lower layer model adaptively. Finally, a mask of selection history is applied to the fused attention map to bias the attention selection towards the task related regions. Efficient learning algorithm with single-pass polynomial complexity is derived. We evaluate our model on a set of natural egocentric videos captured from a wearable glass in real-world environments. Our model outperforms the baseline and state-of-the-art bottom-up saliency models.
Combining a global SVM and local nearest-neighbor classifiers driven by local discriminative boundaries

Nonlinear support vector machines (SVMs) rely on the kernel trick and tradeoff parameters to build nonlinear models to classify complex problems and balance misclassification and generalization. The inconvenience in determining the kernel and the parameters has motivated the use of local nearest neighbor (NN) classifiers in lieu of global classifiers. This substitution ignores the advantage of SVM in global error minimization. On the other hand, the NN rule assumes that class conditional probabilities are locally constant. Such an assumption does not hold near class boundaries and in any high dimensional space due to the curse of dimensionality. We propose a hybrid classification method combining the global SVM and local NN classifiers. Local classifiers occur only when the global SVM is likely to fail. Furthermore, local NN classifiers adopt an adaptive metric driven by local SVM discriminative boundaries. Improved performance has been demonstrated compared to partially similar.
Rich representation and ranking for photographic image retrieval in ImageCLEF 2007

The task of ad hoc photographic image retrieval in ImageCLEF 2007 international benchmark is to retrieve relevant images in the database to the user query formulated as keywords and image examples. This paper presents rich representation and indexing technologies exploited in our system that participated in ImageCLEF 2007. It uses diverse visual content representation, text representation, pseudo-relevance feedback and fusion, which make our system, with mean average precision 0.2833, in the 4th place among 457 automatic runs submitted from 20 participants to photographic ImageCLEF 2007 and in the 2nd place in terms of participants. Our systematic analysis in the paper demonstrates that 1) combing diverse low-level visual features and ranking technologies significantly improves the content-based image retrieval (CBIR) system; 2) cross-modality pseudo-relevance feedback improves the system performance; and 3) fusion of CBIR and TBIR outperforms individual modality based system.
Learning invariant color features with sparse topographic restricted Boltzmann machines

Our objective is to learn invariant color features directly from data via unsupervised learning. In this paper, we introduce a method to regularize restricted Boltzmann machines during training to obtain features that are sparse and topographically organized. Upon analysis, the features learned are Gabor-like and demonstrate a coding of orientation, spatial position, frequency and color that vary smoothly with the topography of the feature map. There is also differentiation between monochrome and color filters, with some exhibiting color-opponent properties. We also found that the learned representation is more invariant to affine image transformations and changes in illumination color.
Organizing and retrieving episodic memories from first person view

Our life is becoming heavily documented and expressed on the digital substrate, which leads to an increasing demand of multimedia analysis tools to organize those memories and retrieve related information when required. Previous systems for information retrieval from first-person-view (FPV) rely on purely visual features, leaving aside the semantic information and failing to extract the personal nature of FPV videos. We propose a system which is capable of organizing and retrieving our digital memories based on their semantic similarity to a given video query. Moreover, we present a novel segmentation method for egocentric vision using multimedia features such as location, time, interactions or activity. Results show its high potential for real FPV videos, making it useful for a wide range of applications for life logging: from personal memory retrieval, to egocentric video summarization.
Combining local class patterns and discovered semantics for image retrieval

Detecting meaningful visual entities (e.g. faces, sky, foliage, buildings, etc.) based on supervised pattern classifiers has become a trend in content-based image retrieval. However, a drawback of the supervised learning approach is the need for manually labeled regions as training samples. We propose a semi-supervised framework to discover local semantic patterns and generate their samples for training with minimal human intervention. Image classifiers are first trained on local image blocks from a small number of labeled images. Then, local semantic patterns are discovered from clustering the image blocks with high classification output. Training samples are induced from cluster memberships for support vector learning to form local semantic pattern detectors. During retrieval, similarities based on local class pattern indexes and discovered pattern indexes are combined to rank images. Query-by-example experiments on 2400 unconstrained consumer photos with 16 semantic queries show that the combined matching approach outperformed the fusion of color and texture features significantly in average precision by 37%.
Design and Evaluation of Vibrating Footwear for Navigation Assistance to Visually Impaired People

A wearable system is designed to provide directional information to visually impaired people. It consists of a mobile phone and haptic shoes. The former serves as the perceptual and control unit that generates directional instructions. Upon receiving the instructions, the shoes combine them with the user's walking status to produce unique vibration patterns. To enable effective direction sensing, a few alternative configurations are proposed, whereby the position and strength of vibrations are modulated programmatically. The prototype system is evaluated in a usability test with 60 subjects. By comparing different configurations, it is shown that the system achieves varying levels of perception accuracy under various walking conditions, and that the proposed design is advantageous to the benchmarking configuration. The system has great potential to provide smart and sensible navigation guidance to visually impaired people especially when integrated with visual processing units.