nils-herrmann/clean_scirepeval huggingface dataset:
Single image depth estimation from image descriptors

With the rapid emergence of 3D displays, we can enrich the user's viewing experiences by adding depth information to the widely existing 2D contents. However, effectively inferring the associated depth from a single 2D image is still a challenging problem. By taking benefits from the recently appeared image descriptors, we proposed the use of an SVM based framework for addressing the single image depth estimation. One advantage is its direct extension to incorporate the recent researches of large scale classification via SVM to meet the upcoming cloud computing paradigm. Our experimental results showed that the proposed framework outperforms the state-of-the-art approaches in performance, even the ones using more complex graphical models like MRF. Also, we made a brief investigation on the individual effectiveness of a set of commonly used image descriptors and found that spatial descriptors (e.g. texture) would be more effective than frequency ones (e.g. DCT coefficients).
VRank: Voting system on Ranking model for human age estimation

Ranking algorithms have proven the potential for human age estimation. Currently, a common paradigm is to compare the input face with reference faces of known age to generate a ranking relation whereby the first-rank reference is exploited for labeling the input face. In this paper, we proposed a framework to improve upon the typical ranking model, called Voting system on Ranking model (VRank), by leveraging relational information (comparative relations, i.e. if the input face is younger or older than each of the references) to make a more robust estimation. Our approach has several advantages: firstly, comparative relations can be explicitly involved to benefit the estimation task; secondly, few incorrect comparisons will not influence much the accuracy of the result, making this approach more robust than the conventional approach; finally, we propose to incorporate the deep learning architecture for training, which extracts robust facial features for increasing the effectiveness of classification. In comparison to the best results from the state-of-the-art methods, the VRank showed a significant outperformance on all the benchmarks, with a relative improvement of 5.74% ~ 69.45% (FG-NET), 19.09% ~ 68.71% (MORPH), and 0.55% ~ 17.73% (IoG).
Gestalt Rule Feature Points

As the large online repositories of image and video data has emerged and continued to grow in number, the visual variations in such repositories has also increased dramatically. For example, the visual scene of a photograph can be changed into different colors by image editing tools or depicted by multiple representations, such as a painting and a hand-drawn sketch. The large visual variations tend to cause ambiguities for the existing computer vision algorithms to recognize the visual analogies of these images and often limit the potential of related applications. In this paper, therefore, we propose a new approach for detecting reliable visual features from images, with a particular focus on improving the repeatability of the local features in those images containing the same semantic contents (e.g., a landmark) but in different visual styles (e.g., a photo and a painting). We proposed a novel method for establishing visual correspondences between images based on the Gestalt theory, a psychological study of how human visions organize the visual perception. Experiments demonstrated the outperformance of our approach over the state-of-the-art local features in various computer vision tasks, such as cross domain image matching and retrieval.
Unsupervised auxiliary visual words discovery for large-scale image object retrieval

Image object retrieval–locating image occurrences of specific objects in large-scale image collections–is essential for manipulating the sheer amount of photos. Current solutions, mostly based on bags-of-words model, suffer from low recall rate and do not resist noises caused by the changes in lighting, viewpoints, and even occlusions. We propose to augment each image with auxiliary visual words (AVWs), semantically relevant to the search targets. The AVWs are automatically discovered by feature propagation and selection in textual and visual image graphs in an unsupervised manner. We investigate variant optimization methods for effectiveness and scalability in large-scale image collections. Experimenting in the large-scale consumer photos, we found that the the proposed method significantly improves the traditional bag-of-words (111% relatively). Meanwhile, the selection process can also notably reduce the number of features (to 1.4%) and can further facilitate indexing in large-scale image object retrieval.
Whac-a-mole: A head detection scheme by estimating the 3D envelope from depth image

Interactive multimedia display has attracted great attention in recent years. However, most of the existing systems lack the user-aware capability, i.e. blind to the viewer's height and spatial location in a real-world 3D space, and often fail to provide a natural interaction. Therefore, in this work, we propose a probabilistic framework for detecting the viewer, (i.e. human heads) in depth images from a bird's eye view camera. In comparison to the state-of-the-art approaches, the experimental results demonstrated that the proposed framework can provide higher detection rate but also real-time execution. Even a large number of people are walking or standing together shoulder-by-shoulder, the proposed probabilistic head detection scheme is still able to give promising people detection capability.
Demo paper: A depth-based crowded heads detection system through a freely-located camera

This work demonstrates a novel system for locating multiple human subjects in a tight crowd by detecting their heads. By exploiting the 3D point cloud captured from a fixed depth camera, our system first develops a user interface for the users to interactively adjust the observation view to a proper place for head detection. Next, the proposed probabilistic framework can detect the human heads in the depth images from the manually adjusted bird's eye view camera. The proposed interactive head detection system can provide high detection rate and real-time execution. Even a crowd of people are staying together shoulder-by-shoulder, the proposed system is still able to give promising people detection capability.
A practical foveation-based rate-shaping mechanism for MPEG videos

Foveation is one of the nonuniform resolution properties of the human visual system. Recently, different foveation models are proposed and utilized for image and video coding, for the sake of bit-rate saving with no or minor perceptual quality distortion. In the first part of this paper, we propose an efficient and practical DCT-domain foveation model, which is deduced from existing experimental results. In the second part, we present a foveation-based rate-shaping mechanism for MPEG bitstreams, as an application example of the proposed foveation model. The rate shaper is based on eliminating DCT coefficients embedded in MPEG bitstreams. An efficient rate-shaping mechanism is developed to meet various bit-rate requirements. Our simulation confirmed that the proposed foveation model and the rate-shaping mechanism are practical for real-world usage.
Boundary Delineation of Breast Lesions in Series of 2D Sonography by Modeling the Spatial-Temporal Prior and Cell-Based MAP Approach

This paper proposes a general boundary delineation method for 2D serial US images by modeling the spatial-temporal dynamics and generating the object boundary in each slice under the cell-based MAP scheme. The modeling of the spatial-temporal dynamics can serve as a prior to guide the cell-based MAP process and potentially maintain the contextual coherence. Experiments have been conducted on 8 sets of compression breast series and 5 sets of freehand breast acquisitions. The computer-generated results by our algorithm are compared to manual delineations prepared by experts. The experimental results suggest that the boundaries of the proposed method are not significantly different to manual outlines and are quite stable in terms of reproducibility.
Semantic Analysis for Automatic Event Recognition and Segmentation of Wedding Ceremony Videos

Wedding is one of the most important ceremonies in our lives. It symbolizes the birth and creation of a new family. In this paper, we present a system for automatically segmenting a wedding ceremony video into a sequence of recognizable wedding events, e.g., the couple's wedding kiss. Our goal is to develop an automatic tool that helps users to efficiently organize, search, and retrieve his/her treasured wedding memories. Furthermore, the obtained event descriptions could benefit and complement the current research in semantic video understanding. Based on the knowledge of wedding customs, a set of audiovisual features, relating to the wedding contexts of speech/music types, applause activities, picture-taking activities, and leading roles, are exploited to build statistical models for each wedding event. Thirteen wedding events are then recognized by a hidden Markov model, which takes into account both the fitness of observed features and the temporal rationality of event ordering to improve the segmentation accuracy. We conducted experiments on a collection of wedding videos and the promising results demonstrate the effectiveness of our approach. Comparisons with conditional random fields show that the proposed approach is more effective in this application domain.
Optical switching of porous anodic aluminum oxide films embedded with silver nanoparticles

Dielectric films embedded with two-dimensional (2-D) arrays of metal nanoparticles have been fabricated by electro deposition of silver into nanopores of anodic aluminum oxide (AAO) films. Their pore diameter and inter-pore spacing are 80 nm and 20 nm, respectively. With its photoconductivity, the specimen has a higher conductance when being illuminated by a 514 nm wavelength laser. This photoconductivity can be explained by improved electron transport due to plasmonic coupling among neighboring silver nanoparticles induced by light illumination. The 2-D photoconductivity is expected to enable applications which are not suitable for one-dimensional photoconductivity induced by plasmonic coupling of nanoparticles embedded in nanowires. The 2-D photoconductivity of metal-nanoparticle-embedded dielectric films are expected to facilitate optoelectronic integrated circuits in the future.
Attaching-music: An interactive music delivery system for private listening as wherever you go

In this paper, we propose a music listening system, called “Attaching-Music,” which detects a human subject's ear position and projects the sound to a listener's ears by two directional speakers with servo motors. Our probabilistic head detection scheme from depth data can provide precise location for speakers to project the sound to an audience. Using the proposed “Attaching-Music” system, a headset-free user can listen to a music with proper soundstage with stereo effect based on two directional speakers driven by the corresponding servo motors, with the 3D localization parameters provided by our head detection scheme in an operational range.
Animating Still Landscape Photographs Through Cloud Motion Creation

Animating landscape photographs can create an engaging viewing experience and has long been an active area of multimedia research. Images can be used to create an illusion of movement, thereby providing more vivid and interesting visuals . Among numerous objects in visual scenes, clouds appear as a common element in landscape photos but are challenging to manipulate because clouds are ill-posed structures without concrete forms and present motion with very high degrees of freedom and arbitrary shape deformation. This paper addresses the generation of dynamic imagery from a still photo through automatic motion creation. First, a cloud appearance model is acquired by representing each pixel in the sky region as a combination of two computable image properties , i.e., Cloudiness estimates cloud density and Cloud Structure gives the corresponding cloud shape and texture. Then, based on the obtained Cloudiness and Cloud Structure, a content-aware wind field is generated to synthesize the motion of cloud flow. Experimental results demonstrate that our approach can work well on various types of cloud images and produce visually convincing dynamic imagery of landscape photos. Furthermore , we showcased several extended multimedia applications to validate that the proposed framework is generic and widely extensible to other vapor-like objects like mist and smoke. Please see http://mclab.citi.sinica.edu.tw/demo/dynamicimagery.htm for demos.
Generative and Discriminative Modeling toward Semantic Context Detection in Audio Tracks

Semantic-level content analysis is a crucial issue to achieve efficient content retrieval and management. We propose a hierarchical approach that models the statistical characteristics of several audio events over a time series to accomplish semantic context detection. Two stages, including audio event and semantic context modeling/testing, are devised to bridge the semantic gap between physical audio features and semantic concepts. For action movies we focused in this work, hidden Markov models (HMMs) are used to model four representative audio events, i.e. gunshot, explosion, car-braking, and engine sounds. At the semantic context level, generative (ergodic hidden Markov model) and discriminative (support vector machine, SVM) approaches are investigated to fuse the characteristics and correlations among various audio events, which provide cues for detecting gunplay and car-chasing scenes. The experimental results demonstrate the effectiveness of the proposed approaches and draw a sketch for semantic indexing and retrieval. Moreover, the differences between two fusion schemes are discussed to be the reference for future research.
Multi-cue pedestrian detection from 3D point cloud data

Pedestrian detection is one of the key technologies of driver assistance system. In order to prevent potential collisions, pedestrians should be always accurately identified whether during the day or at night. Since the visual images of the night are not clear, this paper proposes a method for recognizing pedestrians by using a high-definition LIDAR without visual images. In order to handle the long-distance sparse point problem, a novel solution is introduced to improve the performance. The proposed method maps the three-dimensional point cloud to the two-dimensional plane by a distance-aware expansion approach and the corresponding 2D contour and its associated 2D features are then extracted. Based on both 2D and 3D cues, the proposed method obtains significant performance boosts over state-of-the-art approaches by 13% in terms of F1-measure.
Medical image denoising using sparse representations

Image noise reduction, or denoising, became an attractive research domain in the recent years. There are several published approaches in the image denoising field and each one has its presumptions, advantages, and limitations. In this paper, we address the problem of eliminating or even decreasing the noise from medical images. Some schemes for image noise reduction are proposed. The schemes include DCT_OMP, DCT_BOMP, Log Gabor_BOMP, DCT_OCMP and Wavelet_OMP. In these schemes, the image content is represented as a sparse linear combination of a set of atoms that can be obtained from some trained dictionaries. In literature, several algorithms are proposed to build up these kinds of dictionaries. In our study, the K-SVD algorithm is utilized to obtain a dictionary that can effectively describe the image. Also, some greedy algorithms are used to perform the sparse-coding of the signal. The proposed schemes have been tested on two different medical images with different levels of an additive noise. The Log Gabor_BOMP scheme showed a significant outperformance, in both the de-noising results and computational time, on all the other schemes.
A study of semantic context detection by using SVM and GMM approaches

Semantic-level content analysis is a crucial issue to achieve efficient content retrieval and management. In this paper, we propose an hierarchical approach that models the statistical characteristics of several audio events over a time series to accomplish semantic context detection. Two stages, including audio event and semantic context modeling/testing, are devised to bridge the semantic gap between physical audio features and semantic concepts. HMM are used to model audio events, and SVM and GMM are used to fuse the characteristics of various audio events related to some specific semantic concepts. The experimental results show that the approach is effective in detecting semantic context. The comparison between SVM- and GMM-based approaches is also studied
A feature fusion framework for hashing

A hash algorithm converts data into compact strings. In the multimedia domain, effective hashing is the key to large-scale similarity search in high-dimensional feature space. A limit of existing hashing techniques is that they typically use single features. In order to improve search performance, it is necessary to utilize multiple features. Due to the compactness requirement, concatenation of hash values from different features is not an optimal solution. Thus a fusion process is desired. In this paper, we solve the multiple feature fusion problem by a hash bit selection framework. Given multiple features, we derive an n-bit hash value of improved performance compared with hash values of the same length computed from each individual feature. The framework utilizes a feature-independent hash algorithm to generate a sufficient number of bits from each feature, and selects n bits from the hash bit pool by leveraging pair-wise label information. The metric bit reliability is used for ranking the bits. It is estimated by bit-level hypothesis testing. In addition, we also take into account the dependence among bits. A weighted graph is constructed for refined bit selection, where the bit reliability is used as vertex weights and the mutual information among hash bits is used as edge weights. We demonstrate our framework with LSH. Extensive experiments confirm that our method is effective, and outperforms several state-of-the-art methods.
Who's Who in a Sports Video? An Individual Level Sports Video Indexing System

Sports video analysis has attracted great attention in recent years. In the past decade, numerous sports video indexing approaches have been proposed at different semantic levels. In this paper, an individual level sports video indexing (ILSVI) scheme is proposed. The individual level refers to the indexing of a sports video on a player basis, i.e. to recognize each player in a multi-player game. Since the jersey number is always "worn'' by a player as the player's identity in a game, it is feasible to recognize jersey numbers for individual level indexing in sports videos. To solve the jersey number recognition problem, a principal-axis based contour descriptor is proposed. Compared to the state-of-the-art approaches, the proposed descriptor can achieve higher recognition rate and only consume much less computation power. In addition, we developed an interactive system to realize the individual level sports video indexing (ILSVI). This interactive system includes a player detection and a jersey number detection sub-systems. The interactive system can help complete the individual level sports video indexing task. We shall use basketball game videos as the basis to develop real-world systems.
Unsupervised Semantic Feature Discovery for Image Object Retrieval and Tag Refinement

We have witnessed the exponential growth of images and videos with the prevalence of capture devices and the ease of social services such as Flickr and Facebook. Meanwhile, enormous media collections are along with rich contextual cues such as tags, geo-locations, descriptions, and time. To obtain desired images, users usually issue a query to a search engine using either an image or keywords. Therefore, the existing solutions for image retrieval rely on either the image contents (e.g., low-level features) or the surrounding texts (e.g., descriptions, tags) only. Those solutions usually suffer from low recall rates because small changes in lighting conditions, viewpoints, occlusions, or (missing) noisy tags can degrade the performance significantly. In this work, we tackle the problem by leveraging both the image contents and associated textual information in the social media to approximate the semantic representations for the two modalities. We propose a general framework to augment each image with relevant semantic (visual and textual) features by using graphs among images. The framework automatically discovers relevant semantic features by propagation and selection in textual and visual image graphs in an unsupervised manner. We investigate the effectiveness of the framework when using different optimization methods for maximizing efficiency. The proposed framework can be directly applied to various applications, such as keyword-based image search, image object retrieval, and tag refinement. Experimental results confirm that the proposed framework effectively improves the performance of these emerging image retrieval applications.
Workshop I: International workshop on Learning Semantics for Multimedia Big Data (LSMBD)

Recent advance and development of multimedia technologies have enlightened the creation of intuitive and ubiquitous human-computer communication environment to make our life easier. Such progress of emerging multimedia technologies would result in revolutionary changes of our daily practices, such as shopping, traveling, and social networking. Recently the increasingly massive and complex multimedia data bring new challenges for learning semantics: as data volumes grow to the extent, the traditional solution will not be able to handle effectively. The workshop on Learning Semantics for Multimedia Big Data, LSMBD 2015, aims to bring together researchers and professionals from worldwide academia and industry for showcasing, discussing, and reviewing the whole spectrum of technological opportunities, challenges, solutions, and emerging applications in learning semantics for big data.
Machine learning-based behavior recognition system for a basketball player using multiple Kinect cameras

In this paper, a real-time behavior recognition demo system is proposed. By utilizing the captured skeletons and depth information from multiple Kinect cameras mounted at different locations with different view points, the occluded parts of a player and the ball information in the depth channels can be compensated by another Kinect camera without occlusion situations. Besides, a machine learning process trained from the the skeletons and depth channel information from two Kinect cameras makes the the behavior recognition rate to be more than 80% in real-time usage from three of the trained behaviors, i.e. right-hand dribble, left-hand dribble, and shooting behaviors.