nils-herrmann/clean_scirepeval huggingface dataset:
Programmable spatially variant multiresolution readout capability on a sensor focal plane

This paper presents a new image sensor with programmable multiresolution readout capability. The proposed image sensor can output data at varying resolutions. Differing to our previously reported spatially variant sampling sensor, which sub-samples without filtering, in the multiresolution sensor blocks of the pixels are averaged and read out so the new sensor does not suffer from aliasing effects. The resolution can be controlled by varying the block size. We made a prototype of 64/spl times/64 pixels.
Summarizing wearable video

"We want to record our entire life by video" is the motivation of this research. Developing wearable devices and huge storage devices will make it possible to keep entire life by video. We could capture 70 years of our life, however, the problem is how to handle such a huge amount of data. Automatic summarization based on personal interest should be required. In this paper we propose an approach to the automatic structuring and summarization of wearable video. (Wearable video is our abbreviation of "video captured by a wearable camera".) In our approach, we make use of a wearable camera and a sensor of brain waves. The video is firstly structured by objective features of video, and the shots are rated by subjective measures based on brain waves. The approach is very successful for real world experiments and it automatically extracted all the events that the subjects reported they had felt interesting.
Enhancement of Depth Maps With Alpha Channel Estimation for 3-D Video

Depth images are widely used for 3-D scene generation. In depth image acquisition, accurate estimation of the depths of object boundaries, which has a critical impact on the visual quality of the generated 3-D scene, is very difficult, especially in the case of objects with a hairy region. We aimed to generate a dynamic 3-D scene without serious degradation in visual quality by developing solutions for the problems that occur in depth images obtained using an active depth sensor. A novel alpha channel estimation algorithm is proposed for seamless composition along with a depth map improvement method for hairy objects. By utilizing additional depth or infrared (IR) information, the existing matting algorithm can be improved significantly. We further enhanced the alpha estimation method in the temporal domain. The depth map was enhanced by filtering depth values along spatiotemporal neighborhoods based on information provided by the color and alpha images. The proposed method was examined mainly using a time-of-flight (TOF) camera, and Kinect is used too. The experimental results demonstrated that the proposed method can generate a 3-D scene with a greater degree of naturalness as compared to other methods.
When disparity meets distance: HEVC compression of double-faced depth map

A depth map is inherently double-faced, one single depth map can provide two closely related 3D-vision parameters (i.e. the disparitys and the distance z). Most existing depth map compression techniques considered only one of them at a time, rather than addressing the effect of compression for the two parameters simultaneously. In order to remedy this shortage, a new distortion function is proposed and integrated into HEVC 3D compression framework to examine the effectiveness with respect to s and z at the same time. Experimental results show that, with the aid of the proposed distortion function, the rate-distortion performance of the distance quality will be significantly enhanced (from 0.5 dB to 8 dB), while keeping the corresponding quality of disparity almost the same. In addition to the coding performance improvements, we also discovered some interesting phenomenons that never occurred in traditional compression framework. We expect this work can inspire more interesting research works on depth map compression.
Software based object tracking with visual feature integration

Multiple visual features can be used to track objects in environments where the use of a single visual feature is not sufficient. We have proposed an alternative method to feature integration by way of feature substitution where the dominant feature at any instant is used for tracking. The problem of increased computational cost resulting from the use of multiple features is addressed by using software optimization techniques. Here we use software threads and special instructions provided in MMX processors to increase the frame rate. Experimental results indicate that it is not always necessary to adopt a hardware based approach for real time object tracking even when the computational cost is high.
Generation of a disparity panorama using a 3-camera capturing system

This paper presents a method to create a panorama disparity map of a natural environment from stereo images captured by our original multi-purpose 3-camera panorama capturing system which features accurate frame synchronization between 3 channels and can be used outdoors through battery operation. For robust determination of correspondences of a stereo pair, we make use of a census transform, a kind of non-parametric local transform. The census transform summarizes local image structure as a bit string, and gives very good stereo-matching results without such processes as image pyramids, bi-directional search, complicated matching evaluation, and too much edge detection dependent interpolation. To interpolate unknown disparities, we introduced a process influenced by the K-means algorithm. Then, use of 3-channel images and overlapping of the maps give multiple disparity values for one pixel, which enables screening of very confident values. We have obtained very good results on such complex scenes as contain branches and leaves of trees.
Image-based Calorie Content Estimation for Dietary Assessment

In this paper, we present an image-analysis based approach to calorie content estimation for dietary assessment. We make use of daily food images captured and stored by multiple users in a public Web service called Food Log. The images are taken without any control or markers. We build a dictionary dataset of 6512 images contained in Food Log the calorie content of which have been estimated by experts in nutrition. An image is compared to the ground truth data from the point of views of multiple image features such as color histograms, color correlograms and SURF fetures, and the ground truth images are ranked by similarities. Finally, calorie content of the input food image is computed by linear estimation using the top n ranked calories in multiple features. The distribution of the estimation shows that 79% of the estimations are correct within ±40% error and 35% correct within ±20% error.
Virtual view generation by linear processing of two differently focused images

Presents a method to render shifting (parallax) and focusing effects on object in a scene for generating a virtual view image with arbitrary focus. Two differently focused images of a same scene-near-focused image and far-focused image-are used as input images under the assumption that a scene has near and far objects. The proposed method can freely handle the shifting and the focusing effect on each object according to the view point and the focus depth of the virtual camera only by linear filtering of the input images without any segmentation or 3D modeling of the objects. Experimental results using real images are shown to test the performance of the method.
cGAN-Based Manga Colorization Using a Single Training Image

The Japanese comic format known as Manga is popular all over the world. It is traditionally produced in black and white, and colorization is time consuming and costly. Automatic colorization methods generally rely on greyscale values, which are not present in manga. Furthermore, due to copyright protection, colorized manga available for training is scarce. We propose a manga colorization method based on conditional Generative Adversarial Networks (cGAN). Unlike previous cGAN approaches that use many hundreds or thousands of training images, our method requires only a single colorized reference image for training, avoiding the need of a large dataset. Colorizing manga using cGANs can produce blurry results with artifacts, and the resolution is limited. We therefore also propose a method of segmentation and color-correction to mitigate these issues. The final results are sharp, clear, and in high resolution, and stay true to the character's original color scheme.
A novel image sensor for video compression

A novel image sensor on which video signals can be compressed is proposed. Since the video signal is compressed on an imager plane by using fast analog processing, the amount of image data read out from the imager can be significantly reduced with very small latency. The proposed system can be potentially applied to high pixel rate cameras such as those for high speed imaging and high resolution imaging. Conditional replenishment is employed for the video compression algorithm. Analog circuits are designed both for processing in each pixel and for controlling the entire data rate. The behavior of the circuit is investigated on the basis of both an analog circuit simulator and a scale-up-circuit. A VLSI chip is designed and is under fabrication.<<ETX>>
Real-time image-based rendering system with random access CMOS imagers

We investigate a real-time image-based rendering (IBR) system. IBR enables one to synthesize a 3D arbitrary view from a 2D array of input images. However, conventional systems have a difficulty to handle dynamic scenes because of the heaviness of the input. We design an IBR system with random access image sensors. In the system, each image sensor outputs only the pixels that are externally selected, so that the total readout data from the camera array is reduced to the necessary and sufficient condition for the rendering. We develop a prototype of random access image sensor by the CMOS process. A prototype of IBR system with 16 of this sensors was constructed and synthesized arbitrary-view at 60 frames/second.
Object detection refinement using Markov random field based pruning and learning based rescoring

Contextual information such as the co-occurrence of objects and the location of objects has played an important role in object detection. We present candidate pruning and object rescoring methods that leverage contextual information and that can improve the state-of-the-art CNN-based object detection methods such as Fast R-CNN and Faster R-CNN. In our pruning method, we formulate candidate reduction as a Markov random field optimization problem. In our rescoring method, we employ a machine learning technique to reconsider the detection scores of candidate windows. We experimentally demonstrate improvements in R-CNN-based object detection methods using two datasets. Moreover, we apply our model to the structured retrieval task to show the potential applications of our model.
PQTable: Nonexhaustive Fast Search for Product-Quantized Codes Using Hash Tables

In this paper, we propose a product quantization table (PQTable)—a fast search method for product-quantized codes via hash tables. An identifier of each database vector is associated with the slot of a hash table by using its PQ-code as a key. For querying, an input vector is PQ-encoded and hashed, and the items associated with that code are then retrieved. The proposed PQTable produces the same results as a linear PQ scan, and is <inline-formula> <tex-math notation="LaTeX">$10^2$</tex-math></inline-formula>–<inline-formula><tex-math notation="LaTeX">$10^5$ </tex-math></inline-formula> times faster. Although the state-of-the-art performance can be achieved by previous inverted-indexing-based approaches, such methods require manually designed parameter setting and significant training; our PQTable is free of these limitations, and therefore offers a practical and effective solution for real-world problems. Specifically, when the vectors are highly compressed, our PQTable achieves one of the fastest search performances on a single CPU to date with significantly efficient memory usage (0.059-ms per query over <inline-formula><tex-math notation="LaTeX">$10^9$</tex-math></inline-formula> data points with just 5.5-GB memory consumption). Finally, we show that our proposed PQTable can naturally handle the codes of an optimized product quantization (OPQTable).
Large-Scale Geosocial Multimedia [Guest editorial]

With the advance of the Web 2.0 era came an explosive growth of geographical multimedia data shared on social network websites such as Flickr, YouTube, Facebook, and Zooomr. Location-aware media description, modeling, learning, and recommendation in pervasive social media analytics have become a key focus of the recent research in computer vision, multimedia, and signal processing societies. A new breed of multimedia applications that incorporates image/video annotation, visual search, content mining and recommendation, and so on may revolutionize the field. Combined with the popularity of location-aware social multimedia, location context data makes traditionally challenging problems more tractable. This special issue brings together active researchers to share recent progress in this exciting area. This issue highlights the latest developments in large-scale multiple evidence-based learning for geosocial multimedia computing and identifies several key challenges and potential innovations.
Computational image sensors for on-sensor-compression

In this paper, we propose novel image sensors which compress image signal. By making use of very fast analog processing on the imager plane, the compression sensor can significantly reduce the amount of pixel data output from the sensor. The proposed sensor is intended to overcome the communication bottle neck for high pixel rate imaging such as high frame rate imaging and high resolution imaging. The compression sensor consists of three parts; transducer, memory and processor. Two architectures for on-sensor-compression are discussed in this paper that are pixel parallel architecture and column parallel architecture. In the former architecture, the three parts are put together in each pixel, and processing is pixel parallel. In the latter architecture, transducer, processor and memory areas are separated, and processing is column parallel. We also describe a prototype chip of pixel-parallel-type sensor with 32/spl times/32 pixels which has been fabricated using 2 /spl mu/m CMOS technology. Some results of examinations are shown in this paper.
New design and implementation of adaptive-integration-time image sensor

We describe a novel approach to enhance the performance of image sensing by integrating the processing element with an image sensor. We have been investigating a smart sensor which controls the integration time of every pixel independently. The integration time is controlled so that it has higher temporal resolution and wider dynamic range. We describe a new adaptive-integration-time image sensor which has 128/spl times/64 pixels. We have designed the VLSI prototype by using a column parallel architecture. In this new chip, the scheme to control the integration time is extended, and the pixel pitch processing speed and power consumption are much improved in comparison with our previous prototype. A 7-bit address encoder is newly implemented. We show some experimental results obtained with the prototype.
Photometric Stereo Using Constrained Bivariate Regression for General Isotropic Surfaces

This paper presents a photometric stereo method that is purely pixelwise and handles general isotropic surfaces in a stable manner. Following the recently proposed sum-of-lobes representation of the isotropic reflectance function, we constructed a constrained bivariate regression problem where the regression function is approximated by smooth, bivariate Bernstein polynomials. The unknown normal vector was separated from the unknown reflectance function by considering the inverse representation of the image formation process, and then we could accurately compute the unknown surface normals by solving a simple and efficient quadratic programming problem. Extensive evaluations that showed the state-of-the-art performance using both synthetic and real-world images were performed.
Photometric Stereo Using Sparse Bayesian Regression for General Diffuse Surfaces

Most conventional algorithms for non-Lambertian photometric stereo can be partitioned into two categories. The first category is built upon stable outlier rejection techniques while assuming a dense Lambertian structure for the inliers, and thus performance degrades when general diffuse regions are present. The second utilizes complex reflectance representations and non-linear optimization over pixels to handle non-Lambertian surfaces, but does not explicitly account for shadows or other forms of corrupting outliers. In this paper, we present a purely pixel-wise photometric stereo method that stably and efficiently handles various non-Lambertian effects by assuming that appearances can be decomposed into a sparse, non-diffuse component (e.g., shadows, specularities, etc.) and a diffuse component represented by a monotonic function of the surface normal and lighting dot-product. This function is constructed using a piecewise linear approximation to the inverse diffuse model, leading to closed-form estimates of the surface normals and model parameters in the absence of non-diffuse corruptions. The latter are modeled as latent variables embedded within a hierarchical Bayesian model such that we may accurately compute the unknown surface normals while simultaneously separating diffuse from non-diffuse components. Extensive evaluations are performed that show state-of-the-art performance using both synthetic and real-world images.
Report of the Exhibition: “Digital Public Art in HANEDA AIRPORT “AIR HARBOR” - Technology Meets Air: A Sensation of a New World”

We produced an exhibition “Digital Public Art in HANEDA AIRPORT ‘AIR HARBOR’ - Technology Meets Air: A Sensation of a New World” in October, 2009 at the Haneda Airport which is public space and is the biggest airport in Japan. This exhibition aims to show the new concept “Digital Public Art” which tries to evolve media art to public art. This exhibition is also an experiment to bring information technology into public spaces as an interactive art works. In this exhibition, we combined virtual reality technologies, human interface methods, and signal processing techniques with art for developing public art artworks. From the view point of technology, we have developed display technologies such as projecting to real objects display, sensing technologies for large area such as carbon dioxide (CO2) sensing for people's congestion, interface technologies such as public interface, especially for the public spaces. We have solved problems that occur when we set information technologies in to the public space. In this paper, we propose the “Digital Public Art” and introduce the exhibition “AIR HARBOR” and its technologies that are suitable in the public space.
A novel image-based rendering method by linear filtering of multiple focused images acquired by a camera array

In this paper, we present a novel approach to image-based rendering (IBR) for generating an arbitrary view image with arbitrary focus for a scene consisting two approximately constant depths. The presented method differs from the conventional IBRs using multiple view images in that we acquire two differently focused images from each camera position and render parallax and focus effects on objects at different depth simply by linear filtering of the acquired images without segmentation. Experimental results on the real images acquired with 4 cameras located in parallel are presented.
A teleconferencing system capable of multiple person eye contact (MPEC) using half mirrors and cameras placed at common points of extended lines of gaze

In this paper a novel method of providing eye contact for multiple participants in a teleconferencing system is presented. This method proposes a simple idea to fulfill the requirement for eye contact, for two or more local and remote participants simultaneously, by using half mirrors and cameras placed at the common points of the extended lines of gaze of each participant. In contrast to the commonly available one to one eye contact of video phone systems, which also use half mirrors, the proposed system has the advantage of serving more than one person at a given location, hence named multiple person eye contact (MPEC) teleconferencing system. We observed the resultant images by using a preliminary experiment and also by building a prototype system. The results from both of these experiments proved the merits of the proposed idea. One of the main features of this system is that, it gives any participant, a feeling of "being looked at" if any remote participant is actually gazing at him. On the other hand if none of the remote participants are looking at him, then he gets the feeling of "not being looked at" which is also extremely important for effective inter personal communication. Furthermore, this system preserves the spatial continuity of neighboring participants, since all the participants at a given location are captured by each video camera.
VenueNet: Fine-Grained Venue Discovery by Deep Correlation Learning

Venue photos, as a new type of multimedia contents, are exploding on the Internet because users like to take photos and share with their friends in which venue they spent time and what impressed them there. Discovering a venue by a social photo is very useful for supplementing venue retrieval and recommendation. However, little research focused on fine-grained venue discovery by leveraging multimodal venue dataset. In this paper, we present the first multimodal dataset specially built for venue discovery, which includes venue photos, descriptions, and categories. Using this dataset, we propose a novel framework for fine-grained venue discovery through correlating venue photos and descriptions, aiming to learn a VenueNet representing a knowledge base and association for venues and their properties in different modalities. In the training phase, visual and textual features of the same venues, by two sub-networks, are respectively mapped to a same semantic space, in which canonical correlation analysis (CCA) is applied to these features to train the two sub-networks. In the query phase, given a photo, its correlation with textual features in the dataset is analyzed to find the most similar venue. Experimental results verify the practicability of the Deep CCA model for fine-grained venue discovery from large-scale multimodal dataset.
Networked large-scale image sensing system using spatially-variant sampling smart image sensors

Large-scale image sensing system is considered to provide better coverage than conventional video surveillance systems with small number of image sensors; however, on the other hand, the bandwidth problem is likely to occur because the network resources should be shared by the image sensors at the same time, which makes it difficult to implement a practical system. As one of the feasible solutions, we present the system using a spatially-variant sampling (SVS) smart image sensor for real-time spatial resolution control that is hard to achieve by the conventional CCD image sensors. The regions with moving objects are detected on the sensor nodes and transmitted with different spatial resolution and temporal rate. The simulation results support the effectiveness of this scheme. The experimental system is being developed.