Towards a quantitative evaluation of multi-imaging systems

Nowadays many cameras embed multi-imaging (MI) technology without always giving the option to the user to explicitly activate or deactivate it. MI means that they capture multiple images, combine them and give a single final image, letting sometimes this procedure being completely transparent to the user. One of the reasons why this technology has become very popular is that natural scenes may have a dynamic range that is larger than the dynamic range of a camera sensor. So as to produce an image without underor over-exposed areas, several input images are captured and later merged into a single high dynamic range (HDR) result. There is an obvious need for evaluating this new technology. In order to do so, we will present laboratory setups conceived so as to exhibit the characteristics and artifacts that are peculiar to MI, and will propose metrics so as to progress toward an objective quantitative evaluation of those systems. On the first part of this paper we will focus on HDR and more precisely on contrast, texture and color aspects. Secondly, we will focus on artifacts that are directly related to moving objects or moving camera during a multi-exposure acquisition. We will propose an approach to measure ghosting artifacts without accessing individual source images as input, as most of the MI devices most often do not provide them. Thirdly, we will expose an open question arising from MI technology about how the different smartphone makers define the exposure time of the single reconstructed image and will describe our work around a timemeasurement solution. The last part of our study concerns the analysis of the degree of correlation between the objective results computed using the proposed laboratory setup and subjective results on real natural scenes captured using HDR ON and OFF modes of a given device. Introduction Multi-image (MI) computational photography applications have received lots of attention in recent years, mostly driven by the smartphone market. MI technology involves combining multiple shots of a scene into a single image. These shots can be taken simultaneously (multi-sensors, multi-cameras) or sequentially. There exist many applications of MI: spatial or temporal noise reduction [11], high dynamic range (HDR) [17, 9, 10, 16], motion blur reduction [12, 13], super-resolution, focus stacking, depth of field manipulation, high frame rate, among others. The creation of a single image from a sequence of images raises several problems. These problems and artifacts are mostly due to motion in the scene or camera. For a review of methods for dealing with these issues we refer to [5, 8] and references therein. A different type of artifacts concern the tone mapping of HDR scenes. Since the images must be displayed on screens with limited range, the choice of tone mapping becomes a critical part of the MI system [16]. This process is more qualitative (a) (b) Figure 1. Picture (a) shows the proposed setup for measuring the capacity of MI devices to capture HDR scenes. The target contains a color chart, a texture chart, and a grayscale of 80 uniform patches linearly distributed between 0 and 100% of transmittance. The luminance of the left light source is always 170 cd/m2. The luminance of the right light source varies between 170 and 17000 cd/m2. In (b) a real scene is displayed in order to correlate the measurement with a perceptual analysis. in nature as it aims at tricking the observer into thinking that the image shown on a low dynamic range medium has actually a high dynamic range [2, 14]. This article is concerned with the artifacts resulting from the MI technology used in modern photographic devices and proposes objective metrics for evaluating some of those artifacts. Currently, evaluating the quality of MI algorithms is very important as this technology is more and more present in different devices. Apart of being present, this technology is often transparent to the user. Nowadays, the MI capture is mostly judged through subjective evaluations [15, 3]. Other authors or assume access to the complete stack of input images which is often hand annotated [4]. But there is an obvious need of end-to-end objective evaluations that do not require access to the intermediate images. On the other hand, the traditional lab setup (tripod, flat shooting targets) is particularly favorable to MI systems, more favorable than real life, so that traditional evaluation methods tend to overestimate the real-world quality of MI systems. The goal of this paper is to propose lab setups and objective metrics so as to simplify the evaluation process of MI technology. Ghosting. The most common artifact related to MI acquisition is ghosting [8, 6]. When combining images acquired at different instants, because of motion, there is no warranty of observing the exact same image. When the underlying MI algorithm fails at detecting or compensating this motion, “ghosts” appear in the combined image as result of combining incoherent frames (see Figure 2). (a) (b) (c) Figure 2. Reproduction of ghosting artifacts using a synthetic video of a moving pattern. (a) shows a real ghosting artifact. (b) is an image taken with the iPhone 6+ with HDR OFF, while (c) is taken with HDR ON. We can observe the same kind of ghosting artifacts. As we do not know what happens when HDR is off (we are not necessarily looking at a "single-source" image, Maybe the device did some other (non-HDR) multi-image fusion without telling us), it makes sense to test this mode. Contrast and texture loss. Very present defects are halos and contrast and texture loss resulting from the tone mapping of HDR images (see Figure 4, 3, 5). We observe that local contrast perception, along with a spatial/intensity coherence in the image is an important factor to account for a good HDR image. Tone mapping algorithms allow to display an HDR image on a support with limited dynamic range. The tone mapping algorithm must produce a pleasant rendering of the image while preserving low contrast details [1]. This is usually done by local contrast adaptations, which are inspired on perceptual principles [2] (i.e. humans do not perceive absolute intensities but rather local contrast changes). Color appearance. Color is a very sensitive matter when dealing with HDR scenes. The human visual system (HVS) adapts differently in front of a real HDR scene and in front of a typical LDR display. Even with an accurate HDR image of a scene, the tone mapped and displayed image may appear different from the real scene due to a different adaptation of the HVS [16]. Understanding the HVS and the models created to predict how colors are be perceived under such different lighting conditions [18] will have a direct impact on color fidelity. Moreover, color treatment in HDR tone mapped images can be used to recover contrast and texture [19], which influence directly the color fidelity. This implies that color appearance is a key factor to a good final image. Noise. Noise adds a new dimension of complexity to the MI problem. When stitching multiple images, the risk is that the produced image may have incoherent noise (see Figure 6). As noise is perceived as part of the image texture, its incoherence results in an apparent boundary. Although the analysis below is not centered on noise coherence, we still observe its impact in the proposed metrics. Evaluation of multi-image systems. There is an obvious need to evaluate these new technologies. Several papers in the literature address the evaluation of MI artifacts, they usually focus on HDR deghosting but their observations can be applied to most of the MI applications. These papers can be roughly divided into two categories. In the first category we find perceptual evaluations carried on by trained observers [3] instructed to look for precise artifacts (a) (b) (c) (d) Figure 3. An important aspect of HDR rendering is texture preservation.The picture (b) is less textured in comparison with (a). The texture target of the HDR setup can highlight the preservation of texture. The picture (d) is less textured than (c), even if it has higher contrast.