# Use of Gradient-Based Shadow Detection for Estimating Environmental Illumination Distribution
## 1 Introduction  
Understanding the environmental illumination information of a scene is important when rendering virtual objects in a way that matches the given image; it particularly helps during the generation of convincing vertical shadows onto the real scene [1; 2]. Without shadows, virtual objects will appear like they are floating, making the rendered image appear unrealistic. The goal of our study is to estimate the environmental illumination distribution and to improve the visual perception of an augmented reality (AR) system by generating virtual shadows. Many studies have described the generation of realistic images that reflect the environmental illumination of their scene [3; 4; 5; 6; 7]. Some of these studies have employed additional camera equipment (e.g., a light probe or a fish-eye camera) to estimate real-world illumination conditions [8; 9]. One previous study used mobile sensors (e.g., ambient sensors), a global positioning system (GPS), and a weather application programming interface (API) to estimate outdoor illumination for a mobile AR application [10]. Shadows are cast by the occluding object and the illumination source in a given scene, so they provide an important clue about the shape and the relative position of the object, as well as the environmental illumination. The shadow generated by a reference object with known geometry can therefore be examined to estimate the environment illumination [1; 3; 4]. However, the performance of point-based estimation methods is heavily dependent on the number of sampling points, and their positions of those points on the shadow's surface. In addition, it is difficult to determine accurate threshold values with which to segment the shadow region. One study introduced an iterative votingscheme that compared inside-region (potential cast shadows region) pixels with outside-region pixels using shadow features [6]. In this method, the inside-region with respect to the three-dimensional (3-D) object was defined in an initial setup procedure, meaning that the dynamic situation in which the position of the illumination sources were changed was not considered. For example, when there is a wide distribution of illumination sources in the scene, the inside-region become too large. The illumination estimation performance is therefore significantly dependent on which point is sampled. However, the study did not present a detailed method for determining the sample points for iterative voting [6]. A further study introduced an illumination estimation method for a mobile AR system, under the assumption that there is a single dominant light source in the scene [3]. By jumping the shadow boundary trace along the current estimate of the shadow vector, this method was able to extract distinct contours from the shadowed region. As there are usually multiple light sources in real scenes, however, the shadow boundaries often have a complex distribution. In this case, the method adopted featured difficulties in tracing the exact shadow boundaries, even though several salient shadow edges were present in the scene [3]. Previous studies have extracted the shadow region and its boundary on a textured shadow surface in a static outdoor scene [5; 11; 12]. Also, the shadow detection is critical for accurate object detection in video streams since shadow points are often misclassified as object points, causing errors in segmentation and tracking [13; 14; 15]. However, these studies mainly examined shadow edges with distinct gradient distributions, because outdoor environments feature a main light source (sunlight). These methods were applied to a situation in which the sun was occluded, but they assumed that most of the shadow boundaries would still have large gradient magnitude values and distinct gradient directions. In general, it is difficult to detect precisely the shadow boundary by only using the image features. One study constructed 3-D shape descriptions of buildings from the fragmented linear features in an aerial image [16].