# A Corner-Highlighting Method for Ambient Occlusion
## 3 Our Algorithm
### Outline-Dependent Masking  
We assume ambient occlusion happens in places where surfaces, or groups of faces, are incoherently arranged. This means we also assume occlusion not to occur where surfaces are consistent and in opened surroundings. With this analogy, we aimed for an algorithm that can highlight surfaces where adjacent surfaces are close enough to occlude illumination. As a result, we brought up a simple approach to search this surface incoherency which is outlining. We define the outline as a replica of the original surface where every vertex is extruded by its integrated normal vectors with an arbitrary extent. Figure 1 shows two-dimensional examples of the outline. Closed polygons are the model objects, or what we refer to a group of faces. The edges of those model objects are the original faces to be rendered, and the dotted lines around them are their outlines. Three-dimensional representation would be polyhedron with faces wrapping around. How to generate these outlines is explained further in Section 3.2. We can easily spot the mismatching surfaces by rendering these prepared outlines. The mismatching surface, or surface incoherency, is the case where nearby surfaces have distant surface normal vector orientations. When a view ray enters more than two outlines without exiting, we consider the hit point of the ray as an outline intersection. This intersection check specifies mismatching surfaces in ease, which is what our algorithm highlights as possibly occluded areas. Figure 1 demonstrates these outline intersections. Dotted lines represent outlines and highlighted areas represent their intersections. View rays hitting the  
Figure 1: Two-dimensional representation of model objects, outlines (in dotted lines), and their intersections (in hatched lines). surface inside this highlighted region are likely to be occluded. As we can see in this figure, concave corners are what our algorithm is locating before computing ambient occlusion. The actual implementation of the corner highlighting takes a few steps. We first render the base model object information to the screen-space \(G\)-buffer. Rendered buffers hold the world-space position, the normal vector, and the depth value of the pixel. Generating this \(G\)-buffer is a necessary step in deferred rendering; it provides essential data for the illumination in later processing. Rendering outlines come next to check the intersection aforementioned. We detect the view ray entering and exiting outlines by rendering them. We consider rendering outline faces equivalent to the view ray crossing the outlines. View ray stops when it confronts the model object surface, thus crossing the outline faces behind the model objects are ignored. Accordingly, depth testing with previously obtained depth buffer is mandatory. In this step, writing to the depth buffer is disabled to avoid depth testing with outline faces. Outline intersection can be measured by counting the outline faces rendered on a pixel. Add 1 when the view ray enters the front face of the outline. Subtract 1 when enters back face vice versa. When this summation ends above 2, it implies that the pixel hit by the view ray is in the intersection region, and thus corresponding pixel can be marked as a possibly occluded area. Figure 2 represents how corner checking is performed. We have the same scene setup with two different view rays, left and right. Filled rectangles represent original model objects and the dotted edges wrapping around are their outlines, and small arrows showing their normal vector directions. The \(+\) and \(-\) symbols on the figure show where view ray enters and exits the outline the model objects. Left view ray sums up to 1, showing its contact point is not inside the corner. Right view ray sums up to 2 on the other hand, meaning its point is inside the corner and is likely to be occluded. With this simple approach, we generate the stencil layers for ambient occlusion. Figure 3 shows the actual implementation results of our algorithm.