# A Novel Spatiotemporal Analysis Framework for Air Pollution Episode Association in Puli, Taiwan
## 2 Literature Review
### Video Analysis  
Video is a time series of image frames, and it is by nature a spatiotemporal stream of data. The existing techniques for video processing have many analogies with the methods for air-pollution spatiotemporal analysis. In particular, the shot boundary detection (SBD) techniques, the gait energy image (GEI) representation scheme, the image retrieval algorithms, and the convolutional neural networks (CNN) are most relevant to our proposed methods, and they are reviewed as follows:  
(1) Shot boundary detection (SBD). Video is a stream of image frames. To deliver situated semantics, a video is composed of a sequence of scenes, and each scene may have one or more shots. Each shot contains contiguous image frames that are uninterruptedly shot by a camera to represent continuous actions in time and space. Video SBD is the temporal segmentation of a video into consecutive shots that are primitive elements for video indexing and retrieval [18; 19; 20]. The video shot boundary appears in the frame(s) transiting from one shot to another shot. The transition between two consecutive shots usually appears in two types. The abrupt transition (hard cut) is a significant and sudden change from one shot to the next. The soft transition (soft cut) is a gradual change between the adjacent shots by applying spatial, chromatic, or spatial-chromatic effects, resulting in several stylish transitions such as dissolve, wipe, and fade in/out. Clearly, soft-cut detection is more challenging than hard-cut detection. Most existing SBD techniques proceed as follows: A score function is defined for estimating the probability of a frame being the true cut separating two consecutive shots. Then an appropriate threshold is used to classify the frames into cuts and non-cuts based on their scores. Several score functions were developed [19; 20]. (1) Sum of absolute differences (SAD). The score function sums up the absolute difference between corresponding pixels at the same location in the two consecutive image frames. This is the earliest SBD approach, which is simple but sensitive to object movement and may produce many false cuts. (2) Histogram differences (HD). Instead of computing the difference at the pixel level, the difference between the histograms of consecutive frames for each color channel is calculated. The HD is less sensitive to object movement. However, it loses the spatial information, i.e., the histogram of two completely different images can be exactly the same. (3) Edge change ratio (ECR). Since the cut appears at the frame(s) where the old edge pixels disappear and the new edge pixels emerge, the cut score can be estimated by calculating the difference at the edge locations. (4) Transform correlation (TC). The image frames are transformed into the frequency domain, and the correlation between the transformed images is evaluated. The cut frame is identified at the local minimum of the correlation. (5) Hybrid score functions(HSF) have been proposed by combing two or more of the above-mentioned approaches to enhance the robustness of the score function for different video contexts. (2) Gait Energy Image (GEI). Human individual recognition in a natural scene is a very challenging problem because the image/video capturing the individual may just capture some parts of the individual (e.g., face, shoulder, waist, and legs) or from any viewing angle (front, side, bird's-eye, and back). Therefore, various biometrics (face, fingerprint, and palm print) and context (posture, gait, and silhouette) were investigated to enhance the accuracy of human individual recognition. One of the novel context-based techniques for human individual recognition is the GEI technique proposed by [PERSON] and [PERSON] [21]. GEI is a spatiotemporal representation to analyze human walking properties for individual recognition by gait. GEI represents human gait motion in a single image while preserving spatiotemporal information. The sequence of binary silhouettes of humans walking in the video was extracted from the original video.