Measuring similarity between geo‐tagged videos using largest common view

Electronics LettersVolume 55, Issue 8 p. 450-452 Image and vision processing and display technologyFree Access Measuring similarity between geo-tagged videos using largest common view Wei Ding, Wei Ding School of Computer, Information, and Communications Engineering, Kunsan National University, Kunsan, Republic of KoreaSearch for more papers by this authorKwangSoo Yang, KwangSoo Yang Department of Computer Science, Florida Atlantic University, Boca Raton, Florida, USASearch for more papers by this authorKwang Woo Nam, Corresponding Author Kwang Woo Nam kwnam@kunsan.ac.kr orcid.org/0000-0001-9970-7863 School of Computer, Information, and Communications Engineering, Kunsan National University, Kunsan, Republic of KoreaSearch for more papers by this author Wei Ding, Wei Ding School of Computer, Information, and Communications Engineering, Kunsan National University, Kunsan, Republic of KoreaSearch for more papers by this authorKwangSoo Yang, KwangSoo Yang Department of Computer Science, Florida Atlantic University, Boca Raton, Florida, USASearch for more papers by this authorKwang Woo Nam, Corresponding Author Kwang Woo Nam kwnam@kunsan.ac.kr orcid.org/0000-0001-9970-7863 School of Computer, Information, and Communications Engineering, Kunsan National University, Kunsan, Republic of KoreaSearch for more papers by this author First published: 01 April 2019 https://doi.org/10.1049/el.2018.7499Citations: 2AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract This Letter presents a novel problem for discovering similar trajectories based on the field of view of the video data. The problem is important for many societal applications such as grouping moving objects, classifying geo-images, and identifying the interesting trajectory patterns. Prior works consider only either spatial locations or spatial relationship between two line-segments. However, these approaches show a limitation to find similar moving objects with common views. In this Letter, the authors propose a new algorithm that can group both spatial locations and points of view to identify similar trajectories. The authors also propose novel methods that reduce the computational cost for the proposed work. Experimental results using real-world datasets demonstrate that the proposed approach outperforms prior work and reduces the computational cost. Introduction Recent advances in mobile technologies such as GPS, accelerometers, gyro sensors, and dash cams have significantly increased the quantity of GeoVideo generated by mobile users. GeoVideo contains a variety of spatial properties, such as the latitude and longitude at which the video was taken, the direction of the view in which the video was taken, and the angle of the camera. These spatial properties enable us to identify new interesting trajectory patterns that can utilise geometric and topological locations, regions of interests, and direction of user's views. Consider the example in Fig. 1. Three people follow the same direction toward a tourist attraction and capture videos. Which trajectories are similar? Prior work, such as Hausdorff or longest common subsequence (LCSS) approaches [1, 2], consider only either spatial locations or spatial relationship between two line-segments and show that GeoVideo2 and GeoVideo3 are the most similar. Is it even semantically true? Even though GeoVideo2 and GeoVideo3 have a similar trajectory, the field of views (FoVs) of the two cameras are different, which results in the semantically incorrect output. Therefore, in terms of semantic similarity, GeoVideo1 and Geovideo3 should be the most similar due to similar FoVs. To satisfy this requirement, we propose a novel approach, namely largest common view subsequence (LCVS), which measures the similarity between trajectories based on the FoV. Fig 1Open in figure viewerPowerPoint Example of the geo-tagged videos with similar trajectories, but different view directions. Geovideos 1 and 2 have similar trajectory distance but in terms of view of the video, geovideos 1 and 3 are more similar The proposed approach is important for many societal applications such as grouping moving objects, classifying geo-images, and identifying the interesting trajectory patterns. For example, suppose you have a video of many parents and children groups moving hand in hand at a mall. Prior work, such as Hausdorff and LCSS, may produce similar trajectories but show a limitation to distinguish groups based on common interests. Since our proposed approach considers FoV in the similarity measurement, it can clearly identify a similar group of trajectories based on common interests (e.g. viewpoints). Background Prior work on identifying the similar trajectories can be categorised into two groups: (i) distanced-based approach and (ii) common subsequence-based approach. Distance-based approaches use distance metrics (e.g. Euclidean or Hausdorff distance) to measure the similarity between two trajectories [1]. However, such approaches do not guarantee the accuracy due to outliers or different speeds in trajectories. Common subsequence-based approaches use the LCSS to identify the similar trajectories [2]. However, these approaches have not been designed to account for the field of the view or the angle of the camera. By contrast, this Letter proposes a novel approach for identifying semantically interesting group of trajectories that considers both spatial locations and the direction of the viewpoints. Our proposed approach utilises the concept of LCSS problem [3]. The LCSS approach uses the distance threshold and the time threshold between two spatial points to identify a common subsequence [3]. The distance threshold is used to find the common locations. If the distance between two spatial locations is within the distance threshold, then two spatial locations are the same. The main issue of the LCSS method is that it may produce semantically incorrect results because spatial locations within the distance threshold may have different or opposite viewpoints. To resolve this issue, our approach uses the weighted distance measurement based on the common views. Largest common view subsequence GeoVideo consists of a series of video frames, associated with spatial properties. Fig. 2 shows an example of GeoVideo. Let i be the timestamp, let pi be the location of the camera, let ri be the maximum visible distance from the camera, let θi be the angle from the north to the direction of the camera, and let δi be the maximum horizontal angle of the camera lens. In this example, the video frame at timestamp i is associated with a set of spatial properties (i.e. pi, ri, θi, and δi). Let fovi be a set of spatial properties (i.e. fovi = (pi, ri, θi, δi)) and let View(fovi) be the region of fovi. Given two FoV regions (e.g. View(fovi) and View(fovj)), the weight of the common view (CVW) between View(fovi) and View(fovj) is defined as follows: (1)where is the union of two FoV regions and is the intersection of two FoV regions. Fig 2Open in figure viewerPowerPoint Example of FoV model in GeoVideo Assume that GeoVideo A consists of m number of FoVs (i.e. A = {fov1, fov2, …, fovm}) and GeoVideo B consists of n number of FoVs (i.e. B = {fov1, fov2, …, fovn}). Let Head(A) be a sequence of FoVs of A from 1 to m − 1. (i.e. Head(A) = {fov1, …, fovm−1}) and let σ be the minimum time threshold. The maximum common view-based similarity distance is defined as (2) LCVS similarity and distance function Given a minimum time threshold σ, the LCVS similarity between A and B is defined as (3)The LCVS distance between A and B is defined as (4)The LCVS distance is metric because it satisfies the following three properties: (i) for all (non-negativity) (ii) for all (symmetry) (iii) for all (triangle inequality) Algebraic cost model of LCVS LCVS enumerate all subsequences of GeoVideo A, which takes O (2m). For each subsequence, LCVS scans n number of FoVs in GeoVideo B to compute . This takes O(n). Therefore, the cost model of LCVS is O(n·2m). Implementation and experimental result The main performance bottleneck of the proposed approach is to compute the FoV region, the union of two FoV regions, and the intersection of two FoV regions. To improve the efficiency of LCVS, we employ three novel methods to reduce the computational cost of computing CVW. Our first method uses the minimum bounding segment (MBS) to approximate the FoV region (see Fig. 3a). This method partitions FoV using the same-sized triangles and sum up the area of these triangles to estimate the FoV region, which takes linear time in terms of the number of triangles. Our second method uses the minimum bounding triangle (MBT) to speed up the computation of the FoV region (see Fig. 3b). Our third method uses the minimum bounding rectangle (MBR) to roughly estimate the FoV region (see Fig. 3c), which can significantly reduce the computation cost for CVW in a special case (e.g. car dash cams). Fig 3Open in figure viewerPowerPoint Simplification of an FoV region using MBS, MBT, MBR a MBS, FoV is approximated by a set of segments (or triangles) b MBT, FoV is approximated by a triangle c MBR, FoV is approiximated by an rectangle We conducted experiments to evaluate the performance of the LCVS algorithm. We tested four different approaches: (i) Original LCSS, (ii) LCVS with MBS, (iii) LCVS with MBT, and (iv) LCVS with MBR. The overall goal was to show the performance improvements of our new three methods to identify similar FoV trajectory patterns based on geo-tagged video datasets. We wanted to answer two questions: (i) What is the effect of the number of FoVs? (ii) What is the effect of viewable distances? Performance measurements were the accuracy of outputs and execution time. We used geo-tagged 4000 real-world driving videos in New York City, taken from BDD100K [4]. We generated two different datasets: (i) straight-ahead direction FoV and (ii) random direction FoV. In straight-ahead direction FoV dataset, the direction of the camera is aligned with the direction of moving object, which can be obtained from dashcam recordings. In random direction FoV dataset, the direction of the camera is randomly varying without an alignment with the direction of moving object, which can be obtained from mobile device camera recordings (e.g. smartphone). In our experimental setup, we fixed the minimum time threshold σ to 1 and the angle of a segment (i.e. the acute angle of a triangle) in MBS to 5. The first experiment evaluated the effect of the number of FoVs. The number of FoVs were varied from 1000 to 4000. Figs. 4a and b give the accuracy of the outputs. As can be seen, our approaches outperform the original LCSS. The performance gap increases as the number of FoVs increases. This is because the original LCSS considers only point distance to measure the similarity between two trajectories. Fig. 4b shows that LCVS with MBS and LCVS with MBT outperform LCVS with MBR. This is because MBR can roughly estimate the FoV region without considering the accuracy of the outputs. Fig 4Open in figure viewerPowerPoint Effect of number of FoVs a Straight-ahead direction FoV dataset b Randon direction FoV dataset The second experiment evaluated the effect of viewable distances. We varied the distance interval from 10 to 60 m. Figs. 5a and b give the accuracy of the outputs. As can be observed, both LCVS with MBS and LCVS with MBT outperform other approaches. Fig. 5b shows that LCVS with MBR performs poorly because the error field is increased with viewable distance in random direction dataset. Fig 5Open in figure viewerPowerPoint Effect of viewable distances a Straight-ahead direction FoV dataset b Randon direction FoV dataset The third experiment evaluated the runtime of the LCVS algorithm. We varied the number of FoVs from 1000 to 4000 (see Figs. 6a and b). The results show that the original LCSS performs slightly better than LCVS with MBT and LCVS with MBR. This is because our approaches compute CVW to measure the similarity between the two trajectories. However, given that the accuracy of the original LCSS is much lower than our approaches, choosing LCVS to identify similar FoV trajectory patterns is a better choice. We can see that LCVS with MBS is slower than other approaches because MBS takes a linear time to compute CVW. Fig 6Open in figure viewerPowerPoint Effect of the number of FoV a Straight-ahead direction FoV dataset b Randon direction FoV dataset Conclusion We described a novel approach to group both spatial locations and points of view to identify similar trajectories. We proposed three new methods to improve the performance of the proposed work. Experimental results using real world dataset demonstrated that our proposed approach outperforms related work and reduces the computational cost. Acknowledgments This work was supported by the National Research Foundation of Korea grant (no. 2018R1A2B6007982), and the National Land Space Information Research Program grant (no. 14NSIP-B080144-01) of the Korean government. References 1Gastaldo, P., Zunino, R.: 'Hausdorff distance for robust and adaptive template selection in visual target detection', Electron. Lett., 2002, 38, (25), pp. 1651– 1653 (https://doi/org/10.1049/el:20021179) 2Vlachos, M., Gunopulos, D., Kollios, G.: ' Discovering similar multidimensional trajectories'. Int. Conf. of Data Engineering, San Jose, February 2002, pp. 673– 684 3Atev, S., Miller, G., Papanikolopoulos, N.P.: 'Clustering of vehicle trajectories', Trans. Intell. Transp. Syst., 2010, 11, (3), pp. 647– 657 (https://doi/org/10.1109/TITS.2010.2048101) 4Yu, F., Xian, W., Chen, Y. et. al.,: ' BDD100K: a diverse driving video database with scalable annotation tooling', arXiv preprint, 2018, arXiv:1805.04687 Citing Literature Volume55, Issue8April 2019Pages 450-452 FiguresReferencesRelatedInformation