Geometry aware convolutional filters for omnidirectional images representation

1 INTRODUCTION . Drone vision , autonomous cars and robot navigation systems often use omnidirectional cameras , as they allow recording the scene with wide field of view . Despite their obvious advantages , images obtained by such cameras have different statistics compared to planar images . Nevertheless , omnidirectional images are often processed with standard techniques , which are unfortunately poorly adapted to the specific geometry of such images . In this paper we improve one of the most popular frameworks for image processing , namely convolutional neural network ( CNN ) for omnidirectional images . CNNs prove to be effective , as they permit to achieve very good performance in many different tasks like image classification , segmentation , generation and compression . In the context of omnidirectional cameras , CNNs are typically applied directly to the unwrapped and distorted spherical images . This approach , however , is suboptimal : due to specific geometry of these images , and , in particular , the change in the image statistics with the position in the image . The latter forces the network to learn different filters for different locations in the omnidirectional images ( see Fig . 1 ) . To solve this issue , we replace ordinary convolutional filters with the graph-based ones that can adapt their size and shape depending on the position in the omnidirectional image thanks to the flexible graph structure . In order to overcome the common limitation of these graph-based filters being isotropic ( i.e . invariant to the changes in the objects orientation in the image ) we suggest to use multiple directed graphs instead of a single undirected one , as used in ( Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Khasanova & Frossard , 2017a ) . This permits our method to encode the orientation of the objects that appear in the images and , therefore , extract more meaningful features from them . This together with the inherent ability of our filters to adapt to the image projective geometry allows our method to reach state-of-the-art performance when working not just with spherical projections , as done by Khasanova & Frossard ( 2017b ) ; Cohen et al . ( 2018 ) , but virtually with any type of image projective geometry that can be encoded by a graph . In our experiments we show that apart from the state-of-the-art results on the regular omnidirectional images classification task , our approach outperforms the existing classification techniques when applied to the images projected on a randomly perturbed spherical surface or on a cube via the cube-map projection , which has recently become one of the popular ways to represent 360-images ( Chen et al. , 2018 ) . Finally , we demonstrate that our method can be applied for orientation-dependent task such as compression and reduce artifacts comparing with a standard approaches . 2 RELATED WORK . In this section we first briefly introduce the most recent approaches that combine the power of deep learning methods with graph signal processing techniques , as they are similar in spirit to our work . We then discuss in more details the recent trends in omnidirectional image processing . Geometric Deep learning . In the recent years a number of deep learning methods have been introduced tailored for processing irregular data that is represented as a graph . One way to solve this problem is suggested by Monti et al . ( 2017 ) , where the authors define a local system of ddimensional pseudo-coordinates for every node of the graph and learn both the filters and patch operators that work on these coordinates . A different direction is taken by Wang et al . ( 2018 ) , who propose using edge-based convolutional kernels and dynamically update the graph . While being flexible and effective for general tasks , in the specific context of the omnidirectional images these methods do not directly take the advantage of the knowledge about the projective geometry , which we model using a specifically designed graph representation . While most of the existing methods work with undirected graphs , the recent work of ( Monti et al. , 2018 ) propose an approach for processing data defined on a directed graph by exploiting local graph motifs that describe its connectivity patterns . The main differences of this method with our work are that first , this method assumes that the directed graph is already given . In our problem , building such a graph that is able to fully take advantage of the image projective geometry is one of the contributions . Second , the approach in ( Monti et al. , 2018 ) does not use the knowledge of the coordinate system associated with omnidirectional images , which we however use in our architecture , in order to define filter orientations . The list of the aforementioned works is by no means extensive , therefore we refer the interested reader to the survey ( Bronstein et al. , 2017 ) , which summarizes many geometric deep learning methods in detail . Omnidirectional image processing . The most typical way of dealing with images taken by omnidirectional cameras is to apply standard image processing techniques directly on the equirectangular projection images , which is one of the most common representations for the omnidirectional images ( De Simone et al. , 2016 ) . However , due to the strong distortion effects introduced by the process of unwrapping of the projection surface to a plane , standard techniques lose much of their efficiency , as the appearance of the same object may change depending on its location in the image . To overcome this problem the recent work of Khasanova & Frossard ( 2017b ) suggests using graph-based special convolutional filers to adapt to the geometry of omnidirectional images . This method , however , relies on the convolutional operation defined in the spectral domain , which leads to isotropic filters and may reduce the complexity of trained filters . We , on the other hand , propose to build anisotropic graphs-based convolutional filters that do not have this limitation . A different direction is taken by the authors of ( Su & Grauman , 2017 ) who suggest adapting the size of the convolutional kernel to the elevation of the equirectangular image . The main limitation of this technique , however , is that it requires a significantly larger number of parameters than the competing techniques , as it does not have the weight sharing property of CNNs . It rather requires learning different convolutional filters for different elevations of the equirectangular image . Further , Jeon & Kim ( 2017 ) propose to learn the shape of the convolutional filter , by learning the sampling locations ( position offsets ) , where the elements of the filter are evaluated . The authors of ( Dai et al. , 2017 ) extend later this idea and suggest learning dynamic offsets of the filter elements depending on the image content , which allows the filters to adapt to different parts of the image . This method is quite flexible , however in the context of omnidirectional images requires an extensive training set , as the network needs to learn how it should react to various objects appearing at any possible elevation . In our work we rather take advantage of the knowledge of the image projective geometry and use this knowledge in the design of our architecture to adapt the size and shape of convolutional filters . A different approach is suggested by Cohen et al . ( 2018 ) , who introduces a CNN that is designed for spherical shapes and define filters directly on its surface . This method , however , is specifically designed for processing spherical images , while our approach is easily adapted to different kind of shapes , which we show in our experiments . Further , the methods of ( Coors et al. , 2018 ; Tateno et al. , 2018 ) suggest a different way of compensating for the distortion of omnidirectional image . They suggest adapting the sampling locations of the convolutional filters to the geometry of the lens by projecting kernels to the sphere and using interpolated pixel values on the projected locations for implementing the convolutional filters . While these works are the closest to in spirit to ours , we propose a more general architecture , which permits to adapt the shape and size of the convolutional kernel to the location of omnidirectional image , and therefore to use the information about all the pixels and not only of a subset of them . Then , the authors in ( Monroy et al. , 2018 ; Ruder et al. , 2018 ) suggest a completely different approach to tackle the distortion of omnidirectional images . Instead of working with equirectangular images , they propose to project an omnidirectional image to a cube , where each of its faces represents an image that would have been seen through a regular perspective camera , with the optical center located in the center of the cube ( Chen et al. , 2018 ) . Representing an omnidirectional image in this way allows having less noticeable distortion effects as compared to equirectangular images . This representation , however , suffers from another type of distortion that appears due to discontinuity effect on the borders between the faces of the cube . To mitigate this issue , Monroy et al . ( 2018 ) propose to apply a smoothing filter as a post-processing step and Ruder et al . ( 2018 ) suggest an algorithm that enforces consistency between the neighboring facets of the cube . Contrary to the mentioned approaches , as we model cube surface as a graph , our algorithm can easily handle the discontinuity problem and adapt to image distortions introduced by the cube-map projection . 3 GEOMETRY-AWARE CNN . In this section we describe our algorithm , which adapts convolutional filters to the distortion of omnidirectional images . We start with the introduction of the equirectangular projection , as it is one of common ways to represent images from the omnidirecitonal cameras ( De Simone et al. , 2016 ; Coors et al. , 2018 ) . We then describe our graph-based representation learning framework . 3.1 EQUIRECTANGULAR PROJECTION . Omnidirectional visual content can be represented as a sphere with radius r , where the user is assumed to be located at the center . Each 3D point can be projected to a point on the surface of this sphere , which is described by spherical coordinates , namely a longitude θ ∈ [ −π , π ] and a latitude φ ∈ [ −π2 , π 2 ] . Omnidirectional images are generally not processed directly in their native geometry , but they are first projected to the 2D plane where classical image processing techniques can be activated . One of the popular projections involves sampling the data on the sphere with equal steps ∆θ and ∆φ , which results in an equrectangular image on the 2D plane . Thus , each point of equrectangular image is defined by its spherical coordinates . To describe this projection let us introduce the tangent plane T , which is tangent to the sphere in the point ( θ0 , φ0 ) . Thus , each point ( x , y ) on the tangent plane is projected to the sphere surface ( θ , φ ) as follows ( Coors et al. , 2018 ) : φ ( x , y ) = sin−1 ( cos ν sinφ0 + y sin ν cosφ0 ρ ) θ ( x , y ) = θ0 + tan −1 ( x sin νρ cosφ0 cos ν−y sinφ0 sin ν ) . ( 1 ) where ρ = √ ( x2 + y2 ) and ν = tan−1 ρ . In order to have similar filter response regardless of the position of the object we model distortion of the applied filter . Thus , similarly to the works of ( Khasanova & Frossard , 2017b ) and ( Coors et al. , 2018 ) , we define the filter kernel on this tangent plane T . Fig . 1 illustrates a sample equirectangular image with various kernels corresponding to tangent plane at different positions on the sphere . As we can see the projected area is different for various tangent plane locations . This projected area defines the support of our geometry-aware features , as described in the next section .

Summary: The paper proposes a new way of defining CNNs for omnidirectional images. The method is based on graph convolutional networks, and in contrast to previous work, is applicable to other geometries than spherical ones (e.g. fisheye cameras). Since standard graph CNNs are unable to tell left from right (and up from down, etc.), a key question is how to define anisotropic filters. This is achieved by introducing several directed graphs that have orientation built into the graph structure.