Year: 2018

Abstract:
Due to their wide field of view, omnidirectional cameras are frequently used by autonomous vehicles, drones and robots for navigation and other computer vision tasks. The images captured by such cameras, are often analyzed and classified with techniques designed for planar images that unfortunately fail to properly handle the native geometry of such images. That results in suboptimal performance, and lack of truly meaningful visual features. In this paper we aim at improving popular deep convolutional neural networks so that they can properly take into account the specific properties of omnidirectional data. In particular we propose an algorithm that adapts convolutional layers, which often serve as a core building block of a CNN, to the properties of omnidirectional images. Thus, our filters have a shape and size that adapts with the location on the omnidirectional image. We show that our method is not limited to spherical surfaces and is able to incorporate the knowledge about any kind of omnidirectional geometry inside the deep learning network. As depicted by our experiments, our method outperforms the existing deep neural network techniques for omnidirectional image classification and compression tasks.

heading: 1 INTRODUCTION
text: Drone vision, autonomous cars and robot navigation systems often use omnidirectional cameras, as they allow recording the scene with wide field of view. Despite their obvious advantages, images obtained by such cameras have different statistics compared to planar images. Nevertheless, omnidirectional images are often processed with standard techniques, which are unfortunately poorly adapted to the specific geometry of such images.
In this paper we improve one of the most popular frameworks for image processing, namely convolutional neural network (CNN) for omnidirectional images. CNNs prove to be effective, as they permit to achieve very good performance in many different tasks like image classification, segmentation, generation and compression. In the context of omnidirectional cameras, CNNs are typically applied directly to the unwrapped and distorted spherical images. This approach, however, is suboptimal: due to specific geometry of these images, and, in particular, the change in the image statistics with the position in the image. The latter forces the network to learn different filters for different locations in the omnidirectional images (see Fig. 1).
To solve this issue, we replace ordinary convolutional filters with the graph-based ones that can adapt their size and shape depending on the position in the omnidirectional image thanks to the flexible graph structure. In order to overcome the common limitation of these graph-based filters being isotropic (i.e. invariant to the changes in the objects orientation in the image) we suggest to use multiple directed graphs instead of a single undirected one, as used in (Defferrard et al., 2016; Kipf & Welling, 2017; Khasanova & Frossard, 2017a). This permits our method to encode the orientation of the objects that appear in the images and, therefore, extract more meaningful features from them. This together with the inherent ability of our filters to adapt to the image projective geometry allows our method to reach state-of-the-art performance when working not just with spherical projections, as done by Khasanova & Frossard (2017b); Cohen et al. (2018), but virtually with any type of image projective geometry that can be encoded by a graph. In our experiments we show that apart from the state-of-the-art results on the regular omnidirectional images classification task, our approach outperforms the existing classification techniques when applied to the images projected on a randomly
perturbed spherical surface or on a cube via the cube-map projection, which has recently become one of the popular ways to represent 360-images (Chen et al., 2018). Finally, we demonstrate that our method can be applied for orientation-dependent task such as compression and reduce artifacts comparing with a standard approaches.

heading: 2 RELATED WORK
text: In this section we first briefly introduce the most recent approaches that combine the power of deep learning methods with graph signal processing techniques, as they are similar in spirit to our work. We then discuss in more details the recent trends in omnidirectional image processing.
Geometric Deep learning. In the recent years a number of deep learning methods have been introduced tailored for processing irregular data that is represented as a graph. One way to solve this problem is suggested by Monti et al. (2017), where the authors define a local system of ddimensional pseudo-coordinates for every node of the graph and learn both the filters and patch operators that work on these coordinates. A different direction is taken by Wang et al. (2018), who propose using edge-based convolutional kernels and dynamically update the graph. While being flexible and effective for general tasks, in the specific context of the omnidirectional images these methods do not directly take the advantage of the knowledge about the projective geometry, which we model using a specifically designed graph representation. While most of the existing methods work with undirected graphs, the recent work of (Monti et al., 2018) propose an approach for processing data defined on a directed graph by exploiting local graph motifs that describe its connectivity patterns. The main differences of this method with our work are that first, this method assumes that the directed graph is already given. In our problem, building such a graph that is able to fully take advantage of the image projective geometry is one of the contributions. Second, the approach in (Monti et al., 2018) does not use the knowledge of the coordinate system associated with omnidirectional images, which we however use in our architecture, in order to define filter orientations. The list of the aforementioned works is by no means extensive, therefore we refer the interested reader to the survey (Bronstein et al., 2017), which summarizes many geometric deep learning methods in detail.
Omnidirectional image processing. The most typical way of dealing with images taken by omnidirectional cameras is to apply standard image processing techniques directly on the equirectangular projection images, which is one of the most common representations for the omnidirectional images (De Simone et al., 2016). However, due to the strong distortion effects introduced by the process of unwrapping of the projection surface to a plane, standard techniques lose much of their efficiency, as the appearance of the same object may change depending on its location in the image. To overcome this problem the recent work of Khasanova & Frossard (2017b) suggests using graph-based special convolutional filers to adapt to the geometry of omnidirectional images. This method, however, relies on the convolutional operation defined in the spectral domain, which leads
to isotropic filters and may reduce the complexity of trained filters. We, on the other hand, propose to build anisotropic graphs-based convolutional filters that do not have this limitation. A different direction is taken by the authors of (Su & Grauman, 2017) who suggest adapting the size of the convolutional kernel to the elevation of the equirectangular image. The main limitation of this technique, however, is that it requires a significantly larger number of parameters than the competing techniques, as it does not have the weight sharing property of CNNs. It rather requires learning different convolutional filters for different elevations of the equirectangular image. Further, Jeon & Kim (2017) propose to learn the shape of the convolutional filter, by learning the sampling locations (position offsets), where the elements of the filter are evaluated. The authors of (Dai et al., 2017) extend later this idea and suggest learning dynamic offsets of the filter elements depending on the image content, which allows the filters to adapt to different parts of the image. This method is quite flexible, however in the context of omnidirectional images requires an extensive training set, as the network needs to learn how it should react to various objects appearing at any possible elevation. In our work we rather take advantage of the knowledge of the image projective geometry and use this knowledge in the design of our architecture to adapt the size and shape of convolutional filters.
A different approach is suggested by Cohen et al. (2018), who introduces a CNN that is designed for spherical shapes and define filters directly on its surface. This method, however, is specifically designed for processing spherical images, while our approach is easily adapted to different kind of shapes, which we show in our experiments. Further, the methods of (Coors et al., 2018; Tateno et al., 2018) suggest a different way of compensating for the distortion of omnidirectional image. They suggest adapting the sampling locations of the convolutional filters to the geometry of the lens by projecting kernels to the sphere and using interpolated pixel values on the projected locations for implementing the convolutional filters. While these works are the closest to in spirit to ours, we propose a more general architecture, which permits to adapt the shape and size of the convolutional kernel to the location of omnidirectional image, and therefore to use the information about all the pixels and not only of a subset of them.
Then, the authors in (Monroy et al., 2018; Ruder et al., 2018) suggest a completely different approach to tackle the distortion of omnidirectional images. Instead of working with equirectangular images, they propose to project an omnidirectional image to a cube, where each of its faces represents an image that would have been seen through a regular perspective camera, with the optical center located in the center of the cube (Chen et al., 2018). Representing an omnidirectional image in this way allows having less noticeable distortion effects as compared to equirectangular images. This representation, however, suffers from another type of distortion that appears due to discontinuity effect on the borders between the faces of the cube. To mitigate this issue, Monroy et al. (2018) propose to apply a smoothing filter as a post-processing step and Ruder et al. (2018) suggest an algorithm that enforces consistency between the neighboring facets of the cube. Contrary to the mentioned approaches, as we model cube surface as a graph, our algorithm can easily handle the discontinuity problem and adapt to image distortions introduced by the cube-map projection.

heading: 3 GEOMETRY-AWARE CNN
text: In this section we describe our algorithm, which adapts convolutional filters to the distortion of omnidirectional images. We start with the introduction of the equirectangular projection, as it is one of common ways to represent images from the omnidirecitonal cameras (De Simone et al., 2016; Coors et al., 2018). We then describe our graph-based representation learning framework.

heading: 3.1 EQUIRECTANGULAR PROJECTION
text: Omnidirectional visual content can be represented as a sphere with radius r, where the user is assumed to be located at the center. Each 3D point can be projected to a point on the surface of this sphere, which is described by spherical coordinates, namely a longitude θ ∈ [−π, π] and a latitude φ ∈ [−π2 , π 2 ].
Omnidirectional images are generally not processed directly in their native geometry, but they are first projected to the 2D plane where classical image processing techniques can be activated. One of the popular projections involves sampling the data on the sphere with equal steps ∆θ and ∆φ, which results in an equrectangular image on the 2D plane. Thus, each point of equrectangular image
is defined by its spherical coordinates. To describe this projection let us introduce the tangent plane T , which is tangent to the sphere in the point (θ0, φ0). Thus, each point (x, y) on the tangent plane is projected to the sphere surface (θ, φ) as follows (Coors et al., 2018):
φ(x, y) = sin−1(cos ν sinφ0 + y sin ν cosφ0
ρ )
θ(x, y) = θ0 + tan −1( x sin νρ cosφ0 cos ν−y sinφ0 sin ν )
. (1)
where ρ = √ (x2 + y2) and ν = tan−1 ρ.
In order to have similar filter response regardless of the position of the object we model distortion of the applied filter. Thus, similarly to the works of (Khasanova & Frossard, 2017b) and (Coors et al., 2018), we define the filter kernel on this tangent plane T . Fig. 1 illustrates a sample equirectangular image with various kernels corresponding to tangent plane at different positions on the sphere. As we can see the projected area is different for various tangent plane locations. This projected area defines the support of our geometry-aware features, as described in the next section.

heading: 3.2 GEOMETRY-ADAPTIVE FILTERS
text: In the context of omnidirectional cameras, the main drawback of the classical convolutional approach is that it applies the same rectangular filters to different image positions. However, as we mentioned before, equirectangular images have different statistics at various elevations. Therefore, we propose to adapt the size and shape of the filter to the elevation on the spherical image.
To do so we propose to use a graph-based approach, which has recently become popular. The nodes of this graph vi ∈ G and signal y(vi) defined on the nodes represent pixels and their intensity values respectively. Based on this graph G we can then use the Laplacian polynomial filters F as proposed in (Defferrard et al., 2016; Khasanova & Frossard, 2017a), where normalized Laplacian operator is defined as follows: L = I −D−1/2AD−1/2, (2) where I is an identity matrix, D is a diagonal degree matrix and A is an adjacency matrix, which for each node vp of graph G defines its neighborhood Np. Then, F has the following form:
F = M∑ l=1 αlLl, (3)
where the αl are the trainable parameters, L is a Laplacian matrix and M is the degree of F . Using these filters we can construct a Deep learning architecture and use it for various tasks, e.g., classification of omnidirectional images. The main advantage of our approach is that by appropriately constructing the graph G we make the Laplacian polynomial filters F react similarly to the same object seen at different elevation of the equirectangular image regardless of geometric distortion due to the spherical geometry. Here we call those filters geometry-aware (GA).
In order to adapt the GA-filter F to the elevation we build a graph G in such a way that the neighbourhood of each node is different for different elevations of the omnidirectional image. In the following section we describe two approaches that rely on undirected and directed graphs respectively, which consequently define isotropic and anisotropic filters. Then we describe in more details the polynomial anisotropic filter F that is used for directed graphs..
Undirected graph construction for adaptive filtering. To adapt F to the elevation level we construct a graph G with nodes that have different neighborhoods depending on the elevation. To do so we define a circular area on a tangent plane T , centered in the tangency point. Then we move T such that it becomes tangent to the sphere S in different positions vp and project the circular area onto S. For every point of S this creates a neighborhood Np, which changes its shape and size together with the elevation, as can be seen in Fig. 1.
Based on this geometry adaptive neighborhood, we then construct the graph G in the following way. We connect the node vp ∈ G, corresponding to a tangent point on the sphere, with the node vj ∈ Np. The corresponding edge epi has a weight wpi that is inversely proportional to the Euclidean distance between vp and vi, which are defined on the sphere:
wpi = ||vp − vi||−1L2 , vi ∈ Np , wpi = 0, vi /∈ Np .
(4)
This allows us to vary the size of the neighbourhood according to the geometry of the omnidirectional image for each node in G, and weight the contribution of the nodes to the final filter response according to their distances to vp. Therefore, depending on the elevation the filter is changing its shape and size.
While effective, filter F does not have a defined orientation in space as according to Eq. (3) the filter applies the same weights αl to all nodes in the l-hoop neighborhood, with the contribution of each node being weighted by the distance to vp. This results in F being isotropic, which leads to suboptimal representation, as the network is not able to encode the orientation of the object.
Directed graphs construction. In order to overcome the limitations of isotropic filters, we propose to replace a single undirected graph G with multiple directed graphs Gk, where each Gk defines its own orientation. Let us consider the case of a 3x3 classical convolutional filter. In this case the filter has 9 distinct elements. To mimic the same structure with our graph convolutional filters we employ the following algorithm. First we define 9 non-overlapping areas sk, k = 1..9 on the tangent plane, which together form a rectangular region that is centered in the tangency point vp of T as defined in the Fig. 2. This rectangular region effectively defines the receptive field of the filter on the tangent plane. Then we build a set of nine directed graphs Gk, k = 1..9 in the similar way, as mentioned in the previous section for undirected graph. In particular in order to build graph Gk we do as follows. For the area sk and for every node vp we move the tangent plane at point vp and then project sk from T onto the sphere. This operation defines a specific neighborhood Nk(p) on the sphere that consists of the points that belong to the projection of the region sk from the plane T . We then connect vp with a directed edge to each of these points, where the weight of the edge is defined in Eq. (4). Note that the direction of the edge is very important, because connecting vp and vi with an undirected edge forces vp to be part of the neighborhood Nk(i). This, however, is not possible, as the neighborhood Nk(i) is computed by projecting the area sk from the plane T that is tangent to the sphere at point vi and does not include vp.
This results in construction of the directed graph Gk, which corresponds to the kth region of the filter, illustrated in Fig. 2. We repeat this operation for all the areas sk, k = 1..9 of our filter, which leads to creation of 9 directed graphs Gk, k = 1..9. Given this set of graphs Gk we define the resulting convolutional operation F as follows:
F = 9∑ k=1 Fk, (5)
where Fk is the filtering operation defined on the graph Gk. Note that this filtering operation is slightly different from the operation that is used when working with undirected graphs and is discussed in more details in the following section.
To sum up, the introduced graph construction process allows having anisotropic filters F , defined in Eq. (5) that are capable of capturing the orientation of the object and therefore learn more meaningful feature representation for an image compared to the isotropic graph-based filters. It is important
to note that in this paper we use the set of 9 non-overlapping rectangular areas defined on the tangent plane, as shown by Fig. 2, due to their rough correspondence to the elements of a 3 × 3 convolutional filter. However, our method can be easily extended to an arbitrary number of such areas with arbitrary shapes.
Geometry aware anisotropic filters. For directed graphs Gk Laplacian matrix is not defined, therefore, we use the polynomial filters proposed in (Sakiyama et al., 2017). Instead of the Laplacian matrix these filters rely on the normalized adjacency matrix Ak, which is defined as follows:
Ak = D−1k Ak, (6)
where Ak and Dk are the weighted adjacency and the diagonal degree matrices of graph Gk respectively. The elements of Dk are computed as Dk(m,m) = ∑ nAk(m,n). Then, we define filters in the following way: Fk = α(k)0 + α (k) 1 Ak, (7)
where α(k)0 , α (k) 1 are the training parameters of our filter. Here, we use polynomial filters of degree 1, as they achieve good balance between speed and performance.
Network architecture. The introduced approach focuses on the modification of the convolutional layer to incorporate the knowledge about the image projective geometry inside the neural network. Thus, it can be applied for a broad variety of tasks. In this chapter we focus on the image classification and compression problems. For the former one we use a relatively standard architecture that consists of a sequence of convolutional layers with the introduced graph-based filters, followed by a sequence of the fully connected layers. For the compression task we use the architecture proposed in (Ballé et al., 2017) and replace its convolutional filters with the proposed graph-based ones.
Discussion. Our method can be seen as a generalization of different approaches that have been developed for omnidirectional images. For example, if the node vp at elevation φi has only one neighbor in each direction and the weight of the edges between nodes is always equal to one, it will be the standard CNN method (LeCun et al., 2001). Further, if these neighbors correspond to the projected points it becomes the recently proposed algorithm of (Coors et al., 2018). Finally, if we replace directed graphs with a single undirected one we get the same behavior of the polynomial filters as described in graph-based deep learning methods (Khasanova & Frossard, 2017a; Defferrard et al., 2016; Kipf & Welling, 2017; Khasanova & Frossard, 2017b).

heading: 4 RESULTS
text: In this section we illustrate the performance of our approach. We start by evaluating our method with respect to competing approaches on the task of classification images that are projected to different surfaces. Finally, to show the generality of our approach and illustrate the effectiveness of the anisotropic graph-based filters, we evaluate our method on the image compression task.

heading: 4.1 IMAGE CLASSIFICATION
text: In this section we first introduce the datasets that we used for the evaluation of our method. We then discuss the baseline approaches and architectures, which we use in our experiments. Finally, we show the quantitative comparison of our method with the competing ones.
Datasets. We evaluate our method on three different types of data, which feature different surface geometries, where the images are projected:
• Spherical dataset (S) consists of images projected on different locations on a sphere. The resulting spherical images are then unwrapped to for equirectangular images, as described in Section 3.1.
• Mod-spherical dataset (MS) features image projections on more complicated surfaces that are depicted by Fig. 3 together with the representative examples of projected images. This dataset itself consists of three different versions: MS1, MS2, MS3 which correspond to the
surfaces which are getting further away from the spherical one. A more detailed discussion about the type of projection and the surface geometry used in these datasets can be found in ApendixB.
• Fish-eye dataset (F) consists of images projected on different locations on a sphere using stereographic projection Bettonvil (2005), which is frequently used in fish-eye cameras.
• Cube-map dataset (CM) features projection of the images on the cube as shown by Fig. 4. This type of projection has recently gained popularity for handling omnidirectional images, due to its ability to reduce distortion artifacts that appear due to the spherical geometry.
In all these dataset we use MNIST images from (LeCun & Cortes, 2010) , which are divided into train, validation and test sets with 54k, 6k and 10k samples respectively.
Architecture. We compare our approach with standard ConvNets, the algorithm proposed in (Cohen et al., 2018) and other graph-based methods. We present our result in the Table 11. For the graph-based methods we investigate 3 possible way of constructing G:
• Regular grid-graph with 8 neighbors and all equal weights wij = 1;
• Regular grid-graph with 8 neighbors and weights that depend on the Euclidean distance dij between the nodes, as proposed in (Khasanova & Frossard, 2017b) wij = d−1ij ;
• Irregular GA-graph wij = d−1ij (isotropic filters from Section 3.2).
For all of them we build a normalized Laplacian matrix (Khasanova & Frossard, 2017b) and use polynomial filters of degree 1, which is equivalent to using 3× 3 convolutional kernels. Therefore, for the standard ConvNet we similarly rely on filters of size 3× 3.
1We were unable to compare our method to the recent work of (Coors et al., 2018) as to the best of our knowledge, there is no publicly available implementation.
All the competing approaches use the networks of roughly the same complexity. For all the methods we use the architectures of similar structure and roughly the same number of parameters. For all the graph-based approaches we use the graph-based convolutions with stride two on each layer, which in turn requires building graph for each new layer according to its respective sampling. The exact architecture of the classification network is illustrated in Appendix A. For the method of (Cohen et al., 2018) we used the architecture proposed in the paper with roughly the same number of parameters as in competing approaches.
Evaluation. We present the result of comparison of our approach with the baseline methods in Table 1. Our method significantly outperforms the standard ConvNets, as it is designed to use the geometry of the omnidirectional images. Further it shows a much higher accuracy then other graphbased techniques, which rely on isotropic filters. Further, our method achieves comparably accuracy with (Cohen et al., 2018) on spherical image representation, however, our method is more general, therefore, it outperforms (Cohen et al., 2018) on other datasets. Finally, we are able to run our approach on cub-map projection while the SphericalCNN by design is not applicable to such kind of images.

heading: 4.2 IMAGE COMPRESSION
text: In all our previous experiment we have focused on evaluating our approach on the image classification task. To show the generality of our method and better illustrate the effectiveness of anisotropic graph-based filters, we now evaluate their performance on an image compression problem. For this task, we choose to modify the architecture introduced in (Ballé et al., 2017) by replacing the ordinary convolutional layers with our own graph-based convolutions. In this section we first introduce our approach and then compare the performance of the two graph-based methods, which rely on isotropic and anisotropic graph-based filters respectively.
Image compression framework. The method introduced in (Ballé et al., 2017), presents the process of image compression is an optimization of the tradeoff between having small distortion of the pixel intensity values and the small number of bits that are required for storing the compressed representation of these values. As described in (Ballé et al., 2016; Ballé et al., 2017), this optimization can be represented as a variational autoencoder. For more details we refer to Appendix C. In the context of omnidirectional images, we propose to modify the method proposed in (Ballé et al., 2017) by using our geometry-aware filters instead of standard convolutional ones.
Evaluation. We now evaluate the performance of our approach. For this experiment we have implemented two versions of the system. One with isotropic graph-based filters and the other one with the anisotropic ones. We further evaluate the original method (Ballé et al., 2017) for the sake of completeness. All three methods are trained and tested on the same splits of the modified version of the dataset (Xiao et al., 2012), which consists of omnidirectional images projected onto a cube. From this dataset we use 3900 images for training and 1000 for testing of our approaches.
We compare the methods in terms of the Peak Signal to Noise Ratio (PSNR) with respect to the average number of bits per pixel (bpp). The results of the evaluation are presented in Fig. 5. As we can see, our method with anisotropic filters and (Ballé et al., 2017) show similar PSNR values and
significantly outperform the architecture with isotropic filters. Further, due to the fact that PSNR depends on the average difference in pixel values between the compressed image and the original one it is not able to reliably detect small artifacts that appear in the cube-map images, which are noticeable for humans. These artifacts are clearly seen in Fig. 6, where we illustrate that due to the knowledge about the image projective geometry, our approach correctly reconstructs the areas along cube borders, while the method of Ballé et al. (2017) over-smooths these areas.

heading: 5 CONCLUSION
text: In this paper we have presented generic way of graph construction that allows incorporating the information about the image geometry inside the neural network. Further, we have introduced the graph-based geometry aware convolutional filters that adapt their shape and size to the geometry of the projection surface. In contrast to many existing graph-based filters, our filters are anisotropic, which allows to better adjust to the specific properties of the problem. Our illustrative experiments show state-of-the-art performance of our approach applied to image classification and compression tasks in the presence of various types of image distortions.

heading: A ARCHITECTURE OF CLASSIFICATION FRAMEWORK
text: Fig. 7 illustrates the architecture of our classification network, which we use in Section 4.1.

heading: B MODIFIED SPHERICAL SURFACE.
text: In order to evaluate the performance of our method as a function of deformation of the spherical surface, we have created a set of datasets by projecting the MNIST images to random locations of the surfaces, which have shapes, shown in Fig. 3 (a,c) and unwrap these spherical images to equirectangular ones. The white color in Fig. 3 (a,c) denotes the areas of the generated surface that are the furthest from the spherical surface of the same radius. The Fig. 3 (b,d) illustrates sample images of digits projected onto the respective surfaces. Each of the aforementioned surfaces is the following modification of a spherical one from Eq. (1):
x = cos(φi) sin(θi − θ0) y = (cos(φ0) sin(φi + p(φi, r, l))− sin(φ0 + p(φ0, r, l)) cos(φi) cos(θi − θ0))/c, c = sin(φi + p(φi, r, l)) sin(φ0 + p(φ0, r, l)) + cos(φi) cos(φ0) cos(θi − θ0),
(8)
where (x, y) are the coordinates on the tangent plane and p(φ, r, l) is the perturbation function that can be written as
p(φ, r, l) = r sin−1(sin(lφ)), (9)
where φ is the elevation level; r is the parameter that regulates the perturbation magnitude and l defines frequency of the perturbation signal. In our experiments we have set l = 10. Note that for a specific case of r = 0 we get the ordinary spherical surface. We then use Eq. 8 to construct the graph G that allows our method to adapt to the surface geometry and evaluate our method on each of the generated datasets.

heading: C COMPRESSION
text: In this section we briefly describe compression approach, which proposed by ?. An input image x is encoded using a function ga(x;α), which results in the respective latent representation y. Then, y is quantized into ŷ, which can be losslessly compressed using entropy coding algorithms. This ŷ is then passed to the decoder gs(ŷ;β) at the decompression step, which results in a decompressed image x̂. Here, we denote by α and β the parameters of the encoding and decoding algorithms respectively. While both encoder and decoder can be represented as a differentiable function, the process of quantization is non-differentiable. Therefore the authors of (Ballé et al., 2016) propose to replace quantization with an additive uniform noise at the training step as follows:
ỹi = y + ∆y, (10)
where ∆y denotes additive i.i.d uniform noise. This trick allows to perform the end-to-end optimization of both the encoder and decoder parameters using the following loss function:
L(α, β) = Ex,∆y [ − ∑ i log2pỹi(ga(x;α) + ∆y) + λd(gs(ga(x;α) + ∆y;β), x) ] , (11)
where gs, ga are convolutional deep neural networks, d represents the distance between the images and λ is a weighting parameter. Thus, during the training step, we add noise (according to Eq. (10)) to be able to back propagate the error and at the inference time we apply quantization to the latent representation y. The overall architecture that we use is similar to the one proposed in (Ballé et al., 2017) and is summarized in Fig. 8. Further, the method of (Ballé et al., 2017) relies on the standard convolutional layers, which are practical for ordinary images: they allow learning local image structures independently of their location in the image.
Additional visual results. We run experiment with described compression architecture, where we compare three approaches: original and methods, where we replace convolutional filter from the architecture to graph-based isotropic and geometry-aware filters. Fig. 9 further illustrates some visual comparison of the methods and we can see isotropic filters produce over-smoothed decompressed images, which do not look realistic and result in very low PSNR values. On the other hand our method with anisotropic filters is able to produce sharp results.

Review summaries:
The paper introduces geometry-aware filters based on constructed graphs into the standard CNN for omnidirectional image classification. Overall, the idea is interesting and the authors propose an extrinsic way to respect the underlying geometry by using tangent space projection. Understanding the graph construction and filter definition is not easy from the text description. It would be better to use a figure to illustrate them. 

The paper proposes a new way of defining CNNs for omnidirectional images. The method is based on graph convolutional networks, and in contrast to previous work, is applicable to other geometries than spherical ones (e.g. fisheye cameras). Since standard graph CNNs are unable to tell left from right (and up from down, etc.), a key question is how to define anisotropic filters. This is achieved by introducing several directed graphs that have orientation built into the graph structure.