Learning Spherical Convolution for <inline-formula><tex-math notation="LaTeX">$360^{\circ }$</tex-math><alternatives><mml:math><mml:msup><mml:mn>360</mml:mn><mml:mo>∘</mml:mo></mml:msup></mml:math><inline-graphic xlink:href="su-ieq1-3113612.gif"/></alternatives></inline-formula> Recognition

While <inline-formula><tex-math notation="LaTeX">$360^{\circ }$</tex-math><alternatives><mml:math><mml:msup><mml:mn>360</mml:mn><mml:mo>∘</mml:mo></mml:msup></mml:math><inline-graphic xlink:href="su-ieq2-3113612.gif"/></alternatives></inline-formula> cameras offer tremendous new possibilities in vision, graphics, and augmented reality, the spherical images they produce make visual recognition non-trivial. Ideally, <inline-formula><tex-math notation="LaTeX">$360^{\circ }$</tex-math><alternatives><mml:math><mml:msup><mml:mn>360</mml:mn><mml:mo>∘</mml:mo></mml:msup></mml:math><inline-graphic xlink:href="su-ieq3-3113612.gif"/></alternatives></inline-formula> imagery could inherit the deep convolutional neural networks (CNNs) already trained with great success on perspective projection images. However, spherical images cannot be projected to a single plane without significant distortion, and existing methods to transfer CNNs from perspective to spherical images introduce significant computational costs and/or degradations in accuracy. We propose to <italic>learn</italic> a Spherical Convolution Network (SphConv) that translates a planar CNN to the equirectangular projection of <inline-formula><tex-math notation="LaTeX">$360^{\circ }$</tex-math><alternatives><mml:math><mml:msup><mml:mn>360</mml:mn><mml:mo>∘</mml:mo></mml:msup></mml:math><inline-graphic xlink:href="su-ieq4-3113612.gif"/></alternatives></inline-formula> images. Given a source CNN for perspective images as input, SphConv learns to reproduce the flat filter outputs on <inline-formula><tex-math notation="LaTeX">$360^{\circ }$</tex-math><alternatives><mml:math><mml:msup><mml:mn>360</mml:mn><mml:mo>∘</mml:mo></mml:msup></mml:math><inline-graphic xlink:href="su-ieq5-3113612.gif"/></alternatives></inline-formula> data, sensitive to the varying distortion effects across the viewing sphere. The key benefits are 1) efficient and accurate recognition for <inline-formula><tex-math notation="LaTeX">$360^{\circ }$</tex-math><alternatives><mml:math><mml:msup><mml:mn>360</mml:mn><mml:mo>∘</mml:mo></mml:msup></mml:math><inline-graphic xlink:href="su-ieq6-3113612.gif"/></alternatives></inline-formula> images, and 2) the ability to leverage powerful pre-trained networks for perspective images. We further proposes two instantiation of SphConv—Spherical Kernel learns location dependent kernels on the sphere for SphConv, and Kernel Transformer Network learns a functional transformation that generates SphConv kernels from the source CNN. Among the two variants, Kernel Transformer Network has a much lower memory footprint at the cost of higher computational overhead. Validating our approach with multiple source CNNs and datasets, we show that SphConv using KTN successfully preserves the source CNN’s accuracy, while offering efficiency, transferability, and scalability to typical image resolutions. We further introduce a spherical Faster R-CNN model based on SphConv and show that we can learn a spherical object detector without any object annotation in <inline-formula><tex-math notation="LaTeX">$360^{\circ }$</tex-math><alternatives><mml:math><mml:msup><mml:mn>360</mml:mn><mml:mo>∘</mml:mo></mml:msup></mml:math><inline-graphic xlink:href="su-ieq7-3113612.gif"/></alternatives></inline-formula> images.