OctNet: Learning Deep 3D Representations at High Resolutions
Gernot Riegler1
Ali Osman Ulusoy2
Andreas Geiger2,31Institute for Computer Graphics and Vision, Graz University of Technology
2Autonomous Vision Group, MPI for Intelligent Systems Tübingen
3
Computer Vision and Geometry Group, ETH Zürich
riegler@icg.tugraz.at{osman.ulusoy,andreas.geiger}@tue.mpg.deDense 3D ConvNet

Abstract

OctNetDense 3D ConvNet

We present OctNet, a representation for deep learningwith sparse 3D data. In contrast to existing models, our representation enables 3D convolutional networks which areboth deep and high resolution. Towards this goal, we exploit the sparsity in the input data to hierarchically partition the space using a set of unbalanced octrees where eachleaf node stores a pooled feature representation. This allows to focus memory allocation and computation to therelevant dense regions and enables deeper networks withoutcompromising resolution. We demonstrate the utility of our
OctNet representation by analyzing the impact of resolution
on several 3D tasks including 3D object classification, orientation estimation and point cloud labeling.

1. Introduction
Over the last several years, convolutional networks havelead to substantial performance gains in many areas of computer vision. In most of these cases, the input to the network
is of two-dimensional nature, e.g., in image classification[18], object detection [35] or semantic segmentation [13].
However, recent advances in 3D reconstruction [33] andgraphics [21] allow capturing and modeling large amounts
of 3D data. At the same time, large 3D repositories such as
ModelNet [47], ShapeNet [6] or 3D Warehouse1 as well as
databases of 3D object scans [7] are becoming increasingly
available. These factors have motivated the development ofconvolutional networks that operate on 3D data.
Most existing 3D network architectures [8,29,34,47] replace the 2D pixel array by its 3D analogue, i.e., a dense andregular 3D voxel grid, and process this grid using 3D convolution and pooling operations. However, for dense 3D
data, computational and memory requirements grow cubically with the resolution. Consequently, existing 3D networks are limited to low 3D resolutions, typically in theorder of 303 voxels. To fully exploit the rich and detailed(a) Layer 1: 323(b) Layer 2: 163(c) Layer 3: 83

Figure 1: Motivation. For illustration purposes, wetrained a dense convolutional network to classify 3D shapes
from [47]. Given a voxelized bed as input, we show the
maximum response across all feature maps at intermediate
layers (a-c) of the network before pooling. Higher activations are indicated with darker colors. Voxels with zeroactivation are not displayed. The first row visualizes the
responses in 3D while the second row shows a 2D slice.
Note how voxels close to the object contour respond morestrongly than voxels further away. We exploit the sparsity
in our data by allocating memory and computations using aspace partitioning data structure (bottom row).
geometry of our 3D world, however, much higher resolution
networks are required.
In this work, we build on the observation that 3D data isoften sparse in nature, e.g., point clouds, or meshes, resulting in wasted computations when applying 3D convolutions

1 https://3dwarehouse.sketchup.com

1

naı̈vely. We illustrate this in Fig. 1 for a 3D classification
example. Given the 3D meshes of [47] we voxelize the input
at a resolution of 643 and train a simple 3D convolutional
network to minimize a classification loss. We depict the
maximum of the responses across all feature maps at different layers of the network. It is easy to observe that highactivations occur only near the object boundaries.
Motivated by this observation, we propose OctNet, a 3D
convolutional network that exploits this sparsity property.
Our OctNet hierarchically partitions the 3D space into a setof unbalanced octrees [31]. Each octree splits the 3D spaceaccording to the density of the data. More specifically, werecursively split octree nodes that contain a data point in its
domain, i.e., 3D points, or mesh triangles, stopping at thefinest resolution of the tree. Therefore, leaf nodes vary insize, e.g., an empty leaf node may comprise up to 83 = 512
voxels for a tree of depth 3 and each leaf node in the octree
stores a pooled summary of all feature activations of thevoxel it comprises. The convolutional network operations
are directly defined on the structure of these trees. Therefore, our network dynamically focuses computational andmemory resources, depending on the 3D structure of the input. This leads to a significant reduction in computationaland memory requirements which allows for deep learning
at high resolutions. Importantly, we also show how essential network operations (convolution, pooling or unpooling)
can be efficiently implemented on this new data structure.
We demonstrate the utility of the proposed OctNet on
three different problems involving three-dimensional data:
3D classification, 3D orientation estimation of unknown
object instances and semantic segmentation of 3D point
clouds. In particular, we show that the proposed OctNet enables significant higher input resolutions compared to dense
inputs due to its lower memory consumption, while achieving identical performance compared to the equivalent dense
network at lower resolutions. At the same time we gain significant speed-ups at resolutions of 1283 and above. Using
our OctNet, we investigate the impact of high resolution inputs wrt. accuracy on the three tasks and demonstrate thathigher resolutions are particularly beneficial for orientation
estimation and semantic point cloud labeling. Our code isavailable from the project website2 .

2. Related Work
While 2D convolutional networks have proven very successful in extracting information from images [11, 13, 18,
35, 41, 42, 46, 48, 49], there exists comparably little workon processing three-dimensional data. In this Section, we
review existing work on dense and sparse models.
Dense Models: Wu et al. [47] trained a deep belief networkon shapes discretized to a 303 voxel grid for object classi2 https://github.com/griegler/octnet

fication, shape completion and next best view prediction.
Maturana et al. [29] proposed VoxNet, a feed-forward convolutional network for classifying 323 voxel volumes from
RGB-D data. In follow-up work, Sedaghat et al. [1] showedthat introducing an auxiliary orientation loss increases classification performance over the original VoxNet. Similar
models have also been exploited for semantic point cloudlabeling [20] and scene context has been integrated in [51].
Recently, generative models [36] and auto-encoders [5,39] have demonstrated impressive performance in learning
low-dimensional object representations from collections of
low-resolution (323 ) 3D shapes. Interestingly, these lowdimensional representations can be directly inferred from asingle image [14] or a sequence of images [8].
Due to computational and memory limitations, all aforementioned methods are only able to process and generate
shapes at a very coarse resolution, typically in the order of
303 voxels. Besides, when high-resolution outputs are desired, e.g., for labeling 3D point clouds, inefficient slidingwindow techniques with a limited receptive field must be
adopted [20]. Increasing the resolution naı̈vely [32, 40, 52]
reduces the depth of the networks and hence their expressiveness. In contrast, the proposed OctNets allow for training deep architectures at significant higher resolutions.
Sparse Models: There exist only few network architectures which explicitly exploit sparsity in the data. As these
networks do not require exhaustive dense convolutions they
have the potential of handling higher resolutions.
Engelcke et al. [10] proposed to calculate convolutionsat sparse input locations by pushing values to their target
locations. This has the potential to reduce the number ofconvolutions but does not reduce the amount of memory required. Consequently, their work considers only very shallow networks with up to three layers.
A similar approach is presented in [15, 16] where sparse
convolutions are reduced to matrix operations. Unfortunately, the model only allows for 2 × 2 convolutions andresults in indexing and copy overhead which prevents processing volumes of larger resolution (the maximum resolution considered in [15, 16] is 803 voxels). Besides, eachlayer decreases sparsity and thus increases the number ofoperations, even at a single resolution. In contrast, the number of operations remains constant in our model.
Li et al. [27] proposed field probing networks whichsample 3D data at sparse points before feeding them into
fully connected layers. While this reduces memory andcomputation, it does not allow for exploiting the distributed
computational power of convolutional networks as fieldprobing layers can not be stacked, convolved or pooled.
Jampani et al. [22] introduced bilateral convolution layers (BCL) which map sparse inputs into permutohedral
space where learnt convolutional filters are applied. Theirwork is related to ours with respect to efficiently exploiting

the sparsity in the input data. However, in contrast to BCL
our method is specifically targeted at 3D convolutional networks and can be immediately dropped in as a replacement
in existing network architectures.

3. Octree Networks
To decrease the memory footprint of convolutional networks operating on sparse 3D data, we propose an adaptive
space partitioning scheme which focuses computations on
the relevant regions. As mathematical operations of deep
networks, especially convolutional networks, are best understood on regular grids, we restrict our attention to data
structures on 3D voxel grids. One of the most popular spacepartitioning structures on voxel grids are octrees [30] whichhave been widely adopted due to their flexible and hierarchical structure. Areas of application include depth fusion [23], image rendering [26] and 3D reconstruction [44].
In this paper, we propose 3D convolutional networks on octrees to learn representations from high resolution 3D data.
An octree partitions the 3D space by recursively subdividing it into octants. By subdividing only the cells whichcontain relevant information (e.g., cells crossing a surface
boundary or cells containing one or more 3D points) storage
can be allocated adaptively. Densely populated regions aremodeled with high accuracy (i.e., using small cells) while
empty regions are summarized by large cells in the octree.
Unfortunately, vanilla octree implementations [30] haveseveral drawbacks that hamper its application in deep networks. While octrees reduce the memory footprint of the
3D representation, most versions do not allow for efficient
access to the underlying data. In particular, octrees are typically implemented using pointers, where each node contains
a pointer to its children. Accessing an arbitrary element (orthe neighbor of an element) in the octree requires a traversal
starting from the root until the desired cell is reached. Thus,
the number of memory accesses is equal to the depth of thetree. This becomes increasingly costly for deep, i.e., highresolution, octrees. Convolutional network operations suchas convolution or pooling require frequent access to neighboring elements. It is thus critical to utilize an octree designthat allows for fast data access.
We tackle these challenges by leveraging a hybrid gridoctree data structure which we describe in Section 3.1. In
Section 3.2, we show how 3D convolution and pooling operations can be implemented efficiently on this data structure.

3.1. Hybrid Grid-Octree Data Structure
The above mentioned problems with the vanilla octree
data structure increase with the octree depth. Instead of representing the entire high resolution 3D input with a single
unbalanced octree, we leverage a hybrid grid-octree structure similar to the one proposed by Miller et al. [31]. The
key idea is to restrict the maximal depth of an octree to a

Figure 2: Hybrid Grid-Octree Data Structure. This example illustrates a hybrid grid-octree consisting of 8 shallow octrees indicated by different colors. Using 2 shallow
octrees in each dimension with a maximum depth of 3 leads
to a total resolution of 163 voxels.

1
0

1

0

1

0

0

0

0

01010000 01010000(a) Shallow Octree(b) Bit-Representation

Figure 3: Bit Representation. Shallow octrees can be
efficiently encoded using bit-strings. Here, the bit-string
1 01010000 00000000 01010000 00000000 01010000 0...
defines the octree in (a). The corresponding tree is shownin (b). The color of the voxels corresponds to the split level.
small number, e.g., three, and place several such shallow
octrees along a regular grid (Fig. 2). While this data structure may not be as memory efficient as the standard octree,significant compression ratios can still be achieved. For instance, a single shallow octree that does not contain input
data stores only a single vector, instead of 83 = 512 vectors
for all voxels at the finest resolution at depth 3.
An additional benefit of a collection of shallow octrees
is that their structure can be encoded very efficiently using a bit string representation which further lowers accesstime and allows for efficient GPGPU implementations [31].
Given a shallow octree of depth 3, we use 73 bit to represent the complete tree. The first bit with index 0 indicates,
if the root node is split, or not. Further, bits 1 to 8 indicate
if one of the child nodes is subdivided and bits 9 to 72 denote splits of the grandchildren, see Fig. 3. A tree depth of
3 gives a good trade-off between memory consumption andcomputational efficiency. Increasing the octree depth results
in an exponential growth in the required bits to store the tree
structure and further increases the cell traversal time.
Using this bit-representation, a single voxel in the shal-low octree is fully characterised by its bit index. This index
determines the depth of the voxel in the octree and therefore
also the voxel size. Instead of using pointers to the parent
and child nodes, simple arithmetic can be used to retrieve
the corresponding indices of a voxel with bit index i:pa(i) =

i−1
8


,

(1)

ch(i) = 8 · i + 1 .

(2)

In contrast to [31], we associate a data container (for storing
features vectors) with all leaf nodes of each shallow tree.
We allocate the data of a shallow octree in a contiguous dataarray. The offset associated with a particular voxel in thisarray can be computed as follows:
pa(i)−1

data idx(i) = 8

X
j=0

|

bit(j) + 1 −{z#nodes above i

}

i−1
X

bit(j)

j=0

|

{z

}

#split nodes pre i(3)+ mod (i − 1, 8) .
|{z}
offsetHere, mod denotes the modulo operator and bit returns thetree bit-string value at i. See supp. document for an example. Both sum operations can be efficiently implemented
using bit counting intrinsics (popcnt). The data arrays ofall shallow octrees are concatenated into a single contiguous
data array during training and testing to reduce I/O latency.

3.2. Network Operations
Given the hybrid grid-octree data structure introduced in
the previous Section, we now discuss the efficient implementation of network operations on this data structure. We
will focus on the most common operations in convolutional
networks [13, 18, 35]: convolution, pooling and unpooling.
Note that point-wise operations, like activation functions,do not differ in their implementation as they are independent of the data structure.
Let us first introduce the notation which will be usedthroughout this Section. Ti,j,k denotes the value of a 3D
tensor T at location (i, j, k). Now assume a hybrid gridoctree structure with D × H × W unbalanced shallow octrees of maximum depth 3. Let O[i, j, k] denote the valueof the smallest cell in this structure which comprises thevoxel (i, j, k). Note that in contrast to the tensor notation,O[i1 , j1 , k1 ] and O[i2 , j2 , k2 ] with i1 6= i2 ∨ j1 6= j2 ∨ k1 6=
k2 may refer to the same voxel in the hybrid grid-octree,
depending on the size of the voxels. We obtain the index
of the shallow octree in the grid via (b 8i c, b 8j c, b k8 c) and the
local index of the voxel at the finest resolution in that octreeby (mod (i, 8), mod (j, 8), mod (k, 8)).

(a) Standard Convolution(b) Efficient Convolution

Figure 4: Convolution. This figure illustrates the convolution of a 33 kernel (red) with a 83 grid-octree cell (black).
Only 2 of the 3 dimensions are shown. A naı̈ve implementation evaluates the kernel at every location (i, j, k) withina grid-octree cell as shown in (a). This results in ∼14k multiplications for this example. In contrast, (b) depicts ourefficient implementation of the same operation which requires only ∼3k multiplications. As all 83 voxels inside thegrid-octree cell are the same value, the convolution kernel
inside the cell needs to be evaluated only once. Voxels at thecell boundary need to integrate information from neighboring cells. This can be efficiently implemented by summing
truncated kernels. See our supp. document for details.
Given this notation, the mapping from a grid-octree O toa tensor T with compatible dimensions is given by
oc2ten : Ti,j,k = O[i, j, k] .

(4)

Similarly, the reverse mapping is given by
ten2oc : O[i, j, k] = pool voxels (Tī,j̄,k̄ ) ,(5)(ī,j̄,k̄)∈Ω[i,j,k]

where pool voxels (·) is a pooling function (e.g., averageor max-pooling) which pools all voxels in T over the smallest grid-octree cell comprising location (i, j, k), denoted byΩ[i, j, k]. This pooling is necessary as a single voxel in O
can cover up to 83 = 512 elements of T , depending on its
size |Ω[i, j, k]|.
Remark: With the two functions defined above, we could
wrap any network operation f defined on 3D tensors via
g(O) = ten2oc(f (oc2ten(O))) .

(6)

However, this would require a costly conversion from thememory efficient grid-octrees to a regular 3D tensor andback. Besides, storing a dense tensor in memory limits the
maximal resolution. We therefore define our network operations directly on the hybrid grid-octree data structure.
Convolution The convolution operation is the most important, but also the most computational expensive operation in deep convolutional networks. For a single feature

(a) Input(b) Output(a) Input(b) Output

Figure 5: Pooling. The 23 pooling operation on the gridoctree structure combines 8 neighbouring shallow octrees(a) into one shallow octree (b). The size of each voxel ishalved and copied to the new shallow octree structure. Voxels on the finest resolution are pooled. Different shallow
octrees are depicted in different colors.

Figure 6: Unpooling. The 23 unpooling operation transforms a single shallow octree of depth d as shown in (a)into 8 shallow octrees of depth d − 1, illustrated in (b). For
each node at depth zero one shallow octree is spawned. Allother voxels double in size. Different shallow octrees are
depicted in different colors.

map, convolving a 3D tensor T with a 3D convolution kernel W ∈ RL×M ×N can be written as

within each region. Formally, we haveout
Ti,j,k
=

L−1
−1 N
−1
XM
X
X
l=0 m=0 n=0

Wl,m,n · Tî,inĵ,k̂ ,(7)with î = i − l + bL/2c, ĵ = j − m + bM/2c, k̂ = k − n +bN/2c. Similarly, the convolutions on the grid-octree data
structure are defined as
Oout [i, j, k] = pool voxels (Tī,j̄,k̄ )(8)(ī,j̄,k̄)∈Ω[i,j,k]Ti,j,k =

L−1
−1 N
−1
XM
X
X
l=0 m=0 n=0

Wl,m,n · Oin [î, ĵ, k̂] .

While this calculation yields the same result as the tensor
convolution in Eq. (7) with the oc2ten, ten2oc wrapper, weare now able to define a computationally more efficient convolution operator. Our key observation is that for small convolution kernels and large voxels, Ti,j,k is constant withina small margin of the voxel due to its constant support
Oin [î, ĵ, k̂]. Thus, we only need to compute the convolution within the voxel once, followed by convolution along
the surface of the voxel where the support changes due toadjacent voxels taking different values (Fig. 4). This minimizes the number of calculations by a factor of 4 for voxels
of size 83 , see supp. material for a detailed derivation. At
the same time, it enables a better caching mechanism.
Pooling Another important operation in deep convolutional networks is pooling. Pooling reduces the spatial resolution of the input tensor and aggregates higher-level information for further processing, thereby increasing the receptive field and capturing context. For instance, strided
23 max-pooling divides the input tensor T in into 23 nonoverlapping regions and computes the maximum valueout
Ti,j,k=max

l,m,n∈[0,1]inT2i+l,2j+m,2k+n,(9)where T in ∈ R2D×2H×2W and T out ∈ RD×H×W .
To implement pooling on the grid-octree data structure
we reduce the number of shallow octrees. For an input gridoctree Oin with 2D × 2H × 2W shallow octrees, the output
Oout contains D × H × W shallow octrees. Each voxelof Oin is halved in size and copied one level deeper in theshallow octree. Voxels at depth 3 in Oin are pooled. Thiscan be formalized as(
Oin [2i, 2j, 2k] if vxd(2i, 2j, 2k) < 3outO [i, j, k] =
P
else
P =

max

(Oin [2i + l, 2j + m, 2k + n]) ,l,m,n∈[0,1](10)where vxd(·) computes the depth of the indexed voxel inthe shallow octree. A visual example is depicted in Fig. 5.
Unpooling For several tasks such as semantic segmentation, the desired network output is of the same size as the
network input. While pooling is crucial to increase the receptive field size of the network and capture context, it loses
spatial resolution. To increase the resolution of the network, U-shaped network architectures have become popular [2, 52] which encode information using pooling operations and increase the resolution in a decoder part using
unpooling or deconvolution layers [50], possibly in combination with skip-connections [9, 18] to increase precision. The simplest unpooling strategy uses nearest neighbour interpolation and can be formalized on dense input
T in ∈ RD×H×W and output T out ∈ R2D×2H×2W tensors
as follows:outinTi,j,k
= Tbi/2c,bj/2c,bk/2c.

(11)

This operation also changes the data structure: The numberof shallow octrees increases by a factor of 8, as each nodeat depth 0 spawns a new shallow octree. All other nodes
double their size. Thus, after this operation the tree depth isdecreased. See Fig. 6 for a visual example of this operation.
Remark: To capture fine details, voxels can be split again
at the finest resolution according to the original octree ofthe corresponding pooling layer. This allows us to take fulladvantage of skip connections. We follow this approach inour semantic 3D point cloud labeling experiments.

OctNetDenseNet

163

323

Runtime [s]

80
70
60
50
40
30
20
10
03
8

643

Input Resolution

In this Section we leverage our OctNet representation toinvestigate the impact of input resolution on three different
3D tasks: 3D shape classification, 3D orientation estimationand semantic segmentation of 3D point clouds. To isolate
the effect of resolution from other factors we consider simple network architectures. Orthogonal techniques like data
augmentation, joint 2D/3D modeling or ensemble learning
are likely to further improve the performance of our models.
Implementation We implemented the grid-octree data
structure, all layers including the necessary forward andbackward functions, as well as utility methods to create thedata structure from point clouds and meshes, as a standalone C++/CUDA library. This allows the usage of our code
within all existing deep learning frameworks. For our experimental evaluation we used the Torch3 framework.

4.1. 3D Classification
We use the popular ModelNet10 dataset [47] for the 3D
shape classification task. The dataset contains 10 shape categories and consists of 3991 3D shapes for training and 908
3D shapes for testing. Each shape is provided as a triangular mesh, oriented in a canonical pose. We convert thetriangle meshes to dense respective grid-octree occupancy
grids, where a voxel is set to 1 if it intersects the mesh. We
scale each mesh to fit into a 3D grid of (N − P )3 voxels,
where N is the number of voxels in each dimension of the
input grid and P = 2 is a padding parameter.
We first study the influence of the input resolution on
memory usage, runtime and classification accuracy. Towards this goal, we create a series of networks of different
input resolution from 83 to 2563 voxels. Each network consists of several blocks which reduce resolution by half until
we reach a resolution of 83 . Each block comprises two convolutional layers (33 filters, stride 1) and one max-pooling
3 http://torch.ch/

2563

16
14
12
10
8
6
4
2
03
8

(a) Memory

0.92

0.94

0.90
0.88
0.86
83

OctNetDenseNet

163

323

643

1283

2563

1283

2563

Input Resolution(b) Runtime

OctNet 1
OctNet 2
OctNet 3

0.94

0.92

OctNetDenseNet
VoxNet

0.90
0.88
0.86

163

323

643

Input Resolution(c) Accuracy

4. Experimental Evaluation

1283

Accuracy(12)

Memory [GB]Oout [i, j, k] = Oin [bi/2c, bj/2c, bk/2c] .

Accuracy

Again, we can define the analogous operation on the hybrid
grid-octree data structure by

1283

2563

83

163

323

643

Input Resolution(d) Accuracy

Figure 7: Results on ModelNet10 Classification Task.
layer (23 filters, stride 2). The number of feature maps in thefirst block is 8 and increases by 6 with every block. After thelast block we add a fully-connected layer with 512 units anda final output layer with 10 units. Each convolutional layer
and the first fully-connected layer are followed by a rectified linear unit [25] as activation function and the weights
are initialized as described in [17]. We use the standard
cross-entropy loss for training and train all networks for 20
epochs with a batch size of 32 using Adam [24]. The initial
learning rate is set to 0.001 and we decrease the learning
rate by a factor of 10 after 15 epochs.
Overall, we consider three different types of networks:
the original VoxNet architecture of Maturana et al. [29]which operates on a fixed 323 voxel grid, the proposed OctNet and a dense version of it which we denote “DenseNet”
in the following. While performance gains can be obtained using orthogonal approaches such as network ensembles [5] or a combination of 3D and 2D convolutional networks [19,41], in this paper we deliberately focus on “pure”
3D convolutional network approaches to isolate the effect ofresolution from other influencing factors.
Fig. 7 shows our results. First, we compare the memory consumption and run-time of our OctNet wrt. the dense
baseline approach, see Fig. 7a and 7b. Importantly, OctNets
require significantly less memory and run-time for high input resolutions compared to dense input grids. Using abatch size of 32 samples, our OctNet easily fits in a modern
GPU’s memory (12GB) for an input resolution of 2563 . In
contrast, the corresponding dense model fits into the memory only for resolutions ≤643 . A more detailed analysis of
the memory consumption wrt. the sparsity in the data is provided in the supp. document. OctNets also run faster thantheir dense counterparts for resolutions >643 . For resolutions ≤643 , OctNets run slightly slower due to the overhead
incurred by the grid-octree representation and processing.

bathtub 0.92 0.08

83

toilet

sofa

table

monitorsofa
table
toiletn. standdressermonitorn. standtoilet

sofa

table

monitorn. standdesk

dresser

bed

chair

bathtub

0.01
0.01
0.81 0.01
0.02 0.03 0.12
0.01 0.87
0.12
0.01
0.02 0.96 0.01
0.01 0.17
0.77
0.05
0.01
0.02 0.01
0.01 0.95
0.22
0.78
0.01
0.02
0.97
0.99

desk

desk

dresserchair

0.70 0.02
0.09 0.19
dresser
0.87 0.05 0.08
monitor0.03 0.97
n. stand0.01 0.01 0.30
0.58
0.09
sofa
0.04
0.03 0.01
0.92
table
0.10
0.01
0.89
toilet 0.01
0.01
0.01 0.97
desk

163

0.99

bed

0.03

bed

0.08 0.04
0.03 0.01

bathtub

83

0.02

0.95 0.01
0.97

bed
chair

chair

bathtub 0.76 0.10

323

Bathtub

Bed

DresserN. StandFigure 8: Voxelized 3D Shapes from ModelNet10.
Leveraging our OctNets, we now compare the impact
of input resolution with respect to classification accuracy.
Fig. 7c shows the results of different OctNet architectures
where we keep the number of convolutional layers per block
fixed to 1, 2 and 3. Fig. 7d shows a comparison of accuracy
with respect to DenseNet and VoxNet when keeping the capacity of the model, i.e., the number of parameters, constantby removing max-pooling layers from the beginning of the
network. We first note that despite its pooled representation,
OctNet performs on par with its dense equivalent. This confirms our initial intuition (Fig. 1) that sparse data allows forallocating resources adaptively without loss in performance.
Furthermore, both models outperform the shallower VoxNet
architecture, indicating the importance of network depth.
Regarding classification accuracy we observed improvements for lower resolutions but diminishing returns beyond
an input resolution of 323 voxels. Taking a closer look at
the confusion matrices in Fig. 9, we observe that higher input resolution helps for some classes, e.g., bathtub, whileothers remain ambiguous independently of the resolution,e.g., dresser vs. night stand. We visualize this lack of discriminative power by showing voxelized representations of
3D shapes from the ModelNet10 database Fig. 8. Whilebathtubs look similar to beds (or sofas, tables) at low resolution they can be successfully distinguished at higher resolutions. However, a certain ambiguity between dresser andnight stand remains.

4.2. 3D Orientation Estimation
In this Section, we investigate the importance of input
resolution on 3D orientation estimation. Most existing approaches to 3D pose estimation [3, 4, 38, 43, 45] assume thatthe true 3D shape of the object instance is known. To assess

8.0
7.5
7.0
6.5
6.0
5.5
5.0
4.5
4.0
3.5 3
8

OctNet 1
OctNet 2
OctNet 3
163

323

643

1283

Input Resolution

2563(a) Mean Angular Error

5123

Mean Angular Error µ(φ)[◦]643

Mean Angular Error µ(φ)[◦]323

Figure 9: Confusion Matrices on ModelNet10.
14

OctNet

12
10
8
6
163

323

643

Input Resolution

1283(b) Mean Angular Error

Figure 10: Orientation Estimation on ModelNet10.
the generalization ability of 3D convolutional networks, weconsider a slightly different setup where only the object category is known. After training a model on a hold-out set of
3D shapes from a single category, we test the ability of themodel to predict the 3D orientation of unseen 3D shapes
from the same category.
More concretely, given an instance of an object category
with unknown pose, the goal is to estimate the rotation withrespect to the canonical pose. We utilize the 3D shapes fromthe chair class of the ModelNet10 dataset and rotate themrandomly between ±15◦ around each axis. We use the same
network architectures and training protocol as in the classification experiment, except that the networks regress orientations. We use unit quaternions to represent 3D rotations
and train our networks with an Euclidean loss. For small angles, this loss is a good approximation to the rotation angle
φ = arccos(2hq1 , q2 i2 − 1) between quaternions q1 , q2 .

Fig. 10 shows our results using the same naming convention as in the previous Section. We observe that finedetails are more important compared to the classification
task. For the OctNet 1-3 architectures we observe a steadyincrease in performance, while for networks with constantcapacity across resolutions (Fig. 10b), performance levels
beyond 1283 voxels input resolution. Qualitative results of
the latter experiment are shown in Fig. 11. Each row shows
10 different predictions for two randomly selected chair
instance over several input resolutions, ranging from 163
to 1283 . Darker colors indicate larger errors which occur
more frequently at lower resolutions. In contrast, predictions at higher network resolutions cluster around the truepose. Note that learning a dense 3D representation at a resolution of 1283 voxels or beyond would not be feasible.

Average

Overall

IoU

Riemenschneider et al. [37]Martinovic et al. [28]Gadde et al. [12]

68.5

78.6

42.3
52.2
54.4

OctNet 643
OctNet 1283
OctNet 2563

60.0
65.3
73.6

73.6
76.1
81.5

45.6
50.4
59.2

Table 1: Semantic Segmentation on RueMonge2014.

163

323

643

1283

Figure 11: Orientation Estimation on ModelNet10. This
figure illustrates 10 rotation estimates for 3 chair instances
while varying the input resolution from 163 to 1283 . Darker
colors indicate larger deviations from the ground truth.

(a) Voxelized Input(b) Voxel Estimates

4.3. 3D Semantic Segmentation
In this Section, we evaluate the proposed OctNets on the
problem of labeling 3D point cloud with semantic information. We use the RueMonge2014 dataset [37] that provides
a colored 3D point cloud of several Haussmanian style facades, comprising ∼1 million 3D points in total. The labels
are window, wall, balcony, door, roof, sky and shop.
For this task, we train a U-shaped network [2, 52] on
three different input resolutions, 643 , 1283 and 2563 , wherethe voxel size was selected such that the height of all buildings fits into the input volume. We first map the point cloud
into the grid-octree structure. For all leaf nodes which contain more than one point, we average the input features andcalculate the majority vote of the ground truth labels fortraining. As features we use the binary voxel occupancy, theRGB color, the normal vector and the height above ground.
Due to the small number of training samples, we augment
the data for this task by applying small rotations.
Our network architecture comprises an encoder and a decoder part. The encoder part consists of four blocks which
comprise 2 convolution layers (33 filters, stride 1) followed
by one max-pooling layer each. The decoder consists offour blocks which comprise 2 convolutions (33 filters, stride
1) followed by a guided unpooling layer as discussed in theprevious Section. Additionally, after each unpooling step allfeatures from the last layer of the encoder at the same resolution are concatenated to provide high-resolution details.
All networks are trained with a per voxel cross entropy loss
using Adam [24] and a learning rate of 0.0001.
Table 1 compares the proposed OctNet to several stateof the art approaches on the facade labeling task following
the extended evaluation protocol of [12]. The 3D points of(c) Estimated Point Cloud (d) Ground Truth Point Cloud

Figure 12: OctNet 2563 Facade Labeling Results.
the test set are assigned the label of the corresponding gridoctree voxels. As evaluation measures we use overall pixelTPaccuracy TP+FN
over all 3D points, average class accuTP
over all classes.
racy, and intersection over union TP+FN+FP
Here, FP, FN and TP denote false positives, false negatives
and true positives, respectively.
Our results clearly show that increasing the input resolution is essential to obtain state-of-the-art results, as finer
details vanish at coarser resolutions. Qualitative results for
one facade are provided in Fig. 12. Further results are provided in the supp. document.

5. Conclusion and Future Work
We presented OctNet, a novel 3D representation whichmakes deep learning with high-resolution inputs tractable.
We analyzed the importance of high resolution inputs on
several 3D learning tasks, such as object categorization,pose estimation and semantic segmentation. Our experiments revealed that for ModelNet10 classification lowresolution networks prove sufficient while high input (andoutput) resolution matters for 3D orientation estimation and
3D point cloud labeling. We believe that as the community
moves from low resolution object datasets such as ModelNet10 to high resolution large scale 3D data, OctNet will
enable further improvements. One particularly promisingavenue for future research is in learning representations for
multi-view 3D reconstruction where the ability to process
high resolution voxelized shapes is of crucial importance.

References[1] N. S. Alvar, M. Zolfaghari, and T. Brox. Orientation-boosted
voxel nets for 3d object recognition. arXiv.org, 1604.03351,
2016. 2[2] V. Badrinarayanan, A. Kendall, and R. Cipolla. Segnet: Adeep convolutional encoder-decoder architecture for image
segmentation. arXiv.org, 1511.00561, 2015. 5, 8
[3] E. Brachmann, A. Krull, F. Michel, S. Gumhold, J. Shotton,
and C. Rother. Learning 6d object pose estimation using
3d object coordinates. In Proc. of the European Conf. on
Computer Vision (ECCV), 2014. 7[4] E. Brachmann, F. Michel, A. Krull, M. Y. Yang, S. Gumhold,and C. Rother. Uncertainty-driven 6d pose estimation of objects and scenes from a single rgb image. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2016. 7[5] A. Brock, T. Lim, J. M. Ritchie, and N. Weston. Generativeand discriminative voxel modeling with convolutional neuralnetworks. arXiv.org, 1608.04236, 2016. 2, 6
[6] A. X. Chang, T. A. Funkhouser, L. J. Guibas, P. Hanrahan,
Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su,
J. Xiao, L. Yi, and F. Yu. Shapenet: An information-rich 3dmodel repository. arXiv.org, 1512.03012, 2015. 1[7] S. Choi, Q. Zhou, S. Miller, and V. Koltun. A large datasetof object scans. arXiv.org, 1602.02481, 2016. 1[8] C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese. 3dr2n2: A unified approach for single and multi-view 3d object
reconstruction. In Proc. of the European Conf. on Computer
Vision (ECCV), 2016. 1, 2
[9] A. Dosovitskiy, P. Fischer, E. Ilg, P. Haeusser, C. Hazirbas,
V. Golkov, P. v.d. Smagt, D. Cremers, and T. Brox. Flownet:
Learning optical flow with convolutional networks. In Proc.
of the IEEE International Conf. on Computer Vision (ICCV),
2015. 5[10] M. Engelcke, D. Rao, D. Z. Wang, C. H. Tong, and I. Posner.
Vote3deep: Fast object detection in 3d point clouds using efficient convolutional neural networks. arXiv.org, 609.06666,2016. 2[11] J. Flynn, I. Neulander, J. Philbin, and N. Snavely. Deepstereo: Learning to predict new views from the world’s imagery. arXiv.org, 1506.06825, 2015. 2[12] R. Gadde, V. Jampani, R. Marlet, and P. V. Gehler. Efficient 2d and 3d facade segmentation using auto-context.
arXiv.org, 1606.06437, 2016. 8[13] G. Ghiasi and C. C. Fowlkes. Laplacian pyramid reconstruction and refinement for semantic segmentation. In Proc. of
the European Conf. on Computer Vision (ECCV), 2016. 1, 2,
4[14] R. Girdhar, D. F. Fouhey, M. Rodriguez, and A. Gupta.
Learning a predictable and generative vector representationfor objects. In Proc. of the European Conf. on Computer
Vision (ECCV), 2016. 2[15] B. Graham. Spatially-sparse convolutional neural networks.
arXiv.org, 2014. 2[16] B. Graham. Sparse 3d convolutional neural networks. In
Proc. of the British Machine Vision Conf. (BMVC), 2015. 2[17] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into
rectifiers: Surpassing human-level performance on imagenet
classification. In Proc. of the IEEE International Conf. on
Computer Vision (ICCV), 2015. 6[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learningfor image recognition. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2016. 1, 2, 4, 5
[19] V. Hegde and R. Zadeh. Fusionnet: 3d object classification
using multiple data representations. arXiv.org, 1607.05695,
2016. 6[20] J. Huang and S. You. Point cloud labeling using 3d convolutional neural network. In Proc. of the International Conf. on
Pattern Recognition (ICPR), 2016. 2[21] Q. Huang, H. Wang, and V. Koltun. Single-view reconstruction via joint analysis of image and shape collections. In
ACM Trans. on Graphics (SIGGRAPH), 2015. 1[22] V. Jampani, M. Kiefel, and P. V. Gehler. Learning sparse highdimensional filters: Image filtering, dense crfs and bilateralneural networks. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2016. 2[23] W. Kehl, T. Holl, F. Tombari, S. Ilic, and N. Navab. An
octree-based approach towards efficient variational range
data fusion. arXiv.org, 1608.07411, 2016. 3[24] D. P. Kingma and J. Ba. Adam: A method for stochastic
optimization. In Proc. of the International Conf. on Learning
Representations (ICLR), 2015. 6, 8
[25] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
Advances in Neural Information Processing Systems (NIPS),
2012. 6[26] S. Laine and T. Karras. Efficient sparse voxel octrees. IEEE
Trans. on Visualization and Computer Graphics (VCG),
17(8):1048–1059, 2011. 3[27] Y. Li, S. Pirk, H. Su, C. R. Qi, and L. J. Guibas. FPNN: fieldprobing neural networks for 3d data. arXiv.org, 1605.06240,
2016. 2[28] A. Martinović, J. Knopp, H. Riemenschneider, and L. Van
Gool. 3d all the way: Semantic segmentation of urban scenes
from start to end in 3d. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2015. 8[29] D. Maturana and S. Scherer. Voxnet: A 3d convolutional
neural network for real-time object recognition. In Proc.
IEEE International Conf. on Intelligent Robots and Systems(IROS), 2015. 1, 2, 6[30] D. Meagher. Geometric modeling using octree encoding. Computer Graphics and Image Processing (CGIP),
19(1):85, 1982. 3[31] A. Miller, V. Jain, and J. L. Mundy. Real-time renderingand dynamic updating of 3-d volumetric data. In Proc. of
the Workshop on General Purpose Processing on Graphics
Processing Units (GPGPU), page 8, 2011. 2, 3, 4[32] F. Milletari, N. Navab, and S. Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. arXiv.org, 1606.04797, 2016. 2[33] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux,D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, andA. Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In Proc. of the International Symposiumon Mixed and Augmented Reality (ISMAR), 2011. 1[34] C. R. Qi, H. Su, M. Nießner, A. Dai, M. Yan, and L. Guibas.
Volumetric and multi-view cnns for object classification on
3d data. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2016. 1[35] S. Ren, K. He, R. B. Girshick, and J. Sun. Faster R-CNN:
towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NIPS), 2015. 1, 2, 4[36] D. J. Rezende, S. M. A. Eslami, S. Mohamed, P. Battaglia,
M. Jaderberg, and N. Heess. Unsupervised learning of 3d
structure from images. arXiv.org, 1607.00662, 2016. 2[37] H. Riemenschneider, A. Bódis-Szomorú, J. Weissenberg,and L. V. Gool. Learning where to classify in multi-view
semantic segmentation. In Proc. of the European Conf. on
Computer Vision (ECCV), 2014. 8[38] R. Rios-Cabrera and T. Tuytelaars. Discriminatively trained
templates for 3d object detection: A real-time scalable approach. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), pages 2048–2055, 2013. 7[39] A. Sharma, O. Grau, and M. Fritz. Vconv-dae: Deep volumetric shape learning without object labels. arXiv.org,
1604.03755, 2016. 2[40] S. Song and J. Xiao. Deep sliding shapes for amodal 3d
object detection in RGB-D images. arXiv.org, 1511.02300,
2015. 2[41] H. Su, S. Maji, E. Kalogerakis, and E. G. Learned-Miller.
Multi-view convolutional neural networks for 3d shaperecognition. In Proc. of the IEEE International Conf. on
Computer Vision (ICCV), 2015. 2, 6[42] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Multi-view
3d models from single images with a convolutional network.
In Proc. of the European Conf. on Computer Vision (ECCV),
2016. 2[43] A. Tejani, D. Tang, R. Kouskouridas, and T. Kim. Latentclass hough forests for 3d object detection and pose estimation. In Proc. of the European Conf. on Computer Vision(ECCV), 2014. 7[44] A. O. Ulusoy, A. Geiger, and M. J. Black. Towards probabilistic volumetric reconstruction using ray potentials. In
Proc. of the International Conf. on 3D Vision (3DV), 2015. 3[45] P. Wohlhart and V. Lepetit. Learning descriptors for object
recognition and 3d pose estimation. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2015. 7[46] J. Wu, T. Xue, J. J. Lim, Y. Tian, J. B. Tenenbaum, A. Torralba, and W. T. Freeman. Single image 3d interpreter network. In Proc. of the European Conf. on Computer Vision(ECCV), 2016. 2[47] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, andJ. Xiao. 3d shapenets: A deep representation for volumetricshapes. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2015. 1, 2, 6
[48] J. Xie, R. B. Girshick, and A. Farhadi. Deep3d: Fully automatic 2d-to-3d video conversion with deep convolutional
neural networks. In Proc. of the European Conf. on Computer Vision (ECCV), 2016. 2[49] A. R. Zamir, T. Wekel, P. Agrawal, C. Wei, J. Malik, and
S. Savarese. Generic 3d representation via pose estimationand matching. In Proc. of the European Conf. on Computer
Vision (ECCV), 2016. 2[50] M. D. Zeiler, G. W. Taylor, and R. Fergus. Adaptive deconvolutional networks for mid and high level feature learning.
In Proc. of the IEEE International Conf. on Computer Vision(ICCV), 2011. 5[51] Y. Zhang, M. Bai, P. Kohli, S. Izadi, and J. Xiao. Deepcontext: Context-encoding neural pathways for 3d holistic scene
understanding. arXiv.org, 1603.04922, 2016. 2[52] Özgün Cicek, A. Abdulkadir, S. S. Lienkamp, T. Brox, andO. Ronneberger. 3d u-net: Learning dense volumetric segmentation from sparse annotation. arXiv.org, 1606.06650,
2016. 2, 5, 8