Room Reconstruction from a Single Spherical
Image by Higher-order Energy Minimization
Kosuke Fukano∗ , Yoshihiko Mochizuki∗ , Satoshi Iizuka∗ , Edgar Simo-Serra∗ , Akihiro Sugimoto† ,and Hiroshi Ishikawa∗∗ Waseda University, Tokyo, Japan

Email: ich38940@ruri.waseda.jp, {motchy,iizuka,esimo}@aoni.waseda.jp, hfs@waseda.jp† National Institute of Informatics, Tokyo, Japan
Email: sugimoto@nii.ac.jp

Abstract—We propose a method for understanding a roomfrom a single spherical image, i.e., reconstructing and identifying
structural planes forming the ceiling, the floor, and the walls in aroom. A spherical image records the light that falls onto a single
viewpoint from all directions and does not require correlatinggeometrical information from multiple images, which facilitatesrobust and precise reconstruction of the room structure. In our
method, we detect line segments from a given image, and classify
them into two groups: segments that form the boundaries ofthe structural planes and those that do not. We formulate thisproblem as a higher-order energy minimization problem thatcombines the various measures of likelihood that one, two, or
three line segments are part of the boundary. We minimize theenergy with graph cuts to identify segments forming boundaries,
from which we estimate structural the planes in 3D. Experimental
results on synthetic and real images confirm the effectiveness ofthe proposed method.

II. R ELATED W ORK
A similar method that try to estimate the structure from asingle image without using stereo or structure from motion
is [2]. Similar to our method, it infers the structure of theroom from the line segments detected in the input image. They
however use the images from an ordinary camera, making it
impossible to recover the whole room in a single image. In
contrast, our method not only captures the whole room in the
single image but also takes advantage of the unique geometryof the spherical image to reconstruct the walls.
Y. Zhang et al. [3] uses panorama images to find a 3D
bounding boxes of a room and objects therein. However, their
method is much more elaborate than ours and involves learningfrom a dataset of annotated panorama images. In contrast, ourmethod does not use any training set and infer the geometryof the room solely from a single spherical image.

I. I NTRODUCTION
III. P ROPOSED M ETHOD
We present a method to reconstruct (i.e., to identify thefloor, the ceiling, and the walls of) a simple room in a single
spherical image. An example of the input image is shown in
Fig. 1. We aim to reconstruct the most basic structure of thewhole room such as the walls, even in a cluttered image.
In the common 3D reconstruction technique like stereo andstructure from motion, establishing a correspondence between
the points in multiple images is necessary. In the clutteredindoor scenes, it is often difficult to accurately determine thecorrespondence due to the texture on the wall and obstructing
objects like furnitures [1]. Moreover, because such methods
that depend on multiview input need that each point in thescene to appear in at least two images from different angles,
it is complicated to cover the whole room with the necessary
number of images with adequate parallax.
In our method, we first detect line segments in the input
image. The aim is to determine the borders of the floor, theceiling, and the walls of the room among the detected line
segments. Since the detected line segments include those thatare not part of the borders, we classify them into the edges andother line segments using higher-order energy minimization.
Finally, we infer the planes that form the room (i.e., the ceilingetc.) from the border line segments.

In this work, we use a single spherical image taken indoors
as the input, and reconstruct the shape of the room. We
assume the room and the spherical image satisfy the following
conditions.
The room is roughly a cuboid, consisting of six rectangular planar walls (including the floor and the ceiling).
• All the six planes and their intersecting lines appear atleast partially in the image.
•Fig. 2 depicts the planes and the line segments that constitutethe room.
The objective here can be thought of as dividing the image
into regions corresponding to the six walls of the room. For
that, we solve the problem of identifying the spherical line
segments on the unit sphere that are borders of the walls.
Fig. 3 illustrates the flow of the process. The method consistsof the following three steps.
1) We detect line segments in the input image. Fig. 4
illustrates the process of detecting line segments inspherical images. First, we project the input image onto
six planar images. Then, we detect line segments in theplanar images using the method by Matas et al. [4].
Finally, we project the detected line segments back onto

(a) Input image(b) Line segment detection(c) Line segment extraction(d) Planes estimation

Fig. 1. An example input image. Left: A half of the spherical image. Right:
the whole image projected onto a rectangle.
Line Segment

Spherical line segment

Fig. 3. The flow of the proposed method. (a) Input spherical image. (b) Detect
line segments. The detected line segments are shown in black. (c) Classify
the detected line segments into those which are borders of the walls (S ∗ )and all others, such as lines drawn on the wall. The line segments in S ∗ areshown in red. Other line segments are shown in black. (d) Using the line
segments in S ∗ , infer the six plane walls (including the floor and the ceiling)that constitute the room. Walls facing each other are shown in the same color.

Plane

Fig. 2. Planes and line segments that constitute the room.

the unit sphere. We denote the set of the detected line
segments on the sphere by S = {s1 , s2 , · · · , sn }.
2) We identify the subset S ∗ ⊂ S of line segments
corresponding to the wall borders. Since S contains
line segments other than the wall borders, such as thosecaused by wall texture and objects present in the room,
we classify the line segments into S ∗ and S\S ∗ . We treat
this as a labeling problem and solve it by minimizing ahigher-order energy.
3) We infer the six walls using S ∗ .
We explain the second and the third steps in more detail.
A. Line Segments Extraction
To classify the line segments into S ∗ and S\S ∗ , we treat thisas a labeling problem and minimize a higher-order energy. Letn be the number of the line segments in S. We denote the i’thline segment by si and its length by |si |. Here, we talk about
the line segments on the unit sphere, i.e., parts of great circles.
So |si | is actually the length of an arc on the unit sphere. The
labeling L = (li )i=1,...,n of the line segments determine which
ones belong to S ∗ . That is, the label li is 1 if the line segment
si is in S ∗ and 0 if it is not. We define an energy function
on L such that it has smaller values when it is more likelythat the labeling indicates the correct set of line segments inS ∗ . By finding the labeling that minimize this energy function,
we infer the set S ∗ of line segments that constitutes the wallborders. For minimization, we use the graph-cut algorithm forhigher-order energies in [5]. The energy is a weighted sum of
five potential functions:
E(L) =

wlength Elength (L) + wcollinear Ecollinear (L)+ wcross Ecross (L) + wcorner Ecorner (L)+ wplane Eplane (L).

(1)(a) Spherical image(b) Perspective image(c) Detected line segments(d) Spherical line segments

Fig. 4. Detecting line segments in a spherical image. (a) A spherical image.
(b) Project the input image onto six planar images. (c) Detect line segmentsin the planar images. (d) Project the detected segments back onto the unit
sphere.

The potentials have the following meaning: Elength is about
the length of the segments, Ecollinear is about pairs of segmentson the same great circle, Ecross is about pairs of intersecting
segments, Ecorner is about triples of segments that form acorner, and Eplane is about quadruples of segments that forma rectangle. Next, we discuss each of these potentials.
1) Potential Regulating Length: Since there are various
objects and textures in the scene, line segments with various lengths are detected. Many of shorter segments tend tooriginate from objects and textures. We can threshold theminimum length, but it is conceivable that some segments
appear short even though they are actually long borders of
the walls because of various reasons, such as occlusion by
objects. Instead, we regulate the length of the line segment
in a soft way by introducing a potential function Elength (L)that encourages longer segments to be labeled as a part of thewall borders by giving them lower values. It is given by thefollowing:
Elength (L)=nXflength (li )(2)Fig. 5. Crossing segments and non-crossing segments that are boundary of
the walls. The two line segments in the yellow box on the left cross eachother. The ones in the cyan box on the right do not, and are borders of walls.

i=1


flength (li )10=(li = 1 ∧ |sπi | < dlength )(otherwise)

where 0 < dlength < 1 is a threshold.
2) Potential Measuring Collinearity: In detecting line segments in the spherical image, in some cases two detected
segments are actually parts of one segment, because of an
occlusion by objects, because of the lighting condition, or
because the segment does not fit in one of the six projected
planes. Let si and sj be two detected line segments that are
actually one segment. As such, their labels should always
coincide. Thus, we introduce a potential that lowers the energy
when such a pair are labeled the same. Let C be the set ofpairs of indices of line segments that can be regarded to forma single segment. We define Ecollinear as follows:
X
Ecollinear (L) =fcollinear (li , lj )(3)

Rectangular corner

𝑇𝑗

𝑇𝑖
𝑠𝑗

𝑠𝑖

𝑇𝑘

𝑠𝑘

Fig. 6. A perpendicular vertex and its line segments that constitute a corner.

(i,j)∈C
fcollinear (li , lj )

=

−eij
0

(li = lj )(otherwise)

Here, eij is a positive value that becomes larger when the
distance between the line segments si and sj becomes smaller.
3) Potential Discouraging Segments Crossing: The line
segments that are borders or boundaries of two walls should
not look crossing another line segment. Examples of suchcrossing and non-crossing are shown in Fig. 5. Thus we
introduce a potential that raise the energy value when crossing
segments are labeled as borders of walls. Let F be the set ofpairs of indices of line segments that cross each other. Let dij
be the minimum of the distances between the crossing pointand either endpoint of si or sj . We define Ecross using F and
dij as follows:
X
Ecross (L) =fcross (li , lj )(4)

the projection of T1 , T2 , and T3 onto the unit sphere. Then s1 ,s2 , and s3 are called the line segments that constitute a corner.
Fig. 6 shows a perpendicular vertex and its line segments that
constitute a corner. When s1 , s2 , and s3 are line segments that
constitute a corner, they are likely to be in S ∗ . Thus we add apotential that encourage such triple to have label 1. First, let
q1 , q2 , and q3 be the intersections (points on the unit sphere) of
the three pairs of line segments out of s1 , s2 , and s3 . Let g be
the barycenter of the three points and dijk the average distance
between g and q1 , q2 , and q3 . Then the potential Ecorner (L)is defined as follows:
X
Ecorner (L) =fcorner (li , lj , lk )(6)(i,j,k)∈Afcorner (li , lj , lk )=(li ,lj )∈Ffcross (li , lj ) =dij0(li = lj = 1)
(otherwise)(5)4) Potential Assessing Three Segments Forming A Corner:
Consider three planes P1 , P2 , and P3 in the 3D space such thateach pair among them are perpendicular to each other. Let T1
be the intersection of P2 and P3 , T2 the intersection of P1 andP3 , and T3 the intersection of P1 and P2 . The segments T1 ,T2 , and T3 intersect perpendicularly at one point, called theperpendicular vertex, in the 3D space [6]. Let s1 , s2 , and s3 be− cos(dijk ) (li = lj = lk = 1)0(otherwise)5) Potential Estimating Planes: The potential Eplane (L)estimates how likely sets of four line segments form planes.
It lowers the energy if the four are likely to form a rectangle
and they are all labeled 1. It is defined as follows:X
Eplane (L) =fplane (lh , li , lj , lk )(7)(h,i,j,k)∈Vfplane (lh , li , lj , lk )=−rplane0(lh = li = lj = lk = 1)
(otherwise)Fig. 7. Four line segments that can be part of a rectangle (left) and thatcannot (right).

Here, V is the set of quadruples of line segments that canpossibly form a rectangle in the 3D space. This is determinedby examining necessary conditions on spatial relationships
between the segments. We refer to [7] and omit the details
here because of the lack of space. Fig. 7 illustrates one suchcondition. Also, rplane is the ratio of the sum of the lengthsof the four segments to the perimeter of the rectangle that canbe formed, projected on the unit sphere.
B. Inferring the Walls∗

Using the set S of line segments likely to form the bordersof the walls, we infer the planes that constitute the ceiling, thefloor, and the walls in the following steps:
1) Generate the set Q of candidate quadruples of line
segments that can form planes.
2) Eliminate the candidates that cannot constitute the room,i.e., those that cannot be the four borders of a floor, aceiling, or a wall of the room.
To generate the candidates, we use the same necessary conditions [7] on spatial relationships between the segments weused in §III-A5. The set Q includes both planes that constitutethe room (i.e., are either the ceiling, the floor, or the walls)
and those do not. Examples of both cases are shown in Fig. 8.
Here, planes that constitute the room have common borders
with other such planes. Therefore, if any of the borders of Q
is not also an border of another plane in Q, Q can be excluded
as the candidate. We eliminate the planes that do not constitutethe room as follows:
1) For Q ∈ Q, determine if a plane in Q with a common
border with Q can be found for each of Q’s borders.
2) If they cannot be found, remove Q from Q.
3) Repeat 1 and 2 for all Q ∈ Q.
4) Repeat 1 through 3 Until the number of the elements of
Q no longer decreases.
We take the set Q after this process as the set of the planes
constituting the room. Finally, we cluster the normal directionof the remaining planes with k-means with k = 6.
C. Reconstructing the Room
To obtain the whole room shape as a cuboid, we can utilize
the method in Y. Zhang et al. [3]. The algorithm uses thesegments corresponding to each edge of a cuboid to construct

Fig. 8. Examples of planes that constitute the room (upper row) and thosethat do not (lower row).

a set of constraints for a least-square problem. Since we know
the correspondence between Q and the faces of a cuboid, we
can supply the necessary information to the algorithm. The
method also needs the axes of vanishing point directions towhich the cuboid should be aligned. Thus we need to estimate
the vanishing points corresponding to the input image by someexisting method [8].
IV. E XPERIMENTAL R ESULTS
Here, we describe the experiments and their results weperformed to evaluate our method. Fig. 9 shows the two
input images with resolution 3584 × 1792 pixels (left) and
2048 × 1024 pixels (right), respectively. Fig. 10 shows thedetected line segments in blue and the selected lines by energy
minimization in red.
The weights were automatically chosen by trying different
combinations and then choosing the one that resulted in the
fewest line segments, out of the combinations that the final
wall inference does not fail (by ending up with empty Q).
For the left image, 314 segments were detected, whichcontain all the wall borders. The automatically chosen weights
for the minimization were wlength = 1, wcollinear = 5,
wcross = 100, and wcorner = 6. After the energy minimization,
128 segments were selected as possible borders. Then, 29792
plane candidates were generated, and 1691 planes out of themremained after the wall inference. The planes are clustered into
six directions as shown in Fig. 11.
The right image is the more complicated case. The 551
detected segments include the edges of the table, a woman, andher hand. The automatically chosen weights were wlength = 3,
wcollinear = 1, wcorner = 74, and wcorner = 3. Afterthe energy minimization, 219 segments were selected. Then,
42721 plane candidates were generated, and 135 planes outof them remained after the wall inference. Fig. 12 shows theeliminated candidates.
In both cases, we correctly identified all the six planes.
Finally, we applied the method [3] to estimate the cuboid foreach set of candidates as shown in Fig. 13.
A. Analysis of the weight for Ecross
Changing the weight wcross of Ecross , we computed thePrecision and Recall for the labeling for the right image. OtherFig. 9. Two input images.

Fig. 12. Eliminated candidates of the ceiling, floor, and walls.

Fig. 10. Detected line segments (top) and classification results by energy
minimization (bottom).

weights used were: wlength = 1, wcollinear = 100, wcorner = 6,
and wplane = 0.
Table I shows the classification accuracy by energy minimization with varying wcross . The ground truth for the accuracy calculation is manually annotated as shown in Fig. 14.
Fig. 15 shows the corresponding results.
Discussion. From Table I and Fig. 15, it can be seen that whenwcross is small most segments are classified to be wall borders,
while the accuracy is the highest when wcross = 1500. This
indicates that Ecross is effective. On the other hand the Recall
suffers when wcross is too big.

Fig. 13. Final results.

B. Degenerate case
For some cases, the wall inference may fail, i.e., the set ofplanes Q becomes empty. This is because insufficient segments
are detected to reconstruct the room or the classification result
is not accurate enough.
Fig. 16 is the classification result of the detected segments

TABLE IC LASSIFICATION ACCURACY WITH VARYING wcross .

Fig. 11. Eliminated candidates of the ceiling, floor, and walls.

wcross

0

100

500

1000

1500

2000

2500

Precision
Recall

0.28
1.00

0.31
1.00

0.54
0.98

0.69
0.98

0.94
0.98

0.91
0.69

0.20
0.01

Fig. 14. Ground truth for the accuracy evaluation. Left: Detected line
segments. Right: Those that are actually part of the wall borders are shownin red.

(a) wcross =0(b) wcross =100
Fig. 17. Eliminated candidates at one step before the algorithm failed.

(c) wcross =500(d) wcross =1000(e) wcross =1500(f) wcross =2000

Fig. 15. Classification results with varying wcross . Those segments that werelabeled 1 (i.e., as wall borders) are shown in red.

for an image. In this case, the inference failed. However, anintermediate Q roughly forms a cuboid without the most ofoutliers as shown in Fig. 17. We can also apply the cuboid
estimation method [3] to get the approximate result (Fig. 18).
V. C ONCLUSION
We present a method to reconstruct a simple room from asingle spherical image. We first detect line segments, which weclassify into borders and others using higher-order energy minimization. Then we infer the planes that constitute the roomfrom the boundary line segments. We correctly determined thestructure of the rooms in experiments with real images. Future
work would include loosening the assumption that we imposed
here.

Fig. 16. Detected segments and the classification result by energy minimization.

Fig. 18. The estimated cuboid.

ACKNOWLEDGMENT
This work was partially supported by JST CREST.
R EFERENCES[1] R. Cabral and Y. Furukawa, “Piecewise planar and compact floorplan
reconstruction from images,” in IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), Jun. 2014.
[2] D. C. Lee, M. Hebert, and T. Kanade, “Geometric reasoning for single
image structure recovery,” in IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), Jun. 2009.
[3] Y. Zhang, S. Song, P. Tan, and J. Xiao, “PanoContext: A whole-room 3dcontext model for panoramic scene understanding,” in The 13th European
Conference on Computer Vision (ECCV), Sep. 2014.
[4] J. Matas, C. Galambos, and J. Kittler, “Robust detection of lines using the
progressive probabilistic Hough transform,” Computer Vision and Image
Understanding, vol. 78, no. 1, pp. 119 – 137, 2000.
[5] H. Ishikawa, “Transformation of general binary MRF minimization to
the first-order case,” IEEE Transactions on Pattern Analysis and MachineIntelligence, vol. 33, no. 6, pp. 1234–1249, 2011.
[6] K. Kanatani, Group-Theoretical Methods in Image Understanding, ser.
Springer Series in Information Science. Springer Berlin Heidelberg,
1990, vol. 20.
[7] H. Kato and M. Billinghurst, “Marker tracking and HMD calibration fora video-based augmented reality conferencing system,” in the 2nd IEEEand ACM International Workshop on Augmented Reality, 1999, pp. 85–.
[8] C. Rother, “A new approach for vanishing point detection in architecturalenvironments,” in The British Machine Vision Conference (BMVC), 2000.