Auto-Framing Based on User Camera Movement(a)Tomoya SAWADA

Masahiro TOYOURA

Xiaoyang MAO

Interdisciplinary Graduated School,
University of YAMANASHI

Interdisciplinary Graduated School,
University of YAMANASHI

Interdisciplinary Graduated School,
University of YAMANASHI

An image generated by stitching the frames
obtained during the user’s camera movement.

(b)Result of aspect ratio 16:9.

(c)Result of aspect ratio
3:2.

(d)

Result of aspect ratio
1:1

Figure 1: Optimal composition search results for an image at different aspect ratios

ABSTRACT
We propose a novel approach to assisting users with searching the
optimal composition of a photograph. In existing studies, theprocess of detecting the object in a given photo occurred via only
image processing, however the result does not always include theobject of user’s interest. A major technique contribution of ourapproach is to exploit the user’s motion to understand the user’s
subjective interest in a scene. User’s subjective interest andobjective structure information of the scene are combined toestimate the best composition based on aesthetic measures. We
named this system Auto-Framing. The evaluation result showsthat estimated optimal composition closes to the ground-truth. We
will embed our technique in an actual camera to enable bothautomatic detection of compositions and real-time guidance
functionality. 1

CCS CONCEPTS
・Computing methodologies → Computational photography

KEYWORDS
Auto-Framing, optimal composition, user’s interest estimation,
spatio-temporal analysis, saliency map,

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the fullcitation on the first page. Copyrights for components of this work owned by others
than ACM must be honored. Abstracting with credit is permitted. To copy otherwise,or republish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from Permissions@acm.org.
CGI '17, June 27-30, 2017, Yokohama, Japan© 2017 Association for Computing Machinery.
ACM ISBN 978-1-4503-5228-4/17/06…$15.00
http://dx.doi.org/10.1145/3095140.3095158

1

INTRODUCTION

Thanks to the advance of computer vision and image
processing techniques, anyone can take a photograph ofreasonable quality with ease. For example, equippedwith the modern face detection technique, many digital
cameras provide the face-priority auto focus function,
which can set the focus and appropriate exposure
automatically. In addition to setting the focus, thecomposition is yet another important factor to beconsidered when taking a photo. To the best of ourknowledge, currently there are still no effective
techniques for helping users to obtain photos with
optimal composition, though some methods have been
proposed for post-processing a photograph to have thebest composition. Understanding how to compose
photos generally requires considerable experience and arefined aesthetic taste. For regular users, then,composing photos well can be a challenge. The
composition of a photo also depends on the user’s
intended subject. For this study, we focused on how
users move the camera, which is a form of interaction
between the user and the target scene when attempting tocompose a photo. Using camera movement to estimate
the user’s subject, the proposed auto-framing techniquededuces the subject-image context and automatically
captures photos with the optimal composition under an
objective aesthetic measure. Users naturally move theircameras around when preparing to take photos. By

basing our technique around that instinctive process, weestablish a system that enables well-composed
photography without placing any additional burden on
the user. Figure 1 demonstrated the results generated
with the proposed method. Figure 1(a) is the master
image generated by stitching all the frames obtained
during user’s camera movement. Figure 1(b) to (d) are
the results of optimal composition for given aspect ratios.

2 RELATED RESEARCH
One of the earliest techniques related to the photocomposition was the method by Nielsen et al. [1], asystem that detected undesirable artifacts in digital
photos (lens obstructions by fingers and straps, forexample) and automatically cropped the images
accordingly. However, the cropping process did notaccount for photo composition. While Nielsen et al.
focused on identifying and eliminating artifacts, ourstudy defines auto-framing as a technique for estimating
the user’s intended subject, determining a composition
that places the subject in the optimal position undersome given aesthetic measure, and then automatically
cropping the image. While our study aims to develop atool to assist users taking a photo of ideal composition,
several recent studies used data driven approach forpost-processing photographs to have optimalcompositions ([2]–[4]). Nishiyama et al. [2], for example,
designed a technique for learning photo quality and thendetermining. They prepared a large database of photos
with manually provided aesthetic quality scores, createdan evaluation function for classifying photo quality via
learning processes, trimmed the input image at different
scales and sizes, and used the evaluation function toassign a quality score to each sampling image. The
system then returned the sampling image with the
optimal value as the photo with a good composition.
Yan et al. [3] also proposed a method for learning
optimized photo cropping position with a goodcomposition using large photo database. The learning
process is able to understand a scene of photo and theirapproach can apply to a new photo by extracting andcombining foreground information, intensity difference,
texture difference, isolation of foreground and so on.
Liu et al. [4] also proposed a method that searched for
optimal compositions via an evaluation method
combining visual balance, the rule of thirds, anddiagonal dominance. The Liu et al. method used saliency
2

map to estimate the main subject of the target image and,based on the saliency results, divided the image into
regions to obtain straight-line segments of regionalboundaries. To create a distribution map of theseimportant components, the researchers used the threeaesthetic guidelines to determine evaluation values forthe composition of the image, trim the image at random
locations and different scales, and repeat the trimming
process until arriving at the optimal composition.
Fang et al. [5] also uses saliency map to evaluate avisual appearance. In their approach, to integratesaliency map and edge information with manuallyweight can find good composition. The saliency map
technique allows users to estimate physical locations invideo footage and still imagery that attract the viewer’s
interest. Guo et al. [6] proposed a method forrecomposing a photo to align the main subject with therule of thirds. The approach involved detecting the
subject in an image via saliency map and then estimating
the optimal position of the subject using a rule of thirds
based scale. The researchers found inconspicuous seams
at the edges of the target subject and applied several
iterations of the Seam Carving process [7] to move thesubject to its optimal location.
In these past studies, the process of detecting the subject
in a given photo occurred via image processing
techniques after the corresponding user took the input
photos; the disconnect between the subject-detection
step and the actual shooting process made it quitepossible that the system’s subject-detection “result”
would not match the user’s actual subjective target. The
proposed method operates on a different definition of thesubject. Instead of assuming the subject of a photo to be
the salient region, our process focuses on estimating thephotographer’s subjective target subject during theshooting process. Therefore, the proposed method has
two key components. First, operating on the assumptionthat the user’s camera movement provides valuable clues
for identifying the target subject, the method uses videoof camera movement to estimate the user’s intended
subject—what exactly the photographer is trying tocapture in his or her image. Second, the method assesses
the aesthetic merit of the subject’s position within thecomplete photo, using the objective heuristic of the ruleof thirds.

create a Structure Map, which provides a distribution
profile of local image features in space, and a FixationMap, which presents a concentration distribution of local
image features in a temporal sequence. To also consider
the possible attention on faces, a type of higher-order
semantic information, we create a Master Map byintegrating an image that has undergone face-recognition[10] processing with the Fixation Map and Structure
Map. Based on the Master Map, we then search thestitched composite image to enable trimming at the
optimal scale and in the optimal location. When
identifying the ideal composition, our system searches
for candidates at different scales and different locations
in order to find a composition that adheres to the rule ofthirds. By also incorporating saliency map [11] toestimate locations in the trimmed image that might draw
the viewer’s interest, the system searches for locations
with objective aesthetic merit. After finding the optimalvalue, we crop the section out of the stitched composite
image and present the result to the user as the best
possible composition.

4

ALGORITHM

4.1 Generating the Master Map
Figure 2: An overview of the proposed method

3 OVERVIEW OF THE PROPOSED METHOD
Composition plays a crucial role in conveying thecontent and message of a photo, and images that present
subjects effectively have a good composition [8].
Figure 2 provides an overview of the proposed method.
First, we use the movement of the user’s camera as theinformational basis for our auto-framing approach. To
gather that information, we record video of what the user
sees as he or she holds the camera, moves the device
around, and composes the shot until eventually releasing
the shutter. We then sample the video information at
regular intervals to create a set of sampling images. Next,
we stitch the sampling images together using featurematching and projective transformation techniques. Thistranslates the input video into a large synthesis image,which recreates the user’s perspective from behind thelens. We also extract ORB (Oriented FAST and Rotated
BRIEF) features [9] from each sampling image andperform spatiotemporal analyses on the ORB features to

One can assume that a photographer’s target subject islikely to appear in a vast majority of the corresponding
input video frames. In obtaining time-series imagery
from the input video, we thus sample the footage atregular intervals. The regular-interval approach not only
allows us to cover the full scope of the user’s camera
movement information but also makes it possible toincorporate local image features in a temporal fashion,
thereby enabling estimations of the subject region
present throughout the duration of the video.
The photographer’s intended subject area appears inmost of the frames of the camera-movement video.
Drawing on that subject area hypothesis, we create a
Master Map that represents the probability of containing
the user’s subjective target and the distribution of thesubject’s structural information. The Master Map uses
information from two sources: a Fixation Map, whichuses a temporal analysis of image features to specify the
subject region of the user’s intent, and a Structure Map,
which uses a spatial analysis of image features toprovide structural information on the target subject. Here,3the sizes of both the Fixation Map and the Structure Map
are the same as the size of the stitched image. Eachframe of the video footage contains local image features.
To match in-image locations on a frame-to-frame basis
and thereby facilitate the image synthesis process, weused ORB features [9]. Using the video of the user’s
camera movement, we match the descriptors on a frameby-frame basis and then analyze the descriptors
temporally to create a corresponding Fixation Map. Wet
at frame t as follows.
define Fixation Map M ORB

(

)

t
t
M ORB
= f ORB⊗ G 0, σ 2f ,

(1)

( )
( )

1 if x , y is ORB featuret
fORB x , y = 
0 if x, y is not ORB feature( )Here, G(0, σ 2f ) corresponds to a Gaussian kernel withmean 0 and variance σ 2f and ⊗ indicates a convolution
operation. ( x, y ) , meanwhile, represent a pixel of thetsampling image. Converting f ORB(a local image feature)
into a map estimating the probability of informationcontaining makes it possible to estimate the important
location in a given frame. Integrating that process along
a temporal path produces final Fixation Map M Fixation .

when the photographer places the subject of the photo on
or close to dividing lines or the intersections of thedividing lines. In order to align important subjects withthese dividing lines and thereby achieve a better overallcomposition, one needs structural information on thesubject region, the skeleton of the target subject. The
Structure Map, which is for gathering structural
information on the region containing the user’s intended
subject, is generated from the stitched master image inthe following 4 steps: 1) Extract ORB keypoints frommaster image. 2) Create a black image of the same sizeas the master image. 3) For each ORB keypoint, add awhite disk to the image with the radius of the disk
proportional to the scale of the ORB feature. 4) Apply
distance transform to the binary image to obtain askeleton representing the structure of feature distribution.
The final Master Map is obtained as the weighted
average of Fixation Map M Fixation , Structure Map
M Structure , and the face-recognition results M Face :′M Master= xM Fixation + yM Structure + zM Face

(3)

Here, x , y and z are user controllable constant andare empirically set to 0.4, 0.3 and 0.3, respectively, incurrent implementation.

ni
M Fixation = ∑ M ORB

(2)

i =1

Here, n represents the number of sampled frames. As theuser’s subjective target subject naturally appears innumerous sampling frames, adding together the local
image features on a temporal basis makes Fixation Map
M Fixation a reliable indicator of the region containing theuser’s intended subject.
Drawing on the spatial distribution of the local image
features, one can also infer structural information
characterizing the subject. The proposed method thus
detects the target subject (a subjective element) andevaluates the aesthetic merit of the photo’s composition(from an objective perspective), giving it the ability toidentify ideal scenes reflecting both user intent andcompositional beauty. The process of evaluating aphoto’s composition requires structural information in
the distribution of local image features. The rule ofthirds holds that overall compositional quality improves
4

4.2 Automatically determining the optimalcomposition via objective aesthetic evaluation
4.2.1 The rule of thirds. The rule of thirds is a standard
measure for photo composition. The rule is acompositional guideline that divides an image into nine
equal parts via “power lines” (two horizontal and two
vertical lines). The rule encourages photographers toalign their subjects with the power lines or theirintersection points (“power points”). According to therule, placing points of interest on the key line segments
and intersections gives the composition a better balance.
The rule of thirds also plays into spatial factors. For
example, assigning 1/3 of an image’s space to thebackground and the remaining 2/3 to the foreground (orvice versa) divides the total space into thirds and thereby
stabilizes the composition. After estimating thephotographer’s subjective target subject, our proposed
method satisfies the general composition rules ofphotographic science by applying a model thatrecognizes the importance of aligning subject regions

with the line segments and intersections of the rule-ofthirds guideline.
4.2.2 Creating an evaluation function. Give an aspectratio, to determine the optimal composition locations, weuse a random-sampling approach to search the Master
Map having the same size as the stitched composite
image. By varying the scale with each successive trial
and modifying the sampling locations, our system works
to determine the optimal composition.

In our method, we define the following energy function
to estimate aesthetically pleasing locations in thecomposite image. First, E Line is an energy function forevaluating how well the given location complies withthe central tenet of the rule of thirds: that placing points
of interest on the key line segments and intersections
gives the composition a better balance.
′
E Line = γ ∑ M Master( x, y )α ⋅ M RoT ( x, y )1−α
( x, y)(4)+ (1 − γ ) ∑ M Saliency ( x, y )β ⋅ M RoT ( x, y )1− β( x, y )Here, α , β, and γ are parameters for adjusting theweights of the three integrated map, M Master, M Saliency′

M RoT . ( x, y ) represents a pixel in each map.
is the Master Map trimmed with the given′
M Masteralso prevalent on the M RoT , then, the formula produces ahigh evaluation value.
Our method also needs to account for the spatial elementof the rule of thirds, which holds that assigning 1/3 of animage’s space to the background and the remaining 2/3
to the foreground (or vice versa) divides the total space
into thirds and stabilizes the composition. We define thefinal energy function ESpace as follows.
ESpace =1
Χ mod 3 + 1(6)Here, mod represents a remainder operator. Χ
represents the number of informative blocks among the
9 blocks divided via power line (maximum number is 9).
When the ratio of informative blocks and noninformative blocks is a 1/3 or 2/3 (or vice versa),
formula (6) reaches the highest value 1. We use M Master′and M Saliency to estimate the information volume in eachdivided block and then add together those two maps’
values to find the number of blocks exceeding a given
threshold. The final cost function is defined as follow:
E = ELine × ESpace

(7)

and

aspect ratio and location. M Saliency is a saliency map
computed for the trimmed stitched image. We employ afrequency analysis-based approach (Hou et al. [9]) for
the fast computing of saliency map. M RoT is obtained byapplying a Gaussian filter to the binary image with “1”represents the dividing lines of one third rule.

()

M RoT = f RoT ⊗ G 0, σ 2f ,

(5)

( )
( )

1 if u , v is on LINE / POINT
f RoT u , v = 0 if u , v is not on LINE / POINT( )Formula (4) first uses map multiplication operations todetermine the degree to which the Master Map(reflecting the user’s intention) conforms to the rule ofthirds. The formula then combines that result with thedegree to which the trimmed saliency map meets the ruleof thirds standards, providing a comprehensive result.
When both the locations with high values in the Master
Map and the salient locations after a trimming trial are

The final cost is able to take into account the two phasesof the rule of the thirds. In the composition evaluation
model of the formula (7), a higher value signifies abetter composition. In the formula (7), the evaluation
function provides an objective assessment of how well
the post-trimming information (the Saliency Map) andthe user intention-reflecting information (the MasterMap) conform to the two main principles of the rule ofthirds.

5

EXPERIMENT AND RESULTS

For our experiment, we used a digital single-lens reflex
camera (OLYMPUS E-PL2) and a lens (M. Zuiko 17
mm F2.8) with a fixed focal length (equivalent to 34 mm
on a 35-mm camera). After having the photographers(nine graduate students, all in their 20s) adjust the focus
manually, we started recording video when the user
began the shooting process and stopped the recording
when the user released the shutter. Each video had arecording resolution of 1280 x 720 and a frame rate of5

30 fps. We had the experiment participants shoot freely,
taking time to compose their images carefully. Over thecourse of the experiment, the participants took photos ofnatural subjects, people, landscapes, night views, food,buildings, and other artificial objects for a total of 34
scenes in normal, everyday situations. In landscapephotos (see Figure 1, for example), the aspect ratio of
the output has a considerable effect on the overall
impression of the image. The results also suggest that
subjects in flowing motion (waterfalls, etc.) have little
effect on the image synthesis results. During theexperiment, the proposed method’s process of searchingfor optimal compositions was not always successful indetecting important locations. Night scenes were
problematic, for example. The pervasive darkness in anight scene severely limits the range of feature points tolocations where light is present; the system thus has
minimal feature input, making it much more difficult tocreate maps for defining feature importance.

Figure 3: System evaluation result

We also had the photographers manually extract theirtarget subjects from the stitched images at three different
aspect ratios. Using that input, we then performed area
calculations to determine the accuracy rate of our
method by aspect ratio. Figure 3 is a boxplot of the
distribution of the 34 scenes across the three aspectratios. The high median values and small distribution
widths in all the aspect ratios suggest that the method
delivers high-accuracy results.

6 CONCLUSION
Our study proposed an auto-framing photography
technique that 1) detects the target subject based on the
6

photographer’s subjective interest and 2) evaluates thecomposition of the photo from an objective, aesthetic
perspective. By basing our technique around the fact thata user naturally moves the camera when composing animage, a process that reflects his or her aesthetic intent,
we successfully developed a system that allows the user
to take optimally composed photos without having totake any additional, extraneous action. Moving forward,
we hope to embed our technology in an actual camera toenable both automatic detection of optimal compositions
and real-time guidance functionality.

ACKNOWLEDGEMENTS
This study was funded by JSPS Grants-in-Aid for Scientific
Research (No. 16k12459).

REFERENCES[1]

F. Nielsen, S. Owada, and Y. Hasegawa, “Autoframing: A
Recommendation System for Detecting Undesirable Elements andCropping Automatically Photos,” in Proceedings of the international
conference on Multimedia and Expo, pp.417-420, 2006.
[2] M. Nishiyama, T. Okabe, Y. Sato, and I. Sato, “Sensation-based
Photo Cropping,” in Proceedings of the international conference on
Multimedia (MM 2009), pp.669-672, 2009.
[3] J. Yan, S. Lin, S-B. Kang, X. Tang, “Learning the change for
Automatic Image Cropping,” IEEE Conference on Computer Vision
and Pattern Recognition (CVPR2013), pp. 971-978, 2013.
[4] L. Liu, R. Chen, L. Wolf, and D. Cohen-Or, “Optimizing Photo
Composition,” Computer Graphic Forum (Proceedings ofEurographics 2010), vol.29, no.2, pp.469-478, 2010.
[5] C. Fang, Z. Lin, R. Mech, X. Shen, “Automatic Image Cropping
using Visual Composition, Boundary Simplicity and Content
Preservation Models,”in Proceedings of the 22nd ACM international
conference on Multimedia(MM), pp. 1105-1180, 2014.
[6] Y.W. Guo, M. Liu, T.T. Gu, and W.P. Wang, “Improving Photo
Composition Elegantly: Considering Image similarity During
Composition Optimization,” Computer Graphic Forum(Proceedings of Pacific Graphics 2012), vol.31, no.7, pp.2193-2202,
2012.
[7] S. Avidan, and A. Shamir, “Seam Carving for Content-aware ImageResizing,” ACM Transactions on Graphics (Proceedings of ACM
SIGGRAPH 2007), vol.26, no.3, 2007.
[8] B. Krages, “Photography: The Art of Composition,” Allworth Press,
2005.
[9] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, “ORB: An
Efficient Alternative to SIFT or SURF,” IEEE International
Conference on Computer Vision (ICCV2011), pp. 2564-2571, 2011.
[10] T. Ojala, M. Pietikainen, and T. Maenpaa, “Multiresolution Grayscale and Rotation Invariant Texture Classification with Local
Binary Patterns,” IEEE Transactions on Pattern Analysis andMachine Intelligence (PAMI), vol.24, no.7, pp.971-987, 2002.
[11] X. Hou, and L. Zhang, “Saliency Detection: A Spectral Residual
Approach,” IEEE International Conference on Computer Vision and
Pattern Recognition (CVPR 2007), pp.1-8, 2007.