A Semantic Space Partitioning Approach to Virtual Camera Composition (2005)

Positioning a virtual camera in a 3D virtual environment is generally a non-intuitive task when using a 2D input device such as a mouse. The user has a mental representation of the result in terms of what he wants to see on the screen, and has to apply a mental inversion process to determine the location, orientation and field of view parameters of the camera. This task is generally achieved through a tedious and time-consuming process requiring a succession of “place the camera” and “check the result” operations. Current 3D modelers surprisingly lack integration of tools to assist the user in this task, despite the fact that cinema, in more than a hundred years, has provided a rich grammar that allows a director to unambiguously describe shots. Modelers are based on complex mathematical notions (spline curves, velocity graphs) more or less hidden by high-level manipulators. Manipulators allow positioning and animating the camera, but lack correlation with well established cinematographic notions relative to camera composition (object framing, distance shot specification, relative viewing angles, occlusions). In this paper we propose a new approach to virtual camera composition that fully relies on this grammar, relieves the user from low-level parameter manipulation and offers him classes of possible solutions w.r.t. current cinematographic notions. In related literature, numerous approaches have utilized cinematographic properties such as subject size and location within the frame to assist users in camera composition and camera planning. J. Blinn [Bli88] propose a vector algebra-based solution to pin two objects at given locations on the screen. Gleicher and Witkin [GW92] offer means to control 2D points directly on the display screen through differential manipulation techniques instead of controlling the camera parameters. Blinn's algebraic method has laid the groundwork for cinematographic idiom-based approaches, such as Christianson et al.'s compiler for the Declarative Camera Control Language (DCCL) [CAH*96] or He's et al.'s Virtual Cinematographer [HCS96]. These approaches however suffer from the point representation of the objects that doesn't capture the occlusion properties and thus avoids one of the main problems in camera planning, namely occlusion. Moreover the “point-like” representation cannot take into account the real geometry of the scene. Camera composition and camera planning can be viewed as constrained optimization problems in which the properties of the shot are expressed as numerical constraints on the camera variables (respectively the camera's path variables) and a broad range of solving procedures is available to compute solutions. The solvers differ in the way to manage over-constrained and under-constrained cases, in their complete or incomplete search capacities, in local minima management and possible optimization processes (generally finding the best solution w.r.t. an objective function). In their CAMDROID system for automated camera planning [DZ95], Drucker et al. use an existing numerical constraint solver package (CFSQP) and have to compile the shot specifications in the input script used by this library. Unfortunately, the solving process is sensitive to the initial configuration and is subject to local minima failures. Bares et al. use a partial constraint satisfaction system named CONSTRAINTCAM [BGL98] in order to provide alternate solutions when constraints cannot be completely satisfied. This solution is based on a limited subset of cinematographic properties (viewing angle, viewing distance and occlusion avoidance), which limits the procedure to small problems. P. Olivier et al.[OHPL99] propose a rich set of properties for composition purposes. The authors have developed the CAMPLAN system [HO00] that numerically solves the optimization problem via a metaheuristic search (genetic algorithms) method. The main shortcomings of this purely optimization-based technique is that the CAMPLAN genetic algorithm produces solutions in widely varying amounts of time and is also subject to the initial population of solutions. Gooch et al.[GRMS01] also use optimization procedures in order to produce images fitting artistic composition criterion like rules of thirds and fifths and the notion of canonical viewpoint (viewpoint leading to better identification of an object). The CSP (Constraint Satisfaction Problem) framework has proven to succeed in some camera composition and motion planning approaches. In [BMBT00] Bares et al. propose a heuristic-based complete search algorithm. The process is applied inside promising 3D areas computed through simple geometric intersections. Although efficient, the approach avoids problems related to multiple solutions. Unlike Bares et al. who utilize partial constraint satisfaction through cost functions, Jardillier & Languénou use pure interval methods in The Virtual Cameraman[JL98] to compute camera paths which yield sequences of images fulfilling temporally indexed image properties. This idea has been improved by Christie et al. in [CLG02]. Unfortunately, the benefits of interval-based techniques that guarantee the fulfillment of the properties during the whole sequence are counterbalanced by the computational effort required, and the absence of any mechanism for constraint relaxation. However, whenever the method fails, the user has a guarantee that there are no solutions to his problem (due to completeness of interval-based approaches). Some approaches combine constraints and optimization. One of these was presented by J. Pickering [Pic02] and is closely related to our work. In this solving method, the constraints are used to create feasible regions of space that will serve as bounds for an optimization procedure. The search space is subdivided by a “shadow-volumes” algorithm based on the properties of the image, and the feasible regions are then discretized and stored in an octree structure. Each node of the octree is then used as a starting point for a genetic algorithm that tries to find a solution to the problem. The main shortcomings of this approach lay in the computational effort required to create the octree, and the fact that multiple distinct solutions are ignored. Most of the methods we mention rely upon optimization processes to compute satisfactory camera placements and therefore lead to a unique solution closely related to the objective function. However, the description of a cinematic shot can possibly yield different visual solutions. Therefore, in computing the set of semantically distinct solutions w.r.t. cinematographic properties, one provides the user meaningful results. Figure 1 presents a top view of a simple scene containing three objects A, B and C. Whenever the user describes a shot in which he constrains A and B respectively to lay on the left and on the right of the screen, it clearly yields three possible classes of camera configurations: area (1) object C is on the left of A and B on the screen, (2) object C is between A and B, and (3) object C is on the right of A and B. Moreover, when considering possible occlusions, two classes can be added (4) A occludes C and (5) B occludes C. In such cases, classical optimization and incomplete CSP-based approaches fail in that a unique solution [DZ95,OHPL99,Pic02], or a reduced subset [JL98,CLG02] of solutions is proposed, whereas all classes of solutions should be equally considered. These approaches actually lose the semantics of the problem while relying upon pure numerical approaches. Once a solution is computed, no further information on its characteristics or differences with other possible solutions is provided. Certainly, any classification process can be provided afterhand, but requires an important computational effort. Possible distinct areas for viewing a couple of objects A and B (resp. on the left and right of the screen) w.r.t. to a third object C. In this paper, we propose to integrate a semantic dimension in the solving and interaction processes to assist the user in his camera placement tasks. We follow a threefold declarative approach: describe the desired solution with a set of cinematographic-based properties, compute distinct classes of solutions satisfying the description with related cinematographic properties, explore and interact with the classes of possible solutions. In the description phase, as in previous approaches [OHPL99,HO00,JL98,DZ95], a high-level grammar is offered including composition properties (framing objects on screen surface, relative object orientation and size) and shot properties (close shot, establishing shot, low and high angle). The geometry of the scene (locations and orientation of objects) is considered as an input provided by the user. In the computational phase, two processes are combined. The first process partitions the search space according to cinematographic properties (e.g. area such that A occludes B on the screen) and builds the intersection of the space partitions. The second process computes a nice representative of each possible class of solutions via a continuous domain implementation of a local search metaheuristic algorithm. Finally, in a third phase, the user navigates in the possible solution sets and interacts with the semantic information provided in each area. This paper concentrates on the first two phases and offers solid foundations for high-level interactions with the user. This paper is organized as follows: Section 2 introduces our semantic space partitioning approach to virtual camera composition, Section 3 details the numerical solving process. The exploitation of the semantic volumes is presented in Section 4 and relevant results are then presented in Section 5. Finally Section 6 discusses future research directions and concludes. Our approach to virtual camera compostion (VCC) is based on the primary idea of Binary Space Partition (BSP) and can be considered as an extension of visual aspects[KvD79] and closely related works such as viewpoint space partitioning[PD90] in the field of object recognition. The idea behind visual aspects is to gather all the viewpoints of a single polyhedron that share similar topological characteristics on the image. A change of appearance of the polyhedron with changing viewpoint, gives rise to boundaries in the search space. Computing all the boundaries enables the construction of regions of constant aspect, namely viewpoint space partitions. In this paper, we propose an extension of viewpoint space partitions to multiple objects and replace the topological characteristics of a polyhedron by cinematographic properties such as occlusions, relative viewing angles, distance shots and relative object locations. We introduce the notion of semantic volume as a volume of possible camera locations that give rise to qualitatively equivalent shots w.r.t. to cinematographic properties, i.e. semantically equivalent shots. Each volume is characterized by a set of semantic tags issued from film grammar [Ari76] and each tag is associated to a satisfied property in the volume. Tags are either related to a single object such as viewing angle, or to a couple of objects such as occlusion and relative image location. Therefore, the entire space of possible camera locations is thoroughly partitioned for each object, and each couple of objects. We then derive from the user's description the subset of volumes to be considered and intersect them. This process leads to a set of non-connected regions of which each represents a different class of solutions in terms of visual aspect. The computation of semantic volumes can be formalized as follows. We define the geometric filtering operator Gƒ that inputs a property p and provides a semantic volume sv, which is defined by sv=〈S, V〉 where S is a conjunction of semantic tags (e.g. LeftOf(A) ΛMediumShotOn(B) ΛOccludes(A,B) Λ…) and V is a subset of possible camera locations in camera location inside possibly the property whereas camera location Possible camera are to be further computed by the numerical process Section The filtering operator is complete in that it lose any camera locations. The operator Gƒ computes non-connected volumes by the space of of the camera locations w.r.t. p and a semantic tag to the the user description occludes leads to a volume V with the semantic tag computation partial and occlusions). The the filtering operator provides the semantic volumes related to each we that the camera's of the is to interval and the is in Camera these The property is based on the notion of shots in allows the to a viewing shot for an are different of shots the the the the the the semantic tags all of shots to The between the camera and a according to The semantic volume is computed given the of an object and a cinematographic shot by the user. One can an size to each shot presented in Figure The and desired area in the frame are used to determine the range of camera In order to some to the solving we compute an interval range related to this by and an to it The and bounds of the interval to two the and of an that the set of for the camera is computed by the volumes leading to characteristics shots. The orientation property the virtual the viewing required to an object or a A set of viewing is offered (e.g. relative to object A, there are to and each can be with high and low relative and Computing orientation semantic volumes in a volume of possible camera w.r.t. the vector to Once a with the orientation is in order to relative viewing and related semantic The occlusion property gives the user the to some constraints between two objects of the scene. The can a occlusion of an object by a partial occlusion or an absence of occlusion between two objects. A partial occlusion when a of an some of the second The semantic volumes by an occlusion property are computed given defined with to the of the two objects in the The inside bounds of the define the volumes of partial occlusion. The bounds of the define the volumes where no possible occlusion can The property allows the virtual to an object in a given frame or the screen space. This property the relative locations and of objects on the screen. constrains the distance shot and the camera locations and Moreover, or partial can be from and can be from the absence of A description three objects. description three objects with an The computation and of the semantic volumes related to this property on the of properties defined by the user and their relative location on the screen. We two (1) whenever a single frame is the property will like a the size of the frame and the size of the object in the scene lead to the computation of a volume that limits the all possible camera locations the object are possible w.r.t. this (2) the user has two or more a is to be for couple of objects in order to The of all the distinct frame is provided we propose a two leading to different semantic volumes considering and and that the semantic volumes computed at the of the will the between the on the screen (e.g. object A is left of but the locations of the objects in the the numerical process will compute camera to locations in the Figure 6 a user input related to a Figure provides a 2D of the scene with locations of objects A, B and C. the representation is two but all the computation in The first semantic volume is as 1 A and and with possible camera location in this area can lead to a shot in which the object A on the left of B. as 1 and 2 on the screen, no occlusion should between A and B. 2 and 3 are by computing the occlusion tags are where B A and when A B. the area of Figure for the possible camera locations satisfying the user's description 2D of the semantic volumes related to couple when respectively on the left and right of the screen. This process is for couple of objects in the screen, Figure an the is partial and areas and are considered as possible camera locations and are with or the area between A and B is considered as possible 3 provide any configuration and is with in the areas containing possible camera locations are at the left and right of Figure distinct volumes are the camera is in the right solution B will than A in the shot, whereas it is to the A will than B. 2D of semantic volumes related to on objects A and B the further semantic partitioning and is offered by considering and The of the whole set of distinct frame follow the process. positioning properties allow to some relative between objects on the screen, one can that one wants to see the object A on the right of B, B, These relative placements describe the of objects to within a visual composition as a conjunction of properties is by a intersection of the semantic a set of properties provided by the user results in the computation The intersection process lead to an a unique volume or to a set of non-connected In order to the intersection of 3D we propose to rely on of the semantic volumes than pure geometric Each is the of a field the function. each point of the the of the are A 3D volume is when a such that provides with the (1) of volumes by the (2) in that we or (3) to a point in the via a and simple Our implementation relies upon the that provides and means to create by for each object and with the the whole 3D scene. the possible camera locations the shots between the camera and an object are via the volumes the occlusion property are defined as the objects thus the regions of or partial The to the use of in the computational cost of our approach requires a unique at the in order to determine the of non-connected The cost of the is by the of and for our the computation of the non-connected with The of the space partitioning approach in a semantic volume V containing possible camera locations. the numerical computes a nice representative of each volume in This in in V a camera configuration orientation and that the properties and that each property to a semantic tag of S a cost function). The problem therefore to determine a of variables such that where for the cost associated to property and where is the property given by the user. optimization and constrained optimization techniques a objective (e.g. In order to manage algebraic constraints such as and algebraic constraints such as we rely on The of the framework can be in 1 presents our The algorithm relies on the of the search space a semantic starting from an initial and the the current The a set of in V within a the current introduces the notions of and The allows on promising regions of the search space by the size the in local no has been for a while a of the initial is to allow a of the search space. The procedure is by the of and and the of at each of the function. each the best in terms of constraint satisfaction and cost the new current the user's description lead to a solution is to as objects have been in the 3D we propose to integrate the satisfaction of the constraints in the cost function. Therefore, is given by The cost to the of a property considering the orientation property of for the best are vector and such that the camera's orientation is to This local search technique be used to compute solutions of camera composition problems to Olivier et al.'s However, the process the solver in promising regions and the of the associated to each The constraint is by a simple of the related to a semantic the configuration in V and is The main of this paper is to offer a semantic for and with the We two possible on the computed semantic volumes and on the whole 3D scene the computed a given each computed volume provided by the geometric solver some related to the satisfaction of the properties. the of each distinct volume can be semantically according to properties the user has a semantic volume is characterized by the relative location some further of can be computed by considering the orientation properties related to A and B (e.g. a the user can two semantically volumes and the description and for the differences between them. This to compute all tags in and in that each object and couple of objects in the scene their semantic it is to on a computed volume any possible object or The to computing a new geometric intersection and the of computed by in 2 of Section one can there a possible camera location such that A, B and C can be viewed from the relative where is the computed semantic volume and the geometric that computes the semantic volume related to the property is the is all possible volumes the a computed volume w.r.t. a property a similar the of a volume into all the possible shot relative to an object A can be expressed as The set of properties in Section 2 is provided via a as an extension to the is for the and the of the solutions given to the user. Moreover it the into existing by the of and 2 the related to properties. The property as parameters the of the left and top right of the frame containing the less than or more than 1 frame objects of the screen. The orientation property requires an object and a viewing presented in Section The property an object and a shot class properties with and the occlusion or our the classical shot two and then a shot with possible classes of solutions. The classical shot is when a between two or more and in the camera behind one while the Figure the related semantic volume and a result is presented in Figure The shot is given by the declarative script w.r.t. and view of the search space computed by the shot and of the volume A result of the The geometry of the scene is of objects The user objects A, B and C respectively in the and right of the screen, and constrains and to to the screen any occlusion. Some results are presented in Figure three shots the users description and different classes of solutions. view of the shot and related to possible camera locations with further semantic information related to of A B and C shots associated to the 3 presents the time during the and numerical in computation of one representative of each semantic volume. is directly related to the of the are and 1 and 2 respectively and is the time in the local search with and provide a Although the time to compute all important time representative is which is for interaction purposes. The semantic approach offers the the cinematographic properties provide semantic volumes containing the possible solutions of the problem through a geometric process that areas of the The computation of the boundaries relies on and avoids volume Whenever the intersection process leads to an there is a guarantee of in the user's The numerical process offers a representative of each volume at low computational The use of as object boundaries in the occlusion computation can be the of for objects that are one or for This lead to regions of the search space that possibly lead to shots during and results can be through but to integrate in our can be by computing volumes provided an representation of such volumes is The computational cost of our approach is related to the of objects in the scene and to the user's Most time is in the computation of the of non-connected volumes which is related to the of the function. We to this process by computing the intersection of of each semantic volume and then the on this Finally, the approach to is our main objective to possible camera for a of time the computation of camera A extension is the extension is and the main in with the time criterion in each volume. in this paper we have presented an approach to virtual camera composition that classes of distinct provides means to and computes the notion of visual we the notion of as a set of possible camera locations that share a set of cinematographic results the of our approach and in and to virtual camera