Patent ID: 12192618

DESCRIPTION OF EMBODIMENTS

The following clearly and completely describes the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. The described embodiments are only some, but not all, of the embodiments of the present invention. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protected scope of the present invention.

An example of a series of processes according to the embodiment of the present invention is as follows: A user activates a camera function of a mobile system while maintaining the camera of the system facing toward surroundings of a subject. The system automatically captures one or more images, which may be color or depth formats, and stores them in temporary caches which will be deleted after capturing. Using these temporary images, the system builds scene information which may include color images, depth images, normal maps, meshes or shapes represented with implicit functions. Candidates of the target subjects on a current focused image are estimated from the scene information, and photography composition is suggested with an armature which is a set of finite two-dimensional lines. The user can specify target subjects with a GUI (Graphical User Interface) on the screen of the system. The system combines the focused image and the armature, and verifies whether the current composition is good or not by evaluating the Euclidean distances between subjects on the screen and specified two-dimensional points which may be on lines or intersections of the armature. After the verification, the system shows the combined image and instructions for taking a photo with the best composition. The system can fit the current composition of the current focused image to the best composition with the user's permission. The details of these processes are described below.

The first embodiment of the present invention is implemented in a camera application to recommend and adjust photography composition automatically on a mobile system. The mobile system may be a mobile device having a camera and a display.FIG.1illustrates an example of user operations according to the first embodiment. First, a user103touches a touch display101on the system100to start the main process described in the flowchart ofFIG.4. During the main process, the user103is supposed to maintain the camera102facing toward surroundings of one or more subjects104. For example, the user moves the camera102around the subjects104or the user moves with the camera102so that the camera102faces the subjects104from different directions.

Then, the subjects104are rendered on the touch display101, and the user103can see the rendered subjects105and operate a GUI (Graphical User Interface) on the touch display101to take a picture if needed. In an embodiment, a subject map to specify target subjects is displayed on a focused image of which the user103intends to take a picture. Based on the specified target subjects, an armature and an instruction to the user103are display on the focused image, and the focused image is manually or automatically adjusted.

FIG.2depicts a block diagram of a hardware configuration of the mobile system100according to the first embodiment. The mobile system100includes a touch display101, a camera102, a CPU (Central Processing Unit)200, a data path201, a RAM (Random Access Memory)202, a ROM (Read Only Memory)203and storage systems204.

The touch display101receives touch operations by the user103, and shows a GUI and pictures which are captured with the camera102or stored in the storage systems204. The camera102may include CMOS image sensors, depth sensors, or additional sensors, and capture images or videos, whose formats are color images or depth images. The types of depth sensors may be ToF (Time of Flight) sensors, Structured-light 3D scanner or other known sensors. The depth image is an image in which depth values are visualized, and the depth value is defined as a distance from a subject.

The CPU200controls each component connected through the data path201. The data path201includes an input/output interface or a bus to convey data to each component. The RAM202is used as a main memory of the CPU200. The ROM203stores an OS (Operating System), programs, system drivers and so on. The storage systems204store most data of the mobile system100and may be SSD (Solid-State Drive), HDD (Hard Disk Drive) or other disk drives. The camera application for recommending and adjusting photography composition may be stored in the storage systems204.

FIG.3shows a block diagram of a functional configuration of the mobile system100according to the first embodiment. When the camera application for recommending and adjusting photography composition is executed using the hardware inFIG.2, this functional configuration is realized. The mobile system100includes a user interface control unit300, a camera control unit301, a scene reconstruction unit302, a subject selection unit303, a composition recommendation unit304, a composition adjustment unit305and a storage unit306.

The user interface control unit300receives user inputs or operations on the touch display101such as a touch, a swipe and so on. The user interface control unit300displays a GUI on the touch display101according to the user inputs and the other states of the mobile system100.

The camera control unit301controls the camera102in the mobile system100. The camera control unit301captures one or more images and stores the images in the storage unit306. The images may be a set of static separate images or time-sequential images. Multiple formats of the images such as color and depth may be used. The camera control unit301also controls functions of the camera102such as zooming, angle adjustment, focus correction, and other affine transformations to change the view of the images.

The scene reconstruction unit302reconstructs three-dimensional (3D) scene information (a reconstructed scene) from the images captured by the camera control unit301. The scene information may be a set of color images, depth images or 3D geometry represented as a set of polygonal meshes, point clouds or implicit functions.

The subject selection unit303generates a subject map to specify target subjects in a focused image based on the captured images and the 3D scene information. The phrase “focused image” refers to an image that the user103intends to take a picture, namely, an image to which the camera102is directed when the camera control unit301detects a user operation related to taking a picture, for example, when the camera102is directed in a certain direction for a predefined duration, when the user performs a half-push operation on a shutter button, and so on.

The composition recommendation unit304generates armatures and instructions as a recommendation for the focused image.

When the user accepts automatic composition adjustment, the composition adjustment unit305generates a control signal to adjust the camera102according to the armatures and the instructions, and sends the control signal to the camera control unit301.

The storage unit306reads and writes the images, predefined armatures, models and parameters used in the other units.

FIG.4illustrates an overall flowchart of the first embodiment, andFIG.5illustrates examples of images to be processed in the steps ofFIG.4.

At step400, the user103is photographing subjects500. The user103activates a camera function of the mobile system100and keeps the camera of the mobile system100facing toward surroundings of one or more subjects104, for example, the user103moves the camera102around the subjects104or the user103moves with the camera102so that the camera102faces the subjects104from different directions, and the camera control unit301captures a plurality of temporary images501by using the camera102. In the first embodiment, the temporary images501are temporary sequential pairs of color images and depth images, however, color images only, depth images only; or other type of images such as infrared images may be used as temporary images. The temporary images501are deleted when the user103ends the camera application through the user interface control unit301.

At step401, the scene reconstruction unit302extracts a reconstructed scene502from the temporary images501. In the first embodiment, the reconstructed scene502may be a 3D mesh structure.

The 3D mesh reconstruction from the temporary images501may be implemented with well-known methods such as SfM (Structure-from-Motion), MVS (Multi-View Stereo) or KinectFusion. The scene reconstruction may be done with the CPU200or an additional hardware processor such as a GPU (Graphical Processor Unit) or any other hardware specified for matrix calculations such as convolution. The format of the reconstructed scene502is not limited to mesh. For example, other 3D geometry formats such as point clouds or implicit functions may be used as the reconstructed scene. Further, 2D data such as a set of color images or 2.5D data such as depth images may be used as the reconstructed scene.

At step402, the camera control unit301detects a user operation related to taking a picture, for example, an operation that the camera102is directed in a certain direction for a predefined duration, an operation that the user performs a half-push operation on a shutter button, and so on, and the camera control unit301captures a focused image503from the camera102. The focused image503is used to show armatures and instructions.

At step403, the subject selection unit303generates a subject map504to specify a subject region (a region of a target subject) in the focused image503. Details of the step403will be described later.

At step404, the composition recommendation unit304generates an armature508and an instruction510as a recommendation for the focused image to improve composition quality of the subject region. Then the composition recommendation unit304shows an output image509on the touch display101. Details of the step404will be described later.

At step405, the composition adjustment unit305asks the user103to allow automatic composition adjustment or not. If the user accepts the automatic adjustment, the process proceeds to step406. Otherwise, the automatic adjustment is canceled. The user103may then manually adjust the composition. The user103can follow the armatures and the instructions on the touch display101, and move the camera102to manually generate an adjusted image511.

At step406, the composition adjustment unit305applies the automatic composition adjustment for the focused image to generate the adjusted image511. Details of the step406will be described later.

Next, the subject selection at step403above is described in detail with reference toFIG.5andFIG.6.FIG.6shows a detailed flowchart of the subject selection.

At step600, the subject selection unit303obtains the reconstructed scene502extracted at step401from the scene reconstruction unit302.

At step601, the subject selection unit303generates a subject map504to specify a possible subject region in the space of the focused image503by using the reconstructed scene502. In the first embodiment, the subject map504is a visual saliency map, however, other types of images may be used as a subject map, for examples, a semantic segmentation image, instance segmentation image and so on. Generation of the subject map may be implemented with well-known methods. For example, machine learning based methods such as convolutional neural network (CNN) or image-and-geometry processing based methods may be used. The subject map504may include subjects with low reliability which are not important. In order to improve accuracy of the subject map504, it is important to add the reconstructed scene502as inputs for the estimation of the subject map504. The subject map504generated only from color images may contain incorrect subject candidate areas if it's difficult to segmentize the images or to get the pixel intensity of the images. For example, that may be the case where pictures are taken when the scene is dark or crowded. When the format of scene information is geometry represented as a set of polygonal meshes, the subject selection unit306uses only the geometry in the three dimensional area which can be projected from the focused image503.

At step602, the subject selection unit303applies thresholding to the subject map504to remove subjects with low reliability, and keep automatically selected subjects505which do not reflect user preference.

At step603, the subject selection unit303checks if there are any user inputs to specify user preferences for subjects in the focused image503. The user inputs may be represented as touch positions507for the focused image503. If there are any user inputs, the processing will go to step604. Otherwise, the subject selection402finishes and goes to step403.

At step604, subject selection unit303converts the touch positions507to a mask for the automatically selected subjects505. Then masking is applied to keep only subjects specified by the touch positions507. Finally, user-selected subjects508are left in the subject map504.

The composition recommendation at step404above is described in detail with reference toFIG.5,FIG.7,FIG.8andFIG.9.FIG.7shows a detailed flowchart of the composition recommendation at step404.FIG.8illustrates examples of predefined armatures.FIG.9depicts examples of instructions for user operations.

At step700, the composition recommendation unit304estimates an armature509from the user-selected subjects508. For the estimation, the unit304extracts a feature vector from the selected subjects508with statistical heuristic methods or machine learning methods such as convolutional neural networks. The output of the estimation could be a probability vector that represents a likelihood of predefined armatures. Each predefined armature is composed of finite two-dimensional lines such as rule-of-thirds800, quadrant801, center802, diagonal803, golden804, or multiple armatures combined805. The storage unit308stores the predefined armatures encoded in vectorized formats, run length formats or other formats which can be easily decoded. The predefined armatures are loaded from the storage unit306, and the output armature509will be generated according to the corresponding probability vectors with statistical methods such as maximum likelihood estimation or weighted summation.

At step701, the composition recommendation unit304combines the armature509and the focused image503as the output image510.

At step702, the composition recommendation unit304verifies whether the current composition is good or not by evaluating the Euclidean distances between points in the area which has a size greater than a predefined threshold in the selected subjects508, and the specified points, lines or intersections of the armature509.

At step703, according to the verification at step702, the composition recommendation unit304presents instructions511for taking a good photo which can be operated through a GUI on the touch display101such as zooming, angle adjustment, focus correction, and other affine transformations. The instructions could be no instructions, text instructions901which show text message904, symbol instructions902which show symbols905and voice instructions903which sound a voice906. The user103can select whether to show the armature509and the instructions511or not through the touch display101. If the current composition is good as a result of the verification at step702, the composition recommendation unit304does not present instructions511.

Next, the composition adjustment at step406is described in detail with reference toFIG.10andFIG.11.FIG.10shows a detailed flowchart of the composition adjustment at step406.FIG.11illustrates examples of the composition adjustment.

At step1000, the composition adjustment unit305verifies whether the current composition of the focused image503is good or not in the same way as step702. If the current composition is good, step1000will be finished. Otherwise, the processing goes to step1001.

At step1001, the composition adjustment unit305improves composition quality automatically: At first, it is assumed that a focused image1100before the composition adjustment does not have good composition as a result of the verification at step1000. Then, the composition adjustment unit305sends a signal to the camera control unit301to adjust the camera102according to the instructions511calculated by the composition recommendation unit304to improve the composition quality: The camera control unit302controls the camera102to make the composition better, whose operation includes zooming, angle adjustment, focus correction and white balancing. An intermediate focused image1101during the composition adjustment has better composition in terms of smaller Euclidean distances between points of subjects and specified points of armatures.

At step1002, the composition adjustment unit305sends a signal to user interface control unit300to render the intermediate focused image1101. Then, the processing goes back to step1000. At this step1002, the user103can confirm the intermediate focused image1101. However, this step is not mandatory and could be skipped.

By iteration from step1000to step1002, the focused image1101is updated repeatedly. The focused image1102after the iteration will have good composition.

According to the embodiments of the present invention, the users can take a good photograph without additional knowledge of composition, and given users' permission, the current composition selected by the user can be adjusted and a photo can be automatically taken.

To get scene information surrounding users, additional sensors such as GPS (Global Positioning System) or LiDAR (Light Detection and Ranging) can be used. Vision-based optimization methods, such as differentiable rendering, can be used to adjust the current composition.

What is disclosed above is merely exemplary embodiments of the present invention, and certainly is not intended to limit the protection scope of the present invention. A person of ordinary skill in the art may understand that all or some of processes that implement the foregoing embodiments and equivalent modifications made in accordance with the claims of the present invention shall fall within the scope of the present invention.