Patent Description:
Auto Focus (AF) functions for automatically focusing a camera, a camcorder, or the like are often provided in terminals having camera functions, such as mobile phones or smartphones. For example, some AF functions are configured so that a user touches, for focusing, on a portion of a subject (hereinafter, referred to as "main subject") that is previewed on a screen of a smartphone or the like and on which the user desires to focus. Meanwhile, when the user does not clearly indicate the main subject to the camera, center-weighted AF that automatically focuses on the subject near the center of the screen is mainly adopted.

Cited references include <CIT> disclosing the features of the preamble of claim <NUM> and <CIT>.

However, the center-weighted AF has a problem that the main subject cannot be appropriately brought into focus, when the main subject is off the center, when nearby and faraway subjects are positioned near the center, or the like.

Therefore, the present disclosure proposes an information processing device, an information processing method, and an information processing program that are configured to appropriately bring the main subject into focus.

The present embodiment will be described in detail below with reference to the drawings. Note that in the present description and the drawings, portions having substantially the same configurations are designated by the same reference numerals, and redundant description thereof will be omitted.

Note that the description will be given in the following order.

Next, an example of the functional configuration of an information processing device <NUM> according to the present embodiment will be described. The information processing device <NUM> may be a digital camera or a digital camcorder, or may be a mobile terminal such as a smartphone or tablet personal computer (PC).

<FIG> is a block diagram illustrating an example of the functional configuration of an information processing device <NUM> according to the present embodiment. As illustrated in <FIG>, the information processing device <NUM> according to the present embodiment includes a storage unit <NUM>, an image capture unit <NUM>, a display unit <NUM>, a calculation unit <NUM>, a determination unit <NUM>, a focusing unit <NUM>, a tracking unit <NUM>, a sensor unit <NUM>, and a control unit <NUM>.

The storage unit <NUM> according to the present embodiment is a storage area for temporarily or permanently storing various programs and data. For example, the storage unit <NUM> may store programs and data for performing various functions of the information processing device <NUM>. In a specific example, the storage unit <NUM> may store a program for running a camera application, a learning model for determining a main subject, management data for managing various settings, and the like. In addition, image capture parameters (e.g., various parameters for capturing an image, such as focusing and exposure) and the like input by the user via a user interface (UI) of the camera application may be stored. As a matter of course, the above description is merely examples, and the type of data stored in the storage unit <NUM> is not particularly limited.

The image capture unit <NUM> according to the present embodiment captures a moving image or picture under control by the control unit <NUM>. The image capture unit <NUM> includes an imaging element, a focus ring, a zoom lens, and the like. The moving image or picture captured by the image capture unit <NUM> are converted into digital data and stored in the storage unit <NUM>. Note that the moving image captured by the image capture unit <NUM> is stored together with voice and environmental sound collected during image capture by a voice input unit (not illustrated) such as a microphone. Furthermore, the moving image captured by the image capture unit <NUM> and stored in the storage unit <NUM> includes a moving image captured while a recording is made, and a moving image captured, while the recording is not made, for preview of the subject on the display unit <NUM> or the like. The former moving image is displayed on the display unit <NUM>, temporarily stored in a Random Access Memory (RAM), and then stored in a Read Only Memory (ROM). The latter moving image is also temporarily stored in the RAM, but when the RAM becomes full, an older moving image is deleted and is not stored in the ROM.

The display unit <NUM> according to the present embodiment displays various visual information under the control by the control unit <NUM>. The visual information represents, for example, the UI of the camera application, the subject captured by the image capture unit <NUM>, and the like. For this purpose, the display unit <NUM> includes various display devices such as a Liquid Crystal Display (LCD) device and an Organic Light Emitting Diode (OLED) display device.

The calculation unit <NUM> according to the present embodiment calculates, for each predetermined area of an image (captured image) of one frame of the moving image captured by the image capture unit <NUM>, the probability (main subject degree: e.g., <NUM> to <NUM>) that a subject included in the predetermined area is the main subject. Here, the main subject is an object (object) that the camera operator desires to capture. In the present embodiment, for example, using a learning model that uses, as training data, a plurality of sets of an image obtained by capturing an object that can be the main subject and a mask image in which a main subject area is masked, one main subject is determined from the captured moving image, and the determined main subject is brought into focus.

The determination unit <NUM> according to the present embodiment determines, as a valid data area, the predetermined area having a main subject degree equal to or larger than a predetermined threshold (e.g., <NUM>), and determines one main subject area on the basis of the valid data area. A method of determining the main subject area on the basis of the valid data area will be described later.

The focusing unit <NUM> according to the present embodiment focuses on the main subject area determined by the determination unit <NUM>. When the main subject area is not determined, the focusing unit <NUM> focuses on a center area of the captured image (center-weighted AF).

The tracking unit <NUM> according to the present embodiment tracks the main subject determined by the determination unit <NUM>. The tracking unit <NUM> tracks the main subject, and when the main subject is out of the frame or when the main subject degree of the main subject is significantly decreases in the main subject area where the main subject is captured, the determination unit <NUM> determines another main subject area.

The sensor unit <NUM> according to the present embodiment measures a distance (also referred to as depth or Depth information) to the subject. The sensor unit <NUM> includes, for example, a Time of Flight (ToF) sensor and the like.

Although not illustrated, the information processing device <NUM> may include a voice output unit that outputs various sounds and voice, in addition to the above description. The voice output unit outputs sound or voice, for example, according to the situation of the camera application under the control by the control unit <NUM> (e.g., emits sound at the start and end of image capturing). For this purpose, the voice output unit includes a speaker and an amplifier.

The control unit <NUM> according to the present embodiment controls each configuration included in the information processing device <NUM>. In addition, one of the characteristics of the control unit <NUM> is to control image capture with camera. The control of image capture includes adjusting the image capture parameters, operating the focus ring and the like of the image capture unit <NUM> on the basis of the image capture parameters. Details of the functions of the control unit <NUM> will be described later.

The example of the functional configuration of the information processing device <NUM> according to the present embodiment has been described. Note that the configuration described above with reference to <FIG> is merely an example, and the functional configuration of the information processing device <NUM> according to the present embodiment is not limited to such an example. For example, the information processing device <NUM> may not necessarily include all of the configurations illustrated in <FIG>, and each configuration such as the display unit <NUM> may be included in another device different from the information processing device <NUM>. The functional configuration of the information processing device <NUM> according to the present embodiment can be flexibly modified according to specifications and operations.

In addition, an arithmetic device such as a Central Proccessing Unit (CPU) may perform the function of each component element by reading a control program from a storage medium such as the ROM or RAM storing the control programs in which process procedures implementing these functions are described, and interpreting and executing the program. Therefore, it is possible to appropriately change the configuration to be used according to technical level each time the present embodiment is carried out. Furthermore, an example of a hardware configuration of the information processing device <NUM> will be described later.

Next, functions of the information processing device <NUM> according to the present embodiment will be described in detail. One of the features of the control unit <NUM> of the information processing device <NUM> according to the present embodiment is to determine one main subject from the captured moving image by using the learning model, and appropriately focus on the determined main subject.

First, a method of generating the learning model according to the present embodiment will be described with reference to <FIG>. The learning model of the present embodiment is a model for identifying a main subject portion in the captured image. The learning model includes an input layer to which the captured image (or feature amount of the captured image) is input, an output layer, a first element that belongs to a layer being any of the input layer to the output layer, other than the output layer, and a second element whose value is calculated on the basis of the first element and a weight of the first element. The learning model causes the information processing device <NUM> to function to output, from the output layer, data (e.g., data indicating the main subject degree which is described later) indicating a portion recognized as the main subject in the captured image, according to the captured image (or the feature amount of the captured image) input to the input layer.

Note that a generation device (e.g., the information processing device <NUM> such as a server device) that generates the learning model of the present embodiment may generate the above-described learning model by using any learning algorithm. For example, the generation device may generate the learning model of the present embodiment by using a learning algorithm such as a Neural Network (NN), a Support Vector Machine (SVM), clustering, or reinforcement learning. In an example, it is assumed that the generation device generates the learning model of the present embodiment by using the NN. In this case, the learning model may have the input layer that includes one or more neurons, an intermediate layer that includes one or more neurons, and the output layer that includes one or more neurons.

Here, it is assumed that the learning model according to the present embodiment is implemented by a regression model indicated by "y = a<NUM>*x<NUM> + a<NUM>*x<NUM> +. In this case, the first element included in the learning model corresponds to input data (xi) such as x<NUM> or x<NUM>. Furthermore, the weight of the first element corresponds to a coefficient ai corresponding to xi. Here, the regression model can be regarded as a simple perceptron having an input layer and an output layer. When each model is regarded as the simple perceptron, the first element can be regarded to correspond to any node in the input layer, and the second element can be regarded as a node in the output layer.

Furthermore, it is assumed that the learning model according to the present embodiment is implemented by the NN, such as a Deep Neural Network (DNN), having one or more intermediate layers. In this case, the first element included in the learning model corresponds to any node included in the input layer or any of the intermediate layers. In addition, the second element corresponds to a node in the next layer that is a node to which a value is transmitted from a node corresponding to the first element. In addition, the weight of the first element corresponds to a connection coefficient that is a weight considered for a value transmitted from the node corresponding to the first element to the node corresponding to the second element.

The main subject degree is calculated using the learning model having any structure such as the regression model or the NN described above. More specifically, in the learning model, the coefficient is set to output the main subject degree for each predetermined area of a captured image when the captured image is input. The learning model according to the present embodiment may be a model generated on the basis of a result obtained by repeating input and output of data.

Note that, in the above example, the learning model according to the present embodiment is a model (referred to as a model X) that outputs the main subject degree for each predetermined area of the captured image when the captured image is input. However, the learning model according to the present embodiment may be a model generated on the basis of a result obtained by repeating input and output of data to and from the model X. For example, the learning model according to the present embodiment may be a learning model (referred to as a model Y) in which the captured image is input and the main subject degree output from the model X is output. Alternatively, the learning model according to the present embodiment may be a learning model in which the captured image is input and the main subject degree output from the model Y is output.

<FIG> is a diagram illustrating an example of extraction of a main subject object according to the present embodiment. The left side of <FIG> is a captured image of a cat in a living room with a window from which the outside view can be seen. On the other hand, the right side of <FIG> is an image obtained by extracting the cat that is the main subject. In this manner, the learning model is generated by extracting, from various captured images, objects, such as persons, plants and animals, buildings, moving objects including cars and airplanes, characters and signboards, toys, electrical appliances, food, clothing, accessories, astronomical objects, and the like, that can be the main subject, and being trained to learn images in which the objects that can be the main subjects are captured and the mask images in which the main subject portions are masked. Such a learning model makes it possible to determine an appropriate main subject from the captured moving image (image). Furthermore, the depth of the subject that is acquired by the sensor unit <NUM> may be used as the training data of the learning model. This configuration makes it possible to generate the learning model in consideration of the distance to the subject. Furthermore, it is also possible to perform deep learning by a Recurrent Neural Network (RNN) in which time series data including captured images of past frames of the captured moving image is input.

Next, a method of generating a main subject MAP for determining the main subject from a captured image will be described with reference to <FIG> and <FIG>. <FIG> is a diagram illustrating an example of segmentation for generating the main subject MAP according to the present embodiment. In the present embodiment, the main subject area is determined by calculating the main subject degree for each predetermined area of the captured image and generating the main subject MAP in which each predetermined area and the main subject degree are mapped. As illustrated in <FIG>, each predetermined area is determined, for example, by dividing the captured image into <NUM> rows × <NUM> columns. Note that the number of areas to be divided is not limited to <NUM> rows × <NUM> columns, and the number of pixels included in the areas may be constant or different. For example, the image may be divided so that the number of pixels included in the respective areas is constant, and pixels corresponding to the number of pixels not included in the areas may be included in areas on the outer edge of the captured image.

<FIG> is a diagram illustrating an example of generation of the main subject MAP according to the present embodiment. Using the learning model generated as described in <FIG>, the main subject degree is calculated for each area of the captured image divided in <FIG>, and the main subject MAP is generated. The right side of <FIG> is an example of the generated main subject MAP. As illustrated on the right side of <FIG>, the main subject MAP indicates the main subject degree (<NUM> to <NUM>) calculated for each area. In the example of <FIG>, an area including "flower" in the captured image shows high main subject degree. Note that, in the example of <FIG>, the areas having high main subject degree are gathered at one place, but some captured images may have dispersed areas of high main subject degree or have no area of high main subject degree.

Next, detection of a rectangular area for determining the main subject area from the main subject MAP will be described. <FIG> is a diagram illustrating an example of detection of the rectangular area according to the present embodiment. As illustrated on the left side of <FIG>, a predetermined area having a main subject degree equal to or larger than the predetermined threshold (e.g., <NUM>) is determined as the valid data area, and adjacent valid data areas are treated together as one valid data area. In the example on the left side of <FIG>, the adjacent valid data areas are combined and put together into the one valid data area, but are put together into a plurality of valid data areas having different sizes in some cases. Furthermore, the valid data area can be determined in a case where the main subject degree of a target area is equal to or larger than the predetermined threshold continuously for a certain period of time (e.g., two seconds). In this case, the main subject MAPs is generated for each of the captured images of a plurality of frames, and the valid data area is determined by, for example, processing the plurality of main subject MAPs in chronological order.

Then, as illustrated on the right side of <FIG>, the rectangular area inscribed in the valid data area is detected. When there is a plurality of valid data areas, the rectangular area is detected in each of the valid data areas. The detected rectangular area is determined as the main subject area. Note that, in detection of the rectangular area, a detection condition (e.g., upper limit or lower limit) can be further provided for the width, height, area, and the like of the rectangular area. This is because the rectangular area that is too small or too large is not appropriate as the main subject area to be brought into focus.

In addition, in order to bring the main subject area into focus, it is necessary to determine a single main subject area. Therefore, in a case where there is a plurality of main subject areas to be determined, for example, one main subject area having the largest area and/or closest to the information processing device <NUM> is determined from the plurality of main subject areas. Note that, in determination of the closest main subject area, the distance between the information processing device <NUM> and each main subject area is measured by the sensor unit <NUM> such as a ToF sensor to determine the closest main subject area. Alternatively, the closest main subject area may be determined by using depth information acquired from a phase-contrast image generated by the image capture unit <NUM>. Then, the determined one main subject area is brought into focus.

Next, tracking of the main subject in the main subject area brought into focus will be described. <FIG> is a diagram illustrating an example of object tracking according to the present embodiment. In the example of <FIG>, first, as illustrated on the left side of <FIG>, a bunch of bananas is determined as the main subject by the learning model, and the main subject surrounded by a broken line is tracked by the tracking unit <NUM>. Even if the position of the main subject in a camera frame is moved by, for example, moving the information processing device <NUM> by the camera operator, the tracking unit <NUM> continues to track the main subject (the bunch of bananas). Therefore, the main subject is kept in focus. However, as illustrated on the right side of <FIG>, when the main subject is out of frame (the main subject is lost), another main subject area (apple) is determined and tracked. Note that, for the loss of the main subject, it can be considered that the main subject is hidden by another object or the main subject degree of the main subject area is significantly reduced. In this case as well, another main subject area is determined. Note that the loss of the main subject can be postponed to a certain degree. For example, instead of determining the another main subject area immediately after the loss of the main subject occurs, the another main subject area can be determined when the loss of the main subject continues for a certain period of time (e.g., three seconds).

Next, a procedure of a main subject AF process according to the present embodiment will be described with reference to <FIG> is a flowchart illustrating the main subject AF process according to the present embodiment. The present process is, for example, a process that is triggered by temporarily storing, in the RAM, digital data obtained by converting a moving image obtained by capturing a subject by the image capture unit <NUM> of the information processing device <NUM>, and that determines the main subject and brings the determined main subject into focus.

As illustrated in <FIG>, first, the calculation unit <NUM> of the information processing device <NUM> calculates the main subject degree for each predetermined area of an image (captured image) of one frame of the moving image captured by the image capture unit <NUM> (Step S101). The predetermined area is determined by dividing the captured image into a predetermined number of grids (e.g., <NUM> rows × <NUM> columns).

Next, the determination unit <NUM> of the information processing device <NUM> determines, as the valid data area, the predetermined area having a main subject degree equal to or larger than the predetermined threshold, calculated in Step S101 (Step S102). At this time, the valid data area cannot be determined in some cases, since the captured image does not particularly include an object that can be the main subject and there is no area having a main subject degree equal to or larger than the predetermined threshold. In addition, a predetermined number of captured images including past frames can be processed in chronological order to determine, as the valid data area, the predetermined area having a main subject degree equal to or larger than the predetermined threshold continuously for a certain period of time. Therefore, the valid data area including the main subject to be brought into focus can be further appropriately determined.

When the determination unit <NUM> cannot determine the valid data area (Step S103: No), the process is repeated from Step S101 for an image of the next frame of the captured moving image. At this time, the captured image does not include the object that can be the main subject, and therefore, it is also possible to bring the center area of the captured image into focus.

On the other hand, when the determination unit <NUM> can determine the valid data area (Step S103: Yes), the determination unit <NUM> detects the rectangular area inscribed in the valid data area and satisfying a predetermined detection condition, and determines the rectangular area as the main subject area (Step S104). The rectangular area satisfying the predetermined detection condition is, for example, a rectangular area that has a width equal to or larger than a predetermined width and a height equal to or larger than a predetermined height. Therefore, there may be a plurality of rectangular areas satisfying the detection condition, and a plurality of main subject areas may be determined, in some cases.

When there is a plurality of main subject areas determined in Step S104 (Step S105: No), the determination unit <NUM> determines one main subject area from the plurality of main subject areas (Step S106). The one main subject area is, for example, a main subject area that has the largest area. Alternatively, the main subject area may be a main subject area that is closest to the information processing device <NUM>.

When the one main subject area is determined in Step S106, or when one main subject area is determined in Step S104 (Step S105: Yes), the focusing unit <NUM> of the information processing device <NUM> focuses on the determined one main subject area (Step S107). After Step S107, the present process ends.

Next, a procedure of a main subject tracking process according to the present embodiment will be described with reference to <FIG> is a flowchart illustrating the main subject tracking process according to the present embodiment. The present process is a process that is triggered by determination of the main subject area in the main subject AF process of <FIG> and that tracks the main subject to be brought into focus.

As illustrated in <FIG>, first, the tracking unit <NUM> of the information processing device <NUM> tracks the main subject included in the one main subject area determined in Step S104 or Step S106 (Step S201). The tracking described herein is to monitor the main subject in the camera frame that is moved by, for example, moving the information processing device <NUM> by the camera operator. In other words, monitoring the main subject in the captured image for each frame of the moving image captured by the image capture unit <NUM> is continued.

Next, the tracking unit <NUM> determines whether the main subject being tracked is lost (Step S202). Note that it is also possible to process a predetermined number of captured images including past frames in chronological order to determine that the main subject has been lost when the main subject has been lost continuously for a certain period of time.

When the main subject is not lost (Step S203: No), the process returns to Step S201, and the tracking unit <NUM> tracks the main subject. On the other hand, when the main subject is lost (Step S203: Yes), the determination unit <NUM> of the information processing device <NUM> determines a second valid data area that is different from the valid data area including the main subject having been tracked and that has a main subject degree equal to or larger than the predetermined threshold (Step S204).

When the second valid data area can be determined (Step S205: Yes), the process proceeds to Step S104 in <FIG>, a rectangular area inscribed in the second valid data area determined in Step S204 is detected, and the rectangular area is determined as the main subject area. Then, the process proceeds to Step S105.

On the other hand, when there is no area having a main subject degree equal to or larger than the predetermined threshold and the second valid data area cannot be determined (Step S205: No), the focusing unit <NUM> of the information processing device <NUM> focuses on the center area of the captured image (Step S206). After Step <NUM>, the present process ends.

Next, an example of the hardware configuration of the information processing device <NUM> according to the present embodiment will be described. <FIG> is a block diagram illustrating an example of the hardware configuration of the information processing device <NUM> according to the present embodiment. Referring to <FIG>, the information processing device <NUM> includes, for example, a processor <NUM>, ROM <NUM>, RAM <NUM>, a host bus <NUM>, a bridge <NUM>, an external bus <NUM>, an interface <NUM>, an input device <NUM>, an output device <NUM>, a storage <NUM>, a drive <NUM>, a connection port <NUM>, and a communication device <NUM>. Note that the hardware configuration illustrated here is merely an example, and some of the component elements may be omitted. In addition, a component element other than the component elements illustrated in the drawing may be further included.

The processor <NUM> functions as an arithmetic processing device or a control device, for example, and controls all or part of the operation of component elements, on the basis of various programs or various data (including the learning model) recorded in the ROM <NUM>, the RAM <NUM>, the storage <NUM>, or a removable recording medium <NUM>.

The ROM <NUM> is a unit that stores a program read by the processor <NUM>, data (including the learning model) used for calculation, and the like. The RAM <NUM> temporarily or permanently stores, for example, a program read by the processor <NUM>, various parameters that appropriately change when the program is executed, and the like.

The processor <NUM>, the ROM <NUM>, and the RAM <NUM> are mutually connected, for example, via the host bus <NUM> configured to transmit data at high speed. Meanwhile, the host bus <NUM> is connected to, for example, the external bus <NUM> configured to transmit data at relatively low speed via the bridge <NUM>. In addition, the external bus <NUM> is connected to various component elements via the interface <NUM>.

The input device <NUM> employs, for example, a mouse, a keyboard, a touch panel, a button, a switch, and a lever. Furthermore, for the input device <NUM>, a remote controller configured to transmit a control signal by using infrared rays or another radio wave is sometimes used. Furthermore, the input device <NUM> includes a voice sound input device such as a microphone.

The output device <NUM> is a device that is configured to visually or audibly notify the user of acquired information, for example, a display device such as a Cathode Ray Tube (CRT), LCD, or organic EL, an audio output device such as a speaker or headphone, a printer, a mobile phone, or a facsimile. Furthermore, the output device <NUM> according to the present embodiment includes various vibrating devices configured to output tactile stimulation.

The storage <NUM> is a device for storing various data. The storage <NUM> employs, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

The drive <NUM> is, for example, a device that reads information recorded on the removable recording medium <NUM> such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory or that writes information to the removable recording medium <NUM>.

The removable recording medium <NUM> is, for example, a DVD medium, a Blu-ray (registered trademark) medium, an HD DVD medium, various semiconductor storage media, or the like. As a matter of course, the removable recording medium <NUM> may be, for example, an IC card with a noncontact IC chip, an electronic device, or the like.

The connection port <NUM> is, for example, a Universal Serial Bus (USB) port, an IEEE1394 port, a Small Computer System Interface (SCSI), an RS-232C port, or a port for connecting an external connection device <NUM> such as optical audio terminal.

The external connection device <NUM> is, for example, a printer, a portable music player, a digital camera, a digital camcorder, an IC recorder, or the like.

Claim 1:
An information processing device (<NUM>) including:
an image capture unit (<NUM>) that captures an image of a subject and converts the image into digital data to generate a captured image;
a calculation unit (<NUM>) that calculates, for each predetermined area of the captured image, a probability that the subject included in the predetermined area is a main subject desired to be captured by a camera operator;
a determination unit (<NUM>) that determines the predetermined area in which the probability exceeds a predetermined threshold as a valid data area and determines a main subject area based on the valid data area; and
a focusing unit (<NUM>) that focuses on the determined main subject area;
characterized in that
the determination unit is further configured to determine the valid data area by determining the predetermined area in which the probability exceeds the predetermined threshold for a certain period of time, as the valid data area.