Abstract:
Provided are an image conversion device, a camera, a video system, an image conversion method and a program which are capable of performing a desired image conversion even when the orientation of multiple regions within one image is to be changed in different directions. This system is provided with: a region dividing unit ( 13 ) which divides one inputted image into multiple regions, and an image conversion unit ( 14 ) which converts the image of at least one of the regions created by the region dividing unit ( 13 ) to an image imaged from a virtual viewpoint different from the imaging viewpoint of the inputted image.

Description:
TECHNICAL FIELD 
       [0001]    The present invention relates to an image conversion apparatus, a camera, a video system, an image conversion method and a recording medium including a program recorded therein that convert an image in such a way that a line-of-sight direction is changed. 
       BACKGROUND ART 
       [0002]    There has been known a technique that converts an image captured by a camera into an image captured from a virtual viewpoint different from a viewpoint of capturing the original image. 
         [0003]    PTL 1 discloses a technique that forms, using the abovementioned image conversion technique, an overview image of a wide range by using a plurality of images captured by a plurality of cameras. In this technique, the plurality of images captured by the plurality of cameras installed at different positions are changed to images captured from the same viewpoint, and the plurality of images are combined into a single image to form the above-described overview image of a wide range. 
         [0004]    PTL 2 discloses a technique that is used when an image captured by a main camera includes a blind zone and that fills the blind zone of the image with an image by using the above-described image conversion technique. In this technique, a sub-camera which is different from the main camera is used to capture an image of a range for filling the blind zone. The viewpoint of the captured image is then converted to have the same viewpoint as the viewpoint of the main camera, and the image in the range overlapping with the dead zone is cut out and combined. 
       CITATION LIST 
     Patent Literature 
     PTL 1 
     Japanese Patent Application Laid-Open No. 2005-333565 
     PTL 2 
     Japanese Patent No. 4364471 
     SUMMARY OF INVENTION 
     Technical Problem 
       [0005]    Let us consider a case where a video conference is held, for example, while participants are positioned in three directions of a table and captured by a camera from one remaining direction, and the captured image is output and displayed in a different site for the video conference. In this case, the participant positioned in front of the camera is in a proper display state in which the participant faces the front in the image. However, the participants positioned in the left and right directions of the camera are in a slightly improper display state for the video conference in which the participants turn away from the camera in the image. In this case, the image of all the participants is supposed to be displayed properly if the left region of the image is turned left and the right region of the image is turned right. 
         [0006]    However, the image conversion technique that changes a viewpoint according to the related art is a technique that converts an image in a way as if a single whole image is viewed from a single virtual viewpoint. Therefore, the image conversion technique, for example, cannot handle a situation where the right side of the image is turned 30 degrees and the left side of the image is turned 20 degrees. 
         [0007]    An object of the present invention is to provide an image conversion apparatus, a camera, a video system, an image conversion method and a recording medium including a program recorded therein that make it possible to flexibly handle a case where one image includes a plurality of regions desired to be turned in different directions and thus to convert the image in a desired way. 
       Solution to Problem 
       [0008]    An image conversion apparatus according to an aspect of the present invention includes: a region dividing section that divides one input image into a plurality of regions; and an image conversion section that converts an image of at least one of the plurality of regions into an image captured from a virtual viewpoint different from a viewpoint of capturing the input image, the plurality of regions being obtained by dividing the one input image by the region dividing section. 
       Advantageous Effects of Invention 
       [0009]    According to the present invention, it is possible to convert an image in such a way that one input image is divided into a plurality of regions and a line of sight is changed for each of the regions. Therefore, even in a case where one image includes a plurality of regions desired to be turned in different directions, it is possible to flexibly handle the case. 
     
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         [0010]      FIG. 1  is a block diagram illustrating a video conference system including a camera apparatus and displays according to Embodiment 1 of the present invention; 
           [0011]      FIG. 2  is a plan view illustrating an example of how images are captured using the camera apparatus according to the embodiment; 
           [0012]      FIG. 3  is an image view illustrating a captured image obtained by capturing in  FIG. 2 ; 
           [0013]      FIG. 4  is a plan view illustrating line-of-sight directions set by a Line-of-sight setting section; 
           [0014]      FIG. 5  is an image view illustrating a captured image after image conversion and image combining processing; 
           [0015]      FIG. 6  is a plan view illustrating an example of a display arrangement; 
           [0016]      FIG. 7  is a plan view illustrating a variation of the display arrangement; 
           [0017]      FIG. 8  is a plan view illustrating a variation of a display configuration; 
           [0018]      FIG. 9  is a block diagram illustrating a video conference system including a camera apparatus and displays according to Embodiment 2 of the present invention; 
           [0019]      FIG. 10  is a flowchart illustrating a processing procedure of a face direction detecting section; 
           [0020]      FIG. 11  is an image view of how face detection is performed; 
           [0021]      FIG. 12  is a diagram for describing an example of a detection result of the direction of each face; and 
           [0022]      FIG. 13  is a diagram for describing an example of region setting and line of sight setting results. 
       
    
    
     DESCRIPTION OF EMBODIMENTS 
       [0023]    Hereinafter, embodiments of the present invention will be described based on the drawings. 
       Embodiment 1 
       [0024]      FIG. 1  is a block diagram illustrating a video conference system (video system) including camera apparatus  1  and displays  21  according to Embodiment 1 of the present invention. 
         [0025]    Camera apparatus  1  in the embodiment includes: image input section  12  that includes a camera lens and an imaging device and that inputs image data of a captured image; region dividing section  13  that divides the captured image into a plurality of regions; and image conversion section  14  that performs image conversion on the image of each of the regions obtained by the dividing process. In addition, camera apparatus  1  includes: combining section  15  that performs image combining and output; and shape model database  16  that stores therein a three-dimensional shape model of a human face, or shape models of a wall, a desk and the like in a room. Camera apparatus  1  further includes: region setting section  17  that sets a region to be divided; and Line-of-sight setting section  18  that performs setting of the image conversion. 
         [0026]    Region setting section  17  has a plurality of operation buttons and sets a plurality of regions in the captured image upon reception of an input operation of a user. For example, optional line segments in the captured image are output and displayed by an input operation of the user, and region setting section  17  sets a range surrounded by the line segments or an outline of the captured image as one region. Alternatively, a plurality of points are input in the captured image by the input operation of the user and region setting section  17  obtains regions divided by dashed lines having the plurality of input points as apexes, or a Voronoi region having the plurality of input points as generators. Then, region setting section  17  may set the Voronoi region as a plurality of regions. The information of the set region is transmitted to region dividing section  13  and Line-of-sight setting section  18 . 
         [0027]    Upon reception of information on the set regions from region setting section  17 , region dividing section  13  performs a dividing process of the image data supplied from image input section  12  so that the captured image is divided into the set regions. Then, region dividing section  13  generates image data for each of the regions and transmits the image data to image conversion section  14 . 
         [0028]    Line-of-sight setting section  18  has a plurality of operation buttons and sets a line-of-sight direction of a conversion result (direction corresponding to the line of sight after image conversion) with respect to the plurality of regions in the captured image by receiving the input operation of the user. For example, line-of-sight setting section  18  displays an arrow for each of the set regions of the captured image and makes the direction of the arrow changeable in a three-dimensional manner by the input operation of the user. Then, line-of-sight setting section  18  sets the finally determined direction of the arrow as the line-of-sight direction of conversion target. The setting information of the line-of-sight direction for each of the regions is transmitted to image conversion section  14 . 
         [0029]    Image conversion section  14  performs image conversion processing on the image data of each of the regions so that the image having an optical axis of the camera lens as the line-of-sight direction is converted into an image which is viewed from the line-of-sight direction set for each of the regions. In a case of wide-angle capturing, the line-of-sight directions of the left and right regions of the image are slightly changed from the optical axis of the camera lens according to a viewing angle. Therefore, the information indicating in what shape and how the image before the conversion is placed in a three-dimensional manner is required to accurately perform the image conversion in which the above-described line-of-sight directions are changed. Image conversion section  14  is configured to perform simple image conversion by using the three-dimensional face model for only a face portion of a person requiring accuracy, and handling the other portions as models arranged along a uniform plane. The direction of the uniform plane can be obtained by, for example, extracting a line segment or a polygon which can specify a direction of in the image of each of the regions by an image analysis and estimating an average direction of the line segments or polygons. In addition, image conversion section  14  may be configured to cause the user to input the direction of the plane. 
         [0030]    Image conversion section  14  searches for a face portion of a person in the image of each of the regions by matching processing or the like, and when the face portion of the person is present, further specifies the direction of the face from eyes, a nose, a mouth, a contour and the like. Then, image conversion section  14  associates the face image with a three-dimensional shape using the three-dimensional shape data of shape model database  16 . In addition, the other portions are associated with the plane in which the direction is estimated as described above. By performing the processing, image conversion section  14  can associate each pixel of the image data with a coordinate point in a virtual three-dimensional mapping space. Next, image conversion section  14  performs processing of converting the image mapped in the virtual three-dimensional space into an image captured from a newly set line-of-sight direction, based on the setting information supplied from line-of-sight setting section  18  to generate image data after conversion. By performing such image processing, image conversion section  14  can convert the image into an image which is viewed from the line-of-sight direction newly set from the line-of-sight direction of the camera during capturing the image of each region so that the face portion of the person is relatively accurately converted and the other portions are roughly converted (to be converted as the plane models). 
         [0031]    Combining section  15  arranges the image data of a plurality of regions supplied from image conversion section  14  in the same arrangement as the arrangement when the dividing process is performed, and combines the image data into one piece of image data. The image data is converted into display data for outputting and displaying the image data and is output. When the image data of the plurality of regions obtained by the dividing process is individually output and displayed on a plurality of displays, combining section  15  may be configured to convert the image data of the plurality of regions obtained by the dividing process into display data suitable for the individual displays and output the converted data. 
         [0032]    The video conference system in  FIG. 1  includes above-described camera apparatus  1 , and one or a plurality of displays  21  that output and display an image by receiving display data through a network. For example, in a system including three displays  21 , the plurality of displays  21  are set to mainly display and output a plurality of region portions of the display image corresponding to the division regions of the image. 
         [0033]    Next, the operation of the above-described video conference system will be described. 
         [0034]      FIG. 2  is a plan view illustrating an example of how an image is captured using camera apparatus  1 ,  FIG. 3  is an image view illustrating a captured image obtained by the capturing in  FIG. 2 . The example illustrated in  FIG. 2  indicates a state in which persons P 1  to P 6  are positioned in three directions of table  51  and persons P 1  to P 6  are captured from one remaining direction of table  51  through wide-angle camera lens  11  with wide viewing angle θ 1 . Such capturing makes it possible to obtain the image as illustrated in  FIG. 3 . 
         [0035]    First, the user sets a region through region setting section  17 . Here, as illustrated in  FIG. 3 , the user designates dividing lines L 1  and L 2  by region setting section  17  and sets a left side range, a facing surface range and a right side range of table  51  as each region. 
         [0036]      FIG. 4  is a plan view illustrating line-of-sight directions set by the line-of-sight setting section, and  FIG. 5  is an image view illustrating a captured image after image conversion and image combining processing. 
         [0037]    Next, the user sets a line-of-sight direction to be converted by line-of-sight setting section  18 . For example, as illustrated in  FIG. 4 , the user sets new line-of-sight direction VA 2  with respect to line-of-sight direction VA 1  of an actual camera in the left region through line-of-sight setting section  18 . Further, the user sets line-of-sight direction VB 2  which is not changed from the original line-of-sight direction in the center region through line-of-sight setting section  18 , and sets new line-of-sight direction VC 2  with respect to new line-of-sight direction VC 1  of the actual camera in the right region through line-of-sight setting section  18 . 
         [0038]    Next, image conversion section  14  performs image conversion processing of rotating the image of A plane S 1 , the three-dimensional shapes of the face portions of persons P 1  and P 2 , and the image of the background wall of the persons by rotational angle “−θA” with respect to the image data of the left region. In addition, image conversion section  14  performs image conversion processing of rotating the image of C plane S 3 , the three-dimensional shapes of the face portions of persons P 5  and P 6 , and the image of the background wall of the persons by rotational angle “θC” with respect to the image data of the right region. Image conversion section  14  transmits the image data of the center region in which the line-of-sight direction is not changed to combining section  15  without any change. 
         [0039]    Then, the image data of such a plurality of regions after conversion is combined by combining section  15  to generate an image as illustrated in  FIG. 5 . In the image illustrated in  FIG. 5 , the images of persons P 1  and P 2  positioned on the left side of table  51  and persons P 5  and P 6  positioned on the right side are converted into images directed approximately to the front, and the depth perception of the images is also converted to the same degree as in the images before conversion so that the images are changed into images which are easily viewable in the video conference. 
         [0040]    When the above-described combining processing is performed, combining section  15  may perform smoothing processing of smoothing boundaries between the images of the regions, or processing of matching a position of a characteristic object. For example, in the case of  FIG. 5 , table  51  is set as the characteristic object, and combining section  15  may move the entire image of the A plane vertically to match the end position of the table. In addition, combining section  15  may remove the upper and lower regions of the entire image to make an empty region invisible from the user by moving the image and perform combining processing so that the entire image has a rectangular shape. 
         [0041]    Since combining section  15  makes the image data of each region converted by image conversion section  14  have the same size (shape) as the image data before conversion and combines the image data, combining section  15  selects a region to be used in the image data after conversion, and then, combines the image data. In the image data after conversion, when there is a portion in which the size is not sufficient, combining section  15  may be configured to perform right and left inversion (make a mirror image) on a neighboring image as image data used for the insufficient portion. 
         [0042]      FIG. 6  is a plan view illustrating an arrangement example of a plurality of displays  21 . In the drawing, reference numeral  52  denotes a table in a video conference room, and reference numerals P 7  to P 10  respectively denote persons in the meeting room. Further, a lower side of a dotted line illustrates a viewing space in a connection source (current room), and an upper side of the dotted line illustrates a space image in a room of a connection destination which is expected from the image. 
         [0043]    On the side of viewing the image transmitted from camera apparatus  1 , as illustrated in  FIG. 6 , displays  21  with the same number as the number of the division regions of the image are arranged in the same arrangement as the arrangement of the division regions. At this time, directions VA, VB and VC of each display  21  are arranged to correspond to line-of-sight directions VA 2 , VB 2  and VC 2  after conversion in the image of each of the regions. Alternatively, the information on the directions of displays  21  is transmitted to camera apparatus  1 , and line-of-sight directions VA 2 , VB 2  and VC 2  of new conversion results corresponding to the information may be set in camera apparatus  1 . 
         [0044]    Then, the image data mainly including the corresponding image of each region is output and displayed on each display  21 . When combining section  15  determines displays  21  of output and display destinations, combining section  15  may refer to the arrangement information of each display  21  and the position information of each division region in the image. Then, combining section  15  transmits the image data including the division regions corresponding to the arrangement information of corresponding displays  21  to each display  21 . 
         [0045]    By outputting the images from a plurality of displays  21 , as illustrated in  FIG. 6 , the directions and arrangement of persons P 1  to P 6  as partners of the video conference expected from the images and table  51  are converted to be properly widened (viewed as the direction is changed from the side direction to the front direction). Therefore, the partners of the video conference are easily viewable. 
         [0046]      FIGS. 7 and 8  are plan views each illustrating a variation of the number and arrangement of displays  21 . A plurality of displays  21  illustrated in  FIG. 7  are arranged on the plane without any change in angle. In the arrangement, angle differences between vertical lines of screen planes of displays  21  and line-of-sight directions after conversion corresponding to the easily viewable directions VA, VB and VC may be added to the conversion angles θA and θB of the line-of-sight directions. Even when displays  21  are planarly arranged in this manner, almost similar to the case of  FIG. 6 , the images of the partners of the video conference can be output and displayed in an easily viewable manner. In addition, as illustrated in  FIG. 8 , the images of all of the division regions may be collectively displayed by single display  21 . In this manner, almost similar to the case of  FIG. 6 , the images of the partners of the video conference can be output and displayed in an easily viewable manner. 
       Embodiment 2 
       [0047]      FIG. 9  is a block diagram illustrating a video conference system including camera apparatus  1 A and displays  21  according to Embodiment 2. Camera apparatus  1 A according to Embodiment 2 automatically performs, by processing of face direction detection section  19 , region setting in an image by region setting section  17 , and new line-of-sight direction setting by line-of-sight setting section  18 . 
         [0048]      FIG. 10  is a flowchart illustrating a processing procedure of face direction detection section  19 ;  FIG. 11  is an image view for describing face detection;  FIG. 12  is a diagram for describing an example of a detection result of the direction of each face; and  FIG. 13  is a diagram for describing an example of region setting and line of sight setting results. 
         [0049]    For example, face direction detection section  19  starts processing of the flowchart illustrated in  FIG. 10  based on an instruction operation for starting settings from a user. The user performs the instruction operation for starting settings in a state in which persons are positioned and a capturing frame is determined. 
         [0050]    When the processing starts, first, face direction detection section  19  acquires the captured image at this point from image input section  12  to detect a face portion of a person in the captured image by matching processing (Step J 1 : image search processing). As shown in  FIG. 11 , from image G 1  in which table  51  and persons P 1  to P 6  are captured, detection frames f 1  to f 6  of the face portions of persons P 1  to P 6  are extracted. 
         [0051]    Next, face direction detection section  19  analyzes the contours of the detected face portions and the arrangement of eyes, a nose and a mouth to detect direction of each face (Step J 2 : direction detection processing). As illustrated in  FIG. 12 , the face direction of each of persons P 1  to P 6  of image G 1  is detected and is digitalized from the relationship between the position of the camera lens and the direction of each face. 
         [0052]    Next, face direction detection section  19  categorizes the plurality of faces into groups (Step J 3 ) based on the positions of the detected faces and the directions of the detected faces in the image. Specifically, face direction detection section  19  groups a plurality of faces in which a difference in the directions of the detected faces is within a predetermined range (for example, within 30°) and the positions of the detected faces are arranged in sequence. In the case of image G 1 , since a difference between the face directions of two persons P 1  and P 2  consecutively positioned from the left side is within 30°, and a difference between the face direction of person P 3  who is the third person next to person P 2  and the face direction of person P 2  exceeds 30°, face direction detection section  19  categorizes the faces in detection frames f 1  and f 2  into a first group. By repeating the same processing, face direction detection section  19  categorizes the faces in detection frames f 3  and f 4  into a second group and categorizes the faces in detection frames f 5  and f 6  into a third group. 
         [0053]    After arranging the faces into the groups, face direction detection section  19  divides the image into regions corresponding to each group (Step J 4 : region setting processing). The dividing process of a region can be performed by, for example, using a Voronoi division algorithm. That is, face direction detection section  19  performs dividing process of a region for each face using the center of each of the detected faces as a generator so that each point on the image belongs to the closest generator. Further, face direction detection section  19  combines the regions of the plurality of faces belonging to the same group to set the regions to regions R 1  to R 3  of the corresponding groups (refer to  FIG. 13 ). Regions R 1  to R 3  for each group are determined and then, face direction detection section  19  transmits the information of regions R 1  to R 3  to region setting section  17  to perform region setting. 
         [0054]    After the dividing process of a region, face direction detection section  19  performs processing of determining a line-of-sight direction of conversion result for each group (Step J 5 : line of sight setting processing). As illustrated in  FIG. 13 , the line-of-sight direction is obtained as an average direction of the directions of the plurality of faces belonging to the same group. The obtained line-of-sight directions are associated with regions R 1  to R 3  for each group and transmitted to line-of-sight setting section  18 . Therefore, the line-of-sight direction of the conversion result of each of regions R 1  to R 3  is set in Line-of-sight setting section  18 . When the line-of-sight direction is changed only by a small angle (for example, ±5°) from the line-of-sight direction connecting the center point of each of regions R 1  to R 3  and the camera, an image conversion may be omitted, and the line-of-sight direction of the conversion destination is not set in this case. 
         [0055]    In Embodiment 2, the operations are performed in the same manner as in Embodiment 1 except the region setting of the image and new line-of-sight direction setting. In Embodiment 2, the input operation of the user required for the region setting and the new line-of-sight direction setting can be considerably reduced. 
         [0056]    As described above, in camera apparatuses  1  and  1 A according to Embodiments 1 and 2, and the video conference system, image conversion processing can be performed on one input image by dividing the image into a plurality of regions and converting the division images into images having different line-of-sight directions, respectively. According to this system, when one input image includes a plurality of subjects directed in various directions, the arrangement and the directions of the subjects are flexibly handled, and the image can be converted into an easily viewable image as a whole. Alternatively, the image can be converted into an image that has been deformed in a desired way in various aspects. 
         [0057]    In the above-described embodiments, an example in which the video system of the present invention is applied to a video conference system has been described, but the configuration of the video system may be implemented in one digital still camera or one digital video camera. That is, the configuration to input an image, to divide an image into a plurality of regions, to perform image conversion for changing a line-of-sight direction, and to output and display an image may be included and performed in one apparatus. In addition, an apparatus obtained by removing image input section  12  from camera apparatuses  1  and  1 A in the embodiments may be separately provided as an image conversion apparatus. 
         [0058]    Further, in Embodiment 2, an example in which image conversion is performed by categorizing a plurality of faces into groups and changing a line-of-sight direction in each region of each group has been described. However, the image conversion section may be configured to set each of face detection frames f 1  to f 6  as an individual region and to individually perform image conversion (line-of-sight direction conversion) only on the face portions. 
         [0059]    In addition, line-of-sight direction conversion may be performed on the background portions according to the angles of the faces. Since a camera image is used as input and there is no image data of lateral side and back side unlike CG in the line-of-sight direction conversion, a modest conversion (with a degree smaller than the degree of an expected conversion) may be suitable in some cases. In particular, when the conversion angle exceeds 30° C., the modest conversion may be performed. 
         [0060]    In addition, line-of-sight setting section  18  does not necessarily change all the lines of sight. The line-of-sight setting section may be configured to convert a viewpoint of one region and to maintain the original viewpoints without performing viewpoint conversion for the other regions. As a result, the line-of-sight setting may be performed such that two lines of sight are relatively changed. 
         [0061]    Further, region setting section  17 , line-of-sight setting section  18 , and face direction detection section  19  are not necessarily included in camera apparatus  1  (or camera apparatus  1 A), and these functions may be provided through a network. For example, the video conference apparatus of the connection destination of the video conference system includes the image input section, the region dividing section, the image conversion section, the shape model DB, and the combining section. In addition, the video conference apparatus of the connection source includes the region setting section, the line-of-sight setting section, and the face direction detection section. Then, the video conference apparatus of the connection source may receive an image from the video conference apparatus of the connection destination through the network, perform region setting, line-of-sight direction setting and face detection, and transmit the results to the video conference system of the connection destination so that a desired image is obtained. 
         [0062]    The configuration elements including region dividing section  13 , image conversion section  14 , combining section  15 , region setting section  17 , line-of-sight setting section  18 , and face direction detection section  19 , which have been described in the above-described embodiments, may be configured by hardware, or software which is implemented by a program executed by a computer. The program may be recorded in a computer readable recording medium. The recording medium may be a non-transitory recording medium such as flash memory and the like. 
         [0063]    The disclosure of Japanese Patent Application No. 2011-161910, filed on Jul. 25, 2011, including the specification, drawings and abstract, is incorporated herein by reference in its entirety. 
       INDUSTRIAL APPLICABILITY 
       [0064]    The present invention can be applied to a digital still camera, a digital video camera, and a video system which transmits or broadcasts video to a different place to allow for viewing the video. 
       REFERENCE SIGNS LIST 
       [0000]    
       
           1 ,  1 A Camera apparatus 
           12  Image input section 
           13  Region dividing section 
           14  Image conversion section 
           15  Combining section 
           16  Shape model database 
           17  Region setting section 
           18  Line-of-sight setting section 
           21  Display 
         VA 1  to VC 1  Line-of-sight direction of actual camera 
         VA 2  to VC 2  Line-of-sight direction of conversion result