Patent ID: 12203744

DESCRIPTION OF EMBODIMENTS

First Embodiment

A first embodiment is described.FIG.1is a diagram illustrating a detection device and a detection system according to the embodiment. A detection system50includes a plurality of detection devices (detection device1a, detection device1b) and an illumination device IL. The illumination device IL irradiate an object OB with illumination light L1. The object OB in the first embodiment is irradiated with the illumination light L1controlled by the illumination device IL as well as, for example, environment light L2such as light from interior light and natural light. The detection device1a(first detection device) detects the object OB from a viewpoint Vp1(first viewpoint), and calculates model information (first model information) on the object OB viewed from the viewpoint Vp1. The detection device1b(second detection device) detects the object OB from a viewpoint Vp2different from the viewpoint Vp1, and calculates model information (second model information) on the object OB viewed from the viewpoint Vp2. The detection system50integrates the first model information and the second model information.

For example, at least one of the detection device1aand the detection device1bis an imaging device, and detects an object OB in a detection region (for example, field of view). For example, at least one of the detection device1aand the detection device1bmay be a fixed-point camera, a camera capable of manually or automatically changing the field of view, or a portable information terminal (for example, smartphone, tablet, and mobile phone with camera). The detection device1aand the detection device1beach use the result of detecting the object OB to perform arithmetic processing on information on the object OB. The detection device1aand the detection device1beach perform arithmetic processing on its own terminal to model at least a part of the object OB and calculate model information (model data). For example, the detection device1aand the detection device1beach execute computer graphics processing (CG processing) to calculate model information (for example, CG model data).

For example, the model information includes shape information representing a three-dimensional shape of the object OB. The model information may include texture information on the object OB. For example, the model information includes at least one of three-dimensional point coordinates, relevance information on the point coordinates, texture information on surfaces defined by the three-dimensional point coordinates and their relevance information, spatial information on images, such as illumination conditions and light source information in the entire image, and polygon data serving as shape information. For example, the texture information includes at least one information on a character, a figure, a pattern, information defining unevenness, a particular image, and color (for example, chromatic color, achromatic color) of the surface of the object OB. For example, the detection device1acalculates model information representing the object OB viewed from a viewpoint Vp1(for example, predetermined viewpoint, first viewpoint, single viewpoint, one direction). For example, the detection device1acalculates model information representing the object OB viewed from first viewpoint at a predetermined angle from the object OB.

For example, the detection device1aincludes an imager2A, a detector2B, an information calculator3, a light source calculator4, a characteristics calculator5, a position calculator31, a model integrator32, a memory7, a communicator8, a controller9, and a main body10. Examples of the main body10include a camera body, a case, and a casing. For example, the detector2B, the information calculator3, the light source calculator4, the characteristics calculator5, the position calculator31, and the model integrator32are provided in the main body10.

The imager2A (first imager) images a space including a light source19(illustrated inFIGS.3A and3Breferred to later) that irradiates an object OB with illumination light and including the object. For example, imaging results (first taken image) of the imager2A include an omnidirectional image. For example, the imager2A is an omnidirectional camera, and takes a 360° panorama image as an omnidirectional image. An omnidirectional image is an image that is looked over from the viewpoint of the imager2A in the range of 360° for two declination directions (latitudinal direction and longitudinal direction) in polar coordinates whose pole is the viewpoint. For example, the imager2A stores data on taken omnidirectional images (omnidirectional image data) in the memory7. In the following description, the first taken image is an omnidirectional image, but the first taken image may be a part of an omnidirectional image, and, for example, may be an image taken in the range of a hemisphere corresponding to a half of a sphere centered at the viewpoint of the imager2A, or an image taken in a range equal to or larger than the hemisphere and smaller than 360°. The first taken image is not necessarily required to be an omnidirectional image, and is not necessarily required to be a part of an omnidirectional image. The imager2A may be provided in the main body10, and may be provided separately from the main body10. The detection device1ais not necessarily required to include the imager2A, and, for example, may be an imaging device different from the detection device1ain the detection system50.

FIG.2is a conceptual diagram illustrating an omnidirectional image. For example, an omnidirectional image Ima is an image obtained by imaging the periphery of an object in the horizontal direction in a range of 360°. For example, the omnidirectional image Ima is an image obtained by imaging a sphere from a viewpoint of the imager2A in all directions (in the range of 360° in the longitudinal direction and the latitudinal direction). Symbols Im1to Im5inFIG.2respectively represent angular positions of 0°, 90°, 180°, 270°, and 360° of the periphery of the object. An omnidirectional image Imb is an image obtained by projecting the omnidirectional image Ima onto a cylindrical surface such that the angular position Im1of 0° and the angular position Im5of 360° in the omnidirectional image Ima match with each other. An omnidirectional image Imc is an image obtained by projecting the omnidirectional image Imb onto a spherical surface such that upper ends of the angular positions Im1to Im4in the vertical direction in the omnidirectional image Imb match with one another and lower ends of the angular positions Im1to Im4in the vertical direction in the omnidirectional image Imb match with one another.

Referring back to the description with reference toFIG.1, for example, the detector2B optically detects the object OB from the viewpoint Vp1. For example, the detector2B acquires at least one of an image of the object OB viewed from the viewpoint Vp1and a distance from the viewpoint Vp1to each point on the object OB. For example, the detector2B may detect the object OB from a predetermined angle of sight. For example, the detector2B may detect the object OB from a predetermined line of sight (for example, single line of sight).

FIG.3Ais a diagram illustrating an example of the detector2B. For example, the detector2B includes an imager11(second imager) and a distance measurer12. The imager11images an object OB from a viewpoint Vp1, and outputs data on an image (second taken image) of the object OB as a detection result. For example, the imaging range (for example, field of view, angle of view) of the imager2A (first imager) is wider than the imaging range (for example, field of view, angle of view) of the imager11(second imager). For example, the detection results of the detector2B include image data on the detected object OB. The distance measurer12detects a distance from the viewpoint Vp1(seeFIG.1) to each point on the object OB. The detector2B is not necessarily required to include the imager11or the distance measurer12. For example, the detection system50may use the imager2A instead of the imager11. The imager11may image a space including a light source that irradiates the object OB with illumination light and including the object OB. In this case, the detection system50may use an imaging result of the imager11as a first taken image, and, for example, is not necessarily required to include the imager2A.

The imager11includes an image forming optical system13and an imaging element14. The image forming optical system13forms an image of the object OB. For example, the image forming optical system13is held in a lens barrel and mounted to the main body10(seeFIG.1) together with the lens barrel. Examples of the image forming optical system13and the lens barrel include an interchangeable lens, which is detachable from the main body10. The image forming optical system13and the lens barrel may be a built-in lens, and, for example, the lens barrel may be a part of the main body10and may be undetachable from the main body10.

For example, the imaging element14is a CMOS image sensor or a CCD image sensor in which a plurality of pixels are two-dimensionally arranged. For example, the imaging element14is housed in the main body10. The imaging element14takes an image formed by the image forming optical system13. For example, imaging results (detection results) of the imaging element14include information (for example, RGB data) on grayscale values of colors of pixels. For example, the imaging element14outputs the imaging results in a data format of a full-color image. The imager11may detect the direction and the intensity of light beams from the object OB like a light-field camera, for example.

The distance measurer12detects a distance from each point on the surface of the object OB. For example, the distance measurer12detects the distance by a time of flight (TOF) method. The distance measurer12may detect the distance by another method. For example, the distance measurer12may include a laser scanner and detect the distance by laser scanning. For example, the distance measurer12may project a predetermined pattern to the object OB and measure the distance based on a result of detecting the pattern. The distance measurer12may include a phase difference sensor and detect the distance by the phase difference method. The distance measurer12may detect the distance by a depth from defocus (DFD) method. In the case where a method using imaging such as the DFD method is used, the distance measurer12may use at least one of the image forming optical system13and the imaging element14in the imager11.

For example, the distance measurer12includes an irradiator15, an image forming optical system16, an imaging element17, and a controller18. The irradiator15is able to irradiate the object OB with infrared light. The irradiator15is controlled by the controller18. The controller18temporally changes (for example, amplitude modulates) the intensity of infrared light applied from the irradiator15. The image forming optical system16forms an image of the object OB. The image forming optical system16may, similarly to the image forming optical system13, be at least a part of interchangeable lenses or at least a part of built-in lenses. The imaging element17is, for example, a CMOS image sensor or a CCD image sensor. The imaging element17has sensitivity to at least a wavelength band of light applied by the irradiator15. The imaging element17is, for example, controlled by the controller18to detect infrared light reflected and scattered by the object OB. For example, the imaging element17takes the image formed by the image forming optical system16.

The controller18uses the detection results of the imaging element17to detect distances (depths) from points on the surface of the object OB to the imaging element17. For example, flying time of light entering the imaging element17from a point on the surface of the object OB changes depending on the depth of the point. The output of the imaging element17changes depending on the flying time, and, for example, the controller18calculates the depth based on the output of the imaging element17. For example, the controller18calculates the depth for each partial region (for example, one pixel, a plurality of pixels) of an image taken by the imaging element17, and calculates (generates) depth information by associating the position and the depth in the region. For example, the depth information includes information in which the position of a point on the surface of the object OB and the distance (depth) from the point to the imaging device1a(for example, the distance measurer12) are associated with each other. For example, the depth information includes information (for example, a depth image) representing a distribution of depths (for example, depth map) in the object OB.

Next,FIG.3BandFIG.3Care diagrams illustrating the illumination device IL. The illumination device IL irradiates a space including an object OB with illumination light L1. The illumination device IL is able to change an irradiation direction of the illumination light L1with respect to the object OB. For example, the illumination device IL includes a plurality of light sources19like an LED matrix, and changes the irradiation direction by switching a light source to be turned on among the light sources19. For example, inFIG.3B, in the illumination device IL, a light source19aamong the light sources19is turned on, and the other light sources are turned off. InFIG.3C, in the illumination device IL, a light source19bamong the light sources19is turned on, and the other light sources are turned off. For example, the illumination device IL is able to control the turning-on and the turning-off of the illumination light L1irradiating the object OB, the irradiation direction, and illuminance.

For example, the illumination device IL is connected to the controller9(seeFIG.1) in the detection device1aso as to be communicable in a wired or wireless manner. For example, the controller9supplies a control signal to the illumination device IL, and, for example, the illumination device IL switches the turning-on and the turning-off of each of the light sources19based on the control signal from the controller9. For example, the controller9controls the imager2A and the imager11in the detector2B to take images in synchronization with the irradiation of the illumination light L1. For example, the controller9stores taken image data from the imager2A and the imager11in the memory7in association with irradiation conditions of the illumination device IL. For example, the irradiation conditions of the illumination device IL include the irradiation direction of the illumination light L1(for example, identification information on the turned-on light source). For example, the controller9controls the imager2A and the imager11to take images of the object OB in the state in which the object OB is not irradiated with the illumination light L1from the illumination device IL, and stores taken image data in the memory7. The controller9controls the imager2A and the imager11to take images of the object OB in the state in which the object OB is irradiated with the illumination light L1from the illumination device IL, and stores taken image data in the memory7in association with identification information on light sources that irradiate the object OB with the illumination light L1. The illumination device IL may change the irradiation direction of the illumination light L1by changing at least one of the position and the posture of the illumination device IL.

For example, the illumination light L1that irradiates the object OB is reflected in part and emitted from the object OB. For example, the ratio between the light amount of reflected light L3and the illumination light L1changes depending on optical characteristics (such as reflectivity, scattering rate, absorbance, and transmissivity) of the object surface. For example, the reflected light L3includes a regularly reflected component L3aand a scattered component L3b. For example, a light intensity distribution of the scattered component L3bis represented by a Gaussian distribution centered at the normal direction of the object surface at an incident position of the illumination light L1on the object (for example, the object OB). For example, the regularly reflected component L3ahas a distribution in which the light intensity becomes maximum in a direction in which the regularly reflected component L3ais regularly reflected at the incident position on the object with respect to the irradiation direction of the illumination light L1, and the angular variance is smaller than in the scattered component L3b(for example, the half-value width is smaller).

For example, the intensity of the reflected light L3toward each direction (angular direction) around the incident position on the object changes depending on the ratio between the scattered component L3band the regularly reflected component L3a. For example, the ratio of the scattered component L3bto the reflected light L3changes depending on optical characteristics (for example, scattering rate, scattering coefficient) of the object surface. For example, the ratio of the scattered component L3bto the reflected light L3is high when the object surface is rough (degree of roughness is high), and is low when the object surface is close to a smooth surface (for example, mirror surface). For example, the brightness of the object surface in an image taken by the imager11changes depending on the irradiation direction of the illumination light L1. For example, inFIG.3B, the regularly reflected component L3adoes not enter the imager11, but a part of the scattered component L3benters the imager11. InFIG.3C, the regularly reflected component L3aand a part of the scattered component L3benter the imager11. For example, in an image taken by the imager11, the brightness of a part corresponding to the incident position of the illumination light L1in the object is higher inFIG.3Cthan inFIG.3B.

Referring back to the description with reference toFIG.1, for example, the information calculator3includes a digital signal processor (DSP). The information calculator3uses detection results (for example, depth information) of the detector2B to calculate at least one of shape information and texture information on the object OB. In the description with reference toFIG.3A, the distance measurer12generates the depth information, but the information calculator3may generate depth information based on the depth detected by the distance measurer12.

For example, the information calculator3calculates, as shape information, point cloud data including coordinates of a plurality of points on the object OB based on the detection results of the detector2B. The information calculator3calculate point cloud data (point cloud data processing) by using the detection results of the distance measurer12(for example, depth information). For example, the information calculator3calculates point cloud data by perspective transformation from a distance image (depth image) represented by the depth information to a planar image. For example, the information calculator3stores the calculated point cloud data in the memory7. When the imager11and the distance measurer12have different fields of view, for example, the information calculator3may convert the detection results of the distance measurer12into the result of detecting the object OB from the field of view of the imager11by perspective transformation (projection transformation) or the like. For example, the information calculator3may execute perspective transformation by using parameters that depend on the positional relation between the field of view of the imager11and the field of view of the distance measurer12(for example, position of viewpoint, direction of line of sight).

The information calculator3generates, as shape information, surface information including coordinates of a plurality of points on the object OB and connection information among the points based on the detection results of the detector2B. Examples of the surface information include polygon data, vector data, and draw data. For example, the connection information includes information that associates points of both ends of a line corresponding to a ridge (for example, edge) of the object OB and information that associates a plurality of lines corresponding to the contour of a surface of the object OB with one another. For example, the information calculator3estimates a surface between a point selected from among a plurality of points included in the point cloud data and a neighborhood point, and converts the point cloud data into polygon data having planar information among the points (surface processing). For example, the information calculator3converts the point cloud data into the polygon data by an algorithm using the least squares method. For example, an algorithm published in a point cloud processing library may be applied to this algorithm. For example, the information calculator3stores the calculated surface information in the memory7.

The light source calculator4uses the first taken image (for example, omnidirectional image) obtained by imaging a space including the light source19that irradiates an object OB with illumination light L1and including the object OB, and calculates and estimates light source information on a light source19that irradiates the object OB with the illumination light. For example, the light source information includes information on at least one item of the position of a light source that irradiates the object OB with illumination light L1with reference to the object, the direction in which the light source irradiates the object with light (irradiation direction), the wavelength of light emitted from the light source, and the type of the light source. For example, the light source calculator4estimates the relative position of a light source (for example, the light source19inFIGS.3B and3C) that irradiates the object OB with illumination light L1with respect to the object. For example, the light source calculator4detects a light source19in the illumination device IL that appears in the omnidirectional image. For example, the light source calculator4uses an omnidirectional image projected on a spherical surface centered at the viewpoint of the imager2A to detect the orientation of the light source19with respect to the viewpoint of the imager2A. For example, the light source calculator4uses the orientation of the light source19with respect to the viewpoint of the imager2A and position information on the viewpoint of the imager2A to calculate position information on the light source19. For example, the position information on the viewpoint of the imager2A is determined by the arrangement of the imager2A and is information known in advance.

For example, the light source calculator4may calculate the light source information by using a model that assumes Lambertian reflectance or a model including Albedo estimation. For example, the light source calculator4may use pixel values of respective pixels in an image taken by the imager11to estimate the scattered component L3b(refer toFIGS.3B and3C) derived from the object OB and the regularly reflected component L3a(refer toFIGS.3B and3C) derived from the object OB. For example, the light source calculator4may use the result of estimating the component regularly reflected by the object OB and shape information to calculate the direction in which light enters the object OB from the light source.

For example, the light source calculator4separates the illumination light L1and the environment light L2to calculate light source information on the illumination light L1. For example, the light source calculator4calculates the light source information by comparing and computing an omnidirectional image in the state in which the object OB is irradiated with the illumination light L1and an omnidirectional image in which the object OB is not irradiated with the illumination light L1. For example, the light source calculator4takes a difference between an omnidirectional image in which the irradiation of the illumination light L1is stopped and an omnidirectional image in the state in which the object OB is irradiated with the illumination light L1, thereby extracting influence of the illumination light L1on the object OB (eliminating influence of the environment light L2). In this case, for example, the detection device1ais not necessarily required to detect the object OB in a dark room, and is able to easily acquire the light source information. For example, the light source calculator4stores the calculated light source information in the memory7.

The light source calculator4may generate an image in which the influence of illumination light L1is reduced from a plurality of omnidirectional images. For example, the light source calculator4stores data on an image in which the influence of illumination light L1is reduced from omnidirectional images in the memory7. The communicator8may transmit at least one of an omnidirectional image in the state in which the object OB is not irradiated with the illumination light L1and an image in which the influence of the illumination light L1is reduced from the omnidirectional image to the outside. For example, such an image is global illumination information (scene information) including information on the periphery of the object OB, and is useable for the background when a modeled object OB is rendered.

The characteristics calculator5calculates surface characteristics information on the object OB. For example, the surface characteristics information includes optical characteristics for each part of the object OB, and the optical characteristics include at least one of reflectivity, transmissivity, scattering rate, and absorbance. For example, the surface characteristics information is a part of texture information, and, for example, the texture information includes information on at least one item of pattern information representing design of the surface of the object OB, light source information on the illumination light L1illuminating the object OB, and surface characteristics information representing optical characteristics of the surface of the object OB.

For example, the characteristics calculator5calculates, as the surface characteristics information, at least one of bi-directional reflectance distribution function (BRDF) and spatially varying bidirectional reflectance distribution function (SVBRDF). The characteristics calculator5calculates the surface characteristics information by using light source information, shape information, and taken images obtained by imaging the object OB while changing the irradiation direction of the illumination light L1. The characteristics calculator5calculates, for each part of the object OB, surface characteristics information by using a shape represented by shape information (for example, the normal direction of the surface of each part of the object OB), the irradiation direction of the illumination light L1represented by light source information, and a taken image corresponding to the irradiation direction of the illumination light L1. For example, the characteristics calculator5uses calculation results of the light source calculator4(for example, the irradiation direction of the illumination light L1) as light source information on the illumination light L1. For example, the characteristics calculator5uses calculation results of the information calculator3as shape information. For example, the characteristics calculator5uses imaging results of the imager11as taken images obtained by imaging the object OB while changing the irradiation direction of the illumination light L1. In the case where the imager11detects the direction of light beams (the irradiation direction of the illumination light L1) and the intensity thereof like a light-field camera, as compared with the case where the imaging is performed for each irradiation direction while changing the irradiation direction of the illumination light L1in a plurality of directions, equivalent information is able to be acquired by single imaging. In this case, for example, as compared with the case where a plurality of images are taken, the influence of relative misalignment of the illumination device IL, the object OB, and the viewpoint Vp1is able to be reduced during the imaging. The characteristics calculator5may use imaging results (for example, omnidirectional image) of the imager2A as taken images obtained by imaging the object OB while changing the irradiation direction of the illumination light L1. For example, the characteristics calculator5calculates the surface characteristics information by using a taken image corresponding to the irradiation direction of the illumination light L1.

For example, the characteristics calculator5calculates the surface characteristics information by inverse rendering. For example, the characteristics calculator5solves a relational expression of optical characteristics, shape information, and light source information by defining optical characteristics of each part of the object OB to be unknown, thereby calculating the surface characteristics information. For example, the characteristics calculator5calculates the surface characteristics information by solving a relational expression established from the basic formula of inverse rendering by defining the SVBRDF to be unknown. A BDRF vector of known material is represented by ρj with tilde indicated in Expression (1) below. j is an index representing the kind of material, and m is the number of kinds of materials. For example, when optical characteristics of each of RGB are defined such that a declination of the viewpoint Vp in the latitudinal direction, a declination of the light source in the latitudinal direction, and a declination between the viewpoint Vp and the light source in the longitudinal direction in the polar coordinates whose pole is the object are in 90 steps (for example, in increments of 1°), the above-mentioned BRDF vector is a 90×90×90×3 dimensional vector.
[Math. 1]
{|j=1,2, . . . ,m}(1)

A basis vector Ki of m BRDF vectors above is determined by, for example, principal component analysis (PCA) as a vector to minimize Expression (2) below.

[Math.2]E=∑j=1mρ˜j-∑i=1nKiT⁢ρ˜j⁢Ki2(2)

Examples of the method for acquiring information on a part of an object OB for which surface characteristics information is to be calculated (hereinafter referred to as “part to be calculated”) include a method in which a plurality of images taken for each irradiation direction while changing the irradiation direction of a light source (for example, the illumination light L1) are used to acquire pixel data (observation pixel data) from the same pixel corresponding to a part to be calculated. Another method is, for example, a method in which an image taken with one irradiation direction is used to acquire a plurality of pieces of pixel data (observation pixel data) from a pixel corresponding to a part of the object OB whose optical characteristics (surface characteristics) are regarded as the same as a part to be calculated and a pixel corresponding to the part to be calculated. A method in which the above-mentioned two methods are combined may be used. Regarding each part of the object OB corresponding to a region (for example, one pixel, a plurality of pixels) in the taken image, for example, the positional relation with the light source (for example, irradiation direction) is already known by the light source information, the normal direction at each part of the object OB is already known by the shape information, and the positional relation between each part of the object OB and the viewpoint Vp1is already known by the shape information. Expression (3) below is observation pixel data on the same reflection characteristics. For example, Expression (3) is determined by correcting, for each part of the object OB viewed from the viewpoint Vp1, cosine components of distances between parts of the object OB and the viewpoint Vp1with respect to the normal direction and cosine components of distances between parts of the object OB and the light source with respect to the normal direction.

[Math.3][XR⁢1(θh⁢1,θd⁢1,φd⁢1)XG⁢1(θh⁢1,θd⁢1,φd⁢1)XB⁢1(θh⁢1,θd⁢1,φd⁢1)XR⁢2(θh⁢2,θd⁢2,φd⁢2)XG⁢2(θh⁢2,θd⁢2,φd⁢2)XB⁢21(θh⁢2,θd⁢2,φd⁢2)⋮⋮⋮XRm(θh⁢m,θd⁢m,φd⁢m)XG⁢m(θh⁢m,θd⁢m,φd⁢m)XBm(θh⁢m,θd⁢m,φd⁢m)](3)

Expression (4) below is an expression used to calculate surface characteristics information. The left side of Expression (4) corresponds to observation pixel data in Expression (3). A matrix in the right side of Expression (4) is obtained by arranging basis vectors Ki indicated in Expression (2) in the row direction as column vectors. In Expression (4), when the number of observation pixels (the number of elements of vectors in the left side) is equal to or larger than the number of basis vectors (the number of columns in the matrix in the right side), a vector of a coefficient Wn is determined. For example, the vector of the coefficient Wn may be calculated by a method using optimization such as the method of least squares. Once the vector of the coefficient Wn is determined, the BDRF for any angle is able to be calculated by using the basis vector Ki.

[Math.4][XR⁢1(θh⁢1,θd⁢1,φd⁢1)XG⁢1(θh⁢1,θd⁢1,φd⁢1)XB⁢1(θh⁢1,θd⁢1,φd⁢1)XR⁢2(θh⁢2,θd⁢2,φd⁢2)XG⁢2(θh⁢2,θd⁢2,φd⁢2)XB⁢21(θh⁢2,θd⁢2,φd⁢2)⋮XRm(θh⁢m,θd⁢m,φd⁢m)XG⁢m(θh⁢m,θd⁢m,φd⁢m)XBm(θh⁢m,θd⁢m,φd⁢m)]=[KR⁢11(θ⁢h1,θ⁢d1,φ⁢d1)KR⁢1⁢n(θ⁢h1,θ⁢d1,φ⁢d1)KG⁢11(θ⁢h1,θ⁢d1,φ⁢d1)⋯KG⁢1⁢n(θ⁢h1,θ⁢d1,φ⁢d1)KB⁢11(θ⁢h1,θ⁢d1,φ⁢d1)KB⁢1⁢n(θ⁢h1,θ⁢d1,φ⁢d1)⋮⋱⋮KRm⁢1(θ⁢hm,θ⁢dm,φ⁢dm)KRmn(θ⁢hm,θ⁢hm,φ⁢dm)KGm⁢1(θ⁢hm,θ⁢hm,φ⁢dm)⋯KGmn(θ⁢hm,θ⁢hm,φ⁢dm)KBm⁢1(θ⁢hm,θ⁢hm,φ⁢dm)KBmn(θ⁢hm,θ⁢hm,φ⁢dm)][W1W2⋮Wn](4)

In this manner, for example, the characteristics calculator5calculates surface characteristics information by solving the relational expression (for example, Expression (4)) of optical characteristics, shape information, and light source information while optical characteristics for each part of the object OB are unknown. For example, the characteristics calculator5stores the calculated surface characteristics information in the memory7. For example, the characteristics calculator5calculates the surface characteristics information by solving the relational expression for a first part of the object OB. For example, the first part is a part for which information (for example, observation pixel) necessary for solving the relational expression is sufficient. For example, the characteristics calculator5does not calculate surface characteristics information by the relational expression for a second part different from the first part of the object OB. For example, the second part is a part for which information (for example, the number of observation pixels) necessary for solving the relational expression is insufficient. The characteristics calculator5may calculate optical characteristics of the second part by interpolation using optical characteristics of the first part. When information (for example, observation pixels) necessary for solving the relational expression is able to be obtained, the characteristics calculator5may calculate the irradiation direction of the illumination light L1from surface characteristics information (for example, a plurality of pieces of surface characteristics information having different imaging viewpoints). The detection device1is not necessarily required to calculate the surface characteristics information on the object OB, and is not necessarily required to include the characteristics calculator5.

The position calculator31uses the light source information to calculate and estimate first position information representing a positional relation between the object OB and the viewpoint Vp1. The first position information is used to integrate first model information and second model information including shape information obtained by detecting the object OB from a viewpoint Vp2(second viewpoint). For example, the position calculator31calculates the positional relation between the first viewpoint (viewpoint Vp) and the object OB by using an irradiation direction of the illumination light L1with respect to the object OB as the light source information. For example, a positional relation between the viewpoint of the imager2A (for example, first taken image) and the viewpoint of the detector2B is known in advance, and the orientation of the light source from the viewpoint of the imager2A is able to be known from the light source information, and hence the position calculator31is able to estimate the position of the light source19with respect to the viewpoint (viewpoint Vp) of the detector2B. For example, the information calculator3calculates the positional relation between the first viewpoint (viewpoint Vp) and the object OB, and the position calculator31is also able to estimate a positional relation among the viewpoint Vp, the object OB, and the light source19by using the positional relation between the first viewpoint (viewpoint Vp) and the object OB calculated by the information calculator3and the positional relation between the viewpoint Vp and the light source19. The position calculator31may calculate the above-mentioned positional relation by using a plurality of pieces of light source information obtained from a plurality of first taken images (for example, omnidirectional images) having different irradiation directions of the illumination light L1. The first taken image used by the position calculator31may be a single taken image that has one type of the irradiation direction of the illumination light L1, and, for example, the irradiation direction of the illumination light L1may be one direction. The imager2A may acquire a taken image (for example, omnidirectional image) including an object OB and a detection device (for example, the detection device1a), and the position calculator31may calculate the above-mentioned positional relation by using information obtained from the taken image. The imager2A may acquire a taken image (for example, omnidirectional image) including a light source19and a viewpoint Vp (for example, the detection device1a), and the position calculator31may calculate the above-mentioned positional relation by using information obtained from the taken image. For example, the detection device1bhas the same configuration as that of the detection device1a. Similarly to the detection device1a, the detection device1bcalculates second position information representing a positional relation between an object OB and a viewpoint Vp2. A communicator A in the detection device1areceives second model information and second position information from the detection device1b. The position calculator31may use light source information to calculate a positional relation between the illumination device IL (light source19) and the viewpoint Vp1. The detection device1bmay use light source information to calculate a positional relation between the illumination device IL (light source19) and the viewpoint Vp2. In this case, for example, the model integrator32may calculate a positional relation between the viewpoint Vp1and the viewpoint Vp2based on positional relations of the viewpoint Vp1and the viewpoint Vp2with respect to the illumination device IL (light source19). The model integrator32may use the calculated positional relation between the viewpoint Vp1and the viewpoint Vp2to integrate first model information and second model information.

The model integrator32uses the first position information and the second position information to integrate the first model information and the second model information. For example, the model integrator32converts the coordinate system of the second model information into the same coordinate system as that of the first model information such that the relation between the object OB and a predetermined irradiation direction of the illumination light L1calculated by the position calculator31in the detection device1aand the relation between the object OB and a predetermined irradiation direction of the illumination light L1calculated by the position calculator31in the detection device1bare consistent with each other. For example, the model integrator32integrates model information by adding the first model information and the coordinate-converted second model information. For example, the model integrator32stores the integrated model information in the memory A.

The model integrator32may integrate first model information and second model information by matching feature points of the shape represented by the first model information and feature points of the shape represented by the second model information. For example, the model integrator32uses the first position information and the second position information, enabling the range for searching for feature points in the second model information similar to feature points in the first model information to be narrowed to reduce the processing load.

For example, the memory7is a non-volatile memory such as a USB memory and a memory card, and stores various kinds of information therein. The memory7may include a storage device built in the detection device1a, or may include a port to which a storage device that is releasable from the detection device1ais connectable.

For example, the communicator8includes at least one of an I/O port, such as a USB port and a LAN port, and a communication device that performs wireless communications using radio waves or infrared rays. The communicator8is controlled by the controller9to read information stored in the memory7and transmit the read information to an external device. For example, the communicator8transmits at least a part of calculation results (for example, shape information) of the information calculator3, calculation results (for example, light source information) of the light source calculator4, calculation results (for example, surface characteristics information) of the characteristics calculator5, calculation results (for example, first position information) of the position calculator31, and processing results of the model integrator32to an external device (for example, the information processing device51illustrated inFIG.8referred to later). For example, the communicator8receives information including instructions from the external device. The communicator8is capable of storing the received information in the memory7and supplying the received information to the controller9.

For example, the controller9controls the components in the detection device1ain response to an instruction (control signal) from a user or an external device. The controller9controls the illumination device IL such that the imaging processing by the imager11and the detection processing by the detector2B are executed in synchronization with the irradiation of the illumination light L1. For example, the detection processing includes the imaging processing by the imager11and the distance detection processing by the distance measurer12. For example, the controller9stores at least a part of the detection results of the imager11in the memory7. For example, the controller9stores at least a part of the detection results of the detector2B in the memory7. For example, the controller9controls the information calculator3to calculate model information (for example, shape information), and stores at least a part of the model information calculated by the information calculator3in the memory7. For example, the controller9controls the light source calculator4to calculate light source information, and stores at least a part of the light source information calculated by the light source calculator4in the memory7. For example, the controller9controls the characteristics calculator5to calculate surface characteristics information, and stores at least a part of the surface characteristics information calculated by the characteristics calculator5in the memory7. For example, the controller9controls the position calculator31to calculate position information, and stores at least a part of the position information calculated by the position calculator31in the memory7. For example, the controller9controls the model integrator32to execute model integration processing, and stores processing results in the memory7. For example, the controller9controls the communicator8to transmit at least a part of the information stored in the memory7to an external device.

For example, the detection device1ais capable of outputting at least a part of the model information to a digital device capable of inputting and outputting digital information such as a barcode and a two-dimensional code. Such a digital device is capable of displaying or printing digital information including at least a part of the model information on a display or paper. A reader device including a reader (for example, an optical reader) capable of reading displayed or printed digital information is capable of inputting the digital information to a storage area in the reader device through the reader. The reader device may further include a rendering processor described later. The above-mentioned detection system50may include the digital device or the reader device including the reader described above. The detection device1amay include the digital device or the reader device described above. When the detection device1aincludes the above-mentioned digital device, the communicator6may transmit at least a part of the model information to the above-mentioned digital device. The above-mentioned digital device may generate digital information based on received model information and output the digital information to a medium such as paper.

Next, a detection method according to the first embodiment is described based on the operation of the detection device1ahaving the above-mentioned configuration.FIG.4is a flowchart illustrating an example of the detection method according to the embodiment. The detection device1calculates, by using a first taken image obtained by imaging a space including a light source19that irradiates an object OB with illumination light L1and including the object OB, light source information on the light source that irradiates the object OB with the illumination light. For example, while changing the irradiation direction of the illumination light L1, the detection device1performs the processing from Step S1to Step S5to acquire a first taken image (for example, omnidirectional image) of a space including a light source19corresponding to each irradiation direction and including the object OB and a taken image (for example, second taken image) of the object OB viewed from one viewpoint, and calculate a plurality of pieces of light source information corresponding to the irradiation directions. Prior to the processing in Step S1, the imager2A may take a plurality of omnidirectional images of a space including the object OB in the state in which the irradiation of the illumination light L1is stopped.

FIG.5is a flowchart illustrating an example of processing for calculating light source information (Step S4inFIG.4). In Step S21inFIG.5, the light source calculator4acquires data on a first taken image (for example, omnidirectional image) from the memory7. In Step S22, the light source calculator4converts the omnidirectional image into an image on a spherical surface (seeFIG.2). In Step S23, the light source calculator4separates illumination light L1from environment light L2among light (illumination light L1, environment light L2) irradiating the object OB, thereby obtaining light source information on the illumination light L1. For example, the light source calculator4subtracts an omnidirectional image taken in the state in which the object OB is not irradiated with the illumination light L1from an omnidirectional image taken in the state in which the object OB is irradiated with the illumination light L1to separate the illumination light L1on the omnidirectional image from the environment light L2. The order of the processing in Step S22and the processing in Step S23may be replaced with each other. After the illumination light L1on the omnidirectional image is separated from the environment light L2, the omnidirectional image may be converted into an image on a spherical surface. In Step S24, the light source calculator4calculates light source information on the illumination light L1at the object OB by using the omnidirectional image on the spherical surface.

In Step S5inFIG.4, the controller9determines whether to change the irradiation direction of the illumination light L1. For example, when the taking of omnidirectional images in a part of a plurality of predetermined irradiation directions has not been finished, the controller9determines to change the irradiation direction of the illumination light L1(Yes in Step S5). When the controller9determines to change the irradiation direction of the illumination light L1(Yes in Step S5), the controller9returns to the processing in Step S1to set the irradiation direction to the next direction, and repeats the processing after Step S1. In this manner, while the controller9switches the irradiation direction of the illumination light L1, the imager2A takes an omnidirectional image for each irradiation direction, and the light source calculator4calculates light source information for each irradiation direction. The processing (light source information calculation processing) in Step S4may be performed after an omnidirectional image is acquired for each of a plurality of irradiation directions. For example, the detection device1may repeat the processing from Step S1to Step S5excluding the processing in Step S4to accumulate data on omnidirectional images, and the light source calculator4may use the accumulated data on omnidirectional images to sequentially calculate light source information on irradiation directions.

In Step S5, for example, when the taking of omnidirectional images in predetermined irradiation directions has been finished, the controller9determines not to change the irradiation direction of the illumination light L1(No in Step S5). When the controller9determines not to change the irradiation direction of the illumination light L1(No in Step S5), the position calculator31uses the light source information to calculate position information (first position information) in Step S6.

In Step S7, the detector2B detects the object OB from a viewpoint Vp1. For example, the controller9controls the imager11to execute imaging processing in parallel to (in synchronization with) the imaging by the imager2A. The controller9controls the distance measurer12to detect a distance to the object OB. The detection processing of the distance by the distance measurer12may or need not be performed in parallel to the imaging processing by the imager2A or the imaging processing by the imager11. In Step S8, the information calculator3uses the detection results of the detector2to calculate shape information on the object OB viewed from the viewpoint Vp1. For example, the information calculator3uses the detection results (for example, depth information) of the distance measurer12to calculate point cloud data, and uses the point cloud data to calculate surface information.

In Step S9, the characteristics calculator5calculates surface characteristics information on the object OB by using the shape information calculated by the information calculator3, the light source information calculated by the light source calculator4, and a plurality of taken images obtained by imaging the object OB while changing the irradiation direction. For example, the taken images are images taken by the imager11in Step S2, but may be an omnidirectional image taken by the imager2A in Step S1, and may include an image taken by the imager2A and an image taken by the imager11.

FIG.6is a flowchart illustrating an example of processing for calculating surface characteristics information according to the embodiment (Step S9). In Step S31, the characteristics calculator5sets a part to be calculated of the object OB. For example, the part to be calculated may be one pixel or a plurality of pixels in a taken image, may be a mesh region surrounded by a line sequentially connecting three or more points included in point cloud data, or may be a surface surrounded by three or more lines included in surface information. In Step S32, the characteristics calculator5acquires shape information on the part to be calculated, light source information on irradiation directions of the illumination light L1, and taken images (for example, images taken by the imager11) of the object OB corresponding to the irradiation directions from the memory7.

In Step33, the characteristics calculator5determines whether information for solving a relational expression (for example, Expression (4) above) is sufficient. For example, as described above for Expression (4), when the number of observation pixels is equal to or more than the number of basis vectors, the characteristics calculator5determines that information for solving the relational expression is sufficient (Yes in Step S33). When the characteristics calculator5determines that information for solving the relational expression is sufficient (Yes in Step S33), the characteristics calculator5solves a relational expression in which surface characteristics information is unknown in Step S34. In Step S35, the characteristics calculator5optimizes the unknown by using an evaluation function. For example, the evaluation function is a difference between an image of the object OB obtained by rendering processing using the solution of the above-mentioned relational expression and a taken image used to solve the relational expression. For example, the characteristics calculator5optimizes the unknown such that the value of the above-mentioned evaluation function is minimized.

When it is determined in Step S33that information for solving the relational expression is not sufficient (insufficient) (No in Step S33) or after the processing in Step S35, the characteristics calculator5determines whether there is the next part to be calculated in Step S36. For example, the characteristics calculator5determines that there is the next part to be calculated when the processing in Step S31is not completed for a predetermined part (Yes in Step S36). When the characteristics calculator5determines that there is the next part to be calculated (Yes in Step S36), the characteristics calculator5returns to the processing in Step S31to set the next part to be calculated, and repeats the processing in and after Step S32. In Step S36, for example, when the processing in Step S31has been completed for a predetermined part, the characteristics calculator5determines that there is no next part to be calculated (No in Step S36), and finishes a series of processing.

FIG.7is a flowchart illustrating another example of the processing for calculating surface characteristics information according to the embodiment. In the processing inFIG.7, the characteristics calculator5calculates optical characteristics of other parts by interpolation using optical characteristics obtained by solving the above-mentioned relational expression. For example, the processing inFIG.7is performed after the processing in Step S36inFIG.6. In Step S41, the characteristics calculator5sets a part to be calculated. In Step S42, the characteristics calculator5determines whether optical characteristics of the part to be calculated have been calculated. When the characteristics calculator5determines that the optical characteristics of the part to be calculated have not been calculated (No in Step S42), the characteristics calculator5determines whether to calculate the optical characteristics by interpolation. For example, the characteristics calculator5determines to calculate the optical characteristics by interpolation when the required number of parts, whose optical characteristics have been calculated, for interpolation is present around the part to be calculated (Yes in Step S43). When the characteristics calculator5determines to calculate the optical characteristics by interpolation (Yes in Step S43), the characteristics calculator5calculates the optical characteristics by interpolation in Step S44.

After the processing in Step S44, when the characteristics calculator5determines that the optical characteristics have been calculated in Step S32(No in Step S42) or when the characteristics calculator5determines in Step S43not to calculate the optical characteristics by interpolation (No in Step S43), the characteristics calculator5determines in Step S45whether there is the next part to be calculated. For example, the characteristics calculator5determines that there is the next part to be calculated when the processing in Step S41has not been completed for a predetermined part (Yes in Step S45). When the characteristics calculator5determines that there is the next part to be calculated (Yes in Step S45), the characteristics calculator5returns to the processing in Step S41to set the next part to be calculated, and repeats the processing in and after Step S42. In Step S45, for example, when the processing in Step S41has been completed for a predetermined part, the characteristics calculator5determines that there is no next part to be calculated (No in Step S45), and finishes a series of processing.

As described above, for example, the detection device1ain the first embodiment may include: a detector that detects an object from a first viewpoint (for example, one viewpoint); an information calculator that calculates first model information including shape information on the object from the first viewpoint by using detection results of the detector; a light source calculator that calculates light source information on the illumination light by using a plurality of omnidirectional images obtained by imaging a space including the object while changing an irradiation direction of illumination light with respect to the object; and a position calculator that calculates relative positions between the viewpoints by using the light source information as information used to integrate the first model information and second model information including shape information obtained by detecting the object from a viewpoint (for example, second viewpoint) different from the first viewpoint. In this manner, according to the first embodiment, for example, the position of a light source that irradiates an object with illumination light is able to be estimated from an omnidirectional image, and a relative position between detection devices is able to be specified by using a plurality of omnidirectional images.

Second Embodiment

A second embodiment is described. In the second embodiment, the same configurations as in the above-mentioned embodiment are denoted by the same reference symbols, and descriptions thereof are omitted or simplified.FIG.8is a diagram illustrating a detection device and a detection system according to the embodiment. The detection system A includes an information processing device A that processes at least one of information output from the imaging device1aand information output from the imaging device1b. For example, the information processing device51is provided with an input device52and a display device53.

The information processing device51acquires information from the detection device1athrough communications with the detection device1a. The information processing device51acquires information from the detection device1bthrough communications with the detection device1b. For example, the information processing device51executes the rendering processing by using at least a part of the information acquired from the detection device1aand the information acquired from the detection device1b. For example, on the basis of setting information on a viewpoint input to the input device52by the user, the information processing device51calculates data on an estimated image obtained by viewing the object OB from this viewpoint. For example, the information processing device51supplies the data on the estimated image to the display device53and displays the estimated image on the display device53.

The input device52includes, for example, at least one of a keyboard, a mouse, a touch panel, a sensor such as an acceleration sensor, a voice input machine, and a touch pen. The input device52is connected to the information processing device51. For example, the input device52receives input of information from the user and supplies the input information to the information processing device51. The display device53includes, for example, a liquid crystal display or a touch panel display and is connected to the information processing device51. For example, the display device53displays an image (for example, an estimated image by the rendering processing) by using image data supplied from the information processing device51.

The information processing device51includes a communicator55, a memory56, a model integrator32, a rendering processor57, and a controller58. For example, the communicator55includes at least one of a USB port, a network card, or a communication device that performs wireless communications using radio waves or infrared rays. The communicator55is communicable with the communicator8in the detection device1aand the communicator8in the detection device1b.

For example, the memory56includes a removable storage medium such as a USB memory, or an external or built-in large-capacity storage device such as a hard disk. For example, the memory56stores therein data on at least a part of the information received via the communicator55, an imaging control program for controlling at least one of the detection device1aand the detection device1b, and a processing program for executing each processing in the information processing device51.

The model integrator32is the same as in first embodiment. For example, the information processing device51receives first model information and first position information from the detection device1a, and receives second model information and second position information from the detection device1b. The model integrator32uses the first position information and the second position information to integrate the first model information and the second model information. The detection device1amay or need not include the model integrator32.

The rendering processor57includes, for example, a graphics processing unit (GPU). The rendering processor57may have an aspect in which a CPU and a memory execute each processing in accordance with an image processing program. For example, the rendering processor57executes at least one of drawing processing, texture mapping processing, or shading processing.

In the drawing processing, for example, the rendering processor57is able to calculate an estimated image (for example, reconstructed image) of a shape determined by shape information in model information as viewed from a certain viewpoint. In the following description, the shape represented by the shape information is referred to as “model shape”. In the drawing processing, for example, the rendering processor57is able to reconstruct a model shape (for example, estimated image) from model information (for example, shape information). The rendering processor57performs rendering processing by using surface characteristics information included in the model information obtained by integrating the first model information and the second model information. For example, the rendering processor57stores data on the calculated estimated image in the memory56. In the texture mapping processing, for example, the rendering processor57is able to calculate an estimated image obtained by attaching an image represented by texture information in model information to the surface of an object on the estimated image. The rendering processor57is also able to calculate an estimated image obtained by attaching a texture different from the object OB to the surface of the object on the estimated image. In the shading processing, for example, the rendering processor57is able to calculate an estimated image obtained by adding a shade formed by a light source represented by light source information in model information to an object on the estimated image. In the shading processing, for example, the rendering processor57is able to calculate an estimated image obtained by adding a shade formed by a desired light source to an object on the estimated image.

For example, the controller58controls the components in the information processing device51, the detection device1a, the detection device1b, the illumination device IL, the input device52, and the display device53. For example, the controller58controls the communicator55to transmit an instruction (a control signal) and setting information to the detection device1aand the detection device1b. For example, the controller58stores information received by the communicator55from the detection device1aand the detection device1bin the memory56. For example, the controller58controls the rendering processor57to execute the rendering processing.

The detection system50does not necessarily include the input device52. For example, various kinds of instructions and information may be input to the detection system50via the communicator8. The detection system50does not necessarily include the display device53. For example, the detection system50may output the data on the estimated image generated by the rendering processing to an external display device, and this display device may display the estimated image.

In the above-mentioned embodiments, a calculator including the information calculator3, the light source calculator4, and the position calculator31includes a computer system, for example. For example, this calculator reads a detection program stored in the memory7, and executes various kinds of processing in accordance with the detection program. For example, the detection program causes a computer to execute: calculating, by using detection results of detecting an object from a first viewpoint, first model information including shape information on the object from the first viewpoint; calculating, by using a plurality of first taken images obtained by imaging a space including a light source that irradiates the object with illumination light (for example, the illumination light L1) and including the object, light source information on the light source; and calculating an absolute or relative positional relation between the first viewpoint and the object by using the light source information as information used to integrate the first model information and second model information including shape information obtained by detecting the object from a second viewpoint different from the first viewpoint. The detection program may be recorded in a computer-readable storage medium to be provided.

The technical scope of the present invention is not limited to the modes described in the above-mentioned embodiments and the like. At least one of the elements described in the above-mentioned embodiments and the like may be omitted. The elements described in the above-mentioned embodiments and the like may be combined as appropriate. To the extent allowed by laws, the disclosure of all the literature cited in the above-mentioned embodiments and the like is incorporated herein by reference.

DESCRIPTION OF REFERENCE SIGNS

1a,1bdetection device,2A imager,2B detector,3information calculator,4light source calculator,5characteristics calculator,8communicator,9controller,11imager,12distance measurer,31position calculator,32model integrator,50detection system, IL illumination device, L1illumination light, L2environment light, OB object, Vp1viewpoint, Vp2viewpoint