PROJECTION SYSTEM AND PROJECTION CONTROL METHOD

Provided is a projection system using a projection device that simultaneously projects on a plurality of projection surfaces. A projection system includes: a user recognition unit that recognizes a user existing in a space; a projection environment recognition unit that recognizes a projection surface on which a video can be projected in the space; and a control unit that controls a projection device so as to project a video on the projection surface recognized by the projection environment recognition unit for the user recognized by the user recognition unit. The projection device can simultaneously project a video on a plurality of surfaces. The control unit controls a projection device so as to simultaneously project a video on two or more projection surfaces recognized by the projection environment recognition unit.

TECHNICAL FIELD

The technology disclosed in the present specification (hereinafter, “the present disclosure”) relates to a projection system and a projection control method for projecting an image on one or a plurality of projection surfaces.

BACKGROUND ART

A projection device also called a “projector” can project a large image on a screen and simultaneously present the image to a plurality of persons, and thus has been used for applications such as presentation for a long time. In recent years, the use of projection devices has been further expanded due to the appearance of the projection mapping technology for pasting a projected video to a three-dimensional object. For example, there has been proposed an image projection device that identifies a plurality of projection surfaces within a projection range on the basis of an imaging signal obtained by imaging the projection range of a projection unit by an imaging unit, and allocates a seed image or a UI image to each projection surface to project the image (see Patent Document 1). Such an image projection device can correct the size, luminance, and chromaticity of a video in consideration of the identified state of each projection surface.

CITATION LIST

Patent Document

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

An object of the present disclosure is to provide a projection system and a projection control method using a projection device capable of performing simultaneous projection on a plurality of projection surfaces.

Solutions to Problems

The present disclosure has been made in view of the above problems, and a first aspect thereof is a projection system including:a user recognition unit that recognizes a user existing in a space;a projection environment recognition unit that recognizes a projection surface on which a video can be projected in the space; anda control unit that controls a projection device so as to project a video on the projection surface recognized by the projection environment recognition unit for the user recognized by the user recognition unit. The projection device can simultaneously project a video on a plurality of surfaces. Then, the control unit controls the projection device so as to simultaneously project the video on the two or more projection surfaces recognized by the projection environment recognition unit.

However, the term “system” referred here indicates a logical assembly of multiple of devices (or functional modules that implement specific functions), and it does not matter whether or not each of the devices or functional modules is in a single housing. That is, one device including multiple components or functional modules and an assembly of multiple devices correspond to the “system”.

At least one of the user recognition unit or the projection environment recognition unit performs recognition on the basis of sensor information detected by a sensor installed in the space.

The projection device includes a phase modulation type spatial light modulator, and can simultaneously project a video on a plurality of surfaces different in vertical and horizontal directions and a depth direction. Therefore, the projection environment recognition unit recognizes a plurality of projection surfaces different in the vertical and horizontal directions and the depth direction.

The user recognition unit defines a characteristic and a state of the user. Furthermore, the projection system according to the first aspect further includes: a content selection unit that selects content to be displayed to a user on the basis of defined user information; and a projection surface determination unit that determines a projection surface on which the selected content is projected.

The projection environment recognition unit detects information of the recognized projection surface such as an attribute, a shape, an area, and characteristics (reflectance, luminance, chromaticity) of the projection surface.

Furthermore, the projection system according to the first aspect further includes a projection parameter correction unit that corrects a projection parameter for the projection surface determined by the projection surface determination unit. The projection parameter correction unit limits at least one of a distance between projection surfaces different in a depth direction, the number of projection surfaces, or a projection size on the basis of a design value of the projection device, determines priorities of a plurality of the projection surfaces determined by the projection surface determination unit, and the projection parameter correction unit performs corrects luminance, chromaticity, and a size of a projected video on the projection surface.

Furthermore, a second aspect of the present disclosure is a projection control method including:a user recognition step of recognizing a user existing in a space;a projection environment recognition step of recognizing a projection surface on which a video can be projected in the space; anda control step of controlling a projection device so as to project a video on the projection surface recognized by the projection environment recognition step for the user recognized by the user recognition step.

Effects of the Invention

According to the present disclosure, it is possible to provide a projection system and a projection control method that project content to a user recognized in a space on the projection surfaces recognized in the space by using a projection device capable of simultaneously projecting on a plurality of projection surfaces.

Note that, effects described in the present specification are merely examples, and the effects brought about by the present disclosure are not limited thereto. Furthermore, the present disclosure may further provide additional effects in addition to the effects described above.

Still other objects, features, and advantages of the present disclosure will become apparent from a more detailed description based on embodiments as described later and the accompanying drawings.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, the present disclosure will be described in the following order with reference to the drawings.A. OverviewB. Projection PrincipleC. System ConfigurationC-1. System Configuration Example (1)C-2. System Configuration Example (2)C-3. Projection Device Capable of Simultaneously Projecting on a Plurality of Projection Surfaces Different in Depth DirectionC-4. System Configuration Example (3)C-5. System Configuration Example (4)C-6. System Configuration Example (5)C-7. Correction of Projection ParameterC-7-1. Limitation of Projection SurfaceC-7-2. Priority Determination on Projection SurfaceC-7-3. Correction of Projection SurfaceC-7-3-1. Geometric CorrectionC-7-3-2. Luminance CorrectionC-7-3-3. Chromaticity CorrectionC-7-3-4. Specific ExamplesC-8. Modality of Input InformationC-9. Detection of Meta Information from Device Possessed by UserD. ApplicationD-1. Example 1D-2. Example 2D-3. Example 3D-4. Example 4D-5. Example 5D-6. Example 6

There has already been proposed a projection device that performs simultaneous projection on a plurality of projection surfaces and corrects the size, luminance, chromaticity, and the like of a video according to the state of the projection surfaces (see Patent Document 1). On the other hand, the present disclosure further proposes a projection system that includes a user recognition function of recognizing a user existing in a space and a projection environment recognition function of recognizing a projection environment of the space, and projects content on a projection surface recognized in the space to the user recognized in the space.

In a case where the projection system is mounted on a vehicle, the space includes both the inside and outside of the vehicle. Then, according to the present disclosure, it is possible to project content directed to each of a passenger and a pedestrian near the vehicle, for example, on projection surfaces recognized inside and outside the vehicle. Furthermore, the space may be an indoor space in which an immersive virtual reality (VR) system such as cave automatic virtual environment (CAVE) or Warp is constructed. Then, according to the present disclosure, content can be projected to each user in an immersive space on one or more projection surfaces recognized in the immersive space.

According to the projection system to which the present disclosure is applied, not only the user existing in the space can be recognized by the user recognition function, but also the characteristic and state of each user can be further recognized, and appropriate content can be selected in consideration of the characteristic and state of the user.

Furthermore, according to the projection system to which the present disclosure is applied, the projection surface in the space can be recognized by the projection environment recognition function, and the projection surface can be allocated to each user. At that time, the projection surface can be allocated according to the content selected for each user. Furthermore, projection parameters can be corrected in consideration of the projection environment, the characteristics of the projection surface, the image quality when the content is projected, and the like.

A projection system according to the present disclosure uses a projection device capable of simultaneously projecting on a plurality of projection surfaces. The plurality of projection surfaces means a plurality of projection surfaces different not only in the vertical and horizontal directions (in other words, projection directions) but also in the depth direction. Incidentally, the image projection device described in Patent Literature 1 can simultaneously project on two projection surfaces different in the vertical and horizontal directions, but cannot simultaneously project on a plurality of projection surfaces different in the depth direction.

According to the present disclosure, by using a projection device capable of simultaneously projecting on a plurality of projection surfaces different in the vertical and horizontal directions and in the depth direction, it is possible to realize adaptive projection control of content in consideration of both a state of a user and a projection environment, and it is possible to improve usability. In particular, since simultaneous projection can be performed on a plurality of projection surfaces including the depth direction with only one projection device, space saving is achieved, which is advantageous, for example, as an in-vehicle system. Furthermore, by concentrating the projection light of the projection device only on a necessary portion, energy efficiency and cost efficiency can be improved.

As described above, the projection system according to the present disclosure uses a projection device capable of simultaneously projecting on a plurality of projection surfaces different in the depth direction in addition to the vertical and horizontal directions. In the present embodiment, simultaneous projection of a video on a plurality of projection surfaces different in the depth direction is realized using a light modulation element.

In general, a spatial light modumator (SLM) has only an element that can independently modulate amplitude modulation or phase modulation. In the former amplitude modulation scheme, the interference fringe intensity distribution of object light and reference light is displayed on an amplitude modulation type SLM by, for example, computer-generated hologram (CGH), and the amplitude modulation type SLM is irradiated with the reference light to generate reproduction light of an object. Although detailed description is omitted, in such amplitude modulation scheme, the object light can be reproduced as it is, but there is a problem in that a large amount of unnecessary light is generated.

On the other hand, the phase modulation also includes holography, but any light intensity can be created at any position by wavefront control of the light. In the phase modulation scheme, when the phase distribution of the object light is displayed on the phase modulation type SLM and the phase modulation type SLM is irradiated with the reference light, the wavefront of the passing light is controlled to generate the reproduction light of the object. The phase modulation scheme is more advantageous than the amplitude modulation scheme because the object light can be reproduced with high light utilization efficiency without generating unnecessary components in addition to being able to reproduce the phase of the object light correctly. As the phase modulation type SLM, a phase modulation type liquid crystal on silicon (LCOS) or a phase modulation type microelectromechanical systems (MEMS) is used. For example, Patent Literature 2 refers to a projector using an SLM. Hereinafter, a method of projecting on different projection surfaces in the depth direction using the phase modulation scheme will be described.

Object light O (x, y) on the xy plane can be expressed by an amplitude component A0(x, y) and a phase component exp (iφ0(x, y)) as shown on the right side of the following formula (1). However, the xy plane is defined as a zero position in the depth direction. The zero position in the depth direction corresponds to an “SLM plane” or a “hologram plane” on which the phase modulation type SLM is installed. If the object light O (x, y) can be reproduced at the zero position in the depth direction, it is possible to freely record and reproduce light.

Phase distribution P (x, y) of the object light O (x, y) is as shown in the following formula (2) as can be seen from the above formula (1). In the phase modulation scheme, the phase distribution P (x, y) is displayed on a phase modulation type SLM (phase modulation type LCOS or the like) arranged at the zero position in the depth direction, and the reference light R (x, y) is multiplied, whereby reproduction light O′ (x, y) of the object can be generated as shown in the following formula (3). The right side of the following formula (3) includes only the reproduction light obtained by phase-modulating the reference light R (x, y). Therefore, according to the phase modulation scheme, it can be understood that object light can be efficiently reproduced without generating unnecessary components.

Subsequently, a case of reproducing wavefront information of two pieces of object light O1(x1, y1) and O2(x2, y2) respectively arranged at different positions r1and r2in the depth direction will be described with reference toFIGS.45and46. The object light O1(x1, y1) and the object light O2(x2, y2) are expressed by the following formulas (4) and (5), respectively. However, (x1, y1) is an xy coordinate system on the position r1in the depth direction, and (x2, y2) is an xy coordinate system on the position r2in the depth direction.

As illustrated inFIG.45, the object light O1(x1, y1) and the object light O2(x2, y2) are arranged at different positions r1and r2, respectively, in the depth direction from the zero position. First, wavefront information of each object light O1(x1, y1) and O2(x2, y2) is synthesized on a virtual space, and the object light O (x, y) at the zero position in the depth direction is calculated. The following formula (6) shows a calculation formula for synthesizing the object light O1(x1, y1) and the object light O2(x2, y2). However, since the calculation formula cannot be solved analytically, the following formula (6) shows an approximate formula. Although various calculation methods of the approximate formula are proposed, detailed description thereof is omitted here.

Then, at the time of reproduction of the object light O1(x1, y1) and the object light O2(x2, y2), the phase information exp (iφ0(x, y)) included in the composite wavefront information calculated on the right side of the above formula (6) is displayed on the SLM arranged at the zero position in the depth direction, and the collimated reference light (or parallel light) AR(x, y) is incident on the SLM as illustrated inFIG.46, so that the passing light is wavefront controlled by the SLM. As a result, reproduction light O1′ (x1, y1) and reproduction light O2′ (x2, y2) of each object illustrated in the following formulas (7) and (8), respectively, can be reproduced at the same position as in the calculation illustrated inFIG.45.

Each of the reproduction light O1′ (x1, y1) and the reproduction light O2′ (x2, y2) is equivalent to arbitrary two-dimensional information. Therefore, it can be said that videos V1and V2can be simultaneously projected on the two surfaces of the positions r1and r2different in the vertical and horizontal directions and the depth direction.

As described above, the phase modulation type projection device is used as the best embodiment of the present disclosure because the phase modulation scheme can reproduce object light with high light utilization efficiency without generating unnecessary components. The distribution of light can be reproduced at an arbitrary position in each of the xyz directions, and simultaneous projection on a plurality of projection surfaces different in the depth direction in addition to the vertical and horizontal directions can be performed by one projection device. However, even in the amplitude modulation scheme, it is possible to realize display on a plurality of surfaces having different depths according to the principle of holography, and a projection device of the amplitude modulation scheme may be used if the fact that unnecessary components are generated and light utilization efficiency is lowered is not a problem.

Since simultaneous projection can be performed on a plurality of projection surfaces including the depth direction by only one projection device, space saving is achieved. As a matter of course, if there are no restrictions on space efficiency, energy consumption, and cost, the projection system may adopt a multi-projector in which a plurality of projection devices is integrated. In the following description, unless otherwise specified, it is assumed that the projection system uses only one projection device of the phase modulation scheme.

Note that examples of phase distribution generation algorithm for generating the phase distribution displayed on the phase modulation type SLM include a GS method and a method of calculating a freeform phase, but are not limited thereto.

C. System Configuration

In section C, a configuration of a projection system to which the present disclosure is applied will be described.

C-1. System Configuration Example (1)

FIG.1schematically illustrates a configuration of a projection system100to which the present disclosure is applied. The illustrated projection system100includes a user recognition unit101, a projection environment recognition unit102, and an output control unit103.

The user recognition unit101recognizes a user existing in a space within a range that can be projected by the projection system100. The user recognition unit101basically recognizes the user on the basis of sensor information acquired by a sensor installed in the same space. In a case where the projection system100is mounted on a vehicle, the user recognition unit101recognizes a user inside the vehicle (a passenger or the like) and a user outside the vehicle (a pedestrian or the like around the vehicle). The user recognition unit101further recognizes the characteristic and state of the user, which will be described later in detail.

The projection environment recognition unit102recognizes a portion that can be actually projected by the projection device110as a projection surface in a space within a range that can be projected by the projection system100. The projection environment recognition unit102basically recognizes the projection surface on the basis of sensor information acquired by a sensor installed in the same space. The projection environment recognition unit102further recognizes the characteristic and state of the projection surface, which will be described later in detail.

The output control unit103controls the output of the projection device110on the basis of the respective recognition results of the user recognition unit101and the projection environment recognition unit102so as to project a video on the projection surface and display information to the user. The projection device110is a phase modulation type projection device as described in the above section B, and is a projection device capable of simultaneously projecting on a plurality of projection surfaces different in the vertical and horizontal directions and the depth direction.

The output control unit103basically controls the projection operation of the video on the projection surface of the projection device110by allocating an appropriate one or a plurality of projection surfaces among the one or a plurality of projection surfaces recognized by the projection environment recognition unit102to the one or a plurality of users recognized by the user recognition unit101.FIG.1illustrates an example in which the projection device110simultaneously projects videos on two projection surfaces, a first projection surface (x1, y1, z1) and a second projection surface (x2, y2, z2). Moreover, the output control unit103can select appropriate content in consideration of the characteristic and state of the user recognized by the user recognition unit101and allocate a projection surface suitable for projection of the selected content, but details of this point will be described later.

C-2. System Configuration Example (2)

FIG.2schematically illustrates a configuration of a projection system200to which the present disclosure is applied. The illustrated projection system200includes an input unit201, a user recognition unit202, a projection environment recognition unit203, and an output control unit204. However, the same components as those included in the projection system100illustrated inFIG.1are denoted by the same names.

The input unit201inputs sensor information acquired by a sensor installed in a space within a range that can be projected by the projection system200. Alternatively, the input unit201may be the sensor itself installed in the space. The sensor includes an image sensor, a distance sensor, and the like. The sensor may further include a position sensor such as a thermo camera, an ultrasonic sensor, a touch sensor, or a global positional system (GPS) sensor, and various other sensors capable of sensing information regarding the environment of the space. In a case where the space within a range that can be projected is in the inside of a vehicle, the input unit201inputs, for example, sensor information from an in-vehicle sensor installed inside the vehicle. In a case where the user recognition unit202also recognizes a user outside the vehicle, the input unit201also inputs sensor information outside the vehicle.

The user recognition unit202recognizes the user on the basis of the sensor information supplied from the input unit201, and further recognizes the characteristic and state of the user. In a case where the projection system200is mounted on a vehicle, the user recognition unit202recognizes a user inside the vehicle (a passenger or the like) and a user outside the vehicle (a pedestrian or the like around the vehicle).

The projection environment recognition unit203includes a projection surface detection unit203-1. On the basis of the sensor information supplied from the input unit201, the projection surface detection unit203-1detects, as a projection surface, a portion that can be actually projected by a projection device210in a space within a range that can be projected by the projection system100.

The output control unit204controls the output of the projection device210on the basis of the respective recognition results of the user recognition unit202and the projection surface detection unit203-1so as to project a video on the projection surface and display information to the user. The projection device210is a phase modulation type projection device as described in the above section B, and is a projection device capable of simultaneously projecting on a plurality of projection surfaces different in the vertical and horizontal directions and the depth direction.

The output control unit204basically controls the projection operation of the video on the projection surface of the projection device210by allocating an appropriate one or a plurality of projection surfaces among the one or a plurality of projection surfaces recognized by the projection surface detection unit203-1to the one or a plurality of users recognized by the user recognition unit204. Moreover, the output control unit204selects appropriate content in consideration of the characteristic and state of the user recognized by the user recognition unit202, and allocates a projection surface suitable for projection of the selected content.

The projection surface detection unit203-1will be described more specifically. From the projectable range of the projection device210, the projection surface detection unit203-1detects, as a projection surface, a region that satisfies a condition defined by one or more thresholds, such as an area equal to or larger than a predetermined threshold, a curvature equal to or smaller than a predetermined threshold, or a gradient equal to or larger than a predetermined threshold (an angle formed by projection light and the projection surface). The threshold for detecting the projection surface may be defined for each user, may be defined for each piece of content, or may be defined for a combination of the user and the content. Furthermore, a threshold value for detecting the projection surface may be defined for each application to which the projection system200is applied.

FIGS.3and4illustrate sensor information acquired by a sensor installed in a space within a range that can be projected by the projection system200. Here, it is assumed that the projection system200is mounted on a vehicle and used, and sensor information inside the vehicle and around the vehicle is acquired as the sensor information. The sensor includes an image sensor, a distance sensor, and the like, and the input unit201inputs image data of the inside of the vehicle as illustrated inFIG.3and image data of the periphery of the vehicle as illustrated inFIG.4. Furthermore, the input unit201may simultaneously input sensor information acquired by a thermo camera, an ultrasonic sensor, a touch sensor, and various other sensors capable of sensing information regarding the environment of the space in the same environment.

The projection surface detection unit203-1detects a plurality of projection surfaces suitable for the user inside the vehicle recognized by the user recognition unit202on the basis of the image data of the inside of the vehicle illustrated inFIG.3. In the example illustrated inFIG.5, the projection surface detection unit203-1detects a plurality of projection surfaces having an area equal to or larger than a predetermined threshold, a curvature equal to or smaller than a predetermined threshold, and a gradient equal to or larger than a predetermined threshold (an angle formed by projection light and the projection surface), such as a headrest of the seat in front, a ceiling, and a pillar, as the projection surface suitable for the user (not illustrated) sitting on the last seat. InFIG.5, each detected projection surface is displayed in light gray. Furthermore, the projection surface detection unit203-1does not detect a portion having an area equal to or larger than the predetermined threshold and a curvature equal to or smaller than the predetermined threshold, but having a gradient of less than the threshold as the projection surface.

Furthermore, the projection surface detection unit203-1detects a projection surface suitable for the user around the vehicle recognized by the user recognition unit202on the basis of the image data of the periphery of the vehicle illustrated inFIG.4. In the example illustrated inFIG.6, in a case where a woman waiting for a traffic light at the intersection around the vehicle is recognized as a user outside the vehicle, the projection surface detection unit203-1detects a projection surface having an area equal to or larger than a predetermined threshold, a curvature equal to or smaller than a predetermined threshold, and a gradient equal to or larger than a predetermined threshold (an angle formed by projection light and the projection surface), such as a road surface in front of the woman. InFIG.6, the projection surface detected on the road surface in front of the woman is displayed in light gray.

C-3. Projection Device Capable of Simultaneously Projecting on a Plurality of Projection Surfaces Different in Depth Direction

As described in the above section B, in the present disclosure, simultaneous projection on a plurality of projection surfaces different in the depth direction in addition to the vertical and horizontal directions is realized using the phase modulation type projection device.FIG.7illustrates an operation example of the projection device110inFIG.1. The same applies to the operation of the projection device210inFIG.2.

The projection device110includes a phase modulation type SLM (phase modulation type LCOS or the like)701. Phase information included in composite wavefront information of videos to be projected onto each of two projection surfaces (x1, y1, z1) and (x2, y2, z2) different in the depth direction is displayed on a phase modulation type SLM701, and when reproduction light (substantially parallel light) obtained by collimating irradiation light of a light source (not illustrated inFIG.7) is incident from the rear of the phase modulation type SLM701, desired videos are simultaneously projected onto the respective front projection surfaces (x1, y1, z1) and (x2, y2, z2).

Note that examples of the phase distribution generation algorithm for generating the phase distribution displayed on the phase modulation type SLM701include the GS method and the method of calculating a freeform phase (described above), but are not limited thereto.

Furthermore, a luminance modulation panel (not illustrated) may be arranged at the subsequent stage of the phase modulation type SLM701. By using the luminance modulation panel, the luminance dynamic range of the projected video can be extended and the resolution can be improved. In this case, processing of determining the transmittance or the reflectance of the luminance modulation panel is performed. However, it should be noted that in a case where the luminance dynamic range is extended, the overall luminance decreases, and thus, in addition to this, the resolution of the projected video can be improved.

By using the phase modulation type projection device110, it is possible to simultaneously project on a plurality of projection surfaces different in the depth direction. In a case where the projection system100is mounted on a vehicle, multi-projection on a plurality of projection surfaces such as a headrest, a pillar, and a ceiling inside the vehicle can be realized as illustrated inFIG.5.

Furthermore, according to the projection device110, focusing on a moving object can be performed by changing the phase distribution displayed on the phase modulation type SLM701. For example, in projection mapping interaction with a dish, focusing can be performed even when a plate is lifted. As another example of focusing, moving production can be performed in applications (described later) such as various projection mapping events including a fashion show, bowling, and other sports competitions.

The projection device110can also be used as a light source of structured light. The structured light is one method of three-dimensional measurement, and can irradiate an object with structured light patterned in a dot shape or the like and acquire depth information from distortion of the pattern. By using the phase modulation type projection device110, projection can be performed without changing density according to the depth.

Furthermore, the projection device110can be applied to a VR system such as CAVE or Warp to display a video in a wide area by one device.

Furthermore, the projection device110can be used as an indicator of a touch sensor in an aerial display that displays a video in the air (described later). By using the projection device110, a dark and small point can be presented in a case where the distance is long, and a large and bright point can be presented in a case where the distance is short.

C-4. System Configuration Example (3)

FIG.8schematically illustrates a configuration of a projection system800to which the present disclosure is applied. The illustrated projection system800includes an input unit801, a user information detection unit802, a projection environment recognition unit803, and an output control unit804. However, the same components as those included in the projection system100illustrated inFIG.1or the projection system200illustrated inFIG.2are denoted by the same names. The projection system800has a main feature in a method of detecting user information.

The input unit801inputs sensor information acquired by a sensor installed in a space within a range that can be projected by the projection system800. Alternatively, the input unit801may be the sensor itself installed in the space. The sensor includes an image sensor, a distance sensor, and the like. The sensor may further include a thermo camera, an ultrasonic sensor, a touch sensor, and various other sensors capable of sensing information regarding the environment of the space. The input unit801inputs, for example, sensor information from an in-vehicle sensor installed inside the vehicle. In a case where the user recognition unit802also recognizes a user outside the vehicle, the input unit801also inputs sensor information outside the vehicle.

The user information detection unit802includes a user recognition unit802-1and a user definition unit802-2.

The user recognition unit802-1recognizes the user on the basis of the sensor information supplied from the input unit801. The user recognition unit802-1detects the number of users, the position of the user, and the direction of the face and the line-of-sight of the user by particularly using image information of an RGB camera and a distance sensor as the sensor information. The user recognition unit802-1can realize such posture recognition by using a posture estimation model such as Openpose developed by Carnegie Mellon University, for example. In a case where the projection system800is mounted on a vehicle, the user recognition unit802-1recognizes a user inside the vehicle (a passenger or the like) and a user outside the vehicle (a pedestrian or the like around the vehicle).

The user definition unit802-2defines the characteristic and state of the user recognized by the user recognition unit802-1. When the user recognition unit802-1recognizes a plurality of users, the characteristic and state are defined for each user. The user definition unit802-2compares, for example, with a database describing stereotype information of the user, and defines characteristic data of the user recognized from the image. Furthermore, the user definition unit802-2defines the state of the user such as awakening or sleeping on the basis of the recognition result by the user recognition unit802-1. The user definition unit802-2can estimate the state of the user using, for example, the number of blinks, movement of the line-of-sight, and the like as parameters. Furthermore, the user definition unit802-2may estimate the attribute of the user using a learned machine learning model. The user definition unit802-2stores the characteristic and state defined for each user recognized by the user recognition unit802-1in a user characteristic database.

The projection environment recognition unit803includes a projection surface detection unit803-1. On the basis of the sensor information supplied from the input unit801, the projection surface detection unit803-1detects, as a projection surface, a portion that can be actually projected by a projection device810in a space within a range that can be projected by the projection system800. As already described in section C-2 above, from the projectable range of the projection device810, the projection surface detection unit803-1detects, as a projection surface, a region that satisfies a condition defined by one or more thresholds, such as an area equal to or larger than a predetermined threshold, a curvature equal to or smaller than a predetermined threshold, or a gradient equal to or larger than a predetermined threshold (an angle formed by projection light and the projection surface).

The output control unit804controls the output of the projection device810on the basis of the respective recognition results of the user recognition unit802-1and the projection surface detection unit803-1so as to project a video on the projection surface and display information to the user. The projection device810is a phase modulation type projection device as described in the above section B, and is a projection device capable of simultaneously projecting on a plurality of projection surfaces different in the vertical and horizontal directions and the depth direction.

Subsequently, the operations of the user recognition unit802-1and the user definition unit802-2in the user information detection unit802will be mainly described specifically in detail by taking a case in which the projection system800is mounted on a vehicle as an example.

The user recognition unit802-1recognizes three users (user1, user2, user3) inside the vehicle as illustrated inFIG.9, for example, on the basis of the image data of the inside of the vehicle. Furthermore, the user recognition unit802-1recognizes a user (user4) walking around the vehicle as illustrated inFIG.10, for example, on the basis of the image data of the periphery of the vehicle. Then, an entry of each user (user1, user2, user3, user4) is provided in the user characteristic database.

Subsequently, the user definition unit802-2defines characteristic data for each user by associating stereotype information with each recognized user. For example, as characteristic data, “male, thirties, company employee” is defined for the user1, “female, thirties, housewife” is defined for the user2, “infant” is defined for the user3, and “female, twenties, undergraduate” is defined for the user4.

Furthermore, the user definition unit802-2defines the state of the user such as awakening or sleeping of each recognized user. The user definition unit802-2can estimate the state of the user using, for example, the number of blinks, movement of the line-of-sight, and the like as parameters. For example, as characteristic data, “driving” is defined for the user1, “waking up” is defined for the user2, “waking up” is defined for the user3, and “waking up and waiting at a traffic light” is defined for the user4.

Moreover, the user definition unit802-2defines whether information can be displayed to the user (Yes) or not (No) on the basis of the characteristic and state of the user. The display of the information referred to here means projection of a video onto a projection surface allocated to the user by the projection device810. For example, whether or not to display the information of each user is defined such that the user1is “No”, the user2is “Yes”, the user3is “No”, and the user4is “Yes”. For example, while a user in an awake state is defined as information display “Yes”, a user who is driving or sleeping, a user who is awake but operates a smartphone or listens to music, a baby, or the like is defined as information display “No”.

Then, the characteristic, the state, and the information display defined for each user by the user definition unit802-2are stored in the entry of each user in the user characteristic database.FIG.11illustrates a user characteristic database storing information of each user defined by the user definition unit802-1.

Note that it is assumed that what kind of user characteristic or state is defined for the recognition result by the user recognition unit802-1differs according to the definition rule or the machine learning model used by the user definition unit802-2.FIG.11is an example of the user characteristic database, and it is also assumed that different characteristics and states are defined according to a definition rule or a machine learning model to be used.

The output control unit804controls information display to each user on the basis of the user characteristic database as illustrated inFIG.11. Specifically, the output control unit804controls the output of the projection device810such that the video of content is projected on each of the projection surfaces by allocating any one of the projection surfaces detected by the projection surface detection unit803-1to the user2and the user4defined as the information display “Yes” on the user characteristic database. On the other hand, the output control unit804does not allocate the projection surface to the user1and the user3defined as the information display “No” on the user characteristic database, and therefore does not project the video to these users.

C-5. System Configuration Example (4)

FIG.12schematically illustrates a configuration of a projection system1200to which the present disclosure is applied. The illustrated projection system1200includes an input unit1201, a user information detection unit1202, a projection environment recognition unit1203, and an output control unit1204. However, the same components as those included in the projection system800illustrated inFIG.8are denoted by the same names. The projection system1200has a main feature in that content to be displayed to the user is selected and a projection surface is determined on the basis of user information.

The input unit1201inputs sensor information acquired by a sensor installed in a space within a range that can be projected by the projection system1200. Alternatively, the input unit1201may be the sensor itself installed in the space. The input unit1201inputs, for example, sensor information from an in-vehicle sensor installed inside the vehicle. In a case where the user recognition unit1202also recognizes a user outside the vehicle, the input unit1201also inputs sensor information outside the vehicle.

The user information detection unit1202includes a user recognition unit1202-1and a user definition unit1202-2. The user recognition unit1202-1detects the number of users, the position of the user, and the direction of the face and the line-of-sight of the user from the sensor information supplied from the input unit1201using, for example, the posture estimation model such as Openpose. Then, the user definition unit1202-2defines the characteristic and state of the user recognized by the user recognition unit1202-1, and stores the characteristic and state defined for each user in the entry of the corresponding user in the user characteristic database.

The projection environment recognition unit1203includes a projection surface detection unit1203-1. On the basis of the sensor information supplied from the input unit1201, the projection surface detection unit1203-1detects, as a projection surface, a portion that can be actually projected by a projection device1210in a space within a range that can be projected by the projection system1200. As already described in section C-2 above, from the projectable range of the projection device1210, the projection surface detection unit1203-1detects, as a projection surface, a region that satisfies a condition defined by one or more thresholds, such as an area equal to or larger than a predetermined threshold, a curvature equal to or smaller than a predetermined threshold, or a gradient equal to or larger than a predetermined threshold (an angle formed by projection light and the projection surface).

The output control unit1204includes a content selection unit1204-1and a projection surface determination unit1204-2. The content selection unit1204-1selects content to be displayed to the user on the basis of the user information recognized by the user recognition unit1202. Furthermore, the projection surface determination unit1204-2determines a projection surface on which the video of the content is projected from among the projection surfaces detected by the projection surface detection unit1203-1.

Furthermore, the output control unit1204performs projection size determination processing on the projection surface allocated to each piece of content by the projection surface determination unit1204-2, and projection luminance determination processing for determining the luminance and chromaticity of the video to be projected. As the projection size determination processing, the projectable size with respect to the projection surface is calculated on the basis of the distance between the projection surface and the projection device1210, the projectable size is compared with a recommended size of the content, and content reduction processing is performed as necessary so that the content is within the projectable size. As the projection luminance determination processing, the output and the correction amount of the projection device1210are calculated from the characteristics of the projection surface (chromaticity, luminance, reflectance). As for the chromaticity, a chromaticity correction value of a color space is calculated on the basis of chromaticity information of the projection surface. In a case where the chromaticity of a certain pixel in the content exceeds the displayable chromaticity of the projection device1210, the chromaticity is determined so as to match the displayable chromaticity of the projection device1210. As for the luminance, the possible output of the projection device1210is calculated from the total number of projection surfaces on which the content is projected and the content to be projected, the feasible luminance is further calculated in consideration of the reflectance of the projection surface, and the feasible luminance is compared with the luminance of the original signal of the content is compared. In a case where the luminance of the original signal of the content is less than the luminance of the original signal, the output luminance is decreased to the feasible luminance, or setting values (current value, duty value, and the like) of the projection device1210at that time is calculated. In a case where the determination is made by priority of chromaticity, the determination is made in the order of chromaticity->luminance. However, in a case where the determination is made by priority of luminance, the determination is not limited to this order, and the determination method is not limited to the above. Display information such as the determined projection size, luminance, and chromaticity is stored in the entry of the corresponding content in the user characteristic database.

Then, the output control unit1204controls the output of the projection device1210so as to project a video on the projection surface and display information to the user. The projection device1210is a phase modulation type projection device as described in the above section B, and is a projection device capable of simultaneously projecting on a plurality of projection surfaces different in the vertical and horizontal directions and the depth direction. The output control unit1204performs display image generation processing on the basis of the projection size, the projection luminance, and the projection chromaticity determined for each projection surface on which the content is projected. The display image generation processing includes display target generation processing, display phase distribution generation processing, and drive parameter setting processing. In the display target generation processing, a luminance distribution target to be displayed for monochrome or each color channel is generated. In the display phase distribution generation processing, a phase distribution is generated for each calculated luminance distribution target. In a case where the luminance correction is performed, optical correction information corresponding to the luminance correction amount is added to the generated phase distribution. Examples of the phase distribution generation algorithm include the GS method and the freeform method, but are not limited thereto. In the drive parameter setting processing, a drive parameter (current value in case of CW, duty in case of pulse) of the light source of the projection device1210is set so as to display the determined luminance and chromaticity. Furthermore, in a case where a luminance modulation panel at the subsequent stage of the phase modulation type SLM is arranged, the transmittance or the reflectance of the luminance modulation panel is determined. The output control unit1204outputs the determined information to the projection device1210to project a video.

Hereinafter, each of the content selection unit1204-1and the projection surface determination unit1204-2will be described in detail.

The content selection unit1204-1selects content to be displayed to the user on the basis of the user information (that is, the user characteristic database) defined by the user definition unit1202-1. Specifically, the content selection unit1204-1compares the characteristic and state of the user defined by the user definition unit1202-2with the content database (attribute information of each piece of content or the like), and matches the content to be displayed to the user. Collaborative filtering (CF), other recommendation technologies, or a machine learning model can be applied to the matching, but is not limited to a specific method. Then, the content selection unit1204-1stores the information of the content (access information to content, such as a content name or a uniform resource locator (URL)) selected for the user in the entry of the corresponding user in the user characteristic database.

The content selection unit1204-1may select a plurality of pieces of content for one user. In a case where there is a plurality of pieces of content to be displayed to one user, for example, the content may be stored in the entry of the corresponding user in the user characteristic database in order of priority based on any of the following rules (1) to (3). Furthermore, the priority order of a plurality of pieces of content may be determined on the basis of a learned machine learning model instead of the following rule base.(1) Priority order based on predetermined user preference(2) Priority based on current position information of user (priority of surrounding facility, event information, surrounding weather forecasts, or the like).(3) Priority based on current time (for example, news is prioritized at 7:00 AM, 12:00 in the daytime, and 18:00 in the evening).

Furthermore, the content selection unit1204-1may select a plurality of pieces of content to be displayed to one user. In this case, an entry for storing two or more contents for the corresponding user may be added to the user characteristic database.

Note that it is sufficient that the content selection unit1204-1selects the content only for the user whose information display is defined as “Yes”, and does not select the content for the user whose information display is defined as “No”. The entry of the user whose information display is defined as “No” may be deleted from the user characteristic database.

The projection surface determination unit1204-2determines a projection surface for projecting the content selected by the content selection unit1204-1from among the projection surfaces detected by the projection surface detection unit1203-1. The projection surface determination unit1204-2performs determination processing of a projection surface for each piece of content selected for the user by the content selection unit1204-1. First, the projection surface determination unit1204-2determines whether or not the projection surface detected by the projection surface detection unit1203-1is present in the field of view of the user from which the content has been selected. Here, in a case where the projection surface detected by the projection surface detection unit1203-1is present in the field of view of the target user, the projection surface determination unit1204-2stores the projection surface in association with the user (alternatively, the content selected for the user) in the entry of the corresponding user in the user characteristic database. On the other hand, in a case where the projection surface is not present in the field of view of the target user, the projection surface determination unit1204-2does not associate the projection surface with the user.

The projection surface determination unit1204-2may allocate the projection surface to the content on the basis of any one of the following priority orders (1) to (6).(1) Descending order of area (absolute value) of projection surface(2) Descending order of angle of view with respect to user(3) Ascending order of distance between user and projection surface(4) Order closer to recommended screen size or recommended angle of view of content(5) Descending order of contrast between background of projection surface and content(6) Ascending order of background luminance of projection surface

Subsequently, an operation in which the projection system1200selects content to be displayed to the user on the basis of the user information and determines a projection surface will be specifically described by taking a case of being mounted on a vehicle as an example.

The user recognition unit1202-1recognizes three users (user1, user2, user3) inside the vehicle as illustrated inFIG.9, for example, on the basis of the image data of the inside of the vehicle. Furthermore, the user recognition unit1202-1recognizes a user (user4) walking around the vehicle as illustrated inFIG.10, for example, on the basis of the image data of the periphery of the vehicle. Subsequently, the user definition unit1202-2defines characteristic data for each user by associating stereotype information with each recognized user (user1, user2, user3, user4). Then, the user characteristic database as illustrated inFIG.11is constructed.

On the basis of the sensor information supplied from the input unit1201, the projection surface detection unit1203-1of the projection environment recognition unit1203detects, as a projection surface, a portion that can be actually projected by the projection device1210in each of the inside and the outside of the vehicle. Here, it is assumed that the projection surface detection unit1203-1detects a total of nine projection surfaces #001 to #009 inside the vehicle as illustrated inFIG.13, and detects one projection surface #101 on the road surface near the user4outside the vehicle as illustrated inFIG.14.

The content selection unit1204-1selects content to be displayed for each of two users of the user2and the user4whose information display is defined as “Yes” in the user characteristic database illustrated inFIG.11. Here, the content selection unit1204-1selects three pieces of content of “Okinawa resort advertisement”, “news”, and “map” for the user2, selects one piece of content of “shopping mall advertisement” for the user4, and stores the selected pieces of content in the entry of the corresponding user in the user characteristic database.

Subsequently, the projection surface determination unit1204-2determines, from among the projection surfaces detected by the projection surface detection unit1203-1, the projection surface on which each piece of content “Okinawa resort advertisement”, “news”, “map”, and “shopping mall advertisement” selected by the content selection unit1204-1is projected. As described above, the projection surface determination unit1204-2determines the projection surface on which the content is present in the field of view of the selected user, and determines the projection surface on which the content selected by the user is displayed from among the projection surfaces in the field of view of the user according to the priority order (described above). Here, the projection surface #009 detected in the field of view of the user2is determined as a projection surface on which the content “Okinawa resort advertisement” and “news” are displayed, the other projection surface #003 detected in the field of view of the user2is determined as a projection surface on which the content “map” is displayed, and the projection surface #101 detected in the field of view of the user4is determined as a projection surface on which the content “shopping mall advertisement” is displayed.

Then, the projection surface determination unit1204-2stores the projection surface allocated to each piece of content in the entry of the corresponding user in the user characteristic database.FIG.15illustrates a user characteristic database storing the content selected by the content selection unit1204-1for the user and the information on the projection surface allocated to each piece of content by the projection surface determination unit1204-2.

C-6. System Configuration Example (5)

FIG.16schematically illustrates a configuration of a projection system1600to which the present disclosure is applied. The illustrated projection system1600includes an input unit1601, a user information detection unit1602, a projection environment recognition unit1603, and an output control unit1604, a projection parameter correction unit1605. However, the same components as those included in the projection system1200illustrated inFIG.12are denoted by the same names. The projection system1600has a main feature in that the characteristic of a projection surface is defined on the basis of sensor information, and a projection surface having a characteristic suitable for content projection is determined.

The input unit1601inputs sensor information acquired by a sensor installed in a space within a range that can be projected by the projection system1600. Alternatively, the input unit1601may be the sensor itself installed in the space. The input unit1601inputs, for example, sensor information from an in-vehicle sensor installed inside the vehicle. In a case where the user recognition unit1602also recognizes a user outside the vehicle, the input unit1601also inputs sensor information outside the vehicle.

The user information detection unit1602includes a user recognition unit1602-1and a user definition unit1602-2. The user recognition unit1602-1detects the number of users, the position of the user, and the direction of the face and the line-of-sight of the user from the sensor information supplied from the input unit1601using, for example, the posture estimation model such as Openpose. Then, the user definition unit1602-2defines the characteristic and state of the user recognized by the user recognition unit1602-1, and stores the characteristic and state defined for each user in the entry of the corresponding user in the user characteristic database.

The projection environment recognition unit1603includes a projection surface detection unit1603-1and a projection surface definition unit1603-2. On the basis of the sensor information supplied from the input unit1601, the projection surface detection unit1603-1detects, as a projection surface, a portion that can be actually projected by the projection device1610in a space within a range that can be projected by the projection system1600(the same as above). In the projection system1600, a projection surface database is used to manage the characteristics of each projection surface detected by the projection surface detection unit1603-1. An entry of each projection surface detected by the projection surface detection unit1603-1is provided in the projection surface database.

The projection surface definition unit1603-2recognizes the characteristics of the projection surface detected by the projection surface detection unit1603-1and stores the information in the corresponding entry of the projection surface database. Specifically, the projection surface definition unit1603-2allocates characteristic information such as attribute, shape, area, reflectance, chromaticity, and luminance to the projection surface, and stores the characteristic information in the projection surface database. As for the attribute of the projection surface, the projection surface definition unit1603-2may perform clustering in comparison with a predetermined database, or may perform clustering using a learned machine learning model. The projection surface definition unit1603-2calculates the shape, area, reflectance, chromaticity, and luminance of the projection surface on the basis of the sensor information acquired by the input unit1601.FIG.17illustrates an example of the projection surface database storing characteristic information defined for each projection surface detected inside and outside the vehicle as illustrated inFIGS.13and14. However, the chromaticity is represented by a coordinate position on the chromaticity map.

The characteristic data of the projection surface defined by the projection surface definition unit1603-2can be used when the projection surface suitable for the content is determined by the projection surface determination unit1604-2in the subsequent stage and, moreover, when projection parameters are corrected by the projection parameter correction unit1605.

The output control unit1604includes a content selection unit1604-1and a projection surface determination unit1604-2. The content selection unit1604-1selects content to be displayed to the user on the basis of the user information recognized by the user recognition unit1602. Furthermore, the projection surface determination unit1604-2determines a projection surface to be allocated to the user or a projection surface having a characteristic of projecting the video of the content selected for the user from among the projection surfaces detected by the projection surface detection unit1603-1. The information regarding the content selected by the content selection unit1604-1and the projection surface of the content determined by the projection surface determination unit1604-2is stored in the user characteristic database (described above).

The projection parameter correction unit1605has a function of correcting the projection parameters such as on the content selected by the content selection unit1604-1and the projection surface determined by the projection surface determination unit1604-2in order to maintain the projection quality. For example, when the area of the projection surface is smaller than the recommended screen size of the content, the projection parameter correction unit1605performs processing of reducing the screen size of the original content. Furthermore, when the luminance and chromaticity of the projected video are greatly different from those of the original content due to the characteristics of the projection surface, which is the background, the projection parameter correction unit1605performs signal processing so that the luminance and chromaticity of the projected video becomes closer to the original luminance and chromaticity. The display information such as the projection size, luminance, and chromaticity corrected by the projection parameter correction unit1605is stored in the entry of the corresponding content in the user characteristic database. The detailed function of the projection parameter correction unit1605will be described in the subsequent section C-7. The correction processing by the projection parameter correction unit1605may be performed in real time.

The projection device1610is a phase modulation type projection device as described in the above section B, and is a projection device capable of simultaneously projecting on a plurality of projection surfaces different in the vertical and horizontal directions and the depth direction. Display image generation processing for generating an image to be projected by the projection device1610is performed on the basis of the projection size, the projection luminance, and the projection chromaticity corrected by the projection parameter correction unit1605. This display image generation processing may be performed by the output control unit1604or may be performed by the projection parameter correction unit1605. Since the display image generation processing is as described in the above section C-5, the detailed description thereof will be omitted here. The projection device1610projects a video on the basis of the information determined by the display image generation processing.

Subsequently, an effect that the projection system1600is equipped with the projection surface definition function will be specifically described by taking a case of being mounted on a vehicle as an example. However, in the projection environment recognition unit1603, it is assumed that the projection surface database is updated in real time (alternatively, in a short control cycle) on the basis of the sensor information acquired by the input unit1601from moment to moment.FIG.18illustrates a projection surface database in which the projection surface detected inside the vehicle in the daytime and the characteristic data of each projection surface defined in the daytime are stored. Furthermore,FIG.19illustrates a projection surface database in which the projection surface detected inside the vehicle at night and the characteristic data of each projection surface defined at night are stored.

Here, an example will be considered in which when the content selection unit1603-1selects content having a recommended screen size of 0.45 m2, an optimum projection surface is determined from among the projection surfaces #001 to #004 inside the vehicle illustrated inFIG.13. However, the luminance of the projection device1610is set to 500 ml, the recommended screen size of the content is set to 0.45 m2, and the maximum area occupancy (the ratio of the effective projection area that can be actually projected in the entire projection range that can be projected by the projection device1610) is set to 10%. Then, the projection surface determination unit1604-2prioritizes the projection surfaces on the basis of the following conditions (1) and (2).(1) Same as or higher than the content recommended screen size, and closest to the content recommended screen size(2) Contrast between the content and the projection surface is 2:1 or more.

First, a case where the projection surface of the content is determined on the basis of the projection surface database defined in the daytime will be described. In the daytime, two projection surfaces #003 and #001 are selected in order of being closest to the content recommended screen size from the above condition (1) on the basis of the projection surface database ofFIG.18. Subsequently, the content luminance at the time of projection on the projection surfaces #001 and #003 will be calculated. The content luminance at the time of projection on the projection surface #001 is 1768 nits as shown in the following formula (9). Then, with reference toFIG.18, the luminance of the projection surface #001 in the daytime is 52 nits. Therefore, the contrast between the content and the projection surface #001 is 1768:52=34:1, and it can be seen that the projection surface #001 in the daytime also satisfies the above condition (2).

On the other hand, the content luminance at the time of projection on the projection surface #003 is 2475 nits as shown in the following formula (10). Then, with reference toFIG.18, the luminance of the projection surface #003 in the daytime is 10000 nits. Therefore, the contrast between the content and the projection surface #003 is 0.24:1, and it can be seen that the projection surface #003 in the daytime does not satisfy the above condition (2). Therefore, in the daytime, the projection surface determination unit1604-2determines the projection surface #001, which satisfies both the conditions (1) and (2), as the projection surface suitable for the projection of the content.

Subsequently, a case where the projection surface of the content is determined on the basis of the projection surface database defined at night will be described. At night, the two projection surfaces #003 and #001 are selected in order of being closest to the content recommended screen size from the above condition (1) on the basis of the projection surface database inFIG.19. Subsequently, the content luminance at the time of projection on the projection surfaces #001 and #003 will be calculated. The content luminance at the time of projection on the projection surface #001 is 1768 nits as shown in the above formula (9), but with reference toFIG.19, the luminance of the projection surface #001 at night is 10 nits. Therefore, the contrast between the content and the projection surface #001 is 177:1, and it can be seen that the projection surface #001 also satisfies the above condition (2) even at night.

Furthermore, the content luminance at the time of projection on the projection surface #003 is 2475 nits as shown in the above formula (10), but with reference toFIG.19, the luminance of the projection surface at night is 20 nits. Therefore, the contrast between the content and the projection surface #003 is 123:1, and the projection surface #003 also satisfies the above condition (2) even at night. Therefore, at night, the projection surface determination unit1604-2determines the projection surface #003 closest to the content recommended area as the projection surface suitable for the projection of the content.

As described above, in the projection system1600, the projection surface determined for the same content varies on the basis of the projection environment sensed in real time such as daytime and nighttime. That is, in the projection system1600, since the projection surface more suitable for the content selected for the user can be dynamically determined by combining the projection surface definition unit1603-2and the projection surface determination unit1604-2, it is possible to flexibly perform projection with higher usability.

C-7. Correction of Projection Surface

The projection system1600illustrated inFIG.16has a function of correcting the projection parameters such as on the content selected by the content selection unit1604-1and the projection surface determined by the projection surface determination unit1604-2in order to maintain the projection quality. Furthermore, when the luminance and chromaticity of the projected video are greatly different from those of the original content due to the characteristics of the projection surface, which is the background, the projection parameter correction unit1605performs signal processing so that the luminance and chromaticity of the projected video becomes closer to the original luminance and chromaticity.

In section C-7, a correction function of the projection parameters performed by the projection parameter correction unit1605on the content selected by the content selection unit1604-1, the projection surface determined by the projection surface determination unit1604-2, and the like will be described.

C-7-1. Limitation of Projection Surface

First, in section C-7-1, a function will be described in which the projection parameter correction unit1605imposes a restriction on the projection surface determined by the projection surface determination unit1604.

The projection device1610can simultaneously project a plurality of projection surfaces different in the depth direction by the phase modulation scheme. However, as illustrated inFIG.20, in a case where the distance between the two different projection surfaces2001and2002is smaller than a design value, the projection device1610may not be able to switch the video because the projection surface on the far side is too close. Specifically,FIGS.21and22illustrate cases where, when a vehicle is parked or stopped in a parking space near a wall surface such as a fence, the projection surface determination unit1604-2determines the window surface of the vehicle window and the wall surface in the vicinity of the vehicle window as the projection surface (FIG.21illustrates a state in which a vehicle parked in the vicinity of a wall surface is viewed from above, andFIG.22illustrates a state in which the vehicle is viewed from the side).

Furthermore, in the examples illustrated inFIGS.20to22, two projection surfaces in which the distance therebetween is small are illustrated, but it is also assumed that the distances of three or more projection surfaces are smaller than the design value. In such a case, the projection parameter correction unit1605may pick up two of each of a plurality of projection surfaces (a vehicle window and a wall surface immediately adjacent to the vehicle window, or the like) and compare the two with each other to determine a projection surface to be prioritized. For example, the projection parameter correction unit1605gives priority to the projection surface closer to the user. In a case where the projection system1600is mounted on a vehicle as illustrated inFIGS.21to22, the projection parameter correction unit1605may prioritize a projection surface inside the vehicle. This is because if a wall surface outside the vehicle is also projected, a video projected on the wall surface and a video on the near side, that is, on the projection surface inside the vehicle are mixed, whereby it is difficult to see the video. In the example illustrated inFIG.20, the projection parameter correction unit1605may correct the ratio of brightness to be distributed to the projection surface2001on the near side (window surface of the vehicle window) and the projection surface2002on the far side (wall surface outside the vehicle) to 100%: 0%.

Furthermore, the projection parameter correction unit1605may impose the following restrictions (1) to (3) on the projection surface.(1) In a case where the number of projection surfaces is close to the design value of the projection device1610, the projection parameter correction unit1605narrows down the number of projection surfaces to a predetermined number or less according to a predetermined prioritization rule. For example, the projection parameter correction unit1605may preferentially select the projection surface closer to the user. In a case where the projection system1600is mounted on a vehicle, the projection parameter correction unit1605may preferentially select a projection surface inside the vehicle.(2) In a case where the projection light amount allocated to the projection surface exceeds the design value of the projection device1610, the projection parameter correction unit1605may perform correction so as to reduce the projection area of the projection surface according to a predetermined prioritization rule. For example, the projection size of each projection surface may be uniformly reduced, or the projection size may be reduced in order from the projection surface closer to the user.(3) When the area of the projection surface is smaller than the recommended screen size of the content, the projection parameter correction unit1605performs processing of reducing the screen size of the original content.

C-7-2. Priority Determination on Projection Surface

In section C-7-2, a function will be described in which the projection parameter correction unit1605determines priorities of the plurality of projection surfaces determined by the projection surface determination unit1604-2will be described.

In a case where the projection light amount allocated to each projection surface exceeds the design value of the projection device1610, the projection parameter correction unit1605adjusts the distribution of the luminance to each projection surface according to a predetermined prioritization rule. Examples of the prioritization rule include the following rules (1) and (2). However, the following rules are merely examples, and other rules can be determined.(1) On the basis of position information, advertisement is prioritized in a place with a high priority set by the user.(2) Luminance at a position close to the user is prioritized.

FIG.23illustrates an operation example of the projection parameter correction unit1605in a case where the content selection unit1604-1selects each piece of content of news “ABC” for the inside of the vehicle and an advertisement “DEF” for the outside of the vehicle, and the projection surface determination unit1604-2further determines the window surface as the projection surface of the news “ABC” for the inside of the vehicle and determines the road surface as the projection surface of the advertisement “DEF” for the outside of the vehicle. For example, in a case where a vehicle is traveling in a downtown where a large amount of advertisement revenue can be expected by displaying outside the vehicle in the daytime when brightness is required, the allocation of the projection light amount to each projection surface (or each piece of content) is corrected with priority given to the projection of the advertisement “DEF” for the outside of the vehicle over the news “ABC” for the inside of the vehicle. In the example illustrated inFIG.23, the projection parameter correction unit1605allocates 20% of the projection light amount to the news “ABC” for the inside of the vehicle and allocates 80% of the projection light amount to the advertisement “DEF” for the outside of the vehicle. Furthermore,FIG.24illustrates a state in which when a vehicle equipped with the projection system1600is traveling in town, news “ABC” for the inside of the vehicle is projected on a window surface of the vehicle window, and an advertisement “DEF” for the outside of the vehicle is projected on a road surface with a respective corrected projection light amount.

C-7-3. Correction of Projection Surface

In section C-7-3, a function will be described in which the projection parameter correction unit1605corrects the projection parameter so that the actual video projected on the projection surface is appropriate on the basis of the information obtained by the projection surface determination unit1604-2and the projection surface definition unit1603-2. Examples of the correction function include geometric correction, luminance correction, and chromaticity correction. Hereinafter, each correction function will be described below.

The projection parameter correction unit1605calculates the size and shape of the video when the content selected by the content selection unit1604-1is actually projected on the projection surface on the basis of the relative positional relationship between the projection surface determined by the projection surface determination unit1604-1and the projection device1610and the angle of view information of the projection device1610. In a case where the projection system1600is mounted on a vehicle, the relative position between the projection surface and the projection device1610can be calculated on the basis of current position information of the vehicle measured by a GPS sensor or the like, predicted route information of the vehicle, or the like. Then, the projection parameter correction unit1605corrects the projection parameter so as to minimize the difference between the recommended size and shape of the content and the calculated size and shape of the projected video. The projection parameter correction unit1605also performs what is-called trapezoid correction accompanying the inclination of the projection surface.

In a case where there is a difference between the luminance value of the original signal of the content and the luminance value projected on the projection surface, the projection parameter correction unit1605corrects the video signal so that the projected luminance value becomes closer to the luminance value of the original signal. Of course, the projection parameter correction unit1605may perform correction such that the projected luminance value becomes closer to the luminance value of the original signal using means other than the signal processing.

The projection parameter correction unit1605may calculate the luminance value projected on the projection surface in advance on the basis of the information in the projection surface database (attribute, shape, reflectance, luminance, chromaticity, and the like of the projection surface) and the design value of the projection device1610. Alternatively, the projection parameter correction unit1605may receive sensor information obtained by sensing the luminance value of the actually projected projection surface from the input unit1601to be used for the luminance correction.

Furthermore, the projection parameter correction unit1605calculates the possible output of the projection device1610from the total number of projection surfaces on which the content is projected and the content to be projected, and further calculates the feasible luminance in consideration of the reflectance of the projection surface. Then, the feasible luminance is compared with the luminance of the original signal of the content, and in a case where the feasible luminance is less than the luminance of the original signal, the output luminance is reduced to the feasible luminance, or setting values (current value, duty value, and the like) of the projection device1610at that time is calculated.

In a case where there is a difference between the chromaticity of the original signal of the content and the chromaticity projected on the projection surface, the projection parameter correction unit1605performs chromaticity correction on the color space of the video signal such that the projected chromaticity becomes closer to the chromaticity of the original signal. Of course, the projection parameter correction unit1605may perform chromaticity correction such that the projected chromaticity becomes closer to the chromaticity of the original signal using means other than the signal processing. Note that, in a case where the chromaticity of a certain pixel in the content exceeds the displayable chromaticity of the projection device1610, the projection parameter correction unit1605corrects the chromaticity of the pixel so as to match the displayable chromaticity of the projection device1610.

The projection parameter correction unit1605may calculate the chromaticity projected on the projection surface in advance on the basis of the information in the projection surface database (attribute, shape, reflectance, luminance, chromaticity, and the like of the projection surface) and the design value of the projection device1610. Alternatively, the projection parameter correction unit1605may receive sensor information obtained by sensing the chromaticity of the actually projected projection surface from the input unit1601to be used for the chromaticity correction.

White balance correction is performed with priority given to light source control of the projection device1610. Therefore, the projection parameter correction unit1605may calculate the light source control parameter and pass the information to the projection device1610.

C-7-3-4. Specific Examples

In section C-7-3-4, a specific example of the projection parameter correction performed by the projection parameter correction unit1605will be described.

FIG.25illustrates a method in which the projection parameter correction unit1605performs correction according to the distance from the projection device1610. In a case where a video is projected at three locations of a short distance, a middle distance, and a long distance, the projection parameter correction unit1605performs luminance correction such that the longer the distance, the brighter the luminance, and the luminance is dark at a short distance. Furthermore, in a case where the projection surface is bright, the luminance of the projected video on the projection surface may be reduced. Furthermore, as geometric correction, the projection parameter correction unit1605performs enlargement and reduction processing on the video such that the size of the video projected on each projection surface is constant.

FIG.25illustrates an example in which arrows for presenting the traveling direction (or navigation) are projected on projection surfaces at three locations of a short distance, a middle distance, and a long distance. The projection parameter correction unit1605first allocates 20%, 40%, and 100% of the projection light amounts to the respective projection surfaces at a short distance, a middle distance, and a long distance. Here, in a case where the projection surface is bright, the contrast between the background (projection surface) and the projected content is small, and thus, correction for adjusting the contrast by increasing the luminance is performed. In the example illustrated inFIG.25, since the projection surface at the middle distance is bright, correction is performed to increase the allocated 40% of the projection light amount to 60%. With such luminance correction, the projection light amount can be effectively distributed according to the distance and the brightness of the projection surface, so that high luminance of the projection device1610can be realized. Furthermore, as geometric correction, the projection parameter correction unit1605also performs trapezoid correction of performing enlargement processing on the video projected on the projection surface at a short distance and performing reduction processing on the video projected on the projection surface at a long distance. As a result, the size of the video projected on each of projection surfaces at a short distance, a middle distance, and a long distance becomes constant.

FIG.26illustrates a specific example of performing projection parameter correction in a case where the projection system1600is mounted on a vehicle (not illustrated inFIG.26). The drawing is an example in which, when the vehicle approaches a Y-shaped road branching from a main road to a left side road, arrows2601to2604presenting a traveling direction are projected on a plurality of portions on the road surface. The projection parameter correction unit1605first allocates a projection light amount according to the distance to each projection surface so as to make the projection surface bright on the far side and dark on the near side. Therefore, the projection light amount is allocated in the order of the arrow2601, the arrow2602, the arrow2603, and the arrow2604. Moreover, since the projection surface on the side groove is brighter than the projection surface on the asphalt, correction for adjusting the contrast by increasing the projection light amount allocated to the arrow2602projected on the side groove is performed. With such luminance correction, the projection light amount can be effectively distributed according to the distance and the brightness of the projection surface, so that high luminance of the projection device1610can be realized. Furthermore, as geometric correction, the projection parameter correction unit1605performs geometric correction according to the inclination and shape of the projection surface, such as trapezoid correction. Furthermore, the projection parameter correction unit1605performs enlargement processing on the video projected on the projection surface at a short distance and performs reduction processing on the video projected on the projection surface at a long distance such that the size of the video projected on each projection surface is constant (performs reduction and enlargement processing in order of the arrow2601, the arrow2602, the arrow2603, and the arrow2604). With these projection parameter corrections, it is possible to project the arrows2601to2604, which look the same from the user (for example, a driver sitting in a driver's seat) on a plurality of the projection surfaces having different distances, attributes, and the like on the road surfaces.

FIGS.27and28illustrate how the projection system1600mounted on a vehicle projects an advertisement for the outside of the vehicle on a sidewalk toward a pedestrian in the vicinity of the vehicle. However,FIG.27illustrates a state in which an advertisement for the outside of the vehicle is projected from the vehicle onto the sidewalk at a certain time T, andFIG.28illustrates a state in which the advertisement for the outside of the vehicle is projected from the vehicle onto the sidewalk at a time T+ΔT that is advanced by ΔT. In the example illustrated inFIG.27, at time T, the content (advertisement for the outside of the vehicle) selected for the pedestrian by the content selection unit1604-1is projected on the projection surface on the sidewalk determined for the pedestrian by the projection surface determination unit1604-2. During the period ΔT, the vehicle moves in the traveling direction, and is further about to turn left at the intersection. Therefore, the distance from the projection device1610to the projection surface on the sidewalk increases, and the inclination of the projection surface with respect to the optical axis also changes. Therefore, the projection parameter correction unit1605corrects (brightens) the projection light amount allocated to the projection surface on the sidewalk according to the change in the distance to the projection surface, and also performs geometric correction according to the inclination of the projection surface with respect to the optical axis. The relative position between the projection surface and the projection device1610can be calculated on the basis of current position information of the vehicle measured by a GPS sensor or the like, predicted route information of the vehicle, or the like. As a result, as illustrated inFIG.28, since the coordinates of the projection surface on the sidewalk with respect to the pedestrian and the size of the projected video become constant, the pedestrian can continue to see the same advertisement for the outside of the vehicle between time T and time T+ΔT.

In the case of continuously presenting the content, the projection surface (sidewalk) allocated to the user (pedestrian) is fixed, and the projection parameter correction unit1605updates the projection parameter (geometric correction, luminance correction, chromaticity correction) for each frame, so that information presentation securing the usability can be performed even while the vehicle on which the projection system1600is mounted is moving.

C-8. Modality of Input Information

Heretofore, the description is given assuming that the input unit1601mainly inputs image information from an image sensor, a distance sensor, or the like, and the recognition of the user by the user recognition unit1602-1and the definition of the user information by the user definition unit1602-2are performed.

The input unit1601may further acquire sound data using a microphone or the like. In such a case, it is possible to more accurately grasp the attribute of the user and the state of the user by acquiring conversation, daily life sounds uttered by the user, other environmental sounds, and the like from the sound data.

C-9. Detection of Meta Information from Device Possessed by User

It has become common for users to carry multifunctional information terminals such as smartphones, tablets, and smartwatches. This type of device often stores meta information of various users, such as schedule information, e-ticket information, and information of services and facilities in which accounts are registered.

Therefore, the input unit1601may acquire the meta information of the user from a device possessed by the user. In such a case, the user definition unit1602-2can more accurately define the user information, and the content selection unit1604-1can select the content suitable for the attribute of the user and the state of the user.

FIG.29schematically illustrates a configuration of a projection system2900to which the present disclosure is applied. The illustrated projection system2900includes an input unit2901, a user information detection unit2902, a projection environment recognition unit2903, and an output control unit2904, a projection parameter correction unit2905, and an application information accumulation unit2906. However, the same components as those included in the projection system1600illustrated inFIG.16are denoted by the same names. The projection system2900has a main feature in that application information including allocation of a projection surface and parameters related to content selection for each application (or use of the system) is managed, so that the projection system can be deployed to a wide range of applications (can be used for various scenes).

The input unit2901inputs sensor information acquired by a sensor installed in a space within a range that can be projected by the projection system2900. Alternatively, the input unit2901may be the sensor itself installed in the space. The input unit2901inputs, for example, sensor information from an in-vehicle sensor installed inside the vehicle. In a case where the user recognition unit2902also recognizes a user outside the vehicle, the input unit2901also inputs sensor information outside the vehicle.

The user information detection unit2902includes a user recognition unit2902-1and a user definition unit2902-2. The user recognition unit2902-1detects the number of users, the position of the user, and the direction of the face and the line-of-sight of the user from the sensor information supplied from the input unit2901using, for example, the posture estimation model such as Openpose. Then, the user definition unit2902-2defines the characteristic and state of the user recognized by the user recognition unit2902-1, and stores the characteristic and state defined for each user in the entry of the corresponding user in the user characteristic database.

The projection environment recognition unit2903includes a projection surface detection unit2903-1and a projection surface definition unit2903. On the basis of the sensor information supplied from the input unit2901, the projection surface detection unit2903-1detects, as a projection surface, a portion that can be actually projected by the projection device2910in a space within a range that can be projected by the projection system2900(the same as above). In the projection system2900, a projection surface database is used to manage the characteristics of each projection surface detected by the projection surface detection unit2903-1. An entry of each projection surface detected by the projection surface detection unit2903-1is provided in the projection surface database. A projection surface definition unit2903-2recognizes the characteristics such as attribute, shape, area, reflectance, chromaticity, and luminance of the projection surface detected by the projection surface detection unit2903-1and stores the information in the corresponding entry of the projection surface database.

The output control unit2904includes a content selection unit2904-1and a projection surface determination unit2904-2. The content selection unit2904-1selects content to be displayed to the user on the basis of the user information recognized by the user recognition unit2902. Furthermore, the projection surface determination unit2904-2determines a projection surface on which the video of the content is projected from among the projection surfaces detected by the projection surface detection unit2903-1. The information regarding the content selected by the content selection unit2904-1and the projection surface of the content determined by the projection surface determination unit2904-2is stored in the user characteristic database (described above). However, the content selection unit2904-1selects the content according to a provision related to content selection on the basis of the corresponding application information managed by the application information accumulation unit2906. Furthermore, the projection surface determination unit2904-2determines the projection surface according to a provision related to allocation of the projection surface on the basis of the corresponding application information.

In order to maintain the projection quality, the projection parameter correction unit2905performs correction of projection parameters such as limitation and priority determination of a projection surface when a video is projected from the projection device2910, geometric correction of the projected video, and luminance and chromaticity correction (for details, refer to section C-7 above).

The projection device2910is a phase modulation type projection device as described in the above section B, and is a projection device capable of simultaneously projecting on a plurality of projection surfaces different in the vertical and horizontal directions and the depth direction. Display image generation processing for generating an image to be projected by the projection device2910is performed on the basis of the projection size, the projection luminance, and the projection chromaticity corrected by the projection parameter correction unit2905. This display image generation processing may be performed by the output control unit2904or may be performed by the projection parameter correction unit2905. Since the display image generation processing is as described in the above section C-5, the detailed description thereof will be omitted here. The projection device2910projects a video on the basis of the information determined by the display image generation processing.

The application information accumulation unit2906manages the application information database by storing application information including allocation of a projection surface and parameters related to content selection for each application (or use of the system) as a database. Then, the content selection unit2904-1selects content according to a provision related to content selection on the basis of the corresponding application information. Furthermore, the projection surface determination unit2904-2determines the projection surface according to a provision related to allocation of the projection surface on the basis of the corresponding application information.

FIG.30illustrates a configuration example of an application information database managed by the application information accumulation unit2906. The application information database illustrated in the drawing stores application information related to seven applications of “advertisement projection inside and outside a vehicle”, “multi-surface projection inside a vehicle (headrest, ceiling, pillar, or the like)”, “dish projection mapping”, “projection mapping event (fashion show, bowling)”, “gaze region appreciation in CAVE system (coping with many people)”, “gaze region appreciation in CAVE system (coping with energy saving on a single surface)”, “touch indicator of aerial display”, and “structured light source system”.

In the illustrated application information database, a multi-plane display flag is a flag indicating whether to allocate a plurality of projection surfaces to one user. An application in which the flag is TRUE indicates that multiple projection surfaces are allocated to one user, and an application in which the flag is FALSE indicates that only one projection surface is allocated to one user.

Furthermore, the recommended number of planes is the number of projection surfaces allocated per user. In the case of an application in which the multi-plane display flag is TRUE, the number of projection surfaces used by each user can be defined in advance. On the other hand, in the case of an application in which the multi-plane display flag is FALSE, the recommended number of planes inevitably is one.

Furthermore, in a case where information is presented on a plurality of projection surfaces (however, in a case where the multi-plane display flag is TRUE), a multi-content flag is a flag indicating whether or not the information projected on all the projection surfaces is a series of pieces of content. An application in which the multi-content flag is FALSE indicates that the information related to all projection surfaces is presented (that is, the content is single content). Furthermore, an application in which the multi-content flag is TRUE indicates that irrelevant information is presented for each projection surface (that is, the content is multi-content).

Note that, in the application information database illustrated inFIG.30, the assignment of the multi-plane display flag, the recommended number of planes, and the multi-content flag to each application is an example, and is not necessarily limited to the examples illustrated inFIG.30.

In section D-1, an example in which the projection system2900is applied to an application “multi-surface projection inside a vehicle” will be described.

In this example, as illustrated inFIG.31, a projection device2910and a sensor3101are installed in a rear portion inside a vehicle equipped with three rows of seats. The sensor3101includes an image sensor such as an RGB camera and a distance sensor, and the sensor information acquired by the sensor3101is input from the input unit2901to the projection system2900. Furthermore, the projection device2910can simultaneously project on a plurality of projection surfaces different in the vertical and horizontal directions and the depth direction by the phase modulation scheme. Then, as illustrated inFIG.30, in the application “multi-surface projection inside a vehicle”, the multi-plane display flag is designated as TRUE, the recommended number of planes is designated as four, and the multi-content flag is designated as TRUE.

As illustrated inFIG.5, the projection surface detection unit2903-1detects a plurality of projection surfaces on a headrest, a ceiling, a pillar, or the like. Then, the projection surface definition unit2903-2recognizes the characteristic of each detected projection surface to be stored in the projection surface database.

The content selection unit2904-1selects content to be displayed for the recommended number of planes for each target user. In the user characteristic database, a user with information presentation of Yes is a presentation target. Since the multi-content flag is designated as TRUE in the application, the content selection unit2904-1selects irrelevant content for the recommended number of planes (four) for each target user.

The projection surface determination unit2904-2performs processing of allocating a projection surface to each target user in the user characteristic database. Since the multi-plane display flag is TRUE in the application, the projection surface determination unit2904-2scans all the projection surfaces for each target user and determines the projection surfaces for the recommended number of planes designated in the user characteristic database for each user. However, the number of projection surfaces actually allocated to each user is determined so as not to exceed the maximum number of projection surfaces of the projection device2910. In a case where a certain user can use more projection surfaces than the recommended number of planes, the projection surface determination unit2904-2determines projection surfaces corresponding to the recommended number of planes on the basis of a predetermined prioritization rule (described above).

Note that, in a case where the same projection surface is a candidate for a plurality of target users, the display information is compared between the users, and the same projection surface is determined for the plurality of target users if the display information is the same. Furthermore, in a case where the display information is different between the users, the projection surface is allocated to one user as it is, and is changed to another projection surface for the other users. In a case where no other projection surface remains, the allocation of the projection surface to the user is abandoned.

FIG.32illustrates an example of a user characteristic database constructed in the application “multi-surface projection inside a vehicle”. However, the multi-plane display flag is TRUE, the recommended number of planes is four, and the maximum number of projection surfaces of the projection device2910is 10.

In section D-2, an example in which the projection system2900is applied to the application “dish projection mapping” will be described. In the “dish projection mapping”, for example, as illustrated inFIG.33, an interaction of projecting a video of a character or the like that attracts an appetite onto a plate on which a dish is served is performed. By applying the projection system2900to this application, even if the user lifts the plate, the video projected on the plate can be always focused.

In this example, a projection device2910and a sensor (an image sensor such as an RGB camera, a distance sensor, or the like) are installed on a ceiling of a room such as a kitchen (not illustrated), and the sensor information acquired by the sensor is input from the input unit2901to the projection system2900. Then, as illustrated inFIG.30, in the application “dish projection mapping”, the multi-plane display flag is designated as TRUE, the recommended number of planes is designated as three, and the multi-content flag is designated as FALSE.

As illustrated inFIG.33A, the projection surface detection unit2903-1detects three projection surfaces3301to3303for projecting content related to a dish (cake) in a projection environment in which a plate on which the dish (cake in the illustrated example) is served is placed on a table. Then, the projection surface definition unit2903-2recognizes the characteristic of each projection surface3301to3303detected to be stored in the projection surface database. In the projection environment in which the plate is placed on the table as illustrated in the drawing, the projection surface is detected on both the table and the plate.

The content selection unit2904-1selects content for the recommended number of planes (three) for the target user to be stored in the user characteristic database. Since the multi-content flag is designated as FALSE in the application, the content selection unit2904-1selects a series of three pieces of content related to the dish (cake) for each target user.

FIG.34illustrates a configuration example of the user characteristic database when the content selection unit2904-1selects a series of pieces of content “rabbit cafe” for the target user by assuming a case where only one target user (the user ID is #Uaaaa) is detected by the user information detection unit2902. The “rabbit cafe” includes a total of six pieces of related content of a-1 to a-3 and b-1 to b-3. However, the user characteristic database illustrated inFIG.34is a provisional database in which content candidates are selected for the target user, and the association between the projection surface and the content has not yet been determined. Furthermore, since the “rabbit cafe” includes six pieces of related content exceeding the recommended number of planes, which is three, it is necessary to narrow the selected content to the recommended number of planes.

The projection surface determination unit2904-2performs processing of allocating a projection surface to the target user in the user characteristic database. Since the multi-content flag is FALSE in the application, the projection surface determination unit2904-2scans for the number of the series of content selected by the content selection unit2904-1for the target user, and allocates the projection surfaces for the recommended number of planes matching the recommended value (recommended screen size or the like) associated with the content to the target user. Then, the projection surface determination unit2904-2stores the projection surface allocated to each of the series of content in the corresponding entry the user characteristic database.

Note that, in a case where the same projection surface is a candidate for a plurality of target users, the display information is compared between the users, and the same projection surface is determined for the plurality of target users if the display information is the same. Furthermore, in a case where the display information is different between the users, the projection surface is allocated to one user as it is, and is changed to another projection surface for the other users. In a case where no other projection surface remains, the allocation of the projection surface to the user is abandoned.

In the provisional user characteristic database after the content selection illustrated inFIG.34, six pieces of related content exceeding the recommended number of planes, which is three, are selected for the target user (user ID: Uaaaa). In such a case, the projection surface determination unit2904-2narrows down the six pieces of related content to three pieces of related content (a-1, a-3, a-2), which is the recommended number of planes, on the basis of a predetermined prioritization rule (described above), and allocate, to each piece of content, a projection surface (#0001, #0005, #0010) matching the recommended value associated with each piece of content.FIG.35illustrates a final user characteristic database constructed on the basis of the user characteristic database illustrated inFIG.34.FIG.33Billustrates a dish projection mapping example in which three pieces of related content are projected on the projection surfaces3301to3303detected by both the table and the plate. In the projection environment in which the plate is placed on the table as illustrated inFIG.33A, two pieces of content “rabbit cafe a-1” and “rabbit cafe a-2” are projected on the projection surfaces3301and3302detected on the table, and “rabbit cafe a-3” is projected on the projection surface3303detected on the plate. Moreover,FIG.33Cillustrates a dish projection mapping example when the plate is lifted from the table. In the present disclosure, two pieces of content “rabbit cafe a-1” and “rabbit cafe a-2” projected on the table remain when the plate is lifted. On the other hand, the content “rabbit cafe a-3” projected on the plate is projected on the lifted plate as a video with the same size, the same luminance, and in-focus as in a state where the plate is placed on the table. Incidentally, when similar projection mapping is performed using a general projection device, only the content on the plate is reduced when the plate is lifted, and a video having high luminance and a shifted focus is obtained, which is a problem.

In section D-3, an example in which the projection system2900is applied to the application “projection mapping event” to perform the moving production will be described. In normal projection mapping, a projection object serving as a screen is fixed and does not move dynamically. On the other hand, by applying the projection system2900, it is possible to continue projection mapping with a high degree of freedom in the depth direction even when the projection object dynamically moves, whereby usability is improved.

In a case where the projection system2900is applied to the application “projection mapping event” to perform the moving production, as illustrated inFIG.30, the multi-plane display flag is designated as TRUE, the recommended number of planes is designated as seven, and the multi-content flag is designated as FALSE.

In a bowling site, for example, an image sensor such as an RGB camera or a distance sensor is installed on the ceiling or other places, and the sensor information is input from the input unit2901. The user recognition unit2902-1recognizes the user in the bowling site from the sensor information supplied from the input unit2901, and the user definition unit2902-2defines the recognized characteristic and state of the user to be stored in the user characteristic database. The user definition unit2902-2defines information display as Yes for the target user to be a target of information presentation.

The projection surface detection unit2903-1detects a portion that can be projected in the bowling site as a projection surface on the basis of the sensor information supplied from the input unit2901. In the application, not only an object that does not move like the floor of the lane but also the surface of an object that dynamically moves like a rolling ball is detected as a projection surface. Consequently, a video can be projected following the rolling ball, and the moving production can be performed. The projection surface definition unit2903-2recognizes characteristics of a projection surface such as the lane or the rolling ball to be stored in the projection surface database.FIG.36illustrates a plurality of projection surfaces detected on the lane and the surface of the rolling ball. In the drawing, each projection surface is displayed in gray.

The content selection unit2904-1selects content for the recommended number of planes (seven) for the target user. Since the multi-content flag is designated as FALSE in the application, the content selection unit2904-1selects a series of pieces of content for the target user. Then, the content selection unit2904-1stores the series of pieces of content and the like) for the content in the user characteristic database.

The projection surface determination unit2904-2performs processing of allocating a projection surface to the target user in the user characteristic database. Since the multi-content flag is designated as FALSE in the application, the projection surface determination unit2904-2scans for the number of series of content selected by the content selection unit2904-1for the target user, and allocates the projection surfaces for the recommended number of planes matching the recommended value (recommended screen size or the like) associated with the content to the target user. Then, the projection surface determination unit2904-2stores the projection surface allocated to each of the series of content in the corresponding entry the user characteristic database.

FIG.37illustrates a final user characteristic database in which a series of pieces of content selected for the target user and information on the projection surface allocated to each piece of content are stored in the application.

As illustrated inFIG.36, in the application, by using the floor of the lane, the surface of the rolling ball, the wall surface (masking) in front, and the like as a projection surface, it is possible to present information to the target user such as a bowler who has thrown the ball. By using the projection system2900according to the present disclosure for the application, display can be continued with the same brightness even while the ball is rolling, and thus usability is improved.

However, when a video is projected on the lane before the ball is pitched, it is difficult for the bowler to read the lane condition, and when the video is projected in front of the rolling ball, it is difficult to observe the course of the ball. Therefore, as illustrated inFIGS.38and39, the projection surface determination unit2904-2may allocate the projection surface on the lane through which the ball has passed to the content. As illustrated inFIG.38, only one projection surface3801in front of the lane is allocated to the content immediately after the pitching. In this way, since the video is not projected on the lane before the ball is pitched and before the ball passes, visibility is not impaired, and usability is further improved. Thereafter, when the ball rolls on the lane and reaches near the pin deck, four projection surfaces3901to3904are allocated to the content on the lane. Therefore, as the ball moves, the region that can be projected on the lane expands, and more projection surfaces can be effectively used.

Note that, although a specific example in which the application “projection mapping event” is applied to bowling has been described in section D-3, the application can be similarly applied to other sports competitions and events other than sports to perform moving production. For example, in a fashion show, a moving production can be performed by using a floor of a runway, or a costume or a body of a model walking on the runway as a projection surface.

In section D-4, an example in which the projection system2900is applied to the application “gaze region appreciation (coping with many people) in the CAVE system” will be described.

FIG.40illustrates an example of a CAVE system4000. The illustrated CAVE system4000includes a space (such as a room) surrounded by wall surfaces on four sides, and each wall surface can be used as a projection surface. The user can enter the space and enjoy the video projected on the wall surface. By applying the projection system2900to the CAVE system4000, it is possible to present the content selected for each user in a region at which each user is gazing. Furthermore, since the projection system2900can simultaneously project on a plurality of projection surfaces different in the vertical and horizontal directions and the depth direction by using the projection device2910, space efficiency and cost efficiency are high and energy saving can be realized as compared with a case where the number of projectors corresponding to the number of projection surfaces is used.

In a case where the projection system2900is applied to the application “gaze region appreciation in CAVE system (coping with many people)”, as illustrated inFIG.30, the multi-plane display flag is designated as TRUE, the recommended number of planes is designated as five, and the multi-content flag is designated as TRUE.

An image sensor4020such as an RGB camera or a distance sensor is installed above or on the ceiling of the space in which the CAVE system4000is constructed, and captures a user and a projection environment existing in the space. The user recognition unit2902-1recognizes the user in the space from the sensor information supplied from the input unit2901, and the user definition unit2902-2defines the recognized characteristic and state of the user to be stored in the user characteristic database. Since the user who has entered the space basically has an intention of watching the video, the user definition unit2902-2defines information display of all users in the space as Yes (that is, the target user). Furthermore, the projection surface detection unit2903-1detects four wall surfaces surrounding the space as a projection surface on the basis of the sensor information supplied from the input unit2901. Note that one wall surface may be detected as one projection surface, or one wall surface may be divided into a plurality of regions and detected as a projection surface for each region.

The content selection unit2904-1selects content for the recommended number of planes (five) for each target user. Since the multi-content flag is designated as TRUE in the application, the content selection unit2904-1selects content irrelevant to the target user. Then, the content selection unit2904-1stores the content selected for each target user and the like) for the content in the user characteristic database.

The projection surface determination unit2904-2performs processing of allocating a projection surface to the target user in the user characteristic database. Since the multi-content flag is designated as TRUE in the application, the projection surface determination unit2904-2scans all the projection surfaces for each target user and determines the projection surfaces for the recommended number of planes designated in the user characteristic database for each user. However, the number of projection surfaces actually allocated to each user is determined so as not to exceed the maximum number of projection surfaces of the projection device2910. In a case where a certain user can use more projection surfaces than the recommended number of planes, the projection surface determination unit2904-2determines projection surfaces corresponding to the recommended number of planes on the basis of a predetermined prioritization rule (described above).

Note that, in a case where the same projection surface is a candidate for a plurality of target users, the display information is compared between the users, and the same projection surface is determined for the plurality of target users if the display information is the same. Furthermore, in a case where the display information is different between the users, the projection surface is allocated to one user as it is, and is changed to another projection surface for the other users. In a case where no other projection surface remains, the allocation of the projection surface to the user is abandoned.

FIG.41illustrates an example of the user characteristic database constructed in the application “gaze region appreciation (coping with many people) in the CAVE system”. However, the multi-plane display flag is TRUE, the recommended number of planes is five, and the maximum number of projection surfaces of the projection device2910is 10.

By performing content output control in the CAVE system4000on the basis of the user characteristic database as illustrated inFIG.41, the projection system2900can simultaneously project each piece of content determined on the basis of the attributes of users4001and4002on projection surfaces4011and4012determined on the wall surfaces in the vicinity of the users4001and4002as illustrated inFIG.40. In order to simplify the drawing, in the CAVE system400illustrated inFIG.40, only two users are accommodated in the space, but even if a large number of users of three or more are accommodated, it is possible to similarly allocate a projection surface to each user and present individual content.

An example of the CAVE system in which many people appreciate a video is described in the above section D-4, but an example in which the projection system2900is applied to the application “gaze region appreciation in the CAVE system (coping with energy saving on a single surface)” will be described in section D-5. Here, the CAVE system4000having the configuration illustrated inFIG.40is also assumed.

In a case where the projection system4000is applied to the application “gaze region appreciation (coping with energy-saving on a single surface) in the CAVE system”, as illustrated inFIG.30, the multi-plane display flag is designated as FALSE, the recommended number of planes is designated as one, and the multi-content flag is designated as FALSE.

Since the operations of the user information detection unit2902and the projection environment recognition unit2903are similar to those in section D-4 above, detailed description thereof will be omitted here.

Since the multi-content flag is designated as FALSE, the content selection unit2904-1selects content one by one for the target user. Furthermore, since the multi-plane display flag is designated as FALSE, the projection surface determination unit2904-2determines projection surfaces matching the recommended screen information of the content assigned to the target user one by one. Then, the information selected by the content selection unit2904-1and determined by the projection surface determination unit2904-2is stored in the user characteristic database.

As a result, since a single projection surface is allocated to each user accommodated in the space to perform video appreciation, energy saving of the CAVE system4000can be realized by suppressing the output of the projection device2910as compared with the above-described Example 4 in which a plurality of projection surfaces is allocated to each user.

In section D-6, an example in which the projection system2900is applied to the application “touch indicator of the aerial display” will be described.

Here, the aerial display is a display device capable of stereoscopically displaying an image in an empty real space (air), and for example, one or a plurality of stationary projectors is used to form an image in the air by combining a lens, a half mirror, or the like, thereby displaying the image in the air (see, for example, Patent Document 3).

One application of the aerial display may include display of a user interface (UI) screen in an empty air, for example, for a user to operate a device. With the aerial display, the UI screen can be installed at any place without the need to install a real display device.FIG.42illustrates an example in which a UI screen4200including menu buttons for file operation such as open, close, save, and print is displayed on an aerial display. However, even if the user tries to touch the menu button on the UI screen displayed by the aerial display, since there is no entity that can be actually touched and operated, the user cannot obtain the tactile sensation of the fingertip touching the menu button, and thus the operation is difficult.

Therefore, an indicator indicating a distance between the aerial display UI screen and the fingertip to be touched is displayed using the projection system2900, and the tactile sensation of the fingertip is compensated by visual information given by the indicator. Specifically, a dark and small indicator is displayed when the distance of the user's fingertip with respect to the UI screen is long, and a bright and large indicator is displayed when the user's fingertip approaches the UI screen.

In a case where the projection system2900is applied to the application “touch indicator of the aerial display”, as illustrated inFIG.30, the multi-plane display flag is designated as TRUE, the recommended number of planes is designated as four, and the multi-content flag is designated as FALSE.

Since the operations of the user information detection unit2902and the projection environment recognition unit2903are similar to those in section D-4 above, detailed description thereof will be omitted here. Since the multi-content flag is designated as FALSE, the content selection unit2904-1selects a series of pieces of content to be assigned to the target user on the basis of the user information defined by the user definition unit2902-1. Furthermore, the projection surface determination unit2904-2determines projection surfaces matching the recommended screen information of the content assigned to the target user one by one. Then, the information selected by the content selection unit2904-1and determined by the projection surface determination unit2904-2is stored in the user characteristic database.

FIGS.43and44illustrate an example in which the projection system2900is applied to the application “touch indicator of the aerial display”.FIG.43illustrates a state in which a dark small indicator4301is displayed when the distance of the user's fingertip is long with respect to the UI screen4200illustrated inFIG.42. Furthermore,FIG.44illustrates a state in which a bright and large indicator4401is displayed when the distance of the user's fingertip approaches the UI screen4200. The user can grasp the sense of distance to the UI screen4200even if the user cannot obtain the tactile sensation of the UI screen4200on the basis of the visual information obtained from the indicators4301and4401illustrated inFIGS.43and44.

INDUSTRIAL APPLICABILITY

The present disclosure has been described in detail with reference to a specific embodiment. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiment without departing from the gist of the present disclosure.

In the present specification, the embodiment in which the projection system according to the present disclosure is applied to a vehicle or the like has been mainly described, but the gist of the present disclosure is not limited thereto.

Furthermore, the projection system according to the present disclosure basically uses the phase modulation type projection device to simultaneously project on a plurality of projection surfaces different in the vertical and horizontal directions and the depth direction. However, as long as there is no restriction such as space efficiency and energy efficiency, another type of projection device such as an amplitude modulation type projection device (even in the amplitude modulation scheme, display on a plurality of surfaces having different depths can be realized by the principle of holography) or a multi-projector can also be used.

In short, the present disclosure has been described in an illustrative manner, and the contents disclosed in the present specification should not be interpreted in a limited manner. To determine the gist of the present disclosure, the claims should be taken into consideration.

Note that the present disclosure may also have the following configurations.(1) A projection system including:a user recognition unit that recognizes a user existing in a space;a projection environment recognition unit that recognizes a projection surface on which a video can be projected in the space; anda control unit that controls a projection device so as to project a video on the projection surface recognized by the projection environment recognition unit for the user recognized by the user recognition unit.(1-1) The projection system according to (1), in whichthe space is a vehicle or another mobile device,the user recognition unit recognizes a user inside and outside the mobile device, andthe projection environment recognition unit recognizes a projection surface inside and outside the mobile device.(1-2) The projection system according to (1), in whichthe space is surrounded by a wall, andthe projection environment recognition unit recognizes a projection surface from a wall surface surrounding the space.(2) The projection system according to claim1, in whichthe projection device is capable of simultaneously projecting a video on a plurality of surfaces, andthe control unit controls a projection device so as to simultaneously project a video on two or more projection surfaces recognized by the projection environment recognition unit.(3) The projection system according to (1) or (2), in whichat least one of the user recognition unit or the projection environment recognition unit performs recognition on the basis of sensor information detected by a sensor installed in the space.(3-1) The projection system according to (3), in whichthe sensor includes at least one of an image sensor, a depth sensor, a thermo camera, an ultrasonic sensor, or a touch sensor.(4) The projection system according to any one of (1) to (3), in whichthe projection environment recognition unit detects, as a projection surface, a region that satisfies a condition defined by one or more threshold values, such as an area equal to or larger than a predetermined threshold value, a curvature equal to or smaller than a predetermined threshold value, or a gradient equal to or larger than a predetermined threshold value, from a projectable range of the projection device.(4-1) The projection system according to (4), in whichthe threshold value is defined for each user, each piece of content to be projected, or each application to which the projection system is applied.(5) The projection system according to any one of (1) to (4), in whichthe projection device is capable of simultaneously projecting a video on a plurality of surfaces different in vertical and horizontal directions and a depth direction, andthe projection environment recognition unit recognizes a plurality of projection surfaces different in vertical and horizontal directions and a depth direction.(6) The projection system according to (5), in whichthe projection device is a projection device including a phase modulation type spatial light modulator.(7) The projection system according to any one of (1) to (6), in whichthe user recognition unit defines a characteristic and a state of a user.(7-1) The projection system according to (7), in whichthe user recognition unit detects at least one of the number of users, a position of the user, a direction of a face of the user, or a line-of-sight of the user, and defines a characteristic and a state of the user on the basis of the detection result.(7-2) The projection system according to (7-1), in whichthe user recognition unit recognizes a posture of a user from image information obtained by capturing the space with an RGB camera or a depth sensor, using a posture estimation model (Openpose).(7-3) The projection system according to (7), in whichthe user recognition unit defines an attribute (gender, age, occupation, or the like) of a user and a state (awakening or sleeping) of the user as user characteristic data.(7-4) The projection system according to (7-3), in whichthe user recognition unit defines an attribute of a user by associating stereotype information.(7-5) The projection system according to (7-3), in whichthe user recognition unit defines a state of a user on the basis of the number of blinks or movement of line-of-sight.(7-6) The projection system according to (7), in whichthe user recognition unit defines a characteristic or a state of a user on the basis of meta information possessed by a device possessed by the user.(8) The projection system according to (7), further includinga content selection unit that selects content to be displayed to a user on the basis of defined user information.(8-1) The projection system according to (8), in whichthe content selection unit selects content on the basis of a predetermined priority.(8-2) The projection system according to (8-1), in whichthe priority is based on at least one of preference of a user, position information of the user, or current time.(8-3) The projection system according to (8), in whichthe selected content is associated with the user.(8-4) The projection system according to (8), in whichthe selected content and display information including a recommended size and recommended luminance of the selected content are associated with the user.(9) The projection system according to (8), further includinga projection surface determination unit that determines a projection surface on which the selected content is projected.(9-1) The projection system according to (9), in whichthe projection surface determination unit determines whether or not there is a projection surface recognized by the projection environment recognition unit in a field of view of a user, and allocates a projection surface of the user on the basis of the determination result.(9-2) The projection system according to (9), in whichthe projection surface determination unit determines a projection surface on the basis of a predetermined priority.(9-3) The projection system according to (9-2), in whichthe priority includes at least one of an area of a projection surface, a size of an angle of view with respect to a user, a distance between the user and the projection surface, a degree of matching between the projection surface and a recommended screen size or a recommended angle of view of content selected by the user, a magnitude of contrast between a background of the projection surface and the content, or smallness of background luminance of the projection surface.(9-4) The projection system according to (9), in whichthe determined projection surface is associated with the user or the content.(10) The projection system according to any one of (1) to (9), in whichthe projection environment recognition unit further detects information of the recognized projection surface.(11) The projection system according to (10), in whichthe projection environment recognition unit defines an attribute, a shape, an area, and characteristics (reflectance, luminance, chromaticity) of a projection surface.(11-1) The projection system according to (11), in whichthe projection environment recognition unit calculates a shape, an area, and characteristics of a projection surface on the basis of sensor information detected by a sensor installed in the space.(11-2) The projection system according to (11), in whichthe projection environment recognition unit performs clustering of the projection surfaces on the basis of data of each projection surface.(12) The projection system according to (9), further includinga projection parameter correction unit that corrects a projection parameter for the projection surface determined by the projection surface determination unit.(13) The projection system according to (12), in whichthe projection parameter correction unit limits at least one of a distance between projection surfaces different in a depth direction, the number of projection surfaces, or a projection size on the basis of a design value of the projection device.(14) The projection system according to (12) or (13), in whichthe projection parameter correction unit determines priorities of a plurality of the projection surfaces determined by the projection surface determination unit.(15) The projection system according to any one of (12) to (14), in whichthe projection parameter correction unit corrects at least one of luminance, chromaticity, or a size of a projected video on a projection surface.(16) The projection system according to (3), in whichthe sensor includes a microphone, andthe user recognition unit recognizes a state of a user on the basis of sound data, a conversation, or a daily life sound input from the microphone.(17) The projection system according to (8) or (9), in whichthe content selection unit selects content to be displayed to a user on the basis of user information recognized from meta information included in a device possessed by the user.(18) The projection system according to any one of (1) to (17), further includingan application information accumulation unit that accumulates application information defining a display method for each application, in whichat least one of selection of content or determination of a projection surface is performed on the basis of the application information.(19) The projection system according to (18), in whichthe application information includes information indicating whether or not content can be displayed using a plurality of projection surfaces, a recommended number of projection surfaces, and whether or not content irrelevant to each projection surface can be displayed in a case where a plurality of projection surfaces is used.(19-1) The projection system according to (19), in whichin a case of an application that permits use of a plurality of projection surfaces, a projection surface is allocated to a target user on the basis of a recommended number of projection surfaces so as not to exceed a maximum display number of projection surfaces of the projection device from among projection surfaces determined for the target user.(19-2) The projection system according to (19), in whichin a case where the same projection surface is redundantly allocated to a plurality of users, if there is display information common to the users, the overlapping allocation of the projection surfaces is maintained.(20) A projection control method including:a user recognition step of recognizing a user existing in a space;a projection environment recognition step of recognizing a projection surface on which a video can be projected in the space; anda control step of controlling a projection device so as to project a video on the projection surface recognized by the projection environment recognition step for the user recognized by the user recognition step.

REFERENCE SIGNS LIST