Patent Publication Number: US-2023143429-A1

Title: Display controlling device and display controlling method

Description:
FIELD OF INVENTION 
     The present invention relates to an in-vehicle display controlling device and display controlling method. 
     PRIOR ART 
     Conventionally, various types of display devices have been mounted in vehicle. 
     For example, in Patent Document 1 there are descriptions of, for example: “A cockpit camera 8 for imaging a head (a bust), where the location of a viewpoint 30 of the driver is detected through image recognition by image processing means is provided, and a view field image (3 a ) corresponding to the position of the actual viewpoint 30 is displayed on a display screen 35 of a display device 3”; “When the driver wishes to check the ground surface 9 further to the front side than the front of the vehicle 1, viewpoint movement is carried out naturally, without requiring an operation on a switch, or the like, through movement of the head forward so as to view the display screen 35 of the display device 3”; and “The effect is as looking through the display screen 35 to view a region 9 a  that is a hidden surface of the ground surface 9 in front of the vehicle 1. In other words, an effect is produced as if an observation window is provided that passes through the vehicle body structure at the position of the display screen 35.” 
     Moreover, in Document 2 there is a description of “comprising: a display monitor 2 that is disposed movably in front of the driver&#39;s seat; moving means 3 and 4 for moving the display monitor 2; imaging means 5 through 7 for capturing images of the region in front of the vehicle, including a blind region of the vehicle; gazing point identifying means 12 for identifying a gazing point Pa that is viewed by the driver during travel; display controlling means 13 for causing the display monitor 2 to display an image of the vicinity of the gazing point Pa, identified by the gaze line identifying means 12, from images captured by the imaging means 5 through 7; and movement controlling means 13 for controlling the moving means 3 and 4 so that the image of the vicinity of the gazing point Pa that is displayed on the display monitor 2 will be displayed on the display monitor 2 in the gaze line Le of the driver that is facing the gazing point Pa that is identified by the gazing point identifying means 12.” 
     Moreover, Patent Document 3 describes: “A display system for a vehicle that includes a plurality of display screens that are mounted in a vehicle cabin. Cameras are mounted on the outside of the vehicle, corresponding to the positions of the display screens. A (plurality of) sensor(s) is oriented facing the driver of the vehicle to sense the positions of the eyes of the driver. An ECU for a display system analyzes the positions of the eyes of the driver to calculate a gaze line. An image from a camera is displayed on a display screen corresponding to the gaze line of the driver. The result is that the display system provides, to the driver, an image that appears as if the vehicle body were not there, without the vehicle obstructing the view by the driver. Consequently, the display system provides parts wherein the vehicle body is virtually transparent, to thereby expand the field of view provided to the driver.” 
     PRIOR ART DOCUMENTS 
     Patent Documents 
     [Patent Document 1] Japanese Unexamined Patent Application Publication 2019-043317 
     [Patent Document 2] Japanese Unexamined Patent Application Publication 2009-149230 
     [Patent Document 3] Japanese Translation of PCT International Application Publication 2014-534697 
     SUMMARY OF THE INVENTION 
     Problem Solved by the Present Invention 
     The prior art can output a display that provides a virtual view that compensates for blind spots caused by the vehicle body, or the like. However, the prior art cannot be said to envision completely active operations, or the like, of vehicle occupants in relation to the display output. For example, if one wishes to look closely at something that is being viewed directly, moving one&#39;s face closer is a natural action. However, when one wishes to look closely at an object that is displayed on a display device, moving one&#39;s face closer to the display device does not change the resolution, or the like of the display device. Thus, there is no improvement in visibility as there would be when moving one&#39;s face closer when viewing an object directly. While it is necessary to, for example, zoom the image on the display when one wishes to view a displayed object more closely, a display device that, as its primary purpose, is to supplement the field of view in particular, such an operation is not provided. Moreover, in some cases there is a need for not only zooming the image on the display, but also other operations, such as adding other information, to the display output. 
     Moreover, from the perspective of ensuring safe operation or of limitations depending on the seating position in in-vehicle display devices, when a variety of different operations are to be performed the ease of operations is extremely important. 
     Given this, the object of the present invention is to achieve better ease of operations in relation to the display output of an in-vehicle display device. 
     Means for Solving the Problem 
     In order to achieve the object set forth above, one typical display controlling device according to the present invention comprises: image acquiring means for acquiring a captured image wherein the vicinity of a vehicle has been imaged; region setting means for setting a prescribed region between displaying means, installed within a cabin of the vehicle and a position wherein a vehicle occupant is within the cabin of the vehicle; function setting means for setting a display function associated with a prescribed region; viewpoint position detecting means for detecting a position relating to a viewpoint of the vehicle occupant; and image controlling means for controlling the displaying means so as to display the captured image, wherein: when the position relating to the viewpoint is detected as being in a prescribed region, the controlling means execute the display function on the captured image displayed by the displaying means. 
     Effects of the Invention 
     The present invention enables achievement of improved ease of operations with respect to a display output of an in-vehicle display device. Other objects, structures, and effects will be understood from the explanation of an embodiment, given below. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is an explanatory diagram for explaining an overview of a display controlling device according to a first embodiment. 
         FIG.  2    is a structural diagram for a display controlling device according to the first embodiment. 
         FIG.  3    is a flowchart showing a procedure when setting a region. 
         FIG.  4    is a flowchart showing a procedure when operating. 
         FIG.  5    is a specific example of a screen output when setting settings. 
         FIG.  6    is a specific example of a process for assigning a function to a region. 
         FIG.  7    is a specific example of a process for changing a region size. 
         FIG.  8    is a specific example of increasing or decreasing the number of regions. 
         FIG.  9    is an explanatory diagram for a specific example of a display function. 
         FIG.  10    is an explanatory diagram of a zooming function that uses a plurality of regions. 
     
    
    
     DETAILED DESCRIPTION 
     An embodiment will be explained below using the drawings. 
     Embodiment 1 
       FIG.  1    is an explanatory diagram for explaining an overview of a display controlling device according to a first embodiment. A display  30 , which is displaying means, is installed in a dashboard of a vehicle  10 . The display  30  has, for example, a display region from the driver seat side to the passenger seat side of the dashboard. The display control device that is mounted in the vehicle  10 . The display control device acquires the state of a vehicle occupant U 1  (the driver) from an interior camera, to detect a location relating to a viewpoint of the vehicle occupant U 1  (U 1   a  and U 1   b ) (for example, the head thereof) in order to control the display on the display  30 . 
     Specifically, the display controlling device performs image processing on an image captured by an exterior camera based on the position of the head of the vehicle occupant U 1  to calculate an exterior image that would be visible from the viewpoint of the vehicle occupant U 1 .And the display controlling device displays it on the display  30  to provide a view V 2  that virtually passes through the display  30 . The result is that the vehicle occupant U 1  can concatenate the directly visible view V 1  and the virtual view V 2 , expanding the view of the vehicle occupant U 1 . 
     A case wherein an obstacle D 1  exists to the front of the vehicle  10  is shown in  FIG.  1   . The obstacle D 1  is blocked by the hood of the vehicle  10 , so is not included in the view V 1  that could be viewed directly. However, the vehicle occupant U 1  is able to see the obstacle D 1  through a virtual view V 2  that is provided by the display  30  displaying a virtual view image. 
     Moreover, depending on the positional relationship between the head of the vehicle occupant U 1  and the display  30 , the display controlling device executes a function in relation to the display of the display  30 . 
     As a specific example, the vehicle occupant U 1   a  in  FIG.  1    shows a state wherein a driving operation is performed in a normal driving position. On the other hand, the vehicle occupant U 1   b  shows a state that is leaning forward with the face nearer to the display  30 . When the head of the vehicle occupant U 1  is in a prescribed region, set in the vicinity of the display  30 , the display controlling device executes a function to zoom the image on the display of the obstacle D 1 . 
     If the vehicle occupant U 1  is in the state shown as U 1   a , the display  30  displays an image of the obstacle D 1  that is calculated as being visible from that viewpoint as an obstacle image D 1   a . If the vehicle occupant U 1  is in the state shown as U 1   b , an obstacle image D 1   b  wherein the image of the obstacle D 1  is shown with zoomed is displayed. 
     In other words, the obstacle image D 1   b  will be zoomed rather than the image that is calculated as being visible from the position of the vehicle occupant U 1   a .Therefore, the vehicle occupant U 1   b  enable to view the obstacle D 1  in detail. 
     Moving one&#39;s face closer to the object is a natural action, when one wishes to examine an object closely. To handle this action as an operation for requesting an the zoom the image displayed on the display enables the display zooming function to be executed with improved ease of operation. Moreover, the fact that there is no need to touch the display  30  contributes to an improvement in the ease of operation. 
       FIG.  2    is a structural diagram for the display controlling device according to the first embodiment. As illustrated in  FIG.  2    the display controlling device has an exterior camera  11 , an interior camera  12 , the processing unit  20 , and a display  30 . Additionally, although not illustrated, the display controlling device is provided with an arbitrary input interface. The input interface can be a touch panel that is integrated with the display  30 , or can be a physical button. In this embodiment, the explanation will be for that wherein a touch panel that is integrated with the display  30  is provided. 
     The exterior camera  11  is image acquiring means for acquiring a captured image wherein the surroundings of the vehicle have been imaged. 
     The interior camera  12  is imaging means for imaging the interior of the vehicle cabin, to function as a sensor for acquiring the state of a vehicle occupant within the vehicle cabin. 
     The display  30  is displaying means equipped within the cabin of the vehicle. 
     The processing unit  20  has a viewpoint position detecting unit  21 , a region setting unit  22 , a function setting unit  23 , and an image controlling unit  24 . If the processing unit is a computer, functions corresponding to the viewpoint position detecting unit  21 , the region setting unit  22 , the function setting unit  23 , and the image controlling unit  24  should be achieved through execution of a program by a CPU (Central Processing Unit). 
     The viewpoint position detecting unit  21  is viewpoint position detecting means for detecting a position relating to the viewpoint of a vehicle occupant. Specifically, the viewpoint position detecting unit  21  carries out an image recognizing process on an image captured by an interior camera  12  to detect the position of the head of the vehicle occupant, as the position relating to the viewpoint of the vehicle occupant. 
     The region setting unit  22  is region setting means for setting a prescribed region, between the occupancy position of the vehicle occupant within the cabin of the vehicle, and a display  30 . 
     The function setting unit  23  is function setting means for setting a display function corresponding to a prescribed region. An arbitrary function can be set for the display function, such as zooming a captured image, adding the display of route information to the captured image, displaying information regarding playback content, or the like. 
     The image controlling unit  24  is image controlling means for controlling the display of a captured image on the display  30 . The captured image that is displayed on the display  30  can be an image captured by the exterior camera  11 . Preferably, the captured image that is displayed on the display  30  can be a virtual view image that is generated from an image captured by the exterior camera  11 . 
     The virtual view image is that wherein an image of outside of the vehicle, as calculated as being visible from the viewpoint of the vehicle occupant, is generated based on the viewpoint position detected by the viewpoint position detecting unit  21 . 
     Moreover, the image controlling unit  24  executes a display function that has been set by the function setting unit  23  on the captured image that is displayed on the display  30 , when detecting the head of the vehicle occupant (a position relating to the viewpoint) is in a region which is set by the region setting unit  22 . In this case, if the captured image that is displayed on the display  30  is a virtual view image, the image controlling unit  24  will execute the display function on the virtual view image. 
     For example, the region setting unit  22  set a region near the display  30 , and, if the function setting unit  23  associates the display zooming function with that region. In this case, the image controlling unit  24  displays the visible view image with zooming the image displayed on the display  30  if the head of the vehicle occupant enters into a region that is near the display  30 . A part of the visible view image can be zoomed. For example, an object in the visible view image viewed by the vehicle occupant can be identified based on detecting the gaze line of the vehicle occupant and the identified object can be zoomed selectively. 
     Additionally, if the function associated by the function setting unit  23  is that of adding route information to the display, the image controlling unit  24  would superimpose route information on the virtual view image when the head of the vehicle occupant enters into that region. Similarly, if the function associated by the function setting unit  23  is that of displaying music information, the image controlling unit  24  would display information relating to music when the head of the vehicle occupant enters into that region. For example, if the vehicle occupant has interest in a song that is playing and brings his or her face toward the display  30 , information can be provided by displaying the name of the song and the name of the artist. 
     The region setting unit  22  can set multiple divided regions as prescribed regions with different distances from the display  30 . In this case, the function setting unit  23  can set respective functions for the plurality of divided regions. For example, if functions for zooming the image displayed on the display by different scales are set for the plurality of divided regions, the object can be displayed more largely when the face is closer to the display  30 . 
       FIG.  3    is a flowchart showing a set-up procedure to set a region. When the function set for a region is called, the processing unit  20  of the display controlling device sequentially executes the procedure shown in  FIG.  3   . 
     At the beginning of the process, the interior camera  12  starts to acquire an image (Step S 101 ). The viewpoint position detecting unit  21  begins a process for detecting the viewpoint position (for example, the head position of the vehicle occupant) from the image acquired by the interior camera  12  (Step S 102 ). Additionally, the image controlling unit  24  begins to display the positional relationship between the viewpoint position and the display  30  (Step S 103 ). 
     The region setting unit  22  receives an input operation from a vehicle occupant and increases or decreases the number of regions (Step S 104 ) and changes the region size (Step S 105 ). The function setting unit  23  receives selections for functions corresponding to the individual regions (Step S 106 ). 
     Thereafter, Step S 104  through S 016  are repeated until a confirmation operation is received. When the confirmation operation is received (Step S 107 : YES), the image controlling unit  24  stores the detailed settings for the regions and functions (Step S 108 ) and concludes the process for setting up regions. 
       FIG.  4    is a flowchart showing a procedure when operating. When the in-vehicle device is supplied with power, the processing unit  20  of the display controlling device repeatedly executes the procedure shown in  FIG.  4   . 
     First the exterior camera  11  and interior camera  12  acquire respective images (Step S 201 ). 
     The viewpoint position detecting unit  21  detects the viewpoint position (for example, the position of a head) from the image acquired by the interior camera  12  (Step S 202 ). The image controlling unit  24  generates a virtual view image from the viewpoint position and the image of the exterior camera  11  (Step S 203 ). And the image controlling unit  24  executes a virtual view image displaying process for displaying a virtual view image on the display  30  (Step S 204 ). 
     Moreover, the image controlling unit  24  identifies the region wherein the viewpoint position is located (Step S 205 ). 
     If a function is set corresponding to the region wherein the viewpoint position is located (Step S 206 : YES), the image controlling unit  24  calls and executes that function (Step S 207 ), to complete the process. After the process has been completed, processing starts again from Step S 201 . If no function has been set corresponding to the region wherein the viewpoint position is located (Step S 206 : NO), then processing is terminated at that point, and processing begins again with Step S 201 . 
       FIG.  5    is a specific example of a screen output during the set-up procedure. 
     During the set-up procedure, the display  30  displays a seat image G 1 , a head image G 2  (G 2   a  and G 2   b ), a display image G 3 , an Add Region button G 4 , and a Remove Region button G 5 . 
     The seat image G 1  is an image for displaying schematically the position of the vehicle occupant within the vehicle. The head image G 2  is an image for displaying schematically the position of the head of the vehicle occupant. The display image G 3  is an image for indicating schematically the position of the display  30 . The head image G 2  is located based on the detection result by the viewpoint position detecting unit  21 , where the head can be shown in any position with relation to the seat and the display  30 . 
     A boundary indicator G 10  indicates a boundary of a region that is set between the position of the occupant within the vehicle and the display  30 . In  FIG.  5   , region A 1  is set from the seat image G 1  to the boundary indicator G 10 , and region A 2  is set from the boundary indicator G 10  to the display image G 3 . 
     Additionally, a function icon that indicates a display function is displayed for each individual region. The function icon Gil that is displayed in region A 1  indicates that no display function has been set. The function icon G 12  that is displayed in the region A 2  indicates a function for zooming the image displayed on the display. 
     Here the position of the head image G 2  is changed in real time using detection results from the viewpoint position detecting unit  21 . The head image G 2   a  in  FIG.  5    indicates the position of the head in a state wherein the driving operation is carried out in the normal driving position. The head image G 2   b  indicates the position of the head in a state wherein the posture is that of leaning forward with the face near the display  30 . Indicating the position of the head in real time enables the vehicle occupant to identify easily the relationship between the position of the occupant&#39;s own head and the region. 
       FIG.  6    is a specific example of a process for associating a function with a region. As illustrated in  FIG.  6   , when a long-push operation has been received by the function icon G 12  in the region A 2 , the image controlling unit  24  displays the selectable display functions. In  FIG.  6   , the function icon G 12  for the function for zooming the image displayed on the display, the function icon G 13  for the function for adding route information to the display, and the function icon G 14  for displaying music information are displayed. When the function icon G 13  is selected, the function icon G 13  will be displayed thereafter in region A 2 . In this case, the function setting unit  23  assigns “Add route information to the display” corresponding to the region A 2 . 
       FIG.  7    is a specific example of a process for changing the size of a region. As shown in  FIG.  7   , when a boundary indicator G 10  is subjected to a sliding operation, the image controlling unit  24  changes the position of the boundary indicator G 10  in accordance with the amount of sliding. In this case, the region setting unit  22  changes the sizes of the region A 1  and the region A 2  depending on the position after the change. 
       FIG.  8    is a specific example of increasing or decreasing the number of regions. As illustrated in  FIG.  8   , when the Add Region button G 4  is pressed, a new region A 3  is added in a state wherein region A 1  and region A 2  have been set up. 
     As an example, the additional region can be generated through dividing into two the existing region that is nearest to the display  30 . Moreover, the display function of the added region can be identical to that of the region prior to division, or a display function that is not set in the existing regions can be applied as a default value. 
     When the Remove Region button G 5  is pressed in a state wherein region A 1  and region A 2  have been set up, region A 2  will be removed. The result is that only region A 1 , for which no display function has been set, will remain. In other words, all regions wherein display functions have been set can be removed. 
     If the removal operation is received in a state wherein there is a plurality of regions, the region that is to be removed may be set arbitrarily. However, as an example, the regions wherein display functions have been set can be removed with priority when there are both of regions for which display functions have been set and regions for which display functions have not been set. This priority sequence positions the display function as an additional function that is executed in response to a clear operation by the vehicle occupant. If this positioning is used, preferably the region corresponding to the normal driving region will always have no display function set. 
     If the Add Region button G 4  is pressed in a state wherein there is only the region A 1 , for which the display function has not been set, a new region A 2  can be added on the display  30  side. And the new region is associate with the function for zooming the image displayed on the display  30  as the display function in the default state. 
       FIG.  9    is an explanatory diagram of a specific example of a display function. 
     If the head of the vehicle occupant moves so as to enter into the region for which the function for zooming the image displayed on the display has been set, the image controlling unit  24  zooms the obstacle image D 1   a  to display an obstacle image D 1   b.    
     Similarly, if the head of the vehicle occupant moves so as to enter into a region for which the function for adding route information to the display has been set, the image controlling unit  24  will display route information N 1 . 
       FIG.  10    is an explanatory diagram for a zooming function that uses a plurality of regions. In  FIG.  10   , regions A 1  through A 3  are set up sequentially from the position at which the vehicle occupant is present in the vehicle up to the display  30 . The region A 1  has no display function set. Functions to zoom the imaged displayed on the display are set for region A 2  and region A 3 . In other words, region A 2  and region A 3  are divided regions provided in a prescribed region wherein display functions are set, where different scales are set for region A 2  and region A 3 . The zoom displays with different scales can be set up as individual display functions, or different scales can be produced through repetitively applying a single function. For example, with the magnification rate of the display zooming function being a uniform 1.2, when entering into region A 2  from region A 1  the magnification rate will be 1.2, and when entering region A 3  from region A 2  the magnification rate can be applied repetitively with the magnification rate being 1.2×1.2=1.44. 
     In  FIG.  10   , when the head of the vehicle occupant is in region A 1 , an obstacle image D 1   a  that corresponds to a scale of “1” is displayed. When the head has entered region A 2 , an obstacle image D 1   b  with a scale of “1.2” is displayed. When it has entered into region A 3 , an obstacle image D 1   c  with a scale of “1.44” is displayed. In this way, associating different scales to the divided regions makes it possible to carry out zooming in stages depending on the position of the head. 
     As described above, the device of the disclosure comprises: image acquiring means for acquiring a captured image wherein the vicinity of a vehicle has been imaged; region setting means for setting a prescribed region between displaying means, installed within a cabin of the vehicle and a position wherein a vehicle occupant is within the cabin of the vehicle; function setting means for setting a display function associated with a prescribed region; viewpoint position detecting means for detecting a position relating to the viewpoint of the vehicle occupant; and image controlling means for controlling the displaying means so as to display the captured image, wherein: when the position relating to the viewpoint is detected as being in the prescribed region, the controlling means executes the display function on the captured image displayed by the displaying means. This structure and these operations make it possible to check the state far from the vicinity of the vehicle (through zooming the image displayed on the display of the camera image), or to check information required in operation, other than a camera image (such as navigation information) without carrying out any special operation, through merely moving the viewpoint of the vehicle occupant (the eyes, head, or face) into a specific area (which is preferably a region near the display). 
     This makes it possible to achieve excellent ease of operations in relation to the display output of the in-vehicle display device. 
     Moreover, in the device according to the present disclosure, the image controlling means performs processing on the captured image based on the position relating to the viewpoint of the vehicle occupant to generate a virtual view image that is an image of outside of the vehicle that is calculated as being visible from the viewpoint of the vehicle occupant, and if the position relating to the viewpoint is identified as being within the prescribed region, executes the display function on the virtual view image. 
     This makes it possible to provide a virtual view depending on the viewpoint of the vehicle occupant, and to call an additional function when the face of the vehicle occupant nears the displaying means. 
     Moreover, in the device according to the present disclosure, the region setting means set up a plurality of divided regions in the prescribed region with different distances to the displaying means, and the function setting means set different scales for the plurality of divided regions. 
     This structure and these operations make it possible to carry out zooming in stages depending on the position of the head of the vehicle occupant. 
     Note that the present invention is not limited to the embodiment set forth above, but rather includes a variety of modified examples. For example, the embodiments set forth above were for explaining the details so as to explain the present invention so as to be easily understood, and there is no limitation to necessarily providing all of the features that were explained. Moreover, insofar as these features are not removed, features may be substituted or added. 
     For example, if there is another vehicle occupant other than the driver, the present invention can be applied to the other vehicle occupant. Moreover, the present invention can be applied through designating a specific vehicle occupant among a plurality of vehicle occupants. 
     Moreover, the state of the vehicle occupant may be acquired using an arbitrary sensor other than an interior camera, such as, for example, a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), a smart phone that is installed, or the like. Moreover, outputs of a plurality of sensors or a plurality of types of sensors can be combined for use. 
     Moreover, the functions related to each other can be set when setting up functions for a plurality of regions. For example, a process can be performed an air conditioning main menu is started first when the face of the vehicle occupant is brought closer to the displaying means, and the temperature setting is increased when the face is brought closer. 
     Moreover, the embodiment was presented wherein the regions were set on the display screen, but the positional relationships between the display and the viewpoint can be designated through an arbitrary method. For example, distances from the display or spaces within the vehicle classified prescribed standards in advance can be designated. The regions in the present invention should identify whether or not to start a display function based on the viewpoint position. 
     Moreover, the present invention can be applied to a pillar monitor that provides a view looking through the pillar, an electronic mirror that provides a side mirror function, a backing mirror, or the like. 
     Moreover, for example, a plurality of display devices can be installed (each having the functions of the present invention) as a pillar monitor or an electronic mirror. And regions and functions can be set for each individual displayed device. In this case, preferably the view of the vehicle occupant is identified to select the display device that is to execute the functions of the present invention. 
     REFERENCE LIST 
       11 : Exterior Camera 
       12 : Interior Camera 
       20 : Processing Unit 
       21 : Viewpoint Position Detecting Unit 
       22 : Region Setting Unit 
       23 : Function Setting Unit 
       24 : Image Controlling Unit 
       30 : Display