Patent Description:
Mirror replacement systems, and camera systems for supplementing mirror views, are utilized in commercial vehicles to enhance the ability of a vehicle operator to see a surrounding environment. Camera monitoring systems (CMS) utilize one or more cameras to provide an enhanced field of view to a vehicle operator. In some examples, the camera monitoring systems cover a larger field of view than a conventional mirror, or include views that are not fully obtainable via a conventional mirror.

Existing CMS utilize a set of displays that replicate the views provided by one or more conventional mirrors as well as any additional views. The views provided by the CMS are flat two-dimensional views. Due to the nature of camera imaging, the flat two-dimensional views do not shift or adjust based on the position of the vehicle operator's head or the direction of the operator's gaze as a conventional mirror would.

<CIT> that corresponds to the preamble of claims <NUM> and <NUM>, discloses a vision system of a vehicle including a camera having a lens and a two dimensional imaging array sensor. The camera is disposed at a side of a body of a vehicle and has a field of view exterior of the vehicle and at least rearward of the vehicle. A video display is disposed in the vehicle and is viewable by a driver of the vehicle when the driver is normally operating the vehicle. <CIT> discloses a mirror replacement system for a motor vehicle, with a processor unit, an optical sensor unit, and a screen, which is set up to display at least part of image and/or video data generated. The mirror replacement system comprises a viewing direction sensor which is set up to detect the direction in which the vehicle driver is looking at the screen or the position of the vehicle driver's head.

An aspect relates to a mirror replacement system for a commercial vehicle including a camera monitoring system (CMS) including at least one camera, and at least one display configured to display images from the at least one camera to a vehicle operator, a driver monitoring system (DMS) including at least one interior camera, the at least one interior camera having a field of view including at least a portion of the vehicle operator's head, the DMS being configured to identify at least one of a head position and orientation of the vehicle operator, a pose of the vehicle operator, and a gaze direction of the vehicle operator, and a controller configured to: detect a shift in at least one of the vehicle operator's head position, the vehicle operator's pose and the vehicle operator's gaze; determine a rotation of the vehicle operator relative to the at least one display based on the detected shift; and digitally manipulate at least one image from the CMS by simulating at least one parallax effect corresponding to the determined rotation; wherein the digital manipulation includes an adjustment of a relative position of objects within the at least one image relative to an anchor point; and wherein the anchor point is at least one object in the at least one image, is a focal point of the vehicle operator, or is a position midway between at least two identified objects in the at least one image.

In another example of the above described mirror replacement system for a commercial vehicle the at least one display includes a first display disposed at or near a first A-pillar and a second display disposed at or near a second A-pillar, each of the first display and the second display configured to display a corresponding rear view of the vehicle.

In another example of any of the above described mirror replacement systems for a commercial vehicle the corresponding rear view of the vehicle includes at least one of a Class II and a Class IV view.

In another example of any of the above described mirror replacement systems for a commercial vehicle the digital manipulation further includes an adjustment of at least one of the relative skew and the relative size of the objects in each display in response to determining the rotation of the vehicle operator relative to the at least one display.

In another example of any of the above described mirror replacement systems for a commercial vehicle the digital manipulation further includes an adjustment of the relative size of objects in the image and the relative skew of objects in the image in response to the change in the rotation of the vehicle operator relative to the at least one display.

In another example of any of the above described mirror replacement systems for a commercial vehicle the controller further includes an object identification module configured to distinguish objects within an image feed.

In another example of any of the above described mirror replacement systems for a commercial vehicle the at least one interior camera includes at least two interior cameras.

Another aspect relates to a method for operating a camera monitoring system including detecting a shift in at least one of a vehicle operator's head position, the vehicle operator's pose and the vehicle operator's gaze, determining a rotation of the vehicle operator relative to a display based on the detected shift, and digitally manipulating at least one camera monitoring system (CMS) image by simulating at least one parallax effect corresponding to the determined rotation; wherein said digitally manipulating the at least one CMS image comprises adjusting a relative position of objects within the at least one CMS image relative to an anchor point; wherein the anchor point is at least one object in the image, is a focal point of the vehicle operator, or is a position midway between at least two identified objects in the image.

In another example of the above described method for operating a camera monitoring system digitally manipulating the at least one CMS image comprises adjusting at least one of a relative size of objects within the CMS image and a relative skew of the objects within the CMS image.

In another example of any of the above described methods for operating a camera monitoring system digitally manipulating the at least one CMS image comprises adjusting the relative position of the objects within the CMS image and a relative skew of the objects within the CMS image.

In another example of any of the above described methods for operating a camera monitoring system digitally manipulating by adjusting at least one of a relative size of objects within the CMS image and a relative skew of the objects within the CMS image includes adjusting the image relative to the anchor point.

In another example of any of the above described methods for operating a camera monitoring system the anchor point is at least one object in the image.

In another example of any of the above described methods for operating a camera monitoring system the anchor point is a focal point of the vehicle operator.

In another example of any of the above described methods for operating a camera monitoring system the anchor point is position midway between at least two identified objects.

In another example of any of the above described methods for operating a camera monitoring system detecting the rotation of the vehicle operator relative to the display includes monitoring each of the vehicle operator's head position, pose, and gaze position using a driver monitoring system.

In another example of any of the above described methods for operating a camera monitoring system the CMS image is a 2D image.

In another example of any of the above described methods for operating a camera monitoring system the CMS image is a 3D image.

In another example of any of the above described methods for operating a camera monitoring system the method is continuously iterated and the at least one CMS image includes a video feed.

In another example of any of the above described methods for operating a camera monitoring system the video feed includes a first camera corresponding to a first mirror position and a second camera corresponding to a second mirror position.

In another example of any of the above described methods for operating a camera monitoring system simulating at least one parallax effect comprises magnifying an actual parallax effect and simulating the magnified parallax effect.

A schematic view of a commercial vehicle <NUM> is illustrated in <FIG> and <FIG>. The vehicle <NUM> includes a vehicle cab or tractor <NUM> for pulling a trailer <NUM>. Although a commercial truck is contemplated in this disclosure, the invention may also be applied to other types of vehicles. The vehicle <NUM> incorporates a camera monitoring system (CMS) <NUM> (<FIG>) that has driver and passenger side camera arms 16a, 16b mounted to the outside of the vehicle cab <NUM>. If desired, the camera arms 16a, 16b may include conventional mirrors integrated with them as well, although the CMS <NUM> can be used to entirely replace mirrors. In additional examples, each side can include multiple camera arms, each arm housing one or more cameras and/or mirrors.

Each of the camera arms 16a, 16b includes a base that is secured to, for example, the cab <NUM>. A pivoting arm is supported by the base and may articulate relative thereto. At least one rearward facing camera 20a, 20b is arranged respectively within camera arms. The exterior cameras 20a, 20b respectively provide an exterior field of view FOVEX1, FOVEX2 that each include at least one of the Class II and Class IV views (<FIG>), which are legal prescribed views in the commercial trucking industry. Multiple cameras also may be used in each camera arm 16a, 16b to provide these views, if desired. Each arm 16a, 16b may also provide a housing that encloses electronics that are configured to provide various features of the CMS <NUM>.

First and second video displays 18a, 18b are arranged on each of the driver and passenger sides within the vehicle cab <NUM> on or near the A-pillars 19a, 19b to display Class II and Class IV views on the respective sides of the vehicle <NUM>. The views provide rear facing side views along the vehicle <NUM> that are captured by the exterior cameras 20a, 20b.

If video of Class V and Class VI views are also desired, a camera housing 16c and camera 20c may be arranged at or near the front of the vehicle <NUM> to provide those views (<FIG>). A third display 18c arranged within the cab <NUM> near the top center of the windshield can be used to display the Class V and Class VI views, which are toward the front of the vehicle <NUM>, to the driver.

If class VIII views are desired, camera housings can be disposed at the sides and rear of the vehicle <NUM> to provide fields of view including some or all of the class VIII zones around the vehicle <NUM>. In such examples, the third display 18c can include one or more frames displaying the class VIII views. Alternatively, additional displays can be added near the first, second and third displays 18a, 18b, 18c and provide a display dedicated to providing a class VIII view. In yet another alternative the additional displays can be included in alternative locations throughout the vehicle <NUM> cabin allowing the user a good view.

The displays 18a, 18b, 18c face a driver region <NUM> within the cabin <NUM> where an operator is seated on a driver seat <NUM>. It may be desirable to incorporate a driver monitoring system (DMS) <NUM> into the CMS <NUM> to evaluate driver activity in the driver region <NUM> and to identify driver parameters that may be indicative of distracted driving or of any other driver characteristic being monitored. One example DMS <NUM> is illustrated in <FIG>. First and second interior cameras 28a, 28b are arranged in the vehicle cabin <NUM> near the first and second displays 18a, 18b and are configured to capture driver images within the vehicle cabin <NUM>. The first and second interior cameras 28a, 28b respectively provide first and second interior fields of view FOVIN1, FOVIN2 that at least partially overlap one another in the driver region <NUM>. The first display 18a is positioned such that it is in the second interior field of view FOVIN2, and the second display 18b is positioned such that it is in the first interior field of view FOVIN1. Driver activity is determined in at least some examples by monitoring the gaze and/or pose of the driver by using operating the vehicle while using the first and second interior cameras 28a, 28b.

Due to the nature of two-dimensional video imaging, video feed(s) generated by a static camera do not replicate a parallax effect that occurs on physical mirrors. The parallax effect is the effect where the position, size or orientation of an object appears to be different (relative to surrounding objects) when viewed from different positions or angles. By way of example changing an angle that you look at an object can make the object look closer to, or farther away from, another object within your field of view. The shifting relative positions, sizes, and orientations of objects due to the parallax effect aids humans in depth perception and can provide valuable information. In the case of a vehicle operator, the added depth perception of the parallax effect can assist a driver in planning maneuvers to avoid an object and/or prioritizing which object to attempt to avoid when multiple objects are presented. Conventional mirrors, and other reflective surfaces, maintain the parallax effect when used to view objects. Movements that cause the parallax affect to make relative positions of objects to change are referred to as rotations.

The CMS <NUM> included in the vehicle <NUM> is interconnected with the DMS <NUM> as described above. A CMS controller within the CMS <NUM> receives head position, gaze orientation, and/or pose information from the DMS <NUM> and digitally manipulates the objects in the image corresponding to adjustments in the driver's head position, gaze direction or pose in order to simulate the parallax effect of an actual mirror view. This simulation can be either through adjusting a true 3D image to account for parallax shifts or a partial simulation of parallax shifts using 2D image manipulation, referred to as quasi-3D. The two types of imaging are referred to together as "at least quasi-3D" herein.

With continued reference to <FIG>, <FIG> and <FIG>, <FIG> illustrates an exemplary display screen including a view at least two identified objects, a tree <NUM> and a sign <NUM> displayed in a flat 2D image <NUM>. The image <NUM> provides a view of a real scene <NUM>, illustrated in <FIG>. The actual physical location of the objects <NUM>, <NUM> within the scene <NUM> does not change when a vehicle operators position causes a rotation. Alternative examples can include any number of additional objects within the image and operate similarly. The objects <NUM>, <NUM> can be identified via an object identification system within the CMS <NUM> including a neural network analysis of the received image feeds, a rule-based analysis using predetermined image analysis rules, manual identification by the vehicle operator, and the like. The distance between each object <NUM>, <NUM> and the vehicle <NUM>, and each object <NUM>, <NUM> and each other object <NUM>, <NUM> is determined by the CMS <NUM>, and a "Center" image <NUM> is generated at the corresponding display 18a or 18b. The center image <NUM> is the image as it would appear to an operator looking directly at the display and does not include any parallax manipulation. This image is reflective of the left scene <NUM> in <FIG>. In the center image <NUM>, the objects <NUM>, <NUM> are separated by a distance <NUM>. Alternatively, the object identification can occur at a remote controller, a general vehicle controller, or any other processing system configured to identify objects within the image.

When the vehicle operator shifts their gaze, pose, and/or head position to the left, this shift causes a rotation of the of the operator's view relative to the display screen. The DMS <NUM> monitors the shift and the relative motion of the driver's viewpoint relative to the display screen, resulting from the rotation, is provided to a parallax control within the CMS <NUM>. The parallax control determines a relative position change between the two or more items corresponding to the parallax shift resulting from the change of perspective. In the example of <FIG>, the driver's rotation causes the two objects <NUM>, <NUM> to appear closer together or farther apart, depending on the rotation. Rotation about other axis can cause corresponding position shifts The CMS <NUM> digitally manipulates the display to move the identified objects <NUM>, <NUM> to match the relative position change. In the simplified examples of <FIG>, the parallax control determines that the viewpoint rotating left (view <NUM>) will shift the sign <NUM> closer to the tree within the plane of the image <NUM>, resulting in an overlap with the sign <NUM> being in front of the tree. Once this is determined the CMS <NUM> responds to the identification of a driver's position rotating left, by manipulating the image <NUM> to reduce the distance <NUM>. This process is performed in real time, allowing the operator to continuously rely on the presented images to provide accurate information.

Conversely, when the operator rotates their viewpoint to the right, the DMS <NUM> identifies the rotation and informs the CMS <NUM>. The CMS <NUM> then identifies that the sign <NUM> and the tree <NUM> should appear farther apart, and digitally manipulates the image <NUM> to move the sign <NUM> farther away from the tree <NUM>.

This operation simulates a parallax effect from shifting perspectives and provides additional depth perception, without generating a truly 3D image. In the described example, one of the objects (the tree <NUM>) is used as an anchor point, and all other objects (the sign <NUM>) are moved relative to the anchor point. The anchor point is a position within the image that is maintained "constant". In alternative examples, the anchor point can be identified in the image, and objects can be moved relative to that anchor point. In yet another example, the shift between two objects <NUM>, <NUM> can be identified, and both objects <NUM>, <NUM> can be shifted away from each other along the identified dimension. The anchor point in the image of this example is the simulated "focal point", or point in the image that the operator is looking at.

In addition to shifting the identified objects, the at least quasi-3D parallax effect is achieved in some examples by altering the relative sizes of the objects based on the head position and gaze direction of the driver. With continued reference to <FIG>, <FIG>, <FIG> and <FIG> schematically illustrates the parallax effect of altering the relative sizes of objects <NUM>, <NUM> in the image <NUM>. For ease of explanation, the same scene is illustrated in the image <NUM> of <FIG> as was used in the scene of image <NUM> in <FIG>. As with the relative positions example, the DMS <NUM> rotation of the vehicle operator's viewpoint about the display and alters the relative sizes of the objects <NUM>, <NUM> to correspond to the detected rotation.

With reference to both <FIG>, a CMS <NUM> can further create or enhance the quasi-3D parallax effect by skewing objects <NUM>, <NUM>, with the relative position of the objects <NUM>, <NUM> determining the magnitude of skew. In addition, the CMS <NUM> can combine the identified quasi-3D parallax effects described herein to further improve the simulated 3D nature. Further, identical process can be applied to a 3D display, allowing for a similar magnitude of parallax simulation and improving the realism and depth of the 3D display.

In one exemplary implementation, the simulated parallax effect is exaggerated within the display in order to assist a driver in more accurately determining the relative positions of objects within the actual scene. The exaggeration can be achieved in one example, by determining that the operator's rotation has exceeded a predefined magnitude (e.g., greater than <NUM> degrees of rotation) and begin exaggerating the simulated parallax effect by applying a multiplier to the additional rotation beyond <NUM> degrees. In another example, a smaller magnification is applied constantly to the determined rotation.

With continued reference to <FIG>, <FIG> illustrates a method for implementing the at least quasi-3D parallax shifting within a vehicle <NUM>. Initially the CMS <NUM> identifies each object within the image in an "Identify Objects within Image" step <NUM>. The objects can be identified as described above, with regards to <FIG>. Further, in some examples, the CMS <NUM> can define a "bounding box" (illustrated as bounding boxes <NUM> and <NUM>) around the identified object and the bounding box defines what is and is not shifted. Alternatively, the object can be re-identified during each operation, depending on the available computational power of the CMS <NUM>.

Once each object within the image is identified, the CMS <NUM> determines the relative position of the objects in a "Determine Relative Object Position" step <NUM>. In some examples, a position of each object is determined relative to a single fixed point. The single fixed point can be the identified anchor point within the image, the vehicle <NUM>, or a static object within the image. In some examples, the single fixed point is the anchor point in the image, while in other examples the single fixed point is used only for determining the relative positioning of the objects and the anchor point is dependent on the image.

Once the relative positions of each identified object are determined, the CMS <NUM> uses the relative positions of the objects, and the position of the camera(s) capturing the image to determines how the objects will shift, resize, or skew relative to each other in response to the driver rotating their point of view relative to the display in an "Identify Parallax Shift Between Objects" step <NUM>. The parallax shift is determined via comparing sequential images based on a known camera movement, utilizing expected or learned parallax shifts, based on known relative positioning relationships, based on expected positioning, or based on a combination of the two.

Concurrently with the method <NUM> of <FIG>, the CMS <NUM> performs a method <NUM> that determines how to modify the two or three dimensional image to simulate, or partially simulate, the parallax shift and create an at least partially 3D effect. Initially the CMS <NUM> uses the DMS <NUM> to monitor the position and orientation of the drivers head, the pose of the driver and/or the direction of the driver's gaze in a "Monitor Driver Position" step <NUM>.

The DMS <NUM> detects a rotation of the vehicle operator about the display in a "Detect Rotation" step <NUM>, and provides a signal to the CMS <NUM> identifying the change in the monitored characteristic. The identified change includes at least a direction and magnitude of the change. In some examples the head position, operator pose, and the gaze direction are simultaneously monitored, and the detection is a detection of the combined change.

Once the identified detection is provided to the CMS <NUM>, the CMS <NUM> responds by shifting the relative position, size, and skew of objects within the image in an "Adjust Parallax Effect" step <NUM>. This process is constantly reiterated during vehicle operation, with the constant adjustments to the parallax effect providing at least quasi-3D by adapting the image to include the shifting parallax effects.

It is appreciated that while example views and figures are included, the examples are simplified and exaggerated for explanatory purposes. In a practical implementation the magnitude of object shifting, resizing, or skewing could be much smaller dependent on which type(s) of quasi-3D parallax effects are being implemented in the given CMS.

Claim 1:
A mirror replacement system for a commercial vehicle comprising:
a camera monitoring system, CMS, (<NUM>) including at least one camera (20a, 20b), and at least one display (18a, 18b, 18c) configured to display images from the at least one camera (20a, 20b) to a vehicle operator;
a driver monitoring system, DMS, (<NUM>) including at least one interior camera (28a, 28b), the at least one interior camera (28a, 28b) having a field of view including at least a portion of the vehicle operator's head, the DMS (<NUM>) being configured to identify at least one of a head position and orientation of the vehicle operator, a pose of the vehicle operator, and a gaze direction of the vehicle operator; and
a controller configured to:
detect a shift in at least one of the vehicle operator's head position, the vehicle operator's pose and the vehicle operator's gaze;
determine a rotation of the vehicle operator relative to the at least one display (18a, 18b, 18c) based on the detected shift; characterized in that the controller is configured to
digitally manipulate at least one image from the CMS (<NUM>) by simulating at least one parallax effect corresponding to the determined rotation;
wherein the digital manipulation includes an adjustment of a relative position of objects (<NUM>, <NUM>) within the at least one image relative to an anchor point; and
wherein the anchor point is at least one object (<NUM>, <NUM>) in the at least one image, is a focal point of the vehicle operator, or is a position midway between at least two identified objects (<NUM>, <NUM>) in the at least one image.