Patent ID: 12195007

DETAILED DESCRIPTION

The following detailed description is merely exemplary in nature and is not intended to limit the application and uses. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding technical field, background, brief summary or the following detailed description. As used herein, the term module refers to any hardware, software, firmware, electronic control component, processing logic, and/or processor device, individually or in any combination, including without limitation: application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory that executes one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.

Embodiments of the present disclosure may be described herein in terms of functional and/or logical block components and various processing steps. It should be appreciated that such block components may be realized by any number of hardware, software, and/or firmware components configured to perform the specified functions. For example, an embodiment of the present disclosure may employ various integrated circuit components, e.g., memory elements, digital signal processing elements, logic elements, look-up tables, or the like, which may carry out a variety of functions under the control of one or more microprocessors or other control devices. In addition, those skilled in the art will appreciate that embodiments of the present disclosure may be practiced in conjunction with any number of systems, and that the systems described herein is merely exemplary embodiments of the present disclosure.

For the sake of brevity, conventional techniques related to signal processing, data transmission, signaling, control, and other functional aspects of the systems (and the individual operating components of the systems) may not be described in detail herein. Furthermore, the connecting lines shown in the various figures contained herein are intended to represent example functional relationships and/or physical couplings between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may be present in an embodiment of the present disclosure.

FIG.1illustrates a vehicle10, according to an exemplary embodiment. In certain embodiments, the vehicle10comprises an automobile. In various embodiments, the vehicle10may be any one of a number of different types of automobiles, such as, for example, a sedan, a wagon, a truck, or a sport utility vehicle (SUV), and may be two-wheel drive (2WD) (i.e., rear-wheel drive or front-wheel drive), four-wheel drive (4WD) or all-wheel drive (AWD), and/or various other types of vehicles in certain embodiments. In addition, in various embodiments, it will also be appreciated that the vehicle10may comprise any number of other types of mobile platforms.

As depicted inFIG.1, the exemplary vehicle10generally includes a chassis12, a body14, front wheels16, and rear wheels18. The body14is arranged on the chassis12and substantially encloses components of the vehicle10. The body14and the chassis12may jointly form a frame. The wheels16-18are each rotationally coupled to the chassis12near a respective corner of the body14.

The vehicle10further includes a propulsion system20, a transmission system22, a steering system24, a sensor system28, at least one data storage device32, at least one controller34, and a display system35, and a driver monitoring system48. The propulsion system20includes an engine26, such as a gasoline or diesel fueled combustion engine, or an electric engine. The transmission system22is configured to transmit power from the propulsion system20to the wheels16-18according to selectable speed ratios. According to various embodiments, the transmission system22may include a step-ratio automatic transmission, a continuously-variable transmission, or other appropriate transmission.

The steering system24influences a position of the wheels16-18. While depicted as including a steering wheel23for illustrative purposes, in some embodiments contemplated within the scope of the present disclosure, the steering system24may not include the steering wheel23. Unlike conventional steering systems, the steering wheel23is not mechanically coupled to the front wheels16. Instead, the steering system24includes a steer by wire system25that functionally couples the steering wheel23to systems (e.g., actuators) configured to turn the front wheels16with electrical cables configured to transmit electronic signals. The steer by wire system25may be adjusted to modify a steering ratio between the steering wheel23and the turning radius of the front wheels16.

The sensor system28includes one or more sensing devices40a-40nthat sense observable conditions of the exterior and/or interior environment of the vehicle and/or of the vehicle itself. The sensing devices40a-40ncan include, but are not limited to, radars, lidars, global positioning systems, optical cameras, thermal cameras, ultrasonic sensors, inertial measurement units, pressure sensors, position sensors, speed sensors, and/or other sensors.

The data storage device32stores data for use in controlling the vehicle10and/or systems and components thereof. As can be appreciated, the data storage device32may be part of the controller34, separate from the controller34, or part of the controller34and part of a separate system. The data storage device32can be any suitable type of storage apparatus, including various different types of direct access storage and/or other memory devices. In one exemplary embodiment, the data storage device32comprises a program product from which a computer readable memory device can receive a program that executes one or more embodiments of one or more processes of the present disclosure, such as the steps of the process discussed further below in connection withFIG.3. In another exemplary embodiment, the program product may be directly stored in and/or otherwise accessed by the memory device and/or one or more other disks and/or other memory devices.

The controller34includes at least one processor44, a communication bus45, and a computer readable storage device or media46. The processor44performs the computation and control functions of the controller34. The processor44can be any custom made or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among several processors associated with the controller34, a semiconductor-based microprocessor (in the form of a microchip or chip set), a macroprocessor, any combination thereof, or generally any device for executing instructions. The computer readable storage device or media46may include volatile and nonvolatile storage in read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM), for example. KAM is a persistent or non-volatile memory that may be used to store various operating variables while the processor44is powered down. The computer-readable storage device or media46may be implemented using any of a number of known memory devices such as PROMs (programmable read-only memory), EPROMs (electrically PROM), EEPROMs (electrically erasable PROM), flash memory, or any other electric, magnetic, optical, or combination memory devices capable of storing data, some of which represent executable instructions, used by the controller34in controlling the vehicle10. The bus45serves to transmit programs, data, status and other information or signals between the various components of the vehicle10. The bus45can be any suitable physical or logical means of connecting computer systems and components. This includes, but is not limited to, direct hard-wired connections, fiber optics, infrared, and wireless bus technologies.

The instructions may include one or more separate programs, each of which comprises an ordered listing of executable instructions for implementing logical functions. The instructions, when executed by the processor44, receive and process signals from the sensor system28, perform logic, calculations, methods and/or algorithms, and generate data based on the logic, calculations, methods, and/or algorithms. Although only one controller34is shown inFIG.1, embodiments of the vehicle10can include any number of controllers34that communicate over any suitable communication medium or a combination of communication mediums and that cooperate to process the sensor signals, perform logic, calculations, methods, and/or algorithms, and generate data.

In various embodiments, the instructions, when executed by the processor44, generally process received data in order to determine a cause of a blockage of the camera50, adjust an escalation algorithm of the DMS48based on the determined cause of the blockage, and, in some embodiments, mitigate the cause of the blockage as further described with respect to theFIGS.2-5. As such, the vehicle10includes a driver monitoring adjustment system100configured to interact with the DMS48and improve functionality thereof and/or communication between the DMS48and the driver. In various embodiments, the driver monitoring adjustment system100may include the controller34, the DMS48, and/or an entirety of the vehicle10.

As can be appreciated, that the controller34may otherwise differ from the embodiment depicted inFIG.1. For example, the controller34may be coupled to or may otherwise utilize one or more remote computer systems and/or other control systems, for example as part of one or more of the above-identified vehicle devices and systems. It will be appreciated that while this exemplary embodiment is described in the context of a fully functioning computer system, those skilled in the art will recognize that the mechanisms of the present disclosure are capable of being distributed as a program product with one or more types of non-transitory computer-readable signal bearing media used to store the program and the instructions thereof and carry out the distribution thereof, such as a non-transitory computer readable medium bearing the program and containing computer instructions stored therein for causing a computer processor (such as the processor44) to perform and execute the program. Such a program product may take a variety of forms, and the present disclosure applies equally regardless of the particular type of computer-readable signal bearing media used to carry out the distribution. Examples of signal bearing media include recordable media such as floppy disks, hard drives, memory cards and optical disks, and transmission media such as digital and analog communication links. It will be appreciated that cloud-based storage and/or other techniques may also be utilized in certain embodiments. It will similarly be appreciated that the computer system of the controller34may also otherwise differ from the embodiment depicted inFIG.1, for example in that the computer system of the controller34may be coupled to or may otherwise utilize one or more remote computer systems and/or other control systems. In various embodiments, the controller34may be a component of the DMS48.

The DMS48may include various controllers, memory devices, data storage devices, sensors, etc. that in combination are configured to monitor a driver of the vehicle10and continuously or periodically generate an assessment of the attentiveness of the driver during operation of the vehicle10. In this example, the DMS48includes a camera50configured to observe the driver, for example, the driver's face. Such observations are used by the DMS48, at least in part, for generating the assessment of the driver.

The vehicle10further includes a sunroof60, that is, a panel on the roof of the vehicle10that allows light to enter the vehicle10. The sunroof60includes a “smart” glass that incorporates a suspended particle device technology to allow for adjustment of opacity between translucent and opaque. Specifically, a user is able to adjust a tint of the sunroof with controls in the vehicle10. Interaction with the controls selectively apply an electrical voltage to a thin film on the glass causing a corresponding change in transmissivity thereof via alignment and misalignment of nanoparticles of the thin film (e.g., polyiodide nanoparticles). The transmissivity of the sunroof may be adjusted over an entirety thereof and/or limited to individual portions or sections thereof. The vehicle10may further include an opaque, retractable panel configured to selectively slide over and cover the sunroof60.

With reference toFIG.2and with continued reference toFIG.1, a dataflow diagram illustrates elements of the driver monitoring adjustment system100ofFIG.1in accordance with various embodiments. As can be appreciated, various embodiments of the system100according to the present disclosure may include any number of modules embedded within the controller34which may be combined and/or further partitioned to similarly implement systems and methods described herein. Furthermore, inputs to the system100may be received from other control modules (not shown) associated with the vehicle10, and/or determined/modeled by other sub-modules (not shown) within the controller34. Furthermore, the inputs might also be subjected to preprocessing, such as sub-sampling, noise-reduction, normalization, feature-extraction, missing data reduction, and the like. In various embodiments, the system100includes an analysis module110, a mitigation module120, and an escalation module130.

In various embodiments, the analysis module110receives as input sensor data140generated by the sensor system28, temporal and location data142stored in the data storage device32, and/or DMS data144generated by the DMS48. The sensor data140includes various data indicating observable conditions of the exterior and/or interior environment of the vehicle10and/or of the vehicle10itself. In various embodiments, the sensor data140includes light intensity measurements generated by a brightness sensor, pressure measurements generated by a touch sensor, and/or a location and/or a trajectory of the vehicle10generated by a global positioning system (GPS) device. The temporal and location data142includes various data indicating a time of year (e.g., date) and a time of day. The DMS data144includes various data indicating that the DMS48is unable to determine an attention state of the driver due to the camera50being unable to monitor the driver of the vehicle10(referred to herein as an unknown attention state), and, optionally, attention states of the driver over a period of time as determined by the DMS48.

The analysis module110performs an analysis of the received data to determine a likely cause of the camera50being unable to monitor the driver of the vehicle10. In various embodiments, the analysis module10includes a driver score submodule112, a sun glare submodule114, a hand position submodule116, and a steering wheel position submodule118.

In various embodiments, the driver score submodule112may analyze the sensor data140, the temporal and location data142, and/or the DMS data144to determine a driver's score to be attributed to the driver of the vehicle10. This driver's score may be used at a given time in determining whether to begin or proceed with escalation of the DMS48.FIGS.3and4illustrate an exemplary process for determining the driver's score. In this example (FIG.3), the DMS data144provides a sequence of determined attention states (i.e.,310-334) that include “On Road” indicating that the driver is attentive, “Off Road” indicating that the driver is inattentive, and “Unknown” indicating that the DMS48is unable to determine the attentive state of the driver. In this example, the driver's score is determined using equation (1) below.

DS(t)=XY-Z×100(1)

Wherein Ds is the driver's score, “X” is the total number of On Road attention states in the sequence at time (t), “Y” is the total number of attention states in the sequence at time (t), and “Z” is the total number of Unknown attention states in the sequence at time (t).FIG.5presents a graph that shows the determined driver's score at various times (t) based on the previously recorded attention states in the sequence. The graph includes the driver's scores on the y-axis (labeled as410) and the times (t) on the x-axis (labeled as412). The sequence includes On Road attention states at310,320,322,330, and334, Off Road attention states at316,320,324,326, and328, and Unknown attention states at312,314,318, and332for a total of thirteen attention states in the sequence. Each of the calculated driver's scores Ds are determined based on four of the attention states.

At a first data point on the graph, the first four attention states310-316are used in equation 1 to result in a Ds=50. The second data point on the graph inputs the second through fifth attention states312-318into equation 1 to result in a Ds=0. This process is continued to calculate the remaining data points.

In various embodiments, the sun glare submodule114may analyze the sensor data140, the temporal and location data142, and/or the DMS data144to determine a probability of the cause of the camera50being unable to monitor the driver being due to sun glare. In various embodiments, the sun glare submodule114may analyze the location of the vehicle10, the date, the time of day, the trajectory of the vehicle10, sensed ambient light levels (e.g., intensity), and/or weather information to determine the probability that sun glare is interfering with the camera50. For example, the sun glare submodule114may determine an absolute position of the sun based on the date, time, and location of the vehicle10. The position of the sun may then be determined in relation to the vehicle10based on, for example, the trajectory of the vehicle10. Various factors such as the sensed ambient light levels, the weather information, and the transmissibility of the sunroof60may be used to determine the predicted light level at the location of the camera50which may then be used to determine a probability that the cause of the camera50being unable to monitor the driver is due to sun glare. During this determination, the relative azimuth and elevation angles of the sun in combination with the structure of the vehicle10may be considered to determine a normalized light level at a specific position (e.g., at the camera50).

In various embodiments, the hand position submodule116may analyze the sensor data140, the temporal and location data142, and/or the DMS data144to determine positions of the driver's hands on the steering wheel and a probability of the cause of the camera50being unable to monitor the driver being due to the driver's hand covering the camera50. In various embodiments, the hand position submodule116may analyze the steering wheel angle, and states of touch zones of the steering wheel to determine the probability that the driver's hands are blocking the camera50.

In various embodiments, the steering wheel position submodule118may analyze the sensor data140, the temporal and location data142, and/or the DMS data144to determine the steering wheel angle and, in combination with the structure of the steering wheel and the position of the camera50, determine a probability of the steering wheel blocking the camera50.

In various embodiments, the mitigation module120receives as input analysis data146generated by the analysis module110. The analysis data146includes various data indicating a determination by the analysis module110as to the likely cause of the camera50being unable to monitor the driver. In various embodiments, the analysis data146may include probabilities corresponding to various possible causes of the camera50being unable to monitor the driver such as, but not limited to, probabilities of the cause being due to sun glare, hand position of the driver on the steering wheel, and/or the position of the steering wheel. In various embodiments, the analysis data146may include the driver score.

The mitigation module120performs modification, if possible, to settings of one or more systems of the vehicle10to mitigate the cause of the camera50being unable to monitor the driver. In various embodiments, the mitigation module120includes a sunroof submodule122and a steering wheel submodule124.

If the vehicle10includes a smart sunroof (e.g., sunroof60) capable of adjusting a tint thereof (e.g., shading, transmissivity, etc.) and/or controlling operation of a retractable panel configured to selectively slide over and cover the sunroof60, and the probability of the cause of the camera50being unable to monitor the driver being due to sun glare exceeds a sun glare threshold, the sunroof submodule122may generate sunroof control data148that includes instructions configured to cause the sunroof60to adjust the tint thereof, either for an entirety of the sunroof60or at least a portion thereof. For example, the sunroof control data148may initiate an electronic signal to be sent to the sunroof60that causes nanoparticles therein to misalign and thereby block incoming sunlight. In this manner, the system100may be able to mitigate the sun glare and thereby allow the DMS48to monitor the driver. In addition, or as an alternative, the sunroof control data148may include instructions configured to cause the retractable panel to entirely or partially cover the sunroof60.

If the vehicle10includes a steer by wire system (e.g., the steer by wire system25), and the probability of the cause of the camera50being unable to monitor the driver being due to the steering wheel blocking the camera50exceeds a steering wheel angle threshold, the steering wheel submodule124may generate steering wheel control data150that includes instructions configured to cause an adjustment to the steering wheel steering ratio to avoid blocking the camera50and thereby allow the DMS48to monitor the driver. In some embodiments, the steering wheel submodule124may analyze a planned route for the vehicle10or a road on which the vehicle10is traveling and continuously adjust the steering wheel steering ratio to prevent or reduce the likelihood of the steering wheel blocking the camera50in upcoming turns.

In various embodiments, the escalation module130receives as input mitigation data152generated by the mitigation module120. The mitigation data152includes various data indicating whether the mitigation module120was able to successfully perform actions to mitigate for the cause of the camera50being unable to monitor the driver.

Upon a determination that the mitigation module120was unable to perform mitigating actions, or the mitigating actions performed failed to allow the camera50to monitor the driver, the escalation module130may generate escalation data154that includes instructions for the DMS48to adjust an escalation algorithm of the DMS48and/or for the DMS48to generate one or more alerts for the driver based on the determined cause of the camera50being unable to monitor the driver. Such escalation algorithm may be configured to perform actions based on activity of the driver, such as generating alerts that indicate instructions or notifications for the driver and/or modifying operation of the vehicle10. In various embodiments, the alert(s) may be rendered on the display system35.

With reference now toFIG.3and with continued reference toFIGS.1-2, a flowchart provides a method200for improving the DMS48as performed by the system100, in accordance with exemplary embodiments. As can be appreciated in light of the disclosure, the order of operation within the method200is not limited to the sequential execution as illustrated inFIG.3, but may be performed in one or more varying orders as applicable and in accordance with the present disclosure. In various embodiments, the method200can be scheduled to run based on one or more predetermined events, and/or can run continuously during operation of the vehicle10.

In one example, the method200may start at210. The method200includes determining, at212, a likely cause of the camera50being unable to monitor the driver of the vehicle10. In various embodiments, the determination may include producing probabilities of various likely causes such as, but not limited to, sun glare, blocking of the camera50by the steering wheel, and blocking of the camera50by a hand of the driver. If the vehicle10includes a smart sunroof, a determined probability of the cause being due to sun glare is compared, at214, to a sun glare threshold. If the sun glare probability exceeds the sun glare threshold, the method200includes adjusting, at216, a transmittance of the sunroof.

If the vehicle10does not include a smart sunroof, or the sun glare probability is below the sun glare threshold, the method200includes comparing a determined probability of the cause being due to the position of the steering wheel is compared, at218, to a steering wheel angle threshold. If the steering wheel probability exceeds the steering wheel angle threshold at218, and the vehicle10includes a steer by wire system as determined at220, the method200includes adjusting, at224, a steering wheel ratio such that the steering wheel does not block the camera50.

If the steering wheel probability is below the steering wheel angle threshold at218, the method200includes adjusting, at222, parameters of the escalation algorithm of the DMS48. If the vehicle10does not include a steer by wire system as determined at220, the method200includes adjusting, at226, pre-escalation parameters of the DMS48.

At228, the method200includes determining whether, based on the escalation algorithm, escalation should begin or proceed. If a determination is made that escalation should not begin or proceed, the method200may return to the start at210. In various embodiments, the determination as to whether the escalation should begin or proceed may include a consideration of the driver's score (e.g., as determined by the driver score submodule112). For example, a high driver's score (i.e., a typically attentive driver) may be consideration for delaying escalation, whereas a poor driver's score (i.e., a typically inattentive driver) may be consideration for accelerating the escalation process.

If a determination is made that escalation should begin or proceed, the method200includes comparing, at230, a probability of the cause of the camera50being unable to monitor the driver being due to a position of a hand of the driver to a hand position threshold. If the hand position probability exceeds the hand position threshold, the method200includes generating, at232, a notification for the driver to reposition the hand such that the camera50is not blocked thereby.

If the hand position probability is below the hand position threshold, the method200includes comparing, at234, probabilities that the cause of the camera50being unable to monitor the driver being due to the position of the steering wheel or sun glare. If the steering wheel position probability exceeds the steering wheel angle threshold or if the sun glare probability exceeds the sun glare threshold, the method200includes generating, at236, a notification for the driver that the camera50is unable to monitor the driver. If the steering wheel position probability is below the steering wheel angle threshold and the sun glare probability is below the sun glare threshold, the method200includes generating, at236, a notification for the driver to pay attention to the road. Once a mitigation action has been performed (i.e., at216or224) and/or a notification generated (i.e., at232,236, or238), the method200may end at240.

While at least one exemplary embodiment has been presented in the foregoing detailed description, it should be appreciated that a vast number of variations exist. It should also be appreciated that the exemplary embodiment or exemplary embodiments are only examples, and are not intended to limit the scope, applicability, or configuration of the disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient road map for implementing the exemplary embodiment or exemplary embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the scope of the disclosure as set forth in the appended claims and the legal equivalents thereof.