Patent ID: 12211294

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and/or systems described herein. Accordingly, various changes, modifications, and equivalents of the methods, apparatuses, and/or systems described herein will be suggested to those of ordinary skill in the art. The progression of processing operations described is an example; however, the sequence of and/or operations is not limited to that set forth herein and may be changed as is known in the art, with the exception of operations necessarily occurring in a particular order. In addition, respective descriptions of well-known functions and constructions may be omitted for increased clarity and conciseness.

Additionally, exemplary embodiments will now be described more fully hereinafter with reference to the accompanying drawings. The exemplary embodiments may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. These embodiments are provided so that this disclosure will be thorough and complete and will fully convey the exemplary embodiments to those of ordinary skill in the art. Like numerals denote like elements throughout.

It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. As used herein, the term “and/or,” includes any and all combinations of one or more of the associated listed items.

It will be understood that when an element is referred to as being “connected,” or “coupled,” to another element, it can be directly connected or coupled to the other element or intervening elements may be present. In contrast, when an element is referred to as being “directly connected,” or “directly coupled,” to another element, there are no intervening elements present.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms “a,” “an,” and “the,” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

Reference will now be made in detail to the exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout.

The expression, “at least one of a, b, and c,” should be understood as including only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.

FIG.1illustrates the configuration of a vehicle according to an exemplary embodiment.FIG.2is a view for illustrating the operation of the second camera (Schlieren camera) according to an exemplary embodiment.FIG.3illustrates an example of the configuration of the second camera according to an exemplary embodiment.FIG.4illustrates the arrangement of the first camera and the second camera according to an exemplary embodiment.FIG.5illustrates an example of an image taken by the second camera according to an exemplary embodiment.

Referring toFIGS.1,2,3,4and5, the vehicle1includes a driver monitoring apparatus100for detecting the driver's status, a cluster150displaying operation information of the vehicle1, a multimedia device160playing music, images, etc., a power steering170that assists the steering of the driver, and a seat driving device180that moves the position of the seat181.

The driver monitoring apparatus100includes a first camera120for photographing a general image, a second camera130for photographing an image in a Schlieren method, and an input switch140for activation and deactivation of driver detection and a controller110for controlling the operation of the driver monitoring apparatus100.

The first camera120may be installed in front of the driver's seat as shown inFIG.4, and may have a field of view from the front of the driver's seat to the rear.

When the driver is seated in the driver's seat, the first camera120may photograph the driver and provide the first image data of the driver to the controller110. The controller110may process the first image data of the driver and identify the state of the driver based on the processing result of the image data of the driver.

The second camera130may photograph the driver in a Schlieren method. The photographing direction of the second camera130may be substantially perpendicular to the photographing direction of the first camera120.

A Schlieren-type camera can visualize the refraction of light due to a change in density of a medium (or space) through which light passes.

The Schlieren type camera includes, for example, a first lens L1and a second lens L2as shown inFIG.2, and a blocker B disposed between the first lens L1and the second lens L2.

A center line of the first lens L1may be aligned with a center line of the second lens L2so that an image is formed on the camera. In other words, the center line of the first lens L1may coincide with the center line of the second lens L2.

In order to block a portion of the light passing through the first lens L1, the blocker B may be disposed below (or above, left or right) the center line of the first lens L1as shown inFIG.2.

Even if a portion of the light is blocked by the blocker B, there are countless paths passing through the first lens L1. As a result, the image taken by the camera is only lowered in brightness and the image itself is not distorted.

For example, as shown inFIG.2A, the light may be uniformly incident on the first lens L1, and the light passing through the first lens L1passes over the blocker B and reaches the second lens L2. The light may pass through the second lens L2and travel uniformly again.

At this time, as shown inFIG.2B, the density of the air positioned in front of the first lens L1may be changed. For example, a flow of air may occur, such as when a wind blows or a strong updraft occurs, and the density of the air may become non-uniform due to the flow of air.

It is known that light is refracted by a change in density of a medium (air). As the density of the air positioned in front of the first lens L1is changed, light is refracted as shown inFIG.2B, and the path of the light may be changed due to the refraction of the light. Also, the path of the light passing through the first lens L1may be changed.

Accordingly, a part of the light passing through the first lens L1is blocked by the blocker B as shown inFIG.2B, and the other part may pass without being blocked by blocker B. Light not blocked by the blocker B may reach the second lens L2, and may be refracted by the second lens L2.

The intensity of the light passing through the second lens L2is non-uniform, and a contrast may occur between a portion having a high light intensity and a portion having a low light intensity. The contrast of light may be photographed by a camera or recognized by a human eye.

As such, the Schlieren-type camera can image (or visualize) the density change of the medium (air) using blocker B.

The second camera130, which is a Schlieren-type camera, may image a change in density of air. In particular, the second camera130may image the change in density of air due to the driver's respiration.

For clearer imaging, the second camera130includes a light source module131and an image sensor module132. The light source module131may transmit light such as infrared light toward the image sensor module132, and the image sensor module132may receive the light transmitted by the image sensor module132.

The light source module131may include a light source131a, a first lens131b, a pinhole131c, a first plane mirror131d, and a first Schlieren mirror131eas shown inFIG.3. The light source131amay emit light to image the drivers respiration. The first lens131b, the pinhole131c, and the first plane mirror131dmay focus and reflect the light emitted from the light source131atoward the first Schlieren mirror131e. The first Schlieren mirror131emay be a concave mirror, and may reflect the light emitted from the light source131aso that the light passes the front of the driver.

The light passing through the first Schlieren mirror131emay travel in parallel and uniformly, and may pass through the front of the driver. Also, light can be refracted by the air the driver discharges during respiration.

The light passing through the front of the driver may be received by the image sensor module132.

The image sensor module132includes a second Schlieren mirror132a, a second plane mirror132b, a blocker132c, and an image sensor132das shown inFIG.3.

The second Schlieren mirror132amay be a concave mirror, and may receive light passing through the front of the driver and reflect it toward the second plane mirror132b. The light reflected by the second Schlieren mirror132a, which is a concave mirror, may be focused. The second plane mirror132bmay reflect the light reflected by the second Schlieren mirror132atoward the image sensor132d.

The blocker132cmay block some of the light that has passed through the front of the driver.

The image sensor132dmay acquire an image by light not blocked by the blocker132c, and may provide image data corresponding to the acquired image to the controller110.

As such, the light source module131may transmit light through the image sensor module132, and the image sensor module132may receive the light transmitted from the light source module131.

At this time, the blocker132cof the image sensor module132may block some of the light transmitted from the light source module131and passed through the front of the driver.

Since there are countless paths through which the light passing through the front of the driver passes, even if some of the light is blocked by the blocker132c, the information of the image is not distorted, and the brightness of the image is only lowered. In other words, even if the light uniformly transmitted from the light source module131is partially blocked by the blocker132c, uniform light may be incident on the image sensor132d. Since uniform light is incident on the image sensor132d, the image acquired by the image sensor132ddoes not include any shape or pattern.

On the other hand, the density of the air through which light passes may be changed by the driver's respiration, and the light emitted from the light source module131may be refracted due to the change in the density of the air. In other words, the driver's respiration can cause refraction of light passing the front of the driver.

Accordingly, the path of the light incident on the image sensor module132is changed, the light blocked by the blocker132cmay change due to the change of the path of the light, and the light incident on the image sensor132dbecomes non-uniform. Accordingly, the image acquired by the image sensor132dmay represent a change in density of air due to the driver's respiration as shown inFIG.5.

As such, the second camera130may image the change in density of air due to the drivers respiration, and provide the controller110with second image data in which the change in density of air due to the driver's respiration is imaged.

The light source module131and the image sensor132dmay be disposed on both sides of the driver so that the light emitted by the light source module131passes through the front of the driver and is incident on the image sensor module132.

For example, as shown inFIG.4, the light source module131is disposed in the console box of the vehicle1, and the image sensor module132may be disposed at the upper left side of the driver (upper right side of the driver when the driver's seat is provided on the right side of the vehicle). The arrangement of the light source module131and the image sensor module132is not limited to that shown inFIG.4, and any arrangement may be used as long as the light emitted by the light source module131can pass through the front of the driver and be incident on the image sensor module132.

The light source module131may be omitted, and the image sensor module132may be installed in the cluster150or a room mirror.

The input switch140may acquire a driver input for activating or deactivating the operation of the driver monitoring apparatus100. The input switch140may be installed on the steering wheel171, for example. Also, the input switch140may include, for example, a tact switch, a push switch, a slide switch, a toggle switch, a micro switch, or a touch switch.

The controller110may be electrically connected to the first camera120, the second camera130, and the input switch140. In addition, the controller110may be connected to the cluster150, the multimedia device160, the power steering170, and the seat driving device180of the vehicle1through vehicle communication.

The controller110may include a processor111that processes the image of the first camera120and the image of the second camera130and provides a control signal for controlling the operation of the driver monitoring apparatus100, and a memory112for processing the image of the first camera120and the image of the second camera130and storing programs and data for controlling the operation of the driver monitoring apparatus100. The controller110may include, for example, one or more processors or one or more memories. The processor111and the memory112may be implemented as separate semiconductor element or as a single semiconductor element.

The memory112includes volatile memories such as Static Random Access Memory (S-RAM) and Dynamic Random Access Memory (D-RAM), and non-volatile memories such as Read Only Memory (ROM), Erasable Programmable Read Only Memory (EPROM). The memory112may include one memory element or a plurality of memory elements.

The processor111may include an arithmetic circuit, a memory circuit, and a control circuit. The processor111may include one chip or a plurality of chips. Also, the processor111may include one core or a plurality of cores.

The controller110may process the image of the first camera120and the image of the second camera130by the program and data stored in the memory112and the operation of the processor111.

The controller110may obtain first image data from the first camera120and process the first image data. For example, the controller110may extract an image of the driver's face from the first image data, and identify whether the driver is in a sleep state from the driver's face. The controller110may include an identification engine obtained through machine learning using, for example, a Convolutional Neural Network (CNN) to identify whether the driver is in a sleep state.

The controller110, based on the driver's sleep state, may transmit a control request for warning of the driver's drowsy driving to the cluster150, the multimedia device160, the power steering170and the seat driving device180of the vehicle1.

The controller110may also obtain second image data from the second camera130and process the second image data.

The controller110may identify a respiratory cycle (e.g., respiratory rate per minute) of the driver based on the second image data. For example, the second camera130may acquire a Schlieren image in front of the driver at a predetermined period, and may transmit second image data corresponding to the acquired Schlieren image. Based on the second image data, the controller110may identify region A in which the air density is changed by the driver's respiration as shown inFIG.5. The size of region A can be changed periodically by the driver's periodic respiration. Based on the period (or number) at which the size of region A becomes the maximum (or number of times), the controller110may identify the driver's respiratory cycle (e.g., respiratory rate per minute).

Also, the controller110may identify the driver's respiratory volume (e.g., the amount of air discharged during one breath) based on the second image data. Based on the second image data, the controller110may identify region A in which the air density is changed by the driver's respiration as shown inFIG.5. The controller110may identify the respiratory volume of the driver based on region A. When the size of region A is maximum, a table including the respiratory volume corresponding to the size of region A can be stored in the controller, the controller110may identify a respiratory volume corresponding to the maximum size of region A using the table. As another example, the controller110may be provided with an identification engine that identifies the respiratory volume corresponding to the maximum size of region A by machine learning, and the controller110may identify the respiratory volume corresponding to the maximum size of region A using the identification engine.

In addition, the controller110may identify respiration by the nose and respiration by the mouth of the driver based on the first image data. For example, the controller110may identify the driver's nose respiration and mouth respiration based on the direction in which air is discharged by respiration.

The controller110may identify the driver's stress state and/or drowsy driving state based on the driver's respiratory cycle and/or respiratory volume.

The controller110may transmit a control request for warning the driver's stress state and/or drowsy driving state based on the driver's stress state and/or drowsy driving state to the cluster150, the multimedia device160, the power steering170, and the seat driving device180of the vehicle1.

As such, the driver monitoring apparatus100may identify the state of the driver based on the facial image of the first camera120and/or the Schlieren image of the second camera130, and may transmit a control request for warning the driver's state in response to the driver's identification to the cluster150, the multimedia device160, the power steering170and the seat driving device180of the vehicle1.

The cluster150displays driving information of the vehicle1including the driving speed of the vehicle1, the engine RPM and/or fuel amount, etc., and may be located in front of the driver as shown inFIG.4. The cluster150may display an image message for warning the driver's stress state and/or drowsy driving state in response to a control request of the driver monitoring apparatus100.

The multimedia device160includes a display161that displays a still image (or moving image) for the convenience and fun of the driver, and a speaker162that outputs sound for the convenience and fun of the driver. The display161may display an image message for warning of a stress state and/or a drowsy driving state of the driver in response to a control request of the driver monitoring apparatus100. The speaker162may output an sound message for warning of a stress state and/or a drowsy driving state of the driver in response to a request of the driver monitoring apparatus100.

The power steering170may detect the driver's will to steer through the steering wheel171and assist the steering of the vehicle1in response to the driver's will to steer. In addition, the power steering170may generate vibrations of the steering wheel171in response to a control request of the driver monitoring apparatus100to warn of a stress state and/or a drowsy driving state of the driver.

The seat driving device180may adjust the position of the seat181in response to the seat movement command of the driver. Also, the seat driving device180may generate vibration of the seat181in response to a control request of the driver monitoring apparatus100to warn of a stress state and/or a drowsy driving state of the driver.

FIG.6illustrates another example of a second camera according to an exemplary embodiment.

InFIGS.3and4, the second camera130in which the light source module131and the image sensor module132are separated from each other has been described, but the present disclosure is not limited thereto.

For example, the driver monitoring apparatus100may include a second camera200and a controller110in which a light source module131and an image sensor module132are integrated.

As shown inFIG.6, the driver monitoring apparatus100may include a second camera200, and the second camera200may include a light source module131and an image sensor module132. The light source module131and the image sensor module132may be the same as the light source module and image sensor module shown inFIG.3, and the description is replaced with the description of the light source module and the image sensor module shown inFIG.3.

In addition, the second camera200may further include a housing201, and the housing201may accommodate the light source module131and the image sensor module132.

At one side of the housing201, an inlet201athrough which air by the driver's respiration can be introduced into the housing201may be formed. The second camera200can image the density change of the air inside the housing201by the air introduced into the housing201by using the light source module131and the image sensor module132. Also, the second camera200may provide the controller110with second image data in which the density change of air due to the driver's respiration is imaged.

The controller110may identify the driver's respiratory cycle (e.g., respiratory rate per minute) and/or the driver's respiratory volume (e.g., the amount of air discharged during one breath) based on the second image data, and identify the driver's stress state and/or drowsy driving state based on the driver's respiratory cycle and/or the driver's respiratory volume. Also, the controller110may warn the driver's stress state and/or drowsy driving state based on the driver's stress state and/or drowsy driving state.

FIG.7illustrates an operation of a vehicle according to an exemplary embodiment.

Referring toFIG.7, an operation1000of the vehicle1for identifying and warning the driver's stress state and/or drowsy driving state is described.

The vehicle1acquires a facial image of the driver (1010).

The driver monitoring apparatus100may acquire a face image of the driver using the first camera120. The first camera120may photograph the driver's face and provide facial image data to the controller110.

The vehicle1identifies whether the driver is in sleep state (1020).

The controller110of the driver monitoring apparatus100may identify whether the driver is in a sleep state based on the facial image data. The controller110may include an identification engine obtained through machine learning using, for example, CNN to identify whether the driver is in a sleep state.

When the driver's sleep state is identified (YES in1020), the vehicle1warns the driver of drowsy driving (1075).

The driver monitoring apparatus100may transmit a control request for warning of the driver's drowsy driving to the cluster150, the multimedia device160, the power steering170, and the seat driving device180of the vehicle1.

If the sleep state of the driver is not identified (NO in1020), the vehicle1acquires a Schlieren image (1030).

The driver monitoring apparatus100may acquire a Schlieren image using the second camera130. The second camera130may photograph the front of the driver and provide Schlieren image data to the controller110.

The vehicle1identifies whether the respiratory rate per minute of the driver is “0” (1040).

The driver monitoring apparatus100may identify a respiratory rate (or respiratory cycle) per minute of the driver based on the Schlieren image data. The controller110may identify region A in which the air density changes due to the driver's respiration based on the Schlieren image data, and identify the respiratory rate per minute of the driver based on the period (or number of times) at which the size of region A becomes the maximum (or minimum).

If the respiratory rate per minute of the driver is “0” (YES in1040), the vehicle1identifies whether the time at which the respiratory rate is “0” is the reference time or more (1050).

The reference time may be set as an acceptable respiratory arrest time.

If the respiratory rate per minute of the driver is “0” and the time at which the respiratory rate is “0” is the reference time or more (YES in1050), the vehicle1warns the driver's respiratory arrest (1055).

The driver monitoring apparatus100may transmit a control request for warning of respiratory arrest of the driver to the cluster150, the multimedia device160, the power steering170and the seat driving device180of the vehicle1.

In addition, the driver monitoring apparatus100may transmit an emergency report to an emergency medical center or an emergency report center through a wireless communication device installed in the vehicle1in order to warn the driver's respiratory arrest.

If the respiratory rate per minute of the driver is not “0” (NO in1040), the vehicle1identifies whether the respiratory rate per minute is the first respiratory rate or more (1060).

The driver monitoring apparatus100may compare the respiratory rate per minute based on the Schlieren image data with the first respiratory rate. It is known that the respiratory rate of the driver increases when the driver is in a sleep state or a stress state. The first respiratory rate may be set as a respiratory rate indicating the driver's sleep state or stress state.

If the respiratory rate per minute is not more than the first respiratory rate (NO in1060), the vehicle1may acquire the facial image and the Schlieren image again.

If the respiratory rate per minute is the first respiratory rate or more (YES in1060), vehicle1identifies whether the driver's respiratory volume is less than the first respiratory volume (1070).

The driver monitoring apparatus100may identify the respiratory volume by one respiration of the driver based on the Schlieren image data. The controller110may identify region A in which air density is changed by the driver's respiration based on the Schlieren image data, and may identify the driver's respiratory volume based on the maximum size of region A.

The controller110may compare the driver's respiratory volume with the first respiratory volume. It is known that the respiratory volume of the driver decreases in the sleep state. The first respiratory volume may be set as a respiratory volume indicating the sleep state of the driver.

If the respiratory rate per minute is the first respiratory rate or more and the driver's respiratory volume is less than the first respiratory volume (YES in1070), the vehicle1warns the driver of drowsy driving (1075).

The driver monitoring apparatus100may transmit a control request for warning of the driver's drowsy driving to the cluster150, the multimedia device160, the power steering170, and the seat driving device180of the vehicle1.

If the respiratory rate per minute is the first respiratory rate or more and the respiratory volume of the driver is not less than the first respiratory volume (NO in1070), the vehicle1identifies whether the respiratory volume of the driver is the second respiratory volume or more (1080).

The controller110of the driver monitoring apparatus100may compare the respiratory volume of the driver with the second respiratory volume. It is known that the respiratory volume of the driver increases in a stress state. The second respiratory volume may be set as a respiratory volume representing the driver's stress state, and may be a value greater than the first respiratory volume.

If the respiratory rate per minute is the first respiratory rate or more and the respiratory volume of the driver is the second respiratory volume or more (YES in1080), the vehicle1warns the stress state of the driver (1085).

The driver monitoring apparatus100may transmit a control request for warning the driver's stress state to the cluster150, the multimedia device160, the power steering170, and the seat driving device180of the vehicle1.

If the respiratory rate per minute is the first respiratory rate or more and the driver's respiratory volume is greater than the first respiratory volume and less than the second respiratory volume (NO in1080), the vehicle1can acquire the facial image and the Schlieren image again.

As described above, the vehicle1may identify the driver's state based on the driver's facial image and/or the driver's Schlieren image, and transmit a control request for warning the driver's state in response to the driver's identification to the cluster150, the multimedia device160, the power steering170and the seat driving device180of the vehicle1.

According to one aspect of the present disclosure, it is possible to provide a driver monitoring apparatus capable of detecting the state of a driver using a Schlieren camera, a vehicle, and a control method thereof.

According to one aspect of the present disclosure, a driver monitoring apparatus, a vehicle, and a control method thereof use a Schlieren camera to image the flow of gas by the driver's respiration as well as the driver's face using a Schlieren camera, and can detect the driver's state based on this.

According to one aspect of the present disclosure, the driver monitoring apparatus, vehicle, and control method thereof can detect the state of the driver by detecting the periodic air flow caused by the driver's respiration, and can significantly reduce false detection compared to the existing driver monitoring system that only relied on the image of the driver.

Exemplary embodiments of the present disclosure have been described above. In the exemplary embodiments described above, some components may be implemented as a “module”. Here, the term ‘module’ means, but is not limited to, a software and/or hardware component, such as a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), which performs certain tasks. A module may advantageously be configured to reside on the addressable storage medium and configured to execute on one or more processors.

Thus, a module may include, by way of example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The operations provided for in the components and modules may be combined into fewer components and modules or further separated into additional components and modules. In addition, the components and modules may be implemented such that they execute one or more CPUs in a device.

With that being said, and in addition to the above described exemplary embodiments, embodiments can thus be implemented through computer readable code/instructions in/on a medium, e.g., a computer readable medium, to control at least one processing element to implement any above described exemplary embodiment. The medium can correspond to any medium/media permitting the storing and/or transmission of the computer readable code.

The computer-readable code can be recorded on a medium or transmitted through the Internet. The medium may include Read Only Memory (ROM), Random Access Memory (RAM), Compact Disk-Read Only Memories (CD-ROMs), magnetic tapes, floppy disks, and optical recording medium. Also, the medium may be a non-transitory computer-readable medium. The media may also be a distributed network, so that the computer readable code is stored or transferred and executed in a distributed fashion. Still further, as only an example, the processing element could include at least one processor or at least one computer processor, and processing elements may be distributed and/or included in a single device.

While exemplary embodiments have been described with respect to a limited number of embodiments, those skilled in the art, having the benefit of this disclosure, will appreciate that other embodiments can be devised which do not depart from the scope as disclosed herein. Accordingly, the scope should be limited only by the attached claims.