Patent Description:
Modern vehicles are generally equipped with various types of monitoring systems, such as cameras, or video recorders to monitor surrounding environment of vehicles and provide a driver of a vehicle with useful data regarding the surrounding environment for improved driving. Such monitoring systems may be installed, for instance, on a roof of the vehicle or on the front portion, back portion of the vehicle to have a broad view of the surrounding environment and capture data associated with objects, pedestrians or vehicles within the surrounding environment.

In addition, the monitoring systems may also monitor the driver of the vehicle for facial pose and gaze. For instance, the driver may be monitored for orientation of the face and the gaze to be in a forward direction and determine if the driver is paying attention on the road. The collected data is then subjected to processing to derive meaningful information that may be used in assisting the driver for navigation, changing lanes, and averting a potential collision. An event, such as an approaching vehicle, a pedestrian on the road may be detected and a warning may be issued to the driver to help the driver initiate a precautionary action.

Application no <CIT> discloses a cognitive-driver-assist system that varies the warning- intensity of a warning or varies the degree to which the system takes over control of a host- vehicle driven by an operator. The warning-intensity and the take-over authority are determined in accordance with a measure of how aware the operator is of a particular object proximate to or in the travel-path of the host-vehicle. US application no. <CIT> disclosed a long term driving danger prediction system and methods to assist a driver with a dangerous condition by combining several sensor modalities available on modern cars to detect traffic participants as well as static elements. Another US application no. <CIT> relates to an on-vehicle situation detection apparatus and method, which grasp a driver's mental and physical condition to determine whether or not a driver drives a vehicle with safety and induce the driver to drive the vehicle with safety in various ways when the driver is determined to be not in a safe driving state so as to protect the driver. Further, it can detect situations in real time through rapid calculation since a considerable time is required when a great deal of data is collected from a plurality of sensors and the situations are detected according to technological development. Patent application no. <CIT> provided a hazard notification system for a vehicle operable by a driver, the notification system comprising: an input to receive information of a current event; a database comprising a plurality of records, a processor arranged to determine a record from the database, by comparing the received information associated with the current event to the information associated with the earlier events, and retrieve a response of the driver corresponding to the determined record from the database; and an output arranged to notify the driver of the current event in dependence on the response of the driver corresponding to the determined record.

However, such monitoring systems, on many occasions, fail to detect events with accuracy due to various factors such as incomplete data or incorrect data, and issue false or irrelevant warnings to the driver. These warnings are generally issued at high volumes to alert the driver that on many instances may startle or distract the driver, thereby inciting a sudden action that could be potentially harmful for the safety of the driver. Further, such irrelevant warnings issued regularly at high volumes may cause a general discomfort, and impact driving of the driver. Therefore, the monitoring systems are not efficient in detecting events and issuing warning to the drivers for enhancing driving experience and safety.

This summary is provided to introduce concepts related to monitoring drivers and external environment for vehicles.

According to the present invention, an ADAS includes a driver monitoring module, an exterior monitoring module, a processor coupled to the driver monitoring module and the exterior monitoring module, and a warning generating module coupled to the processor.

The driver monitoring module may be positioned facing a driver to monitor a driver state, such as drowsiness, sleepiness, and inattentiveness of the driver. The driver state may be monitored based on various factors, such as blinking of eyes, head movement, color of skin, frowning, eye ball movements. For instance, a closing of eyes more than <NUM> seconds may be classified as drowsiness, and a high degree of head movement may be classified as inattentiveness.

Simultaneously, the exterior monitoring module, facing the road, captures data regarding objects, lane markings, potholes, speed signs, and traffic conditions from external environment. The exterior monitoring module may also monitor driving pattern of the driver. For example, whether the driver is driving the vehicle in line with the lanes of the road or driving inconsistently and frequently crossing the lanes and the boundaries of the road.

According to the present invention, the exterior monitoring module adjusts a Region of Interest (ROI) of the view of the road or external environment based on data associated with a path of travel of the vehicle, comprising at least one of a driver indication for a turn, a steering wheel angle change, a path travelled by the vehicle, and a Global Positioning System (GPS) signal. In one example, when the driver is planning to take an exit route and provides an indicator of taking the exit route on right then based on the indicator input, the exterior monitoring module may shift the ROI to right on the exit route and monitor pedestrians, objects, and other vehicles on the exit route.

The data captured from the driver monitoring module and the exterior monitoring module are sent to the processor. The processor processes the data to detect occurrence of an event, for instance, an approaching pedestrian or a vehicle, an object on the road. Upon detection of an event, the processor signals the warning generating module to generate a warning based on the detected event. For instance, a continuous beep sound or a voice based alert may be generated to alert the driver about the event.

According to the invention, intensity of the warning is varied based on the data received from at least one of the exterior monitoring module and the driver monitoring module. For instance, when the vehicle is approaching a pedestrian, who is at a distance of around <NUM> meters, and the driver is attentive, then volume of the beep sound may be low. However, when the vehicle is approaching the pedestrian with speed, the volume of the beep sound is increased until a preventive action is taken to avert a collision or until the pedestrian reaches a safe place that is outside the view of the exterior monitoring module.

It would be noted that the driver monitoring module, the exterior monitoring module, the processor, and the warning generating module may operate in real-time to capture and process the data to generate the warning. Further, the intensity of the warning is also varied in real-time based on criticality of the event.

Although, the present invention has been described with reference to an integrated ADAS comprising the modules, the present invention may also be applicable to monitoring driver and the external environment by the modules placed at different areas within an autonomous vehicle, wherein the modules are communicatively coupled to each other.

Thus, the present invention provides efficient techniques for detecting events and alerting a driver. The techniques provide adaptive warning to the driver of the vehicle, wherein intensity level of the warning is varied based on driver state data, driving pattern data and the external environment data. Further, the events detected are accurate and the warning generated are relevant to a specific situation to enable the driver respond aptly to the events thereby enhancing driver safety.

Other and further aspects and features of the invention will be evident from reading the following detailed description of the embodiments, which are intended to illustrate, not limit, the present invention.

The illustrated embodiments of the invention will be best understood by reference to the drawings, wherein like parts are designated by like numerals throughout. The following description is intended only by way of example, and simply illustrates certain selected embodiments of devices, systems, and processes that are consistent with the invention as defined by the appended claims.

A few inventive aspects of the disclosed embodiments are explained in detail below with reference to the various figures. Embodiments are described to illustrate the disclosed invention, not to limit its scope, which is defined by the claims. Those of ordinary skill in the art will recognize a number of equivalent variations of the various features provided in the description that follows.

Reference throughout the specification to "various embodiments," "some embodiments," "one embodiment," or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, appearances of the phrases "in various embodiments," "in some embodiments," "in one embodiment," or "in an embodiment" in places throughout the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments.

Referring now to <FIG>, an example environment <NUM> in which various embodiments may function is illustrated. As shown the environment <NUM> includes a vehicle <NUM> moving or being driven on a road <NUM>. The vehicle <NUM> may be a car, a jeep, a truck, a bus, or a three-wheeler vehicle. The vehicle <NUM> may have parts like steering wheel, tires, brake, engine, carburetor, doors, horn, lights, etc. not shown in the figure. Also, the vehicle <NUM> may be provided with physical actuators connected to critical function parts like brakes, engine control unit, steering wheel, horn and lights.

The vehicle <NUM> further includes an Advanced Driver Assistance System (ADAS) <NUM> positioned such that the ADAS <NUM> may monitor facial expressions of the driver and may monitor the external environment. In one example, the ADAS <NUM> may be positioned close to the rear view mirror of the vehicle <NUM>. It would be noted that, although the ADAS <NUM> is shown positioned near the rear view mirror, the ADAS <NUM> may be positioned at other places with in the vehicle <NUM>. For instance, the ADAS <NUM> may be positioned on one of a windshield behind an internal rear view mirror, an "A" pillar of the vehicle <NUM>, and on a dashboard.

The ADAS <NUM> may have various modules to collect external data, such as data associated with roads, pedestrians, objects, road edges, lane marking, potential collision, speed signs, potholes, vehicles, and a driving pattern of the driver on the road. Additionally, the ADAS may monitor driving pattern of the driver such as whether the driving is in line with the lanes and boundaries of a road. Further, the modules may also capture data related to driver state, such as facial expressions and features, blink rate of eyes, eyeball movement, opening of the eye, and head movement of the driver. The ADAS <NUM> may also warn the driver corresponding to events, such as a pedestrian crossing the road, or a cyclist in front of the vehicle.

It would be noted that ADAS <NUM>, in one example, may have the modules placed at different positions within the vehicle. For instance, the module for monitoring the driver may be coupled to the windscreen and the module to generate the warning may be coupled to the A-pillar. Such components may either be connected through a wired connection or through a wireless communication to communicate and share data.

In one example, the ADAS <NUM> may be connected to an external server (not shown in figure) through a wireless network, such as a datacenter for cloud backup and data archiving purpose. For instance, information associated with occurrence of an event and preventive action taken by the driver may be recorded for a predefined time span of <NUM> minute, <NUM> seconds, or <NUM> seconds and relayed to the datacenter. Such information may be stored within the datacenter and may be used for analyzing driver pattern during the events and providing useful information to other drivers in similar situations. Also, the information may be utilized for validating insurance claims or insurance premium calculations.

In another example, the ADAS <NUM> may be connected to the actuators as mentioned above. This helps to take over control of these critical function parts in an event of user failing to react.

The details of the components or modules of the ADAS <NUM> and functionality of the modules have been further explained with reference to description of the forthcoming figures.

2A illustrates various modules of the ADAS <NUM>, in accordance with an implementation of the present invention. The ADAS <NUM> includes an exterior monitoring module <NUM>, a driver monitoring module <NUM>, a ranging module <NUM>, a processor <NUM>, a memory <NUM> and a warning generation module <NUM>. The processor <NUM> may be communicably connected to the exterior monitoring module <NUM>, the driver monitoring module <NUM>, and the ranging module <NUM>. The processor <NUM> may also be communicably connected to a memory <NUM> and the warning generation module <NUM>.

In an implementation, the modules, such as the exterior monitoring module <NUM>, the driver monitoring module <NUM>, the ranging module <NUM>, and the driver monitoring module <NUM> may include routines, programs, objects, components, data structures, and the like, which perform particular tasks or implement particular abstract data types. The modules may further include modules that supplement applications on the ADAS <NUM>, for example, modules of an operating system. Further, the modules can be implemented in hardware, instructions executed by a processing unit, or by a combination thereof.

In another aspect of the present invention, the modules may be machine-readable instructions which, when executed by a processor/processing unit, perform any of the described functionalities. The machine-readable instructions may be stored on an electronic memory device, hard disk, optical disk or other machine-readable storage medium or non-transitory medium. In an implementation, the machine-readable instructions can also be downloaded to the storage medium via a network connection.

The exterior monitoring module <NUM> is configured to be facing out of the vehicle <NUM>. This configuration helps the exterior monitoring module <NUM> to capture data from environment external to the vehicle <NUM>.

In an embodiment of the present invention, the exterior monitoring module <NUM> may include a stereo camera 202A and a long range narrow field camera 202A. The stereo camera 202A may be a dual lens camera having a short range. This helps the stereo camera 202A to capture data within a short distance of the vehicle <NUM>. The stereo camera 202A captures the nearby objects, events and data. Further, the long range narrow field camera 202B is configured to capture events at a farther distance and hence captures objects, events and data at a longer distance from the vehicle <NUM>. The stereo camera 202A and the long range narrow camera 202B may be configured to adjust autofocus with the changing environment. The capturing ranges of the stereo camera 202A and the long range narrow field camera 202B may overlap to capture maximum data from external environment. The exterior monitoring module <NUM> is configured to shift its region of interest. The shifting of the region of interest may be based upon a condition of path of travel of the vehicle <NUM>. Details of shifting of region of interest will be described in detail in conjunction with FIGs 4A-4C.

The driver monitoring module <NUM> is positioned to face the driver of the vehicle <NUM> and monitors driver state of the driver. The driver state is determined utilizing driver's eye gaze, facial expressions and head movement. Various driver states that may be determined by the driver monitoring camera are fatigue, sleepiness, anger, happy, jolly, sad, neutral, etc. Hence the driver monitoring module <NUM> is capable of determining multiple driver states. In another implementation of the present invention, the driver monitoring module <NUM> may be a charged coupled device camera, or a Complementary Metal Oxide Semiconductor (CMOS) camera.

In yet another embodiment of the present invention, the ranging module <NUM>, used for determining distance to objects may be one of a light detection and ranging (LiDAR) unit, a radio detection and ranging (RADAR), a sonic detection and ranging (SODAR), and a sound navigation and ranging (SONAR).

In another embodiment of the present invention, the warning generation module <NUM> may include an audio, visual, or haptic warning interfaces. The warning generation module <NUM> may include a Light Emitting Diode (LED) display, a Liquid Crystal Display (LCD), a plasma display, a warning light emitter, a speaker, a haptic feedback module, or a combination thereof.

The processor <NUM>, amongst other capabilities, may be configured to fetch and execute computer-readable instructions stored in a memory. The processor <NUM> may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. The functions of the various elements shown in the figure, including any functional blocks labelled as "processor(s)", may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software.

The processor <NUM> and other modules like the exterior monitoring module <NUM>, the driver monitoring module <NUM>, the ranging module <NUM> and the warning generation module <NUM> as described above may be implemented as hardware or software. If such modules are implemented in software, one or more processors of the associated computing system that performs the operation of the module direct the operation of the computing system in response to having executed computer-executable instructions. In another implementation, the processor <NUM> may also be connected to GPS, indicator of the vehicle <NUM> or pre-fed path of the route to be covered by the vehicle <NUM>.

In yet another embodiment of the present invention, a memory <NUM> may be utilized to store the collected external environment and internal environment data collected. The memory <NUM> may be without limitation, memory drives, removable disc drives, etc., employing connection protocols such as serial advanced technology attachment (SATA), integrated drive electronics (IDE), IEEE-<NUM>, universal serial bus (USB), fiber channel, small computer systems interface (SCSI), etc. The memory drives may further include a drum, magnetic disc drive, magneto-optical drive, optical drive, redundant array of independent discs (RAID), solid-state memory devices, solid-state drives, etc..

In operation, the exterior monitoring module <NUM> may continuously record the surrounding environment of the vehicle <NUM> and provide the input to the processor <NUM>. In one example instant, the surrounding environment may include a pedestrian crossing the road and about <NUM> meters away from the vehicle <NUM>, two vehicles ahead of the vehicle <NUM> at distances <NUM> meters and <NUM> meters and a cyclist on a left lane at a distance of <NUM> meters from the vehicle <NUM>. The objects in the surrounding environment, such as the pedestrian, the two vehicles, and the cyclist are continuously monitored by the exterior monitoring module <NUM> in real-time and sent to the processor <NUM>.

In another example, the exterior monitoring module <NUM> may detect the lanes or boundaries of a road or path travelled by the vehicle <NUM>, as shown in <FIG>. The <FIG> illustrates the road <NUM> travelled by the vehicle <NUM>, and the Region of Interest (ROI) <NUM> as focused by the exterior monitoring module <NUM>. In an example implementation, the exterior monitoring module <NUM> may move the ROI <NUM> to capture the boundary of the road <NUM> and a lane present on the road <NUM>. In one example, the frames 308A, 308B, and 308C illustrate the ROI <NUM> as captured by the exterior monitoring module <NUM>.

According to the invention , the shape of the ROI is adjusted to adapt to the changing external environment data. For example, the shape of the ROI may be changed to fit to a particular portion of a curved road. Similarly, size change may be effected to cover cross roads conditions.

In one example, the ROI may be moved in a linear direction from left to right or from right to left. In another example, the ROI may be moved in a non-linear manner from one point to another point depending upon the external conditions. For example, the ROI may be moved from a first point on a straight road to a second point on an elevated road, where the second point is above the plane of the first point. Thereafter, the exterior monitoring module <NUM> may capture the driving pattern of the driver based on the area of the road <NUM> covered by the vehicle <NUM> during travel. For instance, the exterior monitoring module <NUM> may capture when the vehicle is driven along the lane or the boundary of the road <NUM>, wherein the area covered by the vehicle <NUM> completely aligns with one of the lane and the boundary of the road <NUM>. In a scenario, the vehicle may be driven in a zig-zag manner that does not align with the one of the road boundary and the lane and frequently crosses the lanes. In an example implementation, the vehicle <NUM> may be driven in a wave-manner as illustrated in the <FIG>.

It would be noted that the driving pattern is indicative of the manner in which the vehicle <NUM> is being driven on the road <NUM>. For instance, when a driver is attentive then the driver is likely to drive in alignment with the lane or the boundary and when the driver is inattentive, drowsy, or under the influence of a drug, the driver may drive inconsistently and not in alignment with the lane or the boundary.

Similarly, driver state data is captured by the driver monitoring module <NUM>. For example, the driver monitoring module <NUM> may continuously record facial expressions of the driver for eye gaze, blink rate of eyelids, change in skin tone, nostrils, jaw movements, frowning, baring teeth, movement of cheeks, movement of lips and head movements when the driver is driving the vehicle on the road <NUM>. The continuous recording of the driver state is fed to the processor <NUM>.

In an example, the processor <NUM> receives the data from the exterior monitoring module <NUM> and the driver monitoring module <NUM> and processes the received data. The processing of the data as received by the processor <NUM> is explained in conjunction with the description of <FIG>.

<FIG> illustrates various engines of the processor <NUM>, in accordance with an implementation of the present invention. Engines may be microcontrollers functioning in tandem with each other to achieve coordinated output from the processor <NUM>. The processor <NUM> includes a data receiving engine <NUM>, a determination engine <NUM>, a rating engine <NUM>, a weightage engine <NUM>, and a driving pattern engine <NUM>. The determination engine <NUM> may be communicably connected to the data receiving engine <NUM>, the rating engine <NUM>, the weightage engine <NUM> and the driving pattern engine <NUM>.

In an implementation, the engines such as the data receiving engine <NUM>, the determination engine <NUM>, the rating engine <NUM>, the weightage engine <NUM> and the driving pattern engine <NUM> may include routines, programs, objects, components, data structure and the like, which perform particular tasks or implement particular abstract data types. The engines may further include engines that supplement applications on the processor <NUM>, for example, modules of an operating system. Further, the engine can be implemented in hardware, instructions executed by a processing unit, or by a combination thereof.

In another aspect of the present invention, the engines may be machine-readable instructions which, when executed by a processor/processing unit, perform any of the described functionalities. The machine-readable instructions may be stored on an electronic memory device, hard disk, optical disk or other machine-readable storage medium or non-transitory medium. In an implementation, the machine-readable instructions can also be downloaded to the storage medium via a network connection.

The data receiving engine <NUM> is communicably connected to the determination engine <NUM>, the driving pattern engine <NUM> and the weightage engine <NUM>. The data receiving engine <NUM> forwards the data received simultaneously to the determination engine <NUM>, the weightage engine <NUM> and the driving pattern engine <NUM> for simultaneous processing. The determination engine <NUM> may also be communicably connected to the rating engine <NUM>.

In an example operation, the data receiving engine <NUM> may receive data associated with path of travel of the vehicle <NUM>. For instance, the data regarding the changes may be generated by steering angle change more than a predetermined threshold, indicator switched on by the driver, path change like a curving road or crossroads or lane change as per GPS or on the basis of a pre-fed route in navigation system of the vehicle. The data receiving engine <NUM> may also be configured to receive data from a vehicle console or an Electronic Control Unit (ECU) of the vehicle <NUM>. The ECU may communicate data regarding changes the driving path of the vehicle <NUM>. The data may be then forwarded to the determination engine <NUM>.

The determination engine <NUM> may analyze the change data and generate a signal to be sent to the external monitoring module <NUM> to change its focus position in order to change ROI. The signal generated may be received by the external monitoring module <NUM> through an interface (not shown in figure). The interface may be configured to interpret the information within the signal and change the orientation of the external monitoring module <NUM> in order to change the ROI as per the change data. For orientation change of the external monitoring module <NUM>, the ADAS <NUM> may include low power DC motors (not shown in figure) to help in movement of the external monitoring module <NUM>. In another example, the ROI may be shifted without a change in the orientation of the exterior monitoring module <NUM>. The exterior monitoring module <NUM> may change the focal length or zone to adjust the ROI.

<FIG>, illustrate shifting of the ROI, in accordance to an embodiment of the present invention. <FIG> depicts shifting of region of interest 306A (similar to region of interest <NUM>) based on the indicator initiation by the driver of the vehicle <NUM>. As shown, if the driver of the vehicle <NUM> faces a situation wherein the driver has to shift to its existing lane <NUM> to one on the left <NUM>, the driver may turn on the indicator. Upon the turning on of the indicator, the data receiving engine <NUM> may receive this information from the vehicle console or the ECU and direct the exterior monitoring module <NUM> to shift or adjust the ROI 306A. The ROI 306A may then be shifted a new ROI 306B. After shifting the ROI, the exterior monitoring module <NUM> may monitor the external environment data for the new ROI <NUM> B. In other implementations, the region of interest 306A may also be shifted based on the steering wheel angle change. If the steering wheel angle is changed beyond a threshold value, the region of interest 306A is shifted to new region of interest 306B.

In another scenario, as depicted in <FIG>, in case of an approaching cross road <NUM>, the region of interest 306A is adjusted in a perspective manner. The ROI 306A is expanded to new region of interest 306B to cover more area of the road so as to monitor traffic on other roads and detect if other vehicles approach the vehicle <NUM> for a potential collision.

Similarly, region of interest 306A may also be shifted based on pre-fed route input, as illustrated by <FIG>. The driver may input a route into the vehicle <NUM> navigation console. Based on the navigation route, in case the vehicle <NUM> needs to take a right lane from a bifurcation <NUM> on road <NUM>, as shown in the <FIG>, then the region of interest 306A is gradually shifted to a new region of interest 306B. The region of interest 306A may be gradually shifted and not suddenly as may be the case, with indicator initiation or steering wheel angle change. Thereafter, the external environment data is provided to the processor <NUM>.

In an example, the data receiving engine <NUM> of the processor <NUM>, receives the external environment data, and the driving pattern data from the exterior monitoring module <NUM> and the driver state data from the driver monitoring module <NUM>. The combined data collected by the data receiving engine <NUM> may be saved in the memory <NUM> connected to the processor <NUM>. Further, this data may also be sent to the determination engine <NUM>, the weightage engine <NUM> and the driving pattern engine <NUM> for simultaneous processing.

For determining the driving pattern by the processor <NUM>, the determination engine <NUM> may utilize various techniques. In one example, pixel mapping as shown in <FIG>, may be utilized for calculating the driving pattern. The determination engine <NUM> determines extremes of the road 104A and 104B. From the image captured, a pixel map 308A may be generated, that may be made up of multiple pixels 310A-310A.

Each pixel image may have multiple pixel rows for e.g. I, II, III, IV, V, VI, VII within which pixels 310A-310N lines. It should be appreciated by a person having ordinary skill in the art that there may be more pixel rows and the number of rows may be manipulated based on the requirement and the equipment scope. Pixel maps for the subsequent time frames captured may be generated, like 308B and 308C, as shown in <FIG> respectively. The pixel maps 308A, 308B and 308C are then compared to each other in ascending order of time and the frequency of variation. The values of frequency variation Fv may be compared to a threshold value Ft.

There may be a variation captured as shown in pixel row VII and I, II, and III in pixel maps 308B and 308C as the vehicle <NUM> moves from its position on road during its motion. Hence the variation may be calculated for subsequent frames too and the frequency variation may be then calculated. For example, in <FIG>, the driving pattern may be determined as "Rash" as the path traversed <NUM> has high variation frequency.

The data receiving engine <NUM> may also forward the data to the driving pattern engine <NUM>. The driving pattern engine <NUM> determines driving pattern of the driver by using the data captured by the exterior monitoring module <NUM>. Details of driving pattern determination have been explained earlier in conjunction with <FIG>. In one example, the driving pattern engine <NUM> determines the criticality of the event in terms of likelihood or probability of having a damage caused to the vehicle <NUM> or the driver such as a potential collision with a pedestrian, an object on the road, another vehicle. The criticality may be determined based on various parameters, such as closeness of an object to the vehicle, speed of the vehicle, multiple objects present in front of the vehicle, the surrounding terrain. The driving pattern of the driver, after determination, may be forwarded to the determination engine <NUM> in order to be added to the event data already determined.

Thereafter, the determination engine <NUM>, analyzes external environment data for any events like object identification, lane markings, speed limit signs, etc. In an example, the determination engine <NUM> may utilize known techniques of object detection and image processing to detect the events. Further, simultaneously the determination engine <NUM> determines the driver state using the driver data. The determination engine <NUM>, may use a pixel mapping technique to identify the facial expression and related driver emotions.

The determination engine <NUM> may utilize pre-stored templates to which pixel maps of the driver data may be utilized to determine driver state. The pre-stored templates may be stored within the memory <NUM> that may be fetched in real-time.

In one aspect, there could be a conflicting condition wherein one set of data may contradict with another set of data. For instance, it may be determined by the determination engine <NUM> that the driver is drowsy based on the driver state data received by the processor <NUM>, and the driver is driving cautiously from the driving pattern data. In another example, when the driver appears to be attentive and looking towards the road however, the driving pattern is not proper and is zig-zag.

In such a scenario, the weightage engine <NUM> may receive the data from the data receiving engine <NUM>. In situations of conflict in between the data from the driver monitoring module <NUM> and the exterior monitoring module <NUM>, the weightage engine <NUM> determines which data should be given more weightage. In an implementation, the external environment data is given more priority. For example, when the driver is appearing to be drowsy but the driving of the driver is perfectly fine, then the weightage engine <NUM> provides more value to the driving pattern data for generating the warning. Similarly, when the driver is determined to be attentive based on the driver state data, and the driving is determined to be inattentive based on the driving pattern data, then the driving pattern is given more weight and the warning issued is generally at high intensity.

Based on the events determined from the external environment data, the driver state data, and the driving pattern, the determination engine <NUM> may determine to issue a warning to the driver. Thereafter, the intensity of the warning may be ascertained by the rating engine <NUM>. The rating engine <NUM> receives the criticality of the event, the driver state data, and the driving pattern. The table below shows an example relation utilized by the relation engine <NUM> to determine the intensity of the warning by utilizing the parameter of closeness of the object to the vehicle <NUM>. It is to be noted that the below example is not to be considered limiting to the scope of the present invention and is provided for illustrative purposes.

For example, if the pedestrian is determined to be close, for instance less than <NUM> meters, the driver state is drowsy, and the driving pattern is rash, then the intensity of the warning issued may be extremely high. In such a scenario, the volume of the beep sound may be very high and may be issued along with a glaring light. In another scenario, when the pedestrian is at a distance of about <NUM> meters from the vehicle <NUM>, the driver is attentive and is cautiously driving the vehicle <NUM>, then the intensity of the warning is very low. In another scenario, there would be no warning issued, when the driver is cautiously driving and the pedestrian is at a distance of about <NUM> meters.

It would be noted that the warning to be issued is determined based on multitude of factors, for instance, driver state, criticality of the event, and driving pattern. Therefore, the event detection is accurate and relevant for a specific situation. Further, on many occasions, it becomes difficult to capture any one set of data, for instance, driver state data when the driver is wearing a cap or sunglasses that may hide a portion of the face and the eyes of the driver, or the driving pattern during foggy climate or heavy rain when outside visibility is low. In such situations, the determination engine <NUM> may utilize the remaining parameters captured to detect the event and issue a warning with a suitable intensity thereby enhancing robustness and reliability of such systems. There may be other rating methods applicable or parameters for ratings may be changed as per the requirements of a user.

Based on the intensity determined by the rating engine <NUM>, the warning generating module <NUM> may issue the warning to the driver.

There may also be provided a switch button on the ADAS <NUM> that may help the driver to acknowledge the warning. In another implementation, the processor <NUM> of the ADAS <NUM> may also be configured to determine whether there is any action taken by the driver in response to the generated warning. In an embodiment, the action may include, checking drive line change of the vehicle <NUM>. Drive lane change of the vehicle <NUM> may be a straight-line path change calculated simultaneously when the vehicle <NUM> is being driven. If the straight-line path change is so much so that the situation is averted, then the warning may be stopped. Also, in another embodiment, the action may also be to notice any change is steering wheel angle of the vehicle <NUM> by the driver. In a scenario, after the steering angle is changed after the warning is generated to avert the event such as collision with a vehicle or a pedestrian, then the warning may be stopped.

The above description does not provide specific details of manufacture or design of the various components. Those of skill in the art are familiar with such details, and unless departures from those techniques are set out, techniques, known, related art or later developed designs and materials should be employed. Those in the art are capable of choosing suitable manufacturing and design details.

Note that throughout the following discussion, numerous references may be made regarding servers, services, engines, modules, interfaces, portals, platforms, or other systems formed from computing devices. It should be appreciated that the use of such terms is deemed to represent one or more computing devices having at least one processor configured to or programmed to execute software instructions stored on a computer readable tangible, non-transitory medium or also referred to as a processor-readable medium. For example, a server can include one or more computers operating as a web server, database server, or other type of computer server in a manner to fulfill described roles, responsibilities, or functions. Within the context of this document, the disclosed devices or systems are also deemed to comprise computing devices having a processor and a non-transitory memory storing instructions executable by the processor that cause the device to control, manage, or otherwise manipulate the features of the devices or systems.

Some portions of the detailed description herein are presented in terms of algorithms and symbolic representations of operations on data bits performed by conventional computer components, including a central processing unit (CPU), memory storage devices for the CPU, and connected display devices. An algorithm is generally perceived as a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated.

It should be understood, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, as apparent from the discussion herein, it is appreciated that throughout the description, discussions utilizing terms such as "generating," or "monitoring," or "displaying," or "tracking," or "identifying," "or receiving," or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

Now referring to <FIG>, illustrates a method <NUM> for providing assistance to the driver of the vehicle <NUM>, in accordance to an embodiment of the present invention.

Furthermore, the method can be implemented in any suitable hardware, software, firmware, or combination thereof. However, for ease of explanation, in the embodiments described below, the method may be considered to be implemented in the above described system and/or the apparatus and/or any electronic device (not shown).

At step <NUM>, external environment data and driving pattern data are captured. In an example the exterior monitoring module <NUM> may capture the external environment data and the driving pattern data. In an implementation of the present invention, stereo camera 202A and the long-range narrow filed camera 202B have different capturing ranges in order to capture maximum possible area ahead of the vehicle <NUM>. At step <NUM>, the driver state data is captured. In one example, the driver monitoring module <NUM> of the ADAS <NUM> captures the driver state data.

Thereafter, at step <NUM>, an event is detected based on the external environment data, the driver state data, and the driving pattern data. In an example implementation, the processor <NUM>, may detect the event. The external environment data is processed and analyzed to identify any event that may pose a threat from the exterior of the vehicle <NUM>. The processor <NUM>, may identify the external environment for stationary and mobile objects, the road edges, lane markings, vehicles in front, pedestrians, street lights, speed signs, or crossroads etc..

The processor <NUM> may utilize pixel mapping techniques wherein various images are taken at subsequent time frames detect the event. Pixel maps are generated for these images and compared in real-time to capture any variation or aberrations. For example, for a vehicle moving in-front, various images or video may be taken by the exterior monitoring module <NUM>. Images may be taken at subsequent time intervals, whereas video may be taken continuously. However, if the pixel map determines size increasing in subsequent time frame images or video frames, then the vehicle in-front may be slowing down and is an event that the driver should attend to. Similarly, the processor <NUM> may use a pixel mapping technique to identify the facial expression and related driver emotions. Also, the determination engine <NUM> may utilize pre-stored templates to which pixel maps of the driver data may be utilized to determine driver state. Event determined may be based on either the external event, or driver state or a combination of both.

At step <NUM>, after determination of the event, a warning may be generated. IN one example, the warning generation module <NUM> may generate the warning. The warning generation module <NUM> may include an audio interface. Therefore, the warning generated may be an audio alarm. This warning may be varied based on the rating values ascertained for the events and the data accompanying the event. Also, the warning may be varied based on driving pattern determined from the external environment data.

The above described method enables varying of intensity of warning based on the event determined. The event determination is based on the external environment data and the driver state data. The variation of warning is based on the severity of the event based on the data.

Now referring to <FIG>, a method <NUM> for providing assistance to the driver of the vehicle <NUM>, in accordance to an embodiment of the present invention.

At step <NUM>, an event is detected. The event may be for instance, closeness of an object to the vehicle, number of objects on the road, terrain, speed of the vehicle <NUM>. In one example, the processor <NUM> may detect the event based on the external environment data received from the exterior monitoring module <NUM>. Thereafter, at step <NUM>, driver state data is received. The driver state data may be received from the driver monitoring module <NUM>. After receiving the driver state data, driving pattern data may be received at step <NUM>. In an example implementation, the processor <NUM> may receive the driving pattern data from the exterior monitoring module <NUM>.

Thereafter, at step <NUM>, the received driver state information and the driving pattern information is compared. The comparison may be made with an existing table of values containing pre-fed values. In one example, the intensity of the warning may be determined from a table (Table-I described earlier) that stores various combinations of the driver state data, the driving pattern data and the criticality of the event data and a corresponding intensity of warning for each combination. Based on the pre-fed values, the intensity of the warning is determined. At step <NUM>, a warning of the intensity level determined at previous step us generated. Further, the warning generation module <NUM>, then generates the warning of the desired intensity. At step <NUM>, the intensity of the warning may be varied. The warning may be varied based on the rating values ascertained for the events and the data accompanying the event. Also, the warning may be varied based on driving pattern determined from the external environment data.

In another embodiment of the invention, the vehicle may take an auto corrective action in case there is no response or acknowledgement from the driver of the vehicle. The auto corrective action may be based on a distance threshold that is if for a particular distance from the event, the driver takes no corrective action, the ADAS <NUM> may take an action automatically. The auto corrective action may be a braking action, lane change, sounding horn, etc. For instance, if the vehicle is approaching a pedestrian and the driver of the vehicle has not performed any action when the vehicle is <NUM> meters away from the pedestrian, then a braking action to stop the vehicle may be performed. To perform the braking action, the processor may send signals to the actuators coupled to the brakes and upon receiving the signal, the actuators may aid in braking.

Referring now to <FIG> illustrates an exemplary computer system <NUM> for implementing various embodiments is disclosed. The computer system <NUM> may comprise a central processing unit ("CPU" or "processor") <NUM>. The processing unit <NUM> may comprise at least one data processor for executing program components for executing user- or system-generated requests. The processing unit <NUM> may include specialized processing units such as integrated system (bus) controllers, memory management control units, floating point units, graphics processing units, digital signal processing units, etc. The processing unit <NUM> may be implemented using mainframe, distributed processor, multi-core, parallel, grid, or other architectures. Some embodiments may utilize embedded technologies like application-specific integrated circuits (ASICs), digital signal processors (DSPs), Field Programmable Gate Arrays (FPGAs), etc..

In some embodiments, the processing unit <NUM> may be disposed in communication with a communication network <NUM> via a network interface (not shown in figure). The network interface may communicate with the communication network <NUM>. The network interface may employ connection protocols including, without limitation, direct connect, Ethernet (e.g., twisted pair <NUM>/<NUM>/<NUM> Base T), transmission control protocol/internet protocol (TCP/IP), token ring, IEEE <NUM>. 11a/b/g/n/x, etc. The communication network <NUM> may include, without limitation, a direct interconnection, local area network (LAN), wide area network (WAN), wireless network (e.g., using Wireless Application Protocol) etc..

In some embodiments, the processing unit <NUM> may be disposed in communication with one or more databases <NUM> (e.g., a RAM, a ROM, etc.) via the network <NUM>. The network <NUM> may connect to the database <NUM> including, without limitation, memory drives, removable disc drives, etc., employing connection protocols such as serial advanced technology attachment (SATA), integrated drive electronics (IDE), IEEE-<NUM>, universal serial bus (USB), fiber channel, small computer systems interface (SCSI), etc. The memory drives may further include a drum, magnetic disc drive, magneto-optical drive, optical drive, redundant array of independent discs (RAID), solid-state memory devices, solid-state drives, etc. The database may include database from the exterior monitoring module <NUM>, the ranging module <NUM> and the driver monitoring module <NUM>.

The processing unit <NUM> may also be disposed in communication with a computer readable medium <NUM> (e.g. a compact disk, a USB drive, etc.) via the network <NUM>. The network <NUM> may connect the computer readable medium <NUM> including without limitation, floppy disks, flexible disks, hard disks, magnetic tape, or any other magnetic storage medium, CD-ROM, DVD, or any other optical medium, a RAM, a PROM, an EPROM, a FLASH-EPROM, or other memory chip or cartridge, or any other tangible medium. The computer readable medium <NUM> may be processed by the computer system <NUM> or in any other computer system. The computer readable medium <NUM> may include instructions like instruction to monitor driver state, instruction to monitor external environment, instruction to detect events, instruction to generate warnings, or instructions to vary warning intensity.

It will be appreciated that, for clarity purposes, the above description has described embodiments of the present invention with reference to different functional units and processors. However, it will be apparent that any suitable distribution of functionality between different functional units, processors or domains may be used.

The methods illustrated throughout the specification, may be implemented in a computer program product that may be executed on a computer. The computer program product may comprise a non-transitory computer-readable recording medium on which a control program is recorded, such as a disk, hard drive, or the like. Common forms of non-transitory computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, or any other magnetic storage medium, CD-ROM, DVD, or any other optical medium, a RAM, a PROM, an EPROM, a FLASH-EPROM, or other memory chip or cartridge, or any other tangible medium from which a computer can read and use.

Alternatively, the method may be implemented in transitory media, such as a transmittable carrier wave in which the control program is embodied as a data signal using transmission media, such as acoustic or light waves, such as those generated during radio wave and infrared data communications, and the like.

Alternatively, the method may be implemented using a combination of a processing unit <NUM>, a non-transitory Computer Readable Medium (CRM) <NUM>, Database <NUM> all connected to a network <NUM>. The computer readable medium may include instructions that may be fetched by the processing unit <NUM>. The instructions may include instruction to monitor driver state <NUM>, instruction to monitor external environment <NUM>, instruction to detect events <NUM>, instruction to generate warning <NUM>, and instruction to vary warning intensity <NUM>.

In one example, the processing unit <NUM> may execute the instruction to monitor driver state <NUM> to initiate monitoring of the driver state by an exterior monitoring module <NUM>. The exterior monitoring module <NUM> may monitor the driver's facial expressions, reactions, and features to determine if the driver is drowsy, or inattentive, as described earlier. Further, the processing unit <NUM> may also execute the instruction to monitor the external environment <NUM> to operate an exterior monitoring module, such as the exterior monitoring module <NUM> as described earlier to record the surrounding of the vehicle and provide the external environment data to the processing unit <NUM> for processing.

In an example implementation, the processing unit <NUM> may execute the instruction to detect events <NUM> to process the inputs received from the exterior monitoring module <NUM>, and the driver monitoring module <NUM> and detect whether an event has occurred. The event may be an approaching vehicle, a pedestrian on a road, or a cyclist close to the vehicle. After detecting the event, the processing unit <NUM> may execute the instruction to generate warning <NUM> to issue a warning to the driver of the vehicle. In one example, the warning may be issued by a warning generating module, such as the warning generating module <NUM> of the vehicle <NUM> as described earlier.

Thereafter, the processing unit <NUM> executes the instruction to vary warning intensity <NUM> to adjust intensity of waring issued to the driver. In one example, the intensity of the warning may be varied based on various factors such as criticality of the event for instance when the vehicle is very close to the pedestrian, or when the vehicle is close to another vehicle and about to collide. During such critical events, the intensity of the warning is increased. In an aspect of the present invention, the warning issued may be a voice based alert.

Further, the CRM <NUM> may include an instruction to prioritize data. In a scenario, the processing unit <NUM> may process the instruction to prioritize data to prioritize data from one of the exterior monitoring module <NUM> and the driver monitoring module <NUM> in a case when there is a mismatch between the data of the modules.

Claim 1:
An Advanced Driver Assistance System (ADAS) for a vehicle comprising:
a driver monitoring module to monitor a driver state of a driver in the vehicle;
an exterior monitoring module to monitor at least one of an external environment of the vehicle and a driving pattern of the driver, and adjust region of interest of view of the external environment, wherein the region of interest is
adjusted based on data associated with path of travel of the vehicle, wherein the data associated with path of travel is at least one of an indicator initiation, a steering wheel angle change, a pre-fed route, a Global Positioning System input, and wherein a shape of the region of interest is adjusted to adapt to
changes in external environment data received from the exterior monitoring module;
a processor, coupled to the driver monitoring module and the exterior monitoring module, to:
receive driver state data from the driver monitoring module and external environment data from the exterior monitoring module; and
detect an event based on at least one of the external environment data and the driver state data; and
a warning generating module, coupled to the processor, to generate a warning for the driver in response to the event, wherein the intensity of the
warning is based on at least one of the external environment data and the driver state data.