Patent Description:
Modern day vehicle technology is advancing towards higher degree of autonomous ability. Autonomous vehicles are defining a new era in modern day transportation. Autonomous vehicles either require minimum or no intervention from its driver. There are some autonomous vehicles that may only require an initial input from the driver, whereas some other designs of the autonomous vehicles are continuously under control of the driver. There are some autonomous vehicles that can be remotely controlled. For example, automatic parking in vehicles is an example of the autonomous vehicle in operation. According to the Society of Automotive Engineers (SAE) cars and vehicles in general are classified into <NUM> different classifications:.

Autonomous vehicles of Level <NUM> and onwards let the driver to take away their attention from the vehicle. However, autonomous vehicles face dynamic environment that is the environment keeps changing every time. The autonomous vehicles need to keep a track of lane markings, road edges, track road curves, varying surfaces that may be include flat surfaces, winding roads, hilly roads etc. Alongside, the autonomous vehicles also need to keep a check on objects that are both stationary or mobile like a tree or a human or an animal. While such autonomous vehicles relieve the driver of routine tasks, the vehicle driver may perceive his driving activity to be monotonous due to this relief, and may possibly be no longer attentive to the degree that is required for safe control of the motor vehicle in certain situations.

<CIT> discloses an autonomous driving system for a vehicle.

<CIT> discloses a method for enhancing driving performance by monitoring a driver's distraction from driving tasks due to the driver's attention to secondary tasks. <CIT> discloses systems and methods for reducing driver boredom for the driver of a vehicle.

Therefore, there is a need of an efficient system for maintaining driver attentiveness even while the driver is not participating in the controlling of the vehicle.

The foregoing summary, as well as the following detailed description of various embodiments, is better understood when read in conjunction with the drawings provided herein. For the purpose of illustration, there is shown in the drawings exemplary embodiments.

<FIG> shows a block diagram of an autonomous vehicle <NUM> (termed as vehicle <NUM> interchangeably within the description) and its various subsystems, in accordance with an embodiment of the invention. According to an embodiment of the invention, the autonomous vehicle <NUM> may be a fully or a semi-autonomous vehicle. The autonomous vehicle <NUM> includes multiple sub systems to control various important processes and functions. The autonomous vehicle <NUM> may include Engine control module <NUM>, Steering control module <NUM>, Brake control module <NUM>, Alerts control module <NUM>, Lights control module110, Handoff control module <NUM>, Processing module <NUM>, Sensor control module <NUM>, Navigation control module <NUM>, Lane control module <NUM>, Driver monitoring module <NUM>, and Drive mode monitoring module <NUM>.

Engine control module <NUM> controls various functions and processes of an engine of the vehicle <NUM>. Functions and processes to be controlled may be speed of rotation, engine condition, servicing requirements, load on engine, power of engine, etc..

Steering control module <NUM> may help in movement of the vehicle <NUM>. The steering control module <NUM> helps vehicle <NUM> to be driven and controlled in transverse and longitudinal direction. Steering module <NUM> may include actuators that may control the steering module <NUM> in autonomous mode.

Brake control module <NUM> of the autonomous vehicle <NUM> may help in braking function of the vehicle <NUM>. Brake control module <NUM> may control brakes of all four wheels using disc or horse-shoe brake parts. The brake control module <NUM> may also include actuators connected to brake parts in order to control braking while in autonomous drive mode.

Alerts control module <NUM> may control various alerts to be provided during various situations. The alerts may include ranging from servicing requirement of the vehicle <NUM> to lane change assist alerts during manual and autonomous mode.

Lights control module <NUM> may control various lighting functions of the vehicle <NUM>. The lighting functions may be for example, switching on lights while ambient light is below a threshold or changing low beam to high beam while road is empty and high beam is required due to night lighting conditions on road.

Handoff control module <NUM> takes care of drive handling control of the vehicle <NUM>. The handoff control module <NUM> may be responsible for switching control of the vehicle <NUM> to autonomous from manual or vice versa. The handoff control module takes over full control function of the vehicle <NUM> while switching to autonomous mode.

Processing module <NUM> provides computing power to the vehicle <NUM>. The processing module <NUM> helps the vehicle <NUM> in all the calculations required for autonomous, or semi-autonomous driving modes as well. It may also be useful in manual driving mode as well wherein the processing module <NUM> may process route calculations, fuel requirements, etc. In autonomous mode, the processing module <NUM> may take in data from various sensors and use the sensor data for efficient drive control during autonomous drive mode.

Sensor control module <NUM> collects data from the physical sensors provided all over the vehicle <NUM>. The sensors may be RADAR sensors, ultrasonic sensors, LiDAR sensor, proximity sensors, weather sensors, heat sensors, tire pressure sensors, etc. the sensor control module <NUM> in association with the processing module <NUM> may also calibrate the sensors regularly due to dynamic environment around the vehicle <NUM>.

Navigation control module <NUM> helps the autonomous vehicle <NUM> during active autonomous drive mode in navigation. In general, the navigation control module <NUM> may include route calculation, maps, road sign identification etc. for efficient navigation of the vehicle <NUM>.

Lane control module <NUM> may help the vehicle <NUM> to control lane changing and drive within a lane as marked on the road. Lane control module <NUM> may be take input data from image and RADAR sensors to identify lanes and help the vehicle to change lanes during an active autonomous drive mode.

Driver monitoring module <NUM> collects data about driver during an active autonomous drive mode, semiautonomous mode and manual mode. It collects data about driver like face expressions, eye gaze, emotions, facial identity etc. Data about driver may be collected using various cameras facing into a cabin of the vehicle <NUM>.

Drive monitoring module <NUM> collects data about drive of the vehicle <NUM>. The drive may be autonomous drive or manual drive. Data collected may be like drive behavior in various situations, various conditions, confidence level, stress induced mistakes etc. Drive monitoring module <NUM> may help in ascertaining drive behavior during the drive that may be kept for records and utilized for improving future drive interactions, and mistakes while driving the vehicle <NUM>.

It is to be noted, that the vehicle <NUM> may further include some more modules that may help in functioning of the vehicle <NUM> and some modules as mentioned above may be combined together to perform similar functions.

<FIG> is a line diagram of a dashboard <NUM> of a vehicle, in accordance with an embodiment of the invention. The dashboard <NUM> includes an instrument cluster <NUM>, an infotainment system <NUM>, Air conditioning vents <NUM>, steering space <NUM>, and a central console <NUM>.

The instrument cluster <NUM> may include indicators (not shown in figure) for speed, distance, rotations per minute, fuel indications, heating indications, etc. The infotainment system <NUM> provides various entertainment features like music system, navigation, various alerts, etc. to the driver of the vehicle. Air conditioning vents <NUM> may be provided in order to control climate of a cabin of the vehicle. As depicted, there may be multiple air conditioning vents provided within the dashboard <NUM>. The dashboard <NUM> may also include a steering space <NUM> wherein steering wheel of the vehicle is accommodated. Further, there may also be provided a central console <NUM> for driver's use like storage, bottle holders, etc..

<FIG> is a line diagram of the dashboard <NUM> of the vehicle including a projection module <NUM> placed near roof of the vehicle <NUM> in accordance with an embodiment of the invention. The projection module <NUM>, may be configured to display an interactive heads up display on the windscreen <NUM>. The projection module <NUM> is connected to a training module <NUM> (to be described later) and a user device (not shown in the figure) through wireless connection like PAN etc..

The projection module <NUM> may display the interactive training program on the windscreen <NUM> fully or partially. Some portion of the windscreen may be configured to provide certain notifications to the driver for example, milestone of distance achieved, economy of the car for last <NUM> hour of drive, fuel level, etc. The projected visual is interactive and may be interacted by simply touching the air space of the visual. The projection module <NUM> determines the various hand and finger gestures to determine input gestures of the driver. In another embodiment of the invention, the user device connected with the projection module <NUM> may be utilized as a remote control to provide inputs of the driver.

In another embodiment of the invention, the projection may also be provided on a side of the windscreen so as not to obstruct driver's field view. The projection may be configured in a holographic manner and in a see-through fashion that does not hinder the view of the surrounding around the vehicle for the driver.

For example, the projection may be in a small <NUM>-inch display on the left bottom of the windscreen. Such a size will enable the driver to interact as well as keep an eye on the road for emergency situations. The projections may be HD or non-HD projections to take care of non-obstruction of the surrounding view.

According to an embodiment of the invention, the projection module <NUM> may be configured to report an upcoming event to which the driver must pay special attention to or be ready to intervene in. For example, crowded place notification etc..

According to yet another embodiment of the invention, the projection module <NUM> may also be configured to remove the heads up display in a situation wherein intervention of the driver may be required at appropriate time. or, the heads up display may be reduced to a smaller part of the windscreen with small notifications to the driver.

In an embodiment of the invention, the projection module <NUM> may be a standalone device capable of gathering information about the exterior of the vehicle, the interior of the vehicle, projection of information on the windscreen <NUM> etc..

<FIG> is a block diagram of a system <NUM> for engaging a driver during an active autonomous mode, in accordance with an embodiment of the invention. The system <NUM> may include multiple modules like a drive mode monitoring module <NUM>; a driver monitoring module <NUM>; an environment condition module <NUM>; a vehicle information module <NUM>; an interactive module <NUM> which includes, a processing module <NUM>, a training module <NUM>; a memory <NUM>, and a display <NUM>. The system may also be a device with all modules, as mentioned above in the system.

In an implementation, some of the modules such as the drive mode monitoring module <NUM>, the driver monitoring module <NUM>, the environment condition module <NUM>, the vehicle information module <NUM>, the processing module <NUM>, the training module <NUM> may include routines, programs, objects, components, data structure and the like, which perform particular tasks or implement particular abstract data types. The modules may further include modules that supplement applications on the processing module <NUM>, for example, modules of an operating system. Further, the modules may be implemented in hardware, instructions executed by a processing unit, or by a combination thereof.

In another aspect of the present invention, the modules may be machine-readable instructions which, when executed by a processor/processing module, perform any of the described functionalities. The machine-readable instructions may be stored on an electronic memory device, hard disk, optical disk or other machine-readable storage medium or non-transitory medium. In an implementation, the machine-readable instructions may also be downloaded to the storage medium via a network connection.

Memory <NUM> may be without limitation, memory drives, removable disc drives, etc., employing connection protocols such as serial advanced technology attachment (SATA), integrated drive electronics (IDE), IEEE-<NUM>, universal serial bus (USB), fiber channel, small computer systems interface (SCSI), etc. The memory drives may further include a drum, magnetic disc drive, magneto-optical drive, optical drive, redundant array of independent discs (RAID), solid-state memory devices, solid-state drives, etc..

Drive mode monitoring module <NUM> determines, the active driving mode. Driving mode may be manual, semi-autonomous or autonomous. The drive mode monitoring module <NUM> may accept input from user to activate any of the three drive modes. The drive mode monitoring module <NUM> may be a touch button or a physical button or the like. A driver may provide input to the drive mode monitoring module <NUM> to initiation of the driving mode as required by the driver.

Driver monitoring module <NUM> is positioned to face the driver of a vehicle and monitors presence of the driver. The driver monitoring module <NUM> may be a combination of image sensors, occupancy sensors, thermal sensors etc. In operation, the driver monitoring module <NUM> may sense presence or absence of the driver. The driver's presence may be determined using techniques like motion detection, occupancy sensing, thermal vision etc. The driver monitoring module <NUM>, extracts attributes of the driver, once it is established that the driver is present, within the vehicle to identify the driver. Extracted attributes may include, but not limited to a facial scan, a retinal scan, thermal signatures, a fingerprint scan etc. In another example, the user's picture may be taken by the driver monitoring module <NUM>. In yet another example, the driver's driving behavior may be used as an attribute.

The environment condition module <NUM> acquires information from nearby surroundings of the vehicle. Various sensors, like RADAR, LiDAR, image sensors, ultrasonic sensors, infrared sensors, rain sensors, may be employed within the environment condition module <NUM>. Information like traffic, lane markings, pavement, road signs, position of the vehicle with respect to surroundings, other objects around the vehicle, upcoming bad road conditions, vehicle to server communication, vehicle to vehicle communication etc. may be collected by the environment condition module <NUM>.

The vehicle information module <NUM> acquires information regarding speed of the vehicle, or position of the vehicle, etc. Position of the vehicle may be sensed using a Global Positioning system (GPS) whereas speed may be ascertained by utilizing speed sensors affixed on the vehicle.

The interactive module <NUM> includes processing module <NUM> and training module <NUM>. The interactive module <NUM> is configured to initiate an interactive training program for the driver which is connected to the monitoring modules that gets activated the training in case alertness level of the driver falls below a first threshold value.

The processing module <NUM> gathers information from the drive mode monitoring module <NUM>, the driver monitoring module <NUM>, the environment condition module <NUM> and the vehicle information module <NUM> and processes the information for further usage. The processing module <NUM> processes information and determines whether to activate the training module <NUM> or not. The activation is determined on the information received from the driver monitoring module <NUM> and the drive mode monitoring module <NUM>. If the drive mode is autonomous drive mode, and the driver's attentive level is below a first threshold level the processing module <NUM> activates the training module <NUM>. The first threshold level may be preset or may be varied by the processing module <NUM> based on the environment information, and the vehicle information received from the environment condition module <NUM>, and the vehicle information module <NUM> respectively. For example, in case of an empty road, good road conditions, and average speed of the vehicle, the threshold level for attentiveness may be lowered however, in a situation wherein there is traffic, speed of the vehicle is over average speed of surrounding cars, road conditions not good, or weather conditions not favorable for driving the threshold level may then be lowered dynamically in anticipation of avoiding a mishap. Furthermore, in event of driver alertness level being more than a second threshold level, the processing module <NUM> may send deactivation instructions to the training program.

According to an embodiment of the invention, the processing module <NUM>, may further define driving behavior of the driver by using the driver monitoring module <NUM>, the environment condition module <NUM>, and the vehicle information module <NUM>. For example, the processing module <NUM> may utilize the driver emotions while driver captured by the driver monitoring module <NUM> corresponding to an event within the surrounding like sudden braking by a car in-front captured by the environment condition module <NUM>, or speed of the vehicle gathered by the vehicle information module <NUM>. The processing module <NUM> may also continuously monitor and compares the driving behavior of the driver with average or good driving behavior classified of drivers of other vehicles connected and stored in a central server accessible to the processing module <NUM> when the system <NUM> is connected to the central server (described later in detail). The processing module <NUM>, after comparison, may identify shortcomings of the driver and may prepare a training program for the driver.

According to another embodiment of the invention, the processing module <NUM>, may further define driving behavior of the driver by using the driver monitoring module <NUM>, the environment condition module <NUM>, and the vehicle information module <NUM>. For example, the processing module <NUM> may utilize the driver emotions while driver captured by the driver monitoring module <NUM> corresponding to an event within the surrounding like sudden braking by a car in-front captured by the environment condition module <NUM>, or speed of the vehicle gathered by the vehicle information module <NUM>. The processing module <NUM> may also continuously monitor and determine the driving behavior with respect to following of traffic rules that is also stored in a central server accessible to the processing module <NUM> when the system <NUM> is connected to the central server (described later in detail). The processing module <NUM>, after determination, identifies traffic rules violated by the driver and may prepare a training program for the driver for the identified violated traffic rules and other related traffic rules as well.

According to yet another embodiment of the invention, the processing module <NUM>, may further define driving behavior of the driver by using the driver monitoring module <NUM>, the environment condition module <NUM>, and the vehicle information module <NUM>. For example, the processing module <NUM> may utilize the driver emotions while driver captured by the driver monitoring module <NUM> corresponding to an event within the surrounding like sudden braking by a car in-front captured by the environment condition module <NUM>, or speed of the vehicle gathered by the vehicle information module <NUM>. The processing module <NUM> may also continuously monitor and determine the driving behavior with respect to following of traffic rules that is also stored in a central server accessible to the processing module <NUM> when the system <NUM> is connected to the central server (described later in detail). The processing module <NUM>, after determination, may identify various vehicle operation errors faced by the driver and may prepare a training program for the driver to enhance knowledge of the driver about various operational errors and issues that may have been faced by the driver in the past. Further, the interactive training may include small video tutorials to overcome various operational issues indicated by various signs to the driver.

The training module <NUM>, receives activation or deactivation instructions from the processing module <NUM>. The training module <NUM>, on activation may connect to the memory <NUM> for fetching an interactive training program designed specifically for the driver of the vehicle. The training program may specifically have a list of questions, with multiple choice answers, for the driver. In another embodiment, the training may be in graphical form or video tutorials. The list of questions may be based upon shortcomings of the driver as identified by the processing module <NUM>. The training program may be displayed to the driver through the display <NUM> that may be a screen of an infotainment system of the vehicle. The display <NUM> may be configured to receive inputs of the driver. The inputs may be through a touch, physical button, a remote control, voice input, or gesture recognition. The display <NUM> may include a circuitry (not shown in figure) like a printed circuit board (PCB) or an integrated circuit containing appropriate components for receiving and recognizing the driver inputs.

Simultaneously, with providing the training program to the driver, the processing module <NUM> continuously gather the data of driver monitoring modules to monitor the attentiveness level of the driver. In case the attentiveness level of the driver is still continuously below the first threshold level, as defined by the processing module <NUM>, the vehicle is brought to a halt. However, in case the attentive level is above the second threshold level that is predefined only, the training program may be terminated, or some entertainment program may be displayed.

The driver may receive points for the inputs made in the training program that may be further used to provide incentives in insurance etc. the driver may also be ranked in a leaderboard based on his score. If the driver is provided with a training program and he still does not provide any input a warning may be provided to halt the vehicle. In case, the driver still does not provide any input for a predetermined time, the vehicle may be brought to a halt.

<FIG> displays multiple displays displaying various kinds of questions that may be provided to the driver.

<FIG> is a flow chart of a method <NUM> for engaging the driver during an active autonomous drive mode of the vehicle. The order in which the method is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method or alternate methods. Additionally, individual blocks may be deleted from the method without departing from the scope of the invention defined by the appended claims. Furthermore, the method can be implemented in any suitable hardware, software, firmware, or combination thereof. However, for ease of explanation, in the embodiments described below, the method may be considered to be implemented in the above described system and/or the apparatus and/or any electronic device (not shown).

At step <NUM>, a drive mode transition is detected. The drive mode may be changed to autonomous from manual driving mode. The transition may be driver initiated or initiated by the processing module <NUM> itself. The automatic transition may be based on the traffic predictions, road condition predictions that the processing module <NUM> may request from and receive from the central server. After determination of the drive mode transition, at step <NUM>, driver monitoring is initiated. The driver monitoring may utilize cameras to take continuous footage of the driver and capture information from eyes of the driver. At step <NUM>, it is determined, whether the alertness of the driver is below a first threshold or not. If the alertness of the driver is not below the first threshold, then the method <NUM>, moves back to step <NUM>. However, if the alertness level of the driver falls below the first threshold then at step <NUM>, a training module <NUM> is activated that provides an interactive training program to the driver. Details of the training program have been described earlier. At step <NUM>, that may also be simultaneously while the training program is being displayed, the alertness level of the driver is checked by comparing it to a second threshold. If the alertness level is still below the second threshold level, then the method <NUM> goes back to step <NUM>. However, if the alertness level of the driver is equal or greater than the second threshold level, then at step <NUM> the training module <NUM> is deactivated.

It will be appreciated that, for clarity purposes, the above description has described embodiments of the present invention with reference to different functional units and processors. However, it will be apparent that any suitable distribution of functionality between different functional units, processors or domains may be used without detracting from the scope of the invention defined by the appended claims.

The methods illustrated throughout the specification, may be implemented in a computer program product that may be executed on a computer. The computer program product may comprise a non-transitory computer-readable recording medium on which a control program is recorded, such as a disk, hard drive, or the like. Common forms of non-transitory computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, or any other magnetic storage medium, CD-ROM, DVD, or any other optical medium, a RAM, a PROM, an EPROM, a FLASH-EPROM, or other memory chip or cartridge, or any other tangible medium from which a computer can read and use.

Claim 1:
A device for engaging a driver during an autonomous mode (<NUM>) comprising:
a drive mode monitoring module (<NUM>) configured to detect a transition from a manual drive mode to an autonomous drive mode;
a driver monitoring module (<NUM>) to monitor driver alertness during the autonomous drive mode; and
an interactive module (<NUM>), further comprising;
a processing module (<NUM>) configured to activate an interactive training program in case alertness of the driver falls below a first threshold value wherein the first threshold value for alertness is determined based on plurality of attributes;
a training module (<NUM>) configured to receive activation or deactivation instructions from the processing module (<NUM>); characterized in that the interactive training program enhances traffic rule knowledge of the driver, wherein the traffic rule knowledge of the driver is determined based on violations of traffic rules determined by the processing module (<NUM>)